Skill Seeker

What is Skill Seeker?

Why Use This?

Key Features

🌐 Documentation Scraping

📄 PDF Support (v1.2.0)

🐙 GitHub Repository Scraping (v2.0.0)

🔄 Unified Multi-Source Scraping (NEW - v2.0.0)

🤖 AI & Enhancement

⚡ Performance & Scale

✅ Quality Assurance

Quick Example

Option 1: Use from Claude Code (Recommended)

# One-time setup (5 minutes)./setup_mcp.sh# Then in Claude Code, just ask:"Generate a React skill from https://react.dev/""Scrape PDF at docs/manual.pdf and create skill"

Option 2: Use CLI Directly (HTML Docs)

# Install dependencies (2 pip packages)pip3 install requests beautifulsoup4# Generate a React skill in one commandpython3 cli/doc_scraper.py --config configs/react.json --enhance-local# Upload output/react.zip to Claude - Done!

Option 3: Use CLI for PDF Documentation

# Install PDF supportpip3 install PyMuPDF# Basic PDF extractionpython3 cli/pdf_scraper.py --pdf docs/manual.pdf --name myskill# Advanced featurespython3 cli/pdf_scraper.py --pdf docs/manual.pdf --name myskill \    --extract-tables\# Extract tables    --parallel\# Fast parallel processing    --workers 8# Use 8 CPU cores# Scanned PDFs (requires: pip install pytesseract Pillow)python3 cli/pdf_scraper.py --pdf docs/scanned.pdf --name myskill --ocr# Password-protected PDFspython3 cli/pdf_scraper.py --pdf docs/encrypted.pdf --name myskill --password mypassword# Upload output/myskill.zip to Claude - Done!

Option 4: Use CLI for GitHub Repository

# Install GitHub supportpip3 install PyGithub# Basic repository scrapingpython3 cli/github_scraper.py --repo facebook/react# Using a config filepython3 cli/github_scraper.py --config configs/react_github.json# With authentication (higher rate limits)export GITHUB_TOKEN=ghp_your_token_herepython3 cli/github_scraper.py --repo facebook/react# Customize what to includepython3 cli/github_scraper.py --repo django/django \    --include-issues\# Extract GitHub Issues    --max-issues 100\# Limit issue count    --include-changelog\# Extract CHANGELOG.md    --include-releases# Extract GitHub Releases# MCP usage in Claude Code"Scrape GitHub repository facebook/react"# Upload output/react.zip to Claude - Done!

Option 5: Unified Multi-Source Scraping (NEW - v2.0.0)

# Create unified config (mix documentation + GitHub)cat> configs/myframework_unified.json<< 'EOF'{  "name": "myframework",  "description": "Complete framework knowledge from docs + code",  "merge_mode": "rule-based",  "sources": [    {      "type": "documentation",      "base_url": "https://docs.myframework.com/",      "extract_api": true,      "max_pages": 200    },    {      "type": "github",      "repo": "owner/myframework",      "include_code": true,      "code_analysis_depth": "surface"    }  ]}EOF# Run unified scraperpython3 cli/unified_scraper.py --config configs/myframework_unified.json# Upload output/myframework.zip to Claude - Done!

####`move_local_x(delta: float)`⚠️**Conflict**: Documentation signature differs from implementation**Documentation says:**

**Code implementation:**```pythondef move_local_x(delta: float, snap: bool = False) -> None

✅ **Advantages:**- **Identifies documentation gaps** - Find outdated or missing docs automatically- **Catches code changes** - Know when APIs change without docs being updated- **Single source of truth** - One skill showing intent (docs) AND reality (code)- **Actionable insights** - Get suggestions for fixing each conflict- **Development aid** - See what's actually in the codebase vs what's documented**Example Unified Configs:**- `configs/react_unified.json` - React docs + GitHub repo- `configs/django_unified.json` - Django docs + GitHub repo- `configs/fastapi_unified.json` - FastAPI docs + GitHub repo**Full Guide:** See [docs/UNIFIED_SCRAPING.md](docs/UNIFIED_SCRAPING.md) for complete documentation.## How It Works```mermaidgraph LR    A[Documentation Website] --> B[Skill Seeker]    B --> C[Scraper]    B --> D[AI Enhancement]    B --> E[Packager]    C --> F[Organized References]    D --> F    F --> E    E --> G[Claude Skill .zip]    G --> H[Upload to Claude AI]

📋 Prerequisites

🚀 Quick Start

Method 1: MCP Server for Claude Code (Easiest)

# Clone repositorygit clone https://github.com/yusufkaraaslan/Skill_Seekers.gitcd Skill_Seekers# One-time setup (5 minutes)./setup_mcp.sh# Restart Claude Code, then just ask:

List all available configsGenerate config for Tailwind at https://tailwindcss.com/docsScrape docs using configs/react.jsonPackage skill at output/react/

Method 2: CLI (Traditional)

One-Time Setup: Create Virtual Environment

# Clone repositorygit clone https://github.com/yusufkaraaslan/Skill_Seekers.gitcd Skill_Seekers# Create virtual environmentpython3 -m venv venv# Activate virtual environmentsource venv/bin/activate# macOS/Linux# OR on Windows: venv\Scripts\activate# Install dependenciespip install requests beautifulsoup4 pytest# Save dependenciespip freeze> requirements.txt# Optional: Install anthropic for API-based enhancement (not needed for LOCAL enhancement)# pip install anthropic

source venv/bin/activate# Run this each time you start a new terminal session

Easiest: Use a Preset

# Make sure venv is activated (you should see (venv) in your prompt)source venv/bin/activate# Optional: Estimate pages first (fast, 1-2 minutes)python3 cli/estimate_pages.py configs/godot.json# Use Godot presetpython3 cli/doc_scraper.py --config configs/godot.json# Use React presetpython3 cli/doc_scraper.py --config configs/react.json# See all presetsls configs/

Interactive Mode

python3 cli/doc_scraper.py --interactive

Quick Mode

python3 cli/doc_scraper.py \  --name react \  --url https://react.dev/ \  --description"React framework for UIs"

📤 Uploading Skills to Claude

Option 1: Automatic Upload (API-based)

# Set your API key (one-time)export ANTHROPIC_API_KEY=sk-ant-...# Package and upload automaticallypython3 cli/package_skill.py output/react/ --upload# OR upload existing .zippython3 cli/upload_skill.py output/react.zip

Option 2: Manual Upload (No API Key)

# Package skillpython3 cli/package_skill.py output/react/# This will:# 1. Create output/react.zip# 2. Open the output/ folder automatically# 3. Show upload instructions# Then manually upload:# - Go to https://claude.ai/skills# - Click "Upload Skill"# - Select output/react.zip# - Done!

Option 3: Claude Code (MCP) - Smart & Automatic

In Claude Code, just ask:"Package and upload the React skill"# With API key set:# - Packages the skill# - Uploads to Claude automatically# - Done! ✅# Without API key:# - Packages the skill# - Shows where to find the .zip# - Provides manual upload instructions

📁 Simple Structure

doc-to-skill/├── cli/│   ├── doc_scraper.py      # Main scraping tool│   ├── package_skill.py    # Package to .zip│   ├── upload_skill.py     # Auto-upload (API)│   └── enhance_skill.py    # AI enhancement├── mcp/                    # MCP server for Claude Code│   └── server.py           # 9 MCP tools├── configs/                # Preset configurations│   ├── godot.json         # Godot Engine│   ├── react.json         # React│   ├── vue.json           # Vue.js│   ├── django.json        # Django│   └── fastapi.json       # FastAPI└── output/                 # All output (auto-created)    ├── godot_data/        # Scraped data    ├── godot/             # Built skill    └── godot.zip          # Packaged skill

✨ Features

1. Fast Page Estimation (NEW!)

python3 cli/estimate_pages.py configs/react.json# Output:📊 ESTIMATION RESULTS✅ Pages Discovered: 180📈 Estimated Total: 230⏱️  Time Elapsed: 1.2 minutes💡 Recommended max_pages: 280

2. Auto-Detect Existing Data

python3 cli/doc_scraper.py --config configs/godot.json# If data exists:✓ Found existing data: 245 pagesUse existing data? (y/n): y⏭️  Skipping scrape, using existing data

3. Knowledge Generation

4. Smart Categorization

5. Code Language Detection

# Automatically detects:-Python (def,import,from)-JavaScript (const,let,=>)-GDScript (func,var,extends)-C++ (#include, int main)-Andmore...

5. Skip Scraping

# Scrape oncepython3 cli/doc_scraper.py --config configs/react.json# Later, just rebuild (instant)python3 cli/doc_scraper.py --config configs/react.json --skip-scrape

6. Async Mode for Faster Scraping (2-3x Speed!)

# Enable async mode with 8 workers (recommended for large docs)python3 cli/doc_scraper.py --config configs/react.json --async --workers 8# Small docs (~100-500 pages)python3 cli/doc_scraper.py --config configs/mydocs.json --async --workers 4# Large docs (2000+ pages) with no rate limitingpython3 cli/doc_scraper.py --config configs/largedocs.json --async --workers 8 --no-rate-limit

7. AI-Powered SKILL.md Enhancement

# Option 1: During scraping (API-based, requires API key)pip3 install anthropicexport ANTHROPIC_API_KEY=sk-ant-...python3 cli/doc_scraper.py --config configs/react.json --enhance# Option 2: During scraping (LOCAL, no API key - uses Claude Code Max)python3 cli/doc_scraper.py --config configs/react.json --enhance-local# Option 3: After scraping (API-based, standalone)python3 cli/enhance_skill.py output/react/# Option 4: After scraping (LOCAL, no API key, standalone)python3 cli/enhance_skill_local.py output/react/

7. Large Documentation Support (10K-40K+ Pages)

# 1. Estimate first (discover page count)python3 cli/estimate_pages.py configs/godot.json# 2. Auto-split into focused sub-skillspython3 cli/split_config.py configs/godot.json --strategy router# Creates:# - godot-scripting.json (5K pages)# - godot-2d.json (8K pages)# - godot-3d.json (10K pages)# - godot-physics.json (6K pages)# - godot-shaders.json (11K pages)# 3. Scrape all in parallel (4-8 hours instead of 20-40!)forconfigin configs/godot-*.json;do  python3 cli/doc_scraper.py --config$config&donewait# 4. Generate intelligent router/hub skillpython3 cli/generate_router.py configs/godot-*.json# 5. Package all skillspython3 cli/package_multi.py output/godot*/# 6. Upload all .zip files to Claude# Users just ask questions naturally!# Router automatically directs to the right sub-skill!

{"name":"godot","max_pages":40000,"split_strategy":"router","split_config": {"target_pages_per_skill":5000,"create_router":true,"split_by_categories": ["scripting","2d","3d","physics"]  }}

8. Checkpoint/Resume for Long Scrapes

# Enable in config{"checkpoint": {"enabled": true,"interval": 1000  // Save every 1000 pages  }}# If scrape is interrupted (Ctrl+C or crash)python3 cli/doc_scraper.py --config configs/godot.json --resume# Resume from last checkpoint✅ Resuming from checkpoint (12,450 pages scraped)⏭️  Skipping 12,450 already-scraped pages🔄 Continuing from where we left off...# Start fresh (clear checkpoint)python3 cli/doc_scraper.py --config configs/godot.json --fresh

🎯 Complete Workflows

First Time (With Scraping + Enhancement)

# 1. Scrape + Build + AI Enhancement (LOCAL, no API key)python3 cli/doc_scraper.py --config configs/godot.json --enhance-local# 2. Wait for new terminal to close (enhancement completes)# Check the enhanced SKILL.md:cat output/godot/SKILL.md# 3. Packagepython3 cli/package_skill.py output/godot/# 4. Done! You have godot.zip with excellent SKILL.md

Using Existing Data (Fast!)

# 1. Use cached data + Local Enhancementpython3 cli/doc_scraper.py --config configs/godot.json --skip-scrapepython3 cli/enhance_skill_local.py output/godot/# 2. Packagepython3 cli/package_skill.py output/godot/# 3. Done!

Without Enhancement (Basic)

# 1. Scrape + Build (no enhancement)python3 cli/doc_scraper.py --config configs/godot.json# 2. Packagepython3 cli/package_skill.py output/godot/# 3. Done! (SKILL.md will be basic template)

📋 Available Presets

Using Presets

# Godotpython3 cli/doc_scraper.py --config configs/godot.json# Reactpython3 cli/doc_scraper.py --config configs/react.json# Vuepython3 cli/doc_scraper.py --config configs/vue.json# Djangopython3 cli/doc_scraper.py --config configs/django.json# FastAPIpython3 cli/doc_scraper.py --config configs/fastapi.json# Ansiblepython3 cli/doc_scraper.py --config configs/ansible-core.json

🎨 Creating Your Own Config

Option 1: Interactive

python3 cli/doc_scraper.py --interactive# Follow prompts, it will create the config for you

Option 2: Copy and Edit

# Copy a presetcp configs/react.json configs/myframework.json# Edit itnano configs/myframework.json# Use itpython3 cli/doc_scraper.py --config configs/myframework.json

Config Structure

{"name":"myframework","description":"When to use this skill","base_url":"https://docs.myframework.com/","selectors": {"main_content":"article","title":"h1","code_blocks":"pre code"  },"url_patterns": {"include": ["/docs","/guide"],"exclude": ["/blog","/about"]  },"categories": {"getting_started": ["intro","quickstart"],"api": ["api","reference"]  },"rate_limit":0.5,"max_pages":500}

📊 What Gets Created

output/├── godot_data/              # Scraped raw data│   ├── pages/              # JSON files (one per page)│   └── summary.json        # Overview│└── godot/                   # The skill    ├── SKILL.md            # Enhanced with real examples    ├── references/         # Categorized docs    │   ├── index.md    │   ├── getting_started.md    │   ├── scripting.md    │   └── ...    ├── scripts/            # Empty (add your own)    └── assets/             # Empty (add your own)

🎯 Command Line Options

# Interactive modepython3 cli/doc_scraper.py --interactive# Use config filepython3 cli/doc_scraper.py --config configs/godot.json# Quick modepython3 cli/doc_scraper.py --name react --url https://react.dev/# Skip scraping (use existing data)python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape# With descriptionpython3 cli/doc_scraper.py \  --name react \  --url https://react.dev/ \  --description"React framework for building UIs"

💡 Tips

1. Test Small First

{"max_pages":20// Test with just 20 pages}

2. Reuse Scraped Data

# Scrape oncepython3 cli/doc_scraper.py --config configs/react.json# Rebuild multiple times (instant)python3 cli/doc_scraper.py --config configs/react.json --skip-scrapepython3 cli/doc_scraper.py --config configs/react.json --skip-scrape

3. Finding Selectors

# Test in Pythonfrombs4importBeautifulSoupimportrequestsurl="https://docs.example.com/page"soup=BeautifulSoup(requests.get(url).content,'html.parser')# Try different selectorsprint(soup.select_one('article'))print(soup.select_one('main'))print(soup.select_one('div[role="main"]'))

4. Check Output Quality

# After building, check:cat output/godot/SKILL.md# Should have real examplescat output/godot/references/index.md# Categories

🐛 Troubleshooting

No Content Extracted?

Data Exists But Won't Use It?

# Force re-scraperm -rf output/myframework_data/python3 cli/doc_scraper.py --config configs/myframework.json

Categories Not Good?

Want to Update Docs?

# Delete old datarm -rf output/godot_data/# Re-scrapepython3 cli/doc_scraper.py --config configs/godot.json

📈 Performance

✅ Summary

📚 Documentation

Getting Started

Guides

Technical

🎮 Ready?

# Try Godotpython3 cli/doc_scraper.py --config configs/godot.json# Try Reactpython3 cli/doc_scraper.py --config configs/react.json# Or go interactivepython3 cli/doc_scraper.py --interactive

📝 License