gavindiaz/vibe-coding-cn
1
0
Fork 0
mirror of https://github.com/tradecatlabs/vibe-coding-cn.git synced 2026-08-16 20:38:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: move workflow/repo/documents/config under assets

Browse Source
This commit is contained in:
tukuaiai
2026-02-27 01:46:51 +08:00
parent 4207c8fbb8
commit dffbb976e7
488 changed files with 203 additions and 196 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/repo/Skill_Seekers-development/docs/CLAUDE.md
+400
View File
@@ -0,0 +1,400 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview
This is a Python-based documentation scraper that converts ANY documentation website into a Claude skill. It's a single-file tool (`doc_scraper.py`) that scrapes documentation, extracts code patterns, detects programming languages, and generates structured skill files ready for use with Claude.
## Dependencies
```bash
pip3 install requests beautifulsoup4
```
## Core Commands
### Run with a preset configuration
```bash
python3 cli/doc_scraper.py --config configs/godot.json
python3 cli/doc_scraper.py --config configs/react.json
python3 cli/doc_scraper.py --config configs/vue.json
python3 cli/doc_scraper.py --config configs/django.json
python3 cli/doc_scraper.py --config configs/fastapi.json
```
### Interactive mode (for new frameworks)
```bash
python3 cli/doc_scraper.py --interactive
```
### Quick mode (minimal config)
```bash
python3 cli/doc_scraper.py --name react --url https://react.dev/ --description "React framework"
```
### Skip scraping (use cached data)
```bash
python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape
```
### Resume interrupted scrapes
```bash
# If scrape was interrupted
python3 cli/doc_scraper.py --config configs/godot.json --resume
# Start fresh (clear checkpoint)
python3 cli/doc_scraper.py --config configs/godot.json --fresh
```
### Large documentation (10K-40K+ pages)
```bash
# 1. Estimate page count
python3 cli/estimate_pages.py configs/godot.json
# 2. Split into focused sub-skills
python3 cli/split_config.py configs/godot.json --strategy router
# 3. Generate router skill
python3 cli/generate_router.py configs/godot-*.json
# 4. Package multiple skills
python3 cli/package_multi.py output/godot*/
```
### AI-powered SKILL.md enhancement
```bash
# Option 1: During scraping (API-based, requires ANTHROPIC_API_KEY)
pip3 install anthropic
export ANTHROPIC_API_KEY=sk-ant-...
python3 cli/doc_scraper.py --config configs/react.json --enhance
# Option 2: During scraping (LOCAL, no API key - uses Claude Code Max)
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
# Option 3: Standalone after scraping (API-based)
python3 cli/enhance_skill.py output/react/
# Option 4: Standalone after scraping (LOCAL, no API key)
python3 cli/enhance_skill_local.py output/react/
```
The LOCAL enhancement option (`--enhance-local` or `enhance_skill_local.py`) opens a new terminal with Claude Code, which analyzes reference files and enhances SKILL.md automatically. This requires Claude Code Max plan but no API key.
### MCP Integration (Claude Code)
```bash
# One-time setup
./setup_mcp.sh
# Then in Claude Code, use natural language:
"List all available configs"
"Generate config for Tailwind at https://tailwindcss.com/docs"
"Split configs/godot.json using router strategy"
"Generate router for configs/godot-*.json"
"Package skill at output/react/"
```
9 MCP tools available: list_configs, generate_config, validate_config, estimate_pages, scrape_docs, package_skill, upload_skill, split_config, generate_router
### Test with limited pages (edit config first)
Set `"max_pages": 20` in the config file to test with fewer pages.
## Architecture
### Single-File Design
The entire tool is contained in `doc_scraper.py` (~737 lines). It follows a class-based architecture with a single `DocToSkillConverter` class that handles:
- **Web scraping**: BFS traversal with URL validation
- **Content extraction**: CSS selectors for title, content, code blocks
- **Language detection**: Heuristic-based detection from code samples (Python, JavaScript, GDScript, C++, etc.)
- **Pattern extraction**: Identifies common coding patterns from documentation
- **Categorization**: Smart categorization using URL structure, page titles, and content keywords with scoring
- **Skill generation**: Creates SKILL.md with real code examples and categorized reference files
### Data Flow
1. **Scrape Phase**:
- Input: Config JSON (name, base_url, selectors, url_patterns, categories, rate_limit, max_pages)
- Process: BFS traversal starting from base_url, respecting include/exclude patterns
- Output: `output/{name}_data/pages/*.json` + `summary.json`
2. **Build Phase**:
- Input: Scraped JSON data from `output/{name}_data/`
- Process: Load pages → Smart categorize → Extract patterns → Generate references
- Output: `output/{name}/SKILL.md` + `output/{name}/references/*.md`
### Directory Structure
```
Skill_Seekers/
├── cli/ # CLI tools
│ ├── doc_scraper.py # Main scraping & building tool
│ ├── enhance_skill.py # AI enhancement (API-based)
│ ├── enhance_skill_local.py # AI enhancement (LOCAL, no API)
│ ├── estimate_pages.py # Page count estimator
│ ├── split_config.py # Large docs splitter (NEW)
│ ├── generate_router.py # Router skill generator (NEW)
│ ├── package_skill.py # Single skill packager
│ └── package_multi.py # Multi-skill packager (NEW)
├── mcp/ # MCP server
│ ├── server.py # 9 MCP tools (includes upload)
│ └── README.md
├── configs/ # Preset configurations
│ ├── godot.json
│ ├── godot-large-example.json # Large docs example (NEW)
│ ├── react.json
│ └── ...
├── docs/ # Documentation
│ ├── CLAUDE.md # Technical architecture (this file)
│ ├── LARGE_DOCUMENTATION.md # Large docs guide (NEW)
│ ├── ENHANCEMENT.md
│ ├── MCP_SETUP.md
│ └── ...
└── output/ # Generated output (git-ignored)
├── {name}_data/ # Raw scraped data (cached)
│ ├── pages/ # Individual page JSONs
│ ├── summary.json # Scraping summary
│ └── checkpoint.json # Resume checkpoint (NEW)
└── {name}/ # Generated skill
├── SKILL.md # Main skill file with examples
├── SKILL.md.backup # Backup (if enhanced)
├── references/ # Categorized documentation
│ ├── index.md
│ ├── getting_started.md
│ ├── api.md
│ └── ...
├── scripts/ # Empty (for user scripts)
└── assets/ # Empty (for user assets)
```
### Configuration Format
Config files in `configs/*.json` contain:
- `name`: Skill identifier (e.g., "godot", "react")
- `description`: When to use this skill
- `base_url`: Starting URL for scraping
- `selectors`: CSS selectors for content extraction
- `main_content`: Main documentation content (e.g., "article", "div[role='main']")
- `title`: Page title selector
- `code_blocks`: Code sample selector (e.g., "pre code", "pre")
- `url_patterns`: URL filtering
- `include`: Only scrape URLs containing these patterns
- `exclude`: Skip URLs containing these patterns
- `categories`: Keyword-based categorization mapping
- `rate_limit`: Delay between requests (seconds)
- `max_pages`: Maximum pages to scrape
- `split_strategy`: (Optional) How to split large docs: "auto", "category", "router", "size"
- `split_config`: (Optional) Split configuration
- `target_pages_per_skill`: Pages per sub-skill (default: 5000)
- `create_router`: Create router/hub skill (default: true)
- `split_by_categories`: Category names to split by
- `checkpoint`: (Optional) Checkpoint/resume configuration
- `enabled`: Enable checkpointing (default: false)
- `interval`: Save every N pages (default: 1000)
### Key Features
**Auto-detect existing data**: Tool checks for `output/{name}_data/` and prompts to reuse, avoiding re-scraping.
**Language detection**: Detects code languages from:
1. CSS class attributes (`language-*`, `lang-*`)
2. Heuristics (keywords like `def`, `const`, `func`, etc.)
**Pattern extraction**: Looks for "Example:", "Pattern:", "Usage:" markers in content and extracts following code blocks (up to 5 per page).
**Smart categorization**:
- Scores pages against category keywords (3 points for URL match, 2 for title, 1 for content)
- Threshold of 2+ for categorization
- Auto-infers categories from URL segments if none provided
- Falls back to "other" category
**Enhanced SKILL.md**: Generated with:
- Real code examples from documentation (language-annotated)
- Quick reference patterns extracted from docs
- Common pattern section
- Category file listings
**AI-Powered Enhancement**: Two scripts to dramatically improve SKILL.md quality:
- `enhance_skill.py`: Uses Anthropic API (~$0.15-$0.30 per skill, requires API key)
- `enhance_skill_local.py`: Uses Claude Code Max (free, no API key needed)
- Transforms generic 75-line templates into comprehensive 500+ line guides
- Extracts best examples, explains key concepts, adds navigation guidance
- Success rate: 9/10 quality (based on steam-economy test)
**Large Documentation Support (NEW)**: Handle 10K-40K+ page documentation:
- `split_config.py`: Split large configs into multiple focused sub-skills
- `generate_router.py`: Create intelligent router/hub skills that direct queries
- `package_multi.py`: Package multiple skills at once
- 4 split strategies: auto, category, router, size
- Parallel scraping support for faster processing
- MCP integration for natural language usage
**Checkpoint/Resume (NEW)**: Never lose progress on long scrapes:
- Auto-saves every N pages (configurable, default: 1000)
- Resume with `--resume` flag
- Clear checkpoint with `--fresh` flag
- Saves on interruption (Ctrl+C)
## Key Code Locations
- **URL validation**: `is_valid_url()` doc_scraper.py:47-62
- **Content extraction**: `extract_content()` doc_scraper.py:64-131
- **Language detection**: `detect_language()` doc_scraper.py:133-163
- **Pattern extraction**: `extract_patterns()` doc_scraper.py:165-181
- **Smart categorization**: `smart_categorize()` doc_scraper.py:280-321
- **Category inference**: `infer_categories()` doc_scraper.py:323-349
- **Quick reference generation**: `generate_quick_reference()` doc_scraper.py:351-370
- **SKILL.md generation**: `create_enhanced_skill_md()` doc_scraper.py:424-540
- **Scraping loop**: `scrape_all()` doc_scraper.py:226-249
- **Main workflow**: `main()` doc_scraper.py:661-733
## Workflow Examples
### First time scraping (with scraping)
```bash
# 1. Scrape + Build
python3 cli/doc_scraper.py --config configs/godot.json
# Time: 20-40 minutes
# 2. Package
python3 cli/package_skill.py output/godot/
# Result: godot.zip
```
### Using cached data (fast iteration)
```bash
# 1. Use existing data
python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape
# Time: 1-3 minutes
# 2. Package
python3 cli/package_skill.py output/godot/
```
### Creating a new framework config
```bash
# Option 1: Interactive
python3 cli/doc_scraper.py --interactive
# Option 2: Copy and modify
cp configs/react.json configs/myframework.json
# Edit configs/myframework.json
python3 cli/doc_scraper.py --config configs/myframework.json
```
### Large documentation workflow (40K pages)
```bash
# 1. Estimate page count (fast, 1-2 minutes)
python3 cli/estimate_pages.py configs/godot.json
# 2. Split into focused sub-skills
python3 cli/split_config.py configs/godot.json --strategy router --target-pages 5000
# Creates: godot-scripting.json, godot-2d.json, godot-3d.json, etc.
# 3. Scrape all in parallel (4-8 hours instead of 20-40!)
for config in configs/godot-*.json; do
python3 cli/doc_scraper.py --config $config &
done
wait
# 4. Generate intelligent router skill
python3 cli/generate_router.py configs/godot-*.json
# 5. Package all skills
python3 cli/package_multi.py output/godot*/
# 6. Upload all .zip files to Claude
# Result: Router automatically directs queries to the right sub-skill!
```
**Time savings:** Parallel scraping reduces 20-40 hours to 4-8 hours
**See full guide:** [Large Documentation Guide](LARGE_DOCUMENTATION.md)
## Testing Selectors
To find the right CSS selectors for a documentation site:
```python
from bs4 import BeautifulSoup
import requests
url = "https://docs.example.com/page"
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
# Try different selectors
print(soup.select_one('article'))
print(soup.select_one('main'))
print(soup.select_one('div[role="main"]'))
```
## Running Tests
**IMPORTANT: You must install the package before running tests**
```bash
# 1. Install package in editable mode (one-time setup)
pip install -e .
# 2. Run all tests
pytest
# 3. Run specific test files
pytest tests/test_config_validation.py
pytest tests/test_github_scraper.py
# 4. Run with verbose output
pytest -v
# 5. Run with coverage report
pytest --cov=src/skill_seekers --cov-report=html
```
**Why install first?**
- Tests import from `skill_seekers.cli` which requires the package to be installed
- Modern Python packaging best practice (PEP 517/518)
- CI/CD automatically installs with `pip install -e .`
- conftest.py will show helpful error if package not installed
**Test Coverage:**
- 391+ tests passing
- 39% code coverage
- All core features tested
- CI/CD tests on Ubuntu + macOS with Python 3.10-3.12
## Troubleshooting
**No content extracted**: Check `main_content` selector. Common values: `article`, `main`, `div[role="main"]`, `div.content`
**Poor categorization**: Edit `categories` section in config with better keywords specific to the documentation structure
**Force re-scrape**: Delete cached data with `rm -rf output/{name}_data/`
**Rate limiting issues**: Increase `rate_limit` value in config (e.g., from 0.5 to 1.0 seconds)
## Output Quality Checks
After building, verify quality:
```bash
cat output/godot/SKILL.md # Should have real code examples
cat output/godot/references/index.md # Should show categories
ls output/godot/references/ # Should have category .md files
```
## llms.txt Support
Skill_Seekers automatically detects llms.txt files before HTML scraping:
### Detection Order
1. `{base_url}/llms-full.txt` (complete documentation)
2. `{base_url}/llms.txt` (standard version)
3. `{base_url}/llms-small.txt` (quick reference)
### Benefits
- ⚡ 10x faster (< 5 seconds vs 20-60 seconds)
- ✅ More reliable (maintained by docs authors)
- 🎯 Better quality (pre-formatted for LLMs)
- 🚫 No rate limiting needed
### Example Sites
- Hono: https://hono.dev/llms-full.txt
If no llms.txt is found, automatically falls back to HTML scraping.
assets/repo/Skill_Seekers-development/docs/ENHANCEMENT.md
+250
View File
@@ -0,0 +1,250 @@
# AI-Powered SKILL.md Enhancement
Two scripts are available to dramatically improve your SKILL.md file:
1. **`enhance_skill_local.py`** - Uses Claude Code Max (no API key, **recommended**)
2. **`enhance_skill.py`** - Uses Anthropic API (~$0.15-$0.30 per skill)
Both analyze reference documentation and extract the best examples and guidance.
## Why Use Enhancement?
**Problem:** The auto-generated SKILL.md is often too generic:
- Empty Quick Reference section
- No practical code examples
- Generic "When to Use" triggers
- Doesn't highlight key features
**Solution:** Let Claude read your reference docs and create a much better SKILL.md with:
- ✅ Best code examples extracted from documentation
- ✅ Practical quick reference with real patterns
- ✅ Domain-specific guidance
- ✅ Clear navigation tips
- ✅ Key concepts explained
## Quick Start (LOCAL - No API Key)
**Recommended for Claude Code Max users:**
```bash
# Option 1: Standalone enhancement
python3 cli/enhance_skill_local.py output/steam-inventory/
# Option 2: Integrated with scraper
python3 cli/doc_scraper.py --config configs/steam-inventory.json --enhance-local
```
**What happens:**
1. Opens new terminal window
2. Runs Claude Code with enhancement prompt
3. Claude analyzes reference files (~15-20K chars)
4. Generates enhanced SKILL.md (30-60 seconds)
5. Terminal auto-closes when done
**Requirements:**
- Claude Code Max plan (you're already using it!)
- macOS (auto-launch works) or manual terminal run on other OS
## API-Based Enhancement (Alternative)
**If you prefer API-based approach:**
### Installation
```bash
pip3 install anthropic
```
### Setup API Key
```bash
# Option 1: Environment variable (recommended)
export ANTHROPIC_API_KEY=sk-ant-...
# Option 2: Pass directly with --api-key
python3 cli/enhance_skill.py output/react/ --api-key sk-ant-...
```
### Usage
```bash
# Standalone enhancement
python3 cli/enhance_skill.py output/steam-inventory/
# Integrated with scraper
python3 cli/doc_scraper.py --config configs/steam-inventory.json --enhance
# Dry run (see what would be done)
python3 cli/enhance_skill.py output/react/ --dry-run
```
## What It Does
1. **Reads reference files** (api_reference.md, webapi.md, etc.)
2. **Sends to Claude** with instructions to:
- Extract 5-10 best code examples
- Create practical quick reference
- Write domain-specific "When to Use" triggers
- Add helpful navigation guidance
3. **Backs up original** SKILL.md to SKILL.md.backup
4. **Saves enhanced version** as new SKILL.md
## Example Enhancement
### Before (Auto-Generated)
```markdown
## Quick Reference
### Common Patterns
*Quick reference patterns will be added as you use the skill.*
```
### After (AI-Enhanced)
```markdown
## Quick Reference
### Common API Patterns
**Granting promotional items:**
```cpp
void CInventory::GrantPromoItems()
{
SteamItemDef_t newItems[2];
newItems[0] = 110;
newItems[1] = 111;
SteamInventory()->AddPromoItems( &s_GenerateRequestResult, newItems, 2 );
}
```
**Getting all items in player inventory:**
```cpp
SteamInventoryResult_t resultHandle;
bool success = SteamInventory()->GetAllItems( &resultHandle );
```
[... 8 more practical examples ...]
```
## Cost Estimate
- **Input**: ~50,000-100,000 tokens (reference docs)
- **Output**: ~4,000 tokens (enhanced SKILL.md)
- **Model**: claude-sonnet-4-20250514
- **Estimated cost**: $0.15-$0.30 per skill
## Troubleshooting
### "No API key provided"
```bash
export ANTHROPIC_API_KEY=sk-ant-...
# or
python3 cli/enhance_skill.py output/react/ --api-key sk-ant-...
```
### "No reference files found"
Make sure you've run the scraper first:
```bash
python3 cli/doc_scraper.py --config configs/react.json
```
### "anthropic package not installed"
```bash
pip3 install anthropic
```
### Don't like the result?
```bash
# Restore original
mv output/steam-inventory/SKILL.md.backup output/steam-inventory/SKILL.md
# Try again (it may generate different content)
python3 cli/enhance_skill.py output/steam-inventory/
```
## Tips
1. **Run after scraping completes** - Enhancement works best with complete reference docs
2. **Review the output** - AI is good but not perfect, check the generated SKILL.md
3. **Keep the backup** - Original is saved as SKILL.md.backup
4. **Re-run if needed** - Each run may produce slightly different results
5. **Works offline after first run** - Reference files are local
## Real-World Results
**Test Case: steam-economy skill**
- **Before:** 75 lines, generic template, empty Quick Reference
- **After:** 570 lines, 10 practical API examples, key concepts explained
- **Time:** 60 seconds
- **Quality Rating:** 9/10
The LOCAL enhancement successfully:
- Extracted best HTTP/JSON examples from 24 pages of documentation
- Explained domain concepts (Asset Classes, Context IDs, Transaction Lifecycle)
- Created navigation guidance for beginners through advanced users
- Added best practices for security, economy design, and API integration
## Limitations
**LOCAL Enhancement (`enhance_skill_local.py`):**
- Requires Claude Code Max plan
- macOS auto-launch only (manual on other OS)
- Opens new terminal window
- Takes ~60 seconds
**API Enhancement (`enhance_skill.py`):**
- Requires Anthropic API key (paid)
- Cost: ~$0.15-$0.30 per skill
- Limited to ~100K tokens of reference input
**Both:**
- May occasionally miss the best examples
- Can't understand context beyond the reference docs
- Doesn't modify reference files (only SKILL.md)
## Enhancement Options Comparison
| Aspect | Manual Edit | LOCAL Enhancement | API Enhancement |
|--------|-------------|-------------------|-----------------|
| Time | 15-30 minutes | 30-60 seconds | 30-60 seconds |
| Code examples | You pick | AI picks best | AI picks best |
| Quick reference | Write yourself | Auto-generated | Auto-generated |
| Domain guidance | Your knowledge | From docs | From docs |
| Consistency | Varies | Consistent | Consistent |
| Cost | Free (your time) | Free (Max plan) | ~$0.20 per skill |
| Setup | None | None | API key needed |
| Quality | High (if expert) | 9/10 | 9/10 |
| **Recommended?** | For experts only | ✅ **Yes** | If no Max plan |
## When to Use
**Use enhancement when:**
- You want high-quality SKILL.md quickly
- Working with large documentation (50+ pages)
- Creating skills for unfamiliar frameworks
- Need practical code examples extracted
- Want consistent quality across multiple skills
**Skip enhancement when:**
- Budget constrained (use manual editing)
- Very small documentation (<10 pages)
- You know the framework intimately
- Documentation has no code examples
## Advanced: Customization
To customize how Claude enhances the SKILL.md, edit `enhance_skill.py` and modify the `_build_enhancement_prompt()` method around line 130.
Example customization:
```python
prompt += """
ADDITIONAL REQUIREMENTS:
- Focus on security best practices
- Include performance tips
- Add troubleshooting section
"""
```
## See Also
- [README.md](../README.md) - Main documentation
- [CLAUDE.md](CLAUDE.md) - Architecture guide
- [doc_scraper.py](../doc_scraper.py) - Main scraping tool
assets/repo/Skill_Seekers-development/docs/LARGE_DOCUMENTATION.md
+431
View File
@@ -0,0 +1,431 @@
# Handling Large Documentation Sites (10K+ Pages)
Complete guide for scraping and managing large documentation sites with Skill Seeker.
---
## Table of Contents
- [When to Split Documentation](#when-to-split-documentation)
- [Split Strategies](#split-strategies)
- [Quick Start](#quick-start)
- [Detailed Workflows](#detailed-workflows)
- [Best Practices](#best-practices)
- [Examples](#examples)
- [Troubleshooting](#troubleshooting)
---
## When to Split Documentation
### Size Guidelines
| Documentation Size | Recommendation | Strategy |
|-------------------|----------------|----------|
| < 5,000 pages | **One skill** | No splitting needed |
| 5,000 - 10,000 pages | **Consider splitting** | Category-based |
| 10,000 - 30,000 pages | **Recommended** | Router + Categories |
| 30,000+ pages | **Strongly recommended** | Router + Categories |
### Why Split Large Documentation?
**Benefits:**
- ✅ Faster scraping (parallel execution)
- ✅ More focused skills (better Claude performance)
- ✅ Easier maintenance (update one topic at a time)
- ✅ Better user experience (precise answers)
- ✅ Avoids context window limits
**Trade-offs:**
- ⚠️ Multiple skills to manage
- ⚠️ Initial setup more complex
- ⚠️ Router adds one extra skill
---
## Split Strategies
### 1. **No Split** (One Big Skill)
**Best for:** Small to medium documentation (< 5K pages)
```bash
# Just use the config as-is
python3 cli/doc_scraper.py --config configs/react.json
```
**Pros:** Simple, one skill to maintain
**Cons:** Can be slow for large docs, may hit limits
---
### 2. **Category Split** (Multiple Focused Skills)
**Best for:** 5K-15K pages with clear topic divisions
```bash
# Auto-split by categories
python3 cli/split_config.py configs/godot.json --strategy category
# Creates:
# - godot-scripting.json
# - godot-2d.json
# - godot-3d.json
# - godot-physics.json
# - etc.
```
**Pros:** Focused skills, clear separation
**Cons:** User must know which skill to use
---
### 3. **Router + Categories** (Intelligent Hub) ⭐ RECOMMENDED
**Best for:** 10K+ pages, best user experience
```bash
# Create router + sub-skills
python3 cli/split_config.py configs/godot.json --strategy router
# Creates:
# - godot.json (router/hub)
# - godot-scripting.json
# - godot-2d.json
# - etc.
```
**Pros:** Best of both worlds, intelligent routing, natural UX
**Cons:** Slightly more complex setup
---
### 4. **Size-Based Split**
**Best for:** Docs without clear categories
```bash
# Split every 5000 pages
python3 cli/split_config.py configs/bigdocs.json --strategy size --target-pages 5000
# Creates:
# - bigdocs-part1.json
# - bigdocs-part2.json
# - bigdocs-part3.json
# - etc.
```
**Pros:** Simple, predictable
**Cons:** May split related topics
---
## Quick Start
### Option 1: Automatic (Recommended)
```bash
# 1. Create config
python3 cli/doc_scraper.py --interactive
# Name: godot
# URL: https://docs.godotengine.org
# ... fill in prompts ...
# 2. Estimate pages (discovers it's large)
python3 cli/estimate_pages.py configs/godot.json
# Output: ⚠️ 40,000 pages detected - splitting recommended
# 3. Auto-split with router
python3 cli/split_config.py configs/godot.json --strategy router
# 4. Scrape all sub-skills
for config in configs/godot-*.json; do
python3 cli/doc_scraper.py --config $config &
done
wait
# 5. Generate router
python3 cli/generate_router.py configs/godot-*.json
# 6. Package all
python3 cli/package_multi.py output/godot*/
# 7. Upload all .zip files to Claude
```
---
### Option 2: Manual Control
```bash
# 1. Define split in config
nano configs/godot.json
# Add:
{
"split_strategy": "router",
"split_config": {
"target_pages_per_skill": 5000,
"create_router": true,
"split_by_categories": ["scripting", "2d", "3d", "physics"]
}
}
# 2. Split
python3 cli/split_config.py configs/godot.json
# 3. Continue as above...
```
---
## Detailed Workflows
### Workflow 1: Router + Categories (40K Pages)
**Scenario:** Godot documentation (40,000 pages)
**Step 1: Estimate**
```bash
python3 cli/estimate_pages.py configs/godot.json
# Output:
# Estimated: 40,000 pages
# Recommended: Split into 8 skills (5K each)
```
**Step 2: Split Configuration**
```bash
python3 cli/split_config.py configs/godot.json --strategy router --target-pages 5000
# Creates:
# configs/godot.json (router)
# configs/godot-scripting.json (5K pages)
# configs/godot-2d.json (8K pages)
# configs/godot-3d.json (10K pages)
# configs/godot-physics.json (6K pages)
# configs/godot-shaders.json (11K pages)
```
**Step 3: Scrape Sub-Skills (Parallel)**
```bash
# Open multiple terminals or use background jobs
python3 cli/doc_scraper.py --config configs/godot-scripting.json &
python3 cli/doc_scraper.py --config configs/godot-2d.json &
python3 cli/doc_scraper.py --config configs/godot-3d.json &
python3 cli/doc_scraper.py --config configs/godot-physics.json &
python3 cli/doc_scraper.py --config configs/godot-shaders.json &
# Wait for all to complete
wait
# Time: 4-8 hours (parallel) vs 20-40 hours (sequential)
```
**Step 4: Generate Router**
```bash
python3 cli/generate_router.py configs/godot-*.json
# Creates:
# output/godot/SKILL.md (router skill)
```
**Step 5: Package All**
```bash
python3 cli/package_multi.py output/godot*/
# Creates:
# output/godot.zip (router)
# output/godot-scripting.zip
# output/godot-2d.zip
# output/godot-3d.zip
# output/godot-physics.zip
# output/godot-shaders.zip
```
**Step 6: Upload to Claude**
Upload all 6 .zip files to Claude. The router will intelligently direct queries to the right sub-skill!
---
### Workflow 2: Category Split Only (15K Pages)
**Scenario:** Vue.js documentation (15,000 pages)
**No router needed - just focused skills:**
```bash
# 1. Split
python3 cli/split_config.py configs/vue.json --strategy category
# 2. Scrape each
for config in configs/vue-*.json; do
python3 cli/doc_scraper.py --config $config
done
# 3. Package
python3 cli/package_multi.py output/vue*/
# 4. Upload all to Claude
```
**Result:** 5 focused Vue skills (components, reactivity, routing, etc.)
---
## Best Practices
### 1. **Choose Target Size Wisely**
```bash
# Small focused skills (3K-5K pages) - more skills, very focused
python3 cli/split_config.py config.json --target-pages 3000
# Medium skills (5K-8K pages) - balanced (RECOMMENDED)
python3 cli/split_config.py config.json --target-pages 5000
# Larger skills (8K-10K pages) - fewer skills, broader
python3 cli/split_config.py config.json --target-pages 8000
```
### 2. **Use Parallel Scraping**
```bash
# Serial (slow - 40 hours)
for config in configs/godot-*.json; do
python3 cli/doc_scraper.py --config $config
done
# Parallel (fast - 8 hours) ⭐
for config in configs/godot-*.json; do
python3 cli/doc_scraper.py --config $config &
done
wait
```
### 3. **Test Before Full Scrape**
```bash
# Test with limited pages first
nano configs/godot-2d.json
# Set: "max_pages": 50
python3 cli/doc_scraper.py --config configs/godot-2d.json
# If output looks good, increase to full
```
### 4. **Use Checkpoints for Long Scrapes**
```bash
# Enable checkpoints in config
{
"checkpoint": {
"enabled": true,
"interval": 1000
}
}
# If scrape fails, resume
python3 cli/doc_scraper.py --config config.json --resume
```
---
## Examples
### Example 1: AWS Documentation (Hypothetical 50K Pages)
```bash
# 1. Split by AWS services
python3 cli/split_config.py configs/aws.json --strategy router --target-pages 5000
# Creates ~10 skills:
# - aws (router)
# - aws-compute (EC2, Lambda)
# - aws-storage (S3, EBS)
# - aws-database (RDS, DynamoDB)
# - etc.
# 2. Scrape in parallel (overnight)
# 3. Upload all skills to Claude
# 4. User asks "How do I create an S3 bucket?"
# 5. Router activates aws-storage skill
# 6. Focused, accurate answer!
```
### Example 2: Microsoft Docs (100K+ Pages)
```bash
# Too large even with splitting - use selective categories
# Only scrape key topics
python3 cli/split_config.py configs/microsoft.json --strategy category
# Edit configs to include only:
# - microsoft-azure (Azure docs only)
# - microsoft-dotnet (.NET docs only)
# - microsoft-typescript (TS docs only)
# Skip less relevant sections
```
---
## Troubleshooting
### Issue: "Splitting creates too many skills"
**Solution:** Increase target size or combine categories
```bash
# Instead of 5K per skill, use 8K
python3 cli/split_config.py config.json --target-pages 8000
# Or manually combine categories in config
```
### Issue: "Router not routing correctly"
**Solution:** Check routing keywords in router SKILL.md
```bash
# Review router
cat output/godot/SKILL.md
# Update keywords if needed
nano output/godot/SKILL.md
```
### Issue: "Parallel scraping fails"
**Solution:** Reduce parallelism or check rate limits
```bash
# Scrape 2-3 at a time instead of all
python3 cli/doc_scraper.py --config config1.json &
python3 cli/doc_scraper.py --config config2.json &
wait
python3 cli/doc_scraper.py --config config3.json &
python3 cli/doc_scraper.py --config config4.json &
wait
```
---
## Summary
**For 40K+ Page Documentation:**
1. ✅ **Estimate first**: `python3 cli/estimate_pages.py config.json`
2. ✅ **Split with router**: `python3 cli/split_config.py config.json --strategy router`
3. ✅ **Scrape in parallel**: Multiple terminals or background jobs
4. ✅ **Generate router**: `python3 cli/generate_router.py configs/*-*.json`
5. ✅ **Package all**: `python3 cli/package_multi.py output/*/`
6. ✅ **Upload to Claude**: All .zip files
**Result:** Intelligent, fast, focused skills that work seamlessly together!
---
**Questions? See:**
- [Main README](../README.md)
- [MCP Setup Guide](MCP_SETUP.md)
- [Enhancement Guide](ENHANCEMENT.md)
assets/repo/Skill_Seekers-development/docs/LLMS_TXT_SUPPORT.md
+60
View File
@@ -0,0 +1,60 @@
# llms.txt Support
## Overview
Skill_Seekers now automatically detects and uses llms.txt files when available, providing 10x faster documentation ingestion.
## What is llms.txt?
The llms.txt convention is a growing standard where documentation sites provide pre-formatted, LLM-ready markdown files:
- `llms-full.txt` - Complete documentation
- `llms.txt` - Standard balanced version
- `llms-small.txt` - Quick reference
## How It Works
1. Before HTML scraping, Skill_Seekers checks for llms.txt files
2. If found, downloads and parses the markdown
3. If not found, falls back to HTML scraping
4. Zero config changes needed
## Configuration
### Automatic Detection (Recommended)
No config changes needed. Just run normally:
```bash
python3 cli/doc_scraper.py --config configs/hono.json
```
### Explicit URL
Optionally specify llms.txt URL:
```json
{
"name": "hono",
"llms_txt_url": "https://hono.dev/llms-full.txt",
"base_url": "https://hono.dev/docs"
}
```
## Performance Comparison
| Method | Time | Requests |
|--------|------|----------|
| HTML Scraping (20 pages) | 20-60s | 20+ |
| llms.txt | < 5s | 1 |
## Supported Sites
Sites known to provide llms.txt:
- Hono: https://hono.dev/llms-full.txt
- (More to be discovered)
## Fallback Behavior
If llms.txt download or parsing fails, automatically falls back to HTML scraping with no user intervention required.
assets/repo/Skill_Seekers-development/docs/MCP_SETUP.md
+618
View File
@@ -0,0 +1,618 @@
# Complete MCP Setup Guide for Claude Code
Step-by-step guide to set up the Skill Seeker MCP server with Claude Code.
**✅ Fully Tested and Working**: All 9 MCP tools verified in production use with Claude Code
- ✅ 34 comprehensive unit tests (100% pass rate)
- ✅ Integration tested via actual Claude Code MCP protocol
- ✅ All 9 tools working with natural language commands (includes upload support!)
---
## Table of Contents
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Configuration](#configuration)
- [Verification](#verification)
- [Usage Examples](#usage-examples)
- [Troubleshooting](#troubleshooting)
- [Advanced Configuration](#advanced-configuration)
---
## Prerequisites
### Required Software
1. **Python 3.10 or higher**
```bash
python3 --version
# Should show: Python 3.10.x or higher
```
2. **Claude Code installed**
- Download from [claude.ai/code](https://claude.ai/code)
- Requires Claude Pro or Claude Code Max subscription
3. **Skill Seeker repository cloned**
```bash
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git
cd Skill_Seekers
```
### System Requirements
- **Operating System**: macOS, Linux, or Windows (WSL)
- **Disk Space**: 100 MB for dependencies + space for generated skills
- **Network**: Internet connection for documentation scraping
---
## Installation
### Step 1: Install Python Dependencies
```bash
# Navigate to repository root
cd /path/to/Skill_Seekers
# Install MCP server dependencies
pip3 install -r skill_seeker_mcp/requirements.txt
# Install CLI tool dependencies (for scraping)
pip3 install requests beautifulsoup4
```
**Expected output:**
```
Successfully installed mcp-0.9.0 requests-2.31.0 beautifulsoup4-4.12.3
```
### Step 2: Verify Installation
```bash
# Test MCP server can start
timeout 3 python3 skill_seeker_mcp/server.py || echo "Server OK (timeout expected)"
# Should exit cleanly or timeout (both are normal)
```
**Optional: Run Tests**
```bash
# Install test dependencies
pip3 install pytest
# Run MCP server tests (25 tests)
python3 -m pytest tests/test_mcp_server.py -v
# Expected: 25 passed in ~0.3s
```
### Step 3: Note Your Repository Path
```bash
# Get absolute path
pwd
# Example output: /Users/username/Projects/Skill_Seekers
# or: /home/username/Skill_Seekers
```
**Save this path** - you'll need it for configuration!
---
## Configuration
### Step 1: Locate Claude Code MCP Configuration
Claude Code stores MCP configuration in:
- **macOS**: `~/.config/claude-code/mcp.json`
- **Linux**: `~/.config/claude-code/mcp.json`
- **Windows (WSL)**: `~/.config/claude-code/mcp.json`
### Step 2: Create/Edit Configuration File
```bash
# Create config directory if it doesn't exist
mkdir -p ~/.config/claude-code
# Edit the configuration
nano ~/.config/claude-code/mcp.json
```
### Step 3: Add Skill Seeker MCP Server
**Full Configuration Example:**
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": [
"/Users/username/Projects/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/Users/username/Projects/Skill_Seekers",
"env": {}
}
}
}
```
**IMPORTANT:** Replace `/Users/username/Projects/Skill_Seekers` with YOUR actual repository path!
**If you already have other MCP servers:**
```json
{
"mcpServers": {
"existing-server": {
"command": "node",
"args": ["/path/to/existing/server.js"]
},
"skill-seeker": {
"command": "python3",
"args": [
"/Users/username/Projects/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/Users/username/Projects/Skill_Seekers"
}
}
}
```
### Step 4: Save and Restart Claude Code
1. Save the file (`Ctrl+O` in nano, then `Enter`)
2. Exit editor (`Ctrl+X` in nano)
3. **Completely restart Claude Code** (quit and reopen)
---
## Verification
### Step 1: Check MCP Server Loaded
In Claude Code, type:
```
List all available MCP tools
```
You should see 9 Skill Seeker tools:
- `generate_config`
- `estimate_pages`
- `scrape_docs`
- `package_skill`
- `upload_skill`
- `list_configs`
- `validate_config`
- `split_config`
- `generate_router`
### Step 2: Test a Simple Command
```
List all available configs
```
**Expected response:**
```
Available configurations:
1. godot - Godot Engine documentation
2. react - React framework
3. vue - Vue.js framework
4. django - Django web framework
5. fastapi - FastAPI Python framework
6. kubernetes - Kubernetes documentation
7. steam-economy-complete - Steam Economy API
```
### Step 3: Test Config Generation
```
Generate a config for Tailwind CSS at https://tailwindcss.com/docs
```
**Expected response:**
```
✅ Config created: configs/tailwind.json
```
**Verify the file exists:**
```bash
ls configs/tailwind.json
```
---
## Usage Examples
### Example 1: Generate Skill from Scratch
```
User: Generate config for Svelte docs at https://svelte.dev/docs
Claude: ✅ Config created: configs/svelte.json
User: Estimate pages for configs/svelte.json
Claude: 📊 Estimated pages: 150
Recommended max_pages: 180
User: Scrape docs using configs/svelte.json
Claude: ✅ Skill created at output/svelte/
Run: python3 cli/package_skill.py output/svelte/
User: Package skill at output/svelte/
Claude: ✅ Created: output/svelte.zip
Ready to upload to Claude!
```
### Example 2: Use Existing Config
```
User: List all available configs
Claude: [Shows 7 configs]
User: Scrape docs using configs/react.json with max 50 pages
Claude: ✅ Skill created at output/react/
User: Package skill at output/react/
Claude: ✅ Created: output/react.zip
```
### Example 3: Validate Before Scraping
```
User: Validate configs/godot.json
Claude: ✅ Config is valid
- Base URL: https://docs.godotengine.org/en/stable/
- Max pages: 500
- Rate limit: 0.5s
- Categories: 3
User: Estimate pages for configs/godot.json
Claude: 📊 Estimated pages: 450
Current max_pages (500) is sufficient
User: Scrape docs using configs/godot.json
Claude: [Scraping starts...]
```
---
## Troubleshooting
### Issue: MCP Server Not Loading
**Symptoms:**
- Skill Seeker tools don't appear in Claude Code
- No response when asking about configs
**Solutions:**
1. **Check configuration path:**
```bash
cat ~/.config/claude-code/mcp.json
```
2. **Verify Python path:**
```bash
which python3
# Should show: /usr/bin/python3 or /usr/local/bin/python3
```
3. **Test server manually:**
```bash
cd /path/to/Skill_Seekers
python3 skill_seeker_mcp/server.py
# Should start without errors
```
4. **Check Claude Code logs:**
- macOS: `~/Library/Logs/Claude Code/`
- Linux: `~/.config/claude-code/logs/`
5. **Completely restart Claude Code:**
- Quit Claude Code (don't just close window)
- Reopen Claude Code
### Issue: "ModuleNotFoundError: No module named 'mcp'"
**Solution:**
```bash
pip3 install -r skill_seeker_mcp/requirements.txt
```
### Issue: "Permission denied" when running server
**Solution:**
```bash
chmod +x skill_seeker_mcp/server.py
```
### Issue: Tools appear but don't work
**Symptoms:**
- Tools listed but commands fail
- "Error executing tool" messages
**Solutions:**
1. **Check working directory in config:**
```json
{
"cwd": "/FULL/PATH/TO/Skill_Seekers"
}
```
2. **Verify CLI tools exist:**
```bash
ls cli/doc_scraper.py
ls cli/estimate_pages.py
ls cli/package_skill.py
```
3. **Test CLI tools directly:**
```bash
python3 cli/doc_scraper.py --help
```
### Issue: Slow or hanging operations
**Solutions:**
1. **Check rate limit in config:**
- Default: 0.5 seconds
- Increase if needed: 1.0 or 2.0 seconds
2. **Use smaller max_pages for testing:**
```
Generate config with max_pages=20 for testing
```
3. **Check network connection:**
```bash
curl -I https://docs.example.com
```
---
## Advanced Configuration
### Custom Environment Variables
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": ["/path/to/Skill_Seekers/skill_seeker_mcp/server.py"],
"cwd": "/path/to/Skill_Seekers",
"env": {
"ANTHROPIC_API_KEY": "sk-ant-...",
"PYTHONPATH": "/custom/path"
}
}
}
}
```
### Multiple Python Versions
If you have multiple Python versions:
```json
{
"mcpServers": {
"skill-seeker": {
"command": "/usr/local/bin/python3.11",
"args": ["/path/to/Skill_Seekers/skill_seeker_mcp/server.py"],
"cwd": "/path/to/Skill_Seekers"
}
}
}
```
### Virtual Environment
To use a Python virtual environment:
```bash
# Create venv
cd /path/to/Skill_Seekers
python3 -m venv venv
source venv/bin/activate
pip install -r skill_seeker_mcp/requirements.txt
pip install requests beautifulsoup4
which python3
# Copy this path for config
```
```json
{
"mcpServers": {
"skill-seeker": {
"command": "/path/to/Skill_Seekers/venv/bin/python3",
"args": ["/path/to/Skill_Seekers/skill_seeker_mcp/server.py"],
"cwd": "/path/to/Skill_Seekers"
}
}
}
```
### Debug Mode
Enable verbose logging:
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": [
"-u",
"/path/to/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/path/to/Skill_Seekers",
"env": {
"DEBUG": "1"
}
}
}
}
```
---
## Complete Example Configuration
**Minimal (recommended for most users):**
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": [
"/Users/username/Projects/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/Users/username/Projects/Skill_Seekers"
}
}
}
```
**With API enhancement:**
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": [
"/Users/username/Projects/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/Users/username/Projects/Skill_Seekers",
"env": {
"ANTHROPIC_API_KEY": "sk-ant-your-key-here"
}
}
}
}
```
---
## End-to-End Workflow
### Complete Setup and First Skill
```bash
# 1. Install
cd ~/Projects
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git
cd Skill_Seekers
pip3 install -r skill_seeker_mcp/requirements.txt
pip3 install requests beautifulsoup4
# 2. Configure
mkdir -p ~/.config/claude-code
cat > ~/.config/claude-code/mcp.json << 'EOF'
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": [
"/Users/username/Projects/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/Users/username/Projects/Skill_Seekers"
}
}
}
EOF
# (Replace paths with your actual paths!)
# 3. Restart Claude Code
# 4. Test in Claude Code:
```
**In Claude Code:**
```
User: List all available configs
User: Scrape docs using configs/react.json with max 50 pages
User: Package skill at output/react/
```
**Result:** `output/react.zip` ready to upload!
---
## Next Steps
After successful setup:
1. **Try preset configs:**
- React: `scrape docs using configs/react.json`
- Vue: `scrape docs using configs/vue.json`
- Django: `scrape docs using configs/django.json`
2. **Create custom configs:**
- `generate config for [framework] at [url]`
3. **Test with small limits first:**
- Use `max_pages` parameter: `scrape docs using configs/test.json with max 20 pages`
4. **Explore enhancement:**
- Use `--enhance-local` flag for AI-powered SKILL.md improvement
---
## Getting Help
- **Documentation**: See [mcp/README.md](../mcp/README.md)
- **Issues**: [GitHub Issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues)
- **Examples**: See [.github/ISSUES_TO_CREATE.md](../.github/ISSUES_TO_CREATE.md) for test cases
---
## Quick Reference Card
```
SETUP:
1. Install dependencies: pip3 install -r skill_seeker_mcp/requirements.txt
2. Configure: ~/.config/claude-code/mcp.json
3. Restart Claude Code
VERIFY:
- "List all available configs"
- "Validate configs/react.json"
GENERATE SKILL:
1. "Generate config for [name] at [url]"
2. "Estimate pages for configs/[name].json"
3. "Scrape docs using configs/[name].json"
4. "Package skill at output/[name]/"
TROUBLESHOOTING:
- Check: cat ~/.config/claude-code/mcp.json
- Test: python3 skill_seeker_mcp/server.py
- Logs: ~/Library/Logs/Claude Code/
```
---
Happy skill creating! 🚀
assets/repo/Skill_Seekers-development/docs/PDF_ADVANCED_FEATURES.md
+579
View File
@@ -0,0 +1,579 @@
# PDF Advanced Features Guide
Comprehensive guide to advanced PDF extraction features (Priority 2 & 3).
## Overview
Skill Seeker's PDF extractor now includes powerful advanced features for handling complex PDF scenarios:
**Priority 2 Features (More PDF Types):**
- ✅ OCR support for scanned PDFs
- ✅ Password-protected PDF support
- ✅ Complex table extraction
**Priority 3 Features (Performance Optimizations):**
- ✅ Parallel page processing
- ✅ Intelligent caching of expensive operations
## Table of Contents
1. [OCR Support for Scanned PDFs](#ocr-support)
2. [Password-Protected PDFs](#password-protected-pdfs)
3. [Table Extraction](#table-extraction)
4. [Parallel Processing](#parallel-processing)
5. [Caching](#caching)
6. [Combined Usage](#combined-usage)
7. [Performance Benchmarks](#performance-benchmarks)
---
## OCR Support
Extract text from scanned PDFs using Optical Character Recognition.
### Installation
```bash
# Install Tesseract OCR engine
# Ubuntu/Debian
sudo apt-get install tesseract-ocr
# macOS
brew install tesseract
# Install Python packages
pip install pytesseract Pillow
```
### Usage
```bash
# Basic OCR
python3 cli/pdf_extractor_poc.py scanned.pdf --ocr
# OCR with other options
python3 cli/pdf_extractor_poc.py scanned.pdf --ocr --verbose -o output.json
# Full skill creation with OCR
python3 cli/pdf_scraper.py --pdf scanned.pdf --name myskill --ocr
```
### How It Works
1. **Detection**: For each page, checks if text content is < 50 characters
2. **Fallback**: If low text detected and OCR enabled, renders page as image
3. **Processing**: Runs Tesseract OCR on the image
4. **Selection**: Uses OCR text if it's longer than extracted text
5. **Logging**: Shows OCR extraction results in verbose mode
### Example Output
```
📄 Extracting from: scanned.pdf
Pages: 50
OCR: ✅ enabled
Page 1: 245 chars, 0 code blocks, 2 headings, 0 images, 0 tables
OCR extracted 245 chars (was 12)
Page 2: 389 chars, 1 code blocks, 3 headings, 0 images, 0 tables
OCR extracted 389 chars (was 5)
```
### Limitations
- Requires Tesseract installed on system
- Slower than regular text extraction (~2-5 seconds per page)
- Quality depends on PDF scan quality
- Works best with high-resolution scans
### Best Practices
- Use `--parallel` with OCR for faster processing
- Combine with `--verbose` to see OCR progress
- Test on a few pages first before processing large documents
---
## Password-Protected PDFs
Handle encrypted PDFs with password protection.
### Usage
```bash
# Basic usage
python3 cli/pdf_extractor_poc.py encrypted.pdf --password mypassword
# With full workflow
python3 cli/pdf_scraper.py --pdf encrypted.pdf --name myskill --password mypassword
```
### How It Works
1. **Detection**: Checks if PDF is encrypted (`doc.is_encrypted`)
2. **Authentication**: Attempts to authenticate with provided password
3. **Validation**: Returns error if password is incorrect or missing
4. **Processing**: Continues normal extraction if authentication succeeds
### Example Output
```
📄 Extracting from: encrypted.pdf
🔐 PDF is encrypted, trying password...
✅ Password accepted
Pages: 100
Metadata: {...}
```
### Error Handling
```
# Missing password
❌ PDF is encrypted but no password provided
Use --password option to provide password
# Wrong password
❌ Invalid password
```
### Security Notes
- Password is passed via command line (visible in process list)
- For sensitive documents, consider environment variables
- Password is not stored in output JSON
---
## Table Extraction
Extract tables from PDFs and include them in skill references.
### Usage
```bash
# Extract tables
python3 cli/pdf_extractor_poc.py data.pdf --extract-tables
# With other options
python3 cli/pdf_extractor_poc.py data.pdf --extract-tables --verbose -o output.json
# Full skill creation with tables
python3 cli/pdf_scraper.py --pdf data.pdf --name myskill --extract-tables
```
### How It Works
1. **Detection**: Uses PyMuPDF's `find_tables()` method
2. **Extraction**: Extracts table data as 2D array (rows × columns)
3. **Metadata**: Captures bounding box, row count, column count
4. **Integration**: Tables included in page data and summary
### Example Output
```
📄 Extracting from: data.pdf
Table extraction: ✅ enabled
Page 5: 892 chars, 2 code blocks, 4 headings, 0 images, 2 tables
Found table 0: 10x4
Found table 1: 15x6
✅ Extraction complete:
Tables found: 25
```
### Table Data Structure
```json
{
"tables": [
{
"table_index": 0,
"rows": [
["Header 1", "Header 2", "Header 3"],
["Data 1", "Data 2", "Data 3"],
...
],
"bbox": [x0, y0, x1, y1],
"row_count": 10,
"col_count": 4
}
]
}
```
### Integration with Skills
Tables are automatically included in reference files when building skills:
```markdown
## Data Tables
### Table 1 (Page 5)
| Header 1 | Header 2 | Header 3 |
|----------|----------|----------|
| Data 1 | Data 2 | Data 3 |
```
### Limitations
- Quality depends on PDF table structure
- Works best with well-formatted tables
- Complex merged cells may not extract correctly
---
## Parallel Processing
Process pages in parallel for 3x faster extraction.
### Usage
```bash
# Enable parallel processing (auto-detects CPU count)
python3 cli/pdf_extractor_poc.py large.pdf --parallel
# Specify worker count
python3 cli/pdf_extractor_poc.py large.pdf --parallel --workers 8
# With full workflow
python3 cli/pdf_scraper.py --pdf large.pdf --name myskill --parallel --workers 8
```
### How It Works
1. **Worker Pool**: Creates ThreadPoolExecutor with N workers
2. **Distribution**: Distributes pages across workers
3. **Extraction**: Each worker processes pages independently
4. **Collection**: Results collected and merged
5. **Threshold**: Only activates for PDFs with > 5 pages
### Example Output
```
📄 Extracting from: large.pdf
Pages: 500
Parallel processing: ✅ enabled (8 workers)
🚀 Extracting 500 pages in parallel (8 workers)...
✅ Extraction complete:
Total characters: 1,250,000
Code blocks found: 450
```
### Performance
| Pages | Sequential | Parallel (4 workers) | Parallel (8 workers) |
|-------|-----------|---------------------|---------------------|
| 50 | 25s | 10s (2.5x) | 8s (3.1x) |
| 100 | 50s | 18s (2.8x) | 15s (3.3x) |
| 500 | 4m 10s | 1m 30s (2.8x) | 1m 15s (3.3x) |
| 1000 | 8m 20s | 3m 00s (2.8x) | 2m 30s (3.3x) |
### Best Practices
- Use `--workers` equal to CPU core count
- Combine with `--no-cache` for first-time processing
- Monitor system resources (RAM, CPU)
- Not recommended for very large images (memory intensive)
### Limitations
- Requires `concurrent.futures` (Python 3.2+)
- Uses more memory (N workers × page size)
- May not be beneficial for PDFs with many large images
---
## Caching
Intelligent caching of expensive operations for faster re-extraction.
### Usage
```bash
# Caching enabled by default
python3 cli/pdf_extractor_poc.py input.pdf
# Disable caching
python3 cli/pdf_extractor_poc.py input.pdf --no-cache
```
### How It Works
1. **Cache Key**: Each page cached by page number
2. **Check**: Before extraction, checks cache for page data
3. **Store**: After extraction, stores result in cache
4. **Reuse**: On re-run, returns cached data instantly
### What Gets Cached
- Page text and markdown
- Code block detection results
- Language detection results
- Quality scores
- Image extraction results
- Table extraction results
### Example Output
```
Page 1: Using cached data
Page 2: Using cached data
Page 3: 892 chars, 2 code blocks, 4 headings, 0 images, 0 tables
```
### Cache Lifetime
- In-memory only (cleared when process exits)
- Useful for:
- Testing extraction parameters
- Re-running with different filters
- Development and debugging
### When to Disable
- First-time extraction
- PDF file has changed
- Different extraction options
- Memory constraints
---
## Combined Usage
### Maximum Performance
Extract everything as fast as possible:
```bash
python3 cli/pdf_scraper.py \
--pdf docs/manual.pdf \
--name myskill \
--extract-images \
--extract-tables \
--parallel \
--workers 8 \
--min-quality 5.0
```
### Scanned PDF with Tables
```bash
python3 cli/pdf_scraper.py \
--pdf docs/scanned.pdf \
--name myskill \
--ocr \
--extract-tables \
--parallel \
--workers 4
```
### Encrypted PDF with All Features
```bash
python3 cli/pdf_scraper.py \
--pdf docs/encrypted.pdf \
--name myskill \
--password mypassword \
--extract-images \
--extract-tables \
--parallel \
--workers 8 \
--verbose
```
---
## Performance Benchmarks
### Test Setup
- **Hardware**: 8-core CPU, 16GB RAM
- **PDF**: 500-page technical manual
- **Content**: Mixed text, code, images, tables
### Results
| Configuration | Time | Speedup |
|--------------|------|---------|
| Basic (sequential) | 4m 10s | 1.0x (baseline) |
| + Caching | 2m 30s | 1.7x |
| + Parallel (4 workers) | 1m 30s | 2.8x |
| + Parallel (8 workers) | 1m 15s | 3.3x |
| + All optimizations | 1m 10s | 3.6x |
### Feature Overhead
| Feature | Time Impact | Memory Impact |
|---------|------------|---------------|
| OCR | +2-5s per page | +50MB per page |
| Table extraction | +0.5s per page | +10MB |
| Image extraction | +0.2s per image | Varies |
| Parallel (8 workers) | -66% total time | +8x memory |
| Caching | -50% on re-run | +100MB |
---
## Troubleshooting
### OCR Issues
**Problem**: `pytesseract not found`
```bash
# Install pytesseract
pip install pytesseract
# Install Tesseract engine
sudo apt-get install tesseract-ocr # Ubuntu
brew install tesseract # macOS
```
**Problem**: Low OCR quality
- Use higher DPI PDFs
- Check scan quality
- Try different Tesseract language packs
### Parallel Processing Issues
**Problem**: Out of memory errors
```bash
# Reduce worker count
python3 cli/pdf_extractor_poc.py large.pdf --parallel --workers 2
# Or disable parallel
python3 cli/pdf_extractor_poc.py large.pdf
```
**Problem**: Not faster than sequential
- Check CPU usage (may be I/O bound)
- Try with larger PDFs (> 50 pages)
- Monitor system resources
### Table Extraction Issues
**Problem**: Tables not detected
- Check if tables are actual tables (not images)
- Try different PDF viewers to verify structure
- Use `--verbose` to see detection attempts
**Problem**: Malformed table data
- Complex merged cells may not extract correctly
- Try extracting specific pages only
- Manual post-processing may be needed
---
## Best Practices
### For Large PDFs (500+ pages)
1. Use parallel processing:
```bash
python3 cli/pdf_scraper.py --pdf large.pdf --parallel --workers 8
```
2. Extract to JSON first, then build skill:
```bash
python3 cli/pdf_extractor_poc.py large.pdf -o extracted.json --parallel
python3 cli/pdf_scraper.py --from-json extracted.json --name myskill
```
3. Monitor system resources
### For Scanned PDFs
1. Use OCR with parallel processing:
```bash
python3 cli/pdf_scraper.py --pdf scanned.pdf --ocr --parallel --workers 4
```
2. Test on sample pages first
3. Use `--verbose` to monitor OCR performance
### For Encrypted PDFs
1. Use environment variable for password:
```bash
export PDF_PASSWORD="mypassword"
python3 cli/pdf_scraper.py --pdf encrypted.pdf --password "$PDF_PASSWORD"
```
2. Clear history after use to remove password
### For PDFs with Tables
1. Enable table extraction:
```bash
python3 cli/pdf_scraper.py --pdf data.pdf --extract-tables
```
2. Check table quality in output JSON
3. Manual review recommended for critical data
---
## API Reference
### PDFExtractor Class
```python
from pdf_extractor_poc import PDFExtractor
extractor = PDFExtractor(
pdf_path="input.pdf",
verbose=True,
chunk_size=10,
min_quality=5.0,
extract_images=True,
image_dir="images/",
min_image_size=100,
# Advanced features
use_ocr=True,
password="mypassword",
extract_tables=True,
parallel=True,
max_workers=8,
use_cache=True
)
result = extractor.extract_all()
```
### Configuration Options
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `pdf_path` | str | required | Path to PDF file |
| `verbose` | bool | False | Enable verbose logging |
| `chunk_size` | int | 10 | Pages per chunk |
| `min_quality` | float | 0.0 | Min code quality (0-10) |
| `extract_images` | bool | False | Extract images to files |
| `image_dir` | str | None | Image output directory |
| `min_image_size` | int | 100 | Min image dimension |
| `use_ocr` | bool | False | Enable OCR |
| `password` | str | None | PDF password |
| `extract_tables` | bool | False | Extract tables |
| `parallel` | bool | False | Parallel processing |
| `max_workers` | int | CPU count | Worker threads |
| `use_cache` | bool | True | Enable caching |
---
## Summary
✅ **6 Advanced Features** implemented (Priority 2 & 3)
✅ **3x Performance Boost** with parallel processing
✅ **OCR Support** for scanned PDFs
✅ **Password Protection** support
✅ **Table Extraction** from complex PDFs
✅ **Intelligent Caching** for faster re-runs
The PDF extractor now handles virtually any PDF scenario with maximum performance!
assets/repo/Skill_Seekers-development/docs/PDF_CHUNKING.md
+521
View File
@@ -0,0 +1,521 @@
# PDF Page Detection and Chunking (Task B1.3)
**Status:** ✅ Completed
**Date:** October 21, 2025
**Task:** B1.3 - Add PDF page detection and chunking
---
## Overview
Task B1.3 enhances the PDF extractor with intelligent page chunking and chapter detection capabilities. This allows large PDF documentation to be split into manageable, logical sections for better processing and organization.
## New Features
### ✅ 1. Page Chunking
Break large PDFs into smaller, manageable chunks:
- Configurable chunk size (default: 10 pages per chunk)
- Smart chunking that respects chapter boundaries
- Chunk metadata includes page ranges and chapter titles
**Usage:**
```bash
# Default chunking (10 pages per chunk)
python3 cli/pdf_extractor_poc.py input.pdf
# Custom chunk size (20 pages per chunk)
python3 cli/pdf_extractor_poc.py input.pdf --chunk-size 20
# Disable chunking (single chunk with all pages)
python3 cli/pdf_extractor_poc.py input.pdf --chunk-size 0
```
### ✅ 2. Chapter/Section Detection
Automatically detect chapter and section boundaries:
- Detects H1 and H2 headings as chapter markers
- Recognizes common chapter patterns:
- "Chapter 1", "Chapter 2", etc.
- "Part 1", "Part 2", etc.
- "Section 1", "Section 2", etc.
- Numbered sections like "1. Introduction"
**Chapter Detection Logic:**
1. Check for H1/H2 headings at page start
2. Pattern match against common chapter formats
3. Extract chapter title for metadata
### ✅ 3. Code Block Merging
Intelligently merge code blocks split across pages:
- Detects when code continues from one page to the next
- Checks language and detection method consistency
- Looks for continuation indicators:
- Doesn't end with `}`, `;`
- Ends with `,`, `\`
- Incomplete syntax structures
**Example:**
```
Page 5: def calculate_total(items):
total = 0
for item in items:
Page 6: total += item.price
return total
```
The merger will combine these into a single code block.
---
## Output Format
### Enhanced JSON Structure
The output now includes chunking and chapter information:
```json
{
"source_file": "manual.pdf",
"metadata": { ... },
"total_pages": 150,
"total_chunks": 15,
"chapters": [
{
"title": "Getting Started",
"start_page": 1,
"end_page": 12
},
{
"title": "API Reference",
"start_page": 13,
"end_page": 45
}
],
"chunks": [
{
"chunk_number": 1,
"start_page": 1,
"end_page": 12,
"chapter_title": "Getting Started",
"pages": [ ... ]
},
{
"chunk_number": 2,
"start_page": 13,
"end_page": 22,
"chapter_title": "API Reference",
"pages": [ ... ]
}
],
"pages": [ ... ]
}
```
### Chunk Object
Each chunk contains:
- `chunk_number` - Sequential chunk identifier (1-indexed)
- `start_page` - First page in chunk (1-indexed)
- `end_page` - Last page in chunk (1-indexed)
- `chapter_title` - Detected chapter title (if any)
- `pages` - Array of page objects in this chunk
### Merged Code Block Indicator
Code blocks merged from multiple pages include a flag:
```json
{
"code": "def example():\n ...",
"language": "python",
"detection_method": "font",
"merged_from_next_page": true
}
```
---
## Implementation Details
### Chapter Detection Algorithm
```python
def detect_chapter_start(self, page_data):
"""
Detect if a page starts a new chapter/section.
Returns (is_chapter_start, chapter_title) tuple.
"""
# Check H1/H2 headings first
headings = page_data.get('headings', [])
if headings:
first_heading = headings[0]
if first_heading['level'] in ['h1', 'h2']:
return True, first_heading['text']
# Pattern match against common chapter formats
text = page_data.get('text', '')
first_line = text.split('\n')[0] if text else ''
chapter_patterns = [
r'^Chapter\s+\d+',
r'^Part\s+\d+',
r'^Section\s+\d+',
r'^\d+\.\s+[A-Z]', # "1. Introduction"
]
for pattern in chapter_patterns:
if re.match(pattern, first_line, re.IGNORECASE):
return True, first_line.strip()
return False, None
```
### Code Block Merging Algorithm
```python
def merge_continued_code_blocks(self, pages):
"""
Merge code blocks that are split across pages.
"""
for i in range(len(pages) - 1):
current_page = pages[i]
next_page = pages[i + 1]
# Get last code block of current page
last_code = current_page['code_samples'][-1]
# Get first code block of next page
first_next_code = next_page['code_samples'][0]
# Check if they're likely the same code block
if (last_code['language'] == first_next_code['language'] and
last_code['detection_method'] == first_next_code['detection_method']):
# Check for continuation indicators
last_code_text = last_code['code'].rstrip()
continuation_indicators = [
not last_code_text.endswith('}'),
not last_code_text.endswith(';'),
last_code_text.endswith(','),
last_code_text.endswith('\\'),
]
if any(continuation_indicators):
# Merge the blocks
merged_code = last_code['code'] + '\n' + first_next_code['code']
last_code['code'] = merged_code
last_code['merged_from_next_page'] = True
# Remove duplicate from next page
next_page['code_samples'].pop(0)
return pages
```
### Chunking Algorithm
```python
def create_chunks(self, pages):
"""
Create chunks of pages respecting chapter boundaries.
"""
chunks = []
current_chunk = []
current_chapter = None
for i, page in enumerate(pages):
# Detect chapter start
is_chapter, chapter_title = self.detect_chapter_start(page)
if is_chapter and current_chunk:
# Save current chunk before starting new one
chunks.append({
'chunk_number': len(chunks) + 1,
'start_page': chunk_start + 1,
'end_page': i,
'pages': current_chunk,
'chapter_title': current_chapter
})
current_chunk = []
current_chapter = chapter_title
current_chunk.append(page)
# Check if chunk size reached (but don't break chapters)
if not is_chapter and len(current_chunk) >= self.chunk_size:
# Create chunk
chunks.append(...)
current_chunk = []
return chunks
```
---
## Usage Examples
### Basic Chunking
```bash
# Extract with default 10-page chunks
python3 cli/pdf_extractor_poc.py manual.pdf -o manual.json
# Output includes chunks
cat manual.json | jq '.total_chunks'
# Output: 15
```
### Large PDF Processing
```bash
# Large PDF with bigger chunks (50 pages each)
python3 cli/pdf_extractor_poc.py large_manual.pdf --chunk-size 50 -o output.json -v
# Verbose output shows:
# 📦 Creating chunks (chunk_size=50)...
# 🔗 Merging code blocks across pages...
# ✅ Extraction complete:
# Chunks created: 8
# Chapters detected: 12
```
### No Chunking (Single Output)
```bash
# Process all pages as single chunk
python3 cli/pdf_extractor_poc.py small_doc.pdf --chunk-size 0 -o output.json
```
---
## Performance
### Chunking Performance
- **Chapter Detection:** ~0.1ms per page (negligible overhead)
- **Code Merging:** ~0.5ms per page (fast)
- **Chunk Creation:** ~1ms total (very fast)
**Total overhead:** < 1% of extraction time
### Memory Benefits
Chunking large PDFs helps reduce memory usage:
- **Without chunking:** Entire PDF loaded in memory
- **With chunking:** Process chunk-by-chunk (future enhancement)
**Current implementation** still loads entire PDF but provides structured output for chunked processing downstream.
---
## Limitations
### Current Limitations
1. **Chapter Pattern Matching**
- Limited to common English chapter patterns
- May miss non-standard chapter formats
- No support for non-English chapters (e.g., "Capitulo", "Chapitre")
2. **Code Merging Heuristics**
- Based on simple continuation indicators
- May miss some edge cases
- No AST-based validation
3. **Chunk Size**
- Fixed page count (not by content size)
- Doesn't account for page content volume
- No auto-sizing based on memory constraints
### Known Issues
1. **Multi-Chapter Pages**
- If a single page has multiple chapters, only first is detected
- Workaround: Use smaller chunk sizes
2. **False Code Merges**
- Rare cases where separate code blocks are merged
- Detection: Look for `merged_from_next_page` flag
3. **Table of Contents**
- TOC pages may be detected as chapters
- Workaround: Manual filtering in downstream processing
---
## Comparison: Before vs After
| Feature | Before (B1.2) | After (B1.3) |
|---------|---------------|--------------|
| Page chunking | None | ✅ Configurable |
| Chapter detection | None | ✅ Auto-detect |
| Code spanning pages | Split | ✅ Merged |
| Large PDF handling | Difficult | ✅ Chunked |
| Memory efficiency | Poor | Better (structure for future) |
| Output organization | Flat | ✅ Hierarchical |
---
## Testing
### Test Chapter Detection
Create a test PDF with chapters:
1. Page 1: "Chapter 1: Introduction"
2. Page 15: "Chapter 2: Getting Started"
3. Page 30: "Chapter 3: API Reference"
```bash
python3 cli/pdf_extractor_poc.py test.pdf -o test.json --chunk-size 20 -v
# Verify chapters detected
cat test.json | jq '.chapters'
```
Expected output:
```json
[
{
"title": "Chapter 1: Introduction",
"start_page": 1,
"end_page": 14
},
{
"title": "Chapter 2: Getting Started",
"start_page": 15,
"end_page": 29
},
{
"title": "Chapter 3: API Reference",
"start_page": 30,
"end_page": 50
}
]
```
### Test Code Merging
Create a test PDF with code spanning pages:
- Page 1 ends with: `def example():\n total = 0`
- Page 2 starts with: ` for i in range(10):\n total += i`
```bash
python3 cli/pdf_extractor_poc.py test.pdf -o test.json -v
# Check for merged code blocks
cat test.json | jq '.pages[0].code_samples[] | select(.merged_from_next_page == true)'
```
---
## Next Steps (Future Tasks)
### Task B1.4: Improve Code Block Detection
- Add syntax validation
- Use AST parsing for better language detection
- Improve continuation detection accuracy
### Task B1.5: Add Image Extraction
- Extract images from chunks
- OCR for code in images
- Diagram detection and extraction
### Task B1.6: Full PDF Scraper CLI
- Build on chunking foundation
- Category detection for chunks
- Multi-PDF support
---
## Integration with Skill Seeker
The chunking feature lays groundwork for:
1. **Memory-efficient processing** - Process PDFs chunk-by-chunk
2. **Better categorization** - Chapters become categories
3. **Improved SKILL.md** - Organize by detected chapters
4. **Large PDF support** - Handle 500+ page manuals
**Example workflow:**
```bash
# Extract large manual with chapters
python3 cli/pdf_extractor_poc.py large_manual.pdf --chunk-size 25 -o manual.json
# Future: Build skill from chunks
python3 cli/build_skill_from_pdf.py manual.json
# Result: SKILL.md organized by detected chapters
```
---
## API Usage
### Using PDFExtractor with Chunking
```python
from cli.pdf_extractor_poc import PDFExtractor
# Create extractor with 15-page chunks
extractor = PDFExtractor('manual.pdf', verbose=True, chunk_size=15)
# Extract
result = extractor.extract_all()
# Access chunks
for chunk in result['chunks']:
print(f"Chunk {chunk['chunk_number']}: {chunk['chapter_title']}")
print(f" Pages: {chunk['start_page']}-{chunk['end_page']}")
print(f" Total pages: {len(chunk['pages'])}")
# Access chapters
for chapter in result['chapters']:
print(f"Chapter: {chapter['title']}")
print(f" Pages: {chapter['start_page']}-{chapter['end_page']}")
```
### Processing Chunks Independently
```python
# Extract
result = extractor.extract_all()
# Process each chunk separately
for chunk in result['chunks']:
# Get pages in chunk
pages = chunk['pages']
# Process pages
for page in pages:
# Extract code samples
for code in page['code_samples']:
print(f"Found {code['language']} code")
# Check if merged from next page
if code.get('merged_from_next_page'):
print(" (merged from next page)")
```
---
## Conclusion
Task B1.3 successfully implements:
- ✅ Page chunking with configurable size
- ✅ Automatic chapter/section detection
- ✅ Code block merging across pages
- ✅ Enhanced output format with structure
- ✅ Foundation for large PDF handling
**Performance:** Minimal overhead (<1%)
**Compatibility:** Backward compatible (pages array still included)
**Quality:** Significantly improved organization
**Ready for B1.4:** Code block detection improvements
---
**Task Completed:** October 21, 2025
**Next Task:** B1.4 - Improve code block extraction with syntax detection
assets/repo/Skill_Seekers-development/docs/PDF_EXTRACTOR_POC.md
+420
View File
@@ -0,0 +1,420 @@
# PDF Extractor - Proof of Concept (Task B1.2)
**Status:** ✅ Completed
**Date:** October 21, 2025
**Task:** B1.2 - Create simple PDF text extractor (proof of concept)
---
## Overview
This is a proof-of-concept PDF text and code extractor built for Skill Seeker. It demonstrates the feasibility of extracting documentation content from PDF files using PyMuPDF (fitz).
## Features
### ✅ Implemented
1. **Text Extraction** - Extract plain text from all PDF pages
2. **Markdown Conversion** - Convert PDF content to markdown format
3. **Code Block Detection** - Multiple detection methods:
- **Font-based:** Detects monospace fonts (Courier, Mono, Consolas, etc.)
- **Indent-based:** Detects consistently indented code blocks
- **Pattern-based:** Detects function/class definitions, imports
4. **Language Detection** - Auto-detect programming language from code content
5. **Heading Extraction** - Extract document structure from markdown
6. **Image Counting** - Track diagrams and screenshots
7. **JSON Output** - Compatible format with existing doc_scraper.py
### 🎯 Detection Methods
#### Font-Based Detection
Analyzes font properties to find monospace fonts typically used for code:
- Courier, Courier New
- Monaco, Menlo
- Consolas
- DejaVu Sans Mono
#### Indentation-Based Detection
Identifies code blocks by consistent indentation patterns:
- 4 spaces or tabs
- Minimum 2 consecutive lines
- Minimum 20 characters
#### Pattern-Based Detection
Uses regex to find common code structures:
- Function definitions (Python, JS, Go, etc.)
- Class definitions
- Import/require statements
### 🔍 Language Detection
Supports detection of 19 programming languages:
- Python, JavaScript, Java, C, C++, C#
- Go, Rust, PHP, Ruby, Swift, Kotlin
- Shell, SQL, HTML, CSS
- JSON, YAML, XML
---
## Installation
### Prerequisites
```bash
pip install PyMuPDF
```
### Verify Installation
```bash
python3 -c "import fitz; print(fitz.__doc__)"
```
---
## Usage
### Basic Usage
```bash
# Extract from PDF (print to stdout)
python3 cli/pdf_extractor_poc.py input.pdf
# Save to JSON file
python3 cli/pdf_extractor_poc.py input.pdf --output result.json
# Verbose mode (shows progress)
python3 cli/pdf_extractor_poc.py input.pdf --verbose
# Pretty-printed JSON
python3 cli/pdf_extractor_poc.py input.pdf --pretty
```
### Examples
```bash
# Extract Python documentation
python3 cli/pdf_extractor_poc.py docs/python_guide.pdf -o python_extracted.json -v
# Extract with verbose and pretty output
python3 cli/pdf_extractor_poc.py manual.pdf -o manual.json -v --pretty
# Quick test (print to screen)
python3 cli/pdf_extractor_poc.py sample.pdf --pretty
```
---
## Output Format
### JSON Structure
```json
{
"source_file": "input.pdf",
"metadata": {
"title": "Documentation Title",
"author": "Author Name",
"subject": "Subject",
"creator": "PDF Creator",
"producer": "PDF Producer"
},
"total_pages": 50,
"total_chars": 125000,
"total_code_blocks": 87,
"total_headings": 45,
"total_images": 12,
"languages_detected": {
"python": 52,
"javascript": 20,
"sql": 10,
"shell": 5
},
"pages": [
{
"page_number": 1,
"text": "Plain text content...",
"markdown": "# Heading\nContent...",
"headings": [
{
"level": "h1",
"text": "Getting Started"
}
],
"code_samples": [
{
"code": "def hello():\n print('Hello')",
"language": "python",
"detection_method": "font",
"font": "Courier-New"
}
],
"images_count": 2,
"char_count": 2500,
"code_blocks_count": 3
}
]
}
```
### Page Object
Each page contains:
- `page_number` - 1-indexed page number
- `text` - Plain text content
- `markdown` - Markdown-formatted content
- `headings` - Array of heading objects
- `code_samples` - Array of detected code blocks
- `images_count` - Number of images on page
- `char_count` - Character count
- `code_blocks_count` - Number of code blocks found
### Code Sample Object
Each code sample includes:
- `code` - The actual code text
- `language` - Detected language (or 'unknown')
- `detection_method` - How it was found ('font', 'indent', or 'pattern')
- `font` - Font name (if detected by font method)
- `pattern_type` - Type of pattern (if detected by pattern method)
---
## Technical Details
### Detection Accuracy
**Font-based detection:** ⭐⭐⭐⭐⭐ (Best)
- Highly accurate for well-formatted PDFs
- Relies on proper font usage in source document
- Works with: Technical docs, programming books, API references
**Indent-based detection:** ⭐⭐⭐⭐ (Good)
- Good for structured code blocks
- May capture non-code indented content
- Works with: Tutorials, guides, examples
**Pattern-based detection:** ⭐⭐⭐ (Fair)
- Captures specific code constructs
- May miss complex or unusual code
- Works with: Code snippets, function examples
### Language Detection Accuracy
- **High confidence:** Python, JavaScript, Java, Go, SQL
- **Medium confidence:** C++, Rust, PHP, Ruby, Swift
- **Basic detection:** Shell, JSON, YAML, XML
Detection based on keyword patterns, not AST parsing.
### Performance
Tested on various PDF sizes:
- Small (1-10 pages): < 1 second
- Medium (10-100 pages): 1-5 seconds
- Large (100-500 pages): 5-30 seconds
- Very Large (500+ pages): 30+ seconds
Memory usage: ~50-200 MB depending on PDF size and image content.
---
## Limitations
### Current Limitations
1. **No OCR** - Cannot extract text from scanned/image PDFs
2. **No Table Extraction** - Tables are treated as plain text
3. **No Image Extraction** - Only counts images, doesn't extract them
4. **Simple Deduplication** - May miss some duplicate code blocks
5. **No Multi-column Support** - May jumble multi-column layouts
### Known Issues
1. **Code Split Across Pages** - Code blocks spanning pages may be split
2. **Complex Layouts** - May struggle with complex PDF layouts
3. **Non-standard Fonts** - May miss code in non-standard monospace fonts
4. **Unicode Issues** - Some special characters may not preserve correctly
---
## Comparison with Web Scraper
| Feature | Web Scraper | PDF Extractor POC |
|---------|-------------|-------------------|
| Content source | HTML websites | PDF files |
| Code detection | CSS selectors | Font/indent/pattern |
| Language detection | CSS classes + heuristics | Pattern matching |
| Structure | Excellent | Good |
| Links | Full support | Not supported |
| Images | Referenced | Counted only |
| Categories | Auto-categorized | Not implemented |
| Output format | JSON | JSON (compatible) |
---
## Next Steps (Tasks B1.3-B1.8)
### B1.3: Add PDF Page Detection and Chunking
- Split large PDFs into manageable chunks
- Handle page-spanning code blocks
- Add chapter/section detection
### B1.4: Extract Code Blocks from PDFs
- Improve code block detection accuracy
- Add syntax validation
- Better language detection (use tree-sitter?)
### B1.5: Add PDF Image Extraction
- Extract diagrams as separate files
- Extract screenshots
- OCR support for code in images
### B1.6: Create `pdf_scraper.py` CLI Tool
- Full-featured CLI like `doc_scraper.py`
- Config file support
- Category detection
- Multi-PDF support
### B1.7: Add MCP Tool `scrape_pdf`
- Integrate with MCP server
- Add to existing 9 MCP tools
- Test with Claude Code
### B1.8: Create PDF Config Format
- Define JSON config for PDF sources
- Similar to web scraper configs
- Support multiple PDFs per skill
---
## Testing
### Manual Testing
1. **Create test PDF** (or use existing PDF documentation)
2. **Run extractor:**
```bash
python3 cli/pdf_extractor_poc.py test.pdf -o test_result.json -v --pretty
```
3. **Verify output:**
- Check `total_code_blocks` > 0
- Verify `languages_detected` includes expected languages
- Inspect `code_samples` for accuracy
### Test with Real Documentation
Recommended test PDFs:
- Python documentation (python.org)
- Django documentation
- PostgreSQL manual
- Any programming language reference
### Expected Results
Good PDF (well-formatted with monospace code):
- Detection rate: 80-95%
- Language accuracy: 85-95%
- False positives: < 5%
Poor PDF (scanned or badly formatted):
- Detection rate: 20-50%
- Language accuracy: 60-80%
- False positives: 10-30%
---
## Code Examples
### Using PDFExtractor Class Directly
```python
from cli.pdf_extractor_poc import PDFExtractor
# Create extractor
extractor = PDFExtractor('docs/manual.pdf', verbose=True)
# Extract all pages
result = extractor.extract_all()
# Access data
print(f"Total pages: {result['total_pages']}")
print(f"Code blocks: {result['total_code_blocks']}")
print(f"Languages: {result['languages_detected']}")
# Iterate pages
for page in result['pages']:
print(f"\nPage {page['page_number']}:")
print(f" Code blocks: {page['code_blocks_count']}")
for code in page['code_samples']:
print(f" - {code['language']}: {len(code['code'])} chars")
```
### Custom Language Detection
```python
from cli.pdf_extractor_poc import PDFExtractor
extractor = PDFExtractor('input.pdf')
# Override language detection
def custom_detect(code):
if 'SELECT' in code.upper():
return 'sql'
return extractor.detect_language_from_code(code)
# Use in extraction
# (requires modifying the class to support custom detection)
```
---
## Contributing
### Adding New Languages
To add language detection for a new language, edit `detect_language_from_code()`:
```python
patterns = {
# ... existing languages ...
'newlang': [r'pattern1', r'pattern2', r'pattern3'],
}
```
### Adding Detection Methods
To add a new detection method, create a method like:
```python
def detect_code_blocks_by_newmethod(self, page):
"""Detect code using new method"""
code_blocks = []
# ... your detection logic ...
return code_blocks
```
Then add it to `extract_page()`:
```python
newmethod_code_blocks = self.detect_code_blocks_by_newmethod(page)
all_code_blocks = font_code_blocks + indent_code_blocks + pattern_code_blocks + newmethod_code_blocks
```
---
## Conclusion
This POC successfully demonstrates:
- ✅ PyMuPDF can extract text from PDF documentation
- ✅ Multiple detection methods can identify code blocks
- ✅ Language detection works for common languages
- ✅ JSON output is compatible with existing doc_scraper.py
- ✅ Performance is acceptable for typical documentation PDFs
**Ready for B1.3:** The foundation is solid. Next step is adding page chunking and handling large PDFs.
---
**POC Completed:** October 21, 2025
**Next Task:** B1.3 - Add PDF page detection and chunking
assets/repo/Skill_Seekers-development/docs/PDF_IMAGE_EXTRACTION.md
+553
View File
@@ -0,0 +1,553 @@
# PDF Image Extraction (Task B1.5)
**Status:** ✅ Completed
**Date:** October 21, 2025
**Task:** B1.5 - Add PDF image extraction (diagrams, screenshots)
---
## Overview
Task B1.5 adds the ability to extract images (diagrams, screenshots, charts) from PDF documentation and save them as separate files. This is essential for preserving visual documentation elements in skills.
## New Features
### ✅ 1. Image Extraction to Files
Extract embedded images from PDFs and save them to disk:
```bash
# Extract images along with text
python3 cli/pdf_extractor_poc.py manual.pdf --extract-images
# Specify output directory
python3 cli/pdf_extractor_poc.py manual.pdf --extract-images --image-dir assets/images/
# Filter small images (icons, bullets)
python3 cli/pdf_extractor_poc.py manual.pdf --extract-images --min-image-size 200
```
### ✅ 2. Size-Based Filtering
Automatically filter out small images (icons, bullets, decorations):
- **Default threshold:** 100x100 pixels
- **Configurable:** `--min-image-size`
- **Purpose:** Focus on meaningful diagrams and screenshots
### ✅ 3. Image Metadata
Each extracted image includes comprehensive metadata:
```json
{
"filename": "manual_page5_img1.png",
"path": "output/manual_images/manual_page5_img1.png",
"page_number": 5,
"width": 800,
"height": 600,
"format": "png",
"size_bytes": 45821,
"xref": 42
}
```
### ✅ 4. Automatic Directory Creation
Images are automatically organized:
- **Default:** `output/{pdf_name}_images/`
- **Naming:** `{pdf_name}_page{N}_img{M}.{ext}`
- **Formats:** PNG, JPEG, GIF, BMP, etc.
---
## Usage Examples
### Basic Image Extraction
```bash
# Extract all images from PDF
python3 cli/pdf_extractor_poc.py tutorial.pdf --extract-images -v
```
**Output:**
```
📄 Extracting from: tutorial.pdf
Pages: 50
Metadata: {...}
Image directory: output/tutorial_images
Page 1: 2500 chars, 3 code blocks, 2 headings, 0 images
Page 2: 1800 chars, 1 code blocks, 1 headings, 2 images
Extracted image: tutorial_page2_img1.png (800x600)
Extracted image: tutorial_page2_img2.jpeg (1024x768)
...
✅ Extraction complete:
Images found: 45
Images extracted: 32
Image directory: output/tutorial_images
```
### Custom Image Directory
```bash
# Save images to specific directory
python3 cli/pdf_extractor_poc.py manual.pdf --extract-images --image-dir docs/images/
```
Result: Images saved to `docs/images/manual_page*_img*.{ext}`
### Filter Small Images
```bash
# Only extract images >= 200x200 pixels
python3 cli/pdf_extractor_poc.py guide.pdf --extract-images --min-image-size 200 -v
```
**Verbose output shows filtering:**
```
Page 5: 3200 chars, 4 code blocks, 3 headings, 3 images
Skipping small image: 32x32
Skipping small image: 64x48
Extracted image: guide_page5_img3.png (1200x800)
```
### Complete Extraction Workflow
```bash
# Extract everything: text, code, images
python3 cli/pdf_extractor_poc.py documentation.pdf \
--extract-images \
--min-image-size 150 \
--min-quality 6.0 \
--chunk-size 20 \
--output documentation.json \
--verbose \
--pretty
```
---
## Output Format
### Enhanced JSON Structure
The output now includes image extraction data:
```json
{
"source_file": "manual.pdf",
"total_pages": 50,
"total_images": 45,
"total_extracted_images": 32,
"image_directory": "output/manual_images",
"extracted_images": [
{
"filename": "manual_page2_img1.png",
"path": "output/manual_images/manual_page2_img1.png",
"page_number": 2,
"width": 800,
"height": 600,
"format": "png",
"size_bytes": 45821,
"xref": 42
}
],
"pages": [
{
"page_number": 1,
"images_count": 3,
"extracted_images": [
{
"filename": "manual_page1_img1.jpeg",
"path": "output/manual_images/manual_page1_img1.jpeg",
"width": 1024,
"height": 768,
"format": "jpeg",
"size_bytes": 87543
}
]
}
]
}
```
### File System Layout
```
output/
├── manual.json # Extraction results
└── manual_images/ # Image directory
├── manual_page2_img1.png # Page 2, Image 1
├── manual_page2_img2.jpeg # Page 2, Image 2
├── manual_page5_img1.png # Page 5, Image 1
└── ...
```
---
## Technical Implementation
### Image Extraction Method
```python
def extract_images_from_page(self, page, page_num):
"""Extract images from PDF page and save to disk"""
extracted = []
image_list = page.get_images()
for img_index, img in enumerate(image_list):
# Get image data from PDF
xref = img[0]
base_image = self.doc.extract_image(xref)
image_bytes = base_image["image"]
image_ext = base_image["ext"]
width = base_image.get("width", 0)
height = base_image.get("height", 0)
# Filter small images
if width < self.min_image_size or height < self.min_image_size:
continue
# Generate filename
image_filename = f"{pdf_basename}_page{page_num+1}_img{img_index+1}.{image_ext}"
image_path = Path(self.image_dir) / image_filename
# Save image
with open(image_path, "wb") as f:
f.write(image_bytes)
# Store metadata
image_info = {
'filename': image_filename,
'path': str(image_path),
'page_number': page_num + 1,
'width': width,
'height': height,
'format': image_ext,
'size_bytes': len(image_bytes),
}
extracted.append(image_info)
return extracted
```
---
## Performance
### Extraction Speed
| PDF Size | Images | Extraction Time | Overhead |
|----------|--------|-----------------|----------|
| Small (10 pages, 5 images) | 5 | +200ms | ~10% |
| Medium (100 pages, 50 images) | 50 | +2s | ~15% |
| Large (500 pages, 200 images) | 200 | +8s | ~20% |
**Note:** Image extraction adds 10-20% overhead depending on image count and size.
### Storage Requirements
- **PNG images:** ~10-500 KB each (diagrams)
- **JPEG images:** ~50-2000 KB each (screenshots)
- **Typical documentation (100 pages):** ~50-200 MB total
---
## Supported Image Formats
PyMuPDF automatically handles format detection and extraction:
- ✅ PNG (lossless, best for diagrams)
- ✅ JPEG (lossy, best for photos)
- ✅ GIF (animated, rare in PDFs)
- ✅ BMP (uncompressed)
- ✅ TIFF (high quality)
Images are extracted in their original format.
---
## Filtering Strategy
### Why Filter Small Images?
PDFs often contain:
- **Icons:** 16x16, 32x32 (UI elements)
- **Bullets:** 8x8, 12x12 (decorative)
- **Logos:** 50x50, 100x100 (branding)
These are usually not useful for documentation skills.
### Recommended Thresholds
| Use Case | Min Size | Reasoning |
|----------|----------|-----------|
| **General docs** | 100x100 | Filters icons, keeps diagrams |
| **Technical diagrams** | 200x200 | Only meaningful charts |
| **Screenshots** | 300x300 | Only full-size screenshots |
| **All images** | 0 | No filtering |
**Set with:** `--min-image-size N`
---
## Integration with Skill Seeker
### Future Workflow (Task B1.6+)
When building PDF-based skills, images will be:
1. **Extracted** from PDF documentation
2. **Organized** into skill's `assets/` directory
3. **Referenced** in SKILL.md and reference files
4. **Packaged** in final .zip file
**Example:**
```markdown
# API Architecture
See diagram below for the complete API flow:
![API Flow](assets/images/api_flow.png)
The diagram shows...
```
---
## Limitations
### Current Limitations
1. **No OCR**
- Cannot extract text from images
- Code screenshots are not parsed
- Future: Add OCR support for code in images
2. **No Image Analysis**
- Cannot detect diagram types (flowchart, UML, etc.)
- Cannot extract captions
- Future: Add AI-based image classification
3. **No Deduplication**
- Same image on multiple pages extracted multiple times
- Future: Add image hash-based deduplication
4. **Format Preservation**
- Images saved in original format (no conversion)
- No optimization or compression
### Known Issues
1. **Vector Graphics**
- Some PDFs use vector graphics (not images)
- These are not extracted (rendered as part of page)
- Workaround: Use PDF-to-image tools first
2. **Embedded vs Referenced**
- Only embedded images are extracted
- External image references are not followed
3. **Image Quality**
- Quality depends on PDF source
- Low-res source = low-res output
---
## Troubleshooting
### No Images Extracted
**Problem:** `total_extracted_images: 0` but PDF has visible images
**Possible causes:**
1. Images are vector graphics (not raster)
2. Images smaller than `--min-image-size` threshold
3. Images are page backgrounds (not embedded images)
**Solution:**
```bash
# Try with no size filter
python3 cli/pdf_extractor_poc.py input.pdf --extract-images --min-image-size 0 -v
```
### Permission Errors
**Problem:** `PermissionError: [Errno 13] Permission denied`
**Solution:**
```bash
# Ensure output directory is writable
mkdir -p output/images
chmod 755 output/images
# Or specify different directory
python3 cli/pdf_extractor_poc.py input.pdf --extract-images --image-dir ~/my_images/
```
### Disk Space
**Problem:** Running out of disk space
**Solution:**
```bash
# Check PDF size first
du -h input.pdf
# Estimate: ~100-200 MB per 100 pages with images
# Use higher min-image-size to extract fewer images
python3 cli/pdf_extractor_poc.py input.pdf --extract-images --min-image-size 300
```
---
## Examples
### Extract Diagram-Heavy Documentation
```bash
# Architecture documentation with many diagrams
python3 cli/pdf_extractor_poc.py architecture.pdf \
--extract-images \
--min-image-size 250 \
--image-dir docs/diagrams/ \
-v
```
**Result:** High-quality diagrams extracted, icons filtered out.
### Tutorial with Screenshots
```bash
# Tutorial with step-by-step screenshots
python3 cli/pdf_extractor_poc.py tutorial.pdf \
--extract-images \
--min-image-size 400 \
--image-dir tutorial_screenshots/ \
-v
```
**Result:** Full screenshots extracted, UI icons ignored.
### API Reference with Small Charts
```bash
# API docs with various image sizes
python3 cli/pdf_extractor_poc.py api_reference.pdf \
--extract-images \
--min-image-size 150 \
-o api.json \
--pretty
```
**Result:** Charts and graphs extracted, small icons filtered.
---
## Command-Line Reference
### Image Extraction Options
```
--extract-images
Enable image extraction to files
Default: disabled
--image-dir PATH
Directory to save extracted images
Default: output/{pdf_name}_images/
--min-image-size PIXELS
Minimum image dimension (width or height)
Filters out icons and small decorations
Default: 100
```
### Complete Example
```bash
python3 cli/pdf_extractor_poc.py manual.pdf \
--extract-images \
--image-dir assets/images/ \
--min-image-size 200 \
--min-quality 7.0 \
--chunk-size 15 \
--output manual.json \
--verbose \
--pretty
```
---
## Comparison: Before vs After
| Feature | Before (B1.4) | After (B1.5) |
|---------|---------------|--------------|
| Image detection | ✅ Count only | ✅ Count + Extract |
| Image files | ❌ Not saved | ✅ Saved to disk |
| Image metadata | ❌ None | ✅ Full metadata |
| Size filtering | ❌ None | ✅ Configurable |
| Directory organization | ❌ N/A | ✅ Automatic |
| Format support | ❌ N/A | ✅ All formats |
---
## Next Steps
### Task B1.6: Full PDF Scraper CLI
The image extraction feature will be integrated into the full PDF scraper:
```bash
# Future: Full PDF scraper with images
python3 cli/pdf_scraper.py \
--config configs/manual_pdf.json \
--extract-images \
--enhance-local
```
### Task B1.7: MCP Tool Integration
Images will be available through MCP:
```python
# Future: MCP tool
result = mcp.scrape_pdf(
pdf_path="manual.pdf",
extract_images=True,
min_image_size=200
)
```
---
## Conclusion
Task B1.5 successfully implements:
- ✅ Image extraction from PDF pages
- ✅ Automatic file saving with metadata
- ✅ Size-based filtering (configurable)
- ✅ Organized directory structure
- ✅ Multiple format support
**Impact:**
- Preserves visual documentation
- Essential for diagram-heavy docs
- Improves skill completeness
**Performance:** 10-20% overhead (acceptable)
**Compatibility:** Backward compatible (images optional)
**Ready for B1.6:** Full PDF scraper CLI tool
---
**Task Completed:** October 21, 2025
**Next Task:** B1.6 - Create `pdf_scraper.py` CLI tool
assets/repo/Skill_Seekers-development/docs/PDF_MCP_TOOL.md
+437
View File
@@ -0,0 +1,437 @@
# PDF Scraping MCP Tool (Task B1.7)
**Status:** ✅ Completed
**Date:** October 21, 2025
**Task:** B1.7 - Add MCP tool `scrape_pdf`
---
## Overview
Task B1.7 adds the `scrape_pdf` MCP tool to the Skill Seeker MCP server, making PDF documentation scraping available through the Model Context Protocol. This allows Claude Code and other MCP clients to scrape PDF documentation directly.
## Features
### ✅ MCP Tool Integration
- **Tool name:** `scrape_pdf`
- **Description:** Scrape PDF documentation and build Claude skill
- **Supports:** All three usage modes (config, direct, from-json)
- **Integration:** Uses `cli/pdf_scraper.py` backend
### ✅ Three Usage Modes
1. **Config File Mode** - Use PDF config JSON
2. **Direct PDF Mode** - Quick conversion from PDF file
3. **From JSON Mode** - Build from pre-extracted data
---
## Usage
### Mode 1: Config File
```python
# Through MCP
result = await mcp.call_tool("scrape_pdf", {
"config_path": "configs/manual_pdf.json"
})
```
**Example config** (`configs/manual_pdf.json`):
```json
{
"name": "mymanual",
"description": "My Manual documentation",
"pdf_path": "docs/manual.pdf",
"extract_options": {
"chunk_size": 10,
"min_quality": 6.0,
"extract_images": true,
"min_image_size": 150
},
"categories": {
"getting_started": ["introduction", "setup"],
"api": ["api", "reference"],
"tutorial": ["tutorial", "example"]
}
}
```
**Output:**
```
🔍 Extracting from PDF: docs/manual.pdf
📄 Extracting from: docs/manual.pdf
Pages: 150
...
✅ Extraction complete
🏗️ Building skill: mymanual
📋 Categorizing content...
✅ Created 3 categories
📝 Generating reference files...
Generated: output/mymanual/references/getting_started.md
Generated: output/mymanual/references/api.md
Generated: output/mymanual/references/tutorial.md
✅ Skill built successfully: output/mymanual/
📦 Next step: Package with: python3 cli/package_skill.py output/mymanual/
```
### Mode 2: Direct PDF
```python
# Through MCP
result = await mcp.call_tool("scrape_pdf", {
"pdf_path": "manual.pdf",
"name": "mymanual",
"description": "My Manual Docs"
})
```
**Uses default settings:**
- Chunk size: 10
- Min quality: 5.0
- Extract images: true
- Chapter-based categorization
### Mode 3: From Extracted JSON
```python
# Step 1: Extract to JSON (separate tool or CLI)
# python3 cli/pdf_extractor_poc.py manual.pdf -o manual_extracted.json
# Step 2: Build skill from JSON via MCP
result = await mcp.call_tool("scrape_pdf", {
"from_json": "output/manual_extracted.json"
})
```
**Benefits:**
- Separate extraction and building
- Fast iteration on skill structure
- No re-extraction needed
---
## MCP Tool Definition
### Input Schema
```json
{
"name": "scrape_pdf",
"description": "Scrape PDF documentation and build Claude skill. Extracts text, code, and images from PDF files (NEW in B1.7).",
"inputSchema": {
"type": "object",
"properties": {
"config_path": {
"type": "string",
"description": "Path to PDF config JSON file (e.g., configs/manual_pdf.json)"
},
"pdf_path": {
"type": "string",
"description": "Direct PDF path (alternative to config_path)"
},
"name": {
"type": "string",
"description": "Skill name (required with pdf_path)"
},
"description": {
"type": "string",
"description": "Skill description (optional)"
},
"from_json": {
"type": "string",
"description": "Build from extracted JSON file (e.g., output/manual_extracted.json)"
}
},
"required": []
}
}
```
### Return Format
Returns `TextContent` with:
- Success: stdout from `pdf_scraper.py`
- Failure: stderr + stdout for debugging
---
## Implementation
### MCP Server Changes
**Location:** `skill_seeker_mcp/server.py`
**Changes:**
1. Added `scrape_pdf` to `list_tools()` (lines 220-249)
2. Added handler in `call_tool()` (lines 276-277)
3. Implemented `scrape_pdf_tool()` function (lines 591-625)
### Code Implementation
```python
async def scrape_pdf_tool(args: dict) -> list[TextContent]:
"""Scrape PDF documentation and build skill (NEW in B1.7)"""
config_path = args.get("config_path")
pdf_path = args.get("pdf_path")
name = args.get("name")
description = args.get("description")
from_json = args.get("from_json")
# Build command
cmd = [sys.executable, str(CLI_DIR / "pdf_scraper.py")]
# Mode 1: Config file
if config_path:
cmd.extend(["--config", config_path])
# Mode 2: Direct PDF
elif pdf_path and name:
cmd.extend(["--pdf", pdf_path, "--name", name])
if description:
cmd.extend(["--description", description])
# Mode 3: From JSON
elif from_json:
cmd.extend(["--from-json", from_json])
else:
return [TextContent(type="text", text="❌ Error: Must specify --config, --pdf + --name, or --from-json")]
# Run pdf_scraper.py
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode == 0:
return [TextContent(type="text", text=result.stdout)]
else:
return [TextContent(type="text", text=f"Error: {result.stderr}\n\n{result.stdout}")]
```
---
## Integration with MCP Workflow
### Complete Workflow Through MCP
```python
# 1. Create PDF config (optional - can use direct mode)
config_result = await mcp.call_tool("generate_config", {
"name": "api_manual",
"url": "N/A", # Not used for PDF
"description": "API Manual from PDF"
})
# 2. Scrape PDF
scrape_result = await mcp.call_tool("scrape_pdf", {
"pdf_path": "docs/api_manual.pdf",
"name": "api_manual",
"description": "API Manual Documentation"
})
# 3. Package skill
package_result = await mcp.call_tool("package_skill", {
"skill_dir": "output/api_manual/",
"auto_upload": True # Upload if ANTHROPIC_API_KEY set
})
# 4. Upload (if not auto-uploaded)
if "ANTHROPIC_API_KEY" in os.environ:
upload_result = await mcp.call_tool("upload_skill", {
"skill_zip": "output/api_manual.zip"
})
```
### Combined with Web Scraping
```python
# Scrape web documentation
web_result = await mcp.call_tool("scrape_docs", {
"config_path": "configs/framework.json"
})
# Scrape PDF supplement
pdf_result = await mcp.call_tool("scrape_pdf", {
"pdf_path": "docs/framework_api.pdf",
"name": "framework_pdf"
})
# Package both
await mcp.call_tool("package_skill", {"skill_dir": "output/framework/"})
await mcp.call_tool("package_skill", {"skill_dir": "output/framework_pdf/"})
```
---
## Error Handling
### Common Errors
**Error 1: Missing required parameters**
```
❌ Error: Must specify --config, --pdf + --name, or --from-json
```
**Solution:** Provide one of the three modes
**Error 2: PDF file not found**
```
Error: [Errno 2] No such file or directory: 'manual.pdf'
```
**Solution:** Check PDF path is correct
**Error 3: PyMuPDF not installed**
```
ERROR: PyMuPDF not installed
Install with: pip install PyMuPDF
```
**Solution:** Install PyMuPDF: `pip install PyMuPDF`
**Error 4: Invalid JSON config**
```
Error: json.decoder.JSONDecodeError: Expecting value: line 1 column 1
```
**Solution:** Check config file is valid JSON
---
## Testing
### Test MCP Tool
```bash
# 1. Start MCP server
python3 skill_seeker_mcp/server.py
# 2. Test with MCP client or via Claude Code
# 3. Verify tool is listed
# Should see "scrape_pdf" in available tools
```
### Test All Modes
**Mode 1: Config**
```python
result = await mcp.call_tool("scrape_pdf", {
"config_path": "configs/example_pdf.json"
})
assert "✅ Skill built successfully" in result[0].text
```
**Mode 2: Direct**
```python
result = await mcp.call_tool("scrape_pdf", {
"pdf_path": "test.pdf",
"name": "test_skill"
})
assert "✅ Skill built successfully" in result[0].text
```
**Mode 3: From JSON**
```python
# First extract
subprocess.run(["python3", "cli/pdf_extractor_poc.py", "test.pdf", "-o", "test.json"])
# Then build via MCP
result = await mcp.call_tool("scrape_pdf", {
"from_json": "test.json"
})
assert "✅ Skill built successfully" in result[0].text
```
---
## Comparison with Other MCP Tools
| Tool | Input | Output | Use Case |
|------|-------|--------|----------|
| `scrape_docs` | HTML URL | Skill | Web documentation |
| `scrape_pdf` | PDF file | Skill | PDF documentation |
| `generate_config` | URL | Config | Create web config |
| `package_skill` | Skill dir | .zip | Package for upload |
| `upload_skill` | .zip file | Upload | Send to Claude |
---
## Performance
### MCP Tool Overhead
- **MCP overhead:** ~50-100ms
- **Extraction time:** Same as CLI (15s-5m depending on PDF)
- **Building time:** Same as CLI (5s-45s)
**Total:** MCP adds negligible overhead (<1%)
### Async Execution
The MCP tool runs `pdf_scraper.py` synchronously via `subprocess.run()`. For long-running PDFs:
- Client waits for completion
- No progress updates during extraction
- Consider using `--from-json` mode for faster iteration
---
## Future Enhancements
### Potential Improvements
1. **Async Extraction**
- Stream progress updates to client
- Allow cancellation
- Background processing
2. **Batch Processing**
- Process multiple PDFs in parallel
- Merge into single skill
- Shared categories
3. **Enhanced Options**
- Pass all extraction options through MCP
- Dynamic quality threshold
- Image filter controls
4. **Status Checking**
- Query extraction status
- Get progress percentage
- Estimate time remaining
---
## Conclusion
Task B1.7 successfully implements:
- ✅ MCP tool `scrape_pdf`
- ✅ Three usage modes (config, direct, from-json)
- ✅ Integration with MCP server
- ✅ Error handling
- ✅ Compatible with existing MCP workflow
**Impact:**
- PDF scraping available through MCP
- Seamless integration with Claude Code
- Unified workflow for web + PDF documentation
- 10th MCP tool in Skill Seeker
**Total MCP Tools:** 10
1. generate_config
2. estimate_pages
3. scrape_docs
4. package_skill
5. upload_skill
6. list_configs
7. validate_config
8. split_config
9. generate_router
10. **scrape_pdf** (NEW)
---
**Task Completed:** October 21, 2025
**B1 Group Complete:** All 8 tasks (B1.1-B1.8) finished!
**Next:** Task group B2 (Microsoft Word .docx support)
assets/repo/Skill_Seekers-development/docs/PDF_PARSING_RESEARCH.md
+491
View File
@@ -0,0 +1,491 @@
# PDF Parsing Libraries Research (Task B1.1)
**Date:** October 21, 2025
**Task:** B1.1 - Research PDF parsing libraries
**Purpose:** Evaluate Python libraries for extracting text and code from PDF documentation
---
## Executive Summary
After comprehensive research, **PyMuPDF (fitz)** is recommended as the primary library for Skill Seeker's PDF parsing needs, with **pdfplumber** as a secondary option for complex table extraction.
### Quick Recommendation:
- **Primary Choice:** PyMuPDF (fitz) - Fast, comprehensive, well-maintained
- **Secondary/Fallback:** pdfplumber - Better for tables, slower but more precise
- **Avoid:** PyPDF2 (deprecated, merged into pypdf)
---
## Library Comparison Matrix
| Library | Speed | Text Quality | Code Detection | Tables | Maintenance | License |
|---------|-------|--------------|----------------|--------|-------------|---------|
| **PyMuPDF** | ⚡⚡⚡⚡⚡ Fastest (42ms) | High | Excellent | Good | Active | AGPL/Commercial |
| **pdfplumber** | ⚡⚡ Slower (2.5s) | Very High | Excellent | Excellent | Active | MIT |
| **pypdf** | ⚡⚡⚡ Fast | Medium | Good | Basic | Active | BSD |
| **pdfminer.six** | ⚡ Slow | Very High | Good | Medium | Active | MIT |
| **pypdfium2** | ⚡⚡⚡⚡⚡ Very Fast (3ms) | Medium | Good | Basic | Active | Apache-2.0 |
---
## Detailed Analysis
### 1. PyMuPDF (fitz) ⭐ RECOMMENDED
**Performance:** 42 milliseconds (60x faster than pdfminer.six)
**Installation:**
```bash
pip install PyMuPDF
```
**Pros:**
- ✅ Extremely fast (C-based MuPDF backend)
- ✅ Comprehensive features (text, images, tables, metadata)
- ✅ Supports markdown output
- ✅ Can extract images and diagrams
- ✅ Well-documented and actively maintained
- ✅ Handles complex layouts well
**Cons:**
- ⚠️ AGPL license (requires commercial license for proprietary projects)
- ⚠️ Requires MuPDF binary installation (handled by pip)
- ⚠️ Slightly larger dependency footprint
**Code Example:**
```python
import fitz # PyMuPDF
# Extract text from entire PDF
def extract_pdf_text(pdf_path):
doc = fitz.open(pdf_path)
text = ''
for page in doc:
text += page.get_text()
doc.close()
return text
# Extract text from single page
def extract_page_text(pdf_path, page_num):
doc = fitz.open(pdf_path)
page = doc.load_page(page_num)
text = page.get_text()
doc.close()
return text
# Extract with markdown formatting
def extract_as_markdown(pdf_path):
doc = fitz.open(pdf_path)
markdown = ''
for page in doc:
markdown += page.get_text("markdown")
doc.close()
return markdown
```
**Use Cases for Skill Seeker:**
- Fast extraction of code examples from PDF docs
- Preserving formatting for code blocks
- Extracting diagrams and screenshots
- High-volume documentation scraping
---
### 2. pdfplumber ⭐ RECOMMENDED (for tables)
**Performance:** ~2.5 seconds (slower but more precise)
**Installation:**
```bash
pip install pdfplumber
```
**Pros:**
- ✅ MIT license (fully open source)
- ✅ Exceptional table extraction
- ✅ Visual debugging tool
- ✅ Precise layout preservation
- ✅ Built on pdfminer (proven text extraction)
- ✅ No binary dependencies
**Cons:**
- ⚠️ Slower than PyMuPDF
- ⚠️ Higher memory usage for large PDFs
- ⚠️ Requires more configuration for optimal results
**Code Example:**
```python
import pdfplumber
# Extract text from PDF
def extract_with_pdfplumber(pdf_path):
with pdfplumber.open(pdf_path) as pdf:
text = ''
for page in pdf.pages:
text += page.extract_text()
return text
# Extract tables
def extract_tables(pdf_path):
tables = []
with pdfplumber.open(pdf_path) as pdf:
for page in pdf.pages:
page_tables = page.extract_tables()
tables.extend(page_tables)
return tables
# Extract specific region (for code blocks)
def extract_region(pdf_path, page_num, bbox):
with pdfplumber.open(pdf_path) as pdf:
page = pdf.pages[page_num]
cropped = page.crop(bbox)
return cropped.extract_text()
```
**Use Cases for Skill Seeker:**
- Extracting API reference tables from PDFs
- Precise code block extraction with layout
- Documentation with complex table structures
---
### 3. pypdf (formerly PyPDF2)
**Performance:** Fast (medium speed)
**Installation:**
```bash
pip install pypdf
```
**Pros:**
- ✅ BSD license
- ✅ Simple API
- ✅ Can modify PDFs (merge, split, encrypt)
- ✅ Actively maintained (PyPDF2 merged back)
- ✅ No external dependencies
**Cons:**
- ⚠️ Limited complex layout support
- ⚠️ Basic text extraction only
- ⚠️ Poor with scanned/image PDFs
- ⚠️ No table extraction
**Code Example:**
```python
from pypdf import PdfReader
# Extract text
def extract_with_pypdf(pdf_path):
reader = PdfReader(pdf_path)
text = ''
for page in reader.pages:
text += page.extract_text()
return text
```
**Use Cases for Skill Seeker:**
- Simple text extraction
- Fallback when PyMuPDF licensing is an issue
- Basic PDF manipulation tasks
---
### 4. pdfminer.six
**Performance:** Slow (~2.5 seconds)
**Installation:**
```bash
pip install pdfminer.six
```
**Pros:**
- ✅ MIT license
- ✅ Excellent text quality (preserves formatting)
- ✅ Handles complex layouts
- ✅ Pure Python (no binaries)
**Cons:**
- ⚠️ Slowest option
- ⚠️ Complex API
- ⚠️ Poor documentation
- ⚠️ Limited table support
**Use Cases for Skill Seeker:**
- Not recommended (pdfplumber is built on this with better API)
---
### 5. pypdfium2
**Performance:** Very fast (3ms - fastest tested)
**Installation:**
```bash
pip install pypdfium2
```
**Pros:**
- ✅ Extremely fast
- ✅ Apache 2.0 license
- ✅ Lightweight
- ✅ Clean output
**Cons:**
- ⚠️ Basic features only
- ⚠️ Limited documentation
- ⚠️ No table extraction
- ⚠️ Newer/less proven
**Use Cases for Skill Seeker:**
- High-speed basic extraction
- Potential future optimization
---
## Licensing Considerations
### Open Source Projects (Skill Seeker):
- **PyMuPDF:** ✅ AGPL license is fine for open-source projects
- **pdfplumber:** ✅ MIT license (most permissive)
- **pypdf:** ✅ BSD license (permissive)
### Important Note:
PyMuPDF requires AGPL compliance (source code must be shared) OR a commercial license for proprietary use. Since Skill Seeker is open source on GitHub, AGPL is acceptable.
---
## Performance Benchmarks
Based on 2025 testing:
| Library | Time (single page) | Time (100 pages) |
|---------|-------------------|------------------|
| pypdfium2 | 0.003s | 0.3s |
| PyMuPDF | 0.042s | 4.2s |
| pypdf | 0.1s | 10s |
| pdfplumber | 2.5s | 250s |
| pdfminer.six | 2.5s | 250s |
**Winner:** pypdfium2 (speed) / PyMuPDF (features + speed balance)
---
## Recommendations for Skill Seeker
### Primary Approach: PyMuPDF (fitz)
**Why:**
1. **Speed** - 60x faster than alternatives
2. **Features** - Text, images, markdown output, metadata
3. **Quality** - High-quality text extraction
4. **Maintained** - Active development, good docs
5. **License** - AGPL is fine for open source
**Implementation Strategy:**
```python
import fitz # PyMuPDF
def extract_pdf_documentation(pdf_path):
"""
Extract documentation from PDF with code block detection
"""
doc = fitz.open(pdf_path)
pages = []
for page_num, page in enumerate(doc):
# Get text with layout info
text = page.get_text("text")
# Get markdown (preserves code blocks)
markdown = page.get_text("markdown")
# Get images (for diagrams)
images = page.get_images()
pages.append({
'page_number': page_num,
'text': text,
'markdown': markdown,
'images': images
})
doc.close()
return pages
```
### Fallback Approach: pdfplumber
**When to use:**
- PDF has complex tables that PyMuPDF misses
- Need visual debugging
- License concerns (use MIT instead of AGPL)
**Implementation Strategy:**
```python
import pdfplumber
def extract_pdf_tables(pdf_path):
"""
Extract tables from PDF documentation
"""
with pdfplumber.open(pdf_path) as pdf:
tables = []
for page in pdf.pages:
page_tables = page.extract_tables()
if page_tables:
tables.extend(page_tables)
return tables
```
---
## Code Block Detection Strategy
PDFs don't have semantic "code block" markers like HTML. Detection strategies:
### 1. Font-based Detection
```python
# PyMuPDF can detect font changes
def detect_code_by_font(page):
blocks = page.get_text("dict")["blocks"]
code_blocks = []
for block in blocks:
if 'lines' in block:
for line in block['lines']:
for span in line['spans']:
font = span['font']
# Monospace fonts indicate code
if 'Courier' in font or 'Mono' in font:
code_blocks.append(span['text'])
return code_blocks
```
### 2. Indentation-based Detection
```python
def detect_code_by_indent(text):
lines = text.split('\n')
code_blocks = []
current_block = []
for line in lines:
# Code often has consistent indentation
if line.startswith(' ') or line.startswith('\t'):
current_block.append(line)
elif current_block:
code_blocks.append('\n'.join(current_block))
current_block = []
return code_blocks
```
### 3. Pattern-based Detection
```python
import re
def detect_code_by_pattern(text):
# Look for common code patterns
patterns = [
r'(def \w+\(.*?\):)', # Python functions
r'(function \w+\(.*?\) \{)', # JavaScript
r'(class \w+:)', # Python classes
r'(import \w+)', # Import statements
]
code_snippets = []
for pattern in patterns:
matches = re.findall(pattern, text)
code_snippets.extend(matches)
return code_snippets
```
---
## Next Steps (Task B1.2+)
### Immediate Next Task: B1.2 - Create Simple PDF Text Extractor
**Goal:** Proof of concept using PyMuPDF
**Implementation Plan:**
1. Create `cli/pdf_extractor_poc.py`
2. Extract text from sample PDF
3. Detect code blocks using font/pattern matching
4. Output to JSON (similar to web scraper)
**Dependencies:**
```bash
pip install PyMuPDF
```
**Expected Output:**
```json
{
"pages": [
{
"page_number": 1,
"text": "...",
"code_blocks": ["def main():", "import sys"],
"images": []
}
]
}
```
### Future Tasks:
- **B1.3:** Add page chunking (split large PDFs)
- **B1.4:** Improve code block detection
- **B1.5:** Extract images/diagrams
- **B1.6:** Create full `pdf_scraper.py` CLI
- **B1.7:** Add MCP tool integration
- **B1.8:** Create PDF config format
---
## Additional Resources
### Documentation:
- PyMuPDF: https://pymupdf.readthedocs.io/
- pdfplumber: https://github.com/jsvine/pdfplumber
- pypdf: https://pypdf.readthedocs.io/
### Comparison Studies:
- 2025 Comparative Study: https://arxiv.org/html/2410.09871v1
- Performance Benchmarks: https://github.com/py-pdf/benchmarks
### Example Use Cases:
- Extracting API docs from PDF manuals
- Converting PDF guides to markdown
- Building skills from PDF-only documentation
---
## Conclusion
**For Skill Seeker's PDF documentation extraction:**
1. **Use PyMuPDF (fitz)** as primary library
2. **Add pdfplumber** for complex table extraction
3. **Detect code blocks** using font + pattern matching
4. **Preserve formatting** with markdown output
5. **Extract images** for diagrams/screenshots
**Estimated Implementation Time:**
- B1.2 (POC): 2-3 hours
- B1.3-B1.5 (Features): 5-8 hours
- B1.6 (CLI): 3-4 hours
- B1.7 (MCP): 2-3 hours
- B1.8 (Config): 1-2 hours
- **Total: 13-20 hours** for complete PDF support
**License:** AGPL (PyMuPDF) is acceptable for Skill Seeker (open source)
---
**Research completed:** ✅ October 21, 2025
**Next task:** B1.2 - Create simple PDF text extractor (proof of concept)
assets/repo/Skill_Seekers-development/docs/PDF_SCRAPER.md
+616
View File
@@ -0,0 +1,616 @@
# PDF Scraper CLI Tool (Tasks B1.6 + B1.8)
**Status:** ✅ Completed
**Date:** October 21, 2025
**Tasks:** B1.6 - Create pdf_scraper.py CLI tool, B1.8 - PDF config format
---
## Overview
The PDF scraper (`pdf_scraper.py`) is a complete CLI tool that converts PDF documentation into Claude AI skills. It integrates all PDF extraction features (B1.1-B1.5) with the Skill Seeker workflow to produce packaged, uploadable skills.
## Features
### ✅ Complete Workflow
1. **Extract** - Uses `pdf_extractor_poc.py` for extraction
2. **Categorize** - Organizes content by chapters or keywords
3. **Build** - Creates skill structure (SKILL.md, references/)
4. **Package** - Ready for `package_skill.py`
### ✅ Three Usage Modes
1. **Config File** - Use JSON configuration (recommended)
2. **Direct PDF** - Quick conversion from PDF file
3. **From JSON** - Build skill from pre-extracted data
### ✅ Automatic Categorization
- Chapter-based (from PDF structure)
- Keyword-based (configurable)
- Fallback to single category
### ✅ Quality Filtering
- Uses quality scores from B1.4
- Extracts top code examples
- Filters by minimum quality threshold
---
## Usage
### Mode 1: Config File (Recommended)
```bash
# Create config file
cat > configs/my_manual.json <<EOF
{
"name": "mymanual",
"description": "My Manual documentation",
"pdf_path": "docs/manual.pdf",
"extract_options": {
"chunk_size": 10,
"min_quality": 6.0,
"extract_images": true,
"min_image_size": 150
},
"categories": {
"getting_started": ["introduction", "setup"],
"api": ["api", "reference", "function"],
"tutorial": ["tutorial", "example", "guide"]
}
}
EOF
# Run scraper
python3 cli/pdf_scraper.py --config configs/my_manual.json
```
**Output:**
```
🔍 Extracting from PDF: docs/manual.pdf
📄 Extracting from: docs/manual.pdf
Pages: 150
...
✅ Extraction complete
💾 Saved extracted data to: output/mymanual_extracted.json
🏗️ Building skill: mymanual
📋 Categorizing content...
✅ Created 3 categories
- Getting Started: 25 pages
- Api: 80 pages
- Tutorial: 45 pages
📝 Generating reference files...
Generated: output/mymanual/references/getting_started.md
Generated: output/mymanual/references/api.md
Generated: output/mymanual/references/tutorial.md
Generated: output/mymanual/references/index.md
Generated: output/mymanual/SKILL.md
✅ Skill built successfully: output/mymanual/
📦 Next step: Package with: python3 cli/package_skill.py output/mymanual/
```
### Mode 2: Direct PDF
```bash
# Quick conversion without config file
python3 cli/pdf_scraper.py --pdf manual.pdf --name mymanual --description "My Manual Docs"
```
**Uses default settings:**
- Chunk size: 10
- Min quality: 5.0
- Extract images: true
- Min image size: 100px
- No custom categories (chapter-based)
### Mode 3: From Extracted JSON
```bash
# Step 1: Extract only (saves JSON)
python3 cli/pdf_extractor_poc.py manual.pdf -o manual_extracted.json --extract-images
# Step 2: Build skill from JSON (fast, can iterate)
python3 cli/pdf_scraper.py --from-json manual_extracted.json
```
**Benefits:**
- Separate extraction and building
- Iterate on skill structure without re-extracting
- Faster development cycle
---
## Config File Format (Task B1.8)
### Complete Example
```json
{
"name": "godot_manual",
"description": "Godot Engine documentation from PDF manual",
"pdf_path": "docs/godot_manual.pdf",
"extract_options": {
"chunk_size": 15,
"min_quality": 6.0,
"extract_images": true,
"min_image_size": 200
},
"categories": {
"getting_started": [
"introduction",
"getting started",
"installation",
"first steps"
],
"scripting": [
"gdscript",
"scripting",
"code",
"programming"
],
"3d": [
"3d",
"spatial",
"mesh",
"shader"
],
"2d": [
"2d",
"sprite",
"tilemap",
"animation"
],
"api": [
"api",
"class reference",
"method",
"property"
]
}
}
```
### Field Reference
#### Required Fields
- **`name`** (string): Skill identifier
- Used for directory names
- Should be lowercase, no spaces
- Example: `"python_guide"`
- **`pdf_path`** (string): Path to PDF file
- Absolute or relative to working directory
- Example: `"docs/manual.pdf"`
#### Optional Fields
- **`description`** (string): Skill description
- Shows in SKILL.md
- Explains when to use the skill
- Default: `"Documentation skill for {name}"`
- **`extract_options`** (object): Extraction settings
- `chunk_size` (number): Pages per chunk (default: 10)
- `min_quality` (number): Minimum code quality 0-10 (default: 5.0)
- `extract_images` (boolean): Extract images to files (default: true)
- `min_image_size` (number): Minimum image dimension in pixels (default: 100)
- **`categories`** (object): Keyword-based categorization
- Keys: Category names (will be sanitized for filenames)
- Values: Arrays of keywords to match
- If omitted: Uses chapter-based categorization from PDF
---
## Output Structure
### Generated Files
```
output/
├── mymanual_extracted.json # Raw extraction data (B1.5 format)
└── mymanual/ # Skill directory
├── SKILL.md # Main skill file
├── references/ # Reference documentation
│ ├── index.md # Category index
│ ├── getting_started.md # Category 1
│ ├── api.md # Category 2
│ └── tutorial.md # Category 3
├── scripts/ # Empty (for user scripts)
└── assets/ # Assets directory
└── images/ # Extracted images (if enabled)
├── mymanual_page5_img1.png
└── mymanual_page12_img2.jpeg
```
### SKILL.md Format
```markdown
# Mymanual Documentation Skill
My Manual documentation
## When to use this skill
Use this skill when the user asks about mymanual documentation,
including API references, tutorials, examples, and best practices.
## What's included
This skill contains:
- **Getting Started**: 25 pages
- **Api**: 80 pages
- **Tutorial**: 45 pages
## Quick Reference
### Top Code Examples
**Example 1** (Quality: 8.5/10):
```python
def initialize_system():
config = load_config()
setup_logging(config)
return System(config)
```
**Example 2** (Quality: 8.2/10):
```javascript
const app = createApp({
data() {
return { count: 0 }
}
})
```
## Navigation
See `references/index.md` for complete documentation structure.
## Languages Covered
- python: 45 examples
- javascript: 32 examples
- shell: 8 examples
```
### Reference File Format
Each category gets its own reference file:
```markdown
# Getting Started
## Installation
This guide will walk you through installing the software...
### Code Examples
```bash
curl -O https://example.com/install.sh
bash install.sh
```
---
## Configuration
After installation, configure your environment...
### Code Examples
```yaml
server:
port: 8080
host: localhost
```
---
```
---
## Categorization Logic
### Chapter-Based (Automatic)
If PDF has detectable chapters (from B1.3):
1. Extract chapter titles and page ranges
2. Create one category per chapter
3. Assign pages to chapters by page number
**Advantages:**
- Automatic, no config needed
- Respects document structure
- Accurate page assignment
**Example chapters:**
- "Chapter 1: Introduction" → `chapter_1_introduction.md`
- "Part 2: Advanced Topics" → `part_2_advanced_topics.md`
### Keyword-Based (Configurable)
If `categories` config is provided:
1. Score each page against keyword lists
2. Assign to highest-scoring category
3. Fall back to "other" if no match
**Advantages:**
- Flexible, customizable
- Works with PDFs without clear chapters
- Can combine related sections
**Scoring:**
- Keyword in page text: +1 point
- Keyword in page heading: +2 points
- Assigned to category with highest score
---
## Integration with Skill Seeker
### Complete Workflow
```bash
# 1. Create PDF config
cat > configs/api_manual.json <<EOF
{
"name": "api_manual",
"pdf_path": "docs/api.pdf",
"extract_options": {
"min_quality": 7.0,
"extract_images": true
}
}
EOF
# 2. Run PDF scraper
python3 cli/pdf_scraper.py --config configs/api_manual.json
# 3. Package skill
python3 cli/package_skill.py output/api_manual/
# 4. Upload to Claude (if ANTHROPIC_API_KEY set)
python3 cli/package_skill.py output/api_manual/ --upload
# Result: api_manual.zip ready for Claude!
```
### Enhancement (Optional)
```bash
# After building, enhance with AI
python3 cli/enhance_skill_local.py output/api_manual/
# Or with API
export ANTHROPIC_API_KEY=sk-ant-...
python3 cli/enhance_skill.py output/api_manual/
```
---
## Performance
### Benchmark
| PDF Size | Pages | Extraction | Building | Total |
|----------|-------|------------|----------|-------|
| Small | 50 | 30s | 5s | 35s |
| Medium | 200 | 2m | 15s | 2m 15s |
| Large | 500 | 5m | 45s | 5m 45s |
**Extraction**: PDF → JSON (cpu-intensive)
**Building**: JSON → Skill (fast, i/o-bound)
### Optimization Tips
1. **Use `--from-json` for iteration**
- Extract once, build many times
- Test categorization without re-extraction
2. **Adjust chunk size**
- Larger chunks: Faster extraction
- Smaller chunks: Better chapter detection
3. **Filter aggressively**
- Higher `min_quality`: Fewer low-quality code blocks
- Higher `min_image_size`: Fewer small images
---
## Examples
### Example 1: Programming Language Manual
```json
{
"name": "python_reference",
"description": "Python 3.12 Language Reference",
"pdf_path": "python-3.12-reference.pdf",
"extract_options": {
"chunk_size": 20,
"min_quality": 7.0,
"extract_images": false
},
"categories": {
"basics": ["introduction", "basic", "syntax", "types"],
"functions": ["function", "lambda", "decorator"],
"classes": ["class", "object", "inheritance"],
"modules": ["module", "package", "import"],
"stdlib": ["library", "standard library", "built-in"]
}
}
```
### Example 2: API Documentation
```json
{
"name": "rest_api_docs",
"description": "REST API Documentation",
"pdf_path": "api_docs.pdf",
"extract_options": {
"chunk_size": 10,
"min_quality": 6.0,
"extract_images": true,
"min_image_size": 200
},
"categories": {
"authentication": ["auth", "login", "token", "oauth"],
"users": ["user", "account", "profile"],
"products": ["product", "catalog", "inventory"],
"orders": ["order", "purchase", "checkout"],
"webhooks": ["webhook", "event", "callback"]
}
}
```
### Example 3: Framework Documentation
```json
{
"name": "django_docs",
"description": "Django Web Framework Documentation",
"pdf_path": "django-4.2-docs.pdf",
"extract_options": {
"chunk_size": 15,
"min_quality": 6.5,
"extract_images": true
}
}
```
*Note: No categories - uses chapter-based categorization*
---
## Troubleshooting
### No Categories Created
**Problem:** Only "content" or "other" category
**Possible causes:**
1. No chapters detected in PDF
2. Keywords don't match content
3. Config has empty categories
**Solution:**
```bash
# Check extracted chapters
cat output/mymanual_extracted.json | jq '.chapters'
# If empty, add keyword categories to config
# Or let it create single "content" category (OK for small PDFs)
```
### Low-Quality Code Blocks
**Problem:** Too many poor code examples
**Solution:**
```json
{
"extract_options": {
"min_quality": 7.0 // Increase threshold
}
}
```
### Images Not Extracted
**Problem:** No images in `assets/images/`
**Solution:**
```json
{
"extract_options": {
"extract_images": true, // Enable extraction
"min_image_size": 50 // Lower threshold
}
}
```
---
## Comparison with Web Scraper
| Feature | Web Scraper | PDF Scraper |
|---------|-------------|-------------|
| Input | HTML websites | PDF files |
| Crawling | Multi-page BFS | Single-file extraction |
| Structure detection | CSS selectors | Font/heading analysis |
| Categorization | URL patterns | Chapters/keywords |
| Images | Referenced | Embedded (extracted) |
| Code detection | `<pre><code>` | Font/indent/pattern |
| Language detection | CSS classes | Pattern matching |
| Quality scoring | No | Yes (B1.4) |
| Chunking | No | Yes (B1.3) |
---
## Next Steps
### Task B1.7: MCP Tool Integration
The PDF scraper will be available through MCP:
```python
# Future: MCP tool
result = mcp.scrape_pdf(
config_path="configs/manual.json"
)
# Or direct
result = mcp.scrape_pdf(
pdf_path="manual.pdf",
name="mymanual",
extract_images=True
)
```
---
## Conclusion
Tasks B1.6 and B1.8 successfully implement:
**B1.6 - PDF Scraper CLI:**
- ✅ Complete extraction → building workflow
- ✅ Three usage modes (config, direct, from-json)
- ✅ Automatic categorization (chapter or keyword-based)
- ✅ Integration with Skill Seeker workflow
- ✅ Quality filtering and top examples
**B1.8 - PDF Config Format:**
- ✅ JSON configuration format
- ✅ Extraction options (chunk size, quality, images)
- ✅ Category definitions (keyword-based)
- ✅ Compatible with web scraper config style
**Impact:**
- Complete PDF documentation support
- Parallel workflow to web scraping
- Reusable extraction results
- High-quality skill generation
**Ready for B1.7:** MCP tool integration
---
**Tasks Completed:** October 21, 2025
**Next Task:** B1.7 - Add MCP tool `scrape_pdf`
assets/repo/Skill_Seekers-development/docs/PDF_SYNTAX_DETECTION.md
+576
View File
@@ -0,0 +1,576 @@
# PDF Code Block Syntax Detection (Task B1.4)
**Status:** ✅ Completed
**Date:** October 21, 2025
**Task:** B1.4 - Extract code blocks from PDFs with syntax detection
---
## Overview
Task B1.4 enhances the PDF extractor with advanced code block detection capabilities including:
- **Confidence scoring** for language detection
- **Syntax validation** to filter out false positives
- **Quality scoring** to rank code blocks by usefulness
- **Automatic filtering** of low-quality code
This dramatically improves the accuracy and usefulness of extracted code samples from PDF documentation.
---
## New Features
### ✅ 1. Confidence-Based Language Detection
Enhanced language detection now returns both language and confidence score:
**Before (B1.2):**
```python
lang = detect_language_from_code(code) # Returns: 'python'
```
**After (B1.4):**
```python
lang, confidence = detect_language_from_code(code) # Returns: ('python', 0.85)
```
**Confidence Calculation:**
- Pattern matches are weighted (1-5 points)
- Scores are normalized to 0-1 range
- Higher confidence = more reliable detection
**Example Pattern Weights:**
```python
'python': [
(r'\bdef\s+\w+\s*\(', 3), # Strong indicator
(r'\bimport\s+\w+', 2), # Medium indicator
(r':\s*$', 1), # Weak indicator (lines ending with :)
]
```
### ✅ 2. Syntax Validation
Validates detected code blocks to filter false positives:
**Validation Checks:**
1. **Not empty** - Rejects empty code blocks
2. **Indentation consistency** (Python) - Detects mixed tabs/spaces
3. **Balanced brackets** - Checks for unclosed parentheses, braces
4. **Language-specific syntax** (JSON) - Attempts to parse
5. **Natural language detection** - Filters out prose misidentified as code
6. **Comment ratio** - Rejects blocks that are mostly comments
**Output:**
```json
{
"code": "def example():\n return True",
"language": "python",
"is_valid": true,
"validation_issues": []
}
```
**Invalid example:**
```json
{
"code": "This is not code",
"language": "unknown",
"is_valid": false,
"validation_issues": ["May be natural language, not code"]
}
```
### ✅ 3. Quality Scoring
Each code block receives a quality score (0-10) based on multiple factors:
**Scoring Factors:**
1. **Language confidence** (+0 to +2.0 points)
2. **Code length** (optimal: 20-500 chars, +1.0)
3. **Line count** (optimal: 2-50 lines, +1.0)
4. **Has definitions** (functions/classes, +1.5)
5. **Meaningful variable names** (+1.0)
6. **Syntax validation** (+1.0 if valid, -0.5 per issue)
**Quality Tiers:**
- **High quality (7-10):** Complete, valid, useful code examples
- **Medium quality (4-7):** Partial or simple code snippets
- **Low quality (0-4):** Fragments, false positives, invalid code
**Example:**
```python
# High-quality code block (score: 8.5/10)
def calculate_total(items):
total = 0
for item in items:
total += item.price
return total
# Low-quality code block (score: 2.0/10)
x = y
```
### ✅ 4. Quality Filtering
Filter out low-quality code blocks automatically:
```bash
# Keep only high-quality code (score >= 7.0)
python3 cli/pdf_extractor_poc.py input.pdf --min-quality 7.0
# Keep medium and high quality (score >= 4.0)
python3 cli/pdf_extractor_poc.py input.pdf --min-quality 4.0
# No filtering (default)
python3 cli/pdf_extractor_poc.py input.pdf
```
**Benefits:**
- Reduces noise in output
- Focuses on useful examples
- Improves downstream skill quality
### ✅ 5. Quality Statistics
New summary statistics show overall code quality:
```
📊 Code Quality Statistics:
Average quality: 6.8/10
Average confidence: 78.5%
Valid code blocks: 45/52 (86.5%)
High quality (7+): 28
Medium quality (4-7): 17
Low quality (<4): 7
```
---
## Output Format
### Enhanced Code Block Object
Each code block now includes quality metadata:
```json
{
"code": "def example():\n return True",
"language": "python",
"confidence": 0.85,
"quality_score": 7.5,
"is_valid": true,
"validation_issues": [],
"detection_method": "font",
"font": "Courier-New"
}
```
### Quality Statistics Object
Top-level summary of code quality:
```json
{
"quality_statistics": {
"average_quality": 6.8,
"average_confidence": 0.785,
"valid_code_blocks": 45,
"invalid_code_blocks": 7,
"validation_rate": 0.865,
"high_quality_blocks": 28,
"medium_quality_blocks": 17,
"low_quality_blocks": 7
}
}
```
---
## Usage Examples
### Basic Extraction with Quality Stats
```bash
python3 cli/pdf_extractor_poc.py manual.pdf -o output.json --pretty
```
**Output:**
```
✅ Extraction complete:
Total characters: 125,000
Code blocks found: 52
Headings found: 45
Images found: 12
Chunks created: 5
Chapters detected: 3
Languages detected: python, javascript, sql
📊 Code Quality Statistics:
Average quality: 6.8/10
Average confidence: 78.5%
Valid code blocks: 45/52 (86.5%)
High quality (7+): 28
Medium quality (4-7): 17
Low quality (<4): 7
```
### Filter Low-Quality Code
```bash
# Keep only high-quality examples
python3 cli/pdf_extractor_poc.py tutorial.pdf --min-quality 7.0 -v
# Verbose output shows filtering:
# 📄 Extracting from: tutorial.pdf
# ...
# Filtered out 12 low-quality code blocks (min_quality=7.0)
#
# ✅ Extraction complete:
# Code blocks found: 28 (after filtering)
```
### Inspect Quality Scores
```bash
# Extract and view quality scores
python3 cli/pdf_extractor_poc.py input.pdf -o output.json
# View quality scores with jq
cat output.json | jq '.pages[0].code_samples[] | {language, quality_score, is_valid}'
```
**Output:**
```json
{
"language": "python",
"quality_score": 8.5,
"is_valid": true
}
{
"language": "javascript",
"quality_score": 6.2,
"is_valid": true
}
{
"language": "unknown",
"quality_score": 2.1,
"is_valid": false
}
```
---
## Technical Implementation
### Language Detection with Confidence
```python
def detect_language_from_code(self, code):
"""Enhanced with weighted pattern matching"""
patterns = {
'python': [
(r'\bdef\s+\w+\s*\(', 3), # Weight: 3
(r'\bimport\s+\w+', 2), # Weight: 2
(r':\s*$', 1), # Weight: 1
],
# ... other languages
}
# Calculate scores for each language
scores = {}
for lang, lang_patterns in patterns.items():
score = 0
for pattern, weight in lang_patterns:
if re.search(pattern, code, re.IGNORECASE | re.MULTILINE):
score += weight
if score > 0:
scores[lang] = score
# Get best match
best_lang = max(scores, key=scores.get)
confidence = min(scores[best_lang] / 10.0, 1.0)
return best_lang, confidence
```
### Syntax Validation
```python
def validate_code_syntax(self, code, language):
"""Validate code syntax"""
issues = []
if language == 'python':
# Check indentation consistency
indent_chars = set()
for line in code.split('\n'):
if line.startswith(' '):
indent_chars.add('space')
elif line.startswith('\t'):
indent_chars.add('tab')
if len(indent_chars) > 1:
issues.append('Mixed tabs and spaces')
# Check balanced brackets
open_count = code.count('(') + code.count('[') + code.count('{')
close_count = code.count(')') + code.count(']') + code.count('}')
if abs(open_count - close_count) > 2:
issues.append('Unbalanced brackets')
# Check if it's actually natural language
common_words = ['the', 'and', 'for', 'with', 'this', 'that']
word_count = sum(1 for word in common_words if word in code.lower())
if word_count > 5:
issues.append('May be natural language, not code')
return len(issues) == 0, issues
```
### Quality Scoring
```python
def score_code_quality(self, code, language, confidence):
"""Score code quality (0-10)"""
score = 5.0 # Neutral baseline
# Factor 1: Language confidence
score += confidence * 2.0
# Factor 2: Code length (optimal range)
code_length = len(code.strip())
if 20 <= code_length <= 500:
score += 1.0
# Factor 3: Has function/class definitions
if re.search(r'\b(def|function|class|func)\b', code):
score += 1.5
# Factor 4: Meaningful variable names
meaningful_vars = re.findall(r'\b[a-z_][a-z0-9_]{3,}\b', code.lower())
if len(meaningful_vars) >= 2:
score += 1.0
# Factor 5: Syntax validation
is_valid, issues = self.validate_code_syntax(code, language)
if is_valid:
score += 1.0
else:
score -= len(issues) * 0.5
return max(0, min(10, score)) # Clamp to 0-10
```
---
## Performance Impact
### Overhead Analysis
| Operation | Time per page | Impact |
|-----------|---------------|--------|
| Confidence scoring | +0.2ms | Negligible |
| Syntax validation | +0.5ms | Negligible |
| Quality scoring | +0.3ms | Negligible |
| **Total overhead** | **+1.0ms** | **<2%** |
**Benchmark:**
- Small PDF (10 pages): +10ms total (~1% overhead)
- Medium PDF (100 pages): +100ms total (~2% overhead)
- Large PDF (500 pages): +500ms total (~2% overhead)
### Memory Usage
- Quality metadata adds ~200 bytes per code block
- Statistics add ~500 bytes to output
- **Impact:** Negligible (<1% increase)
---
## Comparison: Before vs After
| Metric | Before (B1.3) | After (B1.4) | Improvement |
|--------|---------------|--------------|-------------|
| Language detection | Single return | Lang + confidence | ✅ More reliable |
| Syntax validation | None | Multiple checks | ✅ Filters false positives |
| Quality scoring | None | 0-10 scale | ✅ Ranks code blocks |
| False positives | ~15-20% | ~3-5% | ✅ 75% reduction |
| Code quality avg | Unknown | Measurable | ✅ Trackable |
| Filtering | None | Automatic | ✅ Cleaner output |
---
## Testing
### Test Quality Scoring
```bash
# Create test PDF with various code qualities
# - High-quality: Complete function with meaningful names
# - Medium-quality: Simple variable assignments
# - Low-quality: Natural language text
python3 cli/pdf_extractor_poc.py test.pdf -o test.json -v
# Check quality scores
cat test.json | jq '.pages[].code_samples[] | {language, quality_score}'
```
**Expected Results:**
```json
{"language": "python", "quality_score": 8.5}
{"language": "javascript", "quality_score": 6.2}
{"language": "unknown", "quality_score": 1.8}
```
### Test Validation
```bash
# Check validation results
cat test.json | jq '.pages[].code_samples[] | select(.is_valid == false)'
```
**Should show:**
- Empty code blocks
- Natural language misdetected as code
- Code with severe syntax errors
### Test Filtering
```bash
# Extract with different quality thresholds
python3 cli/pdf_extractor_poc.py test.pdf --min-quality 7.0 -o high_quality.json
python3 cli/pdf_extractor_poc.py test.pdf --min-quality 4.0 -o medium_quality.json
python3 cli/pdf_extractor_poc.py test.pdf --min-quality 0.0 -o all_quality.json
# Compare counts
echo "High quality:"; cat high_quality.json | jq '[.pages[].code_samples[]] | length'
echo "Medium+:"; cat medium_quality.json | jq '[.pages[].code_samples[]] | length'
echo "All:"; cat all_quality.json | jq '[.pages[].code_samples[]] | length'
```
---
## Limitations
### Current Limitations
1. **Validation is heuristic-based**
- No AST parsing (yet)
- Some edge cases may be missed
- Language-specific validation only for Python, JS, Java, C
2. **Quality scoring is subjective**
- Based on heuristics, not compilation
- May not match human judgment perfectly
- Tuned for documentation examples, not production code
3. **Confidence scoring is pattern-based**
- No machine learning
- Limited to defined patterns
- May struggle with uncommon languages
### Known Issues
1. **Short Code Snippets**
- May score lower than deserved
- Example: `x = 5` is valid but scores low
2. **Comments-Heavy Code**
- Well-commented code may be penalized
- Workaround: Adjust comment ratio threshold
3. **Domain-Specific Languages**
- Not covered by pattern detection
- Will be marked as 'unknown'
---
## Future Enhancements
### Potential Improvements
1. **AST-Based Validation**
- Use Python's `ast` module for Python code
- Use esprima/acorn for JavaScript
- Actual syntax parsing instead of heuristics
2. **Machine Learning Detection**
- Train classifier on code vs non-code
- More accurate language detection
- Context-aware quality scoring
3. **Custom Quality Metrics**
- User-defined quality factors
- Domain-specific scoring
- Configurable weights
4. **More Language Support**
- Add TypeScript, Dart, Lua, etc.
- Better pattern coverage
- Language-specific validation
---
## Integration with Skill Seeker
### Improved Skill Quality
With B1.4 enhancements, PDF-based skills will have:
1. **Higher quality code examples**
- Automatic filtering of noise
- Only meaningful snippets included
2. **Better categorization**
- Confidence scores help categorization
- Language-specific references
3. **Validation feedback**
- Know which code blocks may have issues
- Fix before packaging skill
### Example Workflow
```bash
# Step 1: Extract with high-quality filter
python3 cli/pdf_extractor_poc.py manual.pdf --min-quality 7.0 -o manual.json -v
# Step 2: Review quality statistics
cat manual.json | jq '.quality_statistics'
# Step 3: Inspect any invalid blocks
cat manual.json | jq '.pages[].code_samples[] | select(.is_valid == false)'
# Step 4: Build skill (future task B1.6)
python3 cli/pdf_scraper.py --from-json manual.json
```
---
## Conclusion
Task B1.4 successfully implements:
- ✅ Confidence-based language detection
- ✅ Syntax validation for common languages
- ✅ Quality scoring (0-10 scale)
- ✅ Automatic quality filtering
- ✅ Comprehensive quality statistics
**Impact:**
- 75% reduction in false positives
- More reliable code extraction
- Better skill quality
- Measurable code quality metrics
**Performance:** <2% overhead (negligible)
**Compatibility:** Backward compatible (existing fields preserved)
**Ready for B1.5:** Image extraction from PDFs
---
**Task Completed:** October 21, 2025
**Next Task:** B1.5 - Add PDF image extraction (diagrams, screenshots)
assets/repo/Skill_Seekers-development/docs/TERMINAL_SELECTION.md
+94
View File
@@ -0,0 +1,94 @@
# Terminal Selection Guide
When using `--enhance-local`, Skill Seeker opens a new terminal window to run Claude Code. This guide explains how to control which terminal app is used.
## Priority Order
The script automatically detects which terminal to use in this order:
1. **`SKILL_SEEKER_TERMINAL` environment variable** (highest priority)
2. **`TERM_PROGRAM` environment variable** (inherit current terminal)
3. **Terminal.app** (fallback default)
## Setting Your Preferred Terminal
### Option 1: Set Environment Variable (Recommended)
Add this to your shell config (`~/.zshrc` or `~/.bashrc`):
```bash
# For Ghostty users
export SKILL_SEEKER_TERMINAL="Ghostty"
# For iTerm users
export SKILL_SEEKER_TERMINAL="iTerm"
# For WezTerm users
export SKILL_SEEKER_TERMINAL="WezTerm"
```
Then reload your shell:
```bash
source ~/.zshrc # or source ~/.bashrc
```
### Option 2: Set Per-Session
Set the variable before running the command:
```bash
SKILL_SEEKER_TERMINAL="Ghostty" python3 cli/doc_scraper.py --config configs/react.json --enhance-local
```
### Option 3: Inherit Current Terminal (Automatic)
If you run the script from Ghostty, iTerm2, or WezTerm, it will automatically open the enhancement in the same terminal app.
**Note:** IDE terminals (VS Code, Zed, JetBrains) use unique `TERM_PROGRAM` values, so they fall back to Terminal.app unless you set `SKILL_SEEKER_TERMINAL`.
## Supported Terminals
- **Ghostty** (`ghostty`)
- **iTerm2** (`iTerm.app`)
- **Terminal.app** (`Apple_Terminal`)
- **WezTerm** (`WezTerm`)
## Example Output
When terminal detection works:
```
🚀 Launching Claude Code in new terminal...
Using terminal: Ghostty (from SKILL_SEEKER_TERMINAL)
```
When running from an IDE terminal:
```
🚀 Launching Claude Code in new terminal...
⚠️ unknown TERM_PROGRAM (zed)
→ Using Terminal.app as fallback
```
**Tip:** Set `SKILL_SEEKER_TERMINAL` to avoid the fallback behavior.
## Troubleshooting
**Q: The wrong terminal opens even though I set `SKILL_SEEKER_TERMINAL`**
A: Make sure you reloaded your shell after editing `~/.zshrc`:
```bash
source ~/.zshrc
```
**Q: I want to use a different terminal temporarily**
A: Set the variable inline:
```bash
SKILL_SEEKER_TERMINAL="iTerm" python3 cli/doc_scraper.py --enhance-local ...
```
**Q: Can I use a custom terminal app?**
A: Yes! Just use the app name as it appears in `/Applications/`:
```bash
export SKILL_SEEKER_TERMINAL="Alacritty"
```
assets/repo/Skill_Seekers-development/docs/TESTING.md
+716
View File
@@ -0,0 +1,716 @@
# Testing Guide for Skill Seeker
Comprehensive testing documentation for the Skill Seeker project.
## Quick Start
```bash
# Run all tests
python3 run_tests.py
# Run all tests with verbose output
python3 run_tests.py -v
# Run specific test suite
python3 run_tests.py --suite config
python3 run_tests.py --suite features
python3 run_tests.py --suite integration
# Stop on first failure
python3 run_tests.py --failfast
# List all available tests
python3 run_tests.py --list
```
## Test Structure
```
tests/
├── __init__.py # Test package marker
├── test_config_validation.py # Config validation tests (30+ tests)
├── test_scraper_features.py # Core feature tests (25+ tests)
├── test_integration.py # Integration tests (15+ tests)
├── test_pdf_extractor.py # PDF extraction tests (23 tests)
├── test_pdf_scraper.py # PDF workflow tests (18 tests)
└── test_pdf_advanced_features.py # PDF advanced features (26 tests) NEW
```
## Test Suites
### 1. Config Validation Tests (`test_config_validation.py`)
Tests the `validate_config()` function with comprehensive coverage.
**Test Categories:**
- ✅ Valid configurations (minimal and complete)
- ✅ Missing required fields (`name`, `base_url`)
- ✅ Invalid name formats (special characters)
- ✅ Valid name formats (alphanumeric, hyphens, underscores)
- ✅ Invalid URLs (missing protocol)
- ✅ Valid URL protocols (http, https)
- ✅ Selector validation (structure and recommended fields)
- ✅ URL patterns validation (include/exclude lists)
- ✅ Categories validation (structure and keywords)
- ✅ Rate limit validation (range 0-10, type checking)
- ✅ Max pages validation (range 1-10000, type checking)
- ✅ Start URLs validation (format and protocol)
**Example Test:**
```python
def test_valid_complete_config(self):
"""Test valid complete configuration"""
config = {
'name': 'godot',
'base_url': 'https://docs.godotengine.org/en/stable/',
'selectors': {
'main_content': 'div[role="main"]',
'title': 'title',
'code_blocks': 'pre code'
},
'rate_limit': 0.5,
'max_pages': 500
}
errors = validate_config(config)
self.assertEqual(len(errors), 0)
```
**Running:**
```bash
python3 run_tests.py --suite config -v
```
---
### 2. Scraper Features Tests (`test_scraper_features.py`)
Tests core scraper functionality including URL validation, language detection, pattern extraction, and categorization.
**Test Categories:**
**URL Validation:**
- ✅ URL matching include patterns
- ✅ URL matching exclude patterns
- ✅ Different domain rejection
- ✅ No pattern configuration
**Language Detection:**
- ✅ Detection from CSS classes (`language-*`, `lang-*`)
- ✅ Detection from parent elements
- ✅ Python detection (import, from, def)
- ✅ JavaScript detection (const, let, arrow functions)
- ✅ GDScript detection (func, var)
- ✅ C++ detection (#include, int main)
- ✅ Unknown language fallback
**Pattern Extraction:**
- ✅ Extraction with "Example:" marker
- ✅ Extraction with "Usage:" marker
- ✅ Pattern limit (max 5)
**Categorization:**
- ✅ Categorization by URL keywords
- ✅ Categorization by title keywords
- ✅ Categorization by content keywords
- ✅ Fallback to "other" category
- ✅ Empty category removal
**Text Cleaning:**
- ✅ Multiple spaces normalization
- ✅ Newline normalization
- ✅ Tab normalization
- ✅ Whitespace stripping
**Example Test:**
```python
def test_detect_python_from_heuristics(self):
"""Test Python detection from code content"""
html = '<code>import os\nfrom pathlib import Path</code>'
elem = BeautifulSoup(html, 'html.parser').find('code')
lang = self.converter.detect_language(elem, elem.get_text())
self.assertEqual(lang, 'python')
```
**Running:**
```bash
python3 run_tests.py --suite features -v
```
---
### 3. Integration Tests (`test_integration.py`)
Tests complete workflows and interactions between components.
**Test Categories:**
**Dry-Run Mode:**
- ✅ No directories created in dry-run mode
- ✅ Dry-run flag properly set
- ✅ Normal mode creates directories
**Config Loading:**
- ✅ Load valid configuration files
- ✅ Invalid JSON error handling
- ✅ Nonexistent file error handling
- ✅ Validation errors during load
**Real Config Validation:**
- ✅ Godot config validation
- ✅ React config validation
- ✅ Vue config validation
- ✅ Django config validation
- ✅ FastAPI config validation
- ✅ Steam Economy config validation
**URL Processing:**
- ✅ URL normalization
- ✅ Start URLs fallback to base_url
- ✅ Multiple start URLs handling
**Content Extraction:**
- ✅ Empty content handling
- ✅ Basic content extraction
- ✅ Code sample extraction with language detection
**Example Test:**
```python
def test_dry_run_no_directories_created(self):
"""Test that dry-run mode doesn't create directories"""
converter = DocToSkillConverter(self.config, dry_run=True)
data_dir = Path(f"output/{self.config['name']}_data")
skill_dir = Path(f"output/{self.config['name']}")
self.assertFalse(data_dir.exists())
self.assertFalse(skill_dir.exists())
```
**Running:**
```bash
python3 run_tests.py --suite integration -v
```
---
### 4. PDF Extraction Tests (`test_pdf_extractor.py`) **NEW**
Tests PDF content extraction functionality (B1.2-B1.5).
**Note:** These tests require PyMuPDF (`pip install PyMuPDF`). They will be skipped if not installed.
**Test Categories:**
**Language Detection (5 tests):**
- ✅ Python detection with confidence scoring
- ✅ JavaScript detection with confidence
- ✅ C++ detection with confidence
- ✅ Unknown language returns low confidence
- ✅ Confidence always between 0 and 1
**Syntax Validation (5 tests):**
- ✅ Valid Python syntax validation
- ✅ Invalid Python indentation detection
- ✅ Unbalanced brackets detection
- ✅ Valid JavaScript syntax validation
- ✅ Natural language fails validation
**Quality Scoring (4 tests):**
- ✅ Quality score between 0 and 10
- ✅ High-quality code gets good score (>7)
- ✅ Low-quality code gets low score (<4)
- ✅ Quality considers multiple factors
**Chapter Detection (4 tests):**
- ✅ Detect chapters with numbers
- ✅ Detect uppercase chapter headers
- ✅ Detect section headings (e.g., "2.1")
- ✅ Normal text not detected as chapter
**Code Block Merging (2 tests):**
- ✅ Merge code blocks split across pages
- ✅ Don't merge different languages
**Code Detection Methods (2 tests):**
- ✅ Pattern-based detection (keywords)
- ✅ Indent-based detection
**Quality Filtering (1 test):**
- ✅ Filter by minimum quality threshold
**Example Test:**
```python
def test_detect_python_with_confidence(self):
"""Test Python detection returns language and confidence"""
extractor = self.PDFExtractor.__new__(self.PDFExtractor)
code = "def hello():\n print('world')\n return True"
language, confidence = extractor.detect_language_from_code(code)
self.assertEqual(language, "python")
self.assertGreater(confidence, 0.7)
self.assertLessEqual(confidence, 1.0)
```
**Running:**
```bash
python3 -m pytest tests/test_pdf_extractor.py -v
```
---
### 5. PDF Workflow Tests (`test_pdf_scraper.py`) **NEW**
Tests PDF to skill conversion workflow (B1.6).
**Note:** These tests require PyMuPDF (`pip install PyMuPDF`). They will be skipped if not installed.
**Test Categories:**
**PDFToSkillConverter (3 tests):**
- ✅ Initialization with name and PDF path
- ✅ Initialization with config file
- ✅ Requires name or config_path
**Categorization (3 tests):**
- ✅ Categorize by keywords
- ✅ Categorize by chapters
- ✅ Handle missing chapters
**Skill Building (3 tests):**
- ✅ Create required directory structure
- ✅ Create SKILL.md with metadata
- ✅ Create reference files for categories
**Code Block Handling (2 tests):**
- ✅ Include code blocks in references
- ✅ Prefer high-quality code
**Image Handling (2 tests):**
- ✅ Save images to assets directory
- ✅ Reference images in markdown
**Error Handling (3 tests):**
- ✅ Handle missing PDF files
- ✅ Handle invalid config JSON
- ✅ Handle missing required config fields
**JSON Workflow (2 tests):**
- ✅ Load from extracted JSON
- ✅ Build from JSON without extraction
**Example Test:**
```python
def test_build_skill_creates_structure(self):
"""Test that build_skill creates required directory structure"""
converter = self.PDFToSkillConverter(
name="test_skill",
pdf_path="test.pdf",
output_dir=self.temp_dir
)
converter.extracted_data = {
"pages": [{"page_number": 1, "text": "Test", "code_blocks": [], "images": []}],
"total_pages": 1
}
converter.categories = {"test": [converter.extracted_data["pages"][0]]}
converter.build_skill()
skill_dir = Path(self.temp_dir) / "test_skill"
self.assertTrue(skill_dir.exists())
self.assertTrue((skill_dir / "references").exists())
self.assertTrue((skill_dir / "scripts").exists())
self.assertTrue((skill_dir / "assets").exists())
```
**Running:**
```bash
python3 -m pytest tests/test_pdf_scraper.py -v
```
---
### 6. PDF Advanced Features Tests (`test_pdf_advanced_features.py`) **NEW**
Tests advanced PDF features (Priority 2 & 3).
**Note:** These tests require PyMuPDF (`pip install PyMuPDF`). OCR tests also require pytesseract and Pillow. They will be skipped if not installed.
**Test Categories:**
**OCR Support (5 tests):**
- ✅ OCR flag initialization
- ✅ OCR disabled behavior
- ✅ OCR only triggers for minimal text
- ✅ Warning when pytesseract unavailable
- ✅ OCR extraction triggered correctly
**Password Protection (4 tests):**
- ✅ Password parameter initialization
- ✅ Encrypted PDF detection
- ✅ Wrong password handling
- ✅ Missing password error
**Table Extraction (5 tests):**
- ✅ Table extraction flag initialization
- ✅ No extraction when disabled
- ✅ Basic table extraction
- ✅ Multiple tables per page
- ✅ Error handling during extraction
**Caching (5 tests):**
- ✅ Cache initialization
- ✅ Set and get cached values
- ✅ Cache miss returns None
- ✅ Caching can be disabled
- ✅ Cache overwrite
**Parallel Processing (4 tests):**
- ✅ Parallel flag initialization
- ✅ Disabled by default
- ✅ Worker count auto-detection
- ✅ Custom worker count
**Integration (3 tests):**
- ✅ Full initialization with all features
- ✅ Various feature combinations
- ✅ Page data includes tables
**Example Test:**
```python
def test_table_extraction_basic(self):
"""Test basic table extraction"""
extractor = self.PDFExtractor.__new__(self.PDFExtractor)
extractor.extract_tables = True
extractor.verbose = False
# Create mock table
mock_table = Mock()
mock_table.extract.return_value = [
["Header 1", "Header 2", "Header 3"],
["Data 1", "Data 2", "Data 3"]
]
mock_table.bbox = (0, 0, 100, 100)
mock_tables = Mock()
mock_tables.tables = [mock_table]
mock_page = Mock()
mock_page.find_tables.return_value = mock_tables
tables = extractor.extract_tables_from_page(mock_page)
self.assertEqual(len(tables), 1)
self.assertEqual(tables[0]['row_count'], 2)
self.assertEqual(tables[0]['col_count'], 3)
```
**Running:**
```bash
python3 -m pytest tests/test_pdf_advanced_features.py -v
```
---
## Test Runner Features
The custom test runner (`run_tests.py`) provides:
### Colored Output
- 🟢 Green for passing tests
- 🔴 Red for failures and errors
- 🟡 Yellow for skipped tests
### Detailed Summary
```
======================================================================
TEST SUMMARY
======================================================================
Total Tests: 70
✓ Passed: 68
✗ Failed: 2
⊘ Skipped: 0
Success Rate: 97.1%
Test Breakdown by Category:
TestConfigValidation: 28/30 passed
TestURLValidation: 6/6 passed
TestLanguageDetection: 10/10 passed
TestPatternExtraction: 3/3 passed
TestCategorization: 5/5 passed
TestDryRunMode: 3/3 passed
TestConfigLoading: 4/4 passed
TestRealConfigFiles: 6/6 passed
TestContentExtraction: 3/3 passed
======================================================================
```
### Command-Line Options
```bash
# Verbose output (show each test name)
python3 run_tests.py -v
# Quiet output (minimal)
python3 run_tests.py -q
# Stop on first failure
python3 run_tests.py --failfast
# Run specific suite
python3 run_tests.py --suite config
# List all tests
python3 run_tests.py --list
```
---
## Running Individual Tests
### Run Single Test File
```bash
python3 -m unittest tests.test_config_validation
python3 -m unittest tests.test_scraper_features
python3 -m unittest tests.test_integration
```
### Run Single Test Class
```bash
python3 -m unittest tests.test_config_validation.TestConfigValidation
python3 -m unittest tests.test_scraper_features.TestLanguageDetection
```
### Run Single Test Method
```bash
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_valid_complete_config
python3 -m unittest tests.test_scraper_features.TestLanguageDetection.test_detect_python_from_heuristics
```
---
## Test Coverage
### Current Coverage
| Component | Tests | Coverage |
|-----------|-------|----------|
| Config Validation | 30+ | 100% |
| URL Validation | 6 | 95% |
| Language Detection | 10 | 90% |
| Pattern Extraction | 3 | 85% |
| Categorization | 5 | 90% |
| Text Cleaning | 4 | 100% |
| Dry-Run Mode | 3 | 100% |
| Config Loading | 4 | 95% |
| Real Configs | 6 | 100% |
| Content Extraction | 3 | 80% |
| **PDF Extraction** | **23** | **90%** |
| **PDF Workflow** | **18** | **85%** |
| **PDF Advanced Features** | **26** | **95%** |
**Total: 142 tests (75 passing + 67 PDF tests)**
**Note:** PDF tests (67 total) require PyMuPDF and will be skipped if not installed. When PyMuPDF is available, all 142 tests run.
### Not Yet Covered
- Network operations (actual scraping)
- Enhancement scripts (`enhance_skill.py`, `enhance_skill_local.py`)
- Package creation (`package_skill.py`)
- Interactive mode
- SKILL.md generation
- Reference file creation
- PDF extraction with real PDF files (tests use mocked data)
---
## Writing New Tests
### Test Template
```python
#!/usr/bin/env python3
"""
Test suite for [feature name]
Tests [description of what's being tested]
"""
import sys
import os
import unittest
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from doc_scraper import DocToSkillConverter
class TestYourFeature(unittest.TestCase):
"""Test [feature] functionality"""
def setUp(self):
"""Set up test fixtures"""
self.config = {
'name': 'test',
'base_url': 'https://example.com/',
'selectors': {
'main_content': 'article',
'title': 'h1',
'code_blocks': 'pre code'
},
'rate_limit': 0.1,
'max_pages': 10
}
self.converter = DocToSkillConverter(self.config, dry_run=True)
def tearDown(self):
"""Clean up after tests"""
pass
def test_your_feature(self):
"""Test description"""
# Arrange
test_input = "something"
# Act
result = self.converter.some_method(test_input)
# Assert
self.assertEqual(result, expected_value)
if __name__ == '__main__':
unittest.main()
```
### Best Practices
1. **Use descriptive test names**: `test_valid_name_formats` not `test1`
2. **Follow AAA pattern**: Arrange, Act, Assert
3. **One assertion per test** when possible
4. **Test edge cases**: empty inputs, invalid inputs, boundary values
5. **Use setUp/tearDown**: for common initialization and cleanup
6. **Mock external dependencies**: don't make real network calls
7. **Keep tests independent**: tests should not depend on each other
8. **Use dry_run=True**: for converter tests to avoid file creation
---
## Continuous Integration
### GitHub Actions (Future)
```yaml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: '3.7'
- run: pip install requests beautifulsoup4
- run: python3 run_tests.py
```
---
## Troubleshooting
### Tests Fail with Import Errors
```bash
# Make sure you're in the repository root
cd /path/to/Skill_Seekers
# Run tests from root directory
python3 run_tests.py
```
### Tests Create Output Directories
```bash
# Clean up test artifacts
rm -rf output/test-*
# Make sure tests use dry_run=True
# Check test setUp methods
```
### Specific Test Keeps Failing
```bash
# Run only that test with verbose output
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_name -v
# Check the error message carefully
# Verify test expectations match implementation
```
---
## Performance
Test execution times:
- **Config Validation**: ~0.1 seconds (30 tests)
- **Scraper Features**: ~0.3 seconds (25 tests)
- **Integration Tests**: ~0.5 seconds (15 tests)
- **Total**: ~1 second (70 tests)
---
## Contributing Tests
When adding new features:
1. Write tests **before** implementing the feature (TDD)
2. Ensure tests cover:
- ✅ Happy path (valid inputs)
- ✅ Edge cases (empty, null, boundary values)
- ✅ Error cases (invalid inputs)
3. Run tests before committing:
```bash
python3 run_tests.py
```
4. Aim for >80% coverage for new code
---
## Additional Resources
- **unittest documentation**: https://docs.python.org/3/library/unittest.html
- **pytest** (alternative): https://pytest.org/ (more powerful, but requires installation)
- **Test-Driven Development**: https://en.wikipedia.org/wiki/Test-driven_development
---
## Summary
✅ **142 comprehensive tests** covering all major features (75 + 67 PDF)
✅ **PDF support testing** with 67 tests for B1 tasks + Priority 2 & 3
✅ **Colored test runner** with detailed summaries
✅ **Fast execution** (~1 second for full suite)
✅ **Easy to extend** with clear patterns and templates
✅ **Good coverage** of critical paths
**PDF Tests Status:**
- 23 tests for PDF extraction (language detection, syntax validation, quality scoring, chapter detection)
- 18 tests for PDF workflow (initialization, categorization, skill building, code/image handling)
- **26 tests for advanced features (OCR, passwords, tables, parallel, caching)** NEW!
- Tests are skipped gracefully when PyMuPDF is not installed
- Full test coverage when PyMuPDF + optional dependencies are available
**Advanced PDF Features Tested:**
- ✅ OCR support for scanned PDFs (5 tests)
- ✅ Password-protected PDFs (4 tests)
- ✅ Table extraction (5 tests)
- ✅ Parallel processing (4 tests)
- ✅ Caching (5 tests)
- ✅ Integration (3 tests)
Run tests frequently to catch bugs early! 🚀
assets/repo/Skill_Seekers-development/docs/TEST_MCP_IN_CLAUDE_CODE.md
+342
View File
@@ -0,0 +1,342 @@
# Testing MCP Server in Claude Code
This guide shows you how to test the Skill Seeker MCP server **through actual Claude Code** using the MCP protocol (not just Python function calls).
## Important: What We Tested vs What You Need to Test
### What I Tested (Python Direct Calls) ✅
I tested the MCP server **functions** by calling them directly with Python:
```python
await server.list_configs_tool({})
await server.generate_config_tool({...})
```
This verified the **code works**, but didn't test the **MCP protocol integration**.
### What You Need to Test (Actual MCP Protocol) 🎯
You need to test via **Claude Code** using the MCP protocol:
```
In Claude Code:
> List all available configs
> mcp__skill-seeker__list_configs
```
This verifies the **full integration** works.
## Setup Instructions
### Step 1: Configure Claude Code
Create the MCP configuration file:
```bash
# Create config directory
mkdir -p ~/.config/claude-code
# Create/edit MCP configuration
nano ~/.config/claude-code/mcp.json
```
Add this configuration (replace `/path/to/` with your actual path):
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python3",
"args": [
"/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/Skill_Seekers/skill_seeker_mcp/server.py"
],
"cwd": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/Skill_Seekers"
}
}
}
```
Or use the setup script:
```bash
./setup_mcp.sh
```
### Step 2: Restart Claude Code
**IMPORTANT:** Completely quit and restart Claude Code (don't just close the window).
### Step 3: Verify MCP Server Loaded
In Claude Code, check if the server loaded:
```
Show me all available MCP tools
```
You should see 6 tools with the prefix `mcp__skill-seeker__`:
- `mcp__skill-seeker__list_configs`
- `mcp__skill-seeker__generate_config`
- `mcp__skill-seeker__validate_config`
- `mcp__skill-seeker__estimate_pages`
- `mcp__skill-seeker__scrape_docs`
- `mcp__skill-seeker__package_skill`
## Testing All 6 MCP Tools
### Test 1: list_configs
**In Claude Code, type:**
```
List all available Skill Seeker configs
```
**Or explicitly:**
```
Use mcp__skill-seeker__list_configs
```
**Expected Output:**
```
📋 Available Configs:
• django.json
• fastapi.json
• godot.json
• react.json
• vue.json
...
```
### Test 2: generate_config
**In Claude Code, type:**
```
Generate a config for Astro documentation at https://docs.astro.build with max 15 pages
```
**Or explicitly:**
```
Use mcp__skill-seeker__generate_config with:
- name: astro-test
- url: https://docs.astro.build
- description: Astro framework testing
- max_pages: 15
```
**Expected Output:**
```
✅ Config created: configs/astro-test.json
```
### Test 3: validate_config
**In Claude Code, type:**
```
Validate the astro-test config
```
**Or explicitly:**
```
Use mcp__skill-seeker__validate_config for configs/astro-test.json
```
**Expected Output:**
```
✅ Config is valid!
Name: astro-test
Base URL: https://docs.astro.build
Max pages: 15
```
### Test 4: estimate_pages
**In Claude Code, type:**
```
Estimate pages for the astro-test config
```
**Or explicitly:**
```
Use mcp__skill-seeker__estimate_pages for configs/astro-test.json
```
**Expected Output:**
```
📊 ESTIMATION RESULTS
Estimated Total: ~25 pages
Recommended max_pages: 75
```
### Test 5: scrape_docs
**In Claude Code, type:**
```
Scrape docs using the astro-test config
```
**Or explicitly:**
```
Use mcp__skill-seeker__scrape_docs with configs/astro-test.json
```
**Expected Output:**
```
✅ Skill built: output/astro-test/
Scraped X pages
Created Y categories
```
### Test 6: package_skill
**In Claude Code, type:**
```
Package the astro-test skill
```
**Or explicitly:**
```
Use mcp__skill-seeker__package_skill for output/astro-test/
```
**Expected Output:**
```
✅ Package created: output/astro-test.zip
Size: X KB
```
## Complete Workflow Test
Test the entire workflow in Claude Code with natural language:
```
Step 1:
> List all available configs
Step 2:
> Generate config for Svelte at https://svelte.dev/docs with description "Svelte framework" and max 20 pages
Step 3:
> Validate configs/svelte.json
Step 4:
> Estimate pages for configs/svelte.json
Step 5:
> Scrape docs using configs/svelte.json
Step 6:
> Package skill at output/svelte/
```
Expected result: `output/svelte.zip` ready to upload to Claude!
## Troubleshooting
### Issue: Tools Not Appearing
**Symptoms:**
- Claude Code doesn't recognize skill-seeker commands
- No `mcp__skill-seeker__` tools listed
**Solutions:**
1. Check configuration exists:
```bash
cat ~/.config/claude-code/mcp.json
```
2. Verify server can start:
```bash
cd /path/to/Skill_Seekers
python3 skill_seeker_mcp/server.py
# Should start without errors (Ctrl+C to exit)
```
3. Check dependencies installed:
```bash
pip3 list | grep mcp
# Should show: mcp x.x.x
```
4. Completely restart Claude Code (quit and reopen)
5. Check Claude Code logs:
- macOS: `~/Library/Logs/Claude Code/`
- Linux: `~/.config/claude-code/logs/`
### Issue: "Permission Denied"
```bash
chmod +x skill_seeker_mcp/server.py
```
### Issue: "Module Not Found"
```bash
pip3 install -r skill_seeker_mcp/requirements.txt
pip3 install requests beautifulsoup4
```
## Verification Checklist
Use this checklist to verify MCP integration:
- [ ] Configuration file created at `~/.config/claude-code/mcp.json`
- [ ] Repository path in config is absolute and correct
- [ ] Python dependencies installed (`mcp`, `requests`, `beautifulsoup4`)
- [ ] Server starts without errors when run manually
- [ ] Claude Code completely restarted (quit and reopened)
- [ ] Tools appear when asking "show me all MCP tools"
- [ ] Tools have `mcp__skill-seeker__` prefix
- [ ] Can list configs successfully
- [ ] Can generate a test config
- [ ] Can scrape and package a small skill
## What Makes This Different from My Tests
| What I Tested | What You Should Test |
|---------------|---------------------|
| Python function calls | Claude Code MCP protocol |
| `await server.list_configs_tool({})` | Natural language in Claude Code |
| Direct Python imports | Full MCP server integration |
| Validates code works | Validates Claude Code integration |
| Quick unit testing | Real-world usage testing |
## Success Criteria
✅ **MCP Integration is Working When:**
1. You can ask Claude Code to "list all available configs"
2. Claude Code responds with the actual config list
3. You can generate, validate, scrape, and package skills
4. All through natural language commands in Claude Code
5. No Python code needed - just conversation!
## Next Steps After Successful Testing
Once MCP integration works:
1. **Create your first skill:**
```
> Generate config for TailwindCSS at https://tailwindcss.com/docs
> Scrape docs using configs/tailwind.json
> Package skill at output/tailwind/
```
2. **Upload to Claude:**
- Take the generated `.zip` file
- Upload to Claude.ai
- Start using your new skill!
3. **Share feedback:**
- Report any issues on GitHub
- Share successful skills created
- Suggest improvements
## Reference
- **Full Setup Guide:** [docs/MCP_SETUP.md](docs/MCP_SETUP.md)
- **MCP Documentation:** [mcp/README.md](mcp/README.md)
- **Main README:** [README.md](README.md)
- **Setup Script:** `./setup_mcp.sh`
---
**Important:** This document is for testing the **actual MCP protocol integration** with Claude Code, not just the Python functions. Make sure you're testing through Claude Code's UI, not Python scripts!
assets/repo/Skill_Seekers-development/docs/UNIFIED_SCRAPING.md
+633
View File
@@ -0,0 +1,633 @@
# Unified Multi-Source Scraping
**Version:** 2.0 (Feature complete as of October 2025)
## Overview
Unified multi-source scraping allows you to combine knowledge from multiple sources into a single comprehensive Claude skill. Instead of choosing between documentation, GitHub repositories, or PDF manuals, you can now extract and intelligently merge information from all of them.
## Why Unified Scraping?
**The Problem**: Documentation and code often drift apart over time. Official docs might be outdated, missing features that exist in code, or documenting features that have been removed. Separately scraping docs and code creates two incomplete skills.
**The Solution**: Unified scraping:
- Extracts information from multiple sources (documentation, GitHub, PDFs)
- **Detects conflicts** between documentation and actual code implementation
- **Intelligently merges** conflicting information with transparency
- **Highlights discrepancies** with inline warnings (⚠️)
- Creates a single, comprehensive skill that shows the complete picture
## Quick Start
### 1. Create a Unified Config
Create a config file with multiple sources:
```json
{
"name": "react",
"description": "Complete React knowledge from docs + codebase",
"merge_mode": "rule-based",
"sources": [
{
"type": "documentation",
"base_url": "https://react.dev/",
"extract_api": true,
"max_pages": 200
},
{
"type": "github",
"repo": "facebook/react",
"include_code": true,
"code_analysis_depth": "surface",
"max_issues": 100
}
]
}
```
### 2. Scrape and Build
```bash
python3 cli/unified_scraper.py --config configs/react_unified.json
```
The tool will:
1. ✅ **Phase 1**: Scrape all sources (docs + GitHub)
2. ✅ **Phase 2**: Detect conflicts between sources
3. ✅ **Phase 3**: Merge conflicts intelligently
4. ✅ **Phase 4**: Build unified skill with conflict transparency
### 3. Package and Upload
```bash
python3 cli/package_skill.py output/react/
```
## Config Format
### Unified Config Structure
```json
{
"name": "skill-name",
"description": "When to use this skill",
"merge_mode": "rule-based|claude-enhanced",
"sources": [
{
"type": "documentation|github|pdf",
...source-specific fields...
}
]
}
```
### Documentation Source
```json
{
"type": "documentation",
"base_url": "https://docs.example.com/",
"extract_api": true,
"selectors": {
"main_content": "article",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": [],
"exclude": ["/blog/"]
},
"categories": {
"getting_started": ["intro", "tutorial"],
"api": ["api", "reference"]
},
"rate_limit": 0.5,
"max_pages": 200
}
```
### GitHub Source
```json
{
"type": "github",
"repo": "owner/repo",
"github_token": "ghp_...",
"include_issues": true,
"max_issues": 100,
"include_changelog": true,
"include_releases": true,
"include_code": true,
"code_analysis_depth": "surface|deep|full",
"file_patterns": [
"src/**/*.js",
"lib/**/*.ts"
]
}
```
**Code Analysis Depth**:
- `surface` (default): Basic structure, no code analysis
- `deep`: Extract class/function signatures, parameters, return types
- `full`: Complete AST analysis (expensive)
### PDF Source
```json
{
"type": "pdf",
"path": "/path/to/manual.pdf",
"extract_tables": false,
"ocr": false,
"password": "optional-password"
}
```
## Conflict Detection
The unified scraper automatically detects 4 types of conflicts:
### 1. Missing in Documentation
**Severity**: Medium
**Description**: API exists in code but is not documented
**Example**:
```python
# Code has this method:
def move_local_x(self, delta: float, snap: bool = False) -> None:
"""Move node along local X axis"""
# But documentation doesn't mention it
```
**Suggestion**: Add documentation for this API
### 2. Missing in Code
**Severity**: High
**Description**: API is documented but not found in codebase
**Example**:
```python
# Docs say:
def rotate(angle: float) -> None
# But code doesn't have this function
```
**Suggestion**: Update documentation to remove this API, or add it to codebase
### 3. Signature Mismatch
**Severity**: Medium-High
**Description**: API exists in both but signatures differ
**Example**:
```python
# Docs say:
def move_local_x(delta: float)
# Code has:
def move_local_x(delta: float, snap: bool = False)
```
**Suggestion**: Update documentation to match actual signature
### 4. Description Mismatch
**Severity**: Low
**Description**: Different descriptions/docstrings
## Merge Modes
### Rule-Based Merge (Default)
Fast, deterministic merging using predefined rules:
1. **If API only in docs** → Include with `[DOCS_ONLY]` tag
2. **If API only in code** → Include with `[UNDOCUMENTED]` tag
3. **If both match perfectly** → Include normally
4. **If conflict exists** → Prefer code signature, keep docs description
**When to use**:
- Fast merging (< 1 second)
- Automated workflows
- You don't need human oversight
**Example**:
```bash
python3 cli/unified_scraper.py --config config.json --merge-mode rule-based
```
### Claude-Enhanced Merge
AI-powered reconciliation using local Claude Code:
1. Opens new terminal with Claude Code
2. Provides conflict context and instructions
3. Claude analyzes and creates reconciled API reference
4. Human can review and adjust before finalizing
**When to use**:
- Complex conflicts requiring judgment
- You want highest quality merge
- You have time for human oversight
**Example**:
```bash
python3 cli/unified_scraper.py --config config.json --merge-mode claude-enhanced
```
## Skill Output Structure
The unified scraper creates this structure:
```
output/skill-name/
├── SKILL.md # Main skill file with merged APIs
├── references/
│ ├── documentation/ # Documentation references
│ │ └── index.md
│ ├── github/ # GitHub references
│ │ ├── README.md
│ │ ├── issues.md
│ │ └── releases.md
│ ├── pdf/ # PDF references (if applicable)
│ │ └── index.md
│ ├── api/ # Merged API reference
│ │ └── merged_api.md
│ └── conflicts.md # Detailed conflict report
├── scripts/ # Empty (for user scripts)
└── assets/ # Empty (for user assets)
```
### SKILL.md Format
```markdown
# React
Complete React knowledge base combining official documentation and React codebase insights.
## 📚 Sources
This skill combines knowledge from multiple sources:
- ✅ **Documentation**: https://react.dev/
- Pages: 200
- ✅ **GitHub Repository**: facebook/react
- Code Analysis: surface
- Issues: 100
## ⚠️ Data Quality
**5 conflicts detected** between sources.
**Conflict Breakdown:**
- missing_in_docs: 3
- missing_in_code: 2
See `references/conflicts.md` for detailed conflict information.
## 🔧 API Reference
*Merged from documentation and code analysis*
### ✅ Verified APIs
*Documentation and code agree*
#### `useState(initialValue)`
...
### ⚠️ APIs with Conflicts
*Documentation and code differ*
#### `useEffect(callback, deps?)`
⚠️ **Conflict**: Documentation signature differs from code implementation
**Documentation says:**
```
useEffect(callback: () => void, deps: any[])
```
**Code implementation:**
```
useEffect(callback: () => void | (() => void), deps?: readonly any[])
```
*Source: both*
---
```
## Examples
### Example 1: React (Docs + GitHub)
```json
{
"name": "react",
"description": "Complete React framework knowledge",
"merge_mode": "rule-based",
"sources": [
{
"type": "documentation",
"base_url": "https://react.dev/",
"extract_api": true,
"max_pages": 200
},
{
"type": "github",
"repo": "facebook/react",
"include_code": true,
"code_analysis_depth": "surface"
}
]
}
```
### Example 2: Django (Docs + GitHub)
```json
{
"name": "django",
"description": "Complete Django framework knowledge",
"merge_mode": "rule-based",
"sources": [
{
"type": "documentation",
"base_url": "https://docs.djangoproject.com/en/stable/",
"extract_api": true,
"max_pages": 300
},
{
"type": "github",
"repo": "django/django",
"include_code": true,
"code_analysis_depth": "deep",
"file_patterns": [
"django/db/**/*.py",
"django/views/**/*.py"
]
}
]
}
```
### Example 3: Mixed Sources (Docs + GitHub + PDF)
```json
{
"name": "godot",
"description": "Complete Godot Engine knowledge",
"merge_mode": "claude-enhanced",
"sources": [
{
"type": "documentation",
"base_url": "https://docs.godotengine.org/en/stable/",
"extract_api": true,
"max_pages": 500
},
{
"type": "github",
"repo": "godotengine/godot",
"include_code": true,
"code_analysis_depth": "deep"
},
{
"type": "pdf",
"path": "/path/to/godot_manual.pdf",
"extract_tables": true
}
]
}
```
## Command Reference
### Unified Scraper
```bash
# Basic usage
python3 cli/unified_scraper.py --config configs/react_unified.json
# Override merge mode
python3 cli/unified_scraper.py --config configs/react_unified.json --merge-mode claude-enhanced
# Use cached data (skip re-scraping)
python3 cli/unified_scraper.py --config configs/react_unified.json --skip-scrape
```
### Validate Config
```bash
python3 -c "
import sys
sys.path.insert(0, 'cli')
from config_validator import validate_config
validator = validate_config('configs/react_unified.json')
print(f'Format: {\"Unified\" if validator.is_unified else \"Legacy\"}')
print(f'Sources: {len(validator.config.get(\"sources\", []))}')
print(f'Needs API merge: {validator.needs_api_merge()}')
"
```
## MCP Integration
The unified scraper is fully integrated with MCP. The `scrape_docs` tool automatically detects unified vs legacy configs and routes to the appropriate scraper.
```python
# MCP tool usage
{
"name": "scrape_docs",
"arguments": {
"config_path": "configs/react_unified.json",
"merge_mode": "rule-based" # Optional override
}
}
```
The tool will:
1. Auto-detect unified format
2. Route to `unified_scraper.py`
3. Apply specified merge mode
4. Return comprehensive output
## Backward Compatibility
**Legacy configs still work!** The system automatically detects legacy single-source configs and routes to the original `doc_scraper.py`.
```json
// Legacy config (still works)
{
"name": "react",
"base_url": "https://react.dev/",
...
}
// Automatically detected as legacy format
// Routes to doc_scraper.py
```
## Testing
Run integration tests:
```bash
python3 cli/test_unified_simple.py
```
Tests validate:
- ✅ Unified config validation
- ✅ Backward compatibility with legacy configs
- ✅ Mixed source type support
- ✅ Error handling for invalid configs
## Architecture
### Components
1. **config_validator.py**: Validates unified and legacy configs
2. **code_analyzer.py**: Extracts code signatures at configurable depth
3. **conflict_detector.py**: Detects API conflicts between sources
4. **merge_sources.py**: Implements rule-based and Claude-enhanced merging
5. **unified_scraper.py**: Main orchestrator
6. **unified_skill_builder.py**: Generates final skill structure
7. **skill_seeker_mcp/server.py**: MCP integration with auto-detection
### Data Flow
```
Unified Config
↓
ConfigValidator (validates format)
↓
UnifiedScraper.run()
↓
┌────────────────────────────────────┐
│ Phase 1: Scrape All Sources │
│ - Documentation → doc_scraper │
│ - GitHub → github_scraper │
│ - PDF → pdf_scraper │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Phase 2: Detect Conflicts │
│ - ConflictDetector │
│ - Compare docs APIs vs code APIs │
│ - Classify by type and severity │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Phase 3: Merge Sources │
│ - RuleBasedMerger (fast) │
│ - OR ClaudeEnhancedMerger (AI) │
│ - Create unified API reference │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Phase 4: Build Skill │
│ - UnifiedSkillBuilder │
│ - Generate SKILL.md with conflicts│
│ - Create reference structure │
│ - Generate conflicts report │
└────────────────────────────────────┘
↓
Unified Skill (.zip ready)
```
## Best Practices
### 1. Start with Rule-Based Merge
Rule-based is fast and works well for most cases. Only use Claude-enhanced if you need human oversight.
### 2. Use Surface-Level Code Analysis
`code_analysis_depth: "surface"` is usually sufficient. Deep analysis is expensive and rarely needed.
### 3. Limit GitHub Issues
`max_issues: 100` is a good default. More than 200 issues rarely adds value.
### 4. Be Specific with File Patterns
```json
"file_patterns": [
"src/**/*.js", // Good: specific paths
"lib/**/*.ts"
]
// Not recommended:
"file_patterns": ["**/*.js"] // Too broad, slow
```
### 5. Monitor Conflict Reports
Always review `references/conflicts.md` to understand discrepancies between sources.
## Troubleshooting
### No Conflicts Detected
**Possible causes**:
- `extract_api: false` in documentation source
- `include_code: false` in GitHub source
- Code analysis found no APIs (check `code_analysis_depth`)
**Solution**: Ensure both sources have API extraction enabled
### Too Many Conflicts
**Possible causes**:
- Fuzzy matching threshold too strict
- Documentation uses different naming conventions
- Old documentation version
**Solution**: Review conflicts manually and adjust merge strategy
### Merge Takes Too Long
**Possible causes**:
- Using `code_analysis_depth: "full"` (very slow)
- Too many file patterns
- Large repository
**Solution**:
- Use `"surface"` or `"deep"` analysis
- Narrow file patterns
- Increase `rate_limit`
## Future Enhancements
Planned features:
- [ ] Automated conflict resolution strategies
- [ ] Conflict trend analysis across versions
- [ ] Multi-version comparison (docs v1 vs v2)
- [ ] Custom merge rules DSL
- [ ] Conflict confidence scores
## Support
For issues, questions, or suggestions:
- GitHub Issues: https://github.com/yusufkaraaslan/Skill_Seekers/issues
- Documentation: https://github.com/yusufkaraaslan/Skill_Seekers/docs
## Changelog
**v2.0 (October 2025)**: Unified multi-source scraping feature complete
- ✅ Config validation for unified format
- ✅ Deep code analysis with AST parsing
- ✅ Conflict detection (4 types, 3 severity levels)
- ✅ Rule-based merging
- ✅ Claude-enhanced merging
- ✅ Unified skill builder with inline conflict warnings
- ✅ MCP integration with auto-detection
- ✅ Backward compatibility with legacy configs
- ✅ Comprehensive tests and documentation
assets/repo/Skill_Seekers-development/docs/UPLOAD_GUIDE.md
+351
View File
@@ -0,0 +1,351 @@
# How to Upload Skills to Claude
## Quick Answer
**You have 3 options to upload the `.zip` file:**
### Option 1: Automatic Upload (Recommended for CLI)
```bash
# Set your API key (one-time setup)
export ANTHROPIC_API_KEY=sk-ant-...
# Package and upload automatically
python3 cli/package_skill.py output/react/ --upload
# OR upload existing .zip
python3 cli/upload_skill.py output/react.zip
```
✅ **Fully automatic** | No manual steps | Requires API key
### Option 2: Manual Upload (No API Key)
```bash
# Package the skill
python3 cli/package_skill.py output/react/
# This will:
# 1. Create output/react.zip
# 2. Open output/ folder automatically
# 3. Show clear upload instructions
# Then upload manually to https://claude.ai/skills
```
✅ **No API key needed** | Works for everyone | Simple
### Option 3: Claude Code MCP (Easiest)
```
In Claude Code, just say:
"Package and upload the React skill"
# Automatically packages and uploads!
```
✅ **Natural language** | Fully automatic | Best UX
---
## What's Inside the Zip?
The `.zip` file contains:
```
steam-economy.zip
├── SKILL.md ← Main skill file (Claude reads this first)
└── references/ ← Reference documentation
├── index.md ← Category index
├── api_reference.md ← API docs
├── pricing.md ← Pricing docs
├── trading.md ← Trading docs
└── ... ← Other categorized docs
```
**Note:** The zip only includes what Claude needs. It excludes:
- `.backup` files
- Build artifacts
- Temporary files
## What Does package_skill.py Do?
The package script:
1. **Finds your skill directory** (e.g., `output/steam-economy/`)
2. **Validates SKILL.md exists** (required!)
3. **Creates a .zip file** with the same name
4. **Includes all files** except backups
5. **Saves to** `output/` directory
**Example:**
```bash
python3 cli/package_skill.py output/steam-economy/
📦 Packaging skill: steam-economy
Source: output/steam-economy
Output: output/steam-economy.zip
+ SKILL.md
+ references/api_reference.md
+ references/pricing.md
+ references/trading.md
+ ...
✅ Package created: output/steam-economy.zip
Size: 14,290 bytes (14.0 KB)
```
## Complete Workflow
### Step 1: Scrape & Build
```bash
python3 cli/doc_scraper.py --config configs/steam-economy.json
```
**Output:**
- `output/steam-economy_data/` (raw scraped data)
- `output/steam-economy/` (skill directory)
### Step 2: Enhance (Recommended)
```bash
python3 cli/enhance_skill_local.py output/steam-economy/
```
**What it does:**
- Analyzes reference files
- Creates comprehensive SKILL.md
- Backs up original to SKILL.md.backup
**Output:**
- `output/steam-economy/SKILL.md` (enhanced)
- `output/steam-economy/SKILL.md.backup` (original)
### Step 3: Package
```bash
python3 cli/package_skill.py output/steam-economy/
```
**Output:**
- `output/steam-economy.zip` ← **THIS IS WHAT YOU UPLOAD**
### Step 4: Upload to Claude
1. Go to Claude (claude.ai)
2. Click "Add Skill" or skill upload button
3. Select `output/steam-economy.zip`
4. Done!
## What Files Are Required?
**Minimum required structure:**
```
your-skill/
└── SKILL.md ← Required! Claude reads this first
```
**Recommended structure:**
```
your-skill/
├── SKILL.md ← Main skill file (required)
└── references/ ← Reference docs (highly recommended)
├── index.md
└── *.md ← Category files
```
**Optional (can add manually):**
```
your-skill/
├── SKILL.md
├── references/
├── scripts/ ← Helper scripts
│ └── *.py
└── assets/ ← Templates, examples
└── *.txt
```
## File Size Limits
The package script shows size after packaging:
```
✅ Package created: output/steam-economy.zip
Size: 14,290 bytes (14.0 KB)
```
**Typical sizes:**
- Small skill: 5-20 KB
- Medium skill: 20-100 KB
- Large skill: 100-500 KB
Claude has generous size limits, so most documentation-based skills fit easily.
## Quick Reference
### Package a Skill
```bash
python3 cli/package_skill.py output/steam-economy/
```
### Package Multiple Skills
```bash
# Package all skills in output/
for dir in output/*/; do
if [ -f "$dir/SKILL.md" ]; then
python3 cli/package_skill.py "$dir"
fi
done
```
### Check What's in a Zip
```bash
unzip -l output/steam-economy.zip
```
### Test a Packaged Skill Locally
```bash
# Extract to temp directory
mkdir temp-test
unzip output/steam-economy.zip -d temp-test/
cat temp-test/SKILL.md
```
## Troubleshooting
### "SKILL.md not found"
```bash
# Make sure you scraped and built first
python3 cli/doc_scraper.py --config configs/steam-economy.json
# Then package
python3 cli/package_skill.py output/steam-economy/
```
### "Directory not found"
```bash
# Check what skills are available
ls output/
# Use correct path
python3 cli/package_skill.py output/YOUR-SKILL-NAME/
```
### Zip is Too Large
Most skills are small, but if yours is large:
```bash
# Check size
ls -lh output/steam-economy.zip
# If needed, check what's taking space
unzip -l output/steam-economy.zip | sort -k1 -rn | head -20
```
Reference files are usually small. Large sizes often mean:
- Many images (skills typically don't need images)
- Large code examples (these are fine, just be aware)
## What Does Claude Do With the Zip?
When you upload a skill zip:
1. **Claude extracts it**
2. **Reads SKILL.md first** - This tells Claude:
- When to activate this skill
- What the skill does
- Quick reference examples
- How to navigate the references
3. **Indexes reference files** - Claude can search through:
- `references/*.md` files
- Find specific APIs, examples, concepts
4. **Activates automatically** - When you ask about topics matching the skill
## Example: Using the Packaged Skill
After uploading `steam-economy.zip`:
**You ask:** "How do I implement microtransactions in my Steam game?"
**Claude:**
- Recognizes this matches steam-economy skill
- Reads SKILL.md for quick reference
- Searches references/microtransactions.md
- Provides detailed answer with code examples
## API-Based Automatic Upload
### Setup (One-Time)
```bash
# Get your API key from https://console.anthropic.com/
export ANTHROPIC_API_KEY=sk-ant-...
# Add to your shell profile to persist
echo 'export ANTHROPIC_API_KEY=sk-ant-...' >> ~/.bashrc # or ~/.zshrc
```
### Usage
```bash
# Upload existing .zip
python3 cli/upload_skill.py output/react.zip
# OR package and upload in one command
python3 cli/package_skill.py output/react/ --upload
```
### How It Works
The upload tool uses the Anthropic `/v1/skills` API endpoint to:
1. Read your .zip file
2. Authenticate with your API key
3. Upload to Claude's skill storage
4. Verify upload success
### Troubleshooting
**"ANTHROPIC_API_KEY not set"**
```bash
# Check if set
echo $ANTHROPIC_API_KEY
# If empty, set it
export ANTHROPIC_API_KEY=sk-ant-...
```
**"Authentication failed"**
- Verify your API key is correct
- Check https://console.anthropic.com/ for valid keys
**"Upload timed out"**
- Check your internet connection
- Try again or use manual upload
**Upload fails with error**
- Falls back to showing manual upload instructions
- You can still upload via https://claude.ai/skills
---
## Summary
**What you need to do:**
### With API Key (Automatic):
1. ✅ Scrape: `python3 cli/doc_scraper.py --config configs/YOUR-CONFIG.json`
2. ✅ Enhance: `python3 cli/enhance_skill_local.py output/YOUR-SKILL/`
3. ✅ Package & Upload: `python3 cli/package_skill.py output/YOUR-SKILL/ --upload`
4. ✅ Done! Skill is live in Claude
### Without API Key (Manual):
1. ✅ Scrape: `python3 cli/doc_scraper.py --config configs/YOUR-CONFIG.json`
2. ✅ Enhance: `python3 cli/enhance_skill_local.py output/YOUR-SKILL/`
3. ✅ Package: `python3 cli/package_skill.py output/YOUR-SKILL/`
4. ✅ Upload: Go to https://claude.ai/skills and upload the `.zip`
**What you upload:**
- The `.zip` file from `output/` directory
- Example: `output/steam-economy.zip`
**What's in the zip:**
- `SKILL.md` (required)
- `references/*.md` (recommended)
- Any scripts/assets you added (optional)
That's it! 🚀
assets/repo/Skill_Seekers-development/docs/USAGE.md
+811
View File
@@ -0,0 +1,811 @@
# Complete Usage Guide for Skill Seeker
Comprehensive reference for all commands, options, and workflows.
## Table of Contents
- [Quick Reference](#quick-reference)
- [Main Tool: doc_scraper.py](#main-tool-doc_scraperpy)
- [Estimator: estimate_pages.py](#estimator-estimate_pagespy)
- [Enhancement Tools](#enhancement-tools)
- [Packaging Tool](#packaging-tool)
- [Testing Tools](#testing-tools)
- [Available Configs](#available-configs)
- [Common Workflows](#common-workflows)
- [Troubleshooting](#troubleshooting)
---
## Quick Reference
```bash
# 1. Estimate pages (fast, 1-2 min)
python3 cli/estimate_pages.py configs/react.json
# 2. Scrape documentation (20-40 min)
python3 cli/doc_scraper.py --config configs/react.json
# 3. Enhance with Claude Code (60 sec)
python3 cli/enhance_skill_local.py output/react/
# 4. Package to .zip (instant)
python3 cli/package_skill.py output/react/
# 5. Test everything (1 sec)
python3 cli/run_tests.py
```
---
## Main Tool: doc_scraper.py
### Full Help
```
usage: doc_scraper.py [-h] [--interactive] [--config CONFIG] [--name NAME]
[--url URL] [--description DESCRIPTION] [--skip-scrape]
[--dry-run] [--enhance] [--enhance-local]
[--api-key API_KEY]
Convert documentation websites to Claude skills
options:
-h, --help Show this help message and exit
--interactive, -i Interactive configuration mode
--config, -c CONFIG Load configuration from file (e.g., configs/godot.json)
--name NAME Skill name
--url URL Base documentation URL
--description, -d DESCRIPTION
Skill description
--skip-scrape Skip scraping, use existing data
--dry-run Preview what will be scraped without actually scraping
--enhance Enhance SKILL.md using Claude API after building
(requires API key)
--enhance-local Enhance SKILL.md using Claude Code in new terminal
(no API key needed)
--api-key API_KEY Anthropic API key for --enhance (or set ANTHROPIC_API_KEY)
```
### Usage Examples
**1. Use Preset Config (Recommended)**
```bash
python3 cli/doc_scraper.py --config configs/godot.json
python3 cli/doc_scraper.py --config configs/react.json
python3 cli/doc_scraper.py --config configs/vue.json
python3 cli/doc_scraper.py --config configs/django.json
python3 cli/doc_scraper.py --config configs/fastapi.json
```
**2. Interactive Mode**
```bash
python3 cli/doc_scraper.py --interactive
# Wizard walks you through:
# - Skill name
# - Base URL
# - Description
# - Selectors (optional)
# - URL patterns (optional)
# - Rate limit
# - Max pages
```
**3. Quick Mode (Minimal)**
```bash
python3 cli/doc_scraper.py \
--name react \
--url https://react.dev/ \
--description "React framework for building UIs"
```
**4. Dry-Run (Preview)**
```bash
python3 cli/doc_scraper.py --config configs/react.json --dry-run
# Shows what will be scraped without downloading data
# No directories created
# Fast validation
```
**5. Skip Scraping (Use Cached Data)**
```bash
python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape
# Uses existing output/godot_data/
# Fast rebuild (1-3 minutes)
# Useful for testing changes
```
**6. With Local Enhancement**
```bash
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
# Scrapes + enhances in one command
# Opens new terminal for Claude Code
# No API key needed
```
**7. With API Enhancement**
```bash
export ANTHROPIC_API_KEY=sk-ant-...
python3 cli/doc_scraper.py --config configs/react.json --enhance
# Or with inline API key:
python3 cli/doc_scraper.py --config configs/react.json --enhance --api-key sk-ant-...
```
### Output Structure
```
output/
├── {name}_data/ # Scraped raw data (cached)
│ ├── pages/
│ │ ├── page_0.json
│ │ ├── page_1.json
│ │ └── ...
│ └── summary.json # Scraping stats
│
└── {name}/ # Built skill directory
├── SKILL.md # Main skill file
├── SKILL.md.backup # Backup (if enhanced)
├── references/ # Categorized docs
│ ├── index.md
│ ├── getting_started.md
│ ├── api.md
│ └── ...
├── scripts/ # Empty (user scripts)
└── assets/ # Empty (user assets)
```
---
## Estimator: estimate_pages.py
### Full Help
```
usage: estimate_pages.py [-h] [--max-discovery MAX_DISCOVERY]
[--timeout TIMEOUT]
config
Estimate page count for Skill Seeker configs
positional arguments:
config Path to config JSON file
options:
-h, --help Show this help message and exit
--max-discovery, -m MAX_DISCOVERY
Maximum pages to discover (default: 1000)
--timeout, -t TIMEOUT
HTTP request timeout in seconds (default: 30)
```
### Usage Examples
**1. Quick Estimate (100 pages)**
```bash
python3 cli/estimate_pages.py configs/react.json --max-discovery 100
# Time: ~30-60 seconds
# Good for: Quick validation
```
**2. Standard Estimate (1000 pages - default)**
```bash
python3 cli/estimate_pages.py configs/godot.json
# Time: ~1-2 minutes
# Good for: Most use cases
```
**3. Deep Estimate (2000 pages)**
```bash
python3 cli/estimate_pages.py configs/vue.json --max-discovery 2000
# Time: ~3-5 minutes
# Good for: Large documentation sites
```
**4. Custom Timeout**
```bash
python3 cli/estimate_pages.py configs/django.json --timeout 60
# Useful for slow servers
```
### Output Example
```
🔍 Estimating pages for: react
📍 Base URL: https://react.dev/
🎯 Start URLs: 6
⏱️ Rate limit: 0.5s
🔢 Max discovery: 1000
⏳ Discovered: 180 pages (1.3 pages/sec)
======================================================================
📊 ESTIMATION RESULTS
======================================================================
Config: react
Base URL: https://react.dev/
✅ Pages Discovered: 180
⏳ Pages Pending: 50
📈 Estimated Total: 230
⏱️ Time Elapsed: 140.5s
⚡ Discovery Rate: 1.28 pages/sec
======================================================================
💡 RECOMMENDATIONS
======================================================================
✅ Current max_pages (300) is sufficient
⏱️ Estimated full scrape time: 1.9 minutes
(Based on rate_limit: 0.5s)
```
**What It Shows:**
- Estimated total pages to scrape
- Whether current `max_pages` is sufficient
- Recommended `max_pages` value
- Estimated scraping time
- Discovery rate (pages/sec)
---
## Enhancement Tools
### enhance_skill_local.py (Recommended)
**No API key needed - uses Claude Code Max plan**
```bash
# Usage
python3 cli/enhance_skill_local.py output/react/
python3 cli/enhance_skill_local.py output/godot/
# What it does:
# 1. Reads SKILL.md and references/
# 2. Opens new terminal with Claude Code
# 3. Claude enhances SKILL.md
# 4. Backs up original to SKILL.md.backup
# 5. Saves enhanced version
# Time: ~60 seconds
# Cost: Free (uses your Claude Code Max plan)
```
### enhance_skill.py (Alternative)
**Requires Anthropic API key**
```bash
# Install dependency first
pip3 install anthropic
# Usage with environment variable
export ANTHROPIC_API_KEY=sk-ant-...
python3 cli/enhance_skill.py output/react/
# Usage with inline API key
python3 cli/enhance_skill.py output/godot/ --api-key sk-ant-...
# What it does:
# 1. Reads SKILL.md and references/
# 2. Calls Claude API (Sonnet 4)
# 3. Enhances SKILL.md
# 4. Backs up original to SKILL.md.backup
# 5. Saves enhanced version
# Time: ~30-60 seconds
# Cost: ~$0.01-0.10 per skill (depending on size)
```
---
## Packaging Tool
### package_skill.py
```bash
# Usage
python3 cli/package_skill.py output/react/
python3 cli/package_skill.py output/godot/
# What it does:
# 1. Validates SKILL.md exists
# 2. Creates .zip with all skill files
# 3. Saves to output/{name}.zip
# Output:
# output/react.zip
# output/godot.zip
# Time: Instant
```
---
## Testing Tools
### run_tests.py
```bash
# Run all tests (default)
python3 cli/run_tests.py
# 71 tests, ~1 second
# Verbose output
python3 cli/run_tests.py -v
python3 cli/run_tests.py --verbose
# Quiet output
python3 cli/run_tests.py -q
python3 cli/run_tests.py --quiet
# Stop on first failure
python3 cli/run_tests.py -f
python3 cli/run_tests.py --failfast
# Run specific test suite
python3 cli/run_tests.py --suite config
python3 cli/run_tests.py --suite features
python3 cli/run_tests.py --suite integration
# List all tests
python3 cli/run_tests.py --list
```
### Individual Tests
```bash
# Run single test file
python3 -m unittest tests.test_config_validation
python3 -m unittest tests.test_scraper_features
python3 -m unittest tests.test_integration
# Run single test class
python3 -m unittest tests.test_config_validation.TestConfigValidation
# Run single test method
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_valid_complete_config
```
---
## Available Configs
### Preset Configs (Ready to Use)
| Config | Framework | Pages | Description |
|--------|-----------|-------|-------------|
| `godot.json` | Godot Engine | ~500 | Game engine documentation |
| `react.json` | React | ~300 | React framework docs |
| `vue.json` | Vue.js | ~250 | Vue.js framework docs |
| `django.json` | Django | ~400 | Django web framework |
| `fastapi.json` | FastAPI | ~200 | FastAPI Python framework |
| `steam-economy-complete.json` | Steam | ~100 | Steam Economy API docs |
### View Config Details
```bash
# List all configs
ls configs/
# View config content
cat configs/react.json
python3 -m json.tool configs/godot.json
```
### Config Structure
```json
{
"name": "react",
"base_url": "https://react.dev/",
"description": "React - JavaScript library for building UIs",
"start_urls": [
"https://react.dev/learn",
"https://react.dev/reference/react",
"https://react.dev/reference/react-dom"
],
"selectors": {
"main_content": "article",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": ["/learn/", "/reference/"],
"exclude": ["/blog/", "/community/"]
},
"categories": {
"getting_started": ["learn", "tutorial", "intro"],
"api": ["reference", "api", "hooks"],
"guides": ["guide"]
},
"rate_limit": 0.5,
"max_pages": 300
}
```
---
## Common Workflows
### Workflow 1: Use Preset (Fastest)
```bash
# 1. Estimate (optional, 1-2 min)
python3 cli/estimate_pages.py configs/react.json
# 2. Scrape with local enhancement (25 min)
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
# 3. Package (instant)
python3 cli/package_skill.py output/react/
# Result: output/react.zip
# Upload to Claude!
```
### Workflow 2: Custom Documentation
```bash
# 1. Create config
cat > configs/my-docs.json << 'EOF'
{
"name": "my-docs",
"base_url": "https://docs.example.com/",
"description": "My documentation site",
"rate_limit": 0.5,
"max_pages": 200
}
EOF
# 2. Estimate
python3 cli/estimate_pages.py configs/my-docs.json
# 3. Dry-run test
python3 cli/doc_scraper.py --config configs/my-docs.json --dry-run
# 4. Full scrape
python3 cli/doc_scraper.py --config configs/my-docs.json
# 5. Enhance
python3 cli/enhance_skill_local.py output/my-docs/
# 6. Package
python3 cli/package_skill.py output/my-docs/
```
### Workflow 3: Interactive Mode
```bash
# 1. Start interactive wizard
python3 cli/doc_scraper.py --interactive
# 2. Answer prompts:
# - Name: my-framework
# - URL: https://framework.dev/
# - Description: My favorite framework
# - Selectors: (uses defaults)
# - Rate limit: 0.5
# - Max pages: 100
# 3. Enhance
python3 cli/enhance_skill_local.py output/my-framework/
# 4. Package
python3 cli/package_skill.py output/my-framework/
```
### Workflow 4: Quick Mode
```bash
python3 cli/doc_scraper.py \
--name vue \
--url https://vuejs.org/ \
--description "Vue.js framework" \
--enhance-local
```
### Workflow 5: Rebuild from Cache
```bash
# Already scraped once?
# Skip re-scraping, just rebuild
python3 cli/doc_scraper.py --config configs/godot.json --skip-scrape
# Try new enhancement
python3 cli/enhance_skill_local.py output/godot/
# Re-package
python3 cli/package_skill.py output/godot/
```
### Workflow 6: Testing New Config
```bash
# 1. Create test config with low max_pages
cat > configs/test.json << 'EOF'
{
"name": "test-site",
"base_url": "https://docs.test.com/",
"max_pages": 20,
"rate_limit": 0.1
}
EOF
# 2. Estimate
python3 cli/estimate_pages.py configs/test.json --max-discovery 50
# 3. Dry-run
python3 cli/doc_scraper.py --config configs/test.json --dry-run
# 4. Small scrape
python3 cli/doc_scraper.py --config configs/test.json
# 5. Validate output
ls output/test-site/
ls output/test-site/references/
# 6. If good, increase max_pages and re-run
```
---
## Troubleshooting
### Issue: "Rate limit exceeded"
```bash
# Increase rate_limit in config
# Default: 0.5 seconds
# Conservative: 1.0 seconds
# Very conservative: 2.0 seconds
# Edit config:
{
"rate_limit": 1.0
}
```
### Issue: "Too many pages"
```bash
# Estimate first
python3 cli/estimate_pages.py configs/my-config.json
# Set max_pages based on estimate
# Add buffer: estimated + 50
# Edit config:
{
"max_pages": 350 # for 300 estimated
}
```
### Issue: "No content extracted"
```bash
# Wrong selectors
# Test selectors manually:
curl -s https://docs.example.com/ | grep -i 'article\|main\|content'
# Common selectors:
"main_content": "article"
"main_content": "main"
"main_content": ".content"
"main_content": "#main-content"
"main_content": "div[role=\"main\"]"
# Update config with correct selector
```
### Issue: "Tests failing"
```bash
# Run specific failing test
python3 -m unittest tests.test_config_validation.TestConfigValidation.test_name -v
# Check error message
# Verify expectations match implementation
```
### Issue: "Enhancement fails"
```bash
# Local enhancement:
# Make sure Claude Code is running
# Check terminal output
# API enhancement:
# Verify API key is set:
echo $ANTHROPIC_API_KEY
# Or use inline:
python3 cli/enhance_skill.py output/react/ --api-key sk-ant-...
```
### Issue: "Package fails"
```bash
# Verify SKILL.md exists
ls output/my-skill/SKILL.md
# If missing, build first:
python3 cli/doc_scraper.py --config configs/my-skill.json --skip-scrape
```
### Issue: "Can't find output"
```bash
# Check output directory
ls output/
# Skill data (cached):
ls output/{name}_data/
# Built skill:
ls output/{name}/
# Packaged skill:
ls output/{name}.zip
```
---
## Advanced Usage
### Custom Selectors
```json
{
"selectors": {
"main_content": "div.documentation",
"title": "h1.page-title",
"code_blocks": "pre.highlight code",
"navigation": "nav.sidebar"
}
}
```
### URL Pattern Filtering
```json
{
"url_patterns": {
"include": [
"/docs/",
"/guide/",
"/api/",
"/tutorial/"
],
"exclude": [
"/blog/",
"/news/",
"/community/",
"/showcase/"
]
}
}
```
### Custom Categories
```json
{
"categories": {
"getting_started": ["intro", "tutorial", "quickstart", "installation"],
"core_concepts": ["concept", "fundamental", "architecture"],
"api": ["reference", "api", "method", "function"],
"guides": ["guide", "how-to", "example"],
"advanced": ["advanced", "expert", "performance"]
}
}
```
### Multiple Start URLs
```json
{
"start_urls": [
"https://docs.example.com/getting-started/",
"https://docs.example.com/api/",
"https://docs.example.com/guides/",
"https://docs.example.com/examples/"
]
}
```
---
## Performance Tips
1. **Estimate first**: Save 20-40 minutes by validating config
2. **Use dry-run**: Test selectors before full scrape
3. **Cache data**: Use `--skip-scrape` for fast rebuilds
4. **Adjust rate_limit**: Balance speed vs politeness
5. **Set appropriate max_pages**: Don't scrape more than needed
6. **Use start_urls**: Target specific documentation sections
7. **Filter URLs**: Use include/exclude patterns
8. **Run tests**: Catch issues early
---
## Environment Variables
```bash
# Anthropic API key (for API enhancement)
export ANTHROPIC_API_KEY=sk-ant-...
# Optional: Set custom output directory
export SKILL_SEEKER_OUTPUT_DIR=/path/to/output
```
---
## Exit Codes
- `0`: Success
- `1`: Error (general)
- `2`: Warning (estimation hit limit)
---
## File Locations
```
Skill_Seekers/
├── doc_scraper.py # Main tool
├── estimate_pages.py # Estimator
├── enhance_skill.py # API enhancement
├── enhance_skill_local.py # Local enhancement
├── package_skill.py # Packager
├── run_tests.py # Test runner
├── configs/ # Preset configs
├── tests/ # Test suite
├── docs/ # Documentation
└── output/ # Generated output
```
---
## Getting Help
```bash
# Tool-specific help
python3 cli/doc_scraper.py --help
python3 cli/estimate_pages.py --help
python3 cli/run_tests.py --help
# Documentation
cat CLAUDE.md # Quick reference for Claude Code
cat docs/CLAUDE.md # Detailed technical docs
cat docs/TESTING.md # Testing guide
cat docs/USAGE.md # This file
cat docs/ENHANCEMENT.md # Enhancement guide
cat docs/UPLOAD_GUIDE.md # Upload instructions
cat README.md # Project overview
```
---
## Summary
**Essential Commands:**
```bash
python3 cli/estimate_pages.py configs/react.json # Estimate
python3 cli/doc_scraper.py --config configs/react.json # Scrape
python3 cli/enhance_skill_local.py output/react/ # Enhance
python3 cli/package_skill.py output/react/ # Package
python3 cli/run_tests.py # Test
```
**Quick Start:**
```bash
pip3 install requests beautifulsoup4
python3 cli/doc_scraper.py --config configs/react.json --enhance-local
python3 cli/package_skill.py output/react/
# Upload output/react.zip to Claude!
```
Happy skill creating! 🚀
assets/repo/Skill_Seekers-development/docs/plans/2025-10-24-active-skills-design.md
+867
View File
@@ -0,0 +1,867 @@
# Active Skills Design - Demand-Driven Documentation Loading
**Date:** 2025-10-24
**Type:** Architecture Design
**Status:** Phase 1 Implemented ✅
**Author:** Edgar + Claude (Brainstorming Session)
---
## Executive Summary
Transform Skill_Seekers from creating **passive documentation dumps** into **active, intelligent skills** that load documentation on-demand. This eliminates context bloat (300k → 5-10k per query) while maintaining full access to complete documentation.
**Key Innovation:** Skills become lightweight routers with heavy tools in `scripts/`, not documentation repositories.
---
## Problem Statement
### Current Architecture: Passive Skills
**What happens today:**
```
Agent: "How do I use Hono middleware?"
↓
Skill: *Claude loads 203k llms-txt.md into context*
↓
Agent: *answers using loaded docs*
↓
Result: Context bloat, slower performance, hits limits
```
**Issues:**
1. **Context Bloat**: 319k llms-full.txt loaded entirely into context
2. **Wasted Resources**: Agent needs 5k but gets 319k
3. **Truncation Loss**: 36% of content lost (319k → 203k) due to size limits
4. **File Extension Bug**: llms.txt files stored as .txt instead of .md
5. **Single Variant**: Only downloads one file (usually llms-full.txt)
### Current File Structure
```
output/hono/
├── SKILL.md ──────────► Documentation dump + instructions
├── references/
│ └── llms-txt.md ───► 203k (36% truncated from 319k original)
├── scripts/ ──────────► EMPTY (placeholder only!)
└── assets/ ───────────► EMPTY (placeholder only!)
```
---
## Proposed Architecture: Active Skills
### Core Concept
**Skills = Routers + Tools**, not documentation dumps.
**New workflow:**
```
Agent: "How do I use Hono middleware?"
↓
Skill: *runs scripts/search.py "middleware"*
↓
Script: *loads llms-full.md, extracts middleware section, returns 8k*
↓
Agent: *answers using ONLY 8k* (CLEAN CONTEXT!)
↓
Result: 40x less context, no truncation, full access to docs
```
### Benefits
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Context per query | 203k | 5-10k | **20-40x reduction** |
| Content loss | 36% truncated | 0% (no truncation) | **Full fidelity** |
| Variants available | 1 | 3 | **User choice** |
| File format | .txt (wrong) | .md (correct) | **Fixed** |
| Agent workflow | Passive read | Active tools | **Autonomous** |
---
## Design Components
### Component 1: Multi-Variant Download
**Change:** Download ALL 3 variants, not just one.
**File naming (FIXED):**
- `https://hono.dev/llms-full.txt` → `llms-full.md` ✅
- `https://hono.dev/llms.txt` → `llms.md` ✅
- `https://hono.dev/llms-small.txt` → `llms-small.md` ✅
**Sizes (Hono example):**
- `llms-full.md` - 319k (complete documentation)
- `llms-small.md` - 176k (curated essentials)
- `llms.md` - 5.4k (quick reference)
**Storage:**
```
output/hono/references/
├── llms-full.md # 319k - everything (RENAMED from .txt)
├── llms-small.md # 176k - curated (RENAMED from .txt)
├── llms.md # 5.4k - quick ref (RENAMED from .txt)
└── catalog.json # Generated index (NEW)
```
**Implementation in `_try_llms_txt()`:**
```python
def _try_llms_txt(self) -> bool:
"""Download ALL llms.txt variants for active skills"""
# 1. Detect all available variants
detector = LlmsTxtDetector(self.base_url)
variants = detector.detect_all() # NEW method
downloaded = {}
for variant_info in variants:
url = variant_info['url'] # https://hono.dev/llms-full.txt
variant = variant_info['variant'] # 'full', 'standard', 'small'
downloader = LlmsTxtDownloader(url)
content = downloader.download()
if content:
# ✨ FIX: Rename .txt → .md immediately
clean_name = f"llms-{variant}.md"
downloaded[variant] = {
'content': content,
'filename': clean_name
}
# 2. Save ALL variants (not just one)
for variant, data in downloaded.items():
path = os.path.join(self.skill_dir, "references", data['filename'])
with open(path, 'w', encoding='utf-8') as f:
f.write(data['content'])
# 3. Generate catalog from smallest variant
if 'small' in downloaded:
self._generate_catalog(downloaded['small']['content'])
return True
```
---
### Component 2: The Catalog System
**Purpose:** Lightweight index of what exists, not the content itself.
**File:** `assets/catalog.json`
**Structure:**
```json
{
"metadata": {
"framework": "hono",
"version": "auto-detected",
"generated": "2025-10-24T14:30:00Z",
"total_sections": 93,
"variants": {
"quick": "llms-small.md",
"standard": "llms.md",
"complete": "llms-full.md"
}
},
"sections": [
{
"id": "routing",
"title": "Routing",
"h1_marker": "# Routing",
"topics": ["routes", "path", "params", "wildcard"],
"size_bytes": 4800,
"variants": ["quick", "complete"],
"complexity": "beginner"
},
{
"id": "middleware",
"title": "Middleware",
"h1_marker": "# Middleware",
"topics": ["cors", "auth", "logging", "compression"],
"size_bytes": 8200,
"variants": ["quick", "complete"],
"complexity": "intermediate"
}
],
"search_index": {
"cors": ["middleware"],
"routing": ["routing", "path-parameters"],
"authentication": ["middleware", "jwt"],
"context": ["context-handling"],
"streaming": ["streaming-responses"]
}
}
```
**Generation (from llms-small.md):**
```python
def _generate_catalog(self, llms_small_content):
"""Generate catalog.json from llms-small.md TOC"""
catalog = {
"metadata": {...},
"sections": [],
"search_index": {}
}
# Split by h1 headers
sections = re.split(r'\n# ', llms_small_content)
for section_text in sections[1:]:
lines = section_text.split('\n')
title = lines[0].strip()
# Extract h2 topics
topics = re.findall(r'^## (.+)$', section_text, re.MULTILINE)
topics = [t.strip().lower() for t in topics]
section_info = {
"id": title.lower().replace(' ', '-'),
"title": title,
"h1_marker": f"# {title}",
"topics": topics + [title.lower()],
"size_bytes": len(section_text),
"variants": ["quick", "complete"]
}
catalog["sections"].append(section_info)
# Build search index
for topic in section_info["topics"]:
if topic not in catalog["search_index"]:
catalog["search_index"][topic] = []
catalog["search_index"][topic].append(section_info["id"])
# Save to assets/catalog.json
catalog_path = os.path.join(self.skill_dir, "assets", "catalog.json")
with open(catalog_path, 'w', encoding='utf-8') as f:
json.dump(catalog, f, indent=2)
```
---
### Component 3: Active Scripts
**Location:** `scripts/` directory (currently empty)
#### Script 1: `scripts/search.py`
**Purpose:** Search and return only relevant documentation sections.
```python
#!/usr/bin/env python3
"""
ABOUTME: Searches framework documentation and returns relevant sections
ABOUTME: Loads only what's needed - keeps agent context clean
"""
import json
import sys
import re
from pathlib import Path
def search(query, detail="auto"):
"""
Search documentation and return relevant sections.
Args:
query: Search term (e.g., "middleware", "cors", "routing")
detail: "quick" | "standard" | "complete" | "auto"
Returns:
Markdown text of relevant sections only
"""
# Load catalog
catalog_path = Path(__file__).parent.parent / "assets" / "catalog.json"
catalog = json.load(open(catalog_path))
# 1. Find matching sections using search index
query_lower = query.lower()
matching_section_ids = set()
for keyword, section_ids in catalog["search_index"].items():
if query_lower in keyword or keyword in query_lower:
matching_section_ids.update(section_ids)
# Get section details
matches = [s for s in catalog["sections"] if s["id"] in matching_section_ids]
if not matches:
return f"❌ No sections found for '{query}'. Try: python scripts/list_topics.py"
# 2. Determine detail level
if detail == "auto":
# Use quick for overview, complete for deep dive
total_size = sum(s["size_bytes"] for s in matches)
if total_size > 50000: # > 50k
variant = "quick"
else:
variant = "complete"
else:
variant = detail
variant_file = catalog["metadata"]["variants"].get(variant, "complete")
# 3. Load documentation file
doc_path = Path(__file__).parent.parent / "references" / variant_file
doc_content = open(doc_path, 'r', encoding='utf-8').read()
# 4. Extract matched sections
results = []
for match in matches:
h1_marker = match["h1_marker"]
# Find section boundaries
start = doc_content.find(h1_marker)
if start == -1:
continue
# Find next h1 (or end of file)
next_h1 = doc_content.find("\n# ", start + len(h1_marker))
if next_h1 == -1:
section_text = doc_content[start:]
else:
section_text = doc_content[start:next_h1]
results.append({
'title': match['title'],
'size': len(section_text),
'content': section_text
})
# 5. Format output
output = [f"# Search Results for '{query}' ({len(results)} sections found)\n"]
output.append(f"**Variant used:** {variant} ({variant_file})")
output.append(f"**Total size:** {sum(r['size'] for r in results):,} bytes\n")
output.append("---\n")
for result in results:
output.append(result['content'])
output.append("\n---\n")
return '\n'.join(output)
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python search.py <query> [detail]")
print("Example: python search.py middleware")
print("Example: python search.py routing --detail quick")
sys.exit(1)
query = sys.argv[1]
detail = sys.argv[2] if len(sys.argv) > 2 else "auto"
print(search(query, detail))
```
#### Script 2: `scripts/list_topics.py`
**Purpose:** Show all available documentation sections.
```python
#!/usr/bin/env python3
"""
ABOUTME: Lists all available documentation sections with sizes
ABOUTME: Helps agent discover what documentation exists
"""
import json
from pathlib import Path
def list_topics():
"""List all available documentation sections."""
catalog_path = Path(__file__).parent.parent / "assets" / "catalog.json"
catalog = json.load(open(catalog_path))
print(f"# Available Documentation Topics ({catalog['metadata']['framework']})\n")
print(f"**Total sections:** {catalog['metadata']['total_sections']}")
print(f"**Variants:** {', '.join(catalog['metadata']['variants'].keys())}\n")
print("---\n")
# Group by complexity if available
by_complexity = {}
for section in catalog["sections"]:
complexity = section.get("complexity", "general")
if complexity not in by_complexity:
by_complexity[complexity] = []
by_complexity[complexity].append(section)
for complexity in ["beginner", "intermediate", "advanced", "general"]:
if complexity not in by_complexity:
continue
sections = by_complexity[complexity]
print(f"## {complexity.title()} ({len(sections)} sections)\n")
for section in sections:
size_kb = section["size_bytes"] / 1024
topics_str = ", ".join(section["topics"][:3])
print(f"- **{section['title']}** ({size_kb:.1f}k)")
print(f" Topics: {topics_str}")
print(f" Search: `python scripts/search.py {section['id']}`\n")
if __name__ == "__main__":
list_topics()
```
#### Script 3: `scripts/get_section.py`
**Purpose:** Extract a complete section by exact title.
```python
#!/usr/bin/env python3
"""
ABOUTME: Extracts a complete documentation section by title
ABOUTME: Returns full section from llms-full.md (no truncation)
"""
import json
import sys
from pathlib import Path
def get_section(title, variant="complete"):
"""
Get a complete section by exact title.
Args:
title: Section title (e.g., "Middleware", "Routing")
variant: Which file to use (quick/standard/complete)
Returns:
Complete section content
"""
catalog_path = Path(__file__).parent.parent / "assets" / "catalog.json"
catalog = json.load(open(catalog_path))
# Find section
section = None
for s in catalog["sections"]:
if s["title"].lower() == title.lower():
section = s
break
if not section:
return f"❌ Section '{title}' not found. Try: python scripts/list_topics.py"
# Load doc
variant_file = catalog["metadata"]["variants"].get(variant, "complete")
doc_path = Path(__file__).parent.parent / "references" / variant_file
doc_content = open(doc_path, 'r', encoding='utf-8').read()
# Extract section
h1_marker = section["h1_marker"]
start = doc_content.find(h1_marker)
if start == -1:
return f"❌ Section '{title}' not found in {variant_file}"
next_h1 = doc_content.find("\n# ", start + len(h1_marker))
if next_h1 == -1:
section_text = doc_content[start:]
else:
section_text = doc_content[start:next_h1]
return section_text
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python get_section.py <title> [variant]")
print("Example: python get_section.py Middleware")
print("Example: python get_section.py Routing quick")
sys.exit(1)
title = sys.argv[1]
variant = sys.argv[2] if len(sys.argv) > 2 else "complete"
print(get_section(title, variant))
```
---
### Component 4: Active SKILL.md Template
**New template for llms.txt-based skills:**
```markdown
---
name: {name}
description: {description}
type: active
---
# {Name} Skill
**⚡ This is an ACTIVE skill** - Uses scripts to load documentation on-demand instead of dumping everything into context.
## 🎯 Strategy: Demand-Driven Documentation
**Traditional approach:**
- Load 300k+ documentation into context
- Agent reads everything to answer one question
- Context bloat, slower performance
**Active approach:**
- Load 5-10k of relevant sections on-demand
- Agent calls scripts to fetch what's needed
- Clean context, faster performance
## 📚 Available Documentation
This skill provides access to {num_sections} documentation sections across 3 detail levels:
- **Quick Reference** (`llms-small.md`): {small_size}k - Curated essentials
- **Standard** (`llms.md`): {standard_size}k - Core concepts
- **Complete** (`llms-full.md`): {full_size}k - Everything
## 🔧 Tools Available
### 1. Search Documentation
Find and load only relevant sections:
```bash
python scripts/search.py "middleware"
python scripts/search.py "routing" --detail quick
```
**Returns:** 5-10k of relevant content (not 300k!)
### 2. List All Topics
See what documentation exists:
```bash
python scripts/list_topics.py
```
**Returns:** Table of contents with section sizes and search hints
### 3. Get Complete Section
Extract a full section by title:
```bash
python scripts/get_section.py "Middleware"
python scripts/get_section.py "Routing" quick
```
**Returns:** Complete section from chosen variant
## 💡 Recommended Workflow
1. **Discover:** `python scripts/list_topics.py` to see what's available
2. **Search:** `python scripts/search.py "your topic"` to find relevant sections
3. **Deep Dive:** Use returned content to answer questions in detail
4. **Iterate:** Search more specific topics as needed
## ⚠️ Important
**DON'T:** Read `references/*.md` files directly into context
**DO:** Use scripts to fetch only what you need
This keeps your context clean and focused!
## 📊 Index
Complete section catalog available in `assets/catalog.json` with search mappings and size information.
## 🔄 Updating
To refresh with latest documentation:
```bash
python3 cli/doc_scraper.py --config configs/{name}.json
```
```
---
## Implementation Plan
### Phase 1: Foundation (Quick Fixes)
**Tasks:**
1. Fix `.txt` → `.md` renaming in downloader
2. Download all 3 variants (not just one)
3. Store all variants in `references/` with correct names
4. Remove content truncation (2500 chars → unlimited)
**Time:** 1-2 hours
**Files:** `cli/doc_scraper.py`, `cli/llms_txt_downloader.py`
### Phase 2: Catalog System
**Tasks:**
1. Implement `_generate_catalog()` method
2. Parse llms-small.md to extract sections
3. Build search index from topics
4. Generate `assets/catalog.json`
**Time:** 2-3 hours
**Files:** `cli/doc_scraper.py`
### Phase 3: Active Scripts
**Tasks:**
1. Create `scripts/search.py`
2. Create `scripts/list_topics.py`
3. Create `scripts/get_section.py`
4. Make scripts executable (`chmod +x`)
**Time:** 2-3 hours
**Files:** New scripts in `scripts/` template directory
### Phase 4: Template Updates
**Tasks:**
1. Create new active SKILL.md template
2. Update `create_enhanced_skill_md()` to use active template for llms.txt skills
3. Update documentation to explain active skills
**Time:** 1 hour
**Files:** `cli/doc_scraper.py`, `README.md`, `CLAUDE.md`
### Phase 5: Testing & Refinement
**Tasks:**
1. Test with Hono skill (has all 3 variants)
2. Test search accuracy
3. Measure context reduction
4. Document examples
**Time:** 2-3 hours
**Total Estimated Time:** 8-12 hours
---
## Migration Path
### Backward Compatibility
**Existing skills:** No changes (passive skills still work)
**New llms.txt skills:** Automatically use active architecture
**User choice:** Can disable via config flag
### Config Option
```json
{
"name": "hono",
"llms_txt_url": "https://hono.dev/llms-full.txt",
"active_skill": true, // NEW: Enable active architecture (default: true)
"base_url": "https://hono.dev/docs"
}
```
### Detection Logic
```python
# In _try_llms_txt()
active_mode = self.config.get('active_skill', True) # Default true
if active_mode:
# Download all variants, generate catalog, create scripts
self._build_active_skill(downloaded)
else:
# Traditional: single file, no scripts
self._build_passive_skill(downloaded)
```
---
## Benefits Analysis
### Context Efficiency
| Scenario | Passive Skill | Active Skill | Improvement |
|----------|---------------|--------------|-------------|
| Simple query | 203k loaded | 5k loaded | **40x reduction** |
| Multi-topic query | 203k loaded | 15k loaded | **13x reduction** |
| Deep dive | 203k loaded | 30k loaded | **6x reduction** |
### Data Fidelity
| Aspect | Passive | Active |
|--------|---------|--------|
| Content truncation | 36% lost | 0% lost |
| Code truncation | 600 chars max | Unlimited |
| Variants available | 1 | 3 |
### Agent Capabilities
**Passive Skills:**
- ❌ Cannot choose detail level
- ❌ Cannot search efficiently
- ❌ Must read entire context
- ❌ Limited by context window
**Active Skills:**
- ✅ Chooses appropriate detail level
- ✅ Searches catalog efficiently
- ✅ Loads only what's needed
- ✅ Unlimited documentation access
---
## Trade-offs
### Advantages
1. **Massive context reduction** (20-40x less per query)
2. **No content loss** (all 3 variants preserved)
3. **Correct file format** (.md not .txt)
4. **Agent autonomy** (tools to fetch docs)
5. **Scalable** (works with 1MB+ docs)
### Disadvantages
1. **Complexity** (scripts + catalog vs simple files)
2. **Initial overhead** (catalog generation)
3. **Agent learning curve** (must learn to use scripts)
4. **Dependency** (Python required to run scripts)
### Risk Mitigation
**Risk:** Scripts don't work in Claude's sandbox
**Mitigation:** Test thoroughly, provide fallback to passive mode
**Risk:** Catalog generation fails
**Mitigation:** Graceful degradation to single-file mode
**Risk:** Agent doesn't use scripts
**Mitigation:** Clear SKILL.md instructions, examples in quick reference
---
## Success Metrics
### Technical Metrics
- ✅ Context per query < 20k (down from 203k)
- ✅ All 3 variants downloaded and named correctly
- ✅ 0% content truncation
- ✅ Catalog generation < 5 seconds
- ✅ Search script < 1 second response time
### User Experience Metrics
- ✅ Agent successfully uses scripts without prompting
- ✅ Answers are equally or more accurate than passive mode
- ✅ Agent can handle queries about all documentation sections
- ✅ No "context limit exceeded" errors
---
## Future Enhancements
### Phase 6: Smart Caching
Cache frequently accessed sections in SKILL.md quick reference:
```python
# Track access frequency in catalog.json
"sections": [
{
"id": "middleware",
"access_count": 47, # NEW: Track usage
"last_accessed": "2025-10-24T14:30:00Z"
}
]
# Include top 10 most-accessed sections directly in SKILL.md
```
### Phase 7: Semantic Search
Use embeddings for better search:
```python
# Generate embeddings for each section
"sections": [
{
"id": "middleware",
"embedding": [...], # NEW: Vector embedding
"topics": ["cors", "auth"]
}
]
# In search.py: Use cosine similarity for better matches
```
### Phase 8: Progressive Loading
Load increasingly detailed docs:
```python
# First: Load llms.md (5.4k - overview)
# If insufficient: Load llms-small.md section (15k)
# If still insufficient: Load llms-full.md section (30k)
```
---
## Conclusion
Active skills represent a fundamental shift from **documentation repositories** to **documentation routers**. By treating skills as intelligent intermediaries rather than static dumps, we can:
1. **Eliminate context bloat** (40x reduction)
2. **Preserve full fidelity** (0% truncation)
3. **Enable agent autonomy** (tools to fetch docs)
4. **Scale indefinitely** (no size limits)
This design maintains backward compatibility while unlocking new capabilities for modern, LLM-optimized documentation sources like llms.txt.
**Recommendation:** Implement in phases, starting with foundation fixes, then catalog system, then active scripts. Test thoroughly with Hono before making it the default for all llms.txt-based skills.
---
## References
- Original brainstorming session: 2025-10-24
- llms.txt convention: https://llmstxt.org/
- Hono example: https://hono.dev/llms-full.txt
- Skill_Seekers repository: Current project
---
## Appendix: Example Workflows
### Example 1: Agent Searches for "Middleware"
```bash
# Agent runs:
python scripts/search.py "middleware"
# Script returns ~8k of middleware documentation from llms-full.md
# Agent uses that 8k to answer the question
# Total context used: 8k (not 319k!)
```
### Example 2: Agent Explores Documentation
```bash
# 1. Agent lists topics
python scripts/list_topics.py
# Returns: Table of contents (2k)
# 2. Agent picks a topic
python scripts/get_section.py "Routing"
# Returns: Complete Routing section (5k)
# 3. Agent searches related topics
python scripts/search.py "path parameters"
# Returns: Routing + Path section (7k)
# Total context used across 3 queries: 14k (not 3 × 319k = 957k!)
```
### Example 3: Agent Needs Quick Answer
```bash
# Agent uses quick variant for overview
python scripts/search.py "cors" --detail quick
# Returns: Short CORS explanation from llms-small.md (2k)
# If insufficient, agent can follow up with:
python scripts/get_section.py "Middleware" # Full section from llms-full.md
```
---
**Document Status:** Ready for review and implementation planning.
assets/repo/Skill_Seekers-development/docs/plans/2025-10-24-active-skills-phase1.md
+682
View File
@@ -0,0 +1,682 @@
# Active Skills Phase 1: Foundation Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Fix fundamental issues in llms.txt handling: rename .txt→.md, download all 3 variants, remove truncation.
**Architecture:** Modify existing llms.txt download/parse/build workflow to handle multiple variants correctly, store with proper extensions, and preserve complete content without truncation.
**Tech Stack:** Python 3.10+, requests, BeautifulSoup4, existing Skill_Seekers architecture
---
## Task 1: Add Multi-Variant Detection
**Files:**
- Modify: `cli/llms_txt_detector.py`
- Test: `tests/test_llms_txt_detector.py`
**Step 1: Write failing test for detect_all() method**
```python
# tests/test_llms_txt_detector.py (add new test)
def test_detect_all_variants():
"""Test detecting all llms.txt variants"""
from unittest.mock import patch, Mock
detector = LlmsTxtDetector("https://hono.dev/docs")
with patch('cli.llms_txt_detector.requests.head') as mock_head:
# Mock responses for different variants
def mock_response(url, **kwargs):
response = Mock()
# All 3 variants exist for Hono
if 'llms-full.txt' in url or 'llms.txt' in url or 'llms-small.txt' in url:
response.status_code = 200
else:
response.status_code = 404
return response
mock_head.side_effect = mock_response
variants = detector.detect_all()
assert len(variants) == 3
assert any(v['variant'] == 'full' for v in variants)
assert any(v['variant'] == 'standard' for v in variants)
assert any(v['variant'] == 'small' for v in variants)
assert all('url' in v for v in variants)
```
**Step 2: Run test to verify it fails**
Run: `source .venv/bin/activate && pytest tests/test_llms_txt_detector.py::test_detect_all_variants -v`
Expected: FAIL with "AttributeError: 'LlmsTxtDetector' object has no attribute 'detect_all'"
**Step 3: Implement detect_all() method**
```python
# cli/llms_txt_detector.py (add new method)
def detect_all(self) -> List[Dict[str, str]]:
"""
Detect all available llms.txt variants.
Returns:
List of dicts with 'url' and 'variant' keys for each found variant
"""
found_variants = []
for filename, variant in self.VARIANTS:
parsed = urlparse(self.base_url)
root_url = f"{parsed.scheme}://{parsed.netloc}"
url = f"{root_url}/{filename}"
if self._check_url_exists(url):
found_variants.append({
'url': url,
'variant': variant
})
return found_variants
```
**Step 4: Add import for List and Dict at top of file**
```python
# cli/llms_txt_detector.py (add to imports)
from typing import Optional, Dict, List
```
**Step 5: Run test to verify it passes**
Run: `source .venv/bin/activate && pytest tests/test_llms_txt_detector.py::test_detect_all_variants -v`
Expected: PASS
**Step 6: Commit**
```bash
git add cli/llms_txt_detector.py tests/test_llms_txt_detector.py
git commit -m "feat: add detect_all() for multi-variant detection"
```
---
## Task 2: Add File Extension Renaming to Downloader
**Files:**
- Modify: `cli/llms_txt_downloader.py`
- Test: `tests/test_llms_txt_downloader.py`
**Step 1: Write failing test for get_proper_filename() method**
```python
# tests/test_llms_txt_downloader.py (add new test)
def test_get_proper_filename():
"""Test filename conversion from .txt to .md"""
downloader = LlmsTxtDownloader("https://hono.dev/llms-full.txt")
filename = downloader.get_proper_filename()
assert filename == "llms-full.md"
assert not filename.endswith('.txt')
def test_get_proper_filename_standard():
"""Test standard variant naming"""
downloader = LlmsTxtDownloader("https://hono.dev/llms.txt")
filename = downloader.get_proper_filename()
assert filename == "llms.md"
def test_get_proper_filename_small():
"""Test small variant naming"""
downloader = LlmsTxtDownloader("https://hono.dev/llms-small.txt")
filename = downloader.get_proper_filename()
assert filename == "llms-small.md"
```
**Step 2: Run test to verify it fails**
Run: `source .venv/bin/activate && pytest tests/test_llms_txt_downloader.py::test_get_proper_filename -v`
Expected: FAIL with "AttributeError: 'LlmsTxtDownloader' object has no attribute 'get_proper_filename'"
**Step 3: Implement get_proper_filename() method**
```python
# cli/llms_txt_downloader.py (add new method)
def get_proper_filename(self) -> str:
"""
Extract filename from URL and convert .txt to .md
Returns:
Proper filename with .md extension
Examples:
https://hono.dev/llms-full.txt -> llms-full.md
https://hono.dev/llms.txt -> llms.md
https://hono.dev/llms-small.txt -> llms-small.md
"""
# Extract filename from URL
from urllib.parse import urlparse
parsed = urlparse(self.url)
filename = parsed.path.split('/')[-1]
# Replace .txt with .md
if filename.endswith('.txt'):
filename = filename[:-4] + '.md'
return filename
```
**Step 4: Run test to verify it passes**
Run: `source .venv/bin/activate && pytest tests/test_llms_txt_downloader.py::test_get_proper_filename -v`
Expected: PASS (all 3 tests)
**Step 5: Commit**
```bash
git add cli/llms_txt_downloader.py tests/test_llms_txt_downloader.py
git commit -m "feat: add get_proper_filename() for .txt to .md conversion"
```
---
## Task 3: Update _try_llms_txt() to Download All Variants
**Files:**
- Modify: `cli/doc_scraper.py:337-384` (_try_llms_txt method)
- Test: `tests/test_integration.py`
**Step 1: Write failing test for multi-variant download**
```python
# tests/test_integration.py (add to TestFullLlmsTxtWorkflow class)
def test_multi_variant_download(self):
"""Test downloading all 3 llms.txt variants"""
from unittest.mock import patch, Mock
import tempfile
import os
config = {
'name': 'test-multi-variant',
'base_url': 'https://hono.dev/docs'
}
# Mock all 3 variants
sample_full = "# Full\n" + "x" * 1000
sample_standard = "# Standard\n" + "x" * 200
sample_small = "# Small\n" + "x" * 500
with tempfile.TemporaryDirectory() as tmpdir:
with patch('cli.llms_txt_detector.requests.head') as mock_head, \
patch('cli.llms_txt_downloader.requests.get') as mock_get:
# Mock detection (all exist)
mock_head_response = Mock()
mock_head_response.status_code = 200
mock_head.return_value = mock_head_response
# Mock downloads
def mock_download(url, **kwargs):
response = Mock()
response.status_code = 200
if 'llms-full.txt' in url:
response.text = sample_full
elif 'llms-small.txt' in url:
response.text = sample_small
else: # llms.txt
response.text = sample_standard
return response
mock_get.side_effect = mock_download
# Run scraper
scraper = DocumentationScraper(config, dry_run=False)
result = scraper._try_llms_txt()
# Verify all 3 files created
refs_dir = os.path.join(scraper.skill_dir, 'references')
assert os.path.exists(os.path.join(refs_dir, 'llms-full.md'))
assert os.path.exists(os.path.join(refs_dir, 'llms.md'))
assert os.path.exists(os.path.join(refs_dir, 'llms-small.md'))
# Verify content not truncated
with open(os.path.join(refs_dir, 'llms-full.md')) as f:
content = f.read()
assert len(content) == len(sample_full)
```
**Step 2: Run test to verify it fails**
Run: `source .venv/bin/activate && pytest tests/test_integration.py::TestFullLlmsTxtWorkflow::test_multi_variant_download -v`
Expected: FAIL - only one file created, not all 3
**Step 3: Modify _try_llms_txt() to use detect_all()**
```python
# cli/doc_scraper.py (replace _try_llms_txt method, lines 337-384)
def _try_llms_txt(self) -> bool:
"""
Try to use llms.txt instead of HTML scraping.
Downloads ALL available variants and stores with .md extension.
Returns:
True if llms.txt was found and processed successfully
"""
print(f"\n🔍 Checking for llms.txt at {self.base_url}...")
# Check for explicit config URL first
explicit_url = self.config.get('llms_txt_url')
if explicit_url:
print(f"\n📌 Using explicit llms_txt_url from config: {explicit_url}")
downloader = LlmsTxtDownloader(explicit_url)
content = downloader.download()
if content:
# Save with proper .md extension
filename = downloader.get_proper_filename()
filepath = os.path.join(self.skill_dir, "references", filename)
os.makedirs(os.path.dirname(filepath), exist_ok=True)
with open(filepath, 'w', encoding='utf-8') as f:
f.write(content)
print(f" 💾 Saved {filename} ({len(content)} chars)")
# Parse and save pages
parser = LlmsTxtParser(content)
pages = parser.parse()
if pages:
for page in pages:
self.save_page(page)
self.pages.append(page)
self.llms_txt_detected = True
self.llms_txt_variant = 'explicit'
return True
# Auto-detection: Find ALL variants
detector = LlmsTxtDetector(self.base_url)
variants = detector.detect_all()
if not variants:
print("ℹ️ No llms.txt found, using HTML scraping")
return False
print(f"✅ Found {len(variants)} llms.txt variant(s)")
# Download ALL variants
downloaded = {}
for variant_info in variants:
url = variant_info['url']
variant = variant_info['variant']
print(f" 📥 Downloading {variant}...")
downloader = LlmsTxtDownloader(url)
content = downloader.download()
if content:
filename = downloader.get_proper_filename()
downloaded[variant] = {
'content': content,
'filename': filename,
'size': len(content)
}
print(f" ✓ {filename} ({len(content)} chars)")
if not downloaded:
print("⚠️ Failed to download any variants, falling back to HTML scraping")
return False
# Save ALL variants to references/
os.makedirs(os.path.join(self.skill_dir, "references"), exist_ok=True)
for variant, data in downloaded.items():
filepath = os.path.join(self.skill_dir, "references", data['filename'])
with open(filepath, 'w', encoding='utf-8') as f:
f.write(data['content'])
print(f" 💾 Saved {data['filename']}")
# Parse LARGEST variant for skill building
largest = max(downloaded.items(), key=lambda x: x[1]['size'])
print(f"\n📄 Parsing {largest[1]['filename']} for skill building...")
parser = LlmsTxtParser(largest[1]['content'])
pages = parser.parse()
if not pages:
print("⚠️ Failed to parse llms.txt, falling back to HTML scraping")
return False
print(f" ✓ Parsed {len(pages)} sections")
# Save pages for skill building
for page in pages:
self.save_page(page)
self.pages.append(page)
self.llms_txt_detected = True
self.llms_txt_variants = list(downloaded.keys())
return True
```
**Step 4: Add llms_txt_variants attribute to __init__**
```python
# cli/doc_scraper.py (in __init__ method, after llms_txt_variant line)
self.llms_txt_variants = [] # Track all downloaded variants
```
**Step 5: Run test to verify it passes**
Run: `source .venv/bin/activate && pytest tests/test_integration.py::TestFullLlmsTxtWorkflow::test_multi_variant_download -v`
Expected: PASS
**Step 6: Commit**
```bash
git add cli/doc_scraper.py tests/test_integration.py
git commit -m "feat: download all llms.txt variants with proper .md extension"
```
---
## Task 4: Remove Content Truncation
**Files:**
- Modify: `cli/doc_scraper.py:714-730` (create_reference_file method)
**Step 1: Write failing test for no truncation**
```python
# tests/test_integration.py (add new test)
def test_no_content_truncation():
"""Test that content is NOT truncated in reference files"""
from unittest.mock import Mock
import tempfile
import os
config = {
'name': 'test-no-truncate',
'base_url': 'https://example.com/docs'
}
# Create scraper with long content
scraper = DocumentationScraper(config, dry_run=False)
# Create page with content > 2500 chars
long_content = "x" * 5000
long_code = "y" * 1000
pages = [{
'title': 'Long Page',
'url': 'https://example.com/long',
'content': long_content,
'code_samples': [
{'code': long_code, 'language': 'python'}
],
'headings': []
}]
# Create reference file
scraper.create_reference_file('test', pages)
# Verify no truncation
ref_file = os.path.join(scraper.skill_dir, 'references', 'test.md')
with open(ref_file, 'r') as f:
content = f.read()
assert long_content in content # Full content included
assert long_code in content # Full code included
assert '[Content truncated]' not in content
assert '...' not in content or content.count('...') == 0
```
**Step 2: Run test to verify it fails**
Run: `source .venv/bin/activate && pytest tests/test_integration.py::test_no_content_truncation -v`
Expected: FAIL - content contains "[Content truncated]" or "..."
**Step 3: Remove truncation from create_reference_file()**
```python
# cli/doc_scraper.py (modify create_reference_file method, lines 712-731)
# OLD (line 714-716):
# if page.get('content'):
# content = page['content'][:2500]
# if len(page['content']) > 2500:
# content += "\n\n*[Content truncated]*"
# NEW (replace with):
if page.get('content'):
content = page['content'] # NO TRUNCATION
lines.append(content)
lines.append("")
# OLD (line 728-730):
# lines.append(code[:600])
# if len(code) > 600:
# lines.append("...")
# NEW (replace with):
lines.append(code) # NO TRUNCATION
# No "..." suffix
```
**Complete replacement of lines 712-731:**
```python
# cli/doc_scraper.py:712-731 (complete replacement)
# Content (NO TRUNCATION)
if page.get('content'):
lines.append(page['content'])
lines.append("")
# Code examples with language (NO TRUNCATION)
if page.get('code_samples'):
lines.append("**Examples:**\n")
for i, sample in enumerate(page['code_samples'][:4], 1):
lang = sample.get('language', 'unknown')
code = sample.get('code', sample if isinstance(sample, str) else '')
lines.append(f"Example {i} ({lang}):")
lines.append(f"```{lang}")
lines.append(code) # Full code, no truncation
lines.append("```\n")
```
**Step 4: Run test to verify it passes**
Run: `source .venv/bin/activate && pytest tests/test_integration.py::test_no_content_truncation -v`
Expected: PASS
**Step 5: Run full test suite to check for regressions**
Run: `source .venv/bin/activate && pytest tests/ -v`
Expected: All 201+ tests pass
**Step 6: Commit**
```bash
git add cli/doc_scraper.py tests/test_integration.py
git commit -m "feat: remove content truncation in reference files"
```
---
## Task 5: Update Documentation
**Files:**
- Modify: `docs/plans/2025-10-24-active-skills-design.md`
- Modify: `CHANGELOG.md`
**Step 1: Update design doc status**
```markdown
# docs/plans/2025-10-24-active-skills-design.md (update header)
**Status:** Phase 1 Implemented ✅
```
**Step 2: Add CHANGELOG entry**
```markdown
# CHANGELOG.md (add new section at top)
## [Unreleased]
### Added - Phase 1: Active Skills Foundation
- Multi-variant llms.txt detection: downloads all 3 variants (full, standard, small)
- Automatic .txt → .md file extension conversion
- No content truncation: preserves complete documentation
- `detect_all()` method for finding all llms.txt variants
- `get_proper_filename()` for correct .md naming
### Changed
- `_try_llms_txt()` now downloads all available variants instead of just one
- Reference files now contain complete content (no 2500 char limit)
- Code samples now include full code (no 600 char limit)
### Fixed
- File extension bug: llms.txt files now saved as .md
- Content loss: 0% truncation (was 36%)
```
**Step 3: Commit**
```bash
git add docs/plans/2025-10-24-active-skills-design.md CHANGELOG.md
git commit -m "docs: update status for Phase 1 completion"
```
---
## Task 6: Manual Verification
**Files:**
- None (manual testing)
**Step 1: Test with Hono config**
Run: `source .venv/bin/activate && python3 cli/doc_scraper.py --config configs/hono.json`
**Expected output:**
```
🔍 Checking for llms.txt at https://hono.dev/docs...
📌 Using explicit llms_txt_url from config: https://hono.dev/llms-full.txt
💾 Saved llms-full.md (319000 chars)
📄 Parsing llms-full.md for skill building...
✓ Parsed 93 sections
✅ Used llms.txt (explicit) - skipping HTML scraping
```
**Step 2: Verify all 3 files exist with correct extensions**
Run: `ls -lah output/hono/references/llms*.md`
Expected:
```
llms-full.md 319k
llms.md 5.4k
llms-small.md 176k
```
**Step 3: Verify no truncation in reference files**
Run: `grep -c "Content truncated" output/hono/references/*.md`
Expected: 0 matches (no truncation messages)
**Step 4: Check file sizes are correct**
Run: `wc -c output/hono/references/llms-full.md`
Expected: Should match original download size (~319k), not reduced to 203k
**Step 5: Verify all tests still pass**
Run: `source .venv/bin/activate && pytest tests/ -v`
Expected: All tests pass (201+)
---
## Completion Checklist
- [ ] Task 1: Multi-variant detection (detect_all)
- [ ] Task 2: File extension renaming (get_proper_filename)
- [ ] Task 3: Download all variants (_try_llms_txt)
- [ ] Task 4: Remove truncation (create_reference_file)
- [ ] Task 5: Update documentation
- [ ] Task 6: Manual verification
- [ ] All tests passing
- [ ] No regressions in existing functionality
---
## Success Criteria
**Technical:**
- ✅ All 3 variants downloaded when available
- ✅ Files saved with .md extension (not .txt)
- ✅ 0% content truncation (was 36%)
- ✅ All existing tests pass
- ✅ New tests cover all changes
**User Experience:**
- ✅ Hono skill has all 3 files: llms-full.md, llms.md, llms-small.md
- ✅ Reference files contain complete documentation
- ✅ No "[Content truncated]" messages in output
---
## Related Skills
- @superpowers:test-driven-development - Used throughout for TDD approach
- @superpowers:verification-before-completion - Used in Task 6 for manual verification
---
## Notes
- This plan implements Phase 1 from `docs/plans/2025-10-24-active-skills-design.md`
- Phase 2 (Catalog System) and Phase 3 (Active Scripts) will be separate plans
- All changes maintain backward compatibility with existing HTML scraping
- File extension fix (.txt → .md) is critical for proper skill functionality
---
## Estimated Time
- Task 1: 15 minutes
- Task 2: 15 minutes
- Task 3: 30 minutes
- Task 4: 20 minutes
- Task 5: 10 minutes
- Task 6: 15 minutes
**Total: ~1.5 hours**