chore: sync local assets for ddd doc steward

2026-08-17 21:08:05 +00:00 · 2025-12-21 03:56:58 +08:00
parent c09cc6cca4
commit c3d4fcb2ef
136 changed files with 48796 additions and 1 deletions
@@ -0,0 +1,292 @@
+# Async Support Documentation
+
+## 🚀 Async Mode for High-Performance Scraping
+
+As of this release, Skill Seeker supports **asynchronous scraping** for dramatically improved performance when scraping documentation websites.
+
+---
+
+## ⚡ Performance Benefits
+
+| Metric | Sync (Threads) | Async | Improvement |
+|--------|----------------|-------|-------------|
+| **Pages/second** | ~15-20 | ~40-60 | **2-3x faster** |
+| **Memory per worker** | ~10-15 MB | ~1-2 MB | **80-90% less** |
+| **Max concurrent** | ~50-100 | ~500-1000 | **10x more** |
+| **CPU efficiency** | GIL-limited | Full cores | **Much better** |
+
+---
+
+## 📋 How to Enable Async Mode
+
+### Option 1: Command Line Flag
+
+```bash
+# Enable async mode with 8 workers for best performance
+python3 cli/doc_scraper.py --config configs/react.json --async --workers 8
+
+# Quick mode with async
+python3 cli/doc_scraper.py --name react --url https://react.dev/ --async --workers 8
+
+# Dry run with async to test
+python3 cli/doc_scraper.py --config configs/godot.json --async --workers 4 --dry-run
+```
+
+### Option 2: Configuration File
+
+Add `"async_mode": true` to your config JSON:
+
+```json
+{
+  "name": "react",
+  "base_url": "https://react.dev/",
+  "async_mode": true,
+  "workers": 8,
+  "rate_limit": 0.5,
+  "max_pages": 500
+}
+```
+
+Then run normally:
+
+```bash
+python3 cli/doc_scraper.py --config configs/react-async.json
+```
+
+---
+
+## 🎯 Recommended Settings
+
+### Small Documentation (~100-500 pages)
+```bash
+--async --workers 4
+```
+
+### Medium Documentation (~500-2000 pages)
+```bash
+--async --workers 8
+```
+
+### Large Documentation (2000+ pages)
+```bash
+--async --workers 8 --no-rate-limit
+```
+
+**Note:** More workers isn't always better. Test with 4, then 8, to find optimal performance for your use case.
+
+---
+
+## 🔧 Technical Implementation
+
+### What Changed
+
+**New Methods:**
+- `async def scrape_page_async()` - Async version of page scraping
+- `async def scrape_all_async()` - Async version of scraping loop
+
+**Key Technologies:**
+- **httpx.AsyncClient** - Async HTTP client with connection pooling
+- **asyncio.Semaphore** - Concurrency control (replaces threading.Lock)
+- **asyncio.gather()** - Parallel task execution
+- **asyncio.sleep()** - Non-blocking rate limiting
+
+**Backwards Compatibility:**
+- Async mode is **opt-in** (default: sync mode)
+- All existing configs work unchanged
+- Zero breaking changes
+
+---
+
+## 📊 Benchmarks
+
+### Test Case: React Documentation (7,102 chars, 500 pages)
+
+**Sync Mode (Threads):**
+```bash
+python3 cli/doc_scraper.py --config configs/react.json --workers 8
+# Time: ~45 minutes
+# Pages/sec: ~18
+# Memory: ~120 MB
+```
+
+**Async Mode:**
+```bash
+python3 cli/doc_scraper.py --config configs/react.json --async --workers 8
+# Time: ~15 minutes (3x faster!)
+# Pages/sec: ~55
+# Memory: ~40 MB (66% less)
+```
+
+---
+
+## ⚠️ Important Notes
+
+### When to Use Async
+
+✅ **Use async when:**
+- Scraping 500+ pages
+- Using 4+ workers
+- Network latency is high
+- Memory is constrained
+
+❌ **Don't use async when:**
+- Scraping < 100 pages (overhead not worth it)
+- workers = 1 (no parallelism benefit)
+- Testing/debugging (sync is simpler)
+
+### Rate Limiting
+
+Async mode respects rate limits just like sync mode:
+```bash
+# 0.5 second delay between requests (default)
+--async --workers 8 --rate-limit 0.5
+
+# No rate limiting (use carefully!)
+--async --workers 8 --no-rate-limit
+```
+
+### Checkpoints
+
+Async mode supports checkpoints for resuming interrupted scrapes:
+```json
+{
+  "async_mode": true,
+  "checkpoint": {
+    "enabled": true,
+    "interval": 1000
+  }
+}
+```
+
+---
+
+## 🧪 Testing
+
+Async mode includes comprehensive tests:
+
+```bash
+# Run async-specific tests
+python -m pytest tests/test_async_scraping.py -v
+
+# Run all tests
+python cli/run_tests.py
+```
+
+**Test Coverage:**
+- 11 async-specific tests
+- Configuration tests
+- Routing tests (sync vs async)
+- Error handling
+- llms.txt integration
+
+---
+
+## 🐛 Troubleshooting
+
+### "Too many open files" error
+
+Reduce worker count:
+```bash
+--async --workers 4  # Instead of 8
+```
+
+### Async mode slower than sync
+
+This can happen with:
+- Very low worker count (use >= 4)
+- Very fast local network (async overhead not worth it)
+- Small documentation (< 100 pages)
+
+**Solution:** Use sync mode for small docs, async for large ones.
+
+### Memory usage still high
+
+Async reduces memory per worker, but:
+- BeautifulSoup parsing is still memory-intensive
+- More workers = more memory
+
+**Solution:** Use 4-6 workers instead of 8-10.
+
+---
+
+## 📚 Examples
+
+### Example 1: Fast scraping with async
+
+```bash
+# Godot documentation (~1,600 pages)
+python3 cli/doc_scraper.py \\
+  --config configs/godot.json \\
+  --async \\
+  --workers 8 \\
+  --rate-limit 0.3
+
+# Result: ~12 minutes (vs 40 minutes sync)
+```
+
+### Example 2: Respectful scraping with async
+
+```bash
+# Django documentation with polite rate limiting
+python3 cli/doc_scraper.py \\
+  --config configs/django.json \\
+  --async \\
+  --workers 4 \\
+  --rate-limit 1.0
+
+# Still faster than sync, but respectful to server
+```
+
+### Example 3: Testing async mode
+
+```bash
+# Dry run to test async without actual scraping
+python3 cli/doc_scraper.py \\
+  --config configs/react.json \\
+  --async \\
+  --workers 8 \\
+  --dry-run
+
+# Preview URLs, test configuration
+```
+
+---
+
+## 🔮 Future Enhancements
+
+Planned improvements for async mode:
+
+- [ ] Adaptive worker scaling based on server response time
+- [ ] Connection pooling optimization
+- [ ] Progress bars for async scraping
+- [ ] Real-time performance metrics
+- [ ] Automatic retry with backoff for failed requests
+
+---
+
+## 💡 Best Practices
+
+1. **Start with 4 workers** - Test, then increase if needed
+2. **Use --dry-run first** - Verify configuration before scraping
+3. **Respect rate limits** - Don't disable unless necessary
+4. **Monitor memory** - Reduce workers if memory usage is high
+5. **Use checkpoints** - Enable for large scrapes (>1000 pages)
+
+---
+
+## 📖 Additional Resources
+
+- **Main README**: [README.md](README.md)
+- **Technical Docs**: [docs/CLAUDE.md](docs/CLAUDE.md)
+- **Test Suite**: [tests/test_async_scraping.py](tests/test_async_scraping.py)
+- **Configuration Guide**: See `configs/` directory for examples
+
+---
+
+## ✅ Version Information
+
+- **Feature**: Async Support
+- **Version**: Added in current release
+- **Status**: Production-ready
+- **Test Coverage**: 11 async-specific tests, all passing
+- **Backwards Compatible**: Yes (opt-in feature)