ZhijuCen
151 lines
4.5 KiB
Markdown
151 lines
4.5 KiB
Markdown
# Extraction Workflow — Web Content to Markdown
|
|
|
|
Two-phase pipeline: download HTML (lossless) → convert to Markdown (offline).
|
|
|
|
```
|
|
Phase 1: sitemap XML → URL list → fetch HTML → save to html_cache/
|
|
Phase 2: html_cache/ HTML → parse → download images → write Markdown
|
|
```
|
|
|
|
- **Input**: `sitemaps/sitemap_book_en.xml`, `sitemaps/sitemap_docs_en.xml`
|
|
- **HTML cache**: `html_cache/book/`, `html_cache/docs/`
|
|
- **Output**: `skills/mql5/references/book/`, `skills/mql5/references/docs/`
|
|
- **Tool**: Python script using `requests` + `beautifulsoup4`
|
|
|
|
## Script — `scripts/extract.py`
|
|
|
|
### CLI
|
|
|
|
```bash
|
|
# Phase 1: download HTML
|
|
python scripts/extract.py download --sitemap sitemaps/sitemap_book_en.xml
|
|
python scripts/extract.py download --all
|
|
|
|
# Phase 2: convert to Markdown
|
|
python scripts/extract.py convert --sitemap sitemaps/sitemap_book_en.xml
|
|
python scripts/extract.py convert --all
|
|
|
|
# Debug single page (fetches + analyzes)
|
|
python scripts/extract.py debug URL
|
|
```
|
|
|
|
Options (download/convert):
|
|
- `--sitemap PATH` — single sitemap
|
|
- `--all` — all known sitemaps
|
|
- `--force` — re-process everything (ignore progress log)
|
|
- `--dry-run` — show plan without processing
|
|
|
|
### Phase 1 — Download
|
|
|
|
1. Parse sitemap XML → URL list
|
|
2. Build file map (naming convention, see `naming.md`)
|
|
3. For each URL:
|
|
- Skip if already cached (idempotent)
|
|
- Fetch HTML with retry + rate limiting (0.5s delay)
|
|
- Save raw HTML to `html_cache/{label}/{NN-chapter}/{NNNN-name}.html`
|
|
4. Progress: `html_cache/{label}/.download.log`
|
|
|
|
HTML files are saved as-is — no parsing, no transformation.
|
|
|
|
### Phase 2 — Convert
|
|
|
|
1. Parse sitemap XML → URL list (same mapping)
|
|
2. For each URL:
|
|
- Skip if already converted (idempotent)
|
|
- Read local HTML from `html_cache/`
|
|
- Parse `#help` content → elements
|
|
- Convert elements to Markdown
|
|
- Download images to `pics/` subfolder
|
|
- Write `.md` file
|
|
3. Progress: `{output_dir}/.convert.log`
|
|
|
|
### HTML Cache Structure
|
|
|
|
```
|
|
html_cache/
|
|
├── book/
|
|
│ ├── 0000-book.html
|
|
│ ├── .download.log
|
|
│ ├── 00-intro/
|
|
│ │ ├── 0001-intro.html
|
|
│ │ ├── 0002-intro-edit-compile-run.html
|
|
│ │ └── ...
|
|
│ └── ...
|
|
└── docs/
|
|
├── 0000-docs.html
|
|
├── .download.log
|
|
├── 00-basis/
|
|
│ ├── 0001-basis.html
|
|
│ └── ...
|
|
└── ...
|
|
```
|
|
|
|
Filenames mirror the markdown output (same 4-digit prefix, same derived name).
|
|
|
|
## Content Extraction Rules
|
|
|
|
### Main Content Selector
|
|
|
|
```css
|
|
#help > h1:nth-child(1)
|
|
```
|
|
|
|
Extract from this `<h1>` through to the last content element.
|
|
|
|
**Fallback**: If no `h1` exists (e.g. root pages like `/en/book`), use the
|
|
first `<Tag>` child of `#help` as the start element.
|
|
|
|
### Table Handling
|
|
|
|
| Has `<thead>` | Meaning | Render as |
|
|
|---------------|---------|-----------|
|
|
| Yes | Data table with headers | Markdown table with `|---|` separator |
|
|
| No | Code block or console output | Fenced code block (` ``` `) |
|
|
|
|
Tables without `<thead>` on mql5.com typically contain:
|
|
- MQL5 code (with `<span>` color classes for syntax highlighting)
|
|
- MetaTrader 5 Strategy Tester console output
|
|
|
|
### Image Handling
|
|
|
|
The `<img>` is often nested inside `<p class="p_ImageCaption">`.
|
|
|
|
**Alt text extraction**:
|
|
1. Try `img.find_next_sibling("p")`
|
|
2. If not found, try `img.parent.find_next_sibling("p")`
|
|
3. Fall back to `img.get("alt", "")`
|
|
|
|
**`<p>` containing `<img>`**: The `<p>` is treated as an image block —
|
|
the `<img>` is routed through the image download path.
|
|
|
|
### Code Blocks
|
|
|
|
- `<pre><code>` → fenced code block with language detection
|
|
- Language detected from CSS classes (`mql`, `cpp`, `sql`, etc.)
|
|
- Text whitespace preserved (no space-joining)
|
|
|
|
## Debug Target — TesterStatistics
|
|
|
|
URL: `https://www.mql5.com/en/book/automation/tester/tester_testerstatistics`
|
|
|
|
Content boundaries:
|
|
- Start: `#help > h1:nth-child(1)` → "Getting testing financial statistics: TesterStatistics"
|
|
- End: `p.p_Text:nth-child(45)`
|
|
|
|
Exercises: data tables (thead), code blocks (no thead), images, console output.
|
|
|
|
## Error Handling
|
|
|
|
| Error | Phase 1 | Phase 2 |
|
|
|-------|---------|---------|
|
|
| HTTP 404 | Skip, log, continue | N/A (offline) |
|
|
| HTTP 429/5xx | Retry with backoff | N/A |
|
|
| Missing HTML | N/A | Log warning, skip |
|
|
| Parse error | N/A | Log error, skip |
|
|
| Image download fail | N/A | Log warning, use original URL |
|
|
|
|
## Resumability
|
|
|
|
Each phase has its own progress log. Re-running skips completed items.
|
|
Use `--force` to re-process everything.
|