Add Codex workflow support

.agents/skills/bprr/SKILL.md

@@ -0,0 +1,61 @@
+---
+name: bprr
+description: Bulk PR reviewer for awesome-quant. Use when the user asks to review all open PRs, review unreviewed PRs, bulk review, or mentions "bprr". Reviews open PRs lacking the reviewed label and presents a summary before any merge/comment/label action.
+---
+
+# BPRR: Bulk PR Reviewer
+
+Review multiple open pull requests for README entry contributions.
+
+## Hard Rules
+
+1. Use GitHub MCP tools for PR operations. Do not use `gh` for PR review, comments, labels, closing, or merging.
+2. Filter out PRs with the `reviewed` label unless the user asks to re-review them.
+3. Do not comment, label, close, or merge until the user explicitly approves each action.
+4. Do not auto-merge all approved PRs unless the user explicitly selects that option.
+5. If GitHub MCP tools are unavailable, report the blocker and point to `docs/codex-setup.md`.
+
+## Workflow
+
+1. List open PRs sorted oldest first.
+2. Filter to unreviewed PRs by default.
+3. Fetch the default-branch `README.md` once for duplicate checks.
+4. For each PR, fetch details, labels, files, body, and diff.
+5. Review only added `README.md` entries unless the PR changes other files; flag other file changes as unusual.
+6. Validate each PR with the same checks as `$sprr`. Use `scripts/validate_readme.py --diff-from <base-ref>` when the PR branch is available locally; otherwise apply the validator's rules manually from the MCP diff.
+
+## Validation Checklist
+
+For each added entry, check:
+
+- Parser regex match.
+- Required language tags for new non-commercial entries.
+- Description period before optional `[GitHub](...)`.
+- `https://` URLs.
+- Exact optional `[GitHub](https://github.com/owner/repo)` format.
+- Correct category section.
+- Commercial placement under `Commercial & Proprietary Services`.
+- Duplicate project names or URLs.
+- Activity, archived status, and basic documentation for GitHub repos.
+- Clear rationale for multiple related projects in one PR.
+
+## Summary Output
+
+Present a table:
+
+```text
+| PR | Title | Author | Entries | Format | URL | Section | Duplicate | Verdict |
+|----|-------|--------|---------|--------|-----|---------|-----------|---------|
+```
+
+After the table, list any PRs that need detailed notes.
+
+Ask what to do next. Accept selections like:
+
+- `merge 123, 124`
+- `comment 125`
+- `close 126`
+- `all approved`
+- `none`
+
+Confirm each merge before executing it.

.agents/skills/sprr/SKILL.md

@@ -0,0 +1,81 @@
+---
+name: sprr
+description: Single PR reviewer for awesome-quant. Use when the user asks to review, validate, comment on, label, close, or merge one specific pull request that adds README.md entries. Triggers include "sprr", "review PR", "check PR", and "validate contribution".
+---
+
+# SPRR: Single PR Reviewer
+
+Review one pull request that adds entries to `README.md`.
+
+## Hard Rules
+
+1. Review only the PR number the user requested. If no PR number is given, ask for one.
+2. Use GitHub MCP tools for PR operations. Do not use `gh` for PR review, comments, labels, closing, or merging.
+3. Do not modify the PR until the user explicitly approves that action.
+4. Always present findings before asking whether to comment, label, close, or merge.
+5. Always ask before merging.
+6. Enforce `CONTRIBUTING.md` and `AGENTS.md` strictly for new entries.
+
+If GitHub MCP tools are unavailable, report that PR operations are blocked and point the user to `docs/codex-setup.md`.
+
+## Workflow
+
+1. Fetch PR details, files changed, labels, comments, and diff with GitHub MCP.
+2. Confirm whether the `reviewed` label or prior maintainer comments exist, then proceed with full validation anyway.
+3. Focus on added lines in `README.md`. Flag any other changed files as unusual for a normal contribution.
+4. Save or reconstruct the PR diff locally only if needed, then validate added README entries with:
+
+```bash
+uv run python scripts/validate_readme.py --diff-from <base-ref>
+```
+
+If a base ref is not locally available, apply the same checks manually from the fetched diff.
+
+## Validation Checklist
+
+For every added entry:
+
+- It matches `^\s*- \[(.*)\]\((.*)\) - (.*)$`.
+- New non-commercial entries include backtick language tags followed by ` - `.
+- Description ends with a period before optional `[GitHub](...)`.
+- URLs use `https://`.
+- Optional GitHub link uses `[GitHub](https://github.com/owner/repo)`.
+- Section placement matches the project's purpose.
+- Commercial/proprietary projects are under `Commercial & Proprietary Services`.
+- Project name and URLs are not duplicates of existing README entries.
+- GitHub projects are active, not archived, and documented.
+- Multiple projects in one PR are closely related and explained in the PR body.
+
+## Verdicts
+
+Use these verdicts:
+
+- `APPROVE`: entry is ready to merge.
+- `NEEDS CHANGES`: fixable format, section, URL, description, or documentation issue.
+- `REJECT`: duplicate, unrelated multi-project PR, empty PR description, archived/abandoned project, or other hard rejection.
+
+## Output Shape
+
+Report:
+
+```text
+PR #<number>: <title>
+Author: <author>
+Prior review: YES/NO
+Files changed: <files>
+Entries reviewed: <count>
+
+Findings:
+- <entry>: <status and reason>
+
+Verdict: APPROVE | NEEDS CHANGES | REJECT
+Recommended action: <merge/comment/close/no action>
+```
+
+Then ask for explicit approval before doing the recommended action.
+
+## Approved Actions After User Consent
+
+- If approved and user says to merge: merge with squash unless user requests otherwise.
+- If needs changes and user says to comment: leave one concise comment with all requested fixes, then add `reviewed` if available.
+- If rejected and user says to close: leave a polite rejection comment and close the PR.

.agents/skills/update-pypi-dates/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: update-pypi-dates
+description: Refresh tracked PyPI last-updated dates in awesome-quant README.md. Use when the user asks to update PyPI dates, refresh PyPI metadata, or run update-pypi-dates.
+---
+
+# Update PyPI Dates
+
+Refresh last release dates for selected PyPI packages listed in `README.md`.
+
+## Workflow
+
+1. Run the checker:
+
+```bash
+uv run python .agents/skills/update-pypi-dates/scripts/check_pypi_dates.py
+```
+
+2. Show the user which packages changed.
+3. Run README validation:
+
+```bash
+uv run python scripts/validate_readme.py
+```
+
+4. If the user asked for a commit, stage only the relevant files and commit.
+5. Do not push unless the user explicitly asks.
+
+## Tracked Packages
+
+The script currently tracks:
+
+- `qfrm`
+- `chinesestockapi`
+- `tushare`
+- `metatrader5` / `MetaTrader5`
+
+To add packages, edit `PYPI_PROJECTS` in `scripts/check_pypi_dates.py`.

.agents/skills/update-pypi-dates/scripts/check_pypi_dates.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+"""Check tracked PyPI projects and update README.md last-updated notes."""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+from pathlib import Path
+from urllib.request import urlopen
+
+PYPI_PROJECTS = {
+    "qfrm": "qfrm",
+    "chinesestockapi": "chinesestockapi",
+    "tushare": "tushare",
+    "metatrader5": "MetaTrader5",
+}
+
+README_PATH = Path(__file__).resolve().parents[4] / "README.md"
+
+
+def get_pypi_last_updated(package_name: str) -> str | None:
+    url = f"https://pypi.org/pypi/{package_name}/json"
+    try:
+        with urlopen(url, timeout=10) as response:
+            data = json.loads(response.read().decode("utf-8"))
+    except Exception as exc:
+        print(f"ERROR fetching {package_name}: {exc}", file=sys.stderr)
+        return None
+
+    releases = data.get("releases", {})
+    for version in sorted(releases.keys(), reverse=True):
+        files = releases[version]
+        if not files:
+            continue
+        upload_time = files[0].get("upload_time_iso_8601")
+        if upload_time:
+            return upload_time.split("T", 1)[0]
+    return None
+
+
+def find_readme_entry(readme_content: str, display_name: str) -> tuple[int | None, str | None]:
+    pattern = rf"- \[{re.escape(display_name)}\]\(https://pypi\.org/project/[\w.-]+/\)[^\n]*"
+    for line_num, line in enumerate(readme_content.splitlines(), 1):
+        if re.search(pattern, line):
+            return line_num, line
+    return None, None
+
+
+def update_readme_entry(readme_content: str, display_name: str, new_date: str) -> tuple[str, bool]:
+    lines = readme_content.splitlines()
+    line_num, entry = find_readme_entry(readme_content, display_name)
+    if line_num is None or entry is None:
+        print(f"WARN {display_name}: README entry not found")
+        return readme_content, False
+
+    old_line = lines[line_num - 1]
+    new_date_text = f"(Last updated: {new_date})"
+    existing_date_pattern = r"\(Last updated: \d{4}-\d{2}-\d{2}\)"
+
+    if re.search(existing_date_pattern, old_line):
+        new_line = re.sub(existing_date_pattern, new_date_text, old_line)
+    elif old_line.endswith("."):
+        new_line = old_line[:-1] + f" {new_date_text}."
+    else:
+        new_line = old_line + f" {new_date_text}"
+
+    if new_line == old_line:
+        return readme_content, False
+
+    lines[line_num - 1] = new_line
+    trailing_newline = "\n" if readme_content.endswith("\n") else ""
+    return "\n".join(lines) + trailing_newline, True
+
+
+def main() -> int:
+    if not README_PATH.exists():
+        print(f"ERROR README.md not found at {README_PATH}", file=sys.stderr)
+        return 1
+
+    readme_content = README_PATH.read_text(encoding="utf-8")
+    updated_content = readme_content
+    changes: list[tuple[str, str]] = []
+
+    print("Checking PyPI projects for last-updated dates...")
+    for display_name, package_name in PYPI_PROJECTS.items():
+        last_updated = get_pypi_last_updated(package_name)
+        if last_updated is None:
+            print(f"{display_name}: skipped")
+            continue
+
+        updated_content, changed = update_readme_entry(updated_content, display_name, last_updated)
+        if changed:
+            changes.append((display_name, last_updated))
+            print(f"{display_name}: updated to {last_updated}")
+        else:
+            print(f"{display_name}: {last_updated} (no change)")
+
+    if changes:
+        README_PATH.write_text(updated_content, encoding="utf-8")
+        print("Updated README.md:")
+        for name, date in changes:
+            print(f"- {name}: {date}")
+    else:
+        print("All tracked PyPI dates are current.")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.agents/skills/update-pypi-dates/scripts/update_pypi_dates.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+project_root="$(cd "$script_dir/../../../.." && pwd)"
+
+cd "$project_root"
+uv run python "$script_dir/check_pypi_dates.py"
+uv run python scripts/validate_readme.py

.github/workflows/build.yml

@@ -22,7 +22,7 @@ jobs:
     - uses: astral-sh/setup-uv@v6
     - uses: actions/setup-python@v5
       with:
-        python-version: '3.x'
+        python-version: '3.11'
     - name: Install dependencies
       run: |
         uv sync --no-install-project

.github/workflows/pr-validate.yml

@@ -0,0 +1,35 @@
+name: Validate PR
+
+on:
+  pull_request:
+    branches: [ main ]
+    paths:
+      - 'README.md'
+      - 'CONTRIBUTING.md'
+      - 'AGENTS.md'
+      - 'scripts/**'
+      - 'site/**'
+      - '.github/workflows/pr-validate.yml'
+
+permissions:
+  contents: read
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 0
+    - uses: astral-sh/setup-uv@v6
+    - uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+    - name: Install dependencies
+      run: uv sync --no-install-project
+    - name: Fetch base branch
+      run: git fetch origin ${{ github.base_ref }} --depth=1
+    - name: Validate added README entries
+      run: uv run python scripts/validate_readme.py --diff-from origin/${{ github.base_ref }}
+    - name: Generate static site
+      run: uv run python site/generate.py

AGENTS.md

@@ -1,11 +1,36 @@
 # AGENTS.md
 
+## Project Overview
+
+`awesome-quant` is a curated list of quantitative finance libraries, packages,
+and resources. The source of truth is `README.md`; the static website is
+generated into `site/index.html`.
+
+The README is organized by category headings (`##`), not by language headings.
+Language support is stored in inline backtick tags inside each entry.
+
+## Architecture
+
+1. `README.md` contains curated entries.
+2. `parse.py` parses `README.md`, enriches GitHub/CRAN/PyPI metadata, and writes
+   `site/projects.csv`.
+3. `site/generate.py` reads `site/projects.csv` when present, otherwise parses
+   `README.md` directly for local previews, then writes `site/index.html`.
+4. `.github/workflows/build.yml` runs the parser and generator, then deploys
+   `site/` to GitHub Pages.
+
 ## Commands
 
 ```bash
 # Install deps (requires Python 3.11+)
 uv sync --no-install-project
 
+# Validate README entries changed in a PR
+uv run python scripts/validate_readme.py --diff-from origin/main
+
+# Validate the full README; legacy format/duplicate/URL issues are warnings by default
+uv run python scripts/validate_readme.py
+
 # Run parser (requires GitHub token)
 GITHUB_ACCESS_TOKEN=<token> uv run python parse.py
 
@@ -13,36 +38,76 @@ GITHUB_ACCESS_TOKEN=<token> uv run python parse.py
 uv run python site/generate.py
 ```
 
-## GitHub MCP Tools
-
-This repo uses GitHub MCP tools for PR operations (not `gh` CLI). Key tools:
-- `github_list_pull_requests` - list open PRs
-- `github_get_pull_request` - get PR details
-- `github_get_pull_request_files` - get files changed
-- `github_merge_pull_request` - merge PRs
-- `github_add_issue_comment` - comment on PRs
-- `github_update_issue` - close/update issues and PRs
-- `github_get_file_contents` - fetch README.md from repo
-
 ## Entry Format
 
-The critical regex: `^\s*- \[(.*)\]\((.*)\) - (.*)$`
+The critical parser regex is:
 
-- Must include backtick language tags (e.g., `` `Python` ``)
+```text
-- Description must end with period
+^\s*- \[(.*)\]\((.*)\) - (.*)$
-- Commercial sites in "Commercial & Proprietary Services" section
+```
-- Language tag optional for commercial sites
 
-See CLAUDE.md and CONTRIBUTING.md for full details.
+Accepted entry examples:
+
+```markdown
+- [Project Name](https://github.com/owner/repo) - `Python` - Description ending with a period.
+- [Project Name](https://github.com/owner/repo) - `Python` `Rust` - Description ending with a period.
+- [Project Name](https://site.com) - `Python` - Description ending with a period. [GitHub](https://github.com/owner/repo)
+- [Package Name](https://cran.r-project.org/package=pkg) - `R` - Description ending with a period.
+- [package-name](https://pypi.org/project/pkg/) - `Python` - Description ending with a period.
+```
+
+Rules:
+
+- New non-commercial entries must include one or more backtick language tags
+  followed by ` - `.
+- Descriptions must end with a period before the optional `[GitHub](...)` link.
+- URLs in new entries must use `https://`.
+- Commercial/proprietary projects belong in `## Commercial & Proprietary Services`.
+- Language tags are optional only for commercial services and other non-language
+  sections such as `Related Lists`.
+- Use `CONTRIBUTING.md` as the user-facing source for contribution rules.
 
 ## PR Review Workflow
 
-1. Always ask user before merging
+This repo receives most contributions as pull requests adding README entries.
-2. Check section placement (not Market Data if commercial)
+Use Codex repo skills for repeatable review workflows:
-3. Check for duplicates in README
-4. Use "reviewed" label after commenting
 
-## Skill Names
+- `$sprr` - single PR reviewer.
+- `$bprr` - bulk PR reviewer.
+- `$update-pypi-dates` - refresh tracked PyPI last-updated dates.
 
-- `sprr` - Single PR reviewer (formerly review-pr)
+Review requirements:
-- `bprr` - Bulk PR reviewer
+
+1. Only review the PRs the user asked you to review.
+2. Use GitHub MCP tools for PR operations, not the `gh` CLI.
+3. Present findings before any PR-modifying action.
+4. Always ask the user before commenting, labeling, closing, or merging.
+5. Check section placement, duplicates in `README.md`, entry format, URL format,
+   and project activity/documentation.
+6. Use the `reviewed` label only after the user approves a review comment.
+
+## GitHub MCP Tools
+
+This project expects Codex to have a GitHub MCP server configured for PR work.
+Useful tool capabilities include:
+
+- List open pull requests.
+- Get PR details, files, comments, labels, and diffs.
+- Fetch `README.md` from the default branch for duplicate checks.
+- Add PR comments and labels.
+- Merge pull requests after explicit user approval.
+
+See `docs/codex-setup.md` for local Codex setup notes. If GitHub MCP tools are
+not available in a session, report that PR operations are blocked and ask the
+user to configure the MCP server.
+
+## Local Validation Notes
+
+- `scripts/validate_readme.py --diff-from <ref>` is intended for PR checks. It
+  validates only added README entries and treats duplicates/URL issues as errors
+  for those additions.
+- `scripts/validate_readme.py` validates the full README. Legacy format, duplicate, and
+  URL problems are reported as warnings unless strict flags are used.
+- `parse.py` requires `GITHUB_ACCESS_TOKEN`; `site/generate.py` can run without
+  the token if `site/projects.csv` already exists or if it falls back to direct
+  README parsing.

docs/codex-setup.md

@@ -0,0 +1,80 @@
+# Codex Setup
+
+This repo is ready for Codex through `AGENTS.md` and repo-scoped skills in
+`.agents/skills`.
+
+## What Codex Loads
+
+- `AGENTS.md` contains durable repository instructions.
+- `.agents/skills/sprr` reviews one PR.
+- `.agents/skills/bprr` reviews open PRs in bulk.
+- `.agents/skills/update-pypi-dates` refreshes tracked PyPI dates.
+
+Restart Codex after adding or changing skills if they do not appear in skill
+selection.
+
+## GitHub MCP Requirement
+
+PR review workflows require a GitHub MCP server. Configure it in your own Codex
+configuration, usually `~/.codex/config.toml`. Do not commit personal access
+tokens or local auth files to this repository.
+
+After configuring GitHub MCP, start Codex in this repository and run:
+
+```text
+/mcp
+```
+
+Confirm that a GitHub MCP server is enabled and exposes tools that can:
+
+- list pull requests,
+- read PR details, files, labels, comments, and diffs,
+- read `README.md` from the default branch,
+- add comments and labels,
+- merge pull requests after explicit user approval.
+
+If those tools are missing, `$sprr` and `$bprr` should report that PR operations
+are blocked instead of falling back to `gh`.
+
+## Local Validation
+
+Install dependencies once:
+
+```bash
+uv sync --no-install-project
+```
+
+Validate README entries added relative to the base branch:
+
+```bash
+uv run python scripts/validate_readme.py --diff-from origin/main
+```
+
+Validate the full README:
+
+```bash
+uv run python scripts/validate_readme.py
+```
+
+Full validation reports legacy duplicate, URL, and format issues as warnings by
+default. PR-added entries treat those issues as errors.
+
+## Parser And Site Commands
+
+`parse.py` needs a GitHub token because it enriches entries with GitHub metadata:
+
+```bash
+GITHUB_ACCESS_TOKEN=<token> uv run python parse.py
+```
+
+The site generator can run locally without a token when it can use existing CSV
+data or parse `README.md` directly:
+
+```bash
+uv run python site/generate.py
+```
+
+## Claude Files
+
+The existing `.claude/` files and `CLAUDE.md` are kept for compatibility. Codex
+uses `AGENTS.md` and `.agents/skills` as the canonical workflow surfaces.

parse.py

@@ -8,9 +8,24 @@ from urllib.request import urlopen
 import pandas as pd
 from github import Auth, Github
 
-# using an access token
+from scripts.readme_entries import (
-auth = Auth.Token(os.environ["GITHUB_ACCESS_TOKEN"])
+    BADGE_RE,
-g = Github(auth=auth)
+    ENTRY_RE,
+    HEADING_RE,
+    extract_languages,
+    slugify,
+)
+
+_github_client = None
+
+
+def get_github_client():
+    """Create the GitHub client lazily so importing this module is token-free."""
+    global _github_client
+    if _github_client is None:
+        auth = Auth.Token(os.environ["GITHUB_ACCESS_TOKEN"])
+        _github_client = Github(auth=auth)
+    return _github_client
 
 
 def extract_repo(url):
@@ -128,39 +143,11 @@ def get_pypi_last_updated(url):
         return ""
 
 
-def slugify(text):
-    """Convert text to lowercase hyphen-separated slug."""
-    text = text.lower().strip()
-    text = re.sub(r"[&/]+", "-", text)
-    text = re.sub(r"[^\w\s-]", "", text)
-    text = re.sub(r"[\s_]+", "-", text)
-    text = re.sub(r"-+", "-", text)
-    return text.strip("-")
-
-
-re_langs = re.compile(r'^((?:`[^`]+`\s*)+)-\s*(.*)$')
-
-
-def extract_languages(description: str) -> tuple[list[str], str]:
-    """Extract inline language tags from description.
-
-    Returns (languages, clean_description).
-    E.g. "`Python` `Rust` - High-performance..." -> (["Python", "Rust"], "High-performance...")
-    """
-    m = re_langs.match(description)
-    if m:
-        lang_str = m.group(1)
-        clean_desc = m.group(2)
-        langs = re.findall(r'`([^`]+)`', lang_str)
-        return langs, clean_desc
-    return [], description
-
-
 def get_repo_info(repo):
     """Fetch last commit date and star count from GitHub."""
     try:
         if repo:
-            r = g.get_repo(repo)
+            r = get_github_client().get_repo(repo)
             cs = r.get_commits()
             last_commit = cs[0].commit.author.date.strftime("%Y-%m-%d")
             stars = r.stargazers_count
@@ -204,7 +191,7 @@ class Project(Thread):
 
         is_cran = "cran.r-project.org" in primary_url
         is_pypi = "pypi.org" in primary_url or "pypi.python.org" in primary_url
-        is_commercial = self._language == "Commercial & Proprietary Services"
+        is_commercial = self._category == "Commercial & Proprietary Services"
 
         # For CRAN projects, scrape the CRAN page for GitHub URL and published date
         cran_date = ""
@@ -248,48 +235,51 @@ class Project(Thread):
         )
 
 
-projects = []
+def main():
+    projects = []
 
-with open("README.md", "r", encoding="utf8") as f:
+    with open("README.md", "r", encoding="utf8") as f:
-    ret = re.compile(r"^(#+) (.*)$")
+        ret = HEADING_RE
-    rex = re.compile(r"^\s*- \[(.*)\]\((.*)\) - (.*)$")
+        rex = ENTRY_RE
-    re_badge = re.compile(r"\s*!\[[^\]]*\]\([^)]*\)\s*")
+        re_badge = BADGE_RE
-    m_titles = []
+        current_category = ""
-    last_head_level = 0
+        for line in f:
-    current_category = ""
+            line = re_badge.sub(" ", line)
-    for line in f:
+            m = rex.match(line)
-        line = re_badge.sub(" ", line)
-        m = rex.match(line)
-        if m:
-            raw_desc = m.group(3).strip()
-
-            # Extract language tags from description
-            languages, clean_description = extract_languages(raw_desc)
-            primary_language = languages[0] if languages else ""
-
-            p = Project(
-                m,
-                primary_language,
-                current_category,
-                current_category,
-            )
-            p.languages = languages
-            p.clean_description = clean_description
-            p.start()
-            projects.append(p)
-        else:
-            m = ret.match(line)
             if m:
-                hrs = m.group(1)
+                raw_desc = m.group(3).strip()
-                title = m.group(2).strip()
-                if len(hrs) == 2 and title != "Contents":
-                    current_category = title
 
-while True:
+                # Extract language tags from description
-    checks = [not p.is_alive() for p in projects]
+                languages, clean_description = extract_languages(raw_desc)
-    if all(checks):
+                primary_language = languages[0] if languages else ""
-        break
 
-projects = [p.regs for p in projects]
+                p = Project(
-df = pd.DataFrame(projects)
+                    m,
-df.to_csv("site/projects.csv", index=False)
+                    primary_language,
+                    current_category,
+                    current_category,
+                )
+                p.languages = languages
+                p.clean_description = clean_description
+                p.start()
+                projects.append(p)
+            else:
+                m = ret.match(line)
+                if m:
+                    hrs = m.group(1)
+                    title = m.group(2).strip()
+                    if len(hrs) == 2 and title != "Contents":
+                        current_category = title
+
+    while True:
+        checks = [not p.is_alive() for p in projects]
+        if all(checks):
+            break
+
+    projects = [p.regs for p in projects]
+    df = pd.DataFrame(projects)
+    df.to_csv("site/projects.csv", index=False)
+
+
+if __name__ == "__main__":
+    main()

scripts/readme_entries.py

@@ -0,0 +1,117 @@
+"""Shared README entry parsing helpers for awesome-quant."""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Iterable
+
+ENTRY_RE = re.compile(r"^\s*- \[(.*)\]\((.*)\) - (.*)$")
+HEADING_RE = re.compile(r"^(#+) (.*)$")
+BADGE_RE = re.compile(r"\s*!\[[^\]]*\]\([^)]*\)\s*")
+LANGUAGE_PREFIX_RE = re.compile(r"^((?:`[^`]+`\s*)+)-\s*(.*)$")
+GITHUB_LINK_RE = re.compile(r"\[GitHub\]\((https://github\.com/[\w-]+/[-\w.]+)\)")
+MARKDOWN_URL_RE = re.compile(r"\[[^\]]+\]\(([^)]+)\)")
+
+NO_LANGUAGE_REQUIRED_SECTIONS = {
+    "Commercial & Proprietary Services",
+    "Cross-Language Frameworks",
+    "Reproducing Works, Training & Books",
+    "Related Lists",
+}
+
+VALID_SECTIONS = [
+    "Numerical Libraries & Data Structures",
+    "Financial Instruments & Pricing",
+    "Technical Indicators",
+    "Trading & Backtesting",
+    "Portfolio Optimization & Risk Analysis",
+    "Factor Analysis",
+    "Sentiment Analysis & Alternative Data",
+    "Time Series Analysis",
+    "Market Data & Data Sources",
+    "Prediction Markets",
+    "Calendars & Market Hours",
+    "Visualization",
+    "Excel & Spreadsheet Integration",
+    "Quant Research Environments",
+    "Cross-Language Frameworks",
+    "Reproducing Works, Training & Books",
+    "Commercial & Proprietary Services",
+    "Related Lists",
+]
+
+
+@dataclass(frozen=True)
+class ReadmeEntry:
+    line_number: int
+    raw_line: str
+    section: str
+    name: str
+    url: str
+    tail: str
+    languages: list[str]
+    description: str
+
+    @property
+    def markdown_urls(self) -> list[str]:
+        return MARKDOWN_URL_RE.findall(self.raw_line)
+
+    @property
+    def github_url(self) -> str:
+        if "github.com" in self.url:
+            return self.url
+        match = GITHUB_LINK_RE.search(self.description)
+        return match.group(1) if match else ""
+
+
+def slugify(text: str) -> str:
+    """Convert text to lowercase hyphen-separated slug."""
+    text = text.lower().strip()
+    text = re.sub(r"[&/]+", "-", text)
+    text = re.sub(r"[^\w\s-]", "", text)
+    text = re.sub(r"[\s_]+", "-", text)
+    text = re.sub(r"-+", "-", text)
+    return text.strip("-")
+
+
+def extract_languages(description: str) -> tuple[list[str], str]:
+    """Extract leading backtick language tags from an entry description."""
+    match = LANGUAGE_PREFIX_RE.match(description)
+    if not match:
+        return [], description
+    lang_str = match.group(1)
+    clean_description = match.group(2)
+    return re.findall(r"`([^`]+)`", lang_str), clean_description
+
+
+def iter_readme_entries(path: str | Path) -> Iterable[ReadmeEntry]:
+    """Yield parsed README entries with line numbers and current h2 section."""
+    current_section = ""
+    with Path(path).open("r", encoding="utf-8") as handle:
+        for line_number, raw_line in enumerate(handle, 1):
+            line = BADGE_RE.sub(" ", raw_line.rstrip("\n"))
+            heading = HEADING_RE.match(line)
+            if heading:
+                level, title = heading.groups()
+                if len(level) == 2 and title.strip() != "Contents":
+                    current_section = title.strip()
+                continue
+
+            match = ENTRY_RE.match(line)
+            if not match:
+                continue
+
+            tail = match.group(3).strip()
+            languages, description = extract_languages(tail)
+            yield ReadmeEntry(
+                line_number=line_number,
+                raw_line=raw_line.rstrip("\n"),
+                section=current_section,
+                name=match.group(1).strip(),
+                url=match.group(2).strip(),
+                tail=tail,
+                languages=languages,
+                description=description.strip(),
+            )

scripts/validate_readme.py

@@ -0,0 +1,260 @@
+#!/usr/bin/env python3
+"""Validate awesome-quant README entries.
+
+Default mode validates the full README and reports legacy duplicate/http URL
+issues as warnings. With --diff-from, only added README entry lines are checked
+and duplicate/url issues are errors for those additions.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+from collections import defaultdict
+from dataclasses import dataclass
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+from scripts.readme_entries import (
+    ENTRY_RE,
+    GITHUB_LINK_RE,
+    HEADING_RE,
+    MARKDOWN_URL_RE,
+    NO_LANGUAGE_REQUIRED_SECTIONS,
+    VALID_SECTIONS,
+    ReadmeEntry,
+    extract_languages,
+    iter_readme_entries,
+)
+
+
+@dataclass(frozen=True)
+class Issue:
+    severity: str
+    line_number: int
+    code: str
+    message: str
+
+
+def normalize_name(name: str) -> str:
+    return " ".join(name.casefold().split())
+
+
+def normalize_url(url: str) -> str:
+    return url.strip().rstrip("/").casefold()
+
+
+def read_added_line_numbers(diff_from: str, readme_path: Path) -> set[int]:
+    path_arg = str(readme_path)
+    commands = [
+        ["git", "diff", "--unified=0", f"{diff_from}...HEAD", "--", path_arg],
+        ["git", "diff", "--unified=0", diff_from, "--", path_arg],
+    ]
+    last_error = ""
+    for command in commands:
+        proc = subprocess.run(command, text=True, capture_output=True, check=False)
+        if proc.returncode == 0:
+            return parse_added_lines_from_diff(proc.stdout)
+        last_error = proc.stderr.strip()
+    raise RuntimeError(f"git diff failed for {diff_from}: {last_error}")
+
+
+def parse_added_lines_from_diff(diff_text: str) -> set[int]:
+    added: set[int] = set()
+    new_line: int | None = None
+
+    for line in diff_text.splitlines():
+        if line.startswith("@@ "):
+            marker = line.split(" ")[2]
+            start = marker.split(",", 1)[0].lstrip("+")
+            new_line = int(start)
+            continue
+
+        if new_line is None:
+            continue
+
+        if line.startswith("+++") or line.startswith("---"):
+            continue
+        if line.startswith("+"):
+            added.add(new_line)
+            new_line += 1
+        elif line.startswith("-"):
+            continue
+        else:
+            new_line += 1
+
+    return added
+
+
+def build_duplicate_indexes(entries: list[ReadmeEntry]) -> tuple[dict[str, list[ReadmeEntry]], dict[str, list[ReadmeEntry]]]:
+    names: dict[str, list[ReadmeEntry]] = defaultdict(list)
+    urls: dict[str, list[ReadmeEntry]] = defaultdict(list)
+    for entry in entries:
+        names[normalize_name(entry.name)].append(entry)
+        for url in entry.markdown_urls:
+            urls[normalize_url(url)].append(entry)
+    return names, urls
+
+
+def validate_entry(
+    entry: ReadmeEntry,
+    *,
+    duplicate_names: dict[str, list[ReadmeEntry]],
+    duplicate_urls: dict[str, list[ReadmeEntry]],
+    strict_duplicates: bool,
+    strict_urls: bool,
+    strict_format: bool,
+) -> list[Issue]:
+    issues: list[Issue] = []
+    line = entry.line_number
+
+    if entry.section not in VALID_SECTIONS:
+        issues.append(Issue("error", line, "section", f"entry is under unknown section {entry.section!r}"))
+
+    format_severity = "error" if strict_format else "warning"
+
+    if entry.section not in NO_LANGUAGE_REQUIRED_SECTIONS and not entry.languages:
+        issues.append(Issue(format_severity, line, "language", "missing required backtick language tag prefix"))
+
+    if entry.languages:
+        _languages, clean_description = extract_languages(entry.tail)
+        if clean_description == entry.tail:
+            issues.append(Issue(format_severity, line, "language", "language tags must be followed by ' - '"))
+
+    github_label_count = entry.description.count("[GitHub](")
+    if github_label_count and not GITHUB_LINK_RE.search(entry.description):
+        issues.append(Issue(format_severity, line, "github-link", "optional GitHub link must use [GitHub](https://github.com/owner/repo)"))
+
+    github_match = GITHUB_LINK_RE.search(entry.description)
+    description_to_check = entry.description[: github_match.start()].rstrip() if github_match else entry.description.rstrip()
+    # Allow trailing reference links after the sentence, e.g. [GitHub], [Website], or ([Demo](...)).
+    previous = None
+    while previous != description_to_check:
+        previous = description_to_check
+        description_to_check = re.sub(r"\s*\[[^\]]+\]\([^)]+\)\s*$", "", description_to_check).rstrip()
+        description_to_check = re.sub(r"\s*\(\[[^\]]+\]\([^)]+\)\)\s*$", "", description_to_check).rstrip()
+    if description_to_check and not description_to_check.endswith("."):
+        issues.append(Issue(format_severity, line, "period", "description must end with a period before optional trailing link"))
+
+    for url in entry.markdown_urls:
+        if not url.startswith("https://"):
+            severity = "error" if strict_urls else "warning"
+            issues.append(Issue(severity, line, "url", f"URL should use https://: {url}"))
+
+    duplicate_name_entries = [item for item in duplicate_names[normalize_name(entry.name)] if item.line_number != line]
+    if duplicate_name_entries:
+        severity = "error" if strict_duplicates else "warning"
+        lines = ", ".join(str(item.line_number) for item in duplicate_name_entries)
+        issues.append(Issue(severity, line, "duplicate-name", f"project name duplicates line(s): {lines}"))
+
+    seen_duplicate_url_lines: set[int] = set()
+    for url in entry.markdown_urls:
+        duplicate_url_entries = [item for item in duplicate_urls[normalize_url(url)] if item.line_number != line]
+        for item in duplicate_url_entries:
+            seen_duplicate_url_lines.add(item.line_number)
+    if seen_duplicate_url_lines:
+        severity = "error" if strict_duplicates else "warning"
+        lines = ", ".join(str(line_number) for line_number in sorted(seen_duplicate_url_lines))
+        issues.append(Issue(severity, line, "duplicate-url", f"URL duplicates line(s): {lines}"))
+
+    return issues
+
+
+def validate_malformed_added_lines(readme_path: Path, added_lines: set[int]) -> list[Issue]:
+    issues: list[Issue] = []
+    lines = readme_path.read_text(encoding="utf-8").splitlines()
+    for line_number in sorted(added_lines):
+        if line_number < 1 or line_number > len(lines):
+            continue
+        line = lines[line_number - 1]
+        stripped = line.strip()
+        if not stripped or HEADING_RE.match(stripped):
+            continue
+        if stripped.startswith("- ") and not ENTRY_RE.match(line):
+            issues.append(Issue("error", line_number, "format", "added README bullet does not match entry regex"))
+    return issues
+
+
+def format_issue(issue: Issue) -> str:
+    location = f"line {issue.line_number}" if issue.line_number else "README"
+    return f"{issue.severity.upper()} {location} [{issue.code}] {issue.message}"
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="Validate awesome-quant README entries.")
+    parser.add_argument("--readme", default="README.md", help="README path to validate")
+    parser.add_argument("--diff-from", help="validate only README lines added relative to this git ref")
+    parser.add_argument("--strict-duplicates", action="store_true", help="treat duplicate names/URLs as errors in full validation")
+    parser.add_argument("--strict-urls", action="store_true", help="treat non-https URLs as errors in full validation")
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    readme_path = Path(args.readme)
+    if not readme_path.exists():
+        print(f"ERROR README not found: {readme_path}", file=sys.stderr)
+        return 2
+
+    entries = list(iter_readme_entries(readme_path))
+    duplicate_names, duplicate_urls = build_duplicate_indexes(entries)
+
+    added_lines: set[int] | None = None
+    if args.diff_from:
+        try:
+            added_lines = read_added_line_numbers(args.diff_from, readme_path)
+        except RuntimeError as exc:
+            print(f"ERROR {exc}", file=sys.stderr)
+            return 2
+        selected_entries = [entry for entry in entries if entry.line_number in added_lines]
+        strict_duplicates = True
+        strict_urls = True
+        strict_format = True
+    else:
+        selected_entries = entries
+        strict_duplicates = args.strict_duplicates
+        strict_urls = args.strict_urls
+        strict_format = False
+
+    issues: list[Issue] = []
+    if added_lines is not None:
+        issues.extend(validate_malformed_added_lines(readme_path, added_lines))
+
+    for entry in selected_entries:
+        issues.extend(
+            validate_entry(
+                entry,
+                duplicate_names=duplicate_names,
+                duplicate_urls=duplicate_urls,
+                strict_duplicates=strict_duplicates,
+                strict_urls=strict_urls,
+                strict_format=strict_format,
+            )
+        )
+
+    errors = [issue for issue in issues if issue.severity == "error"]
+    warnings = [issue for issue in issues if issue.severity == "warning"]
+
+    scope = f"added README lines relative to {args.diff_from}" if args.diff_from else "full README"
+    print(f"Validated {len(selected_entries)} entries in {scope}.")
+    for issue in sorted(issues, key=lambda item: (item.line_number, item.severity, item.code)):
+        print(format_issue(issue))
+
+    if errors:
+        print(f"Validation failed: {len(errors)} error(s), {len(warnings)} warning(s).")
+        return 1
+
+    if warnings:
+        print(f"Validation passed with {len(warnings)} warning(s).")
+    else:
+        print("Validation passed.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())