Add skills for managing PyPI project update dates and syncing README.md

2026-07-27 20:07:50 +00:00 · 2026-03-22 09:43:18 -03:00
parent 9405b48526
commit fc206ea4f3
6 changed files with 566 additions and 0 deletions
@@ -0,0 +1,108 @@
+---
+name: review-pr
+description: Review and validate pull requests that add new library entries to the awesome-quant README.md. Use this skill whenever the user asks to review PRs, check contributions, validate submissions, triage pull requests, or mentions anything about incoming entries or additions to the curated list. Also triggers for "review pr", "check prs", "merge contributions", or any PR-related workflow in this repo.
+---
+
+# Review PR Skill
+
+Review one open pull request at a time for the awesome-quant curated list. Always start with the oldest unreviewed open PR.
+
+## Workflow
+
+### Step 1: Find the oldest open PR
+
+Use the GitHub MCP tools to list open pull requests sorted by creation date (ascending). Pick the oldest one. Show the user which PR you're reviewing (number, title, author, creation date).
+
+### Step 2: Check for merge conflicts
+
+Use the GitHub MCP tools to read the PR details. If the PR has merge conflicts, report this to the user and stop — the contributor needs to resolve conflicts before review can proceed. Leave a polite comment asking them to rebase.
+
+### Step 3: Fetch the diff and extract added lines
+
+Read the PR diff. Focus only on changes to `README.md`. If the PR modifies files other than `README.md` (like `parse.py`, `site/`, etc.), flag this as unusual — most contributions should only touch `README.md`.
+
+### Step 4: Validate each added entry
+
+For every new line added to `README.md`, check the following:
+
+#### 4a. Entry format
+
+Each entry MUST match this exact pattern:
+
+```
+- [Project Name](https://url) - Short description ending with a period.
+```
+
+Specifically:
+- Starts with `- ` (dash + space)
+- Followed by a markdown link `[Name](URL)`
+- Followed by ` - ` (space, dash, space)
+- Followed by a description that ends with a period `.`
+
+The regex used by `parse.py` to extract entries is: `^\s*- \[(.*)\]\((.*)\) - (.*)$`
+
+If the entry doesn't match, report exactly what's wrong (missing period, wrong separator, etc.).
+
+#### 4b. URL validation
+
+- **GitHub URLs are preferred.** If the URL points to `github.com`, that's ideal — no warning needed.
+- **Non-GitHub URLs**: If the URL points somewhere else (PyPI, personal site, docs site, etc.), flag it with a warning: "This is not a GitHub URL. GitHub repos are preferred because they allow automated tracking of activity and archive status. Consider whether a GitHub link exists for this project."
+- Check that the URL looks well-formed (starts with `https://`).
+
+#### 4c. Section placement
+
+Look at which `##` (language) and `###` (category) heading the entry was added under. Evaluate whether the project fits that section based on its description and URL:
+
+- Does the project's language match the section? (e.g., a Python library should be under `## Python`)
+- Does the project's purpose match the category? (e.g., a backtesting framework should be under `### Trading & Backtesting`, not `### Indicators`)
+- If the placement seems wrong, suggest a better section.
+
+The current sections in the README are:
+
+**Python**: Numerical Libraries & Data Structures, Financial Instruments and Pricing, Indicators, Trading & Backtesting, Risk Analysis, Factor Analysis, Sentiment Analysis, Quant Research Environment, Time Series, Calendars, Data Sources, Excel Integration, Visualization
+
+**R**: Numerical Libraries & Data Structures, Data Sources, Financial Instruments and Pricing, Trading, Backtesting, Risk Analysis, Factor Analysis, Time Series, Calendars
+
+**Other languages**: Matlab, Julia, Java, JavaScript, Haskell, Scala, Ruby, Elixir/Erlang, Golang, CPP, CSharp, Rust
+
+**Cross-language**: Frameworks, Reproducing Works Training & Books
+
+If the project doesn't fit any existing section, suggest the closest match or recommend creating a new subsection (rare).
+
+#### 4d. Duplicate check
+
+Grep the current `README.md` for the project name and URL to ensure it's not already listed.
+
+### Step 5: Summarize findings
+
+Present a clear summary:
+
+```
+PR #<number>: <title>
+Author: <author>
+Created: <date>
+
+Entries reviewed: <count>
+
+<For each entry>
+  - [Name](URL) - Description
+    Format: OK / ISSUE: <details>
+    URL: GitHub / WARNING: Non-GitHub URL (<domain>)
+    Section: OK (<section>) / SUGGESTION: Move to <section>
+    Duplicate: No / YES: Already listed at line <N>
+</For each entry>
+
+Conflicts: None / YES: Needs rebase
+
+Verdict: APPROVE / NEEDS CHANGES
+```
+
+### Step 6: Take action
+
+- **If everything passes**: Ask the user for confirmation, then approve and merge the PR.
+- **If there are issues**: Leave a constructive review comment on the PR listing what needs to be fixed. Be polite and specific — these are open-source contributors.
+
+When leaving comments, be friendly and grateful for the contribution. Example tone:
+> Thanks for the contribution! A couple of things to address before we can merge:
+> - The description should end with a period.
+> - Consider linking to the GitHub repo instead of the docs site so we can track activity.
@@ -0,0 +1,79 @@
+---
+name: sync-site
+description: Sync site/index.qmd with README.md content. Use this skill whenever README.md has been updated and the site needs to reflect those changes — after merging PRs, editing entries, adding libraries, or any modification to README.md. Also triggers for "update site", "sync site", "deploy site", "update qmd", or mentions of site/index.qmd being out of date.
+---
+
+# Sync Site
+
+Update `site/index.qmd` to match the current `README.md` content, then commit and push.
+
+## How it works
+
+`site/index.qmd` is composed of two parts:
+
+1. **YAML frontmatter** (lines 1-11) — Quarto metadata that never changes. Preserve it exactly as-is.
+2. **Body content** — Everything from `## Python` to the end of the file. This must match `README.md` from the `## Python` heading onward.
+
+The preamble in README.md (title, badges, language table of contents) is intentionally excluded from `index.qmd` because the Quarto frontmatter replaces it.
+
+## Steps
+
+### 1. Extract the frontmatter from index.qmd
+
+Read `site/index.qmd` and capture everything up to and including the closing `---` and the blank line + description + badge lines that follow it (lines 1-16).
+
+The header looks like this:
+```
+---
+title: "Awesome Quant"
+...
+---
+
+A curated list of insanely awesome libraries, packages and resources for Quants (Quantitative Finance).
+
+[![](https://awesome.re/badge.svg)](https://awesome.re)
+```
+
+### 2. Extract the body from README.md
+
+Read `README.md` and capture everything starting from the line `## Python` (inclusive) to the end of the file.
+
+### 3. Combine and write
+
+Concatenate the frontmatter header and the README body, then write to `site/index.qmd`.
+
+Use a shell script for reliability:
+
+```bash
+# Extract line number where "## Python" starts in README.md
+START=$(grep -n '^## Python' README.md | head -1 | cut -d: -f1)
+
+# Extract frontmatter + preamble from index.qmd (up to the badge line)
+END=$(grep -n 'awesome.re/badge' site/index.qmd | head -1 | cut -d: -f1)
+
+# Combine: header from qmd + body from README
+{ head -n "$END" site/index.qmd; echo; tail -n +"$START" README.md; } > site/index.qmd.tmp
+mv site/index.qmd.tmp site/index.qmd
+```
+
+### 4. Verify
+
+Run a quick diff to confirm the update looks correct:
+
+```bash
+# Show line count to sanity-check
+wc -l site/index.qmd
+
+# Optionally diff the body portion to confirm they match
+diff <(tail -n +"$START" README.md) <(sed -n "$((END+2)),\$p" site/index.qmd)
+```
+
+### 5. Commit and push
+
+Stage, commit, and push the change:
+
+```bash
+git add site/index.qmd
+git commit -m "Sync site/index.qmd with README.md"
+git push origin master
+```
@@ -0,0 +1,132 @@
+# Update PyPI Dates Skill
+
+Automatically check PyPI projects for their latest update dates and keep README.md entries current.
+
+## Overview
+
+This skill monitors Python packages listed in awesome-quant and updates their entries with the latest release dates from PyPI. This ensures that outdated project information is easily identifiable.
+
+## Features
+
+- ✅ Checks 4 PyPI projects: qfrm, chinesestockapi, tushare, metatrader5
+- ✅ Fetches latest release dates from PyPI API
+- ✅ Updates README.md entries with `(Last updated: YYYY-MM-DD)` format
+- ✅ Commits changes automatically when updates found
+- ✅ No dependencies beyond Python standard library
+
+## Usage
+
+### Invoke from Claude Code
+
+```bash
+/update-pypi-dates
+```
+
+Or call the script directly:
+
+```bash
+./.claude/skills/update-pypi-dates/scripts/check_pypi_dates.py
+```
+
+### Manual Python Execution
+
+```bash
+python3 .claude/skills/update-pypi-dates/scripts/check_pypi_dates.py
+```
+
+### Bash Wrapper
+
+```bash
+./.claude/skills/update-pypi-dates/scripts/update_pypi_dates.sh
+```
+
+## How It Works
+
+1. **Queries PyPI API** for each tracked package
+2. **Extracts last release date** from the releases data
+3. **Updates README.md** entries with format: `(Last updated: 2024-08-27)`
+4. **Commits changes** if any updates were found
+
+## Adding New Packages
+
+To track additional PyPI packages, edit the `PYPI_PROJECTS` dictionary in `scripts/check_pypi_dates.py`:
+
+```python
+PYPI_PROJECTS = {
+    "display_name": "pypi_package_name",
+    # Add new entries here
+}
+```
+
+Then update the skill description if needed.
+
+## Output Examples
+
+### No Changes
+```
+📦 Checking PyPI projects for last updated dates...
+
+Checking qfrm...
+  ✓ 2015-12-12 (no change)
+
+✓ All packages up to date!
+```
+
+### With Updates
+```
+📦 Checking PyPI projects for last updated dates...
+
+Checking tushare...
+  ✓ 2024-08-27 (no change)
+Checking metatrader5...
+  ✅ Updated to 2026-02-20
+
+📝 Found 1 update(s):
+  - metatrader5: 2026-02-20
+
+✅ Updated README.md
+```
+
+## Configuration
+
+### Periodic Execution
+
+Set up a cron job to run weekly:
+
+```bash
+# Add to crontab (runs Monday at 2 AM)
+0 2 * * 1 cd /path/to/awesome-quant && ./.claude/skills/update-pypi-dates/scripts/update_pypi_dates.sh
+```
+
+Or use Claude Code's scheduling:
+
+```bash
+/loop 1w /update-pypi-dates
+```
+
+## Files
+
+- `SKILL.md` - Skill definition and documentation
+- `scripts/check_pypi_dates.py` - Main Python script
+- `scripts/update_pypi_dates.sh` - Bash wrapper with git integration
+- `README.md` - This file
+
+## Requirements
+
+- Python 3.6+
+- Network access to PyPI API
+- Git (for automatic commits)
+
+## Error Handling
+
+- Network timeouts: Gracefully skipped with warning
+- Package not found: Skipped with message
+- API errors: Logged and continued with other packages
+
+## Future Enhancements
+
+- [ ] Support for more PyPI packages
+- [ ] GitHub Actions workflow integration
+- [ ] Email notifications on updates
+- [ ] Comparison with published versions in README
+- [ ] Support for other package registries (npm, Ruby Gems, etc.)
@@ -0,0 +1,66 @@
+---
+name: update-pypi-dates
+description: Check PyPI projects for last updated dates and update README.md entries. Use this skill to keep PyPI project information current with latest release dates from PyPI. Run periodically to maintain up-to-date metadata on Python packages listed in awesome-quant.
+---
+
+# Update PyPI Dates
+
+Automatically check PyPI projects for their latest update dates and update corresponding README.md entries with the "(Last updated: YYYY-MM-DD)" information.
+
+## How it works
+
+The skill maintains a list of PyPI projects in awesome-quant and:
+1. Queries the PyPI JSON API for each package
+2. Extracts the last release date
+3. Updates README.md entries with the new date if changed
+4. Commits changes to git if updates were made
+
+## PyPI Projects to Track
+
+Current projects tracked:
+- qfrm
+- chinesestockapi
+- tushare
+- metatrader5
+
+## Steps
+
+### 1. Fetch PyPI data for each project
+
+For each PyPI project, query: `https://pypi.org/pypi/{package_name}/json`
+
+Extract the last release date from the releases data.
+
+### 2. Update README.md entries
+
+For each project with a new date:
+- Find the entry in README.md
+- Update or add the "(Last updated: YYYY-MM-DD)" note
+- Use the exact entry format: `(Last updated: 2024-08-27)`
+
+### 3. Verify changes
+
+Show the user:
+- Which packages were checked
+- Which packages have new update dates
+- The differences (old date → new date)
+
+### 4. Commit if changes detected
+
+If any updates were made:
+```bash
+git add README.md non-github-urls.md
+git commit -m "Update PyPI project last updated dates"
+git push origin master
+```
+
+## Usage
+
+The user can:
+- Run the skill directly: `/update-pypi-dates`
+- Schedule it to run weekly/monthly via cron
+- Include it in CI/CD pipelines to keep data fresh
+
+## Script
+
+Use `scripts/check_pypi_dates.sh` for the core update logic.
@@ -0,0 +1,154 @@
+#!/usr/bin/env python3
+"""
+Check PyPI projects for last updated dates and update README.md entries.
+"""
+
+import json
+import re
+import subprocess
+import sys
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from urllib.request import urlopen
+
+# PyPI projects to track
+PYPI_PROJECTS = {
+    "qfrm": "qfrm",
+    "chinesestockapi": "chinesestockapi",
+    "tushare": "tushare",
+    "metatrader5": "MetaTrader5",
+}
+
+README_PATH = Path(__file__).parent.parent.parent.parent.parent / "README.md"
+
+
+def get_pypi_last_updated(package_name: str) -> Optional[str]:
+    """
+    Fetch the last updated date for a PyPI package.
+
+    Returns: date string in format "YYYY-MM-DD" or None if not found
+    """
+    try:
+        url = f"https://pypi.org/pypi/{package_name}/json"
+        with urlopen(url, timeout=5) as response:
+            data = json.loads(response.read().decode())
+
+        # Get releases (ordered dict, latest first after sorting)
+        releases = data.get("releases", {})
+        if not releases:
+            return None
+
+        # Find the latest release with upload time
+        for version in sorted(releases.keys(), reverse=True):
+            release_data = releases[version]
+            if release_data and len(release_data) > 0:
+                upload_time = release_data[0].get("upload_time_iso_8601")
+                if upload_time:
+                    # Extract just the date part (YYYY-MM-DD)
+                    return upload_time.split("T")[0]
+
+        return None
+    except Exception as e:
+        print(f"  ❌ Error fetching {package_name}: {e}", file=sys.stderr)
+        return None
+
+
+def find_readme_entry(readme_content: str, package_display_name: str) -> Tuple[Optional[int], Optional[str]]:
+    """
+    Find a README entry for a package.
+
+    Returns: (line_number, match_text) or (None, None)
+    """
+    # Pattern to match the entry - look for [package_name](pypi.org...)
+    pattern = rf"\- \[{re.escape(package_display_name)}\]\(https://pypi\.org/project/\w+/\)[^\n]*"
+
+    for line_num, line in enumerate(readme_content.split("\n"), 1):
+        if re.search(pattern, line):
+            return line_num, line.strip()
+
+    return None, None
+
+
+def update_readme_entry(readme_content: str, package_display_name: str, new_date: str) -> Tuple[str, bool]:
+    """
+    Update or add the last updated date to a README entry.
+
+    Returns: (updated_content, was_changed)
+    """
+    lines = readme_content.split("\n")
+    line_num, entry_text = find_readme_entry(readme_content, package_display_name)
+
+    if line_num is None:
+        return readme_content, False
+
+    old_line = lines[line_num - 1]
+
+    # Check if entry already has a date
+    existing_date_pattern = r"\(Last updated: \d{4}-\d{2}-\d{2}\)"
+    existing_match = re.search(existing_date_pattern, old_line)
+
+    if existing_match:
+        old_date = existing_match.group(0)
+        new_date_str = f"(Last updated: {new_date})"
+        if old_date == new_date_str:
+            return readme_content, False  # No change needed
+        new_line = old_line.replace(old_date, new_date_str)
+    else:
+        # Add date to the end of the entry (before period if present)
+        if old_line.endswith("."):
+            new_line = old_line[:-1] + f" (Last updated: {new_date})."
+        else:
+            new_line = old_line + f" (Last updated: {new_date})"
+
+    lines[line_num - 1] = new_line
+    return "\n".join(lines), True
+
+
+def main():
+    """Check PyPI projects and update README.md."""
+    if not README_PATH.exists():
+        print(f"❌ README.md not found at {README_PATH}")
+        sys.exit(1)
+
+    readme_content = README_PATH.read_text(encoding="utf-8")
+    updated_content = readme_content
+    changes_made = []
+
+    print("📦 Checking PyPI projects for last updated dates...\n")
+
+    for display_name, pypi_name in PYPI_PROJECTS.items():
+        print(f"Checking {display_name}...")
+
+        last_updated = get_pypi_last_updated(pypi_name)
+        if last_updated is None:
+            print(f"  ⚠️  Could not find last updated date")
+            continue
+
+        updated_content, was_changed = update_readme_entry(
+            updated_content, display_name, last_updated
+        )
+
+        if was_changed:
+            print(f"  ✅ Updated to {last_updated}")
+            changes_made.append((display_name, last_updated))
+        else:
+            print(f"  ✓ {last_updated} (no change)")
+
+    print()
+
+    if changes_made:
+        print(f"📝 Found {len(changes_made)} update(s):")
+        for pkg_name, date in changes_made:
+            print(f"  - {pkg_name}: {date}")
+
+        README_PATH.write_text(updated_content, encoding="utf-8")
+        print(f"\n✅ Updated {README_PATH.name}")
+        return 0
+    else:
+        print("✓ All packages up to date!")
+        return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,27 @@
+#!/bin/bash
+# Wrapper script to check PyPI dates and update README.md
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../../../.." && pwd)"
+
+cd "$PROJECT_ROOT" || exit 1
+
+echo "🚀 Starting PyPI date update check..."
+python3 "$SCRIPT_DIR/check_pypi_dates.py"
+exit_code=$?
+
+if [ $exit_code -eq 0 ]; then
+    # Check if there are any changes to commit
+    if git diff --quiet README.md; then
+        echo "✓ No changes needed"
+    else
+        echo ""
+        echo "📝 Changes detected, committing..."
+        git add README.md
+        git commit -m "Update PyPI project last updated dates"
+        git push origin master
+        echo "✅ Changes pushed to remote"
+    fi
+fi
+
+exit $exit_code