chore: migrate repository to standard knowledge base layout

2026-08-20 06:18:05 +00:00 · 2026-05-02 03:29:06 +08:00
parent 40a721c24d
commit 628a3bc832
565 changed files with 687 additions and 711 deletions
@@ -0,0 +1,148 @@
+---
+name: twscrape
+description: "twscrape Twitter/X scraping skill: account pool setup, async search, user/tweet collection, CLI usage, proxy configuration, and rate-limit troubleshooting. Use when extracting public Twitter/X data with twscrape."
+---
+
+# twscrape Skill
+
+Use this skill to build or debug `twscrape` workflows for public Twitter/X data extraction with account rotation, async collection, CLI commands, and proxy-aware operation.
+
+## When to Use This Skill
+
+Trigger when any of these applies:
+- Scraping Twitter/X search results, profiles, followers, timelines, replies, retweeters, media, or trends with `twscrape`.
+- Setting up account pools, cookies, login flows, email verification, or account rotation.
+- Choosing between Python async API and the `twscrape` CLI.
+- Diagnosing rate limits, empty results, login failures, proxy failures, or suspended accounts.
+- Exporting normalized tweet/user data for monitoring, analytics, research, or archival pipelines.
+
+## Not For / Boundaries
+
+- Not for bypassing access controls, private content, paid-only data, or platform restrictions.
+- Not for guaranteed high-volume scraping; account health, platform changes, and endpoint limits can invalidate assumptions.
+- Do not place real Twitter/X passwords, cookies, email passwords, or proxy credentials in examples, commits, logs, or issue reports.
+- Required inputs: target query/user/tweet/list, collection limit, output format, account source, proxy requirements, and compliance constraints.
+- If behavior differs from these notes, verify against `references/` and the upstream repository before changing production collectors.
+
+## Quick Reference
+
+### Common Patterns
+
+**Install the library**
+```bash
+pip install twscrape
+```
+
+**Create an API client and add a cookie-backed account**
+```python
+from twscrape import API
+
+api = API("accounts.db")
+await api.pool.add_account(
+    "username",
+    "password",
+    "email@example.com",
+    "email-password",
+    cookies="ct0=...; auth_token=...",
+)
+```
+
+**Login all configured accounts**
+```python
+await api.pool.login_all()
+```
+
+**Search recent tweets**
+```python
+from twscrape import gather
+
+tweets = await gather(api.search("python lang:en", limit=50))
+```
+
+**Fetch a user then collect timeline data**
+```python
+user = await api.user_by_login("xdevelopers")
+tweets = await gather(api.user_tweets(user.id, limit=100))
+```
+
+**Collect followers or following**
+```python
+followers = await gather(api.followers(user.id, limit=100))
+following = await gather(api.following(user.id, limit=100))
+```
+
+**Inspect tweet details and replies**
+```python
+tweet = await api.tweet_details(1234567890)
+replies = await gather(api.tweet_replies(tweet.id, limit=50))
+```
+
+**Use the CLI for a small search**
+```bash
+twscrape search "python lang:en" --limit=20
+```
+
+**Manage accounts from the CLI**
+```bash
+twscrape add_accounts accounts.txt username:password:email:email_password
+twscrape login_accounts --manual
+twscrape accounts
+```
+
+**Set a global proxy**
+```bash
+export TWS_PROXY=socks5://user:pass@127.0.0.1:1080
+twscrape search "bitcoin" --limit=20
+```
+
+**Enable debug logging**
+```python
+from twscrape.logger import set_log_level
+
+set_log_level("DEBUG")
+```
+
+## Examples
+
+### Example 1: Search Export
+
+- Input: query `python lang:en`, limit `50`, output JSON Lines.
+- Steps:
+  1. Confirm at least one healthy account with `twscrape accounts`.
+  2. Use `await gather(api.search(query, limit=50))`.
+  3. Serialize selected fields such as `id`, `date`, `user.username`, and `rawContent`.
+- Expected output / acceptance: a JSONL file with up to 50 tweet records and no credentials in logs.
+
+### Example 2: User Monitoring
+
+- Input: username `xdevelopers`, timeline limit `100`.
+- Steps:
+  1. Resolve the account with `await api.user_by_login(username)`.
+  2. Collect `api.user_tweets(user.id, limit=100)`.
+  3. Store tweet IDs and timestamps so later runs can deduplicate.
+- Expected output / acceptance: user metadata plus a deduplicated timeline batch.
+
+### Example 3: Rate-Limit Triage
+
+- Input: collector returns no data or waits indefinitely.
+- Steps:
+  1. Run `twscrape accounts` and identify locked, suspended, or rate-limited accounts.
+  2. Enable debug logging and retry the smallest failing query.
+  3. Add healthy accounts or wait for endpoint-specific reset before scaling up.
+- Expected output / acceptance: the failing mode is classified as account health, query syntax, proxy/network, or platform limit.
+
+## References
+
+- `references/index.md`: navigation for the local twscrape reference set.
+- `references/installation.md`: installation and dependency notes.
+- `references/account_management.md`: account pool, login, and rotation behavior.
+- `references/api_methods.md`: Python API method reference.
+- `references/cli_usage.md`: command-line usage.
+- `references/proxy_config.md`: proxy configuration and precedence.
+- `references/examples.md`: longer code examples and extraction patterns.
+
+## Maintenance
+
+- Sources: local `references/` extracted from upstream twscrape material and the upstream repository noted there.
+- Last updated: 2026-04-28
+- Known limits: Twitter/X endpoints and account policies change without notice; validate live collectors against a small sample before large runs.
@@ -0,0 +1,14 @@
+# twscrape：账号管理（参考）
+
+本文件为 `twscrape` 的账号管理参考页。
+
+## 推荐阅读顺序
+
+1. `skills/twscrape/references/installation.md`
+2. `skills/twscrape/SKILL.md`（包含账号池、登录、轮换等完整示例）
+3. `skills/twscrape/references/examples.md`
+
+## 入口
+
+- 账号添加/登录的核心示例：`skills/twscrape/SKILL.md`
+
@@ -0,0 +1,8 @@
+# twscrape：API 方法（参考）
+
+本文件为 `twscrape` 的 API 方法参考页（仓库内精简版）。
+
+## 入口
+
+- 完整方法速查与示例：`skills/twscrape/SKILL.md`
+
@@ -0,0 +1,8 @@
+# twscrape：CLI 用法（参考）
+
+本文件为 `twscrape` 的命令行使用参考页。
+
+## 入口
+
+- CLI 相关命令示例：`skills/twscrape/SKILL.md`
+
@@ -0,0 +1,327 @@
+# twscrape Examples
+
+## Basic Search Example
+
+```python
+import asyncio
+from twscrape import API, gather
+
+async def main():
+    api = API()
+
+    # Search for tweets
+    tweets = await gather(api.search("elon musk", limit=20))
+
+    for tweet in tweets:
+        print(f"{tweet.user.username}: {tweet.rawContent}")
+
+asyncio.run(main())
+```
+
+## User Profile Analysis
+
+```python
+async def analyze_user(username):
+    api = API()
+
+    # Get user info
+    user = await api.user_by_login(username)
+    print(f"User: {user.displayname}")
+    print(f"Followers: {user.followersCount}")
+    print(f"Following: {user.followingCount}")
+
+    # Get recent tweets
+    tweets = await gather(api.user_tweets(user.id, limit=50))
+    print(f"Recent tweets: {len(tweets)}")
+
+    return user, tweets
+```
+
+## Follower Network Collection
+
+```python
+async def collect_network(user_id):
+    api = API()
+
+    # Collect followers
+    followers = await gather(api.followers(user_id, limit=100))
+    print(f"Collected {len(followers)} followers")
+
+    # Collect following
+    following = await gather(api.following(user_id, limit=100))
+    print(f"Collected {len(following)} following")
+
+    return followers, following
+```
+
+## Advanced Search with Filters
+
+```python
+async def advanced_search():
+    api = API()
+
+    # Search with language filter
+    en_tweets = await gather(api.search("python lang:en", limit=20))
+
+    # Search with date filter
+    recent_tweets = await gather(api.search("AI since:2024-01-01", limit=20))
+
+    # Search from specific user
+    user_tweets = await gather(api.search("from:elonmusk", limit=20))
+
+    # Search with media
+    media_tweets = await gather(api.search("cats filter:media", limit=20))
+
+    return en_tweets, recent_tweets, user_tweets, media_tweets
+```
+
+## Tweet Thread Analysis
+
+```python
+async def analyze_thread(tweet_id):
+    api = API()
+
+    # Get tweet details
+    tweet = await api.tweet_details(tweet_id)
+    print(f"Tweet: {tweet.rawContent}")
+
+    # Get replies
+    replies = await gather(api.tweet_replies(tweet_id, limit=100))
+    print(f"Replies: {len(replies)}")
+
+    # Get retweeters
+    retweeters = await gather(api.retweeters(tweet_id, limit=100))
+    print(f"Retweeters: {len(retweeters)}")
+
+    return tweet, replies, retweeters
+```
+
+## Batch User Processing
+
+```python
+async def process_multiple_users(usernames):
+    api = API()
+    results = []
+
+    tasks = []
+    for username in usernames:
+        task = api.user_by_login(username)
+        tasks.append(task)
+
+    users = await asyncio.gather(*tasks)
+
+    for user in users:
+        if user:
+            print(f"Processed: {user.displayname}")
+            results.append(user)
+
+    return results
+
+# Usage
+usernames = ["elonmusk", "xdevelopers", "github"]
+users = await process_multiple_users(usernames)
+```
+
+## Real-time Monitoring
+
+```python
+async def monitor_keywords(keywords, limit=100):
+    api = API()
+
+    for keyword in keywords:
+        print(f"\\nMonitoring: {keyword}")
+
+        async for tweet in api.search(keyword, limit=limit):
+            print(f"[{tweet.date}] @{tweet.user.username}: {tweet.rawContent[:100]}")
+
+            # Process tweet
+            if tweet.likeCount > 1000:
+                print(f"  -> Popular tweet! {tweet.likeCount} likes")
+
+# Usage
+await monitor_keywords(["python", "javascript", "ai"], limit=50)
+```
+
+## Data Export to JSON
+
+```python
+import json
+
+async def export_user_data(username, output_file):
+    api = API()
+
+    user = await api.user_by_login(username)
+    tweets = await gather(api.user_tweets(user.id, limit=100))
+
+    data = {
+        'user': user.dict(),
+        'tweets': [tweet.dict() for tweet in tweets]
+    }
+
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump(data, f, indent=2, ensure_ascii=False)
+
+    print(f"Exported to {output_file}")
+
+# Usage
+await export_user_data("elonmusk", "elon_data.json")
+```
+
+## Trends Analysis
+
+```python
+async def analyze_trends():
+    api = API()
+
+    # Get different trend categories
+    news_trends = await gather(api.trends("news"))
+    sport_trends = await gather(api.trends("sport"))
+
+    print("News Trends:")
+    for trend in news_trends[:10]:
+        print(f"  - {trend}")
+
+    print("\\nSport Trends:")
+    for trend in sport_trends[:10]:
+        print(f"  - {trend}")
+
+    return news_trends, sport_trends
+```
+
+## Using Context Manager for Early Termination
+
+```python
+from contextlib import aclosing
+
+async def find_specific_tweet(query, target_id):
+    api = API()
+
+    async with aclosing(api.search(query)) as gen:
+        async for tweet in gen:
+            if tweet.id == target_id:
+                print(f"Found target tweet: {tweet.rawContent}")
+                return tweet
+
+            if tweet.id < target_id:
+                print("Target not found in results")
+                break
+
+    return None
+```
+
+## Account Setup Example
+
+```python
+async def setup_accounts():
+    api = API()
+
+    # Add accounts with cookies (more stable)
+    cookies = "abc=12; ct0=xyz"
+    await api.pool.add_account(
+        "user1",
+        "password1",
+        "user1@example.com",
+        "mail_password1",
+        cookies=cookies
+    )
+
+    # Add account with credentials
+    await api.pool.add_account(
+        "user2",
+        "password2",
+        "user2@example.com",
+        "mail_password2"
+    )
+
+    # Login all accounts
+    await api.pool.login_all()
+
+    print("Accounts setup complete")
+```
+
+## Proxy Configuration Example
+
+```python
+async def use_proxy():
+    # Global proxy
+    proxy = "http://user:pass@proxy.example.com:8080"
+    api = API(proxy=proxy)
+
+    # Make requests through proxy
+    user = await api.user_by_login("elonmusk")
+    print(f"User: {user.displayname}")
+
+    # Change proxy dynamically
+    api.proxy = "socks5://user:pass@127.0.0.1:1080"
+    tweets = await gather(api.search("python", limit=10))
+
+    # Disable proxy
+    api.proxy = None
+    more_tweets = await gather(api.search("javascript", limit=10))
+```
+
+## Error Handling
+
+```python
+async def safe_user_lookup(username):
+    api = API()
+
+    try:
+        user = await api.user_by_login(username)
+        return user
+    except Exception as e:
+        print(f"Error fetching user {username}: {e}")
+        return None
+
+async def bulk_lookup_with_errors(usernames):
+    results = []
+    for username in usernames:
+        user = await safe_user_lookup(username)
+        if user:
+            results.append(user)
+
+    return results
+```
+
+## Complete Workflow Example
+
+```python
+import asyncio
+import json
+from twscrape import API, gather
+from twscrape.logger import set_log_level
+
+async def complete_workflow():
+    # Setup
+    api = API("my_data.db")
+    set_log_level("INFO")
+
+    # Add accounts
+    await api.pool.add_account(
+        "user1", "pass1", "email1@example.com", "mail_pass1",
+        cookies="cookie_string_here"
+    )
+
+    # Search and analyze
+    query = "python programming"
+    tweets = await gather(api.search(query, limit=100))
+
+    # Extract user data
+    users = {}
+    for tweet in tweets:
+        if tweet.user.username not in users:
+            users[tweet.user.username] = {
+                'user': tweet.user.dict(),
+                'tweets': []
+            }
+        users[tweet.user.username]['tweets'].append(tweet.dict())
+
+    # Export results
+    with open('results.json', 'w', encoding='utf-8') as f:
+        json.dump(users, f, indent=2, ensure_ascii=False)
+
+    print(f"Processed {len(tweets)} tweets from {len(users)} users")
+
+if __name__ == "__main__":
+    asyncio.run(complete_workflow())
+```
@@ -0,0 +1,39 @@
+# twscrape Reference Documentation
+
+## Overview
+
+This directory contains detailed reference documentation for twscrape, a Python library for scraping Twitter/X data.
+
+## Reference Files
+
+### Core Documentation
+- **[installation.md](installation.md)** - Installation instructions and dependencies
+- **[api_methods.md](api_methods.md)** - Complete API method reference with parameters
+- **[account_management.md](account_management.md)** - Account setup, login, and rotation
+- **[cli_usage.md](cli_usage.md)** - Command-line interface guide
+- **[proxy_config.md](proxy_config.md)** - Proxy configuration and setup
+- **[examples.md](examples.md)** - Practical code examples and patterns
+
+## Quick Navigation
+
+### Getting Started
+1. Read [installation.md](installation.md) for setup
+2. Review [account_management.md](account_management.md) for adding accounts
+3. Check [examples.md](examples.md) for quick start code
+
+### API Reference
+- For programmatic usage: [api_methods.md](api_methods.md)
+- For command-line usage: [cli_usage.md](cli_usage.md)
+
+### Advanced Topics
+- Proxy configuration: [proxy_config.md](proxy_config.md)
+- Rate limit handling: See [api_methods.md](api_methods.md#rate-limits)
+
+## Key Features
+
+- ✅ Async/await support for parallel operations
+- ✅ Automatic account rotation
+- ✅ Session persistence
+- ✅ Multiple proxy support
+- ✅ SNScrape-compatible data models
+- ✅ Both CLI and Python API
@@ -0,0 +1,66 @@
+# Installation
+
+## Standard Installation
+
+```bash
+pip install twscrape
+```
+
+## Development Version
+
+Install the latest development version directly from GitHub:
+
+```bash
+pip install git+https://github.com/vladkens/twscrape.git
+```
+
+## Requirements
+
+- Python 3.7+
+- asyncio support
+- Internet connection for Twitter/X access
+
+## Dependencies
+
+The library automatically installs required dependencies:
+- `httpx` - Async HTTP client
+- `aiosqlite` - Async SQLite database
+- Additional dependencies as specified in setup.py
+
+## Verification
+
+Verify installation:
+
+```bash
+# Check CLI is available
+twscrape --help
+
+# Check Python import works
+python -c "from twscrape import API; print('OK')"
+```
+
+## Upgrading
+
+```bash
+pip install --upgrade twscrape
+```
+
+## Uninstallation
+
+```bash
+pip uninstall twscrape
+```
+
+## Database Location
+
+By default, twscrape creates `accounts.db` in your current working directory. You can specify a custom location:
+
+```python
+api = API("path/to/custom.db")
+```
+
+Or via CLI:
+
+```bash
+twscrape --db path/to/custom.db <command>
+```
@@ -0,0 +1,8 @@
+# twscrape：代理配置（参考）
+
+本文件为 `twscrape` 的代理配置参考页。
+
+## 入口
+
+- 代理参数示例：`skills/twscrape/SKILL.md`
+