gavindiaz/vibe-coding-cn
1
0
Fork 0
mirror of https://github.com/tradecatlabs/vibe-coding-cn.git synced 2026-08-21 14:58:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: skills - upgrade skill instructions

Browse Source
This commit is contained in:
tukuaiai
2026-04-28 19:14:07 +08:00
parent f78d03236a
commit 7a902b3348
16 changed files with 1228 additions and 3070 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/skills/twscrape/SKILL.md
+102 -392
View File
@@ -1,438 +1,148 @@
# twscrape
---
name: twscrape
description: "twscrape Twitter/X scraping skill: account pool setup, async search, user/tweet collection, CLI usage, proxy configuration, and rate-limit troubleshooting. Use when extracting public Twitter/X data with twscrape."
---
Python library for scraping Twitter/X data using GraphQL API with account rotation and session management.
# twscrape Skill
## When to use this skill
Use this skill to build or debug `twscrape` workflows for public Twitter/X data extraction with account rotation, async collection, CLI commands, and proxy-aware operation.
Use this skill when:
- Working with Twitter/X data extraction and scraping
- Need to bypass Twitter API limitations with account rotation
- Building social media monitoring or analytics tools
- Extracting tweets, user profiles, followers, trends from Twitter/X
- Need async/parallel scraping operations for large-scale data collection
- Looking for alternatives to official Twitter API
## When to Use This Skill
Trigger when any of these applies:
- Scraping Twitter/X search results, profiles, followers, timelines, replies, retweeters, media, or trends with `twscrape`.
- Setting up account pools, cookies, login flows, email verification, or account rotation.
- Choosing between Python async API and the `twscrape` CLI.
- Diagnosing rate limits, empty results, login failures, proxy failures, or suspended accounts.
- Exporting normalized tweet/user data for monitoring, analytics, research, or archival pipelines.
## Not For / Boundaries
- Not for bypassing access controls, private content, paid-only data, or platform restrictions.
- Not for guaranteed high-volume scraping; account health, platform changes, and endpoint limits can invalidate assumptions.
- Do not place real Twitter/X passwords, cookies, email passwords, or proxy credentials in examples, commits, logs, or issue reports.
- Required inputs: target query/user/tweet/list, collection limit, output format, account source, proxy requirements, and compliance constraints.
- If behavior differs from these notes, verify against `references/` and the upstream repository before changing production collectors.
## Quick Reference
### Installation
### Common Patterns
**Install the library**
```bash
pip install twscrape
```
### Basic Setup
**Create an API client and add a cookie-backed account**
```python
import asyncio
from twscrape import API, gather
from twscrape import API
async def main():
api = API() # Uses accounts.db by default
# Add accounts (with cookies - more stable)
cookies = "abc=12; ct0=xyz"
await api.pool.add_account("user1", "pass1", "email@example.com", "mail_pass", cookies=cookies)
# Or add accounts (with login/password - less stable)
await api.pool.add_account("user2", "pass2", "email2@example.com", "mail_pass2")
await api.pool.login_all()
asyncio.run(main())
api = API("accounts.db")
await api.pool.add_account(
"username",
"password",
"email@example.com",
"email-password",
cookies="ct0=...; auth_token=...",
)
```
### Common Operations
**Login all configured accounts**
```python
# Search tweets
await gather(api.search("elon musk", limit=20))
# Get user info
await api.user_by_login("xdevelopers")
user = await api.user_by_id(2244994945)
# Get user tweets
await gather(api.user_tweets(user_id, limit=20))
await gather(api.user_tweets_and_replies(user_id, limit=20))
await gather(api.user_media(user_id, limit=20))
# Get followers/following
await gather(api.followers(user_id, limit=20))
await gather(api.following(user_id, limit=20))
# Tweet operations
await api.tweet_details(tweet_id)
await gather(api.retweeters(tweet_id, limit=20))
await gather(api.tweet_replies(tweet_id, limit=20))
# Trends
await gather(api.trends("news"))
```
## Key Features
### 1. Multiple API Support
- **Search API**: Standard Twitter search functionality
- **GraphQL API**: Advanced queries and data extraction
- **Automatic switching**: Based on rate limits and availability
### 2. Async/Await Architecture
```python
# Parallel scraping
async for tweet in api.search("elon musk"):
print(tweet.id, tweet.user.username, tweet.rawContent)
```
### 3. Account Management
- Add multiple accounts for rotation
- Automatic rate limit handling
- Session persistence across runs
- Email verification support (IMAP or manual)
### 4. Data Models
- SNScrape-compatible models
- Easy conversion to dict/JSON
- Raw API response access available
## Core API Methods
### Search Operations
#### `search(query, limit, kv={})`
Search tweets by query string.
**Parameters:**
- `query` (str): Search query (supports Twitter search syntax)
- `limit` (int): Maximum number of tweets to return
- `kv` (dict): Additional parameters (e.g., `{"product": "Top"}` for Top tweets)
**Returns:** AsyncIterator of Tweet objects
**Example:**
```python
# Latest tweets
async for tweet in api.search("elon musk", limit=20):
print(tweet.rawContent)
# Top tweets
await gather(api.search("python", limit=20, kv={"product": "Top"}))
```
### User Operations
#### `user_by_login(username)`
Get user information by username.
**Example:**
```python
user = await api.user_by_login("xdevelopers")
print(user.id, user.displayname, user.followersCount)
```
#### `user_by_id(user_id)`
Get user information by user ID.
#### `followers(user_id, limit)`
Get user's followers.
#### `following(user_id, limit)`
Get users that the user follows.
#### `verified_followers(user_id, limit)`
Get only verified followers.
#### `subscriptions(user_id, limit)`
Get user's Twitter Blue subscriptions.
### Tweet Operations
#### `tweet_details(tweet_id)`
Get detailed information about a specific tweet.
#### `tweet_replies(tweet_id, limit)`
Get replies to a tweet.
#### `retweeters(tweet_id, limit)`
Get users who retweeted a specific tweet.
#### `user_tweets(user_id, limit)`
Get tweets from a user (excludes replies).
#### `user_tweets_and_replies(user_id, limit)`
Get tweets and replies from a user.
#### `user_media(user_id, limit)`
Get tweets with media from a user.
### Other Operations
#### `list_timeline(list_id)`
Get tweets from a Twitter list.
#### `trends(category)`
Get trending topics by category.
**Categories:** "news", "sport", "entertainment", etc.
## Account Management
### Adding Accounts
**With cookies (recommended):**
```python
cookies = "abc=12; ct0=xyz" # String or JSON format
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", cookies=cookies)
```
**With credentials:**
```python
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass")
await api.pool.login_all()
```
### CLI Account Management
**Search recent tweets**
```python
from twscrape import gather
tweets = await gather(api.search("python lang:en", limit=50))
```
**Fetch a user then collect timeline data**
```python
user = await api.user_by_login("xdevelopers")
tweets = await gather(api.user_tweets(user.id, limit=100))
```
**Collect followers or following**
```python
followers = await gather(api.followers(user.id, limit=100))
following = await gather(api.following(user.id, limit=100))
```
**Inspect tweet details and replies**
```python
tweet = await api.tweet_details(1234567890)
replies = await gather(api.tweet_replies(tweet.id, limit=50))
```
**Use the CLI for a small search**
```bash
twscrape search "python lang:en" --limit=20
```
**Manage accounts from the CLI**
```bash
# Add accounts from file
twscrape add_accounts accounts.txt username:password:email:email_password
# Login all accounts
twscrape login_accounts
# Manual email verification
twscrape login_accounts --manual
# List accounts and status
twscrape accounts
# Re-login specific accounts
twscrape relogin user1 user2
# Retry failed logins
twscrape relogin_failed
```
## Proxy Configuration
### Per-Account Proxy
```python
proxy = "http://login:pass@example.com:8080"
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", proxy=proxy)
```
### Global Proxy
```python
api = API(proxy="http://login:pass@example.com:8080")
```
### Environment Variable
**Set a global proxy**
```bash
export TWS_PROXY=socks5://user:pass@127.0.0.1:1080
twscrape search "elon musk"
twscrape search "bitcoin" --limit=20
```
### Dynamic Proxy Changes
```python
api.proxy = "socks5://user:pass@127.0.0.1:1080"
doc = await api.user_by_login("elonmusk")
api.proxy = None # Disable proxy
```
**Priority:** `api.proxy` > `TWS_PROXY` env var > account-specific proxy
## CLI Usage
### Search Operations
```bash
twscrape search "QUERY" --limit=20
twscrape search "elon musk lang:es" --limit=20 > data.txt
twscrape search "python" --limit=20 --raw # Raw API responses
```
### User Operations
```bash
twscrape user_by_login USERNAME
twscrape user_by_id USER_ID
twscrape followers USER_ID --limit=20
twscrape following USER_ID --limit=20
twscrape verified_followers USER_ID --limit=20
twscrape user_tweets USER_ID --limit=20
```
### Tweet Operations
```bash
twscrape tweet_details TWEET_ID
twscrape tweet_replies TWEET_ID --limit=20
twscrape retweeters TWEET_ID --limit=20
```
### Trends
```bash
twscrape trends sport
twscrape trends news
```
### Custom Database
```bash
twscrape --db custom-accounts.db <command>
```
## Advanced Usage
### Raw API Responses
```python
async for response in api.search_raw("elon musk"):
print(response.status_code, response.json())
```
### Stopping Iteration
```python
from contextlib import aclosing
async with aclosing(api.search("elon musk")) as gen:
async for tweet in gen:
if tweet.id < 200:
break
```
### Convert Models to Dict/JSON
```python
user = await api.user_by_id(user_id)
user_dict = user.dict()
user_json = user.json()
```
### Enable Debug Logging
**Enable debug logging**
```python
from twscrape.logger import set_log_level
set_log_level("DEBUG")
```
## Environment Variables
## Examples
- **`TWS_PROXY`**: Global proxy for all accounts
Example: `socks5://user:pass@127.0.0.1:1080`
### Example 1: Search Export
- **`TWS_WAIT_EMAIL_CODE`**: Timeout for email verification (default: 30 seconds)
- Input: query `python lang:en`, limit `50`, output JSON Lines.
- Steps:
1. Confirm at least one healthy account with `twscrape accounts`.
2. Use `await gather(api.search(query, limit=50))`.
3. Serialize selected fields such as `id`, `date`, `user.username`, and `rawContent`.
- Expected output / acceptance: a JSONL file with up to 50 tweet records and no credentials in logs.
- **`TWS_RAISE_WHEN_NO_ACCOUNT`**: Raise exception when no accounts available instead of waiting
Values: `false`, `0`, `true`, `1` (default: `false`)
### Example 2: User Monitoring
## Rate Limits & Limitations
- Input: username `xdevelopers`, timeline limit `100`.
- Steps:
1. Resolve the account with `await api.user_by_login(username)`.
2. Collect `api.user_tweets(user.id, limit=100)`.
3. Store tweet IDs and timestamps so later runs can deduplicate.
- Expected output / acceptance: user metadata plus a deduplicated timeline batch.
### Rate Limits
- Rate limits reset **every 15 minutes** per endpoint
- Each account has **separate limits** for different operations
- Accounts automatically rotate when limits are reached
### Example 3: Rate-Limit Triage
### Tweet Limits
- `user_tweets` and `user_tweets_and_replies` return approximately **3,200 tweets maximum** per user
- This is a Twitter/X platform limitation
### Account Status
- Rate limits vary based on:
- Account age
- Account verification status
- Account activity history
### Handling Rate Limits
The library automatically:
- Switches to next available account
- Waits for rate limit reset if all accounts exhausted
- Tracks rate limit status per endpoint
## Common Patterns
### Large-Scale Data Collection
```python
async def collect_user_data(username):
user = await api.user_by_login(username)
# Collect tweets
tweets = await gather(api.user_tweets(user.id, limit=100))
# Collect followers
followers = await gather(api.followers(user.id, limit=100))
# Collect following
following = await gather(api.following(user.id, limit=100))
return {
'user': user,
'tweets': tweets,
'followers': followers,
'following': following
}
```
### Search with Filters
```python
# Language filter
await gather(api.search("python lang:en", limit=20))
# Date filter
await gather(api.search("AI since:2024-01-01", limit=20))
# From specific user
await gather(api.search("from:elonmusk", limit=20))
# With media
await gather(api.search("cats filter:media", limit=20))
```
### Batch Processing
```python
async def process_users(usernames):
tasks = []
for username in usernames:
task = api.user_by_login(username)
tasks.append(task)
users = await asyncio.gather(*tasks)
return users
```
## Troubleshooting
### Login Issues
- **Use cookies instead of credentials** for more stable authentication
- Enable **manual email verification** with `--manual` flag
- Check **email password** is correct for IMAP access
### Rate Limit Problems
- **Add more accounts** for better rotation
- **Increase wait time** between requests
- **Monitor account status** with `twscrape accounts`
### No Data Returned
- **Check account status** - they may be suspended or rate limited
- **Verify query syntax** - use Twitter search syntax
- **Try different accounts** - some may have better access
### Connection Issues
- **Configure proxy** if behind firewall
- **Check network connectivity**
- **Verify Twitter/X is accessible** from your location
## Resources
- **GitHub Repository**: https://github.com/vladkens/twscrape
- **Installation**: `pip install twscrape`
- **Development Version**: `pip install git+https://github.com/vladkens/twscrape.git`
- Input: collector returns no data or waits indefinitely.
- Steps:
1. Run `twscrape accounts` and identify locked, suspended, or rate-limited accounts.
2. Enable debug logging and retry the smallest failing query.
3. Add healthy accounts or wait for endpoint-specific reset before scaling up.
- Expected output / acceptance: the failing mode is classified as account health, query syntax, proxy/network, or platform limit.
## References
For detailed API documentation and examples, see the reference files in the `references/` directory:
- `references/index.md`: navigation for the local twscrape reference set.
- `references/installation.md`: installation and dependency notes.
- `references/account_management.md`: account pool, login, and rotation behavior.
- `references/api_methods.md`: Python API method reference.
- `references/cli_usage.md`: command-line usage.
- `references/proxy_config.md`: proxy configuration and precedence.
- `references/examples.md`: longer code examples and extraction patterns.
- `references/installation.md` - Installation and setup
- `references/api_methods.md` - Complete API method reference
- `references/account_management.md` - Account configuration and management
- `references/cli_usage.md` - Command-line interface guide
- `references/proxy_config.md` - Proxy configuration options
- `references/examples.md` - Code examples and patterns
## Maintenance
---
**Repository**: https://github.com/vladkens/twscrape
**Stars**: 1998+
**Language**: Python
**License**: MIT
- Sources: local `references/` extracted from upstream twscrape material and the upstream repository noted there.
- Last updated: 2026-04-28
- Known limits: Twitter/X endpoints and account policies change without notice; validate live collectors against a small sample before large runs.