gavindiaz/vibe-coding-cn
1
0
Fork 0
mirror of https://github.com/tradecatlabs/vibe-coding-cn.git synced 2026-08-17 21:08:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor: 重构目录结构以支持 i18n

Browse Source 
创建 'i18n' 目录以存放多语言内容。将所有现有的中文内容（文档、提示词、技能、README）移动到 'i18n/zh/' 中。添加了新的根 README 作为语言入口，并为英文（'en'）翻译创建了占位符结构。
This commit is contained in:
tukuaiai
2025-12-16 21:30:13 +08:00
parent 1b235161ec
commit 624ef8d5f9
199 changed files with 674 additions and 654 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/twscrape/SKILL.md
-438
View File
@@ -1,438 +0,0 @@
# twscrape
Python library for scraping Twitter/X data using GraphQL API with account rotation and session management.
## When to use this skill
Use this skill when:
- Working with Twitter/X data extraction and scraping
- Need to bypass Twitter API limitations with account rotation
- Building social media monitoring or analytics tools
- Extracting tweets, user profiles, followers, trends from Twitter/X
- Need async/parallel scraping operations for large-scale data collection
- Looking for alternatives to official Twitter API
## Quick Reference
### Installation
```bash
pip install twscrape
```
### Basic Setup
```python
import asyncio
from twscrape import API, gather
async def main():
api = API() # Uses accounts.db by default
# Add accounts (with cookies - more stable)
cookies = "abc=12; ct0=xyz"
await api.pool.add_account("user1", "pass1", "email@example.com", "mail_pass", cookies=cookies)
# Or add accounts (with login/password - less stable)
await api.pool.add_account("user2", "pass2", "email2@example.com", "mail_pass2")
await api.pool.login_all()
asyncio.run(main())
```
### Common Operations
```python
# Search tweets
await gather(api.search("elon musk", limit=20))
# Get user info
await api.user_by_login("xdevelopers")
user = await api.user_by_id(2244994945)
# Get user tweets
await gather(api.user_tweets(user_id, limit=20))
await gather(api.user_tweets_and_replies(user_id, limit=20))
await gather(api.user_media(user_id, limit=20))
# Get followers/following
await gather(api.followers(user_id, limit=20))
await gather(api.following(user_id, limit=20))
# Tweet operations
await api.tweet_details(tweet_id)
await gather(api.retweeters(tweet_id, limit=20))
await gather(api.tweet_replies(tweet_id, limit=20))
# Trends
await gather(api.trends("news"))
```
## Key Features
### 1. Multiple API Support
- **Search API**: Standard Twitter search functionality
- **GraphQL API**: Advanced queries and data extraction
- **Automatic switching**: Based on rate limits and availability
### 2. Async/Await Architecture
```python
# Parallel scraping
async for tweet in api.search("elon musk"):
print(tweet.id, tweet.user.username, tweet.rawContent)
```
### 3. Account Management
- Add multiple accounts for rotation
- Automatic rate limit handling
- Session persistence across runs
- Email verification support (IMAP or manual)
### 4. Data Models
- SNScrape-compatible models
- Easy conversion to dict/JSON
- Raw API response access available
## Core API Methods
### Search Operations
#### `search(query, limit, kv={})`
Search tweets by query string.
**Parameters:**
- `query` (str): Search query (supports Twitter search syntax)
- `limit` (int): Maximum number of tweets to return
- `kv` (dict): Additional parameters (e.g., `{"product": "Top"}` for Top tweets)
**Returns:** AsyncIterator of Tweet objects
**Example:**
```python
# Latest tweets
async for tweet in api.search("elon musk", limit=20):
print(tweet.rawContent)
# Top tweets
await gather(api.search("python", limit=20, kv={"product": "Top"}))
```
### User Operations
#### `user_by_login(username)`
Get user information by username.
**Example:**
```python
user = await api.user_by_login("xdevelopers")
print(user.id, user.displayname, user.followersCount)
```
#### `user_by_id(user_id)`
Get user information by user ID.
#### `followers(user_id, limit)`
Get user's followers.
#### `following(user_id, limit)`
Get users that the user follows.
#### `verified_followers(user_id, limit)`
Get only verified followers.
#### `subscriptions(user_id, limit)`
Get user's Twitter Blue subscriptions.
### Tweet Operations
#### `tweet_details(tweet_id)`
Get detailed information about a specific tweet.
#### `tweet_replies(tweet_id, limit)`
Get replies to a tweet.
#### `retweeters(tweet_id, limit)`
Get users who retweeted a specific tweet.
#### `user_tweets(user_id, limit)`
Get tweets from a user (excludes replies).
#### `user_tweets_and_replies(user_id, limit)`
Get tweets and replies from a user.
#### `user_media(user_id, limit)`
Get tweets with media from a user.
### Other Operations
#### `list_timeline(list_id)`
Get tweets from a Twitter list.
#### `trends(category)`
Get trending topics by category.
**Categories:** "news", "sport", "entertainment", etc.
## Account Management
### Adding Accounts
**With cookies (recommended):**
```python
cookies = "abc=12; ct0=xyz" # String or JSON format
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", cookies=cookies)
```
**With credentials:**
```python
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass")
await api.pool.login_all()
```
### CLI Account Management
```bash
# Add accounts from file
twscrape add_accounts accounts.txt username:password:email:email_password
# Login all accounts
twscrape login_accounts
# Manual email verification
twscrape login_accounts --manual
# List accounts and status
twscrape accounts
# Re-login specific accounts
twscrape relogin user1 user2
# Retry failed logins
twscrape relogin_failed
```
## Proxy Configuration
### Per-Account Proxy
```python
proxy = "http://login:pass@example.com:8080"
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", proxy=proxy)
```
### Global Proxy
```python
api = API(proxy="http://login:pass@example.com:8080")
```
### Environment Variable
```bash
export TWS_PROXY=socks5://user:pass@127.0.0.1:1080
twscrape search "elon musk"
```
### Dynamic Proxy Changes
```python
api.proxy = "socks5://user:pass@127.0.0.1:1080"
doc = await api.user_by_login("elonmusk")
api.proxy = None # Disable proxy
```
**Priority:** `api.proxy` > `TWS_PROXY` env var > account-specific proxy
## CLI Usage
### Search Operations
```bash
twscrape search "QUERY" --limit=20
twscrape search "elon musk lang:es" --limit=20 > data.txt
twscrape search "python" --limit=20 --raw # Raw API responses
```
### User Operations
```bash
twscrape user_by_login USERNAME
twscrape user_by_id USER_ID
twscrape followers USER_ID --limit=20
twscrape following USER_ID --limit=20
twscrape verified_followers USER_ID --limit=20
twscrape user_tweets USER_ID --limit=20
```
### Tweet Operations
```bash
twscrape tweet_details TWEET_ID
twscrape tweet_replies TWEET_ID --limit=20
twscrape retweeters TWEET_ID --limit=20
```
### Trends
```bash
twscrape trends sport
twscrape trends news
```
### Custom Database
```bash
twscrape --db custom-accounts.db <command>
```
## Advanced Usage
### Raw API Responses
```python
async for response in api.search_raw("elon musk"):
print(response.status_code, response.json())
```
### Stopping Iteration
```python
from contextlib import aclosing
async with aclosing(api.search("elon musk")) as gen:
async for tweet in gen:
if tweet.id < 200:
break
```
### Convert Models to Dict/JSON
```python
user = await api.user_by_id(user_id)
user_dict = user.dict()
user_json = user.json()
```
### Enable Debug Logging
```python
from twscrape.logger import set_log_level
set_log_level("DEBUG")
```
## Environment Variables
- **`TWS_PROXY`**: Global proxy for all accounts
Example: `socks5://user:pass@127.0.0.1:1080`
- **`TWS_WAIT_EMAIL_CODE`**: Timeout for email verification (default: 30 seconds)
- **`TWS_RAISE_WHEN_NO_ACCOUNT`**: Raise exception when no accounts available instead of waiting
Values: `false`, `0`, `true`, `1` (default: `false`)
## Rate Limits & Limitations
### Rate Limits
- Rate limits reset **every 15 minutes** per endpoint
- Each account has **separate limits** for different operations
- Accounts automatically rotate when limits are reached
### Tweet Limits
- `user_tweets` and `user_tweets_and_replies` return approximately **3,200 tweets maximum** per user
- This is a Twitter/X platform limitation
### Account Status
- Rate limits vary based on:
- Account age
- Account verification status
- Account activity history
### Handling Rate Limits
The library automatically:
- Switches to next available account
- Waits for rate limit reset if all accounts exhausted
- Tracks rate limit status per endpoint
## Common Patterns
### Large-Scale Data Collection
```python
async def collect_user_data(username):
user = await api.user_by_login(username)
# Collect tweets
tweets = await gather(api.user_tweets(user.id, limit=100))
# Collect followers
followers = await gather(api.followers(user.id, limit=100))
# Collect following
following = await gather(api.following(user.id, limit=100))
return {
'user': user,
'tweets': tweets,
'followers': followers,
'following': following
}
```
### Search with Filters
```python
# Language filter
await gather(api.search("python lang:en", limit=20))
# Date filter
await gather(api.search("AI since:2024-01-01", limit=20))
# From specific user
await gather(api.search("from:elonmusk", limit=20))
# With media
await gather(api.search("cats filter:media", limit=20))
```
### Batch Processing
```python
async def process_users(usernames):
tasks = []
for username in usernames:
task = api.user_by_login(username)
tasks.append(task)
users = await asyncio.gather(*tasks)
return users
```
## Troubleshooting
### Login Issues
- **Use cookies instead of credentials** for more stable authentication
- Enable **manual email verification** with `--manual` flag
- Check **email password** is correct for IMAP access
### Rate Limit Problems
- **Add more accounts** for better rotation
- **Increase wait time** between requests
- **Monitor account status** with `twscrape accounts`
### No Data Returned
- **Check account status** - they may be suspended or rate limited
- **Verify query syntax** - use Twitter search syntax
- **Try different accounts** - some may have better access
### Connection Issues
- **Configure proxy** if behind firewall
- **Check network connectivity**
- **Verify Twitter/X is accessible** from your location
## Resources
- **GitHub Repository**: https://github.com/vladkens/twscrape
- **Installation**: `pip install twscrape`
- **Development Version**: `pip install git+https://github.com/vladkens/twscrape.git`
## References
For detailed API documentation and examples, see the reference files in the `references/` directory:
- `references/installation.md` - Installation and setup
- `references/api_methods.md` - Complete API method reference
- `references/account_management.md` - Account configuration and management
- `references/cli_usage.md` - Command-line interface guide
- `references/proxy_config.md` - Proxy configuration options
- `references/examples.md` - Code examples and patterns
---
**Repository**: https://github.com/vladkens/twscrape
**Stars**: 1998+
**Language**: Python
**License**: MIT
skills/twscrape/references/examples.md
-327
View File
@@ -1,327 +0,0 @@
# twscrape Examples
## Basic Search Example
```python
import asyncio
from twscrape import API, gather
async def main():
api = API()
# Search for tweets
tweets = await gather(api.search("elon musk", limit=20))
for tweet in tweets:
print(f"{tweet.user.username}: {tweet.rawContent}")
asyncio.run(main())
```
## User Profile Analysis
```python
async def analyze_user(username):
api = API()
# Get user info
user = await api.user_by_login(username)
print(f"User: {user.displayname}")
print(f"Followers: {user.followersCount}")
print(f"Following: {user.followingCount}")
# Get recent tweets
tweets = await gather(api.user_tweets(user.id, limit=50))
print(f"Recent tweets: {len(tweets)}")
return user, tweets
```
## Follower Network Collection
```python
async def collect_network(user_id):
api = API()
# Collect followers
followers = await gather(api.followers(user_id, limit=100))
print(f"Collected {len(followers)} followers")
# Collect following
following = await gather(api.following(user_id, limit=100))
print(f"Collected {len(following)} following")
return followers, following
```
## Advanced Search with Filters
```python
async def advanced_search():
api = API()
# Search with language filter
en_tweets = await gather(api.search("python lang:en", limit=20))
# Search with date filter
recent_tweets = await gather(api.search("AI since:2024-01-01", limit=20))
# Search from specific user
user_tweets = await gather(api.search("from:elonmusk", limit=20))
# Search with media
media_tweets = await gather(api.search("cats filter:media", limit=20))
return en_tweets, recent_tweets, user_tweets, media_tweets
```
## Tweet Thread Analysis
```python
async def analyze_thread(tweet_id):
api = API()
# Get tweet details
tweet = await api.tweet_details(tweet_id)
print(f"Tweet: {tweet.rawContent}")
# Get replies
replies = await gather(api.tweet_replies(tweet_id, limit=100))
print(f"Replies: {len(replies)}")
# Get retweeters
retweeters = await gather(api.retweeters(tweet_id, limit=100))
print(f"Retweeters: {len(retweeters)}")
return tweet, replies, retweeters
```
## Batch User Processing
```python
async def process_multiple_users(usernames):
api = API()
results = []
tasks = []
for username in usernames:
task = api.user_by_login(username)
tasks.append(task)
users = await asyncio.gather(*tasks)
for user in users:
if user:
print(f"Processed: {user.displayname}")
results.append(user)
return results
# Usage
usernames = ["elonmusk", "xdevelopers", "github"]
users = await process_multiple_users(usernames)
```
## Real-time Monitoring
```python
async def monitor_keywords(keywords, limit=100):
api = API()
for keyword in keywords:
print(f"\\nMonitoring: {keyword}")
async for tweet in api.search(keyword, limit=limit):
print(f"[{tweet.date}] @{tweet.user.username}: {tweet.rawContent[:100]}")
# Process tweet
if tweet.likeCount > 1000:
print(f" -> Popular tweet! {tweet.likeCount} likes")
# Usage
await monitor_keywords(["python", "javascript", "ai"], limit=50)
```
## Data Export to JSON
```python
import json
async def export_user_data(username, output_file):
api = API()
user = await api.user_by_login(username)
tweets = await gather(api.user_tweets(user.id, limit=100))
data = {
'user': user.dict(),
'tweets': [tweet.dict() for tweet in tweets]
}
with open(output_file, 'w', encoding='utf-8') as f:
json.dump(data, f, indent=2, ensure_ascii=False)
print(f"Exported to {output_file}")
# Usage
await export_user_data("elonmusk", "elon_data.json")
```
## Trends Analysis
```python
async def analyze_trends():
api = API()
# Get different trend categories
news_trends = await gather(api.trends("news"))
sport_trends = await gather(api.trends("sport"))
print("News Trends:")
for trend in news_trends[:10]:
print(f" - {trend}")
print("\\nSport Trends:")
for trend in sport_trends[:10]:
print(f" - {trend}")
return news_trends, sport_trends
```
## Using Context Manager for Early Termination
```python
from contextlib import aclosing
async def find_specific_tweet(query, target_id):
api = API()
async with aclosing(api.search(query)) as gen:
async for tweet in gen:
if tweet.id == target_id:
print(f"Found target tweet: {tweet.rawContent}")
return tweet
if tweet.id < target_id:
print("Target not found in results")
break
return None
```
## Account Setup Example
```python
async def setup_accounts():
api = API()
# Add accounts with cookies (more stable)
cookies = "abc=12; ct0=xyz"
await api.pool.add_account(
"user1",
"password1",
"user1@example.com",
"mail_password1",
cookies=cookies
)
# Add account with credentials
await api.pool.add_account(
"user2",
"password2",
"user2@example.com",
"mail_password2"
)
# Login all accounts
await api.pool.login_all()
print("Accounts setup complete")
```
## Proxy Configuration Example
```python
async def use_proxy():
# Global proxy
proxy = "http://user:pass@proxy.example.com:8080"
api = API(proxy=proxy)
# Make requests through proxy
user = await api.user_by_login("elonmusk")
print(f"User: {user.displayname}")
# Change proxy dynamically
api.proxy = "socks5://user:pass@127.0.0.1:1080"
tweets = await gather(api.search("python", limit=10))
# Disable proxy
api.proxy = None
more_tweets = await gather(api.search("javascript", limit=10))
```
## Error Handling
```python
async def safe_user_lookup(username):
api = API()
try:
user = await api.user_by_login(username)
return user
except Exception as e:
print(f"Error fetching user {username}: {e}")
return None
async def bulk_lookup_with_errors(usernames):
results = []
for username in usernames:
user = await safe_user_lookup(username)
if user:
results.append(user)
return results
```
## Complete Workflow Example
```python
import asyncio
import json
from twscrape import API, gather
from twscrape.logger import set_log_level
async def complete_workflow():
# Setup
api = API("my_data.db")
set_log_level("INFO")
# Add accounts
await api.pool.add_account(
"user1", "pass1", "email1@example.com", "mail_pass1",
cookies="cookie_string_here"
)
# Search and analyze
query = "python programming"
tweets = await gather(api.search(query, limit=100))
# Extract user data
users = {}
for tweet in tweets:
if tweet.user.username not in users:
users[tweet.user.username] = {
'user': tweet.user.dict(),
'tweets': []
}
users[tweet.user.username]['tweets'].append(tweet.dict())
# Export results
with open('results.json', 'w', encoding='utf-8') as f:
json.dump(users, f, indent=2, ensure_ascii=False)
print(f"Processed {len(tweets)} tweets from {len(users)} users")
if __name__ == "__main__":
asyncio.run(complete_workflow())
```
skills/twscrape/references/index.md
-39
View File
@@ -1,39 +0,0 @@
# twscrape Reference Documentation
## Overview
This directory contains detailed reference documentation for twscrape, a Python library for scraping Twitter/X data.
## Reference Files
### Core Documentation
- **[installation.md](installation.md)** - Installation instructions and dependencies
- **[api_methods.md](api_methods.md)** - Complete API method reference with parameters
- **[account_management.md](account_management.md)** - Account setup, login, and rotation
- **[cli_usage.md](cli_usage.md)** - Command-line interface guide
- **[proxy_config.md](proxy_config.md)** - Proxy configuration and setup
- **[examples.md](examples.md)** - Practical code examples and patterns
## Quick Navigation
### Getting Started
1. Read [installation.md](installation.md) for setup
2. Review [account_management.md](account_management.md) for adding accounts
3. Check [examples.md](examples.md) for quick start code
### API Reference
- For programmatic usage: [api_methods.md](api_methods.md)
- For command-line usage: [cli_usage.md](cli_usage.md)
### Advanced Topics
- Proxy configuration: [proxy_config.md](proxy_config.md)
- Rate limit handling: See [api_methods.md](api_methods.md#rate-limits)
## Key Features
- ✅ Async/await support for parallel operations
- ✅ Automatic account rotation
- ✅ Session persistence
- ✅ Multiple proxy support
- ✅ SNScrape-compatible data models
- ✅ Both CLI and Python API
skills/twscrape/references/installation.md
-66
View File
@@ -1,66 +0,0 @@
# Installation
## Standard Installation
```bash
pip install twscrape
```
## Development Version
Install the latest development version directly from GitHub:
```bash
pip install git+https://github.com/vladkens/twscrape.git
```
## Requirements
- Python 3.7+
- asyncio support
- Internet connection for Twitter/X access
## Dependencies
The library automatically installs required dependencies:
- `httpx` - Async HTTP client
- `aiosqlite` - Async SQLite database
- Additional dependencies as specified in setup.py
## Verification
Verify installation:
```bash
# Check CLI is available
twscrape --help
# Check Python import works
python -c "from twscrape import API; print('OK')"
```
## Upgrading
```bash
pip install --upgrade twscrape
```
## Uninstallation
```bash
pip uninstall twscrape
```
## Database Location
By default, twscrape creates `accounts.db` in your current working directory. You can specify a custom location:
```python
api = API("path/to/custom.db")
```
Or via CLI:
```bash
twscrape --db path/to/custom.db <command>
```