vibe-coding-cn/assets/skills/twscrape/SKILL.md

name, description

name	description
twscrape	twscrape Twitter/X scraping skill: account pool setup, async search, user/tweet collection, CLI usage, proxy configuration, and rate-limit troubleshooting. Use when extracting public Twitter/X data with twscrape.

twscrape Skill

Use this skill to build or debug twscrape workflows for public Twitter/X data extraction with account rotation, async collection, CLI commands, and proxy-aware operation.

When to Use This Skill

Trigger when any of these applies:

• Scraping Twitter/X search results, profiles, followers, timelines, replies, retweeters, media, or trends with twscrape.
• Setting up account pools, cookies, login flows, email verification, or account rotation.
• Choosing between Python async API and the twscrape CLI.
• Diagnosing rate limits, empty results, login failures, proxy failures, or suspended accounts.
• Exporting normalized tweet/user data for monitoring, analytics, research, or archival pipelines.

Not For / Boundaries

• Not for bypassing access controls, private content, paid-only data, or platform restrictions.
• Not for guaranteed high-volume scraping; account health, platform changes, and endpoint limits can invalidate assumptions.
• Do not place real Twitter/X passwords, cookies, email passwords, or proxy credentials in examples, commits, logs, or issue reports.
• Required inputs: target query/user/tweet/list, collection limit, output format, account source, proxy requirements, and compliance constraints.
• If behavior differs from these notes, verify against references/ and the upstream repository before changing production collectors.

Quick Reference

Common Patterns

Install the library

pip install twscrape

Create an API client and add a cookie-backed account

from twscrape import API

api = API("accounts.db")
await api.pool.add_account(
    "username",
    "password",
    "email@example.com",
    "email-password",
    cookies="ct0=...; auth_token=...",
)

Login all configured accounts

await api.pool.login_all()

Search recent tweets

from twscrape import gather

tweets = await gather(api.search("python lang:en", limit=50))

Fetch a user then collect timeline data

user = await api.user_by_login("xdevelopers")
tweets = await gather(api.user_tweets(user.id, limit=100))

Collect followers or following

followers = await gather(api.followers(user.id, limit=100))
following = await gather(api.following(user.id, limit=100))

Inspect tweet details and replies

tweet = await api.tweet_details(1234567890)
replies = await gather(api.tweet_replies(tweet.id, limit=50))

Use the CLI for a small search

twscrape search "python lang:en" --limit=20

Manage accounts from the CLI

twscrape add_accounts accounts.txt username:password:email:email_password
twscrape login_accounts --manual
twscrape accounts

Set a global proxy

export TWS_PROXY=socks5://user:pass@127.0.0.1:1080
twscrape search "bitcoin" --limit=20

Enable debug logging

from twscrape.logger import set_log_level

set_log_level("DEBUG")

Examples

Example 1: Search Export

• Input: query python lang:en, limit 50, output JSON Lines.
• Steps:
  1. Confirm at least one healthy account with twscrape accounts.
  2. Use await gather(api.search(query, limit=50)).
  3. Serialize selected fields such as id, date, user.username, and rawContent.
• Expected output / acceptance: a JSONL file with up to 50 tweet records and no credentials in logs.

Example 2: User Monitoring

• Input: username xdevelopers, timeline limit 100.
• Steps:
  1. Resolve the account with await api.user_by_login(username).
  2. Collect api.user_tweets(user.id, limit=100).
  3. Store tweet IDs and timestamps so later runs can deduplicate.
• Expected output / acceptance: user metadata plus a deduplicated timeline batch.

Example 3: Rate-Limit Triage

• Input: collector returns no data or waits indefinitely.
• Steps:
  1. Run twscrape accounts and identify locked, suspended, or rate-limited accounts.
  2. Enable debug logging and retry the smallest failing query.
  3. Add healthy accounts or wait for endpoint-specific reset before scaling up.
• Expected output / acceptance: the failing mode is classified as account health, query syntax, proxy/network, or platform limit.

References

• references/index.md: navigation for the local twscrape reference set.
• references/installation.md: installation and dependency notes.
• references/account_management.md: account pool, login, and rotation behavior.
• references/api_methods.md: Python API method reference.
• references/cli_usage.md: command-line usage.
• references/proxy_config.md: proxy configuration and precedence.
• references/examples.md: longer code examples and extraction patterns.

Maintenance

• Sources: local references/ extracted from upstream twscrape material and the upstream repository noted there.
• Last updated: 2026-04-28
• Known limits: Twitter/X endpoints and account policies change without notice; validate live collectors against a small sample before large runs.