diff --git a/assets/skills/ccxt/SKILL.md b/assets/skills/ccxt/SKILL.md index 2da5afb..1ffec13 100644 --- a/assets/skills/ccxt/SKILL.md +++ b/assets/skills/ccxt/SKILL.md @@ -1,105 +1,139 @@ --- name: ccxt -description: CCXT cryptocurrency trading library. Use for cryptocurrency exchange APIs, trading, market data, order management, and crypto trading automation across 150+ exchanges. Supports JavaScript/Python/PHP. +description: "CCXT crypto exchange API skill: unified market data, order books, balances, order creation, sandbox mode, rate limits, verbose debugging, and exchange capability checks for JavaScript/Python/PHP trading automation." --- -# Ccxt Skill +# ccxt Skill -Comprehensive assistance with ccxt development, generated from official documentation. +Use this skill to implement or debug CCXT integrations against cryptocurrency exchanges while keeping exchange-specific behavior explicit and testable. ## When to Use This Skill -This skill should be triggered when: -- Working with ccxt -- Asking about ccxt features or APIs -- Implementing ccxt solutions -- Debugging ccxt code -- Learning ccxt best practices +Trigger when any of these applies: +- Fetching tickers, order books, trades, OHLCV, balances, markets, or exchange metadata through CCXT. +- Creating, cancelling, or inspecting orders with the unified CCXT API. +- Checking exchange capabilities with `exchange.has`, `exchange.features`, `loadMarkets()`, or market metadata. +- Handling sandbox/testnet mode, rate limits, authentication, precision, contract size, or market-buy cost semantics. +- Producing minimal reproducible debug reports with verbose HTTP request/response output. + +## Not For / Boundaries + +- Not financial advice, strategy validation, custody guidance, or a guarantee that an exchange supports a specific feature. +- Never run live orders before sandbox or dry-run validation; call `set_sandbox_mode(True)` immediately after exchange construction when supported. +- Do not log, commit, or paste API keys, secrets, passphrases, cookies, or private wallet information. +- Required inputs: language/runtime, exchange id, market symbol, operation type, sandbox/live mode, auth status, and exact error/verbose output. +- Exchange APIs change independently; verify feature support against `exchange.has`, `exchange.features`, and the relevant reference file before production use. ## Quick Reference ### Common Patterns -**Pattern 1:** Frequently Asked Questions I'm trying to run the code, but it's not working, how do I fix it? If your question is formulated in a short manner like the above, we won't help. We don't teach programming. If you're unable to read and understand the Manual or you can't follow precisely the guides from the CONTRIBUTING doc on how to report an issue, we won't help either. Read the CONTRIBUTING guides on how to report an issue and read the Manual. You should not risk anyone's money and time without reading the entire Manual very carefully. You should not risk anything if you're not used to a lot of reading with tons of details. Also, if you don't have the confidence with the programming language you're using, there are much better places for coding fundamentals and practice. Search for python tutorials, js videos, play with examples, this is how other people climb up the learning curve. No shortcuts, if you want to learn something. What is required to get help? When asking a question: Use the search button for duplicates first! Post your request and response in verbose mode! Add exchange.verbose = true right before the line you're having issues with, and copypaste what you see on your screen. It's written and mentioned everywhere, in the Troubleshooting section, in the README and in many answers to similar questions among previous issues and pull requests. No excuses. The verbose output should include both the request and response from the exchange. Include the full error callstack! Write your programming language and language version number Write the CCXT / CCXT Pro library version number Which exchange it is Which method you're trying to call Post your code to reproduce the problem. Make it a complete short runnable program, don't swallow the lines and make it as compact as you can (5-10 lines of code), including the exchange instantation code. Remove all irrelevant parts from it, leaving just the essence of the code to reproduce the issue. DON'T POST SCREENSHOTS OF CODE OR ERRORS, POST THE OUTPUT AND CODE IN PLAIN TEXT! Surround code and output with triple backticks: ```GOOD```. Don't confuse the backtick symbol (`) with the quote symbol ('): '''BAD''' Don't confuse a single backtick with triple backticks: `BAD` DO NOT POST YOUR apiKey AND secret! Keep them safe (remove them before posting)! I am calling a method and I get an error, what am I doing wrong? You're not reporting the issue properly ) Please, help the community to help you ) Read this and follow the steps: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#how-to-submit-an-issue. Once again, your code to reproduce the issue and your verbose request and response ARE REQUIRED. Just the error traceback, or just the response, or just the request, or just the code – is not enough! I got an incorrect result from a method call, can you help? Basically the same answer as the previous question. Read and follow precisely: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#how-to-submit-an-issue. Once again, your code to reproduce the issue and your verbose request and response ARE REQUIRED. Just the error traceback, or just the response, or just the request, or just the code – is not enough! Can you implement feature foo in exchange bar? Yes, we can. And we will, if nobody else does that before us. There's very little point in asking this type of questions, because the answer is always positive. When someone asks if we can do this or that, the question is not about our abilities, it all boils down to time and management needed for implementing all accumulated feature requests. Moreover, this is an open-source library which is a work in progress. This means, that this project is intended to be developed by the community of users, who are using it. What you're asking is not whether we can or cannot implement it, in fact you're actually telling us to go do that particular task and this is not how we see a voluntary collaboration. Your contributions, PRs and commits are welcome: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#how-to-contribute-code. We don't give promises or estimates on the free open-source work. If you wish to speed it up, feel free to reach out to us via info@ccxt.trade. When will you add feature foo for exchange bar ? What's the estimated time? When should we expect this? We don't give promises or estimates on the open-source work. The reasoning behind this is explained in the previous paragraph. When will you add the support for an exchange requested in the Issues? Again, we can't promise on the dates for adding this or that exchange, due to reasons outlined above. The answer will always remain the same: as soon as we can. How long should I wait for a feature to be added? I need to decide whether to implement it myself or to wait for the CCXT Dev Team to implement it for me. Please, go for implemeting it yourself, do not wait for us. We will add it as soon as we can. Also, your contributions are very welcome: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#how-to-contribute-code What's your progress on adding the feature foo that was requested earlier? How do you do implementing exchange bar? This type of questions is usually a waste of time, because answering it usually requires too much time for context-switching, and it often takes more time to answer this question, than to actually satisfy the request with code for a new feature or a new exchange. The progress of this open-source project is also open, so, whenever you're wondering how it is doing, take a look into commit history. What is the status of this PR? Any update? If it is not merged, it means that the PR contains errors, that should be fixed first. If it could be merged as is – we would merge it, and you wouldn't have asked this question in the first place. The most frequent reason for not merging a PR is a violation of any of the CONTRIBUTING guidelines. Those guidelines should be taken literally, cannot skip a single line or word from there if you want your PR to be merged quickly. Code contributions that do not break the guidelines get merged almost immediately (usually, within hours). Can you point out the errors or what should I edit in my PR to get it merged into master branch? Unfortunately, we don't always have the time to quickly list out each and every single error in the code that prevents it from merging. It is often easier and faster to just go and fix the error rather than explain what one should do to fix it. Most of them are already outlined in the CONTRIBUTING guidelines. The main rule of thumb is to follow all guidelines literally. Hey! The fix you've uploaded is in TypeScript, would you fix JavaScript / Python / PHP as well, please? Our build system generates exchange-specific JavaScript, Python and PHP code for us automatically, so it is transpiled from TypeScript, and there's no need to fix all languages separately one by one. Thus, if it is fixed in TypeScript, it is fixed in JavaScript NPM, Python pip and PHP Composer as well. The automatic build usually takes 15-20 minutes. Just upgrade your version with npm, pip or composer after the new version arrives and you'll be fine. More about it here: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#multilanguage-support https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#transpiled-generated-files How to create an order with takeProfit+stopLoss? Some exchanges support createOrder with the additional "attached" stopLoss & takeProfit sub-orders - view StopLoss And TakeProfit Orders Attached To A Position. However, some exchanges might not support that feature and you will need to run separate createOrder methods to add conditional order (e.g. *trigger order | stoploss order | takeprofit order) to the already open position - view [Conditional orders](Manual.md#Conditional Orders). You can also check them by looking at exchange.has['createOrderWithTakeProfitAndStopLoss'], exchange.has['createStopLossOrder'] and exchange.has['createTakeProfitOrder'], however they are not as precise as .features property. How to create a spot market buy with cost? To create a market-buy order with cost, first, you need to check if the exchange supports that feature (exchange.has['createMarketBuyOrderWithCost']). If it does, then you can use the createMarketBuyOrderWithCost` method. Example: order = await exchange.createMarketBuyOrderWithCost(symbol, cost) What does the createMarketBuyRequiresPrice option mean? Many exchanges require the amount to be in the quote currency (they don't accept the base amount) when placing spot-market buy orders. In those cases, the exchange will have the option createMarketBuyRequiresPrice set to true. Example: If you wanted to buy BTC/USDT with a market buy-order, you would need to provide an amount = 5 USDT instead of 0.000X. We have a check to prevent errors that explicitly require the price because users will usually provide the amount in the base currency. So by default, if you do, create_order(symbol, 'market,' 'buy,' 10) will throw an error if the exchange has that option (createOrder() requires the price argument for market buy orders to calculate the total cost to spend (amount * price), alternatively set the createMarketBuyOrderRequiresPrice option or param to false...). If the exchange requires the cost and the user provided the base amount, we need to request an extra parameter price and multiply them to get the cost. If you're aware of this behavior, you can simply disable createMarketBuyOrderRequiresPrice and pass the cost in the amount parameter, but disabling it does not mean you can place the order using the base amount instead of the quote. If you do create_order(symbol, 'market', 'buy', 0.001, 20000) ccxt will use the required price to calculate the cost by doing 0.01*20000 and send that value to the exchange. If you want to provide the cost directly in the amount argument, you can do exchange.options['createMarketBuyOrderRequiresPrice'] = False (you acknowledge that the amount will be the cost for market-buy) and then you can do create_order(symbol, 'market', 'buy', 10) This is basically to avoid a user doing this: create_order('SHIB/USDT', market, buy, 1000000) and thinking he's trying to buy 1kk of shib but in reality he's buying 1kk USDT worth of SHIB. For that reason, by default ccxt always accepts the base currency in the amount parameter. Alternatively, you can use the functions createMarketBuyOrderWithCost/ createMarketSellOrderWithCost if they are available. See more: Market Buys What's the difference between trading spot and swap/perpetual futures? Spot trading involves buying or selling a financial instrument (like a cryptocurrency) for immediate delivery. It's straightforward, involving the direct exchange of assets. Swap trading, on the other hand, involves derivative contracts where two parties exchange financial instruments or cash flows at a set date in the future, based on the underlying asset. Swaps are often used for leverage, speculation, or hedging and do not necessarily involve the exchange of the underlying asset until the contract expires. Besides that, you will be handling contracts if you're trading swaps and not the base currency (e.g., BTC) directly, so if you create an order with amount = 1, the amount in BTC will vary depending on the contractSize. You can check the contract size by doing: await exchange.loadMarkets() symbol = 'XRP/USDT:USDT' market = exchange.market(symbol) print(market['contractSize']) How to place a reduceOnly order? A reduceOnly order is a type of order that can only reduce a position, not increase it. To place a reduceOnly order, you typically use the createOrder method with a reduceOnly parameter set to true. This ensures that the order will only execute if it decreases the size of an open position, and it will either partially fill or not fill at all if executing it would increase the position size. Javascript const params = { 'reduceOnly': true, // set to true if you want to close a position, set to false if you want to open a new position } const order = await exchange.createOrder (symbol, type, side, amount, price, params) Python params = { 'reduceOnly': True, # set to True if you want to close a position, set to False if you want to open a new position } order = exchange.create_order (symbol, type, side, amount, price, params) PHP $params = { 'reduceOnly': true, // set to true if you want to close a position, set to false if you want to open a new position } $order = $exchange->create_order ($symbol, $type, $side, $amount, $price, $params); See more: Trailing Orders How to check the endpoint used by the unified method? To check the endpoint used by a unified method in the CCXT library, you would typically need to refer to the source code of the library for the specific exchange implementation you're interested in. The unified methods in CCXT abstract away the details of the specific endpoints they interact with, so this information is not directly exposed via the library's API. For detailed inspection, you can look at the implementation of the method for the particular exchange in the CCXT library's source code on GitHub. See more: Unified API How to differentiate between previousFundingRate, fundingRate and nextFundingRate in the funding rate structure? The funding rate structure has three different funding rate values that can be returned: previousFundingRaterefers to the most recently completed rate. fundingRate is the upcoming rate. This value is always changing until the funding time passes and then it becomes the previousFundingRate. nextFundingRate is only supported on a few exchanges and is the predicted funding rate after the upcoming rate. This value is two funding rates from now. As an example, say it is 12:30. The previousFundingRate happened at 12:00 and we're looking to see what the upcoming funding rate will be by checking the fundingRate value. In this example, given 4-hour intervals, the fundingRate will happen in the future at 4:00 and the nextFundingRate is the predicted rate that will happen at 8:00. - -``` -python tutorials +**Install CCXT for Python** +```bash +pip install ccxt ``` -**Pattern 2:** To create a market-buy order with cost, first, you need to check if the exchange supports that feature (exchange.has['createMarketBuyOrderWithCost']). If it does, then you can use the createMarketBuyOrderWithCost` method. Example: +**Create an exchange with built-in rate limiting** +```python +import ccxt -``` -exchange.has['createMarketBuyOrderWithCost']). If it does, then you can use the +exchange = ccxt.binance({"enableRateLimit": True}) ``` -**Pattern 3:** Example: If you wanted to buy BTC/USDT with a market buy-order, you would need to provide an amount = 5 USDT instead of 0.000X. We have a check to prevent errors that explicitly require the price because users will usually provide the amount in the base currency. - -``` -create_order(symbol, 'market,' 'buy,' 10) +**Enable sandbox mode before any other call** +```python +exchange = ccxt.binance({"apiKey": "KEY", "secret": "SECRET"}) +exchange.set_sandbox_mode(True) ``` -**Pattern 4:** For a complete list of all exchanges and their supported methods, please, refer to this example: https://github.com/ccxt/ccxt/blob/master/examples/js/exchange-capabilities.js - -``` -exchange.rateLimit +**Load and inspect markets** +```python +markets = exchange.load_markets() +market = exchange.market("BTC/USDT") ``` -**Pattern 5:** The ccxt library supports asynchronous concurrency mode in Python 3.5+ with async/await syntax. The asynchronous Python version uses pure asyncio with aiohttp. In async mode you have all the same properties and methods, but most methods are decorated with an async keyword. If you want to use async mode, you should link against the ccxt.async_support subpackage, like in the following example: - -``` -ccxt.async_support +**Fetch one ticker instead of all tickers** +```python +ticker = exchange.fetch_ticker("BTC/USDT") ``` -## Reference Files +**Fetch an order book for bid/ask work** +```python +book = exchange.fetch_order_book("BTC/USDT", limit=20) +best_bid = book["bids"][0] if book["bids"] else None +best_ask = book["asks"][0] if book["asks"] else None +``` -This skill includes comprehensive documentation in `references/`: +**Check capability before using an endpoint** +```python +if exchange.has.get("fetchOHLCV"): + candles = exchange.fetch_ohlcv("BTC/USDT", timeframe="1m", limit=100) +``` -- **cli.md** - Cli documentation -- **exchanges.md** - Exchanges documentation -- **faq.md** - Faq documentation -- **getting_started.md** - Getting Started documentation -- **manual.md** - Manual documentation -- **other.md** - Other documentation -- **pro.md** - Pro documentation -- **specification.md** - Specification documentation +**Create a reduce-only order when supported by the exchange** +```python +params = {"reduceOnly": True} +order = exchange.create_order("BTC/USDT:USDT", "limit", "sell", 1, 70000, params) +``` -Use `view` to read specific reference files when detailed information is needed. +**Debug with verbose request/response output** +```python +exchange.verbose = True +exchange.fetch_balance() +``` -## Working with This Skill +**Use async support in Python** +```python +import ccxt.async_support as ccxt -### For Beginners -Start with the getting_started or tutorials reference files for foundational concepts. +exchange = ccxt.binance({"enableRateLimit": True}) +try: + await exchange.load_markets() +finally: + await exchange.close() +``` -### For Specific Features -Use the appropriate category reference file (api, guides, etc.) for detailed information. +## Examples -### For Code Examples -The quick reference section above contains common patterns extracted from the official docs. +### Example 1: Market Data Collector -## Resources +- Input: exchange `binance`, symbol `BTC/USDT`, timeframe `1m`. +- Steps: + 1. Instantiate with `enableRateLimit=True`. + 2. Call `load_markets()` once and verify `exchange.has["fetchOHLCV"]`. + 3. Fetch candles and persist timestamp, open, high, low, close, volume. +- Expected output / acceptance: one normalized candle batch and no repeated `load_markets()` calls inside the polling loop. -### references/ -Organized documentation extracted from official sources. These files contain: -- Detailed explanations -- Code examples with language annotations -- Links to original documentation -- Table of contents for quick navigation +### Example 2: Sandbox Order Flow -### scripts/ -Add helper scripts here for common automation tasks. +- Input: exchange with sandbox support, symbol, side, amount, price. +- Steps: + 1. Create the exchange with credentials and immediately call sandbox mode. + 2. Load markets and inspect precision/limits for the symbol. + 3. Place a tiny limit order, fetch it by id, then cancel it. +- Expected output / acceptance: order lifecycle succeeds in sandbox and no live order endpoint is touched. -### assets/ -Add templates, boilerplate, or example projects here. +### Example 3: Debug a Failing Exchange Call -## Notes +- Input: an exception from `create_order()` or `fetch_balance()`. +- Steps: + 1. Reduce code to a 5-20 line reproduction including exchange construction. + 2. Set `exchange.verbose = True` immediately before the failing call. + 3. Capture language version, CCXT version, exchange id, symbol, method, request, response, and full stack trace. +- Expected output / acceptance: a reproducible report without secrets and with enough evidence to distinguish CCXT misuse from exchange API failure. -- This skill was automatically generated from official documentation -- Reference files preserve the structure and examples from source docs -- Code examples include language detection for better syntax highlighting -- Quick reference patterns are extracted from common usage examples in the docs +## References -## Updating +- `references/index.md`: navigation for the CCXT reference set. +- `references/manual.md`: unified API, markets, rate limits, orders, sandbox, and debugging. +- `references/faq.md`: common trading/order pitfalls and issue-reporting requirements. +- `references/exchanges.md`: exchange support and capability notes. +- `references/pro.md`: CCXT Pro and WebSocket-oriented material. +- `references/cli.md`: CCXT CLI usage. +- `references/specification.md`: generated API/spec material. -To refresh this skill with updated documentation: -1. Re-run the scraper with the same configuration -2. The skill will be rebuilt with the latest information +## Maintenance + +- Sources: local `references/` extracted from CCXT documentation. +- Last updated: 2026-04-28 +- Known limits: exchange-specific parameters and capabilities must be verified per exchange; examples intentionally avoid live credentials. diff --git a/assets/skills/claude-code-guide/SKILL.md b/assets/skills/claude-code-guide/SKILL.md index 780b39d..2b3d990 100644 --- a/assets/skills/claude-code-guide/SKILL.md +++ b/assets/skills/claude-code-guide/SKILL.md @@ -1,470 +1,106 @@ --- name: claude-code-guide -description: Claude Code 高级开发指南 - 全面的中文教程，涵盖工具使用、REPL 环境、开发工作流、MCP 集成、高级模式和最佳实践。适合学习 Claude Code 的高级功能和开发技巧。 +description: "Claude Code guide skill: CLI workflows, project context, tool permissions, slash commands, hooks, MCP, large-file analysis, debugging, and advanced coding-agent operating patterns." --- -# Claude Code 高级开发指南 +# claude-code-guide Skill -全面的 Claude Code 中文学习指南，涵盖从基础到高级的所有核心概念、工具使用、开发工作流和最佳实践。 +Use this skill to answer Claude Code workflow questions and to turn the long Chinese guide in `references/README.md` into focused operating steps. -## 何时使用此技能 +## When to Use This Skill -当需要以下帮助时使用此技能： -- 学习 Claude Code 的核心功能和工具 -- 掌握 REPL 环境的高级用法 -- 理解开发工作流和任务管理 -- 使用 MCP 集成外部系统 -- 实现高级开发模式 -- 应用 Claude Code 最佳实践 -- 解决常见问题和错误 -- 进行大文件分析和处理 +Trigger when any of these applies: +- Learning or explaining Claude Code CLI workflows, command habits, or project-context handling. +- Designing slash commands, hooks, MCP integrations, memory/context conventions, or task workflows. +- Debugging Claude Code permission, tool-use, file analysis, or large-context problems. +- Comparing interactive coding-agent patterns such as explore-plan-edit-verify, review loops, or multi-agent coordination. +- Extracting a compact answer from the full Claude Code guide in `references/README.md`. -## 快速参考 +## Not For / Boundaries -### Claude Code 核心工具（7个） +- Not the source of truth for current Claude product availability, pricing, or model names; verify those with official Anthropic docs when exact current behavior matters. +- Not for bypassing tool permissions, sandbox rules, repository policies, or higher-priority system/developer instructions. +- Do not treat the long guide as automatically correct for every Claude Code version; use it as local project reference material. +- Required inputs: target workflow, OS/shell, Claude Code version if relevant, current repository constraints, and the exact error or behavior observed. +- If a command may mutate files or external systems, identify the blast radius and validation path before recommending it. -1. **REPL** - JavaScript 运行时环境 - - 完整的 ES6+ 支持 - - 预加载库：D3.js, MathJS, Lodash, Papaparse, SheetJS - - 支持 async/await, BigInt, WebAssembly - - 文件读取：`window.fs.readFile()` +## Quick Reference -2. **Artifacts** - 可视化输出 - - React, Three.js, 图表库 - - HTML/SVG 渲染 - - 交互式组件 +### Common Patterns -3. **Web Search** - 网络搜索 - - 仅美国可用 - - 域名过滤支持 - -4. **Web Fetch** - 获取网页内容 - - HTML 转 Markdown - - 内容提取和分析 - -5. **Conversation Search** - 对话搜索 - - 搜索历史对话 - - 上下文检索 - -6. **Recent Chats** - 最近对话 - - 访问最近会话 - - 对话历史 - -7. **End Conversation** - 结束对话 - - 清理和总结 - - 会话管理 - -### 大文件分析工作流 +**Start with project context** +```text +Read README/AGENTS/CONTRIBUTING, inspect git status, then search for local patterns before editing. +``` +**Large-file triage** ```bash -# 阶段 1：定量评估 -wc -l filename.md # 行数统计 -wc -w filename.md # 词数统计 -wc -c filename.md # 字符数统计 - -# 阶段 2：结构分析 -grep "^#{1,6} " filename.md # 提取标题层次 -grep "```" filename.md # 识别代码块 -grep -c "keyword" filename.md # 关键词频率 - -# 阶段 3：内容提取 -Read filename.md offset=0 limit=50 # 文件开头 -Read filename.md offset=N limit=100 # 目标部分 -Read filename.md offset=-50 limit=50 # 文件结尾 +wc -l path/to/file.md +rg -n '^#{1,6} ' path/to/file.md +sed -n '1,120p' path/to/file.md ``` -### REPL 高级用法 - -```javascript -// 数据处理 -const data = [1, 2, 3, 4, 5]; -const sum = data.reduce((a, b) => a + b, 0); - -// 使用预加载库 -// Lodash -_.chunk([1, 2, 3, 4], 2); // [[1,2], [3,4]] - -// MathJS -math.sqrt(16); // 4 - -// D3.js -d3.range(10); // [0,1,2,3,4,5,6,7,8,9] - -// 读取文件 -const content = await window.fs.readFile('path/to/file'); - -// 异步操作 -const result = await fetch('https://api.example.com/data'); -const json = await result.json(); +**Slash command authoring shape** +```text +.claude/commands/.md ``` -### 斜杠命令系统 - -**内置命令：** -- `/help` - 显示帮助 -- `/clear` - 清除对话 -- `/plugin` - 管理插件 -- `/settings` - 配置设置 - -**自定义命令：** -创建 `.claude/commands/mycommand.md`： -```markdown -根据需求执行特定任务的指令 +**Hook design checklist** +```text +event -> allowed command -> timeout -> logging -> rollback/disable path ``` -使用：`/mycommand` - -### 开发工作流模式 - -#### 1. 文件分析工作流 -```bash -# 探索 → 理解 → 实现 -ls -la # 列出文件 -Read file.py # 读取内容 -grep "function" file.py # 搜索模式 -# 然后实现修改 +**MCP integration checklist** +```text +server name -> command/env -> auth source -> minimal permission -> smoke test ``` -#### 2. 算法验证工作流 -```bash -# 设计 → 验证 → 实现 -# 1. 在 REPL 中测试逻辑 -# 2. 验证边界情况 -# 3. 实现到代码 +**Debug a Claude Code failure** +```text +Capture: command, working directory, permission mode, exact error, relevant config, and smallest reproduction. ``` -#### 3. 数据探索工作流 -```bash -# 检查 → 分析 → 可视化 -# 1. 读取数据文件 -# 2. REPL 中分析 -# 3. Artifacts 可视化 +**Context hygiene rule** +```text +Keep entry prompts short; move long references to files and load only the needed section. ``` -## 核心概念 +## Examples -### 工具权限系统 +### Example 1: Add a Custom Slash Command -**自动授予权限的工具：** -- REPL -- Artifacts -- Web Search/Fetch -- Conversation Search +- Input: user wants a repeatable `/review-pr` workflow. +- Steps: + 1. Create or update `.claude/commands/review-pr.md` with scope, required inputs, and output format. + 2. Reference repository review rules instead of duplicating stale policy text. + 3. Test the command on a small diff and refine ambiguous steps. +- Expected output / acceptance: command is discoverable, deterministic, and does not bypass repo review constraints. -**需要授权的工具：** -- Bash (读/写文件系统) -- Edit (修改文件) -- Write (创建文件) +### Example 2: Diagnose a Permission Problem -### 项目上下文 +- Input: Claude Code refuses to run or edit a file. +- Steps: + 1. Identify whether the blocker is tool permission, filesystem sandbox, repository policy, or OS permissions. + 2. Capture the exact path, command, and error. + 3. Propose the least-permissive fix and a smoke test. +- Expected output / acceptance: user gets a concrete unblock path without granting broad unnecessary access. -Claude 自动识别： -- Git 仓库状态 -- 编程语言（从文件扩展名） -- 项目结构 -- 依赖配置 +### Example 3: Summarize a Long Guide Section -### 内存系统 +- Input: need the MCP section from `references/README.md`. +- Steps: + 1. Use heading search to locate the section. + 2. Read only the relevant slice. + 3. Return a short operator checklist plus links back to the local reference file. +- Expected output / acceptance: answer is actionable and avoids dumping the full long document. -**对话内存：** -- 存储在当前会话 -- 200K token 窗口 -- 自动上下文管理 +## References -**持久内存（实验性）：** -- 跨会话保存 -- 用户偏好记忆 -- 项目上下文保留 +- `references/index.md`: structured navigation for the guide. +- `references/README.md`: long-form Claude Code Chinese guide. -## MCP 集成 +## Maintenance -### 什么是 MCP？ - -Model Context Protocol - 连接 Claude 到外部系统的协议。 - -### MCP 服务器配置 - -配置文件：`~/.config/claude/mcp_config.json` - -```json -{ - "mcpServers": { - "my-server": { - "command": "node", - "args": ["path/to/server.js"], - "env": { - "API_KEY": "your-key" - } - } - } -} -``` - -### 使用 MCP 工具 - -Claude 会自动发现 MCP 工具并在对话中使用： - -``` -"使用 my-server 工具获取数据" -``` - -## 钩子系统 - -### 钩子类型 - -在 `.claude/settings.json` 配置： - -```json -{ - "hooks": { - "tool-pre-use": "echo 'About to use tool'", - "tool-post-use": "echo 'Tool used'", - "user-prompt-submit": "echo 'Processing prompt'" - } -} -``` - -### 常见钩子用途 - -- 自动格式化代码 -- 运行测试 -- Git 提交检查 -- 日志记录 -- 通知发送 - -## 高级模式 - -### 多代理协作 - -使用 Task 工具启动子代理： - -``` -"启动一个专门的代理来优化这个算法" -``` - -子代理特点： -- 独立上下文 -- 专注单一任务 -- 返回结果到主代理 - -### 智能任务管理 - -使用 TodoWrite 工具： - -``` -"创建任务列表来跟踪这个项目" -``` - -任务状态： -- `pending` - 待处理 -- `in_progress` - 进行中 -- `completed` - 已完成 - -### 代码生成模式 - -**渐进式开发：** -1. 生成基础结构 -2. 添加核心功能 -3. 实现细节 -4. 测试和优化 - -**验证驱动：** -1. 写测试用例 -2. 实现功能 -3. 运行测试 -4. 修复问题 - -## 质量保证 - -### 自动化测试 - -```bash -# 运行测试 -npm test -pytest - -# 类型检查 -mypy script.py -tsc --noEmit - -# 代码检查 -eslint src/ -flake8 . -``` - -### 代码审查模式 - -使用子代理进行审查： - -``` -"启动代码审查代理检查这个文件" -``` - -审查重点： -- 代码质量 -- 安全问题 -- 性能优化 -- 最佳实践 - -## 错误恢复 - -### 常见错误模式 - -1. **工具使用错误** - - 检查权限 - - 验证语法 - - 确认路径 - -2. **文件操作错误** - - 确认文件存在 - - 检查读写权限 - - 验证路径正确 - -3. **API 调用错误** - - 检查网络连接 - - 验证 API 密钥 - - 确认请求格式 - -### 渐进式修复策略 - -1. 隔离问题 -2. 最小化复现 -3. 逐步修复 -4. 验证解决方案 - -## 最佳实践 - -### 开发原则 - -1. **清晰优先** - 明确需求和目标 -2. **渐进实现** - 分步骤开发 -3. **持续验证** - 频繁测试 -4. **适当抽象** - 合理模块化 - -### 工具使用原则 - -1. **正确的工具** - 选择合适的工具 -2. **工具组合** - 多工具协同 -3. **权限最小化** - 只请求必要权限 -4. **错误处理** - 优雅处理失败 - -### 性能优化 - -1. **批量操作** - 合并多个操作 -2. **增量处理** - 处理大文件 -3. **缓存结果** - 避免重复计算 -4. **异步优先** - 使用 async/await - -## 安全考虑 - -### 沙箱模型 - -每个工具在隔离环境中运行： -- REPL：无文件系统访问 -- Bash：需要明确授权 -- Web：仅特定域名 - -### 最佳安全实践 - -1. **最小权限** - 仅授予必要权限 -2. **代码审查** - 检查生成的代码 -3. **敏感数据** - 不要共享密钥 -4. **定期审计** - 检查钩子和配置 - -## 故障排除 - -### 工具无法使用 - -**症状：** 工具调用失败 - -**解决方案：** -- 检查权限设置 -- 验证语法正确 -- 确认文件路径 -- 查看错误消息 - -### REPL 性能问题 - -**症状：** REPL 执行缓慢 - -**解决方案：** -- 减少数据量 -- 使用流式处理 -- 优化算法 -- 分批处理 - -### MCP 连接失败 - -**症状：** MCP 服务器无响应 - -**解决方案：** -- 检查配置文件 -- 验证服务器运行 -- 确认环境变量 -- 查看服务器日志 - -## 实用示例 - -### 示例 1：数据分析 - -```javascript -// 在 REPL 中 -const data = await window.fs.readFile('data.csv'); -const parsed = Papa.parse(data, { header: true }); -const values = parsed.data.map(row => parseFloat(row.value)); -const avg = _.mean(values); -const std = math.std(values); -console.log(`平均值: ${avg}, 标准差: ${std}`); -``` - -### 示例 2：文件搜索 - -```bash -# 在 Bash 中 -grep -r "TODO" src/ -find . -name "*.py" -type f -``` - -### 示例 3：网络数据获取 - -``` -"使用 web_fetch 获取 https://api.example.com/data 的内容， -然后在 REPL 中分析 JSON 数据" -``` - -## 参考文件 - -此技能包含详细文档： - -- **README.md** (9,594 行) - 完整的 Claude Code 高级指南 - -包含以下主题： -- 核心工具深度解析 -- REPL 高级协同模式 -- 开发工作流详解 -- MCP 集成完整指南 -- 钩子系统配置 -- 高级模式和最佳实践 -- 故障排除和安全考虑 - -使用 `view` 命令查看参考文件获取详细信息。 - -## 资源 - -- **GitHub 仓库**: https://github.com/karminski/claude-code-guide-study -- **原始版本**: https://github.com/Cranot/claude-code-guide -- **Anthropic 官方文档**: https://docs.claude.com - -## 注意事项 - -本指南结合了： -- 官方功能和公告 -- 实际使用观察到的模式 -- 概念性方法和最佳实践 -- 第三方工具集成 - -请在使用时参考最新的官方文档。 - ---- - -**使用这个技能深入掌握 Claude Code 的强大功能！** +- Sources: local `references/README.md` and `references/index.md`. +- Last updated: 2026-04-28 +- Known limits: Claude Code behavior can change; verify current CLI flags, hooks, and MCP details against official docs when precision matters. diff --git a/assets/skills/claude-cookbooks/SKILL.md b/assets/skills/claude-cookbooks/SKILL.md index d64a87b..401efb6 100644 --- a/assets/skills/claude-cookbooks/SKILL.md +++ b/assets/skills/claude-cookbooks/SKILL.md @@ -1,313 +1,127 @@ --- name: claude-cookbooks -description: Claude AI cookbooks - code examples, tutorials, and best practices for using Claude API. Use when learning Claude API integration, building Claude-powered applications, or exploring Claude capabilities. +description: "Claude cookbooks skill: Claude API examples for messages, tool use, vision, RAG, summarization, text-to-SQL, prompt caching, agents, multimodal workflows, and third-party integrations." --- -# Claude Cookbooks Skill +# claude-cookbooks Skill -Comprehensive code examples and guides for building with Claude AI, sourced from the official Anthropic cookbooks repository. +Use this skill to turn cookbook material into runnable Claude API integration patterns while keeping model/version assumptions explicit. ## When to Use This Skill -This skill should be triggered when: -- Learning how to use Claude API -- Implementing Claude integrations -- Building applications with Claude -- Working with tool use and function calling -- Implementing multimodal features (vision, image analysis) -- Setting up RAG (Retrieval Augmented Generation) -- Integrating Claude with third-party services -- Building AI agents with Claude -- Optimizing prompts for Claude -- Implementing advanced patterns (caching, sub-agents, etc.) +Trigger when any of these applies: +- Building applications that call the Claude API. +- Implementing tool use/function calling, structured outputs, RAG, summarization, classification, or text-to-SQL. +- Working with multimodal inputs such as images and document extraction. +- Exploring prompt caching, agents, sub-agent patterns, or third-party integrations. +- Looking up cookbook examples stored in `references/` and adapting them to a project. + +## Not For / Boundaries + +- Not the source of truth for latest Anthropic models, pricing, limits, or API changes; verify current API details with official docs when exact current behavior matters. +- Do not hard-code API keys or leak prompts containing private user data. +- Cookbook examples are starting points, not production architecture; add retries, timeouts, observability, evals, and security checks. +- Required inputs: language/runtime, use case, model policy from the project, data sensitivity, expected output schema, and failure handling requirements. +- If local references conflict with current official docs, prefer the official docs and update the reference notes. ## Quick Reference -### Basic API Usage +### Common Patterns +**Basic Messages API shape** ```python import anthropic -client = anthropic.Anthropic(api_key="your-api-key") - -# Simple message +client = anthropic.Anthropic(api_key="YOUR_API_KEY") response = client.messages.create( - model="claude-3-5-sonnet-20241022", + model="YOUR_APPROVED_MODEL", max_tokens=1024, - messages=[{ - "role": "user", - "content": "Hello, Claude!" - }] + messages=[{"role": "user", "content": "Hello"}], ) ``` -### Tool Use (Function Calling) - +**Tool definition shape** ```python -# Define a tool tools = [{ "name": "get_weather", - "description": "Get current weather for a location", + "description": "Get current weather for a location.", "input_schema": { "type": "object", - "properties": { - "location": {"type": "string", "description": "City name"} - }, - "required": ["location"] - } + "properties": {"location": {"type": "string"}}, + "required": ["location"], + }, }] - -# Use the tool -response = client.messages.create( - model="claude-3-5-sonnet-20241022", - max_tokens=1024, - tools=tools, - messages=[{"role": "user", "content": "What's the weather in San Francisco?"}] -) ``` -### Vision (Image Analysis) - +**Vision content shape** ```python -# Analyze an image -response = client.messages.create( - model="claude-3-5-sonnet-20241022", - max_tokens=1024, - messages=[{ - "role": "user", - "content": [ - { - "type": "image", - "source": { - "type": "base64", - "media_type": "image/jpeg", - "data": base64_image - } - }, - {"type": "text", "text": "Describe this image"} - ] - }] -) +content = [ + {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": base64_image}}, + {"type": "text", "text": "Describe the image."}, +] ``` -### Prompt Caching - +**Prompt caching shape** ```python -# Use prompt caching for efficiency -response = client.messages.create( - model="claude-3-5-sonnet-20241022", - max_tokens=1024, - system=[{ - "type": "text", - "text": "Large system prompt here...", - "cache_control": {"type": "ephemeral"} - }], - messages=[{"role": "user", "content": "Your question"}] -) +system = [{ + "type": "text", + "text": "Large stable system prompt...", + "cache_control": {"type": "ephemeral"}, +}] ``` -## Key Capabilities Covered +**RAG pipeline skeleton** +```text +ingest -> chunk -> embed/index -> retrieve -> rerank/filter -> answer with citations -> evaluate +``` -### 1. Classification -- Text classification techniques -- Sentiment analysis -- Content categorization -- Multi-label classification +**Production hardening checklist** +```text +timeouts, retries, redaction, structured logging, eval set, cost guard, rate-limit handling +``` -### 2. Retrieval Augmented Generation (RAG) -- Vector database integration -- Semantic search -- Context retrieval -- Knowledge base queries +## Examples -### 3. Summarization -- Document summarization -- Meeting notes -- Article condensing -- Multi-document synthesis +### Example 1: Add Tool Use to an App -### 4. Text-to-SQL -- Natural language to SQL queries -- Database schema understanding -- Query optimization -- Result interpretation +- Input: user asks for weather lookup through Claude. +- Steps: + 1. Define a JSON schema for `get_weather`. + 2. Send the user request with the tool definition. + 3. Execute only validated tool calls and return tool results to the model. +- Expected output / acceptance: tool arguments pass schema validation and no unapproved function is called. -### 5. Tool Use & Function Calling -- Tool definition and schema -- Parameter validation -- Multi-tool workflows -- Error handling +### Example 2: Build a RAG Answerer -### 6. Multimodal -- Image analysis and OCR -- Chart/graph interpretation -- Visual question answering -- Image generation integration +- Input: local product docs and user questions. +- Steps: + 1. Chunk and index documents with stable IDs. + 2. Retrieve relevant chunks for each question. + 3. Ask Claude to answer only from retrieved evidence and cite chunk IDs. +- Expected output / acceptance: unsupported claims are refused or marked unknown, and answers include source references. -### 7. Advanced Patterns -- Agent architectures -- Sub-agent delegation -- Prompt optimization -- Cost optimization with caching +### Example 3: Vision Extraction -## Repository Structure +- Input: screenshot or document image. +- Steps: + 1. Convert image to supported media type and base64. + 2. Send image plus extraction instructions. + 3. Validate returned fields against the expected schema. +- Expected output / acceptance: extracted data is structured, missing fields are explicit, and raw sensitive images are not logged. -The cookbooks are organized into these main categories: +## References -- **capabilities/** - Core AI capabilities (classification, RAG, summarization, text-to-SQL) -- **tool_use/** - Function calling and tool integration examples -- **multimodal/** - Vision and image-related examples -- **patterns/** - Advanced patterns like agents and workflows -- **third_party/** - Integrations with external services (Pinecone, LlamaIndex, etc.) -- **claude_agent_sdk/** - Agent SDK examples and templates -- **misc/** - Additional utilities (PDF upload, JSON mode, evaluations, etc.) +- `references/index.md`: navigation for cookbook topics. +- `references/main_readme.md` and `references/README.md`: upstream overview material. +- `references/tool_use.md`: tool-use examples. +- `references/capabilities.md`: classification, RAG, summarization, and text-to-SQL. +- `references/multimodal.md`: image and multimodal examples. +- `references/patterns.md`: agents, caching, and advanced patterns. +- `references/third_party.md`: vector DB and external integrations. +- `scripts/memory_tool.py`: local helper script retained from the cookbook material. -## Reference Files +## Maintenance -This skill includes comprehensive documentation in `references/`: - -- **main_readme.md** - Main repository overview -- **capabilities.md** - Core capabilities documentation -- **tool_use.md** - Tool use and function calling guides -- **multimodal.md** - Vision and multimodal capabilities -- **third_party.md** - Third-party integrations -- **patterns.md** - Advanced patterns and agents -- **index.md** - Complete reference index - -## Common Use Cases - -### Building a Customer Service Agent -1. Define tools for CRM access, ticket creation, knowledge base search -2. Use tool use API to handle function calls -3. Implement conversation memory -4. Add fallback mechanisms - -See: `references/tool_use.md#customer-service` - -### Implementing RAG -1. Create embeddings of your documents -2. Store in vector database (Pinecone, etc.) -3. Retrieve relevant context on query -4. Augment Claude's response with context - -See: `references/capabilities.md#rag` - -### Processing Documents with Vision -1. Convert document to images or PDF -2. Use vision API to extract content -3. Structure the extracted data -4. Validate and post-process - -See: `references/multimodal.md#vision` - -### Building Multi-Agent Systems -1. Define specialized agents for different tasks -2. Implement routing logic -3. Use sub-agents for delegation -4. Aggregate results - -See: `references/patterns.md#agents` - -## Best Practices - -### API Usage -- Use appropriate model for task (Sonnet for balance, Haiku for speed, Opus for complex tasks) -- Implement retry logic with exponential backoff -- Handle rate limits gracefully -- Monitor token usage for cost optimization - -### Prompt Engineering -- Be specific and clear in instructions -- Provide examples when needed -- Use system prompts for consistent behavior -- Structure outputs with JSON mode when needed - -### Tool Use -- Define clear, specific tool schemas -- Validate inputs and outputs -- Handle errors gracefully -- Keep tool descriptions concise but informative - -### Multimodal -- Use high-quality images (higher resolution = better results) -- Be specific about what to extract/analyze -- Respect size limits (5MB per image) -- Use appropriate image formats (JPEG, PNG, GIF, WebP) - -## Performance Optimization - -### Prompt Caching -- Cache large system prompts -- Cache frequently used context -- Monitor cache hit rates -- Balance caching vs. fresh content - -### Cost Optimization -- Use Haiku for simple tasks -- Implement prompt caching for repeated context -- Set appropriate max_tokens -- Batch similar requests - -### Latency Optimization -- Use streaming for long responses -- Minimize message history -- Optimize image sizes -- Use appropriate timeout values - -## Resources - -### Official Documentation -- [Anthropic Developer Docs](https://docs.claude.com) -- [API Reference](https://docs.claude.com/claude/reference) -- [Anthropic Support](https://support.anthropic.com) - -### Community -- [Anthropic Discord](https://www.anthropic.com/discord) -- [GitHub Cookbooks Repo](https://github.com/anthropics/claude-cookbooks) - -### Learning Resources -- [Claude API Fundamentals Course](https://github.com/anthropics/courses/tree/master/anthropic_api_fundamentals) -- [Prompt Engineering Guide](https://docs.claude.com/claude/docs/guide-to-anthropics-prompt-engineering-resources) - -## Working with This Skill - -### For Beginners -Start with `references/main_readme.md` and explore basic examples in `references/capabilities.md` - -### For Specific Features -- Tool use → `references/tool_use.md` -- Vision → `references/multimodal.md` -- RAG → `references/capabilities.md#rag` -- Agents → `references/patterns.md#agents` - -### For Code Examples -Each reference file contains practical, copy-pasteable code examples - -## Examples Available - -The cookbook includes 50+ practical examples including: -- Customer service chatbot with tool use -- RAG with Pinecone vector database -- Document summarization -- Image analysis and OCR -- Chart/graph interpretation -- Natural language to SQL -- Content moderation filter -- Automated evaluations -- Multi-agent systems -- Prompt caching optimization - -## Notes - -- All examples use official Anthropic Python SDK -- Code is production-ready with error handling -- Examples follow current API best practices -- Regular updates from Anthropic team -- Community contributions welcome - -## Skill Source - -This skill was created from the official Anthropic Claude Cookbooks repository: -https://github.com/anthropics/claude-cookbooks - -Repository cloned and processed on: 2025-10-29 +- Sources: local `references/` extracted from Anthropic cookbook material. +- Last updated: 2026-04-28 +- Known limits: examples may carry older model names; replace with the project-approved current model before use. diff --git a/assets/skills/coingecko/SKILL.md b/assets/skills/coingecko/SKILL.md index 8259d9c..c5a2cc6 100644 --- a/assets/skills/coingecko/SKILL.md +++ b/assets/skills/coingecko/SKILL.md @@ -1,80 +1,116 @@ --- name: coingecko -description: CoinGecko API documentation - cryptocurrency market data API, price feeds, market cap, volume, historical data. Use when integrating CoinGecko API, building crypto price trackers, or accessing cryptocurrency market data. +description: "CoinGecko API skill: cryptocurrency prices, market data, coin IDs, historical charts, trending search, onchain endpoints, API-key authentication, and rate-limit-aware request design." --- -# Coingecko Skill +# coingecko Skill -Comprehensive assistance with coingecko development, generated from official documentation. +Use this skill to integrate CoinGecko market data into applications, agents, dashboards, and analysis pipelines with clear API-key and rate-limit handling. ## When to Use This Skill -This skill should be triggered when: -- Working with coingecko -- Asking about coingecko features or APIs -- Implementing coingecko solutions -- Debugging coingecko code -- Learning coingecko best practices +Trigger when any of these applies: +- Querying token prices, market cap, volume, historical charts, trending coins, NFTs, categories, or exchange data from CoinGecko. +- Choosing Demo API vs Pro API root URLs and authentication headers. +- Designing rate-limit-aware polling, cache refresh, or price freshness checks. +- Building crypto dashboards, price alerts, analytics jobs, or market-data enrichers. +- Navigating CoinGecko MCP, REST, or onchain/GeckoTerminal reference material. + +## Not For / Boundaries + +- Not financial advice, token endorsement, trade execution, or market manipulation support. +- Do not expose API keys in query strings unless unavoidable; prefer headers and backend proxy insertion. +- Do not assume symbol uniqueness; resolve assets with CoinGecko coin IDs before price calls. +- Required inputs: plan type, root URL, API key availability, coin IDs/symbols/contracts, quote currencies, date range, and freshness requirements. +- CoinGecko endpoints and plan gates evolve; verify paid-only endpoints and rate limits in `references/` before production use. ## Quick Reference ### Common Patterns -*Quick reference patterns will be added as you use the skill.* +**Demo API ping with header auth** +```bash +curl -X GET "https://api.coingecko.com/api/v3/ping" \ + -H "x-cg-demo-api-key: YOUR_API_KEY" +``` -## Reference Files +**Pro API ping with header auth** +```bash +curl -X GET "https://pro-api.coingecko.com/api/v3/ping" \ + -H "x-cg-pro-api-key: YOUR_API_KEY" +``` -This skill includes comprehensive documentation in `references/`: +**Simple price for coin IDs** +```bash +curl "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin,ethereum&vs_currencies=usd" +``` -- **authentication.md** - Authentication documentation -- **coins.md** - Coins documentation -- **contract.md** - Contract documentation -- **exchanges.md** - Exchanges documentation -- **introduction.md** - Introduction documentation -- **market_data.md** - Market Data documentation -- **nfts.md** - Nfts documentation -- **other.md** - Other documentation -- **pricing.md** - Pricing documentation -- **reference.md** - Reference documentation -- **trending.md** - Trending documentation +**Include freshness fields in price response** +```bash +curl "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_last_updated_at=true" +``` -Use `view` to read specific reference files when detailed information is needed. +**Market list by market cap** +```bash +curl "https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=100&page=1" +``` -## Working with This Skill +**Historical chart by coin ID** +```bash +curl "https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=30" +``` -### For Beginners -Start with the getting_started or tutorials reference files for foundational concepts. +**Trending search** +```bash +curl "https://api.coingecko.com/api/v3/search/trending" +``` -### For Specific Features -Use the appropriate category reference file (api, guides, etc.) for detailed information. +**Onchain trending pools** +```bash +curl "https://api.coingecko.com/api/v3/onchain/networks/trending_pools" +``` -### For Code Examples -The quick reference section above contains common patterns extracted from the official docs. +## Examples -## Resources +### Example 1: Price Widget -### references/ -Organized documentation extracted from official sources. These files contain: -- Detailed explanations -- Code examples with language annotations -- Links to original documentation -- Table of contents for quick navigation +- Input: coin IDs `bitcoin,ethereum`, quote `usd`, refresh interval. +- Steps: + 1. Use `/simple/price` with `include_last_updated_at=true`. + 2. Cache responses according to product freshness needs and plan limits. + 3. Display stale-data warnings when `last_updated_at` is outside the allowed window. +- Expected output / acceptance: a small response with current prices and explicit freshness handling. -### scripts/ -Add helper scripts here for common automation tasks. +### Example 2: Market-Cap Dashboard -### assets/ -Add templates, boilerplate, or example projects here. +- Input: quote currency `usd`, `per_page=100`, page number. +- Steps: + 1. Query `/coins/markets`. + 2. Persist coin ID, symbol, name, price, market cap, and volume. + 3. Avoid treating symbols as primary keys because duplicates exist. +- Expected output / acceptance: stable dashboard rows keyed by CoinGecko ID. -## Notes +### Example 3: Trending Research Batch -- This skill was automatically generated from official documentation -- Reference files preserve the structure and examples from source docs -- Code examples include language detection for better syntax highlighting -- Quick reference patterns are extracted from common usage examples in the docs +- Input: need trending coins, NFTs, and categories in the last 24 hours. +- Steps: + 1. Query `/search/trending`. + 2. Normalize returned entities by type. + 3. For coins that require prices, follow up with `/simple/price` using IDs. +- Expected output / acceptance: a typed trending list with price enrichment only where IDs are available. -## Updating +## References -To refresh this skill with updated documentation: -1. Re-run the scraper with the same configuration -2. The skill will be rebuilt with the latest information +- `references/index.md`: navigation for local CoinGecko docs. +- `references/authentication.md`: Demo/Pro API keys, root URLs, and header names. +- `references/coins.md`: coin IDs, markets, charts, simple price, and trending search. +- `references/market_data.md`: NFT market data notes. +- `references/exchanges.md`: exchange endpoints. +- `references/trending.md`: onchain trending pool endpoints. +- `references/llms.md` and `references/llms-full.md`: LLM-oriented reference exports. + +## Maintenance + +- Sources: local `references/` extracted from CoinGecko documentation. +- Last updated: 2026-04-28 +- Known limits: plan availability and rate limits are account-specific; verify against the active CoinGecko plan before shipping. diff --git a/assets/skills/cryptofeed/SKILL.md b/assets/skills/cryptofeed/SKILL.md index b76cf76..141a77c 100644 --- a/assets/skills/cryptofeed/SKILL.md +++ b/assets/skills/cryptofeed/SKILL.md @@ -1,221 +1,119 @@ --- name: cryptofeed -description: Cryptofeed - Real-time cryptocurrency market data feeds from 40+ exchanges. WebSocket streaming, normalized data, order books, trades, tickers. Python library for algorithmic trading and market data analysis. +description: "Cryptofeed real-time crypto market data skill: WebSocket feeds, normalized tickers/trades/order books, NBBO, exchange subscriptions, authenticated channels, and backend streaming to Redis/Kafka/PostgreSQL." --- -# Cryptofeed Skill +# cryptofeed Skill -Comprehensive assistance with Cryptofeed development - a Python library for handling cryptocurrency exchange data feeds with normalized and standardized results. +Use this skill to build Python market-data pipelines with Cryptofeed across exchanges, channels, callbacks, NBBO aggregation, and storage backends. ## When to Use This Skill -This skill should be triggered when: -- Working with real-time cryptocurrency market data -- Implementing WebSocket streaming from crypto exchanges -- Building algorithmic trading systems -- Processing order book updates, trades, or ticker data -- Connecting to 40+ cryptocurrency exchanges -- Using normalized exchange APIs -- Implementing market data backends (Redis, MongoDB, Kafka, etc.) +Trigger when any of these applies: +- Streaming real-time crypto market data from multiple exchanges. +- Subscribing to tickers, trades, L1/L2/L3 order books, candles, funding, liquidations, balances, fills, or order updates. +- Building NBBO, arbitrage monitors, market-data recorders, or backend writers. +- Debugging symbol/channel support, callback shape, reconnection behavior, or backend configuration. +- Comparing Cryptofeed with exchange-specific WebSocket clients. + +## Not For / Boundaries + +- Not a trading strategy engine, order execution system, or persistence database by itself. +- Authenticated channels require exchange credentials; never commit or print secrets. +- Exchange support, symbols, and channel names vary; confirm against the exchange class and `references/README.md`. +- Required inputs: exchange list, symbols, channels, callback/backend target, auth need, and failure mode. +- For historical backfills, pair this with REST or a storage system; Cryptofeed is WebSocket-first. ## Quick Reference -### Installation +### Common Patterns -```python -# Basic installation +**Install Cryptofeed** +```bash pip install cryptofeed - -# With all optional backends -pip install cryptofeed[all] ``` -### Basic Usage Pattern - +**Create a simple feed handler** ```python from cryptofeed import FeedHandler -from cryptofeed.exchanges import Coinbase, Bitfinex -from cryptofeed.defines import TICKER, TRADES, L2_BOOK +from cryptofeed.defines import TICKER +from cryptofeed.exchanges import Coinbase -# Define callbacks -def ticker_callback(data): - print(f"Ticker: {data}") +def ticker(data, receipt_timestamp): + print(data) -def trade_callback(data): - print(f"Trade: {data}") - -# Create feed handler fh = FeedHandler() - -# Add exchange feeds -fh.add_feed(Coinbase( - symbols=['BTC-USD'], - channels=[TICKER], - callbacks={TICKER: ticker_callback} -)) - -fh.add_feed(Bitfinex( - symbols=['BTC-USD'], - channels=[TRADES], - callbacks={TRADES: trade_callback} -)) - -# Start receiving data +fh.add_feed(Coinbase(symbols=["BTC-USD"], channels=[TICKER], callbacks={TICKER: ticker})) fh.run() ``` -### National Best Bid/Offer (NBBO) - +**Subscribe to trades and L2 book** +```python +from cryptofeed.defines import TRADES, L2_BOOK +from cryptofeed.exchanges import Gemini + +fh.add_feed(Gemini( + symbols=["BTC-USD", "ETH-USD"], + channels=[TRADES, L2_BOOK], + callbacks={TRADES: trade_callback, L2_BOOK: book_callback}, +)) +``` + +**Build NBBO across exchanges** ```python -from cryptofeed import FeedHandler from cryptofeed.exchanges import Coinbase, Gemini, Kraken -def nbbo_update(symbol, bid, bid_size, ask, ask_size, bid_feed, ask_feed): - print(f'Pair: {symbol} Bid: {bid:.2f} ({bid_size:.6f}) from {bid_feed}') - print(f'Ask: {ask:.2f} ({ask_size:.6f}) from {ask_feed}') - -f = FeedHandler() -f.add_nbbo([Coinbase, Kraken, Gemini], ['BTC-USD'], nbbo_update) -f.run() +fh.add_nbbo([Coinbase, Kraken, Gemini], ["BTC-USD"], nbbo_callback) ``` -## Supported Exchanges (40+) - -### Major Exchanges -- **Binance** (Spot, Futures, Delivery, US) -- **Coinbase**, **Kraken** (Spot, Futures), **Bitfinex** -- **Gemini**, **OKX**, **Bybit** -- **Huobi** (Spot, DM, Swap), **Gate.io** (Spot, Futures) -- **KuCoin**, **Deribit**, **BitMEX**, **dYdX** - -### Additional Exchanges -AscendEX, Bequant, bitFlyer, Bithumb, Bitstamp, Blockchain.com, Bit.com, Bitget, Crypto.com, Delta, EXX, FMFW.io, HitBTC, Independent Reserve, OKCoin, Phemex, Poloniex, ProBit, Upbit - -## Supported Data Channels - -### Market Data (Public) -- **L1_BOOK** - Top of order book -- **L2_BOOK** - Price aggregated sizes -- **L3_BOOK** - Price aggregated orders -- **TRADES** - Executed trades (taker side) -- **TICKER** - Price ticker updates -- **FUNDING** - Funding rate data -- **OPEN_INTEREST** - Open interest statistics -- **LIQUIDATIONS** - Liquidation events -- **INDEX** - Index price data -- **CANDLES** - Candlestick/K-line data - -### Authenticated Channels (Private) -- **ORDER_INFO** - Order status updates -- **TRANSACTIONS** - Deposits and withdrawals -- **BALANCES** - Wallet balance updates -- **FILLS** - User's executed trades - -## Supported Backends - -Write data directly to storage: - -- **Redis** (Streams and Sorted Sets) -- **Arctic** - Time-series database -- **ZeroMQ**, **InfluxDB v2**, **MongoDB** -- **Kafka**, **RabbitMQ**, **PostgreSQL** -- **QuasarDB**, **GCP Pub/Sub**, **QuestDB** -- **UDP/TCP/Unix Sockets** - -## Key Features - -### Real-time Data Normalization -Cryptofeed normalizes data across all exchanges, providing consistent: -- Symbol formatting -- Timestamp handling -- Data structures -- Channel names - -### WebSocket + REST Fallback -- Primarily uses WebSockets for real-time data -- Falls back to REST polling when WebSocket unavailable -- Automatic reconnection handling - -### NBBO Aggregation -Create synthetic National Best Bid/Offer feeds by aggregating data across multiple exchanges to find arbitrage opportunities. - -### Backend Integration -Direct data writing to various storage systems without custom integration code. - -## Requirements - -- **Python**: 3.8 or higher -- **Installation**: Via pip or from source -- **Optional Dependencies**: Install backends as needed - -## Common Use Cases - -### Multi-Exchange Price Monitoring +**Separate callback work from IO-heavy persistence** ```python -fh = FeedHandler() -fh.add_feed(Binance(symbols=['BTC-USDT'], channels=[TICKER], callbacks=ticker_cb)) -fh.add_feed(Coinbase(symbols=['BTC-USD'], channels=[TICKER], callbacks=ticker_cb)) -fh.add_feed(Kraken(symbols=['BTC-USD'], channels=[TICKER], callbacks=ticker_cb)) -fh.run() +def trade_callback(data, receipt_timestamp): + queue.put_nowait((data, receipt_timestamp)) ``` -### Order Book Depth Analysis +**Check supported channel names** ```python -def book_callback(book, receipt_timestamp): - print(f"Bids: {len(book.book.bids)} | Asks: {len(book.book.asks)}") - -fh.add_feed(Coinbase( - symbols=['BTC-USD'], - channels=[L2_BOOK], - callbacks={L2_BOOK: book_callback} -)) +from cryptofeed.defines import L1_BOOK, L2_BOOK, L3_BOOK, TRADES, TICKER ``` -### Trade Flow Analysis -```python -def trade_callback(trade, receipt_timestamp): - print(f"{trade.exchange} - {trade.symbol}: {trade.side} {trade.amount} @ {trade.price}") +## Examples -fh.add_feed(Binance( - symbols=['BTC-USDT', 'ETH-USDT'], - channels=[TRADES], - callbacks={TRADES: trade_callback} -)) -``` +### Example 1: Single-Exchange Ticker Stream -## Reference Files +- Input: exchange `Coinbase`, symbol `BTC-USD`, channel `TICKER`. +- Steps: + 1. Create `FeedHandler`. + 2. Add one exchange feed with a lightweight callback. + 3. Run the handler and observe normalized ticker objects. +- Expected output / acceptance: ticker updates print with timestamps and no callback-blocking persistence work. -This skill includes documentation in `references/`: +### Example 2: NBBO Monitor -- **getting_started.md** - Installation and basic usage -- **README.md** - Complete overview and examples +- Input: exchanges `Coinbase`, `Kraken`, `Gemini`, symbol `BTC-USD`. +- Steps: + 1. Define `nbbo_callback(symbol, bid, bid_size, ask, ask_size, bid_feed, ask_feed)`. + 2. Add NBBO with the exchange class list. + 3. Alert only when spread or venue changes cross configured thresholds. +- Expected output / acceptance: best bid/ask updates include source venues. -Use `view` to read specific reference files when detailed information is needed. +### Example 3: Backend Recorder -## Working with This Skill +- Input: symbols, channels, and a storage backend such as Redis/Kafka/PostgreSQL. +- Steps: + 1. Confirm optional backend dependencies are installed. + 2. Configure the backend callback instead of writing inside a custom callback. + 3. Run a small symbol set before scaling to many exchanges. +- Expected output / acceptance: records arrive in the backend with normalized exchange and symbol fields. -### For Beginners -Start with basic FeedHandler setup and single exchange connections before adding multiple feeds. +## References -### For Advanced Users -Explore NBBO feeds, authenticated channels, and backend integrations for production systems. +- `references/index.md`: navigation for the local Cryptofeed references. +- `references/README.md`: supported exchanges, basic usage, NBBO, channels, and backends. +- `references/other.md`: additional generated reference material. -### For Code Examples -See the quick reference section above and the reference files for complete working examples. +## Maintenance -## Resources - -- **Repository**: https://github.com/bmoscon/cryptofeed -- **PyPI**: https://pypi.python.org/pypi/cryptofeed -- **Examples**: https://github.com/bmoscon/cryptofeed/tree/master/examples -- **Documentation**: https://github.com/bmoscon/cryptofeed/blob/master/docs/README.md -- **Discord**: https://discord.gg/zaBYaGAYfR -- **Related**: Cryptostore (containerized data storage) - -## Notes - -- Requires Python 3.8+ -- WebSocket-first approach with REST fallback -- Normalized data across all exchanges -- Active development and community support -- 40+ supported exchanges and growing +- Sources: local `references/` extracted from Cryptofeed documentation. +- Last updated: 2026-04-28 +- Known limits: channel support is exchange-specific; always validate symbol naming and callback signatures against the installed version. diff --git a/assets/skills/ddd-doc-steward/SKILL.md b/assets/skills/ddd-doc-steward/SKILL.md index ddb118d..4e49b1a 100644 --- a/assets/skills/ddd-doc-steward/SKILL.md +++ b/assets/skills/ddd-doc-steward/SKILL.md @@ -1,3 +1,4 @@ +--- name: ddd-doc-steward description: "文档驱动开发（DDD）文档管家：以仓库真实证据为准，盘点 ~/project 与 docs 目录，生成/更新 SSOT 文档（计划→补丁/全文→摘要→一致性检查）。触发：需要让文档与代码/配置/运行方式同步、补齐 guides/integrations/features/architecture/incidents/archive、无法推导时标注【待确认】并给验证路径。" --- @@ -6,21 +7,21 @@ description: "文档驱动开发（DDD）文档管家：以仓库真实证据为 让 `~/project/docs` 成为单一可信来源（SSOT）：先盘点、后计划、再增量写文档，所有事实有证据来源，没有证据就【待确认】。 -## 何时使用 +## When to Use This Skill - 需要为真实仓库建立/维护文档 SSOT，输出「盘点表 → 计划 → 文档补丁/全文 → 变更摘要 → 一致性检查」的固定交付物。 - 新增/改动功能、集成或事故复盘，需要同步更新 docs 下对应目录。 - 需要在 strict 模式下，避免任何臆测，所有关键事实必须给出代码/配置/命令的路径或说明验证方法。 - 只能获取部分信息时，仍需生成最小可落地模板并标注【待确认】。 -## 不适用 / 边界 +## Not For / Boundaries - 与工程无关的纯创作文案。 - 无法提供最小证据集（目录树、README/依赖/配置/路由位置）且不接受【待确认】输出时。 - 涉及密钥明文输出；文档中仅可写占位符与获取方式。 - 未提供 `project_root/docs_root/output_mode` 等输入时，先将自然语言归一到输入 JSON（参见快速参考）。 -## 快速参考 +## Quick Reference 1) 归一化输入（缺省值） ```json @@ -68,19 +69,19 @@ description: "文档驱动开发（DDD）文档管家：以仓库真实证据为 - SHOULD：优先用 git diff/related_paths 聚焦；guides 与 integrations 先行；为每个【待确认】提供验证路径。 - NEVER：编造端口/环境变量/接口字段；输出密钥明文；跳过盘点直接写文档。 -## 示例 +## Examples -### 示例 1：空仓库初始化 +### Example 1: 空仓库初始化 - 输入：`change_type=baseline`，docs 为空。 - 步骤：A 扫描→B 盘点表标记全部缺失→计划新增 docs/README.md、guides/getting-started.md 等→C 生成骨架补丁→D 摘要与检查。 - 验收：输出 patch diff，所有命令标【待确认】或给寻找路径。 -### 示例 2：新增登录功能 +### Example 2: 新增登录功能 - 输入：`change_type=feature`，`scope_hint="auth 登录"`，`use_git_diff=true`。 - 步骤：用 diff 找受影响路由；盘点 features/integrations；计划新增 PRD/Spec，更新 integrations/auth-api；标记 token 过期时间待确认。 - 验收：变更摘要指出新增文档与更新字段；检查清单含鉴权/错误码/验证 curl。 -### 示例 3：无法读取仓库 +### Example 3: 无法读取仓库 - 输入：仅自然语言“帮我做 SSOT 文档”。 - 步骤：先归一化 JSON，声明“无法真实扫描”→生成六类文档模板，全量标【待确认】并列证据缺口与采集命令。 - 验收：输出 full_files；每条待确认可直接行动补证。 @@ -92,10 +93,18 @@ description: "文档驱动开发（DDD）文档管家：以仓库真实证据为 - Q: 目录过大处理不过来？ A: 按 prefer_priority 分批；声明本次范围与剩余批次计划。 -## 维护 +## References + +- `references/index.md`：导航与长文入口。 +- `references/getting_started.md`：DDD 文档管家流程与术语。 +- `references/api.md`：输入/输出规范、目录命名与质量门禁。 +- `references/examples.md`：可套用场景示例。 +- `references/troubleshooting.md`：降级与故障处理。 + +## Maintenance - 来源：提示词库（在线表格入口见 `assets/prompt/README.md`）；元技能 `assets/skills/auto-skill/`；自动化辅助工具 `assets/repos/Skill_Seekers-development`. -- 最后更新：2025-12-20 +- 最后更新：2026-04-28 - 已知限制：依赖用户提供真实证据；大体量仓库需分批；不输出敏感值。 ## 质量门禁（出厂前自检） diff --git a/assets/skills/hummingbot/SKILL.md b/assets/skills/hummingbot/SKILL.md index 171e5cd..8b2e29f 100644 --- a/assets/skills/hummingbot/SKILL.md +++ b/assets/skills/hummingbot/SKILL.md @@ -1,133 +1,136 @@ --- name: hummingbot -description: Hummingbot trading bot framework - automated trading strategies, market making, arbitrage, connectors for crypto exchanges. Use when working with algorithmic trading, crypto trading bots, or exchange integrations. +description: "Hummingbot trading bot framework skill: connector setup, scripts, market making, arbitrage, Gateway DEX operations, headless quickstart, candles, market data provider, and troubleshooting for crypto trading bots." --- -# Hummingbot Skill +# hummingbot Skill -Comprehensive assistance with hummingbot development, generated from official documentation. +Use this skill to operate, configure, or extend Hummingbot bots and Gateway connectors with explicit risk controls and version-aware references. ## When to Use This Skill -This skill should be triggered when: -- Working with hummingbot -- Asking about hummingbot features or APIs -- Implementing hummingbot solutions -- Debugging hummingbot code -- Learning hummingbot best practices +Trigger when any of these applies: +- Running Hummingbot strategies, scripts, or headless bot instances. +- Configuring CEX/DEX connectors, API keys, Gateway routes, or blockchain RPC providers. +- Building market-making, arbitrage, liquidity, or custom script strategies. +- Using candles, order book snapshots, mid-price, volume-for-price, or market data provider APIs. +- Debugging connector failures, Docker/runtime issues, Gateway command errors, or strategy config problems. + +## Not For / Boundaries + +- Not financial advice, profitability guarantees, or unattended live-trading approval. +- Never paste real exchange API keys, private keys, mnemonics, or wallet secrets into prompts, examples, logs, or commits. +- Use paper/sandbox/small-size validation before live capital; many connectors have exchange-specific limits and failure modes. +- Required inputs: Hummingbot version, install mode, connector, trading pair, strategy/script, config file, live/paper mode, and exact logs. +- Gateway and connector schemas evolve; verify against `references/` and the running version before production deployment. ## Quick Reference ### Common Patterns -**Pattern 1:** For example: candles = [CandlesFactory.get_candle(connector=kucoin, trading_pair="ETH-USDT", interval="1m", max_records=100)] - -``` -candles = [CandlesFactory.get_candle(connector=kucoin, - trading_pair="ETH-USDT", interval="1m", max_records=100)] +**Run a headless quickstart** +```bash +bin/hummingbot_quickstart.py --headless -p PASSWORD -f CONFIG_FILE_NAME ``` -**Pattern 2:** Example: - -``` -bin/hummingbot_quickstart.py -p a -f simple_pmm_example_config.py -c conf_simple_pmm_example_config_1.yml +**Run a script config** +```bash +bin/hummingbot_quickstart.py -p PASSWORD -f simple_pmm_example_config.py -c conf_simple_pmm_example_config_1.yml ``` -**Pattern 3:** >>> gateway swap --help usage: gateway swap [-h] [connector] [args ...] positional arguments: connector Connector name/type (e.g., jupiter/router) args Arguments: [base-quote] [side] [amount] options: -h, --help show this help message and exit - -``` ->>> gateway swap --help -usage: gateway swap [-h] [connector] [args ...] - -positional arguments: - connector Connector name/type (e.g., jupiter/router) - args Arguments: [base-quote] [side] [amount] - -options: - -h, --help show this help message and exit +**List Gateway connectors** +```text +gateway list ``` -**Pattern 4:** usage: gateway list [-h] - -``` -usage: gateway list [-h] +**Inspect Gateway swap syntax** +```text +gateway swap --help ``` -**Pattern 5:** Example: - -``` -price = self.market_data_provider.get_price_by_type('binance', 'BTC-USDT', PriceType.MidPrice) +**Get a mid price from the market data provider** +```python +price = self.market_data_provider.get_price_by_type( + "binance", + "BTC-USDT", + PriceType.MidPrice, +) ``` -**Pattern 6:** Example: - -``` -price = self.market_data_provider.get_price_by_volume('binance', 'BTC-USDT', volume: 10000, True) +**Get price by quote volume** +```python +price = self.market_data_provider.get_price_by_volume( + "binance", + "BTC-USDT", + 10000, + True, +) ``` -**Pattern 7:** Example: - -``` -price = self.market_data_provider.get_volume_for_price('binance', 'BTC-USDT', 70000, True) +**Get an order book snapshot** +```python +snapshot = self.market_data_provider.get_order_book_snapshot("binance", "BTC-USDT") ``` -**Pattern 8:** Example: - -``` -price = self.market_data_provider.get_order_book_snapshot('binance', 'BTC-USDT') +**Create a candle feed** +```python +candles = CandlesFactory.get_candle( + connector="kucoin", + trading_pair="ETH-USDT", + interval="1m", + max_records=100, +) ``` -## Reference Files +**Update Docker deployment images** +```bash +docker compose down +docker pull hummingbot/hummingbot:latest +docker compose up -d +``` -This skill includes comprehensive documentation in `references/`: +## Examples -- **advanced.md** - Advanced documentation -- **configuration.md** - Configuration documentation -- **connectors.md** - Connectors documentation -- **development.md** - Development documentation -- **getting_started.md** - Getting Started documentation -- **other.md** - Other documentation -- **strategies.md** - Strategies documentation -- **trading.md** - Trading documentation -- **troubleshooting.md** - Troubleshooting documentation +### Example 1: Headless Bot Smoke Test -Use `view` to read specific reference files when detailed information is needed. +- Input: password, strategy config file, optional script config. +- Steps: + 1. Start with `bin/hummingbot_quickstart.py --headless`. + 2. Confirm logs show connector initialization and strategy start. + 3. Stop the bot and inspect final status before enabling live size. +- Expected output / acceptance: the bot starts from config without interactive prompts and exits cleanly. -## Working with This Skill +### Example 2: Gateway Connector Triage -### For Beginners -Start with the getting_started or tutorials reference files for foundational concepts. +- Input: failing Gateway command and connector name. +- Steps: + 1. Run `gateway list` to verify connector availability. + 2. Run `gateway swap --help` or the specific command help. + 3. Compare arguments with the connector/network schema in `references/trading.md`. +- Expected output / acceptance: failure is classified as missing connector, bad args, RPC/network issue, or credential/config issue. -### For Specific Features -Use the appropriate category reference file (api, guides, etc.) for detailed information. +### Example 3: Strategy Market Data Hook -### For Code Examples -The quick reference section above contains common patterns extracted from the official docs. +- Input: connector `binance`, pair `BTC-USDT`, needed price type. +- Steps: + 1. Use `market_data_provider` for mid price or volume-aware price. + 2. Keep API calls outside tight loops when cached data is sufficient. + 3. Log connector/pair/price source for later debugging. +- Expected output / acceptance: strategy reads a deterministic price source without blocking order logic. -## Resources +## References -### references/ -Organized documentation extracted from official sources. These files contain: -- Detailed explanations -- Code examples with language annotations -- Links to original documentation -- Table of contents for quick navigation +- `references/index.md`: navigation for local Hummingbot references. +- `references/getting_started.md`: install and first-run material. +- `references/configuration.md`: bot and connector configuration. +- `references/connectors.md`: exchange connector catalog and notes. +- `references/strategies.md`: strategy and script material. +- `references/trading.md`: Gateway and trading operations. +- `references/development.md`: development, headless, and release-related notes. +- `references/troubleshooting.md`: common failure modes. -### scripts/ -Add helper scripts here for common automation tasks. +## Maintenance -### assets/ -Add templates, boilerplate, or example projects here. - -## Notes - -- This skill was automatically generated from official documentation -- Reference files preserve the structure and examples from source docs -- Code examples include language detection for better syntax highlighting -- Quick reference patterns are extracted from common usage examples in the docs - -## Updating - -To refresh this skill with updated documentation: -1. Re-run the scraper with the same configuration -2. The skill will be rebuilt with the latest information +- Sources: local `references/` extracted from Hummingbot documentation. +- Last updated: 2026-04-28 +- Known limits: connector support and Gateway schemas change frequently; validate against the installed Hummingbot version. diff --git a/assets/skills/markdown-to-epub/SKILL.md b/assets/skills/markdown-to-epub/SKILL.md index 0d3e890..60abc23 100644 --- a/assets/skills/markdown-to-epub/SKILL.md +++ b/assets/skills/markdown-to-epub/SKILL.md @@ -1,29 +1,34 @@ --- name: markdown-to-epub -description: "将 Markdown 手稿与本地图片资产转换为可校验的 EPUB：修复/归一化图片引用与扩展名，保持标题层级 TOC，并做基础包结构检查。" +description: "Markdown to EPUB build skill: normalize local image references, copy assets, call Calibre ebook-convert, inspect EPUB package structure, and report missing images. Use when turning Markdown manuscripts into reproducible EPUB files." --- # markdown-to-epub Skill -把 Markdown 手稿（含本地图片）稳定构建为 EPUB：规范化图片引用、拷贝资产到可重复的构建目录、调用 Calibre `ebook-convert` 转换，并输出可核查报告。 +Use this skill to build a reproducible EPUB from Markdown manuscripts with local image assets, without mutating the source manuscript. ## When to Use This Skill -触发条件（满足其一即可）： -- 需要把一份（或多份）Markdown 手稿打包交付为 EPUB。 -- 图片引用混乱（URL 编码、路径飘忽、扩展名不可信如 `.bin/.idunno`），需要自动归一化。 -- 需要在转换后做最基本的 EPUB 包结构检查（OPF/NCX/NAV、图片数量等）。 +Trigger when any of these applies: +- Converting one or more Markdown files into an EPUB deliverable. +- Normalizing broken or inconsistent image references before conversion. +- Recovering local assets with unreliable extensions such as `.bin` or `.idunno`. +- Running Calibre `ebook-convert` non-interactively from a repeatable build directory. +- Checking the resulting EPUB archive for OPF, NCX/NAV, and image inclusion. ## Not For / Boundaries -- 不负责生成/改写正文内容（不会修改源手稿，只在构建目录里产出规范化版本）。 -- 不下载远程图片（`http(s)`/`data:` 引用会保持原样）。 -- 不替代真正的排版/校对流程（这里只做可交付构建与结构验证）。 +- Not for authoring, rewriting, proofreading, or typesetting the manuscript body. +- Not for downloading remote `http(s)` or `data:` images; remote references are preserved unless the user supplies local replacements. +- Not a substitute for full EPUB QA in dedicated readers; it performs structural and asset checks only. +- Required inputs: source Markdown path, desired EPUB path/title/authors/language, source root, and any fallback asset map. +- If Calibre is unavailable, fail clearly and provide the install/`--ebook-convert-bin` verification path instead of producing a fake EPUB. -## Quick Start +## Quick Reference -从仓库根目录执行（推荐 `python3`）： +### Common Patterns +**Build a basic EPUB** ```bash python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ --input-md "./book.md" \ @@ -33,28 +38,16 @@ python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ --language "zh-CN" ``` -脚本会创建构建工作区（默认 `build_epub/`），包含： -- `book.normalized.md` -- `assets/`：拷贝后的图片（会按真实文件签名推断扩展名） -- `conversion.log` -- `report.json` - -## 依赖 - -- 需要安装 Calibre，并确保 `ebook-convert` 在 `PATH` 中（或用 `--ebook-convert-bin` 指定路径）。 - -## Missing Asset Recovery - -如果 Markdown 里引用了图片但文件找不到，可以提供一个 JSON 映射表（按「basename」匹配）： - -```json -{ - "missing-file.idunno": "replacement-file.idunno" -} +**Use a custom source root and build directory** +```bash +python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ + --input-md "./manuscript/book.md" \ + --source-root "./manuscript" \ + --build-dir "./build/book-epub" \ + --output-epub "./dist/book.epub" ``` -然后重跑（示例）： - +**Recover missing assets with a fallback map** ```bash python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ --input-md "./book.md" \ @@ -62,31 +55,63 @@ python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ --fallback-map "./fallback-map.json" ``` -## Operational Rules +**Allow unresolved local images but report them** +```bash +python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ + --input-md "./book.md" \ + --output-epub "./book.epub" \ + --no-strict-missing +``` -- 优先使用 `ebook-convert`；缺失时明确报错并快速失败。 -- 源手稿只读；所有输出写入 `build_dir/`。 -- TOC 以标题层级（`h1/h2/h3`）为准。 -- 缺失资产必须显式报告；严格模式下不允许静默跳过。 -- 命令保持非交互式。 +**Point to a non-standard Calibre binary** +```bash +python3 assets/skills/markdown-to-epub/scripts/build_epub.py \ + --input-md "./book.md" \ + --ebook-convert-bin "/opt/calibre/ebook-convert" +``` -## Script Interface +**Inspect the generated package** +```bash +unzip -l ./book.epub | rg 'content.opf|toc.ncx|nav.xhtml|\\.(png|jpg|jpeg|webp|gif)$' +``` -`scripts/build_epub.py` 参数： -- `--input-md`（必选）：源 Markdown 路径 -- `--output-epub`（可选）：输出 EPUB 路径，默认 `.epub` -- `--source-root`（可选）：解析图片引用的根目录，默认使用 Markdown 所在目录 -- `--build-dir`（可选）：构建工作区目录，默认 `/build_epub` -- `--fallback-map`（可选）：JSON 映射（缺失图片 basename → 替换 basename） -- `--title` / `--authors` / `--language`：传给 `ebook-convert` 的元数据 -- `--input-encoding`：输入 Markdown 编码，默认 `utf-8` -- `--strict-missing`：严格模式（有任何本地图片无法解析则失败，默认开启） -- `--no-strict-missing`：关闭严格模式（保留未解析链接，继续转换） -- `--ebook-convert-bin`：`ebook-convert` 可执行文件名/路径，默认 `ebook-convert` +## Examples -## Validation Checklist +### Example 1: Clean Manuscript Build -- 确认 EPUB 文件生成且大小不是「几 KB 的空壳」。 -- 确认 EPUB（zip）内包含 OPF 与 NCX/NAV。 -- 确认 EPUB 内图片数量不低于对手稿的预期。 -- 严格模式下确认 `report.json` 的 `missing_images` 为空。 +- Input: `book.md` with valid local images and metadata title/author/language. +- Steps: + 1. Run the basic build command. + 2. Inspect `build_epub/report.json`. + 3. Check the EPUB zip listing for OPF and navigation files. +- Expected output / acceptance: `book.epub` exists, `missing_images` is empty, and package structure contains OPF plus NCX or NAV. + +### Example 2: Extension Recovery + +- Input: Markdown references `images/cover.idunno`, but the file signature is a PNG. +- Steps: + 1. Run the build script in strict mode. + 2. Confirm copied assets in `build_epub/assets/` use normalized extensions. + 3. Rebuild after fixing any missing file mapping. +- Expected output / acceptance: EPUB includes the normalized image and the report records no unresolved local image. + +### Example 3: Missing Asset Triage + +- Input: manuscript references old file names that no longer exist. +- Steps: + 1. Create a JSON fallback map from missing basenames to replacement basenames. + 2. Re-run with `--fallback-map`. + 3. Keep strict mode enabled so unmapped missing assets fail the build. +- Expected output / acceptance: every missing local image is either resolved by the map or listed in `report.json` for explicit follow-up. + +## References + +- `references/index.md`: navigation, script contract, and validation notes. +- `scripts/build_epub.py`: executable builder used by this skill. +- `agents/openai.yaml`: agent metadata for this skill package. + +## Maintenance + +- Sources: local script implementation and EPUB/Calibre behavior observed by the build report. +- Last updated: 2026-04-28 +- Known limits: structural checks do not guarantee visual fidelity in every EPUB reader; run reader-specific QA for final publication. diff --git a/assets/skills/markdown-to-epub/references/index.md b/assets/skills/markdown-to-epub/references/index.md new file mode 100644 index 0000000..40bf524 --- /dev/null +++ b/assets/skills/markdown-to-epub/references/index.md @@ -0,0 +1,27 @@ +# markdown-to-epub Reference Index + +This directory keeps long-form notes for the Markdown to EPUB skill. The entrypoint stays in `../SKILL.md`; operational behavior is implemented by `../scripts/build_epub.py`. + +## Navigation + +- `../SKILL.md`: triggers, boundaries, quick commands, and examples. +- `../scripts/build_epub.py`: source of truth for CLI arguments, asset normalization, Calibre invocation, and EPUB inspection. +- `../agents/openai.yaml`: optional agent metadata. + +## Script Contract + +- Source Markdown is read-only. +- Build artifacts are written to `--build-dir`. +- Local images are copied into the build workspace and normalized when the file signature proves a better extension. +- Remote `http(s)` and `data:` image references are not downloaded. +- Strict mode fails when local image references cannot be resolved. + +## Verification + +Run the builder, then inspect: + +```bash +unzip -l ./book.epub | rg 'content.opf|toc.ncx|nav.xhtml|\\.(png|jpg|jpeg|webp|gif)$' +``` + +Also inspect `report.json` in the build directory for `missing_images`, copied assets, and conversion status. diff --git a/assets/skills/polymarket/SKILL.md b/assets/skills/polymarket/SKILL.md index 9cdd261..2e0f69e 100644 --- a/assets/skills/polymarket/SKILL.md +++ b/assets/skills/polymarket/SKILL.md @@ -1,233 +1,141 @@ --- name: polymarket -description: Comprehensive Polymarket skill covering prediction markets, API, trading, market data, and real-time WebSocket data streaming. Build applications with Polymarket services, monitor live trades, and integrate market predictions. +description: "Polymarket prediction-market skill: REST/API research, CLOB market data, trading integration boundaries, WebSocket real-time data client, subscriptions, filters, authentication, and LLM-oriented market monitoring." --- -# Polymarket Comprehensive Skill +# polymarket Skill -Complete assistance with Polymarket development - covering the full platform (API, trading, market data) and the real-time data streaming client (WebSocket subscriptions for live market activity). +Use this skill to build Polymarket research, monitoring, market-data, and integration workflows while keeping trading/authentication risks explicit. ## When to Use This Skill -This skill should be triggered when: +Trigger when any of these applies: +- Querying Polymarket markets, events, prices, trades, comments, RFQ, or CLOB data. +- Building prediction-market monitors, dashboards, alerts, or LLM market-analysis tools. +- Using the `@polymarket/real-time-data-client` WebSocket client. +- Subscribing to activity, comments, RFQ, crypto price, `clob_market`, or authenticated `clob_user` topics. +- Debugging subscription filters, authentication payloads, reconnect handling, or message processing. -**Platform & API:** -- Working with Polymarket prediction markets -- Using Polymarket API for market data -- Implementing trading strategies -- Building applications with Polymarket services -- Learning Polymarket best practices +## Not For / Boundaries -**Real-Time Data Streaming:** -- Connecting to Polymarket's WebSocket service -- Building prediction market monitoring tools -- Processing live trades, orders, and market updates -- Monitoring market comments and social reactions -- Tracking RFQ (Request for Quote) activity -- Integrating crypto price feeds +- Not financial advice, market-making advice, or a guarantee of trading profitability. +- Do not expose CLOB API keys, secrets, passphrases, wallet keys, or private signing material. +- Private user streams and trading actions require explicit authentication and security review. +- Required inputs: market/event slug or CLOB market id, topic/type, auth need, filter payload, runtime, and expected output. +- For live trading, verify API terms, regional restrictions, auth scheme, and risk controls before implementation. ## Quick Reference -### Real-Time Data Client Setup +### Common Patterns -**Installation:** +**Install the real-time data client** ```bash npm install @polymarket/real-time-data-client ``` -**Basic Usage:** +**Subscribe to live trades** ```typescript import { RealTimeDataClient } from "@polymarket/real-time-data-client"; -const onMessage = (message: Message): void => { - console.log(message.topic, message.type, message.payload); -}; +const client = new RealTimeDataClient({ + onMessage: (message) => console.log(message.topic, message.type, message.payload), + onConnect: (c) => c.subscribe({ + subscriptions: [{ topic: "activity", type: "trades" }], + }), +}); -const onConnect = (client: RealTimeDataClient): void => { - client.subscribe({ - subscriptions: [{ - topic: "activity", - type: "trades" - }] - }); -}; - -new RealTimeDataClient({ onMessage, onConnect }).connect(); +client.connect(); ``` -### Supported WebSocket Topics - -**1. Activity (`activity`)** -- `trades` - Completed trades -- `orders_matched` - Order matching events -- Filters: `{"event_slug":"string"}` OR `{"market_slug":"string"}` - -**2. Comments (`comments`)** -- `comment_created`, `comment_removed` -- `reaction_created`, `reaction_removed` -- Filters: `{"parentEntityID":number,"parentEntityType":"Event"}` - -**3. RFQ (`rfq`)** -- Request/Quote lifecycle events -- No filters, no auth required - -**4. Crypto Prices (`crypto_prices`, `crypto_prices_chainlink`)** -- `update` - Real-time price feeds -- Filters: `{"symbol":"BTC"}` (optional) - -**5. CLOB User (`clob_user`)** ⚠️ Requires Auth -- `order` - User's order updates -- `trade` - User's trade executions - -**6. CLOB Market (`clob_market`)** -- `price_change` - Price movements -- `agg_orderbook` - Aggregated order book -- `last_trade_price` - Latest prices -- `market_created`, `market_resolved` - -### Authentication for User Data - +**Filter activity to one market slug** ```typescript client.subscribe({ - subscriptions: [{ - topic: "clob_user", - type: "*", - clob_auth: { - key: "your-api-key", - secret: "your-api-secret", - passphrase: "your-passphrase" - } - }] + subscriptions: [{ + topic: "activity", + type: "trades", + filters: "{\"market_slug\":\"btc-above-100k-2024\"}", + }], }); ``` -### Common Use Cases - -**Monitor Specific Market:** +**Subscribe to CLOB market price changes** ```typescript client.subscribe({ - subscriptions: [{ - topic: "activity", - type: "trades", - filters: `{"market_slug":"btc-above-100k-2024"}` - }] + subscriptions: [{ + topic: "clob_market", + type: "price_change", + filters: "[\"100\",\"101\",\"102\"]", + }], }); ``` -**Track Multiple Markets:** +**Subscribe to comments for an event** ```typescript client.subscribe({ - subscriptions: [{ - topic: "clob_market", - type: "price_change", - filters: `["100","101","102"]` - }] + subscriptions: [{ + topic: "comments", + type: "*", + filters: "{\"parentEntityID\":12345,\"parentEntityType\":\"Event\"}", + }], }); ``` -**Monitor Event Comments:** +**Authenticate a user stream** ```typescript client.subscribe({ - subscriptions: [{ - topic: "comments", - type: "*", - filters: `{"parentEntityID":12345,"parentEntityType":"Event"}` - }] + subscriptions: [{ + topic: "clob_user", + type: "*", + clob_auth: { + key: "YOUR_API_KEY", + secret: "YOUR_API_SECRET", + passphrase: "YOUR_PASSPHRASE", + }, + }], }); ``` -## Reference Files +## Examples -This skill includes comprehensive documentation in `references/`: +### Example 1: Public Trade Monitor -**Platform Documentation:** -- **api.md** - Polymarket API documentation -- **getting_started.md** - Getting started guide -- **guides.md** - Development guides -- **learn.md** - Learning resources -- **trading.md** - Trading documentation -- **other.md** - Additional resources +- Input: market slug and alert threshold. +- Steps: + 1. Subscribe to `activity/trades` with a `market_slug` filter. + 2. Normalize messages into timestamp, market, side, price, size. + 3. Alert only when trade size or price movement crosses the configured threshold. +- Expected output / acceptance: real-time public trade stream with no private authentication material. -**Real-Time Client:** -- **README.md** - WebSocket client API and examples -- **llms.md** - LLM integration guide -- **llms-full.md** - Complete LLM documentation +### Example 2: CLOB Price Dashboard -Use `view` to read specific reference files for detailed information. +- Input: CLOB market IDs. +- Steps: + 1. Subscribe to `clob_market/price_change` with a JSON array filter. + 2. Update an in-memory view keyed by market id. + 3. Persist snapshots at a controlled interval rather than every message if volume is high. +- Expected output / acceptance: dashboard shows latest market prices and handles reconnects idempotently. -## Key Features +### Example 3: Authenticated User Order Feed -**Platform Capabilities:** -✅ Prediction market creation and resolution -✅ Trading API (REST & WebSocket) -✅ Market data queries -✅ User portfolio management -✅ Event and market discovery +- Input: CLOB API credentials stored in a secret manager. +- Steps: + 1. Load credentials at runtime without logging them. + 2. Subscribe to `clob_user` order/trade events. + 3. Validate message schema and write audit logs without secrets. +- Expected output / acceptance: private user updates are received and secrets never appear in source control or logs. -**Real-Time Streaming:** -✅ WebSocket-based persistent connections -✅ Topic-based subscriptions -✅ Dynamic subscription management -✅ Filter support for targeted data -✅ User authentication for private data -✅ TypeScript with full type safety -✅ Initial data dumps on connection +## References -## Best Practices +- `references/index.md`: navigation for local Polymarket references. +- `references/api.md`: platform API documentation. +- `references/getting_started.md`: onboarding and setup notes. +- `references/trading.md`: trading and market operations. +- `references/realtime-client.md`: WebSocket real-time data client. +- `references/README.md`: platform overview. +- `references/llms.md` and `references/llms-full.md`: LLM integration material. -### WebSocket Connection Management -- Use `onConnect` callback for subscriptions -- Implement reconnection logic for production -- Clean up with `disconnect()` when done -- Handle authentication errors gracefully +## Maintenance -### Subscription Strategy -- Use wildcards (`"*"`) sparingly -- Apply filters to reduce data volume -- Unsubscribe from unused streams -- Process messages asynchronously - -### Performance -- Consider batching high-frequency data -- Use filters to minimize client processing -- Validate message payloads before use - -## Requirements - -- **Node.js**: 14+ recommended -- **TypeScript**: Optional but recommended -- **Package Manager**: npm or yarn - -## Resources - -### Official Links -- **Polymarket Platform**: https://polymarket.com -- **Real-Time Client Repo**: https://github.com/Polymarket/real-time-data-client -- **API Documentation**: See references/api.md - -### Working with This Skill - -**For Beginners:** -Start with `getting_started.md` for foundational concepts. - -**For API Integration:** -Use `api.md` and `trading.md` for REST API details. - -**For Real-Time Data:** -Use `README.md` for WebSocket client implementation. - -**For LLM Integration:** -Use `llms.md` and `llms-full.md` for AI/ML use cases. - -## Notes - -- Real-Time Client is TypeScript/JavaScript (not Python) -- Some WebSocket topics require authentication -- Use filters to manage message volume effectively -- All timestamps are Unix timestamps -- Market IDs are strings (e.g., "100", "101") -- Platform documentation covers both REST API and WebSocket usage - ---- - -**This comprehensive skill combines Polymarket platform expertise with real-time data streaming capabilities!** +- Sources: local `references/` extracted from Polymarket platform and real-time client documentation. +- Last updated: 2026-04-28 +- Known limits: API auth, market availability, and regional/legal constraints must be verified outside this skill before live trading. diff --git a/assets/skills/postgresql/SKILL.md b/assets/skills/postgresql/SKILL.md index e699a0f..b9319e5 100644 --- a/assets/skills/postgresql/SKILL.md +++ b/assets/skills/postgresql/SKILL.md @@ -1,143 +1,127 @@ --- name: postgresql -description: PostgreSQL database documentation - SQL queries, database design, administration, performance tuning, and advanced features. Use when working with PostgreSQL databases, writing SQL, or managing database systems. +description: "PostgreSQL database skill: psql usage, schema design, SQL queries, DDL/DML, transactions, RETURNING, indexing, EXPLAIN, permissions, libpq connection basics, and performance troubleshooting." --- -# Postgresql Skill +# postgresql Skill -Comprehensive assistance with postgresql development, generated from official documentation. +Use this skill to design, query, debug, and operate PostgreSQL databases with evidence-based SQL and safe production habits. ## When to Use This Skill -This skill should be triggered when: -- Working with postgresql -- Asking about postgresql features or APIs -- Implementing postgresql solutions -- Debugging postgresql code -- Learning postgresql best practices +Trigger when any of these applies: +- Writing or reviewing PostgreSQL SQL, DDL, DML, indexes, views, transactions, or migrations. +- Using `psql`, connection strings, `libpq`, or PostgreSQL client behavior. +- Debugging slow queries, locking, permissions, schema/search path, connection failures, or data-modifying statements. +- Designing normalized schema, constraints, keys, and query plans. +- Explaining PostgreSQL-specific features such as `RETURNING`, CTEs, roles, or `EXPLAIN`. + +## Not For / Boundaries + +- Not for destructive production changes without backups, migration plan, and rollback path. +- Do not expose database passwords, connection strings with secrets, dumps containing private data, or production credentials. +- Prefer explicit transactions and small verified changes for DDL/DML that touches real data. +- Required inputs: PostgreSQL version, schema/table definitions, query, expected result, observed error or plan, data volume, and environment. +- When docs or behavior differ by version, verify against the installed server with `SHOW server_version;`. ## Quick Reference ### Common Patterns -**Pattern 1:** 32.1. Database Connection Control Functions # 32.1.1. Connection Strings 32.1.2. Parameter Key Words The following functions deal with making a connection to a PostgreSQL backend server. An application program can have several backend connections open at one time. (One reason to do that is to access more than one database.) Each connection is represented by a PGconn object, which is obtained from the function PQconnectdb, PQconnectdbParams, or PQsetdbLogin. Note that these functions will always return a non-null object pointer, unless perhaps there is too little memory even to allocate the PGconn object. The PQstatus function should be called to check the return value for a successful connection before queries are sent via the connection object. Warning If untrusted users have access to a database that has not adopted a secure schema usage pattern, begin each session by removing publicly-writable schemas from search_path. One can set parameter key word options to value -csearch_path=. Alternately, one can issue PQexec(conn, "SELECT pg_catalog.set_config('search_path', '', false)") after connecting. This consideration is not specific to libpq; it applies to every interface for executing arbitrary SQL commands. Warning On Unix, forking a process with open libpq connections can lead to unpredictable results because the parent and child processes share the same sockets and operating system resources. For this reason, such usage is not recommended, though doing an exec from the child process to load a new executable is safe. PQconnectdbParams # Makes a new connection to the database server. PGconn *PQconnectdbParams(const char * const *keywords, const char * const *values, int expand_dbname); This function opens a new database connection using the parameters taken from two NULL-terminated arrays. The first, keywords, is defined as an array of strings, each one being a key word. The second, values, gives the value for each key word. Unlike PQsetdbLogin below, the parameter set can be extended without changing the function signature, so use of this function (or its nonblocking analogs PQconnectStartParams and PQconnectPoll) is preferred for new application programming. The currently recognized parameter key words are listed in Section 32.1.2. The passed arrays can be empty to use all default parameters, or can contain one or more parameter settings. They must be matched in length. Processing will stop at the first NULL entry in the keywords array. Also, if the values entry associated with a non-NULL keywords entry is NULL or an empty string, that entry is ignored and processing continues with the next pair of array entries. When expand_dbname is non-zero, the value for the first dbname key word is checked to see if it is a connection string. If so, it is “expanded” into the individual connection parameters extracted from the string. The value is considered to be a connection string, rather than just a database name, if it contains an equal sign (=) or it begins with a URI scheme designator. (More details on connection string formats appear in Section 32.1.1.) Only the first occurrence of dbname is treated in this way; any subsequent dbname parameter is processed as a plain database name. In general the parameter arrays are processed from start to end. If any key word is repeated, the last value (that is not NULL or empty) is used. This rule applies in particular when a key word found in a connection string conflicts with one appearing in the keywords array. Thus, the programmer may determine whether array entries can override or be overridden by values taken from a connection string. Array entries appearing before an expanded dbname entry can be overridden by fields of the connection string, and in turn those fields are overridden by array entries appearing after dbname (but, again, only if those entries supply non-empty values). After processing all the array entries and any expanded connection string, any connection parameters that remain unset are filled with default values. If an unset parameter's corresponding environment variable (see Section 32.15) is set, its value is used. If the environment variable is not set either, then the parameter's built-in default value is used. PQconnectdb # Makes a new connection to the database server. PGconn *PQconnectdb(const char *conninfo); This function opens a new database connection using the parameters taken from the string conninfo. The passed string can be empty to use all default parameters, or it can contain one or more parameter settings separated by whitespace, or it can contain a URI. See Section 32.1.1 for details. PQsetdbLogin # Makes a new connection to the database server. PGconn *PQsetdbLogin(const char *pghost, const char *pgport, const char *pgoptions, const char *pgtty, const char *dbName, const char *login, const char *pwd); This is the predecessor of PQconnectdb with a fixed set of parameters. It has the same functionality except that the missing parameters will always take on default values. Write NULL or an empty string for any one of the fixed parameters that is to be defaulted. If the dbName contains an = sign or has a valid connection URI prefix, it is taken as a conninfo string in exactly the same way as if it had been passed to PQconnectdb, and the remaining parameters are then applied as specified for PQconnectdbParams. pgtty is no longer used and any value passed will be ignored. PQsetdb # Makes a new connection to the database server. PGconn *PQsetdb(char *pghost, char *pgport, char *pgoptions, char *pgtty, char *dbName); This is a macro that calls PQsetdbLogin with null pointers for the login and pwd parameters. It is provided for backward compatibility with very old programs. PQconnectStartParamsPQconnectStartPQconnectPoll # Make a connection to the database server in a nonblocking manner. PGconn *PQconnectStartParams(const char * const *keywords, const char * const *values, int expand_dbname); PGconn *PQconnectStart(const char *conninfo); PostgresPollingStatusType PQconnectPoll(PGconn *conn); These three functions are used to open a connection to a database server such that your application's thread of execution is not blocked on remote I/O whilst doing so. The point of this approach is that the waits for I/O to complete can occur in the application's main loop, rather than down inside PQconnectdbParams or PQconnectdb, and so the application can manage this operation in parallel with other activities. With PQconnectStartParams, the database connection is made using the parameters taken from the keywords and values arrays, and controlled by expand_dbname, as described above for PQconnectdbParams. With PQconnectStart, the database connection is made using the parameters taken from the string conninfo as described above for PQconnectdb. Neither PQconnectStartParams nor PQconnectStart nor PQconnectPoll will block, so long as a number of restrictions are met: The hostaddr parameter must be used appropriately to prevent DNS queries from being made. See the documentation of this parameter in Section 32.1.2 for details. If you call PQtrace, ensure that the stream object into which you trace will not block. You must ensure that the socket is in the appropriate state before calling PQconnectPoll, as described below. To begin a nonblocking connection request, call PQconnectStart or PQconnectStartParams. If the result is null, then libpq has been unable to allocate a new PGconn structure. Otherwise, a valid PGconn pointer is returned (though not yet representing a valid connection to the database). Next call PQstatus(conn). If the result is CONNECTION_BAD, the connection attempt has already failed, typically because of invalid connection parameters. If PQconnectStart or PQconnectStartParams succeeds, the next stage is to poll libpq so that it can proceed with the connection sequence. Use PQsocket(conn) to obtain the descriptor of the socket underlying the database connection. (Caution: do not assume that the socket remains the same across PQconnectPoll calls.) Loop thus: If PQconnectPoll(conn) last returned PGRES_POLLING_READING, wait until the socket is ready to read (as indicated by select(), poll(), or similar system function). Note that PQsocketPoll can help reduce boilerplate by abstracting the setup of select(2) or poll(2) if it is available on your system. Then call PQconnectPoll(conn) again. Conversely, if PQconnectPoll(conn) last returned PGRES_POLLING_WRITING, wait until the socket is ready to write, then call PQconnectPoll(conn) again. On the first iteration, i.e., if you have yet to call PQconnectPoll, behave as if it last returned PGRES_POLLING_WRITING. Continue this loop until PQconnectPoll(conn) returns PGRES_POLLING_FAILED, indicating the connection procedure has failed, or PGRES_POLLING_OK, indicating the connection has been successfully made. At any time during connection, the status of the connection can be checked by calling PQstatus. If this call returns CONNECTION_BAD, then the connection procedure has failed; if the call returns CONNECTION_OK, then the connection is ready. Both of these states are equally detectable from the return value of PQconnectPoll, described above. Other states might also occur during (and only during) an asynchronous connection procedure. These indicate the current stage of the connection procedure and might be useful to provide feedback to the user for example. These statuses are: CONNECTION_STARTED # Waiting for connection to be made. CONNECTION_MADE # Connection OK; waiting to send. CONNECTION_AWAITING_RESPONSE # Waiting for a response from the server. CONNECTION_AUTH_OK # Received authentication; waiting for backend start-up to finish. CONNECTION_SSL_STARTUP # Negotiating SSL encryption. CONNECTION_GSS_STARTUP # Negotiating GSS encryption. CONNECTION_CHECK_WRITABLE # Checking if connection is able to handle write transactions. CONNECTION_CHECK_STANDBY # Checking if connection is to a server in standby mode. CONNECTION_CONSUME # Consuming any remaining response messages on connection. Note that, although these constants will remain (in order to maintain compatibility), an application should never rely upon these occurring in a particular order, or at all, or on the status always being one of these documented values. An application might do something like this: switch(PQstatus(conn)) { case CONNECTION_STARTED: feedback = "Connecting..."; break; case CONNECTION_MADE: feedback = "Connected to server..."; break; . . . default: feedback = "Connecting..."; } The connect_timeout connection parameter is ignored when using PQconnectPoll; it is the application's responsibility to decide whether an excessive amount of time has elapsed. Otherwise, PQconnectStart followed by a PQconnectPoll loop is equivalent to PQconnectdb. Note that when PQconnectStart or PQconnectStartParams returns a non-null pointer, you must call PQfinish when you are finished with it, in order to dispose of the structure and any associated memory blocks. This must be done even if the connection attempt fails or is abandoned. PQsocketPoll # Poll a connection's underlying socket descriptor retrieved with PQsocket. The primary use of this function is iterating through the connection sequence described in the documentation of PQconnectStartParams. typedef int64_t pg_usec_time_t; int PQsocketPoll(int sock, int forRead, int forWrite, pg_usec_time_t end_time); This function performs polling of a file descriptor, optionally with a timeout. If forRead is nonzero, the function will terminate when the socket is ready for reading. If forWrite is nonzero, the function will terminate when the socket is ready for writing. The timeout is specified by end_time, which is the time to stop waiting expressed as a number of microseconds since the Unix epoch (that is, time_t times 1 million). Timeout is infinite if end_time is -1. Timeout is immediate (no blocking) if end_time is 0 (or indeed, any time before now). Timeout values can be calculated conveniently by adding the desired number of microseconds to the result of PQgetCurrentTimeUSec. Note that the underlying system calls may have less than microsecond precision, so that the actual delay may be imprecise. The function returns a value greater than 0 if the specified condition is met, 0 if a timeout occurred, or -1 if an error occurred. The error can be retrieved by checking the errno(3) value. In the event both forRead and forWrite are zero, the function immediately returns a timeout indication. PQsocketPoll is implemented using either poll(2) or select(2), depending on platform. See POLLIN and POLLOUT from poll(2), or readfds and writefds from select(2), for more information. PQconndefaults # Returns the default connection options. PQconninfoOption *PQconndefaults(void); typedef struct { char *keyword; /* The keyword of the option */ char *envvar; /* Fallback environment variable name */ char *compiled; /* Fallback compiled in default value */ char *val; /* Option's current value, or NULL */ char *label; /* Label for field in connect dialog */ char *dispchar; /* Indicates how to display this field in a connect dialog. Values are: "" Display entered value as is "*" Password field - hide value "D" Debug option - don't show by default */ int dispsize; /* Field size in characters for dialog */ } PQconninfoOption; Returns a connection options array. This can be used to determine all possible PQconnectdb options and their current default values. The return value points to an array of PQconninfoOption structures, which ends with an entry having a null keyword pointer. The null pointer is returned if memory could not be allocated. Note that the current default values (val fields) will depend on environment variables and other context. A missing or invalid service file will be silently ignored. Callers must treat the connection options data as read-only. After processing the options array, free it by passing it to PQconninfoFree. If this is not done, a small amount of memory is leaked for each call to PQconndefaults. PQconninfo # Returns the connection options used by a live connection. PQconninfoOption *PQconninfo(PGconn *conn); Returns a connection options array. This can be used to determine all possible PQconnectdb options and the values that were used to connect to the server. The return value points to an array of PQconninfoOption structures, which ends with an entry having a null keyword pointer. All notes above for PQconndefaults also apply to the result of PQconninfo. PQconninfoParse # Returns parsed connection options from the provided connection string. PQconninfoOption *PQconninfoParse(const char *conninfo, char **errmsg); Parses a connection string and returns the resulting options as an array; or returns NULL if there is a problem with the connection string. This function can be used to extract the PQconnectdb options in the provided connection string. The return value points to an array of PQconninfoOption structures, which ends with an entry having a null keyword pointer. All legal options will be present in the result array, but the PQconninfoOption for any option not present in the connection string will have val set to NULL; default values are not inserted. If errmsg is not NULL, then *errmsg is set to NULL on success, else to a malloc'd error string explaining the problem. (It is also possible for *errmsg to be set to NULL and the function to return NULL; this indicates an out-of-memory condition.) After processing the options array, free it by passing it to PQconninfoFree. If this is not done, some memory is leaked for each call to PQconninfoParse. Conversely, if an error occurs and errmsg is not NULL, be sure to free the error string using PQfreemem. PQfinish # Closes the connection to the server. Also frees memory used by the PGconn object. void PQfinish(PGconn *conn); Note that even if the server connection attempt fails (as indicated by PQstatus), the application should call PQfinish to free the memory used by the PGconn object. The PGconn pointer must not be used again after PQfinish has been called. PQreset # Resets the communication channel to the server. void PQreset(PGconn *conn); This function will close the connection to the server and attempt to establish a new connection, using all the same parameters previously used. This might be useful for error recovery if a working connection is lost. PQresetStartPQresetPoll # Reset the communication channel to the server, in a nonblocking manner. int PQresetStart(PGconn *conn); PostgresPollingStatusType PQresetPoll(PGconn *conn); These functions will close the connection to the server and attempt to establish a new connection, using all the same parameters previously used. This can be useful for error recovery if a working connection is lost. They differ from PQreset (above) in that they act in a nonblocking manner. These functions suffer from the same restrictions as PQconnectStartParams, PQconnectStart and PQconnectPoll. To initiate a connection reset, call PQresetStart. If it returns 0, the reset has failed. If it returns 1, poll the reset using PQresetPoll in exactly the same way as you would create the connection using PQconnectPoll. PQpingParams # PQpingParams reports the status of the server. It accepts connection parameters identical to those of PQconnectdbParams, described above. It is not necessary to supply correct user name, password, or database name values to obtain the server status; however, if incorrect values are provided, the server will log a failed connection attempt. PGPing PQpingParams(const char * const *keywords, const char * const *values, int expand_dbname); The function returns one of the following values: PQPING_OK # The server is running and appears to be accepting connections. PQPING_REJECT # The server is running but is in a state that disallows connections (startup, shutdown, or crash recovery). PQPING_NO_RESPONSE # The server could not be contacted. This might indicate that the server is not running, or that there is something wrong with the given connection parameters (for example, wrong port number), or that there is a network connectivity problem (for example, a firewall blocking the connection request). PQPING_NO_ATTEMPT # No attempt was made to contact the server, because the supplied parameters were obviously incorrect or there was some client-side problem (for example, out of memory). PQping # PQping reports the status of the server. It accepts connection parameters identical to those of PQconnectdb, described above. It is not necessary to supply correct user name, password, or database name values to obtain the server status; however, if incorrect values are provided, the server will log a failed connection attempt. PGPing PQping(const char *conninfo); The return values are the same as for PQpingParams. PQsetSSLKeyPassHook_OpenSSL # PQsetSSLKeyPassHook_OpenSSL lets an application override libpq's default handling of encrypted client certificate key files using sslpassword or interactive prompting. void PQsetSSLKeyPassHook_OpenSSL(PQsslKeyPassHook_OpenSSL_type hook); The application passes a pointer to a callback function with signature: int callback_fn(char *buf, int size, PGconn *conn); which libpq will then call instead of its default PQdefaultSSLKeyPassHook_OpenSSL handler. The callback should determine the password for the key and copy it to result-buffer buf of size size. The string in buf must be null-terminated. The callback must return the length of the password stored in buf excluding the null terminator. On failure, the callback should set buf[0] = '\0' and return 0. See PQdefaultSSLKeyPassHook_OpenSSL in libpq's source code for an example. If the user specified an explicit key location, its path will be in conn->sslkey when the callback is invoked. This will be empty if the default key path is being used. For keys that are engine specifiers, it is up to engine implementations whether they use the OpenSSL password callback or define their own handling. The app callback may choose to delegate unhandled cases to PQdefaultSSLKeyPassHook_OpenSSL, or call it first and try something else if it returns 0, or completely override it. The callback must not escape normal flow control with exceptions, longjmp(...), etc. It must return normally. PQgetSSLKeyPassHook_OpenSSL # PQgetSSLKeyPassHook_OpenSSL returns the current client certificate key password hook, or NULL if none has been set. PQsslKeyPassHook_OpenSSL_type PQgetSSLKeyPassHook_OpenSSL(void); 32.1.1. Connection Strings # Several libpq functions parse a user-specified string to obtain connection parameters. There are two accepted formats for these strings: plain keyword/value strings and URIs. URIs generally follow RFC 3986, except that multi-host connection strings are allowed as further described below. 32.1.1.1. Keyword/Value Connection Strings # In the keyword/value format, each parameter setting is in the form keyword = value, with space(s) between settings. Spaces around a setting's equal sign are optional. To write an empty value, or a value containing spaces, surround it with single quotes, for example keyword = 'a value'. Single quotes and backslashes within a value must be escaped with a backslash, i.e., \' and \\. Example: host=localhost port=5432 dbname=mydb connect_timeout=10 The recognized parameter key words are listed in Section 32.1.2. 32.1.1.2. Connection URIs # The general form for a connection URI is: postgresql://[userspec@][hostspec][/dbname][?paramspec] where userspec is: user[:password] and hostspec is: [host][:port][,...] and paramspec is: name=value[&...] The URI scheme designator can be either postgresql:// or postgres://. Each of the remaining URI parts is optional. The following examples illustrate valid URI syntax: postgresql:// postgresql://localhost postgresql://localhost:5433 postgresql://localhost/mydb postgresql://user@localhost postgresql://user:secret@localhost postgresql://other@localhost/otherdb?connect_timeout=10&application_name=myapp postgresql://host1:123,host2:456/somedb?target_session_attrs=any&application_name=myapp Values that would normally appear in the hierarchical part of the URI can alternatively be given as named parameters. For example: postgresql:///mydb?host=localhost&port=5433 All named parameters must match key words listed in Section 32.1.2, except that for compatibility with JDBC connection URIs, instances of ssl=true are translated into sslmode=require. The connection URI needs to be encoded with percent-encoding if it includes symbols with special meaning in any of its parts. Here is an example where the equal sign (=) is replaced with %3D and the space character with %20: postgresql://user@localhost:5433/mydb?options=-c%20synchronous_commit%3Doff The host part may be either a host name or an IP address. To specify an IPv6 address, enclose it in square brackets: postgresql://[2001:db8::1234]/database The host part is interpreted as described for the parameter host. In particular, a Unix-domain socket connection is chosen if the host part is either empty or looks like an absolute path name, otherwise a TCP/IP connection is initiated. Note, however, that the slash is a reserved character in the hierarchical part of the URI. So, to specify a non-standard Unix-domain socket directory, either omit the host part of the URI and specify the host as a named parameter, or percent-encode the path in the host part of the URI: postgresql:///dbname?host=/var/lib/postgresql postgresql://%2Fvar%2Flib%2Fpostgresql/dbname It is possible to specify multiple host components, each with an optional port component, in a single URI. A URI of the form postgresql://host1:port1,host2:port2,host3:port3/ is equivalent to a connection string of the form host=host1,host2,host3 port=port1,port2,port3. As further described below, each host will be tried in turn until a connection is successfully established. 32.1.1.3. Specifying Multiple Hosts # It is possible to specify multiple hosts to connect to, so that they are tried in the given order. In the Keyword/Value format, the host, hostaddr, and port options accept comma-separated lists of values. The same number of elements must be given in each option that is specified, such that e.g., the first hostaddr corresponds to the first host name, the second hostaddr corresponds to the second host name, and so forth. As an exception, if only one port is specified, it applies to all the hosts. In the connection URI format, you can list multiple host:port pairs separated by commas in the host component of the URI. In either format, a single host name can translate to multiple network addresses. A common example of this is a host that has both an IPv4 and an IPv6 address. When multiple hosts are specified, or when a single host name is translated to multiple addresses, all the hosts and addresses will be tried in order, until one succeeds. If none of the hosts can be reached, the connection fails. If a connection is established successfully, but authentication fails, the remaining hosts in the list are not tried. If a password file is used, you can have different passwords for different hosts. All the other connection options are the same for every host in the list; it is not possible to e.g., specify different usernames for different hosts. 32.1.2. Parameter Key Words # The currently recognized parameter key words are: host # Name of host to connect to. If a host name looks like an absolute path name, it specifies Unix-domain communication rather than TCP/IP communication; the value is the name of the directory in which the socket file is stored. (On Unix, an absolute path name begins with a slash. On Windows, paths starting with drive letters are also recognized.) If the host name starts with @, it is taken as a Unix-domain socket in the abstract namespace (currently supported on Linux and Windows). The default behavior when host is not specified, or is empty, is to connect to a Unix-domain socket in /tmp (or whatever socket directory was specified when PostgreSQL was built). On Windows, the default is to connect to localhost. A comma-separated list of host names is also accepted, in which case each host name in the list is tried in order; an empty item in the list selects the default behavior as explained above. See Section 32.1.1.3 for details. hostaddr # Numeric IP address of host to connect to. This should be in the standard IPv4 address format, e.g., 172.28.40.9. If your machine supports IPv6, you can also use those addresses. TCP/IP communication is always used when a nonempty string is specified for this parameter. If this parameter is not specified, the value of host will be looked up to find the corresponding IP address — or, if host specifies an IP address, that value will be used directly. Using hostaddr allows the application to avoid a host name look-up, which might be important in applications with time constraints. However, a host name is required for GSSAPI or SSPI authentication methods, as well as for verify-full SSL certificate verification. The following rules are used: If host is specified without hostaddr, a host name lookup occurs. (When using PQconnectPoll, the lookup occurs when PQconnectPoll first considers this host name, and it may cause PQconnectPoll to block for a significant amount of time.) If hostaddr is specified without host, the value for hostaddr gives the server network address. The connection attempt will fail if the authentication method requires a host name. If both host and hostaddr are specified, the value for hostaddr gives the server network address. The value for host is ignored unless the authentication method requires it, in which case it will be used as the host name. Note that authentication is likely to fail if host is not the name of the server at network address hostaddr. Also, when both host and hostaddr are specified, host is used to identify the connection in a password file (see Section 32.16). A comma-separated list of hostaddr values is also accepted, in which case each host in the list is tried in order. An empty item in the list causes the corresponding host name to be used, or the default host name if that is empty as well. See Section 32.1.1.3 for details. Without either a host name or host address, libpq will connect using a local Unix-domain socket; or on Windows, it will attempt to connect to localhost. port # Port number to connect to at the server host, or socket file name extension for Unix-domain connections. If multiple hosts were given in the host or hostaddr parameters, this parameter may specify a comma-separated list of ports of the same length as the host list, or it may specify a single port number to be used for all hosts. An empty string, or an empty item in a comma-separated list, specifies the default port number established when PostgreSQL was built. dbname # The database name. Defaults to be the same as the user name. In certain contexts, the value is checked for extended formats; see Section 32.1.1 for more details on those. user # PostgreSQL user name to connect as. Defaults to be the same as the operating system name of the user running the application. password # Password to be used if the server demands password authentication. passfile # Specifies the name of the file used to store passwords (see Section 32.16). Defaults to ~/.pgpass, or %APPDATA%\postgresql\pgpass.conf on Microsoft Windows. (No error is reported if this file does not exist.) require_auth # Specifies the authentication method that the client requires from the server. If the server does not use the required method to authenticate the client, or if the authentication handshake is not fully completed by the server, the connection will fail. A comma-separated list of methods may also be provided, of which the server must use exactly one in order for the connection to succeed. By default, any authentication method is accepted, and the server is free to skip authentication altogether. Methods may be negated with the addition of a ! prefix, in which case the server must not attempt the listed method; any other method is accepted, and the server is free not to authenticate the client at all. If a comma-separated list is provided, the server may not attempt any of the listed negated methods. Negated and non-negated forms may not be combined in the same setting. As a final special case, the none method requires the server not to use an authentication challenge. (It may also be negated, to require some form of authentication.) The following methods may be specified: password The server must request plaintext password authentication. md5 The server must request MD5 hashed password authentication. Warning Support for MD5-encrypted passwords is deprecated and will be removed in a future release of PostgreSQL. Refer to Section 20.5 for details about migrating to another password type. gss The server must either request a Kerberos handshake via GSSAPI or establish a GSS-encrypted channel (see also gssencmode). sspi The server must request Windows SSPI authentication. scram-sha-256 The server must successfully complete a SCRAM-SHA-256 authentication exchange with the client. oauth The server must request an OAuth bearer token from the client. none The server must not prompt the client for an authentication exchange. (This does not prohibit client certificate authentication via TLS, nor GSS authentication via its encrypted transport.) channel_binding # This option controls the client's use of channel binding. A setting of require means that the connection must employ channel binding, prefer means that the client will choose channel binding if available, and disable prevents the use of channel binding. The default is prefer if PostgreSQL is compiled with SSL support; otherwise the default is disable. Channel binding is a method for the server to authenticate itself to the client. It is only supported over SSL connections with PostgreSQL 11 or later servers using the SCRAM authentication method. connect_timeout # Maximum time to wait while connecting, in seconds (write as a decimal integer, e.g., 10). Zero, negative, or not specified means wait indefinitely. This timeout applies separately to each host name or IP address. For example, if you specify two hosts and connect_timeout is 5, each host will time out if no connection is made within 5 seconds, so the total time spent waiting for a connection might be up to 10 seconds. client_encoding # This sets the client_encoding configuration parameter for this connection. In addition to the values accepted by the corresponding server option, you can use auto to determine the right encoding from the current locale in the client (LC_CTYPE environment variable on Unix systems). options # Specifies command-line options to send to the server at connection start. For example, setting this to -c geqo=off or --geqo=off sets the session's value of the geqo parameter to off. Spaces within this string are considered to separate command-line arguments, unless escaped with a backslash (\); write \\ to represent a literal backslash. For a detailed discussion of the available options, consult Chapter 19. application_name # Specifies a value for the application_name configuration parameter. fallback_application_name # Specifies a fallback value for the application_name configuration parameter. This value will be used if no value has been given for application_name via a connection parameter or the PGAPPNAME environment variable. Specifying a fallback name is useful in generic utility programs that wish to set a default application name but allow it to be overridden by the user. keepalives # Controls whether client-side TCP keepalives are used. The default value is 1, meaning on, but you can change this to 0, meaning off, if keepalives are not wanted. This parameter is ignored for connections made via a Unix-domain socket. keepalives_idle # Controls the number of seconds of inactivity after which TCP should send a keepalive message to the server. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket, or if keepalives are disabled. It is only supported on systems where TCP_KEEPIDLE or an equivalent socket option is available, and on Windows; on other systems, it has no effect. keepalives_interval # Controls the number of seconds after which a TCP keepalive message that is not acknowledged by the server should be retransmitted. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket, or if keepalives are disabled. It is only supported on systems where TCP_KEEPINTVL or an equivalent socket option is available, and on Windows; on other systems, it has no effect. keepalives_count # Controls the number of TCP keepalives that can be lost before the client's connection to the server is considered dead. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket, or if keepalives are disabled. It is only supported on systems where TCP_KEEPCNT or an equivalent socket option is available; on other systems, it has no effect. tcp_user_timeout # Controls the number of milliseconds that transmitted data may remain unacknowledged before a connection is forcibly closed. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket. It is only supported on systems where TCP_USER_TIMEOUT is available; on other systems, it has no effect. replication # This option determines whether the connection should use the replication protocol instead of the normal protocol. This is what PostgreSQL replication connections as well as tools such as pg_basebackup use internally, but it can also be used by third-party applications. For a description of the replication protocol, consult Section 54.4. The following values, which are case-insensitive, are supported: true, on, yes, 1 The connection goes into physical replication mode. database The connection goes into logical replication mode, connecting to the database specified in the dbname parameter. false, off, no, 0 The connection is a regular one, which is the default behavior. In physical or logical replication mode, only the simple query protocol can be used. gssencmode # This option determines whether or with what priority a secure GSS TCP/IP connection will be negotiated with the server. There are three modes: disable only try a non-GSSAPI-encrypted connection prefer (default) if there are GSSAPI credentials present (i.e., in a credentials cache), first try a GSSAPI-encrypted connection; if that fails or there are no credentials, try a non-GSSAPI-encrypted connection. This is the default when PostgreSQL has been compiled with GSSAPI support. require only try a GSSAPI-encrypted connection gssencmode is ignored for Unix domain socket communication. If PostgreSQL is compiled without GSSAPI support, using the require option will cause an error, while prefer will be accepted but libpq will not actually attempt a GSSAPI-encrypted connection. sslmode # This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server. There are six modes: disable only try a non-SSL connection allow first try a non-SSL connection; if that fails, try an SSL connection prefer (default) first try an SSL connection; if that fails, try a non-SSL connection require only try an SSL connection. If a root CA file is present, verify the certificate in the same way as if verify-ca was specified verify-ca only try an SSL connection, and verify that the server certificate is issued by a trusted certificate authority (CA) verify-full only try an SSL connection, verify that the server certificate is issued by a trusted CA and that the requested server host name matches that in the certificate See Section 32.19 for a detailed description of how these options work. sslmode is ignored for Unix domain socket communication. If PostgreSQL is compiled without SSL support, using options require, verify-ca, or verify-full will cause an error, while options allow and prefer will be accepted but libpq will not actually attempt an SSL connection. Note that if GSSAPI encryption is possible, that will be used in preference to SSL encryption, regardless of the value of sslmode. To force use of SSL encryption in an environment that has working GSSAPI infrastructure (such as a Kerberos server), also set gssencmode to disable. requiressl # This option is deprecated in favor of the sslmode setting. If set to 1, an SSL connection to the server is required (this is equivalent to sslmode require). libpq will then refuse to connect if the server does not accept an SSL connection. If set to 0 (default), libpq will negotiate the connection type with the server (equivalent to sslmode prefer). This option is only available if PostgreSQL is compiled with SSL support. sslnegotiation # This option controls how SSL encryption is negotiated with the server, if SSL is used. In the default postgres mode, the client first asks the server if SSL is supported. In direct mode, the client starts the standard SSL handshake directly after establishing the TCP/IP connection. Traditional PostgreSQL protocol negotiation is the most flexible with different server configurations. If the server is known to support direct SSL connections then the latter requires one fewer round trip reducing connection latency and also allows the use of protocol agnostic SSL network tools. The direct SSL option was introduced in PostgreSQL version 17. postgres perform PostgreSQL protocol negotiation. This is the default if the option is not provided. direct start SSL handshake directly after establishing the TCP/IP connection. This is only allowed with sslmode=require or higher, because the weaker settings could lead to unintended fallback to plaintext authentication when the server does not support direct SSL handshake. sslcompression # If set to 1, data sent over SSL connections will be compressed. If set to 0, compression will be disabled. The default is 0. This parameter is ignored if a connection without SSL is made. SSL compression is nowadays considered insecure and its use is no longer recommended. OpenSSL 1.1.0 disabled compression by default, and many operating system distributions disabled it in prior versions as well, so setting this parameter to on will not have any effect if the server does not accept compression. PostgreSQL 14 disabled compression completely in the backend. If security is not a primary concern, compression can improve throughput if the network is the bottleneck. Disabling compression can improve response time and throughput if CPU performance is the limiting factor. sslcert # This parameter specifies the file name of the client SSL certificate, replacing the default ~/.postgresql/postgresql.crt. This parameter is ignored if an SSL connection is not made. sslkey # This parameter specifies the location for the secret key used for the client certificate. It can either specify a file name that will be used instead of the default ~/.postgresql/postgresql.key, or it can specify a key obtained from an external “engine” (engines are OpenSSL loadable modules). An external engine specification should consist of a colon-separated engine name and an engine-specific key identifier. This parameter is ignored if an SSL connection is not made. sslkeylogfile # This parameter specifies the location where libpq will log keys used in this SSL context. This is useful for debugging PostgreSQL protocol interactions or client connections using network inspection tools like Wireshark. This parameter is ignored if an SSL connection is not made, or if LibreSSL is used (LibreSSL does not support key logging). Keys are logged using the NSS format. Warning Key logging will expose potentially sensitive information in the keylog file. Keylog files should be handled with the same care as sslkey files. sslpassword # This parameter specifies the password for the secret key specified in sslkey, allowing client certificate private keys to be stored in encrypted form on disk even when interactive passphrase input is not practical. Specifying this parameter with any non-empty value suppresses the Enter PEM pass phrase: prompt that OpenSSL will emit by default when an encrypted client certificate key is provided to libpq. If the key is not encrypted this parameter is ignored. The parameter has no effect on keys specified by OpenSSL engines unless the engine uses the OpenSSL password callback mechanism for prompts. There is no environment variable equivalent to this option, and no facility for looking it up in .pgpass. It can be used in a service file connection definition. Users with more sophisticated uses should consider using OpenSSL engines and tools like PKCS#11 or USB crypto offload devices. sslcertmode # This option determines whether a client certificate may be sent to the server, and whether the server is required to request one. There are three modes: disable A client certificate is never sent, even if one is available (default location or provided via sslcert). allow (default) A certificate may be sent, if the server requests one and the client has one to send. require The server must request a certificate. The connection will fail if the client does not send a certificate and the server successfully authenticates the client anyway. Note sslcertmode=require doesn't add any additional security, since there is no guarantee that the server is validating the certificate correctly; PostgreSQL servers generally request TLS certificates from clients whether they validate them or not. The option may be useful when troubleshooting more complicated TLS setups. sslrootcert # This parameter specifies the name of a file containing SSL certificate authority (CA) certificate(s). If the file exists, the server's certificate will be verified to be signed by one of these authorities. The default is ~/.postgresql/root.crt. The special value system may be specified instead, in which case the trusted CA roots from the SSL implementation will be loaded. The exact locations of these root certificates differ by SSL implementation and platform. For OpenSSL in particular, the locations may be further modified by the SSL_CERT_DIR and SSL_CERT_FILE environment variables. Note When using sslrootcert=system, the default sslmode is changed to verify-full, and any weaker setting will result in an error. In most cases it is trivial for anyone to obtain a certificate trusted by the system for a hostname they control, rendering verify-ca and all weaker modes useless. The magic system value will take precedence over a local certificate file with the same name. If for some reason you find yourself in this situation, use an alternative path like sslrootcert=./system instead. sslcrl # This parameter specifies the file name of the SSL server certificate revocation list (CRL). Certificates listed in this file, if it exists, will be rejected while attempting to authenticate the server's certificate. If neither sslcrl nor sslcrldir is set, this setting is taken as ~/.postgresql/root.crl. sslcrldir # This parameter specifies the directory name of the SSL server certificate revocation list (CRL). Certificates listed in the files in this directory, if it exists, will be rejected while attempting to authenticate the server's certificate. The directory needs to be prepared with the OpenSSL command openssl rehash or c_rehash. See its documentation for details. Both sslcrl and sslcrldir can be specified together. sslsni # If set to 1 (default), libpq sets the TLS extension “Server Name Indication” (SNI) on SSL-enabled connections. By setting this parameter to 0, this is turned off. The Server Name Indication can be used by SSL-aware proxies to route connections without having to decrypt the SSL stream. (Note that unless the proxy is aware of the PostgreSQL protocol handshake this would require setting sslnegotiation to direct.) However, SNI makes the destination host name appear in cleartext in the network traffic, so it might be undesirable in some cases. requirepeer # This parameter specifies the operating-system user name of the server, for example requirepeer=postgres. When making a Unix-domain socket connection, if this parameter is set, the client checks at the beginning of the connection that the server process is running under the specified user name; if it is not, the connection is aborted with an error. This parameter can be used to provide server authentication similar to that available with SSL certificates on TCP/IP connections. (Note that if the Unix-domain socket is in /tmp or another publicly writable location, any user could start a server listening there. Use this parameter to ensure that you are connected to a server run by a trusted user.) This option is only supported on platforms for which the peer authentication method is implemented; see Section 20.9. ssl_min_protocol_version # This parameter specifies the minimum SSL/TLS protocol version to allow for the connection. Valid values are TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3. The supported protocols depend on the version of OpenSSL used, older versions not supporting the most modern protocol versions. If not specified, the default is TLSv1.2, which satisfies industry best practices as of this writing. ssl_max_protocol_version # This parameter specifies the maximum SSL/TLS protocol version to allow for the connection. Valid values are TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3. The supported protocols depend on the version of OpenSSL used, older versions not supporting the most modern protocol versions. If not set, this parameter is ignored and the connection will use the maximum bound defined by the backend, if set. Setting the maximum protocol version is mainly useful for testing or if some component has issues working with a newer protocol. min_protocol_version # Specifies the minimum protocol version to allow for the connection. The default is to allow any version of the PostgreSQL protocol supported by libpq, which currently means 3.0. If the server does not support at least this protocol version the connection will be closed. The current supported values are 3.0, 3.2, and latest. The latest value is equivalent to the latest protocol version supported by the libpq version being used, which is currently 3.2. max_protocol_version # Specifies the protocol version to request from the server. The default is to use version 3.0 of the PostgreSQL protocol, unless the connection string specifies a feature that relies on a higher protocol version, in which case the latest version supported by libpq is used. If the server does not support the protocol version requested by the client, the connection is automatically downgraded to a lower minor protocol version that the server supports. After the connection attempt has completed you can use PQprotocolVersion to find out which exact protocol version was negotiated. The current supported values are 3.0, 3.2, and latest. The latest value is equivalent to the latest protocol version supported by the libpq version being used, which is currently 3.2. krbsrvname # Kerberos service name to use when authenticating with GSSAPI. This must match the service name specified in the server configuration for Kerberos authentication to succeed. (See also Section 20.6.) The default value is normally postgres, but that can be changed when building PostgreSQL via the --with-krb-srvnam option of configure. In most environments, this parameter never needs to be changed. Some Kerberos implementations might require a different service name, such as Microsoft Active Directory which requires the service name to be in upper case (POSTGRES). gsslib # GSS library to use for GSSAPI authentication. Currently this is disregarded except on Windows builds that include both GSSAPI and SSPI support. In that case, set this to gssapi to cause libpq to use the GSSAPI library for authentication instead of the default SSPI. gssdelegation # Forward (delegate) GSS credentials to the server. The default is 0 which means credentials will not be forwarded to the server. Set this to 1 to have credentials forwarded when possible. scram_client_key # The base64-encoded SCRAM client key. This can be used by foreign-data wrappers or similar middleware to enable pass-through SCRAM authentication. See Section F.38.1.10 for one such implementation. It is not meant to be specified directly by users or client applications. scram_server_key # The base64-encoded SCRAM server key. This can be used by foreign-data wrappers or similar middleware to enable pass-through SCRAM authentication. See Section F.38.1.10 for one such implementation. It is not meant to be specified directly by users or client applications. service # Service name to use for additional parameters. It specifies a service name in pg_service.conf that holds additional connection parameters. This allows applications to specify only a service name so connection parameters can be centrally maintained. See Section 32.17. target_session_attrs # This option determines whether the session must have certain properties to be acceptable. It's typically used in combination with multiple host names to select the first acceptable alternative among several hosts. There are six modes: any (default) any successful connection is acceptable read-write session must accept read-write transactions by default (that is, the server must not be in hot standby mode and the default_transaction_read_only parameter must be off) read-only session must not accept read-write transactions by default (the converse) primary server must not be in hot standby mode standby server must be in hot standby mode prefer-standby first try to find a standby server, but if none of the listed hosts is a standby server, try again in any mode load_balance_hosts # Controls the order in which the client tries to connect to the available hosts and addresses. Once a connection attempt is successful no other hosts and addresses will be tried. This parameter is typically used in combination with multiple host names or a DNS record that returns multiple IPs. This parameter can be used in combination with target_session_attrs to, for example, load balance over standby servers only. Once successfully connected, subsequent queries on the returned connection will all be sent to the same server. There are currently two modes: disable (default) No load balancing across hosts is performed. Hosts are tried in the order in which they are provided and addresses are tried in the order they are received from DNS or a hosts file. random Hosts and addresses are tried in random order. This value is mostly useful when opening multiple connections at the same time, possibly from different machines. This way connections can be load balanced across multiple PostgreSQL servers. While random load balancing, due to its random nature, will almost never result in a completely uniform distribution, it statistically gets quite close. One important aspect here is that this algorithm uses two levels of random choices: First the hosts will be resolved in random order. Then secondly, before resolving the next host, all resolved addresses for the current host will be tried in random order. This behaviour can skew the amount of connections each node gets greatly in certain cases, for instance when some hosts resolve to more addresses than others. But such a skew can also be used on purpose, e.g. to increase the number of connections a larger server gets by providing its hostname multiple times in the host string. When using this value it's recommended to also configure a reasonable value for connect_timeout. Because then, if one of the nodes that are used for load balancing is not responding, a new node will be tried. oauth_issuer # The HTTPS URL of a trusted issuer to contact if the server requests an OAuth token for the connection. This parameter is required for all OAuth connections; it should exactly match the issuer setting in the server's HBA configuration. As part of the standard authentication handshake, libpq will ask the server for a discovery document: a URL providing a set of OAuth configuration parameters. The server must provide a URL that is directly constructed from the components of the oauth_issuer, and this value must exactly match the issuer identifier that is declared in the discovery document itself, or the connection will fail. This is required to prevent a class of "mix-up attacks" on OAuth clients. You may also explicitly set oauth_issuer to the /.well-known/ URI used for OAuth discovery. In this case, if the server asks for a different URL, the connection will fail, but a custom OAuth flow may be able to speed up the standard handshake by using previously cached tokens. (In this case, it is recommended that oauth_scope be set as well, since the client will not have a chance to ask the server for a correct scope setting, and the default scopes for a token may not be sufficient to connect.) libpq currently supports the following well-known endpoints: /.well-known/openid-configuration /.well-known/oauth-authorization-server Warning Issuers are highly privileged during the OAuth connection handshake. As a rule of thumb, if you would not trust the operator of a URL to handle access to your servers, or to impersonate you directly, that URL should not be trusted as an oauth_issuer. oauth_client_id # An OAuth 2.0 client identifier, as issued by the authorization server. If the PostgreSQL server requests an OAuth token for the connection (and if no custom OAuth hook is installed to provide one), then this parameter must be set; otherwise, the connection will fail. oauth_client_secret # The client password, if any, to use when contacting the OAuth authorization server. Whether this parameter is required or not is determined by the OAuth provider; "public" clients generally do not use a secret, whereas "confidential" clients generally do. oauth_scope # The scope of the access request sent to the authorization server, specified as a (possibly empty) space-separated list of OAuth scope identifiers. This parameter is optional and intended for advanced usage. Usually the client will obtain appropriate scope settings from the PostgreSQL server. If this parameter is used, the server's requested scope list will be ignored. This can prevent a less-trusted server from requesting inappropriate access scopes from the end user. However, if the client's scope setting does not contain the server's required scopes, the server is likely to reject the issued token, and the connection will fail. The meaning of an empty scope list is provider-dependent. An OAuth authorization server may choose to issue a token with "default scope", whatever that happens to be, or it may reject the token request entirely. - -``` -PGconn +**Connect with psql** +```bash +psql "postgresql://user@localhost:5432/mydb" ``` -**Pattern 2:** 32.1.1. Connection Strings # Several libpq functions parse a user-specified string to obtain connection parameters. There are two accepted formats for these strings: plain keyword/value strings and URIs. URIs generally follow RFC 3986, except that multi-host connection strings are allowed as further described below. 32.1.1.1. Keyword/Value Connection Strings # In the keyword/value format, each parameter setting is in the form keyword = value, with space(s) between settings. Spaces around a setting's equal sign are optional. To write an empty value, or a value containing spaces, surround it with single quotes, for example keyword = 'a value'. Single quotes and backslashes within a value must be escaped with a backslash, i.e., \' and \\. Example: host=localhost port=5432 dbname=mydb connect_timeout=10 The recognized parameter key words are listed in Section 32.1.2. 32.1.1.2. Connection URIs # The general form for a connection URI is: postgresql://[userspec@][hostspec][/dbname][?paramspec] where userspec is: user[:password] and hostspec is: [host][:port][,...] and paramspec is: name=value[&...] The URI scheme designator can be either postgresql:// or postgres://. Each of the remaining URI parts is optional. The following examples illustrate valid URI syntax: postgresql:// postgresql://localhost postgresql://localhost:5433 postgresql://localhost/mydb postgresql://user@localhost postgresql://user:secret@localhost postgresql://other@localhost/otherdb?connect_timeout=10&application_name=myapp postgresql://host1:123,host2:456/somedb?target_session_attrs=any&application_name=myapp Values that would normally appear in the hierarchical part of the URI can alternatively be given as named parameters. For example: postgresql:///mydb?host=localhost&port=5433 All named parameters must match key words listed in Section 32.1.2, except that for compatibility with JDBC connection URIs, instances of ssl=true are translated into sslmode=require. The connection URI needs to be encoded with percent-encoding if it includes symbols with special meaning in any of its parts. Here is an example where the equal sign (=) is replaced with %3D and the space character with %20: postgresql://user@localhost:5433/mydb?options=-c%20synchronous_commit%3Doff The host part may be either a host name or an IP address. To specify an IPv6 address, enclose it in square brackets: postgresql://[2001:db8::1234]/database The host part is interpreted as described for the parameter host. In particular, a Unix-domain socket connection is chosen if the host part is either empty or looks like an absolute path name, otherwise a TCP/IP connection is initiated. Note, however, that the slash is a reserved character in the hierarchical part of the URI. So, to specify a non-standard Unix-domain socket directory, either omit the host part of the URI and specify the host as a named parameter, or percent-encode the path in the host part of the URI: postgresql:///dbname?host=/var/lib/postgresql postgresql://%2Fvar%2Flib%2Fpostgresql/dbname It is possible to specify multiple host components, each with an optional port component, in a single URI. A URI of the form postgresql://host1:port1,host2:port2,host3:port3/ is equivalent to a connection string of the form host=host1,host2,host3 port=port1,port2,port3. As further described below, each host will be tried in turn until a connection is successfully established. 32.1.1.3. Specifying Multiple Hosts # It is possible to specify multiple hosts to connect to, so that they are tried in the given order. In the Keyword/Value format, the host, hostaddr, and port options accept comma-separated lists of values. The same number of elements must be given in each option that is specified, such that e.g., the first hostaddr corresponds to the first host name, the second hostaddr corresponds to the second host name, and so forth. As an exception, if only one port is specified, it applies to all the hosts. In the connection URI format, you can list multiple host:port pairs separated by commas in the host component of the URI. In either format, a single host name can translate to multiple network addresses. A common example of this is a host that has both an IPv4 and an IPv6 address. When multiple hosts are specified, or when a single host name is translated to multiple addresses, all the hosts and addresses will be tried in order, until one succeeds. If none of the hosts can be reached, the connection fails. If a connection is established successfully, but authentication fails, the remaining hosts in the list are not tried. If a password file is used, you can have different passwords for different hosts. All the other connection options are the same for every host in the list; it is not possible to e.g., specify different usernames for different hosts. - -``` -keyword +**Check server version** +```sql +SHOW server_version; ``` -**Pattern 3:** Example: - -``` -host=localhost port=5432 dbname=mydb connect_timeout=10 +**Create a table with a generated id** +```sql +CREATE TABLE users ( + id serial PRIMARY KEY, + firstname text NOT NULL, + lastname text NOT NULL +); ``` -**Pattern 4:** 32.1.1.2. Connection URIs # The general form for a connection URI is: postgresql://[userspec@][hostspec][/dbname][?paramspec] where userspec is: user[:password] and hostspec is: [host][:port][,...] and paramspec is: name=value[&...] The URI scheme designator can be either postgresql:// or postgres://. Each of the remaining URI parts is optional. The following examples illustrate valid URI syntax: postgresql:// postgresql://localhost postgresql://localhost:5433 postgresql://localhost/mydb postgresql://user@localhost postgresql://user:secret@localhost postgresql://other@localhost/otherdb?connect_timeout=10&application_name=myapp postgresql://host1:123,host2:456/somedb?target_session_attrs=any&application_name=myapp Values that would normally appear in the hierarchical part of the URI can alternatively be given as named parameters. For example: postgresql:///mydb?host=localhost&port=5433 All named parameters must match key words listed in Section 32.1.2, except that for compatibility with JDBC connection URIs, instances of ssl=true are translated into sslmode=require. The connection URI needs to be encoded with percent-encoding if it includes symbols with special meaning in any of its parts. Here is an example where the equal sign (=) is replaced with %3D and the space character with %20: postgresql://user@localhost:5433/mydb?options=-c%20synchronous_commit%3Doff The host part may be either a host name or an IP address. To specify an IPv6 address, enclose it in square brackets: postgresql://[2001:db8::1234]/database The host part is interpreted as described for the parameter host. In particular, a Unix-domain socket connection is chosen if the host part is either empty or looks like an absolute path name, otherwise a TCP/IP connection is initiated. Note, however, that the slash is a reserved character in the hierarchical part of the URI. So, to specify a non-standard Unix-domain socket directory, either omit the host part of the URI and specify the host as a named parameter, or percent-encode the path in the host part of the URI: postgresql:///dbname?host=/var/lib/postgresql postgresql://%2Fvar%2Flib%2Fpostgresql/dbname It is possible to specify multiple host components, each with an optional port component, in a single URI. A URI of the form postgresql://host1:port1,host2:port2,host3:port3/ is equivalent to a connection string of the form host=host1,host2,host3 port=port1,port2,port3. As further described below, each host will be tried in turn until a connection is successfully established. - -``` -postgresql://[userspec@][hostspec][/dbname][?paramspec] - -where userspec is: - -user[:password] - -and hostspec is: - -[host][:port][,...] - -and paramspec is: - -name=value[&...] +**Insert and return generated data** +```sql +INSERT INTO users (firstname, lastname) +VALUES ('Joe', 'Cool') +RETURNING id; ``` -**Pattern 5:** 21.5. Predefined Roles # PostgreSQL provides a set of predefined roles that provide access to certain, commonly needed, privileged capabilities and information. Administrators (including roles that have the CREATEROLE privilege) can GRANT these roles to users and/or other roles in their environment, providing those users with access to the specified capabilities and information. For example: GRANT pg_signal_backend TO admin_user; Warning Care should be taken when granting these roles to ensure they are only used where needed and with the understanding that these roles grant access to privileged information. The predefined roles are described below. Note that the specific permissions for each of the roles may change in the future as additional capabilities are added. Administrators should monitor the release notes for changes. pg_checkpoint # pg_checkpoint allows executing the CHECKPOINT command. pg_create_subscription # pg_create_subscription allows users with CREATE permission on the database to issue CREATE SUBSCRIPTION. pg_database_owner # pg_database_owner always has exactly one implicit member: the current database owner. It cannot be granted membership in any role, and no role can be granted membership in pg_database_owner. However, like any other role, it can own objects and receive grants of access privileges. Consequently, once pg_database_owner has rights within a template database, each owner of a database instantiated from that template will possess those rights. Initially, this role owns the public schema, so each database owner governs local use of that schema. pg_maintain # pg_maintain allows executing VACUUM, ANALYZE, CLUSTER, REFRESH MATERIALIZED VIEW, REINDEX, and LOCK TABLE on all relations, as if having MAINTAIN rights on those objects. pg_monitorpg_read_all_settingspg_read_all_statspg_stat_scan_tables # These roles are intended to allow administrators to easily configure a role for the purpose of monitoring the database server. They grant a set of common privileges allowing the role to read various useful configuration settings, statistics, and other system information normally restricted to superusers. pg_monitor allows reading/executing various monitoring views and functions. This role is a member of pg_read_all_settings, pg_read_all_stats and pg_stat_scan_tables. pg_read_all_settings allows reading all configuration variables, even those normally visible only to superusers. pg_read_all_stats allows reading all pg_stat_* views and use various statistics related extensions, even those normally visible only to superusers. pg_stat_scan_tables allows executing monitoring functions that may take ACCESS SHARE locks on tables, potentially for a long time (e.g., pgrowlocks(text) in the pgrowlocks extension). pg_read_all_datapg_write_all_data # pg_read_all_data allows reading all data (tables, views, sequences), as if having SELECT rights on those objects and USAGE rights on all schemas. This role does not bypass row-level security (RLS) policies. If RLS is being used, an administrator may wish to set BYPASSRLS on roles which this role is granted to. pg_write_all_data allows writing all data (tables, views, sequences), as if having INSERT, UPDATE, and DELETE rights on those objects and USAGE rights on all schemas. This role does not bypass row-level security (RLS) policies. If RLS is being used, an administrator may wish to set BYPASSRLS on roles which this role is granted to. pg_read_server_filespg_write_server_filespg_execute_server_program # These roles are intended to allow administrators to have trusted, but non-superuser, roles which are able to access files and run programs on the database server as the user the database runs as. They bypass all database-level permission checks when accessing files directly and they could be used to gain superuser-level access. Therefore, great care should be taken when granting these roles to users. pg_read_server_files allows reading files from any location the database can access on the server using COPY and other file-access functions. pg_write_server_files allows writing to files in any location the database can access on the server using COPY and other file-access functions. pg_execute_server_program allows executing programs on the database server as the user the database runs as using COPY and other functions which allow executing a server-side program. pg_signal_autovacuum_worker # pg_signal_autovacuum_worker allows signaling autovacuum workers to cancel the current table's vacuum or terminate its session. See Section 9.28.2. pg_signal_backend # pg_signal_backend allows signaling another backend to cancel a query or terminate its session. Note that this role does not permit signaling backends owned by a superuser. See Section 9.28.2. pg_use_reserved_connections # pg_use_reserved_connections allows use of connection slots reserved via reserved_connections. - -``` -CREATEROLE +**Update and inspect changed rows** +```sql +UPDATE products +SET price = price * 1.10 +WHERE price <= 99.99 +RETURNING name, price AS new_price; ``` -**Pattern 6:** 6.4. Returning Data from Modified Rows # Sometimes it is useful to obtain data from modified rows while they are being manipulated. The INSERT, UPDATE, DELETE, and MERGE commands all have an optional RETURNING clause that supports this. Use of RETURNING avoids performing an extra database query to collect the data, and is especially valuable when it would otherwise be difficult to identify the modified rows reliably. The allowed contents of a RETURNING clause are the same as a SELECT command's output list (see Section 7.3). It can contain column names of the command's target table, or value expressions using those columns. A common shorthand is RETURNING *, which selects all columns of the target table in order. In an INSERT, the default data available to RETURNING is the row as it was inserted. This is not so useful in trivial inserts, since it would just repeat the data provided by the client. But it can be very handy when relying on computed default values. For example, when using a serial column to provide unique identifiers, RETURNING can return the ID assigned to a new row: CREATE TABLE users (firstname text, lastname text, id serial primary key); INSERT INTO users (firstname, lastname) VALUES ('Joe', 'Cool') RETURNING id; The RETURNING clause is also very useful with INSERT ... SELECT. In an UPDATE, the default data available to RETURNING is the new content of the modified row. For example: UPDATE products SET price = price * 1.10 WHERE price <= 99.99 RETURNING name, price AS new_price; In a DELETE, the default data available to RETURNING is the content of the deleted row. For example: DELETE FROM products WHERE obsoletion_date = 'today' RETURNING *; In a MERGE, the default data available to RETURNING is the content of the source row plus the content of the inserted, updated, or deleted target row. Since it is quite common for the source and target to have many of the same columns, specifying RETURNING * can lead to a lot of duplicated columns, so it is often more useful to qualify it so as to return just the source or target row. For example: MERGE INTO products p USING new_products n ON p.product_no = n.product_no WHEN NOT MATCHED THEN INSERT VALUES (n.product_no, n.name, n.price) WHEN MATCHED THEN UPDATE SET name = n.name, price = n.price RETURNING p.*; In each of these commands, it is also possible to explicitly return the old and new content of the modified row. For example: UPDATE products SET price = price * 1.10 WHERE price <= 99.99 RETURNING name, old.price AS old_price, new.price AS new_price, new.price - old.price AS price_change; In this example, writing new.price is the same as just writing price, but it makes the meaning clearer. This syntax for returning old and new values is available in INSERT, UPDATE, DELETE, and MERGE commands, but typically old values will be NULL for an INSERT, and new values will be NULL for a DELETE. However, there are situations where it can still be useful for those commands. For example, in an INSERT with an ON CONFLICT DO UPDATE clause, the old values will be non-NULL for conflicting rows. Similarly, if a DELETE is turned into an UPDATE by a rewrite rule, the new values may be non-NULL. If there are triggers (Chapter 37) on the target table, the data available to RETURNING is the row as modified by the triggers. Thus, inspecting columns computed by triggers is another common use-case for RETURNING. - -``` -INSERT +**Create an index** +```sql +CREATE INDEX test1_id_index ON test1 (id); ``` -**Pattern 7:** In an UPDATE, the default data available to RETURNING is the new content of the modified row. For example: - -``` -UPDATE +**Read a query plan** +```sql +EXPLAIN (ANALYZE, BUFFERS) +SELECT * FROM users WHERE lastname = 'Cool'; ``` -**Pattern 8:** In a DELETE, the default data available to RETURNING is the content of the deleted row. For example: - -``` -DELETE +**Use a transaction guard for data changes** +```sql +BEGIN; +UPDATE users SET lastname = 'Checked' WHERE id = 1 RETURNING *; +ROLLBACK; ``` -### Example Code Patterns - -**Example 1** (javascript): -```javascript -PGconn *PQconnectdbParams(const char * const *keywords, - const char * const *values, - int expand_dbname); +**Harden search path for untrusted schemas** +```sql +SELECT pg_catalog.set_config('search_path', '', false); ``` -**Example 2** (javascript): -```javascript -PGconn *PQconnectdb(const char *conninfo); -``` +## Examples -## Reference Files +### Example 1: Add a Safe Migration -This skill includes comprehensive documentation in `references/`: +- Input: new table requirement and target PostgreSQL version. +- Steps: + 1. Write `CREATE TABLE` with primary key, `NOT NULL`, and constraints. + 2. Add indexes only for known query predicates. + 3. Run migration in a transaction in staging and verify with `\d`. +- Expected output / acceptance: migration is reversible or has a rollback plan and schema matches expected constraints. -- **getting_started.md** - Getting Started documentation -- **sql.md** - Sql documentation +### Example 2: Debug a Slow Query -Use `view` to read specific reference files when detailed information is needed. +- Input: SQL query, table definitions, estimated data volume, and current indexes. +- Steps: + 1. Run `EXPLAIN (ANALYZE, BUFFERS)` on a safe environment. + 2. Identify sequential scans, bad row estimates, sort/hash spills, or missing predicates. + 3. Propose the smallest index/query rewrite and re-run the plan. +- Expected output / acceptance: before/after plan evidence shows lower runtime or IO for the target workload. -## Working with This Skill +### Example 3: Validate Data-Modifying SQL -### For Beginners -Start with the getting_started or tutorials reference files for foundational concepts. +- Input: `UPDATE` or `DELETE` statement for production data. +- Steps: + 1. Convert the predicate to a `SELECT count(*)` and inspect sample rows. + 2. Run inside `BEGIN` with `RETURNING` on a staging or transaction-guarded session. + 3. Commit only after row count and returned rows match the expected blast radius. +- Expected output / acceptance: affected rows are known before commit and the rollback path is explicit. -### For Specific Features -Use the appropriate category reference file (api, guides, etc.) for detailed information. +## References -### For Code Examples -The quick reference section above contains common patterns extracted from the official docs. +- `references/index.md`: navigation for local PostgreSQL references. +- `references/getting_started.md`: psql, tutorial SQL, build/install, and basics. +- `references/sql.md`: SQL language, DDL/DML, `RETURNING`, monitoring, and advanced examples. -## Resources +## Maintenance -### references/ -Organized documentation extracted from official sources. These files contain: -- Detailed explanations -- Code examples with language annotations -- Links to original documentation -- Table of contents for quick navigation - -### scripts/ -Add helper scripts here for common automation tasks. - -### assets/ -Add templates, boilerplate, or example projects here. - -## Notes - -- This skill was automatically generated from official documentation -- Reference files preserve the structure and examples from source docs -- Code examples include language detection for better syntax highlighting -- Quick reference patterns are extracted from common usage examples in the docs - -## Updating - -To refresh this skill with updated documentation: -1. Re-run the scraper with the same configuration -2. The skill will be rebuilt with the latest information +- Sources: local `references/` extracted from PostgreSQL documentation. +- Last updated: 2026-04-28 +- Known limits: server behavior depends on PostgreSQL version, extensions, configuration, statistics, and workload; validate with live plans. diff --git a/assets/skills/proxychains/SKILL.md b/assets/skills/proxychains/SKILL.md index 156c813..2d02c19 100644 --- a/assets/skills/proxychains/SKILL.md +++ b/assets/skills/proxychains/SKILL.md @@ -1,291 +1,118 @@ --- name: proxychains -description: Auto-detect network issues and force proxy usage with proxychains4. Use this skill when encountering connection timeouts, DNS failures, or blocked network access. Default proxy is http://127.0.0.1:9910 +description: "proxychains4 network fallback skill: detect timeouts, DNS failures, blocked GitHub/PyPI/npm/curl/git access, configure http://127.0.0.1:9910, and retry commands through proxychains." --- -# Proxychains Network Proxy Skill +# proxychains Skill -**自动使用代理解决网络问题** - 当遇到网络连接失败、超时或访问受限时，自动使用 proxychains4 强制走代理。 +Use this skill when network commands fail or known slow/blocked sources need a transparent TCP proxy through `proxychains4`. ## When to Use This Skill -**触发条件（自动应用）:** -- ❌ Connection timeout errors -- ❌ DNS resolution failures -- ❌ "Connection refused" errors -- ❌ HTTP 403/451 blocked access -- ❌ GitHub/PyPI/npm 等国外资源访问失败 -- ❌ curl/wget/git/pip 等命令失败 +Trigger when any of these applies: +- `curl`, `wget`, `git`, `pip`, `npm`, `yarn`, `docker pull`, or `ssh` fails with timeout, DNS, reset, refused, 403, or 451-like access errors. +- Accessing GitHub, raw GitHub content, PyPI, npm registry, Docker registries, or other known slow foreign resources. +- Configuring a local proxychains profile that points to `http://127.0.0.1:9910`. +- Diagnosing whether failure is proxy service down, proxychains config wrong, unsupported UDP/static binary behavior, or upstream blocked. -**Claude 应该：** -1. 检测到网络错误后，自动在命令前添加 `proxychains4` -2. 使用默认代理配置：`http://127.0.0.1:9910` -3. 如果代理失败，提示用户检查代理服务状态 +## Not For / Boundaries + +- Not for bypassing legal, account, regional, or enterprise security restrictions. +- Not useful for UDP-only traffic or statically linked binaries that proxychains cannot intercept. +- Do not proxy commands that should remain on a trusted private network without checking routing and policy. +- Required inputs: failed command, full error text, target domain, proxy endpoint, OS package name, and whether the proxy service is running. +- If proxychains retry also fails, inspect proxy service health before repeating the same command. ## Quick Reference -### 基本用法模式 +### Common Patterns +**Install on Debian/Ubuntu** ```bash -# ❌ 原命令失败 -curl https://github.com/user/repo +sudo apt install proxychains4 +``` -# ✅ 使用代理重试 +**Create user config for the local proxy** +```bash +mkdir -p ~/.proxychains +cp assets/skills/proxychains/references/proxychains.conf ~/.proxychains/proxychains.conf +``` + +**Test the proxy** +```bash +proxychains4 curl https://ipinfo.io/json +``` + +**Retry a failed curl** +```bash proxychains4 curl https://github.com/user/repo ``` -### 常见场景自动应用 - -**场景 1: Git 操作失败** +**Retry a failed git clone** ```bash -# 原命令 -git clone https://github.com/user/repo.git - -# 自动改为 proxychains4 git clone https://github.com/user/repo.git ``` -**场景 2: Python pip 安装失败** +**Retry a failed pip install** ```bash -# 原命令 -pip install requests - -# 自动改为 proxychains4 pip install requests ``` -**场景 3: npm/yarn 安装失败** +**Retry a failed npm install** ```bash -# 原命令 -npm install package-name - -# 自动改为 proxychains4 npm install package-name ``` -**场景 4: wget/curl 下载失败** +**Retry a failed Docker pull** ```bash -# 原命令 -wget https://example.com/file.tar.gz - -# 自动改为 -proxychains4 wget https://example.com/file.tar.gz -``` - -**场景 5: Docker 拉取镜像失败** -```bash -# 原命令 -docker pull image:tag - -# 自动改为 proxychains4 docker pull image:tag ``` -**场景 6: SSH 连接失败** +**Inspect effective config** ```bash -# 原命令 -ssh user@remote-host - -# 自动改为 -proxychains4 ssh user@remote-host +proxychains4 -f ~/.proxychains/proxychains.conf curl https://example.com ``` -## 配置详情 +## Examples -### 默认代理配置 +### Example 1: GitHub Clone Timeout -**本地代理地址：** `http://127.0.0.1:9910` +- Input: `git clone` fails with a connection timeout. +- Steps: + 1. Retry with `proxychains4 git clone ...`. + 2. If it fails, run `proxychains4 curl https://github.com` to isolate Git vs network. + 3. Check local proxy service on `127.0.0.1:9910`. +- Expected output / acceptance: clone succeeds or failure is classified as proxy service/config/upstream issue. -**配置文件位置：** -- `~/.proxychains/proxychains.conf` (推荐) -- `/etc/proxychains.conf` (系统级) +### Example 2: PyPI Install Fails -### 快速配置脚本 +- Input: `pip install` cannot reach PyPI. +- Steps: + 1. Retry with `proxychains4 pip install `. + 2. If DNS still fails, confirm `proxy_dns` is enabled in config. + 3. Avoid writing credentials into `pip.conf` unless explicitly required. +- Expected output / acceptance: package install succeeds through proxy or produces a new non-network error. -创建用户级配置（自动使用 127.0.0.1:9910）： +### Example 3: Unsupported Traffic Diagnosis -```bash -mkdir -p ~/.proxychains -cat > ~/.proxychains/proxychains.conf << 'EOF' -# Proxychains configuration -strict_chain -proxy_dns -remote_dns_subnet 224 -tcp_read_time_out 15000 -tcp_connect_time_out 8000 +- Input: proxychains retry does not affect a command. +- Steps: + 1. Confirm the command uses TCP and dynamic linking. + 2. Test the proxy with a known-good `curl`. + 3. Use application-native proxy options if proxychains cannot intercept it. +- Expected output / acceptance: unsupported mode is identified instead of repeatedly retrying. -[ProxyList] -http 127.0.0.1 9910 -EOF -``` +## References -### 环境变量方式（临时使用） +- `references/index.md`: navigation for local proxychains references. +- `references/proxychains.conf`: default config targeting `127.0.0.1:9910`. +- `references/quick-reference.md`: command snippets and common scenarios. +- `references/setup-guide.md`: installation and setup notes. +- `references/troubleshooting.md`: failure classification. +- `scripts/setup-proxy.sh`: helper script for local setup. -```bash -# 设置代理环境变量 -export PROXYCHAINS_SOCKS5_HOST=127.0.0.1 -export PROXYCHAINS_SOCKS5_PORT=9910 +## Maintenance -# 使用 -proxychains4 curl https://github.com -``` - -## 工作原理 - -Proxychains 通过 LD_PRELOAD 机制拦截程序的网络调用： - -1. **拦截系统调用**：Hook socket 相关的 libc 函数 -2. **重定向连接**：将所有 TCP 连接重定向到代理服务器 -3. **DNS 代理**：通过代理服务器解析域名 -4. **透明代理**：应用程序无需修改 - -**重要限制：** -- 只支持动态链接的程序（statically linked 程序不支持） -- 只支持 TCP 连接（UDP 不支持） -- 部分使用特殊网络库的程序可能不兼容 - -## Claude 自动化规则 - -### 规则 1: 网络错误自动重试 - -``` -IF 命令返回网络相关错误（timeout, connection refused, DNS failure） -THEN 自动使用 proxychains4 重试该命令 -``` - -### 规则 2: 已知慢速源强制代理 - -``` -IF 访问以下域名/服务： - - github.com - - raw.githubusercontent.com - - pypi.org - - npmjs.org - - registry.npmjs.org - - docker.io - - gcr.io -THEN 直接使用 proxychains4（不等待失败） -``` - -### 规则 3: 失败提示 - -``` -IF proxychains4 命令也失败 -THEN 提示用户： - 1. 检查代理服务是否运行（127.0.0.1:9910） - 2. 检查 proxychains 配置文件 - 3. 尝试其他代理地址 -``` - -## 故障排除 - -### 检查代理服务状态 - -```bash -# 测试代理是否可用 -curl -x http://127.0.0.1:9910 https://www.google.com - -# 检查端口是否监听 -netstat -tunlp | grep 9910 -# 或 -ss -tunlp | grep 9910 -``` - -### 验证 proxychains 配置 - -```bash -# 测试配置是否正确 -proxychains4 curl https://ipinfo.io/json -# 应该显示代理服务器的 IP，而不是本机 IP -``` - -### 常见错误处理 - -**错误 1: "proxychains: command not found"** -```bash -# 安装 proxychains4 -sudo apt install proxychains4 # Debian/Ubuntu -sudo yum install proxychains-ng # CentOS/RHEL -``` - -**错误 2: "timeout"** -```bash -# 检查代理地址配置是否正确 -cat ~/.proxychains/proxychains.conf | grep -A 2 "\[ProxyList\]" - -# 修改超时时间（在配置文件中） -tcp_connect_time_out 15000 -tcp_read_time_out 30000 -``` - -**错误 3: "can't read configuration file"** -```bash -# 创建配置文件 -mkdir -p ~/.proxychains -cp /etc/proxychains.conf ~/.proxychains/proxychains.conf -# 然后编辑配置 -``` - -## 高级用法 - -### 多代理链 - -```conf -# ~/.proxychains/proxychains.conf -strict_chain # 按顺序使用所有代理 - -[ProxyList] -http 127.0.0.1 9910 -socks5 127.0.0.1 1080 -``` - -### 动态代理链 - -```conf -dynamic_chain # 自动跳过死代理 - -[ProxyList] -http 127.0.0.1 9910 -http 127.0.0.1 8080 -socks5 127.0.0.1 1080 -``` - -### 随机代理链 - -```conf -random_chain -chain_len = 2 # 随机选择 2 个代理 - -[ProxyList] -http 127.0.0.1 9910 -socks5 127.0.0.1 1080 -socks5 127.0.0.1 1081 -``` - -### 自定义 DNS 服务器 - -```bash -# 使用自定义 DNS 通过代理解析 -export PROXY_DNS_SERVER=8.8.8.8 -proxychains4 curl https://example.com -``` - -## 参考资源 - -- **官方仓库**: https://github.com/haad/proxychains -- **配置文件**: `references/proxychains.conf` (完整示例) -- **故障排除**: `references/troubleshooting.md` -- **命令速查**: `references/quick-reference.md` - -## 总结 - -**记住这些原则：** -1. ❌ **遇到网络错误** → ✅ 自动加上 `proxychains4` -2. 🌐 **访问国外资源** → ✅ 主动使用 `proxychains4` -3. 🔧 **代理也失败** → ✅ 提示用户检查代理服务 - -**默认代理:** `http://127.0.0.1:9910` - ---- - -**这个技能让 Claude 在遇到网络问题时自动使用代理，无需用户手动干预！** +- Sources: local proxychains reference files and config template. +- Last updated: 2026-04-28 +- Known limits: proxychains relies on `LD_PRELOAD` style interception and is not universal; prefer native proxy settings when available. diff --git a/assets/skills/snapdom/SKILL.md b/assets/skills/snapdom/SKILL.md index 62cfe40..8008b02 100644 --- a/assets/skills/snapdom/SKILL.md +++ b/assets/skills/snapdom/SKILL.md @@ -1,274 +1,122 @@ --- name: snapdom -description: snapDOM is a fast, accurate DOM-to-image capture tool that converts HTML elements into scalable SVG images. Use for capturing HTML elements, converting DOM to images (SVG, PNG, JPG, WebP), preserving styles, fonts, and pseudo-elements. +description: "snapDOM DOM-to-image skill: capture HTML elements as SVG/PNG/JPG/WebP/canvas/blob, preserve styles/fonts/pseudo-elements, use scaling/exclusion/CORS proxy options, and compare screenshot output." --- -# SnapDOM Skill +# snapdom Skill -Fast, dependency-free DOM-to-image capture library for converting HTML elements into scalable SVG or raster image formats. +Use this skill to capture browser DOM elements into image outputs with snapDOM while preserving styling and controlling export options. ## When to Use This Skill -Use SnapDOM when you need to: -- Convert HTML elements to images (SVG, PNG, JPG, WebP) -- Capture styled DOM with pseudo-elements and shadows -- Export elements with embedded fonts and icons -- Create screenshots with custom dimensions or scaling -- Handle CORS-blocked resources using proxy fallback -- Implement custom rendering pipelines with plugins -- Optimize performance on large or complex elements +Trigger when any of these applies: +- Exporting a DOM element to SVG, PNG, JPG, WebP, Canvas, or Blob. +- Capturing styled UI with fonts, pseudo-elements, shadows, transforms, or Shadow DOM. +- Building screenshot/export/download features in web apps. +- Debugging missing assets, CORS-blocked images, scaling, tight bounds, or excluded controls. +- Comparing snapDOM with `html2canvas` or browser screenshot workflows. -## Key Features +## Not For / Boundaries -### Universal Export Options -- **SVG** - Scalable vector format, embeds all styles -- **PNG, JPG, WebP** - Raster formats with configurable quality -- **Canvas** - Get raw Canvas element for further processing -- **Blob** - Raw binary data for custom handling +- Not for full-page browser automation screenshots; use Playwright/Puppeteer when viewport, navigation, or browser state is the core need. +- CORS-blocked resources may require a proxy or same-origin setup; do not assume cross-origin images will embed automatically. +- Do not capture sensitive DOM content unless the user explicitly intends to export it. +- Required inputs: target element selector, desired format, dimensions/scale, asset/CORS constraints, and where the image should be used. +- Verify package/API names against the installed version if build errors indicate version drift. -### Performance -- Ultra-fast capture (1.6ms for small elements, ~171ms for 4000×2000) -- **No dependencies** - Uses standard Web APIs only -- Outperforms html2canvas by 10-40x on complex elements +## Quick Reference -### Style Support -- Embedded fonts (including icon fonts) -- CSS pseudo-elements (::before, ::after) -- CSS counters -- CSS line-clamp -- Transform and shadow effects -- Shadow DOM content +### Common Patterns -### Advanced Capabilities -- Same-origin iframe support -- CORS proxy fallback for blocked assets -- Plugin system for custom transformations -- Straighten transforms (remove rotate/translate) -- Selective element exclusion -- Tight bounding box calculation - -## Installation - -### NPM/Yarn +**Install** ```bash npm install @zumer/snapdom -# or -yarn add @zumer/snapdom ``` -### CDN (ES Module) +**Import from CDN as an ES module** ```html ``` -### CDN (UMD) -```html - -``` - -## Quick Start Examples - -### Basic Reusable Capture +**Capture once and export multiple formats** ```javascript -// Create reusable capture object -const result = await snapdom(document.querySelector('#target')); - -// Export to different formats +const result = await snapdom(document.querySelector("#target")); const png = await result.toPng(); -const jpg = await result.toJpg(); const svg = await result.toSvg(); const canvas = await result.toCanvas(); -const blob = await result.toBlob(); - -// Use the result -document.body.appendChild(png); ``` -### One-Step Export +**One-step PNG export** ```javascript -// Direct export without intermediate object -const png = await snapdom.toPng(document.querySelector('#target')); -const svg = await snapdom.toSvg(element); +const png = await snapdom.toPng(document.querySelector("#target")); ``` -### Download Element +**Download an element** ```javascript -// Automatically download as file -await snapdom.download(element, 'screenshot.png'); -await snapdom.download(element, 'image.svg'); +await snapdom.download(document.querySelector("#target"), "screenshot.png"); ``` -### With Options +**Set scale and dimensions** ```javascript -const result = await snapdom(element, { - scale: 2, // 2x resolution - width: 800, // Custom width - height: 600, // Custom height - embedFonts: true, // Include @font-face - exclude: '.no-capture', // Hide elements - useProxy: true, // Enable CORS proxy - straighten: true, // Remove transforms - noShadows: false // Keep shadows -}); - -const png = await result.toPng({ quality: 0.95 }); -``` - -## Essential Options Reference - -| Option | Type | Purpose | -|--------|------|---------| -| `scale` | Number | Scale output (e.g., 2 for 2x resolution) | -| `width` | Number | Custom output width in pixels | -| `height` | Number | Custom output height in pixels | -| `embedFonts` | Boolean | Include non-icon @font-face rules | -| `useProxy` | String\|Boolean | Enable CORS proxy (URL or true for default) | -| `exclude` | String | CSS selector for elements to hide | -| `straighten` | Boolean | Remove translate/rotate transforms | -| `noShadows` | Boolean | Strip shadow effects | - -## Common Patterns - -### Responsive Screenshots -```javascript -// Capture at different scales -const mobile = await snapdom.toPng(element, { scale: 1 }); -const tablet = await snapdom.toPng(element, { scale: 1.5 }); -const desktop = await snapdom.toPng(element, { scale: 2 }); -``` - -### Exclude Elements -```javascript -// Hide specific elements from capture const png = await snapdom.toPng(element, { - exclude: '.controls, .watermark, [data-no-capture]' -}); -``` - -### Fixed Dimensions -```javascript -// Capture with specific size -const result = await snapdom(element, { + scale: 2, width: 1200, - height: 630 // Standard social media size + height: 630, }); ``` -### CORS Handling +**Exclude UI controls** ```javascript -// Fallback for CORS-blocked resources const png = await snapdom.toPng(element, { - useProxy: 'https://cors.example.com/?' // Custom proxy + exclude: ".controls, [data-no-capture]", }); ``` -### Plugin System (Beta) +**Use a CORS proxy fallback** ```javascript -// Extend with custom exporters -snapdom.plugins([pluginFactory, { colorOverlay: true }]); - -// Hook into lifecycle -defineExports(context) { - return { - pdf: async (ctx, opts) => { /* generate PDF */ } - }; -} - -// Lifecycle hooks available: -// beforeSnap → beforeClone → afterClone → -// beforeRender → beforeExport → afterExport +const png = await snapdom.toPng(element, { + useProxy: "https://cors.example.com/?", +}); ``` -## Performance Comparison +## Examples -SnapDOM significantly outperforms html2canvas: +### Example 1: Social Card Export -| Scenario | SnapDOM | html2canvas | Improvement | -|----------|---------|-------------|-------------| -| Small (200×100) | 1.6ms | 68ms | 42x faster | -| Medium (800×600) | 12ms | 280ms | 23x faster | -| Large (4000×2000) | 171ms | 1,800ms | 10x faster | +- Input: element `#card`, target size `1200x630`, PNG output. +- Steps: + 1. Ensure fonts and images are loaded. + 2. Capture with explicit `width`, `height`, and `scale`. + 3. Download or upload the resulting PNG. +- Expected output / acceptance: exported image matches the card bounds and excludes editor controls. -## Development +### Example 2: SVG Snapshot for Documentation -### Setup -```bash -git clone https://github.com/zumerlab/snapdom.git -cd snapdom -npm install -``` +- Input: styled component preview. +- Steps: + 1. Call `snapdom(element)`. + 2. Export `toSvg()` for scalable documentation output. + 3. Inspect missing fonts/assets if the snapshot differs from the page. +- Expected output / acceptance: SVG preserves visible styles and remains inspectable as a vector artifact. -### Build -```bash -npm run compile -``` +### Example 3: CORS Asset Triage -### Testing -```bash -npm test -``` +- Input: export shows missing remote images. +- Steps: + 1. Confirm whether assets are same-origin and CORS-enabled. + 2. Retry with `useProxy` or replace remote assets with local/same-origin URLs. + 3. Validate final output in the target browser. +- Expected output / acceptance: missing images are attributed to CORS, loading, or selector/sizing issues. -## Browser Support +## References -- Chrome/Edge 90+ -- Firefox 88+ -- Safari 14+ -- Mobile browsers (iOS Safari 14+, Chrome Mobile) +- `references/index.md`: local snapDOM reference navigation. +- `references/other.md`: generated upstream notes and API details. -## Resources +## Maintenance -### Documentation -- **Official Website:** https://snapdom.dev/ -- **GitHub Repository:** https://github.com/zumerlab/snapdom -- **NPM Package:** https://www.npmjs.com/package/@zumer/snapdom -- **License:** MIT - -### scripts/ -Add helper scripts here for automation, e.g.: -- `batch-screenshot.js` - Capture multiple elements -- `pdf-export.js` - Convert snapshots to PDF -- `compare-outputs.js` - Compare SVG vs PNG quality - -### assets/ -Add templates and examples: -- HTML templates for common capture scenarios -- CSS frameworks pre-configured with snapdom -- Boilerplate projects integrating snapdom - -## Related Tools - -- **html2canvas** - Alternative DOM capture (slower but more compatible) -- **Orbit CSS Toolkit** - Companion toolkit by Zumerlab (https://github.com/zumerlab/orbit) - -## Tips & Best Practices - -1. **Performance**: Use `scale` instead of `width`/`height` for better performance -2. **Fonts**: Set `embedFonts: true` to ensure custom fonts appear correctly -3. **CORS Issues**: Use `useProxy: true` if images fail to load -4. **Large Elements**: Break into smaller chunks for complex pages -5. **Quality**: For PNG/JPG, use `quality: 0.95` for best quality -6. **SVG Vectors**: Prefer SVG export for charts and graphics - -## Troubleshooting - -### Elements Not Rendering -- Check if element has sufficient height/width -- Verify CSS is fully loaded before capture -- Try `straighten: false` if transforms are causing issues - -### Missing Fonts -- Set `embedFonts: true` -- Ensure fonts are loaded before calling snapdom -- Check browser console for font loading errors - -### CORS Issues -- Enable `useProxy: true` -- Use custom proxy URL if default fails -- Check if resources are from same origin - -### Performance Issues -- Reduce `scale` value -- Use `noShadows: true` to skip shadow rendering -- Consider splitting large captures into smaller sections +- Sources: local `references/` extracted from snapDOM documentation. +- Last updated: 2026-04-28 +- Known limits: screenshot fidelity depends on browser support, loaded assets, CORS, fonts, and installed snapDOM version. diff --git a/assets/skills/telegram-dev/SKILL.md b/assets/skills/telegram-dev/SKILL.md index d064914..26d51c4 100644 --- a/assets/skills/telegram-dev/SKILL.md +++ b/assets/skills/telegram-dev/SKILL.md @@ -1,760 +1,139 @@ --- name: telegram-dev -description: Telegram 生态开发全栈指南 - 涵盖 Bot API、Mini Apps (Web Apps)、MTProto 客户端开发。包括消息处理、支付、内联模式、Webhook、认证、存储、传感器 API 等完整开发资源。 +description: "Telegram development skill: Bot API, Mini Apps/Web Apps, webhooks, long polling, inline/reply keyboards, payments, initData validation, TDLib/MTProto, message formatting, and deployment troubleshooting." --- -# Telegram 生态开发技能 +# telegram-dev Skill -全面的 Telegram 开发指南，涵盖 Bot 开发、Mini Apps (Web Apps)、客户端开发的完整技术栈。 +Use this skill to build Telegram bots, Mini Apps, and client integrations with explicit security boundaries around tokens, webhooks, and user data. -## 何时使用此技能 +## When to Use This Skill -当需要以下帮助时使用此技能： -- 开发 Telegram Bot（消息机器人） -- 创建 Telegram Mini Apps（小程序） -- 构建自定义 Telegram 客户端 -- 集成 Telegram 支付和业务功能 -- 实现 Webhook 和长轮询 -- 使用 Telegram 认证和存储 -- 处理消息、媒体和文件 -- 实现内联模式和键盘 +Trigger when any of these applies: +- Creating or debugging a Telegram Bot with Bot API methods, long polling, webhooks, commands, messages, media, files, or payments. +- Building Telegram Mini Apps/Web Apps with `window.Telegram.WebApp`, buttons, theme params, storage, sensors, or `initData` validation. +- Implementing inline keyboards, reply keyboards, callback queries, command menus, or dynamic aligned message views. +- Working with TDLib/MTProto client development or API ID/hash based integrations. +- Troubleshooting webhook TLS/port issues, bot token errors, callback handling, formatting, or deployment. -## Telegram 开发生态概览 +## Not For / Boundaries -### 三大核心 API +- Not for spam, unauthorized scraping, account abuse, or bypassing Telegram platform rules. +- Never commit or print bot tokens, API hash, API ID plus phone session data, payment secrets, or user private data. +- Webhook examples require HTTPS and public reachability; local-only servers need a tunnel or local Bot API server setup. +- Required inputs: Bot vs Mini App vs TDLib scope, language/framework, token/auth status, update payload, deployment URL, and exact error. +- Telegram APIs evolve; verify current method parameters and limits in official docs when precision matters. -1. **Bot API** - 创建机器人程序 - - HTTP 接口，简单易用 - - 自动处理加密和通信 - - 适合：聊天机器人、自动化工具 +## Quick Reference -2. **Mini Apps API** (Web Apps) - 创建 Web 应用 - - JavaScript 接口 - - 在 Telegram 内运行 - - 适合：小程序、游戏、电商 +### Common Patterns -3. **Telegram API & TDLib** - 创建客户端 - - 完整的 Telegram 协议实现 - - 支持所有平台 - - 适合：自定义客户端、企业应用 - -## Bot API 开发 - -### 快速开始 - -**API 端点：** -``` -https://api.telegram.org/bot/METHOD_NAME +**Bot API endpoint shape** +```text +https://api.telegram.org/bot/ ``` -**获取 Bot Token：** -1. 与 @BotFather 对话 -2. 发送 `/newbot` -3. 按提示设置名称 -4. 获取 token - -**第一个 Bot (Python)：** +**Send a message** ```python import requests -BOT_TOKEN = "your_bot_token_here" -API_URL = f"https://api.telegram.org/bot{BOT_TOKEN}" - -# 发送消息 -def send_message(chat_id, text): - url = f"{API_URL}/sendMessage" - data = {"chat_id": chat_id, "text": text} - return requests.post(url, json=data) - -# 获取更新（长轮询） -def get_updates(offset=None): - url = f"{API_URL}/getUpdates" - params = {"offset": offset, "timeout": 30} - return requests.get(url, params=params).json() - -# 主循环 -offset = None -while True: - updates = get_updates(offset) - for update in updates.get("result", []): - chat_id = update["message"]["chat"]["id"] - text = update["message"]["text"] - - # 回复消息 - send_message(chat_id, f"你说了：{text}") - - offset = update["update_id"] + 1 +requests.post( + f"https://api.telegram.org/bot{BOT_TOKEN}/sendMessage", + json={"chat_id": chat_id, "text": "Hello"}, + timeout=10, +) ``` -### 核心 API 方法 - -**更新管理：** -- `getUpdates` - 长轮询获取更新 -- `setWebhook` - 设置 Webhook -- `deleteWebhook` - 删除 Webhook -- `getWebhookInfo` - 查询 Webhook 状态 - -**消息操作：** -- `sendMessage` - 发送文本消息 -- `sendPhoto` / `sendVideo` / `sendDocument` - 发送媒体 -- `sendAudio` / `sendVoice` - 发送音频 -- `sendLocation` / `sendVenue` - 发送位置 -- `editMessageText` - 编辑消息 -- `deleteMessage` - 删除消息 -- `forwardMessage` / `copyMessage` - 转发/复制消息 - -**交互元素：** -- `sendPoll` - 发送投票（最多 12 个选项） -- 内联键盘 (InlineKeyboardMarkup) -- 回复键盘 (ReplyKeyboardMarkup) -- `answerCallbackQuery` - 响应回调查询 - -**文件操作：** -- `getFile` - 获取文件信息 -- `downloadFile` - 下载文件 -- 支持最大 2GB 文件（本地 Bot API 模式） - -**支付功能：** -- `sendInvoice` - 发送发票 -- `answerPreCheckoutQuery` - 处理支付 -- Telegram Stars 支付（最高 10,000 Stars） - -### Webhook 配置 - -**设置 Webhook：** +**Long polling** ```python -import requests - -BOT_TOKEN = "your_token" -WEBHOOK_URL = "https://yourdomain.com/webhook" +updates = requests.get( + f"https://api.telegram.org/bot{BOT_TOKEN}/getUpdates", + params={"offset": offset, "timeout": 30}, + timeout=35, +).json() +``` +**Set a webhook** +```python requests.post( f"https://api.telegram.org/bot{BOT_TOKEN}/setWebhook", - json={"url": WEBHOOK_URL} + json={"url": "https://example.com/webhook"}, + timeout=10, ) ``` -**Flask Webhook 示例：** +**Inline keyboard** ```python -from flask import Flask, request -import requests - -app = Flask(__name__) -BOT_TOKEN = "your_token" - -@app.route('/webhook', methods=['POST']) -def webhook(): - update = request.get_json() - - chat_id = update["message"]["chat"]["id"] - text = update["message"]["text"] - - # 发送回复 - requests.post( - f"https://api.telegram.org/bot{BOT_TOKEN}/sendMessage", - json={"chat_id": chat_id, "text": f"收到: {text}"} - ) - - return "OK" - -if __name__ == '__main__': - app.run(port=5000) -``` - -**Webhook 要求：** -- 必须使用 HTTPS -- 支持 TLS 1.2+ -- 端口：443, 80, 88, 8443 -- 公共可访问的 URL - -### 内联键盘 - -**创建内联键盘：** -```python -def send_inline_keyboard(chat_id): - keyboard = { - "inline_keyboard": [ - [ - {"text": "按钮 1", "callback_data": "btn1"}, - {"text": "按钮 2", "callback_data": "btn2"} - ], - [ - {"text": "打开链接", "url": "https://example.com"} - ] - ] - } - - requests.post( - f"{API_URL}/sendMessage", - json={ - "chat_id": chat_id, - "text": "选择一个选项：", - "reply_markup": keyboard - } - ) -``` - -**处理回调：** -```python -def handle_callback_query(callback_query): - query_id = callback_query["id"] - data = callback_query["data"] - chat_id = callback_query["message"]["chat"]["id"] - - # 响应回调 - requests.post( - f"{API_URL}/answerCallbackQuery", - json={"callback_query_id": query_id, "text": f"你点击了 {data}"} - ) - - # 更新消息 - requests.post( - f"{API_URL}/editMessageText", - json={ - "chat_id": chat_id, - "message_id": callback_query["message"]["message_id"], - "text": f"你选择了：{data}" - } - ) -``` - -### 内联模式 - -**配置内联模式：** -与 @BotFather 对话，发送 `/setinline` - -**处理内联查询：** -```python -def handle_inline_query(inline_query): - query_id = inline_query["id"] - query_text = inline_query["query"] - - # 创建结果 - results = [ - { - "type": "article", - "id": "1", - "title": "结果 1", - "input_message_content": { - "message_text": f"你搜索了：{query_text}" - } - } - ] - - requests.post( - f"{API_URL}/answerInlineQuery", - json={"inline_query_id": query_id, "results": results} - ) -``` - -## Mini Apps (Web Apps) 开发 - -### 初始化 Mini App - -**HTML 模板：** -```html - - - - - - - My Mini App - - -

Telegram Mini App

- - - - - -``` - -### Mini App 核心 API - -**WebApp 对象主要属性：** -```javascript -// 初始化数据 -tg.initData // 原始初始化字符串 -tg.initDataUnsafe // 解析后的对象 - -// 用户和主题 -tg.initDataUnsafe.user // 用户信息 -tg.themeParams // 主题颜色 -tg.colorScheme // 'light' 或 'dark' - -// 状态 -tg.isExpanded // 是否全屏 -tg.isFullscreen // 是否全屏 -tg.viewportHeight // 视口高度 -tg.platform // 平台类型 - -// 版本 -tg.version // WebApp 版本 -``` - -**主要方法：** -```javascript -// 窗口控制 -tg.ready() // 标记应用准备就绪 -tg.expand() // 展开到全高度 -tg.close() // 关闭 Mini App -tg.requestFullscreen() // 请求全屏 - -// 数据发送 -tg.sendData(data) // 发送数据到 Bot - -// 导航 -tg.openLink(url) // 打开外部链接 -tg.openTelegramLink(url) // 打开 Telegram 链接 - -// 对话框 -tg.showPopup(params, callback) // 显示弹窗 -tg.showAlert(message) // 显示警告 -tg.showConfirm(message) // 显示确认 - -// 分享 -tg.shareMessage(message) // 分享消息 -tg.shareUrl(url) // 分享链接 -``` - -### UI 控件 - -**主按钮 (MainButton)：** -```javascript -tg.MainButton.setText("点击我"); -tg.MainButton.show(); -tg.MainButton.enable(); -tg.MainButton.showProgress(); // 显示加载 -tg.MainButton.hideProgress(); - -tg.MainButton.onClick(() => { - console.log("主按钮被点击"); -}); -``` - -**次要按钮 (SecondaryButton)：** -```javascript -tg.SecondaryButton.setText("取消"); -tg.SecondaryButton.show(); -tg.SecondaryButton.onClick(() => { - tg.close(); -}); -``` - -**返回按钮 (BackButton)：** -```javascript -tg.BackButton.show(); -tg.BackButton.onClick(() => { - // 返回逻辑 -}); -``` - -**触觉反馈：** -```javascript -tg.HapticFeedback.impactOccurred('light'); // light, medium, heavy -tg.HapticFeedback.notificationOccurred('success'); // success, warning, error -tg.HapticFeedback.selectionChanged(); -``` - -### 存储 API - -**云存储：** -```javascript -// 保存数据 -tg.CloudStorage.setItem('key', 'value', (error, success) => { - if (success) console.log('保存成功'); -}); - -// 获取数据 -tg.CloudStorage.getItem('key', (error, value) => { - console.log('值:', value); -}); - -// 删除数据 -tg.CloudStorage.removeItem('key'); - -// 获取所有键 -tg.CloudStorage.getKeys((error, keys) => { - console.log('所有键:', keys); -}); -``` - -**本地存储：** -```javascript -// 普通本地存储 -localStorage.setItem('key', 'value'); -const value = localStorage.getItem('key'); - -// 安全存储（需要生物识别） -tg.SecureStorage.setItem('secret', 'value', callback); -tg.SecureStorage.getItem('secret', callback); -``` - -### 生物识别认证 - -```javascript -const bioManager = tg.BiometricManager; - -// 初始化 -bioManager.init(() => { - if (bioManager.isInited) { - console.log('支持的类型:', bioManager.biometricType); - // 'finger', 'face', 'unknown' - - if (bioManager.isAccessGranted) { - // 已授权，可以使用 - } else { - // 请求授权 - bioManager.requestAccess({reason: '需要验证身份'}, (success) => { - if (success) { - console.log('授权成功'); - } - }); - } - } -}); - -// 执行认证 -bioManager.authenticate({reason: '确认操作'}, (success, token) => { - if (success) { - console.log('认证成功，token:', token); - } -}); -``` - -### 位置和传感器 - -**获取位置：** -```javascript -tg.LocationManager.init(() => { - if (tg.LocationManager.isInited) { - tg.LocationManager.getLocation((location) => { - console.log('纬度:', location.latitude); - console.log('经度:', location.longitude); - }); - } -}); -``` - -**加速度计：** -```javascript -tg.Accelerometer.start({refresh_rate: 100}, (started) => { - if (started) { - tg.Accelerometer.onEvent((event) => { - console.log('加速度:', event.x, event.y, event.z); - }); - } -}); - -// 停止 -tg.Accelerometer.stop(); -``` - -**陀螺仪：** -```javascript -tg.Gyroscope.start({refresh_rate: 100}, callback); -tg.Gyroscope.onEvent((event) => { - console.log('旋转速度:', event.x, event.y, event.z); -}); -``` - -**设备方向：** -```javascript -tg.DeviceOrientation.start({refresh_rate: 100}, callback); -tg.DeviceOrientation.onEvent((event) => { - console.log('方向:', event.absolute, event.alpha, event.beta, event.gamma); -}); -``` - -### 支付集成 - -**发起支付 (Telegram Stars)：** -```javascript -tg.openInvoice('https://t.me/$invoice_link', (status) => { - if (status === 'paid') { - console.log('支付成功'); - } else if (status === 'cancelled') { - console.log('支付取消'); - } else if (status === 'failed') { - console.log('支付失败'); - } -}); -``` - -### 数据验证 - -**服务器端验证 initData (Python)：** -```python -import hmac -import hashlib -from urllib.parse import parse_qs - -def validate_init_data(init_data, bot_token): - # 解析数据 - parsed = parse_qs(init_data) - received_hash = parsed.get('hash', [''])[0] - - # 移除 hash - data_check_arr = [] - for key, value in parsed.items(): - if key != 'hash': - data_check_arr.append(f"{key}={value[0]}") - - # 排序 - data_check_arr.sort() - data_check_string = '\n'.join(data_check_arr) - - # 计算密钥 - secret_key = hmac.new( - b"WebAppData", - bot_token.encode(), - hashlib.sha256 - ).digest() - - # 计算哈希 - calculated_hash = hmac.new( - secret_key, - data_check_string.encode(), - hashlib.sha256 - ).hexdigest() - - return calculated_hash == received_hash -``` - -### 启动 Mini App - -**从键盘按钮：** -```python -keyboard = { - "keyboard": [[ - { - "text": "打开应用", - "web_app": {"url": "https://yourdomain.com/app"} - } - ]], - "resize_keyboard": True -} - -requests.post( - f"{API_URL}/sendMessage", - json={ - "chat_id": chat_id, - "text": "点击按钮打开应用", - "reply_markup": keyboard - } -) -``` - -**从内联按钮：** -```python -keyboard = { +reply_markup = { "inline_keyboard": [[ - { - "text": "启动应用", - "web_app": {"url": "https://yourdomain.com/app"} - } + {"text": "Open", "url": "https://example.com"}, + {"text": "Action", "callback_data": "action:1"}, ]] } ``` -**从菜单按钮：** -与 @BotFather 对话： -``` -/setmenubutton -→ 选择你的 Bot -→ 提供 URL: https://yourdomain.com/app -``` - -## 客户端开发 (TDLib) - -### 使用 TDLib - -**Python 示例 (python-telegram)：** +**Answer a callback query** ```python -from telegram.client import Telegram - -tg = Telegram( - api_id='your_api_id', - api_hash='your_api_hash', - phone='+1234567890', - database_encryption_key='changeme1234', +requests.post( + f"https://api.telegram.org/bot{BOT_TOKEN}/answerCallbackQuery", + json={"callback_query_id": callback_query_id, "text": "OK"}, ) - -tg.login() - -# 发送消息 -result = tg.send_message( - chat_id=123456789, - text='Hello from TDLib!' -) - -# 获取聊天列表 -result = tg.get_chats() -result.wait() -chats = result.update - -print(chats) - -tg.stop() ``` -### MTProto 协议 +**Initialize a Mini App** +```javascript +const tg = window.Telegram.WebApp; +tg.ready(); +tg.expand(); +``` -**特点：** -- 端到端加密 -- 高性能 -- 支持所有 Telegram 功能 -- 需要 API ID/Hash（从 https://my.telegram.org 获取） +**Send Mini App data back to the bot** +```javascript +tg.sendData(JSON.stringify({ action: "submit" })); +``` -## 最佳实践 +**Validate Mini App initData server-side** +```text +Parse initData -> remove hash -> sort key=value pairs -> HMAC with WebAppData-derived secret -> compare hash. +``` -### Bot 开发 +## Examples -1. **错误处理** - ```python - try: - response = requests.post(url, json=data, timeout=10) - response.raise_for_status() - except requests.exceptions.RequestException as e: - print(f"请求失败: {e}") - ``` +### Example 1: Echo Bot with Long Polling -2. **速率限制** - - 群组消息：最多 20 条/分钟 - - 私聊消息：最多 30 条/秒 - - 全局限制：避免过于频繁 +- Input: bot token and a private test chat. +- Steps: + 1. Call `getUpdates` with an offset. + 2. Extract `message.chat.id` and `message.text`. + 3. Reply with `sendMessage` and advance offset. +- Expected output / acceptance: each user message gets one reply and old updates are not processed repeatedly. -3. **使用 Webhook 而非长轮询** - - 更高效 - - 更低延迟 - - 更好的可扩展性 +### Example 2: Webhook Deployment -4. **数据验证** - - 始终验证 initData - - 不要信任客户端数据 - - 服务器端验证所有操作 +- Input: HTTPS URL `https://example.com/webhook`. +- Steps: + 1. Deploy an endpoint that accepts POST JSON updates. + 2. Call `setWebhook` with the public URL. + 3. Use `getWebhookInfo` to verify status and last error. +- Expected output / acceptance: Telegram delivers updates to the endpoint and webhook info has no current delivery error. -### Mini Apps 开发 +### Example 3: Mini App Button Flow -1. **响应式设计** - ```javascript - // 监听主题变化 - tg.onEvent('themeChanged', () => { - document.body.style.backgroundColor = tg.themeParams.bg_color; - }); - - // 监听视口变化 - tg.onEvent('viewportChanged', () => { - console.log('新高度:', tg.viewportHeight); - }); - ``` +- Input: web app URL and bot chat. +- Steps: + 1. Send a reply or inline keyboard button with `web_app.url`. + 2. In the Mini App, call `ready()` and validate `initData` on the backend. + 3. Send final data with `sendData` or a backend API call. +- Expected output / acceptance: Mini App opens inside Telegram, backend authenticates the user, and bot receives structured data. -2. **性能优化** - - 最小化 JavaScript 包大小 - - 使用懒加载 - - 优化图片和资源 +## References -3. **用户体验** - - 适配深色/浅色主题 - - 使用原生 UI 控件（MainButton 等） - - 提供触觉反馈 - - 快速响应用户操作 +- `references/index.md`: Telegram ecosystem navigation and official links. +- `references/Telegram_Bot_按钮和键盘实现模板.md`: button and keyboard implementation templates. +- `references/动态视图对齐实现文档.md`: aligned data display and dynamic message formatting. -4. **安全考虑** - - HTTPS 强制 - - 验证 initData - - 不在客户端存储敏感信息 - - 使用 SecureStorage 存储密钥 +## Maintenance -## 常用库和工具 - -### Python -- `python-telegram-bot` - 功能强大的 Bot 框架 -- `aiogram` - 异步 Bot 框架 -- `telethon` / `pyrogram` - MTProto 客户端 - -### Node.js -- `node-telegram-bot-api` - Bot API 包装器 -- `telegraf` - 现代 Bot 框架 -- `grammy` - 轻量级框架 - -### 其他语言 -- PHP: `telegram-bot-sdk` -- Go: `telegram-bot-api` -- Java: `TelegramBots` -- C#: `Telegram.Bot` - -## 参考资源 - -### 官方文档 -- Bot API: https://core.telegram.org/bots/api -- Mini Apps: https://core.telegram.org/bots/webapps -- Mini Apps Platform: https://docs.telegram-mini-apps.com -- Telegram API: https://core.telegram.org - -### GitHub 仓库 -- Bot API 服务器: https://github.com/tdlib/telegram-bot-api -- Android 客户端: https://github.com/DrKLO/Telegram -- Desktop 客户端: https://github.com/telegramdesktop/tdesktop -- 官方组织: https://github.com/orgs/TelegramOfficial/repositories - -### 工具 -- @BotFather - 创建和管理 Bot -- https://my.telegram.org - 获取 API ID/Hash -- Telegram Web App 测试环境 - -## 参考文件 - -此技能包含详细的 Telegram 开发资源索引和完整实现模板： - -- **index.md** - 完整的资源链接和快速导航 -- **Telegram_Bot_按钮和键盘实现模板.md** - 交互式按钮和键盘实现指南（404 行，12 KB） - - 三种按钮类型详解（Inline/Reply/Command Menu） - - python-telegram-bot 和 Telethon 双实现对比 - - 完整的即用代码示例和项目结构 - - Handler 系统、错误处理和部署方案 -- **动态视图对齐实现文档.md** - Telegram 数据展示指南（407 行，12 KB） - - 智能动态对齐算法（三步法，O(n×m) 复杂度） - - 等宽字体环境的完美对齐方案 - - 智能数值格式化系统（B/M/K 自动缩写） - - 排行榜和数据表格专业展示 - -这些精简指南提供了核心的 Telegram Bot 开发解决方案： -- 按钮和键盘交互的所有实现方式 -- 消息和数据的专业格式化展示 -- 实用的最佳实践和快速参考 - ---- - -**使用此技能掌握 Telegram 生态的全栈开发！** +- Sources: local Telegram reference files plus official links listed in `references/index.md`. +- Last updated: 2026-04-28 +- Known limits: API methods, limits, and Mini App capabilities are version-sensitive; verify against official Telegram docs for production releases. diff --git a/assets/skills/timescaledb/SKILL.md b/assets/skills/timescaledb/SKILL.md index e6880aa..832684e 100644 --- a/assets/skills/timescaledb/SKILL.md +++ b/assets/skills/timescaledb/SKILL.md @@ -1,108 +1,128 @@ --- name: timescaledb -description: TimescaleDB - PostgreSQL extension for high-performance time-series and event data analytics, hypertables, continuous aggregates, compression, and real-time analytics +description: "TimescaleDB time-series skill: PostgreSQL extension setup, hypertables, time_bucket queries, continuous aggregates, compression/columnstore, retention, performance, and migration troubleshooting." --- -# Timescaledb Skill +# timescaledb Skill -Comprehensive assistance with timescaledb development, generated from official documentation. +Use this skill to model and operate time-series workloads on TimescaleDB/Tiger Data using hypertables, time buckets, compression, and continuous aggregates. ## When to Use This Skill -This skill should be triggered when: -- Working with timescaledb -- Asking about timescaledb features or APIs -- Implementing timescaledb solutions -- Debugging timescaledb code -- Learning timescaledb best practices +Trigger when any of these applies: +- Creating or migrating PostgreSQL tables into TimescaleDB hypertables. +- Designing time-series schemas, chunk intervals, indexes, retention, or compression/columnstore policies. +- Writing `time_bucket` analytics queries or continuous aggregates. +- Debugging ingestion performance, query performance, refresh policies, or migration issues. +- Comparing plain PostgreSQL tables with TimescaleDB hypertables for event/time-series data. + +## Not For / Boundaries + +- Not a replacement for PostgreSQL fundamentals; use `postgresql` for generic SQL, transactions, roles, and non-time-series schema work. +- Do not enable retention/compression policies on production data without restore-tested backups and data-loss review. +- Continuous aggregates have version-specific behavior; verify real-time aggregation and refresh policy defaults against the installed version. +- Required inputs: TimescaleDB version, PostgreSQL version, table schema, time column, ingest rate, query patterns, retention/compression goals, and error text. +- Tiger Cloud features and self-hosted extension features may differ; verify deployment type before prescribing commands. ## Quick Reference ### Common Patterns -*Quick reference patterns will be added as you use the skill.* - -### Example Code Patterns - -**Example 1** (bash): -```bash -rails new my_app -d=postgresql - cd my_app -``` - -**Example 2** (ruby): -```ruby -gem 'timescaledb' -``` - -**Example 3** (shell): -```shell -kubectl create namespace timescale -``` - -**Example 4** (shell): -```shell -kubectl config set-context --current --namespace=timescale -``` - -**Example 5** (sql): +**Enable the extension** ```sql -DROP EXTENSION timescaledb; +CREATE EXTENSION IF NOT EXISTS timescaledb; ``` -## Reference Files +**Create a time-series table** +```sql +CREATE TABLE conditions ( + time timestamptz NOT NULL, + location text NOT NULL, + temperature double precision, + humidity double precision +); +``` -This skill includes comprehensive documentation in `references/`: +**Convert a table to a hypertable** +```sql +SELECT create_hypertable('conditions', 'time'); +``` -- **api.md** - Api documentation -- **compression.md** - Compression documentation -- **continuous_aggregates.md** - Continuous Aggregates documentation -- **getting_started.md** - Getting Started documentation -- **hyperfunctions.md** - Hyperfunctions documentation -- **hypertables.md** - Hypertables documentation -- **installation.md** - Installation documentation -- **other.md** - Other documentation -- **performance.md** - Performance documentation -- **time_buckets.md** - Time Buckets documentation -- **tutorials.md** - Tutorials documentation +**Bucket raw data by hour** +```sql +SELECT + time_bucket('1 hour', time) AS bucket, + location, + avg(temperature) AS avg_temp +FROM conditions +GROUP BY bucket, location +ORDER BY bucket DESC; +``` -Use `view` to read specific reference files when detailed information is needed. +**Create a continuous aggregate** +```sql +CREATE MATERIALIZED VIEW conditions_hourly +WITH (timescaledb.continuous) AS +SELECT + time_bucket('1 hour', time) AS bucket, + location, + avg(temperature) AS avg_temp +FROM conditions +GROUP BY bucket, location; +``` -## Working with This Skill +**Inspect hypertable size** +```sql +SELECT * FROM hypertable_detailed_size('conditions'); +``` -### For Beginners -Start with the getting_started or tutorials reference files for foundational concepts. +**Verify extension version** +```sql +SELECT extversion FROM pg_extension WHERE extname = 'timescaledb'; +``` -### For Specific Features -Use the appropriate category reference file (api, guides, etc.) for detailed information. +## Examples -### For Code Examples -The quick reference section above contains common patterns extracted from the official docs. +### Example 1: Convert Metrics Table to Hypertable -## Resources +- Input: existing table `conditions(time, location, temperature, humidity)`. +- Steps: + 1. Confirm `time` is `NOT NULL` and uses a timestamp type. + 2. Enable the extension. + 3. Run `create_hypertable` in staging and test inserts/queries. +- Expected output / acceptance: the table is a hypertable and existing time-range queries still return correct rows. -### references/ -Organized documentation extracted from official sources. These files contain: -- Detailed explanations -- Code examples with language annotations -- Links to original documentation -- Table of contents for quick navigation +### Example 2: Add Hourly Rollups -### scripts/ -Add helper scripts here for common automation tasks. +- Input: dashboard needs hourly average temperature by location. +- Steps: + 1. Write the raw `time_bucket('1 hour', time)` query. + 2. Convert it to a continuous aggregate after correctness is verified. + 3. Add a refresh policy only after deciding freshness and backfill windows. +- Expected output / acceptance: dashboard reads from the aggregate with documented refresh expectations. -### assets/ -Add templates, boilerplate, or example projects here. +### Example 3: Performance Triage -## Notes +- Input: slow time-range query over a hypertable. +- Steps: + 1. Run `EXPLAIN (ANALYZE, BUFFERS)` and confirm chunk pruning. + 2. Check time predicate shape and indexes for dimension filters. + 3. Inspect hypertable/chunk size and compression state before changing policies. +- Expected output / acceptance: root cause is classified as missing time predicate, bad index, chunk sizing, compression side effect, or stale stats. -- This skill was automatically generated from official documentation -- Reference files preserve the structure and examples from source docs -- Code examples include language detection for better syntax highlighting -- Quick reference patterns are extracted from common usage examples in the docs +## References -## Updating +- `references/index.md`: navigation for local TimescaleDB references. +- `references/installation.md`: install and deployment notes. +- `references/hypertables.md`: hypertables, chunks, sizing, and related APIs. +- `references/time_buckets.md`: `time_bucket` usage. +- `references/continuous_aggregates.md`: aggregate creation and refresh behavior. +- `references/compression.md`: compression/columnstore guidance. +- `references/performance.md`: performance notes. +- `references/tutorials.md`: walkthroughs and examples. -To refresh this skill with updated documentation: -1. Re-run the scraper with the same configuration -2. The skill will be rebuilt with the latest information +## Maintenance + +- Sources: local `references/` extracted from TimescaleDB/Tiger Data documentation. +- Last updated: 2026-04-28 +- Known limits: examples use common SQL forms; verify exact function signatures and policy defaults against the installed extension version. diff --git a/assets/skills/twscrape/SKILL.md b/assets/skills/twscrape/SKILL.md index f32d1f4..2ba8202 100644 --- a/assets/skills/twscrape/SKILL.md +++ b/assets/skills/twscrape/SKILL.md @@ -1,438 +1,148 @@ -# twscrape +--- +name: twscrape +description: "twscrape Twitter/X scraping skill: account pool setup, async search, user/tweet collection, CLI usage, proxy configuration, and rate-limit troubleshooting. Use when extracting public Twitter/X data with twscrape." +--- -Python library for scraping Twitter/X data using GraphQL API with account rotation and session management. +# twscrape Skill -## When to use this skill +Use this skill to build or debug `twscrape` workflows for public Twitter/X data extraction with account rotation, async collection, CLI commands, and proxy-aware operation. -Use this skill when: -- Working with Twitter/X data extraction and scraping -- Need to bypass Twitter API limitations with account rotation -- Building social media monitoring or analytics tools -- Extracting tweets, user profiles, followers, trends from Twitter/X -- Need async/parallel scraping operations for large-scale data collection -- Looking for alternatives to official Twitter API +## When to Use This Skill + +Trigger when any of these applies: +- Scraping Twitter/X search results, profiles, followers, timelines, replies, retweeters, media, or trends with `twscrape`. +- Setting up account pools, cookies, login flows, email verification, or account rotation. +- Choosing between Python async API and the `twscrape` CLI. +- Diagnosing rate limits, empty results, login failures, proxy failures, or suspended accounts. +- Exporting normalized tweet/user data for monitoring, analytics, research, or archival pipelines. + +## Not For / Boundaries + +- Not for bypassing access controls, private content, paid-only data, or platform restrictions. +- Not for guaranteed high-volume scraping; account health, platform changes, and endpoint limits can invalidate assumptions. +- Do not place real Twitter/X passwords, cookies, email passwords, or proxy credentials in examples, commits, logs, or issue reports. +- Required inputs: target query/user/tweet/list, collection limit, output format, account source, proxy requirements, and compliance constraints. +- If behavior differs from these notes, verify against `references/` and the upstream repository before changing production collectors. ## Quick Reference -### Installation +### Common Patterns +**Install the library** ```bash pip install twscrape ``` -### Basic Setup - +**Create an API client and add a cookie-backed account** ```python -import asyncio -from twscrape import API, gather +from twscrape import API -async def main(): - api = API() # Uses accounts.db by default - - # Add accounts (with cookies - more stable) - cookies = "abc=12; ct0=xyz" - await api.pool.add_account("user1", "pass1", "email@example.com", "mail_pass", cookies=cookies) - - # Or add accounts (with login/password - less stable) - await api.pool.add_account("user2", "pass2", "email2@example.com", "mail_pass2") - await api.pool.login_all() - -asyncio.run(main()) +api = API("accounts.db") +await api.pool.add_account( + "username", + "password", + "email@example.com", + "email-password", + cookies="ct0=...; auth_token=...", +) ``` -### Common Operations - +**Login all configured accounts** ```python -# Search tweets -await gather(api.search("elon musk", limit=20)) - -# Get user info -await api.user_by_login("xdevelopers") -user = await api.user_by_id(2244994945) - -# Get user tweets -await gather(api.user_tweets(user_id, limit=20)) -await gather(api.user_tweets_and_replies(user_id, limit=20)) -await gather(api.user_media(user_id, limit=20)) - -# Get followers/following -await gather(api.followers(user_id, limit=20)) -await gather(api.following(user_id, limit=20)) - -# Tweet operations -await api.tweet_details(tweet_id) -await gather(api.retweeters(tweet_id, limit=20)) -await gather(api.tweet_replies(tweet_id, limit=20)) - -# Trends -await gather(api.trends("news")) -``` - -## Key Features - -### 1. Multiple API Support -- **Search API**: Standard Twitter search functionality -- **GraphQL API**: Advanced queries and data extraction -- **Automatic switching**: Based on rate limits and availability - -### 2. Async/Await Architecture -```python -# Parallel scraping -async for tweet in api.search("elon musk"): - print(tweet.id, tweet.user.username, tweet.rawContent) -``` - -### 3. Account Management -- Add multiple accounts for rotation -- Automatic rate limit handling -- Session persistence across runs -- Email verification support (IMAP or manual) - -### 4. Data Models -- SNScrape-compatible models -- Easy conversion to dict/JSON -- Raw API response access available - -## Core API Methods - -### Search Operations - -#### `search(query, limit, kv={})` -Search tweets by query string. - -**Parameters:** -- `query` (str): Search query (supports Twitter search syntax) -- `limit` (int): Maximum number of tweets to return -- `kv` (dict): Additional parameters (e.g., `{"product": "Top"}` for Top tweets) - -**Returns:** AsyncIterator of Tweet objects - -**Example:** -```python -# Latest tweets -async for tweet in api.search("elon musk", limit=20): - print(tweet.rawContent) - -# Top tweets -await gather(api.search("python", limit=20, kv={"product": "Top"})) -``` - -### User Operations - -#### `user_by_login(username)` -Get user information by username. - -**Example:** -```python -user = await api.user_by_login("xdevelopers") -print(user.id, user.displayname, user.followersCount) -``` - -#### `user_by_id(user_id)` -Get user information by user ID. - -#### `followers(user_id, limit)` -Get user's followers. - -#### `following(user_id, limit)` -Get users that the user follows. - -#### `verified_followers(user_id, limit)` -Get only verified followers. - -#### `subscriptions(user_id, limit)` -Get user's Twitter Blue subscriptions. - -### Tweet Operations - -#### `tweet_details(tweet_id)` -Get detailed information about a specific tweet. - -#### `tweet_replies(tweet_id, limit)` -Get replies to a tweet. - -#### `retweeters(tweet_id, limit)` -Get users who retweeted a specific tweet. - -#### `user_tweets(user_id, limit)` -Get tweets from a user (excludes replies). - -#### `user_tweets_and_replies(user_id, limit)` -Get tweets and replies from a user. - -#### `user_media(user_id, limit)` -Get tweets with media from a user. - -### Other Operations - -#### `list_timeline(list_id)` -Get tweets from a Twitter list. - -#### `trends(category)` -Get trending topics by category. - -**Categories:** "news", "sport", "entertainment", etc. - -## Account Management - -### Adding Accounts - -**With cookies (recommended):** -```python -cookies = "abc=12; ct0=xyz" # String or JSON format -await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", cookies=cookies) -``` - -**With credentials:** -```python -await api.pool.add_account("user", "pass", "email@example.com", "mail_pass") await api.pool.login_all() ``` -### CLI Account Management +**Search recent tweets** +```python +from twscrape import gather +tweets = await gather(api.search("python lang:en", limit=50)) +``` + +**Fetch a user then collect timeline data** +```python +user = await api.user_by_login("xdevelopers") +tweets = await gather(api.user_tweets(user.id, limit=100)) +``` + +**Collect followers or following** +```python +followers = await gather(api.followers(user.id, limit=100)) +following = await gather(api.following(user.id, limit=100)) +``` + +**Inspect tweet details and replies** +```python +tweet = await api.tweet_details(1234567890) +replies = await gather(api.tweet_replies(tweet.id, limit=50)) +``` + +**Use the CLI for a small search** +```bash +twscrape search "python lang:en" --limit=20 +``` + +**Manage accounts from the CLI** ```bash -# Add accounts from file twscrape add_accounts accounts.txt username:password:email:email_password - -# Login all accounts -twscrape login_accounts - -# Manual email verification twscrape login_accounts --manual - -# List accounts and status twscrape accounts - -# Re-login specific accounts -twscrape relogin user1 user2 - -# Retry failed logins -twscrape relogin_failed ``` -## Proxy Configuration - -### Per-Account Proxy -```python -proxy = "http://login:pass@example.com:8080" -await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", proxy=proxy) -``` - -### Global Proxy -```python -api = API(proxy="http://login:pass@example.com:8080") -``` - -### Environment Variable +**Set a global proxy** ```bash export TWS_PROXY=socks5://user:pass@127.0.0.1:1080 -twscrape search "elon musk" +twscrape search "bitcoin" --limit=20 ``` -### Dynamic Proxy Changes -```python -api.proxy = "socks5://user:pass@127.0.0.1:1080" -doc = await api.user_by_login("elonmusk") -api.proxy = None # Disable proxy -``` - -**Priority:** `api.proxy` > `TWS_PROXY` env var > account-specific proxy - -## CLI Usage - -### Search Operations -```bash -twscrape search "QUERY" --limit=20 -twscrape search "elon musk lang:es" --limit=20 > data.txt -twscrape search "python" --limit=20 --raw # Raw API responses -``` - -### User Operations -```bash -twscrape user_by_login USERNAME -twscrape user_by_id USER_ID -twscrape followers USER_ID --limit=20 -twscrape following USER_ID --limit=20 -twscrape verified_followers USER_ID --limit=20 -twscrape user_tweets USER_ID --limit=20 -``` - -### Tweet Operations -```bash -twscrape tweet_details TWEET_ID -twscrape tweet_replies TWEET_ID --limit=20 -twscrape retweeters TWEET_ID --limit=20 -``` - -### Trends -```bash -twscrape trends sport -twscrape trends news -``` - -### Custom Database -```bash -twscrape --db custom-accounts.db -``` - -## Advanced Usage - -### Raw API Responses -```python -async for response in api.search_raw("elon musk"): - print(response.status_code, response.json()) -``` - -### Stopping Iteration -```python -from contextlib import aclosing - -async with aclosing(api.search("elon musk")) as gen: - async for tweet in gen: - if tweet.id < 200: - break -``` - -### Convert Models to Dict/JSON -```python -user = await api.user_by_id(user_id) -user_dict = user.dict() -user_json = user.json() -``` - -### Enable Debug Logging +**Enable debug logging** ```python from twscrape.logger import set_log_level + set_log_level("DEBUG") ``` -## Environment Variables +## Examples -- **`TWS_PROXY`**: Global proxy for all accounts - Example: `socks5://user:pass@127.0.0.1:1080` +### Example 1: Search Export -- **`TWS_WAIT_EMAIL_CODE`**: Timeout for email verification (default: 30 seconds) +- Input: query `python lang:en`, limit `50`, output JSON Lines. +- Steps: + 1. Confirm at least one healthy account with `twscrape accounts`. + 2. Use `await gather(api.search(query, limit=50))`. + 3. Serialize selected fields such as `id`, `date`, `user.username`, and `rawContent`. +- Expected output / acceptance: a JSONL file with up to 50 tweet records and no credentials in logs. -- **`TWS_RAISE_WHEN_NO_ACCOUNT`**: Raise exception when no accounts available instead of waiting - Values: `false`, `0`, `true`, `1` (default: `false`) +### Example 2: User Monitoring -## Rate Limits & Limitations +- Input: username `xdevelopers`, timeline limit `100`. +- Steps: + 1. Resolve the account with `await api.user_by_login(username)`. + 2. Collect `api.user_tweets(user.id, limit=100)`. + 3. Store tweet IDs and timestamps so later runs can deduplicate. +- Expected output / acceptance: user metadata plus a deduplicated timeline batch. -### Rate Limits -- Rate limits reset **every 15 minutes** per endpoint -- Each account has **separate limits** for different operations -- Accounts automatically rotate when limits are reached +### Example 3: Rate-Limit Triage -### Tweet Limits -- `user_tweets` and `user_tweets_and_replies` return approximately **3,200 tweets maximum** per user -- This is a Twitter/X platform limitation - -### Account Status -- Rate limits vary based on: - - Account age - - Account verification status - - Account activity history - -### Handling Rate Limits -The library automatically: -- Switches to next available account -- Waits for rate limit reset if all accounts exhausted -- Tracks rate limit status per endpoint - -## Common Patterns - -### Large-Scale Data Collection -```python -async def collect_user_data(username): - user = await api.user_by_login(username) - - # Collect tweets - tweets = await gather(api.user_tweets(user.id, limit=100)) - - # Collect followers - followers = await gather(api.followers(user.id, limit=100)) - - # Collect following - following = await gather(api.following(user.id, limit=100)) - - return { - 'user': user, - 'tweets': tweets, - 'followers': followers, - 'following': following - } -``` - -### Search with Filters -```python -# Language filter -await gather(api.search("python lang:en", limit=20)) - -# Date filter -await gather(api.search("AI since:2024-01-01", limit=20)) - -# From specific user -await gather(api.search("from:elonmusk", limit=20)) - -# With media -await gather(api.search("cats filter:media", limit=20)) -``` - -### Batch Processing -```python -async def process_users(usernames): - tasks = [] - for username in usernames: - task = api.user_by_login(username) - tasks.append(task) - - users = await asyncio.gather(*tasks) - return users -``` - -## Troubleshooting - -### Login Issues -- **Use cookies instead of credentials** for more stable authentication -- Enable **manual email verification** with `--manual` flag -- Check **email password** is correct for IMAP access - -### Rate Limit Problems -- **Add more accounts** for better rotation -- **Increase wait time** between requests -- **Monitor account status** with `twscrape accounts` - -### No Data Returned -- **Check account status** - they may be suspended or rate limited -- **Verify query syntax** - use Twitter search syntax -- **Try different accounts** - some may have better access - -### Connection Issues -- **Configure proxy** if behind firewall -- **Check network connectivity** -- **Verify Twitter/X is accessible** from your location - -## Resources - -- **GitHub Repository**: https://github.com/vladkens/twscrape -- **Installation**: `pip install twscrape` -- **Development Version**: `pip install git+https://github.com/vladkens/twscrape.git` +- Input: collector returns no data or waits indefinitely. +- Steps: + 1. Run `twscrape accounts` and identify locked, suspended, or rate-limited accounts. + 2. Enable debug logging and retry the smallest failing query. + 3. Add healthy accounts or wait for endpoint-specific reset before scaling up. +- Expected output / acceptance: the failing mode is classified as account health, query syntax, proxy/network, or platform limit. ## References -For detailed API documentation and examples, see the reference files in the `references/` directory: +- `references/index.md`: navigation for the local twscrape reference set. +- `references/installation.md`: installation and dependency notes. +- `references/account_management.md`: account pool, login, and rotation behavior. +- `references/api_methods.md`: Python API method reference. +- `references/cli_usage.md`: command-line usage. +- `references/proxy_config.md`: proxy configuration and precedence. +- `references/examples.md`: longer code examples and extraction patterns. -- `references/installation.md` - Installation and setup -- `references/api_methods.md` - Complete API method reference -- `references/account_management.md` - Account configuration and management -- `references/cli_usage.md` - Command-line interface guide -- `references/proxy_config.md` - Proxy configuration options -- `references/examples.md` - Code examples and patterns +## Maintenance ---- - -**Repository**: https://github.com/vladkens/twscrape -**Stars**: 1998+ -**Language**: Python -**License**: MIT +- Sources: local `references/` extracted from upstream twscrape material and the upstream repository noted there. +- Last updated: 2026-04-28 +- Known limits: Twitter/X endpoints and account policies change without notice; validate live collectors against a small sample before large runs.