mirror of
https://github.com/jaxperro/winning-wallet-finder.git
synced 2026-07-27 15:57:47 +00:00
jaxperro
5c8ee5961b
The nightly Mac-coupled bulk ingest was the fragile link (today: Mac slept through 08:00, the pipeline started 09:31, and the sftp bulk pull wedged in timeout cascades holding the db lock — digest blocked behind it). Stage 0 moves the fold onto the box and reduces the Mac to an incremental mirror: - recorder/fold.py (sidecar, capture stays PID 1): closed gz segments -> zstd parquet /data/parquet/<fam>/date=*/segment.parquet, row-parity verified, manifest-logged, THEN gz deleted. Deletion invariant STRONGER: raw needs a verified parquet; parquet needs a Mac ACK before the disk guard may prune it. duckdb capped 384MB; VM 256MB -> 1GB. - recorder/sync_tape.py (com.jaxperro.tape-sync every 15 min + daily.sh): manifest-driven incremental sftp pull, per-file row verify, append into rtds.duckdb NATIVE tables (views-over-parquet rejected: sim's per-asset point queries would crawl), segment-keyed idempotence shared with the legacy ingest.py (kept as fallback), ack back to the box. - recorder/bootstrap_parquet.py: pre-fold history exported to the mirror, parity OK (13.84M trades + 3.07M aux). live/parquet/ is now the complete durable layer Stage 1 (MotherDuck/ClickHouse) would consume. - research/tape.py connect(): brief retry — the 15-min sync holds the write lock for seconds. First run: backlog folded in <60s on the box, 36/36 files mirrored+ verified, rtds.duckdb 13.8M -> 18.2M trades, tape age 24h+ -> ~15-45 min. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
53 lines
987 B
Plaintext
53 lines
987 B
Plaintext
__pycache__/
|
|
*.pyc
|
|
.DS_Store
|
|
|
|
# never commit live credentials or runtime state
|
|
config.json
|
|
config.live.json
|
|
copybot_state.live.json
|
|
config.json.bak
|
|
config.example.json
|
|
copytrade_state.json
|
|
copybot_state.json
|
|
copybot_fills.jsonl
|
|
*.log
|
|
*.tmp
|
|
# NOTE: live/copybot_live.json IS tracked — it's the public dashboard feed
|
|
|
|
# generated research data (regenerable via edge_research.py / table_77.py)
|
|
edge_metrics.jsonl
|
|
edge_profitable.json
|
|
copyable_77.csv
|
|
lp_markets.csv
|
|
follow_10.json
|
|
lp_paper_state.json
|
|
|
|
# cross-venue scanner output
|
|
xarb_hits.csv
|
|
hunt.log
|
|
recv.log
|
|
watcher_state.json
|
|
huntwide.csv
|
|
oos.log
|
|
hunt.log
|
|
|
|
# live/ + wide/ — large local data + regenerable artifacts
|
|
*.duckdb
|
|
*.duckdb.wal
|
|
live/candidates.json
|
|
live/scored.json
|
|
live/*_scored.json
|
|
live/watch_prejune*.json
|
|
live/history/
|
|
live/slug_cache.json
|
|
archive/local/
|
|
live/edge_verdict.txt
|
|
live/rtds.duckdb
|
|
research/forward.log
|
|
research/launchd.log
|
|
research/.nightly.lock.d
|
|
research/__pycache__/
|
|
live/parquet/
|
|
live/tape_sync.log
|