mirror of
https://github.com/jaxperro/winning-wallet-finder.git
synced 2026-07-27 15:57:47 +00:00
jaxperro
5c8ee5961b
The nightly Mac-coupled bulk ingest was the fragile link (today: Mac slept through 08:00, the pipeline started 09:31, and the sftp bulk pull wedged in timeout cascades holding the db lock — digest blocked behind it). Stage 0 moves the fold onto the box and reduces the Mac to an incremental mirror: - recorder/fold.py (sidecar, capture stays PID 1): closed gz segments -> zstd parquet /data/parquet/<fam>/date=*/segment.parquet, row-parity verified, manifest-logged, THEN gz deleted. Deletion invariant STRONGER: raw needs a verified parquet; parquet needs a Mac ACK before the disk guard may prune it. duckdb capped 384MB; VM 256MB -> 1GB. - recorder/sync_tape.py (com.jaxperro.tape-sync every 15 min + daily.sh): manifest-driven incremental sftp pull, per-file row verify, append into rtds.duckdb NATIVE tables (views-over-parquet rejected: sim's per-asset point queries would crawl), segment-keyed idempotence shared with the legacy ingest.py (kept as fallback), ack back to the box. - recorder/bootstrap_parquet.py: pre-fold history exported to the mirror, parity OK (13.84M trades + 3.07M aux). live/parquet/ is now the complete durable layer Stage 1 (MotherDuck/ClickHouse) would consume. - research/tape.py connect(): brief retry — the 15-min sync holds the write lock for seconds. First run: backlog folded in <60s on the box, 36/36 files mirrored+ verified, rtds.duckdb 13.8M -> 18.2M trades, tape age 24h+ -> ~15-45 min. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
19 lines
433 B
TOML
19 lines
433 B
TOML
# wwf-recorder — RTDS tape (recorder/recorder.py). Own app + volume so bot
|
|
# deploys never gap the tape. No http_service (gotcha 15).
|
|
app = "wwf-recorder"
|
|
primary_region = "arn"
|
|
|
|
[build]
|
|
dockerfile = "recorder/fly.Dockerfile"
|
|
|
|
[[mounts]]
|
|
source = "tape"
|
|
destination = "/data"
|
|
|
|
[[restart]]
|
|
policy = "always"
|
|
|
|
[[vm]]
|
|
size = "shared-cpu-1x"
|
|
memory = "1gb" # fold sidecar: duckdb capped at 384MB; capture headroom
|