sync_tape: never bless a timed-out sftp partial — fall through to the base64 fallback

A TimeoutExpired get leaves a size>0 partial that the exists-check treated
as success, so the console fallback never ran for timeouts; the caller's
parquet read then failed (no magic bytes), deleted the file, and the next
run repeated the identical loop — 8 hours of tape-head stall overnight
2026-07-23. Partials from timeouts are now removed and the fallback runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jaxperro
2026-07-23 09:34:52 -04:00
parent 2af921d13a
commit 678d7de948
+6 -2
View File
@@ -48,13 +48,17 @@ def sftp_get(remote, local, timeout=240):
(2026-07-22: sftp timeouts left magic-byte-less partial parquets while
console commands stayed fast). Integrity is the caller's manifest
row-count verify either way; partial files are removed here."""
timed_out = False
try:
subprocess.run([FLYCTL, "ssh", "sftp", "get", remote, local, "-a", APP],
capture_output=True, timeout=timeout,
stdin=subprocess.DEVNULL)
except subprocess.TimeoutExpired:
pass
if os.path.exists(local) and os.path.getsize(local) > 0:
timed_out = True # a timed-out get leaves a size>0 PARTIAL — must
# not be blessed, or the base64 fallback never
# runs and the file loops unreadable forever
# (2026-07-23: 8h of exactly that)
if not timed_out and os.path.exists(local) and os.path.getsize(local) > 0:
return True
try:
os.remove(local)