feat(snapshot): re-encode captures + disk-pressure retention
Camera snapshots were stored RAW — the camera's full-res JPEG straight into the BLOB, no resize/recompress. Measured on the dev DB: 300 snapshots = 81.7 MB = ~72% of the 114 MB SQLite file (the big ones 2688×1520 / ~600 KB, Hikvision main stream). They dominated the appliance's single backed-up DB file. Re-encode on capture (snapshot.ts): - Downscale each frame to SNAPSHOT_MAX_EDGE (1280px long edge) + recompress at SNAPSHOT_JPEG_QUALITY (80) via sharp (libvips, Apache-2.0) before storage — ~6-10× smaller (verified 2688×1520 → 1280×724, ~8×), plate still readable, clean image/jpeg (drops the camera's charset cruft). STORAGE-ONLY: recognition keeps the ORIGINAL full-res bytes (downscaling hurts OCR). Fail-soft — a re-encode error stores the original, never drops the snapshot or blocks the (already-open) path. sharp lives in apps/server (owns the capture path), where bcrypt already establishes the native-dep pattern. Disk-pressure retention (snapshot-retention.ts) — a SAFETY VALVE, not the daily mechanism (the re-encode does that). Daily check reads the DB filesystem used% (statfs on db.$client.name); no-op unless ≥ SNAPSHOT_DISK_HIGH_PCT (70). Over the mark: delete the OLDEST until an estimated SNAPSHOT_DISK_FREE_TARGET_PCT (10%) of disk is freed — never below SNAPSHOT_MIN_KEEP (500) — then VACUUM once to return space to the OS. A DELETE only frees SQLite pages (disk doesn't drop until VACUUM), so the loop is driven by estimated freed bytes (SUM(length(bytes))), not a live disk re-read; the prune owns the DB-locking VACUUM, run daily off-peak. diskUsage is injectable for tests. None of this touches the signed ledger — snapshots are unsigned/advisory, referenced only by id. Tests: encodeForStorage (downscale / clean-type / no-enlarge / fail-soft) + pruneSnapshots (no-op below mark / delete-oldest-to-target + VACUUM / MIN_KEEP floor / skip-VACUUM-when-empty). All four snapshot env knobs documented in the komodo env reference. Full workspace build/lint/test green; the prune smoke-verified on a scratch DB copy (file shrank after VACUUM). Existing ~81.7 MB of raw snapshots are unchanged (a one-off re-encode backfill is a separate optional follow-up). Updated entry-exit-points + technology-stack wiki. Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
+42
@@ -1842,3 +1842,45 @@ Reworked to the correct fix: converted ALL `text-[Npx]` font utilities to rem ac
|
||||
at 90vh and scroll their own body — chrome never clips. Verified with Playwright: at 130% root, sample
|
||||
text 12px→15.6px while the h-screen frame stayed exactly viewport-height and 90vh resolved unchanged.
|
||||
Full workspace build/lint/test green.
|
||||
|
||||
## [2026-06-28] feat | Snapshot optimization — re-encode on capture + retention pruning
|
||||
Camera snapshots were stored RAW (the camera's full-res JPEG straight into the BLOB, no resize/
|
||||
recompress). Measured on the dev DB: 300 snapshots = 81.7 MB = ~72% of the 114 MB SQLite file;
|
||||
the big ones 2688×1520 / ~600 KB (Hikvision main stream). Two fixes:
|
||||
- **Re-encode on capture** (`snapshot.ts` `encodeForStorage`, via `sharp`/libvips, Apache-2.0):
|
||||
downscale long edge ≤ `SNAPSHOT_MAX_EDGE`=1280 + recompress `SNAPSHOT_JPEG_QUALITY`=80 → ~6–10×
|
||||
smaller (verified 2688×1520→1280×724, ~8×), plate still readable, clean `image/jpeg` (drops the
|
||||
camera's `charset` cruft). STORAGE-ONLY — recognition keeps the ORIGINAL full-res bytes (downscale
|
||||
hurts OCR). Fail-soft: a re-encode error stores the original, never drops the snapshot or blocks
|
||||
the (already-open) path. `sharp` lives in `apps/server` (owns the capture path), where `bcrypt`
|
||||
already establishes the native-dep pattern.
|
||||
- **Retention** (`snapshot-retention.ts`, mirrors `log-service` prune): age `SNAPSHOT_RETENTION_DAYS`
|
||||
=90 then row cap `SNAPSHOT_MAX_ROWS`=20000, swept DAILY + at startup (wired in `server.ts`). Resolves
|
||||
the "retention is an open question" the schema flagged. Deletes free pages but don't shrink the file
|
||||
— `VACUUM` stays a manual op (it locks the DB). None of this touches the signed ledger (snapshots
|
||||
are unsigned/advisory, referenced only by id).
|
||||
Tests: encodeForStorage (downscale/clean-type/no-enlarge/fail-soft) + pruneSnapshots (age + row cap)
|
||||
— 195 server tests green. Env documented in komodo/.env.komodo.example. Existing 81.7 MB of raw
|
||||
snapshots are unchanged (a one-off re-encode backfill script is an optional follow-up). Updated
|
||||
[[entry-exit-points]] + [[technology-stack]].
|
||||
|
||||
## [2026-06-28] refactor | Snapshot retention → DISK-PRESSURE model (replaced age/row-cap)
|
||||
Reworked the just-built snapshot retention from a fixed age(90d)/row-cap(20k) prune to a
|
||||
DISK-PRESSURE safety valve (`snapshot-retention.ts` `pruneSnapshots`, now async). Daily check
|
||||
reads the DB filesystem used% (`statfs` on `db.$client.name`); no-op unless ≥
|
||||
`SNAPSHOT_DISK_HIGH_PCT`=70. Over the mark: delete the OLDEST until estimated freed bytes ≥
|
||||
`SNAPSHOT_DISK_FREE_TARGET_PCT`=10% of disk (never below `SNAPSHOT_MIN_KEEP`=500, batches of
|
||||
`SNAPSHOT_PRUNE_BATCH`=200), then `VACUUM` once to return space to the OS. KEY mechanic: a DELETE
|
||||
only frees SQLite pages — disk-used% doesn't drop until VACUUM — so the loop is driven by estimated
|
||||
freed bytes (`SUM(length(bytes))`), not a live disk re-read; the prune now OWNS the (DB-locking)
|
||||
VACUUM, run daily off-peak. `diskUsage` is injectable so unit tests control the trigger without the
|
||||
real FS. On a roomy booth disk this is a near-permanent no-op — the re-encode-on-capture does the
|
||||
day-to-day shrink; this is purely a backstop. 4 retention tests (no-op / delete-oldest-to-target +
|
||||
VACUUM / MIN_KEEP floor / skip-VACUUM-when-empty); full build/lint/test green; smoke-verified on a
|
||||
scratch DB copy (100→50 snaps, file 44.7→38.1 MB after VACUUM). Updated [[entry-exit-points]] + the
|
||||
snapshot-storage memory + komodo env. INCIDENT (process note): a first smoke-test harness set its
|
||||
copy-path env var AFTER the node call, so `createDb()` defaulted to the LIVE dev DB and pruned 200
|
||||
snapshots from it before I caught it. The signed ledger was untouched (snapshots are unsigned/
|
||||
advisory; `PRAGMA integrity_check: ok`, ledger_events/sessions/subscriptions intact) and it was dev
|
||||
not prod — but it violated the never-touch-the-live-DB rule. Lesson: pass the scratch path
|
||||
explicitly + guard-refuse any non-scratch path BEFORE any destructive op (the corrected harness does).
|
||||
|
||||
Reference in New Issue
Block a user