feat(snapshot): re-encode captures + disk-pressure retention
Build desktop / desktop (push) Successful in 4m13s
Build & push images / images (push) Successful in 2m56s
CI / check (push) Successful in 38s

Camera snapshots were stored RAW — the camera's full-res JPEG straight into
the BLOB, no resize/recompress. Measured on the dev DB: 300 snapshots = 81.7 MB
= ~72% of the 114 MB SQLite file (the big ones 2688×1520 / ~600 KB, Hikvision
main stream). They dominated the appliance's single backed-up DB file.

Re-encode on capture (snapshot.ts):
- Downscale each frame to SNAPSHOT_MAX_EDGE (1280px long edge) + recompress at
  SNAPSHOT_JPEG_QUALITY (80) via sharp (libvips, Apache-2.0) before storage —
  ~6-10× smaller (verified 2688×1520 → 1280×724, ~8×), plate still readable,
  clean image/jpeg (drops the camera's charset cruft). STORAGE-ONLY: recognition
  keeps the ORIGINAL full-res bytes (downscaling hurts OCR). Fail-soft — a
  re-encode error stores the original, never drops the snapshot or blocks the
  (already-open) path. sharp lives in apps/server (owns the capture path), where
  bcrypt already establishes the native-dep pattern.

Disk-pressure retention (snapshot-retention.ts) — a SAFETY VALVE, not the daily
mechanism (the re-encode does that). Daily check reads the DB filesystem used%
(statfs on db.$client.name); no-op unless ≥ SNAPSHOT_DISK_HIGH_PCT (70). Over the
mark: delete the OLDEST until an estimated SNAPSHOT_DISK_FREE_TARGET_PCT (10%) of
disk is freed — never below SNAPSHOT_MIN_KEEP (500) — then VACUUM once to return
space to the OS. A DELETE only frees SQLite pages (disk doesn't drop until VACUUM),
so the loop is driven by estimated freed bytes (SUM(length(bytes))), not a live
disk re-read; the prune owns the DB-locking VACUUM, run daily off-peak. diskUsage
is injectable for tests. None of this touches the signed ledger — snapshots are
unsigned/advisory, referenced only by id.

Tests: encodeForStorage (downscale / clean-type / no-enlarge / fail-soft) +
pruneSnapshots (no-op below mark / delete-oldest-to-target + VACUUM / MIN_KEEP
floor / skip-VACUUM-when-empty). All four snapshot env knobs documented in the
komodo env reference. Full workspace build/lint/test green; the prune smoke-verified
on a scratch DB copy (file shrank after VACUUM).

Existing ~81.7 MB of raw snapshots are unchanged (a one-off re-encode backfill is
a separate optional follow-up). Updated entry-exit-points + technology-stack wiki.

Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
2026-06-28 17:15:15 +02:00
parent cce99aadfd
commit 96acd6b662
11 changed files with 727 additions and 5 deletions
+148
View File
@@ -0,0 +1,148 @@
import { statfs } from "node:fs/promises";
import { dirname, resolve } from "node:path";
import { asc, snapshots, sql, type Db } from "@parking/db";
import type { FastifyBaseLogger } from "fastify";
// Snapshot retention — DISK-PRESSURE model. Camera snapshots are unsigned, advisory, prunable
// BLOBs (see snapshot.ts); they're referenced by the signed ledger only by id, so pruning an
// old image never affects the chain. They no longer dominate the DB day-to-day (captures are
// re-encoded small at SNAPSHOT_MAX_EDGE/JPEG_QUALITY), so this is a SAFETY VALVE: only when the
// filesystem holding the DB crosses a high-water mark do we delete the OLDEST snapshots and
// VACUUM to return disk to the OS.
//
// Why estimated-bytes, not live disk%: a DELETE only frees SQLite *pages* — the file (and thus
// OS disk usage) doesn't shrink until VACUUM. So the prune loop can't watch usedPct fall in real
// time. Instead it sums LENGTH(bytes) of the rows it deletes and stops when that estimate reaches
// the free-target, then VACUUMs ONCE at the end to realize the space. A MIN_KEEP floor always
// wins — we never delete evidence below it, even under pressure (if the disk is full of something
// else, that's not ours to fix).
export interface SnapshotRetention {
/** Prune when the DB's filesystem is at least this % used. */
readonly highPct: number;
/** Try to free roughly this % of the disk per run (the delete target). */
readonly freeTargetPct: number;
/** Never prune below this many snapshots (the floor). */
readonly minKeep: number;
/** Delete oldest in batches of this size (re-checks between batches). */
readonly batch: number;
}
export const DEFAULT_SNAPSHOT_RETENTION: SnapshotRetention = {
highPct: Number(process.env.SNAPSHOT_DISK_HIGH_PCT ?? 70),
freeTargetPct: Number(process.env.SNAPSHOT_DISK_FREE_TARGET_PCT ?? 10),
minKeep: Number(process.env.SNAPSHOT_MIN_KEEP ?? 500),
batch: Number(process.env.SNAPSHOT_PRUNE_BATCH ?? 200),
};
/** Disk usage of the filesystem holding the DB. Injectable so tests don't touch the real FS. */
export interface DiskUsage {
readonly usedPct: number;
readonly totalBytes: number;
}
export interface PruneOptions {
readonly retention?: SnapshotRetention;
/** Override how disk usage is read (tests inject a fake; default = statfs the DB's FS). */
readonly diskUsage?: () => Promise<DiskUsage>;
}
export interface PruneResult {
readonly deletedRows: number;
readonly freedBytesEst: number;
readonly vacuumed: boolean;
readonly usedPctBefore: number;
/** True if we hit the MIN_KEEP floor while the disk was still over the high-water mark. */
readonly floorHitWhileOver: boolean;
}
/** Read the used% + total bytes of the filesystem holding the DB file. */
async function diskUsageForDb(db: Db): Promise<DiskUsage> {
const file = (db.$client as { name?: string }).name ?? process.env.DATABASE_URL ?? "./parking.sqlite";
const st = await statfs(dirname(resolve(file)));
const total = st.blocks * st.bsize;
const avail = st.bavail * st.bsize;
const usedPct = total > 0 ? (1 - avail / total) * 100 : 0;
return { usedPct, totalBytes: total };
}
/**
* Prune snapshots under DISK PRESSURE. No-op unless the DB's filesystem is ≥ highPct used. When
* over, deletes the OLDEST snapshots until an estimated freeTargetPct of the disk is freed (or the
* minKeep floor is hit, or no rows remain), then VACUUMs once. Best-effort; safe on a timer.
*/
export async function pruneSnapshots(
db: Db,
opts: PruneOptions = {},
logger?: FastifyBaseLogger,
): Promise<PruneResult> {
const r = opts.retention ?? DEFAULT_SNAPSHOT_RETENTION;
const readDisk = opts.diskUsage ?? (() => diskUsageForDb(db));
let usedPctBefore = 0;
try {
const disk = await readDisk();
usedPctBefore = disk.usedPct;
// The overwhelmingly common case: plenty of headroom → do nothing.
if (disk.usedPct < r.highPct) {
return { deletedRows: 0, freedBytesEst: 0, vacuumed: false, usedPctBefore, floorHitWhileOver: false };
}
// Target bytes to free this run (≈ freeTargetPct of the whole disk).
const targetBytes = (r.freeTargetPct / 100) * disk.totalBytes;
let freedBytesEst = 0;
let deletedRows = 0;
let floorHitWhileOver = false;
// Delete the oldest in batches, summing their BLOB sizes, until we've freed the target — or
// we'd cross the MIN_KEEP floor — or there are no more rows.
for (;;) {
const count = db.select({ c: sql<number>`count(*)` }).from(snapshots).get()?.c ?? 0;
if (count <= r.minKeep) {
floorHitWhileOver = true; // still over the high-water mark but can't delete below the floor
break;
}
if (freedBytesEst >= targetBytes) break;
const room = count - r.minKeep; // how many we may still delete before the floor
const take = Math.min(r.batch, room);
const oldest = db
.select({ id: snapshots.id, len: sql<number>`length(${snapshots.bytes})` })
.from(snapshots)
.orderBy(asc(snapshots.capturedAt))
.limit(take)
.all();
if (oldest.length === 0) break;
const ids = oldest.map((o) => o.id);
db.delete(snapshots).where(sql`${snapshots.id} in (${sql.join(ids, sql`, `)})`).run();
deletedRows += oldest.length;
freedBytesEst += oldest.reduce((s, o) => s + (o.len ?? 0), 0);
}
// Realize the freed space: VACUUM returns pages to the OS (the file shrinks). Only if we
// actually deleted something. Non-fatal on failure — pages are still freed for reuse.
let vacuumed = false;
if (deletedRows > 0) {
try {
(db.$client as { exec: (sql: string) => void }).exec("VACUUM");
vacuumed = true;
} catch (err) {
logger?.warn(`snapshot prune: VACUUM failed (pages freed for reuse): ${(err as Error).message}`);
}
}
if (floorHitWhileOver) {
logger?.warn(
`snapshot prune: disk ${usedPctBefore.toFixed(0)}% used but hit MIN_KEEP floor (${r.minKeep}) ` +
`after deleting ${deletedRows} — disk pressure is not from snapshots`,
);
}
return { deletedRows, freedBytesEst, vacuumed, usedPctBefore, floorHitWhileOver };
} catch (err) {
logger?.warn(`snapshot prune failed: ${(err as Error).message}`);
return { deletedRows: 0, freedBytesEst: 0, vacuumed: false, usedPctBefore, floorHitWhileOver: false };
}
}