feat(snapshot): re-encode captures + disk-pressure retention
Camera snapshots were stored RAW — the camera's full-res JPEG straight into the BLOB, no resize/recompress. Measured on the dev DB: 300 snapshots = 81.7 MB = ~72% of the 114 MB SQLite file (the big ones 2688×1520 / ~600 KB, Hikvision main stream). They dominated the appliance's single backed-up DB file. Re-encode on capture (snapshot.ts): - Downscale each frame to SNAPSHOT_MAX_EDGE (1280px long edge) + recompress at SNAPSHOT_JPEG_QUALITY (80) via sharp (libvips, Apache-2.0) before storage — ~6-10× smaller (verified 2688×1520 → 1280×724, ~8×), plate still readable, clean image/jpeg (drops the camera's charset cruft). STORAGE-ONLY: recognition keeps the ORIGINAL full-res bytes (downscaling hurts OCR). Fail-soft — a re-encode error stores the original, never drops the snapshot or blocks the (already-open) path. sharp lives in apps/server (owns the capture path), where bcrypt already establishes the native-dep pattern. Disk-pressure retention (snapshot-retention.ts) — a SAFETY VALVE, not the daily mechanism (the re-encode does that). Daily check reads the DB filesystem used% (statfs on db.$client.name); no-op unless ≥ SNAPSHOT_DISK_HIGH_PCT (70). Over the mark: delete the OLDEST until an estimated SNAPSHOT_DISK_FREE_TARGET_PCT (10%) of disk is freed — never below SNAPSHOT_MIN_KEEP (500) — then VACUUM once to return space to the OS. A DELETE only frees SQLite pages (disk doesn't drop until VACUUM), so the loop is driven by estimated freed bytes (SUM(length(bytes))), not a live disk re-read; the prune owns the DB-locking VACUUM, run daily off-peak. diskUsage is injectable for tests. None of this touches the signed ledger — snapshots are unsigned/advisory, referenced only by id. Tests: encodeForStorage (downscale / clean-type / no-enlarge / fail-soft) + pruneSnapshots (no-op below mark / delete-oldest-to-target + VACUUM / MIN_KEEP floor / skip-VACUUM-when-empty). All four snapshot env knobs documented in the komodo env reference. Full workspace build/lint/test green; the prune smoke-verified on a scratch DB copy (file shrank after VACUUM). Existing ~81.7 MB of raw snapshots are unchanged (a one-off re-encode backfill is a separate optional follow-up). Updated entry-exit-points + technology-stack wiki. Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
@@ -1,6 +1,8 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import sharp from "sharp";
|
||||
import type { CameraDevice, Snapshot } from "@parking/devices";
|
||||
import { captureSnapshotShared } from "./snapshot.js";
|
||||
import { captureSnapshotShared, encodeForStorage } from "./snapshot.js";
|
||||
import { silentLogger } from "./test-helpers.js";
|
||||
|
||||
// captureSnapshotShared: one HTTP pull per camera per vehicle. A Hikvision unit serves
|
||||
// snapshots SINGLE-THREADED (a 2nd concurrent GET → HTTP 503). On an entry the ANPR
|
||||
@@ -91,3 +93,49 @@ describe("captureSnapshotShared", () => {
|
||||
expect(s1.bytes.equals(s2.bytes)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// encodeForStorage: downscale + re-compress a captured frame for STORAGE (smaller, plate
|
||||
// still readable). Recognition uses the original; this never runs on the OCR path. Fail-soft.
|
||||
describe("encodeForStorage", () => {
|
||||
/** A big synthetic JPEG (2688×1520, the Hikvision main-stream size) to downscale. */
|
||||
async function bigJpeg(): Promise<Buffer> {
|
||||
return sharp({
|
||||
create: { width: 2688, height: 1520, channels: 3, background: { r: 120, g: 130, b: 140 } },
|
||||
})
|
||||
.jpeg({ quality: 95 })
|
||||
.toBuffer();
|
||||
}
|
||||
|
||||
it("downscales the long edge to ≤1280 and emits clean image/jpeg", async () => {
|
||||
const bytes = await bigJpeg();
|
||||
const shot: Snapshot = { bytes, contentType: 'image/jpeg; charset="UTF-8"', capturedAt: new Date().toISOString() };
|
||||
const out = await encodeForStorage(shot, silentLogger());
|
||||
expect(out.contentType).toBe("image/jpeg"); // charset cruft stripped
|
||||
const meta = await sharp(out.bytes).metadata();
|
||||
expect(Math.max(meta.width ?? 0, meta.height ?? 0)).toBeLessThanOrEqual(1280);
|
||||
expect(out.bytes.length).toBeLessThan(bytes.length); // smaller than the original
|
||||
});
|
||||
|
||||
it("never enlarges an already-small image", async () => {
|
||||
const small = await sharp({ create: { width: 640, height: 360, channels: 3, background: { r: 0, g: 0, b: 0 } } })
|
||||
.jpeg()
|
||||
.toBuffer();
|
||||
const out = await encodeForStorage(
|
||||
{ bytes: small, contentType: "image/jpeg", capturedAt: new Date().toISOString() },
|
||||
silentLogger(),
|
||||
);
|
||||
const meta = await sharp(out.bytes).metadata();
|
||||
expect(meta.width).toBe(640); // withoutEnlargement
|
||||
expect(meta.height).toBe(360);
|
||||
});
|
||||
|
||||
it("fails soft: a non-image body is stored unchanged with a cleaned type", async () => {
|
||||
const garbage = Buffer.from("this is not an image");
|
||||
const out = await encodeForStorage(
|
||||
{ bytes: garbage, contentType: 'text/plain; charset="UTF-8"', capturedAt: new Date().toISOString() },
|
||||
silentLogger(),
|
||||
);
|
||||
expect(out.bytes.equals(garbage)).toBe(true); // original bytes, never dropped
|
||||
expect(out.contentType).toBe("text/plain"); // charset stripped even on the fallback
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user