fix(backup): persist last-success/error status; wall-clock-based schedule

BackupService tracked last-success/last-error as plain in-process fields
and scheduled the daily backup via setInterval measured from process
start — so any server restart (deploy/crash/OOM/reboot, routine under
`restart: always`) silently reset the admin UI to "last successful
backup: Never" and drifted the actual cadence, independent of whether
backups were writing correctly to disk (they were — a real field
incident at park-buzi showed 7 valid rotating backups on disk with the
status stuck on "Never").

Persist last-success/error to new site_config columns (migration 0025)
and add BackupService.isDue(), computed from the persisted timestamp
instead of process uptime; server.ts now polls every 15 min and lets
isDue() gate the actual run. No API/UI contract change.

Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
2026-08-30 18:11:23 +02:00
parent 3a176c5cc8
commit 2910672b5a
7 changed files with 317 additions and 28 deletions
+61 -5
View File
@@ -2,7 +2,7 @@
type: concept
tags: [parking, durability, backup, recovery, security, crypto]
sources: []
updated: 2026-06-29
updated: 2026-08-30
---
# Backup & Disaster Recovery
@@ -207,12 +207,68 @@ timer + the manual route**. What landed:
**SMB/NFS already work** — they're just a mounted path the admin enters as the target. **Deferred to
follow-up slices:** an **SFTP** target and a **restore runbook / CLI**.
## Field bug — "last successful backup: Never" despite valid, rotating backups on disk (found + fixed 2026-08-30)
**Symptom (park-buzi):** the admin noticed the backup directory held 7 real, correctly-sized,
correctly-rotating encrypted backups (`parking-backup-*.sqlite.enc`, retention working exactly as
designed) — yet the Backup screen's "Kopja e fundit e suksesshme" (last successful backup) showed
**"Asnjëherë" (Never)**. Separately, the most recent file was 2 days old rather than ~1.
**Root cause — two independent, disconnected code paths, both traced to `setInterval`-since-
process-start:**
1. **Status was never persisted.** `BackupService` tracked `lastSuccessAt`/`lastResult`/
`lastErrorAt`/`lastError` as **plain in-process private fields** — set only inside `run()`,
read only by `status()` on the *same running instance*. Nothing wrote them to `site_config` or
anywhere else durable. The actual backup-writing engine (`backup.ts`: consistent copy → encrypt
→ `pruneOldBackups`) is a completely separate code path that only touches the filesystem and
has no notion of this status object. So "7 valid files on disk" and "status says Never" were
never contradictory — they were two unrelated signals, and **any** server restart (deploy,
crash, OOM, host reboot — all routine under `restart: always` in `docker-compose.prod.yml`)
silently reset the in-memory fields to `null` regardless of what had actually happened on disk.
2. **The schedule was measured from process start, not from the last real backup.** The daily
timer was `setInterval(() => backupService.runScheduled(), 24h)` — a fixed 24h period counted
from whenever the *process* last started, not from wall-clock time or from when a backup last
actually succeeded. The exact same restart that wiped the in-memory status also reset this
countdown, which is why the cadence can silently drift or skip past a day with no error ever
surfacing anywhere.
Both symptoms are one cause: **the server process restarted after the Aug 28 backup, and nothing
about this design was built to survive that.**
### Fix (2026-08-30)
- **`packages/db/src/schema.ts`** / migration `0025_backup_last_status.sql` — four new nullable
`site_config` columns: `backup_last_success_at`, `backup_last_result_json`,
`backup_last_error_at`, `backup_last_error`. Same table, same upsert pattern as
`backup_target_dir`/`backup_keep_last`/`backup_keep_daily_days` (migrations 0016/0017).
- **`backup-service.ts`** — `run()` now writes success/error outcomes to these columns (via a
`#persist` upsert helper) instead of private fields; `status()` reads them fresh from the DB on
every call. A brand-new `BackupService` instance (i.e. a fresh process) now sees exactly what
the previous instance last recorded — no more restart amnesia.
- **New `isDue(now, intervalMs = 24h)`** method: due iff `now - backupLastSuccessAt >= 24h` (or
immediately due if no success was ever recorded), computed from the **persisted** timestamp —
never from process uptime.
- **`server.ts`** — the daily `setInterval` was replaced with a **15-minute poll** calling
`runScheduled()`, which now itself no-ops unless `isDue()` is true. This makes the actual backup
cadence immune to restart timing entirely: however often the process happens to restart, the
next backup fires within 15 minutes of 24h having genuinely elapsed since the last real success
— not 24h after whatever moment the process most recently came back up.
- Covered by a new `backup-service.test.ts`: a fresh `BackupService` over the same DB handle
(simulating a restart) sees the prior instance's last success/error and its cleared-on-success
behavior; `isDue()` is exercised directly against injected timestamps rather than real sleeps.
No change to the `BackupStatus` shape returned by `GET /api/backup/status` or to
`BackupSettings.tsx` — this was purely a durability fix underneath the same contract.
## Status
Design settled 2026-06-29; **engine + admin-configured local/mounted target + admin UI BUILT
2026-06-29** (SFTP + restore tooling pending). The target directory is **admin-chosen in the UI**
(`site_config`, migration 0016), not an env var — the on-site admin picks where backups land; only
`BACKUP_KEY` stays a server secret. Resolves the *design* half of [[open-questions]] #5 and the first
build slices; records the key-custody stance that bears on #6 (signing stays decoupled from the TPM) and
#10 (snapshots bloat backups → future exclude toggle). See [[append-only-event-chain]],
[[disk-os-hardening]], [[tpm]], [[fleet-deployment-komodo]], [[reconciliation]].
`BACKUP_KEY` stays a server secret. **Last-success/last-error status + the scheduling cadence are
now restart-durable (migration 0025, 2026-08-30)** — see field bug above. Resolves the *design*
half of [[open-questions]] #5 and the first build slices; records the key-custody stance that bears
on #6 (signing stays decoupled from the TPM) and #10 (snapshots bloat backups → future exclude
toggle). See [[append-only-event-chain]], [[disk-os-hardening]], [[tpm]], [[fleet-deployment-komodo]],
[[reconciliation]].