fix(backup): persist last-success/error status; wall-clock-based schedule
BackupService tracked last-success/last-error as plain in-process fields and scheduled the daily backup via setInterval measured from process start — so any server restart (deploy/crash/OOM/reboot, routine under `restart: always`) silently reset the admin UI to "last successful backup: Never" and drifted the actual cadence, independent of whether backups were writing correctly to disk (they were — a real field incident at park-buzi showed 7 valid rotating backups on disk with the status stuck on "Never"). Persist last-success/error to new site_config columns (migration 0025) and add BackupService.isDue(), computed from the persisted timestamp instead of process uptime; server.ts now polls every 15 min and lets isDue() gate the actual run. No API/UI contract change. Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
type: concept
|
||||
tags: [parking, durability, backup, recovery, security, crypto]
|
||||
sources: []
|
||||
updated: 2026-06-29
|
||||
updated: 2026-08-30
|
||||
---
|
||||
|
||||
# Backup & Disaster Recovery
|
||||
@@ -207,12 +207,68 @@ timer + the manual route**. What landed:
|
||||
**SMB/NFS already work** — they're just a mounted path the admin enters as the target. **Deferred to
|
||||
follow-up slices:** an **SFTP** target and a **restore runbook / CLI**.
|
||||
|
||||
## Field bug — "last successful backup: Never" despite valid, rotating backups on disk (found + fixed 2026-08-30)
|
||||
|
||||
**Symptom (park-buzi):** the admin noticed the backup directory held 7 real, correctly-sized,
|
||||
correctly-rotating encrypted backups (`parking-backup-*.sqlite.enc`, retention working exactly as
|
||||
designed) — yet the Backup screen's "Kopja e fundit e suksesshme" (last successful backup) showed
|
||||
**"Asnjëherë" (Never)**. Separately, the most recent file was 2 days old rather than ~1.
|
||||
|
||||
**Root cause — two independent, disconnected code paths, both traced to `setInterval`-since-
|
||||
process-start:**
|
||||
|
||||
1. **Status was never persisted.** `BackupService` tracked `lastSuccessAt`/`lastResult`/
|
||||
`lastErrorAt`/`lastError` as **plain in-process private fields** — set only inside `run()`,
|
||||
read only by `status()` on the *same running instance*. Nothing wrote them to `site_config` or
|
||||
anywhere else durable. The actual backup-writing engine (`backup.ts`: consistent copy → encrypt
|
||||
→ `pruneOldBackups`) is a completely separate code path that only touches the filesystem and
|
||||
has no notion of this status object. So "7 valid files on disk" and "status says Never" were
|
||||
never contradictory — they were two unrelated signals, and **any** server restart (deploy,
|
||||
crash, OOM, host reboot — all routine under `restart: always` in `docker-compose.prod.yml`)
|
||||
silently reset the in-memory fields to `null` regardless of what had actually happened on disk.
|
||||
2. **The schedule was measured from process start, not from the last real backup.** The daily
|
||||
timer was `setInterval(() => backupService.runScheduled(), 24h)` — a fixed 24h period counted
|
||||
from whenever the *process* last started, not from wall-clock time or from when a backup last
|
||||
actually succeeded. The exact same restart that wiped the in-memory status also reset this
|
||||
countdown, which is why the cadence can silently drift or skip past a day with no error ever
|
||||
surfacing anywhere.
|
||||
|
||||
Both symptoms are one cause: **the server process restarted after the Aug 28 backup, and nothing
|
||||
about this design was built to survive that.**
|
||||
|
||||
### Fix (2026-08-30)
|
||||
|
||||
- **`packages/db/src/schema.ts`** / migration `0025_backup_last_status.sql` — four new nullable
|
||||
`site_config` columns: `backup_last_success_at`, `backup_last_result_json`,
|
||||
`backup_last_error_at`, `backup_last_error`. Same table, same upsert pattern as
|
||||
`backup_target_dir`/`backup_keep_last`/`backup_keep_daily_days` (migrations 0016/0017).
|
||||
- **`backup-service.ts`** — `run()` now writes success/error outcomes to these columns (via a
|
||||
`#persist` upsert helper) instead of private fields; `status()` reads them fresh from the DB on
|
||||
every call. A brand-new `BackupService` instance (i.e. a fresh process) now sees exactly what
|
||||
the previous instance last recorded — no more restart amnesia.
|
||||
- **New `isDue(now, intervalMs = 24h)`** method: due iff `now - backupLastSuccessAt >= 24h` (or
|
||||
immediately due if no success was ever recorded), computed from the **persisted** timestamp —
|
||||
never from process uptime.
|
||||
- **`server.ts`** — the daily `setInterval` was replaced with a **15-minute poll** calling
|
||||
`runScheduled()`, which now itself no-ops unless `isDue()` is true. This makes the actual backup
|
||||
cadence immune to restart timing entirely: however often the process happens to restart, the
|
||||
next backup fires within 15 minutes of 24h having genuinely elapsed since the last real success
|
||||
— not 24h after whatever moment the process most recently came back up.
|
||||
- Covered by a new `backup-service.test.ts`: a fresh `BackupService` over the same DB handle
|
||||
(simulating a restart) sees the prior instance's last success/error and its cleared-on-success
|
||||
behavior; `isDue()` is exercised directly against injected timestamps rather than real sleeps.
|
||||
|
||||
No change to the `BackupStatus` shape returned by `GET /api/backup/status` or to
|
||||
`BackupSettings.tsx` — this was purely a durability fix underneath the same contract.
|
||||
|
||||
## Status
|
||||
|
||||
Design settled 2026-06-29; **engine + admin-configured local/mounted target + admin UI BUILT
|
||||
2026-06-29** (SFTP + restore tooling pending). The target directory is **admin-chosen in the UI**
|
||||
(`site_config`, migration 0016), not an env var — the on-site admin picks where backups land; only
|
||||
`BACKUP_KEY` stays a server secret. Resolves the *design* half of [[open-questions]] #5 and the first
|
||||
build slices; records the key-custody stance that bears on #6 (signing stays decoupled from the TPM) and
|
||||
#10 (snapshots bloat backups → future exclude toggle). See [[append-only-event-chain]],
|
||||
[[disk-os-hardening]], [[tpm]], [[fleet-deployment-komodo]], [[reconciliation]].
|
||||
`BACKUP_KEY` stays a server secret. **Last-success/last-error status + the scheduling cadence are
|
||||
now restart-durable (migration 0025, 2026-08-30)** — see field bug above. Resolves the *design*
|
||||
half of [[open-questions]] #5 and the first build slices; records the key-custody stance that bears
|
||||
on #6 (signing stays decoupled from the TPM) and #10 (snapshots bloat backups → future exclude
|
||||
toggle). See [[append-only-event-chain]], [[disk-os-hardening]], [[tpm]], [[fleet-deployment-komodo]],
|
||||
[[reconciliation]].
|
||||
|
||||
Reference in New Issue
Block a user