docs(wiki): log-storm hardening + reset drift guard (2026-07-07 incident)
button-light-indicator: failure backoff + rate-limited logging rationale; app-logs: storm coalescing invariant + --diagnostics wipe; local-dev-workflow and appliance-provisioning §7d: new reset flag table + drift guard; log entry tying all three layers to the ENETUNREACH incident. Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
type: concept
|
||||
tags: [parking, observability, diagnostics, logging, frontend, backend]
|
||||
sources: []
|
||||
updated: 2026-07-04
|
||||
updated: 2026-07-08
|
||||
status: open
|
||||
---
|
||||
|
||||
@@ -62,6 +62,16 @@ column — the failed request, error name, component stack, anything), plus pull
|
||||
logged — that's the recursion we guard). Diagnostics must never break the path they observe.
|
||||
- **Bounded.** Frontend queue capped (drops oldest); message/stack/context clamped per row;
|
||||
ingest batch capped.
|
||||
- **Storm coalescing (2026-07-08).** A line identical to the *last persisted row*
|
||||
(level+source+message+path) arriving within **5 min** of its previous occurrence **updates that
|
||||
row** instead of inserting: `context._repeat` counts the fold, `context._firstAt` keeps the first
|
||||
occurrence, `createdAt` moves to the latest (so the storm stays at the top of the newest-first
|
||||
viewer, which badges it `×N`). A *continuous* storm refreshes the window each hit, so it stays ONE
|
||||
row however long it rages. Motivation: the 2026-07-07 field incident — one unreachable controller
|
||||
(`ENETUNREACH`) produced hundreds of identical error rows per minute, evicting unrelated history
|
||||
(see [[button-light-indicator]] for the send-side fix: retry backoff + rate-limited logging).
|
||||
In-memory last-row cache only (a restart just starts a fresh row); if the row was pruned
|
||||
underneath, it falls through to a fresh insert.
|
||||
|
||||
## Retention (offline appliance ⇒ must be bounded)
|
||||
|
||||
@@ -71,6 +81,10 @@ was 30) **and** keep only the newest `LOG_RETENTION_MAX_ROWS` (default 50 000).
|
||||
(unref'd timer) + once at startup. Both env-configurable. Same "prunable, not precious"
|
||||
durability class as `device_events` — the opposite of the append-only ledger.
|
||||
|
||||
Also wipeable on demand: `reset-db.mjs --diagnostics` (new category 2026-07-08 — `app_logs`
|
||||
previously belonged to NO category and silently survived even `--all`; a drift guard in the script
|
||||
now refuses to run if any table is uncategorized). See [[local-dev-workflow]].
|
||||
|
||||
## Container (stdout) logs — the OTHER log store (2026-07-04)
|
||||
|
||||
`docker logs` is a separate, size-bounded store from `app_logs` — it holds **everything**
|
||||
|
||||
Reference in New Issue
Block a user