diff --git a/wiki/concepts/device-status-monitoring.md b/wiki/concepts/device-status-monitoring.md index 8238c09..27f6d1d 100644 --- a/wiki/concepts/device-status-monitoring.md +++ b/wiki/concepts/device-status-monitoring.md @@ -2,7 +2,7 @@ type: concept tags: [parking, device, monitoring, reliability, ui] sources: [] -updated: 2026-06-18 +updated: 2026-06-26 status: open --- @@ -25,6 +25,15 @@ talks only to the adapter interfaces ([[device-adapter-pattern]]), never a drive - **Relays / readers / cameras** → the generic `Device.healthCheck()` **reachability** probe every adapter implements (`ready | degraded | offline`). This is presence/up-ness, not a deep fault model — a relay either answers or it doesn't. + > **Reader health was a LIE until 2026-06-26.** The QR-reader adapter (a PUSH device: it GETs our + > backend on each scan and exposes **no TCP port**) had a hardcoded `healthCheck → { ready, "stub" }`, + > so two genuinely-offline readers still showed **green**. A push device that's silent is + > indistinguishable from a dead one — so claiming `ready` unconditionally is the worst failure + > (false-healthy). Fix: an **optional reader IP** (monitor-only; scans still resolve by serial) + + > an **unprivileged ICMP ping** (`drivers/icmp.ts`, shells `/bin/ping` in SOCK_DGRAM mode — no + > CAP_NET_RAW, no native dep; the booth compose sets `net.ipv4.ping_group_range`). Reply → `ready`, + > no reply → `offline`; **no IP set → `degraded` ("set IP to monitor")**, never a false green. + > Verified on hardware: pinged the real readers on the device VLAN. See [[gee-qr-er80]]. Both collapse to one **traffic-light**: `ready | degraded | offline`, plus a `detail` string. Fail **toward "there's a problem"**, never false-healthy: a probe that throws or times out reads diff --git a/wiki/entities/lpr-camera.md b/wiki/entities/lpr-camera.md index d5ba5a6..19e3d27 100644 --- a/wiki/entities/lpr-camera.md +++ b/wiki/entities/lpr-camera.md @@ -2,7 +2,7 @@ type: entity tags: [parking, hardware, readers, offline-first] sources: [parking-system-architecture] -updated: 2026-06-15 +updated: 2026-06-26 --- # LPR Camera @@ -59,6 +59,42 @@ A **Hikvision** unit ("Camera 20", MAC `94:e1:ac:…`, Hikvision OUI) at `10.0.1 - Reaching it from the WSL dev box required forcing the source address (`config.localAddress`, threaded into the driver) — see [[wsl-dev-networking]] (multi-subnet source-selection trap). +### HTTP 503 "Device Busy" — can be PERSISTENT; the real fix is stream selection (2026-06-26) + +The snapshot endpoint returns **HTTP 503** with the ISAPI body `statusCode 2` / `"Device Busy"` / +`subStatusCode deviceBusy` (occasionally **500**). It comes in two flavours, and they need different +fixes — **don't assume it's a momentary blip**: + +- **Transient** — the encoder is briefly occupied (another snapshot in flight, a stream starting). + Clears on retry within a frame or two. +- **Persistent** — the **MAIN-stream encoder is saturated** and 503s on EVERY main-stream snapshot. + Confirmed on hardware (**DS-2CD1047G3H-LIU**, 2026-06-26): `channels/101/picture` → 503 on five + consecutive probes 800 ms apart, while **`channels/102/picture` (the SUB stream) → 200 every time**, + a clean ~15 KB JPEG. So the path/API was correct (the camera answered with a structured Hikvision + status); the main encoder was simply never free. A retry loop **cannot** fix this — it just delays + the failure. + +**The fix that actually works: snapshot from the SUB stream.** The Hikvision ISAPI channel id is +`` (e.g. ch1 main = `101`, ch1 **sub = `102`**). The driver now has a **`stream` +config field** (`1` = main, default for back-compat; `2` = sub). Set the G3H camera to **Sub (02)** in +the setup form → its status flips `degraded → ready` (verified live: pulled a 14.7 KB JPEG in ~87 ms). +The sub-stream is also the better fit for snapshot/ANPR anyway (smaller/faster; doesn't contend with +live-view/recording for the main encoder). + +Two more complementary mitigations (both BUILT, for the *transient* case): +1. **Don't cause concurrent busy.** On a vehicle entry two server paths used to snapshot the same + camera at once (the ANPR bridge + the advisory `snapshotAsync`); the 2nd concurrent GET drew a 503. + They now share ONE pull via `captureSnapshotShared` (deviceId-keyed, `apps/server/src/snapshot.ts`) + — the main cause of the slow 2026-06-25 subscriber entry. See [[lane-presence-and-anpr-entry]]. +2. **Retry a transient one.** `HttpCamera.captureSnapshot` retries 503/500 with a short linear backoff + (250/500/750 ms, ≤4 attempts), then fails naming it `(device busy)`; it does NOT retry 401/404 + (config errors won't self-heal). This recovers a momentary blip but, by design, still fails a + PERSISTENTLY-busy main stream — the cue to switch that camera to the sub-stream. + +Covered by `packages/devices/src/drivers/camera.test.ts` (retry behaviour + the main/sub path +selection). `healthCheck()` deliberately reports a live 503 as `degraded` (it surfaces a genuinely +saturated main stream rather than hiding it behind a retry). + ## Camera PUSH — "Alarm Server" event notifications (2026-06-22) Separate from the **pull** snapshot path above: newer Hikvision firmware can **push** an event to diff --git a/wiki/log.md b/wiki/log.md index ef9fc21..2464be4 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -1636,3 +1636,39 @@ booth `ModuleNotFoundError` is a STALE image (fix: `booth.sh update`). **Fix:** `dev`/`start`/`recognize` now `uv sync --extra alpr &&` first (self-healing), `.env` set to `fast_alpr`, + a `dev:stub` escape hatch. Restored real ANPR locally (`/health` → `fast_alpr` ready, model loaded from cache, no download). Documented in [[vision-service-packaging]] ("Two runtimes, one fragile"). + +## [2026-06-26] fix | Hikvision snapshot 503 "Device Busy" — stream selection + retry + Alarm URL helper +Three camera fixes. (1) **503 Device Busy — the REAL fix is stream selection.** First framed as +"transient, just retry" — WRONG for this camera. Hardware probe of **DS-2CD1047G3H-LIU** (10.0.10.13): +`channels/101/picture` (MAIN) → 503 `deviceBusy` on 5 consecutive probes 800ms apart, while +`channels/102/picture` (SUB) → 200 clean JPEG every time. The main encoder is PERSISTENTLY saturated; +a retry loop can't fix it. Added a **`stream` config field** to the Hikvision driver (1=main default +for back-compat, 2=sub; ISAPI id ``). Verified live: setting the camera to Sub flips +its status degraded→ready (14.7KB JPEG in ~87ms). (2) **Transient retry** (still useful for a genuine +momentary blip + the de-dup case): `HttpCamera.captureSnapshot` retries 503/500 with linear backoff +(250/500/750ms ×4), fails naming it `(device busy)`, does NOT retry 401/404. Plus the already-landed +`captureSnapshotShared` removing concurrent self-collision. `healthCheck` reports a live 503 as +`degraded` (surfaces a saturated main stream rather than hiding it). Covered by `camera.test.ts` +(10 tests: retry + main/sub path). (3) **Alarm Server URL helper:** the camera setup form now generates the camera's Alarm +Settings (Destination IP / URL / Protocol / Port) ready to paste, so the operator never hunts the +deviceId or memorises the endpoint. CRUCIAL: host/port come from the **backend address on the camera's +subnet** (`backendIpForDevice` + server port, the same probe the push-IP picker uses) — NOT +`window.location.origin` (the SPA's dev/proxy origin, which would wrongly say `localhost:5173`). +Verified live: matches the on-camera config field-for-field (10.0.10.203 / …/event / HTTP / 3000). +Shows a "save first" (needs a deviceId) then "test first" (needs the resolved backend IP) hint. +Documented in [[lpr-camera]] ("503 Device Busy"). Devices 6 new tests; server 168 green. + +## [2026-06-26] fix | QR reader status was a LIE (hardcoded "ready") → real ICMP liveness +Two genuinely-OFFLINE QR readers showed GREEN in the status bar. Cause: the QR-reader adapter +(`StubReader`) had `healthCheck → { ready, "stub" }` hardcoded — it never probed anything. These are +PUSH devices (scan → GET our backend, resolve by serial) that expose **no TCP port**, so a connect +probe (cameras/printers) has nothing to hit; the stub "solved" that by lying. False-healthy is the +worst failure for a status bar. Fix: an **optional reader IP** (monitor-ONLY — scans still resolve by +serial, operation unchanged) + an **unprivileged ICMP ping** (`drivers/icmp.ts`: shells `/bin/ping` +`-c1`, exit-0 = reply; no native dep, no CAP_NET_RAW). `healthCheck`: IP replies → `ready`, no reply → +`offline`, **no IP → `degraded` ("set IP to monitor")** (never a false green). Booth compose +(`docker-compose.prod.yml`) sets `net.ipv4.ping_group_range=0 2147483647` so `/bin/ping` works +unprivileged for the non-root container user. Verified on hardware: the readers (10.0.10.7/.8) answer +ICMP on the device VLAN (eth1) — distinct MACs — and the UI Test connection shows "● ready — ping +10.0.10.7". (NB: an earlier "offline" reading was a WSL wrong-route artifact, not the readers.) Covered +by `reader.test.ts` (4 tests). Documented in [[device-status-monitoring]]. Devices +4 tests, all green.