docs(camera): Hik event-push gotchas + dead-camera diagnostic method

Captures the hard-won findings from the field session: the WSL source-IP
rewrite + skipSourceIpCheck fix, the boolean-as-string setup bug, the
unreliable "Test" button, the latching httpBroken flag, and Notify-
Surveillance-Center vs HTTP-Alarm-Server.

Adds a "diagnose a non-pushing camera from its OWN state" runbook
(alertStream heartbeat silence, SSH showStatus EventScribe:except, dmesg
RTC/UBIFS, netstat outbound watch) and documents the verified-dead
DS-2CD1043G2-LIU unit (defective event engine, survives factory reset ->
RMA), with the pull+vision fallback.

Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
2026-06-22 12:50:44 +02:00
parent b7300ec080
commit 547061edf9
2 changed files with 61 additions and 0 deletions
+47
View File
@@ -86,3 +86,50 @@ Center**, then **Alarm Settings → Alarm Server**, makes the camera **HTTP-POST
next step once the real payload is known. If the camera emits its own plate (`<plateNumber>`), we next step once the real payload is known. If the camera emits its own plate (`<plateNumber>`), we
can use it as an advisory read directly; otherwise the server hands the attached/pulled frame to can use it as an advisory read directly; otherwise the server hands the attached/pulled frame to
the [[opencv-anpr-service|vision service]] for ANPR. the [[opencv-anpr-service|vision service]] for ANPR.
### Gotchas learned the hard way (2026-06-22 field session)
Several traps surfaced trying to get a real camera to push. In order of how long each cost:
- **WSL rewrites the inbound source IP.** On the dev host (WSL mirrored mode), an inbound LAN packet
arrives at our server with its **source rewritten to the host's own IP** (`10.0.10.203`), not the
camera's. The source-IP guard then rejects every push as a mismatch. Fix: a per-device
**`skipSourceIpCheck`** config flag (a Setup checkbox) that bypasses the IP guard — the signed
ledger + optional Digest remain the real guards. Leave OFF on a normal LAN.
- **The setup checkbox saved booleans as the STRING `"true"`.** The generic config-field form had no
boolean renderer, so a `type:"boolean"` field fell through to a text input. Fixed (checkbox
renderer); the server also coerces `"true"`/`1`/`yes`/`on` defensively.
- **The camera's "Test" button proves almost nothing.** It does a TCP/connectivity probe and reports
"service available" on ANY HTTP reply (even our 404) — it does **not** POST a real event to your
URL. Only a real detection (or the ISAPI `httpHosts/<id>/test`) actually exercises the path.
- **`httpBroken` latches.** Once the camera marks the host broken (from earlier failed deliveries),
it stays `true` across reboots and won't retry. Clear it by **re-PUTting** the httpHost config
(`PUT /ISAPI/Event/notification/httpHosts/1` with `<httpBroken>false</httpBroken>`).
- **"Notify Surveillance Center" ≠ the HTTP Alarm Server** on some firmware (separate upload
channels); and **basic Motion Detection may not POST to the httpHost** while smart/VCA events do.
Always confirm the **Arming Schedule** covers the test time, too (a silent killer).
### Diagnosing a non-pushing camera — read its OWN state, don't guess
Status APIs lie (a `diskfull` error on a camera with **no disk** is a misleading label). The decisive
checks, in order — they need no cooperation from our server:
1. **`GET /ISAPI/Event/notification/alertStream`** (Digest) — the camera's live event bus. A HEALTHY
camera streams periodic multipart heartbeat frames even when idle, and an event line when one
fires. **Total silence (no heartbeat) for 20s = the event subsystem is producing nothing** — the
fault is internal, upstream of any linkage (HTTP host, FTP, alarm stream all die together).
2. **SSH into the camera** (BusyBox "psh" restricted shell; default creds work): `showStatus` →
look for **`EventScribe: except`** (event subsystem in an exception state); `dmesg` → UBIFS
"recovery needed" on every volume (unclean flash) and **`fh rtc get time error ffffffff`** (dead
RTC — a hardware fault; the clock reads 1970, which wedges time-keyed event logic). `ps` shows the
`davinci` app. `df`/`printPart` are NOT available; partition sizes via the UBIFS lines in `dmesg`.
3. **`netstat` on the camera** while triggering — watch for an OUTBOUND line `cam:port → server:3000`.
None appearing = the camera never even attempts the POST.
> **Verified-dead unit (2026-06-22):** a `DS-2CD1043G2-LIU` (V5.8.10, 8 months old) failed all three:
> silent alertStream, `EventScribe: except`, `diskfull` on `Event/triggers`, dead RTC, and **zero
> outbound to `:3000`** across a 3-minute netstat watch — through reboot, basic reset, AND full
> factory reset. Conclusion: **defective event engine (RMA)**, not our integration. Our endpoint was
> verified correct throughout (it accepts + records a POST the instant one actually arrives). When a
> camera's event push is dead, fall back to the **pull + [[opencv-anpr-service|vision]]** path, which
> works with the same camera (it still serves snapshots fine).
+14
View File
@@ -1384,3 +1384,17 @@ setup offers the backend push IP). NOT yet a barrier trigger or DeviceReadEvent
read-bus/ANPR wiring is the next step once the real payload is captured (advisory-only rule still read-bus/ANPR wiring is the next step once the real payload is captured (advisory-only rule still
governs). Tests: hikvision-alarm.test.ts (6: vehicle XML summary, ANPR plate, raw JSON, wrong-IP governs). Tests: hikvision-alarm.test.ts (6: vehicle XML summary, ANPR plate, raw JSON, wrong-IP
404, disabled 404, unknown-device 404); server 109/109, build+lint 14/14. See [[lpr-camera]]. 404, disabled 404, unknown-device 404); server 109/109, build+lint 14/14. See [[lpr-camera]].
## [2026-06-22] debug | Hikvision event-push field session — fixes + a verified-dead camera
Long session getting a real Hik camera to POST events. Server/integration fixes (committed): listen
on ALL methods (camera probes with GET/etc, not just POST); record rejected pushes too (kind
"alarm-rejected" + reason) so "nothing arrived" is never ambiguous; a GET /api/devices/hikvision/
alarms read endpoint; per-device skipSourceIpCheck (WSL mirrored mode REWRITES the inbound source IP
to the host's own, so the source-IP guard rejected every push); and a real checkbox renderer for
type:"boolean" config fields (they were saving the STRING "true"). Then proved — via the camera's
OWN state, not our server — that the specific DS-2CD1043G2-LIU unit has a DEAD event engine: silent
alertStream (no heartbeat), EventScribe:except, diskfull on Event/triggers, dead RTC (rtc get time
error / clock at 1970), and ZERO outbound to :3000 over a 3-min netstat watch, surviving reboot +
basic reset + FULL factory reset. Verdict: defective camera (RMA), not our code. Captured the
diagnostic method (alertStream silence / SSH showStatus / netstat) in [[lpr-camera]]. Fallback for a
dead-push camera: pull + [[opencv-anpr-service|vision]] (the same camera still serves snapshots).