3a176c5cc8
Full live investigation of the persistent 503 "deviceBusy" on main-stream ISAPI snapshots (10.0.10.13): ruled out config (byte-identical to a working sibling model), ruled out firmware age (reproduced on both the original V5.8.11 and current V5.11.0 builds, ~15 months apart), and ruled out real resource contention (a full channel-ID sweep shows every ID fails identically except the one hardcoded working value, including nonexistent channels) — pointing at a broken/incomplete ISAPI snapshot handler that mislabels itself as "busy," not a real encoder ceiling. RTSP main-stream frame-grab was confirmed as a working route around it, but given the bug and the sub-stream's real-world plate-read accuracy problems, the owner decided to replace the DS-2CD1047G3H-LIU units rather than carry an ffmpeg/RTSP dependency to work around vendor firmware. Ingested the vendor datasheet as a source page along the way. Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
401 lines
29 KiB
Markdown
401 lines
29 KiB
Markdown
---
|
||
type: entity
|
||
tags: [parking, hardware, readers, offline-first]
|
||
sources: [parking-system-architecture, DS-2CD1047G3H-LIU]
|
||
updated: 2026-08-23
|
||
---
|
||
|
||
# LPR Camera
|
||
|
||
License-plate-recognition camera. For **casual/transient** vehicles, the **plate acts as ticket +
|
||
an independent record**. (See [[parking-system-architecture]] §8, §9.)
|
||
|
||
> **Superseded direction (2026-06-15):** recognition now runs **host-side** on snapshots from
|
||
> ordinary Hikvision/Dahua cameras via the [[opencv-anpr-service]], **not** on a dedicated edge-AI
|
||
> LPR camera — see [[vision-service]]. The edge-AI camera below is kept as the original assumption /
|
||
> a fallback option, but is no longer the planned path. The host-side service also does **vehicle
|
||
> verification** (anti-plate-spoofing), which an edge-LPR camera does not.
|
||
|
||
- **Edge AI (original assumption)**: recognition runs **on-device**, so it keeps working with no
|
||
internet — fits [[offline-first]].
|
||
- It's a **host-side** identity source: only the host sees the read; the host decides and
|
||
commands the relay open (the [[uhppote-controller]] is demoted to a commanded relay for that
|
||
lane). See [[entry-exit-readers]].
|
||
- Being host-in-the-loop is **good for fraud detection** — you get two independent records (the
|
||
host's signed [[append-only-event-chain]] entry + the controller's remote-open event) that
|
||
should reconcile one-to-one; any mismatch is an anomaly.
|
||
- Mounting: within ~15° of vehicle travel at a controlled chokepoint for best reads.
|
||
|
||
## Snapshot driver (entry/exit fraud-control record)
|
||
|
||
Separate from edge-AI LPR: the camera driver (`packages/devices/src/drivers/camera.ts`) does
|
||
**snapshot-on-event** — the host pulls a still over HTTP when an entry/exit fires and stores it,
|
||
referenced from the signed [[append-only-event-chain]] entry as an independent record. The camera
|
||
**pulls, it does not push** — so it is NOT `pushesToBackend` and the setup wizard correctly hides
|
||
the "Backend push IP" field for it (gated on the driver's `pushesToBackend` flag; only
|
||
[[dingtian-relay]] sets it).
|
||
|
||
- **Hikvision** uses **ISAPI**: `GET /ISAPI/Streaming/channels/<id>/picture` (`101` = ch1 main
|
||
stream) with **HTTP Digest** auth. The "Enable Hikvision-CGI" toggle (Network → Advanced →
|
||
Integration Protocol) is a *different* legacy CGI surface — **not** needed for ISAPI.
|
||
- **Dahua** uses CGI: `GET /cgi-bin/snapshot.cgi?channel=<n>` (0-based channel; the wizard's
|
||
1-based channel is decremented).
|
||
|
||
**Driver / storage boundary:** the driver FETCHES the image bytes (client-side HTTP Digest in
|
||
`drivers/http-digest.ts`) and returns them on `Snapshot.bytes`; **storage is the caller's job**
|
||
(the future entry/exit flow stores the bytes + mints a durable `imageRef`). This keeps the device
|
||
adapter free of any filesystem/blob-store dependency. `healthCheck()` is honest — it actually pulls
|
||
a frame (exercising reachability + auth + path/channel in one shot), not a fake `ready/stub`.
|
||
|
||
### Verified on hardware (2026-06-15)
|
||
|
||
A **Hikvision** unit ("Camera 20", MAC `94:e1:ac:…`, Hikvision OUI) at `10.0.10.121`, creds
|
||
`admin` / `admin123` (Digest), TCP 80:
|
||
|
||
- Initial `curl` test confirmed the ISAPI path returns a 2688×1520 JPEG (~306 KB).
|
||
- The **real driver** (no longer a stub) was then run end to end against it:
|
||
`healthCheck()` → `ready` (pulled a frame), `captureSnapshot()` → valid `image/jpeg`, ~322 KB,
|
||
correct JPEG magic. Digest handshake works through `HttpCamera`.
|
||
- Reaching it from the WSL dev box required forcing the source address (`config.localAddress`,
|
||
threaded into the driver) — see [[wsl-dev-networking]] (multi-subnet source-selection trap).
|
||
|
||
### HTTP 503 "Device Busy" — can be PERSISTENT; the real fix is stream selection (2026-06-26)
|
||
|
||
The snapshot endpoint returns **HTTP 503** with the ISAPI body `statusCode 2` / `"Device Busy"` /
|
||
`subStatusCode deviceBusy` (occasionally **500**). It comes in two flavours, and they need different
|
||
fixes — **don't assume it's a momentary blip**:
|
||
|
||
- **Transient** — the encoder is briefly occupied (another snapshot in flight, a stream starting).
|
||
Clears on retry within a frame or two.
|
||
- **Persistent** — the **MAIN-stream encoder is saturated** and 503s on EVERY main-stream snapshot.
|
||
Confirmed on hardware (**DS-2CD1047G3H-LIU**, 2026-06-26): `channels/101/picture` → 503 on five
|
||
consecutive probes 800 ms apart, while **`channels/102/picture` (the SUB stream) → 200 every time**,
|
||
a clean ~15 KB JPEG. So the path/API was correct (the camera answered with a structured Hikvision
|
||
status); the main encoder was simply never free. A retry loop **cannot** fix this — it just delays
|
||
the failure.
|
||
|
||
> **Sharper finding (2026-06-27, same DS-2CD1047G3H-LIU).** Re-probed `10.0.10.13` directly after a
|
||
> camera reboot with every web/live-view connection closed. **Sub (102): 10/10 rapid back-to-back
|
||
> pulls → 200** (~50 ms, ~14.7 KB) — flawless even with NO delay, harder than the live ANPR cadence.
|
||
> **Main (101): 3/3 → 503 in ~20 ms** — an *instant* reject, not a timeout. So main isn't merely
|
||
> "saturated/busy" on this model — its snapshot endpoint is **structurally unavailable**; **sub (102)
|
||
> is mandatory**, not just preferable. SEPARATELY, the camera 503s on **any** stream when its
|
||
> **connection slots are exhausted** — a human holding the web UI / live-view, or parallel
|
||
> main-stream experiments, consume slots; a **reboot clears stuck slots**. This was the actual cause
|
||
> of the **2026-06-27 "subscribers auto-enter but don't auto-exit"** scare: manual main-stream
|
||
> testing held the exit camera's slots → the system's sub-stream snapshot pulls got "Device Busy" →
|
||
> the ANPR exit read never got a frame → no auto-exit. **NOT a code/flow bug** — the subscription
|
||
> exit logic, camera→relay binding, and `stream: "2"` config were all correct (auto-exit/entry pairs
|
||
> were clean before the test storm and after the reboot). (Also recorded as the LLM memory
|
||
> `g3h-main-stream-snapshot-503`.)
|
||
|
||
**The fix that actually works: snapshot from the SUB stream.** The Hikvision ISAPI channel id is
|
||
`<channel><stream>` (e.g. ch1 main = `101`, ch1 **sub = `102`**). The driver now has a **`stream`
|
||
config field** (`1` = main, default for back-compat; `2` = sub). Set the G3H camera to **Sub (02)** in
|
||
the setup form → its status flips `degraded → ready` (verified live: pulled a 14.7 KB JPEG in ~87 ms).
|
||
The sub-stream is also the better fit for snapshot/ANPR anyway (smaller/faster; doesn't contend with
|
||
live-view/recording for the main encoder).
|
||
|
||
Two more complementary mitigations (both BUILT, for the *transient* case):
|
||
1. **Don't cause concurrent busy.** On a vehicle entry two server paths used to snapshot the same
|
||
camera at once (the ANPR bridge + the advisory `snapshotAsync`); the 2nd concurrent GET drew a 503.
|
||
They now share ONE pull via `captureSnapshotShared` (deviceId-keyed, `apps/server/src/snapshot.ts`)
|
||
— the main cause of the slow 2026-06-25 subscriber entry. See [[lane-presence-and-anpr-entry]].
|
||
2. **Retry a transient one.** `HttpCamera.captureSnapshot` retries 503/500 with a short linear backoff
|
||
(250/500/750 ms, ≤4 attempts), then fails naming it `(device busy)`; it does NOT retry 401/404
|
||
(config errors won't self-heal). This recovers a momentary blip but, by design, still fails a
|
||
PERSISTENTLY-busy main stream — the cue to switch that camera to the sub-stream.
|
||
|
||
Covered by `packages/devices/src/drivers/camera.test.ts` (retry behaviour + the main/sub path
|
||
selection). `healthCheck()` deliberately reports a live 503 as `degraded` (it surfaces a genuinely
|
||
saturated main stream rather than hiding it behind a retry).
|
||
|
||
### Main-stream ISAPI snapshot 503 is model-specific, not config — and RTSP routes around it (2026-08-23)
|
||
|
||
Live comparison, same site (park-buzi), same day, both cameras reachable via the site's port
|
||
forwards (`park-buzi.msai.al:8081` / `:8082`) and directly on the LAN (`10.0.10.13` = the `:8082`
|
||
unit):
|
||
|
||
| | `:8081` (works) | `:8082` = `10.0.10.13` (503s) |
|
||
|---|---|---|
|
||
| Model | **DS-2CD1043G2-LIU** | **DS-2CD1047G3H-LIU** |
|
||
| Firmware | V5.8.10 | V5.8.11 |
|
||
| Channel 101 config | 2560×1440, VBR, 6144 Kbps cap, 20fps | **identical** — 2560×1440, VBR, 6144 Kbps cap, 20fps |
|
||
| `SmartCodec` | disabled | disabled |
|
||
| `GET /ISAPI/Streaming/channels/101/picture` | **200**, valid JPEG | **503**, `statusCode 2 / deviceBusy` (3/3 retries, instant) |
|
||
| `GET /ISAPI/Streaming/channels/102/picture` (sub) | — | **200**, valid JPEG |
|
||
|
||
Channel-101 config is **byte-identical** between the two units (bitrate, resolution, frame rate,
|
||
SmartCodec) — this rules out "misconfigured over some ceiling" definitively; the only things that
|
||
differ are model + firmware. Combined with the [[#Source: HIKVISION DS-2CD1047G3H-LIU-F datasheet|
|
||
vendor datasheet]] fact that the G3H's **main stream has no MJPEG option** (sub-stream does), the
|
||
working theory is that this SKU's snapshot codepath has to transcode a live H.264/H.265 frame into
|
||
JPEG on demand for main, and its firmware/encoder can't do that reliably at this resolution —
|
||
while sub can serve JPEG more natively. **Treat this as a `DS-2CD1047G3H-LIU`-model limitation
|
||
(this firmware line), not a config or ISAPI-usage bug** — matches every earlier finding on this
|
||
same unit (`10.0.10.13`) in the sections below, now cross-confirmed against a working sibling model
|
||
on the same network with identical settings.
|
||
|
||
**RTSP main-stream frame-grab works and routes around it entirely**, confirmed live against
|
||
`10.0.10.13`:
|
||
|
||
```bash
|
||
ffmpeg -rtsp_transport tcp -y \
|
||
-i "rtsp://admin:<pw>@10.0.10.13:554/Streaming/Channels/101" \
|
||
-frames:v 1 -update 1 snapshot.jpg
|
||
```
|
||
|
||
Returned a valid 2560×1440 JPEG (94 KB) on the first try — same camera, same main-stream
|
||
resolution the ISAPI endpoint 503s on. This makes sense mechanically: RTSP just taps the H.264
|
||
stream the encoder is **already producing continuously** for live-view/recording; there's no
|
||
on-demand "pause and re-encode as standalone JPEG" step for the firmware to choke on, unlike the
|
||
ISAPI snapshot path. Port 554 is open on the LAN (`10.0.10.13`) but **not** forwarded through the
|
||
site's public port-forward (`park-buzi.msai.al` only exposes the HTTP/ISAPI ports, consistent with
|
||
[[network-isolation|the camera/controller network staying LAN-only]] — RTSP was only reachable
|
||
from inside the site network, never tested through the public forward).
|
||
|
||
**Not yet built**: `packages/devices/src/drivers/camera.ts` is HTTP-Digest/ISAPI only today: no
|
||
RTSP client, no `ffmpeg` child-process dependency. Adding an RTSP fallback (or RTSP-first path for
|
||
cameras that report persistent `deviceBusy` on ISAPI main) is a real architectural addition — new
|
||
process-spawn dependency, RTSP auth handling, transport selection (TCP confirmed working; UDP
|
||
untested) — not implemented as of this writing.
|
||
|
||
**Escalated from "nice to have" to a real requirement (2026-08-23): the sub-stream (768×432) is
|
||
too weak for reliable plate reads** — it fails to read plates "from time to time" in practice, so
|
||
sub-stream-only is not an acceptable permanent mitigation for this camera; RTSP-for-main is needed
|
||
for ANPR accuracy, not just for a higher-res evidence photo.
|
||
|
||
**Firmware update tested and RULED OUT as the fix (2026-08-23).** Before building the RTSP path,
|
||
checked whether this was simply a day-one bug: the camera shipped on `V5.8.11` build 250415 —
|
||
confirmed via the official Hikvision release note to be **the very first H13U firmware build that
|
||
added support for the DS-2CD1XX7G3H-LIU family at all** ("Newly add 1 series 4MP fixed-focus
|
||
cameras: DS-2CD1XX7G3H-LIU"), a plausible day-one-bug candidate. Upgraded live to **`V5.11.0` build
|
||
260701** (over a year of firmware progress, incl. an intermediate `V5.8.21_SP1` release explicitly
|
||
noting "Fix network and image potential bugs to enhance device stability"). **Result: NO CHANGE.**
|
||
Post-upgrade, `channels/101/picture` still returns `HTTP 503 / statusCode 2 / deviceBusy`, 5/5
|
||
consecutive attempts, byte-identical error body to pre-upgrade. Sub-stream (`102`) still healthy
|
||
(200, ~15KB) — camera is fine post-upgrade, just this one limitation persists.
|
||
|
||
**Superseded finding, below: "durable hardware/encoder ceiling" was the wrong framing.** At this
|
||
point in the investigation it looked like a real capacity limit (reproduced across ~15 months of
|
||
firmware). The channel-sweep test below shows that's not what's actually happening.
|
||
|
||
### The real cause: a broken/incomplete ISAPI snapshot handler, not "busy" (2026-08-23)
|
||
|
||
`deviceBusy` never meant "busy." Swept every channel/stream ID against the snapshot endpoint in
|
||
one sitting, including IDs that don't exist on this camera at all:
|
||
|
||
| channel | `GET .../channels/<id>/picture` |
|
||
|---|---|
|
||
| 1 | 503 `deviceBusy` |
|
||
| 100 | 503 `deviceBusy` |
|
||
| **101** (real main) | **503 `deviceBusy`** |
|
||
| **102** (real sub) | **200 OK** |
|
||
| 103 | 503 `deviceBusy` |
|
||
| 201 (channel 2 doesn't exist — single-channel camera) | 503 `deviceBusy` |
|
||
| 999 (garbage) | 503 `deviceBusy` |
|
||
|
||
**Every ID fails identically except exactly `102`.** A genuinely busy/saturated encoder would not
|
||
succeed on one specific value and fail the same way on nonexistent channel IDs — a real resource
|
||
contention error would 404 or behave differently on garbage input, not return the identical `Device
|
||
Busy` XML body regardless of whether the target exists. This is the signature of a **generic
|
||
fallback error path**: the firmware's snapshot handler appears to only be correctly wired for
|
||
`102` (the one channel/stream combination Hikvision evidently tested for this SKU) and returns a
|
||
stock, misleading `deviceBusy` for every other case — valid main-stream `101` included. It is a
|
||
**firmware bug that mislabels itself as resource contention**, not a real capacity ceiling — which
|
||
also fits the firmware-upgrade non-result above (a wrong-code-path bug doesn't get fixed by
|
||
"more capacity," so no firmware version fixing it would be surprising).
|
||
|
||
**Decision (2026-08-23): replace this camera line rather than build around it.** RTSP main-stream
|
||
capture is proven to work (see above) and could still be built as a `camera.ts` addition, but given
|
||
the ISAPI snapshot path is flatly broken for anything but one hardcoded channel value, and the
|
||
site's actual need (reliable plate reads — sub-stream alone isn't accurate enough) requires
|
||
full-resolution captures, the owner chose to swap out the `DS-2CD1047G3H-LIU` units rather than
|
||
carry a `ffmpeg`/RTSP dependency to route around a vendor firmware bug. The working `DS-2CD1043G2-
|
||
LIU` (`:8081` in the comparison above) has no such issue — ISAPI main-stream snapshot works
|
||
natively — and is the reference model for replacements. RTSP-frame-grab remains documented above
|
||
as a viable fallback if a `G3H`-family camera is ever unavoidable.
|
||
|
||
## Clock sync — the 1970 power-cut reset (built 2026-07-07)
|
||
|
||
Field observation (park-buzi): after a power cut these cameras come back with their clock at the
|
||
**1970 epoch** (no/dead RTC battery, no NTP) and stay there until a human logs into the web UI —
|
||
Hikvision's web login silently pushes the browser clock. A wrong camera clock corrupts the OSD
|
||
timestamp burned into every snapshot (the evidence trail) and the times on ANPR pushes.
|
||
|
||
Built: the HOST is the site's time authority (offline-first — no NTP infra dependency), and the
|
||
device monitor re-syncs each Hikvision camera over the same Digest-auth ISAPI used for snapshots:
|
||
|
||
- **Trigger:** the camera's **offline → ready transition** (exactly the power-restored moment) +
|
||
a **24h backstop**; the attempt timestamp is stamped BEFORE the async call, so a failing camera
|
||
retries at backstop cadence, never every 8s poll.
|
||
- **Mechanics** (`HikvisionCamera.syncClock`): `GET /ISAPI/System/time`, parse `localTime`; drift ≤
|
||
60s → leave alone. Beyond that → `PUT /ISAPI/System/time` with `timeMode=manual`, the site's
|
||
wall-clock now WITH explicit utc offset (`localIsoWithOffset(siteTz)`, e.g.
|
||
`2026-07-07T15:30:22+02:00` — the offset makes the instant unambiguous), and the camera's own
|
||
`timeZone` string **echoed back verbatim** (we correct the clock, never fight its tz/DST config).
|
||
An unparseable camera reply counts as infinite drift → sync.
|
||
- **Visibility:** a sync after a big jump (>1h — the power-cut signature) logs at **warn**
|
||
(persisted to app_logs); small corrections log info. Failures log warn.
|
||
- **Scope:** Hikvision only (`isClockSyncable` capability guard); the Dahua driver's CGI has no
|
||
ISAPI time endpoint — a Dahua clock sync would be its own driver work.
|
||
- Rejected alternative: camera-side **NTP against the booth** (chrony on the appliance). More
|
||
standard, but adds a provisioning dependency per booth and the camera polls NTP on ITS schedule
|
||
— a freshly power-cycled camera could still sit at 1970 for a while, which is precisely the
|
||
moment that matters.
|
||
|
||
## Camera PUSH — "Alarm Server" event notifications (2026-06-22)
|
||
|
||
Separate from the **pull** snapshot path above: newer Hikvision firmware can **push** an event to
|
||
us. Under **Event → Smart/VCA** (e.g. line crossing / intrusion / "Vehicle Detection") the unit
|
||
exposes **Detection Target: Human / Vehicle** — selecting **Vehicle** + **Notify Surveillance
|
||
Center**, then **Alarm Settings → Alarm Server**, makes the camera **HTTP-POST an
|
||
`EventNotificationAlert`** to a URL we host on each detection. Same machine-call shape as the
|
||
[[dingtian-relay]] Input Link push — no polling.
|
||
|
||
- **Ingress:** `POST /api/devices/hikvision/:deviceId/event` (`apps/server/src/routes/hikvision-alarm.ts`).
|
||
**Source-IP guarded** (must come from the device's configured `host`) + **optional HTTP Digest**
|
||
(some firmware can't authenticate the Alarm Server call → source-IP only). NOT behind the SPA
|
||
cookie/CSRF (it's a device call), exactly like the Dingtian push.
|
||
- **Config:** added to the `hikvision` driver — `alarmPushEnabled` (bool), `pushUser`/`pushPassword`
|
||
(optional Digest). The driver is now `pushesToBackend: true`, so first-run setup offers the backend
|
||
push IP. Point the camera's Alarm Server at `http://<backend-ip>:<port>/api/devices/hikvision/<deviceId>/event`.
|
||
- **Discovery-first:** the endpoint is **permissive** — accepts ANY content-type as raw bytes (event
|
||
XML, multipart-with-JPEG, or JSON; Hik's format varies by model/firmware), records the **verbatim
|
||
body** as a `kind:"alarm"` device_event, and best-effort extracts `eventType` / `target` / `plate`
|
||
/ `dateTime` / `channelID`. The point of this first cut is to **see exactly what a given camera
|
||
sends** (inspect via `GET /api/events` or the server log) before wiring it to the read bus.
|
||
- **Not yet a barrier trigger.** It records + breadcrumbs only; it does NOT emit a `DeviceReadEvent`
|
||
or open anything. A plate read is **advisory, never the sole reason** a barrier opens
|
||
([[append-only-event-chain]], [[opencv-anpr-service]]). Two consumers were since designed off this
|
||
same vehicle event — see **[[lane-presence-and-anpr-entry]]**: (a) BUILT — advisory lane busy/free
|
||
booth lights; (b) BUILT (2026-06-22) — the ANPR "bridge" (`anpr-entry.ts`) that snapshots → ANPR →
|
||
emits a `kind:"plate"` read for a SUBSCRIBER match through the existing gated flow (a small
|
||
`apps/server` handler class, not a service). If the camera ever emits its own `<plateNumber>` we'd
|
||
use it directly; this `DS-2CD1043G2`
|
||
does not, so the server pulls the frame and hands it to the [[opencv-anpr-service|vision service]].
|
||
|
||
### "Subscribers auto-enter but don't auto-exit" — a 4-layer CAMERA fault, NOT our code (2026-06-27)
|
||
|
||
A long debugging session on the **DS-2CD1047G3H-LIU** exit camera (`10.0.10.13`, exit-lane). The
|
||
symptom: subscribers (e.g. Caca) auto-entered via ANPR fine but **never auto-exited**. **Every
|
||
assumption about *our* code was wrong; all four real causes were camera-side.** Method that finally
|
||
cracked it: a **dumb HTTP sink** (`scratch-camera-sink.py`) the camera's Alarm Server was pointed
|
||
at, to see — verbatim — what the camera actually sends, independent of our app's parsing/acceptance.
|
||
|
||
The wrong turns, and what was actually true:
|
||
|
||
1. **Wrong assumption: "the exit camera 503s, so harden the snapshot retry / reduce load."** The 503
|
||
storm in the data was mostly **manual main-stream testing**: on this G3H, `channels/101/picture`
|
||
(MAIN) **503s instantly every time** — structurally unavailable, not "busy" — while `102` (SUB)
|
||
serves 10/10 rapid pulls cleanly. AND the camera **503s on *any* stream when its connection slots
|
||
are exhausted** (a held web UI / live-view, parallel experiments); a **reboot clears stuck slots**.
|
||
So the snapshot retry/flow code was fine. See [[#HTTP 503 "Device Busy"]] + the
|
||
`g3h-main-stream-snapshot-503` memory. (The exit/subscription FLOW logic, camera→relay binding,
|
||
and `stream:"2"` config were all correct the whole time — verified: clean entry/exit pairs before
|
||
the test storm, and after the reboot.)
|
||
|
||
2. **The actual blocker #1 — the exit camera never POSTed at all.** `alarmPushEnabled=true` in our
|
||
config, but `10.0.10.13` had sent **ZERO** alarms ever (entry cam `.12`: 1126). The sink received
|
||
nothing from `.13`; our `/event` endpoint logged no rejections either → the camera wasn't sending.
|
||
Cause found in the camera's own **Diagnose Information** dump: **`Main Db is broken` /
|
||
`db_restore failed` / `Going to reset cfg`** — the camera's internal config DB (`ipc_db`) was
|
||
**CORRUPT**, plus repeated reboots. A broken config DB means the event→linkage→push pipeline can't
|
||
reliably read its own config, so it silently never POSTs. **Fix: factory-reset the camera** (rebuilds
|
||
`ipc_db`), then reconfigure. (If corruption returns after a clean reset → failing flash → RMA.)
|
||
|
||
3. **Wrong assumption: "missing gateway/DNS blocks the push."** A documented Hikvision note says a
|
||
gateway is needed even same-subnet — but here it was **inverted**: the WORKING cam `.12` has NO
|
||
gateway/DNS; the broken `.13` HAD both. Red herring. Gateway/DNS was not the cause.
|
||
|
||
4. **The actual blocker #2 — after reset, the camera pushed PLAIN MOTION, not vehicle.** Post-reset
|
||
`.13` POSTed `<eventType>VMD</eventType>` with **no target tag**. The backend gates on
|
||
`target == "vehicle"` (`hikvision-alarm.ts` `isVehicleActive`, matches `<targetType>` /
|
||
`<detectionTarget>` / `<objectType>`), so a plain-motion push is **ignored** → bridge never fires.
|
||
The working `.12` sends `eventType=VMD` **with `target=vehicle`**. The difference is the AcuSense
|
||
**Detection-Target = Vehicle** filter ON the Motion event — defaulted OFF after factory reset.
|
||
**Fix: enable Vehicle target classification** on `.13`'s Motion Detection. Confirmed live: a real
|
||
drive-through then POSTed `<eventType>VMD</eventType> … <targetType>vehicle</targetType>` — exactly
|
||
what the backend needs. (So "VMD/Motion" IS the right event for this camera class; it's the *target
|
||
filter* that matters, not switching to a different event type.)
|
||
|
||
**Takeaways:** (a) a camera that's silently not-pushing looks identical to "fine" in our logs — the
|
||
sink-to-prove-it-sends method is the fastest disambiguator; flagged as an observability gap (a
|
||
`alarmPushEnabled=true` camera with 0 pushes ever should be a surfaced condition, like the
|
||
[[device-status-monitoring|reader-liveness]] fix). (b) For ANPR the camera must send a **vehicle
|
||
`targetType`** — verify the push body, not just the UI toggles. (c) Hikvision config-DB corruption
|
||
is real; factory reset is the cure. None of this was a code bug. See
|
||
[[lane-presence-and-anpr-entry]] for the push→bridge→exit path.
|
||
|
||
### Gotchas learned the hard way (2026-06-22 field session)
|
||
|
||
Several traps surfaced trying to get a real camera to push. In order of how long each cost:
|
||
|
||
- **WSL rewrites the inbound source IP.** On the dev host (WSL mirrored mode), an inbound LAN packet
|
||
arrives at our server with its **source rewritten to the host's own IP** (`10.0.10.203`), not the
|
||
camera's. The source-IP guard then rejects every push as a mismatch. Fix: a per-device
|
||
**`skipSourceIpCheck`** config flag (a Setup checkbox) that bypasses the IP guard — the signed
|
||
ledger + optional Digest remain the real guards. Leave OFF on a normal LAN.
|
||
- **The setup checkbox saved booleans as the STRING `"true"`.** The generic config-field form had no
|
||
boolean renderer, so a `type:"boolean"` field fell through to a text input. Fixed (checkbox
|
||
renderer); the server also coerces `"true"`/`1`/`yes`/`on` defensively.
|
||
- **The camera's "Test" button proves almost nothing.** It does a TCP/connectivity probe and reports
|
||
"service available" on ANY HTTP reply (even our 404) — it does **not** POST a real event to your
|
||
URL. Only a real detection (or the ISAPI `httpHosts/<id>/test`) actually exercises the path.
|
||
- **`httpBroken` latches.** Once the camera marks the host broken (from earlier failed deliveries),
|
||
it stays `true` across reboots and won't retry. Clear it by **re-PUTting** the httpHost config
|
||
(`PUT /ISAPI/Event/notification/httpHosts/1` with `<httpBroken>false</httpBroken>`).
|
||
- **"Notify Surveillance Center" ≠ the HTTP Alarm Server** on some firmware (separate upload
|
||
channels). Always confirm the **Arming Schedule** covers the test time, too (a silent killer).
|
||
- **⭐ THE ROOT CAUSE (2026-06-22): no detection AREA drawn.** This is what actually defeated us for
|
||
most of a day. On the motion/smart-detection page there's a **Draw Area** step — if **no region is
|
||
drawn on the frame, the camera detects nothing, generates NO event, and therefore posts nothing**
|
||
anywhere (httpHost, FTP, alarm stream all stay silent because there's no event upstream). Enabling
|
||
the detection + ticking Notify Surveillance Center is **not enough** — you must draw the region.
|
||
Once an area was drawn, the very first vehicle produced a clean POST. **Check this FIRST.**
|
||
|
||
### Confirmed real payload (DS-2CD1043G2-LIU, V5.8.10, 2026-06-22)
|
||
|
||
What this camera actually POSTs on a motion event with a target — captured end-to-end:
|
||
|
||
- **`Content-Type: multipart/form-data; boundary=boundary`**, one XML part named `MoveDetection.xml`
|
||
(`Content-Type: application/xml`). A real frame/JPEG *may* be attached as a second part on other
|
||
event types — our endpoint stores the readable head; splitting an image part to `snapshots` is a
|
||
forward step (not needed for plain motion).
|
||
- The XML is an `EventNotificationAlert` with the fields we care about:
|
||
- `<eventType>VMD</eventType>` (Video Motion Detection) + `<eventState>active</eventState>`
|
||
- **`<targetType>vehicle</targetType>`** — the camera classifies **vehicle vs human ON-DEVICE**.
|
||
(Field is `targetType`, NOT `detectionTarget`.) This means simple presence + class comes for
|
||
free, no vision model needed for that part.
|
||
- `<targetInfo><targetRect>` with normalized `X/Y/width/height` (0–1) — the **bounding box**.
|
||
- `<channelID>`, `<macAddress>` (provenance), `<dateTime>` — **but the dateTime is GARBAGE**
|
||
(`2032-…`) because this unit's **RTC is dead** (see below); we use our own server receive time,
|
||
never the camera's. (No `<plateNumber>` — this is a motion event, not an ANPR camera.)
|
||
|
||
### If a camera still won't push — diagnostics (read its OWN state)
|
||
|
||
Only after confirming the **detection area is drawn** + arming schedule covers now + Notify
|
||
Surveillance Center is on. These read the camera directly (no cooperation from our server):
|
||
|
||
1. **`netstat` on the camera (via SSH) while you trigger** — watch for an OUTBOUND line
|
||
`cam:port → server:3000`. It appearing = the camera fired and is delivering (then check our
|
||
`/api/devices/hikvision/alarms`). None = no event was generated (almost always: **no area drawn**).
|
||
2. **`GET /ISAPI/Event/notification/alertStream`** (Digest, needs a clean handshake) — the live event
|
||
bus. NB: a `curl --digest` tap that fails the handshake returns empty and looks like "no events"
|
||
— don't over-read silence here (this misled us); the netstat watch above is more reliable.
|
||
3. **SSH `showStatus` / `dmesg`** expose internal state. ⚠ **Caveat learned the hard way:** these
|
||
surface scary-looking strings that are **red herrings** — `EventScribe: except`, a `diskfull`
|
||
error on `Event/triggers` (on a camera with **no disk**), and `fh rtc get time error` / a 1970
|
||
clock. On our unit ALL of these were present **and the camera worked fine** once an area was
|
||
drawn. The dead RTC is real (hence the bogus `dateTime`) but **harmless** to event push. **Do NOT
|
||
conclude "dead camera / RMA" from these** — they are not proof of a broken event engine.
|
||
|
||
> **Correction (2026-06-22):** an earlier version of this page concluded this DS-2CD1043G2-LIU was a
|
||
> **defective unit needing RMA**, based on the silent alertStream + `diskfull`/`EventScribe:except` +
|
||
> dead RTC surviving a full factory reset. **That was WRONG.** The camera was healthy; the real cause
|
||
> was simply **no detection area drawn**, so no event was ever generated. The `diskfull`/RTC findings
|
||
> were unrelated quirks (RTC genuinely dead, but it doesn't block event push). Lesson: don't
|
||
> escalate to "hardware fault" while a basic config precondition (the drawn region) is unmet — and
|
||
> treat vendor status-API error strings as unreliable. The pull + [[opencv-anpr-service|vision]] path
|
||
> remains a valid fallback, but it was not needed here.
|