parking_solution/wiki/entities/opencv-anpr-service.md

type, tags, sources, updated, status

type	tags	sources	updated	status
entity	parking vision anpr anti-fraud service		2026-06-15	open

OpenCV ANPR / Vision Service

A local microservice that analyses camera snapshots: reads the licence plate (ANPR) and extracts vehicle attributes for verification. Built by us (decision 2026-06-15) to do recognition host-side on ordinary IP-camera snapshots, replacing the dedicated edge-AI lpr-camera. See decision vision-service.

Two jobs

1. Identity (ANPR). snapshot → { plate, confidence, bbox }. Feeds the existing IdentitySource = "lpr" (parking-session): the plate is a session/identity key and the way a plate-bound subscription is matched.
2. Verification (anti-fraud witness). snapshot → vehicle attributes — at minimum { make?, model?, colour, bodyType }, ideally a compact visual fingerprint (an embedding). This is the answer to plate-spoofing: a fraudster prints a registered/paid plate and drives in with a different car. Plate-reading alone can't catch that; comparing the vehicle seen at entry vs. exit (and vs. the subscription's known car) can. A plate that entered on a red hatchback but exits on a black SUV is a reconciliation anomaly — exactly the independent-witness role the append-only-event-chain flags as the unbuilt gap. See reconciliation.

The two jobs are why this is worth building rather than just plate-OCR: the service is both an identity source and an independent witness, the visual analogue of the whole system's "two records that must reconcile" thesis.

Architecture — separate localhost process

• A Python service (e.g. FastAPI) running on the appliance, called by the Node backend over localhost HTTP (POST /analyze with the JPEG bytes the camera driver already pulls — see lpr-camera "driver/storage boundary": Snapshot.bytes). Source lives in this monorepo at apps/vision/ (Turbo shim; uv-managed deps) — co-located source, separate process; see vision-service-packaging.
• Fully offline (offline-first): all inference is local, no cloud. Model weights ship on the appliance.
• Process isolation is deliberate — it keeps a heavy Python/native/AGPL stack out of the Node app's process and license surface (see licensing below), and gives it its own failure domain. If the service is down/slow, the host falls back (transient ticket path) rather than blocking the lane.
• Request/response (first cut):
  • POST /analyze → { plate: {text, confidence, bbox}|null, vehicle: {colour, bodyType, make?, model?, embedding?}, modelVersion, tookMs }
  • GET /health → readiness + model versions.
• The Node side wraps it behind an internal interface (like a device adapter) so the recognizer can be swapped without touching business logic.

Licensing — scoped AGPL exception (amends the standing rule)

The app is strictly MIT/Apache/BSD (technology-stack, standing-decisions). Accurate ANPR/vehicle models were assumed to be mostly AGPL (Ultralytics YOLO detectors, OpenALPR) or commercial — but the fast-alpr stack (above) is MIT end-to-end, so a permissive ANPR baseline now looks achievable (pending the weight-provenance caveat). The exception below still matters for the strongest models (Ultralytics YOLO) and for the vehicle-verification job. Decision (2026-06-15): allow AGPL inside this service only. It is a separate process, not linked into the app, so its obligations don't reach the Node/React codebase; the app's permissive guarantee is preserved. Recorded as an explicit exception in standing-decisions / vision-service.

• OpenCV core itself is Apache-2.0 (clean either way).
• AGPL note: if the appliance is ever offered as a network service to third parties, AGPL's network-use clause could require offering the service's source — relevant only if productised beyond the on-site appliance; flag at that point.

Recognizer evaluation — fast-alpr is the leading baseline (2026-06-19)

YOLO vs OpenCV is a category error — they're different pipeline layers, not competitors. ANPR is a pipeline: (1) plate detection (find the box → YOLO-family detector), (2) plate OCR (read the crop → a CRNN/CCT or OCR engine), (3) glue (capture/crop/deskew/draw → OpenCV, Apache-2.0, always present). So the real choice is which end-to-end recognizer, and OpenCV is used regardless as the image-handling toolkit.

Leading option: fast-alpr (v0.4.0, 15 Mar 2026). A thin orchestrator over two swappable stages, both on ONNX Runtime — which matches THIS service's decided architecture (separate localhost Python process, offline, swappable behind an interface) almost exactly:

Stage	Default model	Library	License
Plate detection	yolo-v9-t-384-license-plate-end2end	open-image-models	MIT
Plate OCR	cct-xs-v2-global-model	fast-plate-ocr	MIT

• MIT top-to-bottom (library and the published model weights), one maintainer (ankandrew) across all three repos. The detector is open-image-models' own YOLOv9 ONNX export — NOT the Ultralytics AGPL package — so fast-alpr is a permissive baseline that may not even need the scoped AGPL exception below. ⚠️ Caveat (verify before relying on it): a repo's LICENSE covers its code; redistributed model weights can carry separate provenance (YOLOv9 upstream is GPL-3.0; Ultralytics YOLO is AGPL). Confirm the weight training/provenance (model card) before treating "MIT weights" as settled for compliance — the AGPL-in-service exception is the safety net if it doesn't hold.
• CPU-only + fully offline. No runtime ships by default; pick a backend extra — fast-alpr[onnx] (CPU), or [onnx-gpu]/[onnx-openvino]/[onnx-directml]/[onnx-qnn] — which maps onto the "CPU now, small GPU/NPU later" compute question (bom, open-questions).
• Albanian/EU plates: fast-plate-ocr also has a European model trained on 40+ countries (newer than the default global model) — benchmark it against the default for AL accuracy.
• Modular, no lock-in: swap either stage via BaseDetector/BaseOCR (their docs plug in Tesseract). So fast-alpr is the baseline you keep while replacing one stage if needed.

Scope: fast-alpr is plate-only — it does Job 1 (ANPR) but NOT Job 2 (vehicle verification). The anti-spoofing vehicle-attribute/fingerprint stage is still ours to build — but since fast-alpr already standardizes on ONNX Runtime + a YOLO-family detector, the vehicle stage shares that runtime (the coherent outcome). Other options, weaker: OpenALPR (permissive but largely unmaintained, the old "permissive-only, weaker" path); Ultralytics YOLO + PaddleOCR (most accurate/tunable, but YOLO is AGPL → needs the in-service exception; most build effort — the "scale" path if fast-alpr's accuracy disappoints).

Recommendation: prototype with fast-alpr now (permissive, offline, ONNX, fits the decided shape); plan a YOLO-detector fine-tune + PaddleOCR only if production accuracy demands it. Choice kept open pending the weight-provenance check (the AL-plate benchmark below is now done).

Albanian-plate OCR benchmark — keep the default (2026-06-19)

Ran the four candidate fast-plate-ocr models through the full pipeline (YOLOv9 detect → OCR) on real AL plate photos (Wikimedia), CPU, scaffolded service:

OCR model	AA 558 EE	AA 687 KE	Speed	Note
cct-xs-v2-global-model (default)	✓ 0.999	✓ 1.000	33–39 ms	best accuracy + fastest; returns region=Albania
cct-s-v2-global-model	✓ 0.998	✓ 0.999	50–65 ms	as accurate, ~50% slower
global-plates-mobile-vit-v2-model	✓ 0.955	✓ 0.959	33–35 ms	fast, lower confidence
european-plates-mobile-vit-v2-model	✓ 0.784	✓ 0.766	38–46 ms	correct but much lower confidence; misread a synthetic AB123FG→AB123FO

Finding (overturns the prior assumption): the default cct-xs-v2-global-model is the best for Albania — most accurate AND fastest. The "European (40+ country)" model is worse here (~0.77 vs ~1.0 confidence, one synthetic misread), despite the "EU model → better for AL" intuition. So no config change: VISION_OCR_MODEL stays cct-xs-v2-global-model. Caveat: both test photos were clean head-on shots; real booth captures (angled, dirty, night, motion-blur) will lower absolute confidence — the min_confidence=0.5 floor (→ low_confidence → ticket-path fallback) covers that. The ranking should hold; re-benchmark on real on-site captures once the cameras are installed.

Anti-fraud / threat-model fit

• Plate spoofing (the motivating case): vehicle-attribute / fingerprint mismatch entry↔exit or vs. a subscription's registered car → anomaly. Doesn't block on its own (recognition is probabilistic) — it flags for reconciliation and is captured in the signed record.
• The recognition result and the source image both attach to the signed append-only-event-chain entry, so the evidence is tamper-evident even though recognition itself is host-side and fallible.
• Recognition is advisory, never the sole authority to open a barrier where money/access is at stake — confidence thresholds + fallback to ticket/manual; a low-confidence read must not strand a car (fail-state-safety).

Fitness for the entry/exit flows (assessment, 2026-06-19)

Asked after the scaffold + AL benchmark: is the service worthy to consume in the entry/exit flows? The benchmark settles accuracy (0.99+ on clean AL plates); "worthy" then turns on what authority the read is given — and the answer splits by role:

• ✅ Worthy NOW — as an ADVISORY identity source (Job 1). The flows are already built for a plate: a kind:"plate" device-events is a first-class identity today — exit-flow.ts signs source:"lpr" for it, and subscription-flow.ts matches a read plate against subscriptionPlates (subscription plate binding). So the service just produces the plate string a snapshot → /analyze → (if confident) a DeviceReadEvent{kind:"plate"} on the existing read bus. No flow rewrite — it feeds an existing input. Concretely worthy for: hands-free subscriber barrier open (plate-bound), and evidence enrichment (plate + image on the signed entry/exit for disputes).
• ⚠️ NOT worthy as the SOLE AUTHORITY to open a TRANSIENT barrier. Two threat-model reasons: (1) a plate is not a payment — a transient still needs a ticket + payment; letting a plate open the exit would be an unpaid-exit bypass. The min_confidence floor → low_confidence → ticket/manual fallback is the guard (already in the scaffold). (2) Plate-spoofing (a printed plate on a different car) — plate-only ANPR cannot catch it; that needs Job 2 (vehicle verification), which is NOT built. So plate-as-identity is convenience + evidence, never the lone reason a paid barrier opens. Consistent with "advisory, never sole authority" above.

Gaps before it's actually consumed (capable ≠ wired): (1) ✅ DONE — the Node→service VisionClient adapter (apps/server/src/vision-client.ts, localhost HTTP to /analyze + /health) now exists: opt-in (VISION_ENABLED, default off), fail-soft (any error/timeout/unreachable → null, never throws into the lane → ticket-path fallback), and re-applies the confidence floor (VISION_MIN_CONFIDENCE) so a low read is flagged advisory. Constructed in server.ts; verified end-to-end against the live service (Node → AA558EE 0.999, region=Albania). (2) ✅ DONE — trigger: ANPR rides the entry/exit SNAPSHOT (snapshot.ts). The real-world trigger is a transient button-press or a subscriber QR/RFID read — which already fires the entry/exit and its evidence snapshot. That is exactly the moment to recognize: snapshotAsync now takes the VisionClient, and after storing each snapshot from an opt-in camera (config.anpr === true), it runs ANPR off the SAME image and records the plate against the SAME session identity — an unsigned device_events{kind:"read"} with plate / confidence / region / model / snapshotId / source:"entry-exit-snapshot". So you can later answer "session X entered on plate AA558EE", with the evidence image linked by snapshotId. No polling — recognition fires only on a real entry/exit, one image serving both evidence and plate extraction. (Superseded the earlier polling VisionReader, now removed — VISION_POLL_MS/VISION_DEDUPE_MS gone.) The flows pass the client (entry/exit/ subscription constructors). It is advisory + fire-and-forget: a low-confidence/no-plate result records nothing, a vision failure never delays or changes the open, and the plate does not feed the access decision (the flow already decided). Verified end-to-end: a simulated entry snapshot on an anpr camera → stored the snapshot for the session AND recorded {identity:"TICKET-…", plate:"AA558EE", confidence:0.999, region:"Albania", snapshotId:…}.

Viewing it: GET /api/snapshots/by-identity/:identity now also returns plates[] (the kind:"read" reads for that session), and the SnapshotStrip renders each as a cyan "Plate: AA558EE 100%" chip above the images — so the recognized plate shows in the booth event-detail modal AND the pay modal beside the evidence photo, with no separate screen. (3) field-accuracy unknown — re-benchmark/tune the threshold on real on-site captures (angle/night/dirt). (4) the weight-provenance check (open). Bottom line: consume it as a gated advisory identity record off the entry/exit snapshot — not as sole authority — and Job 2 is still required for the anti-spoofing value. The adapter + the snapshot-triggered ANPR are now both built and verified end-to-end; remaining is field tuning (3), the provenance check (4), and Job 2.

Configuration (2026-06-19)

Turning it on touches four layers — two env sets (one per process), per-camera data, and deploy. The Python service and the Node server both read the VISION_ prefix but are separate processes, so give each its own .env (apps/vision/.env and apps/server/.env) — don't merge them. .env.example files document both.

1. The Python service (apps/vision/.env): VISION_RECOGNIZER=fast_alpr (the default stub recognizes nothing), VISION_HOST/VISION_PORT (prefer 127.0.0.1 — only the Node backend calls /analyze, so don't expose it off-host), VISION_OCR_MODEL/VISION_DETECTOR_MODEL (leave defaults — the AL-benchmark winners), VISION_MIN_CONFIDENCE. Install the models with uv sync --extra alpr; weights download on first run, so cache them at build/deploy for the air-gapped appliance.

2. The Node server (apps/server/.env): VISION_ENABLED=1 is the master switch (off by default — nothing runs or shows without it); VISION_URL must match the service's host:port; VISION_TIMEOUT_MS (slow-call cap so a lane never hangs) and VISION_MIN_CONFIDENCE (re-applied client-side). ANPR fires on the entry/exit snapshot, so there are no poll/dedupe knobs.

3. Per-camera opt-in (device config, not env): a camera does ANPR only when its config has both anpr: true and a relay binding (controllerId + relay). The anpr flag is a checkbox on the camera form in the first-run-setup (built 2026-06-19). Without the binding the entry-exit-points refuses every read ("reader not bound to a barrier") — so an unbound ANPR camera recognizes but every read is rejected (and logged with its snapshot).

4. Footer health: when VISION_ENABLED, the device-status-monitoring probes the service's /health each tick and shows a "Vision" chip in the booth footer (ready/degraded/offline

• the recognizer name); no chip when disabled. So the operator sees at a glance whether vision is up.

Network isolation (network-isolation): cameras live on the isolated device VLAN, so the vision service must reach that VLAN to pull snapshots — but its own /analyze should bind localhost (Node is the only caller). Keep the AGPL/heavy stack contained to this process.

Open

• Recognizer choice — fast-alpr (MIT, YOLOv9+CCT on ONNX) is the baseline, AL-benchmarked: the default cct-xs-v2-global-model won over the EU model on real AL plates (table above). The one remaining open item is the model-weight-provenance check (the MIT-weights claim). A re-benchmark on real on-site captures (angled/night/dirty) is wanted once cameras are installed. See vision-service; AGPL still permitted in-service for the stronger fallback.
• Vehicle fingerprint: attribute classifier vs. embedding-similarity; what threshold makes a mismatch an anomaly without false-positiving on lighting/angle.
• Compute footprint on the appliance (CPU-only vs. a small GPU/NPU) — procurement input (bom, open-questions).
• Per-camera opt-in — ✅ built: config.anpr === true enables ANPR on a camera (set via the SetupWizard checkbox); ANPR then runs on that camera's entry/exit snapshot.