parking_solution/wiki/entities/opencv-anpr-service.md

type, tags, sources, updated, status

type	tags	sources	updated	status
entity	parking vision anpr anti-fraud service		2026-06-15	open

OpenCV ANPR / Vision Service

A local microservice that analyses camera snapshots: reads the licence plate (ANPR) and extracts vehicle attributes for verification. Built by us (decision 2026-06-15) to do recognition host-side on ordinary IP-camera snapshots, replacing the dedicated edge-AI lpr-camera. See decision vision-service.

Two jobs

1. Identity (ANPR). snapshot → { plate, confidence, bbox }. Feeds the existing IdentitySource = "lpr" (parking-session): the plate is a session/identity key and the way a plate-bound subscription is matched.
2. Verification (anti-fraud witness). snapshot → vehicle attributes — at minimum { make?, model?, colour, bodyType }, ideally a compact visual fingerprint (an embedding). This is the answer to plate-spoofing: a fraudster prints a registered/paid plate and drives in with a different car. Plate-reading alone can't catch that; comparing the vehicle seen at entry vs. exit (and vs. the subscription's known car) can. A plate that entered on a red hatchback but exits on a black SUV is a reconciliation anomaly — exactly the independent-witness role the append-only-event-chain flags as the unbuilt gap. See reconciliation.

The two jobs are why this is worth building rather than just plate-OCR: the service is both an identity source and an independent witness, the visual analogue of the whole system's "two records that must reconcile" thesis.

Architecture — separate localhost process

• A Python service (e.g. FastAPI) running on the appliance, called by the Node backend over localhost HTTP (POST /analyze with the JPEG bytes the camera driver already pulls — see lpr-camera "driver/storage boundary": Snapshot.bytes). Source lives in this monorepo at apps/vision/ (Turbo shim; uv-managed deps) — co-located source, separate process; see vision-service-packaging.
• Fully offline (offline-first): all inference is local, no cloud. Model weights ship on the appliance.
• Process isolation is deliberate — it keeps a heavy Python/native/AGPL stack out of the Node app's process and license surface (see licensing below), and gives it its own failure domain. If the service is down/slow, the host falls back (transient ticket path) rather than blocking the lane.
• Request/response (first cut):
  • POST /analyze → { plate: {text, confidence, bbox}|null, vehicle: {colour, bodyType, make?, model?, embedding?}, modelVersion, tookMs }
  • GET /health → readiness + model versions.
• The Node side wraps it behind an internal interface (like a device adapter) so the recognizer can be swapped without touching business logic.

Licensing — scoped AGPL exception (amends the standing rule)

The app is strictly MIT/Apache/BSD (technology-stack, standing-decisions). Accurate ANPR/vehicle models were assumed to be mostly AGPL (Ultralytics YOLO detectors, OpenALPR) or commercial — but the fast-alpr stack (above) is MIT end-to-end, so a permissive ANPR baseline now looks achievable (pending the weight-provenance caveat). The exception below still matters for the strongest models (Ultralytics YOLO) and for the vehicle-verification job. Decision (2026-06-15): allow AGPL inside this service only. It is a separate process, not linked into the app, so its obligations don't reach the Node/React codebase; the app's permissive guarantee is preserved. Recorded as an explicit exception in standing-decisions / vision-service.

• OpenCV core itself is Apache-2.0 (clean either way).
• AGPL note: if the appliance is ever offered as a network service to third parties, AGPL's network-use clause could require offering the service's source — relevant only if productised beyond the on-site appliance; flag at that point.

Recognizer evaluation — fast-alpr is the leading baseline (2026-06-19)

YOLO vs OpenCV is a category error — they're different pipeline layers, not competitors. ANPR is a pipeline: (1) plate detection (find the box → YOLO-family detector), (2) plate OCR (read the crop → a CRNN/CCT or OCR engine), (3) glue (capture/crop/deskew/draw → OpenCV, Apache-2.0, always present). So the real choice is which end-to-end recognizer, and OpenCV is used regardless as the image-handling toolkit.

Leading option: fast-alpr (v0.4.0, 15 Mar 2026). A thin orchestrator over two swappable stages, both on ONNX Runtime — which matches THIS service's decided architecture (separate localhost Python process, offline, swappable behind an interface) almost exactly:

Stage	Default model	Library	License
Plate detection	yolo-v9-t-384-license-plate-end2end	open-image-models	MIT
Plate OCR	cct-xs-v2-global-model	fast-plate-ocr	MIT

• MIT top-to-bottom (library and the published model weights), one maintainer (ankandrew) across all three repos. The detector is open-image-models' own YOLOv9 ONNX export — NOT the Ultralytics AGPL package — so fast-alpr is a permissive baseline that may not even need the scoped AGPL exception below. ⚠️ Caveat (verify before relying on it): a repo's LICENSE covers its code; redistributed model weights can carry separate provenance (YOLOv9 upstream is GPL-3.0; Ultralytics YOLO is AGPL). Confirm the weight training/provenance (model card) before treating "MIT weights" as settled for compliance — the AGPL-in-service exception is the safety net if it doesn't hold.
• CPU-only + fully offline. No runtime ships by default; pick a backend extra — fast-alpr[onnx] (CPU), or [onnx-gpu]/[onnx-openvino]/[onnx-directml]/[onnx-qnn] — which maps onto the "CPU now, small GPU/NPU later" compute question (bom, open-questions).
• Albanian/EU plates: fast-plate-ocr also has a European model trained on 40+ countries (newer than the default global model) — benchmark it against the default for AL accuracy.
• Modular, no lock-in: swap either stage via BaseDetector/BaseOCR (their docs plug in Tesseract). So fast-alpr is the baseline you keep while replacing one stage if needed.

Scope: fast-alpr is plate-only — it does Job 1 (ANPR) but NOT Job 2 (vehicle verification). The anti-spoofing vehicle-attribute/fingerprint stage is still ours to build — but since fast-alpr already standardizes on ONNX Runtime + a YOLO-family detector, the vehicle stage shares that runtime (the coherent outcome). Other options, weaker: OpenALPR (permissive but largely unmaintained, the old "permissive-only, weaker" path); Ultralytics YOLO + PaddleOCR (most accurate/tunable, but YOLO is AGPL → needs the in-service exception; most build effort — the "scale" path if fast-alpr's accuracy disappoints).

Recommendation: prototype with fast-alpr now (permissive, offline, ONNX, fits the decided shape); plan a YOLO-detector fine-tune + PaddleOCR only if production accuracy demands it. Choice kept open pending the weight-provenance check (the AL-plate benchmark below is now done).

Albanian-plate OCR benchmark — keep the default (2026-06-19)

Ran the four candidate fast-plate-ocr models through the full pipeline (YOLOv9 detect → OCR) on real AL plate photos (Wikimedia), CPU, scaffolded service:

OCR model	AA 558 EE	AA 687 KE	Speed	Note
cct-xs-v2-global-model (default)	✓ 0.999	✓ 1.000	33–39 ms	best accuracy + fastest; returns region=Albania
cct-s-v2-global-model	✓ 0.998	✓ 0.999	50–65 ms	as accurate, ~50% slower
global-plates-mobile-vit-v2-model	✓ 0.955	✓ 0.959	33–35 ms	fast, lower confidence
european-plates-mobile-vit-v2-model	✓ 0.784	✓ 0.766	38–46 ms	correct but much lower confidence; misread a synthetic AB123FG→AB123FO

Finding (overturns the prior assumption): the default cct-xs-v2-global-model is the best for Albania — most accurate AND fastest. The "European (40+ country)" model is worse here (~0.77 vs ~1.0 confidence, one synthetic misread), despite the "EU model → better for AL" intuition. So no config change: VISION_OCR_MODEL stays cct-xs-v2-global-model. Caveat: both test photos were clean head-on shots; real booth captures (angled, dirty, night, motion-blur) will lower absolute confidence — the min_confidence=0.5 floor (→ low_confidence → ticket-path fallback) covers that. The ranking should hold; re-benchmark on real on-site captures once the cameras are installed.

Anti-fraud / threat-model fit

• Plate spoofing (the motivating case): vehicle-attribute / fingerprint mismatch entry↔exit or vs. a subscription's registered car → anomaly. Doesn't block on its own (recognition is probabilistic) — it flags for reconciliation and is captured in the signed record.
• The recognition result and the source image both attach to the signed append-only-event-chain entry, so the evidence is tamper-evident even though recognition itself is host-side and fallible.
• Recognition is advisory, never the sole authority to open a barrier where money/access is at stake — confidence thresholds + fallback to ticket/manual; a low-confidence read must not strand a car (fail-state-safety).

Fitness for the entry/exit flows (assessment, 2026-06-19)

Asked after the scaffold + AL benchmark: is the service worthy to consume in the entry/exit flows? The benchmark settles accuracy (0.99+ on clean AL plates); "worthy" then turns on what authority the read is given — and the answer splits by role:

• ✅ Worthy NOW — as an ADVISORY identity source (Job 1). The flows are already built for a plate: a kind:"plate" device-events is a first-class identity today — exit-flow.ts signs source:"lpr" for it, and subscription-flow.ts matches a read plate against subscriptionPlates (subscription plate binding). So the service just produces the plate string a snapshot → /analyze → (if confident) a DeviceReadEvent{kind:"plate"} on the existing read bus. No flow rewrite — it feeds an existing input. Concretely worthy for: hands-free subscriber barrier open (plate-bound), and evidence enrichment (plate + image on the signed entry/exit for disputes).
• ⚠️ NOT worthy as the SOLE AUTHORITY to open a TRANSIENT barrier. Two threat-model reasons: (1) a plate is not a payment — a transient still needs a ticket + payment; letting a plate open the exit would be an unpaid-exit bypass. The min_confidence floor → low_confidence → ticket/manual fallback is the guard (already in the scaffold). (2) Plate-spoofing (a printed plate on a different car) — plate-only ANPR cannot catch it; that needs Job 2 (vehicle verification), which is NOT built. So plate-as-identity is convenience + evidence, never the lone reason a paid barrier opens. Consistent with "advisory, never sole authority" above.

Gaps before it's actually consumed (capable ≠ wired): (1) ✅ DONE — the Node→service VisionClient adapter (apps/server/src/vision-client.ts, localhost HTTP to /analyze + /health) now exists: opt-in (VISION_ENABLED, default off), fail-soft (any error/timeout/unreachable → null, never throws into the lane → ticket-path fallback), and re-applies the confidence floor (VISION_MIN_CONFIDENCE) so a low read is flagged advisory. Constructed in server.ts; verified end-to-end against the live service (Node → AA558EE 0.999, region=Albania). (2) ✅ DONE — trigger: ANPR rides the entry/exit SNAPSHOT (snapshot.ts). The real-world trigger is a transient button-press or a subscriber QR/RFID read — which already fires the entry/exit and its evidence snapshot. That is exactly the moment to recognize: snapshotAsync now takes the VisionClient, and after storing each snapshot from an opt-in camera (config.anpr === true), it runs ANPR off the SAME image and records the plate against the SAME session identity — an unsigned device_events{kind:"read"} with plate / confidence / region / model / snapshotId / source:"entry-exit-snapshot". So you can later answer "session X entered on plate AA558EE", with the evidence image linked by snapshotId. No polling — recognition fires only on a real entry/exit, one image serving both evidence and plate extraction. (Superseded the earlier polling VisionReader, now removed — VISION_POLL_MS/VISION_DEDUPE_MS gone.) The flows pass the client (entry/exit/ subscription constructors). It is advisory + fire-and-forget: a low-confidence/no-plate result records nothing, a vision failure never delays or changes the open, and the plate does not feed the access decision (the flow already decided). Verified end-to-end: a simulated entry snapshot on an anpr camera → stored the snapshot for the session AND recorded {identity:"TICKET-…", plate:"AA558EE", confidence:0.999, region:"Albania", snapshotId:…}.

Viewing it: GET /api/snapshots/by-identity/:identity now also returns plates[] (the kind:"read" reads for that session), and the SnapshotStrip renders each as a cyan "Plate: AA558EE 100%" chip above the images — so the recognized plate shows in the booth event-detail modal AND the pay modal beside the evidence photo, with no separate screen. (3) field-accuracy unknown — re-benchmark/tune the threshold on real on-site captures (angle/night/dirt). (4) the weight-provenance check (open). Bottom line: consume it as a gated advisory identity record off the entry/exit snapshot — not as sole authority — and Job 2 is still required for the anti-spoofing value. The adapter + the snapshot-triggered ANPR are now both built and verified end-to-end; remaining is field tuning (3), the provenance check (4), and Job 2.

Configuration (2026-06-19)

Turning it on touches four layers — two env sets (one per process), per-camera data, and deploy. The Python service and the Node server both read the VISION_ prefix but are separate processes, so give each its own .env (apps/vision/.env and apps/server/.env) — don't merge them. .env.example files document both.

1. The Python service (apps/vision/.env): VISION_RECOGNIZER=fast_alpr (the default stub recognizes nothing), VISION_HOST/VISION_PORT (prefer 127.0.0.1 — only the Node backend calls /analyze, so don't expose it off-host), VISION_OCR_MODEL/VISION_DETECTOR_MODEL (leave defaults — the AL-benchmark winners), VISION_MIN_CONFIDENCE. Install the models with uv sync --extra alpr; weights download on first run, so cache them at build/deploy for the air-gapped appliance.

2. The Node server (apps/server/.env): VISION_ENABLED=1 is the master switch (off by default — nothing runs or shows without it); VISION_URL must match the service's host:port; VISION_TIMEOUT_MS (slow-call cap so a lane never hangs) and VISION_MIN_CONFIDENCE (re-applied client-side). ANPR fires on the entry/exit snapshot, so there are no poll/dedupe knobs.

3. Per-camera opt-in (device config, not env): a camera does ANPR only when its config has both anpr: true and a relay binding (controllerId + relay). The anpr flag is a checkbox on the camera form in the first-run-setup (built 2026-06-19). Without the binding the entry-exit-points refuses every read ("reader not bound to a barrier") — so an unbound ANPR camera recognizes but every read is rejected (and logged with its snapshot).

4. Footer health: when VISION_ENABLED, the device-status-monitoring probes the service's /health each tick and shows a "Vision" chip in the booth footer (ready/degraded/offline

• the recognizer name); no chip when disabled. So the operator sees at a glance whether vision is up.

Network isolation (network-isolation): cameras live on the isolated device VLAN, so the vision service must reach that VLAN to pull snapshots — but its own /analyze should bind localhost (Node is the only caller). Keep the AGPL/heavy stack contained to this process.

Hardening / fix backlog (vision-service-hardening): the 2026-07-02 code + security reviews logged a prioritised to-do list against apps/vision/ — DoS gaps (body-cap-after-buffering, pixel-bomb, inference on the async event loop), unauthenticated + operator-writable model weights, and 0.0.0.0-by-default binding lead it. Consult it before touching this service.

Open

• Recognizer choice — fast-alpr (MIT, YOLOv9+CCT on ONNX) is the baseline, AL-benchmarked: the default cct-xs-v2-global-model won over the EU model on real AL plates (table above). The one remaining open item is the model-weight-provenance check (the MIT-weights claim). A re-benchmark on real on-site captures (angled/night/dirty) is wanted once cameras are installed. See vision-service; AGPL still permitted in-service for the stronger fallback.
• Vehicle fingerprint: attribute classifier vs. embedding-similarity; what threshold makes a mismatch an anomaly without false-positiving on lighting/angle.
• Compute footprint on the appliance (CPU-only vs. a small GPU/NPU) — procurement input (bom, open-questions).
• Per-camera opt-in — ✅ built: config.anpr === true enables ANPR on a camera (set via the SetupWizard checkbox); ANPR then runs on that camera's entry/exit snapshot.

Vehicle body type (advisory) — the vehicle stage, phase A (2026-09-06)

Phase B (the classifier that knows SUV from sedan), its training loop and the hardware it runs on are decided on bodytype-classifier-training — not built yet.

/analyze populates vehicle.body_type + vehicle.confidence from the shared vocabulary (car, sedan, hatchback, suv, minivan, pickup, van, truck, bus, motorcycle). Node records it beside the plate and the Car Wash desk pre-selects the category the site maps it to; the operator decides, a confident downgrade is flagged, nothing is gated on it (venue-modules §Vehicle category from vision).

Phase A = YOLOX-S (Megvii, Apache-2.0) as ONNX on the same ONNX Runtime the plate stage uses — the licence rule that keeps Ultralytics (AGPL) out. vision_service/vehicle.py: pure numpy/cv2 letterbox (pad 114, raw 0–255 BGR — YOLOX's exported graphs are not normalised), stride-grid decode, class-agnostic NMS, COCO car/motorcycle/bus/truck → the vocabulary, and ONE vehicle per frame: the box holding the plate's centre when a plate was read (the car that was read, not the one behind), else the largest box. WithVehicle in recognizer.py wraps whichever plate recognizer runs (stub included, so the stage is testable without fast-alpr); a failing stage yields vehicle: null and a vehicle: … note in /health.detail — it never costs the plate read. Composed model_version reads <plate>+yolox:yolox_s.onnx@640.

• Config: VISION_VEHICLE_MODEL_PATH (unset = stage off), VISION_VEHICLE_INPUT_SIZE (640), VISION_VEHICLE_MIN_CONFIDENCE (0.4 — the detector's floor; the SITE threshold that decides a flag lives in Setup → Car wash). The Docker image bakes the weights at /app/models/yolox_s.onnx (best-effort curl at build; no network → stage off) and sets the path, so the air-gapped appliance never fetches at runtime and no operator-writable path holds a model (vision-service-hardening). Compose forwards the var; set it EMPTY in the stack env to switch the stage off. Locally: curl the release file into apps/vision/models/ (gitignored).
• Measured on dev (2026-09-06), four real 2560×1440 entry frames from the DS-2CD1047G3H: three with a car → car at 0.83–0.88, ~240–330 ms each on the dev CPU with 2 intra-op threads; the empty-lane frame with a person at the camera → no vehicle (correct: a person is not a class we keep). One frame per entry, so the cost is invisible to the lane.
• What it cannot do: SUV vs sedan — COCO has one car. For a Vetura/SUV price list every car maps to Vetura and no downgrade fires; vans, trucks, buses and motorcycles do separate. Phase B (a body-type classifier on the pilot's own frames — every wash order is a labelled frame) is what closes that gap; the detector's box is the crop it will classify.
• Training (user asked, 2026-09-06): YOLOX-S needs none. It ships trained on COCO and is a finished detector; its limit is vocabulary, not quality. Phase B is a different, smaller model — a classifier over the detector's crop, not a retrained detector — fine-tuned on a small Apache-licensed backbone. Data: a few hundred crops per category to start, a couple of thousand is comfortable, all from the pilot's own lane and camera. Labels are NOT the operator's picks (untrusted — threat-model) but a trusted reviewer's, gathered through the vision-review-outbox. Expect 85–95 % on frontal gate views once tuned — enough to flag, never to bill, which is why the flag records and the site threshold exists.