---
type: entity
tags: [parking, vision, anpr, anti-fraud, service]
sources: []
updated: 2026-06-15
status: open
---

# OpenCV ANPR / Vision Service

A **local microservice** that analyses camera snapshots: reads the licence **plate** (ANPR) and
extracts **vehicle attributes** for verification. Built by us (decision 2026-06-15) to do
recognition **host-side on ordinary IP-camera snapshots**, replacing the dedicated edge-AI
[[lpr-camera]]. See decision [[vision-service]].

## Two jobs

1. **Identity (ANPR).** snapshot → `{ plate, confidence, bbox }`. Feeds the existing
   `IdentitySource = "lpr"` ([[parking-session]]): the plate is a session/identity key and the way
   a plate-bound [[subscription]] is matched.
2. **Verification (anti-fraud witness).** snapshot → vehicle attributes — at minimum
   `{ make?, model?, colour, bodyType }`, ideally a compact **visual fingerprint** (an embedding).
   This is the answer to **plate-spoofing**: *a fraudster prints a registered/paid plate and drives
   in with a different car.* Plate-reading alone can't catch that; comparing the **vehicle** seen at
   entry vs. exit (and vs. the [[subscription]]'s known car) can. A plate that entered on a red hatchback
   but exits on a black SUV is a **reconciliation anomaly** — exactly the independent-witness role
   the [[append-only-event-chain]] flags as the unbuilt gap. See [[reconciliation]].

> The two jobs are why this is worth building rather than just plate-OCR: the service is both an
> **identity source** and an **independent witness**, the visual analogue of the whole system's
> "two records that must reconcile" thesis.

## Architecture — separate localhost process

- A **Python service** (e.g. FastAPI) running **on the appliance**, called by the Node backend over
  **localhost HTTP** (`POST /analyze` with the JPEG bytes the camera driver already pulls — see
  [[lpr-camera]] "driver/storage boundary": `Snapshot.bytes`). **Source lives in this monorepo at
  `apps/vision/`** (Turbo shim; `uv`-managed deps) — co-located source, separate process; see
  [[vision-service-packaging]].
- **Fully offline** ([[offline-first]]): all inference is local, no cloud. Model weights ship on the
  appliance.
- **Process isolation is deliberate** — it keeps a heavy Python/native/AGPL stack out of the
  Node app's process and license surface (see licensing below), and gives it its own failure
  domain. If the service is down/slow, the host falls back (transient ticket path) rather than
  blocking the lane.
- **Request/response (first cut):**
  - `POST /analyze` → `{ plate: {text, confidence, bbox}|null, vehicle: {colour, bodyType, make?, model?, embedding?}, modelVersion, tookMs }`
  - `GET /health` → readiness + model versions.
- The Node side wraps it behind an internal interface (like a device adapter) so the recognizer can
  be swapped without touching business logic.

## Licensing — scoped AGPL exception (amends the standing rule)

The app is strictly **MIT/Apache/BSD** ([[technology-stack]], [[standing-decisions]]). Accurate
ANPR/vehicle models were *assumed* to be mostly **AGPL** (Ultralytics YOLO detectors, OpenALPR) or
commercial — but the **fast-alpr stack (above) is MIT end-to-end**, so a permissive ANPR baseline now
looks achievable (pending the weight-provenance caveat). The exception below still matters for the
*strongest* models (Ultralytics YOLO) and for the vehicle-verification job. Decision (2026-06-15):
**allow AGPL inside this service only.** It is a **separate process**, not
linked into the app, so its obligations don't reach the Node/React codebase; the app's permissive
guarantee is preserved. Recorded as an explicit exception in [[standing-decisions]] /
[[vision-service]].

- OpenCV core itself is **Apache-2.0** (clean either way).
- AGPL note: if the appliance is ever offered as a network service to third parties, AGPL's
  network-use clause could require offering the service's source — relevant only if productised
  beyond the on-site appliance; flag at that point.

## Recognizer evaluation — fast-alpr is the leading baseline (2026-06-19)

`YOLO vs OpenCV` is a **category error** — they're different pipeline layers, not competitors. ANPR
is a **pipeline**: (1) plate **detection** (find the box → YOLO-family detector), (2) plate **OCR**
(read the crop → a CRNN/CCT or OCR engine), (3) **glue** (capture/crop/deskew/draw → OpenCV,
Apache-2.0, always present). So the real choice is *which end-to-end recognizer*, and **OpenCV is
used regardless** as the image-handling toolkit.

**Leading option: [fast-alpr](https://github.com/ankandrew/fast-alpr) (v0.4.0, 15 Mar 2026).** A thin
orchestrator over two **swappable** stages, both on **ONNX Runtime** — which matches THIS service's
decided architecture (separate localhost Python process, offline, swappable behind an interface)
almost exactly:

| Stage | Default model | Library | License |
| --- | --- | --- | --- |
| Plate detection | `yolo-v9-t-384-license-plate-end2end` | [open-image-models](https://github.com/ankandrew/open-image-models) | MIT |
| Plate OCR | `cct-xs-v2-global-model` | [fast-plate-ocr](https://github.com/ankandrew/fast-plate-ocr) | MIT |

- **MIT top-to-bottom** (library *and* the published model weights), one maintainer (ankandrew) across
  all three repos. **The detector is open-image-models' own YOLOv9 ONNX export — NOT the Ultralytics
  AGPL package** — so fast-alpr is a **permissive baseline that may not even need the scoped AGPL
  exception** below. ⚠️ **Caveat (verify before relying on it):** a repo's LICENSE covers its *code*;
  redistributed model *weights* can carry separate provenance (YOLOv9 upstream is GPL-3.0; Ultralytics
  YOLO is AGPL). Confirm the weight training/provenance (model card) before treating "MIT weights" as
  settled for compliance — the AGPL-in-service exception is the safety net if it doesn't hold.
- **CPU-only + fully offline.** No runtime ships by default; pick a backend extra — `fast-alpr[onnx]`
  (CPU), or `[onnx-gpu]`/`[onnx-openvino]`/`[onnx-directml]`/`[onnx-qnn]` — which maps onto the
  "CPU now, small GPU/NPU later" compute question ([[bom]], [[open-questions]]).
- **Albanian/EU plates:** fast-plate-ocr also has a **European model trained on 40+ countries** (newer
  than the default global model) — benchmark it against the default for AL accuracy.
- **Modular, no lock-in:** swap either stage via `BaseDetector`/`BaseOCR` (their docs plug in
  Tesseract). So fast-alpr is the baseline you keep while replacing one stage if needed.

**Scope: fast-alpr is plate-only — it does Job 1 (ANPR) but NOT Job 2 (vehicle verification).** The
anti-spoofing vehicle-attribute/fingerprint stage is still ours to build — but since fast-alpr already
standardizes on **ONNX Runtime + a YOLO-family detector**, the vehicle stage shares that runtime (the
coherent outcome). Other options, weaker: **OpenALPR** (permissive but largely unmaintained, the old
"permissive-only, weaker" path); **Ultralytics YOLO + PaddleOCR** (most accurate/tunable, but YOLO is
AGPL → needs the in-service exception; most build effort — the "scale" path if fast-alpr's accuracy
disappoints).

**Recommendation:** prototype with **fast-alpr** now (permissive, offline, ONNX, fits the decided
shape); plan a YOLO-detector fine-tune + PaddleOCR only if production accuracy demands it. Choice kept
**open** pending the weight-provenance check (the AL-plate benchmark below is now done).

### Albanian-plate OCR benchmark — keep the default (2026-06-19)

Ran the four candidate `fast-plate-ocr` models through the **full pipeline** (YOLOv9 detect → OCR) on
real AL plate photos (Wikimedia), CPU, scaffolded service:

| OCR model | `AA 558 EE` | `AA 687 KE` | Speed | Note |
| --- | --- | --- | --- | --- |
| **`cct-xs-v2-global-model`** (default) | ✓ 0.999 | ✓ **1.000** | **33–39 ms** | best accuracy + fastest; returns `region=Albania` |
| `cct-s-v2-global-model` | ✓ 0.998 | ✓ 0.999 | 50–65 ms | as accurate, ~50% slower |
| `global-plates-mobile-vit-v2-model` | ✓ 0.955 | ✓ 0.959 | 33–35 ms | fast, lower confidence |
| `european-plates-mobile-vit-v2-model` | ✓ 0.784 | ✓ 0.766 | 38–46 ms | correct but **much lower confidence**; misread a synthetic `AB123FG`→`AB123FO` |

**Finding (overturns the prior assumption):** the **default `cct-xs-v2-global-model` is the best for
Albania** — most accurate AND fastest. The "European (40+ country)" model is *worse* here (~0.77 vs
~1.0 confidence, one synthetic misread), despite the "EU model → better for AL" intuition. So **no
config change**: `VISION_OCR_MODEL` stays `cct-xs-v2-global-model`. Caveat: both test photos were
clean head-on shots; real booth captures (angled, dirty, night, motion-blur) will lower absolute
confidence — the `min_confidence=0.5` floor (→ `low_confidence` → ticket-path fallback) covers that.
The ranking should hold; re-benchmark on real on-site captures once the cameras are installed.

## Anti-fraud / threat-model fit

- **Plate spoofing** (the motivating case): vehicle-attribute / fingerprint mismatch entry↔exit or
  vs. a [[subscription]]'s registered car → anomaly. Doesn't *block* on its own (recognition is
  probabilistic) — it **flags for [[reconciliation]]** and is captured in the signed record.
- The recognition result and the source image both attach to the signed [[append-only-event-chain]]
  entry, so the *evidence* is tamper-evident even though recognition itself is host-side and
  fallible.
- Recognition is **advisory, never the sole authority** to open a barrier where money/access is at
  stake — confidence thresholds + fallback to ticket/manual; a low-confidence read must not strand a
  car ([[fail-state-safety]]).

## Fitness for the entry/exit flows (assessment, 2026-06-19)

Asked after the scaffold + AL benchmark: *is the service worthy to consume in the entry/exit flows?*
The benchmark settles **accuracy** (0.99+ on clean AL plates); "worthy" then turns on **what authority
the read is given** — and the answer splits by role:

- **✅ Worthy NOW — as an ADVISORY identity source (Job 1).** The flows are **already built for a
  plate**: a `kind:"plate"` [[device-events|read]] is a first-class identity today — `exit-flow.ts`
  signs `source:"lpr"` for it, and `subscription-flow.ts` matches a read plate against
  `subscriptionPlates` ([[subscription]] plate binding). So the service just **produces** the plate
  string a snapshot → `/analyze` → (if confident) a `DeviceReadEvent{kind:"plate"}` on the existing
  read bus. **No flow rewrite — it feeds an existing input.** Concretely worthy for: hands-free
  **subscriber** barrier open (plate-bound), and **evidence enrichment** (plate + image on the signed
  entry/exit for disputes).
- **⚠️ NOT worthy as the SOLE AUTHORITY to open a TRANSIENT barrier.** Two threat-model reasons: (1) **a
  plate is not a payment** — a transient still needs a ticket + `payment`; letting a plate open the
  exit would be an unpaid-exit bypass. The `min_confidence` floor → `low_confidence` → ticket/manual
  fallback is the guard (already in the scaffold). (2) **Plate-spoofing** (a printed plate on a
  different car) — plate-only ANPR *cannot* catch it; that needs **Job 2 (vehicle verification), which
  is NOT built**. So plate-as-identity is convenience + evidence, never the lone reason a paid barrier
  opens. Consistent with "advisory, never sole authority" above.

**Gaps before it's actually consumed (capable ≠ wired):** (1) ✅ **DONE — the Node→service
`VisionClient`** adapter (`apps/server/src/vision-client.ts`, localhost HTTP to `/analyze` + `/health`)
now exists: **opt-in** (`VISION_ENABLED`, default off), **fail-soft** (any error/timeout/unreachable →
`null`, never throws into the lane → ticket-path fallback), and **re-applies the confidence floor**
(`VISION_MIN_CONFIDENCE`) so a low read is flagged advisory. Constructed in `server.ts`; verified
end-to-end against the live service (Node → `AA558EE` 0.999, `region=Albania`). (2) ✅ **DONE —
trigger: ANPR rides the entry/exit SNAPSHOT (`snapshot.ts`).** The real-world trigger is a **transient
button-press or a subscriber QR/RFID read** — which already fires the entry/exit and its evidence
snapshot. That is exactly the moment to recognize: `snapshotAsync` now takes the `VisionClient`, and
after storing each snapshot from an **opt-in** camera (`config.anpr === true`), it runs ANPR off the
**SAME image** and **records the plate against the SAME session `identity`** — an unsigned
`device_events{kind:"read"}` with plate / confidence / region / model / `snapshotId` /
`source:"entry-exit-snapshot"`. So you can later answer *"session X entered on plate AA558EE"*, with the
evidence image linked by `snapshotId`. **No polling — recognition fires only on a real entry/exit**,
one image serving both evidence and plate extraction. *(Superseded the earlier polling `VisionReader`,
now removed — `VISION_POLL_MS`/`VISION_DEDUPE_MS` gone.)* The flows pass the client (entry/exit/
subscription constructors). It is **advisory + fire-and-forget**: a low-confidence/no-plate result
records nothing, a vision failure never delays or changes the open, and the plate does **not** feed the
access decision (the flow already decided). *Verified end-to-end:* a simulated entry snapshot on an
`anpr` camera → stored the snapshot for the session AND recorded `{identity:"TICKET-…", plate:"AA558EE",
confidence:0.999, region:"Albania", snapshotId:…}`.

**Viewing it:** `GET /api/snapshots/by-identity/:identity` now also returns `plates[]` (the
`kind:"read"` reads for that session), and the **`SnapshotStrip`** renders each as a cyan
"Plate: AA558EE 100%" chip above the images — so the recognized plate shows in the **booth
event-detail modal AND the pay modal** beside the evidence photo, with no separate screen.
(3) **field-accuracy** unknown — re-benchmark/tune
the threshold on real on-site captures (angle/night/dirt). (4) the **weight-provenance** check (open).
**Bottom line: consume it as a gated advisory identity record off the entry/exit snapshot — not as sole
authority — and Job 2 is still required for the anti-spoofing value.** The
adapter + the snapshot-triggered ANPR are now **both built and verified end-to-end**; remaining is
field tuning (3), the provenance check (4), and Job 2.

## Configuration (2026-06-19)

Turning it on touches **four layers** — two env sets (one per process), per-camera data, and deploy.
The Python service and the Node server **both** read the `VISION_` prefix but are **separate
processes**, so give each its **own `.env`** (`apps/vision/.env` and `apps/server/.env`) — don't merge
them. `.env.example` files document both.

**1. The Python service (`apps/vision/.env`):** `VISION_RECOGNIZER=fast_alpr` (the default `stub`
recognizes nothing), `VISION_HOST`/`VISION_PORT` (prefer **`127.0.0.1`** — only the Node backend calls
`/analyze`, so don't expose it off-host), `VISION_OCR_MODEL`/`VISION_DETECTOR_MODEL` (leave defaults —
the AL-benchmark winners), `VISION_MIN_CONFIDENCE`. Install the models with `uv sync --extra alpr`;
weights download on first run, so **cache them at build/deploy** for the air-gapped appliance.

**2. The Node server (`apps/server/.env`):** `VISION_ENABLED=1` is the **master switch** (off by
default — nothing runs or shows without it); `VISION_URL` must match the service's host:port;
`VISION_TIMEOUT_MS` (slow-call cap so a lane never hangs) and `VISION_MIN_CONFIDENCE` (re-applied
client-side). ANPR fires on the entry/exit snapshot, so there are **no poll/dedupe knobs**.

**3. Per-camera opt-in (device config, not env):** a camera does ANPR only when its config has **both**
`anpr: true` **and** a relay binding (`controllerId` + `relay`). The `anpr` flag is a **checkbox on the
camera form in the [[first-run-setup|SetupWizard]]** (built 2026-06-19). Without the binding the
[[entry-exit-points|dispatcher]] refuses every read ("reader not bound to a barrier") — so an
unbound ANPR camera recognizes but every read is rejected (and logged with its snapshot).

**4. Footer health:** when `VISION_ENABLED`, the [[device-status-monitoring|DeviceMonitor]] probes the
service's `/health` each tick and shows a **"Vision" chip** in the booth footer (ready/degraded/offline
+ the recognizer name); no chip when disabled. So the operator sees at a glance whether vision is up.

> **Network isolation** ([[network-isolation]]): cameras live on the isolated device VLAN, so the
> vision service must reach that VLAN to pull snapshots — but its own `/analyze` should bind
> **localhost** (Node is the only caller). Keep the AGPL/heavy stack contained to this process.

> **Hardening / fix backlog** ([[vision-service-hardening]]): the 2026-07-02 code + security reviews
> logged a prioritised to-do list against `apps/vision/` — DoS gaps (body-cap-after-buffering,
> pixel-bomb, inference on the async event loop), unauthenticated + operator-writable model weights,
> and `0.0.0.0`-by-default binding lead it. **Consult it before touching this service.**

## Open

- **Recognizer choice** — **fast-alpr (MIT, YOLOv9+CCT on ONNX) is the baseline, AL-benchmarked**: the
  default `cct-xs-v2-global-model` won over the EU model on real AL plates (table above). The one
  remaining open item is the **model-weight-provenance check** (the MIT-weights claim). A re-benchmark
  on real *on-site* captures (angled/night/dirty) is wanted once cameras are installed. See
  [[vision-service]]; AGPL still permitted in-service for the stronger fallback.
- **Vehicle fingerprint**: attribute classifier vs. embedding-similarity; what threshold makes a
  mismatch an anomaly without false-positiving on lighting/angle.
- **Compute footprint** on the appliance (CPU-only vs. a small GPU/NPU) — procurement input
  ([[bom]], [[open-questions]]).
- Per-camera **opt-in** — ✅ **built**: `config.anpr === true` enables ANPR on a camera (set via the
  SetupWizard checkbox); ANPR then runs on that camera's entry/exit snapshot.

## Vehicle body type (advisory) — the vehicle stage, phase A (2026-09-06)

> Phase B (the classifier that knows SUV from sedan), its training loop and the hardware it runs on
> are decided on [[bodytype-classifier-training]] — not built yet.

`/analyze` populates `vehicle.body_type` + `vehicle.confidence` from the shared vocabulary
(car, sedan, hatchback, suv, minivan, pickup, van, truck, bus, motorcycle). Node records it beside
the plate and the Car Wash desk pre-selects the category the site maps it to; the operator
decides, a confident downgrade is flagged, nothing is gated on it ([[venue-modules]] §Vehicle
category from vision).

**Phase A = YOLOX-S (Megvii, Apache-2.0) as ONNX** on the same ONNX Runtime the plate stage uses
— the licence rule that keeps Ultralytics (AGPL) out. `vision_service/vehicle.py`: pure numpy/cv2
letterbox (pad 114, raw 0–255 BGR — YOLOX's exported graphs are not normalised), stride-grid
decode, class-agnostic NMS, COCO `car/motorcycle/bus/truck` → the vocabulary, and ONE vehicle per
frame: the box holding the plate's centre when a plate was read (the car that was read, not the
one behind), else the largest box. `WithVehicle` in `recognizer.py` wraps whichever plate
recognizer runs (stub included, so the stage is testable without fast-alpr); a failing stage
yields `vehicle: null` and a `vehicle: …` note in `/health.detail` — it never costs the plate
read. Composed `model_version` reads `<plate>+yolox:yolox_s.onnx@640`.

- **Config:** `VISION_VEHICLE_MODEL_PATH` (unset = stage off), `VISION_VEHICLE_INPUT_SIZE` (640),
  `VISION_VEHICLE_MIN_CONFIDENCE` (0.4 — the detector's floor; the SITE threshold that decides a
  flag lives in Setup → Car wash). The Docker image bakes the weights at `/app/models/yolox_s.onnx`
  (best-effort curl at build; no network → stage off) and sets the path, so the air-gapped
  appliance never fetches at runtime and no operator-writable path holds a model
  ([[vision-service-hardening]]). Compose forwards the var; set it EMPTY in the stack env to
  switch the stage off. Locally: curl the release file into `apps/vision/models/` (gitignored).
- **Measured on dev (2026-09-06), four real 2560×1440 entry frames from the DS-2CD1047G3H:** three
  with a car → `car` at 0.83–0.88, ~240–330 ms each on the dev CPU with 2 intra-op threads; the
  empty-lane frame with a person at the camera → no vehicle (correct: a person is not a class we
  keep). One frame per entry, so the cost is invisible to the lane.
- **What it cannot do:** SUV vs sedan — COCO has one `car`. For a Vetura/SUV price list every car
  maps to Vetura and no downgrade fires; vans, trucks, buses and motorcycles do separate. Phase B
  (a body-type classifier on the pilot's own frames — every wash order is a labelled frame) is
  what closes that gap; the detector's box is the crop it will classify.
- **Training (user asked, 2026-09-06): YOLOX-S needs none.** It ships trained on COCO and is a
  finished detector; its limit is vocabulary, not quality. Phase B is a *different, smaller*
  model — a classifier over the detector's crop, not a retrained detector — fine-tuned on a small
  Apache-licensed backbone. Data: a few hundred crops per category to start, a couple of
  thousand is comfortable, all from the pilot's own lane and camera. Labels are NOT the
  operator's picks (untrusted — [[threat-model]]) but a trusted reviewer's, gathered through the
  [[vision-review-outbox]]. Expect 85–95 % on frontal gate views once tuned — enough to flag,
  never to bill, which is why the flag records and the site threshold exists.