feat(vision): vehicle stage, phase A — YOLOX-S (Apache-2.0 ONNX) beside the plate recognizer
Fills /analyze vehicle.body_type + confidence (car / motorcycle / bus / truck from COCO, mapped to the shared vocabulary) for the Car Wash desk's category suggestion (venue-modules.md §Vehicle category from vision). Advisory: the operator decides, a confident downgrade is flagged, nothing is gated on it. - vision_service/vehicle.py: pure numpy/cv2 letterbox (pad 114, raw BGR), stride-grid decode, class-agnostic NMS, one vehicle per frame (the box holding the plate's centre, else the largest); YoloxVehicleDetector on onnxruntime CPU, 2 intra-op threads. - recognizer.py: WithVehicle composes the stage over any plate recognizer (stub included); a failing stage yields vehicle=null + a "vehicle: …" note in /health.detail — never costs the plate read. model_version reads "<plate>+yolox:yolox_s.onnx@640". - settings: VISION_VEHICLE_MODEL_PATH (unset = off), _INPUT_SIZE (640), _MIN_CONFIDENCE (0.4, the detector's floor; the flag threshold is site config). - Dockerfile bakes yolox_s.onnx (best-effort curl at build; no network → stage off) and sets the path; compose forwards it (empty = off); .env.example documents it. - Measured on four real dev entry frames (DS-2CD1047G3H, 2560×1440): car at 0.83–0.88 in ~240–330 ms; empty lane with a person → none. - tests/test_vehicle.py: decode/NMS/pick/letterbox on synthetic tensors, the composition, and a missing-model /health. Wiki: opencv-anpr-service, venue-modules, log. Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
@@ -20,3 +20,12 @@ VISION_OCR_MODEL=cct-xs-v2-global-model
|
|||||||
# Confidence floor — a best plate below this is flagged low_confidence so the Node side
|
# Confidence floor — a best plate below this is flagged low_confidence so the Node side
|
||||||
# treats it as advisory and falls back to the ticket path. Keep in sync with the server.
|
# treats it as advisory and falls back to the ticket path. Keep in sync with the server.
|
||||||
VISION_MIN_CONFIDENCE=0.5
|
VISION_MIN_CONFIDENCE=0.5
|
||||||
|
|
||||||
|
# Vehicle stage (phase A): a YOLOX ONNX graph (Apache-2.0) run on the same frame after the
|
||||||
|
# plate read; fills /analyze `vehicle.body_type` (car/truck/bus/motorcycle) + confidence for
|
||||||
|
# the Car Wash desk's category suggestion. Unset = off. The Docker image bakes the weights
|
||||||
|
# at /app/models/yolox_s.onnx; locally: curl the release file into apps/vision/models/.
|
||||||
|
# https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.onnx
|
||||||
|
# VISION_VEHICLE_MODEL_PATH=models/yolox_s.onnx
|
||||||
|
# VISION_VEHICLE_INPUT_SIZE=640
|
||||||
|
# VISION_VEHICLE_MIN_CONFIDENCE=0.4
|
||||||
|
|||||||
+10
-2
@@ -14,7 +14,7 @@ ENV UV_LINK_MODE=copy \
|
|||||||
|
|
||||||
# System libs the recognizer stack needs (opencv/onnxruntime): GL + glib. Kept minimal.
|
# System libs the recognizer stack needs (opencv/onnxruntime): GL + glib. Kept minimal.
|
||||||
RUN apt-get update \
|
RUN apt-get update \
|
||||||
&& apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \
|
&& apt-get install -y --no-install-recommends libgl1 libglib2.0-0 curl \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# ---- deps: resolve + install the venv from the lockfile (cache-friendly) ----
|
# ---- deps: resolve + install the venv from the lockfile (cache-friendly) ----
|
||||||
@@ -26,6 +26,13 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
|||||||
# ---- project source ----
|
# ---- project source ----
|
||||||
COPY vision_service/ ./vision_service/
|
COPY vision_service/ ./vision_service/
|
||||||
COPY README.md ./
|
COPY README.md ./
|
||||||
|
# Vehicle stage weights (phase A): YOLOX-S, Apache-2.0, ~36 MB, baked into the image so the
|
||||||
|
# air-gapped appliance never fetches at runtime and no operator-writable path holds a model
|
||||||
|
# (vision-service-hardening.md). Best-effort at build: without network the stage stays off.
|
||||||
|
ARG YOLOX_URL=https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.onnx
|
||||||
|
RUN mkdir -p /app/models \
|
||||||
|
&& (curl -fsSL -o /app/models/yolox_s.onnx "$YOLOX_URL" \
|
||||||
|
|| (echo "[build] yolox weights not fetched (no network) — vehicle stage off" && rm -f /app/models/yolox_s.onnx))
|
||||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||||
uv sync --frozen --extra alpr
|
uv sync --frozen --extra alpr
|
||||||
|
|
||||||
@@ -49,7 +56,8 @@ RUN uv run python -c "from fast_alpr import ALPR; ALPR()" \
|
|||||||
# Default to the stub recognizer (offline, no model load); override to fast_alpr in prod.
|
# Default to the stub recognizer (offline, no model load); override to fast_alpr in prod.
|
||||||
ENV VISION_RECOGNIZER=stub \
|
ENV VISION_RECOGNIZER=stub \
|
||||||
VISION_HOST=0.0.0.0 \
|
VISION_HOST=0.0.0.0 \
|
||||||
VISION_PORT=8089
|
VISION_PORT=8089 \
|
||||||
|
VISION_VEHICLE_MODEL_PATH=/app/models/yolox_s.onnx
|
||||||
EXPOSE 8089
|
EXPOSE 8089
|
||||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
||||||
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8089/health').status==200 else 1)" || exit 1
|
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8089/health').status==200 else 1)" || exit 1
|
||||||
|
|||||||
@@ -0,0 +1,143 @@
|
|||||||
|
"""Vehicle stage (phase A) — pure post-processing on synthetic tensors, and the
|
||||||
|
recognizer composition over the stub with a fake detector. No weights needed."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from vision_service.schemas import BBox, VehicleResult
|
||||||
|
from vision_service.vehicle import (
|
||||||
|
COCO_VEHICLE_CLASSES,
|
||||||
|
Detection,
|
||||||
|
decode,
|
||||||
|
letterbox,
|
||||||
|
nms,
|
||||||
|
pick_vehicle,
|
||||||
|
vehicles_from_output,
|
||||||
|
)
|
||||||
|
|
||||||
|
SIZE = 64 # tiny "model" input: grids 8x8 + 4x4 + 2x2 = 84 rows
|
||||||
|
ROWS = (SIZE // 8) ** 2 + (SIZE // 16) ** 2 + (SIZE // 32) ** 2
|
||||||
|
|
||||||
|
|
||||||
|
def raw_output(hits: list[tuple[int, int, int, float, float, float, float]]) -> np.ndarray:
|
||||||
|
"""Build a YOLOX-style raw tensor [ROWS, 85] with the given (row, coco_class, _, obj,
|
||||||
|
cls_score, log_w, log_h) hits; everything else is background."""
|
||||||
|
raw = np.zeros((ROWS, 85), dtype=np.float32)
|
||||||
|
raw[:, 2:4] = -10.0 # exp → ~0 size for background rows
|
||||||
|
for row, cls, _, obj, score, lw, lh in hits:
|
||||||
|
raw[row, 0:2] = 0.5 # centre of its grid cell
|
||||||
|
raw[row, 2] = lw
|
||||||
|
raw[row, 3] = lh
|
||||||
|
raw[row, 4] = obj
|
||||||
|
raw[row, 5 + cls] = score
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def test_decode_maps_grid_offsets_and_log_sizes_to_pixels() -> None:
|
||||||
|
raw = raw_output([(0, 2, 0, 1.0, 1.0, np.log(2.0), np.log(3.0))])
|
||||||
|
dec = decode(raw, SIZE)
|
||||||
|
# Row 0 = stride-8 grid cell (0,0): centre (0.5+0)*8 = 4, size exp(log 2)*8 = 16 / 24.
|
||||||
|
assert dec[0, :4].tolist() == [4.0, 4.0, 16.0, 24.0]
|
||||||
|
# Last row = stride-32 cell (1,1): centre (0.5+1)*32 = 48.
|
||||||
|
raw2 = raw_output([(ROWS - 1, 7, 0, 1.0, 1.0, 0.0, 0.0)])
|
||||||
|
dec2 = decode(raw2, SIZE)
|
||||||
|
assert dec2[ROWS - 1, :4].tolist() == [48.0, 48.0, 32.0, 32.0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_vehicles_only_above_floor_mapped_to_vocabulary_and_scaled_back() -> None:
|
||||||
|
raw = raw_output(
|
||||||
|
[
|
||||||
|
(0, 2, 0, 0.9, 0.9, np.log(2.0), np.log(2.0)), # car, score .81
|
||||||
|
(1, 0, 0, 0.99, 0.99, np.log(2.0), np.log(2.0)), # person → ignored
|
||||||
|
(2, 7, 0, 0.5, 0.5, np.log(2.0), np.log(2.0)), # truck, score .25 → below floor
|
||||||
|
]
|
||||||
|
)
|
||||||
|
found = vehicles_from_output(raw, SIZE, scale=0.5, min_confidence=0.4)
|
||||||
|
assert [d.body_type for d in found] == ["car"]
|
||||||
|
assert round(found[0].confidence, 2) == 0.81
|
||||||
|
# Box 16px wide in the letterboxed input → 32px in the original (scale 0.5).
|
||||||
|
assert round(found[0].x2 - found[0].x1) == 32
|
||||||
|
assert set(COCO_VEHICLE_CLASSES.values()) == {"car", "motorcycle", "bus", "truck"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_nms_keeps_the_best_of_overlapping_boxes() -> None:
|
||||||
|
boxes = np.array([[0, 0, 10, 10], [1, 1, 11, 11], [50, 50, 60, 60]], dtype=np.float32)
|
||||||
|
scores = np.array([0.5, 0.9, 0.7], dtype=np.float32)
|
||||||
|
assert sorted(nms(boxes, scores, 0.45)) == [1, 2]
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_prefers_the_box_holding_the_plate_else_the_largest() -> None:
|
||||||
|
near = Detection("car", 0.9, 0, 0, 100, 100)
|
||||||
|
far = Detection("truck", 0.8, 200, 200, 400, 400) # larger
|
||||||
|
inside = Detection("car", 0.7, 10, 10, 60, 60) # tighter box also holding the plate
|
||||||
|
assert pick_vehicle([near, far], None) is far
|
||||||
|
assert pick_vehicle([near, far], BBox(x1=20, y1=20, x2=30, y2=30)) is near
|
||||||
|
assert pick_vehicle([near, far, inside], BBox(x1=20, y1=20, x2=30, y2=30)) is inside
|
||||||
|
assert pick_vehicle([near, far], BBox(x1=900, y1=900, x2=910, y2=910)) is far # plate outside every box
|
||||||
|
assert pick_vehicle([], None) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_letterbox_keeps_aspect_and_pads_with_114() -> None:
|
||||||
|
frame = np.zeros((30, 60, 3), dtype=np.uint8)
|
||||||
|
tensor, scale = letterbox(frame, 64)
|
||||||
|
assert tensor.shape == (1, 3, 64, 64) and tensor.dtype == np.float32
|
||||||
|
assert abs(scale - 64 / 60) < 1e-9
|
||||||
|
assert tensor[0, 0, 63, 63] == 114.0 # padding
|
||||||
|
assert tensor[0, 0, 0, 0] == 0.0 # image
|
||||||
|
|
||||||
|
|
||||||
|
class FakeDetector:
|
||||||
|
model_version = "fake-vehicle"
|
||||||
|
|
||||||
|
def __init__(self, result: VehicleResult | None) -> None:
|
||||||
|
self.result = result
|
||||||
|
self.calls: list[BBox | None] = []
|
||||||
|
|
||||||
|
def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None:
|
||||||
|
self.calls.append(plate)
|
||||||
|
return self.result
|
||||||
|
|
||||||
|
|
||||||
|
def test_composition_fills_vehicle_over_the_stub_and_survives_a_failing_stage() -> None:
|
||||||
|
from vision_service.recognizer import StubRecognizer, WithVehicle
|
||||||
|
from vision_service.settings import Settings
|
||||||
|
|
||||||
|
det = FakeDetector(VehicleResult(body_type="truck", confidence=0.77))
|
||||||
|
rec = WithVehicle(StubRecognizer(Settings()), det)
|
||||||
|
res = rec.analyze(b"jpeg-bytes")
|
||||||
|
assert res.plate is None
|
||||||
|
assert res.vehicle == VehicleResult(body_type="truck", confidence=0.77)
|
||||||
|
assert res.model_version == "stub-0+fake-vehicle"
|
||||||
|
assert det.calls == [None]
|
||||||
|
|
||||||
|
class Boom:
|
||||||
|
model_version = "boom"
|
||||||
|
|
||||||
|
def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None:
|
||||||
|
raise RuntimeError("no model")
|
||||||
|
|
||||||
|
rec2 = WithVehicle(StubRecognizer(Settings()), Boom())
|
||||||
|
res2 = rec2.analyze(b"jpeg-bytes")
|
||||||
|
assert res2.vehicle is None
|
||||||
|
assert rec2.ready is True
|
||||||
|
assert "vehicle: RuntimeError: no model" in (rec2.error or "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_app_reports_a_missing_model_file_and_keeps_serving() -> None:
|
||||||
|
from vision_service.app import app
|
||||||
|
|
||||||
|
os.environ["VISION_VEHICLE_MODEL_PATH"] = "/nonexistent/yolox.onnx"
|
||||||
|
try:
|
||||||
|
with TestClient(app) as client:
|
||||||
|
health = client.get("/health").json()
|
||||||
|
assert health["ready"] is True # the plate stage (stub) is fine
|
||||||
|
assert "vehicle:" in (health["detail"] or "")
|
||||||
|
res = client.post("/analyze", content=b"x", headers={"content-type": "application/octet-stream"})
|
||||||
|
assert res.status_code == 200
|
||||||
|
assert res.json()["vehicle"] is None
|
||||||
|
finally:
|
||||||
|
os.environ.pop("VISION_VEHICLE_MODEL_PATH", None)
|
||||||
@@ -19,6 +19,7 @@ from typing import Protocol
|
|||||||
|
|
||||||
from .schemas import AnalyzeResponse, BBox, PlateResult
|
from .schemas import AnalyzeResponse, BBox, PlateResult
|
||||||
from .settings import Settings
|
from .settings import Settings
|
||||||
|
from .vehicle import VehicleDetector, YoloxVehicleDetector
|
||||||
|
|
||||||
|
|
||||||
class Recognizer(Protocol):
|
class Recognizer(Protocol):
|
||||||
@@ -165,10 +166,59 @@ class FastAlprRecognizer:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class WithVehicle:
|
||||||
|
"""Composition: any plate recognizer + the vehicle stage. Runs the plate stage first
|
||||||
|
(its box picks WHICH vehicle), then fills `vehicle`. A failing vehicle stage is
|
||||||
|
logged into `error` and yields null — it must never cost the plate read."""
|
||||||
|
|
||||||
|
def __init__(self, inner: Recognizer, detector: VehicleDetector) -> None:
|
||||||
|
self._inner = inner
|
||||||
|
self._detector = detector
|
||||||
|
self.vehicle_error: str | None = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def model_version(self) -> str:
|
||||||
|
return f"{self._inner.model_version}+{self._detector.model_version}"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def ready(self) -> bool:
|
||||||
|
return bool(self._inner.ready)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def error(self) -> str | None:
|
||||||
|
inner = getattr(self._inner, "error", None)
|
||||||
|
det = getattr(self._detector, "error", None) or self.vehicle_error
|
||||||
|
parts = [p for p in (inner, f"vehicle: {det}" if det else None) if p]
|
||||||
|
return "; ".join(parts) if parts else None
|
||||||
|
|
||||||
|
def analyze(self, image_bytes: bytes) -> AnalyzeResponse:
|
||||||
|
started = time.perf_counter()
|
||||||
|
res = self._inner.analyze(image_bytes)
|
||||||
|
try:
|
||||||
|
vehicle = self._detector.detect(image_bytes, res.plate.bbox if res.plate else None)
|
||||||
|
except Exception as exc: # noqa: BLE001 - advisory stage, never fatal
|
||||||
|
self.vehicle_error = f"{type(exc).__name__}: {exc}"
|
||||||
|
vehicle = None
|
||||||
|
took_ms = (time.perf_counter() - started) * 1000.0
|
||||||
|
return res.model_copy(
|
||||||
|
update={"vehicle": vehicle, "model_version": self.model_version, "took_ms": took_ms}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_recognizer(settings: Settings) -> Recognizer:
|
def build_recognizer(settings: Settings) -> Recognizer:
|
||||||
"""Factory: pick the recognizer from settings. Falls back to the stub if the real
|
"""Factory: pick the recognizer from settings. Falls back to the stub if the real
|
||||||
one can't load, so the service always comes up (with ready=False surfaced)."""
|
one can't load, so the service always comes up (with ready=False surfaced). The
|
||||||
|
vehicle stage wraps whichever recognizer runs when a model path is configured."""
|
||||||
|
rec: Recognizer
|
||||||
if settings.recognizer == "fast_alpr":
|
if settings.recognizer == "fast_alpr":
|
||||||
rec = FastAlprRecognizer(settings)
|
rec = FastAlprRecognizer(settings)
|
||||||
|
else:
|
||||||
|
rec = StubRecognizer(settings)
|
||||||
|
if settings.vehicle_model_path:
|
||||||
|
detector = YoloxVehicleDetector(
|
||||||
|
settings.vehicle_model_path,
|
||||||
|
input_size=settings.vehicle_input_size,
|
||||||
|
min_confidence=settings.vehicle_min_confidence,
|
||||||
|
)
|
||||||
|
return WithVehicle(rec, detector)
|
||||||
return rec
|
return rec
|
||||||
return StubRecognizer(settings)
|
|
||||||
|
|||||||
@@ -32,6 +32,16 @@ class Settings(BaseSettings):
|
|||||||
# Node side can fall back to the ticket path rather than trust it.
|
# Node side can fall back to the ticket path rather than trust it.
|
||||||
min_confidence: float = 0.5
|
min_confidence: float = 0.5
|
||||||
|
|
||||||
|
# Vehicle stage (phase A — venue-modules.md §Vehicle category from vision): a YOLOX
|
||||||
|
# ONNX graph (Apache-2.0) run beside the plate recognizer. Unset = stage off (the
|
||||||
|
# response's `vehicle` stays null). Bake the file into the image (models/), never a
|
||||||
|
# path an operator can write (vision-service-hardening.md).
|
||||||
|
vehicle_model_path: str | None = None
|
||||||
|
vehicle_input_size: int = 640
|
||||||
|
# Detection score floor for a vehicle box to count at all (the Node side applies the
|
||||||
|
# site's own, stricter threshold before it FLAGS anything).
|
||||||
|
vehicle_min_confidence: float = 0.4
|
||||||
|
|
||||||
|
|
||||||
def get_settings() -> Settings:
|
def get_settings() -> Settings:
|
||||||
return Settings()
|
return Settings()
|
||||||
|
|||||||
@@ -0,0 +1,247 @@
|
|||||||
|
"""Vehicle stage: a COCO object detector beside the plate recognizer (Job 2, phase A).
|
||||||
|
|
||||||
|
Answers "what KIND of vehicle is in this entry frame?" for the Car Wash desk's category
|
||||||
|
suggestion (wiki/decisions/venue-modules.md §Vehicle category from vision). ADVISORY by
|
||||||
|
design: the Node server records it next to the plate, the desk pre-selects the site
|
||||||
|
category it maps to, the operator decides, a confident downgrade is flagged. Nothing is
|
||||||
|
ever gated on it, so a wrong or missing detection costs nothing but a suggestion.
|
||||||
|
|
||||||
|
Model: YOLOX (Megvii, Apache-2.0) as an ONNX graph on the ONNX Runtime the plate stage
|
||||||
|
already uses — the licence rule that keeps Ultralytics (AGPL) out. COCO's vehicle classes
|
||||||
|
are car / motorcycle / bus / truck: enough to tell a van or a truck from a car, NOT enough
|
||||||
|
for SUV vs sedan — that is phase B (a body-type classifier on the pilot's own frames).
|
||||||
|
The detector's vehicle box is also the crop phase B will classify.
|
||||||
|
|
||||||
|
Pure numpy/cv2 pre/post-processing, no torch: letterbox to the model's square input
|
||||||
|
(pad 114, no normalisation — YOLOX's exported graphs take raw 0–255 BGR), decode the
|
||||||
|
stride grids, class-agnostic NMS, map COCO ids to the shared vocabulary, pick ONE
|
||||||
|
vehicle: the one whose box holds the plate (when a plate was read), else the largest.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Protocol
|
||||||
|
|
||||||
|
from .schemas import BBox, VehicleResult
|
||||||
|
|
||||||
|
# COCO-80 class index → the shared VEHICLE_CLASSES vocabulary (packages/shared).
|
||||||
|
COCO_VEHICLE_CLASSES: dict[int, str] = {2: "car", 3: "motorcycle", 5: "bus", 7: "truck"}
|
||||||
|
|
||||||
|
# YOLOX feature strides; grids are input/stride per level (8400 anchors at 640).
|
||||||
|
_STRIDES = (8, 16, 32)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Detection:
|
||||||
|
body_type: str
|
||||||
|
confidence: float
|
||||||
|
x1: float
|
||||||
|
y1: float
|
||||||
|
x2: float
|
||||||
|
y2: float
|
||||||
|
|
||||||
|
@property
|
||||||
|
def area(self) -> float:
|
||||||
|
return max(0.0, self.x2 - self.x1) * max(0.0, self.y2 - self.y1)
|
||||||
|
|
||||||
|
def contains(self, x: float, y: float) -> bool:
|
||||||
|
return self.x1 <= x <= self.x2 and self.y1 <= y <= self.y2
|
||||||
|
|
||||||
|
|
||||||
|
class VehicleDetector(Protocol):
|
||||||
|
"""What the recognizer composition needs: frame bytes (+ the plate box) → a class."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def model_version(self) -> str: ...
|
||||||
|
|
||||||
|
def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None: ...
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------------
|
||||||
|
# Pre/post-processing (pure functions — unit-tested on synthetic tensors)
|
||||||
|
# ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def letterbox(frame: Any, size: int) -> tuple[Any, float]:
|
||||||
|
"""Resize keeping aspect, pad bottom/right with 114 to size×size. Returns the CHW
|
||||||
|
float32 tensor (batch dim added) and the scale to map boxes back."""
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
h, w = frame.shape[:2]
|
||||||
|
r = min(size / h, size / w)
|
||||||
|
nh, nw = int(round(h * r)), int(round(w * r))
|
||||||
|
resized = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_LINEAR)
|
||||||
|
padded = np.full((size, size, 3), 114, dtype=np.uint8)
|
||||||
|
padded[:nh, :nw] = resized
|
||||||
|
tensor = padded.transpose(2, 0, 1)[None].astype(np.float32)
|
||||||
|
return np.ascontiguousarray(tensor), r
|
||||||
|
|
||||||
|
|
||||||
|
def decode(raw: Any, size: int) -> Any:
|
||||||
|
"""YOLOX raw output [N, 5+classes] (batch squeezed) → same shape with xywh decoded
|
||||||
|
into pixel units of the letterboxed input. Rows are ordered stride 8, 16, 32."""
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
out = raw.astype(np.float32).copy()
|
||||||
|
grids = []
|
||||||
|
strides = []
|
||||||
|
for s in _STRIDES:
|
||||||
|
n = size // s
|
||||||
|
ys, xs = np.meshgrid(np.arange(n), np.arange(n), indexing="ij")
|
||||||
|
grids.append(np.stack((xs, ys), axis=-1).reshape(-1, 2))
|
||||||
|
strides.append(np.full((n * n, 1), s, dtype=np.float32))
|
||||||
|
grid = np.concatenate(grids, axis=0).astype(np.float32)
|
||||||
|
stride = np.concatenate(strides, axis=0)
|
||||||
|
if out.shape[0] != grid.shape[0]:
|
||||||
|
raise ValueError(f"unexpected output rows {out.shape[0]} for input {size} (want {grid.shape[0]})")
|
||||||
|
out[:, :2] = (out[:, :2] + grid) * stride
|
||||||
|
out[:, 2:4] = np.exp(out[:, 2:4]) * stride
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def nms(boxes: Any, scores: Any, iou_threshold: float) -> list[int]:
|
||||||
|
"""Greedy class-agnostic non-max suppression over xyxy boxes; returns kept indices."""
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
if len(boxes) == 0:
|
||||||
|
return []
|
||||||
|
order = scores.argsort()[::-1]
|
||||||
|
x1, y1, x2, y2 = boxes[:, 0], boxes[:, 1], boxes[:, 2], boxes[:, 3]
|
||||||
|
areas = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
|
||||||
|
keep: list[int] = []
|
||||||
|
while order.size > 0:
|
||||||
|
i = int(order[0])
|
||||||
|
keep.append(i)
|
||||||
|
if order.size == 1:
|
||||||
|
break
|
||||||
|
rest = order[1:]
|
||||||
|
xx1 = np.maximum(x1[i], x1[rest])
|
||||||
|
yy1 = np.maximum(y1[i], y1[rest])
|
||||||
|
xx2 = np.minimum(x2[i], x2[rest])
|
||||||
|
yy2 = np.minimum(y2[i], y2[rest])
|
||||||
|
inter = np.clip(xx2 - xx1, 0, None) * np.clip(yy2 - yy1, 0, None)
|
||||||
|
iou = inter / (areas[i] + areas[rest] - inter + 1e-9)
|
||||||
|
order = rest[iou <= iou_threshold]
|
||||||
|
return keep
|
||||||
|
|
||||||
|
|
||||||
|
def vehicles_from_output(
|
||||||
|
raw: Any, size: int, scale: float, min_confidence: float, iou_threshold: float = 0.45
|
||||||
|
) -> list[Detection]:
|
||||||
|
"""Full post-processing: decode → vehicle classes only → confidence floor → NMS →
|
||||||
|
boxes in ORIGINAL frame pixels."""
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
dec = decode(raw, size)
|
||||||
|
cls_scores = dec[:, 5:]
|
||||||
|
cls_idx = cls_scores.argmax(axis=1)
|
||||||
|
score = dec[:, 4] * cls_scores[np.arange(len(dec)), cls_idx]
|
||||||
|
wanted = np.isin(cls_idx, list(COCO_VEHICLE_CLASSES)) & (score >= min_confidence)
|
||||||
|
if not wanted.any():
|
||||||
|
return []
|
||||||
|
d = dec[wanted]
|
||||||
|
s = score[wanted]
|
||||||
|
c = cls_idx[wanted]
|
||||||
|
boxes = np.stack(
|
||||||
|
(d[:, 0] - d[:, 2] / 2, d[:, 1] - d[:, 3] / 2, d[:, 0] + d[:, 2] / 2, d[:, 1] + d[:, 3] / 2), axis=1
|
||||||
|
)
|
||||||
|
keep = nms(boxes, s, iou_threshold)
|
||||||
|
out: list[Detection] = []
|
||||||
|
for i in keep:
|
||||||
|
b = boxes[i] / scale
|
||||||
|
out.append(
|
||||||
|
Detection(
|
||||||
|
body_type=COCO_VEHICLE_CLASSES[int(c[i])],
|
||||||
|
confidence=float(s[i]),
|
||||||
|
x1=float(b[0]),
|
||||||
|
y1=float(b[1]),
|
||||||
|
x2=float(b[2]),
|
||||||
|
y2=float(b[3]),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def pick_vehicle(detections: list[Detection], plate: BBox | None) -> Detection | None:
|
||||||
|
"""ONE vehicle per frame: the box holding the plate's centre (the car that was read —
|
||||||
|
a lane frame can show the car behind too), else the largest box (nearest the camera)."""
|
||||||
|
if not detections:
|
||||||
|
return None
|
||||||
|
if plate is not None:
|
||||||
|
cx = (plate.x1 + plate.x2) / 2
|
||||||
|
cy = (plate.y1 + plate.y2) / 2
|
||||||
|
holders = [d for d in detections if d.contains(cx, cy)]
|
||||||
|
if holders:
|
||||||
|
return min(holders, key=lambda d: d.area) # the tightest box around the plate
|
||||||
|
return max(detections, key=lambda d: d.area)
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------------
|
||||||
|
# The ONNX Runtime detector
|
||||||
|
# ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class YoloxVehicleDetector:
|
||||||
|
"""YOLOX ONNX on onnxruntime (CPU). Loads once; a load failure is surfaced through
|
||||||
|
`error` and the stage simply yields no vehicle (never breaks the plate path)."""
|
||||||
|
|
||||||
|
def __init__(self, model_path: str, input_size: int = 640, min_confidence: float = 0.4) -> None:
|
||||||
|
self._path = Path(model_path)
|
||||||
|
self._size = input_size
|
||||||
|
self._min_confidence = min_confidence
|
||||||
|
self._session = None
|
||||||
|
self._input_name = "images"
|
||||||
|
self._error: str | None = None
|
||||||
|
try:
|
||||||
|
import onnxruntime as ort
|
||||||
|
|
||||||
|
opts = ort.SessionOptions()
|
||||||
|
opts.intra_op_num_threads = 2 # one frame per entry; leave cores to the lane
|
||||||
|
self._session = ort.InferenceSession(
|
||||||
|
str(self._path), sess_options=opts, providers=["CPUExecutionProvider"]
|
||||||
|
)
|
||||||
|
self._input_name = self._session.get_inputs()[0].name
|
||||||
|
except Exception as exc: # noqa: BLE001 - not-ready, never fatal
|
||||||
|
self._error = f"{type(exc).__name__}: {exc}"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def model_version(self) -> str:
|
||||||
|
return f"yolox:{self._path.name}@{self._size}"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def ready(self) -> bool:
|
||||||
|
return self._session is not None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def error(self) -> str | None:
|
||||||
|
return self._error
|
||||||
|
|
||||||
|
def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None:
|
||||||
|
if self._session is None:
|
||||||
|
return None
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
frame = cv2.imdecode(np.frombuffer(image_bytes, dtype=np.uint8), cv2.IMREAD_COLOR)
|
||||||
|
if frame is None:
|
||||||
|
return None
|
||||||
|
tensor, scale = letterbox(frame, self._size)
|
||||||
|
raw = self._session.run(None, {self._input_name: tensor})[0][0]
|
||||||
|
found = vehicles_from_output(raw, self._size, scale, self._min_confidence)
|
||||||
|
best = pick_vehicle(found, plate)
|
||||||
|
if best is None:
|
||||||
|
return None
|
||||||
|
return VehicleResult(body_type=best.body_type, confidence=round(best.confidence, 4))
|
||||||
|
|
||||||
|
|
||||||
|
def time_detect(
|
||||||
|
detector: VehicleDetector, image_bytes: bytes, plate: BBox | None
|
||||||
|
) -> tuple[VehicleResult | None, float]:
|
||||||
|
"""detect() with wall time in ms (for logs/benchmarks)."""
|
||||||
|
started = time.perf_counter()
|
||||||
|
result = detector.detect(image_bytes, plate)
|
||||||
|
return result, (time.perf_counter() - started) * 1000.0
|
||||||
@@ -55,6 +55,9 @@ services:
|
|||||||
environment:
|
environment:
|
||||||
# Engine: stub (no models) by default; prod override sets fast_alpr.
|
# Engine: stub (no models) by default; prod override sets fast_alpr.
|
||||||
VISION_RECOGNIZER: ${VISION_RECOGNIZER:-stub}
|
VISION_RECOGNIZER: ${VISION_RECOGNIZER:-stub}
|
||||||
|
# Vehicle stage (Car Wash category suggestion): the image bakes YOLOX-S at this path.
|
||||||
|
# Set the var to an EMPTY value in the stack env to switch the stage off.
|
||||||
|
VISION_VEHICLE_MODEL_PATH: ${VISION_VEHICLE_MODEL_PATH-/app/models/yolox_s.onnx}
|
||||||
networks:
|
networks:
|
||||||
- parking
|
- parking
|
||||||
|
|
||||||
|
|||||||
@@ -188,11 +188,13 @@ onto the site's own categories ("car, sedan, hatchback → Vetura").
|
|||||||
and the read is at or above the threshold → one `anomaly` (`carwash.categoryDowngrade`, both
|
and the read is at or above the threshold → one `anomaly` (`carwash.categoryDowngrade`, both
|
||||||
categories, both prices, operator, snapshotId) and `downgrade_event_id` on the order. Equal,
|
categories, both prices, operator, snapshotId) and `downgrade_event_id` on the order. Equal,
|
||||||
upgrade, unsure or unmapped reads flag nothing. The order is always created.
|
upgrade, unsure or unmapped reads flag nothing. The order is always created.
|
||||||
- **Model — NOT built.** No bundled recognizer produces `body_type` yet, so today the desk shows
|
- **Model — phase A built (same day).** YOLOX-S (Apache-2.0 ONNX) as a vehicle stage beside the
|
||||||
nothing and nothing is flagged. Phase A = a COCO detector on Apache-2.0 ONNX weights (car /
|
plate recognizer: car / truck / bus / motorcycle + the vehicle crop, ~250 ms per entry frame on
|
||||||
truck / bus / motorcycle, plus the vehicle crop); Phase B = the body-type classifier trained on
|
CPU, weights baked into the vision image. Details and measurements on [[opencv-anpr-service]]
|
||||||
the pilot's own frames — every wash order is a labelled frame (entry snapshot + the category a
|
§Vehicle body type. Phase B = the body-type classifier trained on the pilot's own frames —
|
||||||
person chose), so the dataset builds itself on park-2. Reports (discrepancies per operator per
|
every wash order is a labelled frame (entry snapshot + the category a person chose), so the
|
||||||
|
dataset builds itself on park-2. Until then a Vetura/SUV list sees every car as Vetura and no
|
||||||
|
downgrade fires; van/truck/bus/motorcycle do separate. Reports (discrepancies per operator per
|
||||||
shift) wait for the first real reads.
|
shift) wait for the first real reads.
|
||||||
|
|
||||||
## Car Wash — the pilot module (settled 2026-09-05)
|
## Car Wash — the pilot module (settled 2026-09-05)
|
||||||
|
|||||||
@@ -249,11 +249,36 @@ service's `/health` each tick and shows a **"Vision" chip** in the booth footer
|
|||||||
- Per-camera **opt-in** — ✅ **built**: `config.anpr === true` enables ANPR on a camera (set via the
|
- Per-camera **opt-in** — ✅ **built**: `config.anpr === true` enables ANPR on a camera (set via the
|
||||||
SetupWizard checkbox); ANPR then runs on that camera's entry/exit snapshot.
|
SetupWizard checkbox); ANPR then runs on that camera's entry/exit snapshot.
|
||||||
|
|
||||||
## Vehicle body type (advisory) — contract only, 2026-09-06
|
## Vehicle body type (advisory) — the vehicle stage, phase A (2026-09-06)
|
||||||
|
|
||||||
`/analyze` may now populate `vehicle.body_type` + `vehicle.confidence` from the shared vocabulary
|
`/analyze` populates `vehicle.body_type` + `vehicle.confidence` from the shared vocabulary
|
||||||
(car, sedan, hatchback, suv, minivan, pickup, van, truck, bus, motorcycle). Node records it beside
|
(car, sedan, hatchback, suv, minivan, pickup, van, truck, bus, motorcycle). Node records it beside
|
||||||
the plate and the Car Wash desk pre-selects the category the site maps it to; the operator
|
the plate and the Car Wash desk pre-selects the category the site maps it to; the operator
|
||||||
decides, a confident downgrade is flagged, nothing is gated on it. No bundled recognizer emits
|
decides, a confident downgrade is flagged, nothing is gated on it ([[venue-modules]] §Vehicle
|
||||||
it yet — see [[venue-modules]] §Vehicle category from vision for the model plan (COCO detector
|
category from vision).
|
||||||
first, body-type classifier on own frames second).
|
|
||||||
|
**Phase A = YOLOX-S (Megvii, Apache-2.0) as ONNX** on the same ONNX Runtime the plate stage uses
|
||||||
|
— the licence rule that keeps Ultralytics (AGPL) out. `vision_service/vehicle.py`: pure numpy/cv2
|
||||||
|
letterbox (pad 114, raw 0–255 BGR — YOLOX's exported graphs are not normalised), stride-grid
|
||||||
|
decode, class-agnostic NMS, COCO `car/motorcycle/bus/truck` → the vocabulary, and ONE vehicle per
|
||||||
|
frame: the box holding the plate's centre when a plate was read (the car that was read, not the
|
||||||
|
one behind), else the largest box. `WithVehicle` in `recognizer.py` wraps whichever plate
|
||||||
|
recognizer runs (stub included, so the stage is testable without fast-alpr); a failing stage
|
||||||
|
yields `vehicle: null` and a `vehicle: …` note in `/health.detail` — it never costs the plate
|
||||||
|
read. Composed `model_version` reads `<plate>+yolox:yolox_s.onnx@640`.
|
||||||
|
|
||||||
|
- **Config:** `VISION_VEHICLE_MODEL_PATH` (unset = stage off), `VISION_VEHICLE_INPUT_SIZE` (640),
|
||||||
|
`VISION_VEHICLE_MIN_CONFIDENCE` (0.4 — the detector's floor; the SITE threshold that decides a
|
||||||
|
flag lives in Setup → Car wash). The Docker image bakes the weights at `/app/models/yolox_s.onnx`
|
||||||
|
(best-effort curl at build; no network → stage off) and sets the path, so the air-gapped
|
||||||
|
appliance never fetches at runtime and no operator-writable path holds a model
|
||||||
|
([[vision-service-hardening]]). Compose forwards the var; set it EMPTY in the stack env to
|
||||||
|
switch the stage off. Locally: curl the release file into `apps/vision/models/` (gitignored).
|
||||||
|
- **Measured on dev (2026-09-06), four real 2560×1440 entry frames from the DS-2CD1047G3H:** three
|
||||||
|
with a car → `car` at 0.83–0.88, ~240–330 ms each on the dev CPU with 2 intra-op threads; the
|
||||||
|
empty-lane frame with a person at the camera → no vehicle (correct: a person is not a class we
|
||||||
|
keep). One frame per entry, so the cost is invisible to the lane.
|
||||||
|
- **What it cannot do:** SUV vs sedan — COCO has one `car`. For a Vetura/SUV price list every car
|
||||||
|
maps to Vetura and no downgrade fires; vans, trucks, buses and motorcycles do separate. Phase B
|
||||||
|
(a body-type classifier on the pilot's own frames — every wash order is a labelled frame) is
|
||||||
|
what closes that gap; the detector's box is the crop it will classify.
|
||||||
|
|||||||
@@ -3091,3 +3091,12 @@ category + threshold; the desk pre-selects the mapped category and shows the rea
|
|||||||
a confident, pricier-mapped read with a cheaper choice signs `anomaly carwash.categoryDowngrade`
|
a confident, pricier-mapped read with a cheaper choice signs `anomaly carwash.categoryDowngrade`
|
||||||
(both categories/prices, operator, snapshot) — never blocks. No recognizer emits body_type yet.
|
(both categories/prices, operator, snapshot) — never blocks. No recognizer emits body_type yet.
|
||||||
Tests in carwash.test.ts. Updated [[venue-modules]] (As built), [[opencv-anpr-service]].
|
Tests in carwash.test.ts. Updated [[venue-modules]] (As built), [[opencv-anpr-service]].
|
||||||
|
|
||||||
|
## [2026-09-06] ingest | Vision vehicle stage, phase A: YOLOX-S beside the plate recognizer
|
||||||
|
`vision_service/vehicle.py` (YOLOX ONNX on onnxruntime: letterbox, grid decode, NMS, COCO
|
||||||
|
car/motorcycle/bus/truck → vocabulary, one vehicle per frame — the box holding the plate, else the
|
||||||
|
largest) + `WithVehicle` composition over any plate recognizer; `VISION_VEHICLE_MODEL_PATH` (unset =
|
||||||
|
off), input size, detector floor; Dockerfile bakes yolox_s.onnx (best-effort curl) and sets the
|
||||||
|
path; compose forwards it (empty = off). Measured on four real dev entry frames: car at 0.83–0.88,
|
||||||
|
~240–330 ms, empty lane → none. Tests: tests/test_vehicle.py (pure post-processing + composition +
|
||||||
|
missing-model health). Updated [[opencv-anpr-service]], [[venue-modules]].
|
||||||
|
|||||||
Reference in New Issue
Block a user