feat(trainer): training from the collector UI — the trainer becomes a job service, the review page gains a Training section
Trainer: `parking-trainer serve` — a stdlib HTTP job API on the compose network (never
published): /health, /readiness, /versions, /versions/<v>/report, /jobs. One job at a
time; each job runs the CLI as a subprocess with its output captured, state + log
persisted under /out/jobs/ so a restart keeps history. `publish` takes its URL from
TRAINER_PUBLISH_URL. Dockerfile: CMD serve, EXPOSE 8091, healthcheck.
Collector: COLLECTOR_TRAINER_URL + /api/training/{status,jobs,jobs/:id,versions/:v/report}
— a reviewer-gated proxy that forwards a fixed set of paths and whitelisted knobs and
passes the trainer's status codes through (409 while a job runs; 503 unconfigured, 502
unreachable). /review gains the Training section: labels per class vs the minimum with
Train disabled until two classes clear it, mode / backbone / floor, the running job's
live log, the versions with Report / Evaluate / Publish (publish confirms), and the
reminder that pinning stays a git commit. Fixed on the way: an apostrophe in the page's
inline script broke the whole page — a test now parses the script.
Compose: `trainer` is a service (restart: unless-stopped, read-only data volume, its own
trainer-out volume), the `train` profile and TRAINER_OUT are gone; the Docker-socket
route was rejected (root on the host for a service booths upload to). Verified with both
images running together: a Train started through the proxy finished, version and report
came back, the page rendered.
Wiki: bodytype-classifier-training (loop, running it, operating notes superseded),
vision-review-outbox, fleet-deployment-komodo, log.
Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
@@ -18,8 +18,13 @@ Xeon". This page is the loop as built; what is still outstanding is at the end.
|
||||
Each step is a place where a person decides. Nothing here runs on its own.
|
||||
|
||||
1. **Train** — `apps/trainer` (`parking-trainer`, Python/uv like the vision service; its own
|
||||
image `parking-trainer`, a one-off job on the collector's host — never a booth service).
|
||||
`train` reads the collector's `collector.sqlite` and `crops/` **straight off the volume**
|
||||
image `parking-trainer`, the `trainer` service beside the collector on the reviewer's host —
|
||||
never a booth service). **Started from the collector's UI:** the Training section of
|
||||
`/review` (readiness, a Train button with mode / backbone / floor, the live log, the
|
||||
versions with Report / Evaluate / Publish) drives a small job API the trainer serves on the
|
||||
compose network (`serve`; stdlib HTTP, one job at a time, each job the CLI as a subprocess
|
||||
with its log persisted under `/out/jobs/`). The collector proxies it behind the reviewer's
|
||||
login; the trainer is never published. `train` reads the collector's `collector.sqlite` and `crops/` **straight off the volume**
|
||||
(read-only), takes only reviewed, usable rows (the operator's pick and the camera's class are
|
||||
never labels), **splits by TIME** (validation = the newest 20 % by *time seen*, so the number
|
||||
reflects tomorrow's traffic), drops classes with fewer than `--min-per-class` (20) labels from
|
||||
@@ -129,47 +134,44 @@ What the owner has: an **NVIDIA Quadro FX 3800** (in hand, not installed), and i
|
||||
host, so nothing moves.
|
||||
- **Consequences for the build (done):** the trainer image is **CPU-only PyTorch** (torch
|
||||
2.14+cpu, ~200 MB of wheels, not the ~5 GB CUDA build); the `trainer` service in
|
||||
`docker-compose.collector.yml` is real now — `profiles: ["train"]`, no device reservation
|
||||
(one block to add if a modern card ever lands; the trainer would pick up CUDA), the collector
|
||||
volume mounted read-only, output to `TRAINER_OUT` on the host (default `./models` beside the
|
||||
compose file).
|
||||
`docker-compose.collector.yml` is real — always on, serving the job API, no device
|
||||
reservation (one block to add if a modern card ever lands; the trainer would pick up CUDA),
|
||||
the collector volume mounted read-only, models/reports/logs in its own `trainer-out` volume.
|
||||
- **If faster is ever wanted:** a used mid-range card of the last few generations (~€200) turns
|
||||
the hour into a minute, given a slot and a PSU. **Renting a cloud GPU is rejected**: the crops
|
||||
would leave the premises, and even scrubbed of plates and site that runs against the whole
|
||||
privacy design of the outbox.
|
||||
|
||||
## Running it (on the collector host)
|
||||
## Running it
|
||||
|
||||
```
|
||||
docker compose -f docker-compose.collector.yml --profile train run --rm trainer inspect
|
||||
docker compose -f docker-compose.collector.yml --profile train run --rm trainer train --min-accuracy 0.85
|
||||
docker compose -f docker-compose.collector.yml --profile train run --rm trainer evaluate --model /out/<version>/bodytype.onnx
|
||||
docker compose -f docker-compose.collector.yml --profile train run --rm trainer publish /out/<version> --url https://git.infra.msai.al/api/packages/mca/generic/parking-bodytype
|
||||
```
|
||||
From the collector's `/review` page, Training section: **Train** (mode, backbone, floor) when
|
||||
readiness says enough labels; watch the log; read the report under Versions; **Evaluate** a
|
||||
written version against labels reviewed since; **Publish** it (needs `TRAINER_PUBLISH_TOKEN`
|
||||
in the `wash-collector` stack — commented until the first publish). Then, in git: write the
|
||||
version into `apps/vision/models/bodytype.version`, commit, let the build produce the image,
|
||||
bump the booth's `TAG`. The pin stays a commit on purpose — it is the deploy control.
|
||||
|
||||
Then: write the version into `apps/vision/models/bodytype.version`, commit, let the build produce
|
||||
the image, bump the booth's `TAG`. The trainer is never started by a deploy (a profile), and the
|
||||
`wash-collector` stack's `TRAINER_OUT` / `TRAINER_PUBLISH_TOKEN` lines stay commented until the
|
||||
first publish.
|
||||
The CLI is still there for debugging, inside the running container:
|
||||
`docker compose -f docker-compose.collector.yml exec trainer parking-trainer inspect`.
|
||||
|
||||
## Operating notes (first deploy, 2026-09-07)
|
||||
## Operating notes (2026-09-07)
|
||||
|
||||
- **Only the collector shows as running — that is correct.** The trainer is not a service; it is
|
||||
behind the `train` compose *profile*, so a deploy never starts it and `docker ps` on
|
||||
`art-docker-station` lists one container. The trainer runs when invoked by hand, does its job,
|
||||
exits, and leaves nothing behind (`--rm`).
|
||||
- **A deploy does not pull profile services either.** The first `run` pulls the image itself, so
|
||||
the host's Docker must be logged in to the registry (`docker compose -f
|
||||
docker-compose.collector.yml --profile train pull trainer` is the check; if refused, `docker
|
||||
login git.infra.msai.al` on the host first).
|
||||
- **Why the wash-collector TAG bump mattered** although the collector code did not change: the
|
||||
trainer service uses the stack's TAG, so a bump makes `run` resolve to an image that exists
|
||||
(`stage-f7a262a` is the first tag that carries `parking-trainer`).
|
||||
- **park-2 does not need the bump** until a model is pinned: the new vision image ships with an
|
||||
- **First deploy (`stage-f7a262a`) shipped the trainer as a compose *profile*** — a one-off
|
||||
job the owner had to start by hand with `docker compose … --profile train run …` from
|
||||
wherever Komodo's periphery had cloned the repo (`/etc/komodo/stacks/wash-collector/`).
|
||||
The user rightly called that "not so smart": the host runs a periphery, and the reviewer is
|
||||
already in the collector's UI. **Superseded the same day:** the trainer is now a
|
||||
**service** (`restart: unless-stopped`, the `serve` command) and the collector's
|
||||
`/review` page carries the Training section. A deploy starts both containers; `docker ps`
|
||||
shows two.
|
||||
- **Why not a Docker socket in the collector** (the other way to a button): it would hand
|
||||
root on the host to a service that accepts uploads from booths — the party the
|
||||
[[threat-model]] distrusts. The job API keeps the trainer a normal container with a
|
||||
read-only data mount and its own `trainer-out` volume.
|
||||
- **park-2 does not need a bump** until a model is pinned: the vision image ships with an
|
||||
empty `bodytype.version`, phase B off, nothing for a booth to gain.
|
||||
- **What to run, from the stack's directory on the host:** `inspect` first (label counts per
|
||||
class, `ready: false` until enough are reviewed), then `train`, read `report.md` under
|
||||
`TRAINER_OUT/<version>/`, then `publish`, pin, push, bump the booth. Commands under §Running it.
|
||||
- **Reviewing is the bottleneck**: the Training section shows labels per class against the
|
||||
minimum and keeps Train disabled until two classes clear it.
|
||||
|
||||
## Packaging rule (same as the vision service)
|
||||
|
||||
|
||||
@@ -158,11 +158,10 @@ collector ([[vision-review-outbox]]) runs on the reviewer's GPU host as its own
|
||||
(`wash-collector`, `server = "art-docker-station"`, `file_paths = ["docker-compose.collector.yml"]`).
|
||||
Same repo, branch and pinned `TAG` promotion, its own secret references, and — because a stack
|
||||
names its compose files — nothing booth-side lands on that host and nothing of it on a booth.
|
||||
The same stack carries the phase-B **trainer** as a compose *profile* (`train`,
|
||||
[[bodytype-classifier-training]]): a deploy never starts it; the owner runs it by hand on the host
|
||||
with `docker compose … --profile train run --rm trainer …`. So after a deploy of that stack
|
||||
`docker ps` shows one container — expected; and a deploy pulls nothing for the profile, the
|
||||
first `run` does (host Docker must be logged in to the registry). Its two env lines (`TRAINER_OUT`, the
|
||||
The same stack carries the phase-B **trainer** as a second service ([[bodytype-classifier-training]]):
|
||||
a deploy starts both, `docker ps` shows two containers, and the trainer is driven from the
|
||||
collector's UI, never from the host's shell (a first cut as a compose *profile* run by hand was
|
||||
replaced the same day — the host runs a periphery, nobody should be typing compose there). Its two env lines (`TRAINER_OUT`, the
|
||||
`TRAINER_PUBLISH_TOKEN` secret reference) stay commented in `resources.toml` until the first
|
||||
publish.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user