feat(trainer): training from the collector UI — the trainer becomes a job service, the review page gains a Training section
Trainer: `parking-trainer serve` — a stdlib HTTP job API on the compose network (never
published): /health, /readiness, /versions, /versions/<v>/report, /jobs. One job at a
time; each job runs the CLI as a subprocess with its output captured, state + log
persisted under /out/jobs/ so a restart keeps history. `publish` takes its URL from
TRAINER_PUBLISH_URL. Dockerfile: CMD serve, EXPOSE 8091, healthcheck.
Collector: COLLECTOR_TRAINER_URL + /api/training/{status,jobs,jobs/:id,versions/:v/report}
— a reviewer-gated proxy that forwards a fixed set of paths and whitelisted knobs and
passes the trainer's status codes through (409 while a job runs; 503 unconfigured, 502
unreachable). /review gains the Training section: labels per class vs the minimum with
Train disabled until two classes clear it, mode / backbone / floor, the running job's
live log, the versions with Report / Evaluate / Publish (publish confirms), and the
reminder that pinning stays a git commit. Fixed on the way: an apostrophe in the page's
inline script broke the whole page — a test now parses the script.
Compose: `trainer` is a service (restart: unless-stopped, read-only data volume, its own
trainer-out volume), the `train` profile and TRAINER_OUT are gone; the Docker-socket
route was rejected (root on the host for a service booths upload to). Verified with both
images running together: a Train started through the proxy finished, version and report
came back, the page rendered.
Wiki: bodytype-classifier-training (loop, running it, operating notes superseded),
vision-review-outbox, fleet-deployment-komodo, log.
Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
@@ -22,29 +22,30 @@ services:
|
||||
COLLECTOR_REVIEWER_USER: ${COLLECTOR_REVIEWER_USER:-reviewer}
|
||||
COLLECTOR_REVIEWER_PASS: ${COLLECTOR_REVIEWER_PASS:?set COLLECTOR_REVIEWER_PASS in the stack env}
|
||||
LOG_LEVEL: ${LOG_LEVEL:-info}
|
||||
# The trainer's job API (sibling service above). Unset = no Training section.
|
||||
COLLECTOR_TRAINER_URL: ${COLLECTOR_TRAINER_URL-http://trainer:8091}
|
||||
volumes:
|
||||
- collector-data:/data
|
||||
|
||||
# Phase B trainer — a ONE-OFF JOB on this host's CPU, not a service (profile "train": it
|
||||
# only runs when asked). Reads the collector's SQLite + crops straight off the same volume
|
||||
# (read-only), writes a versioned model folder under TRAINER_OUT on the host. CPU-only
|
||||
# PyTorch: the Xeon E3-1225 v5 trains a few thousand crops in minutes (features mode) to an
|
||||
# hour (full fine-tune) — see wiki/decisions/bodytype-classifier-training.md. If a modern GPU
|
||||
# ever lands in the host, add an nvidia device reservation here; the trainer picks up CUDA.
|
||||
#
|
||||
# docker compose -f docker-compose.collector.yml --profile train run --rm trainer inspect
|
||||
# docker compose -f docker-compose.collector.yml --profile train run --rm trainer train --min-accuracy 0.85
|
||||
# docker compose -f docker-compose.collector.yml --profile train run --rm trainer evaluate --model /out/<version>/bodytype.onnx
|
||||
# docker compose -f docker-compose.collector.yml --profile train run --rm trainer publish /out/<version> --url <gitea generic package url>
|
||||
# Phase B trainer — a small always-on job service beside the collector (CPU-only torch;
|
||||
# idle it is a tiny Python HTTP server, torch loads only when a job runs). It reads the
|
||||
# collector's SQLite + crops off the same volume (read-only) and keeps models, reports and
|
||||
# job logs in its own volume. NOT published: only the collector reaches it, on this compose
|
||||
# network, and the reviewer's login on the collector is the gate. The Training section of
|
||||
# /review is its UI (readiness, Train / Evaluate / Publish, reports, logs).
|
||||
# See wiki/decisions/bodytype-classifier-training.md.
|
||||
trainer:
|
||||
image: ${REGISTRY:-git.infra.msai.al/mca/parking_solution}/parking-trainer:${TAG:-dev}
|
||||
profiles: ["train"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
# Only `publish` needs it: a Gitea token with package:write for the model's generic package.
|
||||
# Where `publish` PUTs a passing model (a Gitea generic package) and the token it uses
|
||||
# (package:write). Only publishing needs the token; training runs without it.
|
||||
TRAINER_PUBLISH_URL: ${TRAINER_PUBLISH_URL:-https://git.infra.msai.al/api/packages/mca/generic/parking-bodytype}
|
||||
TRAINER_PUBLISH_TOKEN: ${TRAINER_PUBLISH_TOKEN:-}
|
||||
volumes:
|
||||
- collector-data:/data:ro
|
||||
- ${TRAINER_OUT:-./models}:/out
|
||||
- trainer-out:/out
|
||||
|
||||
volumes:
|
||||
collector-data:
|
||||
trainer-out:
|
||||
|
||||
Reference in New Issue
Block a user