fix(collector,trainer): migrate an existing collector DB on open; trainer handlers answer 500 JSON

The reviewer host's collector.sqlite was created by an earlier build, before the
`kind` column. CREATE TABLE IF NOT EXISTS shapes only a new database, so every query
naming the column failed: the collector's /health (container unhealthy), every
booth ingest, and the trainer's readiness — whose stdlib server printed the
traceback and dropped the socket, which the collector could only render as
"trainer not reachable: fetch failed". Nine days like that.

- CollectorDb.#migrate(): PRAGMA table_info against the list of columns added
  since the first deploy; ALTER TABLE ADD COLUMN for each missing one (all
  nullable or defaulted). Append to that list whenever a column joins the CREATE.
  Test replays the original schema: health, ingest, stats, a legacy row reads
  back with the defaults.
- Trainer Handler._guarded(): any unexpected exception → 500 JSON naming it,
  never a dropped connection; /health keeps answering. Test drives readiness
  against an old-schema DB.
- The collector's training status proxy includes the trainer's error text.

Wiki: the incident and the schema rule (vision-review-outbox), what the message
means (bodytype-classifier-training), log. Deploy: the new collector migrates on
start; nothing manual.

Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
2026-09-16 10:33:47 +02:00
parent fe3b12a60d
commit f4b806a538
8 changed files with 186 additions and 2 deletions
+20
View File
@@ -154,6 +154,26 @@ its own repo the day it needs its own cadence. Deploy the collector BEFORE a boo
package kind it does not know (a 422 is abandoned, not retried). The export neutralises cells
that start like a spreadsheet formula (category/service names are booth-supplied text).
> **Incident 2026-09-16 — the collector's DB predated the `kind` column; nothing worked for 9
> days and nothing said so.** The reviewer opened /review: *Training — trainer not reachable:
> fetch failed*. On the host: collector `stage-2d9bb15` **unhealthy** (`/health` → 500 *no such
> column: kind*), trainer healthy but every `/readiness` a Python traceback; the volume's
> `collector.sqlite` (created 2026-09-07 by the previous build, **0 items**) had the original
> column set. `CREATE TABLE IF NOT EXISTS` shapes only a NEW database — an existing volume keeps
> its old columns, so every query naming `kind` failed: the collector's health, **every ingest**
> (booths would have got 500s and kept retrying — the log shows none ever arrived, a separate
> question), and the trainer's readiness. The trainer's stdlib server printed the traceback and
> dropped the socket, which the collector could only render as "fetch failed".
>
> Fixes (same day): `CollectorDb` now **migrates on open** — `PRAGMA table_info` vs a list of the
> columns added since the first deploy, `ALTER TABLE … ADD COLUMN` for each missing one (all
> nullable or defaulted; **append to that list whenever a column joins the CREATE**); the trainer's
> handlers are guarded — an unexpected exception is a **500 JSON** naming the error, never a
> dropped connection; the collector's status proxy surfaces the trainer's error text. Rule going
> forward: the collector owns the schema; the trainer only reads; a deploy that changes the table
> must be accompanied by a migration entry, and the Training section is the first place a
> schema/DB mismatch shows — read its error text before suspecting the network.
**Status (2026-09-07).** Live: the collector runs on `art-docker-station` and park-2 is wired to
it (`stage-dbbb051` on both stacks, every entry sampled). The review screen at
`http://docker-station.nb.infra:8090/review` is filling; no labels reviewed yet.
@@ -173,6 +173,13 @@ The CLI is still there for debugging, inside the running container:
- **Reviewing is the bottleneck**: the Training section shows labels per class against the
minimum and keeps Train disabled until two classes clear it.
**"Training — trainer not reachable: fetch failed" (2026-09-16).** Not a network problem: the
trainer answered `/health` but its `/readiness` crashed on the collector's DB (a volume from before
the `kind` column) and the stdlib server dropped the socket without a reply. Since the fix the
trainer answers **500 JSON with the error** and the collector shows that text; a genuine network
failure still reads "fetch failed" / ECONNREFUSED. See [[vision-review-outbox]] §Incident 2026-09-16.
## Packaging rule (same as the vision service)
Core deps are light (numpy, opencv-headless, onnxruntime): `inspect`, `evaluate`, the data and
+14
View File
@@ -3308,3 +3308,17 @@ Live after the fix: degraded "cover open, paper out, printer off-line"; closed
"we are good using the network with this printer." Also: park-lab Periphery was "not loaded"
after a reboot — the unit had never been enabled; `systemctl --user enable --now periphery`
recorded as a §7a gotcha on [[appliance-provisioning]]. Pages: [[k200l-printer]].
## [2026-09-16] fix | Collector DB schema migration; trainer handlers answer 500 JSON; the "trainer not reachable" incident
User: "Training — trainer not reachable: fetch failed". Read-only look at art-docker-station over
SSH: collector stage-2d9bb15 unhealthy (`/health` 500 "no such column: kind"), trainer healthy
but `/readiness` tracebacks on the same column; the volume's collector.sqlite (2026-09-07, 0
items, original columns) predates `kind` — CREATE TABLE IF NOT EXISTS never migrates an existing
table. No `/ingest` request in the container's 9-day log at all (park-2 either not sending or not
reaching the host — to check on the booth). Built: `CollectorDb.#migrate()` (PRAGMA table_info vs
the list of columns added since the first deploy → ALTER TABLE ADD COLUMN; test replays the old
schema: health, ingest, stats, legacy row reads back with defaults); trainer `Handler._guarded`
(any exception → 500 JSON naming it; test: readiness on an old-schema DB → 500 "no such column:
kind", /health still 200); the collector's training proxy includes the trainer's error text. Pages:
[[vision-review-outbox]] (incident + rule), [[bodytype-classifier-training]] (what the message
means). Deploy: nothing manual — the new collector migrates on start.