parking_solution/wiki/decisions/appliance-provisioning.md

type, tags, sources, updated, status

type	tags	sources	updated	status
reference	parking deployment appliance hardening runbook offline-first		2026-09-02	settled

Appliance provisioning runbook (booth PC)

Step-by-step to take a booth PC from factory Windows to a hardened, encrypted, container-running parking appliance. Written from the first real provisioning, 2026-06-23 (hardening) + first Komodo deploy, 2026-06-27 (the runtime) — every command here was run and verified on the actual hardware, including the firmware-specific workaround. Companion to disk-os-hardening (the why), tpm (TPM analysis), container-deployment (the images), and fleet-deployment-komodo (the deploy control plane this runbook's §7 uses).

⚠ This box is the threat-model defence. The load-bearing anti-fraud control is still reconciliation over the append-only-event-chain — disk encryption + Secure Boot raise the cost of offline tamper, they don't replace reconciliation.

Reference hardware (first unit, 2026-06-23)

• Dell OptiPlex 7070, Intel Core i5-8500 (Coffee Lake), 238 GB SATA SSD (/dev/sda).
• TPM 2.0 — discrete Nuvoton (Get-Tpm → ManufacturerIdTxt NTC, fw 7.2.1.0). NOT Intel PTT/fTPM. Discrete ⇒ an external LPC/SPI bus exists (bus-sniff is a theoretical physical attack on PCR-only sealing — accepted; see tpm). Used PC — previous owner irrelevant.
• Shipped Windows 11; formatted to Ubuntu 26.04 LTS (the decided platform — desktop-shell-tauri).

1. BIOS (F2 at the Dell logo)

• TPM: leave On. Used PC → Clear the TPM once (Security → TPM → Clear) so the prior owner's keys are wiped before LUKS enrollment. (PPI "Bypass for Clear" was unchecked → it asks for physical confirmation at next boot; that's normal.)
• Secure Boot: Enabled, Deployed Mode (not Audit). "Enable Custom Mode" UNCHECKED = Standard Mode with stock Microsoft keys — this is what Ubuntu's signed shim needs. Do NOT touch PK/KEK/db/dbx. NB: the 7070's Expert Key Management is edit-only (Save/Replace/Append/Delete — no read-only "View Key"), so you cannot inspect db from BIOS; verify via the live USB instead (step 2).
• Boot: UEFI only (no CSM/Legacy — a Legacy install has no Secure Boot / TPM-seal path).
• Set a BIOS admin password.

2. Boot the Ubuntu 26.04 USB (Secure Boot ON)

• Flash the ISO DIRECTLY (Rufus GPT/UEFI, Etcher, or dd). NOT Ventoy — Ventoy's own bootloader isn't in db, so Secure Boot rejects it with Verification failed: (0x1A) Security Violation (this is Secure Boot working correctly, not a fault). A directly-flashed Ubuntu USB boots the Microsoft-signed shim, which stock db trusts.
• F12 at the Dell logo → pick the USB under UEFI BOOT.
• Reaching the installer with Secure Boot ON = positive proof the MS third-party UEFI CA is in db (the verification the BIOS couldn't show us).

3. Encrypted install — the firmware workaround (IMPORTANT)

The 26.04 installer disk page offers: No Encryption / Encrypt with a passphrase / Use hardware-backed encryption (+ advanced LVM/ZFS, both ZFS experimental).

• "Use hardware-backed encryption" FAILS on this 7070 with: PCR_UNUSABLE … error with secure boot policy (PCR7) measurements: generating secure boot profiles for systems with timestamp revocation (dbt) support is currently not supported. → Ubuntu's automated FDE profiler can't model PCR7 on Dell firmware carrying a dbt (UEFI timestamp revocation list). It is NOT a TPM or Secure-Boot fault — both are fine.
• So: choose "Encrypt with a passphrase". Set a strong passphrase and SAVE IT OFF-MACHINE (phone / password manager). It is both the boot unlock (until TPM sealing) AND the permanent recovery slot. Finish the install.
• Result (verify with lsblk): sda1 vfat /boot/efi, sda2 ext4 /boot, sda3 crypto_LUKS → dm_crypt-0 (LVM2) → ubuntu--vg-ubuntu--lv ext4 /.

4. Seal LUKS to the TPM (manual — PCR 7 only)

Do this AFTER first boot. Manual enrollment sidesteps the installer's dbt profiler and lets us pick PCRs. Bind to PCR 7 only (Secure Boot state): it catches the attack that matters (disabling Secure Boot to boot a tampered kernel) WITHOUT breaking on routine kernel/GRUB updates (which churn PCRs 4/8/9 → would otherwise drop every boot to the passphrase). Firmware-only PCR 0 is the fallback if PCR 7 ever errors.

sudo apt update && sudo apt install -y tpm2-tools
sudo tpm2_pcrread sha256                 # sanity: PCRs 0-10 populated, PCR 7 has a real value

# Enroll the TPM (prompts for the EXISTING install passphrase to authorize the new slot):
sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 /dev/sda3

# Verify TWO slots — keep BOTH (slot 0 password = recovery, slot 1 tpm2 = auto-unlock):
sudo systemd-cryptenroll /dev/sda3
#   SLOT TYPE
#      0 password
#      1 tpm2

Wire it into boot (back up first; the mapping is dm_crypt-0, the LUKS UUID is in /etc/crypttab):

sudo cp /etc/crypttab /etc/crypttab.bak
sudo sed -i 's/none luks$/none luks,tpm2-device=auto/' /etc/crypttab
cat /etc/crypttab     # → dm_crypt-0 UUID=… none luks,tpm2-device=auto
sudo update-initramfs -u
sudo reboot

• Boots straight to login, no passphrase prompt = ✅ TPM auto-unlock works (unattended reboot achieved — VERIFIED on this unit 2026-06-23).
• Still prompts = PCR mismatch; type the passphrase (NOT locked out), then retry with --tpm2-pcrs=0. The password slot + crypttab.bak make this fully reversible.

Re-seal runbook: a BIOS update / Secure Boot change / UEFI dbx (revocation list) update alters PCR 7 → the TPM refuses → boot falls back to the passphrase prompt (not a brick). After such a change, re-run step 4's systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 /dev/sda3 to re-bind, then reboot to confirm unattended unlock returned.

4a. Firmware / UEFI dbx updates break PCR 7 — and are an OPERATOR threat (VERIFIED 2026-06-30)

The PCR-7 re-seal hazard above is not a rare event — the most common trigger is a UEFI dbx (Secure Boot revocation database) update, and it bit the real park-buzi booth on 2026-06-28:

• What dbx is: the Secure Boot blocklist of known-vulnerable bootloader/shim hashes (vendor = Microsoft). It is delivered by fwupd/LVFS — a channel SEPARATE from APT (the GNOME "Firmware Updater", which on Ubuntu is the firmware-updater snap, surfaces it). apt list --upgradable being clean does NOT mean a firmware/dbx update isn't pending.
• The GRUB panic (root cause): applying a new dbx against a stale GRUB/shim revokes the installed bootloader → Secure Boot refuses to load it → unbootable / GRUB "panic". The fix is ordering: apt full-upgrade (current grub-efi/shim-signed) FIRST, then dbx. A fresh reinstall ships a current GRUB, so reinstalling recovers it.
• It moves PCR 7: even with a current GRUB, applying dbx changes the Secure-Boot-policy measurement → the TPM (slot 1) refuses to release the key → next boot drops to the slot-0 passphrase prompt. Recover with the re-seal runbook above. VERIFIED: on park-buzi the dbx update went through, the box rebooted to a passphrase prompt, the slot-0 passphrase unlocked it, and systemd-cryptenroll --wipe-slot=tpm2 … --tpm2-pcrs=7 restored silent auto-unlock.

Threat-model consequence (threat-model: the operator is the adversary). A firmware/dbx update on a TPM-sealed booth → the booth won't boot unattended and needs the slot-0 passphrase. So the operator must be unable to trigger a firmware update, and must never hold the passphrase. Lock it down (DONE on park-buzi 2026-06-30):

# 1. Kill the firmware-update DAEMON (the GUI "Update" button then fails with no daemon):
sudo systemctl mask fwupd.service fwupd-refresh.timer
systemctl is-enabled fwupd.service fwupd-refresh.timer   # → masked / masked (persists across reboot)

# 2. Remove the operator-facing GUI so the screen is never even presented (Ubuntu = a snap):
sudo snap remove firmware-updater
snap list | grep -i firmware                              # → no output (re-check: seeded snaps can re-install)

Plus: the BIOS admin password (§1) must gate entering setup / changing settings (a supervisor/admin password, not just a boot password) so the operator can't disable Secure Boot or change boot order — either of which also breaks the seal. And the slot-0 passphrase stays off-machine / escrowed (same custody as EVENT_SIGNING_KEY / BACKUP_KEY); it is an admin-only recovery secret, used on-site during a maintenance window, never known to operators.

Net: firmware/dbx updates become an admin-only, on-site, deliberate action. The booth is unattended-bootable only while the firmware/Secure-Boot state is frozen — that is the security property, not a bug. Legitimate firmware maintenance now costs: physical presence + the slot-0 passphrase + a PCR-7 re-enroll.

⚠ Gotcha — cryptsetup … --test-passphrase SILENTLY passes via the TPM. Before any firmware/dbx change, you must prove a typed passphrase still unlocks the disk (the TPM-independent safety net). But sudo cryptsetup open --test-passphrase /dev/sda3 with a TPM2 token enrolled will succeed without prompting — the TPM auto-answers (it unlocks the tpm2 slot, e.g. slot 1), a FALSE positive that proves nothing about a human-typeable key. Force a real test with --disable-external-tokens (→ No usable token is available. then it prompts; success on slot 0 = the passphrase genuinely works):

sudo cryptsetup open --test-passphrase /dev/sda3 --disable-external-tokens --verbose

5. GRUB password — EDIT-ONLY (VERIFIED 2026-06-23)

Closes the init=/bin/bash / systemd.unit=rescue.target local-root hole: without it, anyone at the keyboard presses e at the GRUB menu, edits the kernel cmdline, and boots to a root shell with no login. The PCR-7 TPM seal does NOT cover this — editing the GRUB cmdline doesn't change PCR 7 (Secure Boot policy), so the TPM still releases the key and the attacker lands on the decrypted disk. This is the specific countermeasure for the threat-model. Use edit-only mode (--unrestricted) so the box still boots UNATTENDED — the password is required only to EDIT entries, never to boot.

grub-mkpasswd-pbkdf2          # enter a password (twice) → copy the grub.pbkdf2.sha512.* hash

Add the superuser (paste YOUR hash) to the end of /etc/grub.d/40_custom:

set superusers="admin"
password_pbkdf2 admin grub.pbkdf2.sha512.10000.<YOUR_HASH>

Make menu entries bootable WITHOUT the password (edit-only) — in /etc/grub.d/10_linux, set the active CLASS= line to include --unrestricted:

CLASS="--class gnu-linux --class gnu --class os --unrestricted"

Regenerate + VERIFY BOTH HALVES landed in the real config BEFORE rebooting (a GRUB misconfig means a rescue-USB recovery):

sudo update-grub
sudo grep -c "password_pbkdf2" /boot/grub/grub.cfg   # want ≥1 (password present)
sudo grep -c "unrestricted"    /boot/grub/grub.cfg   # want ≥1 (entries bootable w/o password)
sudo reboot

✅ VERIFIED on this unit: boots straight to login (no GRUB prompt, TPM still auto-unlocks) AND pressing e at the menu prompts for admin + password. Store the GRUB password off-machine (alongside the LUKS passphrase).

OS hardening on the first unit is now COMPLETE: LUKS FDE + TPM auto-unlock (PCR 7) + Secure Boot (Deployed) + GRUB edit-lock.

5c. OS user model — admin vs operator (VERIFIED 2026-06-23)

The OS has TWO roles and they must be different identities (threat-model: the operator is the adversary). Create a dedicated admin (real password, sudo, NO auto-login) and keep the operator as an auto-login, UNPRIVILEGED account.

sudo adduser admin && sudo usermod -aG sudo admin
# VERIFY in a second session: log in as admin → `sudo whoami` prints root — BEFORE the next step:
sudo gpasswd -d <operator> sudo          # demote the auto-login operator
groups <operator>                        # confirm: no 'sudo'

Use gpasswd -d, not deluser <user> <group>: on this Ubuntu the perl adduser tooling rejects hyphenated usernames (sanitize_string: invalid characters in 'park-operator' — VERIFIED on park-buzi 2026-07-06). And group removal applies at next login — the auto-login operator session keeps its old memberships until the box reboots (or the session relogs); re-verify groups from inside the operator session afterwards.

⚠ Order matters: confirm the new admin's sudo works before demoting the operator, or you lock yourself out. Keep auto-login on the OPERATOR, not admin. Leave root password disabled (Ubuntu default) — admin+sudo IS the root path; enabling root adds risk, no gain.

Strip latent escalation groups from the operator: sudo gpasswd -d <operator> lxd (lxd group = launch a privileged container that mounts host / as root — undoes the no-sudo hardening) and sudo gpasswd -d <operator> lpadmin (printer admin, unneeded). And NEVER add the operator to docker (also root-equivalent).

5b. Further hardening (TODO — not yet done)

• Key-based SSH only (disable password auth) if SSH is enabled at all. Routine ops no longer need SSH — Komodo Periphery (§7) drives deploys + gives a container terminal over the mesh — so SSH can be locked down hard or disabled, leaving the mesh + Komodo as the management path.
• No/locked-down desktop + kiosk autostart — single-purpose; the operator never reaches a shell (desktop-shell-tauri).
• Consider moving the host event-signing key into the TPM (non-extractable) — tpm, open-questions #12.
• sudo apt autoremove the leftover old kernel once the new one is proven.

6. Runtime — Docker engine (VERIFIED 2026-06-23)

Install Docker Engine + compose (as admin). NB Ubuntu 26.04 codename is resolute, which download.docker.com may not yet publish — pin the repo line to noble, OR use Ubuntu's docker.io. Add only admin to the docker group (root-equivalent — NEVER the operator).

This gives the appliance the engine. How the stack gets ONTO it is step 7 — and as of 2026-06-27 the primary path is Komodo (remote, no-SSH), not a hand-copied dir. The manual docker compose flow survives as a break-glass fallback (§7c).

7. Deploy the stack — Komodo Periphery (PRIMARY, 2026-06-27)

The booth is driven by a central Komodo Core over the NetBird mesh. The appliance runs a small Periphery agent that dials out to Core; Core then deploys the same compose files. No inbound port on the booth, no SSH for routine ops. Full rationale + threat model: fleet-deployment-komodo. Verified end-to-end on the first booth (park-buzi) 2026-06-27.

7a. Install Periphery (on the booth, as admin)

Prereq: the booth is on the NetBird mesh and can reach Core's reverse-proxy URL (https://komodo.infra.msai.al).

1. In Core: Settings → Onboarding → + New Onboarding Key (Name = the booth, e.g. park-buzi; Expiry ~1 day; Pre-Existing Key empty). Copy the one-time O-… key. Single-use — delete it after the agent connects.
2. On the booth, install Periphery in user mode (runs as admin, who is in docker; NO root daemon; outbound → opens no inbound port):

curl -sSL https://raw.githubusercontent.com/moghtech/komodo/main/scripts/setup-periphery.py | python3 - --user \
  --core-address="https://komodo.infra.msai.al" \
  --connect-as="park-buzi" \
  --onboarding-key="O-…"
sudo loginctl enable-linger admin     # so the user service starts at boot without a login

• --connect-as is the Server name in Core — unique, stable, site-meaningful (the fleet's primary key). Booth #2 = a different name (e.g. park-durres); never reuse one. Get this right in the command itself — it's a plain field in periphery.config.toml on the host, so a typo/placeholder here needs a config edit + agent restart to fix, NOT a rename in Core's UI (which only relabels Core's record, not the agent's real identity — gotcha #12 below).
• --core-address is Core's reverse-proxy URL (the URL you load the Core UI at over the mesh), NOT :9120 — Core's container port 9120 is exposed-not-published; the agent reaches it through the proxy. (Gotcha #7 below.)
• Config lands at ~/.config/komodo/periphery.config.toml. The key field is core_address (singular).

⚠ ALWAYS CHECK THIS — every install so far has hit it (lab box 2026-07-07, booth park-2 2026-09-02). root_directory must be a path admin can write, but Periphery's installer writes root_directory = "/etc/komodo" even with --user (still true as of v2.3.3). Result: panic Failed to write private key pem to "/etc/komodo/keys/periphery.key" … Permission denied, crash-loop until systemd gives up (Start request repeated too quickly).

Fix + restart:

sed -i 's|^root_directory = .*|root_directory = "'"$HOME"'/.komodo"|' ~/.config/komodo/periphery.config.toml
systemctl --user reset-failed periphery && systemctl --user restart periphery

NB sudo systemctl restart periphery says unit not found — it's a USER unit; always systemctl --user …. The onboarding key survives a pre-connect crash (unused until first dial).

➜ Do not stop here once it's green. This fix only gets Periphery running — the Stack still isn't deployed. Immediately continue to verify below, then §7b.

Verify: systemctl --user status periphery → active; the server park-buzi appears and goes OK/green in Core → Servers. Then delete the onboarding key.

➜ Next step is §7b below — the Stack itself is not deployed yet. A green Server in Core just means the agent connected; it runs nothing until you add the Registry/Git accounts and deploy.

7b. Deploy the Stack (in Core — by hand once, then code)

Add Registry Account + Git Account for git.infra.msai.al (user komodo, tokens) in Core so Periphery can clone the repo AND pull the private images. Two distinct credential types — the git clone working does NOT imply the image pull is authed (gotcha #8). Per-booth secrets (park_<booth>_jwt_secret, park_<booth>_event_signing_key — distinct values, openssl rand -hex 32) live in Core's Variables/Secrets store, referenced from the Stack as [[…]].

Create a Stack (UI → Stacks → New), name = the booth (park-buzi):

• Server: park-buzi · Source: repo mca/parking_solution, branch dev, files docker-compose.yml + docker-compose.prod.yml · Registry account: komodo (else the pull is anonymous → no basic auth credentials).
• Environment (Komodo writes this to a .env on the booth at deploy, substituting [[…]]):

REGISTRY=git.infra.msai.al/mca/parking_solution
TAG=dev                                  # moving tag (staging). PIN to dev-<sha> for a live booth.
COOKIE_SECURE=0                          # CRITICAL on plain-http or the auth cookie never sends → no login
VISION_ENABLED=1
WS_ALLOWED_ORIGINS=                      # browser at the booth URL is same-origin; leave empty (the
                                         #   Tauri desktop app needs its origin here — separate task)
JWT_SECRET=[[park_buzi_jwt_secret]]
EVENT_SIGNING_KEY=[[park_buzi_event_signing_key]]

Deploy → Periphery pulls + compose ups. All containers (proxy/Caddy, server, vision) green. Seed the FIRST admin (DB starts empty → nobody can log in until this runs; idempotent) via Komodo's terminal on the server container (no SSH):

docker exec -it -e ADMIN_USER=admin -e ADMIN_PASS='<strong-pw>' \
  park-buzi-server-1 node scripts/seed-admin.mjs

Secrets-on-disk note. The generated .env lands on the booth with cleartext secrets (compose needs real values). That's why the disk is LUKS-encrypted (§3–4) and keys are per-booth — the encryption is the control, and a single-booth compromise leaks only that booth's key. See fleet-deployment-komodo (the EVENT_SIGNING_KEY-in-Core blast-radius caveat; ATECC608 is the intended long-term signer).

7b-bis. Fleet-as-code (resources.toml) — optional but recommended

The repo's komodo/resources.toml mirrors the working Stack. Pointing a Core ResourceSync at it makes the fleet git-managed: booth #N is a copy-pasted [[stack]] block; an image bump is a one-line TAG= edit + push + Execute; every change is an auditable commit; a rebuilt Core re-creates everything from the file. Keep the sync Unmanaged + Delete-Unmatched OFF until trusted. An empty diff / disabled Execute = the file already matches the live Stack (success, not an error). See komodo/README.md and fleet-deployment-komodo.

7c. Break-glass — manual compose (mesh/Core down)

When the mesh or Core is unreachable, the same compose files run locally via scripts/booth.sh (or raw docker compose). Needs a local .env and a docker login git.infra.msai.al (a read-only package token). This is the FALLBACK, not the routine path:

docker login git.infra.msai.al
ENV=prod ./booth.sh config   # dry-run the merged env
ENV=prod ./booth.sh up

booth.sh runs from wherever it sits next to the compose files (the booth deploys them flat, e.g. /opt/parking_systems/). See container-deployment.

7d. Reset the DB for TRAINING/DEMO — docker exec, not pnpm (2026-06-30)

A site is sometimes run live to train operators/admins on the real app; afterwards the demo data must go without leaving an obvious self-serve button (the [[threat-model|operator must not be able to wipe history]]). The reset is a CLI script (packages/db/scripts/reset-db.mjs), and on the booth there is no pnpm — only the running containers. So run it the same way as the seed-admin step in §7b: docker exec into the server container, where the script ships inside the deploy bundle at node_modules/@parking/db/scripts/reset-db.mjs (the same place the boot migrator lives — see the entrypoint). DATABASE_URL in-container is /data/parking.sqlite (the parking-data volume).

# On the booth (or via Komodo's terminal on the server container). Category flags:
#   --financial  ledger (entry/exit/payment/void/shift/cash/anomaly) + device_events + snapshots +
#                subscription INSTANCES/credentials/plates + blocklist. KEEPS users/devices/config/
#                tariffs/subscription PLANS.
#   --config     site_config, devices, setup_state (re-runs first-run setup), tariffs + versions
#                + drafts, plans.
#   --users      users, roles, role_permissions, auth sessions.
#   --diagnostics  app_logs (the /setup/logs store).              --all  every table.
# A drift guard refuses to run if the DB has a table no category covers (2026-07-08).
docker exec -it \
  -e RESET_ALLOWED=1 \
  -e DATABASE_URL=/data/parking.sqlite \
  park-buzi-server-1 \
  node node_modules/@parking/db/scripts/reset-db.mjs --financial

⚠ --financial/--all TRUNCATE the append-only, signed append-only-event-chain — the anti-fraud record. A partial delete would break the hash chain, so a financial reset wipes the whole ledger back to empty (re-seeding starts a NEW chain under the same EVENT_SIGNING_KEY/ BACKUP_KEY — the keys are not touched). This is the opposite of how the ledger is meant to behave, hence the two gates: it refuses unless RESET_ALLOWED=1 is set (a real booth never sets it) and you type the DB filename to confirm (parking.sqlite; --yes skips that for scripted setup only). It is a training/demo tool — never run on a production booth's data.

After --users/--all (users cleared), re-seed the first admin exactly as in §7b (docker exec … node scripts/seed-admin.mjs) so someone can log back in. Since 2026-07-06 the seed script self-heals the built-in admin role row that this reset also wipes — before that fix the documented re-seed died on a role_id FOREIGN KEY error (field failure on park-buzi). For dev (where pnpm exists) the same script is pnpm db:reset --financial — see local-dev-workflow.

7e. Lost APP admin password — reset from the Linux admin account (2026-07-06)

The app's admin password lives only as a bcrypt hash in the booth DB; there is no in-app recovery (nobody above the admin exists to send a reset). The recovery path is the Linux admin account (the only user in the docker group): the seed script doubles as the password-reset tool via FORCE=1 — on an existing username it RESETS that user's password (and restores roleId: admin, so it also rescues a demoted admin).

# Interactive (preferred — the password never lands in shell history):
docker exec -it -e FORCE=1 park-buzi-server-1 node scripts/seed-admin.mjs
#   → prompts: username (Enter = admin), new password (min 8 chars)

# Non-interactive (scripted; NB the password enters the HOST's shell history):
docker exec -e FORCE=1 -e ADMIN_USER=admin -e ADMIN_PASS='new-strong-pass' \
  park-buzi-server-1 node scripts/seed-admin.mjs

• Attributable, not gated. Whoever holds Linux root owns the DB file — the app cannot defend against that actor and doesn't pretend to. What it CAN do: the script appends a signed config_change ledger event (admin.passwordReset / admin.seeded on first seed, operator console:seed-admin) so a console reset stays visible in the chain afterwards. If the signing key is unavailable (e.g. a dev shell), it warns loudly and proceeds — locking an admin out to protect an audit line would invert the priority. The threat-model adversary remains the operator, who has no Linux account at all.
• Sessions are NOT revoked by a password reset — issued JWT cookies ride to expiry. A forgotten password needs nothing more; a suspected-stolen one should also rotate the booth's JWT_SECRET (Komodo Variables → redeploy), which invalidates every session instantly.
• Works on a fresh/reset DB too (the role-row self-heal above), so §7b first-seed, §7d post-reset re-seed, and this recovery are all the same one command.

Healthy startup + web-access

Healthy logs: vision Initialized LicensePlateDetector … with NO "Downloading" (baked weights), server [migrate] done → SPA static serving enabled → Server listening. The transient vision-service -> offline at boot then -> ready (fast_alpr) ~8s later is normal (monitor polls before vision finishes loading). Reach the UI at http://<name-or-ip>/ (Caddy on :80).

Web-access gotchas (all fixed in the images/compose — see container-deployment "Web access"): the SPA uses a RELATIVE /api base (works from any host; do NOT bake a domain) + a Caddy proxy gives the clean port-80 URL; the domain (parksystems.msai.al) is pointed at the booth's LAN IP via hosts/DNS ON-SITE, never an image rebuild. The Tauri desktop app (install the .deb from mca/public_releases, NOT the AppImage — see desktop-shell-tauri) asks for the server address on first launch (127.0.0.1:3000 on the booth itself, or any <ip>:3000 / <name> via Caddy); nothing is baked in since v0.1.5. In-app updates need the admin password (polkit) — by decision, updates are an admin action, so plan to be at the box when bringing it online for one.

Quick-reference: the gotchas, in order they bit us

1. Ventoy USB → 0x1A Security Violation under Secure Boot → flash the ISO directly instead.
2. 7070 BIOS has no "View Key" → can't inspect db; the live-USB boot IS the verification.
3. Installer "hardware-backed encryption" → PCR_UNUSABLE/dbt → use passphrase LUKS + manual seal.
4. Bind TPM to PCR 7 only, not a multi-PCR set (kernel updates churn 4/8/9 → passphrase every boot).
5. Always keep the password slot + an off-machine copy of the passphrase (TPM is never the only key).
6. GRUB password MUST be edit-only (--unrestricted on entries) or it prompts on EVERY boot → breaks unattended reboot. Verify grep -c unrestricted /boot/grub/grub.cfg ≥1 before rebooting.

Firmware / dbx gotchas (2026-06-30, §4a)

12. UEFI dbx ships via fwupd/LVFS, NOT APT. apt list --upgradable clean ≠ no firmware update pending. A new dbx vs a stale GRUB → revoked bootloader → unbootable / GRUB panic (apt full-upgrade first, then dbx). And dbx moves PCR 7 → breaks TPM auto-unlock → passphrase prompt → re-seal (§4 runbook). Mask fwupd + remove the firmware-updater snap so the operator can't trigger it.
13. cryptsetup … --test-passphrase silently passes via the TPM token (false safety signal). Use --disable-external-tokens to actually force a typed-passphrase test before any firmware change.

Komodo deploy gotchas (2026-06-27)

7. Periphery core_address is Core's reverse-proxy URL (https://komodo.infra.msai.al), NOT 100.x:9120. Core's 9120 is exposed-not-published (docker ps shows 9120/tcp with no ->) → a direct dial gets Connection refused. Ping/SSH working over the mesh does NOT mean :9120 is reachable.
8. Git auth ≠ registry auth. The repo cloning fine does not mean image pull is authed — they're separate Komodo credentials. A blank registry account on the Stack → anonymous pull → no basic auth credentials. Set the Stack's Registry Account (komodo).
9. User-mode Periphery + /etc/komodo root_directory = Permission denied writing the agent key. User-mode (runs as admin, no root daemon) must keep root_directory under $HOME. Hit on every install so far (lab box 2026-07-07, booth park-2 2026-09-02, still on v2.3.3) — check this first whenever a fresh Periphery install crash-loops; see the boxed callout in §7a for the fix. Easy to fix-and-move-on without realizing the Stack still isn't deployed — §7a's fix only starts the agent, §7b deploys the Stack.
10. The config key is core_address (singular). And --core-address derives wss:// from https:// — if Core were plain-HTTP you'd need http:// (→ ws://).
11. ResourceSync Execute disabled + file shown clean in Info = empty diff = already in sync (success). Execute only enables when the file and Core diverge (e.g. you edit TAG).
12. Renaming a Server in Core's UI does NOT change the agent's actual identity. connect_as is a plain field persisted in the agent's own ~/.config/komodo/periphery.config.toml — Core's UI rename only relabels Core's record, the agent keeps re-announcing under its original connect_as on every reconnect. Symptom (hit 2026-08-30, lab box): a server named via a leftover template placeholder in the install command kept reappearing in Core no matter how many times it was renamed there, while the intended name sat permanently NOT OK (nothing was ever checking in as that name). Fix: edit connect_as directly in periphery.config.toml on the host, then systemctl --user restart periphery — no reinstall/re-onboarding needed. Delete the stray old-name Server record in Core afterward. Lesson: always double-check --connect-as is a REAL name (never leave a template placeholder like <new-server-name> in a copy-pasted install command) — Core will happily create a server with that literal string.
13. Upgrading an already-installed Periphery is: re-run the same installer, unchanged --connect-as. No separate update mechanism, no update-only flag. The installer script explicitly skips rewriting periphery.config.toml if one already exists ("Config already exists, skipping...") — it only stops the service, replaces the binary, and restarts — so a re-run is config-preserving and a fresh/dummy --onboarding-key value on that re-run is simply unused (confirmed against Komodo's own setup-periphery.py source, 2026-08-30; no Periphery-specific breaking changes between v2.2.0 and v2.3.2 per Komodo's release notes). Verified end-to-end on art-docker-station (lab, dry run) then park-buzi (live booth, 2026-08-30): same command as §7a step 2, same --connect-as, app containers untouched throughout (Periphery restarting itself never touches the already-running compose stack). Always dry-run a version bump on a lab/dev box before a live booth, even with a clean release-notes check — this project only had one lab box to test against and used it first.