parking_solution/wiki/decisions/appliance-provisioning.md

type, tags, sources, updated, status

type	tags	sources	updated	status
reference	parking deployment appliance hardening runbook offline-first		2026-06-23	settled

Appliance provisioning runbook (booth PC)

Step-by-step to take a booth PC from factory Windows to a hardened, encrypted, container-running parking appliance. Written from the first real provisioning, 2026-06-23, on the actual hardware below — every command here was run and verified on that machine, including the firmware-specific workaround. Companion to disk-os-hardening (the why), tpm (TPM analysis), and container-deployment (the images this runs).

⚠ This box is the threat-model defence. The load-bearing anti-fraud control is still reconciliation over the append-only-event-chain — disk encryption + Secure Boot raise the cost of offline tamper, they don't replace reconciliation.

Reference hardware (first unit, 2026-06-23)

• Dell OptiPlex 7070, Intel Core i5-8500 (Coffee Lake), 238 GB SATA SSD (/dev/sda).
• TPM 2.0 — discrete Nuvoton (Get-Tpm → ManufacturerIdTxt NTC, fw 7.2.1.0). NOT Intel PTT/fTPM. Discrete ⇒ an external LPC/SPI bus exists (bus-sniff is a theoretical physical attack on PCR-only sealing — accepted; see tpm). Used PC — previous owner irrelevant.
• Shipped Windows 11; formatted to Ubuntu 26.04 LTS (the decided platform — desktop-shell-tauri).

1. BIOS (F2 at the Dell logo)

• TPM: leave On. Used PC → Clear the TPM once (Security → TPM → Clear) so the prior owner's keys are wiped before LUKS enrollment. (PPI "Bypass for Clear" was unchecked → it asks for physical confirmation at next boot; that's normal.)
• Secure Boot: Enabled, Deployed Mode (not Audit). "Enable Custom Mode" UNCHECKED = Standard Mode with stock Microsoft keys — this is what Ubuntu's signed shim needs. Do NOT touch PK/KEK/db/dbx. NB: the 7070's Expert Key Management is edit-only (Save/Replace/Append/Delete — no read-only "View Key"), so you cannot inspect db from BIOS; verify via the live USB instead (step 2).
• Boot: UEFI only (no CSM/Legacy — a Legacy install has no Secure Boot / TPM-seal path).
• Set a BIOS admin password.

2. Boot the Ubuntu 26.04 USB (Secure Boot ON)

• Flash the ISO DIRECTLY (Rufus GPT/UEFI, Etcher, or dd). NOT Ventoy — Ventoy's own bootloader isn't in db, so Secure Boot rejects it with Verification failed: (0x1A) Security Violation (this is Secure Boot working correctly, not a fault). A directly-flashed Ubuntu USB boots the Microsoft-signed shim, which stock db trusts.
• F12 at the Dell logo → pick the USB under UEFI BOOT.
• Reaching the installer with Secure Boot ON = positive proof the MS third-party UEFI CA is in db (the verification the BIOS couldn't show us).

3. Encrypted install — the firmware workaround (IMPORTANT)

The 26.04 installer disk page offers: No Encryption / Encrypt with a passphrase / Use hardware-backed encryption (+ advanced LVM/ZFS, both ZFS experimental).

• "Use hardware-backed encryption" FAILS on this 7070 with: PCR_UNUSABLE … error with secure boot policy (PCR7) measurements: generating secure boot profiles for systems with timestamp revocation (dbt) support is currently not supported. → Ubuntu's automated FDE profiler can't model PCR7 on Dell firmware carrying a dbt (UEFI timestamp revocation list). It is NOT a TPM or Secure-Boot fault — both are fine.
• So: choose "Encrypt with a passphrase". Set a strong passphrase and SAVE IT OFF-MACHINE (phone / password manager). It is both the boot unlock (until TPM sealing) AND the permanent recovery slot. Finish the install.
• Result (verify with lsblk): sda1 vfat /boot/efi, sda2 ext4 /boot, sda3 crypto_LUKS → dm_crypt-0 (LVM2) → ubuntu--vg-ubuntu--lv ext4 /.

4. Seal LUKS to the TPM (manual — PCR 7 only)

Do this AFTER first boot. Manual enrollment sidesteps the installer's dbt profiler and lets us pick PCRs. Bind to PCR 7 only (Secure Boot state): it catches the attack that matters (disabling Secure Boot to boot a tampered kernel) WITHOUT breaking on routine kernel/GRUB updates (which churn PCRs 4/8/9 → would otherwise drop every boot to the passphrase). Firmware-only PCR 0 is the fallback if PCR 7 ever errors.

sudo apt update && sudo apt install -y tpm2-tools
sudo tpm2_pcrread sha256                 # sanity: PCRs 0-10 populated, PCR 7 has a real value

# Enroll the TPM (prompts for the EXISTING install passphrase to authorize the new slot):
sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 /dev/sda3

# Verify TWO slots — keep BOTH (slot 0 password = recovery, slot 1 tpm2 = auto-unlock):
sudo systemd-cryptenroll /dev/sda3
#   SLOT TYPE
#      0 password
#      1 tpm2

Wire it into boot (back up first; the mapping is dm_crypt-0, the LUKS UUID is in /etc/crypttab):

sudo cp /etc/crypttab /etc/crypttab.bak
sudo sed -i 's/none luks$/none luks,tpm2-device=auto/' /etc/crypttab
cat /etc/crypttab     # → dm_crypt-0 UUID=… none luks,tpm2-device=auto
sudo update-initramfs -u
sudo reboot

• Boots straight to login, no passphrase prompt = ✅ TPM auto-unlock works (unattended reboot achieved — VERIFIED on this unit 2026-06-23).
• Still prompts = PCR mismatch; type the passphrase (NOT locked out), then retry with --tpm2-pcrs=0. The password slot + crypttab.bak make this fully reversible.

Re-seal runbook: a BIOS update / Secure Boot change alters PCR 7 → the TPM refuses → boot falls back to the passphrase prompt (not a brick). After such a change, re-run step 4's systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 /dev/sda3 to re-bind.

5. GRUB password — EDIT-ONLY (VERIFIED 2026-06-23)

Closes the init=/bin/bash / systemd.unit=rescue.target local-root hole: without it, anyone at the keyboard presses e at the GRUB menu, edits the kernel cmdline, and boots to a root shell with no login. The PCR-7 TPM seal does NOT cover this — editing the GRUB cmdline doesn't change PCR 7 (Secure Boot policy), so the TPM still releases the key and the attacker lands on the decrypted disk. This is the specific countermeasure for the threat-model. Use edit-only mode (--unrestricted) so the box still boots UNATTENDED — the password is required only to EDIT entries, never to boot.

grub-mkpasswd-pbkdf2          # enter a password (twice) → copy the grub.pbkdf2.sha512.* hash

Add the superuser (paste YOUR hash) to the end of /etc/grub.d/40_custom:

set superusers="admin"
password_pbkdf2 admin grub.pbkdf2.sha512.10000.<YOUR_HASH>

Make menu entries bootable WITHOUT the password (edit-only) — in /etc/grub.d/10_linux, set the active CLASS= line to include --unrestricted:

CLASS="--class gnu-linux --class gnu --class os --unrestricted"

Regenerate + VERIFY BOTH HALVES landed in the real config BEFORE rebooting (a GRUB misconfig means a rescue-USB recovery):

sudo update-grub
sudo grep -c "password_pbkdf2" /boot/grub/grub.cfg   # want ≥1 (password present)
sudo grep -c "unrestricted"    /boot/grub/grub.cfg   # want ≥1 (entries bootable w/o password)
sudo reboot

✅ VERIFIED on this unit: boots straight to login (no GRUB prompt, TPM still auto-unlocks) AND pressing e at the menu prompts for admin + password. Store the GRUB password off-machine (alongside the LUKS passphrase).

OS hardening on the first unit is now COMPLETE: LUKS FDE + TPM auto-unlock (PCR 7) + Secure Boot (Deployed) + GRUB edit-lock.

5c. OS user model — admin vs operator (VERIFIED 2026-06-23)

The OS has TWO roles and they must be different identities (threat-model: the operator is the adversary). Create a dedicated admin (real password, sudo, NO auto-login) and keep the operator as an auto-login, UNPRIVILEGED account.

sudo adduser admin && sudo usermod -aG sudo admin
# VERIFY in a second session: log in as admin → `sudo whoami` prints root — BEFORE the next step:
sudo deluser <operator> sudo            # demote the auto-login operator
groups <operator>                        # confirm: no 'sudo'

⚠ Order matters: confirm the new admin's sudo works before demoting the operator, or you lock yourself out. Keep auto-login on the OPERATOR, not admin. Leave root password disabled (Ubuntu default) — admin+sudo IS the root path; enabling root adds risk, no gain.

Strip latent escalation groups from the operator: sudo deluser <operator> lxd (lxd group = launch a privileged container that mounts host / as root — undoes the no-sudo hardening) and lpadmin (printer admin, unneeded). And NEVER add the operator to docker (also root-equivalent).

5b. Further hardening (TODO — not yet done)

• Key-based SSH only (disable password auth) if SSH is enabled at all.
• No/locked-down desktop + kiosk autostart — single-purpose; the operator never reaches a shell (desktop-shell-tauri).
• Consider moving the host event-signing key into the TPM (non-extractable) — tpm, open-questions #12.
• sudo apt autoremove the leftover old kernel once the new one is proven.

6. Runtime — Docker stack (VERIFIED 2026-06-23)

Install Docker Engine + compose (as admin). NB Ubuntu 26.04 codename is resolute, which download.docker.com may not yet publish — pin the repo line to noble, OR use Ubuntu's docker.io. Add only admin to the docker group (root-equivalent — NEVER the operator).

Deploy from a standalone dir (hand-copied; no repo on the appliance), e.g. /opt/parking_solution: docker-compose.yml + docker-compose.prod.yml (the Caddy/prod override) + Caddyfile + a .env (chmod 600). The .env (driven into the containers by the base compose):

JWT_SECRET=<openssl rand -hex 32>          # server REFUSES to boot without (>=32, no insecure default)
EVENT_SIGNING_KEY=<a DIFFERENT openssl rand -hex 32>
COOKIE_SECURE=0                            # CRITICAL on plain-http or the auth cookie never sends → no login
WS_ALLOWED_ORIGINS=http://<name-or-ip>     # any REMOTE origin admins use (same-origin always passes)
VISION_ENABLED=1
# REGISTRY/TAG default to git.infra.msai.al/mca/parking_solution + dev; set TAG=main to pin.

docker login git.infra.msai.al            # a read-only package token, not the account password
docker compose -f docker-compose.yml -f docker-compose.prod.yml config   # dry-run: verify the merged env
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
# Seed the FIRST admin (DB starts empty → nobody can log in until this runs; idempotent):
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec \
  -e ADMIN_USER=admin -e ADMIN_PASS='<strong-pw>' server node scripts/seed-admin.mjs

Healthy startup logs: vision Initialized LicensePlateDetector … with NO "Downloading" (baked weights), server [migrate] done → SPA static serving enabled → Server listening. The transient vision-service -> offline at boot then -> ready (fast_alpr) ~8s later is normal (monitor polls before vision finishes loading). Reach the UI at http://<name-or-ip>/ (Caddy on :80).

Web-access gotchas (all fixed in the images/compose — see container-deployment "Web access"): the SPA uses a RELATIVE /api base (works from any host; do NOT bake a domain) + a Caddy proxy gives the clean port-80 URL; the domain (parksystems.msai.al) is pointed at the booth's LAN IP via hosts/DNS ON-SITE, never an image rebuild.

Quick-reference: the gotchas, in order they bit us

1. Ventoy USB → 0x1A Security Violation under Secure Boot → flash the ISO directly instead.
2. 7070 BIOS has no "View Key" → can't inspect db; the live-USB boot IS the verification.
3. Installer "hardware-backed encryption" → PCR_UNUSABLE/dbt → use passphrase LUKS + manual seal.
4. Bind TPM to PCR 7 only, not a multi-PCR set (kernel updates churn 4/8/9 → passphrase every boot).
5. Always keep the password slot + an off-machine copy of the passphrase (TPM is never the only key).
6. GRUB password MUST be edit-only (--unrestricted on entries) or it prompts on EVERY boot → breaks unattended reboot. Verify grep -c unrestricted /boot/grub/grub.cfg ≥1 before rebooting.