release: update go.mod for v1.4.0

* feat(vm): direct-boot foundations + config/input extensions + image build pipeline

vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go

config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput

image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts

* feat(daemon): direct-boot launch path for microvm system instances

When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration

PC-machine path (DirectBoot=false) unchanged.

Closes mulga-siv-65

* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service

When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer

When ELBv2Enabled=false: existing PC-machine path unchanged.

Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.

Closes mulga-siv-66

* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold

* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag

- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note

Closes mulga-siv-67

* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed

* fix(microvm): end-to-end fixes for direct-boot LB path

Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).

virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.

sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).

BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.

wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.

Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.

* fix(build): auto-detect docker/podman when apk unavailable for microvm image build

* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs

tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.

Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.

* fix(microvm): fix mgmt NIC routing for all LB schemes

Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:

1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
   buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
   'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
   'ip route add 192.168.1.33 via 10.15.8.1'.

2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
   after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
   before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
   blob and skipping interface setup in init.sh entirely.

Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.

* feat(microvm): Phase 2 kernel stripping + boot optimisation

- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
  field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
  HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
  exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
  during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
  with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches

* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip

The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.

Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.

* fix(build): keep failover modules in initramfs strip

virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.

Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.

* fix(daemon): restore slog default after viperblock.New mutates it

viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.

Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.

Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.

Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).

* perf(microvm): hit Phase 2 boot/memory gates

t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.

Root causes:

1. build-microvm-image.sh staged the entire chroot into the cpio
   without dropping /boot, embedding a duplicate vmlinuz, Alpine's
   stock initramfs-virt, System.map, and a "boot → ." symlink loop
   (~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
   the kernel is copied out, before the cpio. Initramfs drops from
   31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
   42 MiB → 22 MiB.

2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
   launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
   1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
   no firmware, ROM, or block layer). Both extracted to
   directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
   qemuStartSettleWait to 50 ms.

3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
   Added boot-time perf flags: quiet loglevel=3 (silence printk over
   115200-baud isa-serial), mitigations=off (KVM guest on trusted
   host, single-tenant LB workload), tsc=reliable, no_timer_check,
   reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).

4. init.sh dropped modprobe virtio_mmio (built-in:
   CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
   from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).

Phase 2 gates:
  - t1→t2 ≤ 300 ms       ✅ 264 ms / 281 ms
  - artifact ≤ 50 MiB    ✅ 22 MiB (gate already in build script)
  - ELBv2 e2e green      ✅ 32/33 (pre-existing client-VM failure,
                                   unrelated to microvm path)

* build(microvm): wire build into deploy + distro + release pipeline

Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).

Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.

Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.

* build(ansible): include microvm artifacts in build role tarball

Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).

* feat(config): enable microvm direct-boot LB path by default

Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.

Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).

* fix(microvm): bootstrap apk.static when no host rootfs tool exists

Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.

When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.

No-op when apk, docker, or podman is already present.

Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244

* revert: drop apk.static bootstrap in build-microvm-image

Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.

* docs(workbook): drop lb-alpine import for microvm direct-boot LB

LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.

* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN

When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").

Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.

* fix(e2e): redirect lb probe chatter to stderr so stalls are visible

test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.

Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.

* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node

buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.

resolveMgmtRoute already encodes the correct policy:
  - multi-node any scheme        → explicit MgmtRoute{Gateway,Target}
  - single-node internal         → fallback to (MgmtBridgeIP, AdvertiseIP)
  - single-node internet-facing  → empty (reach AWSGW via VPC SNAT)

Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.

Test covers all three branches.

* feat(ansible): dev-deploy playbook for in-place artifact swap

Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.

Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.

Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
  artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.

* chore(microvm): remove legacy LB Alpine image build pipeline

Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.

build-system-image stays — other manifests (e.g. GPU images) still
use it.

* refactor(microvm): remove legacy PC-machine LB boot path

Phase 3 — direct-boot becomes the only ELBv2 LB launch path.

- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
  caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
  getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
  on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
  buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
  isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
  [daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
  paths) and refresh stale comments that still mentioned lbVMUserData.

ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.

* docs: drop lb-alpine catalog entry from cluster setup guide

LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.

* feat(ui): remove LB image import banner + LB-AMI gating

Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:

- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
  system-managed.ts (isSystemManagedImage stays — still used by the
  EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
  wizard pages
- The corresponding tests that asserted the banner appears

* revert: restore lb.conf + lb-setup.sh for mulga install pipeline

Phase 3 microvm removal (commit 6313d74) deleted these too early — the
mulga install/iso artifact builders (scripts/tofu-cluster/build-install
-artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke
scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with
"ERROR: Manifest not found: scripts/images/lb.conf".

Restore both files verbatim so the legacy pipeline keeps building until
the mulga-side migration to microvm-only artifacts lands post-merge.

.github/assets/architecture.svg

@@ -0,0 +1,225 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#161b22"/>
+      <stop offset="45%" style="stop-color:#1f2028"/>
+      <stop offset="100%" style="stop-color:#161b22"/>
+    </linearGradient>
+    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262633"/>
+      <stop offset="100%" style="stop-color:#7F1D1D14"/>
+    </linearGradient>
+    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#F87171"/>
+      <stop offset="50%" style="stop-color:#DC2626"/>
+      <stop offset="100%" style="stop-color:#7F1D1D"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="620" fill="url(#bg)" rx="12"/>
+
+  <!-- Title bar -->
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
+  <text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
+
+  <!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
+  <rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
+  <text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
+  <text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
+
+  <g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
+
+    <rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
+
+    <rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
+
+    <rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
+
+    <rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
+
+    <rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
+
+    <rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
+  </g>
+
+  <!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
+  <line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
+  <text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
+
+  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
+  <rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
+  <text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
+  <text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
+
+  <g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
+    <rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
+
+    <rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
+
+    <rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
+
+    <rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
+
+    <rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
+
+    <rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
+  </g>
+
+  <!-- ═══ NATS BUS ═══════════════════════════════════════ -->
+  <line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
+  <text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
+  <text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.*   ebs.*   s3.*   vpc.*   iam.*</text>
+  <text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
+
+  <!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
+  <g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
+    <line x1="100" y1="306" x2="100" y2="328"/>
+    <line x1="275" y1="306" x2="275" y2="328"/>
+    <line x1="450" y1="306" x2="450" y2="328"/>
+    <line x1="625" y1="306" x2="625" y2="328"/>
+    <line x1="800" y1="306" x2="800" y2="328"/>
+  </g>
+
+  <!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
+  <!-- EC2 -->
+  <g transform="translate(28, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
+  </g>
+
+  <!-- EBS -->
+  <g transform="translate(204, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
+  </g>
+
+  <!-- S3 -->
+  <g transform="translate(380, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
+  </g>
+
+  <!-- VPC -->
+  <g transform="translate(556, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
+  </g>
+
+  <!-- IAM -->
+  <g transform="translate(732, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
+  </g>
+
+  <!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
+  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
+    <line x1="100" y1="410" x2="100" y2="440"/>
+    <line x1="276" y1="410" x2="276" y2="440"/>
+    <line x1="452" y1="410" x2="452" y2="440"/>
+    <line x1="628" y1="410" x2="628" y2="440"/>
+    <line x1="804" y1="410" x2="804" y2="440"/>
+  </g>
+
+  <!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
+  <rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
+  <text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
+
+  <!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
+  <!-- QEMU/KVM -->
+  <g transform="translate(28, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
+  </g>
+
+  <!-- Viperblock -->
+  <g transform="translate(204, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
+  </g>
+
+  <!-- Predastore -->
+  <g transform="translate(380, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
+  </g>
+
+  <!-- OVN/OVS -->
+  <g transform="translate(556, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
+  </g>
+
+  <!-- IAM store -->
+  <g transform="translate(732, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
+  </g>
+
+  <!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
+  <g transform="translate(16, 558)">
+    <rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
+    <text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
+      <tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
+      <tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
+    </text>
+    <text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
+      Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
+      No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
+    </text>
+  </g>
+</svg>

.github/assets/banner.svg

@@ -0,0 +1,154 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="300" viewBox="0 0 900 300">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a;stop-opacity:1" />
+      <stop offset="50%" style="stop-color:#0d1117;stop-opacity:1" />
+      <stop offset="100%" style="stop-color:#0a0a0a;stop-opacity:1" />
+    </linearGradient>
+    <radialGradient id="vignette" cx="50%" cy="50%" r="70%">
+      <stop offset="0%" style="stop-color:#7F1D1D;stop-opacity:0.08"/>
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0"/>
+    </radialGradient>
+    <linearGradient id="mulga-grad" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.95" />
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.95" />
+    </linearGradient>
+    <linearGradient id="rule" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0" />
+      <stop offset="50%" style="stop-color:#F87171;stop-opacity:0.45" />
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0" />
+    </linearGradient>
+    <filter id="node-glow">
+      <feGaussianBlur in="SourceGraphic" stdDeviation="3" />
+    </filter>
+  </defs>
+
+  <!-- Background -->
+  <rect width="900" height="300" fill="url(#bg)" rx="12"/>
+  <rect width="900" height="300" fill="url(#vignette)" rx="12"/>
+
+  <!-- Mesh network background (mulgadc style) -->
+  <g opacity="0.35">
+    <!-- Connections (drawn first, under nodes) -->
+    <g stroke="#7F1D1D" stroke-width="0.6" fill="none">
+      <line x1="80" y1="60" x2="200" y2="40"/>
+      <line x1="80" y1="60" x2="150" y2="140"/>
+      <line x1="200" y1="40" x2="330" y2="90"/>
+      <line x1="150" y1="140" x2="260" y2="180"/>
+      <line x1="150" y1="140" x2="60" y2="220"/>
+      <line x1="330" y1="90" x2="260" y2="180"/>
+      <line x1="260" y1="180" x2="120" y2="260"/>
+      <line x1="60" y1="220" x2="120" y2="260"/>
+
+      <line x1="620" y1="50" x2="760" y2="90"/>
+      <line x1="620" y1="50" x2="700" y2="160"/>
+      <line x1="760" y1="90" x2="850" y2="160"/>
+      <line x1="700" y1="160" x2="820" y2="220"/>
+      <line x1="850" y1="160" x2="820" y2="220"/>
+      <line x1="700" y1="160" x2="600" y2="240"/>
+      <line x1="600" y1="240" x2="820" y2="220"/>
+      <line x1="760" y1="90" x2="700" y2="160"/>
+    </g>
+    <!-- Nodes -->
+    <g fill="#DC2626">
+      <circle cx="80" cy="60" r="2.5"/>
+      <circle cx="200" cy="40" r="2"/>
+      <circle cx="330" cy="90" r="2.5"/>
+      <circle cx="150" cy="140" r="3"/>
+      <circle cx="260" cy="180" r="2"/>
+      <circle cx="60" cy="220" r="2"/>
+      <circle cx="120" cy="260" r="2.5"/>
+
+      <circle cx="620" cy="50" r="2"/>
+      <circle cx="760" cy="90" r="3"/>
+      <circle cx="700" cy="160" r="2.5"/>
+      <circle cx="850" cy="160" r="2"/>
+      <circle cx="820" cy="220" r="2.5"/>
+      <circle cx="600" cy="240" r="2"/>
+    </g>
+    <!-- Pulse rings (static "about-to-pulse" look) -->
+    <g fill="none" stroke="#F87171" stroke-width="0.8" opacity="0.5">
+      <circle cx="150" cy="140" r="6"/>
+      <circle cx="760" cy="90" r="6"/>
+      <circle cx="700" cy="160" r="8"/>
+    </g>
+  </g>
+
+  <!-- Main emblem: cluster-node stack with NATS bus -->
+  <g transform="translate(200, 150)">
+    <!-- Outer glow ring -->
+    <circle cx="0" cy="0" r="64" fill="none" stroke="url(#mulga-grad)" stroke-width="1.5" opacity="0.35" filter="url(#node-glow)"/>
+
+    <!-- Top node -->
+    <rect x="-24" y="-52" width="48" height="22" rx="3" fill="none" stroke="#DC2626" stroke-width="1.8"/>
+    <circle cx="-16" cy="-41" r="1.6" fill="#F87171"/>
+    <circle cx="-8" cy="-41" r="1.6" fill="#F87171"/>
+    <line x1="0" y1="-41" x2="16" y2="-41" stroke="#F87171" stroke-width="0.9" opacity="0.6"/>
+
+    <!-- Middle node (active) -->
+    <rect x="-30" y="-14" width="60" height="28" rx="3" fill="url(#mulga-grad)" opacity="0.22" stroke="#DC2626" stroke-width="2"/>
+    <circle cx="-20" cy="0" r="2" fill="#F87171"/>
+    <circle cx="-10" cy="0" r="2" fill="#F87171"/>
+    <line x1="-2" y1="0" x2="22" y2="0" stroke="#F87171" stroke-width="1.2" opacity="0.8"/>
+
+    <!-- Bottom node -->
+    <rect x="-24" y="30" width="48" height="22" rx="3" fill="none" stroke="#7F1D1D" stroke-width="1.8"/>
+    <circle cx="-16" cy="41" r="1.6" fill="#DC2626"/>
+    <circle cx="-8" cy="41" r="1.6" fill="#DC2626"/>
+    <line x1="0" y1="41" x2="16" y2="41" stroke="#DC2626" stroke-width="0.9" opacity="0.6"/>
+
+    <!-- Radiating mesh ports -->
+    <g stroke-width="0.9" opacity="0.55">
+      <line x1="-30" y1="0" x2="-58" y2="-26" stroke="#DC2626"/>
+      <line x1="-30" y1="0" x2="-64" y2="0" stroke="#DC2626"/>
+      <line x1="-30" y1="0" x2="-58" y2="26" stroke="#DC2626"/>
+      <line x1="30" y1="0" x2="58" y2="-26" stroke="#7F1D1D"/>
+      <line x1="30" y1="0" x2="64" y2="0" stroke="#7F1D1D"/>
+      <line x1="30" y1="0" x2="58" y2="26" stroke="#7F1D1D"/>
+    </g>
+    <g fill="#F87171">
+      <circle cx="-58" cy="-26" r="2.5"/>
+      <circle cx="-64" cy="0" r="2.5"/>
+      <circle cx="-58" cy="26" r="2.5"/>
+    </g>
+    <g fill="#DC2626">
+      <circle cx="58" cy="-26" r="2.5"/>
+      <circle cx="64" cy="0" r="2.5"/>
+      <circle cx="58" cy="26" r="2.5"/>
+    </g>
+  </g>
+
+  <!-- Title -->
+  <text x="340" y="120" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="56" font-weight="700" fill="#f5f5f5" letter-spacing="-1.5">Spinifex</text>
+
+  <!-- Subtitle -->
+  <text x="340" y="155" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">Open-source AWS-compatible platform</text>
+  <text x="340" y="180" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">for secure edge deployments</text>
+
+  <!-- Rule -->
+  <rect x="340" y="200" width="440" height="1" fill="url(#rule)" rx="1"/>
+
+  <!-- Service pills -->
+  <g font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" font-weight="500">
+    <rect x="340" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="365" y="231" fill="#F87171" text-anchor="middle">EC2</text>
+
+    <rect x="398" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="423" y="231" fill="#F87171" text-anchor="middle">EBS</text>
+
+    <rect x="456" y="215" width="45" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="478" y="231" fill="#F87171" text-anchor="middle">S3</text>
+
+    <rect x="509" y="215" width="55" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="536" y="231" fill="#F87171" text-anchor="middle">VPC</text>
+
+    <rect x="572" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="597" y="231" fill="#F87171" text-anchor="middle">IAM</text>
+
+    <rect x="630" y="215" width="78" height="24" rx="12" fill="#18181b" stroke="#52525b" stroke-width="0.5" stroke-opacity="0.8"/>
+    <text x="669" y="231" fill="#a1a1aa" text-anchor="middle">Air-Gapped</text>
+  </g>
+
+  <!-- Tagline -->
+  <text x="340" y="265" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" fill="#71717a" letter-spacing="1.8">YOUR INFRASTRUCTURE · YOUR HARDWARE · YOUR CLOUD</text>
+</svg>

.github/assets/design-architecture.svg

@@ -0,0 +1,175 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="720" viewBox="0 0 900 720">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#161b22"/>
+      <stop offset="45%" style="stop-color:#1f2028"/>
+      <stop offset="100%" style="stop-color:#161b22"/>
+    </linearGradient>
+    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262633"/>
+      <stop offset="100%" style="stop-color:#7F1D1D14"/>
+    </linearGradient>
+    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#F87171"/>
+      <stop offset="50%" style="stop-color:#DC2626"/>
+      <stop offset="100%" style="stop-color:#7F1D1D"/>
+    </linearGradient>
+    <linearGradient id="daemon-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262622"/>
+      <stop offset="100%" style="stop-color:#7F1D1D0a"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="720" fill="url(#bg)" rx="12"/>
+
+  <!-- Title -->
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">HIGH-LEVEL ARCHITECTURE</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">SDK → Gateway → NATS → Daemons → VMs</text>
+
+  <!-- ═══ AWS SDK / CLI ══════════════════════════════════ -->
+  <rect x="16" y="50" width="868" height="76" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
+  <text x="32" y="72" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="1">AWS SDK / CLI</text>
+  <text x="32" y="90" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Users interact with Spinifex using standard AWS SDKs and CLIs against a custom endpoint URL</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
+    <rect x="32" y="98" width="480" height="20" rx="4" fill="#0a0a0a" stroke="#3fb950" stroke-width="0.6" stroke-opacity="0.4"/>
+    <text x="42" y="112" fill="#7ee787">$ aws ec2 run-instances --endpoint-url https://localhost:9999 ...</text>
+  </g>
+
+  <!-- Connector: HTTPS + SigV4 -->
+  <line x1="450" y1="126" x2="450" y2="154" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="380" y="128" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
+  <text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">HTTPS · SigV4 Auth</text>
+
+  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
+  <rect x="16" y="154" width="868" height="104" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
+  <text x="32" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY · port 9999</text>
+  <text x="876" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a" text-anchor="end">spinifex/services/awsgw/awsgw.go</text>
+
+  <!-- Two-column feature bullets -->
+  <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+    <text x="40" y="200">● TLS termination</text>
+    <text x="40" y="220">● EC2 Query Protocol parsing</text>
+    <text x="40" y="240">● XML response generation</text>
+
+    <text x="460" y="200">● AWS SigV4 authentication</text>
+    <text x="460" y="220">● Routes to service handlers</text>
+    <text x="460" y="240">● NATS client connection</text>
+  </g>
+
+  <!-- Connector: JSON over NATS -->
+  <line x1="450" y1="258" x2="450" y2="286" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="380" y="260" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
+  <text x="450" y="274" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">JSON over NATS</text>
+
+  <!-- ═══ NATS BROKER ═══════════════════════════════════ -->
+  <rect x="16" y="286" width="868" height="124" rx="8" fill="#1a0a0a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+  <rect x="16" y="286" width="868" height="28" rx="8" fill="url(#nats-grad)" opacity="0.9"/>
+  <text x="32" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS MESSAGE BROKER</text>
+  <text x="876" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">request/reply · queue groups · fan-out</text>
+
+  <!-- Two-column topics/queue groups -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="40" y="332" fill="#F87171" font-weight="600">Topics</text>
+    <text x="40" y="350" fill="#f5f5f5">ec2.RunInstances</text>
+    <text x="40" y="368" fill="#f5f5f5">ec2.DescribeInstances</text>
+    <text x="40" y="386" fill="#f5f5f5">ec2.cmd.{instanceID}</text>
+    <text x="40" y="404" fill="#f5f5f5">ebs.mount  ·  ebs.unmount</text>
+
+    <text x="460" y="332" fill="#F87171" font-weight="600">Delivery mode</text>
+    <text x="460" y="350" fill="#a1a1aa">spinifex-workers <tspan fill="#71717a">(load-balanced)</tspan></text>
+    <text x="460" y="368" fill="#a1a1aa">no queue group <tspan fill="#71717a">(fan-out to all nodes)</tspan></text>
+    <text x="460" y="386" fill="#a1a1aa">per-instance <tspan fill="#71717a">(start / stop / terminate)</tspan></text>
+    <text x="460" y="404" fill="#a1a1aa">Viperblock request/reply</text>
+  </g>
+
+  <!-- Fan-out connectors to 3 daemons -->
+  <g stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6" fill="none">
+    <line x1="450" y1="410" x2="450" y2="430"/>
+    <line x1="450" y1="430" x2="155" y2="452"/>
+    <line x1="450" y1="430" x2="450" y2="452"/>
+    <line x1="450" y1="430" x2="745" y2="452"/>
+  </g>
+
+  <!-- ═══ DAEMON ROW ════════════════════════════════════ -->
+  <!-- Daemon 1 -->
+  <g transform="translate(32, 452)">
+    <rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
+    <text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 1</text>
+    <line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+      <text x="20" y="52">● NATS subscriber</text>
+      <text x="20" y="70">● Resource manager</text>
+      <text x="20" y="88">● QEMU/KVM launcher</text>
+      <text x="20" y="106">● QMP monitoring</text>
+    </g>
+    <text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
+  </g>
+
+  <!-- Daemon 2 -->
+  <g transform="translate(327, 452)">
+    <rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
+    <text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 2</text>
+    <line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+      <text x="20" y="52">● NATS subscriber</text>
+      <text x="20" y="70">● Resource manager</text>
+      <text x="20" y="88">● QEMU/KVM launcher</text>
+      <text x="20" y="106">● QMP monitoring</text>
+    </g>
+    <text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
+  </g>
+
+  <!-- Daemon N -->
+  <g transform="translate(622, 452)">
+    <rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
+    <text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node N</text>
+    <line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+      <text x="20" y="52">● NATS subscriber</text>
+      <text x="20" y="70">● Resource manager</text>
+      <text x="20" y="88">● QEMU/KVM launcher</text>
+      <text x="20" y="106">● QMP monitoring</text>
+    </g>
+    <text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
+  </g>
+
+  <!-- Daemon → VM connectors -->
+  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
+    <line x1="155" y1="576" x2="155" y2="610"/>
+    <line x1="450" y1="576" x2="450" y2="610"/>
+    <line x1="745" y1="576" x2="745" y2="610"/>
+  </g>
+
+  <!-- ═══ VM BAR + ROW ═══════════════════════════════════ -->
+  <rect x="16" y="610" width="868" height="22" rx="5" fill="url(#bar-grad)"/>
+  <text x="450" y="625" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">WORKLOADS · QEMU/KVM virtual machines per node</text>
+
+  <!-- VM boxes -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <rect x="32" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="155" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
+    <text x="155" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
+    <text x="155" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 1 workloads</text>
+
+    <rect x="327" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="450" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
+    <text x="450" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
+    <text x="450" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 2 workloads</text>
+
+    <rect x="622" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="745" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
+    <text x="745" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
+    <text x="745" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node N workloads</text>
+  </g>
+</svg>

.github/assets/diagrams/tf-bastion.svg

@@ -0,0 +1,46 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="180" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">BASTION · only entry point into a private subnet</text>
+
+  <!-- WAN -->
+  <g transform="translate(40, 56)">
+    <rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
+    <text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">operator / laptop</text>
+  </g>
+
+  <!-- Arrow: SSH -->
+  <line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
+  <rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">SSH</text>
+
+  <!-- Bastion -->
+  <g transform="translate(338, 56)">
+    <rect width="220" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.3" stroke-opacity="0.6"/>
+    <text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Bastion</text>
+    <text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">public subnet</text>
+  </g>
+
+  <!-- Arrow: SSH (private IP) -->
+  <line x1="558" y1="86" x2="638" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="634,82 646,86 634,90" fill="#a855f7" opacity="0.8"/>
+  <rect x="552" y="62" width="140" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="622" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">SSH (private IP)</text>
+
+  <!-- App server -->
+  <g transform="translate(656, 56)">
+    <rect width="220" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
+    <text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">App Server</text>
+    <text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">private subnet · no internet</text>
+  </g>
+
+  <text x="450" y="156" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">App server has no route to the internet — all access hops through the bastion.</text>
+</svg>

.github/assets/diagrams/tf-nginx-alb.svg

@@ -0,0 +1,98 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="480" viewBox="0 0 900 480">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="inet-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb9501f"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f71f"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="480" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · multi-AZ</text>
+
+  <!-- Internet -->
+  <rect x="350" y="54" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
+  <text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
+
+  <!-- connector to IGW -->
+  <line x1="450" y1="94" x2="450" y2="116" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="446,112 454,112 450,120" fill="#71717a" opacity="0.7"/>
+
+  <!-- IGW -->
+  <rect x="380" y="122" width="140" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+  <text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">IGW</text>
+  <text x="450" y="158" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">nginx-alb-igw</text>
+
+  <!-- Fan-out to two public subnets -->
+  <line x1="450" y1="166" x2="450" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="230" y1="184" x2="670" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="230" y1="184" x2="230" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="670" y1="184" x2="670" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+
+  <!-- public_a -->
+  <g transform="translate(108, 202)">
+    <rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_a</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+
+    <rect x="20" y="70" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="85" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">NAT GW</text>
+  </g>
+
+  <!-- public_b -->
+  <g transform="translate(548, 202)">
+    <rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_b</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="56" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="71" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+  </g>
+
+  <!-- SNAT arrow from public_a NAT → private row -->
+  <line x1="230" y1="306" x2="230" y2="340" stroke="#F87171" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.65"/>
+  <polygon points="226,336 234,336 230,344" fill="#F87171" opacity="0.8"/>
+  <rect x="256" y="314" width="240" height="20" rx="10" fill="#0a0a0a" stroke="#F87171" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="376" y="328" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">SNAT for private subnets (egress)</text>
+
+  <!-- private_a -->
+  <g transform="translate(108, 350)">
+    <rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_a</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-a</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_1</text>
+    <text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
+  </g>
+
+  <!-- private_b -->
+  <g transform="translate(548, 350)">
+    <rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_b</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-b</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_2</text>
+    <text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
+  </g>
+</svg>

.github/assets/diagrams/tf-s3-webapp.svg

@@ -0,0 +1,44 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="180" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">S3 WEBAPP · Flask on EC2 with Predastore backend</text>
+
+  <!-- Browser -->
+  <g transform="translate(40, 56)">
+    <rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Browser</text>
+    <text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">end user</text>
+  </g>
+
+  <!-- Arrow: HTTP -->
+  <line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
+  <rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">HTTP</text>
+
+  <!-- EC2 Instance -->
+  <g transform="translate(338, 56)">
+    <rect width="240" height="60" rx="8" fill="#DC26261a" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.65"/>
+    <text x="120" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">EC2 Instance</text>
+    <text x="120" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">Flask webapp · port 80</text>
+  </g>
+
+  <!-- Arrow: S3 API (boto3) -->
+  <line x1="578" y1="86" x2="658" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="654,82 666,86 654,90" fill="#a855f7" opacity="0.8"/>
+  <rect x="570" y="62" width="144" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="642" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">S3 API (boto3)</text>
+
+  <!-- Predastore -->
+  <g transform="translate(676, 56)">
+    <rect width="200" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
+    <text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">Predastore</text>
+    <text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">S3 store · port 8443</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-config-layers.svg

@@ -0,0 +1,78 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="330" viewBox="0 0 900 330">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="cluster-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="pool-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262622"/>
+      <stop offset="100%" style="stop-color:#7F1D1D08"/>
+    </linearGradient>
+    <linearGradient id="node-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#58a6ff22"/>
+      <stop offset="100%" style="stop-color:#58a6ff08"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="330" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">CONFIGURATION LEVELS · spinifex.toml</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">three scopes: cluster · pool · node</text>
+
+  <!-- File badge -->
+  <rect x="24" y="50" width="140" height="24" rx="6" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
+  <text x="94" y="67" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#e4e4e7" text-anchor="middle">spinifex.toml</text>
+
+  <!-- spine -->
+  <line x1="52" y1="74" x2="52" y2="298" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+
+  <!-- ═══ Level 1: [network] cluster-wide ═══════════════════ -->
+  <line x1="52" y1="108" x2="92" y2="108" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+  <rect x="92" y="86" width="764" height="56" rx="8" fill="url(#cluster-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+  <text x="108" y="108" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">[network]</text>
+  <text x="220" y="108" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Cluster-wide · which mode?</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="108" y="128" fill="#a1a1aa">└─</text>
+    <text x="128" y="128" fill="#7ee787">external_mode</text>
+    <text x="224" y="128" fill="#a1a1aa">=</text>
+    <text x="238" y="128" fill="#f5f5f5">"pool"</text>
+  </g>
+
+  <!-- ═══ Level 2: [[network.external_pools]] ═══════════════ -->
+  <line x1="52" y1="176" x2="92" y2="176" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+  <rect x="92" y="154" width="764" height="78" rx="8" fill="url(#pool-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+  <text x="108" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">[[network.external_pools]]</text>
+  <text x="328" y="176" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">IP ranges · what IPs to use?</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="108" y="196" fill="#a1a1aa">├─</text>
+    <text x="128" y="196" fill="#FCA5A5">name</text>
+    <text x="158" y="196" fill="#71717a">·</text>
+    <text x="166" y="196" fill="#FCA5A5">range_start</text>
+    <text x="246" y="196" fill="#71717a">·</text>
+    <text x="254" y="196" fill="#FCA5A5">range_end</text>
+
+    <text x="108" y="212" fill="#a1a1aa">├─</text>
+    <text x="128" y="212" fill="#FCA5A5">gateway</text>
+    <text x="182" y="212" fill="#71717a">·</text>
+    <text x="190" y="212" fill="#FCA5A5">prefix_len</text>
+
+    <text x="108" y="228" fill="#a1a1aa">└─</text>
+    <text x="128" y="228" fill="#FCA5A5">region</text>
+    <text x="174" y="228" fill="#71717a">·</text>
+    <text x="182" y="228" fill="#FCA5A5">az</text>
+    <text x="202" y="228" fill="#71717a">(optional)</text>
+  </g>
+
+  <!-- ═══ Level 3: [nodes.NAME.vpcd] ════════════════════════ -->
+  <line x1="52" y1="268" x2="92" y2="268" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+  <rect x="92" y="246" width="764" height="56" rx="8" fill="url(#node-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
+  <text x="108" y="268" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700">[nodes.NAME.vpcd]</text>
+  <text x="266" y="268" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Per-node · which NIC?</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="108" y="288" fill="#a1a1aa">└─</text>
+    <text x="128" y="288" fill="#8ab4f8">external_interface</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-datapath-localnet.svg

@@ -0,0 +1,63 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="110" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="22" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DATA PATH · VM to WAN via OVN localnet</text>
+
+  <!-- VM TAP -->
+  <g transform="translate(24, 40)">
+    <rect width="118" height="48" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="59" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">VM TAP</text>
+    <text x="59" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">guest interface</text>
+  </g>
+  <polygon points="148,60 160,64 148,68" fill="#71717a" opacity="0.7"/>
+  <line x1="142" y1="64" x2="148" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- br-int -->
+  <g transform="translate(166, 40)">
+    <rect width="102" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="51" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-int</text>
+    <text x="51" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
+  </g>
+  <polygon points="274,60 286,64 274,68" fill="#71717a" opacity="0.7"/>
+  <line x1="268" y1="64" x2="274" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- OVN logical pipeline -->
+  <g transform="translate(292, 40)">
+    <rect width="192" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="96" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN logical pipeline</text>
+    <text x="96" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">switch → router → SNAT/DNAT</text>
+  </g>
+  <polygon points="490,60 502,64 490,68" fill="#71717a" opacity="0.7"/>
+  <line x1="484" y1="64" x2="490" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- localnet -->
+  <g transform="translate(508, 40)">
+    <rect width="120" height="48" rx="7" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
+    <text x="60" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">localnet</text>
+    <text x="60" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">OVN port type</text>
+  </g>
+  <polygon points="634,60 646,64 634,68" fill="#71717a" opacity="0.7"/>
+  <line x1="628" y1="64" x2="634" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- br-wan -->
+  <g transform="translate(652, 40)">
+    <rect width="110" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="55" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+    <text x="55" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
+  </g>
+  <polygon points="768,60 780,64 768,68" fill="#71717a" opacity="0.7"/>
+  <line x1="762" y1="64" x2="768" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- WAN -->
+  <g transform="translate(786, 40)">
+    <rect width="90" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="45" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
+    <text x="45" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">uplink</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-dhcp-conversations.svg

@@ -0,0 +1,74 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="360" viewBox="0 0 900 360">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="host-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#58a6ff22"/>
+      <stop offset="100%" style="stop-color:#58a6ff08"/>
+    </linearGradient>
+    <linearGradient id="vm-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="360" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">TWO DHCP CONVERSATIONS · completely independent</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">pool mode · static or dhcp source</text>
+
+  <!-- ═══ HOST ↔ ROUTER DHCP ══════════════════════════════ -->
+  <rect x="24" y="54" width="852" height="138" rx="10" fill="url(#host-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
+  <text x="44" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700" letter-spacing="0.8">HOST ↔ ROUTER DHCP</text>
+  <text x="876" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="end">on br-ext · veth pair → br-wan</text>
+  <line x1="24" y1="88" x2="876" y2="88" stroke="#58a6ff" stroke-opacity="0.25" stroke-width="0.5"/>
+
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <!-- step 1 -->
+    <circle cx="54" cy="108" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
+    <text x="54" y="112" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">1</text>
+    <text x="74" y="112" fill="#f5f5f5">Spinifex asks router:</text>
+    <text x="228" y="112" fill="#a1a1aa">"give me a public IP for this VM"</text>
+
+    <!-- step 2 -->
+    <circle cx="54" cy="132" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
+    <text x="54" y="136" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">2</text>
+    <text x="74" y="136" fill="#f5f5f5">Router responds:</text>
+    <text x="196" y="136" fill="#8ab4f8" font-weight="600">192.168.1.75</text>
+
+    <!-- step 3 -->
+    <circle cx="54" cy="156" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
+    <text x="54" y="160" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">3</text>
+    <text x="74" y="160" fill="#f5f5f5">Spinifex stores IP, tells OVN:</text>
+    <text x="274" y="160" fill="#F87171" font-weight="600">dnat_and_snat(192.168.1.75 ↔ 172.31.0.4)</text>
+  </g>
+
+  <text x="44" y="181" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">Only used in DHCP source mode. Static mode picks from range.</text>
+
+  <!-- separator -->
+  <line x1="24" y1="206" x2="876" y2="206" stroke="#27272a" stroke-width="0.8"/>
+  <text x="450" y="220" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#52525b" text-anchor="middle" letter-spacing="2">INDEPENDENT · no packets cross between</text>
+
+  <!-- ═══ VM ↔ OVN DHCP ═══════════════════════════════════ -->
+  <rect x="24" y="234" width="852" height="110" rx="10" fill="url(#vm-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+  <text x="44" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="0.8">VM ↔ OVN DHCP</text>
+  <text x="876" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end">on tap → br-int → OVN logical switch</text>
+  <line x1="24" y1="268" x2="876" y2="268" stroke="#3fb950" stroke-opacity="0.25" stroke-width="0.5"/>
+
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <circle cx="54" cy="288" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
+    <text x="54" y="292" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">1</text>
+    <text x="74" y="292" fill="#f5f5f5">cloud-init asks:</text>
+    <text x="190" y="292" fill="#a1a1aa">"what's my IP?"</text>
+
+    <circle cx="54" cy="312" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
+    <text x="54" y="316" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">2</text>
+    <text x="74" y="316" fill="#f5f5f5">OVN DHCP responds:</text>
+    <text x="226" y="316" fill="#a1a1aa">"you are</text>
+    <text x="284" y="316" fill="#7ee787" font-weight="600">172.31.0.4</text>
+    <text x="356" y="316" fill="#a1a1aa">"</text>
+  </g>
+
+  <text x="44" y="336" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">VM never sees 192.168.1.x or the router.</text>
+</svg>

.github/assets/diagrams/vpc-direct-bridge.svg

@@ -0,0 +1,44 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="170" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DIRECT BRIDGE MODE · dedicated WAN NIC</text>
+
+  <!-- Row 1: Management NIC (independent, no WAN bridge path) -->
+  <g transform="translate(24, 42)">
+    <rect width="852" height="44" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
+    <text x="20" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">Management NIC</text>
+    <text x="20" y="36" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">host IP · SSH · spinifex services · overlay (Geneve)</text>
+    <text x="840" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end" font-weight="600">independent path</text>
+  </g>
+
+  <!-- Row 2: WAN NIC → br-wan → localnet -->
+  <g transform="translate(24, 100)">
+    <rect width="160" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="80" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
+    <text x="80" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">dedicated physical</text>
+  </g>
+  <polygon points="198,118 210,122 198,126" fill="#71717a" opacity="0.7"/>
+  <line x1="186" y1="122" x2="198" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <g transform="translate(214, 100)">
+    <rect width="220" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="110" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+    <text x="110" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS port (NIC added directly)</text>
+  </g>
+  <polygon points="448,118 460,122 448,126" fill="#71717a" opacity="0.7"/>
+  <line x1="434" y1="122" x2="448" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <g transform="translate(464, 100)">
+    <rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">OVN localnet</text>
+    <text x="90" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">external switch port</text>
+  </g>
+
+  <text x="670" y="130" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" font-weight="600">distributed NAT · any MAC · all frames</text>
+</svg>

.github/assets/diagrams/vpc-host-flow.svg

@@ -0,0 +1,91 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="440" viewBox="0 0 900 440">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="host-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#27272a88"/>
+      <stop offset="100%" style="stop-color:#27272a22"/>
+    </linearGradient>
+    <linearGradient id="br-int-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f71f"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+    <linearGradient id="br-wan-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#58a6ff1f"/>
+      <stop offset="100%" style="stop-color:#58a6ff08"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="440" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK FLOW · bare-metal host</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">br-int overlay · br-wan WAN uplink</text>
+
+  <!-- ═══ Bare-Metal Host container ═══════════════════════ -->
+  <rect x="40" y="60" width="820" height="284" rx="12" fill="url(#host-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.7"/>
+  <text x="58" y="84" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" font-weight="700" letter-spacing="1">BARE-METAL HOST</text>
+  <line x1="40" y1="96" x2="860" y2="96" stroke="#52525b" stroke-opacity="0.5" stroke-width="0.5"/>
+
+  <!-- br-int bridge (left) -->
+  <g transform="translate(80, 122)">
+    <rect width="260" height="176" rx="10" fill="url(#br-int-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" font-weight="700">br-int</text>
+    <text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="end">(overlay)</text>
+    <line x1="20" y1="34" x2="240" y2="34" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+      <rect x="30" y="52" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
+      <text x="130" y="70" fill="#d8b4fe" text-anchor="middle">tap-eni1</text>
+
+      <rect x="30" y="88" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
+      <text x="130" y="106" fill="#d8b4fe" text-anchor="middle">tap-eni2</text>
+
+      <rect x="30" y="124" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
+      <text x="130" y="142" fill="#d8b4fe" text-anchor="middle">tap-eni3</text>
+    </g>
+  </g>
+
+  <!-- OVN NAT pipeline arrow in middle -->
+  <g transform="translate(352, 200)">
+    <line x1="0" y1="10" x2="148" y2="10" stroke="#DC2626" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+    <polygon points="148,5 160,10 148,15" fill="#DC2626" opacity="0.8"/>
+    <text x="78" y="0" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN</text>
+    <text x="78" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">NAT pipeline</text>
+  </g>
+
+  <!-- br-wan bridge (right) -->
+  <g transform="translate(520, 122)">
+    <rect width="260" height="176" rx="10" fill="url(#br-wan-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" font-weight="700">br-wan</text>
+    <text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#8ab4f8" text-anchor="end">(WAN)</text>
+    <line x1="20" y1="34" x2="240" y2="34" stroke="#58a6ff" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+      <rect x="30" y="52" width="200" height="60" rx="5" fill="#58a6ff1a" stroke="#58a6ff" stroke-opacity="0.45" stroke-width="0.8"/>
+      <text x="130" y="74" fill="#8ab4f8" text-anchor="middle" font-weight="600">spx-ext-{nic}</text>
+      <text x="130" y="92" fill="#a1a1aa" text-anchor="middle" font-size="10">(macvlan)</text>
+    </g>
+  </g>
+
+  <!-- macvlan bridge mode label -->
+  <text x="650" y="314" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">macvlan bridge mode</text>
+
+  <!-- WAN NIC inside host, bottom -->
+  <line x1="650" y1="298" x2="650" y2="320" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <g transform="translate(500, 320)">
+    <rect width="300" height="20" rx="5" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
+    <text x="150" y="14" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">WAN NIC · host IP unchanged</text>
+  </g>
+
+  <!-- connector out -->
+  <line x1="650" y1="344" x2="650" y2="380" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="646,376 654,376 650,384" fill="#71717a" opacity="0.7"/>
+
+  <!-- Physical WAN -->
+  <g transform="translate(500, 388)">
+    <rect width="300" height="40" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
+    <text x="150" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">Physical WAN</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-macvlan.svg

@@ -0,0 +1,57 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="320" viewBox="0 0 900 320">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="320" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">MACVLAN FALLBACK · single-NIC host</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">WAN NIC == SSH NIC · host IP preserved</text>
+
+  <!-- WAN NIC (parent) -->
+  <g transform="translate(280, 58)">
+    <rect width="340" height="54" rx="8" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.3" stroke-opacity="0.65"/>
+    <text x="170" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
+    <text x="170" y="42" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">host IP unchanged · parent interface</text>
+  </g>
+
+  <!-- Split point -->
+  <line x1="450" y1="112" x2="450" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <line x1="260" y1="132" x2="640" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <line x1="260" y1="132" x2="260" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <line x1="640" y1="132" x2="640" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- Child 1: host traffic -->
+  <g transform="translate(120, 148)">
+    <rect width="280" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">host traffic</text>
+    <text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SSH · management · overlay</text>
+  </g>
+
+  <!-- Child 2: macvlan -->
+  <g transform="translate(500, 148)">
+    <rect width="280" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">spx-ext-{nic}</text>
+    <text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">macvlan · bridge mode · no IP</text>
+  </g>
+
+  <!-- br-wan -->
+  <line x1="640" y1="196" x2="640" y2="216" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="636,212 644,212 640,220" fill="#71717a" opacity="0.7"/>
+
+  <g transform="translate(500, 220)">
+    <rect width="280" height="40" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="140" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+  </g>
+
+  <!-- localnet -->
+  <line x1="640" y1="260" x2="640" y2="276" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="636,272 644,272 640,280" fill="#71717a" opacity="0.7"/>
+
+  <g transform="translate(500, 280)">
+    <rect width="280" height="30" rx="6" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
+    <text x="140" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="600">localnet → OVN ext switch</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-network-path.svg

@@ -0,0 +1,131 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="wan-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="ovs-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#58a6ff22"/>
+      <stop offset="100%" style="stop-color:#58a6ff0a"/>
+    </linearGradient>
+    <linearGradient id="router-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#DC262622"/>
+      <stop offset="100%" style="stop-color:#7F1D1D0a"/>
+    </linearGradient>
+    <linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f722"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="560" fill="url(#bg)" rx="12"/>
+
+  <!-- Title -->
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK PATH</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">OVN logical topology · bridge mapping · NAT</text>
+
+  <!-- WAN -->
+  <rect x="300" y="54" width="300" height="40" rx="8" fill="url(#wan-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.55"/>
+  <text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Physical Network (WAN)</text>
+
+  <!-- connector -->
+  <line x1="450" y1="94" x2="450" y2="112" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- br-wan bridge -->
+  <rect x="340" y="112" width="220" height="52" rx="8" fill="url(#ovs-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.65"/>
+  <text x="450" y="134" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+  <text x="450" y="152" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">OVS bridge · physical uplink</text>
+
+  <!-- connector with bridge-mappings label -->
+  <line x1="450" y1="164" x2="450" y2="186" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <rect x="316" y="170" width="268" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.5"/>
+  <text x="450" y="183" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">ovn-bridge-mappings: external:br-wan</text>
+
+  <!-- VPC Logical Router (the big box) -->
+  <rect x="60" y="208" width="780" height="196" rx="10" fill="url(#router-grad)" stroke="#DC2626" stroke-width="1.4" stroke-opacity="0.75"/>
+  <text x="78" y="232" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" font-weight="700" letter-spacing="1">VPC LOGICAL ROUTER</text>
+  <line x1="60" y1="242" x2="840" y2="242" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+
+  <!-- Router ports row -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <rect x="82" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
+    <text x="142" y="270" fill="#F87171" text-anchor="middle" font-weight="600">Gateway port</text>
+    <text x="142" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">public IP</text>
+
+    <text x="215" y="282" fill="#71717a">─────</text>
+
+    <rect x="260" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
+    <text x="320" y="270" fill="#F87171" text-anchor="middle" font-weight="600">ext switch</text>
+    <text x="320" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">OVN logical</text>
+
+    <text x="393" y="282" fill="#71717a">─────</text>
+
+    <rect x="438" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
+    <text x="498" y="270" fill="#F87171" text-anchor="middle" font-weight="600">localnet port</text>
+    <text x="498" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">→ br-wan</text>
+  </g>
+
+  <!-- NAT rules + Routes, two columns on right -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
+    <text x="598" y="260" fill="#F87171" font-weight="600">NAT rules</text>
+    <text x="598" y="276" fill="#f5f5f5">SNAT:</text>
+    <text x="644" y="276" fill="#a1a1aa">VPC CIDR → gw IP</text>
+    <text x="598" y="292" fill="#f5f5f5">dnat_and_snat:</text>
+    <text x="704" y="292" fill="#a1a1aa">pub ↔ priv</text>
+  </g>
+
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
+    <text x="82" y="330" fill="#F87171" font-weight="600">Routes</text>
+    <text x="82" y="346" fill="#f5f5f5">0.0.0.0/0</text>
+    <text x="154" y="346" fill="#a1a1aa">→ WAN gateway</text>
+  </g>
+
+  <!-- Subnet branches inside router -->
+  <line x1="150" y1="362" x2="150" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="150" y1="386" x2="305" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="305" y1="386" x2="305" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+
+  <line x1="150" y1="386" x2="515" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="515" y1="386" x2="515" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+
+  <!-- Subnet A (public) -->
+  <g transform="translate(215, 404)">
+    <rect width="180" height="124" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">subnet-a</text>
+    <text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">(public)</text>
+    <line x1="16" y1="46" x2="164" y2="46" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
+    <text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.1.0/24</text>
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
+      <circle cx="40" cy="88" r="2.5" fill="#3fb950"/>
+      <text x="52" y="92">eni-1</text>
+      <circle cx="40" cy="108" r="2.5" fill="#3fb950"/>
+      <text x="52" y="112">eni-2</text>
+    </g>
+  </g>
+
+  <!-- Subnet B (private) -->
+  <g transform="translate(425, 404)">
+    <rect width="180" height="124" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">subnet-b</text>
+    <text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">(private)</text>
+    <line x1="16" y1="46" x2="164" y2="46" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+    <text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.2.0/24</text>
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
+      <circle cx="40" cy="88" r="2.5" fill="#a855f7"/>
+      <text x="52" y="92">eni-3</text>
+      <circle cx="40" cy="108" r="2.5" fill="#a855f7"/>
+      <text x="52" y="112">eni-4</text>
+    </g>
+  </g>
+
+  <!-- Footnote -->
+  <text x="24" y="548" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a">Cross-host traffic uses Geneve tunnels (UDP 6081) over the overlay NIC · ovn-controller programs OpenFlow rules on br-int</text>
+</svg>

.github/assets/diagrams/vpc-private-subnet-flow.svg

@@ -0,0 +1,56 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="110" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PRIVATE SUBNET · no default route</text>
+
+  <!-- Instance -->
+  <g transform="translate(40, 42)">
+    <rect width="180" height="44" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">Instance</text>
+    <text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">10.0.2.5</text>
+  </g>
+
+  <!-- arrow -->
+  <g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
+    <line x1="220" y1="64" x2="260" y2="64" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="260,60 268,64 260,68" fill="#DC2626" opacity="0.6"/>
+
+  <!-- Router -->
+  <g transform="translate(275, 42)">
+    <rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+    <text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">VPC Router</text>
+    <text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">lookup 0.0.0.0/0</text>
+  </g>
+
+  <!-- arrow -->
+  <g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
+    <line x1="455" y1="64" x2="495" y2="64" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="495,60 503,64 495,68" fill="#DC2626" opacity="0.6"/>
+
+  <!-- [no default route] -->
+  <g transform="translate(510, 42)">
+    <rect width="196" height="44" rx="7" fill="#0a0a0a" stroke="#71717a" stroke-width="1" stroke-opacity="0.7" stroke-dasharray="4,3"/>
+    <text x="98" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">[no default route]</text>
+    <text x="98" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle">no IGW attached</text>
+  </g>
+
+  <!-- arrow to dropped -->
+  <g stroke="#71717a" stroke-width="1.2" opacity="0.6" fill="none">
+    <line x1="706" y1="64" x2="746" y2="64" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="746,60 754,64 746,68" fill="#71717a" opacity="0.6"/>
+
+  <!-- dropped -->
+  <g transform="translate(760, 42)">
+    <rect width="116" height="44" rx="7" fill="#27272a" stroke="#52525b" stroke-width="1" stroke-opacity="0.7"/>
+    <text x="58" y="27" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">dropped</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-public-subnet-flow.svg

@@ -0,0 +1,74 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="170" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PUBLIC SUBNET · 1:1 NAT managed by OVN</text>
+
+  <!-- OUTBOUND row -->
+  <text x="24" y="52" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700">Outbound</text>
+
+  <g transform="translate(112, 38)">
+    <rect width="180" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">Instance</text>
+    <text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">10.0.1.5</text>
+  </g>
+
+  <g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="292" y1="58" x2="352" y2="58" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="352,54 360,58 352,62" fill="#3fb950" opacity="0.7"/>
+
+  <g transform="translate(368, 38)">
+    <rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">SNAT</text>
+  </g>
+
+  <g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="498" y1="58" x2="558" y2="58" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="558,54 566,58 558,62" fill="#3fb950" opacity="0.7"/>
+
+  <g transform="translate(574, 38)">
+    <rect width="272" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">exits as 203.0.113.10</text>
+    <text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">source IP rewritten</text>
+  </g>
+
+  <!-- INBOUND row -->
+  <text x="24" y="110" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" font-weight="700">Inbound</text>
+
+  <g transform="translate(112, 96)">
+    <rect width="180" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">WAN client</text>
+    <text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">→ 203.0.113.10</text>
+  </g>
+
+  <g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="292" y1="116" x2="352" y2="116" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="352,112 360,116 352,120" fill="#58a6ff" opacity="0.7"/>
+
+  <g transform="translate(368, 96)">
+    <rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">DNAT</text>
+  </g>
+
+  <g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="498" y1="116" x2="558" y2="116" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="558,112 566,116 558,120" fill="#58a6ff" opacity="0.7"/>
+
+  <g transform="translate(574, 96)">
+    <rect width="272" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">arrives at 10.0.1.5</text>
+    <text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">dest IP rewritten</text>
+  </g>
+
+  <!-- Footnote -->
+  <text x="24" y="158" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a">The instance OS only sees its private IP (10.0.1.5) — the public address is managed entirely by OVN's dnat_and_snat rule.</text>
+</svg>

.github/assets/platform.svg

@@ -0,0 +1,144 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
+  <defs>
+    <linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
+      <stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
+    </linearGradient>
+    <linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
+      <stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
+    </linearGradient>
+    <linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
+    </linearGradient>
+    <linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
+      <stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
+
+  <!-- Subtle mesh dots background -->
+  <g opacity="0.12" fill="#7F1D1D">
+    <circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
+    <circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
+    <circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
+    <circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
+    <circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
+    <circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
+  </g>
+
+  <!-- Title -->
+  <text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
+  <text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
+
+  <!-- Layer 4: Applications (top) -->
+  <g transform="translate(60, 95)">
+    <rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
+    <text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
+
+    <g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
+      <rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
+
+      <rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
+
+      <rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
+
+      <rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
+
+      <rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
+
+      <rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
+    </g>
+  </g>
+
+  <!-- Down arrow -->
+  <g transform="translate(450, 188)" opacity="0.45">
+    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
+    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
+  </g>
+
+  <!-- Layer 3: Spinifex (highlighted) -->
+  <g transform="translate(60, 215)">
+    <rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
+    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
+    <text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
+
+    <g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
+      <!-- EC2 -->
+      <rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
+      <text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
+      <text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
+
+      <!-- EBS -->
+      <rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
+      <text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
+      <text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
+
+      <!-- S3 -->
+      <rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
+      <text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
+      <text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
+
+      <!-- VPC -->
+      <rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
+      <text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
+      <text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
+      <text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
+
+      <!-- IAM -->
+      <rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
+      <text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
+      <text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
+      <text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
+
+      <!-- Control -->
+      <rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
+      <text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
+      <text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
+    </g>
+  </g>
+
+  <!-- Down arrow -->
+  <g transform="translate(450, 360)" opacity="0.45">
+    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
+    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
+  </g>
+
+  <!-- Layer 2: OS -->
+  <g transform="translate(60, 387)">
+    <rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
+    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
+    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
+  </g>
+
+  <!-- Down arrow -->
+  <g transform="translate(450, 454)" opacity="0.45">
+    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
+    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
+  </g>
+
+  <!-- Layer 1: Hardware -->
+  <g transform="translate(60, 481)">
+    <rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
+    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
+  </g>
+</svg>

.github/dependabot.yml

@@ -8,7 +8,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     groups:
       go-dependencies:
         patterns:
@@ -22,7 +22,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     ignore:
       - dependency-name: "*"
         update-types:
@@ -41,7 +41,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     ignore:
       - dependency-name: "@types/node"
         update-types:

.github/workflows/e2e-baremetal.yml

@@ -0,0 +1,117 @@
+name: E2E Bare-Metal
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  statuses: write
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+
+jobs:
+  e2e_baremetal:
+    name: Bare-Metal Multi-Node E2E
+    runs-on: [ self-hosted, ci-baremetal ]
+    timeout-minutes: 120
+    concurrency:
+      group: e2e-baremetal-cluster
+      cancel-in-progress: false
+    env:
+      BM_ISO_CACHE: /var/cache/spinifex-iso-builder
+    steps:
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/checkout@v6
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: |
+            scripts/iso-builder
+            scripts/baremetal
+          path: mulga
+
+      - name: Install SSH Key
+        run: |
+          echo "${{ secrets.BM_SSH_PRIVATE_KEY }}" > "${RUNNER_TEMP}/bm-ci"
+          chmod 0600 "${RUNNER_TEMP}/bm-ci"
+          ssh-keygen -y -f "${RUNNER_TEMP}/bm-ci" > "${RUNNER_TEMP}/bm-ci.pub"
+
+      - name: Build Artifacts
+        run: |
+          mulga/scripts/baremetal/bm-build-artifacts.sh \
+            "$GITHUB_WORKSPACE" \
+            "${RUNNER_TEMP}/spinifex-tests.tar" \
+            "${RUNNER_TEMP}/bm-bins"
+
+      - name: Build PXE Artifacts
+        run: |
+          sudo make -C mulga/scripts/iso-builder \
+            build-rootfs build-squashfs build-initramfs \
+            CACHE="${BM_ISO_CACHE}" \
+            SPINIFEX_BIN="${RUNNER_TEMP}/bm-bins/spx" \
+            INSTALLER_BIN="${RUNNER_TEMP}/bm-bins/spinifex-installer" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}" \
+            VIPERBLOCK_SRC="${RUNNER_TEMP}/bm-bins/viperblock" \
+            VERSION="$(cd "$GITHUB_WORKSPACE" && git describe --tags --always 2>/dev/null || echo dev)"
+
+      - name: Write ePDU credentials
+        run: |
+          install -m 0600 /dev/null "${RUNNER_TEMP}/epdu.env"
+          cat >> "${RUNNER_TEMP}/epdu.env" <<EOF
+          EPDU_HOST=192.168.1.63
+          EPDU_SNMP_USER=ci-bmpower
+          EPDU_SNMP_SEC_LEVEL=authPriv
+          EPDU_SNMP_AUTH_PROTO=MD5
+          EPDU_SNMP_AUTH_PASS=${{ secrets.BM_EPDU_AUTH_PASS }}
+          EPDU_SNMP_PRIV_PROTO=DES
+          EPDU_SNMP_PRIV_PASS=${{ secrets.BM_EPDU_PRIV_PASS }}
+          EOF
+
+      - name: Cache VM images
+        run: |
+          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-24.04-cloudimg-amd64.img"
+          if [ ! -f "$UBUNTU_IMG" ]; then
+            echo "Downloading Ubuntu 24.04 cloud image..."
+            sudo curl -fsSL -o "$UBUNTU_IMG" \
+              "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
+            echo "  Downloaded $(du -h "$UBUNTU_IMG" | cut -f1)"
+          else
+            echo "Ubuntu 24.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
+          fi
+
+          LB_IMG="${BM_ISO_CACHE}/lb-alpine-3.21.6-x86_64.raw"
+          if [ ! -f "$LB_IMG" ]; then
+            echo "Downloading lb-alpine image..."
+            sudo curl -fsSL -o "$LB_IMG" \
+              "https://iso.mulgadc.com/system-ami/lb-alpine-3.21.6-x86_64.raw"
+            echo "  Downloaded $(du -h "$LB_IMG" | cut -f1)"
+          else
+            echo "lb-alpine image cached ($(du -h "$LB_IMG" | cut -f1))"
+          fi
+
+      - name: Run Bare-Metal E2E
+        env:
+          BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
+          BM_SSH_KEY: ${{ runner.temp }}/bm-ci
+          BM_EPDU_ENV: ${{ runner.temp }}/epdu.env
+          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-24.04-cloudimg-amd64.img
+          BM_LB_ALPINE_IMAGE: ${{ env.BM_ISO_CACHE }}/lb-alpine-3.21.6-x86_64.raw
+          SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
+        run: |
+          sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
+            --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
+            --initrd    "${BM_ISO_CACHE}/initrd.img" \
+            --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
+            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"

.github/workflows/e2e-ddil.yml

@@ -0,0 +1,203 @@
+name: E2E DDIL
+
+# Phase 1 trigger: workflow_dispatch only. Nightly cron + on-main-merge
+# land in mulga-siv-5.8 once ≥4 scenarios are asserting. See
+# docs/development/improvements/ddil-e2e-test-harness.md §4 / §Dependencies.
+on:
+  workflow_dispatch:
+    inputs:
+      quarantined:
+        description: "Comma-separated scenario letters to quarantine (A..F)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+
+jobs:
+  ddil:
+    name: DDIL Real Multi-Node
+    runs-on: [ self-hosted, ci-multi ]
+    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
+    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
+    # dominated by provisioning; the 60m ceiling leaves headroom for
+    # Phase 2/3 when assertions come online.
+    timeout-minutes: 60
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
+    steps:
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      # Full spinifex checkout: the Go harness must run on the runner
+      # (outside the cluster) so its SSH path survives scenario partitions
+      # that drop peer-to-peer cluster traffic. Running inside the cluster
+      # would self-sever when Scenario C isolates a node.
+      - uses: actions/checkout@v6
+        name: Checkout Spinifex
+        with:
+          path: spinifex
+
+      - uses: actions/checkout@v6
+        name: Checkout Mulga (tofu-cluster)
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - name: Install gotestsum
+        env:
+          GOTESTSUM_VERSION: "1.12.0"
+        run: |
+          # Prebuilt tarball, not `go install`: gotestsum v1.12.0 pulls
+          # x/tools v0.19.0 whose tokeninternal static-assert pegs a
+          # narrow Go window (~1.22). The self-hosted runner's Go falls
+          # outside that window, so source-install fails with
+          # `invalid array length -delta * delta`. A prebuilt binary
+          # sidesteps toolchain coupling entirely.
+          BIN_DIR="${RUNNER_TEMP}/gotestsum-bin"
+          mkdir -p "${BIN_DIR}"
+          if [ ! -x "${BIN_DIR}/gotestsum" ]; then
+            curl -fsSL \
+              "https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz" \
+              | tar -xz -C "${BIN_DIR}" gotestsum
+          fi
+          echo "${BIN_DIR}" >> "${GITHUB_PATH}"
+          "${BIN_DIR}/gotestsum" --version
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          tofu init -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Run DDIL Scenarios
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -o pipefail
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
+          REGION=$(echo "$INVENTORY" | jq -r '.region')
+          # DDIL_NODES must be the cluster-traffic IPs: PartitionNode
+          # installs iptables DROP rules for each peer, so the runner's
+          # SSH source IP must differ from these. On single-NIC
+          # tofu-cluster, wan_ip == cluster IP and the runner's outbound
+          # IP is distinct — see e2e-ddil-test-harness.md §3 Scenario C.
+          NODE_IPS=$(echo "$INVENTORY" | jq -r '[.nodes[].wan_ip] | join(",")')
+          PRIMARY_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          echo "DDIL_NODES=${NODE_IPS}"
+
+          # Witness VMs talk to the cluster's AWS gateway via aws-sdk-go,
+          # which resolves credentials from the standard chain. spx admin
+          # init writes [spinifex] to ${SSH_USER}'s ~/.aws/credentials on
+          # node1 — pull them over SSH and export as the SDK's env-var
+          # entries so the harness can launch instances without depending
+          # on a cluster-bootstrapped profile being present on the runner.
+          CREDS=$(ssh -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+            "${SSH_USER}@${PRIMARY_IP}" 'cat ~/.aws/credentials')
+          AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
+            echo "::error::Failed to extract spinifex AWS credentials from ${PRIMARY_IP}"
+            exit 1
+          fi
+
+          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
+          # the matching private material was generated on the primary node at
+          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
+          # the witness via its MapPublicIpOnLaunch-assigned EIP.
+          GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
+          scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+            "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
+          chmod 600 "${GUEST_KEY}"
+
+          mkdir -p "${RUNNER_TEMP}/ddil-results"
+          cd "${GITHUB_WORKSPACE}/spinifex"
+          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout
+          # while still emitting the machine-readable jsonfile artifact.
+          DDIL_NODES="${NODE_IPS}" \
+          DDIL_SSH_USER="${SSH_USER}" \
+          DDIL_SSH_KEY="${SSH_KEY}" \
+          DDIL_GUEST_SSH_KEY="${GUEST_KEY}" \
+          DDIL_QUARANTINED="${{ inputs.quarantined }}" \
+          AWS_REGION="${REGION}" \
+          AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
+          AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
+            gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
+                      --format testname \
+                      -- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
+
+      - name: Collect Cluster Logs
+        if: failure()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          # Pull spinifex-* journals + QEMU process listings from every
+          # node into the artifact bundle so failures can be triaged
+          # without re-provisioning the cluster. Best-effort — a missing
+          # node should not mask the original test failure.
+          set +e
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
+          LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
+          mkdir -p "${LOG_DIR}"
+
+          mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
+          for ip in "${NODE_IPS[@]}"; do
+            echo "=== collecting ${ip} ==="
+            SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
+            ${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
+            ${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
+            ${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
+          done
+
+      - name: Upload DDIL Results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: ddil-results-${{ github.run_id }}
+          path: ${{ runner.temp }}/ddil-results
+          retention-days: 30
+
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true

.github/workflows/e2e-nightly.yml

@@ -0,0 +1,218 @@
+name: E2E Nightly
+
+on:
+  schedule:
+    - cron: "0 14 * * 0-4" # 00:00 Mon-Fri AEST = 14:00 Sun-Thu UTC
+  workflow_dispatch:
+    inputs:
+      cells:
+        description: "Comma-separated cell numbers to run (blank = all)"
+        required: false
+
+permissions:
+  contents: read
+
+# Scheduled runs are loaded from the default branch (main), so the schedule
+# trigger must stay in sync on main for cron to fire.
+env:
+  GIT_BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+  UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-24.04.qcow2
+
+jobs:
+  matrix:
+    name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{
+      matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
+    strategy:
+      fail-fast: false
+      max-parallel: 3
+      matrix:
+        include:
+          - { cell: 1,  install: binary, topo: single,     net: dhcp,   host: debian-13,    arch: x86_64, runner: ci-single, extra: "" }
+          - { cell: 2,  install: binary, topo: single,     net: static, host: ubuntu-24.04, arch: x86_64, runner: ci-single, extra: "" }
+          - { cell: 3,  install: binary, topo: single,     net: dhcp,   host: ubuntu-24.04, arch: x86_64, runner: ci-single, extra: "" }
+          - { cell: 8,  install: binary, topo: real-multi, net: dhcp,   host: debian-13,    arch: x86_64, runner: ci-multi,  extra: "" }
+          - { cell: 9,  install: binary, topo: real-multi, net: static, host: ubuntu-24.04, arch: x86_64, runner: ci-multi,  extra: "" }
+          - { cell: 10, install: binary, topo: real-multi, net: dhcp,   host: ubuntu-24.04, arch: x86_64, runner: ci-multi,  extra: "" }
+          - { cell: 17, install: binary, topo: single,     net: static, host: debian-13,    arch: x86_64, runner: ci-pseudo, extra: "tofu-examples" }
+          - { cell: 18, install: binary, topo: single,     net: static, host: debian-13,    arch: x86_64, runner: ci-single, extra: "reboot-single" }
+
+    runs-on: [ self-hosted, "${{ matrix.runner }}" ]
+    timeout-minutes: 35
+    env:
+      TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+
+    steps:
+      # blank or scheduled run = full matrix.
+      # Input must be comma-separated with no whitespace (e.g. '1,2,17', not '1, 2').
+      - name: Gate on Dispatch Input
+        id: gate
+        env:
+          CELL: ${{ matrix.cell }}
+          CELLS: ${{ github.event_name == 'workflow_dispatch' && inputs.cells || '' }}
+        run: |
+          if [ -z "$CELLS" ] || [[ ",${CELLS}," == *",${CELL},"* ]]; then
+            echo "selected=true" >> "$GITHUB_OUTPUT"
+            echo "cell ${CELL} selected"
+          else
+            echo "selected=false" >> "$GITHUB_OUTPUT"
+            echo "cell ${CELL} not in '${CELLS}' — subsequent steps will skip"
+          fi
+
+      - name: Clean Workspace
+        if: steps.gate.outputs.selected == 'true'
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        if: steps.gate.outputs.selected == 'true'
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - name: Build Install Artifacts
+        if: steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        if: steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        env:
+          MATRIX_HOST: ${{ matrix.host }}
+        run: |
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+
+          TOFU_VAR_ARGS=""
+          if [ "$MATRIX_HOST" = "ubuntu-24.04" ]; then
+            TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
+          fi
+
+          tofu init -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+
+      - name: Bootstrap via Binary Install
+        if: steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        env:
+          MATRIX_NET: ${{ matrix.net }}
+        run: |
+          if [ "$MATRIX_NET" = "dhcp" ]; then
+            unset SPX_EXTERNAL_POOL
+            export SPX_EXTERNAL_SOURCE=dhcp
+          else
+            export SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}"
+          fi
+
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Run Single Node E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+
+      - name: Run Multi Node E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+
+          NODE_IPS=""
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+          done
+
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+
+      - name: Run Tofu Workbooks
+        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+
+      - name: Run Reboot Resilience E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
+        working-directory: mulga/scripts/tofu-cluster
+        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # systemctl-reboot-induced ssh disconnect mid-run.
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+
+          REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
+          mkdir -p "$REBOOT_DIR"
+          tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
+          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+
+          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+
+      - name: Dump Cluster Journals on Failure
+        if: failure() && steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            echo "===== node$((i+1)) ($IP) ====="
+            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
+              echo "--- $svc ---"
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
+                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
+            done
+            echo "--- DHCP client state ---"
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
+              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+          done
+
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        env:
+          MATRIX_HOST: ${{ matrix.host }}
+        run: |
+          TOFU_VAR_ARGS=""
+          if [ "$MATRIX_HOST" = "ubuntu-24.04" ]; then
+            TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
+          fi
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true

.github/workflows/e2e.yml

@@ -1,4 +1,4 @@
-name: Spinifex E2E Tests
+name: E2E
 
 on:
   push:
@@ -13,6 +13,14 @@ on:
       - tests/e2e/**
       - .github/workflows/e2e.yml
   workflow_dispatch:
+    inputs:
+      sha:
+        description: "PR head SHA to report status on"
+        required: false
+
+permissions:
+  contents: read
+  statuses: write
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
@@ -20,18 +28,18 @@ env:
 jobs:
   e2e_single:
     name: Single-Node E2E
-    runs-on: [self-hosted, ci-single]
-    timeout-minutes: 25
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
     steps:
-      - name: Clean workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -42,27 +50,32 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - name: Clone and build
+      - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Provision VM
         working-directory: mulga/scripts/tofu-cluster
         run: |
           export TF_VAR_git_branch="${GIT_BRANCH}"
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
-      - name: Bootstrap Cluster
+      - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          SPX_PREBUILT=true \
-          SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          SPX_EXTERNAL_POOL="192.168.1.230-192.168.1.240" \
-          ./bootstrap.sh env3 >> "$TOFU_LOG" 2>&1
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
@@ -72,30 +85,30 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            'cd ~/Development/mulga/spinifex && ./tests/e2e/run-e2e.sh && ./tests/e2e/run-cert-e2e.sh'
+            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
   e2e_pseudo_multi:
     name: Pseudo Multi-Node E2E
-    runs-on: [self-hosted, ci-pseudo]
-    timeout-minutes: 30
+    runs-on: [ self-hosted, ci-pseudo ]
+    timeout-minutes: 25
     env:
       TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
     steps:
-      - name: Clean workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -106,26 +119,27 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - name: Clone and build
+      - name: Clone and Build
         working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
+        run: ./build-artifacts.sh "${GIT_BRANCH}"
+          "${RUNNER_TEMP}/spinifex-artifacts.tar"
 
       - name: Provision VM
         working-directory: mulga/scripts/tofu-cluster
         run: |
           export TF_VAR_git_branch="${GIT_BRANCH}"
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
       - name: Bootstrap Cluster
         working-directory: mulga/scripts/tofu-cluster
         run: |
           SPX_PREBUILT=true \
           SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          ./bootstrap.sh env4 >> "$TOFU_LOG" 2>&1
+          ./bootstrap.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
 
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
@@ -135,30 +149,30 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            'cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL="192.168.1.220-192.168.1.229" ./tests/e2e/run-pseudo-multinode-e2e.sh'
+            "cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL=\"${SPINIFEX_CI_POOL}\" ./tests/e2e/run-pseudo-multinode-e2e.sh"
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
   e2e_real_multi:
     name: Real Multi-Node E2E
-    runs-on: [self-hosted, ci-multi]
-    timeout-minutes: 25
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 30
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
     steps:
-      - name: Clean workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -169,27 +183,32 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - name: Clone and build
+      - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
           export TF_VAR_git_branch="${GIT_BRANCH}"
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
-      - name: Bootstrap Cluster
+      - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          SPX_PREBUILT=true \
-          SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          SPX_EXTERNAL_POOL="192.168.1.240-192.168.1.250" \
-          ./bootstrap.sh env2 >> "$TOFU_LOG" 2>&1
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
@@ -206,13 +225,41 @@ jobs:
             NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
           done
 
+          NODE2_IP=$(echo "$INVENTORY" | jq -r '.nodes[1].wan_ip')
+
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/Development/mulga/spinifex && ./tests/e2e/run-multinode-e2e.sh ${NODE_IPS} && ./tests/e2e/run-cert-e2e.sh --multinode ${NODE_IPS}"
+            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  e2e_status:
+    name: Report E2E Status
+    if: always() && inputs.sha
+    needs: [ e2e_single, e2e_pseudo_multi, e2e_real_multi ]
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 5
+    steps:
+      - name: Report Status to PR
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          if [[ "${{ needs.e2e_single.result }}" == "success" && \
+                "${{ needs.e2e_pseudo_multi.result }}" == "success" && \
+                "${{ needs.e2e_real_multi.result }}" == "success" ]]; then
+            STATE="success"
+            DESC="All E2E suites passed"
+          else
+            STATE="failure"
+            DESC="E2E failed: single=${{ needs.e2e_single.result }} pseudo=${{ needs.e2e_pseudo_multi.result }} multi=${{ needs.e2e_real_multi.result }}"
+          fi
+          gh api "repos/${{ github.repository }}/statuses/${{ inputs.sha }}" \
+            -f state="$STATE" \
+            -f context=e2e \
+            -f description="$DESC" \
+            -f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"

.github/workflows/release.yml

@@ -2,7 +2,7 @@ name: Release
 
 on:
   push:
-    tags: ["v*"]
+    tags: [ "v*" ]
 
 permissions:
   contents: write
@@ -31,6 +31,14 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v4
 
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.3'
+
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+        run: cd spinifex && make build-microvm-image
+
       - name: Build amd64 Tarball
         env:
           VERSION: ${{ github.ref_name }}
@@ -82,7 +90,7 @@ jobs:
             spinifex/dist/spinifex-${{ github.ref_name }}-linux-arm64.tar.gz.sha256
 
   release:
-    needs: [distro-amd64, distro-arm64]
+    needs: [ distro-amd64, distro-arm64 ]
     runs-on: ubuntu-24.04
     steps:
       - name: Download amd64 Artifacts
@@ -98,7 +106,7 @@ jobs:
           path: artifacts
 
       - name: Create Draft GitHub Release
-        uses: softprops/action-gh-release@v2
+        uses: softprops/action-gh-release@v3
         with:
           draft: true
           files: |
@@ -115,7 +123,7 @@ jobs:
 
   verify:
     needs: release
-    runs-on: [self-hosted, ci-single]
+    runs-on: [ self-hosted, ci-single ]
     timeout-minutes: 25
     env:
       TOFU_LOG: /tmp/release-verify-${{ github.run_id }}.log
@@ -128,7 +136,7 @@ jobs:
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -149,40 +157,30 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
         run: |
           SETUP_SH=$(find "${GITHUB_WORKSPACE}/release-assets" -name "setup.sh" | head -1)
           TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
-          ./bootstrap-install.sh env3 "$SETUP_SH" "$TARBALL"
-
-      - name: Run Smoke Test
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          INVENTORY=$(tofu output -json inventory)
-          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
-            -i "$SSH_KEY" \
-            smoke-test-install.sh \
-            "tf-user@${WAN_IP}:/tmp/smoke-test.sh"
-          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
-            -i "$SSH_KEY" "tf-user@${WAN_IP}" \
-            'bash /tmp/smoke-test.sh'
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" "$SETUP_SH" "$TARBALL"
 
       - name: Promote Release
         if: success()
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          gh release edit "$TAG" --repo "$REPO" --draft=false
-          echo "Release ${TAG} promoted to latest"
+          if [[ "$TAG" =~ -(dev|rc|alpha|beta) ]]; then
+            gh release edit "$TAG" --repo "$REPO" --draft=false --prerelease
+            echo "Release ${TAG} promoted as prerelease"
+          else
+            gh release edit "$TAG" --repo "$REPO" --draft=false
+            echo "Release ${TAG} promoted to latest"
+          fi
 
       - name: Mark Release Failed
         if: failure()
@@ -199,6 +197,153 @@ jobs:
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  iso:
+    needs: release
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 90
+    steps:
+      - name: Clean Workspace
+        run: |
+          ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
+          # Mirror make clean's umount sequence unconditionally — lazy unmount
+          # detaches immediately even if the mount is busy, so rm -rf is safe.
+          sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
+          sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - name: Checkout Spinifex
+        uses: actions/checkout@v6
+        with:
+          path: spinifex
+          ref: ${{ github.ref }}
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/predastore
+          path: predastore
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/viperblock
+          path: viperblock
+
+      - name: Checkout Mulga iso-builder
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/iso-builder
+          path: mulga
+
+      - name: Clean ISO builder cache
+        run: sudo make -C mulga/scripts/iso-builder clean
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Configure Go workspace
+        run: go work init ./spinifex ./predastore ./viperblock
+
+      - name: Install ISO builder dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y \
+            debootstrap squashfs-tools xorriso grub-pc-bin grub-efi-amd64-bin \
+            grub-common mtools busybox-static ovmf zstd \
+            nbdkit nbdkit-plugin-dev pkg-config gcc
+
+      - name: Build spinifex binaries
+        run: cd spinifex && make build
+
+      - name: Build viperblock
+        run: cd viperblock && make build
+
+      - name: Build release ISO
+        id: build-iso
+        env:
+          VERSION: ${{ github.ref_name }}
+        run: |
+          sudo make -C mulga/scripts/iso-builder release \
+            VERSION="${VERSION}" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}/spinifex" \
+            VIPERBLOCK_SRC="${GITHUB_WORKSPACE}/viperblock" \
+            SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
+            INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"
+
+      - name: Wait for verify
+        id: wait-verify
+        if: steps.build-iso.outcome == 'success'
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          echo "Waiting for verify job to complete..."
+          while true; do
+            CONCLUSION=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
+              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            if [ -n "$CONCLUSION" ]; then
+              echo "Verify conclusion: $CONCLUSION"
+              [ "$CONCLUSION" = "success" ] && exit 0
+              exit 1
+            fi
+            echo "Verify still running, retrying in 30s..."
+            sleep 30
+          done
+
+      - name: Ensure AWS CLI is installed
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        run: |
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Upload ISO to Cloudflare R2
+        id: upload
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          VERSION: ${{ github.ref_name }}
+        run: |
+          aws --endpoint-url=https://e80e329f7dbcf75b80e019c1dce640f2.r2.cloudflarestorage.com \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+          aws --endpoint-url=https://e80e329f7dbcf75b80e019c1dce640f2.r2.cloudflarestorage.com \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+
+      - name: Mark ISO failed
+        if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TAG: ${{ github.ref_name }}
+          REPO: ${{ github.repository }}
+        run: |
+          EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
+          NOTES_FILE=$(mktemp)
+          printf 'ISO build/upload failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
+          rm -f "$NOTES_FILE"

.github/workflows/spinifex.yml

@@ -1,4 +1,4 @@
-name: Spinifex Build and Test
+name: Build and Test
 
 on:
   push:
@@ -17,9 +17,12 @@ on:
       - dev
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   build_and_test:
-    name: Spinifex Build and Test
+    name: Build and Test
     runs-on: ubuntu-24.04
 
     steps:
@@ -33,12 +36,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -60,7 +65,7 @@ jobs:
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
   race_detection:
-    name: Spinifex Race Detection
+    name: Race Detection
     runs-on: ubuntu-24.04
 
     steps:
@@ -73,12 +78,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -94,7 +101,7 @@ jobs:
         run: make -C spinifex test-race
 
   lint_and_security:
-    name: Spinifex Lint and Security
+    name: Lint and Security
     runs-on: ubuntu-24.04
 
     steps:
@@ -107,12 +114,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go

.github/workflows/ui.yml

@@ -17,6 +17,9 @@ on:
       - .github/workflows/ui.yml
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   ui_build_and_test:
     name: UI Build and Test
@@ -32,7 +35,7 @@ jobs:
           sparse-checkout: spinifex/services/spinifexui/frontend
 
       - name: Setup pnpm
-        uses: pnpm/action-setup@v5
+        uses: pnpm/action-setup@v6
         with:
           package_json_file: spinifex/services/spinifexui/frontend/package.json
 
@@ -46,8 +49,8 @@ jobs:
       - name: Install Dependencies
         run: pnpm install --frozen-lockfile
 
-      - name: Run Check
-        run: pnpm check
+      - name: Run Lint
+        run: pnpm lint
 
       - name: Run Tests with Coverage
         run: pnpm test:coverage

.gitignore

@@ -7,6 +7,9 @@
 !example.tfvars
 scripts/iac/proxmox/clusters/
 
+# Generated SSH private keys from terraform examples (e.g. local_file.nginx_pem)
+*.pem
+
 # Go workspace
 go.work
 go.work.sum
@@ -14,6 +17,8 @@ go.work.sum
 # Build outputs
 bin/
 dist/
+build/microvm/vmlinuz
+build/microvm/initramfs.cpio.gz
 !spinifex/services/spinifexui/frontend/dist/
 tmp/
 *.test
@@ -28,6 +33,7 @@ Thumbs.db
 
 # Environment
 .env
+.venv/
 
 # Test reports
 *-report.txt

.golangci.yml

@@ -22,6 +22,7 @@ linters:
     - errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
     - exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
     - fatcontext # detects nested contexts in loops
+    - forcetypeassert # flags unchecked type assertions that can panic
     - gocheckcompilerdirectives # validates go compiler directive comments (//go:)
     - goprintffuncname # checks that printf-like functions are named with f at the end
     - gosec # inspects source code for security problems
@@ -29,11 +30,14 @@ linters:
     - ineffassign # detects assignments to existing variables that are not used
     - makezero # finds slice declarations with non-zero initial length followed by append
     - mirror # catches wrong mirror patterns in bytes/strings usage
+    - musttag # checks structs passed to marshal/unmarshal have proper tags
     - modernize # modernizes code to use newer Go idioms (go fix analyzers)
     - nakedret # finds naked returns in functions
     - nilerr # catches code that returns nil even after checking err != nil
     - nilnesserr # catches returning a different nil error variable after err != nil check
     - nolintlint # reports ill-formed or insufficient nolint directives
+    - nosprintfhostport # flags fmt.Sprintf host:port — use net.JoinHostPort for IPv6 safety
+    - perfsprint # detects fmt.Sprintf calls replaceable with faster alternatives
     - predeclared # catches shadowing of Go builtins (len, cap, error, etc.)
     - reassign # checks that package variables are not reassigned
     - recvcheck # checks for receiver type consistency (pointer vs value)
@@ -91,6 +95,12 @@ linters:
         - newexpr # don't replace aws.String() with new() — AWS SDK idiom
         - stringsbuilder # don't replace small string concatenations with strings.Builder
 
+    perfsprint:
+      concat-loop: false
+      err-error: false # don't flag err.Error()
+      errorf: false # don't convert fmt.Errorf to errors.New
+      strconcat: false
+
     nakedret:
       max-func-lines: 0
 
@@ -124,5 +134,3 @@ linters:
         linters: [gosec] # subprocess execution — legitimate in VM orchestration
       - text: "G304"
         linters: [gosec] # file path from variable — necessary for config loading
-      - text: "G402"
-        linters: [gosec] # TLS config — we handle TLS ourselves

Makefile

@@ -34,6 +34,9 @@ else
 endif
 
 # Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
+# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
+# propagates through the filter — otherwise a test failure is swallowed by
+# grep's own (success) exit code and preflight prints "passed" on red.
 ifdef QUIET
   _Q     = @
   _COVQ  = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
@@ -46,8 +49,7 @@ else
   _SECQ  = 2>&1 | tee
 endif
 
-build:
-	$(MAKE) go_build
+build: go_build build-installer build-lb-agent
 
 # Build spinifex-ui frontend (requires pnpm)
 build-ui:
@@ -61,20 +63,41 @@ LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSI
 
 go_build:
 	@echo -e "\n....Building $(GO_PROJECT_NAME)"
-	go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
+	GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
 
-build-alb-agent:
-	@echo -e "\n....Building alb-agent (static)"
-	CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/alb-agent cmd/alb-agent/main.go
+build-installer:
+	@echo -e "\n....Building spinifex-installer"
+	GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
 
-build-system-image: ## Build a system image from manifest (use IMAGE=alb)
+build-lb-agent:
+	@echo -e "\n....Building lb-agent (static)"
+	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
+
+build-system-image: ## Build a system image from manifest (use IMAGE=lb)
 ifndef IMAGE
-	$(error IMAGE is required. Usage: make build-system-image IMAGE=alb)
+	$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
 endif
 	./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
 
-build-alb-image: ## Build ALB Alpine image
-	$(MAKE) build-system-image IMAGE=alb
+MICROVM_OUT_DIR := build/microvm
+MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
+MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
+
+# Grouped target — script writes both files in one run.
+$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
+	./scripts/build-microvm-image.sh
+
+# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
+# build-microvm-image stays correctly stale-aware.
+bin/lb-agent:
+	$(MAKE) build-lb-agent
+
+build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
+
+install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
+	sudo install -d /usr/share/spinifex/microvm
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
 go_run:
 	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
@@ -116,6 +139,26 @@ run:
 	$(MAKE) go_build
 	$(MAKE) go_run
 
+# Fast iteration: build + install binary + restart all services.
+# Microvm artifacts are reinstalled when they already exist on disk — the rule's
+# input timestamps drive a rebuild only if anything actually changed. On a fresh
+# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
+# `make install-microvm` explicitly the first time.
+deploy: build
+	sudo install -m 755 bin/spx /usr/local/bin/spx
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
+		$(MAKE) install-microvm; \
+	else \
+		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
+	fi
+	sudo systemctl daemon-reload
+	sudo systemctl restart spinifex.target
+
+# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
+# Not needed for code-only changes — use deploy for those.
+reinstall:
+	scripts/dev-install.sh
+
 clean:
 	rm -f ./bin/$(GO_PROJECT_NAME)
 	rm -rf spinifex/services/spinifexui/frontend/dist
@@ -132,9 +175,9 @@ install-system:
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
 
 install-go:
-	@echo -e "\n....Installing Go 1.26.1 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
-		curl -L https://go.dev/dl/go1.26.1.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
+		curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
 		echo "Go already installed in /usr/local/go"; \
 	fi
@@ -189,6 +232,14 @@ distro-amd64:
 		-f build/Dockerfile.distro \
 		--output type=local,dest=dist/amd64/ \
 		../
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
+		echo "[distro-amd64] staging microvm artifacts into tarball"; \
+		mkdir -p dist/amd64/microvm; \
+		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
+	else \
+		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
+		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \
+	fi
 	tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
 	sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
 
@@ -207,7 +258,26 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: build build-ui build-alb-agent build-system-image build-alb-image go_build go_run preflight test test-cover test-race diff-coverage bench run clean \
+# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
+# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
+ansible-dev-preflight:
+	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml
+
+ansible-dev-teardown:
+	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml
+
+ansible-dev-install:
+	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml
+
+ansible-dev-reset:
+	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
+
+ansible-dev-deploy:
+	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
+
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run \
+	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
-	distro distro-amd64 distro-arm64 distro-clean
+	distro distro-amd64 distro-arm64 distro-clean \
+	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy

README.md

@@ -1,25 +1,54 @@
+<p align="center">
+  <img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
+</p>
+
+<p align="center">
+  <a href="https://go.dev"><img src="https://img.shields.io/badge/Go-1.26+-00ADD8?style=flat-square&logo=go&logoColor=white" alt="Go"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-3fb950?style=flat-square" alt="License"></a>
+  <a href="https://docs.mulgadc.com"><img src="https://img.shields.io/badge/Docs-docs.mulgadc.com-DC2626?style=flat-square" alt="Docs"></a>
+</p>
+
+<p align="center">
+  <a href="#what-is-spinifex">What is Spinifex?</a> ·
+  <a href="#the-platform">Platform</a> ·
+  <a href="#core-components">Components</a> ·
+  <a href="#architecture-at-a-glance">Architecture</a> ·
+  <a href="#installation">Installation</a> ·
+  <a href="https://docs.mulgadc.com">Docs</a>
+</p>
+
+---
+
 # Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
 
-(Developer Preview) - Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com/) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
+Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
 
 ## What is Spinifex?
 
 Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
 
-It’s designed for developers and operators who need:
+It's designed for developers and operators who need:
 
 - Cloud-like infrastructure without the cloud
 - Minimal dependencies, full control
 - Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
 - A secure cloud environment you control and own the entire hardware, network and software stack
 
-You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren’t feasible.
+You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
+
+## The Platform
+
+<p align="center">
+  <img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
+</p>
+
+From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
 ## Core Components
 
 ### Spinifex (Compute Service – EC2 Alternative)
 
-Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU’s QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
+Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
 
 - EC2-like VM management on bare metal
 - Launches with cloud-init metadata support
@@ -27,7 +56,7 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 
 ### Viperblock (Block Storage – EBS Alternative)
 
-[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It’s built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
+[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
 
 - Fast, durable virtual disks
 - Replication for resilience
@@ -43,29 +72,52 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 - Multipart uploads, streaming reads/writes
 - Data redundancy with Reed-Solomon encoding
 
+## Architecture at a Glance
+
+<p align="center">
+  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
+</p>
+
+Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
+
 ## Key Features
 
 AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
 
-- Designed for Control – Run on your own terms, whether that’s in a datacenter, remote site, or sensitive environment.
+- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
 - Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
 - Works Offline – No reliance on centralized cloud services or external networks.
 - Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
 
 ## Installation
 
-Installation requires an Ubuntu / Debian system. See the detailed documentation at [https://docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
+Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
+
+### Bare Metal ISO
+
+The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
+
+```bash
+curl -fLO https://iso.mulgadc.com/spinifex.iso
+```
+
+Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
+
 
 ### Single Node Install
 
-The installation is straightforward to set up and running on a single node for testing purposes.
+The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
+
+>*Prerequisite:* Linux bridge for networking.
+
+Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
 
 ```bash
-curl https://install.mulgadc.com | bash
+curl -fsSL https://install.mulgadc.com | bash
 
 sudo /usr/local/share/spinifex/setup-ovn.sh --management
 
-sudo spx admin init
+sudo spx admin init --node node1 --nodes 1
 
 sudo systemctl start spinifex.target
 
@@ -76,7 +128,7 @@ aws ec2 describe-instance-types
 
 ### Development Setup
 
-For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/source-install/) documentation
+For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation
 
 ### Component Repositories
 
@@ -87,6 +139,26 @@ Spinifex coordinates these independent components:
 
 Each component can be developed independently. See component-specific documentation for focused development guides.
 
+## Spinifex UI
+
+Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
+
+<p align="center">
+  <img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console — dashboard view" width="900">
+</p>
+
+The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
+
+```bash
+open https://YOUR_NODE_IP:3000
+```
+
+- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
+- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
+- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
+
+For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
+
 ## Development Philosophy
 
 ### Built by Engineers, For Engineers
@@ -122,4 +194,4 @@ This hybrid approach ensures Spinifex benefits from both proven engineering expe
 
 ## License
 
-Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
+Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.

build/Dockerfile.distro

@@ -30,18 +30,19 @@ COPY viperblock/ ./viperblock/
 
 # Build spinifex binary (spx)
 RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
-    cd spinifex && go build -ldflags "-s -w \
+    cd spinifex && GOFIPS140=v1.0.0 go build -ldflags "-s -w \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
     -o /out/spx cmd/spinifex/main.go
 
 # Build nbdkit viperblock plugin (.so)
-RUN cd viperblock && go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
+RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
 
 # Copy setup scripts and systemd units
 RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
 RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
 RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
+RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
 RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex
 
 FROM scratch AS export

build/microvm/init.sh

@@ -0,0 +1,56 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
+# already enumerated by the time PID 1 runs — no modprobe needed.
+# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
+modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
+modprobe virtio_net 2>/dev/null || true
+
+for i in 1 2 3 4 5; do
+    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
+    sleep 0.02
+done
+
+if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+    echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    exit 1
+fi
+
+. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+
+for n in 0 1 2 3 4 5; do
+    eval mac=\$NIC${n}_MAC
+    [ -z "$mac" ] && continue
+
+    iface=""
+    for attempt in 1 2 3 4 5; do
+        iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+        [ -n "$iface" ] && break
+        sleep 0.02
+    done
+
+    if [ -z "$iface" ]; then
+        echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
+        continue
+    fi
+
+    eval cidr=\$NIC${n}_CIDR
+    eval gw=\$NIC${n}_GW
+    eval default=\$NIC${n}_DEFAULT
+    eval rdst=\$NIC${n}_ROUTE_DST
+    eval rvia=\$NIC${n}_ROUTE_VIA
+
+    ip link set "$iface" up
+    [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+    [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+    [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+done
+
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/lb-agent-env/raw /etc/conf.d/lb-agent
+mkdir -p /etc/ssl/certs
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw /etc/ssl/certs/ca-certificates.crt
+
+exec /sbin/init

build/microvm/inittab

@@ -0,0 +1,4 @@
+::sysinit:/etc/rc.local
+::respawn:/bin/sh -c 'set -a; . /etc/conf.d/lb-agent; set +a; exec /usr/local/bin/lb-agent'
+::ctrlaltdel:/sbin/reboot
+::shutdown:/bin/umount -a -r

build/scripts/predastore-start.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+# Helper script for spinifex-predastore.service
+# Detects host, port, backend, and node-id from spinifex.toml
+# then execs spx. Falls back to single-node defaults if no config found.
+
+CONF=/etc/spinifex/spinifex.toml
+BIND="0.0.0.0:8443"
+# -1 = co-located mode (all [[db]] peers in one process). Single-node spinifex
+# templates list 3 DB peers on 127.0.0.1; only -1 can serve that topology.
+# Multi-node spinifex.toml emits node_id = N (N >= 1) which overrides below.
+# predastore rejects NODE_ID=0 since d3a02f0 (strconv.Atoi-of-garbage guard).
+NODE_ID="-1"
+
+if [ -f "$CONF" ]; then
+    H=$(awk -F'"' '/\[nodes\..*\.predastore\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
+    if [ -n "$H" ]; then
+        BIND="$H"
+    fi
+
+    N=$(awk -F'= *' '/node_id/{gsub(/ /,"",$2);print $2;exit}' "$CONF")
+    if [ -n "$N" ]; then
+        NODE_ID="$N"
+    fi
+fi
+
+export SPINIFEX_PREDASTORE_HOST="${BIND%%:*}"
+export SPINIFEX_PREDASTORE_PORT="${BIND##*:}"
+export SPINIFEX_PREDASTORE_NODE_ID="$NODE_ID"
+
+exec /usr/local/bin/spx service predastore start

build/scripts/wait-for-nats.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+# Wait for NATS to be reachable before starting a dependent service.
+# Reads the NATS address from spinifex.toml [nodes.*.nats] section,
+# falls back to nats.conf, then to 127.0.0.1:4222.
+
+CONF=/etc/spinifex/spinifex.toml
+NATS_CONF=/etc/spinifex/nats/nats.conf
+NATS_HOST=""
+NATS_PORT="4222"
+TIMEOUT=30
+
+# Try spinifex.toml first
+if [ -f "$CONF" ]; then
+    NATS_BIND=$(awk -F'"' '/\[nodes\..*\.nats\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
+    if [ -n "$NATS_BIND" ]; then
+        NATS_HOST="${NATS_BIND%%:*}"
+        NATS_PORT="${NATS_BIND##*:}"
+    fi
+fi
+
+# Fall back to nats.conf
+if [ -z "$NATS_HOST" ] && [ -f "$NATS_CONF" ]; then
+    NATS_HOST=$(grep "^listen:" "$NATS_CONF" | sed 's/listen: *//;s/:.*//')
+fi
+
+# Final fallback
+NATS_HOST="${NATS_HOST:-127.0.0.1}"
+
+for i in $(seq 1 "$TIMEOUT"); do
+    if nc -z "$NATS_HOST" "$NATS_PORT" 2>/dev/null; then
+        exit 0
+    fi
+    sleep 1
+done
+
+echo "Timed out waiting for NATS at $NATS_HOST:$NATS_PORT" >&2
+exit 1

build/systemd/regenerate-ssh-host-keys.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Regenerate SSH host keys
+ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key
+Before=ssh.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/ssh-keygen -A
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

build/systemd/spinifex-awsgw.service

@@ -2,21 +2,53 @@
 Description=Spinifex AWS Gateway
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-daemon.service
-Requires=spinifex-nats.service
+Wants=spinifex-nats.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-gw
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service awsgw start
 Restart=on-failure
 RestartSec=5
+
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
-Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/awsgw/
 Environment=SPINIFEX_AWSGW_TLS_CERT=/etc/spinifex/server.pem
 Environment=SPINIFEX_AWSGW_TLS_KEY=/etc/spinifex/server.key
 
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/awsgw /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-daemon.service

@@ -1,21 +1,98 @@
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-viperblock.service
-Requires=spinifex-nats.service
+After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service
+# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
+# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
+# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
+# cold boot.
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-daemon
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
+# standalone mode and reconnects to NATS in the background — process-exiting
+# on a missing NATS would kill the local VM management plane.
+#
+# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
+# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
+# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
+# where silent NATS misconfig is more confusing than helpful. Do NOT set this
+# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
+# DDIL Tier 1 (mulga-siv-58): KillMode=process so systemd only signals the
+# daemon's main PID, not the cgroup. QEMU children stay alive across
+# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
+# via the persisted local state file + pidfile + QMP socket. With the
+# default control-group mode, restart would terminate every guest VM —
+# defeating Scenario B's "daemon restart preserves running instances".
+KillMode=process
+# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
+# skips VM teardown (it now just persists state and exits), so 15s is
+# plenty of slack for the cluster-manager + NATS unsubscribe path.
+TimeoutStopSec=15
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
+# DDIL Tier 1 (mulga-siv-58): QEMU pidfiles, QMP sockets, and the
+# shared qemu.log all resolve via utils.pidPath() — which falls back
+# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
+# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
+# pidfiles written by QEMU (running in the OLD daemon's namespace)
+# are invisible to the new daemon. The /local/instances handler then
+# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
+# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
+# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
+# stable, namespace-independent path that both daemon incarnations and
+# QEMU children resolve identically.
+Environment=XDG_RUNTIME_DIR=/run/spinifex
+
+# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
+# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
+# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
+# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
+ProtectSystem=full
+PrivateDevices=no
+DevicePolicy=closed
+DeviceAllow=/dev/kvm rw
+DeviceAllow=/dev/net/tun rw
+DeviceAllow=char-vfio rw
+DeviceAllow=/dev/vfio/vfio rw
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
+# the daemon no longer needs AF_PACKET or the softened kernel protections.
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+# VFIO GPU passthrough requires locking all guest RAM into physical memory.
+# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+LimitMEMLOCK=infinity
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-nats.service

@@ -5,7 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start
 Restart=on-failure
@@ -13,5 +13,36 @@ RestartSec=5
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/nats/nats.conf
 
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/nats /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/nats /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-predastore.service

@@ -2,14 +2,14 @@
 Description=Spinifex Predastore (S3-compatible Object Storage)
 PartOf=spinifex.target
 After=spinifex-nats.service
-Requires=spinifex-nats.service
+Wants=spinifex-nats.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-storage
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
-ExecStart=/usr/local/bin/spx service predastore start
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
+ExecStart=/var/lib/spinifex/predastore-start.sh
 Restart=on-failure
 RestartSec=5
 OOMScoreAdjust=-500
@@ -17,6 +17,38 @@ Environment=SPINIFEX_PREDASTORE_CONFIG_PATH=/etc/spinifex/predastore/predastore.
 Environment=SPINIFEX_PREDASTORE_BASE_PATH=/var/lib/spinifex/predastore/
 Environment=SPINIFEX_PREDASTORE_TLS_CERT=/etc/spinifex/server.pem
 Environment=SPINIFEX_PREDASTORE_TLS_KEY=/etc/spinifex/server.key
+Environment=SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE=/etc/spinifex/predastore/encryption.key
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/predastore /etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/predastore /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-ui.service

@@ -5,12 +5,45 @@ After=spinifex-awsgw.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start
 Restart=on-failure
 RestartSec=5
 OOMScoreAdjust=-500
+Environment=SPINIFEX_UI_TLS_CERT=/etc/spinifex/server.pem
+Environment=SPINIFEX_UI_TLS_KEY=/etc/spinifex/server.key
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/server.pem /etc/spinifex/server.key
+ReadWritePaths=/var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-viperblock.service

@@ -2,13 +2,13 @@
 Description=Spinifex Viperblock (Block Storage)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-predastore.service
-Requires=spinifex-nats.service
+Wants=spinifex-nats.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-viperblock
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service viperblock start
 Restart=on-failure
 RestartSec=5
@@ -17,5 +17,36 @@ EnvironmentFile=/etc/spinifex/systemd.env
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/viperblock/
 
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/systemd.env /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/viperblock /var/log/spinifex /run/spinifex/nbd
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-vpcd.service

@@ -1,19 +1,59 @@
 [Unit]
 Description=Spinifex VPC Daemon (OVN Networking)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-daemon.service
-Requires=spinifex-nats.service
+After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-vpcd
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service vpcd start
 Restart=on-failure
 RestartSec=5
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
+
+# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
+ProtectSystem=full
+PrivateDevices=no
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=no
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=no
+# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
+# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
+# interface. See docs/development/bugs/dhcp-allocate-error.md.
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
+# CAP_NET_RAW: open AF_PACKET sockets.
+# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
+# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
+# Note, disabled for now, sudo issue with ovs
+AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+NoNewPrivileges=no
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex.target

@@ -2,7 +2,7 @@
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service openvswitch-switch.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

build/udev/99-spinifex-vfio.rules

@@ -0,0 +1,5 @@
+# Allow spinifex group to access VFIO devices for GPU passthrough.
+# QEMU (launched by spinifex-daemon, group spinifex) must open /dev/vfio/N
+# to use vfio-pci passthrough devices. The kernel creates these nodes as
+# root:root 0600 by default; this rule grants group-level read-write access.
+SUBSYSTEM=="vfio", GROUP="spinifex", MODE="0660"

cmd/alb-agent/main.go

@@ -1,62 +0,0 @@
-// alb-agent runs inside ALB VMs and manages HAProxy configuration.
-//
-// It connects to NATS, subscribes to config updates for its load balancer,
-// writes HAProxy configs to disk, and reloads HAProxy on changes. It also
-// responds to health pings so the daemon can verify the agent is alive.
-//
-// Usage:
-//
-//	alb-agent --lb-id=lb-xxxxx --nats=nats://10.0.0.1:4222
-//
-// The LB ID defaults to the hostname (set by cloud-init to the LB ID).
-// The NATS URL defaults to nats://127.0.0.1:4222.
-package main
-
-import (
-	"flag"
-	"log/slog"
-	"os"
-	"os/signal"
-	"syscall"
-
-	"github.com/mulgadc/spinifex/spinifex/albagent"
-)
-
-func main() {
-	var (
-		lbID    string
-		natsURL string
-	)
-
-	hostname, _ := os.Hostname()
-
-	flag.StringVar(&lbID, "lb-id", hostname, "Load balancer ID (defaults to hostname)")
-	flag.StringVar(&natsURL, "nats", "nats://127.0.0.1:4222", "NATS server URL")
-	flag.Parse()
-
-	if lbID == "" {
-		slog.Error("--lb-id is required (or set hostname to the LB ID)")
-		os.Exit(1)
-	}
-
-	slog.Info("Starting ALB agent", "lbId", lbID, "nats", natsURL)
-
-	agent, err := albagent.New(lbID, natsURL)
-	if err != nil {
-		slog.Error("Failed to create agent", "err", err)
-		os.Exit(1)
-	}
-
-	if err := agent.Start(); err != nil {
-		slog.Error("Failed to start agent", "err", err)
-		os.Exit(1)
-	}
-
-	// Wait for shutdown signal
-	sigCh := make(chan os.Signal, 1)
-	signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
-	sig := <-sigCh
-
-	slog.Info("Received signal, shutting down", "signal", sig)
-	agent.Stop()
-}

cmd/dhcptest/main.go

@@ -0,0 +1,172 @@
+// dhcptest is a diagnostic tool that runs a DHCP DORA handshake using the
+// same nclient4 library and options that spinifex-vpcd uses for gateway pool
+// leases. Run it on a node to verify DHCP works at the library level before
+// debugging the full spinifex stack.
+//
+// Usage (must run as root — needs AF_PACKET):
+//
+//	sudo dhcptest --iface=br-wan
+//	sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
+//	sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+
+	"github.com/insomniacslk/dhcp/dhcpv4"
+	"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
+)
+
+func main() {
+	iface := flag.String("iface", "br-wan", "Interface/bridge to bind the DHCP client to")
+	macStr := flag.String("mac", "", "Hardware address for DORA (chaddr + AF_PACKET filter). If empty, uses the real interface MAC.")
+	spinifexID := flag.String("spinifex-id", "", "Generate a spinifex virtual MAC from this resource ID (e.g. 'gateway-wan1'). Overrides --mac.")
+	clientID := flag.String("client-id", "", "DHCP option 61 client identifier. Defaults to --spinifex-id or --mac value.")
+	hostname := flag.String("hostname", "", "DHCP option 12 hostname")
+	vendorClass := flag.String("vendor-class", "mulga-spinifex-gw", "DHCP option 60 vendor class identifier")
+	promisc := flag.Bool("promisc", false, "Enable promiscuous mode on the interface before the test (mirrors vpcd's setBridgePromisc)")
+	timeout := flag.Duration("timeout", 10*time.Second, "Per-attempt DHCP timeout")
+	retries := flag.Int("retries", 3, "Number of DORA attempts before giving up")
+	verbose := flag.Bool("verbose", false, "Print full DHCP packet contents")
+	flag.Parse()
+
+	// Resolve the hardware address to use.
+	var hwAddr net.HardwareAddr
+	var err error
+
+	switch {
+	case *spinifexID != "":
+		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
+		if err != nil {
+			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+		}
+		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
+	case *macStr != "":
+		hwAddr, err = net.ParseMAC(*macStr)
+		if err != nil {
+			log.Fatalf("parse --mac %q: %v", *macStr, err)
+		}
+	default:
+		hwAddr, err = ifaceMAC(*iface)
+		if err != nil {
+			log.Fatalf("get MAC for %s: %v", *iface, err)
+		}
+		fmt.Printf("using real interface MAC: %s\n", hwAddr)
+	}
+
+	// Resolve client-id: default to spinifex-id, then MAC string.
+	cid := *clientID
+	if cid == "" {
+		if *spinifexID != "" {
+			cid = *spinifexID
+		} else {
+			cid = hwAddr.String()
+		}
+	}
+
+	fmt.Printf("iface:        %s\n", *iface)
+	fmt.Printf("hw-addr:      %s\n", hwAddr)
+	fmt.Printf("client-id:    %s\n", cid)
+	fmt.Printf("hostname:     %s\n", *hostname)
+	fmt.Printf("vendor-class: %s\n", *vendorClass)
+	fmt.Printf("promisc:      %v\n", *promisc)
+	fmt.Printf("timeout:      %s  retries: %d\n", *timeout, *retries)
+	fmt.Println()
+
+	if *promisc {
+		if err := setPromisc(*iface, true); err != nil {
+			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+		} else {
+			fmt.Printf("set %s promisc ON\n\n", *iface)
+		}
+	}
+
+	// Build nclient4 options — identical to spinifex/services/vpcd/dhcp/nclient4.go.
+	opts := []nclient4.ClientOpt{
+		nclient4.WithHWAddr(hwAddr),
+		nclient4.WithTimeout(*timeout),
+		nclient4.WithRetry(*retries),
+	}
+	if *verbose {
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+	}
+
+	client, err := nclient4.New(*iface, opts...)
+	if err != nil {
+		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+	}
+	defer client.Close()
+
+	// Identity modifiers — identical to identityModifiers() in vpcd dhcp package.
+	var mods []dhcpv4.Modifier
+	if cid != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
+	}
+	if *hostname != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
+	}
+	if *vendorClass != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
+	}
+
+	fmt.Println("Sending DHCP Discover...")
+	start := time.Now()
+	ctx, cancel := context.WithTimeout(context.Background(), *timeout*time.Duration(*retries+1))
+	defer cancel()
+
+	lease, err := client.Request(ctx, mods...)
+	if err != nil {
+		fmt.Printf("\nFAILED after %s: %v\n", time.Since(start).Round(time.Millisecond), err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("SUCCESS in %s\n\n", time.Since(start).Round(time.Millisecond))
+	ack := lease.ACK
+	fmt.Printf("  Your IP:     %s\n", ack.YourIPAddr)
+	fmt.Printf("  Subnet:      %s\n", ack.SubnetMask())
+	fmt.Printf("  Gateway:     %v\n", ack.Router())
+	fmt.Printf("  DNS:         %v\n", ack.DNS())
+	fmt.Printf("  Server ID:   %s\n", ack.ServerIdentifier())
+	fmt.Printf("  Lease time:  %s\n", ack.IPAddressLeaseTime(24*time.Hour))
+	if *verbose {
+		fmt.Printf("\n--- Full ACK ---\n%s\n", ack.Summary())
+	}
+}
+
+// generateMAC is the same deterministic hash used by vpcd's topology.go.
+// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
+func generateMAC(resourceID string) string {
+	h := uint32(0)
+	for _, c := range resourceID {
+		h = h*31 + uint32(c)
+	}
+	return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
+}
+
+func ifaceMAC(name string) (net.HardwareAddr, error) {
+	ifc, err := net.InterfaceByName(name)
+	if err != nil {
+		return nil, err
+	}
+	return ifc.HardwareAddr, nil
+}
+
+func setPromisc(iface string, on bool) error {
+	state := "on"
+	if !on {
+		state = "off"
+	}
+	out, err := exec.Command("ip", "link", "set", iface, "promisc", state).CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("%w: %s", err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}

cmd/installer/autoinstall/autoinstall.go

@@ -0,0 +1,355 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package autoinstall reads headless install parameters from environment
+// variables exported by spinifex-init.sh from the kernel cmdline.
+// Parameters are set in the GRUB "Headless" menu entry in grub.cfg.
+package autoinstall
+
+import (
+	"fmt"
+	"log/slog"
+	"net"
+	"os"
+	"os/exec"
+	"sort"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/mulgadc/spinifex/cmd/installer/install"
+)
+
+// Load returns an install.Config built from SPINIFEX_* environment variables,
+// or nil if SPINIFEX_AUTO is not set to "1" (interactive mode).
+func Load() (*install.Config, error) {
+	if os.Getenv("SPINIFEX_AUTO") != "1" {
+		return nil, nil
+	}
+
+	slog.Info("autoinstall: headless mode enabled via kernel cmdline")
+
+	cfg, err := buildConfig()
+	if err != nil {
+		return nil, fmt.Errorf("autoinstall config: %w", err)
+	}
+	return cfg, nil
+}
+
+// EjectAndReboot ejects the boot device (best-effort) then reboots.
+func EjectAndReboot() {
+	srcDev, _ := os.ReadFile("/run/iso-dev")
+	if dev := strings.TrimSpace(string(srcDev)); dev != "" {
+		slog.Info("autoinstall: ejecting boot device", "device", dev)
+		_ = exec.Command("eject", dev).Run()
+	}
+
+	fmt.Println()
+	fmt.Println("Installation complete.")
+	fmt.Println("Remove the USB drive now if it was not ejected automatically.")
+	fmt.Println("Rebooting in 10 seconds...")
+	time.Sleep(10 * time.Second)
+	// Use the kernel syscall directly — the live environment runs spinifex-init
+	// as PID 1 (not systemd), so exec("reboot") fails trying to reach D-Bus.
+	if err := syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART); err != nil {
+		slog.Error("autoinstall: reboot syscall failed", "err", err)
+	}
+}
+
+func buildConfig() (*install.Config, error) {
+	password := os.Getenv("SPINIFEX_PASSWORD")
+	if password == "" {
+		return nil, fmt.Errorf("SPINIFEX_PASSWORD is required")
+	}
+
+	hostname := os.Getenv("SPINIFEX_HOSTNAME")
+	if hostname == "" {
+		hostname = "spinifex-node"
+	}
+
+	disk, err := resolveDisk(os.Getenv("SPINIFEX_DISK"))
+	if err != nil {
+		return nil, fmt.Errorf("disk: %w", err)
+	}
+
+	wanIface, err := resolveNIC(os.Getenv("SPINIFEX_WAN_IFACE"), "")
+	if err != nil {
+		return nil, fmt.Errorf("WAN NIC: %w", err)
+	}
+
+	cfg := &install.Config{
+		Disk:         disk,
+		Hostname:     hostname,
+		RootPassword: password,
+		// Optional on the headless path — empty means no --email passed to
+		// spx admin init, which omits the operator identity from telemetry.
+		Email:        strings.TrimSpace(os.Getenv("SPINIFEX_EMAIL")),
+		WANInterface: wanIface,
+		WANDHCPMode:  strings.ToLower(os.Getenv("SPINIFEX_WAN_MODE")) != "static",
+	}
+
+	if !cfg.WANDHCPMode {
+		ip := os.Getenv("SPINIFEX_WAN_IP")
+		mask := os.Getenv("SPINIFEX_WAN_MASK")
+		gw := os.Getenv("SPINIFEX_WAN_GW")
+		if ip == "" || mask == "" || gw == "" {
+			return nil, fmt.Errorf("SPINIFEX_WAN_IP, SPINIFEX_WAN_MASK, SPINIFEX_WAN_GW required for static mode")
+		}
+		cfg.WANAddress = ip
+		cfg.WANMask = mask
+		cfg.WANGateway = gw
+		if dns := os.Getenv("SPINIFEX_WAN_DNS"); dns != "" {
+			cfg.WANDNS = strings.Split(dns, ",")
+		}
+	}
+
+	if lanIface := os.Getenv("SPINIFEX_LAN_IFACE"); lanIface != "" {
+		lan, err := resolveNIC(lanIface, wanIface)
+		if err != nil {
+			return nil, fmt.Errorf("LAN NIC: %w", err)
+		}
+		cfg.LANInterface = lan
+		cfg.LANDHCPMode = strings.ToLower(os.Getenv("SPINIFEX_LAN_MODE")) != "static"
+		if !cfg.LANDHCPMode {
+			cfg.LANAddress = os.Getenv("SPINIFEX_LAN_IP")
+			cfg.LANMask = os.Getenv("SPINIFEX_LAN_MASK")
+			if dns := os.Getenv("SPINIFEX_LAN_DNS"); dns != "" {
+				cfg.LANDNS = strings.Split(dns, ",")
+			}
+		}
+	}
+
+	if os.Getenv("SPINIFEX_GPU_PASSTHROUGH") == "1" {
+		cfg.GPUPassthrough = true
+	}
+
+	role := strings.ToLower(os.Getenv("SPINIFEX_ROLE"))
+	if role == "" {
+		role = "init"
+	}
+	cfg.ClusterRole = role
+	skipFormation := os.Getenv("SPINIFEX_SKIP_FORMATION") == "1"
+	if skipFormation {
+		cfg.SkipFormation = true
+	}
+
+	if role == "join" && !skipFormation {
+		joinAddr := os.Getenv("SPINIFEX_JOIN_ADDR")
+		if joinAddr == "" {
+			return nil, fmt.Errorf("SPINIFEX_JOIN_ADDR required when SPINIFEX_ROLE=join")
+		}
+		cfg.JoinAddr = joinAddr
+	}
+
+	return cfg, nil
+}
+
+// resolveDisk maps the spx_disk value from grub.cfg to a block device path.
+//
+// Supported values:
+//
+//	auto            — use the only non-removable disk; fail if multiple found
+//	largest         — largest non-removable disk (explicit opt-in)
+//	smallest        — smallest non-removable disk (typical OS-on-SSD pattern)
+//	nvme            — the only NVMe disk; fail if multiple found
+//	/dev/sda (etc.) — exact device path; fail if not present
+func resolveDisk(target string) (string, error) {
+	switch strings.ToLower(target) {
+	case "", "auto":
+		return singleDisk()
+	case "largest":
+		return diskBySize(true)
+	case "smallest":
+		return diskBySize(false)
+	case "nvme":
+		return nvmeDisk()
+	default:
+		if _, err := os.Stat(target); err != nil {
+			return "", fmt.Errorf("%q not found", target)
+		}
+		return target, nil
+	}
+}
+
+type diskCandidate struct {
+	dev   string
+	bytes int64
+}
+
+// nonRemovableDisks returns all non-removable, non-virtual block devices.
+func nonRemovableDisks() ([]diskCandidate, error) {
+	entries, err := os.ReadDir("/sys/block")
+	if err != nil {
+		return nil, err
+	}
+	var out []diskCandidate
+	for _, e := range entries {
+		dev := e.Name()
+		switch {
+		case strings.HasPrefix(dev, "loop"),
+			strings.HasPrefix(dev, "ram"),
+			strings.HasPrefix(dev, "dm-"),
+			strings.HasPrefix(dev, "sr"):
+			continue
+		}
+		removable, _ := os.ReadFile("/sys/block/" + dev + "/removable")
+		if strings.TrimSpace(string(removable)) == "1" {
+			continue
+		}
+		sizeRaw, _ := os.ReadFile("/sys/block/" + dev + "/size")
+		sectors, parseErr := strconv.ParseInt(strings.TrimSpace(string(sizeRaw)), 10, 64)
+		if parseErr != nil || sectors == 0 {
+			continue
+		}
+		out = append(out, diskCandidate{dev: dev, bytes: sectors * 512})
+	}
+	return out, nil
+}
+
+// diskList returns a human-readable list of candidates for error messages.
+func diskList(disks []diskCandidate) string {
+	lines := make([]string, len(disks))
+	for i, d := range disks {
+		lines[i] = fmt.Sprintf("  /dev/%s (%dG)", d.dev, d.bytes>>30)
+	}
+	return strings.Join(lines, "\n")
+}
+
+// singleDisk selects the only non-removable disk, or fails listing all found.
+func singleDisk() (string, error) {
+	disks, err := nonRemovableDisks()
+	if err != nil {
+		return "", err
+	}
+	switch len(disks) {
+	case 0:
+		return "", fmt.Errorf("no non-removable disks found")
+	case 1:
+		slog.Info("autoinstall: single disk selected", "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
+		return "/dev/" + disks[0].dev, nil
+	default:
+		return "", fmt.Errorf(
+			"multiple disks found — set SPINIFEX_DISK to one of:\n%s\n"+
+				"  largest   (largest disk)\n"+
+				"  smallest  (smallest disk)\n"+
+				"  nvme      (NVMe only)\n"+
+				"  /dev/sdX  (exact path)",
+			diskList(disks),
+		)
+	}
+}
+
+// diskBySize selects the largest (largest=true) or smallest non-removable disk.
+func diskBySize(largest bool) (string, error) {
+	disks, err := nonRemovableDisks()
+	if err != nil {
+		return "", err
+	}
+	if len(disks) == 0 {
+		return "", fmt.Errorf("no non-removable disks found")
+	}
+	sort.Slice(disks, func(i, j int) bool {
+		if largest {
+			return disks[i].bytes > disks[j].bytes
+		}
+		return disks[i].bytes < disks[j].bytes
+	})
+	label := "smallest"
+	if largest {
+		label = "largest"
+	}
+	slog.Info("autoinstall: disk selected by size", "mode", label, "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
+	return "/dev/" + disks[0].dev, nil
+}
+
+// nvmeDisk selects the only NVMe disk, or fails listing all NVMe disks found.
+func nvmeDisk() (string, error) {
+	disks, err := nonRemovableDisks()
+	if err != nil {
+		return "", err
+	}
+	var nvmes []diskCandidate
+	for _, d := range disks {
+		if strings.HasPrefix(d.dev, "nvme") {
+			nvmes = append(nvmes, d)
+		}
+	}
+	switch len(nvmes) {
+	case 0:
+		return "", fmt.Errorf("no NVMe disks found")
+	case 1:
+		slog.Info("autoinstall: NVMe disk selected", "disk", nvmes[0].dev, "size_gb", nvmes[0].bytes>>30)
+		return "/dev/" + nvmes[0].dev, nil
+	default:
+		return "", fmt.Errorf(
+			"multiple NVMe disks found — set SPINIFEX_DISK to one of:\n%s",
+			diskList(nvmes),
+		)
+	}
+}
+
+// virtualNICPrefixes lists interface name prefixes that identify non-physical
+// interfaces (bridges, tunnels, container/OVS veth pairs, etc.). These are
+// skipped when auto-selecting a NIC so we don't configure docker0 or
+// ovs-system as the management interface on a machine that previously ran
+// Docker or OVN.
+var virtualNICPrefixes = []string{
+	"docker", "veth", "virbr", "br-", "ovs-", "vxlan", "genev", "tun", "tap",
+}
+
+// resolveNIC returns the interface name to use. "auto" or empty picks the
+// first physical (non-loopback, non-virtual) interface that is not exclude.
+func resolveNIC(name, exclude string) (string, error) {
+	if name != "" && name != "auto" {
+		return name, nil
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return "", err
+	}
+	for _, iface := range ifaces {
+		if iface.Flags&net.FlagLoopback != 0 {
+			continue
+		}
+		// Require broadcast capability — filters out point-to-point tunnels.
+		if iface.Flags&net.FlagBroadcast == 0 {
+			continue
+		}
+		// Require a MAC address — virtual/tunnel interfaces have none.
+		if len(iface.HardwareAddr) == 0 {
+			continue
+		}
+		if iface.Name == exclude {
+			continue
+		}
+		// Skip known virtual interface prefixes.
+		virtual := false
+		for _, pfx := range virtualNICPrefixes {
+			if strings.HasPrefix(iface.Name, pfx) {
+				virtual = true
+				break
+			}
+		}
+		if virtual {
+			continue
+		}
+		slog.Info("autoinstall: NIC resolved", "mode", "auto", "selected", iface.Name)
+		return iface.Name, nil
+	}
+	return "", fmt.Errorf("no suitable physical NIC found (non-loopback, broadcast-capable, with MAC)")
+}

cmd/installer/autoinstall/autoinstall_test.go

@@ -0,0 +1,17 @@
+package autoinstall
+
+import (
+	"os"
+	"testing"
+)
+
+func TestLoadReturnsNilWhenNotAuto(t *testing.T) {
+	os.Unsetenv("SPINIFEX_AUTO")
+	cfg, err := Load()
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if cfg != nil {
+		t.Fatal("expected nil config when SPINIFEX_AUTO is not set")
+	}
+}

cmd/installer/branding/branding.go

@@ -0,0 +1,56 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package branding centralises Mulga/Spinifex visual identity constants used
+// by the installer TUI.
+//
+// TODO: Replace placeholder values with official Mulga/Spinifex assets once
+// the brand guide and logo files are located.
+package branding
+
+import "github.com/charmbracelet/lipgloss"
+
+// Logo is the ASCII art rendered on the installer welcome screen.
+// TODO: Replace with official Spinifex/Mulga ASCII logo.
+const Logo = `
+  ███████╗██████╗ ██╗███╗   ██╗██╗███████╗███████╗██╗  ██╗
+  ██╔════╝██╔══██╗██║████╗  ██║██║██╔════╝██╔════╝╚██╗██╔╝
+  ███████╗██████╔╝██║██╔██╗ ██║██║█████╗  █████╗   ╚███╔╝
+  ╚════██║██╔═══╝ ██║██║╚██╗██║██║██╔══╝  ██╔══╝   ██╔██╗
+  ███████║██║     ██║██║ ╚████║██║██║     ███████╗██╔╝ ██╗
+  ╚══════╝╚═╝     ╚═╝╚═╝  ╚═══╝╚═╝╚═╝     ╚══════╝╚═╝  ╚═╝`
+
+// Subtitle is displayed beneath the logo on the welcome screen.
+const Subtitle = "Bare-Metal Node Provisioner"
+
+// Publisher is shown in the welcome screen footer.
+const Publisher = "Mulga Defense Corporation"
+
+// Mulga color palette.
+// TODO: Replace with official hex values from the Mulga brand guide.
+const (
+	ColorPrimary    = lipgloss.Color("#00A3E0") // Mulga blue (placeholder)
+	ColorAccent     = lipgloss.Color("#FF6B00") // Mulga orange (placeholder)
+	ColorBackground = lipgloss.Color("#0D1117") // Dark background
+	ColorSurface    = lipgloss.Color("#161B22") // Slightly lighter surface
+	ColorText       = lipgloss.Color("#E6EDF3") // Light text
+	ColorMuted      = lipgloss.Color("#8B949E") // Muted / secondary text
+	ColorSuccess    = lipgloss.Color("#3FB950") // Green
+	ColorWarning    = lipgloss.Color("#D29922") // Amber
+	ColorError      = lipgloss.Color("#F85149") // Red
+	ColorBorder     = lipgloss.Color("#30363D") // Border
+)

cmd/installer/firstboot/firstboot.go

@@ -0,0 +1,266 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package firstboot writes the oneshot systemd service and configuration that
+// completes Spinifex provisioning on the first real boot after installation.
+package firstboot
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/mulgadc/spinifex/cmd/installer/systemd"
+)
+
+// Config holds the values the firstboot service needs to configure the node.
+type Config struct {
+	Hostname string
+	// EncapIP is the Geneve tunnel IP for OVN. Set to the LAN bridge IP when a
+	// dedicated LAN NIC is present, otherwise the WAN bridge IP. Empty when DHCP
+	// is used — setup-ovn.sh auto-detects the IP from the default route in that case.
+	EncapIP string
+	// ClusterRole is "init" or "join".
+	ClusterRole string
+	// JoinAddr is host:port of the primary node, only used when ClusterRole is "join".
+	JoinAddr string
+	// Email is the operator email collected by the TUI or SPINIFEX_EMAIL on
+	// the headless path. Passed to `spx admin init --email=<value>` when set;
+	// omitted entirely when empty.
+	Email string
+	// GPUPassthrough enables VFIO GPU passthrough by passing --gpu-passthrough
+	// to `spx admin init`, which writes gpu_passthrough = true in the daemon config.
+	GPUPassthrough bool
+	// InstallCallback, when non-empty, is curled once at the end of firstboot
+	// after the success marker is written. Generic phone-home hook for
+	// provisioning controllers (PXE/MAAS-style flows). Best-effort: non-2xx
+	// and network failures are swallowed so the install is not gated on
+	// controller reachability.
+	InstallCallback string
+	// SkipFormation skips spx admin init/join. Used when a provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation and will call
+	// spx admin init/join itself with the correct multi-node parameters.
+	SkipFormation bool
+}
+
+// Write drops the firstboot script and systemd unit into root, which should be
+// the path of the installed system's root filesystem (e.g. /mnt/spinifex-install).
+func Write(root string, cfg Config) error {
+	if err := writeScript(root, cfg); err != nil {
+		return fmt.Errorf("firstboot script: %w", err)
+	}
+	if err := systemd.WriteFirstbootUnit(root); err != nil {
+		return fmt.Errorf("firstboot unit: %w", err)
+	}
+	if err := systemd.EnableUnit(root, "spinifex-firstboot.service"); err != nil {
+		return err
+	}
+	// Banner unit is written but NOT enabled here. firstboot enables it
+	// (along with spinifex.target) at the end of its run, so on the first
+	// boot the only spinifex unit running is firstboot itself — no implicit
+	// pull of spinifex.target via the banner's Wants= relationship, which
+	// was the root cause of the boot-time race against firstboot.
+	if err := systemd.WriteBannerUnit(root); err != nil {
+		return fmt.Errorf("banner unit: %w", err)
+	}
+	return systemd.WriteGettyDropIn(root)
+}
+
+func writeScript(root string, cfg Config) error {
+	clusterCmd := buildClusterCmd(cfg)
+
+	// --encap-ip is optional: when DHCP is used the IP is unknown at install time
+	// and setup-ovn.sh auto-detects it from the default route at boot.
+	setupOVN := "/usr/local/bin/setup-ovn.sh --management"
+	if cfg.EncapIP != "" {
+		setupOVN += fmt.Sprintf(" --encap-ip=%s", cfg.EncapIP)
+	}
+
+	callbackBlock := ""
+	if cfg.InstallCallback != "" {
+		callbackBlock = fmt.Sprintf(
+			"\ncurl -fsS --max-time 10 --retry 2 --retry-delay 2 -o /dev/null %s || true\n",
+			shellEscapeSingle(cfg.InstallCallback),
+		)
+	}
+
+	script := fmt.Sprintf(`#!/bin/bash
+# Spinifex firstboot — runs once after ISO installation. On success, writes
+# /var/lib/spinifex/.firstboot-done and the systemd unit's
+# ConditionPathExists=! prevents re-execution on subsequent boots. A partial
+# run leaves no marker, so the next reboot retries from the top — safe
+# because every step below is idempotent (hostnamectl, setup-ovn.sh, spx
+# admin init are all "set if not set"; systemctl enable on an already-enabled
+# unit is a no-op).
+set -euo pipefail
+
+DONE_MARKER=/var/lib/spinifex/.firstboot-done
+
+# Idempotency: bail early if a previous run completed successfully. The unit
+# also has ConditionPathExists=!$DONE_MARKER so we shouldn't get here on
+# subsequent boots, but defend in depth in case the unit is re-triggered
+# manually before the operator notices the marker exists.
+if [ -f "$DONE_MARKER" ]; then
+    echo "[firstboot] already complete — skipping"
+    exit 0
+fi
+
+# Set hostname
+hostnamectl set-hostname %s
+
+# Pre-start OVS and OVN central so their databases are initialised before
+# setup-ovn.sh runs. On physical hardware, first-boot DB initialisation takes
+# longer than setup-ovn.sh's internal 15-second timeout allows. Starting them
+# here and waiting until the NB DB is ready means setup-ovn.sh sees a live DB
+# the moment it starts — no races, no timeout failures.
+systemctl start openvswitch-switch
+systemctl start ovn-central
+echo "Waiting for OVN NB DB to initialise..."
+for _i in $(seq 1 120); do
+    if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
+        echo "OVN NB DB ready (${_i}s)"
+        break
+    fi
+    sleep 1
+done
+
+# Create default nameservers if DHCP server returns blank
+printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolvconf/resolv.conf.d/base
+resolvconf -u
+
+# Configure OVN networking.
+# br-wan (and br-lan if present) are Linux bridges managed by systemd-networkd
+# (declared in /etc/systemd/network/ by the installer). setup-ovn.sh
+# auto-detects br-wan as the default route device and wires it to OVS via a
+# veth pair — non-destructive, SSH-safe.
+%s
+
+# Write the banner (first time, spinifex-banner service will do this on reboot)
+/usr/local/bin/spx admin banner --boot-check
+
+# Cluster formation — capture credentials to file for display on console.
+%s 2>&1
+
+# Fix ownership of files spx admin init wrote. spx runs as root under
+# systemd, so /etc/spinifex/{spinifex.toml,master.key,ca.key,*.pem} and any
+# per-service files written under /var/lib/spinifex/* land as root:root.
+# Delegate to setup.sh's fix_file_ownership (single source of truth) — it
+# knows the per-file modes (ca.key 0600, *.pem 0644, toml/key 0640) that a
+# blunt recursive chmod would clobber.
+SETUP_STAGES=fixown /usr/local/share/spinifex/setup.sh
+# The /var/lib/spinifex/awsgw/config symlink was created at build time by
+# setup.sh's create_directories — it resolves to /etc/spinifex, so
+# {BaseDir}/config/master.key automatically points at /etc/spinifex/master.key.
+
+# Bootstrap the spinifex user's SSH directory with a generated key pair so
+# ~/.ssh exists and the operator can immediately use ssh-keygen / ssh-copy-id
+# without hitting "No such file or directory" errors documented in the setup guide.
+if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
+    mkdir -p /home/spinifex/.ssh
+    chmod 700 /home/spinifex/.ssh
+    ssh-keygen -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$(hostname)"
+    chown -R spinifex:spinifex /home/spinifex/.ssh
+    chmod 600 /home/spinifex/.ssh/id_ed25519
+    chmod 644 /home/spinifex/.ssh/id_ed25519.pub
+fi
+
+# Copy AWS credentials to the spinifex user's home directory.
+# spx admin init runs with HOME=/root (set by the systemd unit), so credentials
+# land in /root/.aws/. Copy them to the spinifex user's home so the operator
+# can use the AWS CLI without sudo.
+if [ -f /root/.aws/credentials ]; then
+    mkdir -p /home/spinifex/.aws
+    cp /root/.aws/credentials /home/spinifex/.aws/credentials
+    cp /root/.aws/config /home/spinifex/.aws/config 2>/dev/null || true
+    chown -R spinifex:spinifex /home/spinifex/.aws
+    chmod 700 /home/spinifex/.aws
+    chmod 600 /home/spinifex/.aws/credentials
+    [ -f /home/spinifex/.aws/config ] && chmod 600 /home/spinifex/.aws/config
+fi
+
+# Enable + activate spinifex.target and the banner now that all configs are
+# in place. "enable --now" creates the multi-user.target.wants/ symlinks (so
+# they start directly on every subsequent boot — no longer dependent on
+# firstboot running each time, since firstboot is condition-skipped after
+# this run via the marker) and activates them in this boot.
+# NOTE: Moved to firstinstall to enable
+#systemctl enable --now spinifex.target spinifex-banner.service
+
+# Enable services to start, on reboot
+systemctl enable spinifex.target spinifex-banner.service
+
+# Mark complete only after every step above has succeeded. Until this point,
+# any failure (set -e) leaves the marker absent and the next reboot retries
+# firstboot from the top.
+mkdir -p "$(dirname "$DONE_MARKER")"
+touch "$DONE_MARKER"
+%s
+systemctl start spinifex.target
+
+# Wait for the daemon to bring up external networking. When external_mode=pool
+# is configured the daemon creates br-ext (the OVN external bridge) during
+# startup; launching instances before it is ready results in no public IP being
+# assigned. ip link requires no root and avoids parsing daemon logs.
+if grep -q 'external_mode.*pool' /etc/spinifex/spinifex.toml 2>/dev/null; then
+    echo "[firstboot] waiting for external networking (br-ext)..."
+    for _i in $(seq 1 30); do
+        if ip link show br-ext >/dev/null 2>&1; then
+            echo "[firstboot] br-ext ready (${_i}s)"
+            break
+        fi
+        sleep 1
+    done
+    if ! ip link show br-ext >/dev/null 2>&1; then
+        echo "[firstboot] warning: br-ext not up after 30s — external networking may be delayed"
+    fi
+fi
+`, cfg.Hostname, setupOVN, clusterCmd, callbackBlock)
+
+	path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
+	return os.WriteFile(path, []byte(script), 0o755)
+}
+
+func buildClusterCmd(cfg Config) string {
+	if cfg.SkipFormation {
+		return `echo "[firstboot] cluster formation skipped — provisioning controller will handle it"`
+	}
+	emailFlag := ""
+	if cfg.Email != "" {
+		// shellEscapeSingle keeps the email safe if it ever contains a
+		// character shell treats specially — belt-and-braces; the regex
+		// validator already rejects whitespace and @-chains.
+		emailFlag = " --email=" + shellEscapeSingle(cfg.Email)
+	}
+	gpuFlag := ""
+	if cfg.GPUPassthrough {
+		gpuFlag = " --gpu-passthrough"
+	}
+	switch cfg.ClusterRole {
+	case "join":
+		return fmt.Sprintf("spx admin join --node %s --host %s%s", cfg.Hostname, cfg.JoinAddr, emailFlag)
+	default:
+		return fmt.Sprintf("spx admin init --node %s --nodes 1%s%s", cfg.Hostname, emailFlag, gpuFlag)
+	}
+}
+
+// shellEscapeSingle wraps s in single quotes with any embedded single
+// quotes escaped. Minimal — we only need this because the email value is
+// interpolated into a shell script written by Write().
+func shellEscapeSingle(s string) string {
+	return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
+}

cmd/installer/firstboot/firstboot_test.go

@@ -0,0 +1,100 @@
+package firstboot
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func makeRootDirs(t *testing.T, root string) {
+	t.Helper()
+	for _, d := range []string{
+		"usr/local/bin",
+		"etc/systemd/system",
+		"etc/systemd/system/multi-user.target.wants",
+	} {
+		if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil {
+			t.Fatalf("mkdir %s: %v", d, err)
+		}
+	}
+}
+
+func TestWriteScriptNoCallbackWhenEmpty(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	cfg := Config{Hostname: "test-node", ClusterRole: "init"}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+
+	script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	if strings.Contains(string(script), "curl") {
+		t.Error("script should not contain curl when InstallCallback is empty")
+	}
+}
+
+func TestWriteScriptEmbedsCurlWhenCallbackSet(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
+	cfg := Config{
+		Hostname:        "test-node",
+		ClusterRole:     "init",
+		InstallCallback: callbackURL,
+	}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+
+	script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	content := string(script)
+
+	if !strings.Contains(content, "curl") {
+		t.Error("script missing curl command")
+	}
+	if !strings.Contains(content, callbackURL) {
+		t.Errorf("script missing callback URL %q", callbackURL)
+	}
+}
+
+func TestWriteScriptCallbackAfterDoneMarker(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
+	cfg := Config{
+		Hostname:        "node1",
+		ClusterRole:     "init",
+		InstallCallback: callbackURL,
+	}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+
+	script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	content := string(script)
+
+	doneIdx := strings.Index(content, "touch \"$DONE_MARKER\"")
+	curlIdx := strings.Index(content, "curl")
+	if doneIdx < 0 {
+		t.Fatal("done marker not found in script")
+	}
+	if curlIdx < 0 {
+		t.Fatal("curl not found in script")
+	}
+	if curlIdx < doneIdx {
+		t.Error("curl must appear after done marker write")
+	}
+}

cmd/installer/install/config.go

@@ -0,0 +1,79 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package install performs the disk installation steps after the UI has
+// collected configuration.
+package install
+
+// Config holds all values collected by the installer UI.
+type Config struct {
+	// Disk is the block device path to install onto (e.g. /dev/sda).
+	Disk string
+
+	// WAN interface — always present. The management IP lives on br-wan (a Linux
+	// bridge over the NIC), not on the physical NIC itself. This allows OVN/OVS
+	// to safely attach to the bridge without disrupting host connectivity.
+	WANInterface string
+	WANDHCPMode  bool
+	WANAddress   string   // empty if WANDHCPMode
+	WANMask      string   // empty if WANDHCPMode
+	WANGateway   string   // empty if WANDHCPMode
+	WANDNS       []string // empty if WANDHCPMode; e.g. ["1.1.1.1", "8.8.8.8"]
+	WANWiFiSSID  string   // non-empty if a WiFi NIC was selected
+	WANWiFiPass  string   // non-empty if a WiFi NIC was selected
+
+	// LAN interface — only set when 2+ NICs are present. The Geneve encap IP
+	// for OVN tunnels lives on br-lan. When empty, the WAN address is used.
+	LANInterface string
+	LANDHCPMode  bool
+	LANAddress   string
+	LANMask      string
+	LANDNS       []string
+	LANWiFiSSID  string
+	LANWiFiPass  string
+
+	// Node identity
+	Hostname string
+
+	// ClusterRole is "init" or "join".
+	ClusterRole string
+
+	// JoinAddr is the primary node address (host:port) when ClusterRole is "join".
+	JoinAddr string
+
+	// SkipFormation skips spx admin init/join in firstboot; the provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation instead.
+	SkipFormation bool
+
+	// CA certificate (PEM), optional.
+	HasCACert bool
+	CACert    string
+
+	// RootPassword is the password to set for the root account on the installed system.
+	RootPassword string
+
+	// Email is the operator's email address, used by the call-home telemetry
+	// endpoint to notify of important system updates and security advisories.
+	// Required on interactive installs; may be empty on headless/CI installs
+	// when SPINIFEX_EMAIL is not supplied on the kernel cmdline.
+	Email string
+
+	// GPUPassthrough enables VFIO GPU passthrough on this node.
+	// Set via SPINIFEX_GPU_PASSTHROUGH=1 on headless installs.
+	// Passes --gpu-passthrough to `spx admin init` during firstboot.
+	GPUPassthrough bool
+}

cmd/installer/install/install.go

@@ -0,0 +1,863 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+package install
+
+import (
+	"bufio"
+	"fmt"
+	"io"
+	"log/slog"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/mulgadc/spinifex/cmd/installer/firstboot"
+	"github.com/mulgadc/spinifex/cmd/installer/systemd"
+)
+
+const (
+	mountRoot = "/mnt/spinifex-install"
+	efiPart   = mountRoot + "/boot/efi"
+)
+
+// Run executes all installation steps in order. It is intentionally sequential
+// and explicit — each step is visible in logs so failures are easy to diagnose.
+func Run(cfg *Config) error {
+	// The live environment may not have /sbin or /usr/sbin in PATH. Set it
+	// explicitly so exec.Command's LookPath finds system binaries like grub-install.
+	_ = os.Setenv("PATH", "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin")
+
+	// Unmount unconditionally on exit so a failed step never leaves partitions
+	// mounted in the live environment, which would cause a retry to double-mount.
+	defer func() {
+		_ = run("umount", efiPart)
+		_ = run("umount", mountRoot)
+	}()
+
+	steps := []struct {
+		name string
+		fn   func() error
+	}{
+		{"partition disk", func() error { return partitionDisk(cfg.Disk) }},
+		{"format partitions", func() error { return formatPartitions(cfg.Disk) }},
+		{"mount partitions", func() error { return mountPartitions(cfg.Disk) }},
+		{"copy rootfs", copyRootfs},
+		{"write fstab", func() error { return writeFstab(cfg.Disk) }},
+		{"install spinifex", func() error { return installSpinifex(cfg) }},
+		{"write network config", func() error { return writeNetworkConfig(cfg) }},
+		{"write firstboot service", func() error { return firstboot.Write(mountRoot, cfg.toFirstbootConfig()) }},
+		{"install bootloader", func() error { return installBootloader(cfg.Disk) }},
+		{"install CA cert", func() error { return installCACert(cfg) }},
+	}
+
+	for _, s := range steps {
+		slog.Info("installer", "step", s.name)
+		if err := s.fn(); err != nil {
+			return fmt.Errorf("step %q: %w", s.name, err)
+		}
+	}
+
+	slog.Info("installation complete")
+	fireInstallCallback()
+	promptRemoveUSB()
+	return reboot()
+}
+
+func partitionDisk(disk string) error {
+	// GPT table with three partitions:
+	//   p1: 1MiB BIOS Boot Partition — required for grub-install i386-pc on GPT
+	//   p2: 512MiB EFI System Partition
+	//   p3: remainder as root (ext4)
+	if err := run("parted", "--script", disk,
+		"mklabel", "gpt",
+		"mkpart", "bios_boot", "1MiB", "2MiB",
+		"set", "1", "bios_grub", "on",
+		"mkpart", "ESP", "fat32", "2MiB", "514MiB",
+		"set", "2", "esp", "on",
+		"mkpart", "root", "ext4", "514MiB", "100%",
+	); err != nil {
+		return err
+	}
+	// Force the kernel to re-read the partition table and wait for udev to
+	// create the partition device nodes. Without this, mkfs.fat in the next
+	// step may race and fail with "No such file or directory" on /dev/sda2 —
+	// the kernel has accepted the new layout but /dev hasn't been populated
+	// yet. Trixie's udev seems slower at this than Bookworm's was.
+	return waitForPartitions(disk)
+}
+
+// waitForPartitions ensures the EFI and root partition device nodes exist
+// after parted creates them. It runs partprobe (kernel re-read) and
+// udevadm settle (wait for queued events), then polls /dev for the files.
+func waitForPartitions(disk string) error {
+	// Best-effort: partprobe failure isn't fatal — udev may still pick up
+	// the change from the BLKRRPART ioctl that parted itself issued.
+	if err := run("partprobe", disk); err != nil {
+		slog.Warn("partprobe failed, continuing", "disk", disk, "err", err)
+	}
+	if err := run("udevadm", "settle", "--timeout=10"); err != nil {
+		slog.Warn("udevadm settle failed, continuing", "err", err)
+	}
+	efi, root := partitionPaths(disk)
+	deadline := time.Now().Add(15 * time.Second)
+	for _, part := range []string{efi, root} {
+		for {
+			if _, err := os.Stat(part); err == nil {
+				break
+			}
+			if time.Now().After(deadline) {
+				return fmt.Errorf("partition device %s did not appear within timeout — kernel/udev did not pick up new partition table", part)
+			}
+			time.Sleep(200 * time.Millisecond)
+		}
+	}
+	return nil
+}
+
+func formatPartitions(disk string) error {
+	efi, root := partitionPaths(disk)
+	if err := run("mkfs.fat", "-F32", efi); err != nil {
+		return err
+	}
+	return run("mkfs.ext4", "-F", root)
+}
+
+func mountPartitions(disk string) error {
+	_, root := partitionPaths(disk)
+	if err := os.MkdirAll(mountRoot, 0o755); err != nil {
+		return err
+	}
+	if err := run("mount", root, mountRoot); err != nil {
+		return err
+	}
+	if err := os.MkdirAll(efiPart, 0o755); err != nil {
+		return err
+	}
+	efi, _ := partitionPaths(disk)
+	return run("mount", efi, efiPart)
+}
+
+// copyRootfs copies the live squashfs environment onto the target disk using
+// rsync. This is the air-gapped alternative to debootstrap — all packages are
+// already embedded in the ISO so no network access is required.
+func copyRootfs() error {
+	if err := run("rsync", "-aHAX", "--delete", "--info=progress2",
+		"--exclude=/proc",
+		"--exclude=/sys",
+		"--exclude=/dev",
+		"--exclude=/run",
+		"--exclude=/tmp",
+		"--exclude=/cdrom",
+		"--exclude=/mnt",
+		"--exclude=/etc/machine-id",
+		"--exclude=/var/lib/dbus/",
+		"--exclude=/etc/openvswitch/",
+		"--exclude=/var/lib/openvswitch/",
+		"--exclude=/var/lib/dhcpcd/",
+		"--exclude=/etc/ssh/ssh_host_*",
+		"--exclude=/lost+found",
+		"--exclude=/boot/efi",
+		"/", mountRoot+"/",
+	); err != nil {
+		return err
+	}
+
+	// Verify critical paths exist before proceeding. rsync exits 0 on ENOSPC
+	// for individual file writes on some filesystems, which would produce a
+	// partial rootfs that boots into a panic.
+	critical := []string{
+		filepath.Join(mountRoot, "bin/bash"),
+		filepath.Join(mountRoot, "lib/systemd/systemd"),
+		filepath.Join(mountRoot, "usr/local/bin/spx"),
+	}
+	for _, p := range critical {
+		if _, err := os.Stat(p); err != nil {
+			return fmt.Errorf("copyRootfs: critical path missing after rsync (%s): %w", p, err)
+		}
+	}
+
+	// rsync skips excluded paths entirely — recreate the virtual filesystem
+	// mount points that systemd expects to exist on the installed system.
+	mountPoints := []struct {
+		path string
+		mode os.FileMode
+	}{
+		{"proc", 0o555},
+		{"sys", 0o555},
+		{"dev", 0o755},
+		{"run", 0o755},
+		{"run/lock", 0o1777},
+		{"tmp", 0o1777},
+		{"mnt", 0o755},
+	}
+	for _, mp := range mountPoints {
+		if err := os.MkdirAll(filepath.Join(mountRoot, mp.path), mp.mode); err != nil {
+			return fmt.Errorf("create mountpoint /%s: %w", mp.path, err)
+		}
+	}
+	return nil
+}
+
+func installSpinifex(cfg *Config) error {
+	// The rootfs copy already contains spx and spinifex-installer at
+	// /usr/local/bin/ — no binary copy needed. Regenerate machine-specific
+	// identity files so each installed node is unique.
+
+	// Bind-mount /dev, /proc, /sys into the chroot so PAM (chpasswd) and
+	// other chroot commands work correctly. Trixie's PAM requires /proc and
+	// /dev/urandom for password hashing.
+	if err := bindChrootMounts(); err != nil {
+		return err
+	}
+	defer unbindChrootMounts()
+
+	// Generate a unique machine-id from the kernel CSPRNG. We do not use
+	// systemd-machine-id-setup in the chroot: it reads the SMBIOS UUID via
+	// the bind-mounted /sys, which is identical on cloned VMs or hosts with
+	// absent/zeroed DMI data, and falls back to non-unique sources (MAC,
+	// hostname) when that also fails — producing the same ID on every node.
+	// /proc/sys/kernel/random/uuid is always unique and requires no chroot.
+	machineIDPath := filepath.Join(mountRoot, "etc/machine-id")
+	rawUUID, err := os.ReadFile("/proc/sys/kernel/random/uuid")
+	if err != nil {
+		return fmt.Errorf("installSpinifex: read kernel uuid: %w", err)
+	}
+	machineID := strings.ReplaceAll(strings.TrimSpace(string(rawUUID)), "-", "") + "\n"
+	if err := os.WriteFile(machineIDPath, []byte(machineID), 0o444); err != nil {
+		return fmt.Errorf("installSpinifex: write machine-id: %w", err)
+	}
+	// dbus mirrors /etc/machine-id; remove it so it is re-created from the
+	// new ID on first boot rather than carrying a stale value.
+	_ = os.Remove(filepath.Join(mountRoot, "var/lib/dbus/machine-id"))
+
+	// Hostname.
+	hostnamePath := filepath.Join(mountRoot, "etc/hostname")
+	if err := os.WriteFile(hostnamePath, []byte(cfg.Hostname+"\n"), 0o644); err != nil {
+		return err
+	}
+
+	// /etc/hosts entry for the hostname.
+	hosts := fmt.Sprintf("127.0.0.1\tlocalhost\n127.0.1.1\t%s\n", cfg.Hostname)
+	if err := os.WriteFile(filepath.Join(mountRoot, "etc/hosts"), []byte(hosts), 0o644); err != nil {
+		return err
+	}
+
+	// Set root + spinifex passwords. We invoke chpasswd from the LIVE
+	// installer (not via `chroot`), passing the target root with -R and
+	// forcing -c YESCRYPT. This deliberately bypasses PAM:
+	//   * `chroot ... chpasswd` uses /etc/pam.d/chpasswd → common-password →
+	//     pam_unix.so with the "obscure" option, which can return
+	//     "Authentication token manipulation error" inside a chroot for
+	//     reasons that are awkward to diagnose (audit subsystem, locked
+	//     shadow entries from useradd, etc.).
+	//   * -c YESCRYPT tells chpasswd to hash locally with libcrypt and write
+	//     directly to <root>/etc/shadow — no PAM stack involved. YESCRYPT
+	//     matches Trixie's ENCRYPT_METHOD so subsequent logins use the same
+	//     algorithm.
+	//   * -R <root> opens the target's passwd/shadow directly; no bind
+	//     mounts of /dev/urandom or /proc are needed for the password step.
+	if cfg.RootPassword != "" {
+		if err := setShadowPassword("root", cfg.RootPassword); err != nil {
+			return fmt.Errorf("set root password: %w", err)
+		}
+		// The spinifex account is the default interactive login on the
+		// node (console + SSH). Root SSH is disabled, so this is the sole
+		// remote entry point. The user itself is created at rootfs build
+		// time (build-rootfs.sh) — here we just set its password.
+		if err := setShadowPassword("spinifex", cfg.RootPassword); err != nil {
+			return fmt.Errorf("set spinifex password: %w", err)
+		}
+	}
+
+	// Write /etc/spinifex/node.conf — read at runtime by spx admin banner
+	// to look up the current IP dynamically (handles IP changes after install).
+	// MANAGEMENT_IFACE is the bridge (br-wan), not the physical NIC.
+	// MANAGEMENT_IP is empty for DHCP — banner's --boot-check fills it in at boot.
+	nodeConf := fmt.Sprintf("MANAGEMENT_IP=%s\nMANAGEMENT_IFACE=br-wan\nNODE_HOSTNAME=%s\n",
+		cfg.WANAddress, cfg.Hostname)
+	confDir := filepath.Join(mountRoot, "etc/spinifex")
+	if err := os.MkdirAll(confDir, 0o755); err != nil {
+		return err
+	}
+	if err := os.WriteFile(filepath.Join(confDir, "node.conf"), []byte(nodeConf), 0o644); err != nil {
+		return err
+	}
+
+	// dhcpcd-base is present on the installed system (used by setup-ovn.sh for
+	// macvlan mode). Mask the standalone dhcpcd.service so it never auto-starts
+	// and races with systemd-networkd's built-in DHCP client on br-wan.
+	if err := maskSystemdUnit(mountRoot, "dhcpcd.service"); err != nil {
+		slog.Warn("installSpinifex: failed to mask dhcpcd.service", "err", err)
+	}
+
+	return nil
+}
+
+func writeNetworkConfig(cfg *Config) error {
+	// IPs live on Linux bridges (br-wan, br-lan), not on the physical NICs.
+	// systemd-networkd owns the full lifecycle: it creates the bridge NetDevs,
+	// enslaves the physical NICs, and runs the DHCP client. This means the
+	// veth pair that setup-ovn.sh adds later (veth-wan-br) is a port of a
+	// networkd-known bridge, so a networkctl reload or reboot never orphans it.
+	netdDir := filepath.Join(mountRoot, "etc/systemd/network")
+	if err := os.MkdirAll(netdDir, 0o755); err != nil {
+		return err
+	}
+
+	if err := writeNetworkdBridge(netdDir, cfg.WANInterface, "br-wan", false,
+		cfg.WANDHCPMode, cfg.WANAddress, cfg.WANMask, cfg.WANGateway, cfg.WANDNS,
+		cfg.WANWiFiSSID, cfg.WANWiFiPass); err != nil {
+		return err
+	}
+
+	if cfg.LANInterface != "" {
+		if err := writeNetworkdBridge(netdDir, cfg.LANInterface, "br-lan", true,
+			cfg.LANDHCPMode, cfg.LANAddress, cfg.LANMask, "", cfg.LANDNS,
+			cfg.LANWiFiSSID, cfg.LANWiFiPass); err != nil {
+			return err
+		}
+		// br-lan timing is controlled by spinifex-lan-bridge.service, not
+		// networkd auto-activation — ActivationPolicy=manual in the .network
+		// file ensures networkd creates the bridge but does not bring it up.
+		if err := systemd.WriteLANBridgeUnit(mountRoot); err != nil {
+			return fmt.Errorf("lan bridge unit: %w", err)
+		}
+	}
+
+	// Unmask systemd-networkd-wait-online on the installed system and scope
+	// it to br-wan only. The live ISO masks this service (build-rootfs.sh) to
+	// avoid blocking the installer environment (which has no br-wan). The mask
+	// is copied to the installed system by copyRootfs and must be removed here
+	// so that network-online.target — and therefore spinifex-firstboot.service
+	// (After=network-online.target) — does not fire before br-wan has its
+	// DHCP lease and default route. Without this, setup-ovn.sh finds no default
+	// route and exits early, creating no veth pair.
+	//
+	// The --interface=br-wan scope means br-lan (ActivationPolicy=manual) never
+	// blocks the wait; --timeout=60 caps the delay on a cold switch.
+	waitOnlineMask := filepath.Join(mountRoot, "etc/systemd/system/systemd-networkd-wait-online.service")
+	_ = os.Remove(waitOnlineMask)
+	waitOnlineDir := filepath.Join(mountRoot, "etc/systemd/system/systemd-networkd-wait-online.service.d")
+	if err := os.MkdirAll(waitOnlineDir, 0o755); err != nil {
+		return err
+	}
+	waitOnlineConf := "[Service]\nExecStart=\nExecStart=/lib/systemd/systemd-networkd-wait-online --interface=br-wan --timeout=60\n"
+	if err := os.WriteFile(filepath.Join(waitOnlineDir, "spinifex-wan-only.conf"), []byte(waitOnlineConf), 0o644); err != nil {
+		return err
+	}
+
+	// Disable IPv6 via sysctl — belt-and-suspenders alongside IPv6AcceptRA=no
+	// in the networkd .network files.
+	bridges := []string{"br-wan"}
+	if cfg.LANInterface != "" {
+		bridges = append(bridges, "br-lan")
+	}
+	var sysctl strings.Builder
+	sysctl.WriteString("# Generated by Spinifex installer — IPv6 disabled on management bridges\n")
+	for _, br := range bridges {
+		fmt.Fprintf(&sysctl, "net.ipv6.conf.%s.disable_ipv6=1\n", br)
+	}
+	sysctlDir := filepath.Join(mountRoot, "etc/sysctl.d")
+	if err := os.MkdirAll(sysctlDir, 0o755); err != nil {
+		return err
+	}
+	if err := os.WriteFile(filepath.Join(sysctlDir, "99-spinifex-network.conf"), []byte(sysctl.String()), 0o644); err != nil {
+		return err
+	}
+
+	// Pin each NIC name to its MAC address via udev so the installed system
+	// always uses the same interface name regardless of probe order.
+	udevDir := filepath.Join(mountRoot, "etc/udev/rules.d")
+	if err := os.MkdirAll(udevDir, 0o755); err != nil {
+		return err
+	}
+	var udevRules strings.Builder
+	for _, iface := range []string{cfg.WANInterface, cfg.LANInterface} {
+		if iface == "" {
+			continue
+		}
+		mac, err := os.ReadFile("/sys/class/net/" + iface + "/address")
+		if err != nil {
+			slog.Warn("writeNetworkConfig: could not read NIC MAC, skipping udev pin", "iface", iface, "err", err)
+			continue
+		}
+		fmt.Fprintf(&udevRules, "SUBSYSTEM==\"net\", ACTION==\"add\", ATTR{address}==\"%s\", NAME=\"%s\"\n",
+			strings.TrimSpace(string(mac)), iface)
+	}
+	if udevRules.Len() > 0 {
+		return os.WriteFile(filepath.Join(udevDir, "70-spinifex-net.rules"), []byte(udevRules.String()), 0o644)
+	}
+	return nil
+}
+
+// writeNetworkdBridge writes the three systemd-networkd files that configure a
+// physical NIC enslaved to a Linux bridge:
+//
+//   - {prio}-spinifex-{suffix}-nic.network  enslaves the NIC to the bridge
+//   - {prio+1}-spinifex-{suffix}.netdev     declares the bridge device
+//   - {prio+1}-spinifex-{suffix}.network    configures IP on the bridge
+//
+// manual=true sets ActivationPolicy=manual so networkd creates the bridge but
+// does not auto-activate it — used for br-lan, which spinifex-lan-bridge.service
+// activates after network-online.target.
+func writeNetworkdBridge(dir, nicIface, bridgeName string, manual, dhcp bool,
+	addr, mask, gw string, dns []string, wifiSSID, wifiPass string) error {
+	suffix := strings.TrimPrefix(bridgeName, "br-")
+	nicPrio, brPrio := 10, 11
+	if manual {
+		nicPrio, brPrio = 12, 13
+	}
+
+	// NIC .network — enslaves the physical NIC to the bridge.
+	nicNet := fmt.Sprintf("[Match]\nName=%s\n\n[Network]\nBridge=%s\n", nicIface, bridgeName)
+	nicFile := fmt.Sprintf("%02d-spinifex-%s-nic.network", nicPrio, suffix)
+	if err := os.WriteFile(filepath.Join(dir, nicFile), []byte(nicNet), 0o644); err != nil {
+		return err
+	}
+
+	// Bridge .netdev — declares the bridge device.
+	brNetdev := fmt.Sprintf("[NetDev]\nName=%s\nKind=bridge\n\n[Bridge]\nSTP=no\n", bridgeName)
+	brNetdevFile := fmt.Sprintf("%02d-spinifex-%s.netdev", brPrio, suffix)
+	if err := os.WriteFile(filepath.Join(dir, brNetdevFile), []byte(brNetdev), 0o644); err != nil {
+		return err
+	}
+
+	// Bridge .network — IP configuration.
+	var b strings.Builder
+	fmt.Fprintf(&b, "[Match]\nName=%s\n\n", bridgeName)
+	if manual {
+		b.WriteString("[Link]\nActivationPolicy=manual\nRequiredForOnline=no\n\n")
+	}
+	b.WriteString("[Network]\n")
+	if dhcp {
+		b.WriteString("DHCP=ipv4\n")
+	} else {
+		cidr, err := addrCIDR(addr, mask)
+		if err != nil {
+			return fmt.Errorf("bridge %s: %w", bridgeName, err)
+		}
+		fmt.Fprintf(&b, "Address=%s\n", cidr)
+		if gw != "" {
+			fmt.Fprintf(&b, "Gateway=%s\n", gw)
+		}
+		for _, ns := range dns {
+			if ns = strings.TrimSpace(ns); ns != "" {
+				fmt.Fprintf(&b, "DNS=%s\n", ns)
+			}
+		}
+	}
+	b.WriteString("IPv6AcceptRA=no\nConfigureWithoutCarrier=yes\n")
+	if manual && dhcp {
+		// br-lan is non-critical; fail fast on DHCP so a missing LAN cable
+		// does not stall spinifex-lan-bridge.service indefinitely.
+		b.WriteString("\n[DHCP]\nTimeout=10\n")
+	}
+	brNetFile := fmt.Sprintf("%02d-spinifex-%s.network", brPrio, suffix)
+	if err := os.WriteFile(filepath.Join(dir, brNetFile), []byte(b.String()), 0o644); err != nil {
+		return err
+	}
+
+	if wifiSSID != "" {
+		if err := writeWPASupplicant(nicIface, wifiSSID, wifiPass); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// addrCIDR converts a dotted-decimal address + subnet mask to CIDR notation.
+func addrCIDR(addr, dotMask string) (string, error) {
+	if net.ParseIP(addr) == nil {
+		return "", fmt.Errorf("invalid IP: %s", addr)
+	}
+	maskIP := net.ParseIP(dotMask)
+	if maskIP == nil {
+		return "", fmt.Errorf("invalid mask: %s", dotMask)
+	}
+	m := net.IPMask(maskIP.To4())
+	ones, bits := m.Size()
+	if bits == 0 {
+		return "", fmt.Errorf("non-contiguous or zero subnet mask: %s", dotMask)
+	}
+	return fmt.Sprintf("%s/%d", addr, ones), nil
+}
+
+// writeWPASupplicant writes a wpa_supplicant config for nicIface and enables
+// the per-interface wpa_supplicant@{iface}.service so authentication completes
+// before networkd enslaves the NIC to the bridge.
+func writeWPASupplicant(nicIface, ssid, psk string) error {
+	conf := fmt.Sprintf(
+		"ctrl_interface=DIR=/run/wpa_supplicant GROUP=netdev\nupdate_config=1\n\nnetwork={\n\tssid=%q\n\tpsk=%q\n}\n",
+		ssid, psk,
+	)
+	wpaDir := filepath.Join(mountRoot, "etc/wpa_supplicant")
+	if err := os.MkdirAll(wpaDir, 0o755); err != nil {
+		return err
+	}
+	if err := os.WriteFile(filepath.Join(wpaDir, "wpa_supplicant-"+nicIface+".conf"), []byte(conf), 0o600); err != nil {
+		return err
+	}
+	// Enable via symlink into multi-user.target.wants pointing at the
+	// package-provided template unit.
+	wantsDir := filepath.Join(mountRoot, "etc/systemd/system/multi-user.target.wants")
+	if err := os.MkdirAll(wantsDir, 0o755); err != nil {
+		return err
+	}
+	link := filepath.Join(wantsDir, "wpa_supplicant@"+nicIface+".service")
+	_ = os.Remove(link)
+	return os.Symlink("/lib/systemd/system/wpa_supplicant@.service", link)
+}
+
+func installBootloader(disk string) error {
+	// grub-install runs in the live environment (not chroot) using the
+	// grub-pc-bin and grub-efi-amd64-bin packages already present on the ISO.
+	// --boot-directory points at the installed system's /boot.
+	bootDir := filepath.Join(mountRoot, "boot")
+	efiDir := filepath.Join(mountRoot, "boot", "efi")
+
+	efiErr := run("grub-install",
+		"--target=x86_64-efi",
+		"--efi-directory="+efiDir,
+		"--boot-directory="+bootDir,
+		"--bootloader-id=spinifex",
+		"--removable",
+		"--recheck",
+	)
+	if efiErr != nil {
+		slog.Warn("installBootloader: EFI install failed", "err", efiErr)
+	}
+	if biosErr := run("grub-install",
+		"--target=i386-pc",
+		"--boot-directory="+bootDir,
+		"--recheck",
+		disk,
+	); biosErr != nil {
+		if efiErr != nil {
+			// Both targets failed — the system will not boot.
+			return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
+		}
+		return biosErr
+	}
+	// Copy splash image and unicode font from the ISO (mounted at /cdrom) so the
+	// installed GRUB shows the same branded background as the installer GRUB.
+	// The font must be at /boot/grub/fonts/unicode.pf2 so update-grub finds it
+	// there and emits the same loadfont path as the ISO's grub.cfg — GRUB 2.12
+	// (trixie) needs the font in the boot partition, not just /usr/share/grub/.
+	copySplashImage(mountRoot)
+	copyGrubFont(mountRoot)
+
+	// Both consoles listed: tty0 stays primary for local install, ttyS0 mirrors
+	// kernel output to serial so headless installs (CI, racks with serial-only
+	// IPMI, remote-dev SSH-to-QEMU) see boot messages. The last `console=` on
+	// the cmdline becomes the system console, so ttyS0 wins on serial-only
+	// hardware while tty0 still receives output for local display.
+	grubDefault := `GRUB_DEFAULT=0
+GRUB_TIMEOUT=5
+GRUB_DISTRIBUTOR=Spinifex
+GRUB_CMDLINE_LINUX_DEFAULT=""
+GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 systemd.show_status=1"
+`
+
+	if err := os.WriteFile(filepath.Join(mountRoot, "etc/default/grub"), []byte(grubDefault), 0o644); err != nil {
+		return fmt.Errorf("write /etc/default/grub: %w", err)
+	}
+
+	// Mirror the ISO grub.cfg graphical block exactly so the installed GRUB menu
+	// looks identical to the installer menu. gfxterm MUST be activated before
+	// serial is appended — background_image silently does nothing in text mode.
+	// Using exec tail so update-grub includes everything from line 3 as raw GRUB config.
+	grubTheme := `#!/bin/sh
+exec tail -n +3 $0
+insmod all_video
+insmod font
+if loadfont /boot/grub/fonts/unicode.pf2; then
+  set gfxmode=auto
+  insmod gfxterm
+  terminal_output gfxterm
+fi
+insmod serial
+if serial --unit=0 --speed=115200 --timeout=1; then
+  terminal_input  --append serial
+  terminal_output --append serial
+fi
+
+# --- Branding ---
+insmod png
+set theme=/boot/grub/theme.txt
+export theme
+`
+	if err := os.WriteFile(filepath.Join(mountRoot, "etc/grub.d/06_spinifex"), []byte(grubTheme), 0o755); err != nil {
+		return fmt.Errorf("write /etc/grub.d/06_spinifex: %w", err)
+	}
+
+	if err := bindChrootMounts(); err != nil {
+		return err
+	}
+	defer unbindChrootMounts()
+	return run("chroot", mountRoot, "update-grub")
+}
+
+func installCACert(cfg *Config) error {
+	if !cfg.HasCACert || cfg.CACert == "" {
+		return nil
+	}
+	certPath := filepath.Join(mountRoot, "usr/local/share/ca-certificates/spinifex-ca.crt")
+	if err := os.WriteFile(certPath, []byte(cfg.CACert), 0o644); err != nil {
+		return err
+	}
+	if err := bindChrootMounts(); err != nil {
+		return err
+	}
+	defer unbindChrootMounts()
+	return run("chroot", mountRoot, "update-ca-certificates")
+}
+
+// promptRemoveUSB prints a removal reminder and waits up to 10 seconds for
+// the user to press Enter before rebooting. Reading from os.Stdin works because
+// spinifex-init redirects the installer's stdin from $CONSOLE_DEV.
+func promptRemoveUSB() {
+	fmt.Println("\n\033[1mInstallation complete.\033[0m")
+	fmt.Println("Remove the USB drive, then press Enter to reboot (auto-rebooting in 10 seconds)...")
+
+	done := make(chan struct{})
+	go func() {
+		scanner := bufio.NewScanner(os.Stdin)
+		scanner.Scan()
+		close(done)
+	}()
+
+	select {
+	case <-done:
+	case <-time.After(10 * time.Second):
+	}
+}
+
+// fireInstallCallback notifies the boot controller that installation is done
+// so it clears the PXE install flag before the node reboots. Without this,
+// a PXE-first boot order causes the node to reinstall on every reboot until
+// firstboot fires the callback — which it can never do if it never runs.
+// No-op when SPINIFEX_INSTALL_CALLBACK is not set (ISO/USB installs).
+func fireInstallCallback() {
+	url := strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK"))
+	if url == "" {
+		return
+	}
+	client := &http.Client{Timeout: 10 * time.Second}
+	resp, err := client.Get(url) //nolint:noctx // installer has no context; best-effort fire-and-forget
+	if err != nil {
+		slog.Warn("fireInstallCallback: request failed", "url", url, "err", err)
+		return
+	}
+	resp.Body.Close()
+	slog.Info("fireInstallCallback: notified boot controller", "url", url, "status", resp.StatusCode)
+}
+
+func reboot() error {
+	// sync filesystems before reboot so nothing is lost.
+	_ = run("sync")
+	// Use the kernel syscall directly — the live environment runs spinifex-init
+	// as PID 1 (not systemd), so the reboot(8) utility fails trying to reach D-Bus.
+	return syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART)
+}
+
+// toFirstbootConfig maps installer Config to the firstboot package's Config.
+func (c *Config) toFirstbootConfig() firstboot.Config {
+	// Geneve encap IP: use LAN bridge IP when a dedicated LAN NIC is present,
+	// otherwise fall back to WAN bridge IP. Empty for DHCP — setup-ovn.sh
+	// auto-detects the IP from the default route at boot in that case.
+	encapIP := c.WANAddress
+	if c.LANInterface != "" && c.LANAddress != "" {
+		encapIP = c.LANAddress
+	}
+	return firstboot.Config{
+		Hostname:        c.Hostname,
+		EncapIP:         encapIP,
+		ClusterRole:     c.ClusterRole,
+		JoinAddr:        c.JoinAddr,
+		Email:           c.Email,
+		InstallCallback: strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK")),
+		SkipFormation:   c.SkipFormation,
+	}
+}
+
+// writeFstab writes /etc/fstab on the installed system using partition UUIDs so
+// the root filesystem is mounted read-write at boot and the EFI partition is
+// mounted at /boot/efi.
+func writeFstab(disk string) error {
+	efi, root := partitionPaths(disk)
+	rootUUID, err := partUUID(root)
+	if err != nil {
+		return fmt.Errorf("get root UUID: %w", err)
+	}
+	efiUUID, err := partUUID(efi)
+	if err != nil {
+		return fmt.Errorf("get EFI UUID: %w", err)
+	}
+	fstab := fmt.Sprintf("# /etc/fstab — generated by Spinifex installer\nUUID=%s / ext4 errors=remount-ro 0 1\nUUID=%s /boot/efi vfat umask=0077 0 1\n",
+		rootUUID, efiUUID)
+	return os.WriteFile(filepath.Join(mountRoot, "etc/fstab"), []byte(fstab), 0o644)
+}
+
+func partUUID(dev string) (string, error) {
+	out, err := exec.Command("blkid", "-s", "UUID", "-o", "value", dev).Output()
+	if err != nil {
+		return "", fmt.Errorf("blkid %s: %w", dev, err)
+	}
+	uuid := strings.TrimSpace(string(out))
+	if uuid == "" {
+		return "", fmt.Errorf("blkid returned no UUID for %s — partition may not have a filesystem yet", dev)
+	}
+	return uuid, nil
+}
+
+// partitionPaths returns the EFI and root partition device paths for a given
+// disk. p1 is the BIOS Boot Partition (no filesystem), p2 is EFI, p3 is root.
+// Handles both /dev/sdX (→ /dev/sdX2, /dev/sdX3) and /dev/nvmeXnY
+// (→ /dev/nvmeXnYp2, /dev/nvmeXnYp3).
+func partitionPaths(disk string) (efi, root string) {
+	// NVMe devices use a 'p' separator before the partition number.
+	if len(disk) > 0 && disk[len(disk)-1] >= '0' && disk[len(disk)-1] <= '9' {
+		return disk + "p2", disk + "p3"
+	}
+	return disk + "2", disk + "3"
+}
+
+// copyGrubFont copies the unicode.pf2 GRUB font into the installed system's
+// /boot/grub/fonts/ directory. This ensures update-grub finds the font at
+// /boot/grub/fonts/unicode.pf2 — the same path the ISO's grub.cfg uses —
+// so the generated grub.cfg enables gfxterm and the background image.
+// Without this, grub-mkconfig falls back to /usr/share/grub/unicode.pf2,
+// a path that GRUB 2.12 (trixie) may fail to resolve at boot time.
+// Non-fatal — a missing source is logged and skipped.
+func copyGrubFont(root string) {
+	candidates := []string{
+		"/cdrom/boot/grub/fonts/unicode.pf2", // ISO tree (preferred)
+		"/usr/share/grub/unicode.pf2",        // live system grub-common fallback
+	}
+	for _, src := range candidates {
+		in, err := os.Open(src)
+		if err != nil {
+			continue
+		}
+		defer in.Close()
+		dstDir := filepath.Join(root, "boot/grub/fonts")
+		if err := os.MkdirAll(dstDir, 0o755); err != nil {
+			slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
+			return
+		}
+		out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
+		if err != nil {
+			slog.Warn("copyGrubFont: cannot open destination", "err", err)
+			return
+		}
+		defer out.Close()
+		if _, err := io.Copy(out, in); err != nil {
+			slog.Warn("copyGrubFont: copy failed", "err", err)
+		}
+		return
+	}
+	slog.Warn("copyGrubFont: no unicode.pf2 found, splash may not display")
+}
+
+// copySplashImage copies the GRUB splash from the live ISO (/cdrom/boot/grub/splash.png)
+// into the installed system so the post-install GRUB shows the same branded background
+// as the installer. Non-fatal — a missing or unreadable source is logged and skipped.
+func copySplashImage(root string) {
+	const src = "/usr/share/spinifex/grub-splash.png"
+	in, err := os.Open(src)
+	if err != nil {
+		slog.Warn("copySplashImage: splash not found, skipping", "path", src)
+		return
+	}
+	defer in.Close()
+
+	dstDir := filepath.Join(root, "boot/grub")
+	if err := os.MkdirAll(dstDir, 0o755); err != nil {
+		slog.Warn("copySplashImage: cannot create boot/grub dir", "err", err)
+		return
+	}
+	out, err := os.OpenFile(filepath.Join(dstDir, "splash.png"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
+	if err != nil {
+		slog.Warn("copySplashImage: cannot open destination", "err", err)
+		return
+	}
+	defer out.Close()
+	if _, err := io.Copy(out, in); err != nil {
+		slog.Warn("copySplashImage: copy failed", "err", err)
+	}
+}
+
+// maskSystemdUnit creates a symlink to /dev/null for the given unit, which is
+// the standard way to permanently disable a unit so systemd will never start it.
+func maskSystemdUnit(root, unit string) error {
+	unitPath := filepath.Join(root, "etc/systemd/system", unit)
+	_ = os.Remove(unitPath)
+	return os.Symlink("/dev/null", unitPath)
+}
+
+// setShadowPassword sets a Unix password on the installed system without
+// going through PAM. See the long comment in installSpinifex for the
+// rationale.
+func setShadowPassword(user, password string) error {
+	cmd := exec.Command("chpasswd", "-c", "YESCRYPT", "-R", mountRoot)
+	cmd.Stdin = strings.NewReader(user + ":" + password)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	return cmd.Run()
+}
+
+// chrootMountPaths lists virtual filesystems to bind-mount into the chroot.
+// Order matters: unbind in reverse.
+var chrootMountPaths = []string{"dev", "proc", "sys"}
+
+// bindChrootMounts bind-mounts /dev, /proc, and /sys into the installed rootfs
+// so chroot commands (chpasswd, systemd-machine-id-setup, update-grub) can
+// access hardware, process info, and entropy sources. Idempotent — already-
+// mounted paths are skipped.
+func bindChrootMounts() error {
+	for _, m := range chrootMountPaths {
+		dst := filepath.Join(mountRoot, m)
+		if err := os.MkdirAll(dst, 0o755); err != nil {
+			return fmt.Errorf("create chroot mountpoint /%s: %w", m, err)
+		}
+		if err := run("mount", "--bind", "/"+m, dst); err != nil {
+			return fmt.Errorf("bind-mount /%s into chroot: %w", m, err)
+		}
+	}
+	return nil
+}
+
+// unbindChrootMounts unmounts the virtual filesystems in reverse order.
+// Errors are logged but not returned — this is best-effort cleanup.
+func unbindChrootMounts() {
+	for i := len(chrootMountPaths) - 1; i >= 0; i-- {
+		_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
+	}
+}
+
+func run(name string, args ...string) error {
+	cmd := exec.Command(name, args...)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	return cmd.Run()
+}

cmd/installer/install/install_test.go

@@ -0,0 +1,27 @@
+package install
+
+import (
+	"os"
+	"testing"
+)
+
+func TestToFirstbootConfigIncludesInstallCallback(t *testing.T) {
+	const url = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
+	t.Setenv("SPINIFEX_INSTALL_CALLBACK", url)
+
+	cfg := &Config{Hostname: "node1", ClusterRole: "init"}
+	fb := cfg.toFirstbootConfig()
+	if fb.InstallCallback != url {
+		t.Errorf("InstallCallback = %q, want %q", fb.InstallCallback, url)
+	}
+}
+
+func TestToFirstbootConfigEmptyCallbackWhenEnvUnset(t *testing.T) {
+	os.Unsetenv("SPINIFEX_INSTALL_CALLBACK")
+
+	cfg := &Config{Hostname: "node1", ClusterRole: "init"}
+	fb := cfg.toFirstbootConfig()
+	if fb.InstallCallback != "" {
+		t.Errorf("InstallCallback = %q, want empty", fb.InstallCallback)
+	}
+}

cmd/installer/main.go

@@ -0,0 +1,87 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/mulgadc/spinifex/cmd/installer/autoinstall"
+	"github.com/mulgadc/spinifex/cmd/installer/install"
+	"github.com/mulgadc/spinifex/cmd/installer/ui"
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+func main() {
+	// Check for a headless autoinstall config on the boot media before
+	// launching the interactive TUI. If found and enabled, run without
+	// any user input then eject the USB and reboot.
+	if cfg, err := autoinstall.Load(); err != nil {
+		fmt.Fprintf(os.Stderr, "autoinstall: %v\n", err)
+		os.Exit(1)
+	} else if cfg != nil {
+		if err := install.Run(cfg); err != nil {
+			fmt.Fprintf(os.Stderr, "install failed: %v\n", err)
+			os.Exit(1)
+		}
+		autoinstall.EjectAndReboot()
+		return
+	}
+
+	// Normal interactive path.
+	ttyPath := detectTTY()
+
+	cfg, err := ui.Run(ttyPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "installer: %v\n", err)
+		os.Exit(1)
+	}
+
+	if err := install.Run(cfg); err != nil {
+		fmt.Fprintf(os.Stderr, "install failed: %v\n", err)
+		os.Exit(1)
+	}
+}
+
+// detectTTY returns the TTY device path the installer should attach to.
+// It first checks the SPINIFEX_CONSOLE environment variable (set by
+// spinifex-init.sh after console fallback logic), then falls back to parsing
+// /proc/cmdline. spinifex-init.sh may override the GRUB-set value (e.g. when
+// GRUB says tty1 but no framebuffer exists, it falls back to ttyS0 and
+// exports the corrected value).
+func detectTTY() string {
+	// Prefer the env var — spinifex-init.sh sets this after resolving the
+	// actual console (including headless/framebuffer fallback).
+	if val := os.Getenv("SPINIFEX_CONSOLE"); val != "" && val != "auto" {
+		return "/dev/" + val
+	}
+
+	// Fall back to kernel cmdline for direct invocation without spinifex-init.
+	data, err := os.ReadFile("/proc/cmdline")
+	if err == nil {
+		for param := range strings.FieldsSeq(string(data)) {
+			if after, ok := strings.CutPrefix(param, "SPINIFEX_CONSOLE="); ok {
+				val := after
+				if val != "" && val != "auto" {
+					return "/dev/" + val
+				}
+			}
+		}
+	}
+	return "/dev/tty1"
+}

cmd/installer/systemd/systemd.go

@@ -0,0 +1,182 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package systemd writes systemd unit files and drop-ins into an installed
+// system root during the Spinifex ISO installation process.
+package systemd
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+)
+
+// WriteLANBridgeUnit installs a non-critical oneshot service that activates
+// br-lan via networkctl *after* network-online.target. br-lan has
+// ActivationPolicy=manual in its networkd .network file, so networkd creates
+// the bridge at boot but does not bring it up automatically. This service
+// triggers the activation once the WAN path is confirmed online, ensuring a
+// missing LAN cable or DHCP timeout on the secondary bridge never stalls
+// the management interface or firstboot.
+func WriteLANBridgeUnit(root string) error {
+	unit := `[Unit]
+Description=Spinifex LAN bridge (non-critical)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/networkctl up br-lan
+RemainAfterExit=yes
+# Failure is non-critical — cable unplugged or switch not ready at boot.
+SuccessExitStatus=0 1
+
+[Install]
+WantedBy=multi-user.target
+`
+	unitPath := filepath.Join(root, "etc/systemd/system/spinifex-lan-bridge.service")
+	if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
+		return err
+	}
+	wantsDir := filepath.Join(root, "etc/systemd/system/multi-user.target.wants")
+	if err := os.MkdirAll(wantsDir, 0o755); err != nil {
+		return err
+	}
+	link := filepath.Join(wantsDir, "spinifex-lan-bridge.service")
+	if err := os.Remove(link); err != nil && !os.IsNotExist(err) {
+		return fmt.Errorf("remove stale symlink %s: %w", link, err)
+	}
+	return os.Symlink("/etc/systemd/system/spinifex-lan-bridge.service", link)
+}
+
+// WriteFirstbootUnit writes the spinifex-firstboot.service oneshot unit that
+// runs the firstboot provisioning script on the first real boot after
+// installation. The script writes /var/lib/spinifex/.firstboot-done on
+// success; ConditionPathExists=! on that marker condition-skips the unit on
+// every subsequent boot so it never re-runs (replacing the previous
+// "disable on EXIT" trap which couldn't distinguish success from failure).
+func WriteFirstbootUnit(root string) error {
+	unit := `[Unit]
+Description=Spinifex first-boot provisioning
+After=network-online.target
+Wants=network-online.target
+ConditionPathExists=/usr/local/bin/spinifex-firstboot.sh
+ConditionPathExists=!/var/lib/spinifex/.firstboot-done
+
+[Service]
+Type=oneshot
+Environment=HOME=/root
+ExecStart=/usr/local/bin/spinifex-firstboot.sh
+RemainAfterExit=yes
+# On non-zero exit, overwrite /etc/issue with a hint so the operator sees
+# the failure on the console — the banner unit is never enabled in that
+# case (firstboot didn't reach its enable --now line), so /etc/issue would
+# otherwise stay at the Debian default with no signal that anything is wrong.
+ExecStopPost=/bin/sh -c 'if [ "$EXIT_STATUS" != "0" ]; then printf "\n*** Spinifex firstboot FAILED ***\nLogin as spinifex, then run:\n  sudo journalctl -u spinifex-firstboot.service\n\n" > /etc/issue; fi'
+# Cap total firstboot runtime so a hang in setup-ovn.sh / spx admin init /
+# ovn-central startup cannot wedge multi-user.target and keep getty from
+# ever reaching the login prompt.
+TimeoutStartSec=300s
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
+`
+	path := filepath.Join(root, "etc/systemd/system/spinifex-firstboot.service")
+	return os.WriteFile(path, []byte(unit), 0o644)
+}
+
+// WriteBannerUnit writes the spinifex-banner.service unit that runs
+// `spx admin banner --boot-check` on every boot after spinifex.target is up.
+// Running After=spinifex.target ensures the banner reflects a settled system
+// state and that the IP check/restart happens once services are already running.
+//
+// The unit file is written by the installer but NOT enabled at install time.
+// firstboot enables it (along with spinifex.target) at the end of its first
+// run via `systemctl enable --now`. On every subsequent boot both units are
+// driven by the multi-user.target.wants/ symlinks firstboot installed.
+//
+// Note: only After=spinifex.target — no Wants=. A Wants= would implicitly
+// pull spinifex.target up at boot, which is exactly the race this whole
+// change is fixing. After= alone is a no-op when the target isn't started,
+// which is correct on first boot before firstboot has provisioned configs.
+func WriteBannerUnit(root string) error {
+	unit := `[Unit]
+Description=Spinifex console banner and boot health check
+After=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/spx admin banner --boot-check
+RemainAfterExit=yes
+# Banner is oneshot; cap it so a stuck boot-check (IP detection, try-restart)
+# cannot block getty via the spinifex-wait.conf drop-in.
+TimeoutStartSec=30s
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
+`
+	path := filepath.Join(root, "etc/systemd/system/spinifex-banner.service")
+	return os.WriteFile(path, []byte(unit), 0o644)
+}
+
+// WriteGettyDropIn holds the primary consoles (tty1 and ttyS0) until
+// spinifex-banner.service completes (when banner is enabled), so the MOTD
+// banner is visible before the login prompt appears. Drop-ins are scoped to
+// named instances (getty@tty1, serial-getty@ttyS0) so tty2, tty3, etc.
+// remain available as rescue terminals.
+//
+// Only After=, not Wants=. A Wants= would implicitly start the (un-enabled)
+// banner on the very first boot, partially undoing the "only firstboot is
+// enabled on first boot" guarantee. On first boot, banner is not enabled,
+// so After= is a no-op and login appears immediately with the default
+// /etc/issue. Once firstboot completes and enables the banner, subsequent
+// boots have banner enabled at multi-user.target and After= correctly
+// orders getty after the banner has rendered /etc/issue.
+func WriteGettyDropIn(root string) error {
+	dropIn := `[Unit]
+After=spinifex-firstboot.service
+`
+	for _, svc := range []string{"getty@tty1", "serial-getty@ttyS0"} {
+		dropInDir := filepath.Join(root, "etc/systemd/system/"+svc+".service.d")
+		if err := os.MkdirAll(dropInDir, 0o755); err != nil {
+			return err
+		}
+		if err := os.WriteFile(filepath.Join(dropInDir, "spinifex-wait.conf"), []byte(dropIn), 0o644); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// EnableUnit creates the multi-user.target.wants symlink for serviceName,
+// equivalent to `systemctl enable` for a unit that targets multi-user.target.
+func EnableUnit(root, serviceName string) error {
+	wantsDir := filepath.Join(root, "etc/systemd/system/multi-user.target.wants")
+	if err := os.MkdirAll(wantsDir, 0o755); err != nil {
+		return err
+	}
+	link := filepath.Join(wantsDir, serviceName)
+	target := "/etc/systemd/system/" + serviceName
+	if err := os.Remove(link); err != nil && !os.IsNotExist(err) {
+		return fmt.Errorf("remove stale symlink %s: %w", link, err)
+	}
+	return os.Symlink(target, link)
+}

cmd/lb-agent/main.go

@@ -0,0 +1,92 @@
+// lb-agent runs inside load balancer VMs and manages HAProxy configuration.
+//
+// It polls the AWS gateway for config updates and reports health via heartbeats.
+// All communication uses SigV4-signed HTTP requests.
+//
+// Usage:
+//
+//	lb-agent --lb-id=lb-xxxxx --gateway=https://192.168.1.33:9999 --access-key=AKIA... --secret-key=...
+//
+// Flags default to environment variables set by cloud-init:
+// LB_LB_ID, LB_GATEWAY_URL, LB_ACCESS_KEY, LB_SECRET_KEY, LB_REGION.
+package main
+
+import (
+	"flag"
+	"log/slog"
+	"os"
+	"os/signal"
+	"syscall"
+
+	"github.com/mulgadc/spinifex/spinifex/lbagent"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+func main() {
+	var (
+		lbID       string
+		gatewayURL string
+		accessKey  string
+		secretKey  string
+		region     string
+	)
+
+	hostname, _ := os.Hostname()
+
+	flag.StringVar(&lbID, "lb-id", envOrDefault("LB_LB_ID", hostname), "Load balancer ID")
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("LB_GATEWAY_URL"), "Gateway URL (e.g. https://192.168.1.33:9999)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("LB_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("LB_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", envOrDefault("LB_REGION", "us-east-1"), "AWS region for SigV4 signing")
+	flag.Parse()
+
+	if lbID == "" {
+		slog.Error("--lb-id is required (or set LB_LB_ID)")
+		os.Exit(1)
+	}
+	if gatewayURL == "" {
+		slog.Error("--gateway is required (or set LB_GATEWAY_URL)")
+		os.Exit(1)
+	}
+	if accessKey == "" || secretKey == "" {
+		slog.Error("--access-key and --secret-key are required (or set LB_ACCESS_KEY / LB_SECRET_KEY)")
+		os.Exit(1)
+	}
+
+	slog.Info("Starting LB agent", "lbId", lbID, "gateway", gatewayURL)
+
+	agent, err := lbagent.New(lbID, gatewayURL, accessKey, secretKey, region)
+	if err != nil {
+		slog.Error("Failed to create agent", "err", err)
+		os.Exit(1)
+	}
+
+	// Start poll loop in a goroutine — it blocks until Stop is called.
+	errCh := make(chan error, 1)
+	go func() {
+		if err := agent.Start(); err != nil {
+			errCh <- err
+		}
+	}()
+
+	// Wait for shutdown signal or fatal error.
+	sigCh := make(chan os.Signal, 1)
+	signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
+
+	select {
+	case sig := <-sigCh:
+		slog.Info("Received signal, shutting down", "signal", sig)
+	case err := <-errCh:
+		slog.Error("Agent error", "err", err)
+	}
+
+	agent.Stop()
+}
+
+func envOrDefault(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}

cmd/spinifex/cmd/admin_gpu.go

@@ -0,0 +1,486 @@
+package cmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/gpu"
+	"github.com/mulgadc/spinifex/spinifex/types"
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+var adminGpuCmd = &cobra.Command{
+	Use:   "gpu",
+	Short: "Manage GPU passthrough for a node",
+}
+
+var adminGpuStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Show GPU hardware and passthrough state",
+	Run:   runAdminGpuStatus,
+}
+
+var adminGpuEnableCmd = &cobra.Command{
+	Use:   "enable",
+	Short: "Enable GPU passthrough on a node",
+	Run:   runAdminGpuEnable,
+}
+
+var adminGpuDisableCmd = &cobra.Command{
+	Use:   "disable",
+	Short: "Disable GPU passthrough on a node (blocked while GPU instances are running)",
+	Run:   runAdminGpuDisable,
+}
+
+var adminGpuSetupCmd = &cobra.Command{
+	Use:   "setup",
+	Short: "Configure this host for GPU passthrough (IOMMU, vfio-pci, nouveau blacklist)",
+	Long: `Idempotent host setup for GPU passthrough. Run once before a reboot to configure
+GRUB, vfio-pci early binding, and the nouveau blacklist; then run again after the
+reboot to verify bindings and enable GPU passthrough in the daemon.
+
+Must be run as root.`,
+	Run: runAdminGpuSetup,
+}
+
+func init() {
+	adminCmd.AddCommand(adminGpuCmd)
+	adminGpuCmd.AddCommand(adminGpuStatusCmd)
+	adminGpuCmd.AddCommand(adminGpuEnableCmd)
+	adminGpuCmd.AddCommand(adminGpuDisableCmd)
+	adminGpuCmd.AddCommand(adminGpuSetupCmd)
+
+	adminGpuStatusCmd.Flags().String("node", "", "Target node name (default: local node)")
+	// enable, disable, and setup write to the local spinifex.toml and signal the
+	// local daemon — they must be run directly on the target host.
+}
+
+// gpuNodeStatus queries NATS and returns the NodeStatusResponse for the target node.
+func gpuNodeStatus(targetNode string) (*types.NodeStatusResponse, error) {
+	cfg, nc, err := loadConfigAndConnect()
+	if err != nil {
+		return nil, err
+	}
+	defer nc.Close()
+
+	if targetNode == "" {
+		targetNode = cfg.Node
+	}
+
+	responses, err := collectResponses(nc, "spinifex.node.status", 3*time.Second)
+	if err != nil {
+		return nil, fmt.Errorf("collect node status: %w", err)
+	}
+	for _, raw := range responses {
+		var resp types.NodeStatusResponse
+		if err := json.Unmarshal(raw, &resp); err != nil {
+			continue
+		}
+		if resp.Node == targetNode {
+			return &resp, nil
+		}
+	}
+	return nil, fmt.Errorf("node %q not found or not responding", targetNode)
+}
+
+func runAdminGpuStatus(cmd *cobra.Command, _ []string) {
+	targetNode, _ := cmd.Flags().GetString("node")
+	resp, err := gpuNodeStatus(targetNode)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	iommu := "unknown"
+	vfio := "unknown"
+	if resp.GPUCapable || len(resp.GPUModels) > 0 {
+		iommu = "active"
+		vfio = "loaded"
+	}
+
+	fmt.Printf("Node:            %s\n", resp.Node)
+	if len(resp.GPUModels) > 0 {
+		fmt.Printf("GPU hardware:    %s\n", strings.Join(resp.GPUModels, ", "))
+	} else {
+		fmt.Printf("GPU hardware:    none detected\n")
+	}
+	fmt.Printf("IOMMU:           %s\n", iommu)
+	fmt.Printf("vfio-pci:        %s\n", vfio)
+
+	if resp.GPUPassthrough {
+		fmt.Printf("Passthrough:     enabled\n")
+		fmt.Printf("GPU pool:        %d/%d allocated\n", resp.AllocGPUs, resp.TotalGPUs)
+
+		var gpuTypes []string
+		for _, cap := range resp.InstanceTypes {
+			gpuTypes = append(gpuTypes, cap.Name)
+		}
+		if len(gpuTypes) > 0 {
+			fmt.Printf("Instance types:  %s\n", strings.Join(gpuTypes, " "))
+		}
+	} else if resp.GPUCapable {
+		fmt.Printf("Passthrough:     disabled\n")
+		fmt.Printf("Instance types:  (none — run 'spx admin gpu enable' to activate)\n")
+	} else {
+		fmt.Printf("Passthrough:     disabled\n")
+		fmt.Printf("Instance types:  (prerequisites not met)\n")
+	}
+}
+
+// gpuToggle writes the TOML setting, sends SIGHUP, and polls until the daemon confirms the new state.
+// enable/disable always operate on the local node; run the command directly on the target host.
+func gpuToggle(_ *cobra.Command, enable bool) {
+	cfgPath := viper.GetString("config")
+	if cfgPath == "" {
+		cfgPath = DefaultConfigFile()
+	}
+	cfg, err := config.LoadConfig(cfgPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
+		os.Exit(1)
+	}
+	localNode := cfg.Node
+
+	// Check current state first.
+	resp, err := gpuNodeStatus(localNode)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	if enable {
+		if resp.GPUPassthrough {
+			fmt.Println("GPU passthrough is already enabled.")
+			return
+		}
+		if !resp.GPUCapable {
+			fmt.Fprintln(os.Stderr, "Error: prerequisites not met on this node.")
+			fmt.Fprintln(os.Stderr, "  Run 'sudo spx admin gpu setup' to configure the host.")
+			os.Exit(1)
+		}
+	} else {
+		if !resp.GPUPassthrough {
+			fmt.Println("GPU passthrough is already disabled.")
+			return
+		}
+		if resp.AllocGPUs > 0 {
+			fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
+			os.Exit(1)
+		}
+	}
+
+	// Write the TOML setting.
+	if err := admin.SetGPUPassthrough(cfgPath, localNode, enable); err != nil {
+		fmt.Fprintf(os.Stderr, "Error writing config: %v\n", err)
+		os.Exit(1)
+	}
+
+	// Signal the daemon to reload.
+	out, err := exec.Command("systemctl", "kill", "-s", "HUP", "spinifex-daemon").CombinedOutput()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error sending SIGHUP: %v\n%s\n", err, out)
+		os.Exit(1)
+	}
+
+	// Poll until the daemon confirms the new state.
+	action := "enable"
+	if !enable {
+		action = "disable"
+	}
+	fmt.Printf("Waiting for daemon to %s GPU passthrough", action)
+	deadline := time.Now().Add(30 * time.Second)
+	for time.Now().Before(deadline) {
+		time.Sleep(1 * time.Second)
+		fmt.Print(".")
+		r, err := gpuNodeStatus(localNode)
+		if err != nil {
+			continue
+		}
+		if r.GPUPassthrough == enable {
+			fmt.Println(" done.")
+			// Refresh the MOTD banner immediately so the login prompt
+			// reflects the new GPU passthrough state without requiring a reboot.
+			_ = exec.Command("spx", "admin", "banner", "--boot-check").Run()
+			runAdminGpuStatus(adminGpuStatusCmd, nil)
+			return
+		}
+	}
+	fmt.Println()
+	fmt.Fprintln(os.Stderr, "Timed out waiting for daemon. Check: journalctl -u spinifex-daemon -n 50")
+	os.Exit(1)
+}
+
+func runAdminGpuEnable(cmd *cobra.Command, _ []string) {
+	gpuToggle(cmd, true)
+}
+
+func runAdminGpuDisable(cmd *cobra.Command, _ []string) {
+	gpuToggle(cmd, false)
+}
+
+func runAdminGpuSetup(_ *cobra.Command, _ []string) {
+	if os.Getuid() != 0 {
+		fmt.Fprintln(os.Stderr, "Error: spx admin gpu setup must be run as root.")
+		os.Exit(1)
+	}
+
+	fmt.Println("==> Detecting GPU")
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+	if len(devices) == 0 {
+		fmt.Fprintln(os.Stderr, "No NVIDIA or AMD GPU found — is the card seated?")
+		os.Exit(1)
+	}
+	for _, d := range devices {
+		fmt.Printf("    %s - %s\n", d.PCIAddress, d.Model)
+	}
+
+	fmt.Println("==> Collecting PCI IDs for vfio-pci")
+	ids := gpuSetupCollectVFIOIDs(devices)
+	idsCSV := strings.Join(ids, ",")
+	fmt.Printf("    IDs: %s\n", idsCSV)
+
+	rebootNeeded := false
+
+	fmt.Println("==> Checking IOMMU")
+	iommuEntries, _ := os.ReadDir("/sys/kernel/iommu_groups/")
+	if len(iommuEntries) > 0 {
+		fmt.Println("    Active")
+	} else {
+		params := gpuSetupIOMMUParams()
+		changed, err := gpuSetupAddGRUBParams(params)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to update GRUB: %v\n", err)
+			os.Exit(1)
+		}
+		if changed {
+			fmt.Printf("    GRUB updated with: %s\n", params)
+		} else {
+			fmt.Println("    GRUB already has IOMMU params — refreshing")
+		}
+		ug := exec.Command("update-grub")
+		ug.Stdout, ug.Stderr = os.Stdout, os.Stderr
+		if err := ug.Run(); err != nil {
+			fmt.Fprintf(os.Stderr, "update-grub failed: %v\n", err)
+			os.Exit(1)
+		}
+		rebootNeeded = true
+	}
+
+	fmt.Println("==> Configuring vfio udev rule")
+	const vfioUdevRule = "/etc/udev/rules.d/99-spinifex-vfio.rules"
+	if _, err := os.Stat(vfioUdevRule); os.IsNotExist(err) {
+		rule := "SUBSYSTEM==\"vfio\", GROUP=\"spinifex\", MODE=\"0660\"\n"
+		if err := os.WriteFile(vfioUdevRule, []byte(rule), 0o644); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write vfio udev rule: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("    Rule installed")
+		_ = exec.Command("udevadm", "control", "--reload-rules").Run()
+		_ = exec.Command("udevadm", "trigger", "--subsystem-match=vfio").Run()
+	} else {
+		fmt.Println("    Rule already present")
+	}
+
+	fmt.Println("==> Configuring nouveau blacklist")
+	const nouveauConf = "/etc/modprobe.d/blacklist-nouveau.conf"
+	if _, err := os.Stat(nouveauConf); os.IsNotExist(err) {
+		if err := os.WriteFile(nouveauConf, []byte("blacklist nouveau\noptions nouveau modeset=0\n"), 0o644); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write nouveau blacklist: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("    nouveau blacklisted")
+		rebootNeeded = true
+	} else {
+		fmt.Println("    Already blacklisted")
+	}
+
+	fmt.Println("==> Configuring amdgpu blacklist")
+	const amdgpuConf = "/etc/modprobe.d/blacklist-amdgpu.conf"
+	hasAMD := false
+	for _, d := range devices {
+		if d.Vendor == gpu.VendorAMD {
+			hasAMD = true
+			break
+		}
+	}
+	if hasAMD {
+		if _, err := os.Stat(amdgpuConf); os.IsNotExist(err) {
+			if err := os.WriteFile(amdgpuConf, []byte("blacklist amdgpu\n"), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "Failed to write amdgpu blacklist: %v\n", err)
+				os.Exit(1)
+			}
+			fmt.Println("    amdgpu blacklisted")
+			rebootNeeded = true
+		} else {
+			fmt.Println("    Already blacklisted")
+		}
+	} else {
+		fmt.Println("    No AMD GPUs — skipping")
+	}
+
+	fmt.Println("==> Configuring vfio-pci early binding")
+	const vfioPCIConf = "/etc/modprobe.d/vfio-pci.conf"
+	existing, _ := os.ReadFile(vfioPCIConf)
+	if !strings.Contains(string(existing), "ids="+idsCSV) {
+		if err := os.WriteFile(vfioPCIConf, []byte("options vfio-pci ids="+idsCSV+"\n"), 0o644); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write vfio-pci.conf: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("    vfio-pci.conf written (ids=%s)\n", idsCSV)
+		rebootNeeded = true
+	} else {
+		fmt.Printf("    Already configured for %s\n", idsCSV)
+	}
+
+	const initramfsModules = "/etc/initramfs-tools/modules"
+	modData, _ := os.ReadFile(initramfsModules)
+	if !strings.Contains(string(modData), "vfio_pci") {
+		f, err := os.OpenFile(initramfsModules, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0o644)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to open initramfs modules: %v\n", err)
+			os.Exit(1)
+		}
+		_, werr := fmt.Fprint(f, "\n# vfio early binding for GPU passthrough\nvfio\nvfio_iommu_type1\nvfio_pci\n")
+		f.Close()
+		if werr != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write initramfs modules: %v\n", werr)
+			os.Exit(1)
+		}
+		fmt.Println("    vfio modules added to initramfs")
+		rebootNeeded = true
+	} else {
+		fmt.Println("    vfio modules already in initramfs")
+	}
+
+	if rebootNeeded {
+		fmt.Println("==> Updating initramfs")
+		ui := exec.Command("update-initramfs", "-u")
+		ui.Stdout, ui.Stderr = os.Stdout, os.Stderr
+		if err := ui.Run(); err != nil {
+			fmt.Fprintf(os.Stderr, "update-initramfs failed: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("\nSetup complete — reboot required.")
+		fmt.Println("  sudo reboot")
+		fmt.Println("  Then run: sudo spx admin gpu enable")
+		return
+	}
+
+	fmt.Println("==> Verifying vfio-pci module")
+	if _, err := os.Stat("/sys/module/vfio_pci"); os.IsNotExist(err) {
+		fmt.Fprintln(os.Stderr, "vfio_pci not loaded — check: journalctl -b | grep vfio")
+		os.Exit(1)
+	}
+	fmt.Println("    Loaded")
+
+	fmt.Println("==> Verifying GPU driver binding")
+	for _, d := range devices {
+		current, err := gpu.CurrentDriver(d.PCIAddress)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Cannot read driver for %s: %v\n", d.PCIAddress, err)
+			os.Exit(1)
+		}
+		if current == "vfio-pci" {
+			fmt.Printf("    %s → vfio-pci\n", d.PCIAddress)
+			continue
+		}
+		if current != "" {
+			// Bound to native driver — blacklist/initramfs changes may not have taken effect.
+			fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci — reboot may be required\n", d.PCIAddress, current)
+			fmt.Fprintf(os.Stderr, "  Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
+			os.Exit(1)
+		}
+		// Device is unbound (ids= did not auto-claim it) — bind explicitly via driver_override.
+		fmt.Printf("    %s unbound — binding to vfio-pci\n", d.PCIAddress)
+		if _, err := gpu.BindVFIO(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to bind %s to vfio-pci: %v\n", d.PCIAddress, err)
+			fmt.Fprintf(os.Stderr, "  Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
+			os.Exit(1)
+		}
+		fmt.Printf("    %s → vfio-pci\n", d.PCIAddress)
+	}
+
+	fmt.Println("==> Enabling GPU passthrough")
+	gpuToggle(nil, true)
+}
+
+// gpuSetupCollectVFIOIDs collects vendor:device PCI ID pairs for each GPU and
+// all sibling PCI functions on the same device slot (e.g. GPU + HDMI audio).
+// All siblings must bind to vfio-pci together for IOMMU group isolation.
+func gpuSetupCollectVFIOIDs(devices []gpu.GPUDevice) []string {
+	seen := make(map[string]bool)
+	var ids []string
+	for _, d := range devices {
+		dot := strings.LastIndex(d.PCIAddress, ".")
+		if dot < 0 {
+			continue
+		}
+		prefix := d.PCIAddress[:dot]
+		matches, _ := filepath.Glob("/sys/bus/pci/devices/" + prefix + ".*")
+		for _, m := range matches {
+			vb, err1 := os.ReadFile(m + "/vendor")
+			db, err2 := os.ReadFile(m + "/device")
+			if err1 != nil || err2 != nil {
+				continue
+			}
+			v := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(string(vb))), "0x")
+			dv := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(string(db))), "0x")
+			id := v + ":" + dv
+			if !seen[id] {
+				seen[id] = true
+				ids = append(ids, id)
+			}
+		}
+	}
+	return ids
+}
+
+func gpuSetupIOMMUParams() string {
+	data, _ := os.ReadFile("/proc/cpuinfo")
+	if strings.Contains(string(data), "GenuineIntel") {
+		return "intel_iommu=on iommu=pt"
+	}
+	return "amd_iommu=on iommu=pt"
+}
+
+// gpuSetupAddGRUBParams appends iommu kernel params to GRUB_CMDLINE_LINUX_DEFAULT.
+// Returns (true, nil) if the file was modified, (false, nil) if params were already present.
+func gpuSetupAddGRUBParams(params string) (bool, error) {
+	const grubFile = "/etc/default/grub"
+	data, err := os.ReadFile(grubFile)
+	if err != nil {
+		return false, fmt.Errorf("read %s: %w", grubFile, err)
+	}
+	content := string(data)
+	if strings.Contains(content, "intel_iommu=on") || strings.Contains(content, "amd_iommu=on") {
+		return false, nil
+	}
+	re := regexp.MustCompile(`(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*)"`)
+	updated := re.ReplaceAllStringFunc(content, func(match string) string {
+		groups := re.FindStringSubmatch(match)
+		if len(groups) < 2 {
+			return match
+		}
+		return groups[1] + " " + params + `"`
+	})
+	if updated == content {
+		return false, fmt.Errorf("GRUB_CMDLINE_LINUX_DEFAULT not found in %s", grubFile)
+	}
+	if err := os.WriteFile(grubFile, []byte(updated), 0o644); err != nil {
+		return false, fmt.Errorf("write %s: %w", grubFile, err)
+	}
+	return true, nil
+}

cmd/spinifex/cmd/admin_test.go

@@ -0,0 +1,312 @@
+package cmd
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/formation"
+	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/spf13/viper"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Regression guard: source URL must print for every error kind, not just
+// mismatch, so a 404/non-HTTPS/size-cap failure still tells the operator
+// which URL to investigate.
+func TestPrintChecksumError(t *testing.T) {
+	image := utils.Images{Checksum: "https://example.com/SUMS", ChecksumType: "sha512"}
+	const imageFile = "/var/lib/img.tar.xz"
+	const imageName = "debian-13-x86_64"
+
+	errs := []error{
+		fmt.Errorf("%w: expected abc got def", utils.ErrChecksumMismatch),
+		fmt.Errorf("%w: 404", utils.ErrChecksumFetchFailed),
+		errors.New("open /x: no such file"),
+	}
+	for _, e := range errs {
+		var buf bytes.Buffer
+		printChecksumError(&buf, imageFile, imageName, image, e)
+		out := buf.String()
+		assert.Contains(t, out, image.Checksum, "source URL must print for: %v", e)
+		assert.Contains(t, out, imageFile)
+		assert.Contains(t, out, "spx admin images import --name "+imageName+" --force")
+	}
+}
+
+// buildRemoteNodes must prefer AdvertiseIP (off-host dial target) and fall
+// back to BindIP when the peer pre-dates siv-8 and didn't send AdvertiseIP.
+func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
+	nodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "203.0.113.1"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2"}, // legacy joiner
+		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "203.0.113.3"},
+	}
+	got := buildRemoteNodes(nodes, "node3")
+	if assert.Len(t, got, 2) {
+		assert.Equal(t, "node1", got[0].Name)
+		assert.Equal(t, "203.0.113.1", got[0].Host, "advertise wins when set")
+		assert.Equal(t, "node2", got[1].Name)
+		assert.Equal(t, "10.0.0.2", got[1].Host, "bind fallback when advertise empty")
+	}
+}
+
+func TestResolveAdvertiseIP(t *testing.T) {
+	wan := &admin.DetectedNetwork{WAN: &admin.DetectedInterface{IP: "192.168.1.21"}}
+	noWAN := &admin.DetectedNetwork{}
+
+	tests := []struct {
+		name      string
+		bindIP    string
+		advertise string
+		detected  *admin.DetectedNetwork
+		want      string
+		wantErr   bool
+	}{
+		{"single-node default, WAN detected", "0.0.0.0", "", wan, "192.168.1.21", false},
+		{"single-node default, no WAN", "0.0.0.0", "", noWAN, "127.0.0.1", false},
+		{"single-node explicit advertise", "0.0.0.0", "203.0.113.5", wan, "203.0.113.5", false},
+		{"multi-node init, specific bind", "10.11.12.1", "", nil, "10.11.12.1", false},
+		{"multi-node dual-homed override", "10.11.12.1", "203.0.113.5", nil, "203.0.113.5", false},
+		{"loopback-only test", "127.0.0.1", "", noWAN, "127.0.0.1", false},
+		{"invalid advertise flag", "0.0.0.0", "not-an-ip", wan, "", true},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, err := resolveAdvertiseIP(tt.bindIP, tt.advertise, tt.detected)
+			if tt.wantErr {
+				assert.Error(t, err)
+				return
+			}
+			require.NoError(t, err)
+			assert.Equal(t, tt.want, got)
+		})
+	}
+}
+
+// Generated spinifex.toml must include both host (listen, BindIP) and
+// advertise (off-host dial target, AdvertiseIP) for the local node.
+func TestSpinifexTomlTemplate_AdvertiseField(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:        "node1",
+		Az:          "ap-southeast-2a",
+		Port:        "4432",
+		Region:      "ap-southeast-2",
+		BindIP:      "0.0.0.0",
+		AdvertiseIP: "192.168.1.21",
+		AccessKey:   "AKIATEST",
+		SecretKey:   "SECRET",
+		AccountID:   "123456789012",
+		NatsToken:   "token",
+		ConfigDir:   dir,
+		OVNNBAddr:   "tcp:127.0.0.1:6641",
+		OVNSBAddr:   "tcp:127.0.0.1:6642",
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `host = "0.0.0.0"`, "listen address")
+	assert.Contains(t, content, `advertise = "192.168.1.21"`, "off-host dial target")
+}
+
+// Empty AdvertiseIP (e.g. loading an existing cluster pre-siv-8) must NOT
+// render an empty advertise = "" line — downstream fallback to Host kicks in.
+func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:      "node1",
+		Az:        "ap-southeast-2a",
+		Port:      "4432",
+		Region:    "ap-southeast-2",
+		BindIP:    "10.11.12.1",
+		AccessKey: "AKIATEST",
+		SecretKey: "SECRET",
+		AccountID: "123456789012",
+		NatsToken: "token",
+		ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641",
+		OVNSBAddr: "tcp:127.0.0.1:6642",
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+	data, _ := os.ReadFile(path)
+	assert.NotContains(t, string(data), "advertise =")
+}
+
+// detectedDhcpBindBridge must return the default-route interface name verbatim
+// when it's a bridge (Linux or OVS, br-* prefix). The old detectedWanBridge()
+// returned hardcoded "br-ext" for Linux bridges — that value broke DHCP on
+// consumer-router LANs because br-ext never sees LAN DHCP traffic (mulga-998).
+func TestDetectedDhcpBindBridge(t *testing.T) {
+	tests := []struct {
+		name     string
+		detected *admin.DetectedNetwork
+		want     string
+	}{
+		{
+			name:     "Linux bridge default route returns bridge name (not 'br-ext')",
+			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-wan"}},
+			want:     "br-wan",
+		},
+		{
+			name:     "OVS bridge default route returned verbatim",
+			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-ext"}},
+			want:     "br-ext",
+		},
+		{
+			name:     "physical NIC defaults to 'br-wan' (bridge the installer creates)",
+			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "enp0s3"}},
+			want:     "br-wan",
+		},
+		{
+			name:     "nil DetectedNetwork",
+			detected: nil,
+			want:     "",
+		},
+		{
+			name:     "DetectedNetwork with nil WAN",
+			detected: &admin.DetectedNetwork{},
+			want:     "",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equal(t, tt.want, detectedDhcpBindBridge(tt.detected))
+		})
+	}
+}
+
+// Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
+// alias (per mulga-998 D3). Prevents the footgun where operators inherited the
+// old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
+func TestCheckLegacyWanBridgeKey_TOMLRejected(t *testing.T) {
+	dir := t.TempDir()
+	cfgPath := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, os.WriteFile(cfgPath, []byte(`
+node = "node1"
+[nodes.node1.vpcd]
+ovn_nb_addr = "tcp:127.0.0.1:6641"
+wan_bridge = "br-ext"
+`), 0o644))
+
+	viper.Reset()
+	t.Cleanup(viper.Reset)
+	viper.SetConfigFile(cfgPath)
+	viper.SetConfigType("toml")
+	require.NoError(t, viper.ReadInConfig())
+
+	err := checkLegacyWanBridgeKey("node1", cfgPath)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "dhcp_bind_bridge")
+	assert.Contains(t, err.Error(), "wan_bridge")
+}
+
+func TestCheckLegacyWanBridgeKey_EnvVarRejected(t *testing.T) {
+	viper.Reset()
+	t.Cleanup(viper.Reset)
+	t.Setenv("SPINIFEX_VPCD_WAN_BRIDGE", "br-ext")
+
+	err := checkLegacyWanBridgeKey("node1", "/tmp/unused.toml")
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "SPINIFEX_VPCD_WAN_BRIDGE")
+	assert.Contains(t, err.Error(), "dhcp_bind_bridge")
+}
+
+func TestCheckLegacyWanBridgeKey_CleanConfigPasses(t *testing.T) {
+	dir := t.TempDir()
+	cfgPath := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, os.WriteFile(cfgPath, []byte(`
+node = "node1"
+[nodes.node1.vpcd]
+ovn_nb_addr = "tcp:127.0.0.1:6641"
+dhcp_bind_bridge = "br-wan"
+`), 0o644))
+
+	viper.Reset()
+	t.Cleanup(viper.Reset)
+	viper.SetConfigFile(cfgPath)
+	viper.SetConfigType("toml")
+	require.NoError(t, viper.ReadInConfig())
+
+	assert.NoError(t, checkLegacyWanBridgeKey("node1", cfgPath))
+}
+
+// iam.nats_url must not embed a bare 0.0.0.0 — the predastore process dials
+// its own NATS via loopback when listening on wildcard.
+func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
+	nodes := []admin.PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"},
+		{ID: 2, Host: "10.0.0.2"},
+		{ID: 3, Host: "10.0.0.3"},
+	}
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0",
+	)
+	require.NoError(t, err)
+	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
+	assert.NotContains(t, content, `nats_url = "nats://0.0.0.0:4222"`)
+
+	// Specific bind IP → stays as-is.
+	content2, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1",
+	)
+	require.NoError(t, err)
+	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
+}
+
+// writePredastoreEncryptionKey is called from both `spx admin init` and `spx
+// admin join`; both depend on it producing a 32-byte file at 0600 in the
+// expected layout. Predastore's keyfile loader rejects anything that isn't
+// exactly that, so a regression here would break service startup on every
+// node.
+func TestWritePredastoreEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "predastore", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "predastore master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "predastore master key must be mode 0600")
+
+	// Per-node invariant: two successive calls (on what would be two
+	// different nodes) must produce different key material. If they ever
+	// returned the same bytes we'd have silently lost the per-node
+	// blast-radius property.
+	configDir2 := t.TempDir()
+	keyPath2, err := writePredastoreEncryptionKey(configDir2)
+	require.NoError(t, err)
+
+	key1, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	key2, err := os.ReadFile(keyPath2)
+	require.NoError(t, err)
+	assert.NotEqual(t, key1, key2, "per-node keys must differ")
+}
+
+// Joiners run `spx admin join` against a configDir that doesn't yet contain
+// a predastore/ subdir. The helper must create it (predastore subdir is
+// owned by spinifex-storage in prod, but the helper just needs the dir to
+// exist).
+func TestWritePredastoreEncryptionKey_CreatesMissingDir(t *testing.T) {
+	configDir := t.TempDir()
+	// Don't pre-create the predastore subdir — simulate a fresh joiner.
+
+	keyPath, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+
+	dirInfo, err := os.Stat(filepath.Dir(keyPath))
+	require.NoError(t, err)
+	assert.True(t, dirInfo.IsDir())
+}

cmd/spinifex/cmd/admin_upgrade.go

@@ -0,0 +1,92 @@
+package cmd
+
+import (
+	"bufio"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/mulgadc/spinifex/spinifex/migrate"
+	"github.com/spf13/cobra"
+)
+
+var upgradeCmd = &cobra.Command{
+	Use:   "upgrade",
+	Short: "Apply pending config migrations",
+	Long: `Apply pending configuration file migrations for upgrades between Spinifex versions.
+
+When run interactively (without --yes), shows pending migrations and prompts
+for confirmation. When called from setup.sh with --yes, applies immediately.
+
+Operators can skip migrations during install by setting INSTALL_SPINIFEX_SKIP_MIGRATE=1,
+then run 'spx admin upgrade' manually to review and apply.`,
+	Run: runAdminUpgrade,
+}
+
+func runAdminUpgrade(cmd *cobra.Command, _ []string) {
+	configDir, _ := cmd.Root().Flags().GetString("config-dir")
+	dataDir, _ := cmd.Root().Flags().GetString("spinifex-dir")
+	yes, _ := cmd.Flags().GetBool("yes")
+
+	// Check that the installation exists.
+	spinifexToml := configDir + "/spinifex.toml"
+	if _, err := os.Stat(spinifexToml); os.IsNotExist(err) {
+		fmt.Fprintf(os.Stderr, "No Spinifex installation found at %s\nRun 'spx admin init' first.\n", configDir)
+		os.Exit(1)
+	}
+
+	// Show current versions.
+	versions, err := migrate.DefaultRegistry.ConfigVersions(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error reading config versions: %v\n", err)
+		os.Exit(1)
+	}
+
+	fmt.Println("Reading current config versions...")
+	for name, v := range versions {
+		if v == 0 {
+			fmt.Printf("  %-20s %d (no version marker)\n", name+":", v)
+		} else {
+			fmt.Printf("  %-20s %d\n", name+":", v)
+		}
+	}
+	fmt.Println()
+
+	// Check for pending migrations.
+	pending, err := migrate.DefaultRegistry.PendingConfig(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error checking pending migrations: %v\n", err)
+		os.Exit(1)
+	}
+
+	if len(pending) == 0 {
+		fmt.Println("No pending config migrations.")
+		return
+	}
+
+	fmt.Println("Pending config migrations:")
+	for _, p := range pending {
+		fmt.Printf("  [config] %s %d → %d: %s\n", p.Target, p.FromVersion, p.ToVersion, p.Description)
+	}
+	fmt.Println()
+
+	// Prompt for confirmation unless --yes.
+	if !yes {
+		fmt.Print("Apply? [y/N] ")
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		answer = strings.TrimSpace(strings.ToLower(answer))
+		if answer != "y" && answer != "yes" {
+			fmt.Println("Aborted.")
+			return
+		}
+	}
+
+	// Apply migrations.
+	if err := migrate.DefaultRegistry.RunAllConfig(configDir, dataDir); err != nil {
+		fmt.Fprintf(os.Stderr, "Migration failed: %v\n", err)
+		os.Exit(1)
+	}
+
+	fmt.Println("\nDone. Restart services to apply: sudo systemctl restart spinifex.target")
+}

cmd/spinifex/cmd/get.go

@@ -22,9 +22,11 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
+	"strconv"
 	"strings"
 	"time"
 
+	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/utils"
@@ -76,14 +78,20 @@ func loadConfigAndConnect() (*config.ClusterConfig, *nats.Conn, error) {
 	}
 
 	// Backfill BaseDir from config file path when not set in the config.
-	// Config lives at <baseDir>/config/spinifex.toml, so baseDir is two levels up.
 	if nodeConfig, ok := cfg.Nodes[cfg.Node]; ok && nodeConfig.BaseDir == "" {
-		nodeConfig.BaseDir = filepath.Dir(filepath.Dir(cfgPath))
+		if isProductionLayout() {
+			// Production: BaseDir is /var/lib/spinifex (data dir).
+			// /var/lib/spinifex/config is a symlink to /etc/spinifex.
+			nodeConfig.BaseDir = DefaultDataDir()
+		} else {
+			// Dev: config lives at <baseDir>/config/spinifex.toml, so baseDir is two levels up.
+			nodeConfig.BaseDir = filepath.Dir(filepath.Dir(cfgPath))
+		}
 		cfg.Nodes[cfg.Node] = nodeConfig
 	}
 
 	nodeConfig := cfg.Nodes[cfg.Node]
-	nc, err := utils.ConnectNATS(nodeConfig.NATS.Host, nodeConfig.NATS.ACL.Token)
+	nc, err := utils.ConnectNATS(admin.DialTarget(nodeConfig.NATS.Host), nodeConfig.NATS.ACL.Token, nodeConfig.NATS.CACert)
 	if err != nil {
 		return nil, nil, fmt.Errorf("failed to connect to NATS: %w", err)
 	}
@@ -215,7 +223,7 @@ func runGetNodes(cmd *cobra.Command, args []string) {
 				resp.Region,
 				resp.AZ,
 				formatUptime(resp.Uptime),
-				fmt.Sprintf("%d", resp.VMCount),
+				strconv.Itoa(resp.VMCount),
 				strings.Join(resp.Services, ","),
 			})
 		} else {
@@ -295,7 +303,7 @@ func runGetVMs(cmd *cobra.Command, args []string) {
 			v.InstanceID,
 			v.Status,
 			v.InstanceType,
-			fmt.Sprintf("%d", v.VCPU),
+			strconv.Itoa(v.VCPU),
 			formatMemGB(v.MemoryGB),
 			v.Node,
 			v.Host,

cmd/spinifex/cmd/paths.go

@@ -7,7 +7,7 @@ import (
 )
 
 // DefaultConfigDir returns the default configuration directory.
-// Production: /etc/spinifex (when running as root AND /etc/spinifex exists)
+// Production: /etc/spinifex (when /etc/spinifex exists)
 // Development: ~/spinifex/config
 func DefaultConfigDir() string {
 	if isProductionLayout() {
@@ -26,6 +26,16 @@ func DefaultDataDir() string {
 	return filepath.Join(realUserHomeDir(), "spinifex")
 }
 
+// LogDirFor returns the log directory for a given data directory.
+// Production: /var/log/spinifex (matches systemd ReadWritePaths)
+// Development: <dataDir>/logs (supports custom per-node data dirs)
+func LogDirFor(dataDir string) string {
+	if isProductionLayout() {
+		return "/var/log/spinifex"
+	}
+	return filepath.Join(dataDir, "logs")
+}
+
 // realUserHomeDir returns the home directory of the real (non-sudo) user.
 // When running under sudo, SUDO_USER is set to the invoking user — resolve
 // their home directory so config/data land in the right place.
@@ -44,14 +54,16 @@ func DefaultConfigFile() string {
 	return filepath.Join(DefaultConfigDir(), "spinifex.toml")
 }
 
+// productionMarkerPath identifies a production install by directory presence.
+// Created by setup.sh during binary install. Overridable in tests.
+var productionMarkerPath = "/etc/spinifex"
+
 // isProductionLayout returns true when running in a production install.
-// Detected by: running as root AND /etc/spinifex directory already exists
-// (created by setup.sh during binary install).
+// No root check — allows non-root users (e.g. tf-user) to run CLI commands
+// like `spx get nodes` without sudo or --config flags.
 func isProductionLayout() bool {
-	if os.Getuid() == 0 {
-		if info, err := os.Stat("/etc/spinifex"); err == nil && info.IsDir() {
-			return true
-		}
+	if info, err := os.Stat(productionMarkerPath); err == nil && info.IsDir() {
+		return true
 	}
 	return false
 }

cmd/spinifex/cmd/paths_test.go

@@ -1,57 +1,77 @@
 package cmd
 
 import (
-	"os"
 	"path/filepath"
 	"testing"
+
+	"github.com/stretchr/testify/assert"
 )
 
-func TestDefaultConfigDir_Dev(t *testing.T) {
-	// Non-root user without /etc/spinifex should get dev paths
-	if os.Getuid() == 0 {
-		t.Skip("test requires non-root user")
-	}
-
-	homeDir, _ := os.UserHomeDir()
-	expected := filepath.Join(homeDir, "spinifex", "config")
-	got := DefaultConfigDir()
-	if got != expected {
-		t.Errorf("DefaultConfigDir() = %q, want %q", got, expected)
-	}
+// withProductionMarker temporarily overrides the marker path used to detect a
+// production install. Tests that pass an empty string force "dev" mode by
+// pointing at a non-existent path; tests that pass t.TempDir() force
+// "production" mode by pointing at a real directory.
+func withProductionMarker(t *testing.T, path string) {
+	t.Helper()
+	orig := productionMarkerPath
+	productionMarkerPath = path
+	t.Cleanup(func() { productionMarkerPath = orig })
 }
 
-func TestDefaultDataDir_Dev(t *testing.T) {
-	if os.Getuid() == 0 {
-		t.Skip("test requires non-root user")
-	}
-
-	homeDir, _ := os.UserHomeDir()
-	expected := filepath.Join(homeDir, "spinifex")
-	got := DefaultDataDir()
-	if got != expected {
-		t.Errorf("DefaultDataDir() = %q, want %q", got, expected)
-	}
+func TestIsProductionLayout(t *testing.T) {
+	t.Run("absent marker returns false", func(t *testing.T) {
+		withProductionMarker(t, filepath.Join(t.TempDir(), "does-not-exist"))
+		assert.False(t, isProductionLayout())
+	})
+	t.Run("present marker returns true", func(t *testing.T) {
+		withProductionMarker(t, t.TempDir())
+		assert.True(t, isProductionLayout())
+	})
 }
 
-func TestDefaultConfigFile(t *testing.T) {
-	if os.Getuid() == 0 {
-		t.Skip("test requires non-root user")
+func TestDefaultPaths(t *testing.T) {
+	homeDir := t.TempDir()
+	t.Setenv("HOME", homeDir)
+	t.Setenv("SUDO_USER", "")
+
+	tests := []struct {
+		name           string
+		productionMode bool
+		wantConfigDir  string
+		wantDataDir    string
+		wantConfigFile string
+		wantLogDir     string // for LogDirFor("/data/dir")
+	}{
+		{
+			name:           "dev layout",
+			productionMode: false,
+			wantConfigDir:  filepath.Join(homeDir, "spinifex", "config"),
+			wantDataDir:    filepath.Join(homeDir, "spinifex"),
+			wantConfigFile: filepath.Join(homeDir, "spinifex", "config", "spinifex.toml"),
+			wantLogDir:     "/data/dir/logs",
+		},
+		{
+			name:           "production layout",
+			productionMode: true,
+			wantConfigDir:  "/etc/spinifex",
+			wantDataDir:    "/var/lib/spinifex",
+			wantConfigFile: "/etc/spinifex/spinifex.toml",
+			wantLogDir:     "/var/log/spinifex",
+		},
 	}
 
-	got := DefaultConfigFile()
-	homeDir, _ := os.UserHomeDir()
-	expected := filepath.Join(homeDir, "spinifex", "config", "spinifex.toml")
-	if got != expected {
-		t.Errorf("DefaultConfigFile() = %q, want %q", got, expected)
-	}
-}
-
-func TestIsProductionLayout_NonRoot(t *testing.T) {
-	if os.Getuid() == 0 {
-		t.Skip("test requires non-root user")
-	}
-
-	if isProductionLayout() {
-		t.Error("isProductionLayout() = true for non-root user, want false")
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if tt.productionMode {
+				withProductionMarker(t, t.TempDir())
+			} else {
+				withProductionMarker(t, filepath.Join(t.TempDir(), "absent"))
+			}
+
+			assert.Equal(t, tt.wantConfigDir, DefaultConfigDir())
+			assert.Equal(t, tt.wantDataDir, DefaultDataDir())
+			assert.Equal(t, tt.wantConfigFile, DefaultConfigFile())
+			assert.Equal(t, tt.wantLogDir, LogDirFor("/data/dir"))
+		})
 	}
 }

cmd/spinifex/cmd/service.go

@@ -22,7 +22,6 @@ import (
 	"os"
 	"strings"
 
-	"github.com/mulgadc/predastore/s3"
 	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/service"
 	"github.com/mulgadc/spinifex/spinifex/services/awsgw"
@@ -116,13 +115,11 @@ var predastoreStartCmd = &cobra.Command{
 			return
 		}
 
-		backendType := viper.GetString("backend")
-		var backend s3.BackendType
+		encryptionKeyFile := viper.GetString("encryption-key-file")
 
-		if backendType == "distributed" {
-			backend = s3.BackendDistributed
-		} else {
-			backend = s3.BackendFilesystem
+		if encryptionKeyFile == "" {
+			fmt.Println("Encryption key file is not set")
+			return
 		}
 
 		nodeID := viper.GetInt("node-id")
@@ -138,8 +135,9 @@ var predastoreStartCmd = &cobra.Command{
 			TlsCert:    tlsCert,
 			TlsKey:     tlsKey,
 
-			Backend: backend,
-			NodeID:  nodeID,
+			EncryptionKeyFile: encryptionKeyFile,
+
+			NodeID: nodeID,
 
 			PprofEnabled:    pprofEnabled,
 			PprofOutputPath: pprofOutput,
@@ -245,13 +243,13 @@ var viperblockStartCmd = &cobra.Command{
 
 		accessKey := viper.GetString("access-key")
 		if accessKey != "" {
-			fmt.Println("Overwriting access-key:", accessKey)
+			fmt.Println("Overwriting access-key: ****")
 			nodeConfig.Predastore.AccessKey = accessKey
 		}
 
 		secretKey := viper.GetString("secret-key")
 		if secretKey != "" {
-			fmt.Println("Overwriting secret-key:", secretKey)
+			fmt.Println("Overwriting secret-key: ****")
 			nodeConfig.Predastore.SecretKey = secretKey
 		}
 
@@ -287,6 +285,8 @@ var viperblockStartCmd = &cobra.Command{
 
 		service, err := service.New("viperblock", &viperblockd.Config{
 			NatsHost:   nodeConfig.NATS.Host,
+			NatsToken:  nodeConfig.NATS.ACL.Token,
+			NatsCACert: nodeConfig.NATS.CACert,
 			PluginPath: pluginPath,
 			S3Host:     nodeConfig.Predastore.Host,
 			Bucket:     nodeConfig.Predastore.Bucket,
@@ -310,7 +310,7 @@ var viperblockStartCmd = &cobra.Command{
 			return
 		}
 
-		fmt.Println("Viperblock service started", service)
+		fmt.Println("Viperblock service started")
 	},
 }
 
@@ -677,6 +677,32 @@ var spinifexUIStatusCmd = &cobra.Command{
 	},
 }
 
+// checkLegacyWanBridgeKey fails vpcd startup if the deprecated `wan_bridge`
+// TOML key or `SPINIFEX_VPCD_WAN_BRIDGE` env-var is present. The key was
+// renamed to `dhcp_bind_bridge` (see mulga-998.a) because the old name misled
+// operators into pointing at the OVN-side bridge ("br-ext"), which never sees
+// LAN DHCP traffic. Per D3: no silent alias, no auto-rewrite — operator must
+// rename the key before vpcd will start.
+func checkLegacyWanBridgeKey(node, cfgFile string) error {
+	legacyInTOML := viper.IsSet("nodes." + node + ".vpcd.wan_bridge")
+	legacyInEnv := os.Getenv("SPINIFEX_VPCD_WAN_BRIDGE") != ""
+	if !legacyInTOML && !legacyInEnv {
+		return nil
+	}
+	source := cfgFile
+	if legacyInEnv {
+		source = "env SPINIFEX_VPCD_WAN_BRIDGE"
+	}
+	return fmt.Errorf(
+		"vpcd: deprecated 'wan_bridge' key found in %s. "+
+			"Rename 'wan_bridge' to 'dhcp_bind_bridge'. The value may also be wrong — verify it is "+
+			"the Linux bridge holding your WAN NIC (veth mode) or the OVS bridge holding your WAN NIC (direct mode); "+
+			"never 'br-ext'. Typical value on a consumer-router LAN: 'br-wan'. "+
+			"Then: sudo systemctl restart spinifex-vpcd",
+		source,
+	)
+}
+
 var vpcdStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the vpcd service",
@@ -695,32 +721,33 @@ var vpcdStartCmd = &cobra.Command{
 			return
 		}
 
+		if err := checkLegacyWanBridgeKey(clusterConfig.Node, cfgFile); err != nil {
+			slog.Error(err.Error())
+			os.Exit(1)
+		}
+
 		nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
 
 		// Map cluster-wide external pools to vpcd config
 		var extPools []vpcd.ExternalPoolConfig
 		for _, p := range clusterConfig.Network.ExternalPools {
 			extPools = append(extPools, vpcd.ExternalPoolConfig{
-				Name:       p.Name,
-				RangeStart: p.RangeStart,
-				RangeEnd:   p.RangeEnd,
-				Gateway:    p.Gateway,
-				GatewayIP:  p.GatewayIP,
-				PrefixLen:  p.PrefixLen,
-				DNSServers: p.DNSServers,
-				Region:     p.Region,
-				AZ:         p.AZ,
+				Name:            p.Name,
+				Source:          p.Source,
+				RangeStart:      p.RangeStart,
+				RangeEnd:        p.RangeEnd,
+				Gateway:         p.Gateway,
+				GatewayIP:       p.GatewayIP,
+				PrefixLen:       p.PrefixLen,
+				DNSServers:      p.DNSServers,
+				Region:          p.Region,
+				AZ:              p.AZ,
+				DhcpBindBridge:  nodeConfig.VPCD.DhcpBindBridge,
+				GwLrpRangeStart: p.GwLrpRangeStart,
+				GwLrpRangeEnd:   p.GwLrpRangeEnd,
 			})
 		}
 
-		// Fallback chassis names derived from config node names. vpcd prefers
-		// discovering chassis from the OVN Southbound DB at startup; these are
-		// only used if that discovery fails.
-		var chassisNames []string
-		for nodeName := range clusterConfig.Nodes {
-			chassisNames = append(chassisNames, "chassis-"+nodeName)
-		}
-
 		var bootstrap *vpcd.BootstrapVPC
 		if clusterConfig.Bootstrap.VpcId != "" {
 			bootstrap = &vpcd.BootstrapVPC{
@@ -733,19 +760,25 @@ var vpcdStartCmd = &cobra.Command{
 			}
 		}
 
+		baseDir := viper.GetString("base-dir")
+		if baseDir != "" {
+			fmt.Println("Overwriting vpcd base-dir to:", baseDir)
+			nodeConfig.BaseDir = baseDir
+		}
+
 		svc, err := service.New("vpcd", &vpcd.Config{
 			NatsHost:          nodeConfig.NATS.Host,
 			NatsToken:         nodeConfig.NATS.ACL.Token,
+			NatsCACert:        nodeConfig.NATS.CACert,
 			OVNNBAddr:         nodeConfig.VPCD.OVNNBAddr,
 			OVNSBAddr:         nodeConfig.VPCD.OVNSBAddr,
 			BaseDir:           nodeConfig.BaseDir,
 			Debug:             false,
 			ExternalMode:      clusterConfig.Network.ExternalMode,
 			ExternalPools:     extPools,
-			ChassisNames:      chassisNames,
 			Bootstrap:         bootstrap,
 			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			WanBridge:         nodeConfig.VPCD.WanBridge,
+			DhcpBindBridge:    nodeConfig.VPCD.DhcpBindBridge,
 			BridgeMode:        nodeConfig.VPCD.BridgeMode,
 		})
 		if err != nil {
@@ -834,13 +867,20 @@ func init() {
 	viper.BindEnv("tls-key", "SPINIFEX_PREDASTORE_TLS_KEY")
 	viper.BindPFlag("tls-key", predastoreCmd.PersistentFlags().Lookup("tls-key"))
 
+	// Predastore at-rest encryption master key (per node; mode 0600)
+	predastoreCmd.PersistentFlags().String("encryption-key-file", "", "Path to this node's 32-byte AES-256 master key file (required)")
+	viper.BindEnv("encryption-key-file", "SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE")
+	viper.BindPFlag("encryption-key-file", predastoreCmd.PersistentFlags().Lookup("encryption-key-file"))
+
 	// Predastore Backend
 	predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
 	viper.BindEnv("backend", "SPINIFEX_PREDASTORE_BACKEND")
 	viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
 
-	// Predastore Node ID
-	predastoreCmd.PersistentFlags().Int("node-id", 0, "Predastore (S3) node ID")
+	// Predastore Node ID. Default -1 is dev mode (launch every configured
+	// QUIC node in-process). Production deployments set this to the node's
+	// real ID (>= 1) via SPINIFEX_PREDASTORE_NODE_ID or --node-id.
+	predastoreCmd.PersistentFlags().Int("node-id", -1, "Predastore (S3) node ID (-1 = dev mode, >= 1 = production)")
 	viper.BindEnv("node-id", "SPINIFEX_PREDASTORE_NODE_ID")
 	viper.BindPFlag("node-id", predastoreCmd.PersistentFlags().Lookup("node-id"))
 
@@ -860,7 +900,7 @@ func init() {
 
 	serviceCmd.AddCommand(viperblockCmd)
 
-	viperblockCmd.PersistentFlags().String("s3-host", "0.0.0.0:8443", "Predastore (S3) host URI")
+	viperblockCmd.PersistentFlags().String("s3-host", "", "Predastore (S3) host URI")
 	viper.BindEnv("s3-host", "SPINIFEX_VIPERBLOCK_S3_HOST")
 	viper.BindPFlag("s3-host", predastoreCmd.PersistentFlags().Lookup("s3-host"))
 

cmd/spinifex/cmd/templates/awsgw.toml

@@ -1,6 +1,28 @@
 # AWS Gateway configuration file in TOML format.
-version = "1.0"
+version = "2"
 region = "{{.Region}}"
 
 # Enable verbose debugging
 debug = false
+
+# API request throttling (per-account, per-action token bucket)
+[ratelimit]
+enabled = false
+rate = 20           # Default sustained requests per second per account per action
+burst = 100         # Default maximum burst capacity
+
+[ratelimit.action.RunInstances]
+rate = 2
+burst = 40
+
+[ratelimit.action.TerminateInstances]
+rate = 2
+burst = 40
+
+[ratelimit.action.StartInstances]
+rate = 2
+burst = 40
+
+[ratelimit.action.StopInstances]
+rate = 2
+burst = 40

cmd/spinifex/cmd/templates/nats.conf

@@ -1,3 +1,4 @@
+# spinifex-config-version: 3
 # NATS Server Configuration
 # Generated by Spinifex admin init
 
@@ -8,15 +9,28 @@ listen: {{.BindIP}}:4222
 listen: 0.0.0.0:4222
 {{- end }}
 
+tls {
+  cert_file: "{{.ConfigDir}}/server.pem"
+  key_file:  "{{.ConfigDir}}/server.key"
+  ca_file:   "{{.ConfigDir}}/ca.pem"
+}
+
 {{- if and .ClusterBindIP .ClusterRoutes }}
 
 cluster {
   name: {{ .ClusterName }}
   listen: {{.ClusterBindIP}}:4248
 
+  tls {
+    cert_file: "{{.ConfigDir}}/server.pem"
+    key_file:  "{{.ConfigDir}}/server.key"
+    ca_file:   "{{.ConfigDir}}/ca.pem"
+    verify:    true
+  }
+
   routes = [
 {{- range .ClusterRoutes }}
-    "nats://{{ . }}",
+    "nats-route://{{$.NatsToken}}@{{ . }}",
 {{- end }}
   ]
 }
@@ -26,7 +40,7 @@ server_name: spinifex-nats-{{.Node}}
 
 # Authorization
 authorization {
-#  token: "{{.NatsToken}}"
+  token: "{{.NatsToken}}"
 }
 
 # JetStream for persistent messaging and KV store
@@ -40,7 +54,7 @@ jetstream {
 debug: false
 trace: false
 logtime: true
-log_file: "{{.DataDir}}/logs/nats.log"
+log_file: "{{.LogDir}}/nats.log"
 
 # Limits
 max_connections: 1000
@@ -48,9 +62,5 @@ max_payload: 1MB
 max_pending: 64MB
 
 # Monitoring
-{{- if .BindIP }}
-http: {{.BindIP}}:8222
-{{- else }}
-http_port: 8222
-{{- end }}
+http: 127.0.0.1:8222
 

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -1,6 +1,6 @@
 # Predastore S3-compatible storage configuration file in TOML format.
 # Generated for multi-node cluster
-version = "1.0"
+version = "4"
 region = "{{.Region}}"
 
 host = "0.0.0.0"
@@ -9,8 +9,13 @@ port = 8443
 debug = false
 
 [rs]
+{{- if eq (len .Nodes) 2}}
+data = 1
+parity = 1
+{{- else}}
 data = 2
 parity = 1
+{{- end}}
 {{- range .Nodes}}
 
 [[db]]
@@ -39,17 +44,34 @@ region = "{{.Region}}"
 type = "distributed"
 public = false
 encryption = ""
+account_id = "000000000000"
 
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
+account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
 ]
 
+# API request throttling (per-account, per-action token bucket)
+[ratelimit]
+enabled = false
+rate = 100          # Default sustained requests per second per account per action
+burst = 500         # Default maximum burst capacity
+
+[ratelimit.action."s3:PutObject"]
+rate = 50
+burst = 200
+
+[ratelimit.action."s3:DeleteObject"]
+rate = 50
+burst = 200
+
 # IAM authentication via NATS KV (enables multi-account S3 access)
 [iam]
-nats_url = "nats://{{.BindIP}}:4222"
+nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}}:4222"
 nats_token = "{{.NatsToken}}"
+nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
 access_keys_bucket = "spinifex-iam-access-keys"

cmd/spinifex/cmd/templates/predastore.toml

@@ -1,6 +1,6 @@
 # Predastore S3-compatible storage configuration file in TOML format.
 # MULTI NODE used for testing
-version = "1.0"
+version = "4"
 region = "{{.Region}}"
 
 # Optionally configure port and host, or specify via daemon CLI options
@@ -73,17 +73,34 @@ region = "ap-southeast-2"
 type = "distributed"
 public = false
 encryption = ""
+account_id = "000000000000"
 
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
+account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
 ]
 
+# API request throttling (per-account, per-action token bucket)
+[ratelimit]
+enabled = false
+rate = 100          # Default sustained requests per second per account per action
+burst = 500         # Default maximum burst capacity
+
+[ratelimit.action."s3:PutObject"]
+rate = 50
+burst = 200
+
+[ratelimit.action."s3:DeleteObject"]
+rate = 50
+burst = 200
+
 # IAM authentication via NATS KV (enables multi-account S3 access)
 [iam]
 nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}}:4222"
 nats_token = "{{.NatsToken}}"
+nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
 access_keys_bucket = "spinifex-iam-access-keys"

cmd/spinifex/cmd/templates/spinifex.toml

@@ -1,7 +1,12 @@
 # Domain configuration file in TOML format.
-version = "1.0"
+version = "2"
 epoch = 1
 node = "{{.Node}}"
+{{- if .OperatorEmail}}
+
+[operator]
+email = "{{.OperatorEmail}}"
+{{- end}}
 {{- if .ExternalMode}}
 
 [network]
@@ -42,8 +47,12 @@ subnet_cidr = "{{.BootstrapSubnetCidr}}"
 [nodes.{{.Node}}]
 node = "{{.Node}}"
 host = "{{.BindIP}}"
+{{- if .AdvertiseIP}}
+advertise = "{{.AdvertiseIP}}"
+{{- end}}
 region = "{{.Region}}"
 az = "{{.Az}}"
+data_dir = "{{.DataDir}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
@@ -59,6 +68,9 @@ dev_networking = false
 {{- else}}
 dev_networking = true
 {{- end}}
+{{- if .GPUPassthrough}}
+gpu_passthrough = true
+{{- end}}
 
 [nodes.{{.Node}}.awsgw]
 host = "{{.BindIP}}:9999"
@@ -69,6 +81,7 @@ config = "awsgw/awsgw.toml"
 
 [nodes.{{.Node}}.nats]
 host = "{{.BindIP}}:4222"
+cacert = "{{.ConfigDir}}/ca.pem"
 
 [nodes.{{.Node}}.nats.acl]
 token = "{{.NatsToken}}"
@@ -84,8 +97,14 @@ ovn_sb_addr = "{{.OVNSBAddr}}"
 {{- if .ExternalIface}}
 external_interface = "{{.ExternalIface}}"
 {{- end}}
-{{- if .WanBridge}}
-wan_bridge = "{{.WanBridge}}"
+{{- if .DhcpBindBridge}}
+# dhcp_bind_bridge: the bridge where vpcd's DHCP client binds its AF_PACKET
+# socket. On Linux-bridge-WAN hosts (netplan default with the WAN NIC enslaved),
+# this is the Linux bridge name (e.g. "br-wan"). On direct-OVS hosts, this is
+# the OVS bridge holding the WAN NIC.
+# NEVER set this to "br-ext" — that is the OVN-side bridge and never sees LAN
+# DHCP traffic.
+dhcp_bind_bridge = "{{.DhcpBindBridge}}"
 {{- end}}
 
 [nodes.{{.Node}}.predastore]

cmd/spinifex/cmd/top.go

@@ -21,6 +21,7 @@ import (
 	"fmt"
 	"os"
 	"sort"
+	"strconv"
 	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/types"
@@ -87,7 +88,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 	sort.Strings(nodeNames)
 
 	nodeTable := pterm.TableData{
-		{"NAME", "CPU (used/total)", "MEM (used/total)", "VMs"},
+		{"NAME", "CPU (used/total)", "MEM (used/total)", "GPU (used/total)", "VMs"},
 	}
 
 	// Aggregate instance type capacity across all nodes
@@ -95,11 +96,18 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 
 	for _, name := range nodeNames {
 		if resp, ok := respondedNodes[name]; ok {
+			gpuCol := "-"
+			if resp.GPUPassthrough {
+				gpuCol = fmt.Sprintf("%d/%d", resp.AllocGPUs, resp.TotalGPUs)
+			} else if resp.GPUCapable {
+				gpuCol = fmt.Sprintf("0/%d*", len(resp.GPUModels))
+			}
 			nodeTable = append(nodeTable, []string{
 				resp.Node,
 				fmt.Sprintf("%d/%d", resp.AllocVCPU, resp.TotalVCPU),
 				fmt.Sprintf("%s/%s", formatMemGB(resp.AllocMemGB), formatMemGB(resp.TotalMemGB)),
-				fmt.Sprintf("%d", resp.VMCount),
+				gpuCol,
+				strconv.Itoa(resp.VMCount),
 			})
 
 			for _, cap := range resp.InstanceTypes {
@@ -119,6 +127,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 				"-",
 				"-",
 				"-",
+				"-",
 			})
 		}
 	}
@@ -146,8 +155,8 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 		agg := capacityMap[name]
 		capTable = append(capTable, []string{
 			name,
-			fmt.Sprintf("%d", agg.Available),
-			fmt.Sprintf("%d", agg.VCPU),
+			strconv.Itoa(agg.Available),
+			strconv.Itoa(agg.VCPU),
 			formatMemGB(agg.MemoryGB),
 		})
 	}

cmd/spinifex/cmd/version_test.go

@@ -1,29 +0,0 @@
-package cmd
-
-import (
-	"testing"
-)
-
-func TestVersionVarsExist(t *testing.T) {
-	// Verify the version variables are set (defaults when not injected via ldflags)
-	if Version == "" {
-		t.Error("Version should not be empty")
-	}
-	if Commit == "" {
-		t.Error("Commit should not be empty")
-	}
-}
-
-func TestVersionCommand(t *testing.T) {
-	// Verify the version command is registered
-	found := false
-	for _, cmd := range rootCmd.Commands() {
-		if cmd.Use == "version" {
-			found = true
-			break
-		}
-	}
-	if !found {
-		t.Error("version command not registered on rootCmd")
-	}
-}

cmd/spinifex/main.go

@@ -16,7 +16,10 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
 package main
 
-import "github.com/mulgadc/spinifex/cmd/spinifex/cmd"
+import (
+	"github.com/mulgadc/spinifex/cmd/spinifex/cmd"
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
 
 func main() {
 	cmd.Execute()

docs/COMMANDS.md

@@ -13,7 +13,7 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp
 | `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | **DONE** |
 | `spx service predastore stop` | — | Predastore must be running | Stops the predastore service | **DONE** |
 | `spx service predastore status` | — | None | Reports predastore service status | **DONE** |
-| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (default: /opt/spinifex/lib/nbdkit-viperblock-plugin.so) | `--config` required, plugin-path must exist on disk | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | **DONE** |
+| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | `--config` required, plugin-path must exist on disk | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | **DONE** |
 | `spx service viperblock stop` | — | Viperblock must be running | Stops the viperblock service | **DONE** |
 | `spx service viperblock status` | — | None | Reports viperblock service status | **DONE** |
 | `spx service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | None | Starts embedded NATS server with optional JetStream | **DONE** |
@@ -45,20 +45,38 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Output | Status |
 |---------|-------|---------------|-------------|--------|--------|
-| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
+| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows `allocated/total` when passthrough is enabled, `0/count*` when the node is GPU-capable but passthrough is disabled, or `-` when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
 
 ### Cluster Initialization
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------|---------------|-------------|------------|--------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--force` | None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` | 1. Init creates all config files<br>2. Root credentials printed once<br>3. master.key is 32 bytes, mode 0600<br>4. bootstrap.json consumed on first start<br>5. `--force` re-initializes existing config<br>6. AWS CLI profile `spx` auto-configured | **DONE** |
-| `spx admin join` | `--host` (required), `--node` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Leader node must be running | Connects to leader node → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster<br>2. Missing host (error)<br>3. Missing node name (error) | **DONE** |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` | 1. Init creates all config files<br>2. Root credentials printed once<br>3. master.key is 32 bytes, mode 0600<br>4. bootstrap.json consumed on first start<br>5. `--force` re-initializes existing config<br>6. AWS CLI profile `spx` auto-configured<br>7. Multi-node init generates join token and writes to `<config-dir>/join-token` | **DONE** |
+| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Leader node must be running | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster<br>2. Missing host (error)<br>3. Missing node name (error)<br>4. Missing token (error)<br>5. Invalid token (401)<br>6. Expired token (401) | **DONE** |
+
+### Version
+
+| Command | Flags | Prerequisites | Basic Logic | Status |
+|---------|-------|---------------|-------------|--------|
+| `spx version` | — | None | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) | **DONE** |
 
 ### Cluster Operations
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------|---------------|-------------|------------|--------|
-| `spx admin cluster shutdown` | — | Cluster must be running | Gracefully shuts down the entire cluster | 1. Shutdown running cluster<br>2. All nodes stop cleanly | **DONE** |
+| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Cluster must be running | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. | 1. Shutdown running cluster<br>2. All nodes stop cleanly<br>3. Force shutdown with unresponsive nodes<br>4. Dry-run prints plan | **DONE** |
+
+### Certificate Management
+
+| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------|---------------|-------------|------------|--------|
+| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Existing CA from `spx admin init` | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. | 1. Renew with auto-detected IPs<br>2. Renew with extra IPs<br>3. Renew with extra DNS names | **DONE** |
+
+### Upgrade Management
+
+| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------|---------------|-------------|------------|--------|
+| `spx admin upgrade` | `--yes` (apply without prompting), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Existing installation (`spinifex.toml` must exist in config dir) | Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless `--yes` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target`. Invoked non-interactively by `scripts/setup.sh` with `--yes`. | 1. No installation present (error, suggests `spx admin init`)<br>2. No pending migrations (prints "No pending config migrations")<br>3. Pending migrations with interactive confirmation (y/yes applies, anything else aborts)<br>4. Pending migrations with `--yes` (applies without prompt)<br>5. Migration failure surfaces error and exits non-zero | **DONE** |
 
 ### Account Management
 
@@ -71,9 +89,49 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------|---------------|-------------|------------|--------|
-| `spx admin images import` | `--name` | Cluster must be running, image file must exist locally | Imports a local disk image (QCOW2) as an AMI → uploads to Predastore S3 → registers as available AMI | 1. Import valid QCOW2 image<br>2. Image appears in `describe-images` | **DONE** |
+| `spx admin images import` | `--name`, `--file`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, and verify the SHA-256/SHA-512 digest before extraction. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip verification — operator-supplied media is outside Spinifex's trust boundary, and the skip is logged at INFO for audit. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN | **DONE** |
 | `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
 
+#### Image integrity verification (CMMC SI.L1-3.14.2)
+
+Catalog imports (`spx admin images import --name <name>`) verify the image
+against the catalog-declared SHA-256/SHA-512 digest before extraction. The sums
+file is fetched from the catalog `Checksum` URL over HTTPS only (cross-scheme
+redirects refused), and verification runs on both fresh downloads and cache
+hits so a poisoned cache is caught on the next import.
+
+On mismatch the import exits non-zero, the cached file is left on disk for
+inspection, and the printed guidance is `spx admin images import --name <name>
+--force` to re-download.
+
+`--file` imports skip verification by design: operator-supplied media is
+outside Spinifex's trust boundary and the operator is responsible for
+integrity (e.g. `sha256sum` against a trusted upstream digest before import).
+The skip is recorded as an INFO `slog` event with `reason=local-file-import`
+so a CMMC assessor can audit the decision from journald.
+
+`--skip-verify` bypasses the checksum step for catalog imports. The command
+still downloads via the catalog URL but does not compare the image digest
+against the sums file. Intended for narrow cases such as debugging upstream
+mirror issues or running against a transiently-broken `latest/` path; the
+skip is logged at WARN with `reason=skip-verify-flag` and printed to stderr
+so operators and assessors see it. Prefer `--file` with an out-of-band
+verified image over `--skip-verify` whenever possible.
+
+**Limitation:** verification confirms the image matches the digest the mirror
+served. A mirror compromise that swaps both image and sums file is not
+detected; closing that gap requires GPG signature verification of the sums
+file, deferred to a later phase.
+
+### GPU Management
+
+| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------|---------------|-------------|------------|--------|
+| `spx admin gpu status` | `--node` (default: local node) | Cluster must be running (NATS) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. | 1. Local node with passthrough enabled (shows pool and instance types)<br>2. Local node capable but passthrough disabled (shows "run enable")<br>3. Node without GPU hardware (shows "none detected")<br>4. `--node` targets a remote node<br>5. Node not found or not responding (error) | **DONE** |
+| `spx admin gpu enable` | — | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. | 1. Enable on capable node<br>2. Already enabled (no-op)<br>3. Prerequisites not met (error, suggests `setup`)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
+| `spx admin gpu disable` | — | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. | 1. Disable with no GPU instances running<br>2. Already disabled (no-op)<br>3. GPU instances running (error with count)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
+| `spx admin gpu setup` | — | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. | 1. First run on unconfigured host (GRUB updated, reboot required)<br>2. Re-run after reboot (verifies bindings, enables passthrough)<br>3. Already fully configured (all steps skipped, enables passthrough)<br>4. AMD GPU present (blacklists amdgpu in addition to nouveau)<br>5. Not run as root (error)<br>6. No GPU hardware detected (error) | **DONE** |
+
 ## AWS Commands
 
 ### EC2 - Instance Management
@@ -81,16 +139,17 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `run-instances` | `--image-id`, `--instance-type`, `--count` (Min/MaxCount), `--key-name`, `--user-data`, `--subnet-id` (auto-creates ENI, assigns private IP), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName only — routes via spread or cluster strategy) | `--security-group-ids`, `--tag-specifications`, `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--iam-instance-profile`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | `describe-images` (AMI must exist), `create-key-pair` (optional), VPC/SG (optional) | Gateway parses AWS query → if Placement.GroupName set, looks up strategy: spread → `distributeInstancesSpread()` (1 instance per node, atomic CAS reservation), cluster → `distributeInstancesCluster()` (pin all to single node); otherwise NATS `ec2.runinstances` → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → if SubnetId provided, auto-creates ENI with private IP → cloud-init injects user-data/keys → on termination, removes instance from placement group → returns reservation with instance ID | 1. Launch with valid AMI and key pair<br>2. Launch with invalid AMI ID (error)<br>3. Launch with block device mappings (custom volume size)<br>4. Launch multiple instances (MinCount/MaxCount)<br>5. Launch with subnet-id (auto-creates ENI)<br>6. Invalid instance type returns error<br>7. Launch with spread placement group (1 per node)<br>8. Launch with cluster placement group (all on one node)<br>9. Insufficient capacity for placement group (error) | **DONE** |
-| `describe-instances` | `--instance-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes | **DONE** |
+| `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:\*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list. Filters applied per-node before aggregation (reduces payload). Also applies to stopped/terminated instances via `describeInstancesFromKV()`. | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes<br>6. Filter by tag<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | `run-instances` (instance must exist in stopped state) | Gateway sends NATS `ec2.cmd.{instance-id}` → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running | 1. Start a stopped instance<br>2. Start already-running instance (error: IncorrectInstanceState)<br>3. Start with invalid instance ID<br>4. Verify volumes re-mount on start | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_powerdown` for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped | 1. Graceful stop of running instance<br>2. Force stop (kills QEMU process)<br>3. Stop already-stopped instance (error)<br>4. Verify ~30s heartbeat detection | **DONE** |
-| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID | **DONE** |
+| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon checks `DisableApiTermination` first and rejects with `OperationNotPermitted` if set (running or stopped, no side effects) → kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID<br>8. Running protected instance rejected (OperationNotPermitted, state stays running)<br>9. Stopped protected instance rejected (resources intact) | **DONE** |
 | `reboot-instances` | `--instance-ids` | `--dry-run` | `run-instances` (instance must be running) | Gateway validates instance IDs → sends EC2InstanceCommand with `RebootInstance=true` via NATS `ec2.cmd.{instanceId}` → daemon validates instance is in StateRunning (returns IncorrectInstanceState if stopped) → issues QMP `system_reset` → instance reboots without stopping, stays in running state | 1. Reboot running instance<br>2. Reboot multiple instances<br>3. Reboot stopped instance (error: IncorrectInstanceState)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Verify instance stays in running state after reboot | **DONE** |
 | `describe-instance-types` | `--filters` (capacity filter only) | `--instance-types`, `--max-results`, `--next-token`, `--dry-run`, all other filters | None | Gateway fans out NATS `ec2.DescribeInstanceTypes` to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns | 1. List all instance types<br>2. Filter by specific type<br>3. Filter with `capacity=true` shows available slots<br>4. Verify vCPU/memory specs match hardware | **DONE** |
-| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data` | `--disable-api-termination`, `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must be stopped (in NATS KV) | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Modify running instance (error: NotFound — running instances not in KV)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Instance not stopped (error: IncorrectInstanceState)<br>6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>7. StateReason cleared on type change (recovery from capacity-unavailable)<br>8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>9. No attribute set (error: InvalidParameterValue)<br>10. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
+| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data`, `--disable-api-termination` | `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must exist; `instance-type` and `user-data` require stopped state, `disable-api-termination` works on running or stopped | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon dispatches by state: stopped instances loaded from JetStream KV, running instances updated via `vm.Manager.UpdateAndPersist` (atomic flag check + persist under one lock to prevent Modify/Terminate race) → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). **DisableApiTermination**: stamps `RunInstancesInput.DisableApiTermination`; flag survives stop/start and daemon restarts. No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Toggle disable-api-termination on running instance<br>4. Toggle disable-api-termination on stopped instance<br>5. Modify instance-type on running instance (error: NotFound — running instances not in KV)<br>6. Instance not found (error: InvalidInstanceID.NotFound)<br>7. Instance not stopped for type/user-data change (error: IncorrectInstanceState)<br>8. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>9. StateReason cleared on type change (recovery from capacity-unavailable)<br>10. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>11. No attribute set (error: InvalidParameterValue)<br>12. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
 | `get-console-output` | `--instance-id` | `--latest` (always returns latest), `--dry-run` | Instance must be running on a node | Gateway sends NATS `ec2.{instanceId}.GetConsoleOutput` (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). | 1. Get output from running instance<br>2. Empty log file returns empty output<br>3. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
-| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`) return real values; unstored attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, `disableApiTermination`=false, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get default disableApiTermination<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
+| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`, `disableApiTermination`) return real values from `RunInstancesInput`; remaining attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get disableApiTermination (reflects RunInstances flag + Modify updates)<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
 | `describe-instance-credit-specifications` | `--instance-ids` | `--filters`, `--max-results`, `--dry-run` | None | Gateway-only stub — returns `CpuCredits: "standard"` for each requested instance ID. No daemon round-trip. T-series credit mode is not persisted. | 1. Get credit spec for T-series instance<br>2. Multiple instance IDs | **DONE** |
+| `describe-instance-status` | `--instance-ids`, `--include-all-instances`, `--filters` (availability-zone, instance-state-code, instance-state-name, tag:\*) | `--max-results`, `--next-token`, `--dry-run` (silently ignored), `event.*` / `instance-status.*` / `system-status.*` filters (rejected with InvalidParameterValue) | None | Gateway fans out NATS `ec2.DescribeInstanceStatus` to all nodes (no queue group). Each daemon validates input (i- prefix, filter names), iterates local VMs and renders InstanceStatus: `Running`→ok/passed, everything-else-included→not-applicable. When `IncludeAllInstances=true`, gateway also queries `ec2.DescribeStoppedInstances` in parallel and transforms reservation entries into InstanceStatus with `not-applicable`. Aggregated results dedup'd by InstanceId (running > stopped). Terminated and error-state instances never returned. No real health probing — static values reflect Mulga's lifecycle state only. | 1. Status of running instance (ok/passed)<br>2. Filter by instance ID<br>3. Default excludes stopped instances<br>4. IncludeAllInstances=true includes stopped (not-applicable)<br>5. Cross-account instance excluded<br>6. Unknown filter returns InvalidParameterValue<br>7. Malformed instance ID rejected<br>8. Tag filter narrows correctly | **DONE** |
 | `monitor-instances` | — | `--instance-ids` | Instance must exist | Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state | 1. Enable monitoring<br>2. Verify monitoring state in describe-instances | **NOT STARTED** |
 | `unmonitor-instances` | — | `--instance-ids` | Instance must exist | Disable detailed monitoring → revert to basic monitoring → return monitoring state | 1. Disable monitoring<br>2. Verify monitoring state reverts in describe-instances | **NOT STARTED** |
 
@@ -99,7 +158,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `create-key-pair` | `--key-name`, `--key-type` (rsa/ed25519) | `--key-format` (pem/ppk), `--tag-specifications`, `--dry-run` | None | NATS `ec2.CreateKeyPair` → daemon generates SSH keypair → stores public key in Predastore S3 (`/bucket/ec2/{name}.pub`) → returns private key material (one-time) | 1. Create RSA key pair<br>2. Create ED25519 key pair<br>3. Duplicate key name (error: InvalidKeyPair.Duplicate)<br>4. Verify key material returned only on creation | **DONE** |
-| `describe-key-pairs` | `--key-names`, `--key-pair-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeKeyPairs` → daemon lists public keys from Predastore S3 → returns key names and fingerprints | 1. List all key pairs<br>2. Filter by key name<br>3. Filter by key pair ID<br>4. Non-existent key returns empty | **DONE** |
+| `describe-key-pairs` | `--key-names`, `--key-pair-ids`, `--filters` (key-pair-id, key-name, fingerprint, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeKeyPairs` → daemon lists public keys from Predastore S3 → applies filters → returns key names and fingerprints | 1. List all key pairs<br>2. Filter by key name<br>3. Filter by key pair ID<br>4. Non-existent key returns empty<br>5. Unknown filter returns InvalidParameterValue | **DONE** |
 | `delete-key-pair` | `--key-name`, `--key-pair-id` | `--dry-run` | Key must exist | NATS `ec2.DeleteKeyPair` → daemon deletes public key from Predastore S3 → returns success | 1. Delete existing key pair<br>2. Delete non-existent key (idempotent, no error)<br>3. Verify key no longer in describe-key-pairs | **DONE** |
 | `import-key-pair` | `--key-name`, `--public-key-material` | `--tag-specifications`, `--dry-run` | None | NATS `ec2.ImportKeyPair` → daemon stores provided public key in Predastore S3 → returns key name and fingerprint | 1. Import valid SSH public key<br>2. Import invalid key material (error)<br>3. Import duplicate name (error)<br>4. Verify imported key usable with run-instances | **DONE** |
 
@@ -107,27 +166,27 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
-| `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias) | `--executable-users`, `--filters`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds and Owners → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName) | **DONE** |
+| `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias), `--filters` (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:\*) | `--executable-users`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName)<br>6. Filter by name wildcard<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `create-image` | `--instance-id`, `--name`, `--description`, `--tag-specifications` | `--no-reboot`, `--block-device-mappings`, `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.{instanceId}.CreateImage` (per-instance topic) → daemon extracts root volume → snapshots via `ebs.snapshot` NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 (`{amiId}/config.json`) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. | 1. Create image from running instance<br>2. Create image from stopped instance<br>3. Invalid instance ID (error)<br>4. Duplicate AMI name (error)<br>5. Verify new AMI appears in describe-images<br>6. Launch new instance from created AMI | **DONE** |
-| `register-image` | — | `--name`, `--description`, `--architecture`, `--root-device-name`, `--virtualization-type`, `--block-device-mappings`, `--boot-mode`, `--ena-support`, `--sriov-net-support`, `--tpm-support`, `--imds-support` | Image data must exist in S3 | Create AMI metadata entry in Predastore → assign ami-ID → set state to available → return ami-ID | 1. Register with valid metadata<br>2. Missing required name (error)<br>3. Verify registered image in describe-images | **NOT STARTED** |
-| `deregister-image` | — | `--image-id`, `--dry-run` | AMI must exist, no instances using it | Mark AMI metadata as deregistered in S3 → AMI no longer available for new launches → existing instances unaffected | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (error)<br>3. Verify deregistered AMI not in describe-images<br>4. Existing instances from AMI still run | **NOT STARTED** |
-| `copy-image` | — | `--source-image-id`, `--source-region`, `--name`, `--description`, `--encrypted`, `--kms-key-id`, `--client-token`, `--copy-image-tags`, `--tag-specifications` | Source AMI must exist | Copy AMI data between regions/nodes in Predastore → create new AMI metadata with new ami-ID → return new ami-ID | 1. Copy image within same region<br>2. Copy non-existent image (error)<br>3. Verify copied image independent of source | **NOT STARTED** |
+| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images | **DONE** |
+| `deregister-image` | `--image-id` | `--dry-run` | AMI must exist; caller must own it (system AMIs immutable via this API) | Gateway validates `ami-` prefix → NATS `ec2.DeregisterImage` → daemon hard-deletes `{amiId}/config.json` from Predastore. Backing snapshot is left intact (matches AWS — operators run `delete-snapshot` separately to reclaim block storage). Cross-account/system-AMI mutations rejected with `UnauthorizedOperation`. Re-deregister returns `InvalidAMIID.NotFound` (no tombstone). | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (`InvalidAMIID.NotFound`)<br>3. Re-deregister already-deleted AMI (`InvalidAMIID.NotFound`)<br>4. Cross-account AMI (`UnauthorizedOperation`)<br>5. System AMI (`UnauthorizedOperation`)<br>6. Verify deregistered AMI not in describe-images<br>7. Backing snapshot untouched | **DONE** |
+| `copy-image` | `--source-image-id`, `--source-region` (must equal gateway region), `--name`, `--description`, `--client-token` (accepted, not honoured), `--copy-image-tags`, `--tag-specifications` (image only) | `--encrypted`, `--kms-key-id`, `--destination-outpost-arn`, `--dry-run` | Source AMI must exist and be owned by caller OR be a system AMI; backing snapshot must still exist | Gateway validates `ami-` prefix, name length (3–128), same-region — cross-region / `Encrypted` / `KmsKeyId` / `DestinationOutpostArn` rejected with `InvalidParameterValue`. NATS `ec2.CopyImage` → daemon checks name uniqueness, reads source `ami-xxx/config.json`, visibility-filters cross-account sources as `InvalidAMIID.NotFound`, reads source `{snap-xxx}/metadata.json` (missing or empty SnapshotID → `InvalidAMIID.NotFound`), writes new `snap-yyy/metadata.json` that inherits source `VolumeID` (no block copy), writes new `ami-yyy/config.json` owned by caller. `Description` inherits from source when unset. Tag merge: `CopyImageTags=true` seeds tags from source, explicit image-resource `TagSpecifications` override colliding keys and append new ones; non-image tag specs ignored. | 1. Same-region copy produces new `ami-`/`snap-` pair with distinct IDs<br>2. New snap shares source `VolumeID` (metadata-only)<br>3. New AMI owned by caller, source untouched<br>4. System AMI copied into caller's account<br>5. Cross-account source (`InvalidAMIID.NotFound`)<br>6. Missing source AMI (`InvalidAMIID.NotFound`)<br>7. Orphaned source (missing/empty `SnapshotID`) → `InvalidAMIID.NotFound`<br>8. Duplicate name (`InvalidAMIName.Duplicate`)<br>9. Cross-region / `Encrypted` / `KmsKeyId` rejected (`InvalidParameterValue`)<br>10. `CopyImageTags` true/false tag inheritance semantics | **DONE** |
 | `import-image` | — | `--disk-containers` (Format, Url/S3Bucket+S3Key), `--description`, `--architecture`, `--platform` | S3 bucket with disk image | Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI | 1. Import QCOW2 image<br>2. Import RAW image<br>3. Invalid format (error)<br>4. Verify imported image launchable | **NOT STARTED** |
-| `describe-image-attribute` | — | `--image-id`, `--attribute`, `--dry-run` | AMI must exist | Read specific attribute from AMI metadata in S3 → return attribute value | 1. Get description attribute<br>2. Get launch permission<br>3. Invalid attribute name (error) | **NOT STARTED** |
-| `modify-image-attribute` | — | `--image-id`, `--launch-permission`, `--description`, `--operation-type`, `--user-ids`, `--user-groups`, `--organization-arns`, `--dry-run` | AMI must exist | Update specific attribute in AMI metadata → persist to S3 | 1. Add launch permission<br>2. Modify description<br>3. Invalid AMI (error) | **NOT STARTED** |
-| `reset-image-attribute` | — | `--image-id`, `--attribute`, `--dry-run` | AMI must exist | Reset attribute to default value in AMI metadata | 1. Reset launch permission to owner-only<br>2. Invalid AMI (error) | **NOT STARTED** |
+| `describe-image-attribute` | `--image-id`, `--attribute` (`description`, `blockDeviceMapping` only) | `--dry-run`, `--attribute` for `launchPermission`/`bootMode`/`kernel`/`ramdisk`/`sriovNetSupport`/`productCodes`/`tpmSupport`/`uefiData`/`imdsSupport`/`lastLaunchedTime`/`deregistrationProtection` | AMI must exist and be owned by caller OR be a system AMI | Gateway validates `ami-` prefix + allowlisted attribute → NATS `ec2.DescribeImageAttribute` → daemon reads `ami-xxx/config.json`, hides cross-account reads as `InvalidAMIID.NotFound` (matches `DescribeImages`), returns `description` from stored metadata or synthesises a single root `blockDeviceMapping` from `AMIMetadata` (same shape as `DescribeImages`). Unsupported attributes rejected with `InvalidParameterValue` — notably `launchPermission` is deferred (breaks Terraform's `aws_ami` refresh; multi-account AMI sharing needs a separate design). | 1. Read `description`<br>2. Read synthesised `blockDeviceMapping`<br>3. AMI not found (`InvalidAMIID.NotFound`)<br>4. Cross-account AMI hidden (`InvalidAMIID.NotFound`)<br>5. System AMI readable by any caller<br>6. Unsupported attributes (`launchPermission`, `bootMode`, …) rejected with `InvalidParameterValue` | **DONE** |
+| `modify-image-attribute` | `--image-id`, `--description` (top-level `Value=…` form), `--attribute description --value …` (structured form) | `--launch-permission`, `--imds-support`, `--operation-type`, `--user-ids`, `--user-groups`, `--organization-arns`, `--product-codes`, `--dry-run`, `--attribute` for anything other than `description` | AMI must exist and be owned by caller (system AMIs immutable via this API) | Gateway validates `ami-` prefix, rejects `LaunchPermission`/`ImdsSupport`/`ProductCodes`/org+user-id flags with `InvalidParameterValue`, and normalises the overloaded input shape (top-level `Description` → `Attribute=description`+`Value=…`). Both forms set together → `InvalidParameterCombination`; neither → `MissingParameter`. NATS `ec2.ModifyImageAttribute` → daemon loads AMI, runs ownership check (cross-account → `UnauthorizedOperation`, system AMIs also rejected), writes `AMIMetadata.Description` back to S3. Empty `Value` clears the description. | 1. Modify description (top-level form)<br>2. Modify description (structured form)<br>3. Clear description with empty value<br>4. Both top-level + structured set (`InvalidParameterCombination`)<br>5. Cross-account / system AMI (`UnauthorizedOperation`)<br>6. AMI not found (`InvalidAMIID.NotFound`)<br>7. `LaunchPermission` / `ImdsSupport` / unsupported `Attribute` rejected (`InvalidParameterValue`)<br>8. Round-trips via `describe-image-attribute` | **DONE** |
+| `reset-image-attribute` | `--image-id`, `--attribute description` | `--attribute launchPermission`, `--dry-run` | AMI must exist and be owned by caller | Gateway validates `ami-` prefix; only `description` is accepted (`launchPermission` — AWS's default reset target — is out of scope). NATS `ec2.ResetImageAttribute` → daemon loads AMI, ownership check, clears `AMIMetadata.Description` to empty string, persists. | 1. Reset description to empty<br>2. Cross-account / system AMI (`UnauthorizedOperation`)<br>3. AMI not found (`InvalidAMIID.NotFound`)<br>4. `launchPermission` / other attributes rejected (`InvalidParameterValue`)<br>5. Round-trips via `describe-image-attribute` | **DONE** |
 
 ### EC2 - Volume (EBS) Management
 
 | Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
-| `describe-volumes` | `--volume-ids` (fast-path lookup), `DeleteOnTermination` (from persisted VolumeMetadata) | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeVolumes` → daemon queries viperblock for volume metadata → returns volume list with state, size, attachments, type, DeleteOnTermination flag | 1. List all volumes<br>2. Filter by volume ID<br>3. Filter by attachment state<br>4. Non-existent volume returns empty<br>5. DeleteOnTermination reflects persisted value | **DONE** |
+| `describe-volumes` | `--volume-ids` (fast-path lookup), `DeleteOnTermination` (from persisted VolumeMetadata), `--filters` (volume-id, status, size, volume-type, attachment.instance-id, attachment.status, attachment.device, availability-zone, tag:\*) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeVolumes` → daemon queries viperblock for volume metadata → applies filters → returns volume list with state, size, attachments, type, DeleteOnTermination flag | 1. List all volumes<br>2. Filter by volume ID<br>3. Filter by attachment state<br>4. Non-existent volume returns empty<br>5. DeleteOnTermination reflects persisted value<br>6. Filter by status, size, volume-type<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `modify-volume` | `--volume-id`, `--size`, `--volume-type`, `--iops` | `--throughput`, `--dry-run`, `--multi-attach-enabled` | Volume must exist | NATS `ec2.ModifyVolume` → daemon sends resize request to viperblock → NBD does not support live resize, instance must be stopped → returns modification state | 1. Increase volume size<br>2. Modify volume type<br>3. Decrease size (error - not supported)<br>4. Modify attached volume (requires stop/start) | **DONE** |
 | `create-volume` | `--size`, `--availability-zone`, `--volume-type` (gp3 only), `--snapshot-id` (creates volume from snapshot) | `--iops` (hardcoded 3000), `--encrypted` (hardcoded false), `--throughput`, `--tag-specifications` | Valid AZ configured via `spinifex init` | Gateway validates input → NATS `ec2.CreateVolume` → daemon generates vol-ID via viperblock → creates volume (empty or from snapshot) of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available | 1. Create 80GB gp3 volume<br>2. Boundary sizes (1 GiB min, 16384 GiB max)<br>3. Invalid AZ (error)<br>4. Verify volume in describe-volumes<br>5. Unsupported volume type (error - only gp3)<br>6. Size out of range (error)<br>7. Create from snapshot | **DONE** |
 | `delete-volume` | `--volume-id` | `--dry-run` | Volume must exist and be detached (state=available) | Gateway validates vol- prefix → NATS `ec2.DeleteVolume` → daemon confirms state=available and no AttachedInstance → NATS `ebs.delete` to viperblockd (stops nbdkit/WAL) → deletes S3 objects under vol-id/, vol-id-efi/, vol-id-cloudinit/ → returns success | 1. Delete detached volume<br>2. Delete attached volume (error: VolumeInUse)<br>3. Delete non-existent volume (error: InvalidVolume.NotFound)<br>4. Verify volume gone from describe-volumes<br>5. Malformed volume ID (error: InvalidVolumeID.Malformed)<br>6. Double delete (idempotent NotFound) | **DONE** |
 | `attach-volume` | `--volume-id`, `--instance-id`, `--device` (optional, auto-assigns `/dev/sd[f-p]`) | `--dry-run` | Volume must exist (available), instance must exist (running) | Gateway sends to `ec2.cmd.{instanceId}` → daemon validates volume (Predastore) → `ebs.mount` via NATS (viperblock starts NBD server) → QMP `blockdev-add` (nbd-{volId}) → QMP `device_add` (virtio-blk-pci, vdisk-{volId}) → three-phase rollback on failure → update EBSRequests + BlockDeviceMappings → persist to JetStream + Predastore → respond with VolumeAttachment | 1. Attach volume to running instance<br>2. Auto-assign device name<br>3. Attach already-attached volume (VolumeInUse)<br>4. Attach to non-existent instance (InvalidInstanceID.NotFound)<br>5. Attach to stopped instance (IncorrectInstanceState)<br>6. Volume not found (InvalidVolume.NotFound)<br>7. All device slots full (AttachmentLimitExceeded)<br>8. Volume persists across stop/start | **DONE** |
 | `detach-volume` | `--volume-id`, `--instance-id` (optional, resolved via DescribeVolumes), `--device` (optional cross-check), `--force` | `--dry-run` | Volume must be attached, instance must be running | Gateway resolves InstanceId if omitted (via DescribeVolumes) → sends to `ec2.cmd.{instanceId}` → daemon validates (running, attached, not boot/EFI/CloudInit, device match) → three-phase hot-unplug: QMP `device_del` (force continues on failure) → QMP `blockdev-del` (abort if fails, preserves state to prevent double-attach) → `ebs.unmount` via NATS (best-effort) → remove from EBSRequests + BlockDeviceMappings → update volume metadata to available → persist state → respond with VolumeAttachment (state=detaching) | 1. Detach with explicit InstanceId<br>2. Detach without InstanceId (gateway resolution)<br>3. Detach with correct --device cross-check<br>4. Missing VolumeId (InvalidParameterValue)<br>5. Volume not attached (IncorrectState)<br>6. Nonexistent volume (InvalidVolume.NotFound)<br>7. Nonexistent instance (InvalidInstanceID.NotFound)<br>8. Instance not running (IncorrectInstanceState)<br>9. Device mismatch (InvalidParameterValue)<br>10. Boot volume protection (OperationNotPermitted)<br>11. Force flag (continues past device_del failure)<br>12. Volume reusability (re-attach after detach) | **DONE** |
-| `describe-volume-status` | `--volume-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates vol- prefix → NATS `ec2.DescribeVolumeStatus` → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) | 1. List all volume statuses<br>2. Filter by specific volume IDs (fast path)<br>3. Non-existent volume ID returns InvalidVolume.NotFound<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Internal sub-volumes excluded from listing<br>6. Nil/empty input defaults to all volumes | **DONE** |
+| `describe-volume-status` | `--volume-ids`, `--filters` (volume-id, volume-status.status, availability-zone) | `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates vol- prefix → NATS `ec2.DescribeVolumeStatus` → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → applies filters → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) | 1. List all volume statuses<br>2. Filter by specific volume IDs (fast path)<br>3. Non-existent volume ID returns InvalidVolume.NotFound<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Internal sub-volumes excluded from listing<br>6. Nil/empty input defaults to all volumes<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | None | Query pending/completed volume modifications → return modification state, progress, original/target size | 1. Check in-progress modification<br>2. Check completed modification<br>3. No modifications returns empty | **NOT STARTED** |
 
 ### EC2 - Snapshot Management
@@ -137,7 +196,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `create-snapshot` | `--volume-id`, `--description`, `--tag-specifications` | `--dry-run` | Volume must exist in Predastore | Gateway validates vol- prefix → NATS `ec2.CreateSnapshot` → daemon reads VolumeConfig from Predastore → generates snap-ID via viperblock → stores SnapshotConfig (metadata-only, points to source volume) as completed → returns Snapshot with state=completed, progress=100% | 1. Snapshot from valid volume<br>2. Missing volume ID (InvalidParameterValue)<br>3. Volume not found (InvalidVolume.NotFound)<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Snapshot with description<br>6. Snapshot with tag specifications | **DONE** |
 | `create-snapshots` | — | `--instance-specification`, `--description`, `--tag-specifications` | Instance must exist, instance-volume attachment tracking | Create snapshots of all volumes attached to instance → return list of snapshot IDs. Blocked: requires instance-volume attachment tracking to discover which volumes to snapshot. | 1. Snapshot all volumes on instance<br>2. Instance with no volumes | **NOT STARTED** |
 | `delete-snapshot` | `--snapshot-id` | `--dry-run` | Snapshot must exist | Gateway validates snap- prefix → NATS `ec2.DeleteSnapshot` → daemon verifies snapshot exists in Predastore → lists and deletes all objects under snapshot prefix → returns success | 1. Delete existing snapshot<br>2. Delete non-existent snapshot (InvalidSnapshot.NotFound)<br>3. Missing snapshot ID (InvalidParameterValue)<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed) | **DONE** |
-| `describe-snapshots` | `--snapshot-ids` | `--owner-ids`, `--filters`, `--max-results`, `--dry-run` | None | Gateway validates snap- prefix on IDs → NATS `ec2.DescribeSnapshots` → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → filters by snapshot ID if specified → returns snapshot list | 1. List all snapshots<br>2. Filter by snapshot ID<br>3. Empty snapshot list<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed) | **DONE** |
+| `describe-snapshots` | `--snapshot-ids`, `--filters` (snapshot-id, status, volume-id, volume-size, owner-id, tag:\*) | `--owner-ids`, `--max-results`, `--dry-run` | None | Gateway validates snap- prefix on IDs → NATS `ec2.DescribeSnapshots` → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → applies filters → returns snapshot list | 1. List all snapshots<br>2. Filter by snapshot ID<br>3. Empty snapshot list<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed)<br>5. Filter by status, volume-id, volume-size<br>6. Unknown filter returns InvalidParameterValue | **DONE** |
 | `copy-snapshot` | `--source-snapshot-id`, `--source-region`, `--description` | `--encrypted`, `--dry-run` | Source snapshot must exist | Gateway validates snap- prefix + source region → NATS `ec2.CopySnapshot` → daemon reads source SnapshotConfig → generates new snap-ID → copies metadata (preserves tags, description override) → stores as completed → returns new snapshot ID | 1. Copy within same region<br>2. Copy non-existent snapshot (InvalidSnapshot.NotFound)<br>3. Missing source ID (InvalidParameterValue)<br>4. Missing source region (MissingParameter)<br>5. Copy preserves tags<br>6. Copy with description override | **DONE** |
 
 ### EC2 - Tags
@@ -190,7 +249,7 @@ Gateway routing and KV CRUD are wired up but no OVN/OVS integration exists. EIGW
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `create-egress-only-internet-gateway` | `--vpc-id`, `--tag-specifications` | `--client-token`, `--dry-run` | VPC must exist | Gateway validates VpcId → NATS `ec2.CreateEgressOnlyInternetGateway` → daemon generates eigw-ID → creates EgressOnlyIGWRecord in `spinifex-eigw` KV bucket with state=attached (always attached at creation, unlike IGW) → returns EgressOnlyInternetGateway with attachment | 1. Create with valid VpcId<br>2. Missing VpcId (error: MissingParameter)<br>3. Create with inline tags<br>4. Verify ID format eigw-{16hex} | **STARTED** (KV only, no OVN) |
 | `delete-egress-only-internet-gateway` | `--egress-only-internet-gateway-id` | `--dry-run` | EIGW must exist | NATS `ec2.DeleteEgressOnlyInternetGateway` → daemon verifies EIGW exists in KV → deletes from `spinifex-eigw` KV → returns ReturnCode=true. No detach required (unlike IGW). | 1. Delete existing EIGW<br>2. Delete non-existent EIGW (error)<br>3. Missing ID (error: MissingParameter) | **STARTED** (KV only, no OVN) |
-| `describe-egress-only-internet-gateways` | `--egress-only-internet-gateway-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeEgressOnlyInternetGateways` → daemon lists all keys from `spinifex-eigw` KV bucket → filters by ID if specified → always includes Attachments array → returns list | 1. Describe all EIGWs<br>2. Filter by specific ID<br>3. Non-existent ID returns empty list | **STARTED** (KV only, no OVN) |
+| `describe-egress-only-internet-gateways` | `--egress-only-internet-gateway-ids`, `--filters` (egress-only-internet-gateway-id, tag:\*) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeEgressOnlyInternetGateways` → daemon lists all keys from `spinifex-eigw` KV bucket → filters by ID if specified → applies filters → always includes Attachments array → returns list | 1. Describe all EIGWs<br>2. Filter by specific ID<br>3. Non-existent ID returns empty list<br>4. Unknown filter returns InvalidParameterValue | **STARTED** (KV only, no OVN) |
 
 ### EC2 - Placement Groups
 
@@ -246,16 +305,18 @@ EIP operations are conditionally available — only registered when external IPA
 | `release-address` | `--allocation-id` | `--dry-run` | EIP must exist and be disassociated | Gateway validates AllocationId non-empty → NATS `ec2.ReleaseAddress` → daemon retrieves EIP record from KV → checks state != associated (InvalidAddressLocked if still associated) → releases IP back to IPAM pool → deletes KV record → returns success | 1. Release disassociated EIP<br>2. Release associated EIP (error: InvalidAddressLocked)<br>3. Release non-existent (error: InvalidAllocationID.NotFound)<br>4. Missing AllocationId (error: MissingParameter) | **DONE** |
 | `associate-address` | `--allocation-id`, `--network-interface-id` OR `--instance-id`, `--private-ip-address` (optional override) | `--dry-run`, `--allow-reassociation` | EIP and target ENI/instance must exist | Gateway validates AllocationId + one of NetworkInterfaceId/InstanceId required → NATS `ec2.AssociateAddress` → daemon retrieves EIP → resolves target ENI (by ID or instance lookup) → generates AssociationId → updates record state=associated with ENI/instance/VPC/private IP → publishes `vpc.add-nat` event to vpcd (OVN NAT configuration) → returns AssociationId | 1. Associate with instance<br>2. Associate with specific ENI<br>3. Associate with private IP override<br>4. Missing AllocationId (MissingParameter)<br>5. Missing target (MissingParameter)<br>6. Non-existent EIP (InvalidAllocationID.NotFound)<br>7. Non-existent ENI (InvalidNetworkInterfaceID.NotFound) | **DONE** |
 | `disassociate-address` | `--association-id` | `--dry-run` | Association must exist | Gateway validates AssociationId non-empty → NATS `ec2.DisassociateAddress` → daemon scans KV to find EIP by AssociationId → publishes `vpc.delete-nat` event to vpcd → clears association fields → reverts state to allocated → updates KV → returns success | 1. Disassociate existing<br>2. Non-existent association (InvalidAssociationID.NotFound)<br>3. Missing AssociationId (MissingParameter) | **DONE** |
-| `describe-addresses` | `--allocation-ids`, `--public-ips` | `--filters`, `--dry-run` | None | NATS `ec2.DescribeAddresses` → daemon scans all EIP records in account → filters by AllocationIds and/or PublicIps if specified → returns Address list with allocation, association, PublicIp, PrivateIpAddress, NetworkInterfaceId, InstanceId, Domain | 1. List all EIPs<br>2. Filter by allocation ID<br>3. Filter by public IP<br>4. Non-existent AllocationId returns error<br>5. Empty result when no EIPs | **DONE** |
+| `describe-addresses` | `--allocation-ids`, `--public-ips`, `--filters` (allocation-id, public-ip, instance-id, association-id, domain, tag:\*) | `--dry-run` | None | NATS `ec2.DescribeAddresses` → daemon scans all EIP records in account → filters by AllocationIds and/or PublicIps if specified → applies filters → returns Address list with allocation, association, PublicIp, PrivateIpAddress, NetworkInterfaceId, InstanceId, Domain | 1. List all EIPs<br>2. Filter by allocation ID<br>3. Filter by public IP<br>4. Non-existent AllocationId returns error<br>5. Empty result when no EIPs<br>6. Filter by instance-id, domain<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `describe-addresses-attribute` | `--allocation-ids` (optional filter) | `--attribute`, `--dry-run`, `--max-results`, `--next-token` | None | Gateway validates input → NATS `ec2.DescribeAddressesAttribute` with `spinifex-workers` queue group → daemon looks up EIP records by AllocationIds (or scans all account EIPs if none specified) → returns AddressAttribute list with AllocationId and PublicIp per entry. Unlike DescribeAddresses, returns empty list (not error) for non-existent AllocationIds. PtrRecord always nil (reverse-DNS not supported). | 1. List all EIP attributes<br>2. Filter by allocation IDs<br>3. Non-existent AllocationId returns empty (not error)<br>4. Nil input returns error | **DONE** |
 
 ### EC2 - NAT Gateway
 
+NAT gateways are stored in `spinifex-vpc-nat-gateways` KV bucket. Deleted gateways are moved to `spinifex-vpc-deleted-nat-gateways` with a 1-hour TTL for Terraform polling. SNAT rule coordination uses `vpc.add-nat-gateway` and `vpc.delete-nat-gateway` NATS events (published via route table service on CreateRoute and during DeleteNatGateway respectively).
+
 | Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
-| `create-nat-gateway` | — | `--subnet-id`, `--allocation-id`, `--connectivity-type`, `--tag-specifications` | Subnet must exist, EIP must be allocated (for public type) | NATS `ec2.CreateNatGateway` → daemon creates NAT gateway in JetStream KV with nat-ID → configure OVS SNAT rules → state=pending→available → return NatGateway | 1. Create public NAT gateway<br>2. Create private NAT gateway (no allocation-id)<br>3. Invalid subnet (error) | **NOT STARTED** |
-| `delete-nat-gateway` | — | `--nat-gateway-id`, `--dry-run` | NAT gateway must exist | NATS `ec2.DeleteNatGateway` → daemon marks as deleting → remove OVS SNAT rules → delete from KV → return NatGatewayId | 1. Delete existing NAT gateway<br>2. Delete non-existent (error) | **NOT STARTED** |
-| `describe-nat-gateways` | — | `--nat-gateway-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeNatGateways` → daemon lists NAT gateways from KV → return list with state, subnet, addresses | 1. List all NAT gateways<br>2. Filter by ID<br>3. Filter by VPC ID | **NOT STARTED** |
+| `create-nat-gateway` | `--subnet-id`, `--allocation-id` | `--connectivity-type` (public only), `--tag-specifications`, `--dry-run` | Subnet must exist, EIP must be allocated and unassociated | Gateway validates SubnetId + AllocationId (required) → NATS `ec2.CreateNatGateway` with `spinifex-workers` queue group → daemon verifies subnet exists in `spinifex-vpc-subnets` KV → verifies EIP exists in `spinifex-vpc-eips` KV and has no AssociationId (Resource.AlreadyAssociated if in use) → generates nat-ID → stores NatGatewayRecord in `spinifex-vpc-nat-gateways` KV with state=available → returns NatGateway with addresses | 1. Create NAT gateway (happy path)<br>2. Subnet not found (InvalidSubnetID.NotFound)<br>3. EIP not found (InvalidAllocationID.NotFound)<br>4. EIP already associated (Resource.AlreadyAssociated)<br>5. Missing SubnetId or AllocationId (MissingParameter) | **DONE** |
+| `delete-nat-gateway` | `--nat-gateway-id` | `--dry-run` | NAT gateway must exist | Gateway validates NatGatewayId (required) → NATS `ec2.DeleteNatGateway` → daemon retrieves record from KV → scans all route tables in VPC for routes targeting this NAT gateway → publishes `vpc.delete-nat-gateway` event per affected subnet (SNAT rule cleanup) → sets state=deleted → stores in `spinifex-vpc-deleted-nat-gateways` with 1-hour TTL → deletes from active KV bucket | 1. Delete existing NAT gateway<br>2. Delete non-existent (InvalidNatGatewayID.NotFound)<br>3. Deleted gateway visible when queried by ID (state=deleted)<br>4. Deleted gateway excluded from general list queries (TTL auto-expire) | **DONE** |
+| `describe-nat-gateways` | `--nat-gateway-ids`, `--filters` (vpc-id, state) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeNatGateways` → daemon scans `spinifex-vpc-nat-gateways` KV → applies NatGatewayIds filter and/or vpc-id/state filters → for specific IDs not found in active bucket, checks deleted bucket → returns InvalidNatGatewayID.NotFound if ID never existed → returns list with state, addresses, subnet, VPC | 1. List all NAT gateways<br>2. Filter by specific IDs<br>3. Filter by state<br>4. Filter by vpc-id<br>5. Non-existent ID (InvalidNatGatewayID.NotFound)<br>6. Deleted gateway visible by ID query | **DONE** |
 | `assign-private-nat-gateway-address` | — | `--nat-gateway-id`, `--private-ip-addresses` | NAT gateway must exist (private type) | NATS `ec2.AssignPrivateNatGatewayAddress` → daemon assigns private IPs to NAT gateway → return assigned addresses | 1. Assign address<br>2. Missing NAT gateway ID (error) | **NOT STARTED** |
 | `associate-nat-gateway-address` | — | `--nat-gateway-id`, `--allocation-ids` | NAT gateway and EIPs must exist | NATS `ec2.AssociateNatGatewayAddress` → daemon associates EIPs with NAT gateway → return association info | 1. Associate EIP<br>2. Missing NAT gateway ID (error) | **NOT STARTED** |
 
@@ -267,42 +328,46 @@ Requires OVN/OVS (`apt install openvswitch-switch ovn-host`). VPC/Subnet/ENI/SG
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `create-vpc` | `--cidr-block`, `--tag-specifications` | `--instance-tenancy`, `--dry-run` | None | Gateway validates CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS `ec2.CreateVpc` → daemon allocates unique VNI from `spinifex-vpc-vni-counter` KV → generates vpc-ID → stores VPCRecord in `spinifex-vpc-vpcs` KV → publishes `vpc.create` event to vpcd (creates OVN logical switch) → returns VPC with state=available | 1. Create VPC with /16 CIDR<br>2. Invalid CIDR format (error: InvalidVpcRange)<br>3. CIDR prefix outside /16–/28 (error)<br>4. Verify in describe-vpcs<br>5. Tags applied at creation | **DONE** |
 | `delete-vpc` | `--vpc-id` | `--dry-run` | VPC must be empty (no subnets) | Gateway validates VpcId → NATS `ec2.DeleteVpc` → daemon checks for dependent subnets in `spinifex-vpc-subnets` KV → returns DependencyViolation if subnets exist → deletes from `spinifex-vpc-vpcs` KV → publishes `vpc.delete` event to vpcd (removes OVN resources) | 1. Delete empty VPC<br>2. Delete VPC with subnets (DependencyViolation)<br>3. Delete non-existent VPC (error) | **DONE** |
-| `describe-vpcs` | `--vpc-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeVpcs` → daemon lists all keys from `spinifex-vpc-vpcs` KV → filters by VPC IDs if specified → returns error for non-existent requested IDs → returns VPC list with CIDR, state, CIDR associations | 1. List all VPCs<br>2. Filter by VPC ID<br>3. Non-existent VPC ID returns error | **DONE** |
+| `describe-vpcs` | `--vpc-ids`, `--filters` (vpc-id, state, cidr-block, is-default, owner-id, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeVpcs` → daemon lists all keys from `spinifex-vpc-vpcs` KV → filters by VPC IDs if specified → applies filters → returns error for non-existent requested IDs → returns VPC list with CIDR, state, CIDR associations | 1. List all VPCs<br>2. Filter by VPC ID<br>3. Non-existent VPC ID returns error<br>4. Filter by state, cidr-block, is-default<br>5. Unknown filter returns InvalidParameterValue | **DONE** |
 | `modify-vpc-attribute` | `--vpc-id`, `--enable-dns-hostnames`, `--enable-dns-support`, `--enable-network-address-usage-metrics` | `--dry-run` | VPC must exist | Gateway validates VpcId non-empty → NATS `ec2.ModifyVpcAttribute` → daemon retrieves VPC record from `spinifex-vpc-vpcs` KV → updates boolean flags if provided → updates KV with optimistic locking → returns empty output | 1. Enable DNS hostnames<br>2. Enable DNS support<br>3. Enable network address usage metrics<br>4. Missing VPC ID (MissingParameter)<br>5. Non-existent VPC (InvalidVpcID.NotFound) | **DONE** |
 | `describe-vpc-attribute` | `--vpc-id`, `--attribute` (enableDnsHostnames, enableDnsSupport, enableNetworkAddressUsageMetrics) | `--dry-run` | VPC must exist | Gateway validates VpcId + Attribute non-empty → NATS `ec2.DescribeVpcAttribute` → daemon retrieves VPC record → returns single attribute as AttributeBooleanValue (one attribute per call, matches AWS behavior) → validates attribute name | 1. Get enableDnsHostnames<br>2. Get enableDnsSupport<br>3. Get enableNetworkAddressUsageMetrics<br>4. Invalid attribute name (InvalidParameterValue)<br>5. Missing VPC ID (MissingParameter)<br>6. Non-existent VPC (InvalidVpcID.NotFound) | **DONE** |
 | `associate-vpc-cidr-block` | — | `--vpc-id`, `--cidr-block` | VPC must exist | NATS `ec2.AssociateVpcCidrBlock` → daemon adds secondary CIDR to VPC metadata → return association | 1. Add secondary CIDR<br>2. Overlapping CIDR (error)<br>3. Max CIDR blocks exceeded (error) | **NOT STARTED** |
 | `disassociate-vpc-cidr-block` | — | `--association-id` | Association must exist | NATS `ec2.DisassociateVpcCidrBlock` → daemon removes secondary CIDR from VPC → return success | 1. Remove secondary CIDR<br>2. Remove primary CIDR (error) | **NOT STARTED** |
 | `create-subnet` | `--vpc-id`, `--cidr-block`, `--availability-zone`, `--tag-specifications` | `--dry-run` | VPC must exist | Gateway validates VpcId + CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS `ec2.CreateSubnet` → daemon verifies parent VPC exists → validates subnet CIDR is within VPC CIDR range → checks for overlapping subnets in same VPC → calculates available IPs (total hosts - 5 AWS-reserved: .0/.1/.2/.3/.255) → generates subnet-ID → stores SubnetRecord in `spinifex-vpc-subnets` KV → publishes `vpc.create-subnet` event to vpcd → defaults AZ from config if omitted | 1. Create subnet within VPC CIDR<br>2. Subnet CIDR outside VPC range (error: InvalidSubnetRange)<br>3. Overlapping subnet CIDRs (error: InvalidSubnetConflict)<br>4. CIDR prefix outside /16–/28 (error)<br>5. Missing VPC or CIDR (error: MissingParameter) | **DONE** |
 | `delete-subnet` | `--subnet-id` | `--dry-run` | Subnet must exist | NATS `ec2.DeleteSubnet` → daemon verifies subnet exists → deletes from `spinifex-vpc-subnets` KV → publishes `vpc.delete-subnet` event to vpcd | 1. Delete existing subnet<br>2. Delete non-existent subnet (error) | **DONE** |
-| `describe-subnets` | `--subnet-ids`, `--filters` (vpc-id) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSubnets` → daemon lists all keys from `spinifex-vpc-subnets` KV → applies vpc-id filter if present → filters by subnet IDs if specified → recalculates available IPs from CIDR → returns error for non-existent requested IDs | 1. List all subnets<br>2. Filter by VPC ID<br>3. Filter by subnet ID<br>4. Non-existent subnet returns error | **DONE** |
+| `describe-subnets` | `--subnet-ids`, `--filters` (vpc-id, subnet-id, availability-zone, cidr-block, state, default-for-az, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSubnets` → daemon lists all keys from `spinifex-vpc-subnets` KV → applies filters → filters by subnet IDs if specified → recalculates available IPs from CIDR → returns error for non-existent requested IDs | 1. List all subnets<br>2. Filter by VPC ID<br>3. Filter by subnet ID<br>4. Non-existent subnet returns error<br>5. Filter by availability-zone, cidr-block, state<br>6. Unknown filter returns InvalidParameterValue | **DONE** |
 | `modify-subnet-attribute` | `--subnet-id`, `--map-public-ip-on-launch` | `--assign-ipv6-address-on-creation`, `--dry-run` | Subnet must exist | Gateway validates SubnetId non-empty → NATS `ec2.ModifySubnetAttribute` → daemon retrieves Subnet record from `spinifex-vpc-subnets` KV → updates MapPublicIpOnLaunch flag if provided → updates KV with optimistic locking → returns empty output | 1. Enable auto-assign public IP<br>2. Disable auto-assign public IP<br>3. Missing SubnetId (MissingParameter)<br>4. Non-existent subnet (InvalidSubnetID.NotFound) | **DONE** |
 | `associate-subnet-cidr-block` | — | `--subnet-id`, `--ipv6-cidr-block` | Subnet must exist | NATS `ec2.AssociateSubnetCidrBlock` → daemon adds IPv6 CIDR to subnet → return association | 1. Add IPv6 CIDR<br>2. Missing subnet ID (error) | **NOT STARTED** |
 | `disassociate-subnet-cidr-block` | — | `--association-id` | Association must exist | NATS `ec2.DisassociateSubnetCidrBlock` → daemon removes IPv6 CIDR from subnet → return success | 1. Remove IPv6 CIDR<br>2. Invalid association (error) | **NOT STARTED** |
 | `create-security-group` | `--group-name`, `--description`, `--vpc-id`, `--tag-specifications` | `--dry-run` | VPC must exist | Gateway validates GroupName + VpcId required → NATS `ec2.CreateSecurityGroup` → daemon verifies VPC exists → checks duplicate name in same VPC → generates sg-ID → creates SecurityGroupRecord in `spinifex-vpc-security-groups` KV with default egress rule (allow all outbound 0.0.0.0/0), no ingress rules → publishes `vpc.create-sg` event to vpcd (OVN ACL configuration) → returns GroupId | 1. Create SG in VPC<br>2. Duplicate name in same VPC (error: InvalidGroup.Duplicate)<br>3. Verify default egress rule (allow all)<br>4. Verify no default ingress rules<br>5. Missing GroupName (MissingParameter)<br>6. Non-existent VPC (InvalidVpcID.NotFound)<br>7. With tag specifications | **DONE** |
 | `delete-security-group` | `--group-id` | `--dry-run` | SG must exist | Gateway validates GroupId non-empty → NATS `ec2.DeleteSecurityGroup` → daemon retrieves SG from KV → deletes from `spinifex-vpc-security-groups` KV → publishes `vpc.delete-sg` event to vpcd → returns success | 1. Delete existing SG<br>2. Delete non-existent SG (error: InvalidGroup.NotFound)<br>3. Missing GroupId (MissingParameter) | **DONE** |
-| `describe-security-groups` | `--group-ids`, `--filters` (vpc-id, group-name) | `--group-names`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSecurityGroups` → daemon scans all SG records in account → filters by GroupIds, vpc-id filter, or group-name filter → returns SecurityGroup list with ingress/egress rules → returns error for non-existent requested GroupIds | 1. List all SGs<br>2. Filter by VPC ID<br>3. Filter by group name<br>4. Filter by GroupId<br>5. Non-existent GroupId (InvalidGroup.NotFound)<br>6. Empty result when no SGs | **DONE** |
+| `describe-security-groups` | `--group-ids`, `--filters` (vpc-id, group-name, group-id, description, ip-permission.cidr, tag:\*) | `--group-names`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSecurityGroups` → daemon scans all SG records in account → applies filters (AND logic across filter names, OR across values) → returns SecurityGroup list with ingress/egress rules → returns error for non-existent requested GroupIds | 1. List all SGs<br>2. Filter by VPC ID<br>3. Filter by group name<br>4. Filter by GroupId<br>5. Non-existent GroupId (InvalidGroup.NotFound)<br>6. Empty result when no SGs<br>7. Filter by description, ip-permission.cidr<br>8. Unknown filter returns InvalidParameterValue | **DONE** |
 | `authorize-security-group-ingress` | `--group-id`, `--ip-permissions` (IpProtocol, FromPort, ToPort, IpRanges/CidrIp, UserIdGroupPairs/GroupId) | `--dry-run` | SG must exist | Gateway validates GroupId non-empty → NATS `ec2.AuthorizeSecurityGroupIngress` → daemon retrieves SG → converts IpPermissions to SGRule objects → appends to IngressRules → updates KV with optimistic locking (revision check) → publishes `vpc.update-sg` event to vpcd → returns true | 1. Allow SSH (port 22) from 0.0.0.0/0<br>2. Allow from specific CIDR<br>3. Allow from source security group<br>4. Missing GroupId (MissingParameter)<br>5. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `authorize-security-group-egress` | `--group-id`, `--ip-permissions` | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.AuthorizeSecurityGroupEgress` → daemon retrieves SG → appends rules to EgressRules → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Allow HTTPS outbound<br>2. Restrict to specific CIDR<br>3. Missing GroupId (MissingParameter)<br>4. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `revoke-security-group-ingress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupIngress` → daemon retrieves SG → removes matching rules from IngressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `revoke-security-group-egress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupEgress` → daemon retrieves SG → removes matching rules from EgressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
+| `describe-security-group-rules` | `--filters` (group-id, security-group-rule-id, tag:\*, tag-key), `--security-group-rule-ids` | `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates `sgr-` format on SecurityGroupRuleIds (nil input = all rules in account) → NATS `ec2.DescribeSecurityGroupRules` → daemon iterates SG records in `spinifex-vpc-security-groups` KV under account → applies filters → emits one `SecurityGroupRule` per ingress (IsEgress=false) and egress (IsEgress=true) rule, each with stable `sgr-` ID, GroupOwnerId=accountID, ReferencedGroupInfo for SG-source rules → returns `InvalidSecurityGroupRuleId.NotFound` if any requested ID missing → v1→v2 KV migration backfills `sgr-` IDs on existing rules at daemon startup; MaxResults/NextToken accepted but ignored | 1. List all rules in account<br>2. Filter by group-id<br>3. Filter by security-group-rule-id<br>4. Non-existent rule ID (InvalidSecurityGroupRuleId.NotFound)<br>5. Malformed sgr- ID at gateway (InvalidParameterValue)<br>6. Tag filters match zero (Tags not yet populated on rules)<br>7. Unknown filter (InvalidParameterValue)<br>8. Migration backfills sgr- IDs on pre-v2 records | **DONE** |
 | `create-internet-gateway` | `--tag-specifications` | `--dry-run` | None | NATS `ec2.CreateInternetGateway` → daemon generates igw-ID → creates IGWRecord in `spinifex-igw` KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments | 1. Create IGW<br>2. Verify in describe-internet-gateways<br>3. Tags applied at creation | **DONE** |
 | `attach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW and VPC must exist | NATS `ec2.AttachInternetGateway` → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes `vpc.igw-attach` event to vpcd (creates OVN external switch, gateway port, SNAT rules) | 1. Attach IGW to VPC<br>2. Attach already-attached IGW (ResourceAlreadyAssociated)<br>3. Non-existent IGW (error)<br>4. Missing params (MissingParameter) | **DONE** |
 | `detach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW must be attached to specified VPC | NATS `ec2.DetachInternetGateway` → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes `vpc.igw-detach` event to vpcd (removes OVN resources) | 1. Detach attached IGW<br>2. Detach from wrong VPC (GatewayNotAttached)<br>3. Detach unattached IGW (error) | **DONE** |
 | `delete-internet-gateway` | `--internet-gateway-id` | `--dry-run` | IGW must be detached | NATS `ec2.DeleteInternetGateway` → daemon validates IGW exists and VpcId is empty (DependencyViolation if attached) → deletes from `spinifex-igw` KV | 1. Delete detached IGW<br>2. Delete attached IGW (DependencyViolation)<br>3. Delete non-existent (error) | **DONE** |
-| `describe-internet-gateways` | `--internet-gateway-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeInternetGateways` → daemon lists all keys from `spinifex-igw` KV → filters by IGW IDs if specified → returns list with Attachments array (if attached) | 1. List all IGWs<br>2. Filter by IGW ID<br>3. Verify attachment info | **DONE** |
+| `describe-internet-gateways` | `--internet-gateway-ids`, `--filters` (internet-gateway-id, attachment.vpc-id, attachment.state, tag:\*) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeInternetGateways` → daemon lists all keys from `spinifex-igw` KV → filters by IGW IDs if specified → applies filters → returns list with Attachments array (if attached) | 1. List all IGWs<br>2. Filter by IGW ID<br>3. Verify attachment info<br>4. Filter by attachment.vpc-id, attachment.state<br>5. Unknown filter returns InvalidParameterValue | **DONE** |
 
 ### EC2 - Route Tables
 
+Route tables are stored in `spinifex-vpc-route-tables` JetStream KV bucket, keyed by `{accountID}.{routeTableId}`. Each route table has a Routes array (with local route pre-populated) and an Associations array. All operations validate VPC consistency. When a route targeting a NAT gateway is created, `vpc.add-nat-gateway` events are published per associated subnet for vpcd SNAT rule creation.
+
 | Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
-| `create-route-table` | — | `--vpc-id`, `--tag-specifications` | VPC must exist | NATS `ec2.CreateRouteTable` → daemon creates route table in KV with rtb-ID → add default local route for VPC CIDR → return route table | 1. Create route table<br>2. Verify default local route<br>3. Missing VPC ID (error) | **NOT STARTED** |
-| `delete-route-table` | — | `--route-table-id`, `--dry-run` | Route table must exist and not be main route table | NATS `ec2.DeleteRouteTable` → daemon verifies not main table → delete from KV → return success | 1. Delete non-main route table<br>2. Delete main route table (error)<br>3. Delete non-existent (error) | **NOT STARTED** |
-| `describe-route-tables` | — | `--route-table-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeRouteTables` → daemon lists route tables from KV → return tables with routes | 1. List all route tables<br>2. Filter by VPC | **NOT STARTED** |
-| `create-route` | — | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` or `--nat-gateway-id` or `--egress-only-internet-gateway-id` | Route table and target must exist | NATS `ec2.CreateRoute` → daemon adds route entry → configure OVS flow for destination CIDR → update metadata | 1. Add internet route via IGW<br>2. Add route to NAT gateway<br>3. Add route to EIGW<br>4. Conflicting route (error) | **NOT STARTED** |
-| `delete-route` | — | `--route-table-id`, `--destination-cidr-block` | Route must exist | NATS `ec2.DeleteRoute` → daemon removes route entry → remove OVS flow → update metadata | 1. Delete existing route<br>2. Delete non-existent route (error)<br>3. Delete local route (error) | **NOT STARTED** |
-| `replace-route` | — | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` or `--nat-gateway-id` | Route must exist | NATS `ec2.ReplaceRoute` → daemon replaces route target → update OVS flow → update metadata | 1. Replace IGW route with NAT<br>2. Replace non-existent route (error) | **NOT STARTED** |
-| `associate-route-table` | — | `--route-table-id`, `--subnet-id` or `--gateway-id` | Route table and target must exist | NATS `ec2.AssociateRouteTable` → daemon links route table to subnet/gateway → return rtbassoc-ID | 1. Associate with subnet<br>2. Missing route table ID (error) | **NOT STARTED** |
-| `disassociate-route-table` | — | `--association-id` | Association must exist, not main table association | NATS `ec2.DisassociateRouteTable` → daemon unlinks route table → return success | 1. Disassociate existing<br>2. Disassociate main table (error) | **NOT STARTED** |
+| `create-route-table` | `--vpc-id` | `--tag-specifications`, `--dry-run` | VPC must exist | Gateway validates VpcId (required) → NATS `ec2.CreateRouteTable` with `spinifex-workers` queue group → daemon verifies VPC exists in `spinifex-vpc-vpcs` KV → retrieves VPC CIDR → generates rtb-ID → creates RouteTableRecord with IsMain=false, local route pre-populated (destination=VPC CIDR, gateway="local"), empty associations → stores in `spinifex-vpc-route-tables` KV → returns RouteTable | 1. Create route table with local route (happy path)<br>2. VPC not found (InvalidVpcID.NotFound)<br>3. Missing VpcId (MissingParameter) | **DONE** |
+| `delete-route-table` | `--route-table-id` | `--dry-run` | Route table must exist, not main, no subnet associations | Gateway validates RouteTableId (required) → NATS `ec2.DeleteRouteTable` → daemon retrieves record → rejects if IsMain=true (DependencyViolation) → rejects if has non-main subnet associations (DependencyViolation) → deletes from KV → returns success | 1. Delete custom route table<br>2. Delete main route table (DependencyViolation)<br>3. Delete table with subnet associations (DependencyViolation)<br>4. Delete non-existent (InvalidRouteTableID.NotFound) | **DONE** |
+| `describe-route-tables` | `--route-table-ids`, `--filters` (vpc-id, route-table-id, association.main, association.route-table-association-id, association.subnet-id, route.destination-cidr-block, route.gateway-id) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeRouteTables` → daemon scans all keys in `spinifex-vpc-route-tables` KV → filters by RouteTableIds if specified → applies all provided filters (AND logic) → returns error for non-existent requested IDs → converts records to EC2 RouteTable objects | 1. List all route tables<br>2. Filter by vpc-id<br>3. Filter by association.main (true/false)<br>4. Non-existent ID (InvalidRouteTableID.NotFound)<br>5. Empty result when no route tables | **DONE** |
+| `create-route` | `--route-table-id`, `--destination-cidr-block`, `--gateway-id`, `--nat-gateway-id` | `--egress-only-internet-gateway-id`, `--vpc-peering-connection-id`, `--dry-run` | Route table and target must exist in same VPC | Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS `ec2.CreateRoute` → daemon retrieves route table → checks for duplicate destination (RouteAlreadyExists) → validates exactly one of GatewayId/NatGatewayId provided → if GatewayId: verifies IGW exists in `spinifex-igw` KV and is attached to same VPC → if NatGatewayId: verifies NAT GW exists in `spinifex-vpc-nat-gateways` KV and is in same VPC → for NAT GW routes, publishes `vpc.add-nat-gateway` event per associated subnet → appends RouteRecord (state=active, origin=CreateRoute) → updates KV → returns Return=true | 1. Add IGW route (happy path)<br>2. Duplicate destination (RouteAlreadyExists)<br>3. IGW not found (InvalidInternetGatewayID.NotFound)<br>4. NAT GW not found (InvalidNatGatewayID.NotFound)<br>5. Gateway not in same VPC (InvalidParameterValue)<br>6. Missing RouteTableId or DestinationCidrBlock (MissingParameter) | **DONE** |
+| `delete-route` | `--route-table-id`, `--destination-cidr-block` | `--dry-run` | Route must exist | Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS `ec2.DeleteRoute` → daemon retrieves route table → finds matching route → rejects if gateway is "local" (InvalidParameterValue — local routes cannot be deleted) → removes route from Routes array → updates KV → returns success | 1. Delete IGW route (happy path, local route remains)<br>2. Delete local route (InvalidParameterValue)<br>3. Route not found (InvalidRoute.NotFound)<br>4. Missing parameters (MissingParameter) | **DONE** |
+| `replace-route` | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` | `--nat-gateway-id`, `--dry-run` | Route must exist, new target must exist in same VPC | Gateway validates RouteTableId + DestinationCidrBlock + GatewayId (required) → NATS `ec2.ReplaceRoute` → daemon retrieves route table → finds matching route → rejects if local route → verifies IGW exists and is attached to same VPC → updates route GatewayId, clears NatGatewayId, sets state=active → updates KV → returns success | 1. Replace route target (IGW swap)<br>2. Replace local route (InvalidParameterValue)<br>3. Route not found (InvalidRoute.NotFound)<br>4. IGW not found (InvalidInternetGatewayID.NotFound) | **DONE** |
+| `associate-route-table` | `--route-table-id`, `--subnet-id` | `--gateway-id`, `--dry-run` | Route table and subnet must exist in same VPC | Gateway validates RouteTableId + SubnetId (required) → NATS `ec2.AssociateRouteTable` → daemon retrieves route table → verifies subnet exists and is in same VPC → scans ALL route tables in VPC to check subnet isn't already explicitly associated (Resource.AlreadyAssociated) → generates rtbassoc-ID → appends AssociationRecord (Main=false) → updates KV → returns AssociationId with state=associated | 1. Associate subnet (happy path)<br>2. Duplicate subnet association (Resource.AlreadyAssociated)<br>3. Subnet not found (InvalidSubnetID.NotFound)<br>4. Subnet in different VPC (InvalidParameterValue)<br>5. Missing parameters (MissingParameter) | **DONE** |
+| `disassociate-route-table` | `--association-id` | `--dry-run` | Association must exist, not main table association | Gateway validates AssociationId (required) → NATS `ec2.DisassociateRouteTable` → daemon scans ALL route tables in account to find matching AssociationId → rejects if Main=true (InvalidParameterValue — main associations are immutable) → removes association from Associations array → updates KV → returns success | 1. Disassociate subnet (happy path)<br>2. Disassociate main table (InvalidParameterValue)<br>3. Association not found (InvalidAssociationID.NotFound) | **DONE** |
+| `replace-route-table-association` | `--association-id`, `--route-table-id` | `--dry-run` | Association and new route table must exist | Gateway validates AssociationId + RouteTableId (required) → NATS `ec2.ReplaceRouteTableAssociation` → daemon verifies new route table exists → scans all route tables to find matching AssociationId → if Main=true: atomically swaps IsMain flag between old and new table → two-phase commit: removes association from old table, creates new AssociationId on new table → compensating write on failure (logs CRITICAL if compensation fails) → returns new AssociationId with state=associated | 1. Move subnet between tables (happy path)<br>2. Replace main table association (swaps IsMain flags)<br>3. Association not found (InvalidAssociationID.NotFound)<br>4. Route table not found (InvalidRouteTableID.NotFound) | **DONE** |
 
 ### EC2 - VPC Peering
 
@@ -415,19 +480,6 @@ ENIs are managed via `spinifex-vpc-enis` KV bucket with CAS-based IPAM allocatio
 | `list-snapshot-blocks` | — | `--snapshot-id`, `--max-results`, `--next-token` | Snapshot must exist | List all non-empty blocks in snapshot with tokens and sizes | 1. List blocks of populated snapshot<br>2. List blocks of empty snapshot | **NOT STARTED** |
 | `list-changed-blocks` | — | `--second-snapshot-id`, `--first-snapshot-id`, `--max-results` | Snapshots must exist | Compare two snapshots → return list of differing block indexes | 1. Compare parent and child snapshots<br>2. Compare unrelated snapshots | **NOT STARTED** |
 
-### S3 (via Predastore)
-
-Currently S3 operations are handled directly by Predastore. Consider moving control/data plane to Spinifex format for AWS gateway integration.
-
-| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------------------|---------------|---------------|-------------|------------|--------|
-| `s3 mb` (CreateBucket) | — | `s3://bucket-name` | None | NATS `s3.createbucket` → Predastore creates bucket → return success | 1. Create valid bucket<br>2. Duplicate bucket name (error)<br>3. Invalid bucket name (error) | **NOT STARTED** (Predastore handles directly) |
-| `s3 rb` (DeleteBucket) | — | `s3://bucket-name`, `--force` | Bucket must exist | NATS `s3.deletebucket` → verify bucket empty (or force) → Predastore deletes bucket | 1. Delete empty bucket<br>2. Delete non-empty bucket (error)<br>3. Force delete non-empty bucket | **NOT STARTED** |
-| `s3 ls` (ListBuckets/Objects) | — | `s3://bucket/prefix`, `--recursive` | None | NATS `s3.listobjects` → Predastore lists objects with prefix → return list | 1. List all buckets<br>2. List objects in bucket<br>3. List with prefix filter | **NOT STARTED** |
-| `s3 cp` (PutObject/GetObject) | — | `source dest`, `--recursive`, `--acl` | Bucket must exist | NATS `s3.putobject`/`s3.getobject` → Predastore stores/retrieves with Reed-Solomon encoding | 1. Upload single file<br>2. Download single file<br>3. Recursive upload/download<br>4. Large file (multipart) | **NOT STARTED** |
-| `s3 rm` (DeleteObject) | — | `s3://bucket/key`, `--recursive` | Object must exist | NATS `s3.deleteobject` → Predastore deletes object shards | 1. Delete single object<br>2. Recursive delete<br>3. Delete non-existent (idempotent) | **NOT STARTED** |
-| `s3 sync` | — | `source dest`, `--delete`, `--exclude`, `--include` | Bucket must exist | Compare source and dest → upload changed/new files → optionally delete removed files | 1. Sync local dir to S3<br>2. Sync with --delete<br>3. Sync with exclude filter | **NOT STARTED** |
-
 ### IAM - User Management
 
 All IAM operations are account-scoped. Users, policies, and access keys are namespaced per account. The gateway extracts the account ID from the authenticated access key via `c.Locals("sigv4.accountId")` and passes it to all IAM service methods. Root user (account `000000000000`) bypasses policy evaluation entirely.
@@ -485,3 +537,144 @@ The gateway evaluates IAM policies on every request (except root user bypass). P
 | Account-scoped evaluation | Policies resolved from caller's account context | **DONE** |
 | EC2 action mapping | All registered EC2 actions mapped to IAM policy strings | **DONE** |
 | IAM action mapping | 16 IAM actions mapped to IAM policy strings | **DONE** |
+
+### IAM - Roles
+
+IAM roles are required for instance profiles, cross-account access, and service-linked permissions. Without roles, workloads must use static access keys. Roles should be stored in a `spinifex-iam-roles` KV bucket keyed by `{accountID}.{roleName}`. Trust policies (AssumeRolePolicyDocument) define who can assume the role.
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-role` | — | `--role-name`, `--assume-role-policy-document`, `--path`, `--description`, `--max-session-duration`, `--permissions-boundary`, `--tags` | None | IAMService.CreateRole(accountID, input) → validate RoleName + AssumeRolePolicyDocument (required) → validate trust policy JSON → generate AROA-prefixed RoleId → store RoleRecord in `spinifex-iam-roles` KV at `{accountID}.{roleName}` → ARN: `arn:aws:iam::{accountID}:role/{path}{roleName}` → return Role | 1. Create with valid trust policy<br>2. Malformed trust policy (MalformedPolicyDocument)<br>3. Duplicate name (EntityAlreadyExists)<br>4. Missing RoleName (InvalidInput)<br>5. With path and description | **NOT STARTED** |
+| `get-role` | — | `--role-name` | Role must exist | IAMService.GetRole(accountID, input) → reads from KV at `{accountID}.{roleName}` → returns Role with ARN, Path, AssumeRolePolicyDocument, CreateDate | 1. Get existing role<br>2. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+| `list-roles` | — | `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListRoles(accountID, input) → prefix-scans KV for `{accountID}.*` → filters by path prefix → returns role list scoped to caller's account | 1. List all roles<br>2. List with path prefix<br>3. Empty list<br>4. Account isolation | **NOT STARTED** |
+| `delete-role` | — | `--role-name` | Role must exist, no attached policies, no instance profiles | IAMService.DeleteRole(accountID, input) → verify no attached policies (DeleteConflict) → verify not in any instance profile (DeleteConflict) → delete from KV | 1. Delete role with no attachments<br>2. Delete role with policies (DeleteConflict)<br>3. Delete role in instance profile (DeleteConflict)<br>4. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+| `update-role` | — | `--role-name`, `--description`, `--max-session-duration` | Role must exist | IAMService.UpdateRole(accountID, input) → update description and/or session duration in KV → return success | 1. Update description<br>2. Update max session duration<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+| `update-assume-role-policy` | — | `--role-name`, `--policy-document` | Role must exist | IAMService.UpdateAssumeRolePolicy(accountID, input) → validate trust policy JSON → update AssumeRolePolicyDocument in KV → return success | 1. Update trust policy<br>2. Malformed policy (MalformedPolicyDocument)<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+| `attach-role-policy` | — | `--role-name`, `--policy-arn` | Role and policy must exist | IAMService.AttachRolePolicy(accountID, input) → verify policy exists → add ARN to role's AttachedPolicies → idempotent | 1. Attach policy<br>2. Already attached (idempotent)<br>3. Non-existent role (NoSuchEntity)<br>4. Non-existent policy (NoSuchEntity) | **NOT STARTED** |
+| `detach-role-policy` | — | `--role-name`, `--policy-arn` | Policy must be attached | IAMService.DetachRolePolicy(accountID, input) → remove ARN from role's AttachedPolicies | 1. Detach attached policy<br>2. Not attached (NoSuchEntity)<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+| `list-attached-role-policies` | — | `--role-name`, `--path-prefix`, `--max-items`, `--marker` | Role must exist | IAMService.ListAttachedRolePolicies(accountID, input) → reads role's AttachedPolicies list → returns ARNs and names | 1. List attached policies<br>2. No policies (empty list)<br>3. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+
+### IAM - Instance Profiles
+
+Instance profiles are containers for IAM roles that allow EC2 instances to assume a role and obtain temporary credentials. Required for `run-instances --iam-instance-profile`. Stored in `spinifex-iam-instance-profiles` KV bucket keyed by `{accountID}.{profileName}`.
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-instance-profile` | — | `--instance-profile-name`, `--path`, `--tags` | None | IAMService.CreateInstanceProfile(accountID, input) → validate name (required) → generate AIPA-prefixed InstanceProfileId → store in `spinifex-iam-instance-profiles` KV with empty Roles array → ARN: `arn:aws:iam::{accountID}:instance-profile/{path}{name}` → return InstanceProfile | 1. Create profile<br>2. Duplicate name (EntityAlreadyExists)<br>3. Missing name (InvalidInput) | **NOT STARTED** |
+| `get-instance-profile` | — | `--instance-profile-name` | Profile must exist | IAMService.GetInstanceProfile(accountID, input) → reads from KV → returns InstanceProfile with Roles array | 1. Get existing profile<br>2. Non-existent (NoSuchEntity) | **NOT STARTED** |
+| `list-instance-profiles` | — | `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListInstanceProfiles(accountID, input) → prefix-scans KV → returns list | 1. List all profiles<br>2. Empty list | **NOT STARTED** |
+| `list-instance-profiles-for-role` | — | `--role-name`, `--max-items`, `--marker` | Role must exist | IAMService.ListInstanceProfilesForRole(accountID, input) → scan all profiles → filter where Roles contains role name → return list | 1. List profiles for role<br>2. Role not in any profile (empty) | **NOT STARTED** |
+| `delete-instance-profile` | — | `--instance-profile-name` | Profile must exist, Roles must be empty | IAMService.DeleteInstanceProfile(accountID, input) → verify Roles empty (DeleteConflict if not) → delete from KV | 1. Delete empty profile<br>2. Delete profile with role (DeleteConflict)<br>3. Non-existent (NoSuchEntity) | **NOT STARTED** |
+| `add-role-to-instance-profile` | — | `--instance-profile-name`, `--role-name` | Both must exist, max 1 role per profile | IAMService.AddRoleToInstanceProfile(accountID, input) → verify profile exists → verify role exists → check Roles array is empty (LimitExceeded if role already present) → append role → update KV | 1. Add role to empty profile<br>2. Profile already has role (LimitExceeded)<br>3. Non-existent profile (NoSuchEntity)<br>4. Non-existent role (NoSuchEntity) | **NOT STARTED** |
+| `remove-role-from-instance-profile` | — | `--instance-profile-name`, `--role-name` | Role must be in profile | IAMService.RemoveRoleFromInstanceProfile(accountID, input) → remove role from Roles array → update KV | 1. Remove role<br>2. Role not in profile (NoSuchEntity) | **NOT STARTED** |
+
+### IAM - Groups
+
+Groups provide a way to attach policies to multiple users. Stored in `spinifex-iam-groups` KV bucket keyed by `{accountID}.{groupName}`.
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-group` | — | `--group-name`, `--path` | None | IAMService.CreateGroup(accountID, input) → validate GroupName (required) → generate AGPA-prefixed GroupId → store in KV → return Group | 1. Create group<br>2. Duplicate name (EntityAlreadyExists)<br>3. Missing name (InvalidInput) | **NOT STARTED** |
+| `get-group` | — | `--group-name` | Group must exist | IAMService.GetGroup(accountID, input) → reads group from KV → returns Group with member Users list | 1. Get existing group<br>2. Non-existent (NoSuchEntity) | **NOT STARTED** |
+| `list-groups` | — | `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListGroups(accountID, input) → prefix-scans KV → returns list | 1. List all groups<br>2. Empty list | **NOT STARTED** |
+| `delete-group` | — | `--group-name` | Group must exist, no members, no attached policies | IAMService.DeleteGroup(accountID, input) → verify no members → verify no attached policies → delete from KV | 1. Delete empty group<br>2. Delete group with members (DeleteConflict)<br>3. Delete group with policies (DeleteConflict)<br>4. Non-existent (NoSuchEntity) | **NOT STARTED** |
+| `add-user-to-group` | — | `--group-name`, `--user-name` | Both must exist | IAMService.AddUserToGroup(accountID, input) → verify group + user exist → add user to group's Members list → idempotent | 1. Add user<br>2. Already a member (idempotent)<br>3. Non-existent group (NoSuchEntity)<br>4. Non-existent user (NoSuchEntity) | **NOT STARTED** |
+| `remove-user-from-group` | — | `--group-name`, `--user-name` | User must be in group | IAMService.RemoveUserFromGroup(accountID, input) → remove user from Members list → update KV | 1. Remove member<br>2. Not a member (NoSuchEntity) | **NOT STARTED** |
+| `list-groups-for-user` | — | `--user-name`, `--max-items`, `--marker` | User must exist | IAMService.ListGroupsForUser(accountID, input) → scan all groups → filter where Members contains user → return list | 1. List groups for user<br>2. User in no groups (empty) | **NOT STARTED** |
+| `attach-group-policy` | — | `--group-name`, `--policy-arn` | Group and policy must exist | IAMService.AttachGroupPolicy(accountID, input) → verify policy exists → add ARN to group's AttachedPolicies → idempotent | 1. Attach policy<br>2. Already attached (idempotent)<br>3. Non-existent group (NoSuchEntity) | **NOT STARTED** |
+| `detach-group-policy` | — | `--group-name`, `--policy-arn` | Policy must be attached | IAMService.DetachGroupPolicy(accountID, input) → remove ARN from AttachedPolicies | 1. Detach policy<br>2. Not attached (NoSuchEntity) | **NOT STARTED** |
+| `list-attached-group-policies` | — | `--group-name`, `--path-prefix`, `--max-items`, `--marker` | Group must exist | IAMService.ListAttachedGroupPolicies(accountID, input) → reads group's AttachedPolicies → returns ARNs and names | 1. List attached policies<br>2. No policies (empty) | **NOT STARTED** |
+
+### STS (Security Token Service)
+
+STS provides temporary credentials and identity verification. `GetCallerIdentity` is called by every AWS SDK/CLI invocation. `AssumeRole` is required once IAM roles are implemented. STS needs its own gateway service endpoint (`sts` service in the `supportedServices` map) with a dedicated action map.
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `get-caller-identity` | — | *(no flags)* | Valid SigV4 credentials | Gateway extracts accountID and username from authenticated request → returns Account, Arn (`arn:aws:iam::{accountID}:user/{userName}` or `arn:aws:sts::{accountID}:assumed-role/{roleName}/{sessionName}`), UserId. Note: a Spinifex-custom version exists at `spinifex.GetCallerIdentity` but needs a proper STS endpoint. | 1. Get identity for IAM user<br>2. Get identity for assumed role<br>3. Get identity for root user<br>4. Invalid credentials (AuthFailure) | **NOT STARTED** |
+| `assume-role` | — | `--role-arn`, `--role-session-name`, `--duration-seconds`, `--policy`, `--external-id`, `--serial-number`, `--token-code`, `--tags`, `--transitive-tag-keys` | Role must exist, caller must be allowed by trust policy | Gateway validates RoleArn + RoleSessionName (required) → looks up role → evaluates trust policy (AssumeRolePolicyDocument) against caller → generates temporary credentials (AccessKeyId with ASIA prefix, SecretAccessKey, SessionToken) → credentials expire after DurationSeconds (default 3600, max 43200) → return Credentials + AssumedRoleUser (ARN, AssumedRoleId) | 1. Assume role with valid trust policy<br>2. Trust policy denies caller (AccessDenied)<br>3. Non-existent role (NoSuchEntity)<br>4. Expired session token rejected at SigV4<br>5. Custom duration<br>6. Session tags | **NOT STARTED** |
+| `get-session-token` | — | `--duration-seconds`, `--serial-number`, `--token-code` | Valid IAM user credentials (not role) | Gateway validates caller is IAM user (not assumed role) → generates temporary credentials (ASIA prefix) → session inherits user's permissions → return Credentials | 1. Get token for IAM user<br>2. Duration within range<br>3. Reject if caller is assumed role | **NOT STARTED** |
+
+### ELBv2 (Elastic Load Balancing v2 — Application & Network Load Balancer)
+
+All ELBv2 resources are stored in the `spinifex-elbv2` JetStream KV bucket. Key format: `lb.{lbId}`, `tg.{tgId}`, `listener.{listenerId}`. The data plane uses a system-managed LB VM running HAProxy, launched automatically during CreateLoadBalancer. HAProxy configuration is pushed via NATS (`elbv2.lb.{lbId}.config` topic) whenever listeners or targets change. Agent readiness is polled via `elbv2.lb.{lbId}.ping`.
+
+#### ELBv2 - Load Balancers
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-load-balancer` | `--name`, `--subnets`, `--security-groups`, `--scheme` (internet-facing/internal), `--tags`, `--ip-address-type` (ipv4 only) | `--type` (hardcoded application), `--customer-owned-ipv4-pool`, `--dry-run` | None (subnets optional) | Gateway validates Name (required) → NATS `elbv2.CreateLoadBalancer` with `spinifex-workers` queue group → daemon validates scheme (internet-facing or internal, InvalidScheme if other) → checks duplicate name in account (DuplicateLoadBalancerName) → generates LB ID → builds DNS name `{name}-{lbId}.{region}.elb.spinifex.local` → if subnets provided, creates one ENI per subnet with `spinifex:managed-by=elbv2` tag (rolls back on failure) → if system AMI available, launches sys.micro LB VM with cloud-init running `/usr/local/bin/lb-agent` → state=provisioning → background goroutine polls agent ping for ~5min → on agent ready: state=active, on timeout: state=failed → if no VM launcher: state=active immediately → persists to KV → returns LoadBalancer with ARN, DNSName, State | 1. Create with name (happy path)<br>2. Internal scheme<br>3. Invalid scheme (InvalidScheme)<br>4. Duplicate name (DuplicateLoadBalancerName)<br>5. Missing name (MissingParameter)<br>6. With tags<br>7. With subnets (ENI creation) | **DONE** |
+| `delete-load-balancer` | `--load-balancer-arn` | `--dry-run` | LB must exist | Gateway validates LoadBalancerArn (required) → NATS `elbv2.DeleteLoadBalancer` → daemon verifies LB exists and belongs to account → terminates LB VM (async background) → deletes all listeners for this LB → deletes system-managed ENIs → deletes LB record from KV → returns success | 1. Delete existing LB<br>2. Not found (LoadBalancerNotFound)<br>3. Cascades listener cleanup<br>4. ENI cleanup | **DONE** |
+| `describe-load-balancers` | `--load-balancer-arns`, `--names` | `--page-size`, `--marker`, `--dry-run` | None | NATS `elbv2.DescribeLoadBalancers` → daemon lists all LB records from KV → filters by account ID → applies ARN filter and/or Name filter → returns LoadBalancer list with State, DNSName, Scheme, VpcId, AvailabilityZones | 1. List all LBs<br>2. Filter by name<br>3. Account isolation | **DONE** |
+
+#### ELBv2 - Target Groups
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-target-group` | `--name`, `--protocol` (HTTP), `--port` (default 80), `--vpc-id`, `--target-type` (instance only), `--health-check-protocol`, `--health-check-port`, `--health-check-path` (default /), `--health-check-interval-seconds` (default 30), `--health-check-timeout-seconds` (default 5), `--healthy-threshold-count` (default 5), `--unhealthy-threshold-count` (default 2), `--matcher` (HttpCode, default 200), `--tags` | `--health-check-enabled`, `--protocol-version`, `--ip-address-type`, `--dry-run` | None | Gateway validates Name (required) → NATS `elbv2.CreateTargetGroup` → daemon checks duplicate name in same VPC (DuplicateTargetGroupName) → generates TG ID → creates HealthCheckConfig with provided values or defaults → stores record with empty Targets array → returns TargetGroup with ARN | 1. Create with defaults<br>2. Custom health check<br>3. Duplicate name (DuplicateTargetGroupName)<br>4. Missing name (MissingParameter) | **DONE** |
+| `delete-target-group` | `--target-group-arn` | `--dry-run` | TG must exist, not referenced by listener | Gateway validates TargetGroupArn (required) → NATS `elbv2.DeleteTargetGroup` → daemon verifies TG exists → scans all listeners to check if any DefaultActions reference this TG (TargetGroupInUse if referenced) → deletes from KV | 1. Delete TG<br>2. TG in use by listener (TargetGroupInUse)<br>3. Not found (TargetGroupNotFound) | **DONE** |
+| `describe-target-groups` | `--target-group-arns`, `--names`, `--load-balancer-arn` | `--page-size`, `--marker`, `--dry-run` | None | NATS `elbv2.DescribeTargetGroups` → daemon lists all TG records → filters by account → applies ARN/Name filters → if LoadBalancerArn specified, lists listeners for that LB and includes only TGs referenced by those listeners → returns TargetGroup list | 1. List all TGs<br>2. Filter by LB ARN (via listener lookup)<br>3. Account isolation | **DONE** |
+
+#### ELBv2 - Targets
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `register-targets` | `--target-group-arn`, `--targets` (Id=instance-id, Port=optional override) | `--dry-run` | TG must exist, instances should exist | Gateway validates TargetGroupArn + Targets (required) → NATS `elbv2.RegisterTargets` → daemon fetches TG → deduplicates against existing targets (key: `{id}:{port}`, idempotent) → resolves instance ID → private IP via VPC ENI lookup → adds target with state=initial → persists to KV → calls `pushConfigForTargetGroup()` to regenerate HAProxy config on all LBs using this TG → returns success | 1. Register targets<br>2. Idempotent re-registration<br>3. TG not found (TargetGroupNotFound)<br>4. Missing ARN or targets (MissingParameter) | **DONE** |
+| `deregister-targets` | `--target-group-arn`, `--targets` (Id, Port) | `--dry-run` | TG must exist | Gateway validates TargetGroupArn + Targets (required) → NATS `elbv2.DeregisterTargets` → daemon fetches TG → builds removal set (key: `{id}:{port}`) → filters Targets array to keep only non-matching → persists to KV → calls `pushConfigForTargetGroup()` to update HAProxy config → returns success | 1. Deregister targets<br>2. TG not found (TargetGroupNotFound)<br>3. Missing parameters (MissingParameter) | **DONE** |
+| `describe-target-health` | `--target-group-arn`, `--targets` (optional filter to specific targets) | `--include` | TG must exist | Gateway validates TargetGroupArn (required) → NATS `elbv2.DescribeTargetHealth` → daemon fetches TG → builds filter set from Targets param (if specified) → for each target, returns TargetHealthDescription with Target (Id, Port) and TargetHealth (State, Description). Note: health state is stored in records, not actively polled — HAProxy runs its own independent health checks. | 1. Describe all targets<br>2. Filter to specific targets<br>3. TG not found (TargetGroupNotFound) | **DONE** |
+
+#### ELBv2 - Listeners
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP, default), `--port` (default 80) | `--ssl-policy`, `--certificates`, `--alpn-policy`, `--mutual-authentication`, `--dry-run` | LB must exist | Gateway validates LoadBalancerArn + DefaultActions (required) → NATS `elbv2.CreateListener` → daemon verifies LB exists → checks for duplicate port on same LB (DuplicateListener) → generates listener ID → builds ARN → stores ListenerRecord in KV → calls `pushConfig(lb)` to regenerate HAProxy config (creates frontend for listener port, backend for target group with health checks) → returns Listener with ARN, Protocol, Port, DefaultActions | 1. Create listener<br>2. Duplicate port (DuplicateListener)<br>3. LB not found (LoadBalancerNotFound)<br>4. Graceful handling when NATS unavailable | **DONE** |
+| `delete-listener` | `--listener-arn` | `--dry-run` | Listener must exist | Gateway validates ListenerArn (required) → NATS `elbv2.DeleteListener` → daemon verifies listener exists and belongs to account → deletes from KV → fetches parent LB → calls `pushConfig(lb)` to regenerate HAProxy config without this listener → returns success | 1. Delete listener<br>2. Not found (ListenerNotFound) | **DONE** |
+| `describe-listeners` | `--load-balancer-arn`, `--listener-arns` | `--page-size`, `--marker`, `--dry-run` | None | NATS `elbv2.DescribeListeners` → daemon lists listeners → if LoadBalancerArn specified, filters to that LB → applies ARN filter → account isolation → returns Listener list | 1. Filter by LB ARN<br>2. Account isolation | **DONE** |
+
+#### ELBv2 - Tags
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `describe-tags` | `--resource-arns` (loadbalancer, targetgroup, listener) | — | Resource(s) must exist | Gateway validates non-empty ResourceArns → NATS `elbv2.DescribeTags` → daemon parses each ARN's resource segment to dispatch by type → looks up record (LB/TG/listener) → enforces account isolation (cross-account → not-found) → returns TagDescriptions in input order with sorted-by-key Tag list. Listener records don't store tags yet, so they always return an empty Tags slice (matches AWS behaviour for untagged resources). Required by Terraform AWS provider post-create refresh on `aws_lb`, `aws_lb_target_group`, and `aws_lb_listener`. | 1. LB tags round-trip<br>2. TG tags round-trip<br>3. Listener returns empty Tags (no error)<br>4. Multiple ARNs in one call<br>5. Unknown LB/TG/listener (per-type not-found)<br>6. Cross-account ARN (not-found, no leak)<br>7. Invalid/non-ELBv2 ARN (InvalidParameterValue)<br>8. Empty/nil ResourceArns (MissingParameter)<br>9. Untagged LB returns empty Tags | **DONE** |
+
+#### ELBv2 - Not Implemented
+
+| Feature | Description | Priority | Status |
+|---------|-------------|----------|--------|
+| Listener rules | Path-based and hostname-based routing (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | **NOT STARTED** |
+| HTTPS/TLS termination | SSL certificates, SSL policies, ALPN | High | **NOT STARTED** |
+| Modify operations | ModifyLoadBalancerAttributes, ModifyTargetGroup, ModifyTargetGroupAttributes, ModifyListener | Medium | **NOT STARTED** |
+| Connection draining | Deregistration delay for graceful target removal | Medium | **NOT STARTED** |
+| Stickiness | Session affinity / sticky sessions on target groups | Medium | **NOT STARTED** |
+| Active health checking | API-driven health state updates (currently HAProxy-only) | Medium | **NOT STARTED** |
+| Target types | IP and Lambda target types (only instance supported) | Low | **NOT STARTED** |
+| Access logs | S3 access log delivery | Low | **NOT STARTED** |
+| WAF integration | AWS WAF association for web application firewall | Low | **NOT STARTED** |
+
+### CloudWatch (Basic Monitoring)
+
+Basic CloudWatch metrics support is required for `monitor-instances`/`unmonitor-instances` to be meaningful. Daemon nodes should publish CPU, disk, and network metrics for running instances. Metrics stored in a time-series-friendly structure in JetStream KV or dedicated NATS subjects.
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `put-metric-data` | — | `--namespace`, `--metric-data` (MetricName, Value, Unit, Timestamp, Dimensions) | None | Gateway validates Namespace + MetricData (required) → NATS `cloudwatch.PutMetricData` → daemon stores metric data points in time-series KV (key: `{namespace}.{metricName}.{dimensionHash}`) → return success | 1. Put single metric<br>2. Put with dimensions (InstanceId)<br>3. Missing namespace (MissingParameter) | **NOT STARTED** |
+| `get-metric-statistics` | — | `--namespace`, `--metric-name`, `--start-time`, `--end-time`, `--period`, `--statistics` (Average, Sum, Min, Max, SampleCount), `--dimensions` | None | Gateway validates required fields → NATS `cloudwatch.GetMetricStatistics` → daemon queries time-series data → aggregates by period → returns Datapoints with requested statistics | 1. Get CPU utilization over 1 hour<br>2. Get with dimensions (InstanceId)<br>3. No data returns empty | **NOT STARTED** |
+| `list-metrics` | — | `--namespace`, `--metric-name`, `--dimensions`, `--recently-active` | None | NATS `cloudwatch.ListMetrics` → daemon scans metric namespaces → returns Metrics list with Namespace, MetricName, Dimensions | 1. List all metrics<br>2. Filter by namespace<br>3. Filter by metric name | **NOT STARTED** |
+| `describe-alarms` | — | `--alarm-names`, `--alarm-name-prefix`, `--state-value`, `--action-prefix` | None | NATS `cloudwatch.DescribeAlarms` → daemon lists alarms from KV → return MetricAlarms with State, Threshold, ComparisonOperator | 1. List all alarms<br>2. Filter by state (OK, ALARM, INSUFFICIENT_DATA) | **NOT STARTED** |
+| `put-metric-alarm` | — | `--alarm-name`, `--namespace`, `--metric-name`, `--statistic`, `--period`, `--evaluation-periods`, `--threshold`, `--comparison-operator`, `--alarm-actions`, `--dimensions` | None | NATS `cloudwatch.PutMetricAlarm` → daemon stores alarm config in KV → alarm evaluation runs on metric publish → state transitions trigger alarm actions | 1. Create alarm<br>2. Update existing alarm<br>3. Missing required fields | **NOT STARTED** |
+| `delete-alarms` | — | `--alarm-names` | Alarms must exist | NATS `cloudwatch.DeleteAlarms` → daemon deletes alarm records from KV → return success | 1. Delete existing alarm<br>2. Non-existent alarm (ResourceNotFound) | **NOT STARTED** |
+
+### Auto Scaling
+
+Auto Scaling manages instance fleets based on scaling policies. Depends on Launch Templates (EC2) and optionally ELBv2 target groups for health-based scaling. Stored in `spinifex-autoscaling` KV bucket.
+
+| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------------------|---------------|---------------|-------------|------------|--------|
+| `create-auto-scaling-group` | — | `--auto-scaling-group-name`, `--launch-template` (Id/Name, Version), `--min-size`, `--max-size`, `--desired-capacity`, `--vpc-zone-identifier` (subnet IDs), `--target-group-arns`, `--health-check-type` (EC2/ELB), `--health-check-grace-period`, `--tags` | Launch template must exist | NATS `autoscaling.CreateAutoScalingGroup` → daemon validates launch template → stores ASG record in KV → launches DesiredCapacity instances using template → registers with target groups if specified → starts health check loop → return success | 1. Create ASG with launch template<br>2. Scale to desired capacity<br>3. Missing launch template (ValidationError)<br>4. Min > Max (ValidationError) | **NOT STARTED** |
+| `update-auto-scaling-group` | — | `--auto-scaling-group-name`, `--min-size`, `--max-size`, `--desired-capacity`, `--launch-template`, `--health-check-type`, `--health-check-grace-period` | ASG must exist | NATS `autoscaling.UpdateAutoScalingGroup` → daemon updates ASG config in KV → if DesiredCapacity changed, scale up/down accordingly → return success | 1. Update desired capacity<br>2. Update launch template<br>3. Non-existent ASG (ValidationError) | **NOT STARTED** |
+| `delete-auto-scaling-group` | — | `--auto-scaling-group-name`, `--force-delete` | ASG must exist | NATS `autoscaling.DeleteAutoScalingGroup` → if force: terminate all instances → deregister from target groups → delete from KV → return success. Without force: reject if instances > 0 | 1. Delete empty ASG<br>2. Force delete with instances<br>3. Delete with instances (ResourceInUse) | **NOT STARTED** |
+| `describe-auto-scaling-groups` | — | `--auto-scaling-group-names`, `--filters`, `--max-records` | None | NATS `autoscaling.DescribeAutoScalingGroups` → daemon lists ASG records → return list with Instances, DesiredCapacity, MinSize, MaxSize | 1. List all ASGs<br>2. Filter by name | **NOT STARTED** |
+| `set-desired-capacity` | — | `--auto-scaling-group-name`, `--desired-capacity`, `--honor-cooldown` | ASG must exist | NATS `autoscaling.SetDesiredCapacity` → daemon updates desired count → scale up (launch) or scale down (terminate oldest) → return success | 1. Scale up<br>2. Scale down<br>3. Exceeds max (ValidationError) | **NOT STARTED** |
+| `describe-auto-scaling-instances` | — | `--instance-ids`, `--max-records` | None | NATS `autoscaling.DescribeAutoScalingInstances` → daemon lists all ASG-managed instances → return list with ASG name, lifecycle state, health status | 1. List all ASG instances<br>2. Filter by instance ID | **NOT STARTED** |
+| `put-scaling-policy` | — | `--auto-scaling-group-name`, `--policy-name`, `--policy-type` (TargetTrackingScaling, StepScaling, SimpleScaling), `--target-tracking-configuration`, `--scaling-adjustment`, `--cooldown` | ASG must exist | NATS `autoscaling.PutScalingPolicy` → daemon stores policy in KV → links to CloudWatch alarms for metric-based triggers → return PolicyARN | 1. Create target tracking policy<br>2. Create step scaling policy<br>3. Non-existent ASG (ValidationError) | **NOT STARTED** |
+| `delete-scaling-policy` | — | `--auto-scaling-group-name`, `--policy-name` | Policy must exist | NATS `autoscaling.DeleteScalingPolicy` → daemon deletes policy → remove CloudWatch alarm links → return success | 1. Delete existing policy<br>2. Non-existent policy | **NOT STARTED** |

docs/DESIGN.md

@@ -4,181 +4,129 @@ Spinifex is an AWS-compatible infrastructure platform for bare-metal, edge, and
 
 ## High-Level Architecture
 
-```
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                              AWS SDK / CLI                                  │
-│                    (aws ec2 run-instances --endpoint-url ...)               │
-└─────────────────────────────────────────────────────────────────────────────┘
-                                      │
-                                      │ HTTPS (SigV4 Auth)
-                                      ▼
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                           AWS Gateway (Port 9999)                           │
-│                                                                             │
-│   • TLS termination                    • AWS SigV4 authentication           │
-│   • EC2 Query Protocol parsing         • Routes to service handlers         │
-│   • XML response generation            • NATS client connection             │
-│                                                                             │
-│   File: spinifex/services/awsgw/awsgw.go                                        │
-└─────────────────────────────────────────────────────────────────────────────┘
-                                      │
-                                      │ JSON over NATS
-                                      ▼
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                            NATS Message Broker                              │
-│                                                                             │
-│   Topics:                              Queue Groups:                        │
-│   • ec2.RunInstances                   • spinifex-workers (load balanced)       │
-│   • ec2.DescribeInstances              • No queue (fan-out to all nodes)    │
-│   • ec2.cmd.{instanceID}              (per-instance start/stop/terminate)   │
-│   • ebs.mount / ebs.unmount                                                 │
-└─────────────────────────────────────────────────────────────────────────────┘
-                                      │
-                    ┌─────────────────┼─────────────────┐
-                    ▼                 ▼                 ▼
-┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
-│   Spinifex Daemon        │ │   Spinifex Daemon        │ │   Spinifex Daemon        │
-│   (Node 1)           │ │   (Node 2)           │ │   (Node N)           │
-│                      │ │                      │ │                      │
-│ • NATS subscriber    │ │ • NATS subscriber    │ │ • NATS subscriber    │
-│ • Resource manager   │ │ • Resource manager   │ │ • Resource manager   │
-│ • QEMU/KVM launcher  │ │ • QEMU/KVM launcher  │ │ • QEMU/KVM launcher  │
-│ • QMP monitoring     │ │ • QMP monitoring     │ │ • QMP monitoring     │
-│                      │ │                      │ │                      │
-│ File: daemon.go      │ │ File: daemon.go      │ │ File: daemon.go      │
-└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
-          │                       │                       │
-          ▼                       ▼                       ▼
-┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
-│   QEMU/KVM VMs       │ │   QEMU/KVM VMs       │ │   QEMU/KVM VMs       │
-└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
-```
+<p align="center">
+  <img src="../.github/assets/design-architecture.svg" alt="Spinifex high-level architecture — AWS SDK, gateway, NATS broker, per-node daemons, QEMU/KVM VMs" width="900">
+</p>
 
 ## Request Flow
 
 ### 1. AWS SDK Request
 
-Users interact with Spinifex using standard AWS SDKs or the AWS CLI by pointing to a custom endpoint:
+Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the spinifex profile.
 
 ```bash
-aws --endpoint-url https://localhost:9999 --no-verify-ssl ec2 run-instances \
+AWS_PROFILE=spinifex aws ec2 run-instances \
     --image-id ami-debian12 \
     --instance-type t3.micro \
     --key-name my-keypair
 ```
 
-The AWS SDK formats this as an HTTP POST with:
+The AWS SDK formats this as an HTTPS POST with:
 - AWS SigV4 authentication headers
-- EC2 Query Protocol body (Action=RunInstances&ImageId=ami-debian12...)
+- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian12&...`)
 
 ### 2. AWS Gateway
 
-The gateway (`spinifex/services/awsgw/awsgw.go:57-107`) is the entry point:
+The gateway (`spinifex/services/awsgw/awsgw.go`) is the entry point:
 
 ```go
-// awsgw.go:74 - Connect to NATS
-natsConn, err := nats.Connect(nodeConfig.NATS.Host, opts...)
+// Connect to NATS (retries while the local broker comes up)
+natsConn, err := utils.ConnectNATSWithRetry(...)
 
-// awsgw.go:89-94 - Create gateway with NATS connection
+// Load IAM (master key + JetStream KV) and create the gateway
 gw := gateway.GatewayConfig{
-    NATSConn: natsConn,
-    Config:   nodeConfig.AWSGW.Config,
+    NATSConn:   natsConn,
+    Config:     nodeConfig.AWSGW.Config,
+    IAMService: iamService,
+    // ...
 }
 
-// awsgw.go:103 - Start TLS listener
-log.Fatal(app.ListenTLS(nodeConfig.AWSGW.Host, nodeConfig.AWSGW.TLSCert, nodeConfig.AWSGW.TLSKey))
+// Serve over TLS
+server.ListenAndServeTLS("", "")
 ```
 
-Request routing (`spinifex/gateway/gateway.go:125-156`):
+Request routing (`spinifex/gateway/gateway.go`):
 
-1. **Authentication**: SigV4 middleware validates AWS credentials (`gateway.go:104`)
-2. **Service Detection**: Extracts service name (ec2, iam, account) from Authorization header (`gateway.go:158-186`)
-3. **Action Dispatch**: Routes to service-specific handler (`gateway.go:136-145`)
+1. **Authentication**: SigV4 middleware validates AWS credentials and resolves the account ID
+2. **Throttling**: Per-account+action token bucket rejects bursts post-auth
+3. **Service Detection**: Extracts service name (ec2, iam, account, elasticloadbalancing, spinifex) from the Authorization header
+4. **Action Dispatch**: Routes to service-specific handler
 
 ```go
-// gateway.go:136-145
 switch svc {
 case "ec2":
-    err = gw.EC2_Request(ctx)
+    err = gw.EC2_Request(w, r)
 case "account":
-    err = gw.Account_Request(ctx)
+    err = gw.Account_Request(w, r)
 case "iam":
-    err = gw.IAM_Request(ctx)
+    err = gw.IAM_Request(w, r)
+case "elasticloadbalancing":
+    err = gw.ELBv2_Request(w, r)
+case "spinifex":
+    err = gw.Spinifex_Request(w, r)
 }
 ```
 
 ### 3. EC2 Handler
 
-The EC2 handler (`spinifex/gateway/ec2.go`) parses the Action parameter and delegates to specific handlers:
+The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and delegates to specific handlers via a generic `ec2Handler` wrapper. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal:
 
 ```go
-// ec2.go - Action routing
-switch action {
-case "RunInstances":
-    output, err := gateway_ec2_instance.RunInstances(input, gw.NATSConn)
-case "DescribeInstances":
-    output, err := gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, expectedNodes)
-// ... more actions
-}
+"RunInstances": ec2Handler(func(input *ec2.RunInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
+    return gateway_ec2_instance.RunInstances(input, gw.NATSConn, accountID)
+}),
+"DescribeInstances": ec2Handler(func(input *ec2.DescribeInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
+    return gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID)
+}),
+// ... + volumes, snapshots, VPCs, subnets, route tables, IGWs, NAT gateways,
+// security groups, network interfaces, placement groups, key pairs, images, tags
 ```
 
 ### 4. NATS Messaging
 
-The gateway communicates with daemons via NATS request/response (`spinifex/handlers/ec2/instance/service_nats.go:24-57`):
+The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
 
 ```go
-// service_nats.go:24-57
-func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput) (*ec2.Reservation, error) {
-    // Marshal input to JSON
-    jsonData, err := json.Marshal(input)
-
-    // Send request to daemon via NATS with 5 minute timeout
-    msg, err := s.natsConn.Request("ec2.RunInstances", jsonData, 5*time.Minute)
-
-    // Unmarshal response
-    var reservation ec2.Reservation
-    err = json.Unmarshal(msg.Data, &reservation)
-
-    return &reservation, nil
+func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error) {
+    topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
+    return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute, accountID)
 }
 ```
 
+`RunInstances` uses a per-instance-type subject so NATS only delivers the request to a node with spare capacity for that type — no application-level reject-and-retry.
+
 ### 5. Daemon Processing
 
-Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle requests:
+Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle requests. A table-driven `subscribeAll()` registers the static EC2/ELBv2 surface at startup:
 
 ```go
-// daemon.go:319 - Subscribe with queue group for load balancing
-d.natsSubscriptions["ec2.RunInstances"], err = d.natsConn.QueueSubscribe(
-    "ec2.RunInstances",
-    "spinifex-workers",      // Queue group - only one daemon handles each request
-    d.handleEC2RunInstances,
-)
+// Queue group → load-balanced (one daemon handles each request)
+{"ec2.CreateKeyPair", d.handleEC2CreateKeyPair, "spinifex-workers"},
 
-// daemon.go:373 - Subscribe without queue group for fan-out
-d.natsSubscriptions["ec2.DescribeInstances"], err = d.natsConn.Subscribe(
-    "ec2.DescribeInstances",
-    d.handleEC2DescribeInstances,  // All daemons respond
-)
+// No queue group → fan-out (every daemon responds)
+{"ec2.DescribeInstanceTypes", d.handleEC2DescribeInstanceTypes, ""},
 ```
 
-**Queue Groups**: Topics subscribed with a queue group (`spinifex-workers`) are load-balanced - only one daemon handles each request. Topics without a queue group fan out to all daemons.
+`ec2.RunInstances` is dynamic: `ResourceManager` subscribes to `ec2.RunInstances.{type}` (and `ec2.RunInstances.{type}.{nodeId}` for targeted dispatch) only while the node has capacity for that type, and unsubscribes when full. Per-instance commands flow over `ec2.cmd.{instanceID}`, subscribed only by the owning node.
+
+**Queue Groups**: topics with the `spinifex-workers` queue group are load-balanced — only one daemon handles each request. Topics without a queue group fan out to all daemons.
 
 ### 6. VM Launch
 
 When a daemon handles `RunInstances`:
 
-1. **Resource Check**: Validates CPU/memory availability (`daemon.go:66-74`)
+1. **Resource Check**: Validates CPU/memory availability via `ResourceManager`
 2. **Volume Generation**: Creates boot, cloud-init, and EFI volumes via Viperblock
 3. **Volume Mount**: Sends `ebs.mount` request to Viperblock, receives NBD URI
 4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks
 5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management
+6. **Command subscription**: Subscribes to `ec2.cmd.{instanceID}` for subsequent commands on this VM
 
 ```go
-// daemon.go - Volume mounting via NATS
+// Volume mounting via NATS
 msg, err := d.natsConn.Request("ebs.mount", ebsMountRequest, 10*time.Second)
 
-// daemon.go - QEMU launch with NBD storage
+// QEMU launch with NBD storage
 cmd := exec.Command("qemu-system-x86_64",
     "-drive", fmt.Sprintf("file=nbd:%s,format=raw", nbdURI),
     // ... additional QEMU args
@@ -190,66 +138,56 @@ cmd := exec.Command("qemu-system-x86_64",
 ### Daemon Structure
 
 ```go
-// daemon.go:76-102
 type Daemon struct {
-    node            string                  // Node identifier
-    clusterConfig   *config.ClusterConfig   // Cluster-wide configuration
-    config          *config.Config          // Node-specific configuration
-    natsConn        *nats.Conn              // NATS connection
-    resourceMgr     *ResourceManager        // CPU/Memory tracking
-    instanceService *InstanceServiceImpl    // EC2 instance operations
-    keyService      *KeyServiceImpl         // SSH key operations
-    imageService    *ImageServiceImpl       // AMI operations
-    Instances       vm.Instances            // Local VMs
-    clusterApp      *fiber.App              // HTTP cluster manager
+    node          string                  // Node identifier
+    clusterConfig *config.ClusterConfig   // Cluster-wide configuration
+    config        *config.Config          // Node-specific configuration
+    natsConn      *nats.Conn              // NATS connection
+    resourceMgr   *ResourceManager        // CPU/Memory tracking + dynamic RunInstances subscriptions
+    Instances     vm.Instances            // Local VMs
+    // ... + one service struct per EC2/ELBv2 resource
+    //       (instance, key, image, volume, snapshot, tags, vpc, subnet,
+    //        igw, eigw, natgw, routetable, eip, placementgroup, elbv2, ...)
 }
 ```
 
 ### Resource Manager
 
-Tracks available and allocated CPU/memory to prevent overcommit:
+Tracks available and allocated CPU/memory to prevent overcommit, and drives the dynamic `ec2.RunInstances.{type}` subscriptions:
 
 ```go
-// daemon.go:66-74
 type ResourceManager struct {
     mu            sync.RWMutex
-    availableVCPU int
-    availableMem  float64
+    hostVCPU      int                                // raw runtime.NumCPU
+    hostMemGB     float64                            // raw /proc/meminfo MemTotal
+    reservedVCPU  int                                // held back for spinifex services
+    reservedMem   float64                            // held back for spinifex services
     allocatedVCPU int
     allocatedMem  float64
-    instanceTypes map[string]InstanceType  // t3.micro, t3.small, etc.
+    instanceTypes map[string]*ec2.InstanceTypeInfo   // t3.micro, t3.small, etc.
 }
 ```
 
-### NATS Topics
-
-| Topic | Queue Group | Purpose |
-|-------|-------------|---------|
-| `ec2.RunInstances` | `spinifex-workers` | Launch new instances |
-| `ec2.DescribeInstances` | None (fan-out) | Query all nodes for instances |
-| `ec2.cmd.{instanceID}` | None (owner only) | Per-instance commands (start/stop/terminate, attach/detach volume) |
-| `ec2.CreateKeyPair` | `spinifex-workers` | Generate SSH keypair |
-| `ec2.DescribeKeyPairs` | `spinifex-workers` | List SSH keypairs |
-| `ec2.DescribeImages` | `spinifex-workers` | List AMIs |
-| `ebs.mount` | - | Mount volume via Viperblock |
-| `ebs.unmount` | - | Unmount volume |
-| `spinifex.admin.{node}.health` | None | Node health checks |
-
 ### Multi-Node Aggregation
 
 For operations that need data from all nodes (like `DescribeInstances`), the gateway uses inbox-based fan-out (`spinifex/gateway/ec2/instance/DescribeInstances.go`):
 
 ```go
-// DescribeInstances.go:16-111
 func DescribeInstances(...) {
     // Create unique inbox for collecting responses
     inbox := nats.NewInbox()
-    sub, err := natsConn.SubscribeSync(inbox)
+    sub, _ := natsConn.SubscribeSync(inbox)
 
-    // Publish to all nodes (no queue group = all daemons respond)
-    err = natsConn.PublishRequest("ec2.DescribeInstances", inbox, jsonData)
+    // Publish to all nodes (no queue group = all daemons respond).
+    // Account ID is propagated as a NATS header so each daemon scopes its response.
+    pubMsg := nats.NewMsg("ec2.DescribeInstances")
+    pubMsg.Reply = inbox
+    pubMsg.Data = jsonData
+    pubMsg.Header.Set(utils.AccountIDHeader, accountID)
+    _ = natsConn.PublishMsg(pubMsg)
 
-    // Collect responses from all nodes with 3-second timeout
+    // Collect responses, returning early once expectedNodes have replied
+    // or the 3s timeout is hit.
     for {
         msg, err := sub.NextMsg(3 * time.Second)
         allReservations = append(allReservations, nodeOutput.Reservations...)
@@ -300,74 +238,72 @@ type ClusterConfig struct {
 
 type Config struct {
     Node, Host, Region, AZ string
-    DataDir    string
-    Daemon     DaemonConfig     // Cluster manager port
-    NATS       NATSConfig       // NATS broker address
-    Predastore PredastoreConfig // S3 endpoint
-    AWSGW      AWSGWConfig      // Gateway port, TLS certs
-    AccessKey, SecretKey string
+    BaseDir, DataDir       string
+    Daemon     DaemonConfig
+    NATS       NATSConfig       // Broker address, JetStream, ACL token, CA cert
+    AWSGW      AWSGWConfig      // Gateway host, TLS certs, throttle config
+    Predastore PredastoreConfig // S3 endpoint + service credentials
+    Viperblock ViperblockConfig
+    VPCD       VPCDConfig       // OVN/OVS endpoints
+    Network    NetworkConfig
 }
 ```
 
+Configs are generated by `spx admin init` (leader) or `spx admin join` (followers).
+
 ## File Structure
 
 ```
 spinifex/
 ├── spinifex/
-│   ├── services/
-│   │   └── awsgw/
-│   │       └── awsgw.go        # AWS Gateway service entry point
-│   ├── gateway/
-│   │   ├── gateway.go          # Request routing, auth middleware
-│   │   ├── ec2.go              # EC2 action dispatcher
-│   │   └── ec2/
-│   │       └── instance/
-│   │           ├── RunInstances.go
-│   │           └── DescribeInstances.go
-│   ├── handlers/
-│   │   └── ec2/
-│   │       ├── instance/
-│   │       │   ├── service_nats.go  # NATS request/response
-│   │       │   └── service_impl.go  # Business logic
-│   │       ├── key/
-│   │       └── image/
+│   ├── services/                # Long-running services (awsgw, spinifex, viperblockd,
+│   │                            #   predastore, nats, vpcd, spinifexui)
+│   ├── gateway/                 # AWS API surface
+│   │   ├── gateway.go           # chi router, SigV4 auth, throttling
+│   │   ├── auth.go              # SigV4 verification + IAM lookup
+│   │   ├── ec2.go               # EC2 action dispatcher
+│   │   ├── ec2/                 # Per-resource handlers (instance, volume, vpc, ...)
+│   │   ├── elbv2/, iam/, policy/
+│   ├── handlers/                # NATS-side service implementations (ec2, elbv2, iam)
 │   ├── daemon/
-│   │   └── daemon.go           # Daemon entry point, NATS subscriptions
-│   ├── config/
-│   │   └── config.go           # Configuration structures
-│   ├── vm/
-│   │   └── instance.go         # VM instance representation
-│   └── qmp/
-│       └── qmp.go              # QEMU Machine Protocol client
+│   │   ├── daemon.go            # Daemon entry, NATS subscription table
+│   │   └── daemon_handlers_*.go # One file per resource family
+│   ├── admin/                   # Cluster bootstrap (init/join, master.key, CA, tokens)
+│   ├── config/                  # Configuration structures
+│   ├── instancetypes/           # t3.*, m5.*, sys.* definitions
+│   ├── lbagent/                 # In-VM agent for ELB target health
+│   ├── vm/                      # VM instance representation
+│   └── qmp/                     # QEMU Machine Protocol client
 └── cmd/
-    └── spinifex/
-        └── main.go             # CLI entry point
+    ├── spinifex/                # spx CLI
+    ├── lb-agent/                # ELB target agent binary
+    └── installer/               # Platform installer
 ```
 
 ## Development
 
 ### Running the Stack
 
-1. Start NATS broker
-2. Start Predastore (S3)
-3. Start Viperblock (EBS)
-4. Start Spinifex Gateway (`spx awsgw`)
-5. Start Spinifex Daemon(s) (`spx daemon`)
+```bash
+sudo spx admin init --node node1 --nodes 1
+sudo systemctl start spinifex.target
+```
+
+`spx admin init` generates the cluster CA, IAM credentials, TLS certs, and AWS CLI `spinifex` profile, then writes config to `/etc/spinifex/` and data to `/var/lib/spinifex/`. `spinifex.target` brings up all services (NATS, Predastore, Viperblock, vpcd, the daemon, awsgw, the UI) under systemd. Use `spx admin join` for additional nodes.
 
 ### Testing with AWS CLI
 
 ```bash
-# Set endpoint
-export AWS_ENDPOINT_URL=https://localhost:9999
+export AWS_PROFILE=spinifex
 
 # Run instance
-aws --no-verify-ssl ec2 run-instances \
-    --image-id ami-debian12 \
-    --instance-type t3.micro
+aws ec2 run-instances --image-id ami-debian12 --instance-type t3.micro
 
 # List instances (queries all nodes)
-aws --no-verify-ssl ec2 describe-instances
+aws ec2 describe-instances
 
 # Terminate instance
-aws --no-verify-ssl ec2 terminate-instances --instance-ids i-xxxxx
+aws ec2 terminate-instances --instance-ids i-xxxxx
 ```
+
+For cluster-state inspection that bypasses the AWS API surface, use the `spx` CLI directly: `spx get nodes`, `spx get vms`, `spx top nodes`.

docs/TAG-CONVENTIONS.md

@@ -0,0 +1,66 @@
+# Spinifex Tag Conventions
+
+Spinifex uses a small set of reserved tag keys to mark resources that are
+owned and managed by the platform itself (as opposed to resources created
+directly by customer AWS API calls). The UI uses these tags to hide
+system-owned resources from customer-facing listings so operators do not
+mistake them for user workloads and accidentally terminate or modify
+them.
+
+## `spinifex:managed-by`
+
+Identifies the Spinifex platform component that owns a resource.
+
+| Value   | Component                       | Tagged resource types                 |
+|---------|---------------------------------|---------------------------------------|
+| `elbv2` | ALB / HAProxy load balancers    | LB VM (`instance`), LB ENI, LB AMI    |
+
+The tag is applied by the backend at create/launch time:
+
+- **Instance** — set in
+  `spinifex/spinifex/daemon/daemon_system_instance.go` when
+  `LaunchSystemInstance` builds the `RunInstancesInput`, and mirrored onto
+  the stored `vm.VM.ManagedBy` field for fast lookup.
+- **ENI** — set in
+  `spinifex/spinifex/handlers/elbv2/service_impl.go` when
+  `CreateLoadBalancer` calls `CreateNetworkInterface`.
+- **AMI** — declared on the catalog entry in
+  `spinifex/spinifex/utils/images.go` and copied into
+  `AMIMetadata.Tags` by the image importer in
+  `spinifex/cmd/spinifex/cmd/admin.go`.
+
+Tag keys and values are defined once in `spinifex/spinifex/tags/tags.go`;
+new system components should register their value there and reuse the
+constants rather than hardcoding strings.
+
+## UI behaviour
+
+The admin **Nodes** page and the **Images** (AMIs) page filter out
+resources that carry `spinifex:managed-by`. Engineers who need to see
+system resources for debugging can append `?system=1` to the URL — the
+filter is bypassed and a hidden-count banner is replaced with the full
+listing. Direct navigation to a system AMI detail page without
+`?system=1` renders the "Image not found" fallback.
+
+The EC2 **Instances** page does not need UI-side filtering: backend
+SigV4 authentication scopes `DescribeInstances` by account ID, and
+system instances live under the global system account
+(`000000000000`), so customers never see them.
+
+## `spinifex:lb-arn`
+
+Applied to ELBv2-managed ENIs alongside `spinifex:managed-by=elbv2`.
+Stores the ARN of the parent load balancer so reconciliation can map an
+ENI back to its owner without a separate lookup. Customer-facing
+filtering keys off `spinifex:managed-by`; `spinifex:lb-arn` is
+informational.
+
+## Adding a new system component
+
+1. Add a new value constant next to `ManagedByELBv2` in
+   `spinifex/spinifex/tags/tags.go`.
+2. Tag every resource the component creates (instance, ENI, AMI,
+   volume, …) with `spinifex:managed-by=<value>`.
+3. Add the value to the table above so operators can recognise it.
+4. No frontend changes are required — the UI filters on the key, not
+   the value.

docs/administration/spinifex-admin-cli/README.md

@@ -35,13 +35,25 @@ All services in the Spinifex platform are managed through this single binary.
 
 ## Account Management
 
+Create a new isolated account with an admin user and AdministratorAccess policy:
+
 ```bash
 spx admin account create --name myteam
 export AWS_PROFILE=spinifex-myteam
 ```
 
+List all accounts:
+
+```bash
+spx admin account list
+```
+
+
+
 ## Node Management
 
+List nodes in the cluster:
+
 ```bash
 spx get nodes
 ```
@@ -59,15 +71,22 @@ node3   Ready     127.0.0.3       ap-southeast-2   ap-southeast-2a  2m       0
 spx top nodes
 ```
 
+Prints per-node CPU/memory usage and cluster-wide instance type availability.
+
 ## Image Management
 
 ```bash
 spx admin images list
-spx admin images import --name debian-12-arm64
+spx admin images import --name debian-13-arm64
 ```
 
+Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.
+
+
 ## Cluster Shutdown
 
+Coordinated, phased shutdown of the entire cluster (API/UI → VMs → viperblock → predastore → NATS/daemon):
+
 ```bash
 spx admin cluster shutdown
 ```
@@ -86,11 +105,12 @@ If you get permission errors during operations, ensure you're running with appro
 
 ### Services Fail to Start
 
-Check the daemon logs for specific errors:
+Check the daemon logs for specific errors via `systemctl`/`journalctl`:
 
 ```bash
-ls ~/spinifex/logs/
-cat ~/spinifex/logs/spinifex.log
+systemctl status 'spinifex-*'
+journalctl -u spinifex-daemon -f
+journalctl -u 'spinifex-*' -f
 ```
 
 Common causes include port conflicts, missing OVN configuration, or untrusted CA certificates.

docs/compute/gpu-passthrough/README.md

@@ -0,0 +1,223 @@
+---
+title: "GPU Passthrough"
+description: "Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances."
+category: "Compute"
+tags:
+  - gpu
+  - passthrough
+  - vfio
+  - ec2
+resources:
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "AWS EC2 Accelerated Computing Instance Types"
+    url: "https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html"
+---
+
+# GPU Passthrough
+
+> Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances.
+
+> **Note:** GPU passthrough is supported on x86_64 hosts only.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Prerequisites](#prerequisites)
+- [Host Setup](#host-setup)
+- [Enable Passthrough](#enable-passthrough)
+- [Status and Monitoring](#status-and-monitoring)
+- [GPU Instance Types](#gpu-instance-types)
+- [Launching GPU Instances](#launching-gpu-instances)
+- [Disable Passthrough](#disable-passthrough)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.
+
+At startup, Spinifex always probes for GPU hardware and surfaces the result in the node banner and in `spx top nodes` — regardless of whether passthrough is enabled. This lets operators verify hardware before activating the feature.
+
+The passthrough state is reflected in the login banner:
+
+![GPU not yet enabled](../../../.github/assets/images/gpu-banner1.png)
+
+---
+
+## Prerequisites
+
+- IOMMU must be supported and enabled in the host BIOS/UEFI (`Intel VT-d` or `AMD-Vi`).
+- An NVIDIA or AMD GPU must be physically installed.
+- The Spinifex node must be running on bare metal (not inside a VM).
+
+---
+
+## Host Setup
+
+Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.
+
+```bash
+sudo spx admin gpu setup
+```
+
+What it does:
+
+1. Detects installed GPUs and their PCI IDs.
+2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
+3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
+4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
+5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
+6. Adds vfio modules to initramfs.
+
+If any change requires a reboot, the command prints instructions and exits:
+
+```
+Setup complete — reboot required.
+  sudo reboot
+  Then run: sudo spx admin gpu enable
+```
+
+---
+
+## Enable Passthrough
+
+After setup and reboot, enable GPU passthrough:
+
+```bash
+sudo spx admin gpu enable
+```
+
+The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
+
+
+![GPU enabled](../../../.github/assets/images/gpu-enabled.png)
+
+---
+
+## Status and Monitoring
+
+### Per-node status
+
+```bash
+spx admin gpu status
+# or for a specific node in the cluster:
+spx admin gpu status --node <node-name>
+```
+
+Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.
+
+### Cluster view
+
+`spx top nodes` includes a `GPU (used/total)` column:
+
+| Value | Meaning |
+|-------|---------|
+| `1/2` | Passthrough active — 1 of 2 GPUs allocated |
+| `0/1*` | Node has a GPU but passthrough is not enabled |
+| `-` | No GPU hardware detected |
+
+```bash
+spx top nodes
+```
+
+---
+
+## GPU Instance Types
+
+GPU instance types are derived from detected hardware.
+
+To list available GPU instance types on the current node:
+
+```bash
+aws ec2 describe-instance-types \
+  --filters Name=instance-type,Values=g*
+```
+
+---
+
+## Launching GPU Instances
+
+Import the GPU-enabled AMI before launching:
+
+```bash
+spx admin images import --name ubuntu-gpu-nvidia   # NVIDIA hosts
+spx admin images import --name ubuntu-gpu-amd      # AMD hosts
+```
+
+Launch a GPU instance:
+
+```bash
+aws ec2 run-instances \
+  --image-id $GPU_AMI \
+  --instance-type $INSTANCE_TYPE \
+  --key-name spinifex-key
+```
+
+To verify the GPU is visible from inside the instance, SSH in and run:
+
+```bash
+# NVIDIA
+nvidia-smi
+
+# AMD
+rocm-smi
+```
+
+---
+
+## Disable Passthrough
+
+Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:
+
+```bash
+sudo spx admin gpu disable
+```
+
+The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).
+
+---
+
+## Troubleshooting
+
+### IOMMU not active after reboot
+
+Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:
+
+```bash
+cat /proc/cmdline | grep iommu
+ls /sys/kernel/iommu_groups/
+```
+
+If `/sys/kernel/iommu_groups/` is empty, IOMMU is not active. Re-run `sudo spx admin gpu setup` after enabling it in firmware.
+
+### GPU bound to native driver instead of vfio-pci
+
+```bash
+lspci -k | grep -A3 -i "vga\|3d\|display"
+```
+
+If the driver is `amdgpu` or `nvidia` rather than `vfio-pci`, the blacklist or early binding did not take effect. Re-run setup and reboot:
+
+```bash
+sudo spx admin gpu setup
+sudo reboot
+```
+
+### `spx admin gpu enable` fails with "prerequisites not met"
+
+Run `setup` first to configure the host, then retry `enable`:
+
+```bash
+sudo spx admin gpu setup
+sudo spx admin gpu enable
+```
+
+### Daemon does not confirm within 30 seconds
+
+Check the daemon log for errors:
+
+```bash
+journalctl -u spinifex-daemon -n 50
+```

docs/compute/launching-instances/README.md

@@ -149,7 +149,7 @@ For production SSH via public IPs, see [Setting Up Your Cluster](/docs/setting-u
 Check QEMU logs for the instance and verify the AMI architecture matches your host:
 
 ```bash
-cat ~/spinifex/logs/spinifex.log
+journalctl -u spinifex-daemon -f
 aws ec2 describe-images --image-ids $SPINIFEX_AMI
 ```
 
@@ -157,7 +157,7 @@ If the AMI is for a different architecture (e.g. arm64 on an x86_64 host), impor
 
 ```bash
 spx admin images list
-spx admin images import --name debian-12-x86_64
+spx admin images import --name debian-13-x86_64
 ```
 
 ### Cannot SSH Into Instance

docs/compute/vpc-networking/README.md

@@ -38,38 +38,9 @@ Spinifex maps AWS VPC concepts directly to OVN constructs:
 
 ## Network Path
 
-```
-                        Physical Network (WAN)
-                               │
-                        ┌──────┴──────┐
-                        │ br-wan │  OVS bridge (physical uplink)
-                        └──────┬──────┘
-                               │  ovn-bridge-mappings: external:br-wan
-                               │
-                    ┌──────────┴───────────────────────────┐
-                    │     VPC Logical Router               │
-                    │                                      │
-                    │  Gateway port ──── ext switch ─────── localnet port
-                    │    (public IP)     (OVN)              (maps to br-wan)
-                    │                                      │
-                    │  NAT rules:                          │
-                    │    SNAT: VPC CIDR → gateway IP       │
-                    │    dnat_and_snat: public ↔ private   │
-                    │                                      │
-                    │  Routes:                             │
-                    │    0.0.0.0/0 → WAN gateway           │
-                    │                                      │
-                    ├────────────┬──────────────┤          │
-                    │            │              │          │
-              ┌─────┴────┐ ┌─────┴────┐         │          │
-              │ subnet-a │ │ subnet-b │         │          │
-              │ (public) │ │ (private)│         │          │
-              │10.0.1/24 │ │10.0.2/24 │         │          │
-              │  eni-1   │ │  eni-3   │         │          │
-              │  eni-2   │ │  eni-4   │         │          │
-              └──────────┘ └──────────┘         │          │
-                                                └──────────┘
-```
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-network-path.svg" alt="VPC logical topology — WAN, br-wan, VPC logical router, subnets, ENIs" width="900">
+</p>
 
 Cross-host traffic uses **Geneve tunnels** (UDP 6081) over the management/overlay
 NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int`
@@ -86,9 +57,9 @@ Instances get a private IP only. They can communicate with other instances in th
 same VPC (even across subnets and hosts via the overlay). They cannot reach the
 internet or be reached from the WAN.
 
-```
-Instance (10.0.2.5) ──→ VPC Router ──→ [no default route] ──→ dropped
-```
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-private-subnet-flow.svg" alt="Private subnet — instance hits router, no default route, packet dropped" width="900">
+</p>
 
 If the VPC has an IGW attached, private subnet instances CAN reach the internet
 via the VPC router's SNAT rule (outbound only — they share the gateway IP). They
@@ -99,10 +70,9 @@ still cannot be reached from the WAN because they have no public IP.
 Instances get both a private IP and a public IP. The public IP is a 1:1 NAT
 managed by OVN — the instance OS only sees its private IP.
 
-```
-Outbound: Instance (10.0.1.5) ──→ SNAT ──→ exits as 203.0.113.10
-Inbound:  203.0.113.10 ──→ DNAT ──→ arrives at 10.0.1.5
-```
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-public-subnet-flow.svg" alt="Public subnet — outbound SNAT and inbound DNAT between private and public IPs" width="900">
+</p>
 
 **Requirements for a public subnet:**
 
@@ -191,25 +161,9 @@ dns_servers = ["192.168.1.1", "8.8.8.8"]
 Regardless of whether IPs come from a static range or DHCP, the OVN behavior
 is identical:
 
-```
-                          Two DHCP conversations (completely independent):
-
- ┌─────────────────────────────────────────────────────────────────────┐
- │ HOST ↔ ROUTER DHCP (on br-ext, via veth pair → br-wan)              │
- │   Spinifex asks router: "give me a public IP for this VM"           │
- │   Router responds: 192.168.1.75                                     │
- │   Spinifex stores IP, tells OVN: dnat_and_snat(192.168.1.75 ↔       │
- │                                                 172.31.0.4)         │
- │   (Only used in DHCP source mode. Static mode picks from range.)    │
- └─────────────────────────────────────────────────────────────────────┘
-
- ┌─────────────────────────────────────────────────────────────────────┐
- │ VM ↔ OVN DHCP (on tap → br-int → OVN logical switch)                │
- │   cloud-init asks: "what's my IP?"                                  │
- │   OVN DHCP responds: "you are 172.31.0.4"                           │
- │   VM never sees 192.168.1.x or the router                           │
- └─────────────────────────────────────────────────────────────────────┘
-```
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-dhcp-conversations.svg" alt="Two independent DHCP conversations — host-to-router and VM-to-OVN" width="900">
+</p>
 
 ### Choosing Static vs DHCP
 
@@ -273,202 +227,172 @@ define a range (or use `source = "dhcp"`) — no data migration needed.
 
 ## Bridge Setup — Physical Network Wiring
 
-OVN needs an OVS bridge for external connectivity. `setup-ovn.sh` auto-detects
-the best approach:
+The WAN NIC **must** be enslaved to a Linux bridge. This is a hard requirement —
+`setup-ovn.sh` will not attach a physical NIC directly to OVS, and macvlan is
+no longer supported. The Linux bridge owns the host IP, default route, and any
+DHCP lease, so SSH and management traffic stay up while OVS/OVN are configured
+underneath.
 
-- **Linux bridge on default route** (e.g. `br-wan` from cloud-init): Creates
-  OVS bridge `br-ext` and links them with a veth pair. The Linux bridge keeps
-  its IP and routes — no interruption. Bridge-mapping: `external:br-ext`.
-- **OVS bridge on default route**: Uses it directly for bridge-mappings.
-- **Physical NIC**: Stops and prints guidance (cannot safely move the NIC).
-
-## Two Bridges, Two Jobs
-
-Every Spinifex node has two OVS bridges:
-
-| Bridge   | Purpose                              | Ports                        |
-| -------- | ------------------------------------ | ---------------------------- |
-| `br-int` | VM overlay traffic (Geneve tunnels)  | VM TAP devices, tunnel ports |
-| `br-wan` | WAN uplink for public subnet traffic | macvlan on WAN NIC           |
-
-`br-int` is always created by `setup-ovn.sh`. `br-wan` is created only when
-a WAN bridge is configured (auto-detected or via `--wan-bridge`, required for public subnets / external connectivity).
-
-The connection between them is logical, not physical — OVN's `localnet` port type
-maps the logical external switch to `br-wan` via `ovn-bridge-mappings`.
+The full datapath chain looks like this:
 
 ```
-VM TAP ──→ br-int ──→ OVN logical pipeline ──→ localnet ──→ br-wan ──→ WAN
+physical NIC (e.g. `wan`)
+   └─ enslaved to ─▶ br-wan         (Linux bridge — host IP, default route, DHCP)
+                       │
+                       └─ veth pair ─▶ br-ext   (OVS bridge — OVN external uplink)
+                                          │
+                                          └─ localnet ─▶ br-int  (OVS integration bridge)
+                                                            │
+                                                            └─▶ TAP devices  (VM NICs)
 ```
 
-## Bridge Modes
+`setup-ovn.sh` auto-detects the Linux bridge that owns the default route
+(typically `br-wan`, provisioned by cloud-init / netplan / systemd-networkd).
+You can override the detection with `--wan-bridge=<name>`. Once detected, the
+script creates the OVS bridge `br-ext` and links it to the WAN bridge with a
+veth pair. The Linux bridge keeps its IP and routes — no interruption.
+Bridge-mapping is set to `external:br-ext`.
 
-Spinifex supports two ways to wire `br-wan` to the physical network.
-**Direct bridge** is preferred when the host has a dedicated WAN NIC. **Macvlan**
-is the fallback for single-NIC hosts where the WAN NIC is also the SSH NIC.
+If the default route is on a bare physical NIC (no bridge), `setup-ovn.sh`
+stops and prints guidance on how to convert the NIC to a bridge before
+re-running.
 
-### Direct Bridge (Preferred)
+### Example: Required `br-wan` State
 
-The WAN NIC is added directly to `br-wan` as an OVS port. OVS sees all
-traffic on the wire — any MAC, any protocol. No filtering, no workarounds.
+The host must have something resembling this before `setup-ovn.sh` is run:
 
 ```
-Management NIC (host IP, SSH)
-WAN NIC ──→ br-wan (OVS port) ──→ OVN localnet
+7: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+    link/ether 26:df:3c:de:d0:c2 brd ff:ff:ff:ff:ff:ff
+    inet 192.168.1.31/23 brd 192.168.1.255 scope global br-wan
+       valid_lft forever preferred_lft forever
+    inet6 fe80::24df:3cff:fede:d0c2/64 scope link proto kernel_ll
+       valid_lft forever preferred_lft forever
 ```
 
-**Use when:** The host has a NIC dedicated to external/WAN traffic that is NOT
-used for SSH or management. Datacenter servers with separate management +
-public NICs. Homelab hosts with 2+ NICs.
+The physical NIC (e.g. `wan`, `eth0`, `eno1`) is enslaved to `br-wan` and has
+no IP of its own — all L3 state lives on the bridge.
 
-**Benefits over macvlan:**
+Example netplan that produces this:
 
-- **Distributed NAT**: DNAT processed on the VM's own chassis (no hairpin)
-- **No MAC workarounds**: OVS sees all frames, any MAC
-- **Host can reach VMs**: No macvlan parent↔child isolation
-- **Better multi-node performance**: External traffic stays local
-- **DHCP lease stability**: No MAC changes after lease
+```yaml
+network:
+  version: 2
+  ethernets:
+    wan:
+      dhcp4: false
+  bridges:
+    br-wan:
+      interfaces: [wan]
+      dhcp4: true
+```
+
+## Three Bridges, Three Jobs
+
+Every Spinifex node has three bridges in the datapath:
+
+| Bridge   | Type         | Created By                  | Purpose                                       | Ports                        |
+| -------- | ------------ | --------------------------- | --------------------------------------------- | ---------------------------- |
+| `br-wan` | Linux bridge | Host (cloud-init / netplan) | Host WAN uplink — owns host IP and default route | Physical WAN NIC, veth peer  |
+| `br-ext` | OVS bridge   | `setup-ovn.sh`              | OVN external uplink (`localnet`)              | veth peer to `br-wan`        |
+| `br-int` | OVS bridge   | `setup-ovn.sh`              | VM overlay traffic (Geneve tunnels)           | VM TAP devices, tunnel ports |
+
+`br-wan` is provisioned by your distro's network configuration (cloud-init,
+netplan, systemd-networkd, ifupdown). The name is configurable; `br-wan` is
+the convention. `br-int` and `br-ext` are always created by `setup-ovn.sh`.
+
+The link between `br-ext` and the VM datapath is logical, not physical: OVN's
+`localnet` port type maps the logical external switch to `br-ext` via
+`ovn-bridge-mappings`. Frames egressing a VM travel TAP → `br-int` → OVN
+pipeline → `br-ext` → veth → `br-wan` → physical NIC → wire.
+
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-datapath-localnet.svg" alt="Data path — VM TAP through br-int, OVN pipeline, br-ext, veth pair, br-wan, physical NIC" width="900">
+</p>
+
+## Running setup-ovn.sh
 
 ```bash
-sudo setup-ovn.sh --wan-bridge=br-wan --wan-iface=eth1
+# Auto-detect the WAN bridge (recommended)
+sudo setup-ovn.sh
+
+# Explicitly specify the WAN bridge name
+sudo setup-ovn.sh --wan-bridge=br-wan
 ```
 
-### Macvlan (Fallback for Single-NIC)
-
-A macvlan sub-interface is created in bridge mode off the WAN NIC, and that
-macvlan is added to br-wan. The host keeps its IP on the original NIC —
-SSH stays up.
-
-```
-WAN NIC (host IP — unchanged)
-  │
-  ├── host traffic (SSH, management, overlay)
-  │
-  └── spx-ext-{nic} (macvlan, bridge mode, no IP)
-        │
-   br-wan
-        │
-   localnet → OVN ext switch
-```
-
-**Use when:** The WAN NIC is the same NIC used for SSH. Single-NIC homelabs,
-edge deployments. Adding the NIC directly to OVS would break host connectivity.
-
-**Limitations:**
-
-- Centralized NAT only (macvlan filters unicast not matching its MAC)
-- Host cannot reach VMs at their public IPs (macvlan isolation)
-- vpcd must align macvlan MAC with OVN router MAC on startup
-- Multi-node: all external traffic hairpins through the gateway chassis
-
-```bash
-sudo setup-ovn.sh --macvlan --wan-iface=eth0
-```
-
-### Mode Comparison
-
-| Capability               | Direct bridge            | Macvlan                          |
-| ------------------------ | ------------------------ | -------------------------------- |
-| NAT mode                 | Distributed (local DNAT) | Centralized (gateway hairpin)    |
-| Multi-node external perf | Optimal                  | Hairpin through gateway chassis  |
-| Host → VM public IP      | Works                    | Blocked (kernel isolation)       |
-| MAC alignment needed     | No                       | Yes (fragile startup step)       |
-| Gratuitous ARPs needed   | No                       | Yes (`nat-addresses=router`)     |
-| Works on single-NIC      | No (breaks SSH)          | Yes                              |
-| DHCP lease stability     | Clean                    | Fragile (MAC changes post-lease) |
-
-### How setup-ovn.sh Decides
-
-| Flags                                  | Result                                          |
-| -------------------------------------- | ----------------------------------------------- |
-| `--wan-bridge=br-wan --wan-iface=eth1` | Direct bridge: NIC added to br-wan              |
-| `--macvlan --wan-iface=eth0`           | Macvlan: sub-interface created, added to br-wan |
-| (no WAN bridge)                        | Only br-int created, no WAN connectivity        |
-
-### Setup (Macvlan Mode)
-
 In environments where the WAN IP comes from a router's DHCP server (homelab,
-small office), add `--dhcp` to obtain a gateway IP from the router automatically:
+small office), add `--dhcp` to obtain a gateway IP from the router
+automatically:
 
 ```bash
-sudo setup-ovn.sh --macvlan --wan-iface=eth0 --dhcp
+sudo setup-ovn.sh --dhcp
 ```
 
 This requests an IP from the **router's DHCP** (e.g., 192.168.1.1 serving
 addresses on the LAN). This is not Spinifex's internal OVN DHCP that assigns
 private IPs to VMs — it's your network's existing DHCP server.
 
-### Macvlan Internals
+| Flags                          | Result                                                                |
+| ------------------------------ | --------------------------------------------------------------------- |
+| (no flags)                     | Auto-detect WAN bridge from default route, create `br-int` + `br-ext` |
+| `--wan-bridge=<name>`          | Use the specified Linux bridge as the WAN uplink                      |
+| `--dhcp`                       | Obtain the OVN gateway IP from the router's DHCP                      |
 
-macvlan in `bridge` mode creates a virtual interface that shares the parent NIC's
-physical wire but has its own MAC address. The Linux kernel blocks direct L2
-frames between a parent interface and its macvlan children.
-
-| Path                      | Works? | Why                                               |
-| ------------------------- | ------ | ------------------------------------------------- |
-| VM → internet             | Yes    | SNAT through OVN router → macvlan → WAN NIC → WAN |
-| LAN device → VM public IP | Yes    | LAN → WAN NIC → macvlan → br-wan → OVN            |
-| Host → VM public IP       | No     | macvlan isolation (kernel blocks parent↔child)    |
-| Host → VM private IP      | Yes    | Overlay via br-int (unrelated to br-wan)          |
+If no Linux bridge owns the default route, `setup-ovn.sh` exits with guidance
+rather than silently breaking host connectivity.
 
 ## Per-Node Configuration
 
-Different nodes in a cluster can have different WAN NICs:
+Different nodes in a cluster can have different WAN bridges and NICs:
 
 ```toml
 [nodes.node1.vpcd]
-external_interface = "eth1"
+external_interface = "br-wan"
 
 [nodes.node2.vpcd]
-external_interface = "eth0"
+external_interface = "br-public"
 
 [nodes.node3.vpcd]
-external_interface = "bond0"       # bonded NIC
+external_interface = "br-wan"      # br-wan enslaving bond0
 ```
 
-Each node runs `setup-ovn.sh` with its own WAN bridge configuration. OVN doesn't
-care which NIC the macvlan is on — only that `ovn-bridge-mappings` is set.
+`external_interface` is the **Linux bridge** that owns the WAN uplink on this
+node — not the physical NIC. The physical NIC lives underneath the bridge.
+Each node runs `setup-ovn.sh` with its own WAN bridge name (or relies on
+auto-detection). OVN only requires `ovn-bridge-mappings` to point at `br-ext`.
 
 ## Bridge Verification
 
 ```bash
-# Check br-wan exists
-sudo ovs-vsctl br-exists br-wan && echo "OK" || echo "MISSING"
+# OVS bridges created by setup-ovn.sh
+sudo ovs-vsctl br-exists br-int && echo "br-int OK" || echo "br-int MISSING"
+sudo ovs-vsctl br-exists br-ext && echo "br-ext OK" || echo "br-ext MISSING"
 
-# Check the physical port
-sudo ovs-vsctl list-ports br-wan
-# Multi-NIC: shows "eth1" (or your WAN NIC)
-# Single-NIC: shows "spx-ext-eth0" (macvlan name)
+# Linux WAN bridge owns the host IP and default route
+ip -br addr show br-wan
+ip route show default
+# Default route's dev should be br-wan (or your WAN bridge name)
 
-# Check bridge mappings
+# br-ext should have one veth port linking it to br-wan
+sudo ovs-vsctl list-ports br-ext
+# Expect: a veth name (e.g. "veth-wan-ovs")
+
+# Confirm the matching peer is enslaved to the Linux WAN bridge
+sudo bridge link show | grep br-wan
+
+# Bridge mappings must point at br-ext
 sudo ovs-vsctl get Open_vSwitch . external-ids:ovn-bridge-mappings
-# Output: "external:br-wan"
+# Output: "external:br-ext"
 
-# Check macvlan (single-NIC only)
-ip link show spx-ext-eth0
-# Shows: state UP, type macvlan mode bridge
+# Physical NIC is enslaved to br-wan (master should be br-wan)
+ip -d link show wan
 ```
 
 ## Network Flow Diagram
 
-```
-┌───────────────────────────────────────────────────┐
-│ Bare-Metal Host                                   │
-│                                                   │
-│  ┌──────────┐         ┌──────────┐                │
-│  │ br-int   │         │br-wan.   │                │
-│  │ (overlay)│         │  (WAN)   │                │
-│  │          │         │          │                │
-│  │ tap-eni1 │  OVN    │spx-ext-  │                │
-│  │ tap-eni2 ├──NAT───→│{nic}     │                │
-│  │ tap-eni3 │pipeline │(macvlan) │                │
-│  └──────────┘         └────┬─────┘                │
-│                            │ macvlan bridge mode  │
-│                        WAN NIC (host IP unchanged)│
-└───────────────────────┼───────────────────────────┘
-                        │
-                   Physical WAN
-```
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-host-flow.svg" alt="Bare-metal host — br-int overlay, br-ext OVS uplink, br-wan Linux bridge, physical NIC, OVN NAT pipeline" width="900">
+</p>
 
 ## Configuration Reference
 
@@ -477,19 +401,9 @@ levels: cluster-wide mode, IP pool definitions, and per-node NIC settings.
 
 ## Configuration Levels
 
-```
-spinifex.toml
-├── [network]                        # Cluster-wide: which mode?
-│   └── external_mode = "pool"
-│
-├── [[network.external_pools]]       # IP ranges: what IPs to use?
-│   ├── name, range_start, range_end
-│   ├── gateway, prefix_len
-│   └── region, az (optional)
-│
-└── [nodes.NAME.vpcd]                # Per-node: which NIC?
-    └── external_interface
-```
+<p align="center">
+  <img src="../../../.github/assets/diagrams/vpc-config-layers.svg" alt="spinifex.toml configuration layers — cluster mode, IP pools, per-node NIC" width="900">
+</p>
 
 ## Cluster-Wide: external_mode
 
@@ -560,12 +474,12 @@ These are different things:
 [nodes.spx1.vpcd]
 ovn_nb_addr        = "tcp:10.1.3.181:6641"   # OVN Northbound DB
 ovn_sb_addr        = "tcp:10.1.3.181:6642"   # OVN Southbound DB
-external_interface = "eth1"                   # WAN NIC name
+external_interface = "br-wan"                 # WAN Linux bridge name
 ```
 
-| Field                | Description                                                                                                                                                      |
-| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `external_interface` | Physical NIC for WAN traffic. A macvlan sub-interface is created on this NIC for br-wan. Different servers may have different names (eth1, eno2, enp3s0, bond0). |
+| Field                | Description                                                                                                                                                                          |
+| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `external_interface` | Linux bridge that owns the WAN uplink on this node (e.g. `br-wan`, `br-public`). The physical NIC is enslaved to this bridge — not configured here. Different nodes may differ. |
 
 ## Pool Selection Logic
 
@@ -624,11 +538,13 @@ gateway     = "192.168.1.1"
 prefix_len  = 24
 
 [nodes.homelab.vpcd]
-external_interface = "eth0"
+external_interface = "br-wan"
 ```
 
-**Setup:** Change your router's DHCP range to end at .149. Run
-`setup-ovn.sh --macvlan --wan-iface=eth0` (or auto-detected if WAN is already a bridge).
+**Setup:** Configure `br-wan` to enslave your physical WAN NIC (netplan,
+cloud-init, or systemd-networkd). Change your router's DHCP range to end
+at .149. Run `sudo setup-ovn.sh` — it auto-detects the WAN bridge from the
+default route, or specify it with `--wan-bridge=br-wan`.
 
 ## Homelab / Dev (DHCP Pool — No Range Reservation)
 
@@ -650,19 +566,67 @@ prefix_len  = 24
 dns_servers = ["192.168.1.1", "8.8.8.8"]
 
 [nodes.homelab.vpcd]
-external_interface = "eth0"
+external_interface = "br-wan"
 ```
 
 **Setup:** No router changes needed. Spinifex requests IPs from the router's
 DHCP server when VMs launch and releases them on terminate. Requires `dhclient`
 on the host (`apt install isc-dhcp-client`).
 
+## Host-Local Subnet (No Upstream Router)
+
+```
+Network: 192.168.10.0/24 — host-local, reachable from the host only
+Host WAN: 198.51.100.10/24 on br-wan (existing address — unchanged)
+Gateway: 192.168.10.1 — second address added to br-wan
+```
+
+Add the VM pool gateway as a second address on `br-wan` alongside the existing WAN
+IP. The host acts as the gateway for the pool — no upstream router or DHCP server
+needed for this range.
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24      # VM pool gateway — host-local
+      - 198.51.100.10/24     # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: 198.51.100.1
+```
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"         # required — no upstream DHCP for this range
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"   # second address on br-wan
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+**Setup:** Apply with `sudo netplan apply`. VMs are reachable from the host at
+`192.168.10.x`. For internet access through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+
+Persist via `/etc/sysctl.d/99-ip-forward.conf` and `netfilter-persistent save`.
+
 ## Datacenter / Colo (ISP Block)
 
 ```
 ISP-assigned: 203.0.113.0/28 (14 usable IPs)
 ISP gateway: 203.0.113.1
-Servers: 3x with separate mgmt (eth0) and public (eth1) NICs
+Servers: 3x with separate mgmt NIC (eth0) and public NIC enslaved to br-wan
 ```
 
 ```toml
@@ -677,17 +641,21 @@ gateway     = "203.0.113.1"
 prefix_len  = 28
 
 [nodes.dc1.vpcd]
-external_interface = "eth1"
+external_interface = "br-wan"      # br-wan enslaves eth1
 
 [nodes.dc2.vpcd]
-external_interface = "eth1"
+external_interface = "br-wan"      # br-wan enslaves eth1
 
 [nodes.dc3.vpcd]
-external_interface = "eno1"
+external_interface = "br-public"   # br-public enslaves eno1
 ```
 
 ## Enterprise On-Prem (VLAN)
 
+The Linux WAN bridge enslaves a VLAN sub-interface (e.g. `eth1.200`) instead
+of a raw NIC. From OVN's perspective nothing changes — `external_interface`
+still points at the bridge.
+
 ```toml
 [network]
 external_mode = "pool"
@@ -700,10 +668,10 @@ gateway     = "172.16.0.1"
 prefix_len  = 24
 
 [nodes.srv1.vpcd]
-external_interface = "eth1.200"     # VLAN-tagged sub-interface
+external_interface = "br-dmz"      # br-dmz enslaves eth1.200
 
 [nodes.srv2.vpcd]
-external_interface = "bond0.200"    # Bonded NIC with VLAN tag
+external_interface = "br-dmz"      # br-dmz enslaves bond0.200
 ```
 
 ## Edge / Branch (Outbound Only)
@@ -719,7 +687,7 @@ gateway_ip = "10.0.0.50"
 prefix_len = 24
 
 [nodes.edge1.vpcd]
-external_interface = "eth0"
+external_interface = "br-wan"
 ```
 
 ## Multi-Region (Multiple Pools)
@@ -880,14 +848,15 @@ ovn-nbctl lr-route-add vpc-{vpcId} 0.0.0.0/0 192.168.1.1
 ## Per-Instance Public IP
 
 ```bash
-# 1:1 NAT (centralized — processed on gateway chassis, uses router MAC)
-ovn-nbctl lr-nat-add vpc-{vpcId} dnat_and_snat {public_ip} {private_ip}
+# 1:1 NAT — distributed (DNAT processed on the VM's own chassis)
+ovn-nbctl lr-nat-add vpc-{vpcId} dnat_and_snat {public_ip} {private_ip} \
+  port-{eniId} {vm_mac}
 ```
 
-> **Note:** Spinifex uses centralized NAT (no `external_mac` / `logical_port`)
-> because macvlan-based external bridges can only receive unicast traffic matching
-> the macvlan's own MAC. The router port MAC is set on the macvlan, and OVN uses
-> it for all ARP replies. For single-node, there is no performance difference.
+With the WAN NIC on a Linux bridge wired to OVS via veth, OVS sees every frame
+on the wire regardless of MAC, so OVN can use distributed NAT. The DNAT is
+processed on the chassis hosting the VM rather than hairpinning through a
+single gateway chassis.
 
 ## Security Group
 
@@ -943,8 +912,13 @@ sudo ovn-nbctl pg-get-ports sg-{groupId}
 
 ## 1. Set Up OVN Bridges
 
+Make sure the WAN NIC is enslaved to a Linux bridge (e.g. `br-wan`) and that
+bridge owns the default route. Then run:
+
 ```bash
-sudo setup-ovn.sh --macvlan --wan-iface=eth0
+sudo setup-ovn.sh                       # auto-detect WAN bridge
+# or
+sudo setup-ovn.sh --wan-bridge=br-wan   # explicit
 ```
 
 ## 2. Configure External IP Pool
@@ -1017,7 +991,7 @@ sudo ovn-nbctl list Logical_Switch_Port port-eni-{eniId}
 # Gateway chassis assignment
 sudo ovn-nbctl list Logical_Router_Port gw-vpc-{vpcId}
 
-# Localnet port options (check nat-addresses=router)
+# Localnet port options (network_name should be "external")
 sudo ovn-nbctl get Logical_Switch_Port ext-port-vpc-{vpcId} options
 ```
 
@@ -1067,24 +1041,30 @@ sudo ovs-vsctl get Open_vSwitch . external_ids
 ### Network Interfaces
 
 ```bash
-# Check macvlan exists and has correct MAC
-ip -d link show spx-ext-{nic}
+# Physical NIC should be enslaved to br-wan ("master br-wan")
+ip -d link show {nic}
 
-# Check macvlan is attached to OVS (look for "openvswitch_slave")
-ip -d link show spx-ext-{nic} | grep openvswitch
+# Linux WAN bridge — owns the host IP and default route
+ip -br addr show br-wan
+ip route show default
+
+# OVS bridges and the veth linking br-ext to br-wan
+sudo ovs-vsctl list-ports br-ext
+bridge link show | grep br-wan
 
 # Interface traffic stats (RX/TX counts, drops)
-ip -s link show spx-ext-{nic}
+ip -s link show {nic}
+ip -s link show br-wan
 ```
 
 ### Packet Capture
 
 ```bash
-# Capture on the OVS bridge (sees traffic after OVS processing)
-sudo tcpdump -i br-wan -n -e arp
+# Capture on the OVS uplink bridge (frames between OVN and br-wan)
+sudo tcpdump -i br-ext -n -e arp
 
-# Capture on the macvlan (sees what leaves/enters the wire)
-sudo tcpdump -i spx-ext-{nic} -n -e "host {public_ip}"
+# Capture on the Linux WAN bridge (frames between br-ext veth and the NIC)
+sudo tcpdump -i br-wan -n -e "host {public_ip}"
 
 # Capture on the physical NIC (sees everything on the wire)
 sudo tcpdump -i {nic} -n "host {public_ip}"
@@ -1094,13 +1074,13 @@ sudo tcpdump -i {nic} -n "host {public_ip}"
 
 ```bash
 # vpcd log (reconcile, NAT, topology operations)
-cat ~/spinifex/logs/vpcd.log
+journalctl -u spinifex-vpcd -f
 
 # ovn-controller log (port binding, commit failures)
 sudo cat /var/log/ovn/ovn-controller.log | tail -50
 
 # Daemon log (instance launch, network setup)
-cat ~/spinifex/logs/spinifex.log
+journalctl -u spinifex-daemon -f
 ```
 
 ### VPC Creation Fails
@@ -1109,7 +1089,7 @@ Check OVN services and vpcd daemon:
 
 ```bash
 sudo systemctl is-active ovn-controller
-cat ~/spinifex/logs/vpcd.log
+journalctl -u spinifex-vpcd -f
 ```
 
 ### Instances Cannot Reach Each Other
@@ -1162,12 +1142,18 @@ Work through these checks in order — each eliminates a class of issues.
 ### 1. Verify OVS wiring
 
 ```bash
-# br-wan must exist with the macvlan port
-sudo ovs-vsctl show | grep -A5 br-wan
+# br-int and br-ext must exist
+sudo ovs-vsctl show | grep -E "Bridge (br-int|br-ext)"
 
-# Bridge mappings must be set
+# br-ext should have a veth port linking it to br-wan
+sudo ovs-vsctl list-ports br-ext
+
+# The Linux WAN bridge should have the physical NIC and the veth peer
+bridge link show | grep br-wan
+
+# Bridge mappings must point at br-ext
 sudo ovs-vsctl get Open_vSwitch . external-ids:ovn-bridge-mappings
-# Expected: "external:br-wan"
+# Expected: "external:br-ext"
 ```
 
 ### 2. Verify chassis and gateway scheduling
@@ -1191,7 +1177,8 @@ If these don't match, see "Chassis name mismatch" below.
 ```bash
 sudo ovn-nbctl lr-nat-list vpc-$VPC | grep dnat_and_snat
 # Must show the public IP → private IP mapping
-# Must NOT have external_mac or logical_port set (centralized NAT)
+# For distributed NAT, external_mac and logical_port should be set
+# (the VM's MAC and ENI port name)
 ```
 
 ### 4. Verify ARP resolution
@@ -1202,20 +1189,21 @@ From another host on the same LAN, check if OVN responds to ARP:
 # On the remote host:
 ping -c 1 {public_ip}
 ip neigh show {public_ip}
-# Should show the OVN router MAC (02:00:00:xx:xx:xx)
+# Should show the VM's ENI MAC (distributed NAT) or the OVN router MAC
 ```
 
-If ARP fails, check the macvlan MAC alignment:
+If ARP fails, confirm the WAN bridge is forwarding:
 
 ```bash
-# On the Spinifex host:
-# Gateway router MAC
-sudo ovn-nbctl get Logical_Router_Port gw-vpc-{vpcId} mac
-# Macvlan MAC
-ip link show spx-ext-{nic} | grep ether
+# Physical NIC must be enslaved to br-wan
+ip -d link show {nic} | grep "master br-wan"
 
-# These MUST match. If they don't, vpcd failed to set the macvlan MAC.
-# Check vpcd.log for "macvlan MAC aligned" or errors.
+# br-wan must be UP and have an IP
+ip -br addr show br-wan
+
+# br-ext must have a veth port whose peer is enslaved to br-wan
+sudo ovs-vsctl list-ports br-ext
+bridge link show | grep br-wan
 ```
 
 ### 5. Verify packet flow with tcpdump
@@ -1226,15 +1214,16 @@ Capture at each layer to find where packets stop:
 # Layer 1: Does the ARP/ICMP arrive on the physical NIC?
 sudo tcpdump -i {nic} -n "host {public_ip}"
 
-# Layer 2: Does it reach br-wan?
+# Layer 2: Does it cross the Linux WAN bridge?
 sudo tcpdump -i br-wan -n "host {public_ip}"
 
-# Layer 3: Does it reach the macvlan output?
-sudo tcpdump -i spx-ext-{nic} -n -e "host {public_ip}"
+# Layer 3: Does it reach the OVS uplink bridge?
+sudo tcpdump -i br-ext -n -e "host {public_ip}"
 ```
 
-If traffic arrives on the physical NIC but not br-wan, the macvlan is
-filtering it. Check MAC alignment (step 4).
+If traffic arrives on the physical NIC but not br-wan, the NIC is not
+enslaved to the bridge. If it reaches br-wan but not br-ext, the veth
+pair between them is missing or down — re-run `setup-ovn.sh`.
 
 ### 6. Use ovn-trace for pipeline debugging
 
@@ -1277,77 +1266,46 @@ sudo systemctl start ovn-central ovn-controller
 
 ### Chassis Name Mismatch
 
-**Symptom:** Gateway chassis is set in NB but the SB has no matching chassis.
-`ovn-nbctl show` reports a gateway chassis name that doesn't appear in
-`ovn-sbctl show`. Gateway traffic (SNAT/DNAT) doesn't work.
+After mulga-999, vpcd uses the OVS-managed UUID (persisted at
+`/etc/openvswitch/system-id.conf` by the `openvswitch-switch` package and
+re-applied on every boot) as the canonical chassis identity. The chassis name
+across OVS `external_ids:system-id`, OVN SB `Chassis.name`, and NB
+`Gateway_Chassis.chassis_name` is always the same value, so the mismatch
+class that this section previously documented can no longer recur.
 
-**Cause:** Three names must align:
+`ReconcileFromKV` runs a `reconcileGatewayChassis` pre-step on every vpcd
+startup that deletes any stale `Gateway_Chassis` row left over from earlier
+installs (where `setup-ovn.sh` fabricated `chassis-$(hostname -s)`) and
+re-binds every gateway LRP against the live SBDB chassis. Pre-mulga-999
+brokenness recovers automatically on the first vpcd restart after upgrade —
+no manual `ovn-nbctl destroy` required.
 
-1. OVS `external_ids:system-id` → what ovn-controller registers as
-2. OVN SB `Chassis.name` → what actually exists in the SB DB
-3. vpcd's chassis name → `chassis-{nodeName}` from config
+### WAN NIC Not Enslaved to a Bridge
 
-If `setup-ovn.sh` used `chassis-$(hostname -s)` but vpcd uses
-`chassis-{nodeName}`, they'll differ on any host where hostname ≠ node name.
+**Symptom:** `setup-ovn.sh` exits with an error like "default route is on a
+physical NIC, not a bridge" and refuses to continue.
 
-**Diagnosis:**
+**Cause:** Spinifex requires the WAN NIC to be enslaved to a Linux bridge
+(typically `br-wan`). Macvlan is no longer supported, and attaching the NIC
+directly to OVS would break SSH and any other host services using the NIC.
 
-```bash
-# What OVS has
-sudo ovs-vsctl get Open_vSwitch . external-ids:system-id
-# What SB registered
-sudo ovn-sbctl list Chassis | grep name
-# What vpcd expects
-grep node ~/spinifex/config/spinifex.toml | head -1
+**Fix:** Move the host IP and default route onto a Linux bridge. Example
+netplan:
+
+```yaml
+network:
+  version: 2
+  ethernets:
+    wan:
+      dhcp4: false
+  bridges:
+    br-wan:
+      interfaces: [wan]
+      dhcp4: true
 ```
 
-**Fix:** `reset-dev-env.sh` passes `--chassis-id=chassis-node1` to
-`setup-ovn.sh`. For manual fix:
-
-```bash
-sudo ovs-vsctl set Open_vSwitch . external_ids:system-id=chassis-node1
-sudo systemctl restart ovn-controller
-```
-
-### Macvlan Unicast Filtering (Inbound Traffic Lost)
-
-**Symptom:** VM gets DHCP, outbound works (ping 8.8.8.8 from VM), but inbound
-from the LAN to the VM's public IP fails. ARP replies go out but ICMP never
-arrives at br-wan.
-
-**Cause:** macvlan in bridge mode only delivers unicast frames matching its own
-MAC. OVN announces the router MAC for external IPs, but the macvlan has a
-different (random) MAC. Broadcasts (ARP requests) pass through; unicast return
-traffic is filtered by the kernel macvlan subsystem.
-
-**Diagnosis:**
-
-```bash
-# Compare these two — they must match
-sudo ovn-nbctl get Logical_Router_Port gw-vpc-{vpcId} mac
-ip link show spx-ext-{nic} | grep ether
-
-# If they differ, inbound unicast is being dropped
-```
-
-**Fix:** vpcd automatically aligns the macvlan MAC on startup. If it failed:
-
-```bash
-# Get the gateway MAC from OVN
-GW_MAC=$(sudo ovn-nbctl get Logical_Router_Port gw-vpc-{vpcId} mac | tr -d '"')
-
-# Set it on the macvlan
-sudo ip link set spx-ext-{nic} down
-sudo ip link set spx-ext-{nic} address $GW_MAC
-sudo ip link set spx-ext-{nic} up
-```
-
-Also ensure the DNAT rule uses centralized NAT (no `external_mac`):
-
-```bash
-sudo ovn-nbctl lr-nat-list vpc-{vpcId}
-# The dnat_and_snat row should have EMPTY external_mac and logical_port columns
-```
+Apply (`sudo netplan apply`), confirm the host IP is now on `br-wan`
+(`ip -br addr show br-wan`), and re-run `setup-ovn.sh`.
 
 ### Stale ARP on Remote Hosts
 
@@ -1362,7 +1320,7 @@ sudo ip neigh flush dev {nic} {public_ip}
 ping {public_ip}   # should work now
 ```
 
-OVN's `nat-addresses=router` option sends periodic gratuitous ARPs that will
+OVN sends periodic gratuitous ARPs from the chassis hosting the VM that will
 eventually update remote ARP caches, but flushing is faster for testing.
 
 ### Security Group Rules Not Taking Effect

docs/getting-started/install-airgapped/README.md

@@ -0,0 +1,186 @@
+---
+title: "Air-Gapped Install"
+description: "Deploy Spinifex in environments without internet connectivity. Covers using a pre-built release tarball, mirrored APT packages, and locally-staged cloud images."
+category: "Getting Started"
+tags:
+  - install
+  - air-gapped
+  - offline
+resources:
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "External Connection Inventory"
+    url: "/docs/security/network-connections"
+---
+
+# Air-Gapped Installation
+
+> Deploy Spinifex in environments without internet connectivity.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+In air-gapped or disconnected environments, Spinifex can be deployed without internet access. This guide covers preparing offline packages on a connected machine, creating USB deployment media, and installing on the target server with package verification.
+
+## Instructions
+
+## Step 1. Download the Release (on a connected machine)
+
+Each Spinifex release publishes a self-contained tarball, the matching `setup.sh`, and a SHA-256 checksum to GitHub Releases. Resolve the latest tag and download the assets for your architecture:
+
+```bash
+ARCH=amd64   # or arm64
+TAG=$(curl -fsSL https://api.github.com/repos/mulgadc/spinifex/releases/latest \
+  | grep '"tag_name"' | cut -d'"' -f4)
+
+BASE="https://github.com/mulgadc/spinifex/releases/download/${TAG}"
+curl -fsSLO "${BASE}/spinifex-${TAG}-linux-${ARCH}.tar.gz"
+curl -fsSLO "${BASE}/spinifex-${TAG}-linux-${ARCH}.tar.gz.sha256"
+curl -fsSLO "${BASE}/setup.sh"
+
+sha256sum -c "spinifex-${TAG}-linux-${ARCH}.tar.gz.sha256"
+```
+
+## Step 2. Stage Dependencies (on the connected machine)
+
+Pre-download the APT packages installed by the production setup. `apt install --download-only` writes `.deb` files to `/var/cache/apt/archives/` without installing them, so the connected machine isn't modified:
+
+```bash
+sudo apt update
+sudo apt install --download-only -y \
+  nbdkit nbdkit-plugin-dev pkg-config \
+  qemu-system-x86 qemu-utils qemu-kvm \
+  libvirt-daemon-system libvirt-clients libvirt-dev \
+  ovn-central ovn-host openvswitch-switch \
+  dhcpcd-base make gcc jq curl iproute2 netcat-openbsd \
+  wget unzip xz-utils file
+```
+
+Download AWS CLI v2:
+
+```bash
+curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \
+  -o awscliv2.zip
+```
+
+Mirror the cloud image you intend to run as guest VMs:
+
+```bash
+mkdir -p images
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260501-2465/debian-13-genericcloud-amd64-20260501-2465.qcow2" \
+  -o images/debian-13-amd64.qcow2
+```
+
+## Step 3. Assemble Transfer Media
+
+```bash
+mkdir -p /media/spinifex-deploy/{tarball,apt-packages,aws,images}
+cp spinifex-${TAG}-linux-${ARCH}.tar.gz /media/spinifex-deploy/tarball/
+cp setup.sh /media/spinifex-deploy/
+cp /var/cache/apt/archives/*.deb /media/spinifex-deploy/apt-packages/
+cp awscliv2.zip /media/spinifex-deploy/aws/
+cp images/*.qcow2 /media/spinifex-deploy/images/
+```
+
+## Step 4. Install on the Air-Gapped Target
+
+Mount the media, install APT dependencies, install AWS CLI, then run `setup.sh` with the local tarball:
+
+```bash
+sudo mount /dev/sdb1 /mnt/usb
+
+sudo dpkg -i /mnt/usb/apt-packages/*.deb
+sudo apt-get install -f --no-download   # resolve any leftover deps from local cache
+
+cd /tmp && unzip /mnt/usb/aws/awscliv2.zip && sudo ./aws/install
+
+INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-*-linux-*.tar.gz \
+INSTALL_SPINIFEX_SKIP_APT=1 \
+INSTALL_SPINIFEX_SKIP_AWS=1 \
+bash /mnt/usb/setup.sh
+```
+
+Run `setup.sh` without `sudo` — the script handles privilege escalation internally and ends by `exec`-ing into a `newgrp spinifex` subshell so your current shell picks up `spinifex` group membership. Without that membership, AWS CLI cannot traverse `/etc/spinifex/` (mode `0750`) to read `ca.pem` and Step 9 will fail with a TLS error. Type `exit` to leave the subshell when finished.
+
+## Step 5. Setup OVN Networking
+
+`spx admin init` requires OVN/OVS to be configured before the daemon can manage tenant networks. `setup-ovn.sh` ships in the tarball and runs purely against local commands (no network access required):
+
+```bash
+sudo /usr/local/share/spinifex/setup-ovn.sh --management
+```
+
+If your WAN interface is a physical NIC rather than a bridge, pass `--wan-bridge=br-wan --wan-iface=<nic>`.
+
+## Step 6. Initialize Without Telemetry
+
+```bash
+sudo spx admin init --node node1 --nodes 1 --no-telemetry
+```
+
+This generates the cluster configuration, TLS certificates, installs the CA into the system trust store, and writes AWS CLI credentials. See [Single-Node Install](/docs/install) for what `spx admin init` does.
+
+## Step 7. Start Services
+
+`setup.sh` enables `spinifex.target` but does not start it on a fresh install. Start it now so predastore is online before you import images:
+
+```bash
+sudo systemctl start spinifex.target
+```
+
+## Step 8. Import Cloud Images
+
+Register the pre-staged image with Spinifex:
+
+```bash
+sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
+  --distro debian --version 13 --arch x86_64
+```
+
+## Step 9. Verify
+
+```bash
+export AWS_PROFILE=spinifex
+aws ec2 describe-instance-types
+aws ec2 describe-images
+```
+
+If both calls return data, your air-gapped install is working. Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to launch your first instance.
+
+## Troubleshooting
+
+### Missing APT Dependencies
+
+`dpkg -i` does not resolve transitive dependencies. After installing, run:
+
+```bash
+sudo apt-get install -f --no-download
+```
+
+The `--no-download` flag forces apt to use only what's in the local cache. If a dependency is genuinely missing, add it to the `apt --download-only` step on the connected machine and re-stage.
+
+### Setup.sh Tries to Download Anyway
+
+Confirm both skip flags are exported and that `INSTALL_SPINIFEX_TARBALL` points at a readable file:
+
+```bash
+sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
+     INSTALL_SPINIFEX_SKIP_APT=1 \
+     INSTALL_SPINIFEX_SKIP_AWS=1 \
+     bash -x /mnt/usb/setup.sh
+```
+
+### Init Telemetry Attempted
+
+`spx admin init` posts a one-shot record to `https://install.mulgadc.com/install`. Pass `--no-telemetry` on every `init` and `join` invocation, or export `SPX_NO_TELEMETRY=1` in the operator's shell profile.
+
+### Image Import Fails
+
+`spx admin images import --file` requires the distro/version/arch flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.

docs/getting-started/install-multi-node/README.md

@@ -22,6 +22,7 @@ resources:
 ## Table of Contents
 
 - [Overview](#overview)
+- [Prerequisites](#prerequisites)
 - [Instructions](#instructions)
 - [Troubleshooting](#troubleshooting)
 
@@ -31,12 +32,28 @@ resources:
 
 A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for peers to join, then distributes credentials, CA certificates, and configuration.
 
-**Network requirements:**
+**Network Requirements:**
 
 - Minimum 1 NIC per server (2 recommended for production)
 - UDP port 6081 open between hosts (Geneve tunnels)
 - TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
 
+## Prerequisites
+
+> [!IMPORTANT]
+> **Prerequisite — WAN bridge required on every node.**
+>
+> Before running the installer on any server, that server's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
+>
+> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release.
+>
+> **Verify on every node before continuing:**
+>
+> - `ip -br link show br-wan` — bridge exists and is `UP`
+> - `ip route` — default route's `dev` is `br-wan`
+>
+> **Setup references:** [VPC Networking → Bridge Setup](/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.
+
 ## Instructions
 
 ## Step 1. Install Spinifex on Each Server
@@ -59,7 +76,7 @@ export AWS_AZ=us-east-1a
 
 ## Step 3. Setup OVN Networking
 
-Server 1 runs OVN central and must be set up first. If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it. Otherwise use `--wan-bridge=br-wan --wan-iface=eth1` (dedicated WAN NIC) or `--macvlan --wan-iface=eth0` (single-NIC).
+Server 1 runs OVN central and must be set up first. If your WAN interface is already a bridge, setup-ovn.sh auto-detects it. Otherwise use `--wan-bridge=br-wan --wan-iface=eth1` (dedicated WAN NIC).
 
 **Server 1:**
 
@@ -104,14 +121,24 @@ sudo spx admin init \
   --port 4432 --region $AWS_REGION --az $AWS_AZ
 ```
 
-Save the admin credentials — they will not be shown again.
+The init output displays the join command including the token:
+
+```
+📡 Formation server started on 10.0.0.1:4432
+   Waiting for 2 more node(s) to join...
+   Token expires in 30m0s
+
+   Other nodes should run:
+   sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>
+```
 
 **Server 2 — Join** (while init is running):
 
 ```bash
 sudo spx admin join \
   --node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
-  --host $SPINIFEX_NODE1:4432 --region $AWS_REGION --az $AWS_AZ
+  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
+  --region $AWS_REGION --az $AWS_AZ
 ```
 
 **Server 3 — Join** (while init is running):
@@ -119,9 +146,12 @@ sudo spx admin join \
 ```bash
 sudo spx admin join \
   --node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
-  --host $SPINIFEX_NODE1:4432 --region $AWS_REGION --az $AWS_AZ
+  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
+  --region $AWS_REGION --az $AWS_AZ
 ```
 
+**Note:** The join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use `--token-ttl 2h`
+
 ## Step 5. Start Services
 
 On **all servers**:
@@ -152,7 +182,7 @@ Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import a
 The init command must still be running when join executes. If init exited, re-run with `--force`.
 
 ```bash
-curl -s http://$SPINIFEX_NODE1:4432/health
+curl -sk https://$SPINIFEX_NODE1:4432/health
 ```
 
 ### OVN Chassis Not Registering
@@ -165,7 +195,7 @@ sudo ss -tlnp | grep 6642
 ### CA Certificate Not Trusted
 
 ```bash
-sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
+sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
 sudo update-ca-certificates
 ```