gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,095 Commits 4 Branches 45 Tags
2dada2cf9dd5fc8b1811f951a4ebee52fbdec223
Commit Graph

1095 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Julian Sommer 2dada2cf9d release: update go.mod for v1.6.0 v1.6.0 2026-06-03 09:15:38 +10:00
Julian Sommer 9508aaacce chore: bump go 1.26.4 2026-06-03 08:59:25 +10:00
joshsiv-mulga 40deb1e4c2 fix(e2e): add explicit IGW default route in run-lb-e2e 
After PR #270 the VPC-LR-wide 0.0.0.0/0 -> IGW static route is no longer
installed implicitly at attach-internet-gateway. Subnets whose effective
route table lacks an internet-bound route get an OVN drop policy at
SubnetEgressPriorityDrop (1100), which kills LB target reply traffic
before un-DNAT can fire and stalls the LB internal-client probe until
job timeout.

Add the standard AWS pattern: describe-route-tables -> create-route
0.0.0.0/0 -> IGW. This fires vpc.add-igw-route + vpc.ungate-subnet-egress,
removing the drop policy and installing a reroute policy at
SubnetEgressPriorityIGW (1000).
 2026-06-02 23:49:23 +10:00
joshsiv-mulga 7cd054cbf1 chore(e2e): dump Logical_Router_Policy + Logical_Router_Static_Route in W5 diag 2026-06-02 22:15:22 +10:00
joshsiv-mulga 41f840ee85 fix(setup): grant spinifex-vpcd sudo for ovn-appctl (siv-167) 
The W2 inject-garp fix landed in 72a6a59d adds a shellout to
`ovn-appctl -t ovn-controller inject-garp <port>` via the L0
execRunner, which prefixes `sudo` when not running as root.
spinifex-vpcd runs as user spinifex-vpcd, so every emission was
hitting `sudo: a password is required` and silently returning a
WARN that nobody read until W5 diag capture surfaced the journal.

Net: W2 has been a complete no-op since it shipped. Every
recycled-EIP scenario has continued to leave stale upstream ARP.

Evidence: diag bundle from run 26813235295 — 12/12 GARP attempts
denied by sudo, 0 successful emissions; pcap shows runner ARP for
recycled EIP .232 stuck on the prior VPC's gw chassisredirect MAC
(02:75:d9:ef:d8:4b) while OVN NB has handed .232 to a new VPC
with a different gw MAC; pcap also shows runner sending 174 SYNs
to recycled EIP .233 with 0 SYN+ACKs received.

Fix: add /usr/bin/ovn-appctl to the spinifex-vpcd NOPASSWD grant.
Trivial; ovn-appctl is already trusted via the existing NB/SB
read-write grants.

Follow-ups (separate change): tighten host.InjectGARP error
classification so sudo-deny is loud (Error or fatal) rather than
silently warn-and-continue; add a boot-time self-check that
verifies sudo permissions for every host-tool the daemon will
shell out to.
 2026-06-02 21:00:20 +10:00
joshsiv-mulga ba7253e2c6 ci(e2e): fix Start Packet Capture EACCES + PID-tracking on tcpdump (siv-167) 
Two bugs in afd63ed9:

1. `sudo mkdir` makes /tmp/spinifex-diag root-owned 0755. The
   subsequent `> /tmp/spinifex-diag/tcpdump.log` is performed by
   the calling tf-user shell BEFORE sudo gets the command, failing
   with Permission denied. action shell runs with `set -e` so the
   step aborts.

2. `sudo nohup ... &; echo $!` captures the wrong PID — `$!` is the
   backgrounded sudo, which may exit when its target dies, leaving
   stale tcpdump.pid.

Fixes:
- chmod 0777 on /tmp/spinifex-diag after creating it (next steps'
  redirects from tf-user shell now work).
- Wrap the tcpdump launch in `sudo bash -c '...'` so backgrounding
  and `$!` happen inside the sudo'd shell — `$!` is the actual
  tcpdump PID (nohup execs, sudo bash holds it). Captures stdin
  to /dev/null, stdout/stderr to /tmp/spinifex-diag/tcpdump.log.
- sleep 2 (was 1) — gives tcpdump time to fully init pcap header.
- Hard fail (exit 1) if PID is missing or process dead, so the
  step's failure is visible instead of a silent green.

bash -n verified.
 2026-06-02 19:45:53 +10:00
joshsiv-mulga afd63ed967 ci(e2e): Phase 1 diag — tcpdump br-wan + bridge fdb + hairpin modes (siv-167) 
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB
single-host return-path drop class:

e2e_single only:
- New step 'Start Packet Capture' between Bootstrap and Run E2E
  Tests. SSHes to test VM and starts background tcpdump on br-wan
  capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap.
  PID stored at /tmp/spinifex-diag/tcpdump.pid.

All three jobs (single, pseudo-multi, real-multi):
- Collect Diagnostics: kill tcpdump (single only — PID file
  guarded) -INT for graceful flush, falls through to -KILL after 5s.
- Adds journal-spinifex-full.log (no -n truncation) alongside
  the existing -n 10000 capped one.
- Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood
  for every br-wan brif — needed to confirm/disconfirm the
  hairpin-loss hypothesis in the plan doc.
- Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap
  (created by sudo tcpdump as root) is readable by tf-user.

bash -n verified on all four rendered scripts. YAML parses.
Zero new gold-image deps (tcpdump, bridge present in base image).
 2026-06-02 19:28:54 +10:00
joshsiv-mulga 2574956f08 ansible cluster-bootstrap 2026-06-02 15:38:56 +10:00
joshsiv-mulga 3f4613c28b ci(e2e): capture OVN/OVS/journal diagnostics on every run 
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:

- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
  dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list

Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).

OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.

Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
 2026-06-02 14:56:41 +10:00
joshsiv-mulga 72a6a59d80 fix(network): inject GARP after AddEIP to refresh upstream ARP on EIP recycle 
ovn-controller emits gratuitous ARPs automatically when a NAT-bound LSP
migrates chassis, but not when an external IP is reassigned to a different
LR whose chassisredirect MAC differs from the prior occupant's. In CI
single-host runs the WAN pool slice is small enough that the same EIP
recycles across VPCs within the 60-300s kernel ARP timeout, so upstream
caches keep pointing at the dead chassis-redirect MAC and packets to the
new VM never arrive.

Fix: add a post-AddEIP hook that runs `ovn-appctl -t ovn-controller
inject-garp <port>` (OVN 22.09+). Port is the EIP's LSP in distributed
NAT or "cr-gw-<vpcID>" in centralized NAT. Best-effort — failure logs a
warning and the NAT install still succeeds.

- network/host/garp.go: new L0 shell-out helper InjectGARP.
- network/policy/nat.go: GARPEmitter option type, called after the
  flows-ready barrier in AddEIP.
- vpcd: wire the production emitter that derives the right OVN port
  from the EIPSpec.
- invariants: add ovn-appctl to the host-tool ban list so future
  callers are forced into network/host/.
 2026-06-02 13:38:52 +10:00
Josh Siviour dd65a2dfa6 feat(elbv2): ListenerRules CRUD with HAProxy ACL routing (#275) 
* feat(elbv2): ListenerRules CRUD with HAProxy ACL routing

Implement CreateRule, ModifyRule, DeleteRule, DescribeRules, and
SetRulePriorities for ALB listeners. Rules support host-header,
path-pattern, http-header, http-request-method, source-ip, and
query-string condition fields. Actions limited to forward (single
target group); redirect and fixed-response deferred.

HAProxy frontends emit per-rule ACLs with `use_backend ... if <acls>`
ordered by priority. Wildcards in host/path map to HAProxy match
modes (`-m beg/end/sub/str`). Condition values are validated at
store-time and re-validated at render-time to prevent config
injection. Target groups referenced by rule actions are protected
against deletion. Deleting a listener cascades its rules.

Frontend adds a Rules tab beneath each listener row showing rules
with conditions and target groups, plus add/edit/delete dialogs
with field-typed condition inputs.

mulga-950

* test(elbv2): rule CRUD e2e + render coverage for all condition fields

Adds Internal_ALB_ListenerRules subtest exercising CreateRule, ModifyRule,
DeleteRule, and DescribeRules over a real ALB with HAProxy. Two TGs with
disjoint backends (app#0 -> tgA, app#1 -> tgB) let the probe responder
identify which TG actually served the request, so path-pattern routing
can be asserted without inspecting internal state.

Extends app-userdata.sh with alpha/ and beta/ sub-paths and extends
client-userdata.sh trigger to accept optional http_path and host fields
so probes can target specific paths and Host headers. Both changes are
backward-compatible: existing callers default to / with no Host override.

Adds unit-test coverage for every condition-field render path
(http-header, http-request-method, query-string with and without key,
source-ip) plus error-path validation tests for CreateRule, ModifyRule,
DeleteRule, DescribeRules, and SetRulePriorities. Lifts diff coverage
above the 60% gate.

* test(elbv2): wait for tgB healthy after rule creation, not before

Spinifex only health-checks a target group once it is referenced by a
listener default action or a rule action. The previous ordering waited
for tgB healthy before CreateRule wired it, so health checks never ran
and the wait timed out. Move the WaitForTargetsHealthy(tgB) call after
createPathRule.

* fix(elbv2): include rule-action TGs in TargetGroupsForLB

healthChecker uses Store.TargetGroupsForLB to map HAProxy server
statuses back to TargetGroupRecord entries. The previous implementation
walked only listener DefaultActions, so a target group referenced only
by a CreateRule action was invisible to the health checker — HAProxy
probed its backends but the resulting health never propagated to the
store. DescribeTargetHealth reported the targets as 'initial' forever.

Extend TargetGroupsForLB to walk rule actions per listener as well.
Adds a unit test that wires a TG via CreateRule and asserts the
resolver returns it.

* test(elbv2): fix wait-for-rule logic to poll away from default backend

waitForPathRouted required the caller to name the expected new host, but
at the post-CreateRule wait point we only know the default host (appA);
the rule target hostname (appB) is discovered by the subsequent probe.
The previous code passed appA as both "new" and "old", so the wait
checked "see appA" against "/alpha/" responses — which never matched
because the rule had successfully routed appA out.

Replace with two narrower helpers: waitForPathRoutedAway (poll until
host X stops appearing) and waitForPathRoutedTo (poll until host Y
appears). Update the three call sites to use the variant that matches
what the caller actually knows at that point.

* fix(elbv2): IAM-style error envelope + ModifyTargetGroup + listener attr stubs

ELBv2 success responses use the IAM-style XML envelope (<ErrorResponse>),
but error responses were emitted with the EC2 envelope (<Response>),
which aws-sdk-go-v2 cannot deserialize for ELBv2 and surfaces as
"UnknownError: UnknownError" — defeating provider-side InvalidAction
ignores (e.g. terraform-provider-aws v5.100's DescribeCapacityReservation
swallow). Route elasticloadbalancing errors through GenerateIAMErrorResponse.

Add ModifyTargetGroup end-to-end (interface, impl, NATS shim, daemon
handler, gateway dispatch) so terraform-provider-aws can set health-check
fields after CreateTargetGroup without infinite drift. Persist
HealthCheckEnabled on TargetGroupRecord and emit it in tgRecordToSDK so
the provider sees the configured value across refresh cycles.

Stub DescribeListenerAttributes / ModifyListenerAttributes with
hand-rolled input/output structs (aws-sdk-go v1.55 lacks these types).
Returns empty attribute list — sufficient for default-config workflows.

* test(elbv2): cover ModifyTargetGroup + listener attribute stubs

Extend handler tests with all-fields, invalid path/matcher, wrong-account
ModifyTargetGroup cases plus HealthCheckEnabled assertion on
CreateTargetGroup. Add gateway-layer validation tests for the new
DescribeListenerAttributes, ModifyListenerAttributes, and ModifyTargetGroup
dispatchers. Raises diff coverage from 58.8% to 63.7%.
 2026-06-02 12:15:42 +10:00
brynmailer-mdc 0c577da6e2 test: cover encryption wiring on feat/encryption-at-rest 
Lifts diff-coverage on PR #276 from 37.5% to 63.5% (threshold 60%) so
CI's Check Diff Coverage step passes.

- utils: LoadViperblockMasterKey — empty path, happy path + cache reuse,
  load error, and bad-perm failure-not-cached.
- handlers/ec2/image: clusterEncryptionEnabled — nil config, empty key
  file, masterkey load error (warn + false), and key-loaded (true).
- handlers/ec2/volume:
  - GetVolumeConfig surfaces "failed to unmarshal config" when the body
    is neither a valid VBState nor a wrapper.
  - mergeVolumeConfig refuses to re-marshal an encrypted VBState (would
    strip the AES-GCM tag and brick the volume).
  - CreateVolume returns ServerInternal when EncryptionKeyFile is set
    but unreadable.
 2026-06-02 11:03:58 +10:00
brynmailer-mdc 557d3e4d59 feat: derive Encrypted from VBState.EncryptionEnabled, drop IsEncrypted reads 
VolumeMetadata.IsEncrypted is being removed from viperblock — it was
hardcoded false everywhere in spinifex and misrepresented the per-volume
encryption status to AWS SDK callers. The authoritative flag is now
VBState.EncryptionEnabled, persisted by viperblock at New() when
MasterKey != nil.

- volume CreateVolume drops the dead IsEncrypted struct member and
  surfaces Encrypted=(mkey != nil) on the EC2 response.
- volume getVolumeByID now decodes the full VBState (falling back to a
  wrapper-only blob for legacy pre-encryption volumes) so it can read
  state.EncryptionEnabled directly.
- snapshot CreateSnapshot pulls Encrypted from the source volume's
  VBState.EncryptionEnabled (already loaded one level up).
- image synthesizeRootBlockDeviceMapping takes the cluster encryption
  posture (mkey != nil) since AMI metadata carries no per-image flag;
  cached masterkey load keeps the hot path cheap.
- Test fixtures drop the now-defunct IsEncrypted lines; the volume
  fixture that asserts Encrypted=true switches to a full VBState seed
  with EncryptionEnabled=true.
 2026-06-02 11:03:58 +10:00
brynmailer-mdc b5220996e1 fix: only clone AMI on os.ErrNotExist in prepareRootVolume 
LoadStateRequest can fail for many reasons — HMAC integrity, key
mismatch, transient backend 5xx, JSON parse failure. The previous
fall-through treated every error as "volume doesn't exist, clone from
AMI", which would overwrite the legitimate encrypted volume's
config.json with AMI base-map state on a tamper or transient backend
fault. Discriminate os.ErrNotExist explicitly and abort otherwise.
 2026-06-02 11:03:58 +10:00
brynmailer-mdc ef50392f9e feat: wire viperblock at-rest encryption key through daemon and AWS handlers 
Adds EncryptionKeyFile to ViperblockConfig (cluster config) and
viperblockd.Config. viperblockd loads the master key once at
launchService start via masterkey.LoadShared and passes
MasterKey + EncryptionEnabled into the VB constructed for every
mounted volume.

AWS handlers (volume.CreateVolume, instance.newViperblock,
image.snapshotStoppedVolume) and the spx admin AMI-import path now
resolve the key via utils.LoadViperblockMasterKey — a process-wide
cache keyed by path so the per-VB lookup is one map hit after the
first load.

volume.mergeVolumeConfig: switch from json.Unmarshal to json.Decoder
so a trailing AES-GCM tag cannot route the body to the wrapper-only
path, and refuse outright when state.EncryptionEnabled — re-marshaling
an encrypted VBState here would strip the tag and brick the volume.

cmd/spinifex viperblock CLI grows a --encryption-key-file flag (env
SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE) under a distinct viper key
so it does not collide with predastore's identically-named flag.
 2026-06-02 11:03:58 +10:00
Josh Siviour 8883984f3f feat(eks): eks-server + eks-agent AMI manifests (#273) 
* feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES

Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12).
Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in
isolation before CreateCluster orchestration lands.

Builder:
- build-system-image.sh gains optional INSTALL_FILES (src:dst pairs,
  mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES.
- ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts
  the manifest references are on disk before rc-update enables them.

eks-server image (Alpine 3.21.7, q35, 4G):
- K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the
  upstream sha256sum-amd64.txt trust anchor).
- nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream
  release zip; unzip installed + purged in same setup step).
- OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot.
- Cron periodics under /etc/periodic/{15min,daily}/ for state-report +
  etcd-snapshot.
- k3s-first-boot publishes node-token + admin kubeconfig over NATS on
  first successful K3s healthz, then self-disables (sentinel + rc-update
  del).
- config.yaml.skel without webhook-token-auth — wiring it before the real
  webhook lands (cs-eks-6c) would block anonymous probes against /healthz.
  Orchestrator drops the live config + restarts k3s.

eks-agent image (Alpine 3.21.7, q35, 2G):
- Same pinned K3s release for server/agent lock-step.
- k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN —
  fail-closed pre-orchestration.

eks-token-webhook stub (cmd/eks-token-webhook/main.go):
- 503 + Retry-After:30 on every path. Statically linked. Lets the AMI
  build succeed and kube-apiserver boot cleanly. Replaced wholesale by
  cs-eks-6c.

Init system divergence from spec wording: Alpine ships OpenRC, not
systemd. Unit files use .initd extensions and cron periodics instead of
.service / .timer. Bead-level intent (auto-enable on boot, periodic
execution) preserved.

Smoke-tested:
- eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on
  6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401
  is K3s v1.32 default RBAC behaviour against anonymous, not a regression).
- eks-agent: builds 280M qcow2; qemu boot mounts root cleanly.

* feat(eks): make targets for eks-server + eks-agent AMI builds

- build-system-image gains subdir-manifest fallback: tries
  scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf
  so the new EKS layout works alongside the existing flat lb manifest.
- build-eks-server-image / build-eks-agent-image — explicit convenience
  targets that delegate to the generic one.
 2026-06-01 16:24:49 +10:00
Josh Siviour 785a5655e4 feat(sts): EKS Q4 — AssumeRoleWithWebIdentity + presigned GetCallerIdentity verify (#272) 
* feat(sts): EKS Q4 STS extensions — interface + JWKS reader + stubs

Lays the structural slice of AssumeRoleWithWebIdentity + presigned-URL
GetCallerIdentity verify for IRSA. The verify-path bodies (JWT signature
check against fetched JWKS, claims validation, Federated-principal trust
policy evaluation, SigV4 presigned-URL canonical-request reconstruction)
land in follow-up commits on this branch.

Adds:
- handlers/sts/oidc_jwks.go — JWK/JWKS types, ParseEKSIssuerURL (defensive
  iss parsing), FetchClusterJWKS (KV-backed read from
  eks-account-{accountID}/clusters/{name}/oidc-jwks.json via the EKS store
  helpers).
- handlers/sts/oidc_jwks_test.go — unit tests over a real JetStream
  test fixture.
- handlers/sts/assume_role_with_web_identity.go — interface entry, stub
  returns ErrorNotImplemented.
- handlers/sts/presigned_url_verify.go — PresignedCallerIdentity type +
  VerifyPresignedGetCallerIdentity stub.
- gateway/sts/AssumeRoleWithWebIdentity.go — dispatch shell.

Modifies:
- handlers/sts/service.go — STSService interface gains
  AssumeRoleWithWebIdentity + VerifyPresignedGetCallerIdentity.
- gateway/sts.go — registers AssumeRoleWithWebIdentity in stsActions +
  stsSkipPolicyCheck (anonymous; trust-policy-gated in handler).
- gateway/sts_request_test.go, gateway/sts/handler_test.go,
  gateway/auth_test.go — stub implementations updated to satisfy
  extended STSService interface.

Anonymous routing for AssumeRoleWithWebIdentity (eks-v1.md Q5 mux rework)
lands in Sprint 6e — for now the action is registered on the standard STS
gateway and rejects non-SigV4 requests upstream.

Refs: cs-eks-sts-q4 (mulga monorepo bead)

* feat(sts): AssumeRoleWithWebIdentity + presigned GetCallerIdentity verify

Implements EKS Q4 STS extensions for IRSA + EKS token-webhook flows.

AssumeRoleWithWebIdentity:
- ES256 JWT verification against per-cluster JWKS (eks-account-{accountID}
  KV bucket), keyed by issuer host-path.
- Federated trust-policy evaluator (explicit-deny-wins, exact-match
  Principal.Federated, fail-closed on Principal: "*").
- Condition support scoped to StringEquals on {iss}:sub (exact) and
  {iss}:aud (set-membership). All other operators rejected at validator.
- OIDC provider registry check against iam-account-{roleAccountID}; v1
  does not support cross-account registration.
- New awserrors: InvalidIdentityToken, IDPRejectedClaim,
  IDPCommunicationError.

VerifyPresignedGetCallerIdentity:
- Hand-rolled SigV4 canonical-request reconstruction (no sdk dep).
- Mandatory X-K8s-Aws-Id in SignedHeaders + value pinned to
  expectedClusterName during canonicalisation — cross-cluster replay
  surfaces as a signature mismatch.
- AKID-prefix dispatch (ASIA → session credential, AKIA → IAM user).
- Strict envelope checks: algorithm, credential scope service/terminator,
  X-Amz-Date / credential-scope date agreement, expiry within
  X-Amz-Expires, 5-minute wire skew.

Tests:
- 14 web-identity scenarios (happy paths, tampering, expiry, wrong aud,
  unknown kid, HS256 rejection, federated-wildcard fail-closed,
  explicit-deny-wins, provider-not-registered).
- 12 presigned-URL scenarios including cross-cluster replay defence,
  missing-x-k8s-aws-id, tampered query, bad service in credential, AKIA
  + ASIA happy paths.
- Cross-cluster anti-replay invariant test (rejects matrix of variant
  cluster names — case-different, prefix, suffix).
- AWS SigV4 known-vector test for deriveSigningKey.

Gateway:
- TODO documents that AssumeRoleWithWebIdentity is currently dispatched
  on the SigV4-gated path; a pre-auth route is needed so a projected
  ServiceAccount token alone can authenticate the caller.

IAM:
- Relaxes trust-policy Condition validator: now permitted only when
  Action is exactly [sts:AssumeRoleWithWebIdentity] and operator is
  StringEquals. Preserves loud failure for sts:AssumeRole.
 2026-06-01 14:29:35 +10:00
tomnewton-mulga 9d8597317a Feat/mig capable (#271) 
* feat(gpu): add NVIDIA MIG multi-instance GPU support

Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).

Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
  (EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
  DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
  ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
  MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
  admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows

* test(gpu): unit tests for MIG parsers and pool manager

Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.

Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.

* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle

Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.

- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
  slice; slow path dequeues a free MIG GPU, releases the lock, calls
  nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
  commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
  DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
  different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
  AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
  gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
  callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
  GPUs, ListProfiles to collect MIG profile specs for type generation;
  removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
  call sites, and fakeGPUClaimer all updated

* fix(gpu): correct mdev parent detection and make --profile optional for mig enable

enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).

Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).

Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).

* chore: writeups for smci and arcee

* test: add coverage for MIG exec, manager slow-path, instancetypes and admin

- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
  IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter

Diff coverage: 75.6% (threshold: 60%).

* fixed gpu passthrough doc

* Revert "fixed gpu passthrough doc"

This reverts commit 1e20f5ab13.

* feat: Added RTX 6000 GPU MIG support and fixed docs

* feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API

Add Manager.Snapshot() for lock-safe pool inspection. Extend
NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory
with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo
(model, VRAM, profile, mdev path for the attached GPU). Populate
both in handleNodeStatus and handleNodeVMs via buildGPUInventory,
buildPoolLookup, and resolveVMGPU helpers.

* feat: surface GPU info on EC2 instance detail page (mulga-new-03)

Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev
path or PCI address) populated from the GetVMs admin API.  Adds formatVRAMMiB
helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend.

* feat: GPU columns + inventory card on Nodes page (mulga-new-04)

Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable
(alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with
per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type.

* feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05)

Splits the flat instance type dropdown into Standard and GPU-Accelerated groups.
GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces
an info line with GPU name and VRAM.

* testing image builder scripts

* fixed nvidia build script

* more descriptive errors for image build script

* more fixes

* more fixes

* fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist

Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU
passthrough, and fix the instance detail page which was incorrectly
showing instance type in the PCI address field. Rebuilt frontend dist
to include GPU UI changes (inventory card, resource columns, GPU-aware
instance type picker).

* fix: improve kernel version detection in nvidia image build script

Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add
diagnostic output to help debug kernel detection failures.

* fixing nvidia image build script

* fix ami name issue

* correct kernel versions

* another fix

* image build works, added docker and nvidia toolkit for QoL

* fix: add gpg

* fixing amd image builder

* add universe

* another fix

* nvm

* fix docker user group issues

* make rocm land where it is supposed to

* fix docker groups

* feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection

Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages,
hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh
to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP
manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern).

* Updated passthrough doc regarding GPU images

* fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118)

* chore: remove stale spinifex@192.168.3.107 directory

Scripts were accidentally committed with a node hostname as the path
prefix during initial testing. Correct copies live in scripts/images/.
 2026-06-01 13:48:13 +10:00
Josh Siviour e7a25316f0 feat(eks): gateway + handler scaffolding for EKS control plane (#267) 
* feat(eks): gateway + handler scaffolding for EKS control plane

Lands the EKS service contract — gateway REST-JSON dispatcher, handler
interface + stub impl, JetStream KV bucket layout, and daemon wiring —
ahead of the K3s control-plane bodies.

Every action currently returns NotImplemented (HTTP 501 + JSON
NotImplementedException) so downstream work can call into a stable
contract while real bodies land sprint by sprint.

- Gateway: 34-action REST-JSON routing table under svc=eks, per-action
  closures dispatch to handlers_eks.NATSEKSService. ErrorHandler /
  writeClusterUnavailable / writeThrottleError gain EKS JSON branches so
  clients see application/x-amz-json-1.1 envelopes.
- Handler: EKSService interface + NATS wrapper + stub impl with
  var _ EKSService compile checks on both implementations.
- KV: eks-account-{accountID} lazy per-account bucket factory plus the
  shared spinifex-eks-leader bucket (TTL 60s, History 1) bootstrapped
  at daemon start. Key-path helpers cover the full Q2 layout
  (cluster meta, nodegroups, access entries, OIDC providers + signing
  key + JWKS, admin kubeconfig, K3s node-token, events).
- Daemon: 34 eks.<Action> subscriptions on the spinifex-workers queue
  group, init via initServiceWithRetry, mirrors the ELBv2 layout.
- awserrors: new ErrorNotImplemented constant (HTTP 501).
- tags: ManagedByEKS constant for downstream SG/NLB/VM tagging.

* test(eks): cover scaffolding wrappers + daemon NATS handlers

Adds two test files closing the diff-coverage gap on the EKS
scaffolding commit:

- spinifex/gateway/eks/wrappers_test.go drives every per-action
  gateway wrapper (cluster, nodegroup, access, addons, oidc, tags —
  34 actions, 50+ subtest invocations) through an embedded NATS
  server with a `eks.>` catch-all stub responder. Covers the
  wrappers themselves plus handlers/eks/service_nats.go via the
  NewNATSEKSService construction inside each wrapper. Bad-JSON
  branches in unmarshalIfBody are also covered.

- spinifex/daemon/daemon_handlers_eks_test.go constructs a minimal
  Daemon{eksService: real impl} and dispatches each of the 34
  handleEKS* handlers via embedded NATS, asserting the NotImplemented
  error code rides back through handleNATSRequest.

Lifts diff coverage from 27.3% to 62.6%, above the 60% threshold.
 2026-06-01 10:47:23 +10:00
Josh Siviour 3415d9f3de fix(network): IGW route-table AWS parity + per-subnet egress gating (#270) 
* fix(network): IGW default route AWS-parity via per-subnet LR policy

ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.

Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.

Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
  output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
  vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet

Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
  separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
  route; EnsureSubnetEgress installs the per-subnet policy with the
  resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
  Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics

* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry

handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
  (mirror of the IGW DeleteRoute path) so the subscriber tears down both
  the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
  the subscriber can install the matching Logical_Router_Policy.

network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
  install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
  Same nexthop and output port as the IGW priority sibling; lower priority
  so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
  validation, lookup, and slog tagging.

network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
  to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
  has no route to egress NAT'd packets and private-subnet traffic is
  dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
  SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.

Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
  router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
  via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
  empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.

* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate

In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.

DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).

* fix(routetable): publish per-subnet events for implicit main-RT membership

E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.

Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).

Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
  main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
  for main RT's routes after deletes for departing RT

* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list

ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.

* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests

EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.

start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.

Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.

Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.

* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV

Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.

LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.

* fix(routes): serialize OVN egress wiring via request-reply

vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).

Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.

handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.

* Revert "fix(routes): serialize OVN egress wiring via request-reply"

This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.

* fix(igw): restore default static route in AttachIGW/DetachIGW

OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.

Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.

Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).

* feat(network): per-subnet egress drop gating for AWS RT parity

VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.

Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.

Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
  vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
  IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
  Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
  attached VPCs whose effective RT lacks 0.0.0.0/0.

Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.

* fix(igw): fan out subnet egress gate decisions on IGW attach/detach

The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.

Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.

Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).

* fix(igw): skip gate fan-out on attach to avoid bootstrap race

AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).

Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.

Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.

* fix(vpc): prevent duplicate main route table from bootstrap race

Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.

CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).

Two-part fix:

1. createMainRouteTable scans rtbKV for an existing IsMain=true record
   on the same VPC before Put-ing a new one. Race window narrows from
   "always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
   between scan and Put.

2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
   tiebreak when multiple IsMain=true records exist: more routes >
   oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
   that slips through (cross-AZ restore, manual KV edit, residual race)
   still produces consistent egress gate decisions instead of flapping
   on KV scan order.

* fix(policy): atomic ACL replacement for SG mutations

Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).

Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
 2026-06-01 09:54:29 +10:00
Josh Siviour 287487a4f2 feat(elbv2): implement ModifyListener (#268) 
* feat(elbv2): implement ModifyListener

Adds AWS-compatible ModifyListener to the spinifex ELBv2 stack so
listeners can be edited in place — port, protocol, and default
actions — without the current delete+create dance that briefly drops
the listener from HAProxy.

- Handler interface, impl, and NATS wrapper get ModifyListener;
  validation mirrors CreateListener (LB type ↔ protocol, dup-port
  excluding self, target-group existence + protocol compatibility).
- Daemon subscribes to elbv2.ModifyListener on the spinifex-workers
  queue and a new gateway action dispatches it.
- Frontend exposes a per-row edit button that pre-fills a dialog from
  the existing listener and calls ModifyListenerCommand, replacing
  the "Listeners cannot be edited" note.
- Unit + NATS round-trip + frontend tests cover the new flow.

Cert / SslPolicy / Alpn handling stays out of scope and lands with
HTTPS listener support.

* test(e2e): cover ModifyListener in lb harness

Adds an Internal_ALB_ModifyListener subtest to TestLoadBalancer that
exercises in-place listener edits end-to-end:

- Provisions tgA + tgB sharing the same app instances.
- Brings up an internal ALB with HTTP:80 → tgA, baselines round-robin.
- ModifyListener port=8090, asserts DescribeListeners shows the new
  port and that probes on 8090 still round-robin — proves HAProxy is
  reconciled without dropping the listener (the regression mulga-944
  fixes; the frontend previously delete+created on every edit).
- ModifyListener DefaultActions → tgB, asserts traffic still flows.

Also factors a probeAtPort helper so the order JSON can target any
listener port, and updates TEST_COVERAGE.md with phase 3e.

* fix(ui): widen listener add/edit dialog so content stays inside the box

AlertDialogContent defaults to max-w-xs (20rem) which clipped the listener
form on the right edge. Use a 32rem cap with grid-cols-[minmax(0,1fr)] +
min-w-0 on the form so CSS grid tracks shrink instead of expanding to
max-content.

* test: cover ModifyListener + system instance dispatch

Lifts ModifyListener diff coverage by exercising nil/empty arn paths,
NLB protocol acceptance + rejection, protocol-only revalidation against
existing target groups, and the launcher-nats wire format error paths
(missing input, missing instance type, decode failures, no responder,
default timeout fallback).

Adds a new spinifex/daemon/daemon_system_dispatch_test.go covering
splitSubjectTail edge cases, respondWithSystem{Launch,Terminate}{Output,
OK,Error} envelope shape, handler entry points for invalid JSON and
missing instance ID, and subscribeSystemTerminate nil-conn + idempotent
re-subscribe behaviour.

Fixes the flaky TestHandleEC2DescribeInstanceTypes/NoFilter… subtest by
selecting a 2-vCPU instance type that also fits schedulable memory, so
hosts with smaller schedulableMem (CI 4 vCPU / ~13 GiB) don't pick a
16 GiB-mem type that fails allocation.

* test(e2e): wait for tgB after listener swap, not before

Spinifex health-checks only target groups wired into a listener's
HAProxy config. Waiting for tgB to report healthy before
modifyListenerDefaultTG attaches it to the listener is impossible —
the agent never probes its targets, so they stay in initial and the
2m EventuallyErr trips.

Drop the pre-swap tgB wait. After modifyListenerDefaultTG, replace
the fixed 3s sleep with WaitForTargetsHealthy on tgB (post-swap),
which is now the listener's DefaultAction.

* test(e2e): derive LB name from ARN in lbENI/deleteLB ENI filter

The ENI description filter was building the LB-name token from a
hard-coded "alb"/"nlb" template (e.g. `ELB app/lb-e2e-alb-int/<id>`),
which only matched LBs whose name happened to be `lb-e2e-alb-*` or
`lb-e2e-nlb-*`. The ModifyListener suite names its LB `lb-e2e-mod`,
so lbENI's EventuallyErr could never find the ENI and tripped after
30s. deleteLB had the same bug: sysInstanceID stayed empty and
WaitForENICleanup waited for a non-existent filter, allowing the
next suite's createLB to race the sys.micro VM reclaim.

Parse the LB name straight out of lb.ARN (penultimate path segment,
same shape createLB already relies on) so the filter matches any
caller-chosen name. Drop the now-redundant scheme arg from lbENI.

* test(e2e): poll for listener serving on new port after ModifyListener

The post-modifyListenerPort settle was a hard 3s sleep, but lb-agent
polls for new HAProxy config on a 5s tick. After the API returns, the
agent has typically not yet detected the config-hash change, so
HAProxy is still bound to the old port — every probe to the new port
RSTs and probeAtPort sees 0/0 successful.

Replace the sleep with waitForListenerServing: a polling loop that
fires 2 probes via the shared trigger client every 2s until at least
one returns a parsable instance_id, with a 60s ceiling. Factor the
trigger-and-fetch out of probeAtPort into probeOnce so the polling
helper can reuse it without the require.* asserts.
 2026-05-30 09:18:52 +10:00
Josh Siviour c1f44b36da feat: vendor-agnostic instance types + sys.* NATS fan-out (#269) 
* feat(instancetypes): emit cross-vendor sibling families on x86_64 hosts

Each x86_64 cpuGeneration's family list is now extended at generate time
with its cross-vendor counterpart family (Intel t3 ↔ AMD t3a,
c5 ↔ c5a, m6i ↔ m6a, etc.) via the new vendorSiblingFamily map in
definitions.go. Sibling families share an identical vCPU/memory shape
per size (same instanceFamilyDef size tables) and the same x86_64 arch
tag, so any host can schedule either vendor's request without changes
to canAllocate or the subscription path.

ResourceManager.updateInstanceSubscriptions iterates the type catalog
unchanged; it now picks up both vendors automatically and subscribes to
ec2.RunInstances.t3.* AND ec2.RunInstances.t3a.* on the same node.
RunInstances for either family load-balances across mixed Intel + AMD
clusters via the existing spinifex-workers queue group.

ARM (Graviton t4g/c?g/m?g/r?g) and legacy Intel-only families (t2, c4,
m4, r4) have no cross-vendor sibling and are unaffected.

Bead: mulga-siv-157 (vendor aliasing portion). sys.* NATS fan-out for
ALB-VMs is a separate change in the same plan.

* feat(daemon,elbv2): NATS fan-out for system instance launch/terminate

ELBv2 ALB-VMs (sys.micro) previously launched via a direct Go method call
from ELBv2ServiceImpl into the local Daemon, so every ALB-VM piled on
whichever node handled the upstream CreateLoadBalancer call. Mixed
clusters tripped 'insufficient capacity for sys.micro' once that node's
host-reserve trimmed schedulable vCPU below the next launch.

Route system-instance lifecycle through a dedicated NATS subject root:

  system.LaunchInstance.{type}            (queue group spinifex-workers)
  system.LaunchInstance.{type}.{nodeID}   (targeted)
  system.TerminateInstance.{instanceID}   (owning-node only)

ResourceManager.updateInstanceSubscriptions now treats sys.* types under
the new subject root with a separate handler hook (systemHandler) and
the same capacity-aware subscribe/unsubscribe behaviour customer types
already have — sys.micro subscriptions drop when the host saturates,
naturally re-distributing load.

The daemon side keeps LaunchSystemInstance / TerminateSystemInstance as
local methods; the new handleSystemLaunchInstance / handleSystemTerminate
handlers decode the JSON envelope, invoke the local method, and reply
with output or error. The terminate subscription is bound on launch
success so future terminate requests reach the owning node, and dropped
on terminate success.

ELBv2ServiceImpl now consumes a NATSSystemInstanceLauncher (handlers/elbv2/
system_instance_launcher_nats.go) instead of a direct daemon pointer, so
any node in the cluster can carry an ALB-VM.

SystemInstanceInput / Output / NICConfig / ExtraENIInput get explicit
JSON tags because the wire crosses NATS.

Tests cover JSON round-trip, remote-error propagation, terminate routing,
and no-responder failure mode.

Bead: mulga-siv-157 (sys.* fan-out portion completes the bead).
 2026-05-29 23:24:24 +10:00
Julian Sommer fbd5e3a607 fix: reconciler emits configured dns_server instead of hardcoded 8.8.8.8 (#265) 2026-05-29 12:23:03 +10:00
Julian Sommer 10728391af chore: revert test-cover Makefile 2026-05-29 11:38:42 +10:00
Julian Sommer e4b7834b2b chore: move input validation into gateway (#264) 2026-05-29 11:17:12 +10:00
Julian Sommer 94aa482931 feat: sts v1 (#261) 2026-05-29 10:42:37 +10:00
Josh Siviour 65ba60e8bc fix(gateway/ec2): fan out DescribeInstanceAttribute across daemons (#263) 
* fix(gateway/ec2): fan out DescribeInstanceAttribute across daemons

DescribeInstanceAttribute was queue-grouped on `spinifex-workers`, so each
request landed on one daemon at random. The handler only consults per-daemon
state (vmMgr / stoppedStore), so 1/N requests succeeded and the rest returned
InvalidInstanceID.NotFound. Terraform's AWS provider refreshes aws_instance
with three attribute reads per cycle, which on a 5-node cluster fails ~99% of
the time.

Move the daemon subscription into the fan-out block (no queue group) and
rewrite the gateway to mirror DescribeInstances: inbox + N replies + drop
NotFound. The aggregator returns the first parsed success, falls back to
NotFound only when every daemon confirms absence, and propagates deterministic
client errors (InvalidParameterValue, MissingParameter) when no daemon
succeeds.

Tests:
- New: MultipleNodes_OwnerWins, AllNodesNotFound, ClientErrorPropagates,
  NoResponders, SingleNode (rewritten without queue group).
- daemon_handlers_test.go: drop the `spinifex-workers` queue-group string from
  the ten DescribeInstanceAttribute subscriptions so the tests match the new
  production wiring.

* fix(ec2): DescribeInstanceTypes returns supported set, not capacity-gated

The no-filter DescribeInstanceTypes response was filtered to types with
at least one free slot. AWS-compatible callers (e.g. the Terraform AWS
provider's post-RunInstances metadata lookup) expect a stable supported
set: when a launch consumes the last free slot of its type, the next
DescribeInstanceTypes for that type must still return it, or the apply
fails with "reading EC2 Instance Type (X): empty result".

Split the API into two ResourceManager methods so capacity semantics
stay strict for the capacity=true filter (used for cluster aggregation)
while default callers get the supported answer:

- GetAvailableInstanceTypeInfos(showCapacity bool) — unchanged; gates on
  remaining schedulable CPU/memory and (for GPU types) free GPU pool
  slots. Used when capacity=true is requested.
- GetSupportedInstanceTypeInfos() — new; lists every loaded instance
  type once, regardless of current allocation or GPU pool state. Still
  skips system types ("sys.*") and entries with incomplete CPU/memory
  metadata.

The handler picks between the two based on the capacity filter. The
gateway aggregator (DescribeInstanceTypes.go) needs no change — it
already deduplicates by type name when capacity=false, so requesting a
specific name now resolves from whichever node supports it.

Tests:
- Daemon unit tests for the new method cover the three filter rules
  (no free slots, system/incomplete skip, GPU without manager).
- Handler tests assert the supported path is hit by default and the
  capacity-gated path is hit only when capacity=true.
- The NATS-driven integration test is split into two subtests: one
  proves the no-filter response is invariant under allocation, the
  other proves capacity=true still gates on remaining resources.

* fix: persist API device name in volume metadata on AttachVolume

aws_volume_attachment polls DescribeVolumes with
attachment.device=/dev/sd[f-p], the API-form name. Spinifex was
persisting the in-guest virtio path (/dev/vd*) via UpdateVolumeState
after AttachVolume, so the filter rejected every attached volume and
the post-attach wait loop failed with "couldn't find resource".

UpdateVolumeState now receives the API-form device name and the
daemon's AttachVolume response echoes DeviceName instead of
GuestDevice. BlockDeviceMappings continues to carry the guest path
(mulga-599 / PR #55) since UpdateGuestDeviceNames re-applies that
convention on every Launch/Start; only the volume-metadata path
diverges to the API form for the AWS-spec round-trip.

Adds fakeVolumeStateUpdater to manager_deps_test.go and a regression
test that asserts the API-form name flows through
UpdateVolumeState while BDM keeps the guest path.

Resolves mulga-siv-154.
 2026-05-28 23:43:39 +10:00
Josh Siviour f877f5edef feat(network/external): restore source=dhcp on external pools (#262) 
* refactor(network/external): introduce Allocator interface + StaticPoolAllocator

Carve external IP allocation out of handlers/ec2/vpc/external_ipam.go into
network/external/static_pool.go behind a typed Allocator interface. The KV
bucket name, schema, CAS retry count, and gateway-IP reservation semantics
are unchanged so existing static-pool data carries over with no manual
migration.

ExternalIPAM stays in handlers/ec2/vpc (L5 cleanup deferred per plan) but
becomes a thin facade that constructs a StaticPoolAllocator and delegates
Allocate / Release / GetPoolRecord through the new interface. Q4 will plug
DHCPPoolAllocator into the same Allocator slot.

Plan: docs/development/improvements/external-pool-dhcp-source.md (Q1).

* feat(network/external/dhcp): package skeleton — Client + Fake + KV store

New network/external/dhcp/ package, restored from the structure deleted
in Slice C (commit 7ff013a) but rehomed under the ADR-0006 layer tree:

- client.go      Client interface + AcquireRequest + Lease (with
                 RenewAt/RebindAt/ExpiresAt helpers).
- nclient4.go    NClient4Client — AF_PACKET DORA / Renew / Release
                 backed by insomniacslk/dhcp/dhcpv4/nclient4.
- fake.go        In-memory test fake with per-step hooks for fault
                 injection and acquire/renew/release call counters.
- kv.go          Store over the per-AZ spinifex-dhcp-leases-<az> KV
                 bucket: Put / Get / Delete / List. Schema matches the
                 plan (absolute timestamps for acquired/t1/t2/expiry).
- *_test.go      Lease-timer arithmetic, fake lifecycle, nclient4 input
                 validation, KV roundtrip + version-key isolation.

Not yet wired into vpcd, ExternalIPAM, or any allocator — Q3/Q4 plug
it in. Plan: docs/development/improvements/external-pool-dhcp-source.md (Q2).

* feat(network/external/dhcp): DHCPManager + daemon-side NATSClient

Adds the per-vpcd DHCPManager that owns active lease state: KV scan on
startup (re-issues RENEW for non-expired entries, drops the rest),
per-lease renewal goroutines with jittered T1/T2/expiry timers and
post-expiry re-DORA, and NATS request/reply subscribers on
vpc.dhcp.acquire / vpc.dhcp.release. Acquire is idempotent on client-id,
coalesced via singleflight so concurrent retries do not produce
duplicate wire DORAs.

Adds the daemon-side dhcp.NATSClient wrapper (RequestAcquire /
RequestRelease) that ExternalIPAM will route DHCP-sourced allocations
through in Q4. Still no allocator wiring — vpcd does not yet construct
the manager.

* feat(network/external/dhcp): wire DHCP allocators; re-allow source=dhcp

Adds DHCPGatewayLRPAllocator and DHCPPoolAllocator, the two consumers
of the Q3 DHCPManager. The gateway-LRP allocator runs in-process inside
vpcd and DORAs once per VPC (client-id `dhcp-gw-lrp-<vpcID>`, used in
centralized NAT). The pool allocator runs daemon-side and routes
per-EIP / ENI / instance acquisitions through vpcd over the restored
vpc.dhcp.acquire / vpc.dhcp.release topics.

ExternalIPAM now dispatches per-pool: static pools keep the existing
StaticPoolAllocator + KV bucket; DHCP-sourced pools attach a
DHCPPoolAllocator via EnableDHCP. vpcd constructs the dhcp.Manager
when any pool has Source="dhcp" and selects the DHCPGatewayLRPAllocator
for IGWManager; everything else keeps StaticRangeAllocator /
LinkLocalAllocator.

Config validator re-allows source="dhcp", requires bind_bridge, and
rejects range_* / gw_lrp_range_* keys on DHCP pools. Unknown sources
are rejected with the supported list. Allocator.Release uses a
server-side IP→client-id lookup so the existing (pool, ip) signature
keeps working for both static and DHCP pools.

Plan: docs/development/improvements/external-pool-dhcp-source.md (Q4)

* feat(spx admin init): restore --external-source=dhcp + --external-bind-bridge

Source defaults to dhcp when --external-pool is omitted; static remains the
default when a pool range is supplied. Adds PoolBindBridge propagation through
ConfigSettings + formation.NetworkConfig + spinifex.toml template so vpcd's
DHCPManager finds the bridge to DORA on across single-node and joiner paths.

* fix(network/external/dhcp): derive chaddr from client-id when HWAddr unset

DHCPGatewayLRPAllocator and DHCPPoolAllocator (first acquire) own no NIC
and pass an empty HWAddr to the manager. nclient4 rejected the request
with "dhcp acquire: hw_addr is required", breaking nightly cells 1/3/8/10
at AttachIGW. Derive the chaddr from the client-id via utils.HashMAC so
the upstream DHCP server sees a stable 02:xx:xx:xx:xx:xx MAC per AWS
identifier, matching the OVN-side MAC scheme operators already grep for.

* fix(network/external/dhcp): enable IFF_PROMISC on bind_bridge across DORA

Nightly cell 1 timed out at OFFER step with "no matching response packet
received". Upstream router replied unicast to the derived 02:xx chaddr
which the Linux bridge had never learned, so the kernel dropped the
frame before nclient4's AF_PACKET socket could see it.

Reference-counted IFF_PROMISC toggle around every Acquire/Renew/Release
so overlapping leases on the same bridge share one flag flip and the
last release restores prior state. Soft-fails to a warn log when
CAP_NET_ADMIN is unavailable — operator may already have promisc set.

* fix(network/external/dhcp): RFC 2131 retransmission — retry DISCOVER 3×

NClient4 was configured with WithRetry(1) + WithTimeout(60s), meaning a
single DISCOVER per Acquire. When the first frame is lost (STP convergence
on a freshly-up bridge, upstream packet loss) the call blocks the full
60s and fails. Nightly cell 1 hit this at startup: gw-lrp Acquire for two
VPCs timed out back-to-back before any e2e step ran.

Switch to WithRetry(3) with a 5s per-attempt seed; nclient4 doubles the
deadline on each retransmission (5s, 10s, 20s) and sends a fresh DISCOVER
each round. Total budget ~35s, three frames on the wire — recoverable
without the higher layer noticing.

* fix(network/external): three CI-blocking extdhcp bugs

DHCP-source pools failed e2e-nightly cells 1/3/8/10 (net: dhcp) after
the restore-dhcp-source branch landed. Three independent root causes:

* gateway nexthop dropped: GatewayIPAllocator.Allocate had no return
  channel for the upstream router learned from lease.Routers. igw fell
  back to link-local 169.254.0.2, unreachable on the WAN subnet, so
  instance egress and SSH-in died silently. Widen the interface to
  return nexthop; DHCP impl pulls Routers[0]; empty Routers is now a
  hard error. Static impl returns pool.Gateway as before.

* DORA budget shrunk 6x: WithTimeout(5s) x WithRetry(3) gave a 15 s
  wallclock for the full DORA, no outer backoff. Restore the deleted
  dhcp_manager.go schedule [4s, 8s, 16s, 32s] under a 90 s budget with
  +/- 1 s per-attempt jitter; drop nclient4Retries to 1 so the leaf
  stays single-shot and the budget arithmetic lives in one place.

* every vpcd subscribed to vpc.dhcp.acquire: plain nc.Subscribe meant
  every vpcd in real-multi cells DORA'd in parallel with the same
  derived chaddr, causing upstream-server NAK / lease leak. Switch to
  QueueSubscribe with per-AZ queue "vpcd-dhcp-workers-<az>".

Plan: docs/development/bugs/extdhcp-pool-fixes.md (mulga).
Bead: mulga-siv-150.
 2026-05-28 20:48:11 +10:00
Julian Sommer 621e49405e fix: UI bug fixes 2026-05-28 15:08:17 +10:00
tomnewton-mulga 88881ba04c fix: Updated Trinity and SMCI docs 2026-05-28 14:42:03 +10:00
Julian Sommer 7ff02a6020 fix: consistent UI capitlisation 2026-05-28 14:27:35 +10:00
tomnewton-mulga 624c3180ea docs: fix gpu-passthrough doc sections and image tags for docs site 2026-05-28 11:41:00 +10:00
tomnewton-mulga 78d703da78 Feat/mig capable (#260) 
* feat(gpu): add NVIDIA MIG multi-instance GPU support

Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).

Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
  (EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
  DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
  ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
  MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
  admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows

* test(gpu): unit tests for MIG parsers and pool manager

Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.

Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.

* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle

Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.

- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
  slice; slow path dequeues a free MIG GPU, releases the lock, calls
  nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
  commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
  DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
  different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
  AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
  gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
  callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
  GPUs, ListProfiles to collect MIG profile specs for type generation;
  removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
  call sites, and fakeGPUClaimer all updated

* fix(gpu): correct mdev parent detection and make --profile optional for mig enable

enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).

Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).

Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).

* chore: writeups for smci and arcee

* test: add coverage for MIG exec, manager slow-path, instancetypes and admin

- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
  IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter

Diff coverage: 75.6% (threshold: 60%).

* fixed gpu passthrough doc

* Revert "fixed gpu passthrough doc"

This reverts commit 1e20f5ab13.
 2026-05-28 11:38:40 +10:00
Josh Siviour fa09228a36 feat: ENI hot-plug Sprint 3b — live QMP pipeline + AttachENI wiring (stage 2) (#258) 
* feat: ENI hot-plug Sprint 3a scaffolding

Lays the Sprint 3a Tier 0 groundwork from
docs/development/feature/enihotplug-v1.md so the live QMP pipeline in 3b
can plug in without further plumbing changes.

- types.ENIRequests: per-VM PCIe slot allocator state (free-list +
  eniID→slot reverse map), mirrors EBSRequests pattern.
- instancetypes.MaxENIsForType / HotPlugENISlotsForType: instance-type →
  ENI cap lookup with explicit-type-then-family-then-default precedence.
- vm.DeviceController interface + QMPDeviceController (real) +
  StubDeviceController (in-memory): unit-testable QMP device_add /
  device_del / netdev_add / netdev_del surface. QueryPCI body lands in
  Sprint 3b alongside the live response parser.
- buildBaseVMConfig now accepts the instance type and emits two distinct
  PCIe root-port pools at boot: hotplug-ebs{1..11} (fixed, /dev/sd[f-p])
  and hotplug-eni{1..N} (per-type, N = MaxENIs - 1).
- volumes.go bus id renamed hotplug%d → hotplug-ebs%d; query-block QDev
  regex narrowed to hotplug-ebs(\d+) so future virtio-net devices on
  hotplug-eni ports cannot collide with the EBS ordering.
- restore + ResetNodeLocalState reset ENIRequests.Mu alongside the EBS
  mutex so restored VMs start with a fresh allocator.

Closes mulga-cs-enihotplug-3a.

* feat: ENI hot-plug Sprint 3b — live QMP pipeline + AttachENI wiring

Lands the Sprint 3b Tier 1 deliverables from
docs/development/feature/enihotplug-v1.md so AttachNetworkInterface /
DetachNetworkInterface AWS APIs drive the host-side QMP pipeline against
a running VM.

- vm/qmp_device.go: real QueryPCI parser. Walks the per-bus query-pci
  response and recurses into pci_bridge.devices so virtio-net devices
  attached under hotplug-eni{N} root ports surface in the flat result.
- vm/eni_hotplug.go (new): HotPlugENI / HotUnplugENI on Manager. 4-step
  pipeline (tap → ovs-add-port → netdev_add → device_add + query-pci
  poll) with full rollback on step-N failure. Tap + OVS calls are
  stubbed in Sprint 3b; Sprint 3c drops in real NetworkPlumber /
  ovs-vsctl invocations against the same call-site interface.
- vm/eni_hotplug_test.go (new): unit tests with StubDeviceController —
  success path, rollback on each step, AttachmentLimitExceeded,
  idempotent re-attach, non-running rejection, force-detach semantics,
  concurrent serialization, query-pci timeout.
- types/ec2.go: EC2InstanceCommand gains AttachENI / DetachENI attrs +
  AttachENIData / DetachENIData payloads, mirroring the AttachVolume
  command surface.
- handlers/ec2/vpc/eni.go: ENIRecord gains AttachmentStatus,
  HotPlugSlot, LastAttachError, DetachInFlight, DetachForce fields per
  plan §State machine. New helpers: GetENIRecord, UpdateENI (CAS
  patcher), FindENIByAttachment (used by DetachNetworkInterface lookup
  by attachmentId).
- gateway/ec2/vpc/{Attach,Detach}NetworkInterface.go (new): AWS SDK
  request decoders that route via the per-instance ec2.cmd.{id}
  channel; DetachNetworkInterface resolves attachmentId →
  instanceId via DescribeNetworkInterfaces first.
- gateway/ec2.go: registers the two new EC2 actions in the dispatcher.
- daemon/daemon_handlers.go: handleEC2Events dispatches the two new
  attribute flags to per-instance handlers.
- daemon/daemon_handlers_eni.go (new): composes vpcService.AttachENI
  (KV) with vmMgr.HotPlugENI (QMP) and publishes
  vpc.eni-hotplug.{attached,detached}.{instanceID} on success. KV
  state machine: available → attaching → attached → detaching →
  available. On QMP failure: KV rolls back to available with
  LastAttachError persisted for DescribeNetworkInterfaces visibility.
- vm/errors.go: ErrENINotAttached, ErrENIPipelineTimeout,
  ErrQMPUnavailable.

The Sprint 3c bead lands the real tap + ovs-vsctl wiring on the same
pipeline; 3d adds the reconciler that converges the attaching /
detaching KV states left behind by a daemon crash mid-pipeline.

Closes mulga-cs-enihotplug-3b.

* Update ec2.go

* Update ec2.go
 2026-05-27 22:54:46 +10:00
Josh Siviour a8e356b3d2 feat: ENI hot-plug Sprint 3a scaffolding (#257) 
Lays the Sprint 3a Tier 0 groundwork from
docs/development/feature/enihotplug-v1.md so the live QMP pipeline in 3b
can plug in without further plumbing changes.

- types.ENIRequests: per-VM PCIe slot allocator state (free-list +
  eniID→slot reverse map), mirrors EBSRequests pattern.
- instancetypes.MaxENIsForType / HotPlugENISlotsForType: instance-type →
  ENI cap lookup with explicit-type-then-family-then-default precedence.
- vm.DeviceController interface + QMPDeviceController (real) +
  StubDeviceController (in-memory): unit-testable QMP device_add /
  device_del / netdev_add / netdev_del surface. QueryPCI body lands in
  Sprint 3b alongside the live response parser.
- buildBaseVMConfig now accepts the instance type and emits two distinct
  PCIe root-port pools at boot: hotplug-ebs{1..11} (fixed, /dev/sd[f-p])
  and hotplug-eni{1..N} (per-type, N = MaxENIs - 1).
- volumes.go bus id renamed hotplug%d → hotplug-ebs%d; query-block QDev
  regex narrowed to hotplug-ebs(\d+) so future virtio-net devices on
  hotplug-eni ports cannot collide with the EBS ordering.
- restore + ResetNodeLocalState reset ENIRequests.Mu alongside the EBS
  mutex so restored VMs start with a fresh allocator.

Closes mulga-cs-enihotplug-3a.
 2026-05-27 22:15:20 +10:00
Josh Siviour fd3dc6b680 feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255) 
* refactor(vpcd): remove macvlan bridge mode

Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).

Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
  already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh

Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
  verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
  (previously relied on empty-mode defaulting to centralized)

Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
  teardown role (migration for operator systems with stale spx-ext-*
  interfaces from pre-veth installs)

* feat(vpcd): resolve external CIDR at startup with bounded retry

Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.

Skipped when ExternalMode is empty (overlay-only deployments).

Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.

* fix(vpcd): cover external CIDR startup paths

Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.

* refactor(network/ovn): extract L1 OVN client to network/ovn package

Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.

services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.

Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.

First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.

* feat(network/topology): TopologyManager interface + vpcd adapter

Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.

TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).

Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.

Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).

* feat(network/host): L0 HostWiring interface + physical/veth impls

Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.

* feat(network/policy): L3 SecurityGroup/NAT/Route managers

Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:

- SecurityGroupManager: clear+add ACL replacement on an existing port
  group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
  host.UplinkMode). Distributed sets logical_port/external_mac on EIP
  dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
  and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
  duplicate rows (closing the long-standing non-idempotency bug in
  vpcd/topology.go's IGW default-route path).

ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.

bead: mulga-siv-125.3.1

* feat(network/external): L5 IGW/EIP/NATGW managers

Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.

* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup

Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.

Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.

Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.

Replaces and removes the 5-pass startup sequence in services/vpcd:
  - Reconcile + ReconcileFromKV
  - ReconcileSGsOnce + ReconcileSGsLoop
  - RetrofitAllExternalLocalnetOptions
  - RetrofitAllGatewayPortNetworks

IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.

Closes mulga-siv-125.3.2.

* refactor(network/topology): promote live Manager impl to network/topology

Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).

TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.

Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).

Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.

Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).

* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers

Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.

Surface additions:

- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
  not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
  (resolves the apply.go TODO from §2.1 of the redesign plan).
  liveManager implements them with GetPortGroup-then-Create /
  GetPortGroup-then-Clear+Delete idempotency.

Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.

* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager

Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.

Deletes:

- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
  TestSubnetGateway block); parity coverage lives in
  network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
  diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
  topology.WithDNSServer to the live manager) is the live path.

Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.

* refactor(network/reconcile): drive SG port groups via topology.Manager

applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.

DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.

Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.

* feat(network/policy): add FlowsBarrier option to NATManager

NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.

NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.

Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.

* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager

handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.

Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).

Slice 4 of mulga-siv-129, sub-step b.

* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager

handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:

- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
  plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
  duplicate vpc.add-nat publishes never tear down the rule the previous
  invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
  FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
  fire-and-forget "already absent = success" semantic.

The vpcd "errors" import drops out with the inline error-comparison.

Slice 4 of mulga-siv-129, sub-step c.

* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)

handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
  the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
  the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
  delegate to external.IGWManager.AttachIGW. The manager mirrors every
  step the legacy handler ran — external switch + localnet port +
  gateway LRP + switch-side router peer + default route + chassis HA +
  waitForFlowsHV tail barrier — so observable behaviour is unchanged.

handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.

Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.

The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.

* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override

RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.

Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.

* refactor(config): drop upstream-DHCP config surface (static WAN pool only)

Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).

Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.

cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.

service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.

Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.

* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)

Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.

- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
  ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
  strings and tests

Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.

* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)

Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.

handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.

No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).

* feat(ipam): Purpose field on every IPAM record (Slice B)

Part of bead mulga-siv-125.3.3.

Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).

External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
  LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
  elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
  allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
  with v1/v2-union record shape so partial migrations resume cleanly.

Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
  keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
  resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
  ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
  (no owning ENI) warn and become Purpose=unknown.

Migrate framework:
- KVContext gains an optional JetStream field; new
  Registry.RunKVWithJetStream method threads it through. RunKV
  signature unchanged — all other callers are unaffected.

Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
  IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
  with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
  no-JetStream fallback, empty bucket.

* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket

The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.

Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.

* feat(config): add network.ipsec_enabled cluster flag (default true)

Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.

Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).

* feat(admin): IPsec peer cert generator signed by cluster CA

Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.

Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.

The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.

* feat(admin): wire IPsec peer cert into admin init + join paths

Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.

spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).

Bead mulga-siv-125.3.5.

* build: install openvswitch-ipsec for OVN native IPsec

openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.

Required by bead mulga-siv-125.3.5.

* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup

When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.

ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.

Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.

Bead mulga-siv-125.3.5.

* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture

TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
  and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
  writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
  (rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
  logged only; geneve may be idle during the 5s capture window).

Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).

Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.

* fix(igw): write attachment State="attached" for AWS parity

Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).

Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.

Match AWS InternetGatewayAttachment.State semantics:
 - Attach: State="attached"
 - Detach: State="detached"

Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.

Bead: mulga-siv-131

* fix(reconcile): call AttachIGW when IGW topology missing

applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.

Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.

Closes mulga-siv-132.

* fix(test): align NewExternalIPAM call with 2-arg signature

Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.

* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race

vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.

handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.

Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.

Closes mulga-siv-133.

* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS

spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.

- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
  spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
  with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
  prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
  honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
  external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
  through --extra-vars so `make` callers can override without editing
  defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
  styles (direct -e vs make EXTRA_VARS), NAT removal.

* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness

enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.

- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
  is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
  the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
  state on failure, fan out across all 3 multinode nodes

Refs: mulga-siv-136

* test(e2e): sticky-skip downstream SSH tests when probe fails

When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.

First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.

TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.

Refs: mulga-siv-137

* fix(ipsec): use Debian unit name openvswitch-ipsec.service

CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:

- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
  failed, daemon logged a warning and continued without flipping
  ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
  never received the cert pointers

Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.

Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).

Refs: mulga-siv-136

* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture

admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.

- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.

mulga-siv-134

* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR

Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.

Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.

Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.

* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write

Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.

Sudoers gains one read-only line:
  spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service

This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.

If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.

* chore: unblock preflight on feat/network-layer-redesign

Two pre-existing failures blocked preflight on the branch:

- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
  rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
  Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
  fails when the parent VPC router is missing. That contract changed in
  aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC
  idempotently before EnsureSubnet to avoid a NATS publish-order race.
  Delete the now-impossible failure case.

* refactor(network/ovn): relocate nbdb model package from services/vpcd

Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change.

The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions,
etc.) lived under services/vpcd/nbdb/ because that was the original home
of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all
L3/L5 managers (network/policy, network/external, network/topology,
network/reconcile) became the primary callers — 8 importers in network/*
vs 2 in services/vpcd/. The directory move puts the schema model where
its consumers are, and unblocks deleting services/vpcd/ entirely once
the subscriber/launcher relocations land (Slices B–D).

* refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed

Phase 4 / mulga-siv-125.6 Slice B.

Production path no longer touches any of:

- RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions —
  the 5-pass startup leftovers that network/reconcile (Phase 2.2) made
  redundant. Nothing called them outside tests.
- reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the
  ipv4ToUint32 / uint32ToIPv4 utilities — superseded by
  external.IGWManager (Phase 2.4) which carries its own pool logic.
- ensureLocalnetOptions / ensureGatewayPortNetworks /
  expectedGatewayPortNetwork — read-before-write retrofit shims for the
  same code paths.
- generateMAC — only callers were reconcileIGW + its tests.

useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is
replaced by a policy.NATMode value injected via WithNATMode(...). The
mode is resolved once in vpcd.go from host.UplinkMode and threaded into
both the topology-handler lazy managers and the reconciler stack, so
"is this veth?" is no longer a question topology.go can ask. The
TopologyHandler defaults to NATModeDistributed when WithNATMode is
omitted, matching the previous default.

Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC
of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*,
TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions,
TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils +
net imports in topology.go.

Stale "phase 4 migrates those onto the new managers" comment in vpcd.go
fixed in passing — the subscribers were migrated in Phase 2.6
(mulga-siv-129); only the file-relocation parts of Phase 4 remain.
Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in
network/ovn/{live.go,mock/mock.go} doc comments also cleaned up.

* refactor(network): move NATS VPC subscribers to network/subscribers

Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of
services/vpcd into a new spinifex/network/subscribers package. Subscriber
holds the topology/policy/external managers directly — the lazy *Once
adapter shim (topology_manager.go) is deleted along with vpcd's local
event-struct copies.

launchService now constructs topology.LiveManager eagerly with a DNS
callback derived from external pools (preserving the unscoped-pool lookup
old TopologyHandler.lmDNSServer did), builds every manager up front, then
hands them to subscribers.New + Subscribe. reconcile.New receives
topology.Manager directly instead of through the adapter.

Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the
launcher; they were left behind as topology.go's last responsibilities.

Handler-layer tests (topology_test.go, security_test.go, integration_test.go)
are dropped: substance is covered by network/topology, network/policy,
network/external manager tests. subscribers/subscribers_test.go adds
focused coverage: missing-dep rejection, topic registration smoke, and
bad-JSON envelope per topic.

services/vpcd is now down to the launcher + the ovn/mock compat aliases
(removed in Slice D when the launcher promotes to cmd/vpcd/).

* refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases

Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign
package layout. services/vpcd/ disappears entirely; the launcher remains
an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate
binary — spx vpcd start stays the cobra entrypoint).

- Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post
  Slice C; only callsite was launchService's NewLiveOVNClient, inlined to
  ovn.NewLiveClient).
- git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/.
- Retarget three importers (spinifex/service/service.go +
  service_test.go, cmd/spinifex/cmd/service.go) to the new path.
- Fix stale services/vpcd / TopologyHandler comments in network/reconcile/
  + network/external/types.go + spinifex/vpcd/vpcd.go.
- service-interfaces.yaml vpcd path updated.
- Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was
  speculative; never built) in favour of in-process Service impl at
  spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/.

Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location)
satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled
join token, plan archive).

* fix(ovn): install openvswitch-ipsec + strongswan explicitly

ovs-monitor-ipsec.service fails to start without an IKE daemon. The
openvswitch-ipsec package only Recommends strongswan, so any apt
config that drops Recommends (or a base image with the recommends
already skipped) leaves the service in a Restart loop with 'IKE
daemon is not installed'. Hoist both packages into setup-ovn.sh so
provisioning is the single source of truth — the bootstrap shim in
mulga is no longer needed.

* test(network): pin ADR-0006 invariants to executable suite

Add static-analysis + behavioural tests that bind every safety and
liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure
output quotes the ADR clause verbatim and points at the offending
file:line so future debugging sessions can't quietly bypass a layer
contract to make a test green.

Static-analysis suite (spinifex/network/invariants/):
- S1 layers_test.go: parses `go list -json`; rejects upward layer
  imports and layers pulling from cross-cutter packages
- S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl /
  ovs-ofctl shell-out outside network/host/
- S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode()
  calls outside the init-time resolver locations
- S4 names_test.go: AST-detects OVN object name construction (concat,
  fmt.Sprintf, strings.Join) outside topology/names.go
- S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...}
  and DeleteLogical{...} calls inside network/policy/
- S6 federation_isolation_test.go: rejects TransitSwitch /
  TransitRouterPort references and ovn-ic literals outside
  network/federation/ (pre-protects the layer before it lands)
- S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan /
  openswan imports anywhere in network/
- S9 reconciler_singleton_test.go: rejects Reconcile…Loop and
  Reconcile…Pass<n> function names anywhere in network/
- S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing
  the {azID} segment; calls t.Skip with the gap report so every test
  run prints the work-to-do without breaking CI. Flip to t.Fatalf
  when the cluster-wide subject rename ships.

Behavioural suite (spinifex/network/reconcile/liveness_test.go):
- L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile
  property test; asserts second pass is a no-op
- L2 drift bound: structural check that DriftInterval is positive and
  ≤30 min support window; full fake-clock behavioural test deferred
- L3 federation re-enables: t.Skip stub pointing at the missing
  network/federation/ package

Fixes the one production S4 violation the new walker found:
network/external/igw.go was constructing "gw-port-"+vpcID inline.
Adds topology.GatewaySwitchPort(vpcID) helper and retargets both
AttachIGW and DetachIGW callsites through L2.

Plan + per-clause mechanic + phasing in
docs/development/improvements/network-invariants.md.

Refs: mulga-siv-140

* fix(ovn): install strongswan-charon (not strongswan metapackage)

The strongswan metapackage's '\''charon-systemd | strongswan-charon'\''
alternative resolves to charon-systemd by default, which ships
/usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or
/usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan
shells out to. Install strongswan-charon explicitly to pull
strongswan-starter (legacy ipsec controller) + the matching charon binary,
matching what openvswitch-ipsec.service expects.

Also reset-failed + restart openvswitch-ipsec.service after enable to
recover from the deb post-install starting the service before its
dependencies finished unpacking.

* chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/

Fall-through reviewer for any change is the maintainer.
network/ and vpcd/ are explicitly listed so the matching branch
protection rule ("Require review from Code Owners") routes those
PRs to joshsiv-mulga. Those trees are governed by ADR-0006
invariant tests (spinifex/network/invariants/); CODEOWNERS adds a
human review gate on top of the mechanical fence.

Refs: mulga-siv-141

* chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team

Fallthrough reviewer should be the team, not a single maintainer, so
any engineer can approve PRs outside the explicitly-gated paths.
network/ and vpcd/ keep joshsiv-mulga as the required reviewer per
ADR-0006 invariant ownership.

Refs: mulga-siv-141

* feat(ci): add test_run workflow input for -test.run filter

Pass an optional regex through to each suite's '\''go test -test.run'\'' so
dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without
running the full multinode suite. Threaded through single, multinode, and
reboot suite invocations; empty input preserves current behaviour.

* fix(daemon): flip NB_Global.ipsec=true on management node

Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each
chassis is necessary but not sufficient for OVN native IPsec — without
NB_Global.ipsec=true, ovn-controller never adds options:remote_name to
the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled
tunnels and the kernel xfrm tables stay empty. After enabling
ipsec_encapsulation locally, also write NB_Global.ipsec=true when the
local ovnnb_db socket is present (management node only); workers skip
silently — one writer is enough for a cluster-wide flag.

Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl
invocation so the grant can't be reused for other NB mutations.

* fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec

The strongswan-charon profile only allows reads from /etc/ipsec.*,
/etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes
the per-tunnel strongSwan config with absolute paths to our peer cert +
key under /etc/spinifex/ipsec/, so charon hits 'Permission denied'
loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm
SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the
profile's local override hook + reload.

* fix(ovn): pin OVS system-id to hostname for IPsec cert identity match

ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in
the strongSwan config it generates. Our per-node IPsec peer cert
(admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN.
With chassis-id left at the package-generated UUID, charon validates
`@<UUID>` against a cert with dnsName=<hostname> — mismatch returns
AUTHENTICATION_FAILED and no xfrm SAs are negotiated.

Pin system-id to hostname (which equals the node name on Mulga clusters)
by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf,
read by ovs-ctl on boot) and the running OVS DB value. Writing only one
would drift on the next reboot — the original "do not override" comment
was right about that hazard, just wrong about the resolution.

Cross-link the invariant in admin/ipsec.go: changing the cert identity
without updating setup-ovn.sh (or vice versa) breaks IKE silently.

Refs: mulga-siv-136

* fix(ovn): take --node-name flag so chassis-id matches cluster identity

CI hostnames are bead-themed (env20-casuarina) and don't match the
cluster node name (node1/node2) baked into the IPsec peer cert dnsName
SAN at `spx admin init/join --node NAME`. Pinning system-id to
`hostname -s` produced env20-casuarina; charon then validated
`@node1` against dnsName=env20-casuarina and IKE failed with
AUTHENTICATION_FAILED.

Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for
manual/dev invocations, and update the ansible install role to pass
{{ spinifex_node_name }}. The matching bootstrap-install.sh change
lives in mulga.

Refs: mulga-siv-136

* refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2

Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go
per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs
in spinifex/daemon/ (daemon is an L0 consumer; S2 applies).

Moved to network/host/:
- OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber)
- GetBridgeIPv4 → host.GetBridgeIPv4
- OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus
- daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec

Moved to utils:
- sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam
  (cross-package reassign would trip the reassign linter; helper keeps
  the indirection inside utils).

Moved to vm:
- generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC)

Deleted (dead):
- SetupComputeNode (replaced by network/host/bridges.go:ensureBridges +
  scripts/setup-ovn.sh:572-579)
- EnsureDataRoute
- findInterfaceByIP
- daemon/network.go + dead tests in daemon/network_test.go (duplicates of
  vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go)

S2 invariant (network/invariants/host_containment_test.go) now walks
both spinifex/network/ and spinifex/daemon/, skipping network/host/.

* fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert)

The earlier override granted reads on /etc/spinifex/ipsec/** only, which
covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem.
charon then can't load the CA, can't validate the peer cert chain, and
surfaces the failure as `no trusted RSA public key found for '<peer>'`
even though identity matching now succeeds — IKE_AUTH fails with
AUTHENTICATION_FAILED and no xfrm SAs negotiate.

Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key})
and always rewrite the override so re-runs replace the older narrower
rule.

Refs: mulga-siv-136

* fix(ipsec): symlink cluster CA into charon's default cacerts dir

ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references
in the strongSwan config, but strongSwan 6.0 only loads CAs declared in
standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at
startup. Without this, charon authenticates the local side fine but
rejects every inbound peer cert as `no trusted RSA public key found for
'<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate.

Do the symlink in admin.GenerateIPSecPeerCert (right after writing the
peer cert) — setup-ovn.sh runs before `spx admin init/join` creates
/etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and-
rename pattern so concurrent test invocations don't race on the symlink.

/etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no
profile change needed.

Refs: mulga-siv-136

* chore(setup-ovn): drop apt-get install_packages block

OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold
image via mulga's image-builder/scripts/provision.sh (commits 116bb05,
a9b3a9c). Keeping a runtime apt-get install path here pulled CI through
apt-cacher-ng on every bootstrap and tripped on intermittent "Message
has been manipulated" InRelease cache corruption (env21/env20 runs).

Drop the install block. Gold-image rebuild is the canonical place to
add new system packages. If a package is missing, downstream ovs/ovn
commands fail loudly during setup-ovn.sh's later steps — clearer signal
than silently re-installing through a flaky proxy.

setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's
bootstrap-install.sh:237) so this removes the last runtime apt-get call
on the CI provisioning path.

* fix(ovn): gate system-id pin on explicit --node-name

setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't
passed, and then unconditionally wrote that value to
/etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id.

On CI single-node, the gold image already registered a chassis with
name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed
hostname (env10-banksia). vpcd's discoverChassis treats any chassis row
whose name != local system-id but hostname == local hostname as a stale
local chassis and skips it, so vpcd reported "no OVN chassis registered
in SBDB" and refused to start, cascading into the e2e + e2e-go failures
on run 26430839508.

Drop the hostname fallback. If --node-name is given, pin both the
on-disk file and the running OVS DB to it (callers that care about
IPsec cert identity matching still get it). If no --node-name is given,
leave the existing system-id alone — that's always safe whether the
value came from a gold-image UUID, bootstrap-install.sh writing nodeN
before this script runs, or a manual ansible value.

* fix(network/reconcile): skip orphan prune on startup; drift owns GC

The vpcd startup reconcile races against the daemon's EnsureDefaultVPC:
the daemon writes SG KV rows + publishes vpc.create-sg on each node in
parallel with vpcd starting up. The leader-gated reconcile can load
intent before any SG row is visible on its local JetStream replica
while subscribers on peer nodes are concurrently creating port groups
from the same vpc.create-sg events. A startup orphan-port-group sweep
then deletes managed sg_* port groups that intent hasn't caught up to.

CI run 26434124790 trace:
  node2 reconcile/lock: elected holder=env20-bottlebrush
  node2 reconcile: starting intent_sgs=0
  node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a
  node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391
  node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb

Every subsequent RunInstances → vpc.create-port failed with
"port group not found" for ~3 minutes until the 5-minute drift tick
recreated the port groups. Tests give up after 60s; six multinode
suites cascade-fail on InsufficientInstanceCapacity.

Split the reconciler into Reconcile (full prune, drift loop) and
ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans
bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged.
Legitimate orphans get garbage-collected on the next drift tick — at
worst 5 minutes of lingering rows, no production impact.

Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract.

* fix(ipsec): trigger charon rereadcacerts after CA symlink install

strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit
openvswitch-ipsec.service is enabled at provision time and starts before
admin init writes the cluster CA symlink, so charon ends up running with
zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public
key found for '<peer>'` and no SAs negotiate — the failure mode exposed
by TestMultinodeIPSec.

Append a stroke rereadcacerts call at the end of
installCAIntoCharonTrustStore so an already-running charon picks up the
freshly-placed CA without a service restart. Failure is swallowed at
debug — a missing/stopped charon at admin-init time is expected and a
later startup will scan cacerts naturally.

* refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8)

ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2,
XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated
entirely to OVN native IPSec and is invisible above L0."

The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper
lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths,
EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in
network/host. The orchestrator now lives alongside its callees as
host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at
the call site. Helper and test-override package vars (systemctlActiveTimeout,
ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec
directly rather than via the Daemon struct.

Closes mulga-siv-142

* refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4)

ADR-0006 S4 forbids any layer above L2 from constructing OVN object
names. Three daemon sites formatted "port-" + ENIId inline; route them
through topology.Port instead.

Closes mulga-siv-143

* refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4)

Replace inline "port-"+eniID formatting across five sites (eip, instance x2,
vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin
wrapper to preserve the OVN binding contract comment and existing call sites.

Closes mulga-siv-144

* test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006)

Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces
in CI. Backlog of violations in widened roots was drained by
mulga-siv-142/-143/-144; this commit makes the wider scan permanent.

  - S2 (host_containment): add vm/ alongside network/ + daemon/
  - S3 (nat_mode):         add daemon/ alongside network/ + vpcd/
  - S4 (names):            add daemon/, handlers/, vm/ alongside
                           network/ + vpcd/; add narrow AWS-ID exemption
                           (vpc-cidr-assoc-) so AWS resource IDs are not
                           misclassified as OVN object names
  - S6 (federation):       add daemon/, handlers/ alongside network/;
                           convert single-root walk to slice
  - S7 (nats_subjects):    add daemon/, handlers/, testutil/, utils/ —
                           stays t.Skip pending cluster-wide subject
                           rename; gap report now lists 58 subjects

CODEOWNERS: add /spinifex/network/invariants/ rule so widening or
narrowing the scan needs explicit owner review.

Closes mulga-siv-145
Refs: audit-adr-0006.md

* chore(e2e): disable IPsec in pseudo multi-node test

Pseudo multi-node runs three simulated nodes as unprivileged tf-user.
The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts,
which fails with EACCES under a non-root caller and aborts admin init.

Disable --ipsec for this suite — loopback-only nodes don't exercise
real Geneve encryption anyway. Real-multi-node + single-node e2e
(both run admin init via sudo) continue to cover the IPsec path.

* fix(ovn): atomic Ensure* primitives prevent duplicate logical entities

OVN NB has no unique constraint on Name. The previous
GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU
race: when vpc.create on one node and the defensive vpc.create-subnet
EnsureVPC call on another both observed absence, both inserted, yielding
two routers per VPC. The orphan held DNAT references blocking EIP
detach, and DNAT/flow installation landing on the orphan caused
TestMultinodeGuestSSH SSH timeouts.

Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the
libovsdb client. Each builds a single OVSDB transaction combining a
wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the
insert. The server serialises the wait+insert atomically; concurrent
callers see exactly one survivor and the rest fall through to a re-fetch
that returns the canonical row.

Switch all production callers (topology.EnsureVPC / EnsureSubnet /
EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs /
applySubnets) to the Ensure* variants. Bare Create* remains available to
the L1 wrapper.

Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block
future reintroduction of the bare Create* call sites outside
network/ovn.

Mock implements Ensure* under its existing mutex so the concurrency
tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single
UUID survives.

Refs: mulga-siv-146

* fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate

The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus
`Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation
struct tags Rows as `json:"rows,omitempty"`, so the empty slice was
dropped on the wire and ovsdb-server rejected every transaction with
"Required 'rows' member is missing", breaking reconciler bootstrap and
every downstream create-port lookup ("logical switch X not found").

Flip the predicate to the equivalent `Until: !=` formulation with a
single-row projection containing the Name we want absent:

  no row    → projection []           != [{name}] → predicate holds
  row exists → projection [{name}]    == [{name}] → predicate fails

Same atomic guarantee, JSON now well-formed. Mock tests can't catch
this — only a real ovsdb-server validates the envelope.

* chore: remove generateMAC wrapper

* chore: remove all bead / plan references

* fix(network/policy): wait for chassis flows after AddNATGateway

AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every
chassis installs the new dnat_and_snat flows; AddNATGateway returned right
after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would
publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API
call would succeed before the gateway chassis had programmed the SNAT flow
for the private subnet.

Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow
CI hosts (last seen run 26455602615 single-node, identical flake on
unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*).

Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test
TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path.

* chore: remove more plan refs

* chore: remove GenerateIPSecPeerCertIfEnabled

* chore: remove dupliate subnetGatewayCIDR function

* chore(network): trim superfluous comments across network/ tree

Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement
comments, remove section dividers grouping <10 items, and cut prose
narration in tests. ADR clause references, one-line WHY comments for
non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are
preserved. No behavioural changes.

Packages: invariants, topology, host, external, ovn, reconcile, policy,
subscribers.

* fix(igw): restore attachment State="available" for AWS parity

AWS InternetGatewayAttachment.State is "available" when an IGW is
attached to a VPC, per the AWS SDK doc ("the state is available when
attached to a VPC; otherwise, this value is not returned"). Commit
b7505f8 wrote "attached"/"detached" to align with the reconciler
filter, which inverted AWS parity.

Fix the producer to write "available" again and update the consumer:
 - handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway
   both persist State="available"; IGWRecord comment narrowed.
 - network/reconcile/intent.loadIGWs: filter accepts "available" (was
   "attached"); attachment is identified by VpcId != "".
 - tests updated: igw service_impl_test, routetable service_impl_test
   fixture, reconcile intent_test fixture.

* Update spinifex.toml

* refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1)

Subscriber previously called policy.NATManager directly for EIP and NAT
Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager)
that existed on paper. ADR-0006 §S1 requires every cross-layer call to
pass through the immediate lower neighbor's typed interface; IGW already
followed this pattern.

Wire EIPManager and NATGWManager into the Subscriber struct, construct
them in vpcd alongside IGWManager, and swap the four direct-L3 call
sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the
L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway).
AttachEIP now fires the flows-barrier, giving callers the same
"reachable on return" guarantee IGW already had.

The reconciler still calls policy.NATManager directly; that's a
cross-cutter and a separate concern.

* test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh

App VMs' user-data writes (unit file + index.html) sit in the on-host
WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before
that loses them — systemd then refuses to load the zero-byte unit and
the responder never starts. Mirror the bash driver's dd+sync pad so the
WAL flushes pre-reboot. Drop when the upstream WAL fix lands.

* test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure

Phase 8d's existing failure path only prints lr-nat-list and
lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits
nonzero, so debugging the NATGW regression requires inline diagnostics
in the journal. Expand the dump to include nb/sb show, chassis bindings,
port_group ACLs, br-int flows filtered to the NATGW IP/private subnet,
host routes, and the private VM's own ip/route/neigh state.

* fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet

Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure):

1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and
   only mutated the named router afterwards. With AWS subnet CIDRs repeating
   across VPCs, (type, logicalIP/externalIP) is not globally unique — the
   delete picked a NAT row owned by a different router, the mutate was a
   no-op, and the row delete failed with "remaining reference(s)", leaving
   the original router's NAT orphaned. Both calls now resolve the LR first
   and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping.

2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as
   SubnetCIDR, producing an snat for the wrong CIDR that collided with the
   correct event-published snat. loadNATGWs now walks route tables and
   emits one NATGWSpec per (NATGW, associated private subnet) pair,
   mirroring publishNatGatewayEventsForAssociation. Intent map keyed by
   natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs.

Tests:
- nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and
  DeleteNATGateway
- intent_test.go: NATGW snat uses associated private subnet CIDR, not the
  NATGW home subnet; NATGW with no associations produces no spec

---------

Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
 2026-05-27 21:05:01 +10:00
Julian Sommer 3f3ba32317 chore: cleanup COMMANDS.md 2026-05-27 15:54:17 +10:00
Julian Sommer f525006d87 feat: IAM Roles part 2 (#256) 2026-05-27 15:44:18 +10:00
Josh Siviour ae6fcfe7a1 Ansible dev extensions (#254) 
* chore(ansible): add dev-status playbook (read-only health doctor)

New `dev-status` role + playbook + Makefile target. Read-only snapshot of
spinifex.target, failed units, port listeners, OVN/OVS state, gateway
round-trip, AMI/keypair/volume/instance counts, and build-vs-install drift
for spx + nbdkit plugin + microvm artifacts. Pairs with dev-preflight as
the post-install / running-system diagnostic.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.1.

* chore(ansible): add dev-logs playbook (bug-report bundle)

New `logs` role + playbook + Makefile target. Captures journalctl for
spinifex-* / ovn-* / openvswitch-switch / qemu / nbdkit, copies
/var/log/spinifex/, snapshots systemctl status + networking (ip/ss/ovs/
ovn show) + spx/plugin/microvm version manifest + dmesg, then bundles
into /tmp/spinifex-logs-<host>-<ts>.tar.gz owned by the invoking user.

Tail mode deliberately omitted — devs should `journalctl -u 'spinifex-*' -f`
directly; ansible captures stdout and is the wrong tool for follow.

Overrides: -e logs_since=10min, -e logs_output_dir=/tmp,
-e logs_include_dmesg=false.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.2.

* chore(ansible): add dev-snapshot / dev-restore (state checkpoint)

New snapshot + restore roles + playbooks + Makefile targets.

dev-snapshot quiesces the full spinifex+OVN+OVS stack in dependency
order (spinifex.target → ovn-controller → ovn-central →
openvswitch-switch), kills lingering QEMU VMs, then tars /etc/spinifex,
/var/lib/spinifex (viperblock volumes included), /var/lib/ovn (NB+SB
DBs), and /etc/openvswitch/conf.db into /var/lib/spinifex-snapshots/
<name>.tar.gz with a sibling .manifest, restarts services in reverse,
and waits for the AWS gateway to come back.

dev-restore inverses: stop services, wipe the same path set, untar the
named bundle, copy /etc/spinifex/ca.pem back into the system trust
store and rebuild the CA bundle (the snapshot doesn't include
/usr/local/share/ca-certificates/spinifex-ca.crt), daemon-reload,
restart in reverse order.

Both refuse to run without -e snapshot_name=<name>, validate the name
matches [A-Za-z0-9._-]+, and produce honest summaries when the gateway
fails to come back (does not mask other ongoing issues).

Tested on dev box: snapshot/restore of 2.4 GiB state (large viperblock
volumes) completes cleanly with no data drift; services all return to
active.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.3.

* fix(ansible): wrap aws/spx calls in sg spinifex to fix group inheritance

Adds scripts/ansible/scripts/aws-as-spinifex.sh and spx-as-spinifex.sh
helpers that re-enter the command under `sg spinifex -c '...'` with
AWS_PROFILE=spinifex preset. Swaps every AWS / spx admin task across
the status, init, deploy, smoketest, snapshot, and restore roles to
use these via spinifex_aws_cli / spinifex_spx_cli vars.

Root cause (mulga-siv-92): ansible's `become_user` is a no-op when the
target user equals the calling user, so sudo never runs and
supplementary groups are inherited from the operator's invoking shell
rather than re-initialised via initgroups(3). On shells that predate
setup.sh's spinifex group add, the spinifex group is missing and
/etc/spinifex/ca.pem (drwxr-x--- root:spinifex, the AWS CLI's
ca_bundle) is unreadable — aws cli fails with SSL Errno 13. The
helpers re-enter the command in a shell that explicitly assumes the
spinifex group via sg(1), bypassing the issue regardless of how
ansible-playbook is invoked.

Side benefit: drops 50 lines of repeated `become: true / become_user /
environment: AWS_PROFILE` boilerplate across the roles.

Closes mulga-siv-92.

* chore(ansible): add dev-version playbook (build vs install drift)

New `version` role + playbook + Makefile target. Compares sha256 of
built artifacts (bin/spx, viperblock/lib/nbdkit-viperblock-plugin.so,
build/microvm/{vmlinuz,initramfs.cpio.gz}) against their installed
counterparts (/usr/local/bin/spx, plugindir, /usr/share/spinifex/
microvm/*). Prints per-artifact drift verdict plus sub-repo HEAD
short-revs (spinifex/viperblock/predastore) for context. Closes with
either "DRIFT detected — run: make ansible-dev-deploy" or "No drift".

Pure read-only. Faster than dev-status (no AWS/OVN/port probing).

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.9.

* chore(ansible): add missing roles/deploy/meta/main.yml

All other roles have meta/main.yml; deploy was the lone outlier.
Consistency fix. Closes mulga-siv-86.10.

* chore(ansible): add dev-vpc playbook (idempotent dev VPC fixture)

New `vpc` role + playbook + Makefile target. Idempotent
present/absent dispatch creates a deterministic spinifex-dev-vpc with
public subnet (10.99.1.0/24, map-public-ip-on-launch), internet
gateway, route table, and default route. Hooks into smoketest's launch
path — smoketest's existing
Name=map-public-ip-on-launch,Values=true filter now picks it up
instead of falling through to the "no subnet → skip launch" branch.

Idempotency: each step describe-then-create, looking up by Name tag
where possible. Re-running on a converged VPC is changed=0, skipped=5
(the create tasks).

Workaround for mulga-siv-94 (gateway silently drops
--tag-specifications and no-ops create-tags for route tables): role
falls back to "RT associated with our subnet" lookup so tag absence
doesn't break idempotency. create-tags task is kept as a no-op-now
forward-looking step for when the gateway gap is closed.

Teardown (-e dev_vpc_state=absent) enumerates ALL non-main route
tables in the VPC (not just the tagged one), so orphans from prior
runs that landed before the fallback was added get cleaned up too.

Tested end-to-end against josh-banksia dev gateway: create / re-run
(idempotent) / teardown all converge cleanly.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.4.
 2026-05-26 16:14:11 +10:00
Julian Sommer 51c48d8c0e feat: IAM Roles part 1 (#253) 2026-05-26 15:45:46 +10:00
Julian Sommer a7f7610176 release: update go.mod for v1.5.0 v1.5.0 2026-05-26 10:04:09 +10:00
dependabot[bot] e34b536b97 chore: bump the go-dependencies group across 1 directory with 3 updates (#252) 2026-05-26 09:10:38 +10:00
dependabot[bot] ef097bf4ab chore: bump actions/setup-go from 5 to 6 in the github-actions group (#251) 2026-05-26 09:09:24 +10:00
Julian Sommer 3b5e3e21c0 chore: tidy 2026-05-26 09:04:46 +10:00
Julian Sommer 71350033f7 chore: bump golang.org/x/net 2026-05-26 09:02:55 +10:00
joshsiv-mulga 80ab3fb37a ci: prune rootless docker + trap container leaks in microvm build 
Self-hosted runner accumulated 18GB in overlay2 from leaked container
layers. Two causes: build-microvm-image.sh only removed the alpine
container on the success path, and no workflow step pruned docker
between runs.

- scripts/build-microvm-image.sh: container id now removed via the
  existing EXIT trap, so failures, signals, and runner kills no longer
  leak the writable layer.
- .github/workflows/e2e-go.yml: prune containers / dangling images /
  buildx cache before each Build Install Artifacts step, in all three
  e2e jobs (single, multinode, reboot).
 2026-05-22 16:52:38 +10:00
Julian Sommer 756a5bb31f feat: media-sanitization doc 2026-05-22 15:38:10 +10:00
Julian Sommer a7f3debd6d fix: log and retry on Describe failure in ensureDefaultVPCInfrastructureFor 2026-05-22 15:22:30 +10:00
Julian Sommer 0362bc9baa fix: only subscribe to elbv2 operations if gateway url configured 2026-05-22 15:16:16 +10:00
Julian Sommer 0f7ff5e574 fix: bootstrap IGW for new tenants on iam.account.created 2026-05-22 14:05:57 +10:00