mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-22 12:16:52 +00:00

Files

T

Josh Siviour fd3dc6b680 feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255 )

* refactor(vpcd): remove macvlan bridge mode

Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).

Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
  already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh

Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
  verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
  (previously relied on empty-mode defaulting to centralized)

Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
  teardown role (migration for operator systems with stale spx-ext-*
  interfaces from pre-veth installs)

* feat(vpcd): resolve external CIDR at startup with bounded retry

Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.

Skipped when ExternalMode is empty (overlay-only deployments).

Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.

* fix(vpcd): cover external CIDR startup paths

Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.

* refactor(network/ovn): extract L1 OVN client to network/ovn package

Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.

services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.

Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.

First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.

* feat(network/topology): TopologyManager interface + vpcd adapter

Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.

TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).

Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.

Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).

* feat(network/host): L0 HostWiring interface + physical/veth impls

Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.

* feat(network/policy): L3 SecurityGroup/NAT/Route managers

Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:

- SecurityGroupManager: clear+add ACL replacement on an existing port
  group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
  host.UplinkMode). Distributed sets logical_port/external_mac on EIP
  dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
  and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
  duplicate rows (closing the long-standing non-idempotency bug in
  vpcd/topology.go's IGW default-route path).

ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.

bead: mulga-siv-125.3.1

* feat(network/external): L5 IGW/EIP/NATGW managers

Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.

* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup

Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.

Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.

Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.

Replaces and removes the 5-pass startup sequence in services/vpcd:
  - Reconcile + ReconcileFromKV
  - ReconcileSGsOnce + ReconcileSGsLoop
  - RetrofitAllExternalLocalnetOptions
  - RetrofitAllGatewayPortNetworks

IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.

Closes mulga-siv-125.3.2.

* refactor(network/topology): promote live Manager impl to network/topology

Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).

TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.

Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).

Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.

Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).

* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers

Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.

Surface additions:

- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
  not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
  (resolves the apply.go TODO from §2.1 of the redesign plan).
  liveManager implements them with GetPortGroup-then-Create /
  GetPortGroup-then-Clear+Delete idempotency.

Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.

* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager

Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.

Deletes:

- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
  TestSubnetGateway block); parity coverage lives in
  network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
  diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
  topology.WithDNSServer to the live manager) is the live path.

Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.

* refactor(network/reconcile): drive SG port groups via topology.Manager

applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.

DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.

Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.

* feat(network/policy): add FlowsBarrier option to NATManager

NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.

NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.

Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.

* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager

handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.

Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).

Slice 4 of mulga-siv-129, sub-step b.

* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager

handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:

- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
  plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
  duplicate vpc.add-nat publishes never tear down the rule the previous
  invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
  FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
  fire-and-forget "already absent = success" semantic.

The vpcd "errors" import drops out with the inline error-comparison.

Slice 4 of mulga-siv-129, sub-step c.

* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)

handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
  the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
  the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
  delegate to external.IGWManager.AttachIGW. The manager mirrors every
  step the legacy handler ran — external switch + localnet port +
  gateway LRP + switch-side router peer + default route + chassis HA +
  waitForFlowsHV tail barrier — so observable behaviour is unchanged.

handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.

Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.

The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.

* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override

RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.

Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.

* refactor(config): drop upstream-DHCP config surface (static WAN pool only)

Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).

Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.

cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.

service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.

Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.

* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)

Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.

- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
  ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
  strings and tests

Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.

* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)

Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.

handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.

No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).

* feat(ipam): Purpose field on every IPAM record (Slice B)

Part of bead mulga-siv-125.3.3.

Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).

External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
  LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
  elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
  allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
  with v1/v2-union record shape so partial migrations resume cleanly.

Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
  keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
  resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
  ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
  (no owning ENI) warn and become Purpose=unknown.

Migrate framework:
- KVContext gains an optional JetStream field; new
  Registry.RunKVWithJetStream method threads it through. RunKV
  signature unchanged — all other callers are unaffected.

Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
  IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
  with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
  no-JetStream fallback, empty bucket.

* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket

The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.

Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.

* feat(config): add network.ipsec_enabled cluster flag (default true)

Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.

Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).

* feat(admin): IPsec peer cert generator signed by cluster CA

Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.

Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.

The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.

* feat(admin): wire IPsec peer cert into admin init + join paths

Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.

spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).

Bead mulga-siv-125.3.5.

* build: install openvswitch-ipsec for OVN native IPsec

openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.

Required by bead mulga-siv-125.3.5.

* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup

When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.

ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.

Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.

Bead mulga-siv-125.3.5.

* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture

TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
  and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
  writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
  (rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
  logged only; geneve may be idle during the 5s capture window).

Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).

Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.

* fix(igw): write attachment State="attached" for AWS parity

Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).

Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.

Match AWS InternetGatewayAttachment.State semantics:
 - Attach: State="attached"
 - Detach: State="detached"

Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.

Bead: mulga-siv-131

* fix(reconcile): call AttachIGW when IGW topology missing

applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.

Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.

Closes mulga-siv-132.

* fix(test): align NewExternalIPAM call with 2-arg signature

Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.

* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race

vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.

handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.

Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.

Closes mulga-siv-133.

* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS

spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.

- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
  spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
  with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
  prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
  honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
  external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
  through --extra-vars so `make` callers can override without editing
  defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
  styles (direct -e vs make EXTRA_VARS), NAT removal.

* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness

enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.

- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
  is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
  the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
  state on failure, fan out across all 3 multinode nodes

Refs: mulga-siv-136

* test(e2e): sticky-skip downstream SSH tests when probe fails

When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.

First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.

TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.

Refs: mulga-siv-137

* fix(ipsec): use Debian unit name openvswitch-ipsec.service

CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:

- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
  failed, daemon logged a warning and continued without flipping
  ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
  never received the cert pointers

Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.

Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).

Refs: mulga-siv-136

* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture

admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.

- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.

mulga-siv-134

* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR

Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.

Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.

Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.

* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write

Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.

Sudoers gains one read-only line:
  spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service

This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.

If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.

* chore: unblock preflight on feat/network-layer-redesign

Two pre-existing failures blocked preflight on the branch:

- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
  rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
  Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
  fails when the parent VPC router is missing. That contract changed in
  aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC
  idempotently before EnsureSubnet to avoid a NATS publish-order race.
  Delete the now-impossible failure case.

* refactor(network/ovn): relocate nbdb model package from services/vpcd

Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change.

The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions,
etc.) lived under services/vpcd/nbdb/ because that was the original home
of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all
L3/L5 managers (network/policy, network/external, network/topology,
network/reconcile) became the primary callers — 8 importers in network/*
vs 2 in services/vpcd/. The directory move puts the schema model where
its consumers are, and unblocks deleting services/vpcd/ entirely once
the subscriber/launcher relocations land (Slices B–D).

* refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed

Phase 4 / mulga-siv-125.6 Slice B.

Production path no longer touches any of:

- RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions —
  the 5-pass startup leftovers that network/reconcile (Phase 2.2) made
  redundant. Nothing called them outside tests.
- reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the
  ipv4ToUint32 / uint32ToIPv4 utilities — superseded by
  external.IGWManager (Phase 2.4) which carries its own pool logic.
- ensureLocalnetOptions / ensureGatewayPortNetworks /
  expectedGatewayPortNetwork — read-before-write retrofit shims for the
  same code paths.
- generateMAC — only callers were reconcileIGW + its tests.

useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is
replaced by a policy.NATMode value injected via WithNATMode(...). The
mode is resolved once in vpcd.go from host.UplinkMode and threaded into
both the topology-handler lazy managers and the reconciler stack, so
"is this veth?" is no longer a question topology.go can ask. The
TopologyHandler defaults to NATModeDistributed when WithNATMode is
omitted, matching the previous default.

Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC
of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*,
TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions,
TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils +
net imports in topology.go.

Stale "phase 4 migrates those onto the new managers" comment in vpcd.go
fixed in passing — the subscribers were migrated in Phase 2.6
(mulga-siv-129); only the file-relocation parts of Phase 4 remain.
Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in
network/ovn/{live.go,mock/mock.go} doc comments also cleaned up.

* refactor(network): move NATS VPC subscribers to network/subscribers

Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of
services/vpcd into a new spinifex/network/subscribers package. Subscriber
holds the topology/policy/external managers directly — the lazy *Once
adapter shim (topology_manager.go) is deleted along with vpcd's local
event-struct copies.

launchService now constructs topology.LiveManager eagerly with a DNS
callback derived from external pools (preserving the unscoped-pool lookup
old TopologyHandler.lmDNSServer did), builds every manager up front, then
hands them to subscribers.New + Subscribe. reconcile.New receives
topology.Manager directly instead of through the adapter.

Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the
launcher; they were left behind as topology.go's last responsibilities.

Handler-layer tests (topology_test.go, security_test.go, integration_test.go)
are dropped: substance is covered by network/topology, network/policy,
network/external manager tests. subscribers/subscribers_test.go adds
focused coverage: missing-dep rejection, topic registration smoke, and
bad-JSON envelope per topic.

services/vpcd is now down to the launcher + the ovn/mock compat aliases
(removed in Slice D when the launcher promotes to cmd/vpcd/).

* refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases

Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign
package layout. services/vpcd/ disappears entirely; the launcher remains
an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate
binary — spx vpcd start stays the cobra entrypoint).

- Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post
  Slice C; only callsite was launchService's NewLiveOVNClient, inlined to
  ovn.NewLiveClient).
- git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/.
- Retarget three importers (spinifex/service/service.go +
  service_test.go, cmd/spinifex/cmd/service.go) to the new path.
- Fix stale services/vpcd / TopologyHandler comments in network/reconcile/
  + network/external/types.go + spinifex/vpcd/vpcd.go.
- service-interfaces.yaml vpcd path updated.
- Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was
  speculative; never built) in favour of in-process Service impl at
  spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/.

Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location)
satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled
join token, plan archive).

* fix(ovn): install openvswitch-ipsec + strongswan explicitly

ovs-monitor-ipsec.service fails to start without an IKE daemon. The
openvswitch-ipsec package only Recommends strongswan, so any apt
config that drops Recommends (or a base image with the recommends
already skipped) leaves the service in a Restart loop with 'IKE
daemon is not installed'. Hoist both packages into setup-ovn.sh so
provisioning is the single source of truth — the bootstrap shim in
mulga is no longer needed.

* test(network): pin ADR-0006 invariants to executable suite

Add static-analysis + behavioural tests that bind every safety and
liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure
output quotes the ADR clause verbatim and points at the offending
file:line so future debugging sessions can't quietly bypass a layer
contract to make a test green.

Static-analysis suite (spinifex/network/invariants/):
- S1 layers_test.go: parses `go list -json`; rejects upward layer
  imports and layers pulling from cross-cutter packages
- S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl /
  ovs-ofctl shell-out outside network/host/
- S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode()
  calls outside the init-time resolver locations
- S4 names_test.go: AST-detects OVN object name construction (concat,
  fmt.Sprintf, strings.Join) outside topology/names.go
- S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...}
  and DeleteLogical{...} calls inside network/policy/
- S6 federation_isolation_test.go: rejects TransitSwitch /
  TransitRouterPort references and ovn-ic literals outside
  network/federation/ (pre-protects the layer before it lands)
- S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan /
  openswan imports anywhere in network/
- S9 reconciler_singleton_test.go: rejects Reconcile…Loop and
  Reconcile…Pass<n> function names anywhere in network/
- S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing
  the {azID} segment; calls t.Skip with the gap report so every test
  run prints the work-to-do without breaking CI. Flip to t.Fatalf
  when the cluster-wide subject rename ships.

Behavioural suite (spinifex/network/reconcile/liveness_test.go):
- L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile
  property test; asserts second pass is a no-op
- L2 drift bound: structural check that DriftInterval is positive and
  ≤30 min support window; full fake-clock behavioural test deferred
- L3 federation re-enables: t.Skip stub pointing at the missing
  network/federation/ package

Fixes the one production S4 violation the new walker found:
network/external/igw.go was constructing "gw-port-"+vpcID inline.
Adds topology.GatewaySwitchPort(vpcID) helper and retargets both
AttachIGW and DetachIGW callsites through L2.

Plan + per-clause mechanic + phasing in
docs/development/improvements/network-invariants.md.

Refs: mulga-siv-140

* fix(ovn): install strongswan-charon (not strongswan metapackage)

The strongswan metapackage's '\''charon-systemd | strongswan-charon'\''
alternative resolves to charon-systemd by default, which ships
/usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or
/usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan
shells out to. Install strongswan-charon explicitly to pull
strongswan-starter (legacy ipsec controller) + the matching charon binary,
matching what openvswitch-ipsec.service expects.

Also reset-failed + restart openvswitch-ipsec.service after enable to
recover from the deb post-install starting the service before its
dependencies finished unpacking.

* chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/

Fall-through reviewer for any change is the maintainer.
network/ and vpcd/ are explicitly listed so the matching branch
protection rule ("Require review from Code Owners") routes those
PRs to joshsiv-mulga. Those trees are governed by ADR-0006
invariant tests (spinifex/network/invariants/); CODEOWNERS adds a
human review gate on top of the mechanical fence.

Refs: mulga-siv-141

* chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team

Fallthrough reviewer should be the team, not a single maintainer, so
any engineer can approve PRs outside the explicitly-gated paths.
network/ and vpcd/ keep joshsiv-mulga as the required reviewer per
ADR-0006 invariant ownership.

Refs: mulga-siv-141

* feat(ci): add test_run workflow input for -test.run filter

Pass an optional regex through to each suite's '\''go test -test.run'\'' so
dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without
running the full multinode suite. Threaded through single, multinode, and
reboot suite invocations; empty input preserves current behaviour.

* fix(daemon): flip NB_Global.ipsec=true on management node

Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each
chassis is necessary but not sufficient for OVN native IPsec — without
NB_Global.ipsec=true, ovn-controller never adds options:remote_name to
the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled
tunnels and the kernel xfrm tables stay empty. After enabling
ipsec_encapsulation locally, also write NB_Global.ipsec=true when the
local ovnnb_db socket is present (management node only); workers skip
silently — one writer is enough for a cluster-wide flag.

Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl
invocation so the grant can't be reused for other NB mutations.

* fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec

The strongswan-charon profile only allows reads from /etc/ipsec.*,
/etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes
the per-tunnel strongSwan config with absolute paths to our peer cert +
key under /etc/spinifex/ipsec/, so charon hits 'Permission denied'
loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm
SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the
profile's local override hook + reload.

* fix(ovn): pin OVS system-id to hostname for IPsec cert identity match

ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in
the strongSwan config it generates. Our per-node IPsec peer cert
(admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN.
With chassis-id left at the package-generated UUID, charon validates
`@<UUID>` against a cert with dnsName=<hostname> — mismatch returns
AUTHENTICATION_FAILED and no xfrm SAs are negotiated.

Pin system-id to hostname (which equals the node name on Mulga clusters)
by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf,
read by ovs-ctl on boot) and the running OVS DB value. Writing only one
would drift on the next reboot — the original "do not override" comment
was right about that hazard, just wrong about the resolution.

Cross-link the invariant in admin/ipsec.go: changing the cert identity
without updating setup-ovn.sh (or vice versa) breaks IKE silently.

Refs: mulga-siv-136

* fix(ovn): take --node-name flag so chassis-id matches cluster identity

CI hostnames are bead-themed (env20-casuarina) and don't match the
cluster node name (node1/node2) baked into the IPsec peer cert dnsName
SAN at `spx admin init/join --node NAME`. Pinning system-id to
`hostname -s` produced env20-casuarina; charon then validated
`@node1` against dnsName=env20-casuarina and IKE failed with
AUTHENTICATION_FAILED.

Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for
manual/dev invocations, and update the ansible install role to pass
{{ spinifex_node_name }}. The matching bootstrap-install.sh change
lives in mulga.

Refs: mulga-siv-136

* refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2

Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go
per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs
in spinifex/daemon/ (daemon is an L0 consumer; S2 applies).

Moved to network/host/:
- OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber)
- GetBridgeIPv4 → host.GetBridgeIPv4
- OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus
- daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec

Moved to utils:
- sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam
  (cross-package reassign would trip the reassign linter; helper keeps
  the indirection inside utils).

Moved to vm:
- generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC)

Deleted (dead):
- SetupComputeNode (replaced by network/host/bridges.go:ensureBridges +
  scripts/setup-ovn.sh:572-579)
- EnsureDataRoute
- findInterfaceByIP
- daemon/network.go + dead tests in daemon/network_test.go (duplicates of
  vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go)

S2 invariant (network/invariants/host_containment_test.go) now walks
both spinifex/network/ and spinifex/daemon/, skipping network/host/.

* fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert)

The earlier override granted reads on /etc/spinifex/ipsec/** only, which
covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem.
charon then can't load the CA, can't validate the peer cert chain, and
surfaces the failure as `no trusted RSA public key found for '<peer>'`
even though identity matching now succeeds — IKE_AUTH fails with
AUTHENTICATION_FAILED and no xfrm SAs negotiate.

Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key})
and always rewrite the override so re-runs replace the older narrower
rule.

Refs: mulga-siv-136

* fix(ipsec): symlink cluster CA into charon's default cacerts dir

ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references
in the strongSwan config, but strongSwan 6.0 only loads CAs declared in
standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at
startup. Without this, charon authenticates the local side fine but
rejects every inbound peer cert as `no trusted RSA public key found for
'<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate.

Do the symlink in admin.GenerateIPSecPeerCert (right after writing the
peer cert) — setup-ovn.sh runs before `spx admin init/join` creates
/etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and-
rename pattern so concurrent test invocations don't race on the symlink.

/etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no
profile change needed.

Refs: mulga-siv-136

* chore(setup-ovn): drop apt-get install_packages block

OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold
image via mulga's image-builder/scripts/provision.sh (commits 116bb05,
a9b3a9c). Keeping a runtime apt-get install path here pulled CI through
apt-cacher-ng on every bootstrap and tripped on intermittent "Message
has been manipulated" InRelease cache corruption (env21/env20 runs).

Drop the install block. Gold-image rebuild is the canonical place to
add new system packages. If a package is missing, downstream ovs/ovn
commands fail loudly during setup-ovn.sh's later steps — clearer signal
than silently re-installing through a flaky proxy.

setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's
bootstrap-install.sh:237) so this removes the last runtime apt-get call
on the CI provisioning path.

* fix(ovn): gate system-id pin on explicit --node-name

setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't
passed, and then unconditionally wrote that value to
/etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id.

On CI single-node, the gold image already registered a chassis with
name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed
hostname (env10-banksia). vpcd's discoverChassis treats any chassis row
whose name != local system-id but hostname == local hostname as a stale
local chassis and skips it, so vpcd reported "no OVN chassis registered
in SBDB" and refused to start, cascading into the e2e + e2e-go failures
on run 26430839508.

Drop the hostname fallback. If --node-name is given, pin both the
on-disk file and the running OVS DB to it (callers that care about
IPsec cert identity matching still get it). If no --node-name is given,
leave the existing system-id alone — that's always safe whether the
value came from a gold-image UUID, bootstrap-install.sh writing nodeN
before this script runs, or a manual ansible value.

* fix(network/reconcile): skip orphan prune on startup; drift owns GC

The vpcd startup reconcile races against the daemon's EnsureDefaultVPC:
the daemon writes SG KV rows + publishes vpc.create-sg on each node in
parallel with vpcd starting up. The leader-gated reconcile can load
intent before any SG row is visible on its local JetStream replica
while subscribers on peer nodes are concurrently creating port groups
from the same vpc.create-sg events. A startup orphan-port-group sweep
then deletes managed sg_* port groups that intent hasn't caught up to.

CI run 26434124790 trace:
  node2 reconcile/lock: elected holder=env20-bottlebrush
  node2 reconcile: starting intent_sgs=0
  node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a
  node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391
  node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb

Every subsequent RunInstances → vpc.create-port failed with
"port group not found" for ~3 minutes until the 5-minute drift tick
recreated the port groups. Tests give up after 60s; six multinode
suites cascade-fail on InsufficientInstanceCapacity.

Split the reconciler into Reconcile (full prune, drift loop) and
ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans
bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged.
Legitimate orphans get garbage-collected on the next drift tick — at
worst 5 minutes of lingering rows, no production impact.

Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract.

* fix(ipsec): trigger charon rereadcacerts after CA symlink install

strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit
openvswitch-ipsec.service is enabled at provision time and starts before
admin init writes the cluster CA symlink, so charon ends up running with
zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public
key found for '<peer>'` and no SAs negotiate — the failure mode exposed
by TestMultinodeIPSec.

Append a stroke rereadcacerts call at the end of
installCAIntoCharonTrustStore so an already-running charon picks up the
freshly-placed CA without a service restart. Failure is swallowed at
debug — a missing/stopped charon at admin-init time is expected and a
later startup will scan cacerts naturally.

* refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8)

ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2,
XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated
entirely to OVN native IPSec and is invisible above L0."

The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper
lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths,
EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in
network/host. The orchestrator now lives alongside its callees as
host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at
the call site. Helper and test-override package vars (systemctlActiveTimeout,
ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec
directly rather than via the Daemon struct.

Closes mulga-siv-142

* refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4)

ADR-0006 S4 forbids any layer above L2 from constructing OVN object
names. Three daemon sites formatted "port-" + ENIId inline; route them
through topology.Port instead.

Closes mulga-siv-143

* refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4)

Replace inline "port-"+eniID formatting across five sites (eip, instance x2,
vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin
wrapper to preserve the OVN binding contract comment and existing call sites.

Closes mulga-siv-144

* test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006)

Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces
in CI. Backlog of violations in widened roots was drained by
mulga-siv-142/-143/-144; this commit makes the wider scan permanent.

  - S2 (host_containment): add vm/ alongside network/ + daemon/
  - S3 (nat_mode):         add daemon/ alongside network/ + vpcd/
  - S4 (names):            add daemon/, handlers/, vm/ alongside
                           network/ + vpcd/; add narrow AWS-ID exemption
                           (vpc-cidr-assoc-) so AWS resource IDs are not
                           misclassified as OVN object names
  - S6 (federation):       add daemon/, handlers/ alongside network/;
                           convert single-root walk to slice
  - S7 (nats_subjects):    add daemon/, handlers/, testutil/, utils/ —
                           stays t.Skip pending cluster-wide subject
                           rename; gap report now lists 58 subjects

CODEOWNERS: add /spinifex/network/invariants/ rule so widening or
narrowing the scan needs explicit owner review.

Closes mulga-siv-145
Refs: audit-adr-0006.md

* chore(e2e): disable IPsec in pseudo multi-node test

Pseudo multi-node runs three simulated nodes as unprivileged tf-user.
The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts,
which fails with EACCES under a non-root caller and aborts admin init.

Disable --ipsec for this suite — loopback-only nodes don't exercise
real Geneve encryption anyway. Real-multi-node + single-node e2e
(both run admin init via sudo) continue to cover the IPsec path.

* fix(ovn): atomic Ensure* primitives prevent duplicate logical entities

OVN NB has no unique constraint on Name. The previous
GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU
race: when vpc.create on one node and the defensive vpc.create-subnet
EnsureVPC call on another both observed absence, both inserted, yielding
two routers per VPC. The orphan held DNAT references blocking EIP
detach, and DNAT/flow installation landing on the orphan caused
TestMultinodeGuestSSH SSH timeouts.

Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the
libovsdb client. Each builds a single OVSDB transaction combining a
wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the
insert. The server serialises the wait+insert atomically; concurrent
callers see exactly one survivor and the rest fall through to a re-fetch
that returns the canonical row.

Switch all production callers (topology.EnsureVPC / EnsureSubnet /
EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs /
applySubnets) to the Ensure* variants. Bare Create* remains available to
the L1 wrapper.

Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block
future reintroduction of the bare Create* call sites outside
network/ovn.

Mock implements Ensure* under its existing mutex so the concurrency
tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single
UUID survives.

Refs: mulga-siv-146

* fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate

The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus
`Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation
struct tags Rows as `json:"rows,omitempty"`, so the empty slice was
dropped on the wire and ovsdb-server rejected every transaction with
"Required 'rows' member is missing", breaking reconciler bootstrap and
every downstream create-port lookup ("logical switch X not found").

Flip the predicate to the equivalent `Until: !=` formulation with a
single-row projection containing the Name we want absent:

  no row    → projection []           != [{name}] → predicate holds
  row exists → projection [{name}]    == [{name}] → predicate fails

Same atomic guarantee, JSON now well-formed. Mock tests can't catch
this — only a real ovsdb-server validates the envelope.

* chore: remove generateMAC wrapper

* chore: remove all bead / plan references

* fix(network/policy): wait for chassis flows after AddNATGateway

AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every
chassis installs the new dnat_and_snat flows; AddNATGateway returned right
after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would
publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API
call would succeed before the gateway chassis had programmed the SNAT flow
for the private subnet.

Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow
CI hosts (last seen run 26455602615 single-node, identical flake on
unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*).

Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test
TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path.

* chore: remove more plan refs

* chore: remove GenerateIPSecPeerCertIfEnabled

* chore: remove dupliate subnetGatewayCIDR function

* chore(network): trim superfluous comments across network/ tree

Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement
comments, remove section dividers grouping <10 items, and cut prose
narration in tests. ADR clause references, one-line WHY comments for
non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are
preserved. No behavioural changes.

Packages: invariants, topology, host, external, ovn, reconcile, policy,
subscribers.

* fix(igw): restore attachment State="available" for AWS parity

AWS InternetGatewayAttachment.State is "available" when an IGW is
attached to a VPC, per the AWS SDK doc ("the state is available when
attached to a VPC; otherwise, this value is not returned"). Commit
b7505f8 wrote "attached"/"detached" to align with the reconciler
filter, which inverted AWS parity.

Fix the producer to write "available" again and update the consumer:
 - handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway
   both persist State="available"; IGWRecord comment narrowed.
 - network/reconcile/intent.loadIGWs: filter accepts "available" (was
   "attached"); attachment is identified by VpcId != "".
 - tests updated: igw service_impl_test, routetable service_impl_test
   fixture, reconcile intent_test fixture.

* Update spinifex.toml

* refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1)

Subscriber previously called policy.NATManager directly for EIP and NAT
Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager)
that existed on paper. ADR-0006 §S1 requires every cross-layer call to
pass through the immediate lower neighbor's typed interface; IGW already
followed this pattern.

Wire EIPManager and NATGWManager into the Subscriber struct, construct
them in vpcd alongside IGWManager, and swap the four direct-L3 call
sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the
L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway).
AttachEIP now fires the flows-barrier, giving callers the same
"reachable on return" guarantee IGW already had.

The reconciler still calls policy.NATManager directly; that's a
cross-cutter and a separate concern.

* test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh

App VMs' user-data writes (unit file + index.html) sit in the on-host
WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before
that loses them — systemd then refuses to load the zero-byte unit and
the responder never starts. Mirror the bash driver's dd+sync pad so the
WAL flushes pre-reboot. Drop when the upstream WAL fix lands.

* test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure

Phase 8d's existing failure path only prints lr-nat-list and
lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits
nonzero, so debugging the NATGW regression requires inline diagnostics
in the journal. Expand the dump to include nb/sb show, chassis bindings,
port_group ACLs, br-int flows filtered to the NATGW IP/private subnet,
host routes, and the private VM's own ip/route/neigh state.

* fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet

Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure):

1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and
   only mutated the named router afterwards. With AWS subnet CIDRs repeating
   across VPCs, (type, logicalIP/externalIP) is not globally unique — the
   delete picked a NAT row owned by a different router, the mutate was a
   no-op, and the row delete failed with "remaining reference(s)", leaving
   the original router's NAT orphaned. Both calls now resolve the LR first
   and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping.

2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as
   SubnetCIDR, producing an snat for the wrong CIDR that collided with the
   correct event-published snat. loadNATGWs now walks route tables and
   emits one NATGWSpec per (NATGW, associated private subnet) pair,
   mirroring publishNatGatewayEventsForAssociation. Intent map keyed by
   natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs.

Tests:
- nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and
  DeleteNATGateway
- intent_test.go: NATGW snat uses associated private subnet CIDR, not the
  NATGW home subnet; NATGW with no associations produces no spec

---------

Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>

2026-05-27 21:05:01 +10:00

39 KiB

Raw Blame History

E2E Test Coverage

Single-Node (`run-e2e.sh`)

Phase 1: Environment Setup

KVM support check
spx admin init (region/az/node config)
CA certificate trust
Start all services (start-dev.sh)
Wait for AWS gateway on localhost:9999

Phase 1b: Cluster Stats CLI

spx get nodes — verify node shows Ready
spx top nodes — verify CPU/MEM resource stats
spx get vms — verify "No VMs found" before any launches

Phase 2: Discovery & Metadata

describe-regions
describe-availability-zones (verify zone name and state)
describe-instance-types (discover available types)
Select nano instance type and detect architecture

Phase 2b: Serial Console Access Settings

get-serial-console-access-status (verify default disabled)
enable-serial-console-access (verify returns true, get confirms true)
disable-serial-console-access (verify returns false, get confirms false)

Phase 3: SSH Key Management

create-key-pair (test-key-1, verify private key material)
import-key-pair (test-key-2, from local RSA key)
describe-key-pairs (verify both keys exist)
delete-key-pair (test-key-2, verify only test-key-1 remains)

Phase 4: Image Management

spx admin images import (file-based, architecture-aware)
describe-images (verify AMI by ID)

Phase 5: Instance Lifecycle

run-instances (launch VM with key pair)
describe-instances (poll pending -> running)

Phase 5a-pre: Cluster Stats CLI (with running VM)

spx get vms — verify running instance appears in output

Phase 5a: Instance Metadata Validation

describe-instances — verify InstanceType matches requested type
Verify KeyName matches requested key
Verify ImageId matches requested AMI
Verify at least 1 BlockDeviceMapping present

Phase 5a-ii: SSH Connectivity & Volume Verification

SSH into instance via QEMU hostfwd port
Verify SSH connectivity (id command returns ec2-user)
Verify root volume size from inside VM matches API-reported size (lsblk vs describe-volumes)
Verify VM hostname

Phase 5a-iii: Console Output

get-console-output succeeds (verify InstanceId in response)

Phase 5 (cont): Root Volume

describe-volumes (verify root volume attached)

Phase 5b: Volume Lifecycle

create-volume (10GB, ap-southeast-2a)
modify-volume (resize to 20GB, poll to verify)
attach-volume (to running instance, /dev/sdf)
describe-volumes (verify in-use + attached state)
detach-volume (verify available state)
delete-volume (verify gone)

Phase 5b-ii: DescribeVolumeStatus

describe-volume-status (on root volume, verify VolumeId in response)

Phase 5c: Snapshot Lifecycle

Uses root volume already attached to running instance (snapshots require a mounted VB instance)
create-snapshot (from attached root volume, with description)
Verify create response fields (VolumeId, VolumeSize, State, Progress)
describe-snapshots (by ID, verify VolumeId/Size/Description)
copy-snapshot (with new description, verify distinct ID)
describe-snapshots (verify both original + copy visible)
delete-snapshot (original, verify gone while copy survives)
delete-snapshot (copy, cleanup)

Phase 5d: Verify Snapshot-Backed Instance Launch

All run-instances calls use the snapshot path (cloneAMIToVolume -> OpenFromSnapshot), so the Phase 5 instance is already snapshot-backed
Verify AMI snapshot exists in Predastore (snap-{amiId}/config.json)
Read Phase 5 root volume's config.json from Predastore
Verify SnapshotID and SourceVolumeName are set (proves zero-copy clone)

Phase 5e: CreateImage Lifecycle

create-image (from running instance, with name and description)
Verify returned ImageId is non-empty
describe-images (verify custom AMI name and state)
Extract backing snapshot ID from Predastore config (for cleanup before termination)

Phase 5f: Security Group Enforcement (egress ACL)

Proves vpcd programs OVN ACLs that actually drop traffic, not just nominal SG records. Tests egress only — in single-node dev_networking mode, runner→VM SSH bypasses OVN via the hostfwd NIC, so ingress isn't a faithful probe. Egress goes through the OVN tap NIC (default route).

Discover VM's OVN gateway from inside the VM (ip route show default)
Test 5f-1 (baseline): ICMP from VM to gateway works under default SG (allow-all egress)
- Skipped gracefully if env blocks ICMP regardless of SG (no false fail)
Test 5f-2 (revoke): revoke-security-group-egress strips the allow-all rule → ICMP must drop
Test 5f-3 (re-authorize): authorize-security-group-egress restores the rule → ICMP must work again
Restore the default egress rule on any failure path so subsequent phases (6, 7, 8) keep their SSH probes working

Phase 7: Instance State Transitions

stop-instances (poll -> stopped)

Phase 7a: Attach Volume to Stopped Instance (Error Path)

create-volume (for attach test)
attach-volume to stopped instance (expect IncorrectInstanceState error)
delete-volume (cleanup)

Phase 7b: ModifyInstanceAttribute

modify-instance-attribute (change instance type from nano → xlarge while stopped)
describe-instances (verify type updated in KV)
describe-instance-types (get expected vCPU count and memory for new type)
start-instances (poll -> running with new type)
SSH: nproc — verify vCPU count matches xlarge (4 vCPUs)
SSH: /proc/meminfo MemTotal — verify memory matches xlarge (within 85% of expected)

Phase 7c: RunInstances with count > 1

run-instances --count 2 (launch 2 instances in a single call)
Verify 2 instances returned in response
Poll both to running state
terminate-instances (both, poll -> terminated)

Phase 8: Negative / Error Path Tests

run-instances with malformed AMI ID (expect InvalidAMIID.Malformed)
run-instances with invalid instance type (expect InvalidInstanceType)
attach-volume on in-use volume (expect VolumeInUse)
detach-volume on boot volume (expect OperationNotPermitted)
delete-snapshot on non-existent snapshot (expect InvalidSnapshot.NotFound)
Unsupported Action via raw HTTP (expect InvalidAction or error response)
run-instances with non-existent AMI ID (expect InvalidAMIID.NotFound)
run-instances with non-existent key pair (expect InvalidKeyPair.NotFound)
delete-volume on non-existent volume (expect InvalidVolume.NotFound)
create-key-pair with duplicate name (expect InvalidKeyPair.Duplicate)
import-key-pair with duplicate name (expect InvalidKeyPair.Duplicate)
import-key-pair with invalid key format (expect InvalidKey.Format)
describe-volumes with non-existent volume ID (expect InvalidVolume.NotFound)
describe-images with non-existent AMI ID (expect InvalidAMIID.NotFound)
create-image with duplicate name (expect InvalidAMIName.Duplicate)
delete-key-pair on non-existent key (expect success — idempotent, matches AWS)
modify-instance-attribute on running instance (expect InvalidInstanceID.NotFound — running instances not in stopped KV)

IAM Phase 1: User CRUD

Root auth via iam list-users (root user exists)
create-user (alice, bob with path)
Duplicate user (expect EntityAlreadyExists)
get-user (alice)
get-user non-existent (expect NoSuchEntity)
list-users (verify count)
list-users with --path-prefix filter

IAM Phase 2: Access Key Lifecycle

create-access-key (alice key 1, key 2)
Third key exceeds limit (expect LimitExceeded)
create-access-key for non-existent user (expect NoSuchEntity)
list-access-keys (alice: 2 keys, bob: 0 keys)
update-access-key (Inactive, verify status, reactivate)
delete-access-key (verify count decremented)

IAM Phase 3: User Authentication

Configure IAM user profile with access key
Deactivate key → auth rejected (InvalidClientTokenId)
Reactivate key
Wrong secret → SignatureDoesNotMatch
Non-existent key ID → InvalidClientTokenId
Multi-user simultaneous auth (root + bob)

IAM Phase 4: Policy CRUD

create-policy (EC2ReadOnly, FullAdmin, DenyTerminate, IAMReadOnly, EC2DescribeAll)
Duplicate policy (expect EntityAlreadyExists)
Malformed JSON (expect MalformedPolicyDocument)
get-policy (by ARN)
get-policy non-existent (expect NoSuchEntity)
get-policy-version (v1)
list-policies (verify count)

IAM Phase 5: Policy Attachment & Enforcement

Create charlie with access key
attach-user-policy (alice: EC2ReadOnly + IAMReadOnly, bob: DenyTerminate)
list-attached-user-policies (verify count)
Idempotent attach (no duplicate)
Attach non-existent policy (expect NoSuchEntity)
Attach to non-existent user (expect NoSuchEntity)
Default Deny: charlie (no policies) → AccessDenied on ec2 + iam
Explicit Allow: alice → ec2:Describe{Instances,Vpcs} allowed, ec2:DescribeKeyPairs denied, iam:ListUsers allowed, iam:CreateUser denied
Deny Override: bob → ec2:Describe allowed, ec2:TerminateInstances denied (explicit Deny), iam denied
Root Bypass: root user → all actions succeed
Prefix Wildcard: swap alice to EC2DescribeAll (ec2:Describe*) → Describe* allowed, non-Describe denied
FullAdmin: charlie with FullAdmin → all actions allowed

IAM Phase 6: Policy Lifecycle — Detach & Delete

detach-user-policy → user loses access
delete-policy while attached (expect DeleteConflict)
Detach then delete → get-policy returns NoSuchEntity

IAM Phase 7: IAM Cleanup

Delete all test users (remove keys + policies first)
Verify root-only remains (list-users count = 1)
Delete all test policies (list-policies count = 0)
Clean up AWS CLI profiles

Phase 8: EC2 Account Scoping

Tests that EC2 resources are properly isolated between tenant accounts (Alpha, Beta). Based on docs/development/feature/iam-phase4-e2e-test-guide.md. Skips CreateImage (mulga-612) and instance tags (mulga-613).

Step 1: Account Setup

spx admin account create (Team Alpha, Team Beta)
Configure AWS CLI profiles
Verify auth for both accounts

Step 2: Instance Scoping

Alpha + Beta each launch 1 instance
describe-instances isolation (each sees only own)
OwnerId field matches account ID
Cross-account stop-instances → InvalidInstanceID.NotFound
Cross-account terminate-instances → InvalidInstanceID.NotFound
Cross-account start-instances → InvalidInstanceID.NotFound
Cross-account modify-instance-attribute → InvalidInstanceID.NotFound
Cross-account get-console-output → InvalidInstanceID.NotFound

Step 3: Volume Scoping

Alpha + Beta each create 1 volume
describe-volumes isolation
Cross-account describe-volumes by ID → InvalidVolume.NotFound
Cross-account delete-volume → InvalidVolume.NotFound
Cross-account attach-volume → InvalidVolume.NotFound
Cross-account detach-volume → InvalidVolume.NotFound
Cross-account modify-volume → InvalidVolume.NotFound

Step 4: Key Pair Scoping

Alpha + Beta each create key pair
describe-key-pairs isolation
Same key name in both accounts (namespace isolation — different KeyPairIds)
Cross-account delete-key-pair → no effect on other account's key
import-key-pair → invisible to other account

Step 5: Snapshot Scoping

Alpha + Beta each create snapshot from own volume
describe-snapshots isolation + OwnerId verification
Cross-account delete-snapshot → UnauthorizedOperation
Cross-account create-snapshot from other's volume → InvalidVolume.NotFound

Step 6: VPC/Subnet Scoping

Alpha + Beta each create VPC (same CIDR — no conflict)
describe-vpcs isolation
Cross-account describe-vpcs by ID → InvalidVpcID.NotFound
Cross-account delete-vpc → InvalidVpcID.NotFound
Alpha + Beta each create subnet
describe-subnets isolation
Cross-account create-subnet in other's VPC → InvalidVpcID.NotFound
Cross-account delete-subnet → InvalidSubnetID.NotFound

Step 7: IGW + EIGW Scoping

Alpha + Beta each create IGW
describe-internet-gateways isolation
Cross-account describe-internet-gateways by ID → InvalidInternetGatewayID.NotFound
Cross-account delete-internet-gateway → InvalidInternetGatewayID.NotFound
Cross-account attach-internet-gateway → InvalidInternetGatewayID.NotFound
Cross-account detach-internet-gateway → InvalidInternetGatewayID.NotFound
Alpha + Beta each create EIGW
describe-egress-only-internet-gateways isolation
Cross-account EIGW delete → no effect

Step 8: Account Settings

Alpha enable-ebs-encryption-by-default → Beta unaffected
Independent toggle verification

Step 9: Global Resources

describe-regions identical for both accounts
describe-availability-zones identical
describe-instance-types identical

Step 10: Edge Cases

Empty account (Gamma) — no resources visible
Root isolation from tenants (root cannot see tenant instances)
Non-existent resource IDs → same error as cross-account (no info leakage)

Step 11: Cleanup

Terminate instances, delete volumes/snapshots/keys/VPCs/IGWs/EIGWs/subnets
Clean up AWS CLI profiles

Phase 8b: VPC Public/Private Subnet E2E

Gated on external_mode = "pool" (skipped when only dev_networking is configured). Validates AWS-parity public/private subnet semantics end-to-end through OVN.

Step 1: Discover default VPC + default subnet (MapPublicIpOnLaunch=true)
Step 2: Public subnet instance
- run-instances into default subnet → auto-assigned PublicIpAddress
- ovn-nbctl lr-nat-list shows per-instance dnat_and_snat rule for PUB_PRIVATE_IP
- Asserts NO blanket VPC CIDR SNAT (mulga-754 AWS-parity regression)
- SSH via public IP succeeds (when bridge reachable)
- Outbound internet from inside the VM (via OVN SNAT)
- Step 2b: In-guest TLS trust — cloud-init injects Spinifex CA into the guest, so curl https://awsgw and https://predastore-s3 work without --insecure
Step 3: Private subnet isolation
- create-subnet 172.31.16.0/20 with MapPublicIpOnLaunch=false
- run-instances into private subnet → no public IP, no internet access from inside the VM

Phase 8c: Route Table Validation

Step 1: Default VPC main route table — local route + 0.0.0.0/0 → IGW present
Step 2: Custom route table CRUD lifecycle — create, list local route, create-route, associate-route-table, disassociate-route-table, delete-route, delete-route-table
Step 3: Error paths — cannot delete the local route; cannot delete the main route table

Phase 8d: NAT Gateway E2E (mulga-763)

Reuses PUB_INSTANCE_ID (bastion) + PRIV_INSTANCE_ID from Phase 8b. End-to-end proof that NAT GW creation/teardown flips private-instance outbound connectivity.

Step 1: Bastion → private SSH ready (via ProxyJump through PUB_IP)
Step 2: Baseline — private VM cannot reach internet
Step 3: allocate-address + create-nat-gateway + create-route (0.0.0.0/0 → NAT GW); verify OVN SNAT rule appears
Step 4: Private VM CAN now reach internet (retried, NAT propagation can take a few seconds)
Step 5: delete-nat-gateway + cleanup route + release-address; verify private VM loses internet again

Phase 8e: Security Group Enforcement (SG-to-SG datapath)

End-to-end proof that the RunInstances → ENI → vpcd → OVN port group + ACL → datapath chain actually drops unauthorized traffic. This is the deepest SG enforcement test in the suite and supersedes Phase 5f for SG-to-SG semantics.

Capacity note: Phase 8b's instance termination (PUB+PRIV) runs before Phase 8e on single-node to free 2 nano slots. Without that, 5 simultaneous nanos (main + PUB + PRIV + client + target) exhaust the runner and the gateway masks InsufficientInstanceCapacity as InvalidInstanceType via isKnownInstanceType.
Step 1: Create sge-client and sge-target SGs; authorize tcp/22 on client-sg from 0.0.0.0/0; authorize tcp/8080 on target-sg from client-sg (SG-to-SG via UserIdGroupPairs)
Step 2: Launch client-vm (sge-client) and target-vm (sge-target) in default subnet; target VM's HTTP server started via cloud-init user-data (port 8080) — target-sg has no SSH ingress, so we never SSH into it
Step 3: ovn-nbctl port_group membership inspection — both client + target LSPs are in their port groups; verifies the SG-to-SG match expression resolves
Step 4: Allowed — curl client → target:8080 succeeds (SG-to-SG allow)
Step 5: Denied — nc client → target:22 blocked (target-sg has no SSH ingress; default-deny ACL drops)
Step 6: Revoke target-sg's tcp/8080 rule and retest immediately — confirms vpcd's sync RequestEvent contract (no eventually-consistent gap)
Step 7: Cleanup — terminate both VMs, delete both SGs

Phase 9: Terminate and Verify Cleanup

delete-snapshot (CreateImage backing snapshot, so DeleteOnTermination is not blocked)
terminate-instances (poll -> terminated)

Phase 9a: SSH Unreachable Verification

Verify SSH connection is refused/unreachable after termination

Phase 9b: Volume Cleanup Verification

describe-volumes on root volume after termination
Verify root volume is deleted (DeleteOnTermination)

Pseudo Multi-Node (`run-pseudo-multinode-e2e.sh`)

Phase 1: Environment Setup

KVM support check
Simulated network IPs (10.11.12.{1,2,3} on loopback)
Simulated network IPs (no ramdisk — start-dev.sh uses disk-backed WAL/VB in CI)

Phase 2: Cluster Initialization

spx admin init (leader node1) — generates join token, writes to ~/node1/config/join-token
CA certificate trust
Read join token from node1 config
spx admin join with --token (node2, node3 — concurrent)
Start all 3 node services concurrently

Phase 3: Cluster Health Verification

Verify NATS cluster (3 nodes)
Verify Predastore cluster (3 nodes)
Wait for gateway on node1
Wait for daemon NATS readiness
describe-regions (gateway connectivity check)

Phase 3b: Cluster Stats CLI

spx get nodes — verify all 3 nodes show Ready
spx top nodes — verify instance type capacity table
spx get vms — verify empty (no instances yet)

Phase 4: Image and Key Setup

describe-instance-types (discover + select nano)
create-key-pair
spx admin images import (with node1 config paths)
describe-images (verify AMI)
create-security-group + 2× authorize-security-group-ingress (tcp/22 + icmp from 0.0.0.0/0)
- All subsequent run-instances use this SG so VMs are reachable. The default SG is egress-only, so without this every Phase 5 SSH probe would be dropped by OVN ACL.

Phase 4b: Multi-Node Key Pair Operations

import-key-pair (multinode-test-key-2, from local RSA key)
describe-key-pairs (verify both keys visible across cluster)
delete-key-pair (multinode-test-key-2, verify deletion)

Phase 5: Multi-Node Instance Tests

Test 1: Instance Distribution

run-instances x3 (distribute across nodes)
Poll all instances to running state
Check instance distribution across nodes
spx get vms — verify all instances visible

Test 1a-ii: SSH Connectivity & Volume Verification

SSH into all 3 instances via QEMU hostfwd port
Verify SSH connectivity (id command returns ec2-user)
Verify root volume size from inside VM matches API-reported size (lsblk vs describe-volumes)
Verify VM hostname

Test 1b: Volume Lifecycle

create-volume (10GB)
modify-volume (resize to 20GB)
attach-volume (to first instance)
detach-volume
delete-volume

Test 1c: Snapshot Lifecycle

Uses root volume of first instance (snapshots require a mounted VB instance)
create-snapshot (from attached root volume, with description)
Verify create response fields (VolumeId, VolumeSize, State)
describe-snapshots (by ID, verify fields)
copy-snapshot (with new description)
describe-snapshots (verify both exist)
delete-snapshot (original, verify copy survives)
delete-snapshot (copy, cleanup)

Test 1c-ii: Verify Snapshot-Backed Instance Launch

All run-instances calls use the snapshot path (cloneAMIToVolume -> OpenFromSnapshot), so the Test 1 instances are already snapshot-backed
Verify AMI snapshot exists in Predastore (snap-{amiId}/config.json)
Read first instance's root volume config.json from Predastore
Verify SnapshotID and SourceVolumeName are set (proves zero-copy clone)

Test 1d-ii: Tag Management (Volumes)

create-tags (2 tags on root volume)
describe-tags (filter by resource-id, verify count)
describe-tags (filter by resource-type=volume)
delete-tags (both tags, verify cleanup)

Test 2: DescribeInstances Aggregation

describe-instances (fan-out across all nodes, verify count)

Test 3: Cross-Node Operations

stop-instances (poll -> stopped)
start-instances (poll -> running)

Test 4: NATS Cluster Health (Post-Operations)

Verify NATS cluster still healthy after all operations

Test 5: VM Crash Recovery

Kill QEMU process with SIGKILL (simulate OOM kill)
Verify daemon detects crash (instance transitions to error/pending)
Wait for auto-restart (backoff starts at 5s)
Verify new QEMU PID differs from original
Verify instance reaches running state
Verify SSH connectivity after recovery

Test 5b: Crash Loop Prevention

Kill QEMU 4 times rapidly on a third instance
Verify crash loop is detected and restarts stop after max attempts (3 in 10-min window)
Verify instance reaches error state (won't restart further)

Phase 5c: IAM Accounts & Cross-Account Isolation

Step 1: Create Accounts

spx admin account create (Team Alpha, Team Beta)
Verify sequential 12-digit account IDs
spx admin account list (verify both accounts)

Step 2: Account Admin Auth

Alpha admin authenticates via ec2 + iam
Beta admin authenticates via ec2 + iam

Step 3: Account-Scoped Users

create-user alice in Alpha + Beta (same name, different accounts)
create-user team-member (Alpha), dev-user (Beta)
list-users scoped per account (verify different user lists)
Cross-account isolation: Alpha cannot see Beta's users and vice versa (NoSuchEntity)

Step 4: Account-Scoped Access Keys

create-access-key for alice in Alpha + Beta
Configure separate AWS CLI profiles per account user

Step 5: Account-Scoped Policies & Enforcement

create-policy EC2ReadOnly in Alpha (narrow: DescribeInstances + DescribeVpcs)
create-policy EC2ReadOnly in Beta (broad: ec2:Describe*)
attach-user-policy to alice in both accounts
Alpha alice: DescribeInstances allowed, DescribeKeyPairs denied (narrow)
Beta alice: DescribeInstances allowed, DescribeKeyPairs allowed (broad Describe*)
Both denied: CreateKeyPair

Step 6: Cross-Account Delete Isolation

Delete Alpha's alice (detach policy, delete key, delete user)
Verify Alpha alice gone (NoSuchEntity)
Verify Beta alice unaffected (still exists, auth still works)

Step 7: EC2 Resource Scoping

Tests that EC2 resources are properly isolated between the Alpha/Beta accounts. Skips CreateImage (mulga-612) and instance tags (mulga-613).

7a: Instance Scoping

Alpha + Beta each launch 1 instance
describe-instances isolation (each sees only own)
OwnerId field matches account ID
Cross-account stop-instances → InvalidInstanceID.NotFound
Cross-account terminate-instances → InvalidInstanceID.NotFound
Cross-account get-console-output → InvalidInstanceID.NotFound

7b: Volume Scoping

Alpha + Beta each create 1 volume
describe-volumes isolation
Cross-account describe-volumes by ID → InvalidVolume.NotFound
Cross-account delete-volume → InvalidVolume.NotFound
Cross-account attach-volume → InvalidVolume.NotFound
Cross-account detach-volume → InvalidVolume.NotFound
Cross-account modify-volume → InvalidVolume.NotFound

7c: Key Pair Scoping

Alpha + Beta each create key pair
describe-key-pairs isolation
Same key name in both accounts (namespace isolation — different KeyPairIds)
Cross-account delete-key-pair → no effect on other account's key
import-key-pair → invisible to other account

7d: Snapshot Scoping

Alpha + Beta each create snapshot from own volume
describe-snapshots isolation + OwnerId verification
Cross-account delete-snapshot → UnauthorizedOperation
Cross-account create-snapshot from other's volume → InvalidVolume.NotFound

7e: VPC/Subnet Scoping

Alpha + Beta each create VPC (same CIDR — no conflict)
describe-vpcs / describe-subnets isolation
Cross-account VPC/subnet operations → NotFound

7f: IGW + EIGW Scoping

Alpha + Beta each create IGW + EIGW
Describe isolation for both
Cross-account attach/detach/delete → NotFound

7g: Account Settings

enable-ebs-encryption-by-default per account — independent toggle

7h: Global Resources

describe-regions, describe-availability-zones, describe-instance-types identical for both accounts

Step 8: Edge Cases

Empty account (Gamma) — no resources visible
Root isolation from tenants
Non-existent resource IDs → same error as cross-account
Parallel key creation (race condition) — no cross-contamination

Step 9: EC2 + IAM Cleanup

Terminate instances, delete volumes/snapshots/keys/VPCs/IGWs/EIGWs/subnets
Delete all IAM users, policies, access keys
Clean up AWS CLI profiles

Phase 5d: VPC Networking

Step 1: create-vpc (10.100.0.0/16) + create-subnet (10.100.1.0/24)
Step 2: run-instances x3 with --subnet-id (launch into VPC subnet)
Poll all VPC instances to running state
Step 3: Verify PrivateIpAddress in describe-instances for each instance
Verify SubnetId and VpcId match requested values
Verify at least 1 NetworkInterface per instance
Verify all IPs are unique and in subnet range (10.100.1.x)
Step 4: SSH + ping connectivity (skipped in CI — OVN DHCP wait too slow)
Step 5: Stop/start IP persistence
- stop-instances (all VPC instances, poll -> stopped)
- Verify PrivateIpAddress persists in stopped state
- start-instances (all VPC instances, poll -> running)
- Verify PrivateIpAddress identical after restart
Step 6: Cleanup — terminate VPC instances, delete subnet, delete VPC

Phase 6: Cluster Shutdown + Restart

Test 6a: Dry-Run Shutdown

spx admin cluster shutdown --dry-run
Validate output contains all 5 phases (GATE, DRAIN, STORAGE, PERSIST, INFRA)

Test 6b: Coordinated Cluster Shutdown

spx admin cluster shutdown --force --timeout 60s
Verify all services down on all nodes (gateway, NATS, QEMU)

Test 6c: Cluster Restart + Recovery

Restart all 3 node services
Verify NATS cluster reforms (3 members)
Wait for gateway and daemon readiness
Smoke test: describe-instance-types returns valid results

Test 6d: Instance Relaunch + Terminate

Wait for instances to finish relaunching after restart (pending → running/error)
terminate-instances (all 3 instances)
Poll all to terminated state

Cleanup

Coordinated cluster shutdown (with fallback to per-node PID stops)
Remove simulated IPs

Real Multi-Node (`run-multinode-e2e.sh`)

Runs on a real 3-node libvirt cluster provisioned by OpenTofu (scripts/tofu-cluster/). Each node is a separate VM with its own OVN, NATS, Predastore, and Spinifex daemon. Bootstrap (bootstrap.sh) handles all provisioning before the test script runs.

Go port: every phase below has a TestMultinode* counterpart in tests/e2e/multinode/ (phase N -> phaseN_* helper). CI runs the Go suite via the e2e_go_multinode job in .github/workflows/e2e-go.yml; the bash driver stays in place as the executable spec until the Go run has a soak-tested cadence on ci-multi.

Bootstrap (pre-test)

OpenTofu provisions 3 libvirt VMs (bottlebrush, ironbark, casuarina) with cloud-init
Wait for all nodes to be SSH-reachable
Copy SSH key to node1 for peer access
git fetch + checkout test branch on all nodes (all 3 repos)
make build on all nodes (parallel)
setup-ovn.sh — primary gets --management, secondaries connect to primary's OVN central
spx admin init on primary (generates join token), read token via SSH, spx admin join --token on secondaries
start-dev.sh on all nodes
Wait for /health (awsgw=ok) on all nodes
Install Spinifex CA certificate on all nodes
import-key-pair (spinifex-key) + spx admin images import (Ubuntu 24.04)
create-vpc (10.200.0.0/16) + create-subnet (10.200.1.0/24)

Phase 1: Pre-flight Validation

KVM support check (/dev/kvm writable)
SSH connectivity to all peer nodes

Phase 2: Cluster Health

NATS cluster: verify 2 unique route peers (3-node quorum)
Predastore reachable on all 3 nodes
AWS gateway reachable on all 3 nodes
Daemon readiness (describe-instance-types returns results)
spx get nodes — verify all 3 nodes show Ready
spx get vms — verify empty (no instances yet)

Phase 3: Instance Lifecycle + Distribution

Discover nano instance type and AMI (from bootstrap import)
authorize-security-group-ingress (tcp/22 on default VPC's default SG, 0.0.0.0/0)
- Required because the default SG is egress-only and the runner's IP is not a member; without this Phase 4 SSH would be dropped by OVN port-group ACL.
- Idempotent (tolerates InvalidPermission.Duplicate on re-runs).
run-instances x3 with staggered launches
Poll all instances to running state
Check instance distribution across physical nodes (QEMU process check)
Verify at least 2 different hosting nodes (non-deterministic, non-fatal if all on 1)
spx get vms — verify all 3 instances visible

Phase 4: SSH into Guest VMs

For each instance across all nodes:
- Find hosting node via QEMU process scan
- Extract SSH hostfwd port from QEMU command line on remote node
- Wait for SSH readiness (up to 60s)
- Verify SSH connectivity (id returns ec2-user)
- Verify block device (lsblk)

Phase 5: Volume Lifecycle

create-volume (10GB)
attach-volume (to first instance, /dev/sdf)
Poll for in-use + attached state
detach-volume (poll for available)
delete-volume (verify gone)

Phase 6: Cross-Node Gateway Access

describe-instances via node1 gateway (baseline count)
describe-instances via node2 and node3 gateways
Verify all gateways return the same instance count (fan-out aggregation)

Phase 7: Cross-Node Operations

Find which node hosts instance 0
Pick a different node's gateway for stop
stop-instances via remote gateway (poll -> stopped)
Pick a third node's gateway for start
start-instances via another remote gateway (poll -> running)

Phase 8: Node Failure

stop-dev.sh on node2 (simulate node failure)
Verify node1 still serves requests (describe-instance-types)
Verify node3 still serves requests
Check NATS degraded state (1 peer instead of 2)
describe-instances still works from surviving nodes

Phase 9: Node Recovery

start-dev.sh on node2 (restart failed node)
Wait for NATS cluster to reform (2 peers again)
Verify node2 gateway is back
spx get nodes — verify all 3 nodes Ready again
Verify node2 serves requests after recovery

Phase 10: Cleanup

terminate-instances (all 3 instances, poll -> terminated)

Consolidated LB Data Plane (`run-lb-e2e.sh`)

Consolidated LB data plane test covering all 4 variants: ALB internet-facing, ALB internal, NLB internet-facing, NLB internal. Shares a single VPC, subnet, and set of dual-purpose app instances (HTTP:80 + TCP:9000) across all suites. Requires pool mode with external IPAM (NOT dev_networking).

Phase 0: Prerequisites

Dev-mode gate (skip if dev_networking enabled)
SSH to peer node (if --peer provided)
Discover instance types (nano), AMI (non-Alpine)
Create key pair (lb-e2e-key)

Phase 1: Shared VPC + Subnet

create-vpc (10.200.0.0/16)
create-internet-gateway + attach
create-subnet (10.200.1.0/24) + MapPublicIpOnLaunch

Phase 2: Launch App Instances

run-instances x2 with dual-purpose cloud-init (HTTP on 80, TCP echo on 9000)
Poll instances to running state, collect private IPs
Poll app ports for readiness (replaces sleep 100)

Phase 3a: ALB Internet-Facing [skip if no --peer]

Create HTTP target group, register 2 targets
create-load-balancer (scheme=internet-facing), verify scheme
Verify ENI has public IP, DNS has no internal- prefix
create-listener (HTTP:80)
Wait for ALB active (agent heartbeat)
Host HTTP traffic test (curl 20x, verify round-robin)
Peer validation (curl from --peer node, 20x)
Cleanup: delete listener, LB, verify ENI removal, delete TG

Phase 3b: ALB Internal

Create HTTP target group, register 2 targets
create-load-balancer (scheme=internal), verify scheme
Verify ENI has NO public IP, DNS has internal- prefix
create-listener (HTTP:80)
Wait for ALB active (mgmt NIC heartbeat)
Wait for targets healthy
Launch client VM (cloud-init curls ALB private IP, serves results)
Poll client status.txt, fetch results, verify round-robin
Terminate client VM
Cleanup: delete LB, verify ENI removal, delete TG

Phase 3c: NLB Internet-Facing [skip if no --peer]

Create TCP target group (port 9000), verify protocol=TCP
Register 2 targets
create-load-balancer (type=network), verify type, ARN /net/
create-listener (TCP:9000)
Wait for NLB active (agent heartbeat)
Verify ENI has public IP
Wait for targets healthy (TCP health check)
Host TCP traffic test (nc 20x, verify round-robin)
Peer validation (nc from --peer node)
Deregister target, verify draining state
Cleanup: delete listener, LB, verify ENI removal, delete TG

Phase 3d: NLB Internal

Create TCP target group (port 9000), register 2 targets
create-load-balancer (type=network, scheme=internal)
Verify type=network, scheme=internal, ARN /net/
Verify ENI has NO public IP, DNS has internal- prefix
create-listener (TCP:9000)
Wait for NLB active (mgmt NIC heartbeat)
Wait for targets healthy (TCP health check)
Launch client VM (cloud-init nc's NLB private IP, serves results)
Poll client status.txt, fetch results, verify TCP responses
Terminate client VM
Cleanup: delete LB, verify ENI removal, delete TG

Phase 4: Cleanup (trap)

Terminate app instances, wait for terminated
Delete key pair, subnet, IGW, VPC

DDIL (`tests/e2e/ddil/`)

Go-based harness and scenarios that exercise the Denied / Disrupted / Intermittent / Limited failure modes called out in the DDIL vulnerability assessment. Design and rationale live in docs/development/improvements/ddil-e2e-test-harness.md; the Go helper API lives in tests/e2e/ddil/harness/ with the authoritative link profile values in harness/profiles.go.

Scenarios begin as t.Skip("requires <dep>") and are flipped to real assertions by the hardening epics they cover (daemon-local-autonomy, predastore-ddil-hardening). The status column below is the authoritative live state — update it in the same PR that changes the test body. TestCoverageDrift enforces the letter column against the TestScenario<L>_... functions in ddil/scenarios/.

Scenarios

Letter	Scenario	DDIL finding closed	Status
A	NATS-only kill (daemon stays up, standalone mode)	Finding 1	ENABLED
B	Daemon restart without NATS (recovers from local state)	Finding 1	ENABLED
C	Clean cluster-network partition (majority + isolated both progress)	Finding 1/2	SKIPPED (needs predastore §2 — see notes)
D	Degraded link under SATCOM profile (fan-out + Raft)	Finding 3	SKIPPED
E	Predastore write under partition (repair journal drains on heal)	Finding 2	SKIPPED
F	Raft under SATCOM latency (≤1 election over 5 min)	Finding 3	SKIPPED

Scenario C quarantine note: the witness VM's block I/O is backed by distributed predastore, so an iptables peer-partition (PartitionNode's OUTPUT DROP rules) also severs the witness disk writes. The result is BLOCK_IO_ERROR → QEMU power-down → daemon termination → ssh handshake EOF on the next AssertProgressed. Needs predastore-ddil-hardening §2 (write-quorum with deferred parity repair) before C can assert without flapping; flip the row back to ENABLED in the same PR that lands §2.

Link profile validation

Values in harness/profiles.go are unvalidated approximations derived from public specifications — real-hardware measurement is deferred until SATCOM/HF access is available. Update the last column when a measurement lands.

Profile	Specification source	Validation status	Last measured against real hardware
LAN	baseline	unvalidated	—
WAN	public site-to-site norms	unvalidated	—
LTEDegraded	public congested-cellular norms	unvalidated	—
SATCOM	published GEO SATCOM figures	unvalidated	—
HFData	HF modem specifications	unvalidated	—
Flapping	synthetic (10s up / 5s down)	unvalidated	—

Known gaps

Security group enforcement breadth — Phase 5f covers egress authorize/revoke for the default SG (dev_networking single-node); Phase 8e covers SG-to-SG ingress + revoke under external pool mode. Not yet covered: ModifyInstanceAttribute --groups swap (verify port group membership changes mid-flight), reconciler convergence after a vpcd restart with drift (kill vpcd, delete an OVN port group, restart vpcd, assert it converges), and per-port-range allow/deny combinations on a single SG. Follow-up: mulga-js-79 (in progress).
Chaos framework (randomised fault injection, long-running stress) — follow-up plan TBD.
Multi-day partition — requires long-running CI infrastructure; follow-up plan TBD.
Real-hardware SATCOM/HF validation — follow-up plan TBD.
Clock skew injection — no hardening epic claims Finding 7 yet; revisit when one lands.
Dual-NIC cluster network — Scenario C uses peer-IP iptables rules on the single-NIC tofu-cluster today; dedicated cluster NIC is deferred to docs/development/improvements/e2e-multinode-dual-nic.md (TBC).
Dockerised per-node pseudo-multinode isolation — candidate under docs/development/improvements/docker-e2e-local.md.
Full Go migration of the rest of the E2E suite — TBD follow-up to the DDIL harness pilot.

39 KiB Raw Blame History Unescape Escape

E2E Test Coverage

Single-Node (run-e2e.sh)

Phase 1: Environment Setup

Phase 1b: Cluster Stats CLI

Phase 2: Discovery & Metadata

Phase 2b: Serial Console Access Settings

Phase 3: SSH Key Management

Phase 4: Image Management

Phase 5: Instance Lifecycle

Phase 5a-pre: Cluster Stats CLI (with running VM)

Phase 5a: Instance Metadata Validation

Phase 5a-ii: SSH Connectivity & Volume Verification

Phase 5a-iii: Console Output

Phase 5 (cont): Root Volume

Phase 5b: Volume Lifecycle

Phase 5b-ii: DescribeVolumeStatus

Phase 5c: Snapshot Lifecycle

Phase 5d: Verify Snapshot-Backed Instance Launch

Phase 5e: CreateImage Lifecycle

Phase 5f: Security Group Enforcement (egress ACL)

Phase 6: Tag Management

Phase 7: Instance State Transitions

Phase 7a: Attach Volume to Stopped Instance (Error Path)

Phase 7b: ModifyInstanceAttribute

Phase 7c: RunInstances with count > 1

Phase 8: Negative / Error Path Tests

IAM Phase 1: User CRUD

IAM Phase 2: Access Key Lifecycle

IAM Phase 3: User Authentication

IAM Phase 4: Policy CRUD

IAM Phase 5: Policy Attachment & Enforcement

IAM Phase 6: Policy Lifecycle — Detach & Delete

IAM Phase 7: IAM Cleanup

Phase 8: EC2 Account Scoping

Step 1: Account Setup

Step 2: Instance Scoping

Step 3: Volume Scoping

Step 4: Key Pair Scoping

Step 5: Snapshot Scoping

Step 6: VPC/Subnet Scoping

Step 7: IGW + EIGW Scoping

Step 8: Account Settings

Step 9: Global Resources

Step 10: Edge Cases

Step 11: Cleanup

Phase 8b: VPC Public/Private Subnet E2E

Phase 8c: Route Table Validation

Phase 8d: NAT Gateway E2E (mulga-763)

Phase 8e: Security Group Enforcement (SG-to-SG datapath)

Phase 9: Terminate and Verify Cleanup

Phase 9a: SSH Unreachable Verification

Phase 9b: Volume Cleanup Verification

Pseudo Multi-Node (run-pseudo-multinode-e2e.sh)

Phase 1: Environment Setup

Phase 2: Cluster Initialization

Phase 3: Cluster Health Verification

Phase 3b: Cluster Stats CLI

Phase 4: Image and Key Setup

Phase 4b: Multi-Node Key Pair Operations

Phase 5: Multi-Node Instance Tests

Test 1: Instance Distribution

Test 1a-ii: SSH Connectivity & Volume Verification

Test 1b: Volume Lifecycle

Test 1c: Snapshot Lifecycle

Test 1c-ii: Verify Snapshot-Backed Instance Launch

Test 1d: Tag Management (Instances)

Test 1d-ii: Tag Management (Volumes)

Test 2: DescribeInstances Aggregation

Test 3: Cross-Node Operations

Test 4: NATS Cluster Health (Post-Operations)

Test 5: VM Crash Recovery

Test 5b: Crash Loop Prevention

Phase 5c: IAM Accounts & Cross-Account Isolation

Step 1: Create Accounts

Step 2: Account Admin Auth

Step 3: Account-Scoped Users

Step 4: Account-Scoped Access Keys

Step 5: Account-Scoped Policies & Enforcement

Step 6: Cross-Account Delete Isolation

39 KiB

Raw Blame History

Single-Node (`run-e2e.sh`)

Pseudo Multi-Node (`run-pseudo-multinode-e2e.sh`)

Real Multi-Node (`run-multinode-e2e.sh`)

Consolidated LB Data Plane (`run-lb-e2e.sh`)

DDIL (`tests/e2e/ddil/`)