applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
Self-hosted runner accumulated 18GB in overlay2 from leaked container
layers. Two causes: build-microvm-image.sh only removed the alpine
container on the success path, and no workflow step pruned docker
between runs.
- scripts/build-microvm-image.sh: container id now removed via the
existing EXIT trap, so failures, signals, and runner kills no longer
leak the writable layer.
- .github/workflows/e2e-go.yml: prune containers / dangling images /
buildx cache before each Build Install Artifacts step, in all three
e2e jobs (single, multinode, reboot).
TestClusterStatsCLI asserts baseline cluster state ("0/" CPU used, "No VMs found") which doesn't hold once the singleton VM is up. Moving it into bucket #1 parallel (siv-128) ran it AFTER all sequential VM launches, so the spx top nodes output reported 4/12 CPU + 1 VM and the assertion failed.
Move it back into the sequential block, ahead of TestInstanceLaunch, so the baseline assertion sees the cluster pre-launch.
* refactor: switch to aws sdk for sigv4 via predastore/auth
* refactor: properly verify against gateway region
* fix(gateway): revert gw.Region binding, surface SigMismatchError details
The gw.Region pin from 65c8444b causes every signed EC2/IAM/etc. request
to fail verification in CI: "auth: signature mismatch" on every health
probe, daemon never reports ready. Restoring sig.Region (pre-refactor
behaviour) unblocks the path while we identify whether nodeConfig.Region
diverges from the AWS CLI profile region, or whether the SDK signer's
canonical-request building differs from the client's.
Add SigMismatchError unwrap to the auth-failure log so operators see the
expected vs provided Authorization headers and can pinpoint the
divergence (credential scope, signed headers, signature) instead of just
the generic mismatch sentinel.
The supportedServices pre-Verify allowlist check stays — it's a
structural gate that doesn't depend on cross-config consistency.
* fix(gateway): re-enable gw.Region binding in SigV4 Verify
With predastore ab6ec8b stripping transport-added headers before
re-signing, the SigV4 verification path is no longer over-sensitive
to wire-vs-signed header divergence. The earlier "auth: signature
mismatch" CI failure was solely from Accept-Encoding/Content-Length
inflation of SignedHeaders — not from a region mismatch — so gw.Region
can be reinstated safely.
The diagnostic SigMismatchError unwrap stays as a permanent
operability improvement; future auth failures will still surface
expected vs provided Authorization headers in the warn log.
* fix: change predastore node ips to 127.0.0.1 (#244)
---------
Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* refactor(e2e/multinode): flatten phase 11 + parallelize bucket #1 + drop phase numbers
Split monolithic phase11_PlacementAndNATGateway (~430 lines, 8 inline
sub-steps) into 6 top-level Tests, each owning its own VPC graph via
t.Cleanup and layered setup helpers (spreadSetupVPC, spreadSetupBastion,
spreadSetupPrivateTrio, spreadSetupNATGW). The cleanup-ordering test
flips *bool latches as each explicit teardown step succeeds so the
LIFO unwind is a no-op on the success path.
Add t.Parallel() to read-only / independent Tests: ClusterHealth,
InstanceDistribution, GuestSSH, VPCNetworking. Deviates from the
mulga-siv-127 bead's bucket-#1 listing by keeping CrossNodeGateway
sequential (asserts a snapshot equality across gateways that the
parallel VPC test would break by churning instance count mid-assert)
and CrossNodeOps sequential (stops/starts trio[0], would race
GuestSSH). Both deviations documented in tests_test.go + plan doc.
Drop the bash phase numbering (phaseN_X funcs + "Phase N" log strings)
from the rest of the suite — 10 funcs renamed to runX. Phase numbers
were a port-time crutch; the Test names now describe each scenario and
phase numbers ageing into noise as bash is retired in Stage H.
* refactor(e2e/multinode): collapse spread Tests + control execution order
Two changes wedged together since they touch the same files:
1. Collapse the 6 top-level TestMultinodeSpread* Tests into a single
TestMultinodeSpread with 6 t.Run sub-tests sharing one VPC + bastion +
private trio + NAT GW setup chain. The 6-test split paid setup cost
6× — bastion launch + private trio launch + NAT GW provision per Test
pushed phase-11 wall-clock near 60min vs ~15min monolithic. Shared-
setup layout keeps JUnit granularity (one row per sub-test) while
restoring the monolithic budget. Setup helpers register t.Cleanup
against the outer test so teardown runs once at the end (LIFO:
NAT GW → trio → bastion → VPC).
2. Move the TestMultinodeSpread wrapper into tests_test.go so file-source
order controls execution position. Previously the func lived in
placement_nat_test.go which sorts before preflight_test.go and
tests_test.go alphabetically — Spread ran FIRST in the sequential
bucket, defeating Preflight's fail-fast on a sick cluster. Wrapper now
runs after NodeRecovery so the cluster has been fully degrade-tested
before Spread's 4-VM + NAT GW + custom-VPC graph spins up.
* feat(e2e/harness): multinode cluster wait helpers + AWSClientForGateway (Stage A)
Stage A of mulga-siv-90. Compile-only; no tests wired yet.
multinode_cluster.go:
- Cluster.WaitNATSPeers polls /routez per node, dedupes by remote_id.
want=2 on healthy 3-node, =1 after a node-down event.
- Cluster.WaitPredastoreHealthy / WaitGatewayHealthy probe HTTPS per node
via insecureHTTPClient (multinode certs are self-signed across peers).
- Cluster.WaitDaemonReady runs DescribeInstanceTypes through every
node's gateway, not just env.ServiceIPs[0] — catches a daemon that's
only registered on one node.
multinode_aws_via.go:
- AWSClientForGateway clones *Env, swaps ServiceIPs to a single node's
Addr, returns a fresh AWSClient. Lets phases 6/7 issue describe/
stop/start across every node in turn (bash aws_via equivalent).
Init/Join helpers deferred to the pseudo-multinode bead (mulga-siv-91)
where they belong — real-multinode bootstrap is owned by tofu +
bootstrap-install.sh before the Go suite starts.
* feat(e2e/multinode): TestMultinodePreflight + TestMultinodeClusterHealth (Stage B)
Stage B of mulga-siv-90. First runnable Test* against a 3-node cluster.
multinode/main_test.go:
- Package-scoped singleton fixture, mirrors single/main_test.go.
- requireMultiNodeFixture skips when SPINIFEX_E2E unset or mode != multinode.
- Fixture = {Env, AWS, Harness, Cluster, Artifacts, TmpDir} — adds Cluster
to the single-node shape via harness.ClusterFromEnv (reads SPINIFEX_NODES
+ SPINIFEX_SSH_USER/KEY).
- TestMain drains harness.Fixture.Close + removes TmpDir at process exit.
multinode/helpers_test.go:
- needAZ / needInstanceTypeArch / needAMI shims (memoized via harness).
- readyNodeCount substring counter — matches bash `grep -c "Ready"`.
multinode/tests_test.go:
- Top-level Test* wrappers (TestMultinodePreflight, TestMultinodeClusterHealth).
multinode/preflight_test.go (phase 1):
- /dev/kvm writable check (no /dev/kvm = test environment is wrong, fail fast).
- peer_ssh hostname on every remote node. Skips node[0] to match bash —
the runner's loopback SSHd is not part of the contract.
multinode/cluster_health_test.go (phase 2):
- Cluster.WaitNATSPeers >= 2 per node.
- Cluster.WaitPredastoreHealthy / WaitGatewayHealthy across nodes.
- Cluster.WaitDaemonReady (DescribeInstanceTypes per gateway).
- spx get nodes shows N Ready, where N = cluster size.
- spx get vms returns (length only — content may be empty).
* feat(e2e/multinode): TestMultinodeInstanceDistribution + GuestSSH (Stage C)
Stage C of mulga-siv-90. Phase 3 (distribution) + phase 4 (guest SSH).
harness/multinode_distribution.go:
- InstanceHostingNode: SSH each peer, ps grep qemu-system | <id>.
Reuses NewPeerSSH; returns nil on no match (caller decides fatality).
- CountInstancesPerNode + AssertSpreadAcrossNodes: fan-out + minNodes
gate. Bash phase 3 warns-on-stacking; Go matches by accepting
minNodes=1 + caller-side log.
harness/multinode_guest_ssh.go:
- GuestSSHPort polls hosting node's qemu cmdline for hostfwd=tcp:...
reusing the existing hostfwdRE from ec2helpers.go (single regex, two
capture groups: host + port).
- GuestSSHEndpoint prefers PublicIpAddress, falls back to hosting-node
+ hostfwd. Mirrors the phase 4 conditional.
- GuestSSHReady polls `echo ready` via SSH with bash-equivalent -o flags.
multinode/helpers_test.go:
- needKeyPair (EnsureKeyPair shim).
- needInstanceTrio: pkg-level sync.Once launches 3 nano instances on
default VPC, registers terminate-on-process-exit. Phase 4+ reuse the
same trio without re-launch.
multinode/distribution_test.go (phase 3):
- launch trio + assert >= 1 node hosting (log warn on stacking).
- spx get vms contains every trio ID.
multinode/guest_ssh_test.go (phase 4):
- For each trio instance: resolve endpoint, poll SSH ready, assert id
reports ec2-user, log lsblk line count.
multinode/tests_test.go: TestMultinodeInstanceDistribution + TestMultinodeGuestSSH.
* feat(e2e/multinode): TestMultinode{Volume,CrossNodeGateway,CrossNodeOps} (Stage D)
Stage D of mulga-siv-90. Phases 5/6/7 — pure AWS API exercise (no
node-state churn).
volume_test.go (phase 5):
- Create 10GiB volume in discovered AZ, attach to trio[0] at /dev/sdf,
detach, delete. Inline lifecycle — DeleteVolume in-test would race a
registered Ensure* cleanup, surface as InvalidVolume.NotFound.
gateway_test.go (phase 6):
- Baseline DescribeInstances via the local gateway, then fan out
AWSClientForGateway per non-local node. Require.Equal across all —
cross-node fan-out regression surfaces immediately.
crossnode_ops_test.go (phase 7):
- Find trio[0]'s hosting node, stop via a different gateway, start via
a third (or same other on 2-node fallback). Returns to running so
later Test*s (phase 4 SSH, etc.) don't cascade-fail. otherTwoGateways
mirrors the bash fallback chain.
* feat(e2e/multinode): TestMultinodeNode{Failure,Recovery} (Stage E)
Stage E of mulga-siv-90. Phase 8 (failure) + phase 9 (recovery) +
diagnostic primitives.
harness/multinode_node_lifecycle.go:
- StopNode SSHes `sudo systemctl stop spinifex.target`. Non-fatal on
exit error (bash matches — cluster shutdown races the SSH conn).
- StartNode mirrors with sudo systemctl start.
- WaitNodeServiceReady polls a single node's HTTPS gateway. Distinct
from cluster-wide WaitGatewayHealthy so a recovering node can be
tracked without re-checking peers.
harness/multinode_diag.go:
- DumpAllNodeLogs fans out journalctl -u 'spinifex-*' per node into
artifactsDir/journal-<node>.log. Tolerates per-node SSH failure.
- SpxGetNodesAcrossCluster strips trailing whitespace off SpxGetNodes.
multinode/node_failure_test.go (phase 8):
- Stop node2, register t.Cleanup restart safety net.
- WaitNATSPeers want=1 (degraded), DescribeInstanceTypes via node1+node3,
DescribeInstances still returns the trio.
multinode/node_recovery_test.go (phase 9):
- StartNode node2 (idempotent — phase 9 in isolation still works).
- WaitNATSPeers want=2, WaitNodeServiceReady, spx get nodes 3 Ready,
DescribeInstanceTypes via the recovered gateway.
* feat(e2e/multinode): TestMultinodeVPCNetworking (Stage F)
Ports phase 10 from run-multinode-e2e.sh. Stands up a fresh non-default
VPC + subnet (10.200.0.0/16 + 10.200.1.0/24), launches 3 instances into
the subnet, then asserts:
- PrivateIpAddress is non-empty and matches 10.200.1.0/24
- All 3 IPs are unique
- SubnetId / VpcId on the instance match the freshly-created ids
- IPs persist across stop -> start cycles
Independent of the package-level trio (which uses the default VPC) —
this phase needs full control over subnet allocation so resources are
created and cleaned up inline via t.Cleanup (LIFO: instances ->
subnet -> VPC).
* feat(e2e/multinode): TestMultinodePlacementAndNAT (Stage G)
Ports phase 11 from run-multinode-e2e.sh — the largest single phase
(~340 bash lines covering spread placement + NAT Gateway egress). Eight
sub-steps preserved with matching harness.Step markers so the Go run
cross-references cleanly with the bash log:
Step 1: VPC + pub/priv subnets + IGW + main RT route + 2 SGs
(bastion: tcp/22 anywhere; private: tcp/22 from bastion-sg,
icmp from VPC)
Step 2: bastion launch, wait SSH, scp key
Step 3: spread placement group + 3 private instances via the group
Step 4: validate spread (>=2 unique hosting nodes; >=3 = full PASS)
Step 5: pre-NAT baseline — confirm NO internet from private VMs
Step 6: allocate EIP + create NAT GW + private RT + 0.0.0.0/0 -> NGW
Step 7: post-NAT — every private VM reaches 8.8.8.8 via NAT GW
Step 8: inline teardown of NAT GW + private RT + EIP
Each resource is owned by the test (no Ensure* memoization) so a failure
mid-flight tears the full VPC graph down via t.Cleanup in LIFO order,
matching the bash teardown sequence. NAT GW state polling is duplicated
locally (waitForNATGatewayStateFatal/Best); the equivalents in
single/natgw_test.go are unexported. Promote to harness/poll.go once a
third consumer lands.
* ci(e2e-go): add multi-node Go E2E job + cert/lb on 3-node cluster (Stage H)
Adds e2e_go_multinode as a parallel job alongside e2e_go_single on the
e2e-go workflow. The single-node job is untouched apart from extending
its SUITES filter to drop the multinode suite (which only runs on the
ci-multi cluster).
What the new job does:
- Provisions a 3-node tofu cluster via the same build-install-artifacts
+ bootstrap-install path as e2e_real_multi (bash).
- Builds multinode.test / cert.test / lb.test on the runner, ships
them to node1.
- Runs each suite with SPINIFEX_MODE=multinode and the cluster csv
in SPINIFEX_NODES / SPINIFEX_NODE_IPS. cert / lb branch internally
on env.Mode so the same binaries cover both single + multi.
- Pulls per-node journals (spinifex-journal-node1..N.log) for the
e2e-analyze action so a node2/3 failure isn't blind.
- Same JUnit + analyze + artifact-upload + status pipeline as the
single-node job.
Other moves:
- Default SUITES now includes e2e-multinode so push triggers fan
out across single + multi automatically. e2e_go_reboot stays
explicit-opt-in (unchanged).
- tests/e2e/Makefile: e2e-multinode now sets MODE=multinode so
`make e2e-multinode` doesn't fall back to single (which would
skip every TestMultinode* on the SPINIFEX_MODE check).
- tests/e2e/TEST_COVERAGE.md: cross-reference the Go port at the
head of the Real Multi-Node section. The bash driver stays in
place as executable spec until the Go run is soak-tested.
- e2e_go_status now needs e2e_go_multinode; multi=skipped counts
as success so a single-only dispatch still posts green.
* fix(e2e/multinode): NATS routez via peer_ssh + StartNode timeout + spx best-effort
CI run 26161792146 surfaced three regressions in the freshly-landed
multinode suite:
1. WaitNATSPeers dialled node.Addr:8222 directly. NATS monitor binds
127.0.0.1:8222 only (cmd/spinifex/cmd/templates/nats.conf:65), so
every routez poll returned "connection refused". Switched to
peer_ssh + curl http://127.0.0.1:8222/routez per node — matches
the bash run-multinode-e2e.sh phase 2 pattern.
2. StartNode / StopNode capped ssh at a 30s context. spinifex.target
has 6+ dependent units (predastore, NATS, awsgw, vpcd, daemon, ui)
and a clean systemctl start runs 60-90s; the 30s budget was
killing ssh mid-command ("signal: killed"). Bump StartNode to 5min
and StopNode to 3min.
3. SpxGetVMs is best-effort in bash (`spx get vms --timeout 5s
2>/dev/null` with no exit-code check) because the CLI's NATS dial
can race the cluster join shortly after bootstrap. The Go port
fatal'd on non-zero exit and broke TestMultinodeClusterHealth +
TestMultinodeInstanceDistribution downstream. Added
harness.SpxRunBestEffort and rewired both call sites.
* fix(e2e/multinode): spx CLI lenience + trio capacity retry + diag
Run 26163994815 cascaded from two issues that bash silently tolerated.
The Go port hard-failed where bash counts the failure and continues, so
3+ tests cascaded off transient first-failures.
- spx get nodes: cluster_health + node_recovery now downgrade <Ready
count to WARN with SpxRunBestEffort, matching bash phase 2
fail_test counter and phase 9 `|| echo ""`. Gateway+daemon health
checks above already prove end-to-end NATS routing.
- needInstanceTrio: retry RunInstances up to 6× on
InsufficientInstanceCapacity (10s backoff). Daemon resourceMgr can
briefly report 0 capacity right after node bootstrap before its
first inventory scan completes — AWS itself surfaces this
transiently. Stagger 2s between launches to match bash phase 3.
- CI workflow: new "Pre-flight diagnostics on node1" step dumps
groups/perms/spx output/systemctl status/journal tail to
preflight-diag.log so next run's artifacts surface what tf-user
sees before the test binary starts.
* fix(e2e): strip SPINIFEX_* from spx child env
Root cause of `spx get nodes` returning "nats: no servers available for
connection" in the multinode Go suite (mulga-siv-90 runs 26163994815,
26195522455): spx CLI calls viper.AutomaticEnv() with prefix "SPINIFEX"
inside config.LoadConfig. The CI workflow exports SPINIFEX_NODES (csv
of cluster WAN IPs) so harness.ClusterFromEnv can build the *Cluster.
viper picks SPINIFEX_NODES up as the `nodes` config key and silently
clobbers the loaded `[nodes.*]` map → nodeConfig.NATS.Host = "" →
nats.Connect("") → ErrNoServers.
Bash never set SPINIFEX_NODES so the bash port was unaffected. Direct
ssh `spx get nodes` from the runner also worked because the env was
clean. Only the test-binary-spawned spx hit the collision.
Fix: SpxRun / SpxRunBestEffort build a sanitized child env that
preserves the small SPINIFEX_* set spx's root command explicitly
BindEnvs (config path, AWS keys, host, base-dir, NATS dial overrides)
and drops everything else.
* fix(vpcd): make handleAddNAT idempotent (mulga-siv-124)
PR #232 (commit 5887927) added a second vpc.add-nat publish in
daemon/vm_adapters.go:onInstanceUpHook to cover instance recovery.
The publish fires on every instance up event including fresh launches,
racing the original publish from handlers/ec2/instance/service_impl.go.
Both publishes are queue-grouped to "vpcd-workers" so two cluster vpcd
instances handle them ~13s apart. handleAddNAT's existing
DeleteAllNATsByExternalIP -> AddNAT sequence is not idempotent under
concurrent writers: the second invocation deletes the rule the first
installed (after ovn-northd compiled SB flows and chassis installed
them), creating a flow-install gap during which the EIP is unreachable.
Symptom: TestMultinodePlacementAndNAT fails with ssh: Connection timed
out to the bastion EIP for 150s (mulga-siv-90 CI runs 26195522455,
26197248656). Bash port last green on e2e-baremetal run 26008057481
before #232 merged.
Fix: before delete+add, look up the existing rule via
FindNATByExternalIP. If it exists and its LogicalIP + spinifex:vpc_id
ExternalID (plus ExternalMAC + LogicalPort under distributed-NAT mode)
already match the target, no-op. Falls through to the existing
delete-then-add for the IP-reuse-across-VPCs case the original code
already handled.
Adds TestTopologyHandler_AddNAT_Idempotent covering the UUID-witness +
flow-barrier counter (two adds with identical state must not recreate
the row or re-arm the barrier).
* test(e2e/multinode): bump bastion SSH wait + ICMP probe
Run 26198221557 confirmed the vpcd dnat_and_snat race (mulga-siv-124)
was fixed — no "cleaned up stale NAT" lines for the bastion EIP, only
one writer + an idempotent skip. But TestMultinodePlacementAndNAT
still timed out at exactly 150s on SSH to the bastion's public IP,
suggesting a separate OVN flow-install lag on the freshly-created
VPC's external switch.
- Bump waitForBastionSSH budget 150s -> 5min. Default-VPC GuestSSH
reaches sshd in ~70s; a brand-new VPC needs longer for ovn-northd
to compile SB + every chassis to install the external ARP
responder + dnat_and_snat flow. 5min covers cloud-init + OVN warm.
- Add an ICMP probe per iteration. Pure diagnostic — ping failure
does not fail the step, but the next iteration's SSH error
carries ping_output so the log splits "datapath down" (ping fail)
from "sshd not up" (ping ok, ssh refused/timeout). Matches the
RCA agent's fix candidate 3.
* test(e2e/multinode): dump cluster datapath state on phase 11 failure
waitForBastionSSH currently fails opaquely when the EIP datapath is
broken — the log shows "ping=fail" but doesn't pin which node owns the
gw chassis or whether ARP / OVN NAT rules / chassis encap IPs are sane.
Register a t.Cleanup hook (after bastionPubIP is known) that fans out
arp / ip route / ovs-vsctl / ovn-sbctl / ovn-nbctl probes to every
cluster node and writes results to <Artifacts>/diag-<node>-<probe>.log
on failure only. Plus a local-side ping / traceroute / ip route block
so a future post-mortem can distinguish runner-network breakage from
cluster datapath breakage.
Best-effort: per-node SSH errors log but never re-fail the test.
* test(e2e/multinode): use IpPermissions form for SG authorize calls
Two AuthorizeSecurityGroupIngress calls in phase 11 used the legacy flat
form (top-level IpProtocol/FromPort/ToPort/CidrIp) instead of the
IpPermissions array. The Spinifex VPC handler at
spinifex/handlers/ec2/vpc/security_group.go:667 reads only
input.IpPermissions and silently drops the flat fields — newRules=0,
no error returned.
Run 26200626265 journal:
groupId=sg-922de57bc74b85c9c (natgw-bastion) newRules=0
groupId=sg-758b05b4a5e7161fd (natgw-private) newRules=1
Bastion SG had zero ingress rules; TCP SYN to bastion :22 was dropped
by ACL, causing waitForBastionSSH to time out after 5min.
Bash version passes because AWS CLI v2 normalizes --protocol/--port/--cidr
shorthand into IpPermissions.N.* on the wire.
Fix: re-shape both calls (bastion tcp/22, private icmp) to use
IpPermissions, matching the third authorize call in the same function
(line 163-175) that already works.
Server-side fix to honor flat form follows in a separate bead.
joshsiv-mulga