* feat: join nodegroup workers via cluster NLB endpoint
Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.
Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.
mulga-siv-298
* feat: gate nodegroup ACTIVE on workers registering Ready
launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.
Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.
mulga-siv-299
* feat(eks): private endpoint access via cross-account Set A ENI
Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.
Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.
Refs: mulga-siv-301
* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress
Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.
Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).
Refs: mulga-siv-321
* fix(vpc): reap the main route table on DeleteVpc
CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.
Refs: mulga-siv-322
* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS
Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').
Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.
* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)
Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
with an ACM-imported self-signed certificate
Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.
* fix: idempotent Delete* across VPC family + volume (RLC1)
Delete handlers now return empty success when the resource is already
absent (nats.ErrKeyNotFound), instead of *NotFound. Transient read
errors still map to ServerInternal. Covers DeleteVpc, DeleteSubnet,
DeleteSecurityGroup, DeleteNetworkInterface, DeleteInternetGateway,
DeleteNatGateway, DeleteRouteTable and DeleteVolume.
Makes destroy retries converge instead of failing mid-graph, removing a
source of orphaned/zombie resources between deploys. Adds a
TestRLC1_*DeleteIdempotentOnAbsent invariant per package and updates the
daemon delegation table; supersedes the old absent->NotFound tests.
* feat: prune orphan ENI LSPs in reconcile applyPorts (ADR-0003 §4)
applyPorts was create-or-update only, so a guest LSP whose
spinifex:eni_id no longer maps to an intent ENI survived instance
terminate and VM-host reinstall — the orphan-LSP capacity-drift root
cause. It now takes pruneOrphans (mirroring applySGs): on a full
reconcile it lists LSPs and tears down each managed ENI port absent from
intent via the composed topology.DeletePort cascade (clears port-group
memberships, then deletes the LSP). The apply-only startup path does not
prune, preserving in-flight ports until subscribers converge.
Adds ListLogicalSwitchPorts to the OVN client (live + mock). Invariant
TestRLC5_OrphanLSPPrunedAfterReconcile guards ADR-0003 §4; a startup-race
test guards the apply-only path.
* feat: idempotent + tracked instance terminate (ADR-0003 §1/§2)
Make terminate idempotent across all three layers and break the
un-terminable-ENI deadlock, and record per-dependency teardown so the
GC backstop can later tell a fully-reaped instance from a dangling one.
- vm.Manager.Terminate: absent / already-terminated / shutting-down now
return nil instead of NotFound / invalid-transition (ADR-0003 §2).
- VM.Teardown map + teardown.go marks: sync steps stamped Done/Failed via
cleaner error returns; fire-and-forget OVN LSP and NAT rule stamped
Pending for the reconciler / GC reaper (ADR-0003 §1).
- InstanceCleaner: DeleteVolumes / ReleasePublicIP / DetachAndDeleteENI /
RemoveFromPlacementGroup / ReleaseGPU now return error.
- vpc ForceDeleteInstanceENI bypasses the in-use guard for the owning
instance, breaking the failed-detach deadlock; plain delete still
rejects an in-use ENI held by another instance.
- gateway TerminateInstances: KV-health-gated idempotent absence (a failed
terminated-bucket query does not fabricate success, ADR-0003 §3).
- Invariants: TestRLC1_TerminateIdempotentOnAbsent, TestRLC4 teardown-map
stamping + failed-dependency, TestRLC_ENINonDeadlock.
* feat: reconciler GC backstop framework + terminated-teardown reaper (ADR-0003 §3)
Add the shared reality->desired garbage-collection framework (inverse of
Restore) and its first reaper, closing the gap 295.3 deferred: nothing
retried pending/failed teardown or purged the terminated record when its
dependents reached done.
- vm/gc.go: GarbageCollector + Reaper interface (Class/Scope/Sweep).
KV-health gated — holds and alarms when the desired-state cannot be
trusted, never reaps against it. ClusterWide reapers run only on the
elected leader (optional elector); NodeLocal run on every node. Start
sweeps immediately then every GCInterval.
- vm/teardown_reaper.go: TerminatedTeardownReaper (node-local). Scans this
node's terminated records, re-drives each not-done dependent through the
idempotent cleaner, and purges the record once all reach done. Records
already complete on arrival are left to the bucket's 1h TTL.
- StateStore gains DeleteTerminatedInstance (JetStream impl already existed).
- daemon: wires the GC after the pending watchdog, gated on a cheap
JetStream AccountInfo health probe.
- Cluster OVN-LSP arm already lands via the reconciler drift prune (295.2).
- Invariants: TestRLC5_GCRetriesPendingTeardownToDoneThenPurges,
TestRLC5_GCNeverPurgesIncompleteRecord, TestRLC5_GCHoldsWhenKVDegraded,
TestRLC5_GCSkipsOtherNodesRecords.
* feat: VPC-family dependency guards (ADR-0004 rule #3)
DeleteSubnet blocks while a live ENI attachment (a resident instance)
remains; available/orphan ENIs do not pin it. DeleteNetworkInterface
guards on the structured attachment (eniIsLiveAttachment), not a bare
in-use string. DeleteNatGateway blocks while a live route table forwards
to it and disassociates its EIP keeping the allocation (AWS parity);
CreateNatGateway now marks the EIP associated so the disassociate is
real and a second NAT GW cannot reuse the allocation.
Binding invariants: TestRLC3 dependency-guard completeness across the
subnet/ENI/NAT-gateway handlers.
* feat: EBS volume data-safety GC override (ADR-0005)
The shared GC backstop reaps actual-desired for every other class; for
volumes that default is forbidden because they carry customer data. Add
VolumeLeakReaper (node-local, rides the 295.5 GC): it marks a volume left
attached to a definitively-gone instance as orphaned (a non-destructive
tag) and alarms, but NEVER issues a delete. The daemon feeds it only this
node's terminated instances whose volumes-teardown failed, so the shared
predastore scan can never false-mark another node's live-instance volume.
DeleteVolume idempotency + the detach/snapshot guards + the
DeleteOnTermination teardown retry already landed (295.1/295.3/295.5).
Binding invariants: TestRLC5 (GC marks+alarms, never deletes data),
TestRLC3 (detach-before-delete).
* test(elbv2): lock ADR-0002 §3 live-only TG in-use guard (mulga-siv-295.4)
* feat(eks): idempotent DeleteCluster on absent + ADR-0006 lifecycle invariants (mulga-siv-295.8)
DeleteCluster returns empty success when the cluster meta is already swept
(Common Resource Lifecycle Contract #1), so tofu destroy retries converge.
Adds TestRLC1 (idempotent delete), TestRLC2 (billable-before-KV-sweep ordering)
and TestRLC3 (NLB target-group no-orphan, riding ADR-0002/172).
* fix(eks): GC reaper terminates orphan control-plane VM after cluster delete (mulga-siv-294)
A normal DeleteCluster can leave a running control-plane VM behind (terminate
completes async; a daemon restart re-launches it from saved running-state) after
the cluster meta is swept — a billable orphan with no owner. Add EKSBillableReaper
on the 0003 GC backstop: a running managed-by=eks CP VM whose cluster meta is
definitively gone is terminated. The CP ENI now carries the customer-account tag
so a system-account VM can be mapped back to its cluster meta; the reaper never
reaps on uncertainty (ENI gone/untagged or meta present).
* feat: reclaim orphan EKS NLB + NAT-GW EIP in billable reaper
The billable GC reaper terminated an orphan control-plane VM but left the
cluster's NAT-gateway EIP and NLB front-end NAT behind when the meta was
gone. After a successful terminate, run the name/tag-driven DeleteClusterNLB
+ DeleteClusterCPVPC teardown (idempotent, meta-independent) so the billable
NAT-GW EIP and NLB NAT are reclaimed too. ADR-0006 §5.
* fix: persist NAT-GW & route-table tags and reorder CP-VPC teardown to reclaim EKS NAT-GW EIP
CreateNatGateway and CreateRouteTable both dropped input.TagSpecifications,
so the tag-driven DeleteClusterCPVPC teardown could not discover the cluster
NAT gateway or its private route table. The explicit DeleteNatGateway +
ReleaseAddress were skipped and the billable NAT-GW EIP leaked under the
system account on every clean DeleteCluster.
Three fixes:
- CreateNatGateway persists ExtractTags(TagSpecifications, natgateway).
- CreateRouteTable persists ExtractTags(TagSpecifications, route-table).
- DeleteClusterCPVPC tears down route tables before the NAT gateway, so the
rule #3 route-ref guard (DependencyViolation / InvalidAddress.Locked) is
not tripped and the EIP releases.
Adds tag-persistence regression tests for both create paths and a teardown
ordering invariant (TestRLC5). Validated live: NAT-GW EIP released cleanly,
no DependencyViolation/Locked.
* fix: route EKS control-plane VM terminate cross-node + DELETING teardown backstop
Multi-node EKS cluster delete wedged in DELETING and stranded the billable
NAT-GW EIP. Two root causes, both invisible on a single-node box:
Cross-node terminate was never routed. TerminateSystemInstance was node-local,
so terminating a CP VM owned by another node returned NotFound (treated as
idempotent) and the teardown deleted a still-attached ENI
(InvalidNetworkInterface.InUse). That error short-circuited purgeClusterInfra
before the managed CP-VPC teardown that releases the NAT-GW EIP. Split
TerminateSystemInstance into a local body plus a routing wrapper that issues
system.TerminateInstance.<id> to the owning node. Bind that subscription for
every system-managed VM (elbv2 + eks) via the OnInstanceUp funnel, so CP VMs
placed on the coordinator's own node (and recovered after a daemon restart) are
reachable too, not only remotely-launched ones.
No DELETING backstop. A failed synchronous DeleteCluster left the cluster
DELETING with nothing to re-drive it. Add EKSDeletingReaper (vm.Reaper, class
eks-deleting) that re-runs purgeClusterInfra for clusters wedged past a min-age,
gated by the per-cluster reconciler leader lease. Stamp ClusterMeta.DeletingSince
on the transition into DELETING.
TerminateK3sServerVM additionally waits out any residual ENI-InUse with a bounded
budget as defense-in-depth.
Validated live on env19 3-node: first delete-cluster converges in 29s, both
remote CP VMs terminated cross-node, NAT-GW EIP released, backstop never fires.
* fix(network): gate gateway reconcile on real datapath reachability
Post-reboot the external OVN datapath can stay dead while every
control-plane signal is green: the chassisredirect SB binding is claimed
(up=true), NAT rows are present and the host neigh is primed, yet
ovn-controller leaves stale gateway/localnet flows so EIP north-south and
the VM->gateway heartbeat forward nothing. ensureGatewayClaimed only
nudges recompute when the binding is unclaimed, so a claimed-but-dead
datapath is never detected or repaired.
Add ensureGatewayDatapath: after the SB claim converges, probe the
gateway LRP IP (OVN answers ICMP natively, no guest or security-group
dependency), and on a miss force one ovn-controller recompute and
re-probe until a short deadline. Reuses the existing recompute nudge and
no-ops when no verifier is wired or no gateway IP resolves.
* fix: self-heal external uplink after reboot boot-race
Post-reboot, ovs-vswitchd/ovn-controller order after the passive
network.target rather than network-online.target, so they start
concurrently with systemd-networkd. Under a heavy boot the WAN-glue veth
(veth-wan-ovs) can come up admin-down before networkd brings it up,
leaving br-wan with no carrier and the external datapath dead while every
control-plane signal reconciles green.
Broaden the datapath health-gate remediation from recompute-only to
RepairDatapath: re-assert the veth admin state (covers the admin-down
link, which a flow recompute cannot revive) then recompute (covers a
stale gateway ofport). Idempotent; the veth ops are skipped in
physical-uplink mode where the device is absent.
Also capture ip -d link (veth admin/carrier) and ovs-ofctl show br-ext
(ofport) in the reboot diag bundle to disambiguate the two failure shapes
from CI artifacts.
* fix: restore AWS-faithful Delete* NotFound; move idempotency to teardown callers (mulga-siv-306)
RLC1 (f9218982) made EC2/VPC/EBS Delete* handlers return success on an absent
resource, breaking AWS compatibility and the cross-account isolation signal (a
cross-account delete reads as absent -> success instead of *.NotFound).
Restore the AWS contract at the API while keeping lifecycle convergence:
- Handlers return canonical *.NotFound on absent: DeleteVpc, DeleteSubnet,
DeleteSecurityGroup, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable,
DeleteVolume, and the public DeleteNetworkInterface (ForceDeleteInstanceENI
keeps its idempotent force path, ADR-0003 section 2).
- Add awserrors.IsNotFound; destroy orchestration (instance terminate, EKS
CP-VPC teardown, EKS IGW/SG cleanup, k3s rollback, ELBv2 ENI/SG teardown)
tolerates NotFound so re-issued/double-targeted deletes converge.
- Flip the RLC1 unit invariants to assert *.NotFound on absent; revert the
daemon round-trip cases; add IsNotFound + CP-VPC convergence tests.
ELBv2 deletes and TerminateInstances stay idempotent (AWS itself is).
* fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence
TestMultinodeGuestSSH intermittently times out (Connection timed out, not
#355's reset-by-peer) connecting to a fresh trio instance's public IP after
the suite churns instances across all three chassis. The instance is running
and AddEIP has installed the DNAT row, flushed/primed the host neigh, and
barriered the gateway-chassis flow — but none of that proves the
gatewayLRP->guest geneve hop: until the guest ENI's southbound Port_Binding is
up on its hosting chassis the cross-node flow is missing and the DNAT-translated
packet blackholes.
Add ensureGuestPortDatapath, the guest-side analog of ensureGatewayDatapath:
after AddEIP, probe the guest LSP's SB Port_Binding (ovn-sbctl up column); on a
miss force one ovn-controller recompute (binds the port / installs the remote
flow) and re-probe until a short deadline. Recompute, not RepairDatapath: this
is flow/binding convergence, not a WAN-uplink fault. No-op when no verifier is
wired or the EIP carries no guest port. spec.PortName is populated for every
associated EIP regardless of NAT mode.
Extends GatewayClaimVerifier with GuestPortUp and wires the gate into
applyEIPs so it runs in the continuous reconcile loop on every node.
* Revert "fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence"
This reverts commit 728b1a9cec6eaf5c7ef5e9eda4469f239b4d8fc0.
* fix(network): key EIP DNAT delete on external IP, not logical IP
A dnat_and_snat row's identity on the VPC gateway router is its external
IP. The logical (private) IP is not a stable key: the EIP pool round-robins
and instance private IPs are reused as instances recycle, so the same
logical IP is held by different EIPs over time.
DeleteEIP deleted the row by logical IP, so a stale or retried delete for a
freed EIP clobbered whichever live EIP now held that recycled private IP --
the next owner's row vanished with no re-add and its public IP went dark
(SSH SYN never left br-wan: Connection timed out). Lifecycle-resilience
teardown/GC retries amplified this by firing vpc.delete-nat repeatedly.
Key the delete on the external IP via DeleteNATByExternalIP and skip when
no external IP is supplied (no EIP means no dnat_and_snat row). A stale
delete now targets the freed EIP's own external IP (ErrNATNotFound no-op)
and can no longer touch the live row.
* refactor: relax DeleteNatGateway to AWS-faithful
Remove the route-reference dependency guard (checkNatGatewayRouteRefs) so a
NAT gateway deletes even while route tables still forward to it, matching AWS
(the route blackholes). The reconciler's loadNATGWs drops the SNAT once the
record leaves the active bucket, so a forwarding route is not a delete-blocking
dependent. EIP is still disassociated on delete (no leak).
* test: invert NAT-GW route-ref invariant to AWS-faithful
TestRLC3_NatGatewayBlocksWhileRouted asserted DependencyViolation while a
route forwarded to the NAT gateway, which is AWS-infaithful. Replace with
TestRLC3_NatGatewayDeletesWhileRouted: delete must succeed whether or not a
live route still targets it (the route blackholes; the reconciler drops the
SNAT). Per ADR-0004 §2.
* fix: tear down NAT-GW SNAT on delete (restore delete-time fan-out)
The guard commit (53ed55a9) dropped publishDeleteEventsForNatGateway on the
assumption a routed NAT GW could never be deleted. With the route-ref guard
removed, deleting a NAT GW while routes still target it left the SNAT in
place (applyNATGWs is create-only, so the reconciler never prunes it) and
egress kept working — TestNATGateway's post-teardown 'lost internet' check
failed. Restore the fan-out: on delete, scan route tables and publish
vpc.delete-nat-gateway per associated subnet so egress stops, while the
route entry survives as a blackhole (AWS-faithful).
* fix: probe EIP datapath in reboot self-heal, not the gateway LRP IP
The post-reboot gateway datapath self-heal probed the gateway LRP IP for
liveness, which OVN answers natively at the router — so it stayed green even
when the EIP NAT pipeline was stranded (ofproto/trace dropped EIP traffic at
table 0). RepairDatapath never fired and TestRebootResilience timed out, with
the drift loop re-probing the same false-green every cycle.
Prefer probing an associated EIP: force a fresh ARP of the dnat_and_snat
external IP (flush neigh, ping to trigger, read the resolved MAC). OVN answers
ARP for the external IP from the gateway chassis independent of the guest, so a
resolved MAC proves the WAN uplink forwards and the NAT flows are installed —
the signal the LRP IP cannot give. Fall back to the LRP IP for VPCs with no
EIP. The existing drift-loop cadence now re-heals a later flow strand.
* fix: lb-agent heartbeats over mgmt port, not the WAN/EIP datapath
The lb-agent heartbeat to awsgw is control-plane and must survive a host
reboot that strands the OVN/EIP north-south data plane. Customer
(internet-facing) lb-agents heartbeated over the WAN GatewayURL, riding the
EIP datapath, so after a reboot the ALB never reported healthy until the
datapath self-healed. System LBs already heartbeated over the on-link
br-mgmt NIC; this extends that path to every lb-agent.
buildLBAgentEnv now always uses mgmtGatewayURL (on-link MgmtBridgeIP, the
mgmt NIC the lb-agent VM already carries), falling back to the WAN URL only
when no mgmt bridge exists. Drops the account-id branch.
* fix: re-heal guest ENI SB Port_Binding post-reboot so EIP ingress converges
Post-reboot every EIP in a VPC goes dark north-south: the DNAT row, host
neigh and gateway-chassis flow are all in place (ARP to the EIP resolves)
but the gatewayLRP->guest flow is missing until ovn-controller binds the
guest ENI's SB Port_Binding, so the DNAT-translated packet blackholes and
the public IP is unreachable on an otherwise-running instance.
Restore the guest-port convergence gate in applyEIPs: probe the guest LSP's
SB Port_Binding up column; on a miss force an ovn-controller recompute and
re-probe until a deadline. Unlike the earlier reverted form, recompute on
every miss, not once: after a host reboot the guest tap is replumbed several
seconds after the reconcile runs, so a single early nudge fires before the
port exists and never binds it. The window is widened to span guest boot.
This is the southbound analog of the gateway datapath self-heal; the EIP
ARP probe cannot detect it because the gateway chassis answers ARP for the
external IP even while the guest delivery flow is absent.
* fix(network): force post-reboot EIP recompute on guest-port bind, startup-only
The reboot suite passed only by accident: the pre-#351 gateway-claim verifier
polled the always-chassis-less gateway LRP, so it fired an ovn-controller
inc-engine/recompute on every reconcile. That spurious recompute was load-bearing
- it reinstalled the centralised gateway DNAT flows that ovn-controller's
incremental engine misses on a cold (post-reboot) start. #351 correctly switched
the check to the chassisredirect port, eliminating the every-cycle recompute, and
the EIP datapath went dark post-reboot with every control-plane signal green.
Restore the kick, scoped to the post-reboot startup path: applyEIPs now gates a
single recompute on the guest ENI's SB Port_Binding chassis claim - the reliable
signal the tap is back, unlike the up column which stays stale-true across a
reboot. Drift never recomputes, preserving the #351 fix for freshly attached
EIPs. Probe renamed GuestPortUp -> GuestPortBound (chassis, not up).
* Revert "fix(network): force post-reboot EIP recompute on guest-port bind, startup-only"
This reverts commit 792e318211ef03efbe24febeb470749d88c0b03c.
* test(e2e/reboot): capture dataplane diag on ALB serve failure
The post-reboot ALB-serve assertion (8.6) failed with no flow/ARP evidence:
the OVN/OVS dataplane dumps only ran on the 8.5 targets-unhealthy path, but
8.5 passes on the daemon's own health view while the north-south EIP datapath
is dark, so the serve failure (8.6) left nothing to localise the drop.
On a serve failure, capture dumpALBProbe/dumpOVNDataplane/dumpOVNClaimState/
dumpHeartbeatDiagnostics/dumpAppDiagnostics before failing. Add to the OVN
dataplane dump: dump-flows br-ext and fdb/show br-ext (north-south bridge,
previously only br-int), and a real ARP probe that flushes the neigh entry
first so primeReachability's pre-seeded reachable entry can no longer mask an
unanswered ARP. Pure diagnostics; no production change.
* test(e2e/reboot): decisive datapath diag — untruncate br-int, real-ingress ofproto/trace, conntrack + tap capture
* test(e2e/reboot): add south->north return-path ofproto/trace to localize egress drop
* test(e2e/reboot): derive LRP MAC from realized flow + remediation experiment (recompute vs ovn-controller restart)
* test(e2e/reboot): dump persisted SB MAC_Binding + external MACs; flush experiments (conntrack vs MAC_Binding) to isolate recompute-resistant return-path drop
* test(e2e/reboot): untruncate Port_Binding + enumerate DGP/chassisredirect claim state (does cr-gw row exist + resident chassis)
* test(e2e/reboot): dump NB source of return-path drop (lflow stage, NAT external_mac/options, LR policy, LSP port_security)
* fix: exempt public-IP instances from subnet egress drop gate (mulga-siv-311)
Post-reboot reconcile installs the priority-1100 subnet egress drop gate for an
IGW-attached subnet that has no 0.0.0.0/0 route, but nothing punched the subnet's
public-IP instances through it. lr_in_policy runs before lr_out un-DNAT/SNAT, so an
inbound connection's reply still carries its private source when it hits the gate and
is dropped. The ALB/EIP datapath goes dark south->north post-reboot while NAT, ARP,
chassis and guest-port binding all read green; the pre-reboot path survives only
because the event-driven gate never evaluates a subnet created after IGW attach.
Install the existing priority-1200 /32 reroute (above the gate) for every public-IP
instance during reconcile. Reroute only: the instance's dnat_and_snat already supplies
SNAT, unlike EnsureSystemInstanceEgress which also adds a plain snat. Public IPs come
from two disjoint sources -- auto-assigned/ELB addresses on the ENI record and user
EIPs in the EIP bucket -- so drive the exemption from both intent.Ports (new
PortSpec.PublicIP) and intent.EIPs.
* refactor: scope public-IP drop-gate exemption to gated subnets
Only install the priority-1200 reroute when the instance's subnet actually carries
a drop gate. Routed subnets egress via their priority-1000 reroute and need no
exemption, so bounding on gate presence keeps the change inert for IGW-routed and
NATGW-routed subnets (no redundant or overriding policy). Adds a negative test.
* refactor: drop unused RemoveEIPInstanceEgress
No production caller; the exemption is install-only in reconcile. Removed the
interface method and impl rather than leave dead API surface.
* test: dump guest console + OVN/datapath diag on SG-to-SG client SSH timeout
TestSGToSGDatapath waited for client-vm SSH via EventuallyErr and reported
only 'exit status 255' on timeout — no guest-side or datapath state. When the
fresh public-IP VM is unreachable for the full window the failure was
undiagnosable from CI artifacts.
Swap the wait to the non-fatal trySSHReady probe and, on timeout, call
DumpVPCFlowDiagnostics (guest console tail, ovn-nbctl/sbctl show, NAT rows,
br-int flows / conntrack / host ARP keyed on the public + private IP) before
Fatal, mirroring vpc_test/baseline_test. Captures persist to the artifact dir
for the Stage 2 analyzer bundle.
* fix: ownership-scope public-IP teardown to survive IP recycling
A live instance's dnat_and_snat rule, host ARP, and IPAM lease were torn down
by the delayed garbage-collection teardown of a different, long-dead instance
that previously held the same recycled public IP. The 2-min vm/gc
instance-teardown reaper re-runs ReleasePublicIP on a stale terminated record;
because the external IPAM round-robins a small pool, the IP had already been
reassigned to a live instance, and both teardown primitives acted on the IP by
value with no ownership check:
- policy.DeleteEIP deleted the dnat_and_snat row by external IP alone,
discarding the logical IP it receives.
- StaticPoolAllocator.Release freed the pool entry by IP value.
Scope both to the current owner so a stale or duplicated teardown for a
recycled IP is a no-op:
- DeleteEIP looks up the row by external IP and skips the delete (and the ARP
flush) when its logical IP no longer matches, mirroring AddEIP's idempotency
lookup. Defeats both external- and logical-IP reuse races.
- Allocator.Release / ExternalIPAM.ReleaseIP take an ownerENIID; the static
allocator no-ops a release whose ENI differs from the live allocation, and
treats an already-freed owner-scoped release as idempotent. Teardown and
rollback callers pass their ENI; user-driven EIP release passes none.
Proven via E2E single TestSGToSGDatapath (run 27515293563): client-vm SSH
black-holed after its NAT was deleted by the dead instance's GC sweep.
* fix(eks): single-flight DeleteCluster teardown per cluster
SDK retries of one DeleteCluster get fanned across nodes by the worker
queue group, so purgeClusterInfra raced the same cluster on multiple
nodes: duplicate ENI/NLB teardown, duplicate TargetGroupNotFound
deregisters, and a real double-release risk on the billable NAT-GW EIP.
Idempotency kept it correct but noisy.
Add a dedicated per-cluster teardown lease (teardownLeaderKey =
reconcilerLeaderKey + "/teardown"), distinct from the reconciler lease
so a delete never contends with the still-live reconciler that holds the
reconciler lease until it observes DELETING and exits. DeleteCluster
acquires it after the NotFound-idempotency check: the loser returns the
cluster as DELETING (AWS-async semantics) without running the teardown;
the winner drives purgeClusterInfra and releases. The DELETING backstop
reaper shares the same lease, so at most one purgeClusterInfra runs per
cluster cluster-wide.
A nil leaderKV (single-node) skips gating, so local behavior is
unchanged.
* fix: env-driven SSH-ready budget for slow baremetal guests
Add SPINIFEX_SSH_READY_TIMEOUT override for the single-node SSH-handshake
probes (default unchanged at 3m). Baremetal boots q35 guests slower (OVMF +
NBD-backed disks) than CI loop devices and intermittently misses the 3m
window; the harness can now widen it to 5m without affecting shared-runner
e2e. Applied at all single-node probe sites; failure messages now report the
actual budget.
* fix: shared external switch to stop per-VPC localnet ARP collision
Per-VPC external switches each attached an untagged localnet port to the same
physical network (br-ext). OVN cannot keep multiple untagged localnets on one
physical network L2-separate, so with two or more concurrent public-IP VPCs ARP
for a public IP landed on the wrong VPC's switch and the owning gateway router's
ARP responder never fired, leaving the public IP unreachable.
Move to a single shared external switch (provider-network pattern): one switch
with one localnet on network_name=external. Each VPC's gateway LRP attaches via
its own router-type switch port; per-VPC NAT isolation is unchanged. AttachIGW
ensures the shared switch idempotently and is migration-safe; DetachIGW removes
only the per-VPC port + LRP and preserves the shared switch. DeleteVPC removes
the per-VPC gateway switch port from the shared switch so it does not dangle.
Reconcile detects attachment via the gateway LRP since the shared switch carries
no vpc_id.
* fix: owner-scope DeleteEIP by ENI port to survive identical pair recycling
A stale or GC-re-emitted vpc.delete-nat for a terminated instance tore down a
live instance's dnat_and_snat rule (and flushed the host ARP) when the public
IP and private IP both recycled, together, to that live instance in the same
VPC. The (external_ip, logical_ip) pair-key matches under identical recycling,
so the existing logical-IP guard fell through.
Stamp the owning ENI port (spinifex:logical_port) on the dnat_and_snat row at
AddEIP, thread the port through handleDeleteNAT -> DetachEIP -> DeleteEIP, and
skip the delete when the row's stamped port differs from the incoming one. The
IPAM release was already owner-scoped; this mirrors that guard at the NAT layer
so any stale delete for a recycled pair is a no-op. Empty port preserves the
prior pair-key behaviour.
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
CI e2e-build compiles the spinifex module with GOWORK=off, resolving the
go.mod pins instead of the go.work local submodule redirect. v1.7.0 of both
modules predates symbols spinifex now calls (viperblock.StateBody,
predastore auth.ParseRoleARN), so the build failed with undefined-symbol
errors that go.work masked locally. Pin both to current dev-branch
pseudo-versions until a coordinated release is cut.
* feat: enable viperblock at-rest encryption by default on fresh installs
Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which
viperblockd treats as legacy cleartext mode. The encryption machinery existed
but the install path never generated a key or set the field.
Provision a cluster-wide viperblock master key in all three fresh-install
flows and render its path into spinifex.toml:
- admin init (single node): generate the key locally.
- admin init --multi-node (leader): generate once, save locally, and
distribute via the formation server (mirrors IAM master key distribution).
- admin join (joiner): receive the shared key from the leader rather than
minting its own, so volumes sealed on any node open everywhere. Lenient on
absence — an older leader returns no key, so the joiner warns and stays
cleartext instead of failing the join.
The key file is omitted from the template when unset, so upgrades with no
existing key keep running cleartext.
* fix: make viperblock encryption key group-readable for both readers
The key is loaded by two services per node — viperblockd (spinifex-viperblock)
and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it
must be group-readable, not owner-only like the per-node predastore key.
The previous commit generated the key but never registered its path with the
deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root
0700 (non-traversable) and the key as root:root 0600. The daemon failed at
startup with "stat ... permission denied".
Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and
SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key
to root:spinifex 0640.
* fix: read viperblock config.json through StateBody for encrypted volumes
The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly,
which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a
valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages
silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the
whole launch path.
Route every config.json read through viperblock.StateBody, which unwraps the
envelope (no key needed — the metadata is authenticated, not encrypted) and is
a no-op for unencrypted volumes:
- image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig
- volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the
refuse-to-merge-encrypted guard sees the real VBState instead of clobbering
the envelope with a tag-less blob)
- admin: readAMIConfig, readVolumeConfig
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
* fix(eks): surface CREATING block reason at Info, not Debug
A cluster stuck in CREATING logged its per-tick bootstrap/health block
reason at Debug, invisible at the default level, so a create that ran to
the 15m timeout FAILED gave no diagnosable cause. Bump both reasons to
Info; they fire only during the CREATING window.
* feat(iam): round-trip AWS-managed policy ARNs for stock EKS tooling
Stock terraform-aws-eks / eksctl attach AWS-managed policies
(AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy,
...) that Spinifex does not provision. AttachRolePolicy rejected them
NoSuchEntity and ListAttachedRolePolicies dropped them, breaking the
node/cluster role round-trip DescribeNodegroup relies on. Accept and
store managed ARNs opaquely (no backing document), deriving the policy
name from the ARN on read.
* feat(eks): echo cluster tags + store-only Tag/Untag/ListTags
DescribeCluster never returned create-time tags and ClusterMeta did not
store them, so a stock terraform-aws provider reconciling default_tags
saw perpetual drift and issued TagResource on every apply — which itself
returned NotImplemented. Store tags on the cluster record, echo them from
DescribeCluster, and back the three tag ops store-only against the record
(cluster ARNs only) so the default_tags round-trip is clean.
* fix(eks): persist nodegroup workers incrementally + boot reclaim sweep
launchWorkers only wrote the worker IDs to the record after the whole
launch loop, so a daemon crash mid-loop stranded already-launched workers
under a record that still showed empty InstanceIDs — nothing could reclaim
them. Persist each worker before launching the next, and add a boot-time
reclaim sweep (SpawnRegisteredReconcilers) that terminates recorded
workers for any nodegroup left CREATING by a crashed launcher, or
CREATE_FAILED still holding workers, settling it to CREATE_FAILED with
InstanceIDs cleared.
* fix: plumb query params into EKS gateway dispatcher
EKS_Request never passed the query string to per-action handlers, so
UntagResource (DELETE /tags/{arn}?tagKeys=k1&tagKeys=k2) saw an empty
TagKeys and silently no-op'd. Fold url.Values into the JSON body when
the body is empty so the handler unmarshal sees the keys.
* feat(iam): resolve AWS-managed EKS policy grants for authorization
Assumed-role/user authorization (checkPolicy -> Get{Role,User}Policies ->
EvaluateAccess) could not honour AWS-managed policy grants: Spinifex stores
no document for managed ARNs, so stock EKS roles whose permissions come
entirely from managed policies (node role: AmazonEKSWorkerNodePolicy +
AmazonEC2ContainerRegistryReadOnly + AmazonEKS_CNI_Policy) evaluated to an
empty grant set and were denied every call.
Add a builtin registry of well-known AWS-managed policy documents, resolved
via a shared resolveAttachedPolicy helper used by both GetRolePolicies and
GetUserPolicies. Unmodeled managed ARNs resolve to no grant (fail-closed
deny) rather than erroring the whole request; customer-managed ARNs still
fail closed when unresolvable.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
The gateway-claim verifier polled the SB Port_Binding for the distributed
gateway port LRP (gw-<vpc>), which is always chassis-less by OVN design — the
chassis claim lands on the chassisredirect port cr-gw-<vpc>. So
ensureGatewayClaimed always read "unclaimed", fired ovn-appctl
inc-engine/recompute every reconcile cycle, and logged a spurious "gateway SB
chassis claim did not converge" error after its 30s budget. The repeated forced
recomputes rebuilt the entire OVN flow table every cycle, intermittently
darkening freshly attached EIP datapaths and flaking the single-node e2e suite
(observed: every br-int EIP flow at duration~10s with n_packets=0).
Check the chassisredirect Port_Binding instead. The nudge now fires only on a
genuinely unclaimed redirect — the real reboot race #339 targets — rather than
on every healthy reconcile.
- topology: add GatewayChassisRedirectPort(vpcID) -> "cr-gw-<vpc>"
- reconcile/apply: pass the cr- port to ensureGatewayClaimed
- reconcile/host: doc the verifier checks the chassisredirect binding
- test: capture the queried port; assert the cr- port is checked, no nudge
TestMultinodeGuestSSH ran with t.Parallel(), so it resumed immediately
after the sequential bucket and probed trio[0] — which CrossNodeOps
stop/starts but only waits to reach state "running", not for sshd to
answer. The guest was often still booting, producing
kex_exchange_identification resets and 2m timeouts that flaked CI red.
Drop t.Parallel() and declare the test last in the sequential bucket so
it runs after CrossNodeOps + NodeFailure/Recovery + Spread have
restabilised the cluster, giving every trio member time to settle and
removing concurrent VM churn.
CreateLoadBalancer is async: InstanceID is only set when the LB VM's QEMU
process starts, ~600ms after the create burst. The reactive
updateStoredConfig calls from CreateListener/RegisterTargets all no-op
during that window (InstanceID empty), and nothing rebuilds config once
the VM launches. The lb-agent then heartbeats, gets an empty ConfigHash,
never sees it change, and probes zero backends -> e2e 0/2 healthy.
Build the data-plane config on the provisioning->active heartbeat
transition instead: that is the one point where InstanceID is guaranteed
set and the agent is alive to consume config. The InstanceID guard in
updateStoredConfig stays as a cheap no-data-plane skip, no longer the
primary gate.
DHCP-source external-pool leases are DHCPRELEASEd only on explicit
instance/EIP terminate. Manager.Stop() preserves leases for
adopt-on-reboot, so env teardown/reset kills vpcd and destroys the
per-AZ lease KV without releasing — stranding every live lease
(instance ENIs, EIP allocations, per-VPC LRP gateways) on the upstream
DHCP server until TTL. Across CI churn with fresh client-ids the
upstream scope fills and new launches get no public IP.
Add Manager.DrainAll + a vpc.dhcp.drain RPC and a standalone
'spx admin cluster drain-dhcp' command that releases every held lease.
Wired into both teardown paths (single-node teardown role and
multi-node cluster-teardown.yml) before services stop. Stop() is
unchanged, so normal restarts still adopt leases on reboot.
* fix(ec2): fail RunInstances when public-IP auto-assign fails
PrepareRunInstances swallowed an AllocateIP failure for a
MapPublicIpOnLaunch subnet (logged WARN, fell through), so the instance
booted running with no public IP. With no qemu-hostfwd fallback such an
instance is unreachable. Mirror the adjacent AddNAT-failure path: detach
and delete the auto-created ENI, deallocate capacity, drop the instance,
and surface InsufficientAddressCapacity.
* test(daemon): deterministically pick a fitting 2-vCPU type in capacity filter test
CapacityFilterRespectsAllocation selected the first 2-vCPU type by Go
map-iteration order, intermittently picking a type near the
schedulable-memory boundary the allocator then rejected (~3/5 failures in
isolation on small hosts). Select the smallest-memory 2-vCPU type from the
capacity-aware set so it always fits.
CreateLoadBalancer's PutLoadBalancer-failure path only released the name
claim, orphaning the ENIs and managed front-end SG already created. Extract
the ENI+SG+claim teardown into rollbackLBInfra and call it from both the
ENI-create-failure and PutLoadBalancer-failure paths.
CreateListener persisted the listener record before authorizing the NLB
managed-SG port and reloading config; a failure in either left an orphan
listener record (and, after a config failure, a dangling SG authorization).
Add rollbackListener to revoke the port (when already opened) and
cascade-delete the listener on every post-persist failure.
Also wait for the async worker launch in the two CreateNodegroup disk-size
tests so they observe the launch under the record-before-launch contract.
* fix: make platform create handlers async to stop responder starvation
EKS CreateCluster/CreateNodegroup and ELBv2 CreateLoadBalancer ran their
full VM/control-plane launch synchronously inside the NATS subscription
callback on the shared spinifex-workers queue. A multi-minute create
head-of-line blocked the worker, so system.LaunchInstance/TerminateInstance
found no responder within the gateway timeout and failed with
'nats: no responders available' (mulga-siv-267.4).
Each create now runs validation + the atomic name claim synchronously,
snapshots the CREATING/provisioning output, then spawns the slow launch on
a per-impl launchWG and returns immediately. Launch failures mark the
record FAILED/CREATE_FAILED so 267.3 reclaim makes a retry idempotent. The
service interfaces are shared by the gateway NATS clients and the daemon
impls, so the async split lives inside the impl method and leaves the
interface, handleNATSRequest, and all daemon handlers unchanged.
The host-pinned system.LaunchInstance/TerminateInstance responders now
dispatch each request on its own goroutine (systemDispatchWg) so a slow
boot, which must respond after it completes, no longer blocks the subject.
ELBv2 keeps ENI creation synchronous (cheap VPC calls that yield the
AZ/VpcId the create response needs); only the LB-VM boot goes async,
folding InstanceID/IPs back into the record for a later Describe.
* fix: keep node-targeted launch subscription live under capacity pressure
The node-targeted (unicast) system.LaunchInstance.{type}.{node} subject was
torn down whenever the node filled, same as the queue (anycast) subject. For
the queue subject that is correct — NATS reroutes the launch to a node with
room. For the node-targeted subject there is no other node: a committed spread
reservation (EKS HA control-plane leg) that targets the now-unsubscribed node
gets 'nats: no responders available' instead of a capacity decision.
Keep the node-targeted subscription live for any type the node can host;
enforce capacity at launch time via ResourceManager.allocate, which already
rejects an over-capacity launch atomically. A reserved launch always reaches a
responder and, when the slot is still good, succeeds.
launchWorkers built RunInstancesInput with no BlockDeviceMappings, so
parseVolumeParams applied the 4GB root-volume default and the nodegroup
DiskSize was silently ignored. Wire rec.DiskSize into a BlockDeviceMapping
when set.
Run 27254808154 showed TestRebootResilience fail with ovn-controller on
the live chassis claiming NO local Port_Bindings post-reboot: ENIs stuck
on a dead chassis UUID, cr-gw chassis empty, NAT external_mac empty —
while NB intent was correct and taps carried the right iface-id. The SB
Chassis _uuid churned across the reboot (old UUID gone, new one created)
while the SB DB persisted, orphaning every claim.
The current artifacts cannot pin WHY the new chassis won't reclaim: the
ovn-controller vlog lives under /var/log/ovn, not journald, so it is
absent from the systemd-journal capture.
Add:
- phase5Snapshot: capture chassis identity PRE-reboot (system-id, SB
Chassis/Chassis_Private, get-connection) to ovn-pre-chassis.txt as the
churn baseline.
- dumpOVNClaimState: at post-reboot failure, capture the same chassis
state POST-reboot plus ovn-controller connection-status/debug, the
gw/eni Port_Binding claim detail, ovn-central-vs-ovn-controller boot
ordering, and tails of ovn-controller.log / ovn-northd.log →
diag-ovn-claim.txt.
Disambiguates chassis-identity churn from an SB/NB startup-ordering race
on the next reboot-suite run. Diagnostic only; no behavior change.
* feat: atomic create idempotency across EKS, ELBv2 and EC2 ClientToken
Every create handler took a non-atomic Get-check then a blind Put, so a
gateway/NATS retry that spawned a second concurrent handler could pass the
check in both and double-launch + leak VMs (267 failure-mode C).
Make each create claim its identity key atomically before any launching work:
- EKS CreateCluster: claim the cluster meta via kv.Create; a prior FAILED
attempt is reclaimed via a CREATING CAS so exactly one concurrent retry wins.
purgeClusterInfra gains a deleteMeta flag so reclaim purges infra while
keeping its claim key.
- EKS CreateNodegroup: claim the nodegroup record before SG provisioning or
worker launch; duplicates return ResourceInUse without provisioning.
- ELBv2 CreateLoadBalancer: account-scoped name claim (lbname.<acct>.<name>)
taken before ENI/VM work, released on every failure path and in
DeleteLoadBalancer; a name claim whose owner lbID has no live record is a
crashed-create orphan and is reclaimed.
- EC2 RunInstances: ClientToken dedup over a dedicated TTL bucket with a
two-phase claim (in-flight -> done) and reservation replay; a token reused
with different params returns IdempotentParameterMismatch.
Race tests under -race assert exactly one owner and a single launch for
concurrent same-name creates on all paths.
* test: cover ClientToken idempotency; extract orchestration for testability
Extract the claim->replay->launch->finalize/abort flow out of RunInstances
into runInstancesWithClientToken(store, ..., launch func) so the idempotency
path is unit-testable without a live gateway/NATS. RunInstances now just
resolves the store, hashes params, and passes a launch closure.
Make the in-flight poll timeout/step package vars so a duplicate-caller
wait-timeout is deterministically testable.
Adds tests: replay-skips-launch, owner-launches-once-then-replay,
launch-failure-aborts, param-mismatch maps to AWS error, getClientTokenStore
binds once, and in-flight wait-timeout.
After a host reboot the Northbound gateway_chassis intent recovers correctly
(SetGatewayChassis re-asserts it idempotently), but ovn-controller can leave the
Southbound Port_Binding for the gateway router port unclaimed (chassis == []).
With no claimed binding it installs no logical->physical flows for the centralised
gateway redirect, so every floating IP on the VPC is dark: ofproto/trace drops the
FIP ingress at br-int table 0, host->FIP is 100% packet loss, and the ALB lb-agent
heartbeat times out until the 5-min health window expires (TestRebootResilience).
rebindGatewayChassis only asserted NB intent. Now, after SetGatewayChassis, the
reconciler verifies the SB Port_Binding has a non-empty chassis and, if not, nudges
the local ovn-controller to recompute (inc-engine/recompute), polling until the
claim converges or a bounded deadline elapses — logging an error rather than
blocking reconcile. SB read + recompute are exec-only (ovn-sbctl/ovn-appctl) and
live in network/host behind a reconcile.GatewayClaimVerifier interface, wired in
vpcd; reconcile stays NB-pure and mockable. No NB object create/delete (ADR-0006 S5).
Verified by run 27251769281's diag-ovn-flows.txt (SB chassis empty + table-0 drop).
Validation completes on the next reboot-suite nightly (single-node self-heals too
fast to reproduce locally).
* test(reboot): capture heartbeat-path diagnostics on post-reboot health timeout
When post-reboot ALB targets fail to go healthy, the lb-agent's heartbeat to the
gateway times out with 'context deadline exceeded while awaiting headers' — a
symptom that alone cannot distinguish a network-level drop from an app/gateway
stall. Replace the fatal WaitForTargetsHealthy with a non-fatal poll that, on
timeout, tcpdumps br-wan for the heartbeat 5-tuple + ARP for the ALB floating IP
and pulls the lb-agent console + awsgw journal, then fails. Disambiguates the
layer before any fix (mulga-siv-265).
* test(reboot): re-aim heartbeat capture + add OVN dataplane flow-dump
Run 27250238163 proved the post-reboot failure is network-level (all 3 floating
IPs 100% packet-loss from host) but NOT neigh/NAT priming — neigh is primed and
NAT rows are present, only br-int forwarding is dead. The prior capture watched
br-wan for the ALB floating IP, while the real lb-agent heartbeat posts to the
gateway port on the host's advertise interface, so the L4 classifier could never
see the heartbeat and defaulted regardless.
- heartbeat tcpdump now watches the gateway port on `any` (real path) plus a
secondary br-wan floating-IP capture for north-south; classifier keys off
gateway-port L4 presence.
- new dumpOVNDataplane captures ovs-vsctl show, ovs-ofctl dump-flows br-int,
ovn-sbctl chassis + Port_Binding, dnat_and_snat NAT rows, host neigh, and a
host->floating-IP ping + ofproto/trace, to pin the flow-restore step.
- failure path dumps diag-ovn-flows.txt alongside diag-heartbeat.txt.
Test-only, build-tagged e2e; no production code.
Julian Sommer