* feat: enable viperblock at-rest encryption by default on fresh installs
Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which
viperblockd treats as legacy cleartext mode. The encryption machinery existed
but the install path never generated a key or set the field.
Provision a cluster-wide viperblock master key in all three fresh-install
flows and render its path into spinifex.toml:
- admin init (single node): generate the key locally.
- admin init --multi-node (leader): generate once, save locally, and
distribute via the formation server (mirrors IAM master key distribution).
- admin join (joiner): receive the shared key from the leader rather than
minting its own, so volumes sealed on any node open everywhere. Lenient on
absence — an older leader returns no key, so the joiner warns and stays
cleartext instead of failing the join.
The key file is omitted from the template when unset, so upgrades with no
existing key keep running cleartext.
* fix: make viperblock encryption key group-readable for both readers
The key is loaded by two services per node — viperblockd (spinifex-viperblock)
and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it
must be group-readable, not owner-only like the per-node predastore key.
The previous commit generated the key but never registered its path with the
deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root
0700 (non-traversable) and the key as root:root 0600. The daemon failed at
startup with "stat ... permission denied".
Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and
SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key
to root:spinifex 0640.
* fix: read viperblock config.json through StateBody for encrypted volumes
The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly,
which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a
valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages
silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the
whole launch path.
Route every config.json read through viperblock.StateBody, which unwraps the
envelope (no key needed — the metadata is authenticated, not encrypted) and is
a no-op for unencrypted volumes:
- image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig
- volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the
refuse-to-merge-encrypted guard sees the real VBState instead of clobbering
the envelope with a tag-less blob)
- admin: readAMIConfig, readVolumeConfig
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
* fix(eks): surface CREATING block reason at Info, not Debug
A cluster stuck in CREATING logged its per-tick bootstrap/health block
reason at Debug, invisible at the default level, so a create that ran to
the 15m timeout FAILED gave no diagnosable cause. Bump both reasons to
Info; they fire only during the CREATING window.
* feat(iam): round-trip AWS-managed policy ARNs for stock EKS tooling
Stock terraform-aws-eks / eksctl attach AWS-managed policies
(AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy,
...) that Spinifex does not provision. AttachRolePolicy rejected them
NoSuchEntity and ListAttachedRolePolicies dropped them, breaking the
node/cluster role round-trip DescribeNodegroup relies on. Accept and
store managed ARNs opaquely (no backing document), deriving the policy
name from the ARN on read.
* feat(eks): echo cluster tags + store-only Tag/Untag/ListTags
DescribeCluster never returned create-time tags and ClusterMeta did not
store them, so a stock terraform-aws provider reconciling default_tags
saw perpetual drift and issued TagResource on every apply — which itself
returned NotImplemented. Store tags on the cluster record, echo them from
DescribeCluster, and back the three tag ops store-only against the record
(cluster ARNs only) so the default_tags round-trip is clean.
* fix(eks): persist nodegroup workers incrementally + boot reclaim sweep
launchWorkers only wrote the worker IDs to the record after the whole
launch loop, so a daemon crash mid-loop stranded already-launched workers
under a record that still showed empty InstanceIDs — nothing could reclaim
them. Persist each worker before launching the next, and add a boot-time
reclaim sweep (SpawnRegisteredReconcilers) that terminates recorded
workers for any nodegroup left CREATING by a crashed launcher, or
CREATE_FAILED still holding workers, settling it to CREATE_FAILED with
InstanceIDs cleared.
* fix: plumb query params into EKS gateway dispatcher
EKS_Request never passed the query string to per-action handlers, so
UntagResource (DELETE /tags/{arn}?tagKeys=k1&tagKeys=k2) saw an empty
TagKeys and silently no-op'd. Fold url.Values into the JSON body when
the body is empty so the handler unmarshal sees the keys.
* feat(iam): resolve AWS-managed EKS policy grants for authorization
Assumed-role/user authorization (checkPolicy -> Get{Role,User}Policies ->
EvaluateAccess) could not honour AWS-managed policy grants: Spinifex stores
no document for managed ARNs, so stock EKS roles whose permissions come
entirely from managed policies (node role: AmazonEKSWorkerNodePolicy +
AmazonEC2ContainerRegistryReadOnly + AmazonEKS_CNI_Policy) evaluated to an
empty grant set and were denied every call.
Add a builtin registry of well-known AWS-managed policy documents, resolved
via a shared resolveAttachedPolicy helper used by both GetRolePolicies and
GetUserPolicies. Unmodeled managed ARNs resolve to no grant (fail-closed
deny) rather than erroring the whole request; customer-managed ARNs still
fail closed when unresolvable.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
The gateway-claim verifier polled the SB Port_Binding for the distributed
gateway port LRP (gw-<vpc>), which is always chassis-less by OVN design — the
chassis claim lands on the chassisredirect port cr-gw-<vpc>. So
ensureGatewayClaimed always read "unclaimed", fired ovn-appctl
inc-engine/recompute every reconcile cycle, and logged a spurious "gateway SB
chassis claim did not converge" error after its 30s budget. The repeated forced
recomputes rebuilt the entire OVN flow table every cycle, intermittently
darkening freshly attached EIP datapaths and flaking the single-node e2e suite
(observed: every br-int EIP flow at duration~10s with n_packets=0).
Check the chassisredirect Port_Binding instead. The nudge now fires only on a
genuinely unclaimed redirect — the real reboot race #339 targets — rather than
on every healthy reconcile.
- topology: add GatewayChassisRedirectPort(vpcID) -> "cr-gw-<vpc>"
- reconcile/apply: pass the cr- port to ensureGatewayClaimed
- reconcile/host: doc the verifier checks the chassisredirect binding
- test: capture the queried port; assert the cr- port is checked, no nudge
TestMultinodeGuestSSH ran with t.Parallel(), so it resumed immediately
after the sequential bucket and probed trio[0] — which CrossNodeOps
stop/starts but only waits to reach state "running", not for sshd to
answer. The guest was often still booting, producing
kex_exchange_identification resets and 2m timeouts that flaked CI red.
Drop t.Parallel() and declare the test last in the sequential bucket so
it runs after CrossNodeOps + NodeFailure/Recovery + Spread have
restabilised the cluster, giving every trio member time to settle and
removing concurrent VM churn.
CreateLoadBalancer is async: InstanceID is only set when the LB VM's QEMU
process starts, ~600ms after the create burst. The reactive
updateStoredConfig calls from CreateListener/RegisterTargets all no-op
during that window (InstanceID empty), and nothing rebuilds config once
the VM launches. The lb-agent then heartbeats, gets an empty ConfigHash,
never sees it change, and probes zero backends -> e2e 0/2 healthy.
Build the data-plane config on the provisioning->active heartbeat
transition instead: that is the one point where InstanceID is guaranteed
set and the agent is alive to consume config. The InstanceID guard in
updateStoredConfig stays as a cheap no-data-plane skip, no longer the
primary gate.
DHCP-source external-pool leases are DHCPRELEASEd only on explicit
instance/EIP terminate. Manager.Stop() preserves leases for
adopt-on-reboot, so env teardown/reset kills vpcd and destroys the
per-AZ lease KV without releasing — stranding every live lease
(instance ENIs, EIP allocations, per-VPC LRP gateways) on the upstream
DHCP server until TTL. Across CI churn with fresh client-ids the
upstream scope fills and new launches get no public IP.
Add Manager.DrainAll + a vpc.dhcp.drain RPC and a standalone
'spx admin cluster drain-dhcp' command that releases every held lease.
Wired into both teardown paths (single-node teardown role and
multi-node cluster-teardown.yml) before services stop. Stop() is
unchanged, so normal restarts still adopt leases on reboot.
* fix(ec2): fail RunInstances when public-IP auto-assign fails
PrepareRunInstances swallowed an AllocateIP failure for a
MapPublicIpOnLaunch subnet (logged WARN, fell through), so the instance
booted running with no public IP. With no qemu-hostfwd fallback such an
instance is unreachable. Mirror the adjacent AddNAT-failure path: detach
and delete the auto-created ENI, deallocate capacity, drop the instance,
and surface InsufficientAddressCapacity.
* test(daemon): deterministically pick a fitting 2-vCPU type in capacity filter test
CapacityFilterRespectsAllocation selected the first 2-vCPU type by Go
map-iteration order, intermittently picking a type near the
schedulable-memory boundary the allocator then rejected (~3/5 failures in
isolation on small hosts). Select the smallest-memory 2-vCPU type from the
capacity-aware set so it always fits.
CreateLoadBalancer's PutLoadBalancer-failure path only released the name
claim, orphaning the ENIs and managed front-end SG already created. Extract
the ENI+SG+claim teardown into rollbackLBInfra and call it from both the
ENI-create-failure and PutLoadBalancer-failure paths.
CreateListener persisted the listener record before authorizing the NLB
managed-SG port and reloading config; a failure in either left an orphan
listener record (and, after a config failure, a dangling SG authorization).
Add rollbackListener to revoke the port (when already opened) and
cascade-delete the listener on every post-persist failure.
Also wait for the async worker launch in the two CreateNodegroup disk-size
tests so they observe the launch under the record-before-launch contract.
* fix: make platform create handlers async to stop responder starvation
EKS CreateCluster/CreateNodegroup and ELBv2 CreateLoadBalancer ran their
full VM/control-plane launch synchronously inside the NATS subscription
callback on the shared spinifex-workers queue. A multi-minute create
head-of-line blocked the worker, so system.LaunchInstance/TerminateInstance
found no responder within the gateway timeout and failed with
'nats: no responders available' (mulga-siv-267.4).
Each create now runs validation + the atomic name claim synchronously,
snapshots the CREATING/provisioning output, then spawns the slow launch on
a per-impl launchWG and returns immediately. Launch failures mark the
record FAILED/CREATE_FAILED so 267.3 reclaim makes a retry idempotent. The
service interfaces are shared by the gateway NATS clients and the daemon
impls, so the async split lives inside the impl method and leaves the
interface, handleNATSRequest, and all daemon handlers unchanged.
The host-pinned system.LaunchInstance/TerminateInstance responders now
dispatch each request on its own goroutine (systemDispatchWg) so a slow
boot, which must respond after it completes, no longer blocks the subject.
ELBv2 keeps ENI creation synchronous (cheap VPC calls that yield the
AZ/VpcId the create response needs); only the LB-VM boot goes async,
folding InstanceID/IPs back into the record for a later Describe.
* fix: keep node-targeted launch subscription live under capacity pressure
The node-targeted (unicast) system.LaunchInstance.{type}.{node} subject was
torn down whenever the node filled, same as the queue (anycast) subject. For
the queue subject that is correct — NATS reroutes the launch to a node with
room. For the node-targeted subject there is no other node: a committed spread
reservation (EKS HA control-plane leg) that targets the now-unsubscribed node
gets 'nats: no responders available' instead of a capacity decision.
Keep the node-targeted subscription live for any type the node can host;
enforce capacity at launch time via ResourceManager.allocate, which already
rejects an over-capacity launch atomically. A reserved launch always reaches a
responder and, when the slot is still good, succeeds.
launchWorkers built RunInstancesInput with no BlockDeviceMappings, so
parseVolumeParams applied the 4GB root-volume default and the nodegroup
DiskSize was silently ignored. Wire rec.DiskSize into a BlockDeviceMapping
when set.
Run 27254808154 showed TestRebootResilience fail with ovn-controller on
the live chassis claiming NO local Port_Bindings post-reboot: ENIs stuck
on a dead chassis UUID, cr-gw chassis empty, NAT external_mac empty —
while NB intent was correct and taps carried the right iface-id. The SB
Chassis _uuid churned across the reboot (old UUID gone, new one created)
while the SB DB persisted, orphaning every claim.
The current artifacts cannot pin WHY the new chassis won't reclaim: the
ovn-controller vlog lives under /var/log/ovn, not journald, so it is
absent from the systemd-journal capture.
Add:
- phase5Snapshot: capture chassis identity PRE-reboot (system-id, SB
Chassis/Chassis_Private, get-connection) to ovn-pre-chassis.txt as the
churn baseline.
- dumpOVNClaimState: at post-reboot failure, capture the same chassis
state POST-reboot plus ovn-controller connection-status/debug, the
gw/eni Port_Binding claim detail, ovn-central-vs-ovn-controller boot
ordering, and tails of ovn-controller.log / ovn-northd.log →
diag-ovn-claim.txt.
Disambiguates chassis-identity churn from an SB/NB startup-ordering race
on the next reboot-suite run. Diagnostic only; no behavior change.
* feat: atomic create idempotency across EKS, ELBv2 and EC2 ClientToken
Every create handler took a non-atomic Get-check then a blind Put, so a
gateway/NATS retry that spawned a second concurrent handler could pass the
check in both and double-launch + leak VMs (267 failure-mode C).
Make each create claim its identity key atomically before any launching work:
- EKS CreateCluster: claim the cluster meta via kv.Create; a prior FAILED
attempt is reclaimed via a CREATING CAS so exactly one concurrent retry wins.
purgeClusterInfra gains a deleteMeta flag so reclaim purges infra while
keeping its claim key.
- EKS CreateNodegroup: claim the nodegroup record before SG provisioning or
worker launch; duplicates return ResourceInUse without provisioning.
- ELBv2 CreateLoadBalancer: account-scoped name claim (lbname.<acct>.<name>)
taken before ENI/VM work, released on every failure path and in
DeleteLoadBalancer; a name claim whose owner lbID has no live record is a
crashed-create orphan and is reclaimed.
- EC2 RunInstances: ClientToken dedup over a dedicated TTL bucket with a
two-phase claim (in-flight -> done) and reservation replay; a token reused
with different params returns IdempotentParameterMismatch.
Race tests under -race assert exactly one owner and a single launch for
concurrent same-name creates on all paths.
* test: cover ClientToken idempotency; extract orchestration for testability
Extract the claim->replay->launch->finalize/abort flow out of RunInstances
into runInstancesWithClientToken(store, ..., launch func) so the idempotency
path is unit-testable without a live gateway/NATS. RunInstances now just
resolves the store, hashes params, and passes a launch closure.
Make the in-flight poll timeout/step package vars so a duplicate-caller
wait-timeout is deterministically testable.
Adds tests: replay-skips-launch, owner-launches-once-then-replay,
launch-failure-aborts, param-mismatch maps to AWS error, getClientTokenStore
binds once, and in-flight wait-timeout.
After a host reboot the Northbound gateway_chassis intent recovers correctly
(SetGatewayChassis re-asserts it idempotently), but ovn-controller can leave the
Southbound Port_Binding for the gateway router port unclaimed (chassis == []).
With no claimed binding it installs no logical->physical flows for the centralised
gateway redirect, so every floating IP on the VPC is dark: ofproto/trace drops the
FIP ingress at br-int table 0, host->FIP is 100% packet loss, and the ALB lb-agent
heartbeat times out until the 5-min health window expires (TestRebootResilience).
rebindGatewayChassis only asserted NB intent. Now, after SetGatewayChassis, the
reconciler verifies the SB Port_Binding has a non-empty chassis and, if not, nudges
the local ovn-controller to recompute (inc-engine/recompute), polling until the
claim converges or a bounded deadline elapses — logging an error rather than
blocking reconcile. SB read + recompute are exec-only (ovn-sbctl/ovn-appctl) and
live in network/host behind a reconcile.GatewayClaimVerifier interface, wired in
vpcd; reconcile stays NB-pure and mockable. No NB object create/delete (ADR-0006 S5).
Verified by run 27251769281's diag-ovn-flows.txt (SB chassis empty + table-0 drop).
Validation completes on the next reboot-suite nightly (single-node self-heals too
fast to reproduce locally).
* test(reboot): capture heartbeat-path diagnostics on post-reboot health timeout
When post-reboot ALB targets fail to go healthy, the lb-agent's heartbeat to the
gateway times out with 'context deadline exceeded while awaiting headers' — a
symptom that alone cannot distinguish a network-level drop from an app/gateway
stall. Replace the fatal WaitForTargetsHealthy with a non-fatal poll that, on
timeout, tcpdumps br-wan for the heartbeat 5-tuple + ARP for the ALB floating IP
and pulls the lb-agent console + awsgw journal, then fails. Disambiguates the
layer before any fix (mulga-siv-265).
* test(reboot): re-aim heartbeat capture + add OVN dataplane flow-dump
Run 27250238163 proved the post-reboot failure is network-level (all 3 floating
IPs 100% packet-loss from host) but NOT neigh/NAT priming — neigh is primed and
NAT rows are present, only br-int forwarding is dead. The prior capture watched
br-wan for the ALB floating IP, while the real lb-agent heartbeat posts to the
gateway port on the host's advertise interface, so the L4 classifier could never
see the heartbeat and defaulted regardless.
- heartbeat tcpdump now watches the gateway port on `any` (real path) plus a
secondary br-wan floating-IP capture for north-south; classifier keys off
gateway-port L4 presence.
- new dumpOVNDataplane captures ovs-vsctl show, ovs-ofctl dump-flows br-int,
ovn-sbctl chassis + Port_Binding, dnat_and_snat NAT rows, host neigh, and a
host->floating-IP ping + ofproto/trace, to pin the flow-restore step.
- failure path dumps diag-ovn-flows.txt alongside diag-heartbeat.txt.
Test-only, build-tagged e2e; no production code.
CreateCluster set the control-plane VM/ENI/spread-group refs into the cluster
meta but first persisted them to KV only after allocateClusterEgressIP. A
failure between placement and that persist left the live sys.medium CP VMs
unrecorded, so neither DeleteCluster nor the FAILED-cluster reclaim could reach
them and they leaked, reserving host capacity.
Persist the meta immediately after placeControlPlane returns. The reclaim path
keys off the persisted ControlPlaneNodes + ControlPlaneSpreadGroup, so this also
frees reserved spread capacity on teardown.
A retried DeleteCluster (or the egress-delete cascade) can release the
hidden-pool allocation before the release step runs, so ReleaseAddress
returns InvalidAllocationID.NotFound. That was treated as a fatal teardown
error, so the cluster wedged in DELETING forever and its KV meta plus
downstream resources leaked. Swallow the NotFound variants as idempotent
success — mirroring the 165.17 ENI-teardown fix — so the SG + KV sweep
proceeds. A genuine release failure still wedges DELETING for retry.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* feat(eks): k3s server-join role for HA control-plane quorum
Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.
- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
join URL from its ENI IP, then launches the rest in parallel as join servers;
a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
but not k3s-first-boot (the first server already publishes the bootstrap
artifacts; a join re-publish only races the bootstrap bus).
* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join
Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.
Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.
* feat: register all control-plane ENI IPs as EKS cluster NLB targets
Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.
Associated Elastic IPs lost external connectivity after a host reboot:
vpcd's reconcile re-applied the dnat_and_snat rule without the ENI MAC,
collapsing it to the centralised shape (external_mac/logical_port unset),
and the daemon's relaunch republish only fired for auto-assigned public
IPs, never for store-backed EIPs.
- EIPRecord gains MacAddress, captured at AssociateAddress and cleared at
DisassociateAddress, so reconcile can re-apply the distributed shape
without re-querying the ENI.
- loadEIPs feeds rec.MacAddress into the reconcile spec; policy.AddEIP
self-heals a stale centralised row when given a MAC.
- AssociatedPublicIPForInstance lets the relaunch hook resolve an
associated EIP from the store and republish vpc.add-nat with the MAC.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* fix: strip duplicate lb- prefix from haproxy socket and cert paths
LoadBalancerID, ListenerID and friends already carry their type prefix
(lb-/lst-), so the stats-socket, frontend cert, and HAProxyManager
config/pid filename templates double-prefixed: lb-lb-<id>.sock,
lb-lb-<id>-<lst>.pem, alb-lb-<id>.cfg.
The socket path is computed independently on both sides (gateway render
in buildHAProxyConfig, agent poll in lbagent.New); any divergence in the
lb- prefix convention between the two leaves the agent polling a socket
HAProxy never bound, spamming 'HAProxy stats unavailable: no such file'
while the data plane serves fine. Drop the redundant literal prefix so
both sides resolve to <id>.sock and the alb-/nginx cert paths match the
delivered config.
* refactor: remove unused HAProxyManager
HAProxyManager was never constructed (NewHAProxyManager had no callers).
The live ALB data plane runs in the lb-agent via reloadHAProxy with the
config generated daemon-side and shipped; this per-daemon process manager
was dead code. Remove the type, its methods, tests, and now-unused
imports (log/slog, os/exec, sync, syscall).
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
The LB VM's external-pool floating-IP NAT (dnat_and_snat external_ip -> LB
VPC IP) was reaped only conditionally on teardown: DeleteNetworkInterface
skips EIP-owned addresses (defers to the EIP lifecycle), and that lifecycle's
reap runs in the async TerminateSystemInstance goroutine gated on the EIP
association. So repeated create/delete cycles accumulated stale rules that
collapsed onto the reused LB VIP, making OVN SNAT/un-DNAT ambiguous and
dropping the lb-agent heartbeat return path.
DeleteLoadBalancer now publishes vpc.delete-nat deterministically from the LB
record (external=AvailZones public IP, logical=VPCIP). DeleteEIP matches on
external+logical and is idempotent, so redundant reaps from other paths are
harmless. Internal LBs (no floating IP) are a no-op.
* fix: re-prime EIP reachability on AddEIP idempotent-skip path
AddEIP early-returned on the idempotent-skip path (OVN DNAT row already
matches) before emitting GARP and priming the host neighbour. Stop->start
and daemon reboot-recovery re-attach the same EIP with an unchanged row, so
the host neigh entry / OVS datapath never got re-primed and the EIP/ALB-VIP
stayed dark until the kernel ARP entry timed out (60-300s). inject-garp is
unsupported on some chassis, making neighPrime the only working remediation
— exactly the step the skip bypassed.
Extract garp + neighPrime/flush into primeReachability and run it on the
skip path as well as after a fresh AddNAT. The FlowsBarrier still fires only
on row churn. Resolves cell-9/10 TestMultinodeGuestSSH and cell-18
TestRebootResilience nightly failures.
* refactor: remove dead inject-garp GARP emitter from EIP path
ovn-appctl inject-garp is not a valid command on the OVN build Spinifex
deploys and tests against — every AddEIP GARP attempt failed with
'inject-garp is not a valid command' and fell through to the host neigh
prime/flush, which is the only remediation that works. The GARP hook was a
guaranteed-failing shell-out plus a misleading WARN on every EIP attach.
Delete host.InjectGARP, the GARPEmitter type, WithGARPEmitter option, the
garp field + its m.garp() call in primeReachability, and the vpcd
injectGARPEmitter wiring. host.ReplaceNeigh (distributed) and host.FlushNeigh
(centralised) already cover both NAT shapes and become the sole EIP
reachability path. Reword the neigh hook docs that framed themselves as an
inject-garp-independent fallback. Prune GARP-only tests; move the shared
recordingRunner helper into neigh_test.go.
tomnewton-mulga