* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT
bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.
e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.
* test: validate ECS host-netns :80 binding via bridge task
The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.
purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:
- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
the DependencyViolation retry budget), in addition to the named cluster
SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.
Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.
On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.
Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.
Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.
Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.
Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.
bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.
Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.
Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
with ImpairedSince) / initializing (2-min launch grace), non-running to
not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
HostUnderMemoryPressure (live MemAvailable below the host reserve).
Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.
Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.
Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.
- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.
Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
Josh Siviour