* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
Julian Sommer