66 Commits

Author	SHA1	Message	Date
Josh Siviour	68eeff069e	feat: project EKS access policies into Kubernetes RBAC (#1049) ... AssociateAccessPolicy recorded a policy association but never granted it in the cluster. Bake static ClusterRoleBindings for the mulga: groups into the eks-node image, project a principal's cluster-scoped associated policies onto those groups in the TokenReview webhook, and reject namespace scope at association time since there is no namespace-scoped RBAC writer.	2026-09-16 16:45:24 +10:00
Josh Siviour	8e11908dfb	fix: make the EKS control plane's disk latency observable, and give it room to run (#1009) ... * fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>	2026-09-07 21:53:41 +10:00
Ben Duncan	18853cef31	fix: close the four defects keeping the e2e nightly red (#1007) ... * fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main.	2026-09-06 16:53:32 -05:00
Josh Siviour	bed994859c	fix: durable Ochre/RDS appliance lifecycle (static ENI, teardown GC, stale LSP) (#852) ... * fix: join the daemon host-port ENI to the appliance security group The customer ENI's self-referencing SG drops all ingress except its own members, so the daemon's host-port ENI could open TCP but never complete a postgres handshake. Carry the appliance's security groups onto the daemon ENI at create, and re-associate an already-created one in place. * fix: complete teardown GC when a volume's metadata doc is already gone DeleteVolumeOnTerminate and DetachVolumeOnTerminate propagated the NoSuchKey error from a missing volume metadata doc as a hard failure, so the teardown reaper backed off forever and never purged the terminated record. A missing metadata doc means the volume is already gone, so treat it as success via objectstore.IsNoSuchKeyError at both call sites, with a matching tolerance in the instance-cleaner adapter. * fix: statically address the RDS customer ENI so it survives the DHCP lease The customer ENI was configured via a one-shot udhcpc with no renewal against OVN's 3600s lease, so the DB endpoint went address-less an hour after every boot. Resolve the customer subnet's prefix and gateway at launch and emit the ENI as a static NIC (CIDR + gateway, never the default route), mirroring the mgmt NIC, so it no longer depends on the lease being renewed. The guest static branch now installs the same source-based return-path policy a non-default DHCP NIC gets. * fix: delete the OVN logical switch port when an ENI is deleted An ENI delete published vpc.delete-port fire-and-forget, so a lost or slow message let the ENI record and its IP free through with the logical switch port still in OVN. A recreate onto the same IP then had two ports carrying it and the router latched the stale MAC, blackholing the datapath. Publish the delete synchronously so the port is confirmed gone before reuse, and make DeletePort idempotent so an already-absent port is a no-op.	2026-08-21 15:11:02 +10:00
Julian Sommer	bdc79108e9	fix: make the DB endpoint the only path to the engine (#830)	2026-08-19 11:04:21 +10:00
Julian Sommer	eac0f175ba	feat: require TLS on RDS connections by default, on both engines (#790)	2026-08-17 09:01:42 +10:00
Josh Siviour	d52245001e	feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788) ... * feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate	2026-08-13 15:54:48 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Julian Sommer	53b31cb76c	fix: keep the RDS master password out of the server and console logs (#754)	2026-08-10 10:16:53 +10:00
Julian Sommer	3379b16586	fix: encrypt and acknowledge the RDS bootstrap password (#739)	2026-08-06 15:56:06 +10:00
Josh Siviour	272b4b22e8	feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740) ... * feat(images): add a vllm-serving AMI as a stacked mkosi profile LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and Docker but no Python, no vLLM and no serving unit, so tagging it would have moved the failure from admission to first boot. Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side of the launcher contract: the runtime, vllm-serve.service, the weights mount and the tags. vLLM is baked into a uv-managed venv rather than pulled as a container at boot — a runtime docker pull of vllm/vllm-openai is known to stall this platform's write plane, and mkosi's unprivileged build has no daemon to bake an image from anyway. Also bring images/mkosi.profiles/ under make test-images. Its scripts have no .sh suffix, so the existing name glob skipped exactly the files worth linting; sources are found by shebang instead. * fix(images): make the vllm postinst survive the build sandbox Two failures found by actually running the build. uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so `command -v uv` reported a good install as missing. Call it by absolute path, and check it the same way. `vllm --version` initialises device config and dies with "Failed to infer device type" on a builder with no GPU, so it could never have verified anything. Assert the pinned version through package metadata instead, which also catches a version mismatch rather than merely a broken entrypoint. * fix(images): pin the serving AMI datasource and mirror cloud-init to the console A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as the EKS control plane. That path carries a cloud-init datasource pin; the vllm profile never got one, and the base config deliberately leaves datasource_list unpinned, so the serving image was the only system VM booting without it. Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing: read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_' sorts above '-') and the pin is discarded in silence. The postinst asserts the MERGED datasource_list via cloud-init's own read_conf_with_confd rather than checking the file exists, because it existed at 90- too and did nothing there. Also mirror every cloud-init stage to the serial console. A serving VM is headless, and a guest that wedges before handover never surrenders /var/log/cloud-init-output.log, so the console get-console-output reads is the only channel an operator has. The images_test.go case covers the two tags resolveServingAMI filters on; it matches on those alone and never on name, so a typo hides the image entirely. * fix(images): make the dhcpcd data-NIC lease genuinely one-shot dhcp_oneshot() promised to configure the link and exit "leaving no daemon to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd daemon survived on the data NIC holding the control socket. cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same NIC. Instead of spawning, dhcpcd forwards the invocation to the resident daemon as a control command and exits 0, and --dumplease returns the resident daemon's lease, so discovery looks successful. Cleanup then waits for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote, spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s on every system-instance boot. Adding -1 exits once the interface is configured and -p keeps the address and routes across that exit. Both are needed; -1 alone tears the link back down. The test now models the real clients — the stub stays resident unless given the flag that makes its client exit, and the dhcpcd branch is forced by dropping udhcpc from PATH rather than left to whether the host happens to have /usr/share/udhcpc/default.script. * fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles that kernel on first use. The serving image ships the driver and the container toolkit but no nvcc, so the sampler warmup died with "Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes into a GPU cold start, after the model had already loaded. VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler, which is the same operation with no CUDA toolchain at runtime. Everything else on the serving path is already precompiled in the wheels: the attention backend resolves to FLASH_ATTN and CUDA graph capture completes, both without nvcc. Baking FlashInfer's AOT modules is the alternative if the fused kernel is ever wanted, and it keeps the toolchain in the builder rather than the appliance. The unit had Restart=on-failure with no bound, so an unfixable start failure restarted forever and buried its own first traceback under identical copies on the serial console — the only view an operator has of a headless guest. StartLimitIntervalSec/StartLimitBurst make it report instead. The postinst asserts the installed unit carries the sampler setting, for the same reason the datasource pin is asserted there: the alternative is finding out at runtime, on a GPU, five minutes in. Also corrects the shared datasource pin's header, which still claimed it installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm profile precisely because a 90- name loses the reverse-sorted, first-source- wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg. * fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes The 5 minute bound was chosen before any cold start had been measured, and its own comment asked to be corrected once a model actually ran on wattle. It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup complete" 257.6s after launch, leaving ~42s of margin on the smallest model this platform can serve, with a cold torch.compile accounting for 26.3s of it. A bigger card and a bigger model are unmeasured. The two failure modes are not symmetric. Too low tears down an endpoint that was about to come up and reports it as a failed launch; too high only makes a genuinely stuck launch take longer to give up. So the bound is set well clear of the single data point rather than close to it. The CLI's own wait moves to 20 minutes to stay above it. The daemon starts its clock when the VM is launched, up to a minute after the CLI starts its own, so leaving both at 15 would expire the client first and report a bare "timed out" in place of the daemon's abort, which carries the reason. * fix: give the Ochre daemon a routed port in the Bedrock system VPC The readiness probe and every InvokeModel call dial a serving VM's system-VPC address from inside the daemon, but that subnet lives in OVN on br-int and the host has no route to it, so a healthy VM never reached READY. Each daemon now owns an ENI in the system VPC's private subnet, bound to an OVS internal port on br-int by iface-id, exactly as ovn-controller binds a guest tap. The ENI address is installed at the subnet's prefix length, which is what supplies the connected route. BaseURL is unchanged: the address it already carried is now reachable. Describe-or-create keyed on a per-node description, since ENIs carry no tag filter, so a restart or a second endpoint reuses one address instead of leaking one per launch. The port is ensured before anything else the launch creates: a VM the daemon cannot dial holds a GPU for the whole readiness window before unwinding.	2026-08-06 15:26:05 +10:00
Julian Sommer	1c4d41bffc	fix: reach the DB endpoint from any subnet of the VPC (#738)	2026-08-06 13:56:03 +10:00
Julian Sommer	e3b5701f15	fix: authorize RDS data-volume formatting by exact volume identity (#737)	2026-08-06 11:12:01 +10:00
Julian Sommer	5dd3af70eb	fix: rds handoff path	2026-08-05 15:52:55 +10:00
Julian Sommer	615578a2e7	refactor: simplify RDS internals (#692)	2026-08-04 14:42:45 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Julian Sommer	050dbdb060	fix: disallow reserved master names during rds password apply	2026-08-04 13:49:40 +10:00
Julian Sommer	970bf18ab5	fix: guard a half bootstapped datadir against crashes	2026-08-04 13:42:16 +10:00
Julian Sommer	faa0519cf1	fix: the RDS master user is no longer a PostgreSQL superuser (#687)	2026-08-04 10:59:45 +10:00
Julian Sommer	9dfef26e48	fix: rds-init	2026-08-04 10:24:48 +10:00
Julian Sommer	5e17169661	feat(e2e): complete the RDS end-to-end suite (#608)	2026-08-03 15:38:07 +10:00
Josh Siviour	a3fb4cccec	fix(eks): backoff and terminal state on delete re-drive, CSI volume reclaim reporting, independent e2e subtests (#642) ... * fix: bound EKS DELETING re-drive with backoff and a terminal give-up reapCluster re-drove purgeClusterInfra on every GC tick with no attempt tracking, so a permanently-failing teardown (e.g. an unretriable DependencyViolation) was retried forever. Track attempt count and last attempt time on ClusterMeta, back off exponentially between re-drives, and give up after maxDeleteReapAttempts with a single error-level log. Status stays DELETING on give-up so the cluster's infra remains tracked and billable pending operator intervention, preserving the ADR-0006 §6 billable-before-sweep invariant. * fix: create the EKS e2e node role independently in each subtest runEBSCSIVolume string-built the worker node role ARN without ever creating it, only passing because runIRSAPod's t.Cleanup discarded DeleteRole's error while the role was still attached to its instance profile, leaving it behind for EBSCSIVolume to reuse by naming convention. Extract ensureNodeRole and call it independently from both subtests, detaching the role from its instance profile before deleting it and asserting that error instead of discarding it. * fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete purgeClusterInfra tore down the control plane, nodegroups, and networking but never looked at EBS volumes, so any volume the in-cluster CSI driver had provisioned for a PVC was left behind untracked and unbilled for. Add a report-only reclaim step that scans for volumes tagged with the cluster's CSI owner tag and logs them for operator follow-up. Volumes are never auto-deleted: a PV's reclaim policy (Delete vs Retain) is a Kubernetes-side property the CSI driver does not tag onto the EC2 volume, so there is no safe way to tell from tags alone whether deleting a given volume is intended. This mirrors the volume-leak reaper's existing mark-and-alarm policy for the same reason. The cluster-owner tag requires the EBS CSI controller's --k8s-tag-cluster-id flag, added here to the controller manifest; without it a CSI volume carries no cluster-scoped tag and reclaim would have to guess across every cluster sharing the account. * fix: persist the last purge error on a wedged DELETING cluster DeleteReapExhausted told an operator a cluster's teardown had given up, but not why: nothing recorded the failing purge's error. Persist it on every failed re-drive, not only once the reaper exhausts its attempts, so it is visible on a still-retrying cluster too. Status stays DELETING throughout, preserving the billing invariant that untorn-down infra remains tracked. * fix: make the EKS e2e suite confirm its CSI volume is actually deleted The EBSCSIVolume subtest deleted its PVC on cleanup but never confirmed the underlying volume was reclaimed, so a CSI driver that silently failed to delete it would leak a volume with nobody noticing. The suite provisions and owns this volume, so unlike production teardown it can safely wait on its StorageClass's Delete reclaim policy: capture the CSI volumeHandle while the PVC is bound, then have cleanup poll EC2 for that exact volume to disappear before the addon/nodegroup that run the CSI controller are torn down.	2026-08-02 08:21:25 +10:00
Josh Siviour	9c9bb9430c	test: gate image shell scripts in CI and pin the spot vCPU quota ordering (#641) ... * test: gate scripts/images/ shell suites + shellcheck in CI The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran by hand: spinifex.yml filters pushes on Go-only paths and no job invoked them, so a broken helper could land unnoticed. Add a `make test-images` target and a path-filtered image-scripts.yml workflow so both run automatically on scripts/images/** and images/** changes, kept separate from preflight so image-asset churn doesn't run on every Go contributor's commit. Also clears the shellcheck findings that would have started this gate red (SC1090 on a runtime-resolved source path, SC2154 x2 on eval-assigned variables shellcheck can't trace). * test: pin vCPU quota gate ordering in RequestSpotInstances Nothing asserted that RequestSpotInstances runs the vCPU quota gate before dispatching capacity or launching a VM, so a regression that dropped or reordered EnforceLaunch would pass silently. Add an ordering test using a real quota.Service (embedded JetStream KV) and real NATS responders standing in for the capacity and launch daemons, recording a global call sequence and asserting the gate's index precedes both. Verified by removing the gate: the first quota-KV read then comes from the post-launch charge instead, landing after capacity dispatch and launch, and the ordering assertion fails as expected.	2026-08-02 08:21:04 +10:00
Josh Siviour	6873875248	fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634) ... On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane.	2026-08-02 08:16:00 +10:00
Josh Siviour	3e195506d3	feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. * refactor(images): extract eks-node initd logic into init-agnostic helpers The OpenRC runscripts carried their start_pre logic inline, so porting a service to systemd meant re-deriving that logic under a distro change rather than just rewriting the init wiring. Move it into /bin/sh helpers that either init system can drive, keeping eks-node on Alpine/OpenRC. k3s-prestart takes the restore-block guard, config.yaml derivation, konnectivity-agent and CoreDNS staging, and the token-webhook wait. konnectivity-server-prestart takes the CA wait and cert mint, plus the server-count derivation that used to run at parse time. Both keep every guard, bound and message as-is. Values flow back the way the GPU image already does it: the helper writes shell assignments to a /run file and the init system sources them, so command_args is built in start_pre where it used to be a top-level assignment. k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU manifest claimed its helpers were reused verbatim from eks-node while hand-copying this one; it now points at the shared file, so the two cannot drift. k3s-agent.initd still sources agent.env and exports itself rather than delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon inherits only what the runscript exports, so a helper subprocess sourcing it would not reach k3s. The helpers run under set -eu, which the runscripts did not. The awk over first-boot.env is guarded on the konnectivity manifest, not on the env file, so a missing env file exits awk nonzero; without || true that turned a warn-and-continue into a refusal to start k3s. Covered by a test that fails on rc=2 if the guard is dropped. * feat(images): build system images in a pinned mkosi container Move the image-build toolchain into a container so a runner needs only Docker. The toolchain cannot drift between runners, and the same build runs identically on a developer's machine. mkosi builds in a user namespace, which two of Docker's default confinements block: the seccomp profile masks CLONE_NEWUSER, and the docker-default AppArmor profile separately blocks unsharing the mount namespace. Both are relaxed; neither is a privilege grant, as capabilities stay dropped and no devices are exposed. mkosi also runs as an ordinary user rather than root, since as root it takes a path that needs real CAP_SYS_ADMIN. Verified end to end: an Ubuntu 26.04 Format=disk image builds from the Debian container and the artefact is a valid GPT disk, with the package cache on a named volume cutting a forced rebuild from ~2min to ~34s. * feat(images): publish system images from the build container Convert mkosi's raw output to a compressed qcow2, write the .sha256 sidecar the spx catalog verifier matches, and upload both to R2 using the same pinned toolchain as the build, so publishing needs nothing on the host either. The toolchain image build moves to a shared helper rather than being duplicated across the two entry points. Publishing needs no relaxed confinement, unlike the build: qemu-img and aws create no namespaces, so it keeps Docker's default seccomp and AppArmor profiles. The object path carries no version and the catalog resolves newest-wins, so an upload overwrites the live image with no way back. Copy the current object to a timestamped archive key first, making a bad publish a restore rather than an unrecoverable loss. * fix(images): build the image on the output's mount, not the cache volume mkosi assembles the image in its workspace and moves the result to the output directory. The workspace lived under the cache volume, which is a different mount from the output bind mount even when both sit on one filesystem, so the move failed EXDEV and mkosi fell back to copying — requiring the finished image to exist twice at once. That is invisible on a small image and fatal on a real one: a ~4G GPU image needed ~8G to land and died with ENOSPC after a successful build. Point the workspace at the output's own mount so the move is a rename again and the image only exists once. The package cache stays on the volume, which is the part worth persisting across runs. * feat(images): add mkosi base config and gpu-nvidia profile Establish images/ with a base mkosi.conf plus a gpu-nvidia profile, replacing the throwaway smoke-test config. The base builds a bootable UEFI disk; the profile adds the NVIDIA stack. Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version hunt, the do_bootloader=no kernel-img.conf hack and the manual update-initramfs all exist only because that pipeline chroots into a prebuilt cloud image that already has a kernel. mkosi installs the kernel and owns the bootloader, so the version is known rather than detected, and the postinst asserts exactly one kernel instead of searching for it. The static configuration (nouveau blacklist, unattended-upgrades blacklist, cloud-init docker group) becomes files under mkosi.extra/ rather than heredocs. linux-image-generic drags in every linux-firmware split package, none of it relevant to a headless VM guest; left at the default the modules initrd embeds all of it into the UKI, which grew to ~880MB and overflowed the 512M ESP. The initrd module set is trimmed to mkosi-initrd's curated default, which is a boot fix and not only a size one. Verified in the built artefact, not just the build log: nvidia.ko.zst and the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the nouveau blacklist, and the nvidia runtime in docker's daemon.json. * fix(images): unblock guest boot on wait-online and firstboot A guest built from these images stalled twice on the way to a login prompt, both times on a unit waiting for input that never arrives in a VM. systemd-networkd-wait-online holds network-online.target until every managed link is up, and docker.service pulls that target in with Wants=, so any image carrying docker inherits the wait whether or not it wants it. A guest whose links are slow or absent ran the unit to its full 120s default with multi-user.target queued behind it. Override it to --any with a short timeout: one link up is what a single-NIC guest actually needs. systemd-firstboot separately prompts on the console for anything the image does not already carry, before multi-user.target. With no timezone baked in, boot stopped dead at "Please enter the new timezone name" and waited for a keystroke. Setting locale, keymap and timezone at build time leaves firstboot with nothing to ask. Both are baked into the base rather than delivered as first-boot drop-ins: an image built from scratch has no cloud-init to write them. Measured on wattle with an A1000 passed through: login at 42.8s, down from ~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports "NVIDIA RTX A1000, 595.71.05, 8188 MiB". Also widen the images gitignore to output-*/ so a scratch build cannot leave multi-GB artefacts stageable. * feat(images): make the base image reachable — cloud-init, sshd, console An image built from scratch launched unreachable. The images this replaces start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a network renderer, dbus and a serial console from it; mkosi installs only what is listed, so none of it was there and nothing said so. Guests get their SSH key and user-data from IMDS via cloud-init, so without the package a launched instance has no authorized_keys at all — no error, just no way in. It also made any cloud-init-delivered config inert, including the gpu profile's docker-group drop-in. Each addition is load-bearing on its own: cloud-init consumes IMDS: keys, user-data, hostname, network openssh-server the key needs a sshd to be installed into netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is never configured, IMDS is never reached, and cloud-init falls back to DataSourceNone dbus systemd only Recommends it and mkosi installs no Recommends, so systemctl could not reach the system bus — and the EKS and ECS bootstraps drive guests with systemctl from user-data lsb-release cloud-init shells out to it; without it every boot reports "degraded" forever and the status stops being a usable signal The kernel also needs console=ttyS0 baked in. The launcher gives every guest a serial port and get-console-output reads it, but a kernel with no console= writes to none of it: the console stopped after the UEFI bootloader and every boot failure was invisible. ttyS0 goes last because Linux makes the final console= the controlling /dev/console. manage_etc_hosts is set because cloud-init renames the guest per launch but at its default never writes a matching hosts entry, so sudo warned on every call. datasource_list is deliberately NOT pinned: the guest identifies the platform from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on its own, and pinning per image is the customisation this migration deletes. Verified on wattle with an A1000 passed through: ssh in as ubuntu with spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s. * feat(images): let the guest grow its root into the volume it was given mkosi sizes the disk to its content, so the image ships with a root partition that fills the disk to its last sector and ~1.4G free. Nothing in it can expand that: cloud-init's growpart module is enabled, but the growpart binary lives in cloud-guest-utils, which is not installed and is not pulled in as a Recommends by anything in the package set. systemd-repart is not an alternative — Ubuntu's systemd package carries no repart binary, unit or definitions at all. The result was a node handed a larger volume that it could only ever use the first few GiB of, with the rest allocated and unreachable. That is fatal for a worker whose job is to pull container images. gdisk is deliberately left out despite being a cloud-init Recommends: growpart resizes GPT through sfdisk, which arrives via the hard fdisk dependency, and e2fsprogs brings the resize2fs that the resizefs module runs afterwards. Verified on wattle: the gpu-nvidia image launched with a 20G root reports 20G on df -h / (was 4.4G), via "'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with nvidia-smi still reporting the A1000 and no failed units. * feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles Adds the eks-common, eks-agent and ecs profiles, so both GPU node images build from the composed base rather than a hand-authored tree, and the three verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone. Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS node images run k3s or containerd and reach the GPU through CDI. Fusing the two was fatal for ECS rather than merely wasteful: gpu-nvidia installed containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs nvidia-ctk to generate its CDI spec. Closes three gaps the pilot left, each of which only shows up on a booted guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the modules on a headless node and CDI generation finds no GPU; mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on the toolkit packages, which the unattended-upgrades blacklist does not cover for a plain `apt upgrade`. mkosi-build.sh gains --image, naming an output and expanding it to an ordered profile list. The order is load-bearing: mkosi runs each profile's postinst in the order the profiles are passed, so docker's nvidia-ctk wiring must follow gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a healthy-looking image with no GPU wiring, so docker's postinst also fails loudly when gpu-nvidia is selected but nvidia-ctk is absent. Profiles reference build inputs where they already live via ExtraTrees= instead of copying them, keeping one copy of the helpers shared with the Alpine image. Go binaries are built on the host into a staging tree, since the builder container carries no Go on purpose: go.mod is the single pin for the toolchain and a second pin in the Dockerfile could disagree with it silently. * fix(images): install systemd-resolved so the guest has a DNS resolver mkosi installs only the packages named in Packages= plus their hard dependency closure, and systemd-resolved is merely a Recommends of systemd. So it was never installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated, because netplan renders to systemd-networkd and networkd alone does not write resolv.conf. glibc then fell back to localhost and every lookup failed with "[::1]:53 connection refused", which reads as a network fault rather than a missing package. A node that cannot resolve cannot pull a container image. The ECS GPU node booted, registered as a container instance, then failed every task at image pull. The EKS image carries the same defect but is masked by the cloud-init bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS launch path has no equivalent, so only ECS exposed it. * fix(images): name build output by image and never report a skipped build as success Two defects in the build wrapper that let the wrong bytes reach a boot test. Every composition wrote the same spinifex.raw, taken from the base config's ImageId, so an artefact carried no record of which profiles produced it and an EKS image was distinguishable from an ECS one only by which build ran last. Passing --image-id names the output after the image, and as a side effect the name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses its default input: --name is the published object and --image the local artefact, and with nothing tying the two together a defaulted input meant whatever raw happened to be in the output directory could be uploaded under the name the caller asked for. mkosi also skips the build whenever the output path merely exists. It does not compare inputs, so it never notices an edited mkosi.conf or a restaged binary, prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable from a fresh one. Builds are now always forced, and the run additionally asserts the expected raw was written during this invocation rather than trusting the exit status. The cache worth keeping is the package download volume, which --force does not touch. * feat: add gpu-amd mkosi profile for the ROCm AMI Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to stay alive for one remaining image. Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04 universe ships its own vendored rocminfo, which unpinned apt preferred over AMD's, mixing release trains within one image. Uses a systemd unit for render/video group membership instead of a cloud-init runcmd, because cloud-init's config merge takes a duplicated runcmd key wholesale from one fragment and silently discards the other, which would have dropped the docker profile's own usermod. Runtime GPU verification is deferred — no AMD hardware is available. * feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine systemd units, a templated etcd-snapshot service with daily and 15-minute timers, the addon/coredns/konnectivity assets, and the six server binaries. konnectivity-server is not an in-repo package, so it gets a pinned host-side clone-and-build rather than an entry in profile_binaries(). Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any attached volume and mulga-mgmt-net configures any multi-NIC system VM, so neither belongs behind an EKS profile that a database image would have to compose to reach them. Start-at-boot moves into preset files. mkosi runs preset-all after every postinst, so a postinst enable is discarded before the image is finished, and with no policy file the fallthrough is enable — which built an image that ran a control plane and a worker at once. Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the directory in reverse and the merge keeps the first source, so a 90- prefixed file lost the key silently. The postinst now asserts the merged result. * refactor(eks-node): make the role selector init-agnostic eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image the selector enabled nothing and the node never started k3s. Both now dispatch through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT overriding detection so either branch is testable on a host that has only one. Detection prefers rc-update over systemctl: the test suite stubs rc-update on PATH, and preferring systemctl would have silently rerouted those cases to a path they were not written for. Enable and start stay two separate passes. The server path starts mulga-eks-k3s-recovery before k3s so a recovery directive applies before the apiserver comes up, and collapsing to systemctl enable --now would interleave them. The dispatch is duplicated in both scripts rather than shared: they are single-file deliverables with no library-path convention between them. Tests cover both branches across all three roles, including that server-join still gets no k3s-first-boot, and that rc-update wins when both inits are present. * fix(eks-node): enqueue role-selected units instead of blocking on them eks-node-role.service declares Before=k3s.service, and mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a blocking systemctl start deadlocks: systemd will not run the queued job while the selector is still activating, and the selector cannot finish until that job runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie and the node wedges mid-boot with the role written but no control plane. On a booted guest the selector sat in activating for 261s until the VM was torn down. Start with --no-block on systemd. The ordering the blocking call was implicitly providing is already declared between the units themselves, so systemd sequences them correctly on its own. OpenRC runs the script directly rather than scheduling it, so that branch keeps blocking and keeps surfacing failures through set -e. * fix: escape $ in konnectivity-server ExecStart so server-count survives systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before the command runs, substituting an empty string for a name it does not know. KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes it to /run/konnectivity-server-args.env, which the wrapper shell sources at runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw it. konnectivity-server then died on flag parsing, which is why the unit logged Started and produced nothing despite --logtostderr=true and journal+console, and why the apiserver's cluster egress selector found no UDS for the life of the cluster. This is a port artefact. konnectivity-server.initd is flag-for-flag identical and shares the same prestart script, but under OpenRC the shell was the only thing reading ${KONN_SERVER_COUNT}, so the same text was correct. Escape it as $$ to defer expansion to the wrapper shell, and add lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of a profile's units. The linter joins line continuations first, since the offending reference sat mid-way through a wrapped multi-line ExecStart, and runs on the host before the container starts so a bad unit fails the build in seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd leaves it literal. Verified on env12 with a rebuilt image: TestEKS/IRSAPod and TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed out` through the egress tunnel, now pass, with zero UDS connect errors across the run. * fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw mkosi only emits the raw. A qcow2 next to it came from a previous run's hand conversion, and nothing about its name says which build it belongs to, so it survives a rebuild and silently ships the old image to a node while the raw beside it carries the new one. Observed on a rebuild that left the raw at 15:34 carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit. This is the same failure the existing freshness assertion guards against, one file over, so extend that block rather than add a second check. Refresh rather than delete: the qcow2 is what gets copied to a node and imported, so removing it would push the manual conversion, and this trap, back onto the caller. Convert via a temp name and rename so an interrupted run leaves the old file rather than a half-written one that looks current. * fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns, which systemd-networkd does not manage, so it shells out to a DHCP client. That call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not. `ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the failure surfaced as an opaque "network setup failed" with the task stuck in STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and ServiceWithELB both failed on the mkosi image, the latter as runningCount=0 because its target group targets task ENI IPs. Resolve the client at setup time, preferring udhcpc so Alpine behaviour is unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not sufficient to select it: Debian and Ubuntu's busybox ship the applet without /usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits nonzero having configured nothing. Same order and same reasoning as dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no daemon behind in the netns. Install dhcpcd-base in base, because neither client was present in any profile. That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved to nothing on Ubuntu and would have failed the first time a cross-host control-plane data NIC needed it — the S5 path the step-6 plan notes has never been exercised. The DHCP-client choice is injected so the tests assert a named platform's branch rather than whatever happens to be installed on the machine running them; without that the same test exercises different code on an Alpine CI box and an Ubuntu dev box. * refactor: retire the ec2-node profile, fold its helpers into the base image ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition had to remember to list it — eks-common even guarded on its presence and aborted the build without it. ECS did not list it, so the ECS image shipped with no by-id symlinks for hot-attached volumes at all. Neither helper is answerable at build time: any image may later be attached a volume, and any image may later be launched as a control plane. Both are inert when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its fw_cfg blob is absent), so the base image is where they belong. That deletes the composition rule, the guard, and the ECS gap in one move. Also drops ec2-node from the two eks image compositions in mkosi-build.sh. Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both carry the helpers and the udev rule, mulga-ebs-byid is enabled via multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants. * fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link mulga-vpc-mtu only won the boot race by ordering: netplan renders UseMTU=true (it does not inherit networkd's own false default), so a DHCP lease renewal at T1 could silently re-apply the advertised MTU to the link. usemtu_off() forces UseMTU=false via a runtime drop-in in the systemd-documented [DHCPv4] section for every rendered unit and reloads networkd, removing the dependency on boot ordering entirely. assert_pinned() then verifies the link and its routes actually match before exiting, so any regression is a loud boot-time failure instead of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has no systemd-networkd. * fix(images): state UseMTU=false under both the current section and the legacy alias netplan renders the DHCP settings under [DHCP] while systemd documents [DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the running systemd disagrees, which is the silent no-op the drop-in exists to prevent. * fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch credendpoint.go DNATs the ECS task credential endpoint via iptables; neither the mkosi ecs profile nor the legacy Alpine manifest installed it, so the endpoint silently failed to start and any task with a task role could not fetch credentials while the instance still looked healthy. Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only, which this platform's HTTPS AssumeRole path can never match) and adds a TaskRoleCredentials subtest that fetches real credentials from inside a task container, so a dead endpoint fails the suite instead of passing unnoticed. * fix(images): ship preset files for base, eks-common and ecs start-at-boot mkosi runs systemctl preset-all after every profile's postinst, discarding whatever systemctl enable a postinst performed. eks-server and eks-agent already carry preset files for this reason; base, eks-common and ecs still relied on the default-enable fallthrough, which happens to match today but is one upstream package change from reversing. Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net), 30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and 40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables cni-dhcp.service/socket, which containernetworking-plugins enables by default with nothing to serve under the baked host-local IPAM conflist). Removes the now-redundant systemctl enable calls from each postinst, matching the eks-server/eks-agent precedent. * fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal denied every Service clause other than the IMDS-only EC2 one. A role carrying the trust policy AWS documents for an ECS task role was unassumable by construction. Service attribution is now derived from the authenticated SigV4 caller: an assumed-role session of the account's ecsInstanceRole — the only credential the in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles in its own account. Nothing in the request body contributes, so a task holding its own task-role session cannot claim the service principal. * fix(images): remove baked SSH host keys, pin Seed= for reproducibility openssh-server's postinst bakes an identical host key pair into every build, making every guest from one build mutually MITM-able; sshd-keygen already regenerates keys on first boot (ConditionFirstBoot=yes), so the baked pair is pure non-determinism with no purpose. Also pin Seed= so systemd-repart stops assigning fresh partition UUIDs on every build. Verified on a built spinifex-ecs-node image: no ssh_host_* files ship, cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service is wired via ssh.service.wants/ (redundant regeneration path). A two-build comparison dropped content-differing files from 9 to 1 (only the UKI, whose zstd internals remain sensitive to unrelated input changes). --------- Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>	2026-07-30 12:57:43 +10:00
Julian Sommer	32d80b473a	feat: RDS v1 foundations (#592)	2026-07-29 10:38:22 +10:00
tomnewton-mulga	6389b37433	fix(gpu): Blackwell GPU support fixes for EKS/ECS image builds and VFIO (#559) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. --------- Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>	2026-07-27 12:11:18 +10:00
Josh Siviour	4793b5ac09	fix(eks): node route MTU blackholing image pulls, and teardown wedged in DELETING (#562) ... * fix(eks-node): stop the DHCP MTU overriding the pinned link MTU Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which honours DHCP option 26 and stamps the advertised MTU onto every route it installs: default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \ metric 1002 mtu 1442 Linux prefers a route's MTU metric over the device MTU, so the node kept advertising an MSS derived from 1442 while the real path MTU through the NAT gateway is 1408. Small packets pass and full-MSS inbound segments — a TLS ServerHello with its certificate chain, image layers — are silently dropped. It presents as `net/http: TLS handshake timeout` on image pulls, not as anything MTU-shaped, and it only affects the host netns, so containerd pulls fail while pod traffic sized off cni0 is fine. Per-destination PMTU exceptions expiring in and out of the route cache made it look intermittent and host-specific. Three changes, innermost first: - dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a route MTU. mulga-vpc-mtu is the single source of truth for the MTU of these nodes. - mulga-vpc-mtu re-stamps any route that already carries a different MTU, covering an image built before this fix or a dhcpcd already running. Routes with no MTU metric inherit the device MTU and are left alone. - the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and ignores the IPsec ESP overhead on the NAT egress path, so it overstates the usable MTU for every guest, not just these nodes. Also corrects the header comment, which claimed Alpine's udhcpc ignores option 26 — this image runs dhcpcd, which honours it. That wrong premise is why the route metric was never considered. Adds a stubbed-`ip` test covering the rewrite, the dev token that `ip route show dev X` omits, idempotency, and routes without an MTU metric. * fix(eks): sweep the CP VPC by identity so teardown can finish DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs by role tag. A create that fails partway — the first control-plane server failing to launch, say — leaves resources behind that never got tagged, so the sweep cannot see them. DeleteVpc rejects while any subnet or non-default SG remains, and teardown is re-driven on a timer, so the cluster sits in DELETING forever retrying the same doomed delete every couple of minutes: eks-deleting: re-driving wedged DELETING teardown cluster=toc eks-deleting: re-drive teardown failed err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation" Nothing converges: the cluster name cannot be reused and terraform's delete wait can never pass. Observed on env19, wedged for ~11 hours until this change went in, at which point the next re-drive cleared it. After the tagged sweep, clear anything still in the VPC scoped by vpc-id: subnets, and non-default SGs (the default SG goes with the VPC cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG provisioner — nil is tolerated, since only teardown uses it. * chore: pin viperblock + predastore to the merged storage fixes viperblock f8cfd77 (#42) makes read-modify-write atomic per block, removing the lost update that silently reverted a 4096-byte block to a previous generation of itself. Confirmed as the env19 corruption cause by an A/B: the pre-fix build corrupted image layers within ~15 minutes (failed to extract layer / invalid tar header / unpigz: corrupted) while the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW conflicts on the worker root disks. predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore restart on a store whose metadata set outgrew one badger transaction fails with "Txn is too big to fit into one request" on every node at once, taking the metadata plane — and with it the AMI catalogue and cluster creation — down permanently. Both were verified in the built artifact rather than from go.mod alone: go version -m reports the new pseudo-versions, and the rmw_conflicts counter plus the batched-restore error strings are present in the binary. Pinning by go.mod while the submodule pointer says otherwise is how an earlier analysis ended up reading dead code. * test: record the corrected DHCP MTU in the kitchen-sink golden The NB golden pins the DHCP options published per subnet, including the advertised MTU. Dropping the subnet option from 1442 to 1408 changes what the reconciler writes, so the golden diverged and the live scenario test failed — the test doing exactly its job. Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408 on the two subnet DHCP records; nothing else in the snapshot moves.	2026-07-22 12:57:36 +10:00
Josh Siviour	b41a193cbc	fix(eks): make provider-id.sh the single k3s node-label authority for nodegroup + GPU labels (#539)	2026-07-15 00:39:13 +10:00
Josh Siviour	4d8a34b545	fix(eks): apply eks.amazonaws.com/nodegroup label via k3s node-label drop-in (#534) ... The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but k3s has no such env var, so non-GPU workers never received the label. waitWorkersReady buckets Ready nodes by that label (added in #511), so the affected nodegroups never counted toward their ACTIVE transition and stayed CREATING until timeout. Ride the label on a real k3s node-label config drop-in (15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the inert K3S_NODE_LABEL from agent.env and the initd export.	2026-07-14 12:43:42 +10:00
Josh Siviour	1e1d2f3c6d	ci: GPU e2e workflow on ci-gpu self-hosted runner (#518) ... * ci: GPU e2e workflow on ci-gpu self-hosted runner Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent, hand-bootstrapped wattle shared host, driven over SSH by a banksia runner labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner installed on the shared box. - runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress false (single A1000 = single-flight, queue not stomp) - workflow_dispatch + GPU-path filter (not every push) - manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU testing blocks CI - if: always() cleanup terminates launched GPU instances; never resets the box (no setup-ovn.sh/dev-reset/cluster-reset) - drives the control plane with spx + the already-configured aws spinifex profile directly (no wrapper scripts, nothing to install on wattle) * ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs The suite is self-guarding — each test skips when its AMI is absent. The hard-fail on the base ubuntu GPU AMI blocked the whole run even when the ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and fail only when none are present. * ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported the self-guarding suite skipped every test, passing as a false green. Make the raw-EC2 GPU passthrough test the always-on baseline. - Add an import-if-absent setup step that pulls the prebuilt base GPU image from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's persistent loop-fs. Runs spx under sg spinifex (local storage write). - Preflight now hard-requires the base AMI (fail, not skip) so a silent import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn. - Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and launch-gpu.sh's nvidia default now use the same canonical name, so a locally built+imported image also satisfies the suite. * ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches spx admin images import --name selects the catalog entry but names the resulting AMI generically (ami-<distro>-<version>-<arch>), so the import landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same flag build-system-image.sh uses) and poll briefly for availability. * fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout Auto-assigned public IPs were never carried through the network reconciler. applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no gatewayLRP->guest convergence. Their dnat_and_snat was created once at RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the guest-port hop never converged: inbound DNATs at the gateway but never reaches the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK). floatingIPSpecs now folds every auto-assigned/ELB public IP into the same AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user EIP wins), making the datapath self-healing across reconcile ticks and deploys. qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB, floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers. * fix(vm): scale QMP handshake command deadline for VFIO launches The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU guest sends its QMP greeting before the RAM pin completes, then cannot service qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and the launch cleanup SIGKILLed a QEMU that would have come up. Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s) for qmp_capabilities. Other callers keep the 30s default. * test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle	2026-07-13 21:28:48 +10:00
Josh Siviour	f7bf921ce7	fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511) ... * fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group names exceed the ELBv2 32-char limit, failing cluster creation asynchronously with no visible cause. safeELBv2Name now deterministically truncates the name and appends a stable hash of the full cluster name when needed, so the same name is derived on every call (create, lookup, delete). DescribeCluster also now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster is FAILED, instead of leaving callers to guess why. * fix(eks): plumb per-nodegroup Ready-node counts through state reports The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready count keyed by the eks.amazonaws.com/nodegroup label value, alongside the existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's observe step, and SetClusterHealthState now carry this map through to ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next commit) can gate on a nodegroup's own workers instead of the cluster total. * fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches waitWorkersReady compared the cluster-wide Ready-node count against a nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's Ready workers could mask another whose own workers never registered. It now reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the eks.amazonaws.com/nodegroup=<name> label. Separately, launchNodegroupInfra never retried a failed worker launch (a transient QMP timeout or host capacity pressure), leaving the nodegroup stuck CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches only the shortfall on failure, logging each retry, until the desired count is reached or workerLaunchRetryTimeout elapses. Both fixes are covered by regression tests: a two-nodegroup scenario where only one has Ready workers, a transient-failure-then-success relaunch, and an exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which this change does not add. * fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and kubectl logs/exec against pods on that node failed "No agent available". Adds an Exists toleration for nvidia.com/gpu:NoSchedule. Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster available to test against right now. * fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS	2026-07-12 12:11:16 +10:00
Josh Siviour	9f349e86ca	feat(eks): expose NVIDIA GPUs to pods (Epic B) — VFIO QMP, worker registration, device-plugin GPU access (#507) ... * feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%).	2026-07-12 10:38:36 +10:00
Josh Siviour	d663760dec	feat(eks): control-plane disaster recovery — etcd snapshot + restore-snapshot (#503) ... * fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run * test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the mgmt-bridge branches and EKSServiceDeps assembly had no direct test. * feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point) * fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status) * fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing) * fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host * fix(daemon): log NATS handler failures at Error (were only on the trace span)	2026-07-10 22:14:58 +10:00
Josh Siviour	6879a3425a	feat(gpu): Ubuntu NVIDIA GPU node images + GPU AMI selection (Epic A) (#502) ... * feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection Epic A of GPU passthrough to EKS pods / ECS task containers. - Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers (instance-family prefix -> vendor from knownGPUModels). - EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU instance types; no fallback to the driverless image (hard error instead). - Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no negation, so exclude client-side). - NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes. - Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged). - New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the proven DKMS -server driver recipe (drop Docker CE + dev payload), add nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation. - build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags. * build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the build onto a real disk.	2026-07-10 19:52:12 +10:00
joshsiv-mulga	2c1ddea124	Remove backed-in creds for eks addon sync - prefer IMDS	2026-07-09 09:43:32 +10:00
Josh Siviour	e47baf321c	feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480) ... * feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight.	2026-07-09 09:43:32 +10:00
Josh Siviour	7bbd99c792	feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459) ... * feat(eks): multinode control-plane networking, IAM, and CoreDNS parity Integrates the EKS multinode fix set for the managed-control-plane topology so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it. EKS: - Route apiserver egress (exec/logs/webhooks) through konnectivity (apiserver-network-proxy): CP servers run konnectivity-server behind the cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod traffic rides the agent-initiated reverse tunnel instead of an unroutable CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every replica (HA-correct). - Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay flannel mesh SG rule authorized at cluster launch with ordered teardown. - Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay. - Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity. - Spread nodegroup workers across hosts; addon readiness falls back on workload rollout when gvks are absent. - elbv2: allow the proxy_protocol_v2.enabled target-group attribute. EBS: - Move volume attachment state to state.json so a live nbdkit VB cannot clobber it; route mounted-volume state updates through the live VB with a retry for in-use marks racing the mount responder; keep the boot volume in-use across stop/crash unmount. * fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM missingOrchestrationDeps gated on s.deps.IAM, but production wires only the lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only). The gate therefore reported missing=[IAM] on every node forever, so CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even though the IAM service was fully up via the provider. Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else deps.IAMProvider()) the nodegroup/role code already uses — so the protective check still rejects a node with no IAM service but passes when the provider supplies one. Add a regression test that sets only IAMProvider and asserts the gate is satisfied; the existing nil-both test still rejects. * fix(eks): let eks-token-webhook use IMDS creds when static keys absent The CP VM launches on an instance profile, so buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys before eksgw.New could fall back to the IMDS SDK chain, so the webhook aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver crashed on the missing --authentication-token-webhook-config-file file, stalling the cluster at CREATING until it timed out to FAILED. Drop the static-key precondition (matching the sibling eks-gateway-publish and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/ cluster gates. * fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt, which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a 169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing 169.254.169.254 and steering IMDS off the data NIC away from the per-tap br-imds datapath, where the host RST it. eks-gateway-publish then failed to sign requests and no bootstrap artifacts were ever published, so clusters sat CREATING until they timed out to FAILED. Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI) are unaffected and IMDS falls through the data NIC's default route to the br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net deletes any default route on a NIC declared NIC<n>_DEFAULT=0. * fix(eks): grant eks:WebhookTokenReview to the control-plane instance role The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway broker at POST /clusters/{name}/token-review, signing with the control-plane VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes against eks:WebhookTokenReview, but the role's inline policy granted only eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every relay and the apiserver returned 401 to all external kubectl. Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile re-asserts the inline policy on every control-plane instance-profile ensure, and the gateway evaluates authorization per request, so existing clusters converge without a rebuild once the shared role is updated.	2026-07-09 09:43:30 +10:00
Josh Siviour	2d4f0fa94c	feat: IMDS instance-role creds for lb-agent + EKS server VMs (siv-231.6) (#450) ... * feat: IMDS-capable gateway signer for lb-agent + EKS clients Add internal/gwsign, a SigV4 signer that draws credentials from either fixed static keys or the AWS SDK default credential chain. In a guest VM the chain falls back to the in-VM IMDS endpoint, yielding scoped, auto-rotating instance-role credentials; a session token is signed as X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds. Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client (internal/eksgw). When static keys are absent the clients now sign with IMDS credentials instead of failing; when present, behaviour is unchanged (static path). This is the client-side foundation for dropping long-lived secrets from VM user-data. Credentials are retrieved per request so rotated IMDS creds take effect without an agent restart. * feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C) Attach a Spinifex-managed system instance profile to the k3s control-plane VM and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating IMDS instance-role credentials instead of a long-lived static secret baked into user-data. - Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a role (EC2 trust) plus its instance profile and asserts an inline permission policy. The gateway enforces per-action authz on assumed-role principals, so the roles carry exactly the actions each VM calls: - EKS server: eks:PublishInternal, eks:ListInternalAddons. - LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig. - Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves the role. - Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/ EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from the lb-agent fw_cfg env; relax k3s server validation to accept profile-only; drop the EKS key requires from the first-boot and state-report scripts. - Both verticals fall back to baked static creds when IAM is unwired, so an absent master key never blocks a launch. * fix: resolve IMDS instance-role creds under the instance account for system VMs Three boot-path defects kept the LB VM's lb-agent from authenticating to the gateway with IMDS-served instance-role creds; each produced a silent empty IMDS role list and a heartbeat failing with "unexpected empty EC2 IMDS role list". Found and fixed against env19. - Build the system IAM ensurer lazily on the first system-VM launch instead of eagerly at daemon startup, where it raced the NATS KV backend and fell back to baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider resolved at launch time. - Create the LB instance profile in the system account the LB VM runs under (utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile under the instance's account, so a customer-account role was invisible to it. - Record the attached instance's owning account on the ENI (InstanceOwnerId, mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every IAM/STS lookup under it. The LB ENI lives in the customer account while the system LB VM and its role live in the system account; keying the whole IMDS chain off the ENI account left the instance invisible and the role lookup missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account. A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role principal with no static keys in user-data. * test(e2e): assert LB VM authenticates via IMDS instance-role creds The LB suite's targets-healthy assertion already depends on the lb-agent authenticating to the gateway, and with static keys dropped from user-data that auth is IMDS-only. Add an explicit check that a recent gateway authentication for the LB used an assumed-role (ASIA) principal, so a regression to the instance-account IMDS resolution fails with a clear message instead of a mysterious datapath timeout. The LB VM is a system-account instance invisible to the customer AWS API, so the assertion observes the gateway journal on the cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and skips when no node is SSH-reachable.	2026-07-09 09:43:29 +10:00
Josh Siviour	a7241e01f9	feat: run EKS apiserver with anonymous-auth=false (CIS) (#447) ... Flip the k3s server apiserver to anonymous-auth=false. The only consumer of the anonymous path was mulga-eks-state-report's /healthz probe; switch it to an authenticated kubectl get --raw=/healthz through the node admin kubeconfig (an unauthenticated curl now returns 401). The NLB target group uses a TCP health check, and the host-side reconciler rides the authenticated NATS state-report, so neither depends on the anonymous apiserver path.	2026-06-30 09:21:07 +10:00
Josh Siviour	de64d6c63d	feat(ecs): agent control-plane over the AWS gateway (drop direct NATS) (#419) ... * feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs") client the ecs-agent will use to reach the AWS gateway over HTTPS instead of connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal. * feat(ecs): SubmitTaskStateChange gateway action → recordTaskState Wires the AWS-API SubmitTaskStateChange action end to end (gateway action → ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a gateway-routed agent reports task state without touching the NATS bus. The handler maps the SDK input onto the same bus.TaskState shape and converges on recordTaskState; the account is authoritative from the SigV4 caller, so an instance cannot report state for another account's task. * feat(ecs): durable assign KV inbox + PollAssignments gateway action Scheduler writes task assigns into a per-instance KV inbox (clusters/{cluster}/assignments/{instance}/{task}) instead of a fire-and-forget core-NATS publish, and a new PollAssignments gateway action lets a gateway-routed agent drain its inbox over TLS+SigV4. At-least-once: an unacked assign is re-delivered on the next poll; the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions reclaim stale inbox entries. The Poll response carries bus.Assign with an RFC3339 time, so it is encoded with encoding/json (RawJSONActions) rather than the jsonutil epoch-second marshaler. * feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway The agent no longer connects to NATS. It registers, heartbeats, reports task state, and receives assignments entirely through the AWS gateway over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus host-internal and closing the trust-boundary gap (a tenant container instance must not hold a shared bus token). - controlPlane seam over ecsgw.Client replaces the NATS publisher: Register (RegisterContainerInstance), SubmitTaskState (SubmitTaskStateChange), PollAssignments. - Heartbeat folds into a periodic idempotent re-register (refreshes LastSeen); bus.Heartbeat is dropped from the agent. - subscribeAssign (NATS subscribe) becomes a poll loop: drain the inbox, dispatch each assign once, ack accepted taskIDs on the next poll (at-least-once). - config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added, ECS_NATS_URL removed; cloud-init initd seeds the IAM creds. Tests move from the NATS publisher fake to a controlPlane fake. * feat(ecs): cloud-init cred-seed + ecs-node bring-up script ECS container instances are plain EC2 booted from the spinifex-ecs-node AMI (AWS-faithful: no ECS-specific launch API). This renders the cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the instance. The agent registers, heartbeats, polls assignments, and reports task state entirely through the AWS gateway over TLS+SigV4 — never NATS. The guest dials the customer-facing gateway address (network advertise IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL split. Seeds the system key for now; scoped per-instance creds are a follow-up (4g task-role endpoint). * fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring The 'Merge branch dev' resolution dropped code where Sprint 4d (#418, awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch the same paths: - recordTaskState STOPPED branch: reclaimTaskENI was clobbered by reclaimAssignInbox; both are needed (release the task ENI AND drain the KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce. - ecs-agent Agent: the netns field, its init, and the setupTaskNetns/ teardownTaskNetns wrappers were lost when the controlPlane Agent struct won the merge, leaving taskrun.go calling undefined methods. - eni_test assign assertion read the assign over NATS; #419 delivers it via the durable KV inbox, so it now polls via PollAssignments. - taskrun_test netns cases used the removed historyPub fake; switched to fakeCP to match the controlPlane seam.	2026-06-25 18:42:24 +10:00
Josh Siviour	9c76df3373	feat(ecs): awsvpc mode — ENI-per-task allocation + guest datapath (Sprint 4d) (#418) ... * feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d) Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the task subnet on placement, hot-plug onto the container instance, plumb ENI identity through the assign payload + task record, and detach+delete on STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged. * feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI The agent builds a per-task network namespace for awsvpc tasks: it finds the hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs the OVN-assigned ENI IP. The container then joins that netns via the OCI network namespace path; bridge/host tasks keep the host (VM) netns as before. * build(ecs-node): add iproute2 for awsvpc task netns The ecs-agent builds awsvpc task network namespaces with `ip netns`, which busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI; udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.	2026-06-25 16:01:14 +10:00
Julian Sommer	8a94886b13	feat: cloud-init EC2 datasource (#411)	2026-06-25 13:48:08 +10:00
Josh Siviour	24bd50a53c	feat: ecs-node AMI (alpine + containerd + ecs-agent baked) (#415) ... Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc + cni-plugins + ca-certificates with the ecs-agent binary baked to /usr/local/bin and three OpenRC services wired in dependency order (ecs-ca-install -> containerd -> ecs-agent). - manifest.conf drives build-system-image.sh (no host mounts). - ecs-ca-install.initd: first-boot oneshot installing the cloud-init delivered gateway CA into the system trust; no-op when absent. - ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the agent after net + containerd. - 10-bridge.conflist: CNI bridge for task networking (Sprint 4d). - setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0 console reorder + fast boot menu. - Makefile: build-ecs-node-image / import-ecs-node-image targets.	2026-06-24 21:54:35 +10:00
Josh Siviour	5025a37006	feat: Containers — internal-ECR pulls, DNS-free registry, EKS hardening + docs (#409) ... * docs: add Containers section with EKS and ECR service pages New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks. * docs(eks): rework Terraform workbooks into a capability ladder Rework the three EKS workbooks into a coherent ladder and publish them to the Containers section. All tiers are single- or three-node capable via node_desired_size and serve a shared Spinifex-themed demo app from ECR, replacing nginxdemos/hello. - demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/ region + hit counter; optional /data persistence), distroless image. - eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort. - eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer- controller addon + ACM, public/private VPC with NAT, managed-ingress=false. - eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access entry RBAC, HTTPS Ingress carried over. Pending live e2e verification (image build/push, apply, ALB + PVC bind). * fix(ecr): implement ListTagsForResource (empty tags) for client parity The AWS provider calls ecr:ListTagsForResource on every repository read, which returned 501 NotImplementedException and broke 'tofu apply' of any workbook that creates an aws_ecr_repository. Implement it to return an empty tag set (resource tagging stays a deferred feature) so reads succeed. Drop the tags block from the three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed TagResource on create. * feat(eks): pull from internal ECR via kubelet credential provider Worker nodes could not pull custom images from the internal Spinifex ECR registry: the eks-node had no DNS for the registry host, no containerd CA trust, no port rewrite to :9999, and no pull credentials. Only public images worked. Add cmd/ecr-credential-provider, a kubelet exec credential provider that fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic auth for the registry host. Thread the gateway URL, account, region, and internal suffix into the agent userdata, which now writes the gateway CA, a registries.yaml (CA trust + :9999 endpoint), the kubelet credential-provider config and config.yaml.d drop-in, and an /etc/hosts entry mapping the registry host to the gateway IP. Ship the provider binary in the eks-node AMI. Also remove a stray paragraph from the admin-cli README. * fix: dial EKS ECR registry mirror by gateway IP cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on every boot, wiping the registry-host mapping appended via bootcmd, so worker nodes could not resolve the internal ECR host and pulls failed with no such host. Point the k3s registries.yaml mirror endpoint and tls config at the gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image ref port-less and dropping the DNS leg until northstar lands. Fall back to the hostname endpoint when the gateway is not an IP. * fix: attach node-role instance profile to EKS workers for ECR EKS workers launched with no IAM instance profile, so IMDS served no role and the ECR credential provider could not mint a registry token, leaving internal ECR pulls failing with 401. Real EKS implicitly creates an instance profile for the node role; do the same here. Wire a KV-backed IAM service into the EKS deps and, at worker launch, find-or-create an instance profile named for the node role, attach the role, and pass it to RunInstances so IMDS exposes the role to the credential provider. * feat: use mulga logo in EKS demo app * feat: add ACM certificate tag support (List/Add/RemoveTags) Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate. * test: cover ACM tag gateway and NATS wrapper paths * feat: support EKS nodegroup tags (Tag/Untag/ListTags) Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException. * fix: provision EKS ALB ingress via node credentials The aws-load-balancer-controller addon could not provision an ALB. Three fixes: - TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE, falling back to the Go system trust pool and rejecting the gateway cert as 'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool. - Credentials: an empty AWS_ROLE_ARN beside the web-identity token file half-arms the SDK web-identity provider, which then signs anonymously (MissingAuthenticationToken). Inject the IRSA env only when a service-account role ARN is staged; otherwise omit it so the default chain falls back to the node instance-profile credentials served over IMDS. - Authorization: the node role lacked the ec2/elbv2 actions the controller needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController managed policy to the node role (CreatePolicy + AttachRolePolicy). * fix: add force_delete to EKS workbook demo ECR repos tofu destroy of an EKS workbook left the demo ECR repository behind: it still holds the pushed demo image, so DeleteRepository (force=false) returns RepositoryNotEmptyException. Add force_delete = true to each workbook's aws_ecr_repository.demo so teardown removes the repo with its images. * fix: force-delete EKS control-plane ENIs on teardown TerminateK3sServerVM deleted the CP ENI with a plain force=false DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the record's Status/InstanceId/AttachmentId are set, so when the K3s VM is gone but the ENI record's attachment fields were never cleared the delete returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper re-drove the same non-idempotent call every 90s, wedging the cluster in DELETING. Teardown owns its CP ENIs and has already terminated the VM, so the attachment is authoritatively dead. Replace the InUse-retry budget loop with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the stale attachment fields, then DeleteNetworkInterface passes the in-use guard. Both calls tolerate an already-gone ENI (NotFound) so a race with the async instance-terminate cascade resolves to idempotent success. purgeClusterInfra's private-endpoint ENI path reuses the same helper. * fix: detach+delete worker ENIs on EKS nodegroup teardown A terminated nodegroup worker could leave its primary ENI in spinifex-vpc-enis as in-use/attached, and a dangling ENI throws DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two gaps: TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped any worker owned by another node, stranding both the VM and its ENI on multi-node clusters. Route the terminate per-instance via ec2.cmd.<id> (mirroring the gateway TerminateInstances path) so the owning daemon runs the full teardown — detach + force-delete the primary ENI and clear its record. No responder / NotFound payload is treated as already-gone so a retried DeleteNodegroup stays idempotent. deleteInstanceENI (stopped-instance terminate) issued a force=false DeleteNetworkInterface with no prior detach: a stale in-use attachment made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind after the instance was already written to the terminated bucket. Detach first, then delete, tolerating NotFound. * fix: heal volume State drift on detach/terminate teardown A detach/terminate could leave a volume's State empty instead of 'available'. DeleteVolume gated on State=='available' exactly, so it returned VolumeInUse for an unattached volume nothing was using, leaving it undeletable and blocking stack teardown. DescribeVolumes cosmetically rendered an empty State as 'available', masking the drift. - DeleteVolume now treats an empty State with no attachment as deletable (an attached volume still returns VolumeInUse). - DescribeVolumes derives an empty State from the attachment (attached -> in-use, unattached -> available) instead of blindly rendering available, so an empty-but-attached volume is no longer hidden. - UpdateVolumeState normalizes an unattached volume with an empty state to available, so a detach writeback that omits the state cannot strand the volume in drift. * fix: break all SG cross-references on EKS teardown DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked only ingress (IpPermissions), never egress (IpPermissionsEgress), so a cross-reference via an egress rule kept the SG pinned on DependencyViolation. It also resolved only the two named cluster SGs, so any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG, the private-endpoint SG, user/TF SGs — was never revoked and pinned the referenced SG on delete, leaving the VPC stuck. Enumerate every non-default SG in the VPC and revoke BOTH ingress and egress on each before deleting, breaking any cycle order-independently, then delete only the cluster-owned SGs (referrer SGs are deleted by their own teardown). The default SG is left untouched. Since DeleteClusterSGs runs against both the customer VPC and the managed control-plane VPC, the broadened sweep covers both teardown paths. * feat: support TargetGroup ProtocolVersion in elbv2 CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target groups to HTTP1) and DescribeTargetGroups/Create output return it. Without this the aws-load-balancer-controller sees a perpetual immutable-field mismatch (desired HTTP1 vs empty) and recreates the target group every reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it omitted to match AWS. * fix: omit IRSA env from ebs-csi addon when no role staged The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}} plus the web-identity token volume. With no service-account role staged the placeholder rendered empty, half-arming the SDK web-identity provider, which panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}} markers like the LBC bundle so the IRSA env and token volume are omitted when no role is staged, falling back to node instance-profile credentials over IMDS. The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes. Requires an eks-node AMI rebake to take effect. * docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup with host routing (app./argocd. on a wildcard cert) instead of one ALB per ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers need (Spinifex wires controller creds at the node level, not IRSA) and an ignore_changes block for the cluster's access_config and vpc_config security_group_ids, which Spinifex DescribeCluster does not echo back, so the provider stops trying to replace the cluster on every plan. * fix: stop addon-sync substituting IRSA markers named in comments The ebs-csi controller manifest's header comment spelled out the literal {{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync matched each marker by substring, so the comment lines were replaced with the env/mount blocks, injecting a bare YAML sequence between two documents. K3s' auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal array into unstructured), so the ebs-csi Addon CR never applied and the EKS addon hung in CREATING. Reword the comment to not contain the literal tokens, and anchor the marker substitution to lines that are exactly the marker (whitespace aside) so a future comment mentioning a marker can't retrigger it. * fix: disable K3s local-storage on EKS clusters in parity mode In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server disabled traefik and servicelb but left local-storage enabled. Its local-path provisioner ships a StorageClass marked default, which races the EBS CSI ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind nondeterministically. EKS has no local-path, so disable local-storage too and let ebs-gp3 be the sole default. The built-in-ingress path is unchanged. * fix: re-encrypt HTTPS target groups in haproxy ALB backends An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's own check then spoke HTTP to a TLS port, failed, and marked the server DOWN, returning 503 — even though the lb-agent's separate HTTPS health probe reported the target healthy, so describe-target-health masked the fault. Emit check-ssl ssl verify none on the server line for HTTPS target groups so both the check and proxied traffic re-encrypt to the backend. * fix: reconnect wedged QMP client on next command A timed-out QMP decode under host contention left the cached client's shared json.Decoder mid-message, wedging every later command and breaking EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead on encode/decode error and redial the retained socket path (re-running the qmp_capabilities handshake in place) on the next send. * fix: round-trip security-group rule descriptions SG ingress/egress rules dropped the per-rule Description on store and describe. The AWS Load Balancer Controller tags the worker-node SG rules it manages with a UserIdGroupPair description and identifies them by that value to revoke them on teardown. Without the round-trip the controller cannot match its rules, never revokes them, and the backend security group stays referenced — DeleteSecurityGroup returns DependencyViolation forever, hanging the ingress finalizer and tofu destroy with orphaned security groups. Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not identity) and render it back through describe. * fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown In the managed control-plane VPC topology, launchNodegroupInfra creates the cluster control-plane and nodegroup security groups in the customer VPC for worker<->control-plane networking, but purgeClusterInfra only reclaimed the security groups in the managed CP VPC. The customer-VPC SGs orphaned, cross-referenced each other, and pinned the customer VPC with DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the managed-CP-VPC branch too, matching the legacy customer-VPC branch. * fix: reclaim ALB EIP when system VM already gone on terminate TerminateSystemInstance only released an internet-facing ALB's Elastic IP on the local path, where the VM is still present in the manager. When the VM was already gone (health-reaped, crashed, or raced by ENI teardown), it took the remote/no-owner path and returned idempotent success without reconciling the EIP, leaking the wan-pool address still associated to a deleted ENI. Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic scan that disassociates then releases any EIP recorded against an instance, and call it as a backstop in TerminateSystemInstance for both branches. * fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook The demo Application carried no resources finalizer, so tofu destroy removed the Application CR without pruning the synced PVC. The PVC survived, the cluster was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the resources-finalizer.argocd.argoproj.io finalizer so deleting the Application cascades to the PVC and ebs-csi reaps the volume before cluster teardown. * feat: raw-IP access + documented Argo CD creds in eks-gitops workbook The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password command as a tofu output and document the credential + IP URLs in the README. * fix: reclaim ALB EIP on the owning-node terminate path The EIP backstop lived only in TerminateSystemInstance, but the ALB delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's cached allocation which is often empty. Extract reclaimSystemInstanceEIP (authoritative KV reconcile via ReleaseAddressByInstanceID) and call it from terminateSystemInstanceLocal as well as the no-owner backstop, so an internet-facing ALB's EIP is released on every terminate route. * chore(eks): drop coredns from the managed addon catalog Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as a managed addon — there is no bundled coredns manifest, so CreateAddon coredns would stage but fail VM-side render. Remove the phantom catalog entry so DescribeAddonVersions and CreateAddon no longer advertise it; repoint the addon tests that used coredns as their sample to argocd. * feat(ui): group EKS and ECR under a Containers nav section Fold the separate EKS and ECR sidebar groups into one Containers group with EKS (clusters) and ECR (repositories) entries. * fix(ui): satisfy strict null checks in ECR component tests Index mock.calls via optional chaining and narrow the getAllByRole result before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix. * refactor(eks): hide internal fixture addons from the public catalog Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays creatable (lookupAddon finds it, so the delivery-transport e2e still drives it) but no longer surfaces in the unfiltered DescribeAddonVersions listing. * refactor(eks): remove K3s built-in ingress; LBC-only Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik, servicelb, and local-storage are now always disabled on EKS nodes (AWS parity — Service type=LoadBalancer and Ingress are the Load Balancer Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in the state reporter. * fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation Parallelize the independent subnet-create and route-table-association loops in the create-VPC mutation with Promise.all (each task records its resource in the rollback list before any further call), and switch the CIDR-validation regex to named capture groups. Clears the no-await-in-loop and prefer-named-capture-group lint errors. * fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal The ECR registry is served on the gateway port (:9999), but proxyEndpoint and repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and failed. Derive the port from the awsgw listen host and append it to the registry host (omitted for 443). Also rename the default internal DNS suffix. * docs: drop verbose ECR :9999 host note * docs: fix literal-asterisk emphasis (single-* not rendered) * docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust * feat: default ECR registry URIs to the gateway host (no-DNS interim) The registry authenticates by token, not hostname, so the per-account <acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port (already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need no DNS; the parity host still works server-side where DNS exists. * fix: serve ECR registry on a DNS-free gateway host end to end Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable host:9999 carried in the server cert SANs rather than the unresolvable per-account parity name. Docker push then dials it directly with no DNS. Mirror the same host on EKS workers: the registries.yaml mirror keys and the credential provider matchImages now cover the port-less parity name, the port-ful parity name, and the gateway IP:9999, all redirecting to the dialed gateway endpoint. A port-ful pull from the API-returned repository_url matches the mirror instead of bypassing it into a failed DNS lookup. Net: docker push host, pod image host, and the API's repository_url are the same value, with no DNS or /etc/hosts. * docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS) * docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS) * docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks	2026-06-23 15:29:14 +10:00
Josh Siviour	9d662f9889	feat: EKS EBS CSI driver (Viperblock) — egress, attach, node discovery, e2e (#402) ... * feat: bake aws-ebs-csi-driver addon + mdev by-id bridge Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet, default gp3 StorageClass) under the eks-node image. The node plugin locates EBS volumes via /dev/disk/by-id, which busybox mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk serial, plus a boot oneshot that points the kernel hotplug helper at mdev and coldplugs volumes present at boot. * test: e2e EBS CSI volume — PVC bind + mount + reschedule survival Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver managed addon bound to an IRSA role, provisions a default gp3 PVC + pod, and asserts a marker written by the first pod survives a reschedule onto a fresh pod backed by the same PVC. Exercises the full data path: CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount. * feat: eks-node provider-id + topology/instance-type labels for EBS CSI mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in from IMDS before K3s starts: kubelet provider-id plus the region, zone and instance-type node labels. The aws-ebs-csi-driver node plugin reads this metadata for NodeGetInfo (instance id from providerID, region/zone from topology labels, instance type for the volume-limit family lookup). k3s runs no AWS cloud-provider so none of it is set otherwise, which left the node plugin's metadata init failing, NodeGetInfo panicking in IsNitroInstanceType, and the CSINode carrying no topology key. Wire the service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and chmod it in setup. * test: e2e EBS CSI controller IRSA policy + failure diagnostics awsgw enforces IAM on assumed-role EC2 calls, so the controller's CreateVolume/AttachVolume need a permission policy on the IRSA role. Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and grant nothing) covering the EBS CSI action set; CreateVolume then succeeds. Add an OnFailure dump of the CSI control path (provisioner and plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume subtest so provisioning/attach failures are diagnosable post-teardown. * feat: EBS CSI obeys control-plane taint; e2e worker nodegroup Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI controller + node manifests so the driver runs only on customer worker nodes, never the system control-plane VM. The e2e creates a 1-node worker nodegroup and the writer/reader pods drop their CP tolerations so the customer-owned volume attaches in the customer space. * fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback virtio-blk-pci hot-plug bus was derived from the AWS device-name letter (/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug — and the attach hung. A failed attach also left the per-volume iothread object behind, so every retry died on object-add 'duplicate property'. nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live query-block state, independent of the device name. delIothreadBestEffort removes the iothread on every attach rollback path. * fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link Alpine eks-node runs busybox mdev (no eudev), which stops at the first matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>. Swap the stock vd* rule in place to route through mulga-ebs-byid, which now delegates to persistent-storage (preserving stock links) before minting the nvme-style link the node plugin expects. * test: EKS e2e — NAT Gateway egress for private worker nodegroup The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS: an IGW serves only public-IP instances), so an IGW-only fixture left the worker unable to reach the public NLB endpoint or pull public CSI images — the nodegroup never joined. createWorkerEgress now builds the real topology: an IGW-fronted public subnet hosting a NAT Gateway, with the worker subnet default-routed to it. Associating the worker route table triggers the subnet-wide SNAT egress reroute. Teardown drains the NAT Gateway before releasing the public subnet + EIP it pins.	2026-06-19 14:43:20 +10:00
Josh Siviour	2acb37342d	feat(eks): aws-load-balancer-controller addon — real ALB Ingress (L7, IRSA) (#376) ... * fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ info ("cannot resolve AZ info for AZs: []") and never built the ALB model. Stamp the single zone-id on every subnet and share it with DescribeAvailabilityZones through one SingleZoneID constant so the two responses can never drift. * feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle Baked manifests for the LBC managed addon (CRDs, controller Deployment/ RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two wirings make it work against the spinifex gateway: - --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/ acm. The aws-sdk-go v1 credential bootstrap resolves the STS client from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the web-identity AssumeRoleWithWebIdentity must be pointed at the gateway here or it falls back to real AWS STS. - --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet count from two to one so an ALB provisions in a single-AZ deployment. * feat(eks): render IRSA into managed addons + node-minted webhook cert Extend the addon-sync renderer to inject the projected-token / role-arn / gateway STS-endpoint + CA block into managed-addon pod specs via opt-in markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl, so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the admission webhook, cached on disk so the render is byte-stable across ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES for the LBC bundle. * feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller Thread the control-plane VPC id through K3sServerInput.VpcID into the first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller can pass --aws-vpc-id to the gateway elbv2/ec2 handlers. * fix(eks): point the LB controller at the customer VPC + customer-facing gateway The aws-load-balancer-controller ran with two control-plane-scoped values that customer worker pods cannot use: - --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID), which the gateway's DescribeVpcs does not return for customer calls, so backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers and their subnets live and where the ALB is provisioned. - The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL / SystemGatewayURL) into pod specs, unreachable from worker pods. Add a distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own bootstrap, state-report and token-webhook paths. * fix(elbv2): resolve forward target group from ForwardConfig listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a forward action carrying its target group via ForwardConfig.TargetGroups (the shape the aws-load-balancer-controller emits) left the action target empty and CreateRule/CreateListener rejected it with MissingParameter. Flatten the first target group from ForwardConfig when the flat field is absent. * fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records The generic S3 tag store and the per-resource KV records are disjoint: DescribeSubnets/DescribeVpcs read tags from their KV record, while CreateTags only wrote the generic store. Subnet tags applied after create were therefore invisible to tag-filtered describes, breaking aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb). Write through CreateTags/DeleteTags to the owning subnet/vpc record so tag-filtered describes observe them. Resource types the VPC service does not own are skipped; the generic store stays their record of truth. * test(daemon): pin synthetic capacity in host-dependent allocate tests TestCanAllocate_CountEdgeCases/Mixed_instance_types and TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation called rm.allocate against live-detected host capacity. allocate gates on readMemAvailableGB (live /proc MemAvailable), so a box already running VMs fails the allocation spuriously, breaking the preflight gate locally. Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic host, matching the sibling Capacity_decreases_after_allocation subtest. * fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster) In the managed-CP topology the apiserver VM sits in the system CP VPC with no route to the worker pod network in the unpeered customer VPC. k3s default egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl logs/exec work but apiserver->pod traffic (admission webhooks, aggregated APIs) is attempted directly and times out -- the aws-load-balancer-controller webhooks fail with context deadline exceeded, blocking Service/Ingress creation cluster-wide. Set egress-selector-mode=cluster so apiserver->pod/service traffic routes through the agent's outbound tunnel, selected by the node hosting the destination, without requiring inbound CP->worker reachability. * feat: bump aws-load-balancer-controller addon bundle to v2.11.0 v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2) and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true. Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings), controller manifest, and image tag v2.11.0, re-applying the spinifex deltas (IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig, alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1. * fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2) LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase names the bundle passed. Those overrides were silently ignored, so every non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the gateway CA -- no ALB could provision. Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for STS) and drop the dead --aws-api-endpoints flag from the controller args. * fix: synthesize taggable ARN for default listener rule Gateway DescribeRules returned the synthetic IsDefault rule with a null RuleArn. LBC read the arn-less default rule after creating a listener and issued DescribeTags with an empty resource ARN, which the gateway rejected with 400 MissingParameter, aborting reconcile. Set a deterministic RuleArn on the default rule, resolve it back to its parent listener in DescribeTags (empty tags) and DescribeRules by-arn. Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls. * feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts Ingress reconcile. Only the explicit-subnet path honors the gate. Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path. An empty set falls back to auto-discovery, preserving the native UX. * fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams Live verification showed two gaps in the LBC addon bundle: LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS). Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs (EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the web-identity STS client still resolves via AWS_ENDPOINT_URL_STS. The injected IngressClassParams was orphaned: the alb IngressClass had no .spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a single-AZ cluster). Link the IngressClass to the IngressClassParams so the explicit-subnet path is taken. * fix: stamp EKS-parity providerID on worker nodes for LBC ip targets k3s registers workers with providerID k3s://<name>, which the AWS Load Balancer Controller rejects (providerID invalid for EC2 instances) when resolving nodes for ip-mode target registration. Resolve the instance-id and AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id> so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps the default providerID and still starts. * fix(elbv2): honor SubnetMappings in CreateLoadBalancer LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list. createLoadBalancer only read input.Subnets, so ENI allocation was skipped, VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the ALB went active but could not serve traffic. Add flattenSubnetIDs to read both sources and reuse it in createLoadBalancer and desiredSubnetSet. * fix(vm): capture GCInterval before spawning GC goroutine TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global GCInterval and restores it in a defer, racing the Start goroutine's read at time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's goroutine into a local before spawning so the global access is ordered with the test's defer; the goroutine touches only the captured local.	2026-06-18 10:53:05 +10:00
Josh Siviour	bc9556d1a9	fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368) ... * fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working.	2026-06-16 20:27:21 +10:00
Josh Siviour	b23fa3bf60	feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356) ... * feat(eks): VM-side managed-addon delivery transport Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/ {accountId}), renders the baked bundles into the K3s auto-deploy dir, and reports per-addon delivery status on a dedicated eks.addon.*.status channel that the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE). - gateway: ListInternalAddons route + ListStagedAddonManifests NATS action - handlers: addon status reconciler (nextAddonStatus), staging installer - eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent - agent renders idempotently (cmp before replace) so K3s auto-deploy does not re-apply every tick and clear the Addon CR .status.gvks readiness signal - agent mirrors per-tick delivery outcomes to the serial console for host-side observability * fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP The cluster NLB front-end IP is assigned by the async LB-VM launch (launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after CreateLoadBalancer returned — losing the race and falling back to the synthetic <lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is not in the apiserver serving-cert SAN, so kubectl against the cluster never connects. Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an inline multi-minute VM boot head-of-line-blocks the gateway responder), and a new CreateLoadBalancerSync drives the data-plane launch inline and returns the LB carrying its front-end address. The EKS cluster-create worker runs off the gateway responder, so it uses the sync entry and reads the IP from the return value instead of racing persisted state. No polling. - elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync - eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result, drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable) when no front-end IP is allocated rather than shipping the DNS-name fallback - eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP * feat: auto-wire IGW for internet-facing EKS cluster endpoint An internet-facing cluster NLB allocates an external front-end IP and the OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP that make that IP answerable on the wire are built only on IGW attach. The control-plane NLB is a system instance, so ensure an IGW on cluster build and reclaim it on teardown. EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership- scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the VPC delete. New IGW dep wired from the daemon. * fix(eks): tear down infra on failed cluster launch to stop LB EIP leak A CreateCluster that fails during its background launch flipped the meta to FAILED but performed no teardown, relying on a later same-name create to reclaim and purge the infra. EKS cluster names are unique per create, so that reclaim never fires: the internet-facing load-balancer VM keeps running with its allocated+associated dnat_and_snat EIP indefinitely. Over repeated create/delete cycles the EIP pool fills with allocations bound to terminated instances, so a later allocation's advertised endpoint and its live OVN NAT diverge onto different pool addresses. - failClusterLaunch now runs a best-effort purgeClusterInfra on the in-memory meta (freshest refs even when the failing step was the meta write itself) before MarkClusterFailed. purgeClusterInfra is idempotent, so a later reclaim or DeleteCluster re-purge is a no-op; the FAILED meta is retained for observability. - Daemon releaseSystemInstanceEIP extracted and called from both TerminateSystemInstance and cleanupFailedSystemInstance, so a launch that fails after AssociateAddress releases the eipService allocation instead of leaking it (MarkFailed's ReleasePublicIP only knows the externalIPAM path). * feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs CreateCluster now auto-builds the managed control-plane VPC under the system account, the spinifex analogue of AWS EKS's hidden managed-account VPC: a public subnet for the internet-facing NLB, private subnet(s) for the k3s control-plane VM(s), and a NAT gateway giving the private CP egress for image pulls. It is composed from the real EC2 VPC-family APIs (CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress gating is wired by the existing topology subscribers off the route-table events those APIs publish — no direct OVN mutation. This resolves the mulga-siv-292 root cause: the internet-facing NLB and the private CP were crammed into one customer subnet with no IGW route, so both were treated as private and dropped by the lr_in_policy 1100 rule, killing the LB-VM return leg and lb-agent heartbeat. The CP only survived via a per-instance 1200 reroute bandaid, now removed — the topology expresses egress instead. Placement is uniform single/multi-node (CP always in the private subnet). The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables, subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A / worker side) remain consumed-not-created and are deferred to the worker round. * fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID The managed control-plane VPC (Set B) launches the K3s control-plane VMs and their ENIs under the system (infra) account, but the on-VM agents — bootstrap publish, state report, token-review, and add-on fetch — must key on the cluster-OWNER (customer) account that owns the cluster meta, namespaces, and AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches under. Without this the VM reported state and fetched add-ons against the system account, so the cluster never observed a healthy control plane (stuck CREATING, null CA) and kubectl/add-on delivery timed out. Also refresh the e2e suite doc comment to describe the Set B topology and the data-plane subtests (kubectl + managed-addon delivery) that now run. * fix(eks): explicitly delete addon objects on unstage K3s does not delete the applied resources when an auto-deploy manifest file is removed from the server manifests dir (documented behaviour). The addon-sync GC pass relied on that, so after DeleteAddon the rendered manifest was dropped but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait timed out. Copy the manifest aside, drop the original so K3s stops tracking it, then kubectl delete the objects (--wait=false to avoid finalizer stalls).	2026-06-12 22:29:24 +10:00
Josh Siviour	74b91b5cc5	feat(eks): HA control plane — 3-node etcd quorum spread across distinct hosts (#335) ... * feat(eks): node-targetable control-plane VM (HA spread transport prereq) Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s control-plane VM placeable on a specific Spinifex host. - Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM as it instead of t3.medium. A sys.* type keeps the CP VM out of customer DescribeInstanceTypes and makes the daemon register the node-targeted system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it). - Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on the launcher: empty/local node launches in-process (unchanged single-CP path); any other node dispatches over the node-targeted subject, where the receiving daemon runs the launch and owns the VM. The spread orchestrator (reserve N hosts, launch per host, verify, rollback, <3-host fallback) lands separately; CreateCluster still launches locally (TargetNodeID empty), so behaviour is unchanged. * feat(eks): HA control-plane spread orchestrator Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster spread placement group, with all-or-nothing rollback and single-CP fallback. - placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate -> ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) -> parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure terminates launched VMs, releases the reservation, drops the group. - <3 schedulable hosts or a lost reservation race falls back to a single local control-plane VM (prior behaviour). - ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster tears down every CP VM and the spread group. - Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate. * fix(eks): size HA control-plane spread on raw host headroom SchedulableHosts matched the CP type by name in node.status's per-type capacity list, but that list omits sys.* types (system types are hidden from customers). The CP VM is sys.medium, so the match never hit and SchedulableHosts returned 0 on every node — placeControlPlane always fell back to a single CP, making the >=3-host spread unreachable in production. System VMs consume host vCPU/memory like any guest, so size the fit against each node's raw schedulable headroom (Total - Reserved - Alloc) and the type's footprint instead. Adds instancetypes.SpecForSystemType for the sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper. * feat(eks): k3s server-join role for HA control-plane quorum Servers 2..N now boot in join mode: server: https://<first-server>:6443 + a shared cluster token, without cluster-init, so they join the first server's embedded-etcd quorum instead of each cluster-initing an independent datastore. - K3sServerInput gains JoinToken (shared cluster secret, set on every server) and ServerURL (set => join server, no cluster-init, server-join role). - buildK3sUserData branches cluster-init vs server:, emits token:, and selects SPINIFEX_K3S_ROLE server vs server-join. - launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the join URL from its ENI IP, then launches the rest in parallel as join servers; a first-server failure skips the joins and rolls back. - CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken) threaded into the placement template. - eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report, but not k3s-first-boot (the first server already publishes the bootstrap artifacts; a join re-publish only races the bootstrap bus). * fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join Join servers (231.7.2) register over the supervisor port 6443 but their embedded etcd could not peer with the quorum: the control-plane SG only admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and kubelet 10250 between control-plane nodes. A single cluster-init server never exercises those ports, so the gap stayed hidden until multi-server HA — the join registered but its etcd never replicated the raft log, leaving the node perpetually NotReady. Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for etcd 2379-2380 and kubelet 10250, called from CreateCluster after EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers reach Ready and form a 3-member etcd quorum that survives loss of the original first server. * feat: register all control-plane ENI IPs as EKS cluster NLB targets Register every HA control-plane server ENI IP in the cluster NLB target group, not just the primary. The TG already health-checks :6443, so a dead member drains and kubectl/apiserver traffic fails over to surviving etcd-quorum members. Add RegisterClusterTargets batch helper; the single RegisterClusterTarget now delegates to it.	2026-06-10 12:25:47 +10:00