mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
dev
15 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
36d5788923 | fix: require iam:PassRole for EKS cluster and node roles (#1139) | ||
|
Julian Sommer
|
2b4ddad78a |
chore: drop dead and redundant nolint directives
Removes directives for linters that are not enabled (noctx, exhaustive, deadcode) and gosec suppressions already covered by config exclusions. Tags ochrevector QueryResult and Filter with their existing field names so musttag passes without changing the NATS wire format, and bounds the RDS window slot conversion instead of suppressing G115. |
||
|
Ben Duncan
|
18853cef31 |
fix: close the four defects keeping the e2e nightly red (#1007)
* fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main. |
||
|
Josh Siviour
|
a3fb4cccec |
fix(eks): backoff and terminal state on delete re-drive, CSI volume reclaim reporting, independent e2e subtests (#642)
* fix: bound EKS DELETING re-drive with backoff and a terminal give-up reapCluster re-drove purgeClusterInfra on every GC tick with no attempt tracking, so a permanently-failing teardown (e.g. an unretriable DependencyViolation) was retried forever. Track attempt count and last attempt time on ClusterMeta, back off exponentially between re-drives, and give up after maxDeleteReapAttempts with a single error-level log. Status stays DELETING on give-up so the cluster's infra remains tracked and billable pending operator intervention, preserving the ADR-0006 §6 billable-before-sweep invariant. * fix: create the EKS e2e node role independently in each subtest runEBSCSIVolume string-built the worker node role ARN without ever creating it, only passing because runIRSAPod's t.Cleanup discarded DeleteRole's error while the role was still attached to its instance profile, leaving it behind for EBSCSIVolume to reuse by naming convention. Extract ensureNodeRole and call it independently from both subtests, detaching the role from its instance profile before deleting it and asserting that error instead of discarding it. * fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete purgeClusterInfra tore down the control plane, nodegroups, and networking but never looked at EBS volumes, so any volume the in-cluster CSI driver had provisioned for a PVC was left behind untracked and unbilled for. Add a report-only reclaim step that scans for volumes tagged with the cluster's CSI owner tag and logs them for operator follow-up. Volumes are never auto-deleted: a PV's reclaim policy (Delete vs Retain) is a Kubernetes-side property the CSI driver does not tag onto the EC2 volume, so there is no safe way to tell from tags alone whether deleting a given volume is intended. This mirrors the volume-leak reaper's existing mark-and-alarm policy for the same reason. The cluster-owner tag requires the EBS CSI controller's --k8s-tag-cluster-id flag, added here to the controller manifest; without it a CSI volume carries no cluster-scoped tag and reclaim would have to guess across every cluster sharing the account. * fix: persist the last purge error on a wedged DELETING cluster DeleteReapExhausted told an operator a cluster's teardown had given up, but not why: nothing recorded the failing purge's error. Persist it on every failed re-drive, not only once the reaper exhausts its attempts, so it is visible on a still-retrying cluster too. Status stays DELETING throughout, preserving the billing invariant that untorn-down infra remains tracked. * fix: make the EKS e2e suite confirm its CSI volume is actually deleted The EBSCSIVolume subtest deleted its PVC on cleanup but never confirmed the underlying volume was reclaimed, so a CSI driver that silently failed to delete it would leak a volume with nobody noticing. The suite provisions and owns this volume, so unlike production teardown it can safely wait on its StorageClass's Delete reclaim policy: capture the CSI volumeHandle while the PVC is bound, then have cleanup poll EC2 for that exact volume to disappear before the addon/nodegroup that run the CSI controller are torn down. |
||
|
Josh Siviour
|
6873875248 |
fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634)
On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane. |
||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Josh Siviour
|
d3ac8c0656 |
test(eks): in-cluster IRSA pod e2e (AssumeRoleWithWebIdentity) (#535)
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN / AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the addon-sync agent injects — and asserts the pod's in-cluster aws sts get-caller-identity resolves to the assumed IRSA role. This closes the gap left by IRSAWebIdentity, which only exchanges the token host-side. Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a 1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted) and reaches awsgw STS over the worker subnet NAT. |
||
|
Josh Siviour
|
9f349e86ca |
feat(eks): expose NVIDIA GPUs to pods (Epic B) — VFIO QMP, worker registration, device-plugin GPU access (#507)
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%). |
||
|
Josh Siviour
|
9d662f9889 |
feat: EKS EBS CSI driver (Viperblock) — egress, attach, node discovery, e2e (#402)
* feat: bake aws-ebs-csi-driver addon + mdev by-id bridge Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet, default gp3 StorageClass) under the eks-node image. The node plugin locates EBS volumes via /dev/disk/by-id, which busybox mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk serial, plus a boot oneshot that points the kernel hotplug helper at mdev and coldplugs volumes present at boot. * test: e2e EBS CSI volume — PVC bind + mount + reschedule survival Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver managed addon bound to an IRSA role, provisions a default gp3 PVC + pod, and asserts a marker written by the first pod survives a reschedule onto a fresh pod backed by the same PVC. Exercises the full data path: CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount. * feat: eks-node provider-id + topology/instance-type labels for EBS CSI mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in from IMDS before K3s starts: kubelet provider-id plus the region, zone and instance-type node labels. The aws-ebs-csi-driver node plugin reads this metadata for NodeGetInfo (instance id from providerID, region/zone from topology labels, instance type for the volume-limit family lookup). k3s runs no AWS cloud-provider so none of it is set otherwise, which left the node plugin's metadata init failing, NodeGetInfo panicking in IsNitroInstanceType, and the CSINode carrying no topology key. Wire the service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and chmod it in setup. * test: e2e EBS CSI controller IRSA policy + failure diagnostics awsgw enforces IAM on assumed-role EC2 calls, so the controller's CreateVolume/AttachVolume need a permission policy on the IRSA role. Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and grant nothing) covering the EBS CSI action set; CreateVolume then succeeds. Add an OnFailure dump of the CSI control path (provisioner and plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume subtest so provisioning/attach failures are diagnosable post-teardown. * feat: EBS CSI obeys control-plane taint; e2e worker nodegroup Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI controller + node manifests so the driver runs only on customer worker nodes, never the system control-plane VM. The e2e creates a 1-node worker nodegroup and the writer/reader pods drop their CP tolerations so the customer-owned volume attaches in the customer space. * fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback virtio-blk-pci hot-plug bus was derived from the AWS device-name letter (/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug — and the attach hung. A failed attach also left the per-volume iothread object behind, so every retry died on object-add 'duplicate property'. nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live query-block state, independent of the device name. delIothreadBestEffort removes the iothread on every attach rollback path. * fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link Alpine eks-node runs busybox mdev (no eudev), which stops at the first matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>. Swap the stock vd* rule in place to route through mulga-ebs-byid, which now delegates to persistent-storage (preserving stock links) before minting the nvme-style link the node plugin expects. * test: EKS e2e — NAT Gateway egress for private worker nodegroup The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS: an IGW serves only public-IP instances), so an IGW-only fixture left the worker unable to reach the public NLB endpoint or pull public CSI images — the nodegroup never joined. createWorkerEgress now builds the real topology: an IGW-fronted public subnet hosting a NAT Gateway, with the worker subnet default-routed to it. Associating the worker route table triggers the subnet-wide SNAT egress reroute. Teardown drains the NAT Gateway before releasing the public subnet + EIP it pins. |
||
|
Julian Sommer
|
2e9501524b | feat(e2e): more e2e coverage (#365) | ||
|
Josh Siviour
|
b23fa3bf60 |
feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356)
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
|
||
|
Julian Sommer
|
50b5495d26 | chore: more comment cleanup | ||
|
Josh Siviour
|
4ef19fc488 | feat(ui): EKS console (#310) | ||
|
Josh Siviour
|
e44221d653 |
feat(eks): public/private endpoint access + publicAccessCidrs-driven NLB scheme & SG (#315)
* feat(eks): public/private endpoint access drives NLB scheme + IP:port reachability Parse resourcesVpcConfig.endpointPublicAccess/endpointPrivateAccess/ publicAccessCidrs (defaults public=true, private=false, cidrs=0.0.0.0/0) and reject the both-disabled combination. Public access provisions an internet-facing NLB and publishes its external frontend IP as the cluster endpoint; private access keeps the NLB internal and uses the VPC frontend IP. Inject the resolved endpoint IP into the k3s apiserver tls-san so the published IP:443 endpoint serves a cert valid for verified TLS, and surface the access fields in DescribeCluster. The e2e suite now connects via the verified endpoint with TLS verification on, dropping the insecure mgmt-IP shim. * fix(eks): authorize NLB->apiserver ingress on control-plane SG The cluster control-plane security group was created with empty ingress, so the NLB's backing LB VM could not reach the apiserver on 6443 from inside the VPC. The target group stayed unhealthy and the published endpoint never served. Authorize tcp/6443 on the control-plane SG from the VPC CIDR after the SGs are created; public exposure remains gated at the NLB front-end by publicAccessCidrs, so this only opens the in-VPC LB->apiserver hop. Extend SubnetVPCResolver with GetVPCCIDR so the service can resolve the VPC CIDR for the rule. * feat(eks): thread publicAccessCidrs to NLB SetLoadBalancerIngressCIDRs Wire the cluster's publicAccessCidrs (parsed from CreateCluster ResourcesVpcConfig) through EnsureClusterNLB to the ELBv2 SetLoadBalancerIngressCIDRs override so a public cluster endpoint can be narrowed below the wide-open scheme default. The override only fires for an internet-facing NLB whose CIDRs restrict below 0.0.0.0/0; internal NLBs and the default keep the LB's out-of-the-box ingress. |
||
|
Josh Siviour
|
496424c41c |
test(eks): EKS control-plane E2E suite (Sprint 6g) (#304)
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
|