mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
53b31cb76ca12f8a48288132bee97d30a8348156
271 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
53b31cb76c | fix: keep the RDS master password out of the server and console logs (#754) | ||
|
Josh Siviour
|
d178ef5992 |
feat: version-aware host-asset preflight for binary-swap deploys (#750)
A binary-swap deploy (make deploy or a manual spx copy) ships a new binary whose host-config prerequisites — the privileged helper scripts and their sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship. The node looks healthy and the gap surfaces later as a cryptic launch failure or a silent security regression. Add a preflight package that checks the host against what this build expects, comparing asset content, not existence, so a stale-but-present asset is caught too. The managed helpers install from quoted heredocs, so their on-disk bytes equal the heredoc body verbatim; the binary carries the canonical sha256s and compares. - spinifex/preflight: CheckHost / CheckHostAt, content hash of the two managed helpers and a NOPASSWD grant-line check for the sysctl helper. - Generated canonicalHashes plus a drift guard that re-extracts the heredocs and fails if the committed manifest no longer matches. - Three surfaces: spx admin preflight, a make deploy gate (SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning. |
||
|
Julian Sommer
|
3379b16586 | fix: encrypt and acknowledge the RDS bootstrap password (#739) | ||
|
Josh Siviour
|
272b4b22e8 |
feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740)
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
|
||
|
Julian Sommer
|
1c4d41bffc | fix: reach the DB endpoint from any subnet of the VPC (#738) | ||
|
Julian Sommer
|
e3b5701f15 | fix: authorize RDS data-volume formatting by exact volume identity (#737) | ||
|
Julian Sommer
|
8c07f80032 | chore: remove the half line breaks | ||
|
Julian Sommer
|
5dd3af70eb | fix: rds handoff path | ||
|
Julian Sommer
|
615578a2e7 | refactor: simplify RDS internals (#692) | ||
|
Julian Sommer
|
2e09461080 | chore: cleanup code comments | ||
|
Julian Sommer
|
050dbdb060 | fix: disallow reserved master names during rds password apply | ||
|
Julian Sommer
|
970bf18ab5 | fix: guard a half bootstapped datadir against crashes | ||
|
Julian Sommer
|
faa0519cf1 | fix: the RDS master user is no longer a PostgreSQL superuser (#687) | ||
|
Julian Sommer
|
9dfef26e48 | fix: rds-init | ||
|
Ben Duncan
|
b76aab8b50 | feat(installer): three-plane networking, ZFS root, distributed NAT and multi-node install (#663) | ||
|
Julian Sommer
|
5e17169661 | feat(e2e): complete the RDS end-to-end suite (#608) | ||
|
Josh Siviour
|
a2c451128d |
fix: unblock the Ubuntu 26.04 installer (qemu-kvm, sudo-rs) (#665)
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two providers there, so apt refuses to resolve it; the token was redundant anyway since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs the qemu-system-* binaries directly. sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it implements no pam_session Defaults setting, and forbids a wildcard inside a command argument. pam_session is now emitted only for classic sudo, and the sysctl grant names a fixed-verb helper instead of a wildcard. The helper is narrower than the rule it replaces: it accepts one interface, a key restricted to rp_filter or accept_local, and a value restricted to 0 or 1, where the old grant permitted any net.ipv4.conf key at any value. Granting sysctl with an unconstrained argument was rejected as root-equivalent. Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime package list is resolved against each supported suite's real indices in CI. |
||
|
Josh Siviour
|
9a1fb44065 |
fix: CreateImage live drain + Alpine .raw sparsify (#647)
* fix(ec2): drain the attached volume before CreateImage reads a live checkpoint CreateImage against a running instance (--no-reboot) read the volume's checkpoint directly from predastore without draining in-flight writes first, unlike CreateSnapshot. A checkpoint captured before a guest- triggered write (e.g. growpart rewriting the GPT after a root volume is launched larger than the base image) could be stale, producing an AMI that boots to a UEFI shell instead of the OS. Extracts the existing snapshot-package drain logic into a shared, reusable DrainVolume so both live-snapshot call sites route through the same drain-before-read guarantee instead of being able to diverge. * test(e2e): boot-verify CreateImage AMIs instead of trusting State=available describe-images reporting State=available is not evidence an AMI boots — a well-formed but unbootable AMI (stuck at the UEFI shell) still registers cleanly. Extend the CreateImage e2e test to launch a probe instance from each produced AMI (running-path and stopped-path) and require a real SSH login, with console-dump-on-failure wired up for diagnosis. * fix(images): sparsify the Alpine guestfish .raw before conversion The Alpine build path grows the qcow2 in place via qemu-img resize + guestfish resize2fs, which can leave metadata scattered across the newly-grown region. virt-sparsify zeroes the guest's unused blocks before the raw conversion so they collapse into real holes, matching the Ubuntu virt-resize path's sparseness. |
||
|
Josh Siviour
|
dad5d702e1 |
fix: stop lbagent heartbeats on positive LB deletion, and actually enable the JetStream watchdog (#643)
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted A LoadBalancerNotFound response is the gateway's positive confirmation that the store has no record for the LB, not a connectivity failure — unlike a dial error, timeout, or 5xx, which leave its existence unknown and must keep retrying. Past the startup registration grace window (or after any prior successful heartbeat), it can no longer be the record's own replication race, so the agent now stops rather than retrying forever and filling the log with heartbeat noise after teardown. * fix(nats): enable the JetStream ENOSPC-latch watchdog timer nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself — that decision is nats-server's own internal state, not spinifex's, so a restart is the only way back. The spinifex-nats-watchdog.timer + service already probe /healthz?js-enabled-only=true and try-restart on a sustained failure, but the timer's WantedBy=timers.target never took effect: neither setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full disk needed a manual restart forever. Enable --now it alongside the rest of the unit set. |
||
|
Josh Siviour
|
a3fb4cccec |
fix(eks): backoff and terminal state on delete re-drive, CSI volume reclaim reporting, independent e2e subtests (#642)
* fix: bound EKS DELETING re-drive with backoff and a terminal give-up reapCluster re-drove purgeClusterInfra on every GC tick with no attempt tracking, so a permanently-failing teardown (e.g. an unretriable DependencyViolation) was retried forever. Track attempt count and last attempt time on ClusterMeta, back off exponentially between re-drives, and give up after maxDeleteReapAttempts with a single error-level log. Status stays DELETING on give-up so the cluster's infra remains tracked and billable pending operator intervention, preserving the ADR-0006 §6 billable-before-sweep invariant. * fix: create the EKS e2e node role independently in each subtest runEBSCSIVolume string-built the worker node role ARN without ever creating it, only passing because runIRSAPod's t.Cleanup discarded DeleteRole's error while the role was still attached to its instance profile, leaving it behind for EBSCSIVolume to reuse by naming convention. Extract ensureNodeRole and call it independently from both subtests, detaching the role from its instance profile before deleting it and asserting that error instead of discarding it. * fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete purgeClusterInfra tore down the control plane, nodegroups, and networking but never looked at EBS volumes, so any volume the in-cluster CSI driver had provisioned for a PVC was left behind untracked and unbilled for. Add a report-only reclaim step that scans for volumes tagged with the cluster's CSI owner tag and logs them for operator follow-up. Volumes are never auto-deleted: a PV's reclaim policy (Delete vs Retain) is a Kubernetes-side property the CSI driver does not tag onto the EC2 volume, so there is no safe way to tell from tags alone whether deleting a given volume is intended. This mirrors the volume-leak reaper's existing mark-and-alarm policy for the same reason. The cluster-owner tag requires the EBS CSI controller's --k8s-tag-cluster-id flag, added here to the controller manifest; without it a CSI volume carries no cluster-scoped tag and reclaim would have to guess across every cluster sharing the account. * fix: persist the last purge error on a wedged DELETING cluster DeleteReapExhausted told an operator a cluster's teardown had given up, but not why: nothing recorded the failing purge's error. Persist it on every failed re-drive, not only once the reaper exhausts its attempts, so it is visible on a still-retrying cluster too. Status stays DELETING throughout, preserving the billing invariant that untorn-down infra remains tracked. * fix: make the EKS e2e suite confirm its CSI volume is actually deleted The EBSCSIVolume subtest deleted its PVC on cleanup but never confirmed the underlying volume was reclaimed, so a CSI driver that silently failed to delete it would leak a volume with nobody noticing. The suite provisions and owns this volume, so unlike production teardown it can safely wait on its StorageClass's Delete reclaim policy: capture the CSI volumeHandle while the PVC is bound, then have cleanup poll EC2 for that exact volume to disappear before the addon/nodegroup that run the CSI controller are torn down. |
||
|
Josh Siviour
|
9c9bb9430c |
test: gate image shell scripts in CI and pin the spot vCPU quota ordering (#641)
* test: gate scripts/images/ shell suites + shellcheck in CI The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran by hand: spinifex.yml filters pushes on Go-only paths and no job invoked them, so a broken helper could land unnoticed. Add a `make test-images` target and a path-filtered image-scripts.yml workflow so both run automatically on scripts/images/** and images/** changes, kept separate from preflight so image-asset churn doesn't run on every Go contributor's commit. Also clears the shellcheck findings that would have started this gate red (SC1090 on a runtime-resolved source path, SC2154 x2 on eval-assigned variables shellcheck can't trace). * test: pin vCPU quota gate ordering in RequestSpotInstances Nothing asserted that RequestSpotInstances runs the vCPU quota gate before dispatching capacity or launching a VM, so a regression that dropped or reordered EnforceLaunch would pass silently. Add an ordering test using a real quota.Service (embedded JetStream KV) and real NATS responders standing in for the capacity and launch daemons, recording a global call sequence and asserting the gate's index precedes both. Verified by removing the gate: the first quota-KV read then comes from the post-launch charge instead, landing after capacity dispatch and launch, and the ordering assertion fails as expected. |
||
|
Josh Siviour
|
9f5f877e50 |
fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637)
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console. |
||
|
Josh Siviour
|
6873875248 |
fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634)
On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane. |
||
|
Josh Siviour
|
6ce8b9ff18 |
fix: fail preflight when a gate tool is killed instead of reporting it clean (#633)
A tool whose child dies by signal can still exit 0 through go tool, so govulncheck printed 'ok' after being OOM-killed and preflight passed without a vulnerability scan having run. Route lint, govulncheck and nilaway through scripts/run-gate.sh, which fails on a non-zero exit or on signal-death output. |
||
|
Josh Siviour
|
628fbbc0c3 |
fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609)
* fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts. |
||
|
Josh Siviour
|
3e195506d3 |
feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605)
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
* refactor(images): extract eks-node initd logic into init-agnostic helpers
The OpenRC runscripts carried their start_pre logic inline, so porting a
service to systemd meant re-deriving that logic under a distro change
rather than just rewriting the init wiring. Move it into /bin/sh helpers
that either init system can drive, keeping eks-node on Alpine/OpenRC.
k3s-prestart takes the restore-block guard, config.yaml derivation,
konnectivity-agent and CoreDNS staging, and the token-webhook wait.
konnectivity-server-prestart takes the CA wait and cert mint, plus the
server-count derivation that used to run at parse time. Both keep every
guard, bound and message as-is.
Values flow back the way the GPU image already does it: the helper writes
shell assignments to a /run file and the init system sources them, so
command_args is built in start_pre where it used to be a top-level
assignment.
k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU
manifest claimed its helpers were reused verbatim from eks-node while
hand-copying this one; it now points at the shared file, so the two
cannot drift.
k3s-agent.initd still sources agent.env and exports itself rather than
delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon
inherits only what the runscript exports, so a helper subprocess sourcing
it would not reach k3s.
The helpers run under set -eu, which the runscripts did not. The awk over
first-boot.env is guarded on the konnectivity manifest, not on the env
file, so a missing env file exits awk nonzero; without || true that turned
a warn-and-continue into a refusal to start k3s. Covered by a test that
fails on rc=2 if the guard is dropped.
* feat(images): build system images in a pinned mkosi container
Move the image-build toolchain into a container so a runner needs only
Docker. The toolchain cannot drift between runners, and the same build
runs identically on a developer's machine.
mkosi builds in a user namespace, which two of Docker's default
confinements block: the seccomp profile masks CLONE_NEWUSER, and the
docker-default AppArmor profile separately blocks unsharing the mount
namespace. Both are relaxed; neither is a privilege grant, as
capabilities stay dropped and no devices are exposed. mkosi also runs as
an ordinary user rather than root, since as root it takes a path that
needs real CAP_SYS_ADMIN.
Verified end to end: an Ubuntu 26.04 Format=disk image builds from the
Debian container and the artefact is a valid GPT disk, with the package
cache on a named volume cutting a forced rebuild from ~2min to ~34s.
* feat(images): publish system images from the build container
Convert mkosi's raw output to a compressed qcow2, write the .sha256
sidecar the spx catalog verifier matches, and upload both to R2 using the
same pinned toolchain as the build, so publishing needs nothing on the
host either. The toolchain image build moves to a shared helper rather
than being duplicated across the two entry points.
Publishing needs no relaxed confinement, unlike the build: qemu-img and
aws create no namespaces, so it keeps Docker's default seccomp and
AppArmor profiles.
The object path carries no version and the catalog resolves newest-wins,
so an upload overwrites the live image with no way back. Copy the current
object to a timestamped archive key first, making a bad publish a restore
rather than an unrecoverable loss.
* fix(images): build the image on the output's mount, not the cache volume
mkosi assembles the image in its workspace and moves the result to the
output directory. The workspace lived under the cache volume, which is a
different mount from the output bind mount even when both sit on one
filesystem, so the move failed EXDEV and mkosi fell back to copying —
requiring the finished image to exist twice at once.
That is invisible on a small image and fatal on a real one: a ~4G GPU
image needed ~8G to land and died with ENOSPC after a successful build.
Point the workspace at the output's own mount so the move is a rename
again and the image only exists once. The package cache stays on the
volume, which is the part worth persisting across runs.
* feat(images): add mkosi base config and gpu-nvidia profile
Establish images/ with a base mkosi.conf plus a gpu-nvidia profile,
replacing the throwaway smoke-test config. The base builds a bootable
UEFI disk; the profile adds the NVIDIA stack.
Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version
hunt, the do_bootloader=no kernel-img.conf hack and the manual
update-initramfs all exist only because that pipeline chroots into a
prebuilt cloud image that already has a kernel. mkosi installs the kernel
and owns the bootloader, so the version is known rather than detected, and
the postinst asserts exactly one kernel instead of searching for it. The
static configuration (nouveau blacklist, unattended-upgrades blacklist,
cloud-init docker group) becomes files under mkosi.extra/ rather than
heredocs.
linux-image-generic drags in every linux-firmware split package, none of
it relevant to a headless VM guest; left at the default the modules initrd
embeds all of it into the UKI, which grew to ~880MB and overflowed the
512M ESP. The initrd module set is trimmed to mkosi-initrd's curated
default, which is a boot fix and not only a size one.
Verified in the built artefact, not just the build log: nvidia.ko.zst and
the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the
nouveau blacklist, and the nvidia runtime in docker's daemon.json.
* fix(images): unblock guest boot on wait-online and firstboot
A guest built from these images stalled twice on the way to a login prompt,
both times on a unit waiting for input that never arrives in a VM.
systemd-networkd-wait-online holds network-online.target until every managed
link is up, and docker.service pulls that target in with Wants=, so any image
carrying docker inherits the wait whether or not it wants it. A guest whose
links are slow or absent ran the unit to its full 120s default with
multi-user.target queued behind it. Override it to --any with a short timeout:
one link up is what a single-NIC guest actually needs.
systemd-firstboot separately prompts on the console for anything the image
does not already carry, before multi-user.target. With no timezone baked in,
boot stopped dead at "Please enter the new timezone name" and waited for a
keystroke. Setting locale, keymap and timezone at build time leaves firstboot
with nothing to ask.
Both are baked into the base rather than delivered as first-boot drop-ins:
an image built from scratch has no cloud-init to write them.
Measured on wattle with an A1000 passed through: login at 42.8s, down from
~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports
"NVIDIA RTX A1000, 595.71.05, 8188 MiB".
Also widen the images gitignore to output-*/ so a scratch build cannot leave
multi-GB artefacts stageable.
* feat(images): make the base image reachable — cloud-init, sshd, console
An image built from scratch launched unreachable. The images this replaces
start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a
network renderer, dbus and a serial console from it; mkosi installs only what
is listed, so none of it was there and nothing said so.
Guests get their SSH key and user-data from IMDS via cloud-init, so without
the package a launched instance has no authorized_keys at all — no error, just
no way in. It also made any cloud-init-delivered config inert, including the
gpu profile's docker-group drop-in.
Each addition is load-bearing on its own:
cloud-init consumes IMDS: keys, user-data, hostname, network
openssh-server the key needs a sshd to be installed into
netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is
never configured, IMDS is never reached, and cloud-init
falls back to DataSourceNone
dbus systemd only Recommends it and mkosi installs no Recommends,
so systemctl could not reach the system bus — and the EKS and
ECS bootstraps drive guests with systemctl from user-data
lsb-release cloud-init shells out to it; without it every boot reports
"degraded" forever and the status stops being a usable signal
The kernel also needs console=ttyS0 baked in. The launcher gives every guest a
serial port and get-console-output reads it, but a kernel with no console=
writes to none of it: the console stopped after the UEFI bootloader and every
boot failure was invisible. ttyS0 goes last because Linux makes the final
console= the controlling /dev/console.
manage_etc_hosts is set because cloud-init renames the guest per launch but at
its default never writes a matching hosts entry, so sudo warned on every call.
datasource_list is deliberately NOT pinned: the guest identifies the platform
from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on
its own, and pinning per image is the customisation this migration deletes.
Verified on wattle with an A1000 passed through: ssh in as ubuntu with
spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed
units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s.
* feat(images): let the guest grow its root into the volume it was given
mkosi sizes the disk to its content, so the image ships with a root
partition that fills the disk to its last sector and ~1.4G free. Nothing
in it can expand that: cloud-init's growpart module is enabled, but the
growpart binary lives in cloud-guest-utils, which is not installed and is
not pulled in as a Recommends by anything in the package set. systemd-repart
is not an alternative — Ubuntu's systemd package carries no repart binary,
unit or definitions at all.
The result was a node handed a larger volume that it could only ever use
the first few GiB of, with the rest allocated and unreachable. That is
fatal for a worker whose job is to pull container images.
gdisk is deliberately left out despite being a cloud-init Recommends:
growpart resizes GPT through sfdisk, which arrives via the hard fdisk
dependency, and e2fsprogs brings the resize2fs that the resizefs module
runs afterwards.
Verified on wattle: the gpu-nvidia image launched with a 20G root reports
20G on df -h / (was 4.4G), via
"'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with
nvidia-smi still reporting the A1000 and no failed units.
* feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles
Adds the eks-common, eks-agent and ecs profiles, so both GPU node images
build from the composed base rather than a hand-authored tree, and the three
verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone.
Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the
standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS
node images run k3s or containerd and reach the GPU through CDI. Fusing the
two was fatal for ECS rather than merely wasteful: gpu-nvidia installed
containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu
containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at
all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs
nvidia-ctk to generate its CDI spec.
Closes three gaps the pilot left, each of which only shows up on a booted
guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the
modules on a headless node and CDI generation finds no GPU;
mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on
the toolkit packages, which the unattended-upgrades blacklist does not cover
for a plain `apt upgrade`.
mkosi-build.sh gains --image, naming an output and expanding it to an ordered
profile list. The order is load-bearing: mkosi runs each profile's postinst in
the order the profiles are passed, so docker's nvidia-ctk wiring must follow
gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a
healthy-looking image with no GPU wiring, so docker's postinst also fails
loudly when gpu-nvidia is selected but nvidia-ctk is absent.
Profiles reference build inputs where they already live via ExtraTrees=
instead of copying them, keeping one copy of the helpers shared with the
Alpine image. Go binaries are built on the host into a staging tree, since the
builder container carries no Go on purpose: go.mod is the single pin for the
toolchain and a second pin in the Dockerfile could disagree with it silently.
* fix(images): install systemd-resolved so the guest has a DNS resolver
mkosi installs only the packages named in Packages= plus their hard dependency
closure, and systemd-resolved is merely a Recommends of systemd. So it was never
installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated,
because netplan renders to systemd-networkd and networkd alone does not write
resolv.conf. glibc then fell back to localhost and every lookup failed with
"[::1]:53 connection refused", which reads as a network fault rather than a
missing package.
A node that cannot resolve cannot pull a container image. The ECS GPU node
booted, registered as a container instance, then failed every task at image
pull. The EKS image carries the same defect but is masked by the cloud-init
bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS
launch path has no equivalent, so only ECS exposed it.
* fix(images): name build output by image and never report a skipped build as success
Two defects in the build wrapper that let the wrong bytes reach a boot test.
Every composition wrote the same spinifex.raw, taken from the base config's
ImageId, so an artefact carried no record of which profiles produced it and an
EKS image was distinguishable from an ECS one only by which build ran last.
Passing --image-id names the output after the image, and as a side effect the
name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses
its default input: --name is the published object and --image the local
artefact, and with nothing tying the two together a defaulted input meant
whatever raw happened to be in the output directory could be uploaded under the
name the caller asked for.
mkosi also skips the build whenever the output path merely exists. It does not
compare inputs, so it never notices an edited mkosi.conf or a restaged binary,
prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable
from a fresh one. Builds are now always forced, and the run additionally asserts
the expected raw was written during this invocation rather than trusting the
exit status. The cache worth keeping is the package download volume, which
--force does not touch.
* feat: add gpu-amd mkosi profile for the ROCm AMI
Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under
mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to
stay alive for one remaining image.
Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04
universe ships its own vendored rocminfo, which unpinned apt preferred over
AMD's, mixing release trains within one image.
Uses a systemd unit for render/video group membership instead of a cloud-init
runcmd, because cloud-init's config merge takes a duplicated runcmd key
wholesale from one fragment and silently discards the other, which would have
dropped the docker profile's own usermod.
Runtime GPU verification is deferred — no AMD hardware is available.
* feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images
Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine
systemd units, a templated etcd-snapshot service with daily and 15-minute
timers, the addon/coredns/konnectivity assets, and the six server binaries.
konnectivity-server is not an in-repo package, so it gets a pinned host-side
clone-and-build rather than an entry in profile_binaries().
Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any
attached volume and mulga-mgmt-net configures any multi-NIC system VM, so
neither belongs behind an EKS profile that a database image would have to
compose to reach them.
Start-at-boot moves into preset files. mkosi runs preset-all after every
postinst, so a postinst enable is discarded before the image is finished, and
with no policy file the fallthrough is enable — which built an image that ran a
control plane and a worker at once.
Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base
ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the
directory in reverse and the merge keeps the first source, so a 90- prefixed
file lost the key silently. The postinst now asserts the merged result.
* refactor(eks-node): make the role selector init-agnostic
eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image
the selector enabled nothing and the node never started k3s. Both now dispatch
through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT
overriding detection so either branch is testable on a host that has only one.
Detection prefers rc-update over systemctl: the test suite stubs rc-update on
PATH, and preferring systemctl would have silently rerouted those cases to a
path they were not written for.
Enable and start stay two separate passes. The server path starts
mulga-eks-k3s-recovery before k3s so a recovery directive applies before the
apiserver comes up, and collapsing to systemctl enable --now would interleave
them.
The dispatch is duplicated in both scripts rather than shared: they are
single-file deliverables with no library-path convention between them.
Tests cover both branches across all three roles, including that server-join
still gets no k3s-first-boot, and that rc-update wins when both inits are
present.
* fix(eks-node): enqueue role-selected units instead of blocking on them
eks-node-role.service declares Before=k3s.service, and
mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a
blocking systemctl start deadlocks: systemd will not run the queued job while
the selector is still activating, and the selector cannot finish until that job
runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie
and the node wedges mid-boot with the role written but no control plane. On a
booted guest the selector sat in activating for 261s until the VM was torn down.
Start with --no-block on systemd. The ordering the blocking call was implicitly
providing is already declared between the units themselves, so systemd
sequences them correctly on its own. OpenRC runs the script directly rather
than scheduling it, so that branch keeps blocking and keeps surfacing failures
through set -e.
* fix: escape $ in konnectivity-server ExecStart so server-count survives
systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before
the command runs, substituting an empty string for a name it does not know.
KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes
it to /run/konnectivity-server-args.env, which the wrapper shell sources at
runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw
it. konnectivity-server then died on flag parsing, which is why the unit logged
Started and produced nothing despite --logtostderr=true and journal+console,
and why the apiserver's cluster egress selector found no UDS for the life of
the cluster.
This is a port artefact. konnectivity-server.initd is flag-for-flag identical
and shares the same prestart script, but under OpenRC the shell was the only
thing reading ${KONN_SERVER_COUNT}, so the same text was correct.
Escape it as $$ to defer expansion to the wrapper shell, and add
lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of
a profile's units. The linter joins line continuations first, since the
offending reference sat mid-way through a wrapped multi-line ExecStart, and
runs on the host before the container starts so a bad unit fails the build in
seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd
leaves it literal.
Verified on env12 with a rebuilt image: TestEKS/IRSAPod and
TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed
out` through the egress tunnel, now pass, with zero UDS connect errors across
the run.
* fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw
mkosi only emits the raw. A qcow2 next to it came from a previous run's hand
conversion, and nothing about its name says which build it belongs to, so it
survives a rebuild and silently ships the old image to a node while the raw
beside it carries the new one. Observed on a rebuild that left the raw at 15:34
carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit.
This is the same failure the existing freshness assertion guards against, one
file over, so extend that block rather than add a second check. Refresh rather
than delete: the qcow2 is what gets copied to a node and imported, so removing
it would push the manual conversion, and this trap, back onto the caller.
Convert via a temp name and rename so an interrupted run leaves the old file
rather than a half-written one that looks current.
* fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc
ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns,
which systemd-networkd does not manage, so it shells out to a DHCP client. That
call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not.
`ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the
failure surfaced as an opaque "network setup failed" with the task stuck in
STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and
ServiceWithELB both failed on the mkosi image, the latter as runningCount=0
because its target group targets task ENI IPs.
Resolve the client at setup time, preferring udhcpc so Alpine behaviour is
unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not
sufficient to select it: Debian and Ubuntu's busybox ship the applet without
/usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits
nonzero having configured nothing. Same order and same reasoning as
dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no
daemon behind in the netns.
Install dhcpcd-base in base, because neither client was present in any profile.
That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved
to nothing on Ubuntu and would have failed the first time a cross-host
control-plane data NIC needed it — the S5 path the step-6 plan notes has never
been exercised.
The DHCP-client choice is injected so the tests assert a named platform's
branch rather than whatever happens to be installed on the machine running
them; without that the same test exercises different code on an Alpine CI box
and an Ubuntu dev box.
* refactor: retire the ec2-node profile, fold its helpers into the base image
ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition
had to remember to list it — eks-common even guarded on its presence and
aborted the build without it. ECS did not list it, so the ECS image shipped
with no by-id symlinks for hot-attached volumes at all.
Neither helper is answerable at build time: any image may later be attached a
volume, and any image may later be launched as a control plane. Both are inert
when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its
fw_cfg blob is absent), so the base image is where they belong. That deletes
the composition rule, the guard, and the ECS gap in one move.
Also drops ec2-node from the two eks image compositions in mkosi-build.sh.
Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both
carry the helpers and the udev rule, mulga-ebs-byid is enabled via
multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants.
* fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link
mulga-vpc-mtu only won the boot race by ordering: netplan renders
UseMTU=true (it does not inherit networkd's own false default), so a
DHCP lease renewal at T1 could silently re-apply the advertised MTU to
the link. usemtu_off() forces UseMTU=false via a runtime drop-in in
the systemd-documented [DHCPv4] section for every rendered unit and
reloads networkd, removing the dependency on boot ordering entirely.
assert_pinned() then verifies the link and its routes actually match
before exiting, so any regression is a loud boot-time failure instead
of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has
no systemd-networkd.
* fix(images): state UseMTU=false under both the current section and the legacy alias
netplan renders the DHCP settings under [DHCP] while systemd documents
[DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the
running systemd disagrees, which is the silent no-op the drop-in exists to
prevent.
* fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch
credendpoint.go DNATs the ECS task credential endpoint via iptables; neither
the mkosi ecs profile nor the legacy Alpine manifest installed it, so the
endpoint silently failed to start and any task with a task role could not
fetch credentials while the instance still looked healthy.
Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only,
which this platform's HTTPS AssumeRole path can never match) and adds a
TaskRoleCredentials subtest that fetches real credentials from inside a task
container, so a dead endpoint fails the suite instead of passing unnoticed.
* fix(images): ship preset files for base, eks-common and ecs start-at-boot
mkosi runs systemctl preset-all after every profile's postinst, discarding
whatever systemctl enable a postinst performed. eks-server and eks-agent
already carry preset files for this reason; base, eks-common and ecs still
relied on the default-enable fallthrough, which happens to match today but
is one upstream package change from reversing.
Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net),
30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and
40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables
cni-dhcp.service/socket, which containernetworking-plugins enables by
default with nothing to serve under the baked host-local IPAM conflist).
Removes the now-redundant systemctl enable calls from each postinst,
matching the eks-server/eks-agent precedent.
* fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers
The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal
denied every Service clause other than the IMDS-only EC2 one. A role carrying the
trust policy AWS documents for an ECS task role was unassumable by construction.
Service attribution is now derived from the authenticated SigV4 caller: an
assumed-role session of the account's ecsInstanceRole — the only credential the
in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles
in its own account. Nothing in the request body contributes, so a task holding
its own task-role session cannot claim the service principal.
* fix(images): remove baked SSH host keys, pin Seed= for reproducibility
openssh-server's postinst bakes an identical host key pair into every
build, making every guest from one build mutually MITM-able; sshd-keygen
already regenerates keys on first boot (ConditionFirstBoot=yes), so the
baked pair is pure non-determinism with no purpose. Also pin Seed= so
systemd-repart stops assigning fresh partition UUIDs on every build.
Verified on a built spinifex-ecs-node image: no ssh_host_* files ship,
cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service
is wired via ssh.service.wants/ (redundant regeneration path). A two-build
comparison dropped content-differing files from 9 to 1 (only the UKI,
whose zstd internals remain sensitive to unrelated input changes).
---------
Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>
|
||
|
Josh Siviour
|
a0659ae7d0 |
fix: gate ovn-controller on SB RAFT readiness and force recompute (#599)
Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then restarted ovn-controller without waiting for the Southbound RAFT to elect a leader, serve clients, and for northd to write the gateway logical flows. With no join step to absorb the boot window the controller attached mid-election, programmed a partial datapath (SNAT ct-commit without output/delivery) and never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed on the single-node baremetal CI host. Add an SB readiness gate before ovn-controller starts (SB reachable and, when clustered, an elected leader), and after ovn-controller restart wait for its SB connection then force inc-engine/recompute to rebuild all OpenFlow from the converged Southbound. Applies to 1/2/3-node RAFT alike. |
||
|
Julian Sommer
|
32d80b473a | feat: RDS v1 foundations (#592) | ||
|
tomnewton-mulga
|
6389b37433 |
fix(gpu): Blackwell GPU support fixes for EKS/ECS image builds and VFIO (#559)
* fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. --------- Co-authored-by: joshsiv-mulga <mrk@mulgadc.com> |
||
|
Julian Sommer
|
e79ca59d4b | fix: configure host DNS from spx admin init/join (#582) | ||
|
Julian Sommer
|
1d55404a73 |
revert: host DNS setup on resolved/resolvconf hosts (#580)
This reverts commit
|
||
|
Julian Sommer
|
f9c9f0b38d | fix: configure host DNS for Spinifex zones on resolved and resolvconf hosts (#580) | ||
|
Josh Siviour
|
4793b5ac09 |
fix(eks): node route MTU blackholing image pulls, and teardown wedged in DELETING (#562)
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.
|
||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
joshsiv-mulga
|
1281381096 |
fix(ci): publish-system-image.sh honors SYSTEM_IMAGE_BUILD_DIR
The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but the publish step hardcoded /tmp and could not find the freshly built raw. GPU legs failed 'Raw image not found'; Alpine legs silently reused stale pre-existing /tmp raws on the persistent runner. Mirror the build script's derivation so publish looks where the build actually wrote. |
||
|
Josh Siviour
|
b41a193cbc | fix(eks): make provider-id.sh the single k3s node-label authority for nodegroup + GPU labels (#539) | ||
|
Josh Siviour
|
4d8a34b545 |
fix(eks): apply eks.amazonaws.com/nodegroup label via k3s node-label drop-in (#534)
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but k3s has no such env var, so non-GPU workers never received the label. waitWorkersReady buckets Ready nodes by that label (added in #511), so the affected nodegroups never counted toward their ACTIVE transition and stayed CREATING until timeout. Ride the label on a real k3s node-label config drop-in (15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the inert K3S_NODE_LABEL from agent.env and the initd export. |
||
|
Josh Siviour
|
1e1d2f3c6d |
ci: GPU e2e workflow on ci-gpu self-hosted runner (#518)
* ci: GPU e2e workflow on ci-gpu self-hosted runner Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent, hand-bootstrapped wattle shared host, driven over SSH by a banksia runner labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner installed on the shared box. - runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress false (single A1000 = single-flight, queue not stomp) - workflow_dispatch + GPU-path filter (not every push) - manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU testing blocks CI - if: always() cleanup terminates launched GPU instances; never resets the box (no setup-ovn.sh/dev-reset/cluster-reset) - drives the control plane with spx + the already-configured aws spinifex profile directly (no wrapper scripts, nothing to install on wattle) * ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs The suite is self-guarding — each test skips when its AMI is absent. The hard-fail on the base ubuntu GPU AMI blocked the whole run even when the ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and fail only when none are present. * ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported the self-guarding suite skipped every test, passing as a false green. Make the raw-EC2 GPU passthrough test the always-on baseline. - Add an import-if-absent setup step that pulls the prebuilt base GPU image from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's persistent loop-fs. Runs spx under sg spinifex (local storage write). - Preflight now hard-requires the base AMI (fail, not skip) so a silent import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn. - Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and launch-gpu.sh's nvidia default now use the same canonical name, so a locally built+imported image also satisfies the suite. * ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches spx admin images import --name selects the catalog entry but names the resulting AMI generically (ami-<distro>-<version>-<arch>), so the import landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same flag build-system-image.sh uses) and poll briefly for availability. * fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout Auto-assigned public IPs were never carried through the network reconciler. applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no gatewayLRP->guest convergence. Their dnat_and_snat was created once at RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the guest-port hop never converged: inbound DNATs at the gateway but never reaches the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK). floatingIPSpecs now folds every auto-assigned/ELB public IP into the same AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user EIP wins), making the datapath self-healing across reconcile ticks and deploys. qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB, floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers. * fix(vm): scale QMP handshake command deadline for VFIO launches The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU guest sends its QMP greeting before the RAM pin completes, then cannot service qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and the launch cleanup SIGKILLed a QEMU that would have come up. Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s) for qmp_capabilities. Other callers keep the 30s default. * test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle |
||
|
Josh Siviour
|
f7bf921ce7 |
fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511)
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group names exceed the ELBv2 32-char limit, failing cluster creation asynchronously with no visible cause. safeELBv2Name now deterministically truncates the name and appends a stable hash of the full cluster name when needed, so the same name is derived on every call (create, lookup, delete). DescribeCluster also now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster is FAILED, instead of leaving callers to guess why. * fix(eks): plumb per-nodegroup Ready-node counts through state reports The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready count keyed by the eks.amazonaws.com/nodegroup label value, alongside the existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's observe step, and SetClusterHealthState now carry this map through to ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next commit) can gate on a nodegroup's own workers instead of the cluster total. * fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches waitWorkersReady compared the cluster-wide Ready-node count against a nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's Ready workers could mask another whose own workers never registered. It now reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the eks.amazonaws.com/nodegroup=<name> label. Separately, launchNodegroupInfra never retried a failed worker launch (a transient QMP timeout or host capacity pressure), leaving the nodegroup stuck CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches only the shortfall on failure, logging each retry, until the desired count is reached or workerLaunchRetryTimeout elapses. Both fixes are covered by regression tests: a two-nodegroup scenario where only one has Ready workers, a transient-failure-then-success relaunch, and an exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which this change does not add. * fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and kubectl logs/exec against pods on that node failed "No agent available". Adds an Exists toleration for nvidia.com/gpu:NoSchedule. Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster available to test against right now. * fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS |
||
|
Josh Siviour
|
9f349e86ca |
feat(eks): expose NVIDIA GPUs to pods (Epic B) — VFIO QMP, worker registration, device-plugin GPU access (#507)
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%). |
||
|
Josh Siviour
|
d663760dec |
feat(eks): control-plane disaster recovery — etcd snapshot + restore-snapshot (#503)
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for
|
||
|
Josh Siviour
|
6879a3425a |
feat(gpu): Ubuntu NVIDIA GPU node images + GPU AMI selection (Epic A) (#502)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection Epic A of GPU passthrough to EKS pods / ECS task containers. - Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers (instance-family prefix -> vendor from knownGPUModels). - EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU instance types; no fallback to the driverless image (hard error instead). - Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no negation, so exclude client-side). - NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes. - Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged). - New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the proven DKMS -server driver recipe (drop Docker CE + dev payload), add nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation. - build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags. * build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the build onto a real disk. |
||
|
Julian Sommer
|
47604f43dc | chore: go 1.26.5 | ||
|
joshsiv-mulga
|
2c1ddea124 | Remove backed-in creds for eks addon sync - prefer IMDS | ||
|
Josh Siviour
|
e1c4f85a36 |
feat: routed-NAT external mode with two-tier ingress (#475)
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets. |
||
|
Josh Siviour
|
e47baf321c |
feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480)
* feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight. |
||
|
Josh Siviour
|
da33e574d4 |
fix: OVN RAFT multi-node bring-up (northd member list, ExecStop, add-nat deadline) (#466)
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline Two related fixes for multi-node OVN clusters: - setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point ovn-northd's client NB/SB connections at the full RAFT member list derived from OVN_REMOTE, so the active northd follows the leader instead of dialing its local member only. Without this, northd stops advancing SB_Global.nb_cfg once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier. Guarded on a real member list so the standalone single-DB-node path is unchanged. - utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to 45s so the client no longer rolls back before the server-side --wait=hv flows barrier (30s) completes. * fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without --ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this races the freshly-started DBs and kills them, so setup-ovn's wait for the NB socket times out and the RAFT create/join aborts. Install an ovn-northd.service drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches the DB lifecycle (mulga-siv-504). |
||
|
Josh Siviour
|
7bbd99c792 |
feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459)
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
|
||
|
Josh Siviour
|
0823bf8c16 |
feat: OVN OVSDB RAFT clustering for multinode Spinifex (epic mulga-siv-486) (#458)
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a management node can run the OVN NB/SB databases as a native OVSDB RAFT cluster instead of a single non-clustered ovn-central. When a local addr is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since the ovn-central aggregator is ExecStart=/bin/true and won't restart its children. Empty remote addr creates the cluster (init node); a set remote addr joins it. NB/SB client set-connection runs only on the create node (RAFT replicates it cluster-wide). The single-node --management path is unchanged when no cluster addrs are supplied, so dev boxes and existing single-node clusters keep working. --ovn-remote already passes through verbatim, so a comma-separated SB endpoint list works for compute nodes; documented in usage. * feat: OVN client multi-endpoint failover for RAFT clusters Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a clustered DB. config.ParseEndpoints splits the list (trimming whitespace, dropping empties); live.go Connect passes one libovsdb WithEndpoint per endpoint, so the client tries them in order, fails over on disconnect, and follows RAFT leader redirects. Single-endpoint strings still yield one endpoint, so single-node dev is unchanged. ovn-sbctl --db= already accepts the raw comma list, so the SB CLI consumers (discoverChassis, gateway_claim) need no change. * feat: write clustered OVN NB/SB endpoints across the RAFT quorum spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback or a single leader address, so vpcd fails over across the cluster. firstboot defers OVN bring-up to the provisioning controller when it owns formation (SkipFormation): a standalone single-node .db would otherwise block the controller's clustered create/join. * test: add multinode OVN RAFT cluster + failover e2e; document clustered topology ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with one leader on every DB node, then stops the NB leader's ovn-central and proves an NB write still lands via a surviving endpoint. Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT cluster-status debugging section; network-connections lists the 6643/6644 RAFT ports. |
||
|
Josh Siviour
|
2d4f0fa94c |
feat: IMDS instance-role creds for lb-agent + EKS server VMs (siv-231.6) (#450)
* feat: IMDS-capable gateway signer for lb-agent + EKS clients Add internal/gwsign, a SigV4 signer that draws credentials from either fixed static keys or the AWS SDK default credential chain. In a guest VM the chain falls back to the in-VM IMDS endpoint, yielding scoped, auto-rotating instance-role credentials; a session token is signed as X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds. Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client (internal/eksgw). When static keys are absent the clients now sign with IMDS credentials instead of failing; when present, behaviour is unchanged (static path). This is the client-side foundation for dropping long-lived secrets from VM user-data. Credentials are retrieved per request so rotated IMDS creds take effect without an agent restart. * feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C) Attach a Spinifex-managed system instance profile to the k3s control-plane VM and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating IMDS instance-role credentials instead of a long-lived static secret baked into user-data. - Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a role (EC2 trust) plus its instance profile and asserts an inline permission policy. The gateway enforces per-action authz on assumed-role principals, so the roles carry exactly the actions each VM calls: - EKS server: eks:PublishInternal, eks:ListInternalAddons. - LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig. - Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves the role. - Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/ EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from the lb-agent fw_cfg env; relax k3s server validation to accept profile-only; drop the EKS key requires from the first-boot and state-report scripts. - Both verticals fall back to baked static creds when IAM is unwired, so an absent master key never blocks a launch. * fix: resolve IMDS instance-role creds under the instance account for system VMs Three boot-path defects kept the LB VM's lb-agent from authenticating to the gateway with IMDS-served instance-role creds; each produced a silent empty IMDS role list and a heartbeat failing with "unexpected empty EC2 IMDS role list". Found and fixed against env19. - Build the system IAM ensurer lazily on the first system-VM launch instead of eagerly at daemon startup, where it raced the NATS KV backend and fell back to baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider resolved at launch time. - Create the LB instance profile in the system account the LB VM runs under (utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile under the instance's account, so a customer-account role was invisible to it. - Record the attached instance's owning account on the ENI (InstanceOwnerId, mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every IAM/STS lookup under it. The LB ENI lives in the customer account while the system LB VM and its role live in the system account; keying the whole IMDS chain off the ENI account left the instance invisible and the role lookup missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account. A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role principal with no static keys in user-data. * test(e2e): assert LB VM authenticates via IMDS instance-role creds The LB suite's targets-healthy assertion already depends on the lb-agent authenticating to the gateway, and with static keys dropped from user-data that auth is IMDS-only. Add an explicit check that a recent gateway authentication for the LB used an assumed-role (ASIA) principal, so a regression to the instance-account IMDS resolution fails with a clear message instead of a mysterious datapath timeout. The LB VM is a system-account instance invisible to the customer AWS API, so the assertion observes the gateway journal on the cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and skips when no node is SSH-reachable. |