mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
a4274f95b537237baeb8cd853bc93abc2675be5f
288 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
57a93fdee3 |
fix(nats): detect the JetStream write latch the health endpoint cannot see (#645)
The watchdog probed only healthz?js-enabled-only=true, which reports the server-wide JetStream state. A per-message-block write error is recorded on the block and returned on every later write, and nothing clears it short of a restart -- the server-wide state is never touched, so healthz stays 200 while writes fail indefinitely after the disk recovers. spx admin node js-probe round-trips a canary key through the cluster-state bucket, which is the only operation that observes the latch. It probes that bucket rather than one of its own because a dedicated bucket is a separate stream that can be placed elsewhere and stay writable. |
||
|
Josh Siviour
|
dad5d702e1 |
fix: stop lbagent heartbeats on positive LB deletion, and actually enable the JetStream watchdog (#643)
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted A LoadBalancerNotFound response is the gateway's positive confirmation that the store has no record for the LB, not a connectivity failure — unlike a dial error, timeout, or 5xx, which leave its existence unknown and must keep retrying. Past the startup registration grace window (or after any prior successful heartbeat), it can no longer be the record's own replication race, so the agent now stops rather than retrying forever and filling the log with heartbeat noise after teardown. * fix(nats): enable the JetStream ENOSPC-latch watchdog timer nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself — that decision is nats-server's own internal state, not spinifex's, so a restart is the only way back. The spinifex-nats-watchdog.timer + service already probe /healthz?js-enabled-only=true and try-restart on a sustained failure, but the timer's WantedBy=timers.target never took effect: neither setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full disk needed a manual restart forever. Enable --now it alongside the rest of the unit set. |
||
|
Josh Siviour
|
59db7d1e52 |
fix: stop calling RemoveSystemImage's deleted bytes "freed" (#638)
BytesFreed counted logical object sizes deleted from the S3 namespace, but predastore only reclaims that disk space later via background compaction. The CLI printed "freed X GiB" immediately on delete, which made a compaction outage that never reclaimed the space invisible. Rename to BytesDeleted and reword the CLI output to say the bytes are marked for deletion and reclaimed by background compaction, not freed immediately. |
||
|
Josh Siviour
|
9f5f877e50 |
fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637)
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console. |
||
|
Josh Siviour
|
2983160374 |
fix(lbagent): emit JSON logs so console attributes reach the sink as fields (#617)
lb-agent configured no logger at all, so it fell back to slog's default text handler. That folds attributes into the message, leaving the cause of a failure trapped in a string: 'Heartbeat failed err="gateway returned 400"' arrived greppable but never queryable. No OTLP export: the agent's only egress is the gateway it is watching, so the host-side serial console remains the path that survives a blackout. |
||
|
Josh Siviour
|
eff11b611b |
fix: stop upstream DHCP leases leaking out of the external pool (#602)
* fix(dhcp): release stranded leases and label them with the owning node Two paths dropped a lease from KV without returning it to the upstream server, so a source="dhcp" pool leaked addresses permanently. Manager.Start deleted an expired entry outright. Dropping the entry alone abandons the binding rather than returning it, and a server that ages leases by liveness holds the address for as long as anything answers ARP for it. Adoption now sends a bounded, best-effort RELEASE first, skipping leases with no stored offer/ack because nclient4 cannot rebuild those. Release-by-IP answered SUCCESS when the IP was absent from KV, telling the caller the address was freed when nothing went on the wire. It now returns an error. Every ReleaseIP call site logs and continues, so this surfaces the strand without wedging teardown. Leases were also unattributable upstream: chaddr is a derived HashMAC and option 61 carries that same MAC, so no field identified the host that took the lease. Option 12 is now "<node>-<client-id>" and option 60 is "mulga-spinifex/<node>", giving the lease table a per-lease identity and a per-host column to group on. HashMAC is untouched — encoding the node there would cut per-node entropy and reverse a documented decision. Observed live: 20 office-LAN addresses held by one node's orphaned OVN gateway LRPs, with no lease records and no renewals. * fix: rebind the datapath when a DHCP lease moves to a new address A lease re-issued on a different IP was persisted to KV with no notification of any kind. doAcquire replaced the lease after an expiry and doRenew accepted whatever a rebind answered with, while the gateway LRP's address is written once at attach and never revisited. Each such event burned one address permanently: the old one squatted by OVN's ARP responder with no renewer and no DHCPRELEASE, the new one renewed forever attached to nothing. Route both paths through applyLease, which releases the superseded address and invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg so the VPC SNAT and host ingress route follow, and waits on the flows barrier. EIP, ENI-public and NATGW addresses are recorded outside vpcd in DescribeAddresses and DescribeInstances, so the hook reports those rather than repointing a datapath the control plane no longer agrees with. The release half still applies to every purpose, so the leak itself is closed either way. A failed rebind does not fail the lease loop: the new address is held regardless, and dropping it would strand both. * fix: reconcile API-visible records when their DHCP lease moves An EIP or auto-assigned public IP whose lease was re-issued elsewhere left DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had already released. The datapath kept the old dnat_and_snat too, so the address was dead on both sides while the upstream server was free to hand it to someone else. vpcd cannot fix this alone: it holds the lease, but the records naming those addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and the daemon serves, dispatching to the service owning the moved address. RebindPublicIP and RebindENIPublicIP commit the new external IP through vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP, so one commit is enough — and only then move the record, so a NAT failure never leaves a record advertising an address that does not route. Both refuse to move a record naming neither address, so a stale request cannot overwrite an allocation the lease does not own, and both are idempotent under duplicate delivery. A failed or unanswered reconcile propagates back to the lease loop rather than reading as success. vpc.dhcp.lease-expired, listed in the service manifest but never implemented, is replaced by the subject that now exists. * fix: dedupe retried AllocateAddress so one call takes one address EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can outlast the AWS client read timeout, so every SDK-level retry minted a fresh allocation ID and drew another upstream DHCP lease. The caller kept one EIP and the rest stayed leased with no record naming them, so nothing ever released them. Retries are now identified by the SDK's amz-sdk-invocation-id, which is stable across attempts of one call. The gateway lifts it off the request, NATSRequest forwards it as a header, and the daemon puts it back on the context, so the EIP service sees the same token every attempt. singleflight joins a retry to the call still in flight — the common case — and a TTL'd KV record answers retries that arrive after the first call completed. * fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones Three leaks that all end the same way: an address stays leased upstream with nothing renewing it on purpose and nothing that will ever release it. Manager.Start deleted any lease that expired while vpcd was down, after releasing it. The resource it was taken for still existed, so the datapath kept forwarding on an address the upstream server now considers free — a duplicate-IP conflict waiting for the server to reissue it — and nothing ever re-acquired. Expired leases now get a renewal loop like any other, which re-DORAs on its first pass; the existing IP-change hook rebinds the datapath when the new address differs. The reaffirm RENEW is skipped for these, since a lease the server has already aged out will only fail it. Nothing reconciled leases against the resources that justify them, so a resource deleted out from under its lease left the address renewed forever. ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check whether the EIP, ENI or VPC still exists, and releases the lease when it does not. A gateway lease also has a router port to remove, so DetachIGW runs before the address goes back — releasing it while OVN still answers ARP for it invites the same conflict. Anything the daemon cannot resolve is reported unknown and kept: an unreachable daemon must never read as a fleet-wide deletion. The acquire ladder ran to ~62s, just past the 60s read timeout botocore defaults to, which is what turned one AllocateAddress into two leases. It now totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the outcome instead of giving up and retrying. This is a mitigation, not a fix, for callers that send no retry token: botocore emits no amz-sdk-invocation-id, so the dedupe added earlier cannot see their retries at all. Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj. * feat: add spx get dhcp-leases so held external addresses are visible The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV bucket with no read path, so working out which addresses were held, by what, and whether the owner still existed meant writing a NATS client or reading journals. That was the main obstacle to diagnosing every leak in this series. The command lists each lease with its purpose, VPC, address, MAC, holding node and remaining time, and resolves the owner over vpc.dhcp.owner-check so an orphan is visible as such. --orphans narrows the output to leaks; --skip-owner-check drops the lookup when only the addresses matter. Fixes mulga-655iu. * fix(vpcd): reconcile gateway ports onto their lease address at startup A gateway LRP configured from a static pool keeps its address when the pool is switched to source="dhcp", so the VPC forwards on an address nothing leases while the lease it holds goes unused. No lease event ever fires to correct it. vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the address the lease actually holds. RebindGatewayIP is idempotent, so a port already in agreement costs one OVSDB read. * test: cover dhcp lease owner-check handlers and the lease CLI The owner-check path decides whether the reaper releases an address, so every inconclusive branch is asserted to report unknown rather than gone. * fix(dhcp): make the outcome of a stale lease release observable A stale RELEASE was logged only when the transport failed. A lease with no raw offer/ack was skipped in silence, and a RELEASE the server ignored looked identical to one it honoured, so an address left bound upstream could not be traced to either cause. The skip now warns, and a successful release logs the chaddr it used, which is what a server matches a RELEASE against. * fix(dhcp): take the RELEASE chaddr from the lease being released A server matches a RELEASE to a binding on chaddr, and the release path re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed since, that sends an address the binding never had: the server drops the RELEASE and says nothing, so the address stays held with no renewer. The chaddr now comes from the lease record, which carries what took the lease out, falling back to live resolution only when the record has none. |
||
|
Josh Siviour
|
3e195506d3 |
feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605)
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
* refactor(images): extract eks-node initd logic into init-agnostic helpers
The OpenRC runscripts carried their start_pre logic inline, so porting a
service to systemd meant re-deriving that logic under a distro change
rather than just rewriting the init wiring. Move it into /bin/sh helpers
that either init system can drive, keeping eks-node on Alpine/OpenRC.
k3s-prestart takes the restore-block guard, config.yaml derivation,
konnectivity-agent and CoreDNS staging, and the token-webhook wait.
konnectivity-server-prestart takes the CA wait and cert mint, plus the
server-count derivation that used to run at parse time. Both keep every
guard, bound and message as-is.
Values flow back the way the GPU image already does it: the helper writes
shell assignments to a /run file and the init system sources them, so
command_args is built in start_pre where it used to be a top-level
assignment.
k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU
manifest claimed its helpers were reused verbatim from eks-node while
hand-copying this one; it now points at the shared file, so the two
cannot drift.
k3s-agent.initd still sources agent.env and exports itself rather than
delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon
inherits only what the runscript exports, so a helper subprocess sourcing
it would not reach k3s.
The helpers run under set -eu, which the runscripts did not. The awk over
first-boot.env is guarded on the konnectivity manifest, not on the env
file, so a missing env file exits awk nonzero; without || true that turned
a warn-and-continue into a refusal to start k3s. Covered by a test that
fails on rc=2 if the guard is dropped.
* feat(images): build system images in a pinned mkosi container
Move the image-build toolchain into a container so a runner needs only
Docker. The toolchain cannot drift between runners, and the same build
runs identically on a developer's machine.
mkosi builds in a user namespace, which two of Docker's default
confinements block: the seccomp profile masks CLONE_NEWUSER, and the
docker-default AppArmor profile separately blocks unsharing the mount
namespace. Both are relaxed; neither is a privilege grant, as
capabilities stay dropped and no devices are exposed. mkosi also runs as
an ordinary user rather than root, since as root it takes a path that
needs real CAP_SYS_ADMIN.
Verified end to end: an Ubuntu 26.04 Format=disk image builds from the
Debian container and the artefact is a valid GPT disk, with the package
cache on a named volume cutting a forced rebuild from ~2min to ~34s.
* feat(images): publish system images from the build container
Convert mkosi's raw output to a compressed qcow2, write the .sha256
sidecar the spx catalog verifier matches, and upload both to R2 using the
same pinned toolchain as the build, so publishing needs nothing on the
host either. The toolchain image build moves to a shared helper rather
than being duplicated across the two entry points.
Publishing needs no relaxed confinement, unlike the build: qemu-img and
aws create no namespaces, so it keeps Docker's default seccomp and
AppArmor profiles.
The object path carries no version and the catalog resolves newest-wins,
so an upload overwrites the live image with no way back. Copy the current
object to a timestamped archive key first, making a bad publish a restore
rather than an unrecoverable loss.
* fix(images): build the image on the output's mount, not the cache volume
mkosi assembles the image in its workspace and moves the result to the
output directory. The workspace lived under the cache volume, which is a
different mount from the output bind mount even when both sit on one
filesystem, so the move failed EXDEV and mkosi fell back to copying —
requiring the finished image to exist twice at once.
That is invisible on a small image and fatal on a real one: a ~4G GPU
image needed ~8G to land and died with ENOSPC after a successful build.
Point the workspace at the output's own mount so the move is a rename
again and the image only exists once. The package cache stays on the
volume, which is the part worth persisting across runs.
* feat(images): add mkosi base config and gpu-nvidia profile
Establish images/ with a base mkosi.conf plus a gpu-nvidia profile,
replacing the throwaway smoke-test config. The base builds a bootable
UEFI disk; the profile adds the NVIDIA stack.
Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version
hunt, the do_bootloader=no kernel-img.conf hack and the manual
update-initramfs all exist only because that pipeline chroots into a
prebuilt cloud image that already has a kernel. mkosi installs the kernel
and owns the bootloader, so the version is known rather than detected, and
the postinst asserts exactly one kernel instead of searching for it. The
static configuration (nouveau blacklist, unattended-upgrades blacklist,
cloud-init docker group) becomes files under mkosi.extra/ rather than
heredocs.
linux-image-generic drags in every linux-firmware split package, none of
it relevant to a headless VM guest; left at the default the modules initrd
embeds all of it into the UKI, which grew to ~880MB and overflowed the
512M ESP. The initrd module set is trimmed to mkosi-initrd's curated
default, which is a boot fix and not only a size one.
Verified in the built artefact, not just the build log: nvidia.ko.zst and
the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the
nouveau blacklist, and the nvidia runtime in docker's daemon.json.
* fix(images): unblock guest boot on wait-online and firstboot
A guest built from these images stalled twice on the way to a login prompt,
both times on a unit waiting for input that never arrives in a VM.
systemd-networkd-wait-online holds network-online.target until every managed
link is up, and docker.service pulls that target in with Wants=, so any image
carrying docker inherits the wait whether or not it wants it. A guest whose
links are slow or absent ran the unit to its full 120s default with
multi-user.target queued behind it. Override it to --any with a short timeout:
one link up is what a single-NIC guest actually needs.
systemd-firstboot separately prompts on the console for anything the image
does not already carry, before multi-user.target. With no timezone baked in,
boot stopped dead at "Please enter the new timezone name" and waited for a
keystroke. Setting locale, keymap and timezone at build time leaves firstboot
with nothing to ask.
Both are baked into the base rather than delivered as first-boot drop-ins:
an image built from scratch has no cloud-init to write them.
Measured on wattle with an A1000 passed through: login at 42.8s, down from
~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports
"NVIDIA RTX A1000, 595.71.05, 8188 MiB".
Also widen the images gitignore to output-*/ so a scratch build cannot leave
multi-GB artefacts stageable.
* feat(images): make the base image reachable — cloud-init, sshd, console
An image built from scratch launched unreachable. The images this replaces
start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a
network renderer, dbus and a serial console from it; mkosi installs only what
is listed, so none of it was there and nothing said so.
Guests get their SSH key and user-data from IMDS via cloud-init, so without
the package a launched instance has no authorized_keys at all — no error, just
no way in. It also made any cloud-init-delivered config inert, including the
gpu profile's docker-group drop-in.
Each addition is load-bearing on its own:
cloud-init consumes IMDS: keys, user-data, hostname, network
openssh-server the key needs a sshd to be installed into
netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is
never configured, IMDS is never reached, and cloud-init
falls back to DataSourceNone
dbus systemd only Recommends it and mkosi installs no Recommends,
so systemctl could not reach the system bus — and the EKS and
ECS bootstraps drive guests with systemctl from user-data
lsb-release cloud-init shells out to it; without it every boot reports
"degraded" forever and the status stops being a usable signal
The kernel also needs console=ttyS0 baked in. The launcher gives every guest a
serial port and get-console-output reads it, but a kernel with no console=
writes to none of it: the console stopped after the UEFI bootloader and every
boot failure was invisible. ttyS0 goes last because Linux makes the final
console= the controlling /dev/console.
manage_etc_hosts is set because cloud-init renames the guest per launch but at
its default never writes a matching hosts entry, so sudo warned on every call.
datasource_list is deliberately NOT pinned: the guest identifies the platform
from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on
its own, and pinning per image is the customisation this migration deletes.
Verified on wattle with an A1000 passed through: ssh in as ubuntu with
spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed
units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s.
* feat(images): let the guest grow its root into the volume it was given
mkosi sizes the disk to its content, so the image ships with a root
partition that fills the disk to its last sector and ~1.4G free. Nothing
in it can expand that: cloud-init's growpart module is enabled, but the
growpart binary lives in cloud-guest-utils, which is not installed and is
not pulled in as a Recommends by anything in the package set. systemd-repart
is not an alternative — Ubuntu's systemd package carries no repart binary,
unit or definitions at all.
The result was a node handed a larger volume that it could only ever use
the first few GiB of, with the rest allocated and unreachable. That is
fatal for a worker whose job is to pull container images.
gdisk is deliberately left out despite being a cloud-init Recommends:
growpart resizes GPT through sfdisk, which arrives via the hard fdisk
dependency, and e2fsprogs brings the resize2fs that the resizefs module
runs afterwards.
Verified on wattle: the gpu-nvidia image launched with a 20G root reports
20G on df -h / (was 4.4G), via
"'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with
nvidia-smi still reporting the A1000 and no failed units.
* feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles
Adds the eks-common, eks-agent and ecs profiles, so both GPU node images
build from the composed base rather than a hand-authored tree, and the three
verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone.
Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the
standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS
node images run k3s or containerd and reach the GPU through CDI. Fusing the
two was fatal for ECS rather than merely wasteful: gpu-nvidia installed
containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu
containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at
all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs
nvidia-ctk to generate its CDI spec.
Closes three gaps the pilot left, each of which only shows up on a booted
guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the
modules on a headless node and CDI generation finds no GPU;
mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on
the toolkit packages, which the unattended-upgrades blacklist does not cover
for a plain `apt upgrade`.
mkosi-build.sh gains --image, naming an output and expanding it to an ordered
profile list. The order is load-bearing: mkosi runs each profile's postinst in
the order the profiles are passed, so docker's nvidia-ctk wiring must follow
gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a
healthy-looking image with no GPU wiring, so docker's postinst also fails
loudly when gpu-nvidia is selected but nvidia-ctk is absent.
Profiles reference build inputs where they already live via ExtraTrees=
instead of copying them, keeping one copy of the helpers shared with the
Alpine image. Go binaries are built on the host into a staging tree, since the
builder container carries no Go on purpose: go.mod is the single pin for the
toolchain and a second pin in the Dockerfile could disagree with it silently.
* fix(images): install systemd-resolved so the guest has a DNS resolver
mkosi installs only the packages named in Packages= plus their hard dependency
closure, and systemd-resolved is merely a Recommends of systemd. So it was never
installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated,
because netplan renders to systemd-networkd and networkd alone does not write
resolv.conf. glibc then fell back to localhost and every lookup failed with
"[::1]:53 connection refused", which reads as a network fault rather than a
missing package.
A node that cannot resolve cannot pull a container image. The ECS GPU node
booted, registered as a container instance, then failed every task at image
pull. The EKS image carries the same defect but is masked by the cloud-init
bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS
launch path has no equivalent, so only ECS exposed it.
* fix(images): name build output by image and never report a skipped build as success
Two defects in the build wrapper that let the wrong bytes reach a boot test.
Every composition wrote the same spinifex.raw, taken from the base config's
ImageId, so an artefact carried no record of which profiles produced it and an
EKS image was distinguishable from an ECS one only by which build ran last.
Passing --image-id names the output after the image, and as a side effect the
name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses
its default input: --name is the published object and --image the local
artefact, and with nothing tying the two together a defaulted input meant
whatever raw happened to be in the output directory could be uploaded under the
name the caller asked for.
mkosi also skips the build whenever the output path merely exists. It does not
compare inputs, so it never notices an edited mkosi.conf or a restaged binary,
prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable
from a fresh one. Builds are now always forced, and the run additionally asserts
the expected raw was written during this invocation rather than trusting the
exit status. The cache worth keeping is the package download volume, which
--force does not touch.
* feat: add gpu-amd mkosi profile for the ROCm AMI
Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under
mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to
stay alive for one remaining image.
Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04
universe ships its own vendored rocminfo, which unpinned apt preferred over
AMD's, mixing release trains within one image.
Uses a systemd unit for render/video group membership instead of a cloud-init
runcmd, because cloud-init's config merge takes a duplicated runcmd key
wholesale from one fragment and silently discards the other, which would have
dropped the docker profile's own usermod.
Runtime GPU verification is deferred — no AMD hardware is available.
* feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images
Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine
systemd units, a templated etcd-snapshot service with daily and 15-minute
timers, the addon/coredns/konnectivity assets, and the six server binaries.
konnectivity-server is not an in-repo package, so it gets a pinned host-side
clone-and-build rather than an entry in profile_binaries().
Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any
attached volume and mulga-mgmt-net configures any multi-NIC system VM, so
neither belongs behind an EKS profile that a database image would have to
compose to reach them.
Start-at-boot moves into preset files. mkosi runs preset-all after every
postinst, so a postinst enable is discarded before the image is finished, and
with no policy file the fallthrough is enable — which built an image that ran a
control plane and a worker at once.
Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base
ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the
directory in reverse and the merge keeps the first source, so a 90- prefixed
file lost the key silently. The postinst now asserts the merged result.
* refactor(eks-node): make the role selector init-agnostic
eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image
the selector enabled nothing and the node never started k3s. Both now dispatch
through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT
overriding detection so either branch is testable on a host that has only one.
Detection prefers rc-update over systemctl: the test suite stubs rc-update on
PATH, and preferring systemctl would have silently rerouted those cases to a
path they were not written for.
Enable and start stay two separate passes. The server path starts
mulga-eks-k3s-recovery before k3s so a recovery directive applies before the
apiserver comes up, and collapsing to systemctl enable --now would interleave
them.
The dispatch is duplicated in both scripts rather than shared: they are
single-file deliverables with no library-path convention between them.
Tests cover both branches across all three roles, including that server-join
still gets no k3s-first-boot, and that rc-update wins when both inits are
present.
* fix(eks-node): enqueue role-selected units instead of blocking on them
eks-node-role.service declares Before=k3s.service, and
mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a
blocking systemctl start deadlocks: systemd will not run the queued job while
the selector is still activating, and the selector cannot finish until that job
runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie
and the node wedges mid-boot with the role written but no control plane. On a
booted guest the selector sat in activating for 261s until the VM was torn down.
Start with --no-block on systemd. The ordering the blocking call was implicitly
providing is already declared between the units themselves, so systemd
sequences them correctly on its own. OpenRC runs the script directly rather
than scheduling it, so that branch keeps blocking and keeps surfacing failures
through set -e.
* fix: escape $ in konnectivity-server ExecStart so server-count survives
systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before
the command runs, substituting an empty string for a name it does not know.
KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes
it to /run/konnectivity-server-args.env, which the wrapper shell sources at
runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw
it. konnectivity-server then died on flag parsing, which is why the unit logged
Started and produced nothing despite --logtostderr=true and journal+console,
and why the apiserver's cluster egress selector found no UDS for the life of
the cluster.
This is a port artefact. konnectivity-server.initd is flag-for-flag identical
and shares the same prestart script, but under OpenRC the shell was the only
thing reading ${KONN_SERVER_COUNT}, so the same text was correct.
Escape it as $$ to defer expansion to the wrapper shell, and add
lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of
a profile's units. The linter joins line continuations first, since the
offending reference sat mid-way through a wrapped multi-line ExecStart, and
runs on the host before the container starts so a bad unit fails the build in
seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd
leaves it literal.
Verified on env12 with a rebuilt image: TestEKS/IRSAPod and
TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed
out` through the egress tunnel, now pass, with zero UDS connect errors across
the run.
* fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw
mkosi only emits the raw. A qcow2 next to it came from a previous run's hand
conversion, and nothing about its name says which build it belongs to, so it
survives a rebuild and silently ships the old image to a node while the raw
beside it carries the new one. Observed on a rebuild that left the raw at 15:34
carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit.
This is the same failure the existing freshness assertion guards against, one
file over, so extend that block rather than add a second check. Refresh rather
than delete: the qcow2 is what gets copied to a node and imported, so removing
it would push the manual conversion, and this trap, back onto the caller.
Convert via a temp name and rename so an interrupted run leaves the old file
rather than a half-written one that looks current.
* fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc
ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns,
which systemd-networkd does not manage, so it shells out to a DHCP client. That
call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not.
`ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the
failure surfaced as an opaque "network setup failed" with the task stuck in
STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and
ServiceWithELB both failed on the mkosi image, the latter as runningCount=0
because its target group targets task ENI IPs.
Resolve the client at setup time, preferring udhcpc so Alpine behaviour is
unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not
sufficient to select it: Debian and Ubuntu's busybox ship the applet without
/usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits
nonzero having configured nothing. Same order and same reasoning as
dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no
daemon behind in the netns.
Install dhcpcd-base in base, because neither client was present in any profile.
That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved
to nothing on Ubuntu and would have failed the first time a cross-host
control-plane data NIC needed it — the S5 path the step-6 plan notes has never
been exercised.
The DHCP-client choice is injected so the tests assert a named platform's
branch rather than whatever happens to be installed on the machine running
them; without that the same test exercises different code on an Alpine CI box
and an Ubuntu dev box.
* refactor: retire the ec2-node profile, fold its helpers into the base image
ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition
had to remember to list it — eks-common even guarded on its presence and
aborted the build without it. ECS did not list it, so the ECS image shipped
with no by-id symlinks for hot-attached volumes at all.
Neither helper is answerable at build time: any image may later be attached a
volume, and any image may later be launched as a control plane. Both are inert
when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its
fw_cfg blob is absent), so the base image is where they belong. That deletes
the composition rule, the guard, and the ECS gap in one move.
Also drops ec2-node from the two eks image compositions in mkosi-build.sh.
Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both
carry the helpers and the udev rule, mulga-ebs-byid is enabled via
multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants.
* fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link
mulga-vpc-mtu only won the boot race by ordering: netplan renders
UseMTU=true (it does not inherit networkd's own false default), so a
DHCP lease renewal at T1 could silently re-apply the advertised MTU to
the link. usemtu_off() forces UseMTU=false via a runtime drop-in in
the systemd-documented [DHCPv4] section for every rendered unit and
reloads networkd, removing the dependency on boot ordering entirely.
assert_pinned() then verifies the link and its routes actually match
before exiting, so any regression is a loud boot-time failure instead
of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has
no systemd-networkd.
* fix(images): state UseMTU=false under both the current section and the legacy alias
netplan renders the DHCP settings under [DHCP] while systemd documents
[DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the
running systemd disagrees, which is the silent no-op the drop-in exists to
prevent.
* fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch
credendpoint.go DNATs the ECS task credential endpoint via iptables; neither
the mkosi ecs profile nor the legacy Alpine manifest installed it, so the
endpoint silently failed to start and any task with a task role could not
fetch credentials while the instance still looked healthy.
Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only,
which this platform's HTTPS AssumeRole path can never match) and adds a
TaskRoleCredentials subtest that fetches real credentials from inside a task
container, so a dead endpoint fails the suite instead of passing unnoticed.
* fix(images): ship preset files for base, eks-common and ecs start-at-boot
mkosi runs systemctl preset-all after every profile's postinst, discarding
whatever systemctl enable a postinst performed. eks-server and eks-agent
already carry preset files for this reason; base, eks-common and ecs still
relied on the default-enable fallthrough, which happens to match today but
is one upstream package change from reversing.
Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net),
30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and
40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables
cni-dhcp.service/socket, which containernetworking-plugins enables by
default with nothing to serve under the baked host-local IPAM conflist).
Removes the now-redundant systemctl enable calls from each postinst,
matching the eks-server/eks-agent precedent.
* fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers
The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal
denied every Service clause other than the IMDS-only EC2 one. A role carrying the
trust policy AWS documents for an ECS task role was unassumable by construction.
Service attribution is now derived from the authenticated SigV4 caller: an
assumed-role session of the account's ecsInstanceRole — the only credential the
in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles
in its own account. Nothing in the request body contributes, so a task holding
its own task-role session cannot claim the service principal.
* fix(images): remove baked SSH host keys, pin Seed= for reproducibility
openssh-server's postinst bakes an identical host key pair into every
build, making every guest from one build mutually MITM-able; sshd-keygen
already regenerates keys on first boot (ConditionFirstBoot=yes), so the
baked pair is pure non-determinism with no purpose. Also pin Seed= so
systemd-repart stops assigning fresh partition UUIDs on every build.
Verified on a built spinifex-ecs-node image: no ssh_host_* files ship,
cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service
is wired via ssh.service.wants/ (redundant regeneration path). A two-build
comparison dropped content-differing files from 9 to 1 (only the UKI,
whose zstd internals remain sensitive to unrelated input changes).
---------
Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>
|
||
|
Josh Siviour
|
2db1034436 |
feat(acm): managed certificate issuance and renewal from a tenant Private CA (#606)
* fix: fan out certificate re-import to referencing load balancers Re-importing new material under an existing ACM ARN previously updated only the stored CertRecord; no load balancer referencing that ARN ever re-rendered its config, so HAProxy kept serving the old leaf until it expired while the API reported success. Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff around every listener create/modify/delete and cert attach/detach in handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook (wired in daemon.go to avoid an import cycle) that re-runs updateStoredConfig for every load balancer in the ARN's InUseBy set, mirroring the existing target-group fan-out. lbagent is unchanged: it already polls ConfigHash and reloads on change. * fix: make InUseBy index updates CAS-safe under concurrent writers AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which loses concurrent writers silently: two listeners on different load balancers attaching the same certificate at once could drop one entry with no error. Because InUseBy is the sole mechanism that fans a renewed certificate out to the data plane, a lost entry means that load balancer never re-renders and its certificate silently expires in HAProxy while ACM reports it as healthy. Switch both to a JetStream KV revision-based compare-and-swap loop with jittered backoff, retrying on a concurrent writer instead of overwriting it. Replace the old concurrency test, which only asserted no errors/panics and so could not detect a lost index entry, with one that asserts exact final state across N concurrent writers. * fix: encrypt ACM certificate private keys at rest CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key. Store.PutCert encrypts when a master key is set; GetCert/ListCerts decrypt, falling back to a narrowly-bounded legacy-plaintext read (only when the stored value is unambiguously a PEM private key block) so pre-existing records keep working and are re-encrypted on next write. A Store never given a key (ELBv2's independent read-only acmStore) behaves exactly as before — encryption is opt-in per Store instance via the new SetMasterKey method, so no other package needs to change. NewACMServiceImplWithNATS now requires a master key and fails construction without one; daemon.go loads master.key inside the initServiceWithRetry closure so a missing key fails daemon startup after the retry window instead of leaving the service permanently degraded. * fix: wire ELBv2's ACM store with the deployment master key ELBv2 constructed its own handlers_acm.Store without a master key, so once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could no longer resolve listener certificates: it read back ciphertext where it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a key so an unkeyed Store can no longer be constructed, thread the deployment master key through NewELBv2ServiceImplWithNATS and the daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM store construction failure fatal instead of silently degrading every HTTPS listener. * feat(acm): tenant Private CA with x509 name constraints Adds an independent tenant root CA (separate from the platform CA) that signs leaf certificates only within its baked-in permitted domains. Name constraints (PermittedDNSDomains, excluded IP ranges, excluded email/URI) are frozen at creation and drift is rejected rather than silently regenerated, so a device's trust store is never invalidated without an explicit operator action. * docs(acm): record OpenSSL verification of the empty email/URI exclusions Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not just Go's own verifier, since this CA is meant for device trust stores. * feat: implement RequestCertificate with PRIVATE_CA managed issuance Adds the ACM RequestCertificate API surface: mode derivation (PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for the deferred modes. Extends CertRecord with issuance/validation/renewal state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md. * feat: wire tenant CA into the running spinifex daemon Add LoadTenantCA as a load-only entry point (never creates), refactor LoadOrCreateTenantCA to share the load path. Add 'spx admin cert create-tenant-ca' to create the tenant CA with explicit permitted domains, idempotent unless --regenerate is passed. Wire tenant CA loading into daemon startup: missing tenant CA logs a warning and degrades PRIVATE_CA issuance only, it does not block startup like the master key does. Fold tenant CA paths into admin's CA path helpers, keeping filenames distinct from the platform CA (tenant-ca.pem/.key vs ca.pem/.key) so the two roots can never be confused. Also give ACM's DeleteCertificate in-use refusal its own awserrors alias instead of reusing the EKS-labelled one directly, and correct COMMANDS.md to reflect that the tagging endpoints are implemented. * fix: derive ACM validation mode from the requested domain, not a global flag Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that northstar is configured, never whether it is authoritative for the domain being requested. Any deployment with a default domain set therefore returned MANUAL_TXT for every request, including domains hosted nowhere, with a TXT record the caller could not publish. Resolve the requested domain against northstar's actual hosted zones, and put MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly delegated so a public CA can resolve the challenge - which is never true of an internal northstar zone. A tenant CA covering every requested domain now wins, and a configured DNS provider still takes precedence over the manual path so operators with a real domain are not downgraded to a private certificate. * feat: renew PRIVATE_CA certificates automatically before they expire Managed issuance signed a leaf once and never renewed it. With 90-day leaves every certificate silently expired a quarter after the first apply, which is the treadmill managed issuance exists to remove. Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully offline, so this needs no ACME client and no network access. The window is proportional - two thirds of the certificate's own lifetime, floored at a minimum - so it self-tunes as issuance lifetimes change rather than degenerating into 'renew immediately, forever' the way a fixed window would. A deterministic per-ARN jitter keeps certificates issued in one apply from all renewing in the same minute. Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes through the ELBv2 fan-out. The push is mandatory rather than incidental: GetLBConfig returns stored configuration and never re-resolves certificate material, so a renewal that did not push would leave HAProxy serving the old leaf until something unrelated changed the config hash. A per-certificate lease keyed on the ARN means one node drives a given renewal without serialising all renewals behind a single leader. Certificates whose domains have fallen outside the tenant CA's name constraints record a failure reason instead of retrying in a tight loop. The renewal scan reads metadata only and never decrypts stored private keys; the lease is a raw compare-and-swap that leaves the encrypted key untouched. * fix: carry handler error messages over the daemon NATS transport handleNATSRequest responded with the sanitized error code alone, so any handler error without a registered code collapsed to a bare ServerInternal and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx has always preserved the message; the two implementations had drifted. Adds respondWithServiceError and routes the 16 service-error sites through it, plus transport-level tests pinning that the message survives for an uncoded error, that a wrapped coded error keeps its real code, and that a bare code carries no message. * feat: implement ACM GetCertificate and stop decrypting keys on read paths GetCertificate was unimplemented end to end: it now returns the PEM certificate body and chain for an owned ARN, ResourceNotFound for an unknown or cross-account ARN, and RequestInProgressException for a certificate that has a record but no body yet, wired through the service interface, the NATS client, the daemon subscription and the gateway action map. Store gains GetCertMetadata alongside GetCert, and ListCerts becomes ListCertMetadata. Every read that answers a question about a certificate rather than serving it - ownership checks, Describe, Delete, tag reads, GetCertificate and listing - now leaves PrivateKey empty instead of running an AES-GCM open per record to produce plaintext key material no caller reads. * fix: reconcile the ACM master-key wiring with dev's new tests The InUseBy and cert fan-out tests that landed on dev construct the ACM and ELBv2 services directly, and both constructors now take the deployment master key. Pass one shared key in the fan-out test - two services over the same bucket disagreeing on the key is the exact failure the wiring exists to stop. A Store is never unkeyed now, so the two test helpers collapse into one. * fix: rebuild the ACM InUseBy index at daemon startup The index that drives certificate fan-out is written only by listener mutations, so a listener whose certificates have not been touched since it was created has no entry. A certificate with an empty index then neither fans out on renewal nor trips DeleteCertificate's in-use guard, and both failures are silent: the renewal reports success while the load balancer keeps serving the old leaf, and a certificate a live listener depends on can be deleted. Reconcile the index from the persisted listener records once at startup. Walking every certificate rather than only the referenced ones makes it a rebuild rather than a backfill, so it also drops entries naming a load balancer that no longer exists. Writing only differences keeps it idempotent, and the index writes are already CAS-guarded, so the nodes of a cluster reconciling the same bucket converge. * fix: deliver listener SNI certificates to HAProxy Add/RemoveListenerCertificates persisted the listener record and then returned, so the config was never re-rendered and lbagent had nothing to poll for. Re-render through reloadListenerLB, the same helper the rule mutations use. That alone changed nothing, because the renderer dropped SNI certificates: resolveFrontendCert returned only the default certificate and the bind line emitted a single crt, with the PEM path keyed on the listener so a second certificate had nowhere to live. Render every attached certificate instead, default first so it stays the no-SNI fallback, each with its own path. HAProxy accepts crt more than once on a bind line and selects by SNI. The default certificate keeps its existing filename so an upgrade does not move a deployed PEM, and the nginx renderer keeps the single-certificate resolver it still relies on. * fix: prune stale TLS certs from the lb-agent cert dir A delivered PEM is cert + chain + private key. writeCertFiles only ever wrote, so every certificate detached from a listener left its key on the load balancer VM's disk indefinitely. Prune after a successful reload, scoped to .pem files in the cert dir's immediate entries. * fix: serve every attached certificate on the nginx NLB engine nginx allows one ssl_certificate per key type in a server block, and the NLB config is a stream block where server_name does not exist. Select by SNI instead: give ssl_certificate a variable and map $ssl_server_name to each certificate's path, keyed on the leaf's DNS SANs. A single-certificate listener keeps a literal path, since a variable costs a per-handshake read. The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so they now take the dir and both engines share one multi-cert resolver. * fix: emit ValidationMethod in DomainValidationOptions DescribeCertificate returned each domain validation entry without a ValidationMethod, so the AWS provider had nothing to compare against aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes. PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits the method on a private certificate, where the provider reads it as NONE whatever the request asked for. |
||
|
Josh Siviour
|
2638a792c5 |
fix(ecs): task-definition round-trip fidelity and a task that dies while the agent is down (#603)
Persist and echo requiresCompatibilities on task definitions, so a Terraform round trip no longer reads null as drift and force-replaces the definition on every apply. Report a task whose containers all exited while the ecs-agent was down. reconcile only re-adopted running containers, so the gateway held such a task RUNNING forever, its service never replaced it, and the load balancer target stayed unhealthy. |
||
|
Julian Sommer
|
d2eb3b11dd | feat: add manual DB snapshots and restore (#601) | ||
|
Julian Sommer
|
d4d9db9e0b | feat: add DB subnet groups and DB parameter groups (#600) | ||
|
Julian Sommer
|
23de2ad85d | feat: ModifyDBInstance and the shared VM-replace helper (#597) | ||
|
Julian Sommer
|
fc6e055ca0 | feat: RDS lifecycle core — reboot, start, stop, delete, events (#596) | ||
|
Julian Sommer
|
32d80b473a | feat: RDS v1 foundations (#592) | ||
|
Josh Siviour
|
6cf9ee9d03 |
fix(dhcp): retransmit within a DORA transaction, resolve the chaddr on every path, stop masking IPAM errors (#593)
* fix(dhcp): retransmit within a DORA transaction instead of restarting it nclient4's retryFn re-sends the same packet under the same xid with a doubling read deadline, which is RFC 2131 4.1 retransmission. WithRetry(1) runs that loop exactly once, so the client transmitted a single DISCOVER per attempt and never retransmitted it. socketTimeout then handed that lone transmission the caller's whole budget, so it sat silent for 4s, then 8s, then 16s, then 32s. Manager does retry, on that schedule, but each attempt builds a fresh client and therefore a fresh xid. Every attempt is a new transaction rather than a retransmission of the previous one, so an OFFER answering an earlier exchange is discarded as belonging to an unknown transaction. Send six transmissions per exchange starting from a 1s read, spanning a 63s ladder — longer than the widest window Manager schedules, so an attempt ends on the caller's context rather than on an exhausted ladder. socketTimeout now clamps to the caller's remaining budget instead of consuming it in one silent read. Observed live recovering an exchange whose OFFER arrived on the third transmission, which a single-shot client would have missed. Both allocation sites also stop replacing the allocator's error with a bare InsufficientAddressCapacity. ResolveErrorCode unwraps, so a genuinely exhausted static pool still surfaces that code while a DHCP fault reports as itself. The flat mapping is why every failure on this path read as a capacity problem regardless of cause. * chore(dhcptest): add --broadcast so the probe can mirror vpcd's Acquire path * fix(dhcp): resolve the chaddr on renew and release, not just on acquire Acquire derived a chaddr when the request carried none. Renew and Release read lease.HWAddr straight out of the store and handed it to nclient4, so a lease persisted before the chaddr was carried renewed with all zeros in chaddr and option 61 fell back to its untyped form. The lease then lands upstream with no hardware address against it — the outcome option 61 was fixed to stop, reached by a different route — and because the server sees an unknown client each time, every renewal takes a new lease. resolveHWAddr now serves all three paths and treats all-zeros as unusable rather than merely empty: six zero bytes passes a width check yet is the exact value that produces the unattributable entry. * test(dhcp): cover chaddr resolution and the unmasked allocation errors |
||
|
Julian Sommer
|
95973eeb3c | refactor: migrate spinifex off legacy nats.JetStreamContext onto nats.go/jetstream (#584) | ||
|
Julian Sommer
|
e79ca59d4b | fix: configure host DNS from spx admin init/join (#582) | ||
|
Julian Sommer
|
12085f029c | feat: render import progress bars in human-readable units (#581) | ||
|
Julian Sommer
|
1d7d5e1750 | fix: disable debug log | ||
|
Julian Sommer
|
1d55404a73 |
revert: host DNS setup on resolved/resolvconf hosts (#580)
This reverts commit
|
||
|
Julian Sommer
|
f9c9f0b38d | fix: configure host DNS for Spinifex zones on resolved and resolvconf hosts (#580) | ||
|
Josh Siviour
|
7e8cbeef6e |
feat(config): enable viperblock chunk GC in rendered node configs (#563)
* feat(config): enable viperblock chunk GC in rendered node configs The gc_enabled toggle was plumbed through every VB construction site but nothing ever set it. GCEnabled is a *bool that every consumer treats as off when nil, and the config template rendered shardwal while omitting gc_enabled, so no deployed node ever carried the key and chunk GC was off everywhere by construction. Superseded chunks were never reclaimed under overwrite churn. Render gc_enabled = true alongside shardwal, and cover it with a test that loads the rendered file through config.LoadConfig rather than matching the template text. A string match would still pass if the value failed to parse into a non-nil pointer, which is the exact way the setting reaches its consumers. Existing deployments keep the old rendering until their config is regenerated; the template governs install and bootstrap only. * chore: pin viperblock to the build with the cross-process GC snapshot marker Enabling chunk GC in rendered node configs is only safe once the sweeping engine can see a snapshot taken by another process. viperblock 3b56d61 adds the per-volume snapshot marker that closes that window, and 2a34add fixes the concurrent-stop panic in the background goroutine lifecycle that GC's own ticker would otherwise widen. Pin both before turning GC on. |
||
|
Josh Siviour
|
291e550e8a |
fix(dhcp): honour the caller's retransmission budget and encode option 61 correctly (#565)
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
|
||
|
Josh Siviour
|
a01e58d90c |
feat: deliver backend ENOSPC to the guest and enable chunk GC in production (#549)
* feat: plumb GCEnabled config through to every VB construction site viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never set it, so every deployed node ran with GC off by construction regardless of which viperblock binary shipped. Add a [viperblock] gc_enabled config key (default false, mirrors shardwal) and thread it through to the three places spinifex builds a VB: the nbdkit plugin it spawns per mounted volume (via a new gc_enabled plugin param), viperblockd's detached-volume opens, and the volume service's CreateVolume. * chore: repin viperblock to GC-enable lineage The gc-enable plumbing references viperblock.VB.GCEnabled, which is absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable commit so a standalone (GOWORK=off) build — the path the ansible deploy uses — resolves the field, and spx plus the nbdkit plugin build from the same GC-bearing viperblock lineage. * fix(storagegrowth): stop launching a nano for image-pull repro ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the synthetic npass suite. A 512MB guest OOM-thrashes containerd before any disk-driven ENOSPC and conflates the storage leak with guest-memory pressure. Add a package-local imagePullInstanceType helper: an exact override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default 16384 MiB). npass keeps using the nano selector unchanged. * feat: report guest block write errors so backend ENOSPC reaches the guest Set werror=report and rerror=report on the guest boot drive so a viperblock ENOSPC (or any other backend write failure) is surfaced to the guest as a real I/O error instead of QEMU silently pausing the VM. Without this the guest kernel never learns its disk is out of space, so it can't fail writes cleanly or let an operator react. * chore: repin predastore and viperblock to the storage-capacity-safety lineage Drop the local-path replace for predastore (a test shortcut) and pin both predastore and viperblock to their feat/storage-capacity-safety commits, so a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC classification, and backpressure plumbing that spinifex's werror/rerror change depends on. * chore: bump viperblock to 054a44a3f0f5 Realigns spx's linked viperblock with the submodule working tree, which the build role's source-drift gate compares against. * chore: repin predastore and viperblock to their dev lineage Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work has now landed on dev in each repo, and both PRs were squash-merged, so the branch commits those pins named are unreachable from any branch. Repin to the dev commits carrying the same code: predastore 5699329 and viperblock 35dd9f2. |
||
|
Josh Siviour
|
08fc5e4c58 |
fix(northstar): initialise OTel SDK and source the telemetry environment (#555)
Northstar carried working northstar.dns.* instruments but shipped no telemetry, so its dashboard was empty and the only northstar data in Elasticsearch was journald scraped under the systemd unit name. Two gaps, both outside the instrumentation. northstarStartCmd never called initTelemetry, unlike every other service command, so the instruments recorded into a no-op provider. The unit also did not source /etc/spinifex/telemetry.env, which every other application unit reads for OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV and MULGA_SOURCE — so even an initialised SDK would have had no endpoint and no env label. Add an invariant covering the second gap: any unit launching an spx service must source the telemetry drop-in. A service missing it starts healthy and silently reports nothing, which only surfaces when someone notices an empty dashboard. Verified the test fails against the unpatched unit. |
||
|
Julian Sommer
|
5b03cf72c4 | fix: prevent live volume tag clobbering (#545) | ||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Josh Siviour
|
3b379fea38 |
fix(ec2,eks,ecs,images): root volume sizing + revive the rotted Ubuntu catalog entries (#542)
* fix(admin): round the AMI root volume up so the image fits
`spx admin images import` sized the root volume with integer division, which
floors: a 4.94 GiB image registered a 4 GiB volume. The image is then copied
into a volume too small to hold it, is truncated, and the guest boots with no
root partition — it stalls on the root device until systemd gives up, and
nothing anywhere reports an undersized volume.
floorVolumeSizeToAMI already exists to prevent exactly this ("Under-sizing
hangs the guest in dracut"), but it raises the caller's requested size to the
AMI's recorded VolumeSizeGiB — the same floored value — so it inherited the
mistake instead of catching it.
This stayed hidden because every current system image is an exact multiple of
1 GiB, where flooring and rounding up agree. It surfaced on a 5306470400-byte
image, which would not boot.
Round up instead, via a helper so the boundaries are testable: an exact
multiple must not gain a spare GiB, and one byte over must.
* fix(eks): default a nodegroup's root volume to 20 GiB, as AWS does
CreateNodegroup stored diskSize verbatim, so omitting it persisted 0 and
the launch skipped its block device mapping entirely, dropping the worker
onto whatever volume the AMI itself declares.
That was harmless while the node AMIs baked generous virtual disks — the
GPU image declares 16 GiB for ~2 GiB of content, so the fall-through
happened to leave plenty of room. It stops being harmless as soon as an
image is sized to its contents: the worker then boots onto a volume with
barely a spare GiB and cannot pull the container images it exists to run.
Sizing the node is the nodegroup's decision either way, not a property
the image should be able to dictate by accident.
20 GiB matches the AWS EKS managed-nodegroup default. It is resolved at
create rather than at launch so DescribeNodegroup reports the size the
node actually gets.
TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping asserted the old
fall-through and is replaced by its inverse; the explicit-size case is
already covered by TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping.
* fix(ecs): size a container instance's root volume to 30 GiB by default
ProvisionCapacity built its RunInstances input without a block device
mapping, so a container instance inherited whatever volume its AMI
declares. EC2 has no default of its own to fall back on — the AMI is the
only thing that decides — and a node AMI is sized to hold the image, which
leaves the agent no room for the container images it exists to pull.
30 GiB mirrors what the AWS ECS-optimized AMI declares. ProvisionCapacity
is spinifex's own API rather than an AWS one, so DiskSize is added to its
input for callers that want to size a cluster explicitly, defaulting when
omitted the same way InstanceType and Count already do.
* fix(images): pull the Ubuntu catalog images from releases, not a pinned daily
Both Ubuntu catalog entries pointed at a dated build under the codename
dailies tree, which Canonical prunes as builds age out. That path now 404s,
so "spx admin images import --name ubuntu-26.04-x86_64" fails outright for
x86_64 and arm64 alike — the entries had rotted with nothing to catch it.
Release builds are retained rather than pruned, so the pin stays resolvable
while keeping a specific, reproducible build. The filenames under releases/
carry the version instead of the codename, which matters because the sums
entry is matched on the URL's basename.
CI is unaffected and could not have caught this: it fetches resolute/current/,
a symlink that never rots, so nothing exercised the pinned path.
Debian pins a dated directory too but is fine — those builds are retained,
and its entries still resolve.
|
||
|
Julian Sommer
|
73eeb87751 | chore: enable testifylint | ||
|
Julian Sommer
|
b5198d55e2 | chore: enable depguard to block log | ||
|
Julian Sommer
|
ac5bd613b4 | chore: enable godot | ||
|
Julian Sommer
|
03d981779b | chore: enable errorlint | ||
|
Julian Sommer
|
b40d538c61 | refactor(admin): remove dead --no-external flag from spx admin init (#537) | ||
|
Julian Sommer
|
744cb1929d | fix: admin init --force preserves keys, credentials and CA (#536) | ||
|
Josh Siviour
|
98fc01764b |
feat(ecs): GPU passthrough to task containers (Epic C) (#512)
* feat(ecs): thread resourceRequirements GPU count through conv, ContainerDef, AssignContainer and RunSpec * feat(ecs): add GPU placement dimension — InstanceRecord capacity, fits/reserve/release, STRINGSET resource reporting * feat(ecs): agent GPU discovery via nvidia-smi, pinning ledger, device-UUID report-back and DescribeTasks gpuIds * feat(ecs): inject assigned GPU devices into containerd OCI spec via CDI (oci.WithCDIDevices) * test(ecs): add GPU task e2e (nvidia-smi container, gpuIds assertion) to gpu e2e suite * test(gpu): decouple base-AMI requirement from GPU fixture so ECS e2e runs without the ubuntu EC2 image * test(gpu): pick smallest advertised GPU instance type to fit a single host * fix(ecs): preserve pinned GPU UUIDs across STOPPED state change so DescribeTasks reports gpuIds for one-shot tasks |
||
|
Josh Siviour
|
183a08353e |
feat(network): harden routed-NAT v1 (VPC cap, e2e gate, docs) (#504)
Three related nat-mode fixes: - ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template so operators can widen any pool. - 724mm: detectPoolMode returns true only for external_mode=pool. nat and disabled clusters now skip Phase 8b/8d, which otherwise failed on AllocateAddress UnsupportedOperation. - pnz2i: document that system instances (ECS/EKS/LB agents) are unsupported in nat v1 (require pool mode) in the vpc-networking README and COMMANDS.md; document the gw_lrp_range_* fields. |
||
|
Josh Siviour
|
d663760dec |
feat(eks): control-plane disaster recovery — etcd snapshot + restore-snapshot (#503)
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for
|
||
|
Julian Sommer
|
635915b5b0 | feat(spx): surface OVN NB/SB Raft roles in get nodes (#493) | ||
|
Julian Sommer
|
20288dbe5d | refactor: dead-code cleanup and helper deduplication (#486) | ||
|
Josh Siviour
|
e1c4f85a36 |
feat: routed-NAT external mode with two-tier ingress (#475)
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets. |
||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
Josh Siviour
|
e47baf321c |
feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480)
* feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight. |
||
|
tomnewton-mulga
|
4f5fd88c86 |
feat: snapshot ami (#279)
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation - Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe) - Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels - Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type) - Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles - GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors * fix(daemon): enforce CPU/memory limits for MIG instance types Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate). MIG types now fall through to canAllocateCount so host CPU and RAM are checked before launch — prevents over-committing a host where one physical GPU backs multiple concurrent slices, each with non-trivial vCPU/memory allocation. * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * fixed testing * more tests * fix: added new graceful process killer in utils, bumped NATS timeout for testing * fix: complete at-rest encryption launch + volume lifecycle (#361) * chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged) * fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) * feat: cloud-init EC2 datasource (#411) * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * merge changes * chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never set post-rebase. Removes the field from MountedVolume and the two nil-checked unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig. * fix: post-rebase gofmt fixes and remove dead snapshot handler code Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go, viperblockd.go, volumes.go, and volume_durability_test.go. Removes the dead unix-socket snapshot handler tests from viperblockd_handlers_test.go. * test: add viperblock snapshot S3 coverage tests Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume past the nil-config guard using an httptest server that returns 403. The AWS SDK credential check fails synchronously without network I/O, keeping the tests fast while covering the S3Config construction, master key load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%. * fix: cloneAMIToVolume reads config via format-agnostic path LoadStateRequest expects a viperblock binary envelope (encrypted), but CreateImageFromInstance writes the AMI config as plain JSON via putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig which uses viperblock.StateBody to handle both formats, matching the path used by ListImages and all other AMI readers. * fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint. CI builds without go.work so it resolved to the published version and failed. Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged. * fix: replace indexOf with slices.Index, revert e2e runner and viperblock version - nbd_test.go: remove local indexOf helper, use slices.Index from stdlib - e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting - go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges) * fix: remove pseudo-version pin from go.sum * feat: drain viperblock via snapshot socket before CreateSnapshot Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock and wait for DrainToBackend() to complete (30s timeout). This ensures WAL chunks and the live checkpoint are current in S3 before the snapshot VB reads them. If the socket is absent the instance is stopped and Close() already drained. fix: socket path used s.config.WalDir which is empty in the daemon config; changed to s.config.DataDir + "/viperblock" to match the path the NBD plugin uses when creating the socket. * feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint - Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot) - Deregister from image detail (DeregisterImage with navigate-away on success) - Promote to System from image detail (admin-only, irreversible via UI) - NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage - signedFetch extended with optional params to support action-keyed extra fields * refactor: address PR review on PromoteImage types and subscription ordering - Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult (add JSON tags; drop the duplicate pair) - Inline the input variable in gateway/spx/promote_image.go - Move spinifex.image.promote subscription to the spinifex.* group in daemon.go * test: cover snapshotVolume/Running/Stopped error paths for diff coverage Add encryption-key-load-error and LoadState-failure tests for the three new viperblock snapshot paths (snapshotVolume in snapshot service, snapshotRunningVolume and snapshotStoppedVolume in image service). Diff coverage: 58.8% → 65.5%. --------- Co-authored-by: Josh Siviour <mrk@mulgadc.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Julian Sommer
|
473185c0ba | chore: remove stale license comments | ||
|
Josh Siviour
|
dfd7cfaa2b |
fix: ECS StopTask reaps container before releasing capacity (siv-484) (#470)
* fix: ECS StopTask reaps container before releasing capacity StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in the control plane, but nothing ever told the ecs-agent to kill the container. The container kept holding its host ports, so DescribeTasks reported a STOPPED task whose container was still RUNNING, and a replacement task placed on the freed capacity that rebound the same host port hit EADDRINUSE. Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165): - Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective. - requestStopTask marks DesiredStatus=STOPPED and posts a stop directive, leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService, and service scale-in use it; the heartbeat reaper and cluster/instance teardown keep the immediate forceStopTask since no live agent can reap. - PollAssignments returns pending stops; the agent lists containers by mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns), then reports STOPPED. recordTaskState performs the single capacity release on the agent's STOPPED report and reclaims the stop inbox. - listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively stopped task drops out of the service's running total immediately. DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the container is actually reaped, matching AWS ordering and eliminating the port race. * test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges * ci: retrigger PR checks |
||
|
Josh Siviour
|
857de1ed80 |
feat: ECS P2 hardening — taskdef validation, logging signal, execution role, rolling deployments (#472)
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException) and warns on unsupported log drivers instead of silently dropping them. - Capture logConfiguration; only host-side json-file is collected, other drivers warn at register and reach the agent for parity. Document json-file retrieval. - Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to authorize ECR pulls, falling back to the instance role when unset. - Add deployment objects with a health-gated rolling update honoring deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment circuit breaker with optional rollback to the last-good task definition. |
||
|
Julian Sommer
|
4c1f7864f8 | chore: more cleanup | ||
|
Julian Sommer
|
bae2dfe30f | chore: use shared ExternalPoolConfig type + fix ip math | ||
|
Josh Siviour
|
7bbd99c792 |
feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459)
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
|
||
|
Julian Sommer
|
7371358537 | chore: more cleanups |