Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then
restarted ovn-controller without waiting for the Southbound RAFT to elect a
leader, serve clients, and for northd to write the gateway logical flows. With
no join step to absorb the boot window the controller attached mid-election,
programmed a partial datapath (SNAT ct-commit without output/delivery) and
never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed
on the single-node baremetal CI host.
Add an SB readiness gate before ovn-controller starts (SB reachable and, when
clustered, an elected leader), and after ovn-controller restart wait for its SB
connection then force inc-engine/recompute to rebuild all OpenFlow from the
converged Southbound. Applies to 1/2/3-node RAFT alike.
The failed state existed and was unreachable: nothing drove available →
failed, so DescribeDBInstances reported available for a database that had
been dead for a week. Add the failure classifier on top of the rds-4
reconciler.
An instance is failed when its system VM is not running AND its agent
heartbeat is stale — never either alone, since a stale beat under a live VM
is an agent fault rather than a dead database. The transition records a
timestamped reason that DescribeDBInstances and DescribeEvents can explain,
and a healthy heartbeat clears it again, which is the recovery the AMI's
in-guest Restart=on-failure and EC2's VM auto-restart provide underneath.
The recovery ladder and its fencing gate stay deferred to v1.1.
The failure clock is persisted rather than held in leader memory, so a
leader change does not restart the window, and it resets only on a healthy
heartbeat so a VM that boots and immediately wedges cannot mask a
persistent fault. Every lifecycle-owned state is skipped outright, which is
what keeps the classifier from racing rds-5a/rds-5b over the same VM.
A persisted heartbeat is judged against the stale window plus the persist
floor: beats are queue-group delivered, so the leader mostly sees them only
through the record, which a healthy agent refreshes no more often than
every five beats. The steady-state path is answered from the record alone,
so a healthy fleet costs no VM describe fan-outs.
* fix(test): reclaim fixture scratch dirs stranded in /tmp
The shared predastore fixture and the integration suite's JetStream store both
outlive any individual test, so neither can use t.TempDir() - the first test to
finish would delete files the daemon still has open. Both used os.MkdirTemp
with nothing removing the directory afterwards, so every run stranded one, and
a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp
is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and
swapped tmpfs pages have no backing store to be dropped to.
Add a shared sweep that reclaims a directory only when it matches both the
prefix and an age no live run can reach. The NATS store also has a clean
shutdown point, so TestMain now removes it outright and the sweep is left to
cover the killed-run case. The predastore daemon runs until the process exits
and has no such point, so the sweep is its only mechanism.
* chore: repin viperblock to the dev tip carrying the scratch-dir fix
Test-only upstream change, so the build output is unchanged - this keeps
go.mod aligned with the submodule pointer rather than delivering a fix.
* fix(dhcp): retransmit within a DORA transaction instead of restarting it
nclient4's retryFn re-sends the same packet under the same xid with a
doubling read deadline, which is RFC 2131 4.1 retransmission. WithRetry(1)
runs that loop exactly once, so the client transmitted a single DISCOVER
per attempt and never retransmitted it. socketTimeout then handed that
lone transmission the caller's whole budget, so it sat silent for 4s,
then 8s, then 16s, then 32s.
Manager does retry, on that schedule, but each attempt builds a fresh
client and therefore a fresh xid. Every attempt is a new transaction
rather than a retransmission of the previous one, so an OFFER answering
an earlier exchange is discarded as belonging to an unknown transaction.
Send six transmissions per exchange starting from a 1s read, spanning a
63s ladder — longer than the widest window Manager schedules, so an
attempt ends on the caller's context rather than on an exhausted ladder.
socketTimeout now clamps to the caller's remaining budget instead of
consuming it in one silent read. Observed live recovering an exchange
whose OFFER arrived on the third transmission, which a single-shot
client would have missed.
Both allocation sites also stop replacing the allocator's error with a
bare InsufficientAddressCapacity. ResolveErrorCode unwraps, so a
genuinely exhausted static pool still surfaces that code while a DHCP
fault reports as itself. The flat mapping is why every failure on this
path read as a capacity problem regardless of cause.
* chore(dhcptest): add --broadcast so the probe can mirror vpcd's Acquire path
* fix(dhcp): resolve the chaddr on renew and release, not just on acquire
Acquire derived a chaddr when the request carried none. Renew and Release
read lease.HWAddr straight out of the store and handed it to nclient4, so
a lease persisted before the chaddr was carried renewed with all zeros in
chaddr and option 61 fell back to its untyped form. The lease then lands
upstream with no hardware address against it — the outcome option 61 was
fixed to stop, reached by a different route — and because the server sees
an unknown client each time, every renewal takes a new lease.
resolveHWAddr now serves all three paths and treats all-zeros as unusable
rather than merely empty: six zero bytes passes a width check yet is the
exact value that produces the unattributable entry.
* test(dhcp): cover chaddr resolution and the unmasked allocation errors
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
* fix: detach before delete on instance terminate to stop root-volume leak
TerminateStoppedInstance and the running-instance terminate path both
deleted a DeleteOnTermination root volume without first clearing its
attachment, so DeleteVolume's in-use guard rejected it and the error was
swallowed (stopped path) or only alarmed (running path), leaking the
volume permanently.
Add VolumeServiceImpl.DeleteVolumeOnTerminate, which clears the stale
attachment via UpdateVolumeState (terminate implies detach) before
deleting, and route both terminate paths through it. Stamp
Teardown[TeardownVolumes] on the stopped path so a delete failure is
visible to the volume leak reaper instead of being lost.
* fix: detach non-DeleteOnTermination root on terminate; cover teardown self-heal
A DeleteOnTermination=false root volume stayed attached to the terminated
instance on both terminate paths: Stop's Unmount deliberately skips Boot
volumes, so the root kept its attachment across stop and neither terminate
path detached it, stranding it attached to an instance that no longer
exists. Add VolumeServiceImpl.DetachVolumeOnTerminate (metadata-only
attachment clear, no delete) and route the DeleteOnTermination=false Boot
volume through it on both paths, matching AWS terminate-implies-detach.
Also add a regression test proving the TerminatedTeardownReaper re-drives a
failed volumes-teardown through the now-detach-first cleaner and self-heals
the leak, so a transient delete failure at terminate does not strand a
volume permanently.
* fix: bound detach chain and reconcile vanished-QEMU wedge
DetachVolume gains a bounded aggregate context deadline over the QMP
hot-unplug chain and a confirmed-dead-QEMU short-circuit that skips the
QMP steps (which would only hang on the wedged process) and seals
directly, so a detach can no longer pin attachMu forever when nbdkit or
QEMU die mid-flight.
OrphanQEMUReaper now also finalizes shutting-down instances whose QEMU
has vanished: a terminate that wedged downstream and never reached
finalizeTerminated is driven to terminated, with outstanding teardown
stamped failed so TerminatedTeardownReaper completes it idempotently.
Gated on a dead process, so a live in-progress terminate is never
stolen.
* feat: add delete-authorized stuck-terminate backstop
A terminate that wedges with QEMU still alive is invisible to the
OrphanQEMUReaper dead-QEMU reconcile and would sit shutting-down forever.
StuckTerminateReaper force-completes it: past stuckTerminateTimeout it
force-kills the wedged process, reclaims DeleteOnTermination volume space
through the cleaner, stamps remaining teardown for TerminatedTeardownReaper,
and drives the record to terminated.
It is the one reaper explicitly authorized to delete volume data, but only
for a DeleteOnTermination volume on an already-terminating instance — a far
narrower licence than ADR-0005 §3 denies VolumeLeakReaper, which stays
mark-and-alarm-only. A new ShuttingDownAt timestamp, stamped once on the
transition, bounds how long a terminate may wedge; a missing timestamp reads
as not-yet-stuck so the backstop never acts on an unbounded record.
TestNPassOverwrite detaches after every pass, so viperblock's drain and GC
only ever run at reopen — it cannot show whether GC reclaims superseded
chunks while a volume stays open, and its serial write rate never nears the
predastore rate limiter's burst. TestContinuousChurn attaches once and drives
concurrent 4 MiB-aligned raw overwrites of per-writer regions for several
minutes without detaching, so a sustained superseded-chunk backlog builds
under one long open, spanning multiple GC ticks.
The workload is bounded and self-contained: each dd writes one chunk and the
clock is re-checked after every write, the loop runs as root under a single
sudo bash with a process-group kill trap, and it is wrapped in a guest-side
timeout. Root ownership is load-bearing — the writers open a raw device, so
only a root-owned process group can reap them; a best-effort guest pkill runs
before detach as a backstop so a stray writer cannot wedge the detach.
Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a
dedicated goroutine instead of sharing the chunk-uploader select, so
sustained overwrite churn no longer starves GC and wedges the volume at
the internal space cap.
This reverts commit f9c9f0b38d.
#580 switched setup.sh to read the northstar :53 listener and zones from
northstar.toml, so the resolved stage now defers at install time (before
formation) and must be re-run afterward. Only firstboot re-runs it; the
manual setup.sh + spx admin init install and CI never do, so those hosts
get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node.
Reverting restores the pre-#580 behaviour: systemd-resolved is configured
at install time from the WAN IP, with no post-formation re-run needed. This
reintroduces the original gap where resolvconf hosts get no host DNS. The
proper fix configures host DNS from spx admin init/join for both resolver
managers (mulga-x30yv).
A freshly launched LB agent begins heartbeating the instant the VM boots,
but the gateway node fielding that first heartbeat may not yet have the LB
record — it is still replicating from the node that created it — so the
gateway answers 400 LoadBalancerNotFound. The IMDS credential chain is
likewise not always warm on the first tick. Both self-heal within a few
ticks, yet every early failure was logged at ERROR, flooding the console
telemetry with startup noise (46 such lines across 8 agents in one run).
Bound these to a registration grace window: before any heartbeat has ever
succeeded, and only within the window, a failure logs at INFO. Past the
window with no success — the fingerprint of a genuinely blackholed agent —
or after any prior success, a failure logs at ERROR as before, so the
serial-console telemetry still surfaces a real blackhole. The only change
for a persistent failure is a bounded delay before it escalates.
* fix: make the seal-failure unmount test deterministic
The test forced the seal to fail by pointing S3Host at an unreachable
endpoint, so the handler only returned once the S3 client had exhausted
its jittered backoff. Measured handler time spanned 1.8s to 5.5s against
a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and
worse on a loaded CI runner.
Config gains a sealVolume seam, nil meaning the real seal, and the test
injects an immediate failure instead. It now runs in 0.53s, 50 for 50,
and still asserts what it was written to assert -- plus that the handler
actually attempted the seal, which the old form could not distinguish.
* chore: bump viperblock to dev tip (GC observability + scratch sweep)
Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info)
and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).
The tier ran inside the Build and Test job, serialised ahead of test-cover, and
again in preflight. Neither placement bought signal that a parallel job does not:
it provisions nothing and finishes in seconds, so the only effect was wall time
on the two paths developers wait for most.
It stays PR-blocking as its own job, which is what distinguishes it from
e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so
never gates a PR before merge.
Julian Sommer