* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
* fix: detach before delete on instance terminate to stop root-volume leak
TerminateStoppedInstance and the running-instance terminate path both
deleted a DeleteOnTermination root volume without first clearing its
attachment, so DeleteVolume's in-use guard rejected it and the error was
swallowed (stopped path) or only alarmed (running path), leaking the
volume permanently.
Add VolumeServiceImpl.DeleteVolumeOnTerminate, which clears the stale
attachment via UpdateVolumeState (terminate implies detach) before
deleting, and route both terminate paths through it. Stamp
Teardown[TeardownVolumes] on the stopped path so a delete failure is
visible to the volume leak reaper instead of being lost.
* fix: detach non-DeleteOnTermination root on terminate; cover teardown self-heal
A DeleteOnTermination=false root volume stayed attached to the terminated
instance on both terminate paths: Stop's Unmount deliberately skips Boot
volumes, so the root kept its attachment across stop and neither terminate
path detached it, stranding it attached to an instance that no longer
exists. Add VolumeServiceImpl.DetachVolumeOnTerminate (metadata-only
attachment clear, no delete) and route the DeleteOnTermination=false Boot
volume through it on both paths, matching AWS terminate-implies-detach.
Also add a regression test proving the TerminatedTeardownReaper re-drives a
failed volumes-teardown through the now-detach-first cleaner and self-heals
the leak, so a transient delete failure at terminate does not strand a
volume permanently.
* fix: bound detach chain and reconcile vanished-QEMU wedge
DetachVolume gains a bounded aggregate context deadline over the QMP
hot-unplug chain and a confirmed-dead-QEMU short-circuit that skips the
QMP steps (which would only hang on the wedged process) and seals
directly, so a detach can no longer pin attachMu forever when nbdkit or
QEMU die mid-flight.
OrphanQEMUReaper now also finalizes shutting-down instances whose QEMU
has vanished: a terminate that wedged downstream and never reached
finalizeTerminated is driven to terminated, with outstanding teardown
stamped failed so TerminatedTeardownReaper completes it idempotently.
Gated on a dead process, so a live in-progress terminate is never
stolen.
* feat: add delete-authorized stuck-terminate backstop
A terminate that wedges with QEMU still alive is invisible to the
OrphanQEMUReaper dead-QEMU reconcile and would sit shutting-down forever.
StuckTerminateReaper force-completes it: past stuckTerminateTimeout it
force-kills the wedged process, reclaims DeleteOnTermination volume space
through the cleaner, stamps remaining teardown for TerminatedTeardownReaper,
and drives the record to terminated.
It is the one reaper explicitly authorized to delete volume data, but only
for a DeleteOnTermination volume on an already-terminating instance — a far
narrower licence than ADR-0005 §3 denies VolumeLeakReaper, which stays
mark-and-alarm-only. A new ShuttingDownAt timestamp, stamped once on the
transition, bounds how long a terminate may wedge; a missing timestamp reads
as not-yet-stuck so the backstop never acts on an unbounded record.
TestNPassOverwrite detaches after every pass, so viperblock's drain and GC
only ever run at reopen — it cannot show whether GC reclaims superseded
chunks while a volume stays open, and its serial write rate never nears the
predastore rate limiter's burst. TestContinuousChurn attaches once and drives
concurrent 4 MiB-aligned raw overwrites of per-writer regions for several
minutes without detaching, so a sustained superseded-chunk backlog builds
under one long open, spanning multiple GC ticks.
The workload is bounded and self-contained: each dd writes one chunk and the
clock is re-checked after every write, the loop runs as root under a single
sudo bash with a process-group kill trap, and it is wrapped in a guest-side
timeout. Root ownership is load-bearing — the writers open a raw device, so
only a root-owned process group can reap them; a best-effort guest pkill runs
before detach as a backstop so a stray writer cannot wedge the detach.
Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a
dedicated goroutine instead of sharing the chunk-uploader select, so
sustained overwrite churn no longer starves GC and wedges the volume at
the internal space cap.
This reverts commit f9c9f0b38d.
#580 switched setup.sh to read the northstar :53 listener and zones from
northstar.toml, so the resolved stage now defers at install time (before
formation) and must be re-run afterward. Only firstboot re-runs it; the
manual setup.sh + spx admin init install and CI never do, so those hosts
get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node.
Reverting restores the pre-#580 behaviour: systemd-resolved is configured
at install time from the WAN IP, with no post-formation re-run needed. This
reintroduces the original gap where resolvconf hosts get no host DNS. The
proper fix configures host DNS from spx admin init/join for both resolver
managers (mulga-x30yv).
A freshly launched LB agent begins heartbeating the instant the VM boots,
but the gateway node fielding that first heartbeat may not yet have the LB
record — it is still replicating from the node that created it — so the
gateway answers 400 LoadBalancerNotFound. The IMDS credential chain is
likewise not always warm on the first tick. Both self-heal within a few
ticks, yet every early failure was logged at ERROR, flooding the console
telemetry with startup noise (46 such lines across 8 agents in one run).
Bound these to a registration grace window: before any heartbeat has ever
succeeded, and only within the window, a failure logs at INFO. Past the
window with no success — the fingerprint of a genuinely blackholed agent —
or after any prior success, a failure logs at ERROR as before, so the
serial-console telemetry still surfaces a real blackhole. The only change
for a persistent failure is a bounded delay before it escalates.
* fix: make the seal-failure unmount test deterministic
The test forced the seal to fail by pointing S3Host at an unreachable
endpoint, so the handler only returned once the S3 client had exhausted
its jittered backoff. Measured handler time spanned 1.8s to 5.5s against
a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and
worse on a loaded CI runner.
Config gains a sealVolume seam, nil meaning the real seal, and the test
injects an immediate failure instead. It now runs in 0.53s, 50 for 50,
and still asserts what it was written to assert -- plus that the handler
actually attempted the seal, which the old form could not distinguish.
* chore: bump viperblock to dev tip (GC observability + scratch sweep)
Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info)
and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).
The tier ran inside the Build and Test job, serialised ahead of test-cover, and
again in preflight. Neither placement bought signal that a parallel job does not:
it provisions nothing and finishes in seconds, so the only effect was wall time
on the two paths developers wait for most.
It stays PR-blocking as its own job, which is what distinguishes it from
e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so
never gates a PR before merge.
MgmtIPAllocator allocated from a process-local map, but br-mgmt is
trunked onto a shared physical NIC making the /24 one flat L2 segment
across the whole cluster. Every node independently allocated from
.10, causing collisions and ARP-race losers being blackholed.
Rewrite the allocator on the spinifex-cluster-state NATS KV bucket
using compare-and-swap, keyed by mgmt-ipam.<subnet>, modeled on IPAM.
Bind the KV handle in startCluster (not startLocal) to respect the
DDIL boundary. Serve already-allocated instance IDs from the local
cache when KV is unhealthy, but refuse new allocations until it
recovers. Rebuild now reconciles this node's running VMs into the
cluster-wide record instead of just refreshing the local cache.
* feat(config): enable viperblock chunk GC in rendered node configs
The gc_enabled toggle was plumbed through every VB construction site but
nothing ever set it. GCEnabled is a *bool that every consumer treats as
off when nil, and the config template rendered shardwal while omitting
gc_enabled, so no deployed node ever carried the key and chunk GC was off
everywhere by construction. Superseded chunks were never reclaimed under
overwrite churn.
Render gc_enabled = true alongside shardwal, and cover it with a test that
loads the rendered file through config.LoadConfig rather than matching the
template text. A string match would still pass if the value failed to parse
into a non-nil pointer, which is the exact way the setting reaches its
consumers.
Existing deployments keep the old rendering until their config is
regenerated; the template governs install and bootstrap only.
* chore: pin viperblock to the build with the cross-process GC snapshot marker
Enabling chunk GC in rendered node configs is only safe once the sweeping
engine can see a snapshot taken by another process. viperblock 3b56d61 adds the
per-volume snapshot marker that closes that window, and 2a34add fixes the
concurrent-stop panic in the background goroutine lifecycle that GC's own ticker
would otherwise widen. Pin both before turning GC on.
LaunchRunInstances is a best-effort batch launcher that reports failure only
through VM state: volume preparation, GPU claim and vmMgr.Run errors each call
MarkFailed and continue. launchAMISystemInstance ignored that and always
returned a SystemInstanceOutput, so an EKS control-plane server, ELB or Bedrock
serving VM whose launch failed handed its caller an instance ID for a VM that
was already being torn down.
Check the instance reached running state before returning. Cleanup stays with
the MarkFailed chain LaunchRunInstances has already run.
buildDrives emitted every non-boot data volume as a bare -drive with no
id= and no node-name=, so a stop/start (or reboot/resize) relaunch left
the volume anonymous: device_del/blockdev-del had nothing to address,
QEMU kept the NBD socket open, and DetachVolume stalled through
nbdkit's full kill grace before giving up.
Add -blockdev support to the QEMU config builder and use it for
non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id
naming and virtio-blk-pci device arguments with AttachVolume's hot-plug
path so cold-boot and hot-attach can no longer drift apart. A data
volume with no persisted HotplugPort now gets one allocated and written
back during buildDrives instead of landing on an unplugged bus.
Also raise unmountSealTimeout past viperblockd's kill grace (it runs
the seal after the SIGKILL, so matching the grace meant an unmount
that used the whole grace could never be observed as successful), and
bound the e2e harness's forced teardown detach by the teardown
deadline so a slow detach is reported as slow rather than as an
unexplained leak.
* test: assert the gateway->vpcd->OVN crossing in the integration tier
The tier already ran the real VPCServiceImpl for the AWS-facing half, but
nothing consumed the vpc.* events it published: the SG topics were answered
by a canned {"success":true} and the fire-and-forget topology topics were
dropped. Both sides of that wire were tested only against stubs, so the join
was asserted nowhere below the live e2e tier.
Wire a real subscribers.Subscriber over a real in-process OVN Northbound DB
(ovntest.StartNB), opt-in per test so only tests that assert OVN state pay
the startup cost. StartDaemonLite gains WithRealVPCD to suppress its stubs,
since a stub and the subscriber both answering one request-reply would race.
Scope is the crossing, not NB row shapes — those stay with the reconcile
scenario tests. Everything host-dependent (ovn-controller, OVS, taps, netns,
nftables, packet paths) remains the live tier's claim.
* fix: make the integration tier re-runnable and quiet the ensure probe
Two defects the vpcd/OVN work surfaced.
The tier was green at -count=1 and red package-wide at -count=2. Test
accounts were keyed by t.Name(), which is unique among tests but repeats
across passes in one TestMain, so a second pass inherited the first pass's
JetStream namespace — including an ECR signing key sealed with a master key
StartGateway regenerates per call. Key each account off a counter instead.
TestRunInstances_ClientTokenIdempotency then collided with itself for the
same reason: its dedup store outlives a pass, so its fixed ClientToken made
the second pass's opening launch look like a replay. Suffix the tokens per
invocation.
ensureNamedRowOps' wait-op is built to fail fast when the row it guards
already exists, which is the ordinary idempotent path, but transactOps
reported it as "OVSDB operation failed" at ERROR. Every re-ensure emitted
one, drowning real faults in the log sink. Classify that one result as the
probe outcome it is and log it at Debug.
* test: cover transactOps' failed-result paths
The classifier split one log line into two branches and neither was
exercised: transactOps sat at 31% because EnsureLogicalRouter's cache
lookup absorbs a re-ensure before it ever transacts, so no unit test
reached a failed operation result.
Drive both kinds directly against a real NB — the guard probe finding an
existing row, and two same-named inserts violating the schema's name
index — since only the second is a fault and they now log at different
levels. transactOps goes to 94%; the remainder is the transport-error
return, which needs a broken connection to reach.
* feat(ci): extend per-cell telemetry scoping to every e2e workflow
ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil
or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it
on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil
and gpu on their single-job workflows.
The best-effort numeric job id lookup moves out of e2e-nightly into a shared
composite action, so the four other provisioning jobs get ci.job_id without
copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather
than interpolating the name into the jq filter, which broke on names carrying
quotes or non-ASCII separators, and no-ops when telemetry is disabled.
gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job
bakes the attrs into a daemon, so the API call would be paid for an env var
no exporter reads.
* fix(ci): define e2e suite topology eligibility once
Three workflows carried three different answers to which suites a topology
can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where
every internet-facing subtest skips for want of a peer, so a fraction of the
suite reported as a green lb run. e2e-suites' multi leg ran multinode alone,
dropping the lb and cert coverage that only a multi-node env can complete —
so full lb coverage never ran there at all.
.github/scripts/suite-sets.sh is now the single definition, sourced by all
three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives
both matrices from it (multi_provision gains the suite-matrix step
single_provision already had), and e2e-nightly's deliberately narrower
per-cell lists are declared there too, guarded by an assert that fails the
cell if a subset ever names a suite its topology cannot fully exercise.
The nightly stays narrow on purpose: its cells have a 35-minute budget and
exist to prove each install/network/host-OS permutation boots and serves.
Suite breadth is the PR gate's job and already runs on every push. That is
now stated in one place rather than implied by a third divergent list.
* test(volume): assert chunk GC stays wired on in CreateVolume
GCEnabled was zero-valued at every viperblock construction site for a long
time and no test noticed. It is not part of viperblock.VBState, so it never
round-trips through a saved config.json and cannot be observed by reading
volume state back.
Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so
the GC wiring is assertable directly against the returned config, without a
live viperblock or S3 backend. The config is returned by pointer because
viperblock.VB carries atomic.Uint64 fields that vet's copylocks check
rightly refuses to see copied.
* test(e2e): snapshot backend byte accounting before volume purge
DeleteVolume purges a volume's entire backend prefix, including any chunks a
leak left behind, before anything measures it. That is the same masking that
hid the storage-growth leak in production: a passing E2E cannot fail on
backend growth when the evidence is deleted first.
Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup
closure, immediately before DeleteVolume, so every volume the shared teardown
path purges is measured rather than silently discarded. The snapshot is
always taken and logged; the hard ratio check additionally runs for callers
that registered a guest-written-bytes baseline, since most callers never
attach or write to the volume they create and so have no baseline to compare
against.
The tolerance is the RS(2,1) erasure floor widened for per-chunk header and
tail-chunk overhead. It is sized to distinguish unbounded growth from a
footprint settling at the floor, not to catch bounded residual garbage.
The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit
would abort the DeleteVolume call queued behind it and turn a reported leak
into a real one.
* test(e2e): add memory-constrained concurrent-write durability regression
The existing durability tests write one 4MiB sentinel, sequentially, on a
comfortably sized guest, which is the exact inverse of the known-open
data-loss bug: acked writes read back zero-filled after a cold reread when a
guest is memory-starved during concurrent writeback. No happy-path test can
reach it, because the fault lives in the intersection of memory pressure,
write concurrency and cache-cold reread.
Add a suite that encodes those three conditions together: a guest discovered
at or below the working set, N parallel large writers launched as background
jobs in a single command so they genuinely overlap, swap disabled, then sync
and drop_caches before rereading. Assertions are per stream rather than
aggregate so a failure names the losing writer instead of leaving whoever
reads it to re-derive which of N writers lost data from one combined digest.
GuestExec gains a timeout-parameterised variant. Its fixed two-minute default
is sized for sub-second payloads, and a multi-gigabyte write to a
network-backed volume would exceed it, so the suite would have failed by
timeout while looking like it had caught the corruption. The budget scales
with the working set, and a deadline is reported distinctly from a data
mismatch so the two red states are never confused.
The suite is dispatchable but deliberately absent from the default SUITES:
the bug it reproduces is still open, so it is expected to fail and must not
sit in a merge-gating path. It is registered in the workflow allow-list so it
stays discoverable rather than being buried behind a skip.
Also record on the live IAM instance-profile test why its assertions were not
moved down to the integration tier, so the trace is not repeated.
* test(e2e): drop backend byte accounting, assert a node-wide delta instead
The accounting helper listed the internal predastore bucket to attribute
bytes per volume. That listing is refused unconditionally: predastore
authorises bucket access by same-account ownership, and the bucket belongs
to the service account, so a test caller gets 403 no matter what it holds.
Granting the test a service credential would invert the trust boundary to
buy a measurement, which is the wrong trade.
Replace it with what the S3 surface actually permits: sample node-wide
allocated bytes either side of the write and bound the delta against the
bytes the guest wrote. Coarser, but it measures something real rather than
depending on a call that cannot succeed.
The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x
expansion floor plus headroom; a cluster running the recommended RS(3,2)
has a 1.667x floor and only ~12% of margin left, so the constant needs
revisiting if the encoding changes.
* revert: drop memdurability e2e suite
The suite asserted the guest read back what it wrote after a
memory-constrained concurrent write, but the assertion could not fail for
the reason it claimed: it compared the guest's own read against the
guest's own write with the page cache still warm, so it passed whether or
not the data reached the backend. A test that is green on a build with a
known-open durability bug is worse than no test — it advertises coverage
that does not exist.
Removes the suite and its Makefile/workflow wiring, plus
DiscoverInstanceTypeAtLeastMemory, which had no other caller.
GuestExecTimeout stays: GuestExec now delegates to it, and the old
two-minute hard cap it replaced was a latent trap for any long guest
command.
* test(e2e): add partial-block concurrent-write durability suite
Replaces the removed memdurability suite, which passed on a build that
still carried the bug it claimed to test.
The engine reaches its read-modify-write path only for a partial block;
a fully covered block takes a full-overwrite branch that reads nothing
and can lose nothing. The old workload wrote through ext4 with 1 MiB
buffers, which the filesystem coalesced into aligned full-block
requests: block-layer tracing recorded zero unaligned and zero sub-4096
requests, so the path under test never ran, and writers 4 MiB apart
could not collide on a block even in principle.
This suite writes the shape the defect needs. Two writers issue
concurrent O_DIRECT writes into disjoint halves of the same 4096-byte
blocks on a raw, unformatted device, stamping every half with the round
number, then drop the page cache and reread with O_DIRECT asserting
every half carries the final round. One half reading back at an earlier
round is the signature of the lost update.
The old suite also could not fail even had it reached the path: its
write phase ended in 'wait; sync; sha256sum' and reported sha256sum's
exit status, while bare wait returns 0 unconditionally, so writer errors
were invisible and a backend ENOSPC presented as a pass. Here each
writer's exit status and exact byte count are asserted before any
durability claim is made.
Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on
teardown; the predecessor left roughly 3 GiB of backend per run and
twice drove a node under the nearfull watermark.
Dispatchable but not in the default suite list until a green run and a
demonstrated fail-first against a pre-fix engine exist.
* test(e2e): let the fixture own the working-set volume delete
The suite deleted the volume in its own cleanup and the fixture deleted
it again on close, so every run ended with an InvalidVolume.NotFound
from the second attempt. Detach stays — DeleteVolume against a still
attached volume is what leaks volumes on teardown — and the delete is
left to the fixture so the volume is released exactly once.
* test(e2e): gate merges on the partial-block durability suite
Verified on one node with the same test binary and workload, varying
only the engine: 1758 of 8192 halves reverted a generation against the
pre-fix build, 0 of 8192 against the fixed one. That is the fail-first
evidence the suite needed before it could be trusted to gate anything.
Adding it to the default suite list rather than leaving it dispatchable
is the point. Its predecessor sat outside every path that runs, which
is how it survived long enough to be discovered green on a build that
still lost data.
* test(integration): cover the ECR data plane in the front gate
The live ecr/ suite drove docker, crane and skopeo against a real
registry, yet none of it boots a guest: it needs a registry, an object
store and an HTTP client. The in-process gateway already serves the OCI
Distribution v2 surface on an ephemeral port with a memory-backed object
store, so these run here in 0.18s instead of behind full provisioning.
Covers the three claims the live tests made beyond authorization, which
is already down here: a push/pull round trip asserting blobs and
manifests read back byte-identical, refusal to auto-create a repository
on push, and retagging an existing manifest without disturbing the
original tag.
Content is compared rather than status codes alone. A registry that
stores a blob under the wrong digest, truncates it, or serves a
different one answers 200 to every request in the sequence.
The capacity/load leg deliberately stays live: it exists to put real
parallel pressure on a real predastore, and a memory-backed object store
cannot make that claim.
* fix(e2e): make volume teardown release the volume before deleting it
Fixture cleanups drain LIFO, so a suite that creates its instance before its
volume tears the volume down first, while it is still attached. DeleteVolume
answered VolumeInUse, nothing retried, and the volume was stranded on the node.
Reordering the chain cannot fix this — registration order has no way to express
the instance/volume dependency. Instead teardown is now self-sufficient for any
creation order: it force-detaches, waits for the volume to reach available, and
only then deletes. Instance teardown waits for terminated for the same reason,
since TerminateInstances returns as soon as the request is accepted.
Two things kept the leak invisible. A failing cleanup was logged and forgotten,
so a run stayed green with resources left behind; failures are now reported as
leaks, failing the bound test or surfacing through Close() for TestMain to fail
the run. And a panic in one cleanup aborted the whole chain, stranding every
resource behind it; each callback is now isolated.
RegisterVolumeTeardown and the fixture had grown separate teardown paths, only
one of which detached. Both now share it.
Re-creating a nodegroup of the same name could never reach ACTIVE. The
Ready-gate took its baseline from the live Ready count for that name, so
a terraform replace after CREATE_FAILED — which deletes then re-creates
— captured the previous incarnation's workers, demanded old+new, then
watched the count fall as the old workers were terminated. The target
was unreachable by construction.
Observed on env19 with a perfectly healthy cluster: three new workers
all joined and the create still failed with
reports 3 Ready nodes, want >= 5 (baseline 2 + 3 workers)
and each retry raised the bar again (baseline 3 => want 6), so the only
way out was untainting the resource by hand.
A create means every worker under that name is new, so the gate is just
"desired Ready nodes carrying this nodegroup label" — baseline 0. The
trade-off is stated in the comment: a stale incarnation's Ready nodes
can briefly inflate the count and satisfy the gate early, but they go
NotReady within about a minute of termination, whereas baselining on
them fails every replacement permanently.
The test models the real sequence — delete, stale Ready count still
reported, re-create — and fails without the fix with the same shape as
the field error.
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
tomnewton-mulga