8 Commits

Author	SHA1	Message	Date
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
joshsiv-mulga	6e13a9d9f4	test: add continuous single-open churn e2e for viperblock 503/GC ... TestNPassOverwrite detaches after every pass, so viperblock's drain and GC only ever run at reopen — it cannot show whether GC reclaims superseded chunks while a volume stays open, and its serial write rate never nears the predastore rate limiter's burst. TestContinuousChurn attaches once and drives concurrent 4 MiB-aligned raw overwrites of per-writer regions for several minutes without detaching, so a sustained superseded-chunk backlog builds under one long open, spanning multiple GC ticks. The workload is bounded and self-contained: each dd writes one chunk and the clock is re-checked after every write, the loop runs as root under a single sudo bash with a process-group kill trap, and it is wrapped in a guest-side timeout. Root ownership is load-bearing — the writers open a raw device, so only a root-owned process group can reap them; a best-effort guest pkill runs before detach as a backstop so a stray writer cannot wedge the detach.	2026-07-23 14:49:01 +10:00
Josh Siviour	cae76ba1b6	test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566) ... * test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it.	2026-07-22 17:02:20 +10:00
Josh Siviour	a01e58d90c	feat: deliver backend ENOSPC to the guest and enable chunk GC in production (#549) ... * feat: plumb GCEnabled config through to every VB construction site viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never set it, so every deployed node ran with GC off by construction regardless of which viperblock binary shipped. Add a [viperblock] gc_enabled config key (default false, mirrors shardwal) and thread it through to the three places spinifex builds a VB: the nbdkit plugin it spawns per mounted volume (via a new gc_enabled plugin param), viperblockd's detached-volume opens, and the volume service's CreateVolume. * chore: repin viperblock to GC-enable lineage The gc-enable plumbing references viperblock.VB.GCEnabled, which is absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable commit so a standalone (GOWORK=off) build — the path the ansible deploy uses — resolves the field, and spx plus the nbdkit plugin build from the same GC-bearing viperblock lineage. * fix(storagegrowth): stop launching a nano for image-pull repro ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the synthetic npass suite. A 512MB guest OOM-thrashes containerd before any disk-driven ENOSPC and conflates the storage leak with guest-memory pressure. Add a package-local imagePullInstanceType helper: an exact override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default 16384 MiB). npass keeps using the nano selector unchanged. * feat: report guest block write errors so backend ENOSPC reaches the guest Set werror=report and rerror=report on the guest boot drive so a viperblock ENOSPC (or any other backend write failure) is surfaced to the guest as a real I/O error instead of QEMU silently pausing the VM. Without this the guest kernel never learns its disk is out of space, so it can't fail writes cleanly or let an operator react. * chore: repin predastore and viperblock to the storage-capacity-safety lineage Drop the local-path replace for predastore (a test shortcut) and pin both predastore and viperblock to their feat/storage-capacity-safety commits, so a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC classification, and backpressure plumbing that spinifex's werror/rerror change depends on. * chore: bump viperblock to 054a44a3f0f5 Realigns spx's linked viperblock with the submodule working tree, which the build role's source-drift gate compares against. * chore: repin predastore and viperblock to their dev lineage Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work has now landed on dev in each repo, and both PRs were squash-merged, so the branch commits those pins named are unreachable from any branch. Repin to the dev commits carrying the same code: predastore 5699329 and viperblock 35dd9f2.	2026-07-21 15:39:14 +10:00
joshsiv-mulga	72353e476c	test(e2e): read the live checkpoint on-node and target the root volume ... The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin with the operator box's gateway identity, which has no S3 read grant on the bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a whole run. It now HEADs the object on the node over PeerSSH using predastore's own [[auth]] credentials from predastore.toml (root-owned, so read via passwordless sudo; the CA is readable through the spinifex group). The credentials are read into shell locals, used inline for one aws s3api head-object, then unset — never echoed, never leaving the node. A probe failure that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe error) rather than reporting a false zero, so a dead instrument can never look like no growth. The dead operator-side S3 client path is removed.	2026-07-17 12:35:43 +10:00
joshsiv-mulga	4cbd2a5c88	test(e2e): add container image-pull storage-growth workload ... Pulls a container image inside a guest onto a viperblock volume and samples guest used bytes, host backing-store bytes, nbdkit RSS and the live checkpoint size as it goes, so backing-store cost can be fitted against the bytes the guest actually wrote. Complements the N-pass overwrite workload, which rewrites one extent and so pins the block map size — that shape cannot show whether cost grows in proportion to bytes written or faster than them. Redirects both dockerd's data-root and containerd's root onto the target volume. Redirecting data-root alone leaves containerd extracting layers into /var/lib/containerd on the AMI's small root disk, which fails with ENOSPC while the target volume sits empty.	2026-07-16 08:10:24 +10:00
joshsiv-mulga	d46d257278	fix(e2e): don't require CA cert when SPINIFEX_AWS_INSECURE=1 ... The storagegrowth gate called ResolveCACert unconditionally before building the AWS client, even though newAWSClient itself skips CA resolution entirely when SPINIFEX_AWS_INSECURE=1. That blocked the documented runner-resident remote path (env12 and similar clusters with no local CA cert on disk) and the skip message falsely claimed no node was running when the real cause was a missing CA. Skip the CA check under SPINIFEX_AWS_INSECURE=1 and state the actual cause plus the escape hatch in the skip message.	2026-07-15 15:17:13 +10:00
joshsiv-mulga	6e043f6453	test(e2e): add storage-growth N-pass overwrite suite ... Rewrites one logical extent N times through full attach/format-write/detach cycles, so the guest-visible footprint stays constant while total bytes written scales with N. That shape isolates the leak where viperblock mints chunk objects under unique monotonic keys and never deletes the superseded ones: a sequential write-once supersedes nothing and so cannot exercise it. The suite drives the workload only and deliberately does not measure the backend. Measuring on-disk bytes from here would mean either a host du (which cannot separate live from dead bytes, and so cannot distinguish this leak from the unrelated one where an overwritten key's old extent is orphaned without a tombstone) or a dependency on tooling outside this repository, which would stop the suite building standalone. It instead reports what it did to a caller-named result file and asserts only what a guest can observe: a sha256 round-trip through a fresh mount per pass, and a footprint that stays constant across them. The volume is retained on request, because deleting it removes the evidence: DeleteVolume enumerates the bucket by prefix and drops every key under it, including the leaked chunks.	2026-07-15 15:03:21 +10:00