306 Commits

Author	SHA1	Message	Date
Ben Duncan	828b5df4aa	fix: pause a guest on a backend I/O error instead of handing it EIO (#977) ... * fix: fail DRAIN when a volume seal failed shutdownAndUnmount logged the volume unmount error and returned nothing, so StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set. The coordinator then advanced to STORAGE and stopped viperblock while the seal was still in flight, tearing the VBState envelope and leaving the guest unable to mount its root volume on the next boot. The deploy reported success. shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup and stopOne carry it without aborting the rest of the teardown, since QEMU is already down by then, and StopAll aggregates the per-VM failures so handleShutdownDrain ACKs with Error set and never reports vms_stopped. volumeMounterAdapter.Unmount was the second swallow site: it logged each failed seal and returned nil unconditionally. It now attempts every volume and returns the aggregate, which is what the VolumeMounter contract already described. Terminate still tolerates a failed seal — the volumes are deleted next, so an unsealed block map loses nothing a caller can ask for. This does not fix the seal itself. It converts silent data loss into a refused upgrade that an operator can act on. * test: add the storagefault e2e suite for backend-outage guest survival Freezes predastore with SIGSTOP to hold connections open while answering nothing, which is the fault a timeout-free path cannot distinguish from a slow peer. On one node that is a total outage; on three it freezes every node but the guest's own, breaking the RS(2,1) shard floor. The suite writes and flushes a verifiable region before the fault and churns a separate unverified region during it, so an interrupted write is never mistaken for lost data. The root-filesystem load runs from the user's home rather than /tmp, which is tmpfs on this image and would move no bytes on the root device. * test: verify the root device is under write pressure, and size the outage The previous guard matched /^ *vda / against /proc/diskstats, where the device name is the third field, and read $7 which is ms_reading rather than sectors_written. It printed empty and asserted nothing, so a green run could not distinguish a surviving guest from an idle disk. It now samples sectors_written twice and fails unless it advances. The outage is sized from viperblock's absorption rather than guessed: a workload must fill DefaultMaxPendingBytes before backpressure engages at all, so a short freeze measures the retry budget rather than the fault. * test: add nbdkit fault injection to the storage-fault e2e suite The predastore freeze takes the whole storage cluster away from every guest on it, which is too broad to run beside anything else. Removing a single volume's NBD server is the same fault at one-volume scope: one host, one guest, nothing cluster-wide stopped. Both signals are here because they sit on opposite sides of the defect. SIGSTOP holds the socket open and answers nothing, so the guest blocks and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at the default reconnect-delay of 0 fails requests at once, which under werror=report aborts the journal. Signalling is guarded: a pid is matched on comm, our plugin and the volume argument together, and re-confirmed immediately before the signal, because these hosts run nbdkit for other volumes and pids are reused. * fix: pause a guest on a backend I/O error instead of handing it EIO Every virtio-blk device was created with werror=report,rerror=report, so a backend outage was delivered to the guest as EIO. ext4 aborts its journal on that and remounts read-only, and the write that failed is simply gone. Under werror=stop QEMU pauses the VM and holds the failed request, so a backend that comes back resumes the guest with nothing lost. The boot drive, the cold-boot data volume and the hot-plugged data volume all move to stop. Alongside it the NBD transport gets reconnect-delay=30, which pauses requests rather than failing them while a storage process restarts on the same socket; on the legacy -drive form it has to be file.reconnect-delay, because the bare name reaches the raw format driver, which has no such option. The QMP heartbeat now recognises run state io-error and issues cont. After ten attempts the instance is reported impaired and the cadence drops to one attempt per ten heartbeats, but it never stops: QEMU holds the request for as long as the guest exists, so a backend that returns an hour later can still be resumed. The storagefault suite gains the fault this was written for — kill the volume's own nbdkit, hold, then put it back on the same socket — and it now verifies a sha256 region across the outage rather than trusting the guest to be reachable. Three harness defects went with it: the socket-liveness poll ran unprivileged against a 0770 directory and could never succeed, the relaunched socket was left 0755 where QEMU needs group write, and the two variants shared one guest so a fault in the first was diagnosed in the second. * test: cover the io-error resume policy * test: stop two parallel tests racing on the terminate backoff variable * fix: report a guest paused on a backend I/O error as impaired A guest held by werror=stop kept reporting ok on both status checks. QMP stays responsive while the guest is paused -- cont is being sent and answered -- so the QMP failure gate that DescribeInstanceStatus keys on never trips, and the one state a customer most needs to see looked healthy. Instance status now reports impaired/failed while the resume budget is non-zero, timestamped from IOErrorSince so impairedSince is when the guest paused rather than when it was escalated. System status reports impaired too: a backend I/O error is an infrastructure fault, not a guest one, and that distinction is what the two checks exist to draw. The counter is cleared by the first poll that sees the guest running, so both clear on recovery. * test: reap the relaunched nbdkit when the fault test ends relaunchNbdkit detaches the server with setsid so it outlives the SSH session that started it, and viperblockd never learned it exists. Nothing reaped it, so every run left an nbdkit holding a socket and a cache for a volume that was later detached or deleted -- four were found running on dev-prod against two detached and two deleted volumes. The PID is resolved from the socket path rather than the volume id, so a legitimate viperblockd server for the same volume is never the one killed. * test: prove the backend is serving before a fault is called cleared Thawing the process a fault stopped is not the same as restoring the service. The outage test treated SIGCONT as the end of the fault and went straight to judging the guest, so it passed while the cluster it ran on stayed unable to serve a single object for the next 35 minutes, and the next test would have inherited that. thawPredastore now puts and gets an object through every gate, with a budget, and fails the test if the backend does not come back. * fix: do not hand back a mount entry whose export is gone A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit was gone. Every later ebs.mount then matched that entry and returned its socket path, so vmMgr waited for a socket nothing would ever create and the instance could not be started again — on dev-prod a guest stopped against a frozen backend was left permanently stopped, four start attempts failing identically on a socket the node had already removed. Drop the entry when there is positive evidence the export is dead: the socket is absent, or the recorded pid is not running. Neither can be true of a live export, so the double-writer hazard the guard exists to prevent is unaffected. Also make the durability test's pattern helpers report failures. The read dd sat in a pipeline, so its exit status was discarded and the harness hashed nothing and returned the empty-string sha256 as a checksum. * test: report the guest console when a restart never answers SSH A restart that reaches running but never comes back is indistinguishable from a dead root filesystem without the serial console, and the log is gone once the fixture terminates the instance. * fix: converge guest port datapaths concurrently, not one deadline after another Each guest port that misses its SB binding burns the full convergence deadline, and applyEIPs spent them in series. Three ports belonging to long-stopped instances therefore held a newly launched guest's EIP dark for the sum of them before its own probe started. AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and sb-cluster-state-reset is now serialised with a minimum interval so concurrent probes cannot each fire a cluster-wide resync. * fix: refuse a stale cross-node start after the owning node is gone The volume lease already refused a cross-node mount, but it is a liveness claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded writes stops renewing, and 45 seconds later nothing refuses a start elsewhere that loads a stale backend checkpoint. A failed seal now writes a durable marker naming the node holding the only current copy, cleared on a seal that reaches the backend and on delete. The lease covers a live holder, the marker a dead one. The refusal also reaches the caller instead of being flattened to ServerInternal, which told an operator to retry something that cannot succeed while the holder is down. * fix: keep the daemon's message when a start is refused Rebuilding the error from the code alone rendered the generic ErrorLookup text, so a refusal naming the node holding the volume reached the caller as boilerplate about attaching volumes. * fix(ec2): terminate detaches every DeleteOnTermination=false volume TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume on the assumption that Stop's Unmount had already cleared its attachment. Unmount only clears it when the seal succeeded, so a volume whose seal failed stayed in-use pointing at an instance that terminate then deleted, leaving it unreachable with no owner to release it. The running-instance path already detaches every DoT=false volume; the stopped path now matches it, and tolerates an already-gone volume the same way. DeleteVolume also keeps its provider error on the way out. The terminate reaper retries the delete every two minutes and logged a bare ServerInternal, which made a blob node being down indistinguishable from a transient fault. * fix(ec2): DetachVolume --force releases a volume whose instance is gone The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a volume left attached by a part-way teardown could never be detached: the instance no longer runs anywhere and nothing answers. That also made it undeletable, since DeleteVolume refuses an attached volume. ForceDetachVolume already existed for this deadlock but was reachable only from account teardown. Force now falls back to it when the instance has no responder, which is the one case where clearing the attachment cannot strand a live guest. * fix(viperblock): unsealed writes steer placement instead of pinning a volume The dirty marker refused any mount from a node other than the one holding unsealed writes. Instance start already forwards to the node that last ran the instance and falls back after a 30s window, so that refusal turned a working fallback into a volume that could run nowhere once its owner was gone. It traded a rare data-loss risk for a permanent availability failure. The marker is now a record and a warning, not a gate. A takeover logs the previous holder and why it could not seal, then adopts the marker so the old node is no longer treated as the better source. Live exclusion is unchanged and still enforced by the lease. It is also written on open rather than on seal failure. A node killed mid-write never reaches its seal, so the old trigger left no trace in exactly the case the marker exists to describe. Adds 'spx admin volumes unsealed' to see which volumes would cost something if started elsewhere. * fix(viperblock): fence a volume whose lease moved instead of logging it The lease was checked once, at mount, and never again. A node that lost it kept its export up and kept writing into a volume another node owned, so both copies advanced and the backend settled it by arrival order. Losing a renewal now decides: another holder means tear the export down, no holder means the entry merely aged out, so reclaim it and leave the guest running. Fencing never seals — this node's copy is the stale one — and leaves the dirty marker for the winner to take over. This is not an epoch fence. It narrows the window from the 45s TTL to one renewal interval at no write-path cost; the backend still cannot refuse a stale writer. * fix(network): advertise EIPs so a recycled address is reachable The gateway switch port only asked ovn-northd to publish the router's NAT addresses in centralised and routed modes, so on a distributed cluster nothing ever announced an elastic IP. OVN still answers ARP for it, which is why a first-time allocation works and hid this — but no gratuitous ARP is sent when the address moves to a new ENI, so every host on the segment keeps resolving it to the previous holder's MAC until its cache ages out. Reusing an address from the pool then lands on a dead MAC and the instance is unreachable from outside. Distributed mode needs the option as much as the others: northd scopes each entry with is_chassis_resident(<vif port>), so the announcement leaves from the chassis running the instance rather than the gateway, which is what distributed NAT wants. Converge on the already-attached path too. AttachIGW returns early once the gateway port exists and nothing else rewrites it, so without that an existing VPC would stay silent for its whole life. Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to carrying their correct external_mac, each scoped to its own ENI's port. * fix(viperblock): make the volume fence and its dirty marker ownership-safe Four corrections to the lease-loss fence, all found in review. Drop the reacquire path. Losing a lease and finding no successor was treated as a lapsed entry worth reclaiming, but leases.acquire returns the cached entry — already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease renewed nothing and the node went on writing a volume it could not prove it owned. The two errors are not symmetric: a false fence costs one guest a restart, a false reclaim leaves two nodes writing one volume. Fence in both cases, and report which happened through the outcome attribute. Surrender a lease this node cannot confirm. Renewal only noticed a loss when the server rejected an update, so a node partitioned from NATS kept its export up while the TTL expired and another node took the volume. Track the last confirmed renewal and give the volume up once that ages past the point where the server may admit another owner. Bound the renewal call too, so a hung JetStream request cannot hold the goroutine past that point. Do not act on a kill that did not take. Both the fence and ebs.unmount removed the mount entry, released the lease and announced the result before knowing nbdkit had exited — the fence then reported a writer stopped that was still running, and the unmount sealed into a directory a live process was writing. Wait for the exit, and on failure leave the volume mounted and say so. An already-exited process now counts as success, which is what the caller asked. Give the dirty marker a generation. It recorded an owner and nothing to order two nodes' claims, so a node returning after a takeover could clear or overwrite the winner's marker and point placement back at the stale copy. Stamp the lease generation, refuse a write from an older one, and only clear a marker this node still owns. A marker that cannot be written now fails the open rather than leaving writes nothing records. * test(viperblock): cover the stall surrender and the unmount kill guard The two ownership cases the fence work left untested. A holder partitioned from NATS while the winner keeps it never gets an update rejected, so revision checking cannot see it — it has to give the volume up on its own clock. Pins that validity sits below the server TTL with room for a check in between, that a lapsed holder reports the loss as a stall rather than as a takeover, and that releasing a surrendered lease leaves the entry for whoever holds it now. The unmount side of the kill guard: sealing under a live nbdkit writes a state torn between two writers, so a kill that did not take has to fail the unmount and leave the entry mounted for a retry. * fix(e2e): keep the guest private key for as long as the key pair EnsureKeyPair computed the PEM path from the caller's directory but wrote the file inside the ensureOnce closure, which runs only on the first call. Callers passed a per-test artifact dir, and ArtifactDir removes that dir when the test passes — so the first test's success deleted the only copy of the private key while the key pair stayed registered, and every later call returned a path nothing had ever been written to. Guests after the first were then launched against a public key whose private half no longer existed, so the harness could never authenticate and gave up after the full five-minute budget. The guests were healthy and reachable on tcp/22 throughout, which is why this read as a storage or network fault for so long. The directory parameter was the defect, so it is gone: the PEM now lives in a directory the fixture owns and removes with the key pair, and no caller can supply one that is shorter-lived. Also reorder the killed-nbdkit teardown. The reaper registered its own cleanup after the detach registered one, so LIFO killed the block server before the platform detached the disk and DetachVolume then failed with "Node ... is in use". The detach cleanup now reaps once it is done. * fix: stop leaking an nbdkit export when recovery cannot adopt it Restart recovery adopts a guest's boot volume, then routinely fails the -efi volume with "claim lease: context deadline exceeded" -- JetStream is still catching up at daemon start and the KV claim runs on the client's 5s default. Recovery gave up on the first error and never retried, so the export stayed live with no registry entry. At terminate the unmount found nothing to seal, returned NotFound and killed nothing, and the process outlived its guest holding the volume and socket open. EFI is always the second volume adopted, which is why it was always the one that leaked. Two changes, because either alone leaves a hole: - rebuildMountedVolume retries the build (4 attempts, 3s apart) so a JetStream that is merely slow no longer orphans an export for the life of the process. - unmountVolume reaps a corroborated export for the volume it was asked to unmount when the registry has no entry for it, so an export orphaned by any means is still cleaned up. The reap goes through reapOrphanedNbdkit, which declines anything a live process references: at unmount the guest's blockdev may not be torn down yet, and killing the server underneath it strands QEMU. * test: hold the backend-serving probe and assert predastore readiness The backend-outage test declared the backend restored on one successful round trip through each gate. A gate can answer the request that arrives just after a thaw and then lose its blob peers under the load that follows, which is a backend that never recovered reported as one that did -- and that is what happened: the probe reported all three gates serving minutes before the guest was still taking BLOCK_IO_ERROR on every resume. The round trip has to hold across three rounds now, and the count restarts rather than decays when one fails, so a gate that serves and then stops is the failure the hold is looking for rather than noise it averages out. It also checks /readyz on each host and reports any failed check by name. The round trip only proves this workstation can write through a gate; it says nothing about which blob peers that gate can reach, and a gate one node short of the write floor serves a small object and fails a guest. A host that serves no admin listener is logged and skipped, since the port is optional configuration. * fix: scope predastore's admin port to the cluster plane in the host policy predastore's admin listener binds the host's cluster bind_addr and serves /healthz and /readyz unauthenticated. Readiness now names which blob peers a gate cannot reach, which is exactly the kind of topology detail 4432 is peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the public one. * fix: keep resuming a paused guest at heartbeat rate after a long outage Past ten attempts the cont dropped to one per ten heartbeats, so a guest whose backend had come back could sit paused for another five minutes before anything retried it. Measured on dev-prod: the backend was proved serving through all three gates while the guest was still waiting on attempt 60, five minutes after attempt 50. The slowdown was there to avoid paying a failed I/O every thirty seconds on a backend still down. That is nearly free, and five minutes of downtime after recovery is not, so only the log line is throttled now. * test: fail the run when a key pair name resolves to more than one pair A create that lands twice leaves two key pairs with one name and different material. RunInstances resolves the name to one of them while the harness holds the private key of the other, and the only symptom is every guest in the run refusing the key at the SSH gate — five minutes of waiting per guest, four guests, and nothing anywhere naming the cause. Observed on dev-prod: two pairs created a millisecond apart, both named e2e-key-30de10ba. One DescribeKeyPairs after the create says so outright instead. * test: tolerate an already-released volume in the detach teardown Teardown runs DetachVolumeWait after a terminate that releases the volume itself, so the detach can arrive at a volume already on its way to available. That is the outcome it wanted, and treating the IncorrectState refusal as fatal failed a backend-outage run whose assertions had all passed — the guest came back, every acknowledged write verified, and the test still went red on the way out. VolumeInUse is still fatal: that one means the volume is attached somewhere it should not be. * test: carry the merged suites onto the reworked key-pair and account-scoped seams The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir this branch removed, and DeleteVolume's metadata lookup is account-scoped now, so the provider-failure test has to seed and delete under the same account. * refactor: make the fence outcome a type in one const block Four separate exported string constants with a paragraph of comment each, passed to RecordVolumeFence as a bare string. Group them, give them a FenceOutcome type, and take that type at both call sites so an arbitrary value can no longer reach the metric attribute. * refactor: move viperblockd's wire contract into a leaf vbwire package The daemon imported viperblockd for exactly two symbols -- the fenced volume subject and its event -- which put the storage service back on the daemon's dependency graph to name something it only listens for. vbwire holds the contract instead: the subject and event, the dirty bucket name and record shape, and the read-only listing an operator tool needs. It imports nothing from spinifex, so both the daemon and the admin CLI drop their edge to viperblockd entirely. cmd/service.go keeps its one, which is genuine: it constructs the service. * docs: drop a specific host name from the start-refusal comment Named a real dev-prod host in a comment and in test fixtures. Say what the name is instead. * ci: give storagefault a dedicated nightly cell The suite is eligible for both topologies and runs on the PR gate, but no nightly cell ran it: the permutation cells run a deliberately narrow subset and none of the dedicated suite cells named it. Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that environment would report the outage as its own failure. * ci: run storagefault nightly on three nodes, not one The previous commit put its dedicated cell on a single-node runner. That was the wrong topology and the wrong justification: single-node is RS(1,0), where the shard floor cannot be broken and a cross-node takeover after a failed seal does not exist, so the two assertions worth having a dedicated cell for were the two it could not make. Cell 29 is now real-multi on ci-multi. It also corrects the claim that the PR gate already covers the multi-node case. The suite is eligible for both topologies, but e2e.yml's default suite list never names it, so it has never run there. A full pass is ~63 minutes on dev-prod bare metal, almost all of it deliberate waiting -- one test holds predastore SIGSTOPped for seven minutes -- which is too long to add to a gate that currently finishes well inside its budget. Nightly is where it belongs. Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since 90 minutes of suite cannot run in a 50-minute job; the permutation steps gated on matrix.suite == '' so the dedicated cells do not also run them; and a Run Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts the suite against E2E_SUITES_MULTI before running it. * fix: stop a startup KV race leaving vpcd serving keyless guests vpcd opens five KV buckets while NATS is still coming up, each attempt carrying a ~5s JetStream API deadline, and applied three different policies to the same failure: spinifex-vpc-enis was fatal, while instance-state, security-groups, subnets and vpcs each degraded silently for the lifetime of the process. A five-second blip during a cold start therefore decided how a node served metadata until someone restarted it. On dev-prod all three nodes hit this after a deploy. Two were saved by accident: their ENI bucket also timed out, so they exited 1 and systemd restarted them cleanly. The third's ENI bucket opened in time, so it never crashed and ran for hours with no instance record loader. With the local state file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's crawl died on ami-id and fell back to DataSourceNone, and every guest launched on that node booted with no authorized_keys. kvstore.OpenWithRetry retries for a bounded window before any policy applies, logs the cause per attempt and the wait on a late success, wraps the last error with the bucket name, and returns at once on a cancelled context. All five opens use it. Failing to open the instance record space is now fatal, matching the ENI bucket: the dependency whose loss stops guests logging in should not be treated more leniently than the one whose loss stops ENI resolution. Also fixes the reason the fallback was load bearing at all. The local state file was written 0600 and vpcd reads it as a different user in the shared spinifex group, so the local-first path had never once worked and its mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so an inherited umask cannot undo it. Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an unresolvable instance as "no key pair" with nothing logged. * fix: scope the ime-* firewall accept to IMDS and VPC DNS only The per-tap OVS demux delivers any guest packet addressed to 169.254.169.254 or .253 onto an ime-* host port regardless of L4 port, and iifname "ime-*" accept then admitted all of it. Every wildcard-bound host service — sshd, the AWS gateway, northstar, the console — answered guest packets on those addresses; only asymmetric reply routing kept the handshake from completing, and a guest could still drive northstar to recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS (.253:53), letting everything else fall through to policy drop. * docs: document the instance-to-host plane and how to protect it without the ISO The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by host services and is invisible to a NIC-level or cloud firewall, since the traffic is intra-host on the OVS bridge. Explain that the ISO ships the scoping rule armed, that the binary installer ships the firewall off, and that a non-ISO install must enable it with setup.sh --firewall=on or firewall_enabled=true. Include verification from an instance and the host. * fix: withhold the IMDS meta-data listing when the instance does not resolve Every omission from the meta-data root is a positive claim about the instance — public-keys/ absent means "launched without a key pair", and cloud-init believes it, writes no authorized_keys and boots the guest keyless. So resolving the instance was best-effort for a listing whose whole meaning depends on it: a backend error or a miss shortened the listing instead of failing it, and nothing anywhere recorded that the answer was a guess. Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot retry a 200 that is merely wrong, so this converts a permanent silent failure into a transient loud one. servePublicKeys already had this policy for the leaf; the root is what contradicted it. A profile lookup that fails still serves the listing minus iam/, and now logs. The key pair is the field a guest cannot recover from losing, while a missing role surfaces through the credentials leaf, which reports the miss itself. Also stops the storagefault cross-node test failing on its own access path. It tolerates the pattern changing across a takeover, but readPattern needs sudo, and a volume opened from the backend checkpoint may predate cloud-init writing its sudoers drop-in — indistinguishable from an unreadable device at the dd call site. It now checks sudo first, and reads cloud-init's recorded datasource to tell a rolled-back root filesystem from IMDS never having answered, failing loudly on the latter rather than accepting it as storage loss. --------- Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>	2026-09-03 19:45:33 -05:00
Julian Sommer	7b4a83ccfe	fix: expect InvalidSnapshot.NotFound for cross-account DeleteSnapshot ... Snapshot metadata is now keyed by owning account, so a snapshot outside the caller's prefix is absent rather than forbidden. The e2e assertion still encoded the pre-partitioning UnauthorizedOperation contract.	2026-09-03 15:51:27 +10:00
Julian Sommer	d8d3e2268d	feat: partition EBS metadata keys by owning account (#989)	2026-09-03 15:38:23 +10:00
Bryn Mailer	18aa67b0e9	feat: add the e2e guest disk-performance gate (#964) ... * feat: add the e2e guest disk-performance gate Nothing measured guest disk performance in CI, which is why a quadratic index scan shipped in viperblock 35dd9f2 and survived three subsequent fixes to adjacent code. It is also why the two availability clauses at the end of the ebs-performance plan's Stage 0 have never been evaluated: a sustained random-write workload took the guest, and then the host, down, and nothing since has demonstrated that it no longer does. tests/e2e/diskperf/ launches an instance, attaches a dedicated blank volume per repetition, and drives a pinned fio profile over guest SSH. Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit criterion names, and randrw-70-30-4k, the mixed workload the headline IOPS came from. The target is a raw device with direct=1 rather than a file on the boot disk, so guest page cache, filesystem allocation and journal traffic are out of the path. Two classes of assertion, kept apart because folding them together lets a run that left the host unreachable pass on a respectable IOPS figure. Availability is hard pass/fail and absolute, so it gates from the first run: the job completes inside its budget, a guest responsiveness probe answers throughout on a second SSH channel, and the host's own sshd answers throughout. The guest probe reads the root volume while the workload runs on a different one, so a pass is evidence of isolation between volumes rather than of the job merely being slow. The host probe is a TCP connect plus an SSH banner read, which needs no credential and still proves the kernel scheduled sshd. Throughput and latency are banded against a committed baseline: fail above 25% regression, warn above 10%, per-metric median across repetitions. baseline.json is committed and embedded, with every figure zeroed, and a zero reference records and warns rather than failing. Only the availability class gates until numbers are captured on the bare-metal cell; a throughput band nobody measured would be a fabricated gate. A fresh volume per repetition is the reset. Extent count grows monotonically, so reusing a volume makes the gate order-dependent and drifts its own numbers upward over time, slowly normalising the regression it exists to catch. Each job is its own guest invocation and fio's JSON is fetched by a separate command, so a wedged job loses only itself and reports as a budget breach rather than a parse failure. e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable workflow serves both the correctness cell and nightly cell 28, which queues behind cell 20 on the shared concurrency group. Cells 25-27 are the dedicated eks, ecs and rds cells. The per-suite Go test timeout is resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares its 180-minute budget beside rds's rather than in a second place. The gate's own arithmetic runs in preflight via test-harness: band edges, the unmeasured cases, per-stream metric selection, fio JSON parsing, and the probe accounting including a slow-but-successful answer counting as a breach. A gate whose comparison logic is untested is the failure mode the plan is otherwise full of. The extent-count ceiling named as a fourth availability assertion is not implemented: it needs viperblock.index.entries from S1.5, which does not exist. * feat: let a manual bare-metal run choose its e2e suites A dispatch run only sees inputs declared under workflow_dispatch, which had none, so suites and suite-env resolved empty and fell back to the correctness set. Capturing a diskperf baseline by hand was therefore unreachable without a workflow_call caller. * fix: build the bare-metal suites the run step will execute The build step packs pre-built binaries into the tarball and did not see the suite selection, so a selected suite was absent from the node unless it happened to be one of the three the build script hardcoded. * feat: commit the measured disk-performance baseline Medians of three repetitions on the bare-metal cell. The throughput and latency bands now gate instead of warning, so a 2x regression in either job fails the suite. randwrite-4k's write tail is two orders above the mixed job's. It is recorded as measured rather than zeroed: an ungated metric would miss the regression class the gate exists to catch.	2026-09-03 14:33:21 +10:00
Julian Sommer	677055a478	test: cover the launch-template boot path end to end (#982)	2026-09-02 14:58:13 +10:00
Julian Sommer	4b4542a4c5	test: retry the multinode same-SG cross-host ping and capture OVN state at first failure ... The cross-host ICMP probe fired once, with no convergence budget, the moment sshd first answered on the source guest. The default SG self-ingress rule matches on the northd-derived $<pg>_ip4 address set, which only reaches the remote chassis after northd regenerates it from port-group membership and ovn-controller installs the flows there, so a first-packet burst can be swallowed by the priority-900 default drop with no log line anywhere. Give the probe the same 45s/2s budget the single-node suite already uses, and snapshot NB/SB address sets, port groups, ACLs, port bindings and br-int flows from every node plus the source guest on the first failed burst rather than at teardown, so the state is captured while the drop is still observable. Emit attempt count and convergence time so a pass that needed most of the budget still surfaces as a propagation delay instead of a silent green.	2026-09-02 11:37:22 +10:00
Julian Sommer	5b09cc9172	test: add a live E2E identity-policy authorization suite (#980)	2026-09-02 11:06:54 +10:00
Julian Sommer	a9a457489d	fix: re-anchor a caller-supplied policy ARN onto the caller's account ... The seven sourcePolicyARN actions resolved to the PolicyArn the caller sent, so the gate decided against an ARN whose account the caller chose. The scope now takes the ARN's path and name and rebuilds it on the caller's account, as sourceOIDCARN beside it already does, and widens to account-wide when the ARN carries no policy component. Lands with the cross-tenant integration suite that found it.	2026-09-02 10:18:50 +10:00
Julian Sommer	ee258fd248	refactor: back the e2e Eventually helpers with testify ... Delegate the hand-rolled deadline loops to require.Eventually and require.EventuallyWithT instead of open-coding the polling.	2026-09-01 12:02:39 +10:00
Julian Sommer	ce9b6e5b41	fix: gate cluster-wide IPsec on every chassis being configured (#953)	2026-08-31 15:31:59 +10:00
Julian Sommer	f919a257ef	fix: make the SG default-deny ACLs match every ethertype (#947)	2026-08-31 12:15:56 +10:00
Josh Siviour	abb817c7d5	feat: add admin Ochre catalog read and drop provider-direct model (#917) ... Expose per-model availability and reason on an operator-only ListOchreCatalog action for the console, keeping the tenant ListFoundationModels omission intact. Remove the unbuilt provider-direct entry from the v1 catalog.	2026-08-27 21:13:44 +10:00
Julian Sommer	fae6b08acc	fix: conform DescribeImages BootMode and IAM list required members ... DescribeImages projected BootMode through aws.String unconditionally, so an AMI recording none emitted "", which is outside the model enum. RegisterImage takes BootMode as an optional input, so this is the normal path for an AMI registered without one, not stale data. Set the member only when present. ListAccessKeys and ListGroups built their results from a nil slice when nothing matched. xmlutil drops a nil slice entirely rather than emitting an empty element, so the response lost a member the model marks required. Start both slices empty and non-nil. Integration conformance drops from 18 violations to 15; the remainder are the EC2 AccessDenied error-code findings, which are unrelated.	2026-08-27 11:46:39 +10:00
Julian Sommer	2da27c4e2a	fix: return InsufficientInstanceCapacity with a 5xx status (#903)	2026-08-27 09:14:30 +10:00
Julian Sommer	3bf0379daa	fix: gate sts:AssumeRole on the caller's identity policy (#889)	2026-08-27 09:04:25 +10:00
Julian Sommer	8e01e76dc0	fix: disable embedlit	2026-08-20 10:52:03 +10:00
Julian Sommer	c84200e519	feat: probe real qemu state (#839)	2026-08-20 09:46:53 +10:00
Ben Duncan	ae2a1830ef	Merge remote-tracking branch 'origin/dev' into account-quota	2026-08-20 04:01:14 +10:00
Ben Duncan	4508193180	fix: fail daemon start when a node with public pools cannot build external IPAM	2026-08-20 03:48:45 +10:00
Ben Duncan	0d8425ef51	test: skip the EIP quota leg where a cell does not serve EIPs from every node	2026-08-20 03:31:40 +10:00
Ben Duncan	19ec127e2a	test: charge a multi-instance launch for every instance it asks for	2026-08-20 02:43:44 +10:00
Ben Duncan	f41f5530a0	test: add an e2e suite proving quotas gate every capped resource	2026-08-20 02:26:27 +10:00
Julian Sommer	894fe84255	test: dump OVN MAC_Binding and route state when natuplink egress fails ... The artifact bundle carried the guest console and the spinifex journals but nothing from OVN, so a phase 6 egress failure could not be attributed to an unresolved gateway nexthop MAC versus a return-path or upstream problem. Register an OnFailure at the top of phase 6 that writes static-mac-binding-list, the SB MAC_Binding table and the VPC router's lr-route-list into the bundle.	2026-08-19 12:08:38 +10:00
Julian Sommer	a8d0ba185e	fix: bound the natuplink guest egress loop to the console verdict budget ... The guest fail-loop ran 36 iterations at ~28 s each (two 10 s curl timeouts, a 3 s ping, a 5 s sleep), so NAT-E2E-EGRESS-FAIL first printed at ~16m50s while the console wait gave up at 8m. The marker was structurally unreachable and an egress failure reported only a byte count. Derive the wait budget from the loop instead of hard-coding both ends: 10 iterations at a 30 s per-iteration allowance plus 3m for boot and cloud-init, keeping the same 8m ceiling. The user-data script is now generated from those constants and the existing marker consts. Dump the last console read via OnFailure so a budget overrun also leaves the console in artifacts.	2026-08-19 12:02:31 +10:00
Julian Sommer	17cea6cd4b	fix: gate the SG datapath probe on the target's HTTP listener ... TestSGPolicyDatapath/AllowedTraffic waited 60s for curl to reach the target's HTTP server but nothing ever waited for that server to start. Its cloud-init runs later than the client's sshd, which is what the preceding stage actually gates on, so the budget was measured from the wrong baseline and the gap between the two is unbounded on a loaded node. The failures reported curl exit 7, a refused connection: a missing SG allow is a drop ACL here, which would time out as exit 28. The target guest now reports its listener on the serial console, the only channel back to a guest whose SG has no SSH ingress, and a new TargetHTTPReady stage waits for that marker before AllowedTraffic runs at all. AllowedTraffic keeps its 60s budget, which now only has to cover chassis flow install, and dumps VPC flow diagnostics when it does fail so a genuine propagation failure is attributable from CI artifacts.	2026-08-19 11:59:02 +10:00
Julian Sommer	bdc79108e9	fix: make the DB endpoint the only path to the engine (#830)	2026-08-19 11:04:21 +10:00
Julian Sommer	95911cbfae	feat: DescribeDBEngineVersions and DescribeOrderableDBInstanceOptions (#807)	2026-08-17 15:04:12 +10:00
Julian Sommer	20a540cb59	chore: update manifest lint	2026-08-17 11:13:07 +10:00
Julian Sommer	eac0f175ba	feat: require TLS on RDS connections by default, on both engines (#790)	2026-08-17 09:01:42 +10:00
Ben Duncan	42d87d7a5a	feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799) ... * feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.	2026-08-15 14:22:49 -05:00
Josh Siviour	f7a2d85691	fix(e2e): stop flagging a Cluster bind to a single-homed node's own address (#789) ... checkSocketScope compared a socket's bind address to node.Addr and called it the node's WAN address, but node.Addr is the cluster peer address (harness.Node's own doc). On CI's single-homed tofu nodes this is the node's only address, so every Cluster-scope service failed no matter what it bound. Compare against a positively-known SPINIFEX_WAN_IP instead, distinct from the peer address, and record via harness.Detail when no such address is available so the skipped check is visible rather than silently passing.	2026-08-13 15:19:23 +10:00
Josh Siviour	b93f6c258a	fix(e2e): resolve the listener inventory without a Go toolchain on the target node (#787) ... * fix(e2e): resolve the listener inventory without a Go toolchain on the target node tests/e2e/multinode/listener_invariant_test.go resolved docs/security/network-connections/README.md at runtime via `go env GOMOD`, which works under `go test` but not inside a compiled .test binary shipped to a node with no Go toolchain. Embed the doc in a new docs/security/network-connections package instead, so the content is compiled into the binary and every caller parses bytes rather than a resolved path. RepoRoot and DocPath are now unused and removed. * docs: trim the embed package comment to the three-line standard	2026-08-13 14:18:02 +10:00
Josh Siviour	7d99705eff	fix(e2e): kill the node-failure phase abruptly instead of draining it (#785) ... * fix(e2e): kill the node-failure phase abruptly instead of draining it Phase 8 used `systemctl stop spinifex.target`, which now drains guests gracefully since the target-stop drain fix landed, so it tested an orderly shutdown rather than a failure. Since a target unit has no process of its own and cannot be signalled, and PartOf= only propagates stop/restart jobs rather than kill, SIGKILL each core service's main process directly instead, leaving orphaned guest qemu/nbdkit processes for recovery to reattach. Also fix a stale comment on spinifex-predastore.service's TimeoutStopSec: it claimed to match viperblock's drain budget, which is now 150s rather than 30s. The value was already correct; only the rationale was outdated. * fix(e2e): hold node2's outage open through Phase 8's assertions Restart=on-failure was respawning each killed unit within ~5s, so every Phase 8 assertion ran against an already-recovered node2 and DEGRADED_PEERS never actually dropped. Stop the same units right after the kill to cancel the queued restart and keep them down for the rest of the phase, and make the NATS degraded-peer count a real assertion (sized off NODE_COUNT) now that it can genuinely fail.	2026-08-13 12:30:15 +10:00
Josh Siviour	57bfbc20df	test: listener invariant static guard + e2e runtime check (#782) ... * test: add the listener invariant static guard and e2e runtime check Adds both halves the plan called for: a static test in spinifex/network/invariants that scans install scripts and config templates for a bare OVSDB ptcp remote, a wildcard bind on a Cluster/Encap port, or a bind port absent from the inventory, and an e2e test in tests/e2e/multinode that asserts the same on live ss -tulnp output. Both read docs/security/network-connections/README.md "## 1. Inbound Listeners" as their fixture through a shared, non-test parser package (spinifex/network/listenerinventory), so a wildcard exception has to be declared in the doc's own prose rather than hardcoded in either test. * fix: make the listener wildcard exception fail closed on an explicit marker The prior heuristic granted an exception for any unnegated "wildcard" or "0.0.0.0" mention within 40 characters of the port's row. That fails open: a future row phrased as "avoids the wildcard" or with a negation further upstream than the window would silently exempt itself. Replace it with a single exact literal phrase, "binds the wildcard by design", that a row must contain to grant the exception — no other wording counts, so an unrecognized mention denies rather than grants. Update the two rows that need it (5300, 500/4500) and document the marker's role in the inventory's own prose. * docs: record that the Geneve tunnel socket binds the wildcard by design	2026-08-13 12:29:26 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Josh Siviour	fd8be8a814	refactor(test): route duplicated NATS bootstraps through testutil (#776) ... * refactor(test): route duplicated NATS bootstraps through testutil Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that spinifex/testutil already provides, and one seedKV clone duplicated testutil.SeedKV. Route them through the shared helpers. Sites whose options genuinely diverge are left alone: fixed ports needed for reconnect and restart scenarios, custom TLS and token auth, a custom MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup. The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than CreateKeyValue, so their failure behaviour differs and they stay as they are. Also moves eight test files to external test packages where they declare no helper used by a sibling in the same package. Coverage is unaffected: Go compiles internal and external test packages into one binary per package, so attribution is identical either way. * chore(test): require new test files to use an external test package Adds a preflight check that fails when a newly added test file declares an in-package test. External test packages bind tests to the exported API and let packages share test doubles, which an in-package double cannot be. Only added files are checked, so the existing in-package tests are untouched. e2e and integration suites are exempt: both share a package-level fixture built in main_test.go, so their files cannot move individually. A file that needs unexported access can opt out with a //test:in-package comment.	2026-08-12 22:38:15 +10:00
Josh Siviour	9427cdfc2d	fix(test): give each predastore fixture its own S3 gate port (#775) ... The fixture pinned its S3 gate to 127.0.0.1:18443. Three test binaries import it and go test runs packages as concurrent processes, so two clusters raced for the port. The loser did not fail cleanly: waitForReady probed the winner's gate on the same port and reported healthy, so the setup client talked to another process's cluster and failed with BucketAlreadyOwnedByYou. Reserve a port per process instead, and reach the gate only through Fixture.Host so it cannot be hardcoded again. A Start that fails partway now drains the cluster before returning; its nodes hold process-wide pipe names that a later attempt would otherwise collide with.	2026-08-12 22:37:54 +10:00
Josh Siviour	91d22d9f12	feat(security): add an nftables host firewall scoping cluster ports to peers (#769) ... * feat(security): add an nftables host firewall scoping cluster ports to peers Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the formation port, NATS, predastore — was reachable from anything that could route to the node. On a single-NIC node the planes collapse onto the public address, which made that the default rather than the exception. setup.sh installs a policy and an apply helper but does not enable them: the peer set is unknown until the cluster's planes resolve, and a drop policy with an empty set would cut the first node that applied it out of its own formation. install-node.sh writes the peers and enables the unit after formation. The policy owns only table inet spinifex_filter and replaces it in one transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip nat/filter tables and reinstalls them only on start, so a whole-ruleset flush — what Debian's nftables.service does on both start and stop — would break elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here could not be rescued by an accept in the other table. Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the host netns, and every guest agent reaches its control plane over the AWS gateway rather than NATS, so no cluster port needs to face the guests. * feat: reconcile the host firewall peer set from the daemon The peer set was written by install-node.sh and by cluster-bootstrap.yml. That missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot), duplicated the same logic in each, and went stale as soon as a node was added or replaced. host.ReconcileFirewall now owns it: every formation path starts the daemon, and it re-runs on each startup. A node's own config names its peers by their wan address only, so a set built from config alone drops every peer's NATS, OVSDB and formation traffic. The lan plane comes from the generated NATS cluster routes and the OVN remote lists, and the vpc plane from the Southbound Encap table; an empty encap answer is an error rather than a narrower set. ProtectSystem=full applies to the sudo child too, so the root helper could not write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths, optional so a node installed before the firewall stage still starts. The apply helper gains set-peers and disable verbs, re-validates every address as a dotted quad with in-range octets, and rolls the peer file back if the ruleset will not load. * fix: retry the host firewall reconcile until every chassis registers The peer set is only safe to write once OVN reports an encap address for every node. On a fresh bootstrap the daemon routinely wins the race against ovn-controller, so a single startup attempt saw a partial chassis list and either wrote a set that dropped Geneve from the nodes missing from it, or failed outright and left the node with no policy until a restart. Gate on the full chassis count and schedule the reconcile instead of running it once: back off from 15s to 5m while it fails, then re-run every 5m so a node added or replaced after formation is picked up without a daemon restart. An unchanged peer set short-circuits before the helper runs. * test: cover the firewall chassis quorum gate in the multinode e2e suite A partial chassis list is the normal state early in a bootstrap, and writing a peer set from one produces a firewall that drops Geneve from every node missing from it. The gate that refuses that write had no live coverage: a 5-node bootstrap converges too fast for any daemon to observe a partial list, so the verification run only ever exercised the OVN-unreachable path. Reach the branch by inflating the expected count instead of breaking OVN: a phantom node in one node's config makes the cluster one chassis short, with nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer file is written while it holds, that the loaded ruleset survives, and that the node recovers to a byte-identical peer set once the phantom is removed.	2026-08-12 22:37:27 +10:00
Josh Siviour	98ec781629	feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770) ... * feat: introduce the EBSProvider interface and control-plane metadata store Phase 1 of decoupling the control plane from viperblock. Adds a provider-neutral block-storage contract that spinifex owns end to end. ebsprovider defines EBSProvider, whose operation set and idempotency rules follow the CSI controller service, with NATS request/reply as the transport rather than CSI's gRPC. Every request and response carries an explicit schema version so wire skew fails loudly at the boundary instead of silently decoding a changed payload. The package deliberately exposes no viperblock types: provider-specific state is an opaque handle. Ships with a NATS client against the ebs.provider.v1.* subjects and an in-memory fake for tests. ebsmetadata gives spinifex its own volume and AMI documents under a spinifex key prefix, so the control plane can answer DescribeVolumes and DescribeImages without reading the storage engine's persisted state. The EC2 volume and snapshot handlers gain provider-backed paths for create, describe, modify, attach-state and delete. Provider injection is optional for now, so the existing engine path stays the default until the viperblockd side serves the new subjects and the metadata migration lands. vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol it used and is now a local helper. * feat: serve the EBS provider contract from viperblockd Gives the ebsprovider NATS client a server. Volume create, describe, expand and delete, plus snapshot create and delete, are now handled by the daemon that owns BaseDir and the mounted-volume registry, which is where viperblock engine construction belongs. Every handler prefers an already-mounted engine over opening a second one on the same volume, and expand refuses outright while the volume is mounted. Opening a second engine on a live volume is the double-writer hazard this decoupling exists to remove, so the boundary enforces it rather than trusting callers. Requests carrying an unrecognised schema version are rejected at the boundary instead of being decoded on a best-effort basis. Snapshot creation replies immediately with an operation ID and completes asynchronously on a per-snapshot subject, since a large volume will outrun the request timeout. Draining the volume stays with the caller: it coordinates with the guest and is not something the storage boundary can reach. The snapshot create subject gains an explicit create verb. Addressing it as ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject and left the wildcard unsubscribable. Publish and unpublish are not served yet; the mount and unmount logic they would front is still inline in launchService and needs extracting first. Also adds a conformance suite that runs one set of contract assertions against both the in-memory fake and the NATS client, so each side is checked against the contract rather than against the other's structs. * feat: let an operator select the EBS provider Adds an [ebs] provider key and wires the NATS provider into the EC2 volume, snapshot, image and instance services at the daemon's composition root. Until now nothing injected a provider, so the provider-backed code paths in those services were unreachable. The key is provider-neutral rather than nested under [viperblock], because it names the boundary rather than one provider's settings; a second provider will not need it renamed. An empty value resolves to "embedded", so deployed config files that predate the key keep today's behaviour, and an unrecognised value is a startup error rather than a silent fallback. Startup probes the provider's capability verb and warns when nothing answers. The daemon and viperblockd start independently, so an unanswered probe is not fatal, but without it a misconfigured node fails much later as an unexplained request timeout on the first EBS call. "viperblockd" stays experimental: under it volume metadata lives in the control plane's own document, and the leak reaper, GC and admin paths still read viperblock's config.json, so they cannot see tenant or tag data for volumes created that way. * fix: make every volume-metadata consumer provider-aware Several readers and writers of volume metadata were never given a provider branch, so under the viperblockd provider they silently used the wrong store. Each of these is a correctness bug rather than a missing feature, because the control plane had already stopped writing the document they read. Modifications had nowhere to live: ModifyVolume built its result in memory and dropped it, and DescribeVolumesModifications read it back out of viperblock's config.json, so a describe straight after a successful modify returned nothing. Volume metadata now carries its own modification record. The leak reaper read attachment and tags from config.json and wrote orphan markers to tags.json, neither of which exists for a provider-managed volume, so a leaked volume was invisible to it and any mark it did write could not be read back. Tag mutations had the same split: a tag applied after creation never reached the document DescribeVolumes reads. Snapshot deletion could not see a provider-managed clone, because a clone records its source only in the control-plane document. That let a delete strip chunks a live clone still reads, so it is fixed here rather than left for the migration. Volume enumeration and DescribeVolumeStatus were likewise scanning object prefixes that a provider-managed volume does not create. Also adds an encryption flag to the volume document, which DescribeVolumes had been reporting as false unconditionally and snapshots could not carry at all, and a list operation for AMIs so the image path can follow later. The legacy path is unchanged; its invariant tests pass unmodified. * feat: backfill ebsmetadata documents from legacy volume and AMI state DescribeVolumes under the viperblockd provider enumerates only ebsmetadata documents, so a volume created before the switch had no document and was invisible rather than merely slow to read. A per-object read fallback cannot fix enumeration, so both a backfill and a fallback are needed. Add an object-store migration kind to the migrate framework. Predastore has no conditional write, so an object-based version stamp would be last-writer-wins across nodes; RunObject stamps progress in a caller-supplied JetStream KV bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a read-then-write rather than a compare-and-swap, so correctness rests on every ObjectMigration.Run being safe to re-execute. The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the numbered config migrations: it needs viperblock's VBState schema to decode the legacy config.json, and migrate is imported by every handler that stamps a KV bucket version. It converts what it can reach and skips unreadable objects with a log, so one corrupt volume cannot hide the rest of the fleet. ebsmetadata gains the legacy read fallback for the window before the migration has run everywhere. It stays free of viperblock by taking the decode as an injected function value, wired at the composition root, and the fallback is retired by leaving it unset. A fallback read never writes the converted value back, which would race the migration. Prefix enumeration pages to exhaustion. Stopping at the first page would let the backfill stamp itself complete having converted only the volumes it happened to see, reintroducing the invisible-volume failure it exists to prevent. The migration runs from configureEBSProvider, so only when the provider is selected; backfilling while the embedded engine is still authoritative would produce documents that immediately go stale. Failure is fatal rather than advisory, unlike the capability probe beside it. This makes the switch to viperblockd one-way, which is now recorded on the config field. * feat: serve volume publish and unpublish from viperblockd PublishVolume and UnpublishVolume were the last unserved verbs on the provider contract, and they are the ones that attach a volume to a VM. Until they are served the provider path cannot boot an instance, so provider = "viperblockd" cannot be verified end to end. The mount and unmount logic was two large anonymous handler bodies inline in launchService. Rather than write a second mount path, extract each into a named function and have both the legacy ebs.mount/ebs.unmount handlers and the new provider handlers call it. Two divergent implementations of nbdkit spawning and the seal decision would be far worse than the inline bodies. The extraction relocates code without rewriting it; response publishing stays in each handler so the provider never publishes to ebs.mount.response. Publish is idempotent: a volume this node already has mounted returns its existing attachment rather than starting a second nbdkit against it, which is the double-writer hazard the boundary exists to prevent. Unpublish preserves the invariant that a failed seal leaves the volume mounted so a retry can re-attempt it, and reports an error rather than success. Both verbs are node-addressed, so they subscribe directly rather than through the spinifex-workers queue group, and are registered only when NodeName is set. There is no queue-group fallback: a subject that already names one node has nothing sensible to fall back to. Read-only attachment is refused rather than silently granted read-write, since nbdkit is started read-write or not at all. Also fixes a latent panic carried by the extracted mount path: viperblock.New returns a nil VB alongside its error, and SetCacheSize was called on it before the error was checked, so a failing New would have taken down the handler. * feat: read and write AMI metadata through ebsmetadata The backfill migration wrote ebsmetadata AMI documents that nothing read: every AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at {imageID}/config.json. Give the image service provider branches for the AMI read, write and enumerate paths, with enumeration going through ListAMIs rather than an object-prefix scan, as DescribeVolumes already does. AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a viperblock type on an interface boundary in the instance package. It now returns ebsmetadata.AMI on both paths, the embedded one converting before it returns, so the boundary names no viperblock type whichever provider is selected. The legacy-to-document mapping had accumulated three copies. Consolidate them in ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata itself stays free of viperblock, and cover it with a reflective test that fails when a field is added to either schema without being wired through. Sixteen fields copied by hand in three places would have dropped one silently. The admin image paths read whichever representation exists and write back to that same one, so a provider-managed AMI never gains a spurious config.json. FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan: missing a dependent there deletes an AMI a live volume still needs, the same class of bug as the snapshot clone path fixed earlier on this branch. That union is built by hand rather than by wiring Store.ListVolumes' fallback, which swallows per-ID read errors. This path deliberately fails closed on a transport error so a delete cannot proceed on a partial picture, and the fallback would have quietly turned that into failing open. The CLI image import now also writes the document, best-effort, so an AMI imported after the migration is not left reachable only through the fallback. * fix(ec2): resolve volume config through ebsmetadata when a provider is configured A provider-created volume has no vol-*/config.json — its metadata lives in the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy key unconditionally, so AttachVolume reported a volume that DescribeVolumes had just listed as available as InvalidVolume.NotFound. Routing through ebsmetadata.Store also puts this path behind the legacy fallback, so a volume created before the provider switch stays attachable. The NoSuchKey remap now names the key it missed; its silence made the original failure invisible in the daemon logs on every node. * test(bench): add an EC2 control-plane latency benchmark Measures the EBS control-plane operations either side of the provider boundary, so the cost of the NATS hop can be stated rather than guessed. Reports median and p95 with a bootstrap CI on the median difference, and labels a delta indicative unless that CI excludes zero. * fix(ec2): drain the volume before a provider snapshot The legacy CreateSnapshot branch drains first so the snapshot reads a current checkpoint; the provider branch did not, and viperblockd documents the drain as the caller's responsibility. Both sides deferred to the other, so an attached volume snapshotted stale data with no error raised. DrainVolume and volumeAttachment now take the fallback state and instance as strings rather than a viperblock.VolumeMetadata, removing a viperblock type from an exported signature. * feat(ec2): route the instance root volume through the EBS provider RunInstances built its own viperblock engine to clone the AMI snapshot into the root volume, which made every launch a second writer against a volume viperblockd already serves. The launch now calls the provider's CreateVolume with the AMI's snapshot and source volume, and records the result in ebsmetadata so DescribeVolumes and attach can see it. A snapshot clone needs both the snapshot ID and its source volume ID to resolve base blocks, so CreateVolume in the volume handler now sends SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected every create-volume --snapshot-id outright. The gp3 IOPS and throughput constants move to spinifex/types so the instance and volume handlers share one definition rather than duplicating literals. * feat(ebsprovider): add a bounded volume seed to CreateVolume The EFI variable store must be initialised from the launching node's firmware VARS template, byte for byte. Passing a host path would not work: CreateVolume is queue-subscribed, so any node can service it, and a foreign node's OVMF package would bind the guest's pflash region to the wrong firmware build. CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd writes them at offset 0 in the engine it already builds. The seed applies only when the create actually allocates the volume, so a relaunch never overwrites a BootOrder the guest has since written. JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes bounds the seed well inside the NATS max_payload the cluster runs with and both ends reject an oversized seed with a clear error rather than letting the publish fail. VolumeSeeding advertises the behaviour so a provider that cannot seed says so instead of silently dropping the bytes. * feat(ec2): route the EFI variable store through the EBS provider prepareEFIVolume built a control-plane viperblock engine to seed the variable store from the firmware VARS template, making every UEFI launch a second writer against a volume viperblockd already serves. It now calls the provider's CreateVolume with the template as seed bytes, so the store is allocated and written by the daemon that owns the volume. The template is still read on the launching node and shipped with the request. CreateVolume is queue-subscribed, so any node can service it, and a VARS region seeded from a foreign node's OVMF build would bind the guest's pflash to the wrong firmware. pflash refuses a VARS region that is not byte-exact, so the returned capacity is checked against the template length; a provider that rounded up to a GiB boundary would otherwise yield a guest that cannot boot. The engine path stays behind a nil-provider guard until the legacy branch goes. * feat(ec2): route image snapshots through the EBS provider snapshotRunningVolume and snapshotStoppedVolume each built a control-plane viperblock engine to take the snapshot backing a new AMI, making CreateImage a second writer against a volume viperblockd already serves. Both now call the provider's CreateSnapshot with the volume's provider handle. The running path keeps its drain, which is what makes the snapshot current rather than one predating a guest-triggered write. The stopped path needs no drain, and the daemon already prefers a live mounted engine over opening a second one, so it handles both cases itself. getVolumeConfig read the legacy config.json, which a provider-created volume does not have, so every CreateImage against one failed. It now reads the ebsmetadata document under a provider, the same way attach does. * fix(daemon): only the owning node creates an image for a stopped instance handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which only the node holding the VM passes, so a running instance is imaged once. A stopped instance lives in the cluster-replicated KV instead, which every node reads successfully, so all of them ran the full snapshot and AMI creation. The caller saw the first reply and the rest leaked: on a three-node cluster one create-image produced three AMIs and three snapshots. A stopped instance records its owner in LastNode, so a node that is not the owner now declines. Gather tallies error frames and keeps waiting, returning only on a success, so declining cannot preempt the owner's reply. An instance stopped by a build that never set LastNode elects an owner through the KV's compare-and-swap, since having every node decline would make create-image fail permanently for those records. * perf(ec2): serve DescribeVolumes by ID from the volume documents The provider branch called ListVolumes unconditionally and filtered in memory, so asking for one volume read and decoded every volume document in the bucket. The non-provider branch already had the equivalent fast path. Explicit IDs now fetch their documents directly. The tenant check moves with them: a document belonging to another account reports not-found rather than being returned, matching what the enumeration filter did. Measured against a memory object store, the fast path stays flat near 20us regardless of bucket size while the enumeration grows linearly, reaching 4.6ms at 500 volumes. * chore(lint): ban the viperblock import from the EC2 control plane The control plane reaches block storage through ebsprovider now, but nothing stopped a handler from importing the engine again. That is not hypothetical: the instance handler first imported viperblock for a config struct alone, and the engine construction followed thirteen days later. depguard denies the import across handlers/, which matters because viperblock.New starts a chunk uploader — every construction is a writer against a volume viperblockd already serves. A types-only import is denied too, since that is exactly how the last one started. The four files still carrying the pre-provider engine path behind a nil-provider guard are exempted by name. The exemption list is the remaining work: it shrinks to nothing when that path goes. * fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot CopySnapshot wrote a control-plane snapshot document under the new ID and nothing else. The copy described correctly and CreateVolume against it succeeded, so it looked like it worked, but the new ID had no viperblock snapshot metadata behind it and the volume failed when the guest attached it. Producing the metadata means re-sealing it under the destination ID. On an encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the sealing helpers are unexported, so spinifex cannot write it. Copying only the unencrypted form would be dead code on any cluster with encryption enabled. Refusing with UnsupportedOperation fails at the call the caller can act on rather than at attach time. Source lookup and the ownership check still run first, so a bad ID or another tenant's snapshot reports that instead. * test(ec2): give handler test fixtures their own legacy config.json shape Handler tests built the legacy config.json fixture by importing viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly. Mirror that shape in ebsmetadata/vblegacy, next to the production code that already reads it (ebsmetadatabackfill), and switch the fixture-only usages across the volume/snapshot/image handler tests to it. A byte-for-byte equivalence test against the real viperblock types guards the mirror from drifting out of sync. Call sites that hand a viperblock type into putVolumeConfig, mergeVolumeConfig, or buildVBConfig are untouched, since those functions still name the real type in their signature. * feat(ebsprovider): add CopySnapshot to the EBS provider contract Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy NATS subject, MemoryProvider and NATSProvider implementations, conformance coverage) and a viperblockd handler that duplicates a snapshot's metadata via viperblock.CopySnapshotMeta on the source volume's VB. * feat(ec2): serve CopySnapshot through the EBS provider CopySnapshot has been refusing with UnsupportedOperation because a copy needs its own snapshot metadata on the backend under the new ID, and on an encrypted volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the control plane cannot re-seal it. The provider contract now carries a copy operation, so the storage engine does it. The provider path generates a new snapshot ID, has the provider duplicate the backend metadata under it, then writes the control-plane document. Rollback mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot rather than leaving one nothing references. Ownership is still checked against the source before anything is created. The no-provider path keeps refusing. It has no way to produce the backend metadata, and the copy it used to write described fine and then failed at attach — a resource that fails as far as possible from the call that made it. Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot reached the provider rather than existing only as a control-plane document. That assertion is what catches the old behaviour. * refactor: remove the embedded EBS engine path from the EC2 control plane viperblockd is now the only EBS provider. The pre-provider path that constructed a viperblock engine in-process is deleted, and with it every github.com/mulgadc/viperblock import under spinifex/handlers/ (tests included). The depguard ec2-control-plane rule drops all four file exemptions and the !$test carve-out, so the rule now reads simply as "nothing under spinifex/handlers/** may import viperblock". viperblock.VolumeConfig was also a data carrier on the provider path. It is replaced by the spinifex-native ebsmetadata.Volume rather than the vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is decoding old objects, while ebsmetadata.Volume already carries every field the call sites read, so the conversion step disappears instead of being reimplemented against a legacy shape. The embedded-only helpers (volumeConfigWrapper, putVolumeConfig, mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached, deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json prefix scan) had exactly one production caller chain each, all through a nil-provider branch, and are removed. The snapshot-dependency guard on DeleteVolume is provider-agnostic and is kept, now running on the provider path where it previously did not. config: viperblockd is the default and an empty Provider resolves to it. An explicit provider = "embedded" is rejected at config load rather than silently upgraded, because starting anyway would migrate a node's volumes to ebsmetadata one-way without the operator asking. The generated spinifex.toml template emits viperblockd. * fix(images): write the imported AMI document from the config the import wrote ImportDiskImage creates the AMI's snapshot and records it on its own *viperblock.VB, which viperblock.New allocates fresh, so the value lands only in {imageID}/config.json. The import command was persisting the ebsmetadata document from its pre-import manifest copy, whose SnapshotID is still empty. Store.GetAMI only consults the legacy config.json fallback when the document is missing, so that empty-SnapshotID document permanently shadowed the real value and every launch from an imported AMI failed the zero-copy clone check with ServerInternal. Rebuild the document by reading back the config.json the import just wrote. A missing or unreadable config is skipped rather than written from the stale manifest: no document still resolves correctly through the legacy fallback. * test(e2e): read NODE from spx get vms instead of VCPU in the spread check spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to field 8. The spread placement check still read field 6, so every instance reported its vCPU count instead of its node, the unique-node count always collapsed to 1, and the check failed on clusters that had spread correctly. Read field 8 and strip the ANSI colour codes wrapping each cell. * feat(viperblockd): add a --debug flag so viperblock debug logging is reachable viperblockd hardcoded Info, and viperblock's VB.logger() falls back to slog.Default() because vb.log is only ever set by SetDebug. That left the engine's debug records, including the chunk uploader's start and disabled lines emitted during New(), unreachable in a deployed environment. Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG rather than the bare debug key, which several commands share. Also sets the Config.Debug field, which existed but was set only by tests. * fix(viperblockd): release the chunk uploader when openVolumeVB fails viperblock.New starts a chunk uploader and a WAL syncer before the backend is touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the caller while leaving both running with no handle to stop them. Repeated backend failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders on a 30s ticker for the life of the process. Adds invariant tests for both halves of the ownership rule: a mounted volume's snapshot create and copy reuse the live engine and log zero volume opens, an unmounted volume opens exactly one, and a failed open leaves no uploader behind. The open counts are read from the same log record a production alarm watches. * fix(ebsprovider): route requests to the node that owns the mount Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one spinifex-workers group, so NATS delivered each request to a random node. Only the node holding the mount can reuse the live engine through findMountedVolume; any other node constructed a second viperblock engine, with a live 30s chunk uploader, against a volume nbdkit was serving on a different host. Measured on a three-node cluster, this was roughly two snapshots in three. Rather than resolve the owner, let the owner answer. A node subscribes ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts a volume and drops it on unmount, mirroring the per-volume ebs.config subscription. Clients try that subject first and fall back to the queue group only on ErrNoResponders, which means nothing is mounted and opening an engine is correct. A timeout deliberately does not fall back, since the owner may still be running the operation. Applies to snapshot create and copy, expand and describe, the four verbs that reuse a mounted engine. Delete keeps queue dispatch. * fix: rebuild viperblockd mounted-volume registry after a restart Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a viperblockd restart while a VM keeps running leaves nbdkit serving a volume the daemon has forgotten. Owner routing then infers from ErrNoResponders that nothing is mounted and falls back to the queue group, which opens a second viperblock engine, with a live chunk uploader, on another node. The same empty registry also defeats handlePublishVolume's double-mount guard, so a republish could start a second nbdkit. Discover surviving nbdkit processes from /proc at startup and rebuild the registry from them, following orphan_scan.go's precedent: match comm exactly, parse the argv for volume=<id> and --unix/-p, and adopt a PID only when nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or killed; anything that fails corroboration is logged and skipped. Recovery runs before the first subscription so no handler can race it. Extract mountVolume's engine construction into constructMountedVB so mount and recovery cannot drift, and release the WAL syncer viperblock.New starts when construction fails after it. * fix: corroborate recovered nbdkit processes from argv, not a pidfile nbdkit is launched with -f (required for the Go plugin) and never writes the --pidfile it is given, so no such file exists on any node; the daemon's own /tmp is a PrivateTmp namespace besides. Corroborating against it rejected every genuine mount, making recovery a no-op in production. Corroborate against what the kernel actually has: the process must serve our plugin, name this daemon's base dir, and still hold its socket. Adopt a volume once, so two live nbdkits for it leave one registry entry and a warning rather than two conflicting ones. * feat: add a second EBS provider backed by qcow2 and qemu-nbd The decoupling was justified by swappability that nothing demonstrated: the conformance suite ran against MemoryProvider and against a NATSProvider wired to a MemoryProvider, so both arms were the same implementation behind different transports. A fake is not a second implementation. Add three pieces. natsserve serves the ebs.provider.v1.* contract from any EBSProvider, depending on nothing but the interface, and replaces the test-only server twin so conformance now exercises production code. The conformance suite moves to its own package and is exported so implementations outside ebsprovider can run it, with no assertion changed. qemunbdd is the second implementation: one qcow2 file per volume, qemu-nbd for publish, real snapshot copies instead of chunk references, and every exec behind a runner seam so its tests need no qemu. Owner-subject routing is deliberately not served neutrally: it needs a per-node mounted-volume registry the interface does not model, so it stays in viperblockd and a neutral server's clients correctly fall back to the queue group. * test: run the conformance suite against the qcow2 provider The suite passes unmodified against an implementation sharing no code with viperblock, both in-process and through natsserve over the real wire contract, which is the swappability claim the provider boundary was built to support. Two real defects it caught. ExpandVolume and GetVolume inspected the image before consulting the publication map, so qemu-img could not open a volume qemu-nbd held the write lock on and the refusal surfaced as an internal error instead of ErrVolumeInUse; qemu-img info now runs with --force-share, so describing a volume no longer depends on whether it is attached. Publish also failed with qemu-nbd's raw exit status when the socket path exceeded the 108-byte AF_UNIX limit, and now reports that reason itself. * feat: make the EBS provider selectable by config Add the qemunbd selector alongside viperblockd, a spx service command that serves the qcow2 provider over natsserve, and service registration. Config validation accepts the new value everywhere viperblockd is accepted, while embedded stays rejected by name. The daemon logged a hard-coded viperblockd constant as the active provider rather than the resolved one, in configureEBSProvider and in both of probeEBSProvider's lines; all three now report what is actually configured. Both daemons answer the same subjects, so exactly one may run on a cluster. * test: add a live lifecycle check for the qcow2 provider Drives create, publish, write, snapshot, restore and read-back entirely through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a cluster node, so the client under test is the same shape the control plane uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE is set, so it stays out of preflight. * fix: publish NBD URIs in the form standard clients accept PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever consume what the EBS provider boundary hands back. Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit all accept. ParseNBDURI now takes both that and the TCP form, and still accepts the legacy nbd:unix: syntax for state written by older daemons, so a conformant third-party provider's URI is no longer rejected on the way in either. Verified against qemu 10.0.11: qemu-img and a real -drive option string both open the URI form, and qemu reports both forms as nbd+unix://?socket=, so it already treats the URI as canonical. * test: validate published exports with libnbd's client tools Every other assertion about the EBS provider goes through our own Go client, so nothing has ever checked whether the export we publish is usable by an NBD client that knows nothing about this codebase. That is the layer neither the conformance suite nor a CSI-shaped contract can reach. RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume: the URI connects, the export advertises its size and block-size constraints, data written externally reads back byte for byte, a ReadOnly publish sets the read-only transmission flag, and UnpublishVolume tears the export down. It skips when libnbd-bin is absent rather than passing vacuously. Wired to qemunbdd, where all five failed before the URI fix. The publish assertion in the main suite now requires a dialable URI instead of merely a non-empty string, which is what let the wrong form through. * test: run the external NBD client suite against a live viperblockd The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit export is the comparison worth having: one external client, two providers it cannot tell apart, and every difference is a candidate leak in the boundary. RunNBDClientSuiteWithConfig takes the node name and volume prefix a live cluster needs, and publishForNBD now unpublishes and deletes what it created so a run against a real cluster leaves nothing behind. Subtests dial a normalised URI so a provider emitting the legacy syntax still gets its export tested rather than every subtest failing on the same string; whether the URI itself is standard is now its own subtest. Adds a reconnect check, since publication holds the volume rather than any one connection. It retries the real operation because a cheaper liveness probe is itself a connection and re-triggers whatever refused it. * feat: make the conformance suite capability-driven The suite asserted a fixed Capabilities literal, so it could only ever pass against a provider advertising exactly that set. Every implementation had to match the suite rather than the suite testing the contract. RunSuite now reads GetCapabilities from the provider under test and gates optional behaviour on what it advertises, and the fixed literal is demoted to ReferenceCapabilities, a convenience for callers constructing a MemoryProvider. Add ReadOnlyPublish to Capabilities. A provider that leaves it false must refuse a ReadOnly publish rather than hand back a writable export, which is what viperblockd already does; it now has a way to say so up front. Add ErrUnsupportedCapability and the unsupported_capability error code. A request for optional behaviour a provider does not advertise is well formed, so classifying it as invalid_argument told the caller to fix a request that was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly. Declared capabilities now match observed behaviour: qemunbdd advertises SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises neither: the nbdkit Go binding exposes no extents callback, and the mount path never passes -r. The external NBD client suite gains a sparse-extent subtest that checks the export's advertised contexts against the capability, so SparseExtentReporting is falsifiable from outside rather than taken on trust, and its read-only subtest now branches on the capability instead of assuming refusal. Cover the new branches with a MemoryProvider advertising no optional capabilities and one advertising OnlineExpansion, the branch no provider we ship exercises. * refactor: rename SourceVolumeID to SourceSnapshotVolumeID CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd required it whenever SourceSnapshotID was set and used it to resolve the snapshot's chunk references, so there it named the snapshot's origin volume. MemoryProvider ignored it entirely. qemunbdd implemented it as an independent volume-to-volume clone, which is what the name and doc comment implied. The conformance suite could not catch the divergence because the reference provider had no branch for the field at all. The contract already has a field meaning "the volume this snapshot came from": Snapshot.SourceVolumeID, which every provider populates. So the request field is the same fact travelling back the other way, and its name was the problem. Rename it to SourceSnapshotVolumeID, document it as the source snapshot's origin volume, and state that it is required with SourceSnapshotID and meaningless without it. All three providers now enforce one rule: a source snapshot must be accompanied by its origin volume, or the request is invalid_argument. MemoryProvider additionally rejects an origin that does not match the snapshot's own SourceVolumeID, so a wrong name fails instead of resolving blocks against a volume the snapshot never came from. qemunbdd loses its volume-to-volume clone branch, which implemented behaviour the contract no longer offers and no caller ever requested. The suite pins both the requirement and a full create-snapshot-restore round trip, so the field is exercised end to end rather than only on the error path. * feat: make the conformance suite runnable against a live provider The suite had only ever run against MemoryProvider and qemunbdd, both of which are discarded with the test binary. viperblockd, the provider production actually uses, was the one implementation the suite never judged, because the suite assumed each subtest got fresh state: hard-coded volume IDs collide with an earlier run's leftovers on a store that keeps them, and a hard-coded node ID addresses a node that does not exist. SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records every volume, snapshot and publication the suite creates and tears them down in dependency order, logging rather than failing so a teardown problem cannot mask the assertion a subtest was making. Two new tests cover this by running the whole suite twice against one shared provider instance, which passes only because cleanup works, and once per prefix to cover concurrent runs sharing a daemon. TestLive_ViperblockdConformance wires this to a real cluster behind SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite. * fix(viperblockd): close the four contract gaps the live conformance run found Unpublishing a volume this node has no mount for now succeeds instead of returning not_found, matching the legacy unmount path: nothing mounted here is already the state the request asks for. CreateSnapshot refuses a snapshot ID that already exists against a different source volume with already_exists, rather than overwriting it. DeleteVolume refuses a published volume with volume_in_use instead of tearing the mount down, on this node or on whichever node answers the volume's owner subject. The export and the guest writing through it survive the metadata deletion, so the old behaviour surfaced later as corruption. CreateVolume reads the source snapshot before writing the volume, returning not_found when it is absent and invalid_argument when it was taken from a different volume than the request names. Previously both produced a blank volume and a successful response. CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument now that it carries a sentinel. * feat(ebsprovider): pin the source snapshot's origin volume in the contract The suite only required source_snapshot_volume_id to be present, so a caller naming a volume the snapshot never came from was accepted everywhere. The snapshot records its own origin, so this is checkable: creating from a snapshot with a mismatched source volume is now invalid_argument for every provider, enforced by qemunbdd against its snapshot metadata and by viperblockd against the snapshot's persisted SourceVolumeName. * test(ebsprovider): run the contract suite against viperblockd in CI CI installed neither qemu nor nbdkit, so every conformance test that needs a real implementation skipped silently and MemoryProvider was the only provider the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails instead of skipping. TestViperblockdConformance stands the provider handlers up over an embedded NATS and a real predastore and runs the same suite against them, which is what caught CopySnapshot accepting a foreign volume's snapshot on unencrypted volumes. * feat(viperblockd): honour read-only publish and advertise block size constraints ReadOnlyPublish was advertised false and read_only=true was rejected, so the capability was true only by omission. nbdkit -r now starts the export when the request asks for it, which sets the NBD read-only transmission flag and makes the plugin refuse writes. Access mode is fixed when nbdkit starts, so a republish asking for the other mode is refused with VolumeInUse rather than handed the running export. Block size constraints were absent entirely: the nbdkit Go binding has no block-size callback, so the plugin cannot advertise them. The blocksize-policy filter can, without a binding callback. Minimum 1 because viperblock's WriteAt handles unaligned requests with a read-modify-write, preferred 4096 to match its block size, maximum 32M to match what qemu-nbd advertises. * test(conformance): judge sparse reporting by the allocation map, not the context The subtest asserted that base:allocation was offered. nbdkit offers that context whether or not the plugin can compute extents, answering 'all allocated' when it cannot, so the assertion passed for a server with no extent support at all — it could not fail. What separates the two is the answer: a volume with nothing written to it is entirely holes to a server that really tracks allocation. Read the map with nbdinfo --map --json and compare holes against the advertised capability. qemunbdd reports holes and advertises true; viperblockd reports none and advertises false. * test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime The predastore fixture is one daemon per test binary, but its cert was exported with t.Setenv, which reverts when the first test that called Start finishes. In-process clients did not notice because crypto/x509 caches the pool behind a sync.Once, loaded while the variable was still set. Child processes did: nbdkit spawned by a later test inherited an environment with no SSL_CERT_FILE and could not verify the daemon, so the whole NBD client suite failed when the package ran as a whole while passing when run alone. Also install libnbd-bin in CI, so the NBD client suite runs there instead of failing on missing tools. * ci: install nbdkit-plugin-dev so the viperblock plugin can build * test(volume): give the snapshot fixture an owner CreateVolume now fails closed on a snapshot whose owner it cannot match, including an unset owner_id. The fixture predates that and wrote none, so the provider test failed with InvalidSnapshot.NotFound before ever reaching the provider. * test(integration): give the volume harness a real provider The control plane no longer builds volumes itself, so CreateVolume in the integration harness failed with "no EBS provider configured": the harness wired a VolumeServiceImpl with nothing behind it. It now serves viperblockd's provider subjects against the same predastore fixture, so the blocks are still written by real viperblock to a real daemon. Three persisted-state assertions moved with it. TenantID, VolumeType and AvailabilityZone are control-plane facts, held in the ebsmetadata document rather than viperblock's own state, so the storage oracle now asserts their absence and DescribeVolumes -- whose answer is account-scoped -- is what checks them. * refactor: route image import through the EBS provider and drop the legacy metadata path The control plane could still only run against viperblock: the AMI import and the ochre weights import both linked the storage library directly, and the ebsmetadata store kept a read fallback onto viperblock's config.json. Import now goes through the provider contract with no new verb: CreateVolume, PublishVolume, a bulk copy over the published NBD export with qemu-img, then UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS, whose request size the contract caps far below a disk image. The migration scaffolding goes with it. ebsmetadata documents are now the only thing the control plane reads: no legacy prefix scan, no config.json fallback, no backfill on daemon start. config.json belongs to the provider, which is the single writer of it. ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is right for DescribeImages, where one corrupt document must not hide the fleet, and wrong for a name-uniqueness check, which cannot rule out a name it failed to read. spinifex and cmd no longer import viperblock outside services/viperblockd and testutil/vbscan. * feat: add a volume enumeration verb so a lost document cannot strand blocks Making ebsmetadata documents the sole index of volumes left one failure with no recovery: a volume whose document is gone disappears from DescribeVolumes, from the leak reaper (which enumerates documents), and from the provider, whose contract had no verb that asks what it holds. ListVolumes closes that. It is capability-gated on VolumeEnumeration and paginated, following CSI's optional LIST_VOLUMES rather than inventing a shape. Refs are deliberately not Volumes: enumeration answers what exists, and a provider must not open every volume's engine to say so. Each implementation reads what actually exists rather than what it remembers. qemunbdd walks its volumes directory, not its in-memory map, which does not survive a restart; viperblockd walks the bucket's top-level prefixes. A volume the provider has forgotten is precisely what a caller is looking for. The consumer is a read-only orphan report: provider volumes minus documented ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is excused only when its base volume is documented, so a stranded pair is still reported. Listing documents stays strict, because reporting a volume as orphaned on the strength of a document that failed to decode would point an operator at live data. * fix: stop the orphan scan accusing storage of holding things it does not A live env19 run reported three orphans against a healthy cluster: an AMI's backing volume, a snapshot, and the cluster key store. All three were wrong, and a report that cries wolf on a clean cluster is worse than no report. Enumeration treated every top-level bucket prefix as a volume. Snapshots sit beside volumes at that level and have a config of their own, so presence of a config cannot separate them; the snap- name is what does, which is already how the AMI removal path reads them back. keys/ and spinifex/ are not volumes at all. An AMI's blocks are a genuine provider volume, so enumeration keeps reporting them. What was missing is on the reading side: an AMI is recorded as an image document, not a volume one, so the scan now treats a documented image as known. Listing images is strict for the same reason listing volumes is — unreadable and absent are different answers, and only one means orphaned. * fix: let an orphaned image's blocks be reclaimed The orphan report tells an operator that reclaiming a stranded volume is their decision, and images remove --force is the tool it points them at. On env19 that tool failed on exactly those volumes: removal drops the ebsmetadata document first, and predastore answers NoSuchKey for a document that is already gone, which is the definition of an orphan. Deleting the document stays first and stays fatal. That ordering is the barrier that stops a launch landing on blocks about to be deleted, and an AMI still advertised after its blocks are gone is the worse failure. Only an already-absent document is now tolerated: the barrier is satisfied, and the blocks are the sole thing left to reclaim. S3ObjectStore.DeleteObject normalises a missing key the way GetObject already does, so this is recognised rather than matched on error text. The obvious unit test passed without the fix, because the in-memory store is silently tolerant on delete and real S3 answers 204 — neither behaves like the backend this runs against. The test now drives a store that reports the miss, and fails without the fix. * feat: add per-verb OTel spans to the EBS provider contract The ten provider verbs cross a single NATS seam and nothing measured it. The only spans on the storage path were the legacy ebs.mount/ebs.unmount pair, so control-plane storage latency could not be attributed to a verb or compared across provider implementations. NATSProvider.request is the one client-side funnel every verb passes through, including both legs of an owner-subject fallback, so a single span there covers the whole contract and measures what the control plane experiences regardless of which provider answers. Span names come from a closed verb set rather than the subject. Owner and snapshot-create subjects embed a volume ID and the mount/unmount subjects embed a node ID; naming spans after them would give every volume a span name of its own. The IDs move to attributes and an unrecognised subject collapses to a single unknown verb. Trace context rides the message headers, so the serving provider joins the caller's trace instead of rooting one of its own. The carrier is a copy of the one in spinifex/utils rather than an import: that package pulls in the aws-sdk and the cluster key store, which the wire contract must not depend on. Responses carry no shared accessor for their error field, so both sides re-read just that field from the encoded reply, and only when the span is recording. Without it a failed verb looks like a successful round trip. viperblockd serves the contract with its own handlers rather than natsserve, so its handlers now take the per-message context and its publish/unpublish pair drops the subject-named consumer span it opened for itself. * feat: add null and hostile providers, and a per-verb bench suite The provider boundary made it possible to put something other than viperblock behind the control plane, and nothing yet did. nullprovider is the floor. It is not a new type: MemoryProvider already answers every verb from a map and natsserve already serves any EBSProvider, so the package names that composition and states what it measures. On loopback with no storage behind it a round trip costs around 195us against 74ns in process, which is the seam's own price and is not attributable to any storage engine. conformance.RunBenchSuite prices one round trip per verb and drives every implementation through the same calls in the same order, so two providers' numbers are comparable. It measures control operations only: guest I/O never crosses these verbs. hostile wraps any provider in one that misbehaves. Faults derive from the seed and a per-target sequence rather than a global counter, so concurrent callers on different volumes cannot reorder each other's faults and a failure found deep into a soak run replays exactly. Error and error-after-work are separate faults on purpose: the second reports failure after the work has already happened, which is how an orphan is made, and is what makes an enumeration scan the right oracle for a soak. Enumeration itself is never faulted, since an oracle that can be told what to say cannot falsify anything. The soak asserts that every volume the provider still holds is attributable to an injected fault. A leftover with nothing behind it is the control plane losing track of storage on its own. Two bugs were found by falsifying the new tests rather than by running them. The FNV-derived draw clustered badly enough that two fault bands were never reachable in 1200 draws, so the draw now goes through a SplitMix64 finalizer. And the first version of the lie test was worthless: MemoryProvider returns copies, so a lie could not corrupt it whether or not the injector copied, and the test now drives a provider that hands out pointers into its own state. * test: bench the provider contract against qemunbdd and a live viperblockd RunBenchSuite was only ever pointed at the null provider, so the numbers priced the seam and nothing else. Point it at the two real implementations. BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a provider that stores nothing and wrong against one that does, so qemunbdd and viperblockd are both measured at 64 MiB. liveProvider splits into a testing.TB flavour. Benchmarks need the same connection and the same node and cannot be handed a *testing.T. * fix(viperblockd): carry the trace context into every viperblock engine call Both S3 clients only emit a span when the request already carries one, so calling the no-context Init, LoadState, SaveState, LoadBlockState and Close left the whole engine open untraceable. Every provider verb traced as one flat server span, and the 342ms describe measured on env19 could not be attributed to anything. viperblock already exposes the Ctx variants and Backend already declares InitCtx, so this is threading, not new instrumentation. openVolumeVB, openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine, copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and mount/unmount keep the context their own span derives instead of dropping it. completeCreateSnapshot gets WithoutCancel: the background half of accept-then-publish has to outlive the request it was accepted by and still be attributable to it. The per-volume ebs.config handler opens a consumer span like the other legacy subjects, so a config update is no longer the one path with no trace at all. * perf: describe a volume without opening an engine on it describeVolumeEngine fell through to openVolumeVB for any volume not mounted on the node serving the request, which constructed a full viperblock engine to read one integer. On env19 that decomposed as a 22 ms reachability list, a 3 ms state read and a 16 ms PutObject of the volume's own config.json — the write coming from the SeqNum bootstrap a read-write open performs. Use viperblock's read-only state load instead. A describe is now the 3 ms read: no background goroutines, no reachability probe, no SeqNum window claimed, and no write issued by a worker that does not own the volume. Falsified: restoring the openVolumeVB call fails TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the list, with the fake backend answering each so neither assertion hides behind the other. * perf: share one HTTP client across every viperblock backend Four sites built the same S3Config by hand, each leaving HTTPClient unset, so every engine open and state read got its own connection pool and paid its own TLS handshake. That was invisible while Backend.Init issued a whole-bucket list — the list paid for the connection and the state read rode it — and it became the dominant cost once describe was reduced to a single request. Route all four through cfg.volumeS3Config, which builds the client once. Falsified: dropping the HTTPClient field takes TestDescribeVolume_ReusesOneConnection from one connection to three. * perf: wait for nbdkit to be ready instead of waiting a fixed second mountVolume spawned nbdkit and then ran a confirm loop until a one-second deadline passed. The loop only broke early when the process exited, so the fast-failure path worked and the success path always cost the full second. That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so the timings never contradicted the assumption that it was process spawn. Wait for the endpoint to accept a connection instead. That is also the question the caller needs answered, since it is about to hand the URI to something that will connect to it — where 'still alive after a second' only correlates with it. A nbdkit that exits during startup still fails the mount immediately, and one that never comes up now fails on a deadline rather than being assumed healthy. Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness timing assertion and the fast-failure one. * test: measure the data path, not just the control verbs The conformance benchmarks price create, publish and describe. Guest I/O never crosses those verbs, so none of those numbers say anything about what a guest experiences. This drives the published export with qemu-img bench, which sits on the same block layer virtio-blk does. It asserts only that the I/O completes and logs a table: what a given store is capable of is not a contract, and pinning a number here would only produce a flaky test. Read profiles stay inside the region the write profiles covered, or they would be timing how fast the export answers with zeroes, and one write profile flushes, because an unflushed write measures how fast the WAL accepts data rather than how fast the store takes it. Profiles are spaced apart because the plugin tears an engine down per disconnect and removes the volume's local directory as it goes, so a connection opened while the previous one is closing fails its handshake. * test: run data-path profiles back to back The spacing was a workaround for a volume lock that did not hold across close. With that fixed, consecutive profiles are worth more than the workaround was: each is its own NBD connection, so a run that completes has also shown that an arriving connection waits out the previous one's close rather than racing it. * fix: refuse a delete that cannot establish whether a volume is published volumeMountedElsewhere returned false on every failure path, including a request timeout, so a busy or partitioned owner was indistinguishable from no owner at all and the delete proceeded. Deleting a published volume leaves the nbdkit export and the guest writing through it alive over storage that is going away. Report the three outcomes separately. NATS distinguishes no-responders from no-answer, so only the former means unmounted; an unanswered probe is refused and the caller may retry once the owner is reachable. * feat: exclude a second viperblock engine with a JetStream volume lease Nothing in the stack stopped two viperblock engines opening one volume from different nodes. For an encrypted volume that is not just a consistency risk: the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte, and the SeqNum window claim is a read-then-write with no CAS, so two engines opening concurrently issue overlapping nonces under the same key. Every engine open now takes a lease first, keyed by volume in a JetStream KV bucket. A lease is a KV create, which JetStream serialises, so a second claimant loses instead of both winning. The revision that won is the generation: monotonic across the bucket, so a later opener always carries a higher one than the writer it displaced. Entries carry a TTL and a renewal loop, so a node that dies mid-mount does not wedge its volumes forever. Losing a lease is logged, not enforced. Until the object store can refuse a stale writer's PUT, failing live guest I/O on lease loss would add a failure mode without stopping the zombie it is aimed at. A daemon with no lease store refuses to open rather than opening blind, so nbdkit recovery binds the store before it adopts surviving exports. Repeat acquisitions on one node share a lease: the unmount seal opens a detached engine while the mount being torn down still holds it, and cross-node exclusion is what the lease is for. A restart reclaims this node's own stale entries, since the exports that outlived the daemon are still running. * refactor: consume masterkey from bluebottle in the test this branch added * feat(ebsprovider): classify refusals a caller may retry Every ErrorCode said what went wrong and none said whether the same request could succeed later, so a caller could not tell a permanent refusal from a transient one. Transient conditions shipped as internal, which tells a caller to give up when it should retry. Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(), which callers branch on instead of on the code itself. DeleteVolume's "could not establish whether the volume is published" refusal moves off internal onto it: the owner that did not answer may answer later. TestErrorCodeRetryable names every code, so adding one without deciding its retry meaning fails rather than silently defaulting to permanent. TestClassifyError pins the sentinel-to-wire-code mapping, which had no coverage at all; a sentinel missing from classifyError falls through to internal and loses both the reason and the retry signal. * fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for Four providers advertised the capability and nothing consumed it: there is no clone verb on the interface. CreateVolume's SourceSnapshotID restores from a snapshot and CopySnapshot duplicates snapshot metadata; neither is a volume clone, and the field documenting SourceSnapshotID says so explicitly. CSI expresses clone as a volume_content_source with a volume source, which this contract does not implement. Advertising it told a caller a feature was available and then gave them no way to use it. Removed rather than backed by a method: adding a clone verb is a feature, not a contract fix, and the capability can come back alongside it. The Capabilities doc comment now states the rule, and qemunbdd's stale comment about sparse extents is corrected while there. * feat(ebsprovider): add ListSnapshots so snapshots are reconcilable The contract could create, copy and delete snapshots but never say which ones it held. That left the control-plane metadata document as the only index of them, so losing one stranded the data it pointed at with no way to find it again. This is the same exposure ListVolumes closed for volumes, and it stayed invisible because ListSnapshots is optional in CSI too. ListSnapshots pages exactly as ListVolumes does, gated on a new SnapshotEnumeration capability. SnapshotRef is deliberately not a Snapshot: enumeration answers what exists, and reading a snapshot's size or state can cost far more than listing it. viperblockd walks the bucket's snap- prefixes, sharing the top-level walk with ListVolumes. It leaves SourceVolumeID empty because the link lives inside the snapshot's own encrypted state and reading it would mean a fetch per snapshot listed. qemunbdd walks its snapshots directory rather than its in-memory map, so a snapshot that outlived the process still appears; its SourceVolumeID is empty in that case for the same reason CreateSnapshot refuses to reuse the ID. Five conformance subtests cover it, and they run green against MemoryProvider, qemunbdd and viperblockd over a real predastore. * feat(ebsprovider): state the exclusion guarantee in the contract A caller could not tell which providers prevent two writers from holding one volume, at what scope, or for how long. The guarantee existed in viperblockd and was invisible from the outside, which is why the conformance suite had no concurrency dimension at all. Capabilities.Exclusion states it, and deliberately separates three questions the contract used to run together: how far exclusion reaches (none, node, cluster), whether a dead owner's claim ever expires, and whether a writer that lost its claim is actually stopped. That last one matters most and is the least flattering: exclusion refuses the second opener, fencing stops the first. viperblockd advertises cluster scope with a 45s TTL and FencesLostClaim false, because a partitioned node whose lease lapsed keeps writing until the object store can refuse it. Four new conformance subtests. Two required a second viperblockd node in the harness: cluster exclusion cannot be exercised by one node, because the second publish goes to a subject nobody serves and times out rather than being refused. Both nodes share the NATS the leases live in and the bucket the volumes live in, so a refusal comes from the lease. Standing that second node up immediately found a real gap, in a test that had been silently skipping for want of one. A publish refused by another node's lease came back as internal rather than volume_in_use, so a caller was told something had broken when the answer was that someone else held the volume. Mapped in handlePublishVolume. * build: pin viperblock to the release carrying the provider's dependencies The branch has been building against a local replace pointing at a viperblock worktree, so its committed pin never actually built: it named a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch, NewStateReader or s3.NewHTTPClient, all of which this branch calls. Points at the merged viperblock dev instead and drops the replace. GOWORK=off now builds clean, which is the deploy path ansible uses. * fix: let viperblockd's delete sweep tolerate an object another sweep removed deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it returned on any DeleteObject error. Two overlapping deletes for one volume left the loser reporting an internal failure once the winner had removed a chunk between the list and the delete. The control-plane copy that carried this fix is gone with the embedded engine path, so the provider path reintroduced the bug it had just fixed. * fix: stop CreateVolume leaving local state on whichever node served it A state read prefers this node's local copy over the backend, and provider RPCs are not owner-routed, so the node that happened to serve a create kept a copy that answered describe as if the volume still existed after another node deleted it. The two-node conformance harness caught it as DeleteVolume/success failing about one run in three. The seeded path already removed its local files after closing. The unseeded path saved state and left them, which was the only difference between the two. * fix: guard the mount path itself against a second export of one volume handlePublishVolume refused a volume this node already exported, but mountVolume did not, and the legacy ebs.<node>.mount subject is the route production takes for instance launch and hot-attach. A retried attach racing a fresh one started two real nbdkit against one volume, with no refusal and no warning: the volume lease only refuses a different node, and a same-node re-acquire finds the volume already held and increments its refcount. That also made the advertised exclusion inaccurate. viperblockd claims cluster-scoped exclusion on the grounds that a second opener is refused, while the conformance subtests reach it only through PublishVolume, so they passed with the legacy route unguarded. Moving the check into mountVolume gives both entry points the same behaviour and makes the claim true for the path production uses. * fix: tolerate a missing legacy AMI config on deregister DeregisterImage deletes the AMI document, then the pre-provider {imageID}/config.json. Predastore answers a delete of an absent key with NoSuchKey, and that was treated as fatal, so every AMI created since the metadata store landed failed to deregister after its document was already gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had just been told it could not delete. The regression test uses a store that answers a missing-key delete the way Predastore does; MemoryObjectStore silently succeeds, which is why the unit suite stayed green. * fix: reap ENIs left behind by an abandoned instance launch A launch creates the instance's ENI and then attaches it. When the daemon dies in between, the record survives with no instance and nothing ever looks at it again. The security group dependency check counts an ENI that lists the group, so the group cannot be deleted for as long as the record exists. The in-process launch path already rolls back on error; a process that dies cannot. This adds a cluster-wide sweep over records that are launch-created, never attached, and older than the launch RPC timeout by a wide margin. The description prefix is now a shared constant because it is the only thing separating an abandoned launch from a standalone ENI a caller asked for.	2026-08-12 16:54:19 +10:00
Julian Sommer	8f930e6275	refactor: consume the shared packages from bluebottle ... Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather than predastore, so the control plane no longer depends on the S3 server for shared auth and policy code.	2026-08-12 10:50:48 +10:00
brynmailer-mdc	a7a62addeb	test: rebuild the predastore fixture and service tests on the facade ... The fixture wrote a config the strict decoder rejects and built its cluster through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and serves under Run. Loading the real file means the fixture trips over the same validation an install does. Its data directory holds node-<id> directly, and it drains on cancellation rather than being abandoned at exit. The runtime tests drove buildRuntime, serve and the old on-disk layout, all of which predastore owns now, so they go. What replaces them covers what spinifex still decides: the flag merge and its precedence, a host id or gate node the config does not have, and the pid file behind Stop and Status. Two fixes to earlier commits in this series: the host id conversion moves behind its own guard, and the node status probes use WaitGroup.Go.	2026-08-11 17:23:42 +10:00
Josh Siviour	0a1d837f40	feat(bedrock): gate Ochre models behind per-account access grants (#741) ... * feat(bedrock): gate Ochre models behind per-account access grants Self-host models were invokable by any authenticated account: tieredCatalog returned them unconditionally and no runtime router checked the caller. GetFoundationModel also described models the caller could not use. Add a bedrock-model-access JetStream KV bucket holding presence-only grants, resolved through a new AccessResolver. Access is deny-by-default: a gateway with no store configured falls back to DenyAllAccessResolver, so an unconfigured deployment serves nothing rather than everything. The system account bypasses grants, matching how quotas exempt it. All four runtime routers share one grantedCatalogEntry gate so they cannot drift on who may invoke what. List and describe agree: an ungranted model is absent from ListFoundationModels and reported as ResourceNotFoundException by GetFoundationModel, so the error cannot confirm a model exists. Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on the spinifex gateway service for the UI, and `spx admin ochre access` for operators and provisioning, which opens the bucket over NATS directly rather than requiring a SigV4 client the CLI does not have. * feat(bedrock): seed the admin account's model grants at gateway start Deny-by-default left a fresh install with a catalog nobody could see, and the first cut filled that gap from ansible — which is a dev tool, so any install that never ran it came up unusable. Seed from the product instead: on start awsgw grants the platform admin account the full catalog, once, behind a marker key in the same bucket. The marker makes it a bootstrap step rather than a policy: tenant accounts still begin with no access, and an operator who revokes a grant does not have it restored by the next restart. Grants are written before the marker so a crash midway repeats the idempotent grants rather than leaving a half-seeded account, and the marker is a conditional create so only the first node to reach it need win. Seeding is best-effort: a KV failure logs and leaves the gateway serving, and the next start retries because the marker is written only on success. * test(bedrock): cover the model-access CLI and gateway admin actions Route the access CLI runners through the same loadConfigAndConnectFn / ochreExit seam the weights runners use, so grant, revoke and list can be driven against an embedded JetStream server, and add gateway tests for the grant admin actions including the non-admin denial.	2026-08-06 16:17:10 +10:00
Julian Sommer	1c4d41bffc	fix: reach the DB endpoint from any subnet of the VPC (#738)	2026-08-06 13:56:03 +10:00
Julian Sommer	8c07f80032	chore: remove the half line breaks	2026-08-06 11:10:15 +10:00
Josh Siviour	688ce5ccfd	feat: add AWS model conformance coverage (#736)	2026-08-05 18:46:37 +10:00
Julian Sommer	2a79bc8473	fix: record RDS instance creation events (#734)	2026-08-05 15:17:06 +10:00
Julian Sommer	f7d5d97e3f	fix: propagate RDS parameter group edits (#733)	2026-08-05 15:08:14 +10:00
Julian Sommer	68671cdf19	fix: prevent parameter groups bricking RDS instances (#731)	2026-08-05 14:16:31 +10:00
Julian Sommer	f240c33778	fix: wait for postgres to be up before running tofu workbook e2e	2026-08-05 11:48:58 +10:00
Julian Sommer	710169e5a6	fix: eliminate RDS Terraform drift (#729)	2026-08-05 10:34:51 +10:00