276 Commits

Author	SHA1	Message	Date
Josh Siviour	f7a2d85691	fix(e2e): stop flagging a Cluster bind to a single-homed node's own address (#789) ... checkSocketScope compared a socket's bind address to node.Addr and called it the node's WAN address, but node.Addr is the cluster peer address (harness.Node's own doc). On CI's single-homed tofu nodes this is the node's only address, so every Cluster-scope service failed no matter what it bound. Compare against a positively-known SPINIFEX_WAN_IP instead, distinct from the peer address, and record via harness.Detail when no such address is available so the skipped check is visible rather than silently passing.	2026-08-13 15:19:23 +10:00
Josh Siviour	b93f6c258a	fix(e2e): resolve the listener inventory without a Go toolchain on the target node (#787) ... * fix(e2e): resolve the listener inventory without a Go toolchain on the target node tests/e2e/multinode/listener_invariant_test.go resolved docs/security/network-connections/README.md at runtime via `go env GOMOD`, which works under `go test` but not inside a compiled .test binary shipped to a node with no Go toolchain. Embed the doc in a new docs/security/network-connections package instead, so the content is compiled into the binary and every caller parses bytes rather than a resolved path. RepoRoot and DocPath are now unused and removed. * docs: trim the embed package comment to the three-line standard	2026-08-13 14:18:02 +10:00
Josh Siviour	7d99705eff	fix(e2e): kill the node-failure phase abruptly instead of draining it (#785) ... * fix(e2e): kill the node-failure phase abruptly instead of draining it Phase 8 used `systemctl stop spinifex.target`, which now drains guests gracefully since the target-stop drain fix landed, so it tested an orderly shutdown rather than a failure. Since a target unit has no process of its own and cannot be signalled, and PartOf= only propagates stop/restart jobs rather than kill, SIGKILL each core service's main process directly instead, leaving orphaned guest qemu/nbdkit processes for recovery to reattach. Also fix a stale comment on spinifex-predastore.service's TimeoutStopSec: it claimed to match viperblock's drain budget, which is now 150s rather than 30s. The value was already correct; only the rationale was outdated. * fix(e2e): hold node2's outage open through Phase 8's assertions Restart=on-failure was respawning each killed unit within ~5s, so every Phase 8 assertion ran against an already-recovered node2 and DEGRADED_PEERS never actually dropped. Stop the same units right after the kill to cancel the queued restart and keep them down for the rest of the phase, and make the NATS degraded-peer count a real assertion (sized off NODE_COUNT) now that it can genuinely fail.	2026-08-13 12:30:15 +10:00
Josh Siviour	57bfbc20df	test: listener invariant static guard + e2e runtime check (#782) ... * test: add the listener invariant static guard and e2e runtime check Adds both halves the plan called for: a static test in spinifex/network/invariants that scans install scripts and config templates for a bare OVSDB ptcp remote, a wildcard bind on a Cluster/Encap port, or a bind port absent from the inventory, and an e2e test in tests/e2e/multinode that asserts the same on live ss -tulnp output. Both read docs/security/network-connections/README.md "## 1. Inbound Listeners" as their fixture through a shared, non-test parser package (spinifex/network/listenerinventory), so a wildcard exception has to be declared in the doc's own prose rather than hardcoded in either test. * fix: make the listener wildcard exception fail closed on an explicit marker The prior heuristic granted an exception for any unnegated "wildcard" or "0.0.0.0" mention within 40 characters of the port's row. That fails open: a future row phrased as "avoids the wildcard" or with a negation further upstream than the window would silently exempt itself. Replace it with a single exact literal phrase, "binds the wildcard by design", that a row must contain to grant the exception — no other wording counts, so an unrecognized mention denies rather than grants. Update the two rows that need it (5300, 500/4500) and document the marker's role in the inventory's own prose. * docs: record that the Geneve tunnel socket binds the wildcard by design	2026-08-13 12:29:26 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Josh Siviour	fd8be8a814	refactor(test): route duplicated NATS bootstraps through testutil (#776) ... * refactor(test): route duplicated NATS bootstraps through testutil Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that spinifex/testutil already provides, and one seedKV clone duplicated testutil.SeedKV. Route them through the shared helpers. Sites whose options genuinely diverge are left alone: fixed ports needed for reconnect and restart scenarios, custom TLS and token auth, a custom MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup. The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than CreateKeyValue, so their failure behaviour differs and they stay as they are. Also moves eight test files to external test packages where they declare no helper used by a sibling in the same package. Coverage is unaffected: Go compiles internal and external test packages into one binary per package, so attribution is identical either way. * chore(test): require new test files to use an external test package Adds a preflight check that fails when a newly added test file declares an in-package test. External test packages bind tests to the exported API and let packages share test doubles, which an in-package double cannot be. Only added files are checked, so the existing in-package tests are untouched. e2e and integration suites are exempt: both share a package-level fixture built in main_test.go, so their files cannot move individually. A file that needs unexported access can opt out with a //test:in-package comment.	2026-08-12 22:38:15 +10:00
Josh Siviour	9427cdfc2d	fix(test): give each predastore fixture its own S3 gate port (#775) ... The fixture pinned its S3 gate to 127.0.0.1:18443. Three test binaries import it and go test runs packages as concurrent processes, so two clusters raced for the port. The loser did not fail cleanly: waitForReady probed the winner's gate on the same port and reported healthy, so the setup client talked to another process's cluster and failed with BucketAlreadyOwnedByYou. Reserve a port per process instead, and reach the gate only through Fixture.Host so it cannot be hardcoded again. A Start that fails partway now drains the cluster before returning; its nodes hold process-wide pipe names that a later attempt would otherwise collide with.	2026-08-12 22:37:54 +10:00
Josh Siviour	91d22d9f12	feat(security): add an nftables host firewall scoping cluster ports to peers (#769) ... * feat(security): add an nftables host firewall scoping cluster ports to peers Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the formation port, NATS, predastore — was reachable from anything that could route to the node. On a single-NIC node the planes collapse onto the public address, which made that the default rather than the exception. setup.sh installs a policy and an apply helper but does not enable them: the peer set is unknown until the cluster's planes resolve, and a drop policy with an empty set would cut the first node that applied it out of its own formation. install-node.sh writes the peers and enables the unit after formation. The policy owns only table inet spinifex_filter and replaces it in one transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip nat/filter tables and reinstalls them only on start, so a whole-ruleset flush — what Debian's nftables.service does on both start and stop — would break elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here could not be rescued by an accept in the other table. Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the host netns, and every guest agent reaches its control plane over the AWS gateway rather than NATS, so no cluster port needs to face the guests. * feat: reconcile the host firewall peer set from the daemon The peer set was written by install-node.sh and by cluster-bootstrap.yml. That missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot), duplicated the same logic in each, and went stale as soon as a node was added or replaced. host.ReconcileFirewall now owns it: every formation path starts the daemon, and it re-runs on each startup. A node's own config names its peers by their wan address only, so a set built from config alone drops every peer's NATS, OVSDB and formation traffic. The lan plane comes from the generated NATS cluster routes and the OVN remote lists, and the vpc plane from the Southbound Encap table; an empty encap answer is an error rather than a narrower set. ProtectSystem=full applies to the sudo child too, so the root helper could not write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths, optional so a node installed before the firewall stage still starts. The apply helper gains set-peers and disable verbs, re-validates every address as a dotted quad with in-range octets, and rolls the peer file back if the ruleset will not load. * fix: retry the host firewall reconcile until every chassis registers The peer set is only safe to write once OVN reports an encap address for every node. On a fresh bootstrap the daemon routinely wins the race against ovn-controller, so a single startup attempt saw a partial chassis list and either wrote a set that dropped Geneve from the nodes missing from it, or failed outright and left the node with no policy until a restart. Gate on the full chassis count and schedule the reconcile instead of running it once: back off from 15s to 5m while it fails, then re-run every 5m so a node added or replaced after formation is picked up without a daemon restart. An unchanged peer set short-circuits before the helper runs. * test: cover the firewall chassis quorum gate in the multinode e2e suite A partial chassis list is the normal state early in a bootstrap, and writing a peer set from one produces a firewall that drops Geneve from every node missing from it. The gate that refuses that write had no live coverage: a 5-node bootstrap converges too fast for any daemon to observe a partial list, so the verification run only ever exercised the OVN-unreachable path. Reach the branch by inflating the expected count instead of breaking OVN: a phantom node in one node's config makes the cluster one chassis short, with nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer file is written while it holds, that the loaded ruleset survives, and that the node recovers to a byte-identical peer set once the phantom is removed.	2026-08-12 22:37:27 +10:00
Josh Siviour	98ec781629	feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770) ... * feat: introduce the EBSProvider interface and control-plane metadata store Phase 1 of decoupling the control plane from viperblock. Adds a provider-neutral block-storage contract that spinifex owns end to end. ebsprovider defines EBSProvider, whose operation set and idempotency rules follow the CSI controller service, with NATS request/reply as the transport rather than CSI's gRPC. Every request and response carries an explicit schema version so wire skew fails loudly at the boundary instead of silently decoding a changed payload. The package deliberately exposes no viperblock types: provider-specific state is an opaque handle. Ships with a NATS client against the ebs.provider.v1.* subjects and an in-memory fake for tests. ebsmetadata gives spinifex its own volume and AMI documents under a spinifex key prefix, so the control plane can answer DescribeVolumes and DescribeImages without reading the storage engine's persisted state. The EC2 volume and snapshot handlers gain provider-backed paths for create, describe, modify, attach-state and delete. Provider injection is optional for now, so the existing engine path stays the default until the viperblockd side serves the new subjects and the metadata migration lands. vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol it used and is now a local helper. * feat: serve the EBS provider contract from viperblockd Gives the ebsprovider NATS client a server. Volume create, describe, expand and delete, plus snapshot create and delete, are now handled by the daemon that owns BaseDir and the mounted-volume registry, which is where viperblock engine construction belongs. Every handler prefers an already-mounted engine over opening a second one on the same volume, and expand refuses outright while the volume is mounted. Opening a second engine on a live volume is the double-writer hazard this decoupling exists to remove, so the boundary enforces it rather than trusting callers. Requests carrying an unrecognised schema version are rejected at the boundary instead of being decoded on a best-effort basis. Snapshot creation replies immediately with an operation ID and completes asynchronously on a per-snapshot subject, since a large volume will outrun the request timeout. Draining the volume stays with the caller: it coordinates with the guest and is not something the storage boundary can reach. The snapshot create subject gains an explicit create verb. Addressing it as ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject and left the wildcard unsubscribable. Publish and unpublish are not served yet; the mount and unmount logic they would front is still inline in launchService and needs extracting first. Also adds a conformance suite that runs one set of contract assertions against both the in-memory fake and the NATS client, so each side is checked against the contract rather than against the other's structs. * feat: let an operator select the EBS provider Adds an [ebs] provider key and wires the NATS provider into the EC2 volume, snapshot, image and instance services at the daemon's composition root. Until now nothing injected a provider, so the provider-backed code paths in those services were unreachable. The key is provider-neutral rather than nested under [viperblock], because it names the boundary rather than one provider's settings; a second provider will not need it renamed. An empty value resolves to "embedded", so deployed config files that predate the key keep today's behaviour, and an unrecognised value is a startup error rather than a silent fallback. Startup probes the provider's capability verb and warns when nothing answers. The daemon and viperblockd start independently, so an unanswered probe is not fatal, but without it a misconfigured node fails much later as an unexplained request timeout on the first EBS call. "viperblockd" stays experimental: under it volume metadata lives in the control plane's own document, and the leak reaper, GC and admin paths still read viperblock's config.json, so they cannot see tenant or tag data for volumes created that way. * fix: make every volume-metadata consumer provider-aware Several readers and writers of volume metadata were never given a provider branch, so under the viperblockd provider they silently used the wrong store. Each of these is a correctness bug rather than a missing feature, because the control plane had already stopped writing the document they read. Modifications had nowhere to live: ModifyVolume built its result in memory and dropped it, and DescribeVolumesModifications read it back out of viperblock's config.json, so a describe straight after a successful modify returned nothing. Volume metadata now carries its own modification record. The leak reaper read attachment and tags from config.json and wrote orphan markers to tags.json, neither of which exists for a provider-managed volume, so a leaked volume was invisible to it and any mark it did write could not be read back. Tag mutations had the same split: a tag applied after creation never reached the document DescribeVolumes reads. Snapshot deletion could not see a provider-managed clone, because a clone records its source only in the control-plane document. That let a delete strip chunks a live clone still reads, so it is fixed here rather than left for the migration. Volume enumeration and DescribeVolumeStatus were likewise scanning object prefixes that a provider-managed volume does not create. Also adds an encryption flag to the volume document, which DescribeVolumes had been reporting as false unconditionally and snapshots could not carry at all, and a list operation for AMIs so the image path can follow later. The legacy path is unchanged; its invariant tests pass unmodified. * feat: backfill ebsmetadata documents from legacy volume and AMI state DescribeVolumes under the viperblockd provider enumerates only ebsmetadata documents, so a volume created before the switch had no document and was invisible rather than merely slow to read. A per-object read fallback cannot fix enumeration, so both a backfill and a fallback are needed. Add an object-store migration kind to the migrate framework. Predastore has no conditional write, so an object-based version stamp would be last-writer-wins across nodes; RunObject stamps progress in a caller-supplied JetStream KV bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a read-then-write rather than a compare-and-swap, so correctness rests on every ObjectMigration.Run being safe to re-execute. The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the numbered config migrations: it needs viperblock's VBState schema to decode the legacy config.json, and migrate is imported by every handler that stamps a KV bucket version. It converts what it can reach and skips unreadable objects with a log, so one corrupt volume cannot hide the rest of the fleet. ebsmetadata gains the legacy read fallback for the window before the migration has run everywhere. It stays free of viperblock by taking the decode as an injected function value, wired at the composition root, and the fallback is retired by leaving it unset. A fallback read never writes the converted value back, which would race the migration. Prefix enumeration pages to exhaustion. Stopping at the first page would let the backfill stamp itself complete having converted only the volumes it happened to see, reintroducing the invisible-volume failure it exists to prevent. The migration runs from configureEBSProvider, so only when the provider is selected; backfilling while the embedded engine is still authoritative would produce documents that immediately go stale. Failure is fatal rather than advisory, unlike the capability probe beside it. This makes the switch to viperblockd one-way, which is now recorded on the config field. * feat: serve volume publish and unpublish from viperblockd PublishVolume and UnpublishVolume were the last unserved verbs on the provider contract, and they are the ones that attach a volume to a VM. Until they are served the provider path cannot boot an instance, so provider = "viperblockd" cannot be verified end to end. The mount and unmount logic was two large anonymous handler bodies inline in launchService. Rather than write a second mount path, extract each into a named function and have both the legacy ebs.mount/ebs.unmount handlers and the new provider handlers call it. Two divergent implementations of nbdkit spawning and the seal decision would be far worse than the inline bodies. The extraction relocates code without rewriting it; response publishing stays in each handler so the provider never publishes to ebs.mount.response. Publish is idempotent: a volume this node already has mounted returns its existing attachment rather than starting a second nbdkit against it, which is the double-writer hazard the boundary exists to prevent. Unpublish preserves the invariant that a failed seal leaves the volume mounted so a retry can re-attempt it, and reports an error rather than success. Both verbs are node-addressed, so they subscribe directly rather than through the spinifex-workers queue group, and are registered only when NodeName is set. There is no queue-group fallback: a subject that already names one node has nothing sensible to fall back to. Read-only attachment is refused rather than silently granted read-write, since nbdkit is started read-write or not at all. Also fixes a latent panic carried by the extracted mount path: viperblock.New returns a nil VB alongside its error, and SetCacheSize was called on it before the error was checked, so a failing New would have taken down the handler. * feat: read and write AMI metadata through ebsmetadata The backfill migration wrote ebsmetadata AMI documents that nothing read: every AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at {imageID}/config.json. Give the image service provider branches for the AMI read, write and enumerate paths, with enumeration going through ListAMIs rather than an object-prefix scan, as DescribeVolumes already does. AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a viperblock type on an interface boundary in the instance package. It now returns ebsmetadata.AMI on both paths, the embedded one converting before it returns, so the boundary names no viperblock type whichever provider is selected. The legacy-to-document mapping had accumulated three copies. Consolidate them in ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata itself stays free of viperblock, and cover it with a reflective test that fails when a field is added to either schema without being wired through. Sixteen fields copied by hand in three places would have dropped one silently. The admin image paths read whichever representation exists and write back to that same one, so a provider-managed AMI never gains a spurious config.json. FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan: missing a dependent there deletes an AMI a live volume still needs, the same class of bug as the snapshot clone path fixed earlier on this branch. That union is built by hand rather than by wiring Store.ListVolumes' fallback, which swallows per-ID read errors. This path deliberately fails closed on a transport error so a delete cannot proceed on a partial picture, and the fallback would have quietly turned that into failing open. The CLI image import now also writes the document, best-effort, so an AMI imported after the migration is not left reachable only through the fallback. * fix(ec2): resolve volume config through ebsmetadata when a provider is configured A provider-created volume has no vol-*/config.json — its metadata lives in the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy key unconditionally, so AttachVolume reported a volume that DescribeVolumes had just listed as available as InvalidVolume.NotFound. Routing through ebsmetadata.Store also puts this path behind the legacy fallback, so a volume created before the provider switch stays attachable. The NoSuchKey remap now names the key it missed; its silence made the original failure invisible in the daemon logs on every node. * test(bench): add an EC2 control-plane latency benchmark Measures the EBS control-plane operations either side of the provider boundary, so the cost of the NATS hop can be stated rather than guessed. Reports median and p95 with a bootstrap CI on the median difference, and labels a delta indicative unless that CI excludes zero. * fix(ec2): drain the volume before a provider snapshot The legacy CreateSnapshot branch drains first so the snapshot reads a current checkpoint; the provider branch did not, and viperblockd documents the drain as the caller's responsibility. Both sides deferred to the other, so an attached volume snapshotted stale data with no error raised. DrainVolume and volumeAttachment now take the fallback state and instance as strings rather than a viperblock.VolumeMetadata, removing a viperblock type from an exported signature. * feat(ec2): route the instance root volume through the EBS provider RunInstances built its own viperblock engine to clone the AMI snapshot into the root volume, which made every launch a second writer against a volume viperblockd already serves. The launch now calls the provider's CreateVolume with the AMI's snapshot and source volume, and records the result in ebsmetadata so DescribeVolumes and attach can see it. A snapshot clone needs both the snapshot ID and its source volume ID to resolve base blocks, so CreateVolume in the volume handler now sends SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected every create-volume --snapshot-id outright. The gp3 IOPS and throughput constants move to spinifex/types so the instance and volume handlers share one definition rather than duplicating literals. * feat(ebsprovider): add a bounded volume seed to CreateVolume The EFI variable store must be initialised from the launching node's firmware VARS template, byte for byte. Passing a host path would not work: CreateVolume is queue-subscribed, so any node can service it, and a foreign node's OVMF package would bind the guest's pflash region to the wrong firmware build. CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd writes them at offset 0 in the engine it already builds. The seed applies only when the create actually allocates the volume, so a relaunch never overwrites a BootOrder the guest has since written. JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes bounds the seed well inside the NATS max_payload the cluster runs with and both ends reject an oversized seed with a clear error rather than letting the publish fail. VolumeSeeding advertises the behaviour so a provider that cannot seed says so instead of silently dropping the bytes. * feat(ec2): route the EFI variable store through the EBS provider prepareEFIVolume built a control-plane viperblock engine to seed the variable store from the firmware VARS template, making every UEFI launch a second writer against a volume viperblockd already serves. It now calls the provider's CreateVolume with the template as seed bytes, so the store is allocated and written by the daemon that owns the volume. The template is still read on the launching node and shipped with the request. CreateVolume is queue-subscribed, so any node can service it, and a VARS region seeded from a foreign node's OVMF build would bind the guest's pflash to the wrong firmware. pflash refuses a VARS region that is not byte-exact, so the returned capacity is checked against the template length; a provider that rounded up to a GiB boundary would otherwise yield a guest that cannot boot. The engine path stays behind a nil-provider guard until the legacy branch goes. * feat(ec2): route image snapshots through the EBS provider snapshotRunningVolume and snapshotStoppedVolume each built a control-plane viperblock engine to take the snapshot backing a new AMI, making CreateImage a second writer against a volume viperblockd already serves. Both now call the provider's CreateSnapshot with the volume's provider handle. The running path keeps its drain, which is what makes the snapshot current rather than one predating a guest-triggered write. The stopped path needs no drain, and the daemon already prefers a live mounted engine over opening a second one, so it handles both cases itself. getVolumeConfig read the legacy config.json, which a provider-created volume does not have, so every CreateImage against one failed. It now reads the ebsmetadata document under a provider, the same way attach does. * fix(daemon): only the owning node creates an image for a stopped instance handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which only the node holding the VM passes, so a running instance is imaged once. A stopped instance lives in the cluster-replicated KV instead, which every node reads successfully, so all of them ran the full snapshot and AMI creation. The caller saw the first reply and the rest leaked: on a three-node cluster one create-image produced three AMIs and three snapshots. A stopped instance records its owner in LastNode, so a node that is not the owner now declines. Gather tallies error frames and keeps waiting, returning only on a success, so declining cannot preempt the owner's reply. An instance stopped by a build that never set LastNode elects an owner through the KV's compare-and-swap, since having every node decline would make create-image fail permanently for those records. * perf(ec2): serve DescribeVolumes by ID from the volume documents The provider branch called ListVolumes unconditionally and filtered in memory, so asking for one volume read and decoded every volume document in the bucket. The non-provider branch already had the equivalent fast path. Explicit IDs now fetch their documents directly. The tenant check moves with them: a document belonging to another account reports not-found rather than being returned, matching what the enumeration filter did. Measured against a memory object store, the fast path stays flat near 20us regardless of bucket size while the enumeration grows linearly, reaching 4.6ms at 500 volumes. * chore(lint): ban the viperblock import from the EC2 control plane The control plane reaches block storage through ebsprovider now, but nothing stopped a handler from importing the engine again. That is not hypothetical: the instance handler first imported viperblock for a config struct alone, and the engine construction followed thirteen days later. depguard denies the import across handlers/, which matters because viperblock.New starts a chunk uploader — every construction is a writer against a volume viperblockd already serves. A types-only import is denied too, since that is exactly how the last one started. The four files still carrying the pre-provider engine path behind a nil-provider guard are exempted by name. The exemption list is the remaining work: it shrinks to nothing when that path goes. * fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot CopySnapshot wrote a control-plane snapshot document under the new ID and nothing else. The copy described correctly and CreateVolume against it succeeded, so it looked like it worked, but the new ID had no viperblock snapshot metadata behind it and the volume failed when the guest attached it. Producing the metadata means re-sealing it under the destination ID. On an encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the sealing helpers are unexported, so spinifex cannot write it. Copying only the unencrypted form would be dead code on any cluster with encryption enabled. Refusing with UnsupportedOperation fails at the call the caller can act on rather than at attach time. Source lookup and the ownership check still run first, so a bad ID or another tenant's snapshot reports that instead. * test(ec2): give handler test fixtures their own legacy config.json shape Handler tests built the legacy config.json fixture by importing viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly. Mirror that shape in ebsmetadata/vblegacy, next to the production code that already reads it (ebsmetadatabackfill), and switch the fixture-only usages across the volume/snapshot/image handler tests to it. A byte-for-byte equivalence test against the real viperblock types guards the mirror from drifting out of sync. Call sites that hand a viperblock type into putVolumeConfig, mergeVolumeConfig, or buildVBConfig are untouched, since those functions still name the real type in their signature. * feat(ebsprovider): add CopySnapshot to the EBS provider contract Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy NATS subject, MemoryProvider and NATSProvider implementations, conformance coverage) and a viperblockd handler that duplicates a snapshot's metadata via viperblock.CopySnapshotMeta on the source volume's VB. * feat(ec2): serve CopySnapshot through the EBS provider CopySnapshot has been refusing with UnsupportedOperation because a copy needs its own snapshot metadata on the backend under the new ID, and on an encrypted volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the control plane cannot re-seal it. The provider contract now carries a copy operation, so the storage engine does it. The provider path generates a new snapshot ID, has the provider duplicate the backend metadata under it, then writes the control-plane document. Rollback mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot rather than leaving one nothing references. Ownership is still checked against the source before anything is created. The no-provider path keeps refusing. It has no way to produce the backend metadata, and the copy it used to write described fine and then failed at attach — a resource that fails as far as possible from the call that made it. Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot reached the provider rather than existing only as a control-plane document. That assertion is what catches the old behaviour. * refactor: remove the embedded EBS engine path from the EC2 control plane viperblockd is now the only EBS provider. The pre-provider path that constructed a viperblock engine in-process is deleted, and with it every github.com/mulgadc/viperblock import under spinifex/handlers/ (tests included). The depguard ec2-control-plane rule drops all four file exemptions and the !$test carve-out, so the rule now reads simply as "nothing under spinifex/handlers/** may import viperblock". viperblock.VolumeConfig was also a data carrier on the provider path. It is replaced by the spinifex-native ebsmetadata.Volume rather than the vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is decoding old objects, while ebsmetadata.Volume already carries every field the call sites read, so the conversion step disappears instead of being reimplemented against a legacy shape. The embedded-only helpers (volumeConfigWrapper, putVolumeConfig, mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached, deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json prefix scan) had exactly one production caller chain each, all through a nil-provider branch, and are removed. The snapshot-dependency guard on DeleteVolume is provider-agnostic and is kept, now running on the provider path where it previously did not. config: viperblockd is the default and an empty Provider resolves to it. An explicit provider = "embedded" is rejected at config load rather than silently upgraded, because starting anyway would migrate a node's volumes to ebsmetadata one-way without the operator asking. The generated spinifex.toml template emits viperblockd. * fix(images): write the imported AMI document from the config the import wrote ImportDiskImage creates the AMI's snapshot and records it on its own *viperblock.VB, which viperblock.New allocates fresh, so the value lands only in {imageID}/config.json. The import command was persisting the ebsmetadata document from its pre-import manifest copy, whose SnapshotID is still empty. Store.GetAMI only consults the legacy config.json fallback when the document is missing, so that empty-SnapshotID document permanently shadowed the real value and every launch from an imported AMI failed the zero-copy clone check with ServerInternal. Rebuild the document by reading back the config.json the import just wrote. A missing or unreadable config is skipped rather than written from the stale manifest: no document still resolves correctly through the legacy fallback. * test(e2e): read NODE from spx get vms instead of VCPU in the spread check spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to field 8. The spread placement check still read field 6, so every instance reported its vCPU count instead of its node, the unique-node count always collapsed to 1, and the check failed on clusters that had spread correctly. Read field 8 and strip the ANSI colour codes wrapping each cell. * feat(viperblockd): add a --debug flag so viperblock debug logging is reachable viperblockd hardcoded Info, and viperblock's VB.logger() falls back to slog.Default() because vb.log is only ever set by SetDebug. That left the engine's debug records, including the chunk uploader's start and disabled lines emitted during New(), unreachable in a deployed environment. Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG rather than the bare debug key, which several commands share. Also sets the Config.Debug field, which existed but was set only by tests. * fix(viperblockd): release the chunk uploader when openVolumeVB fails viperblock.New starts a chunk uploader and a WAL syncer before the backend is touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the caller while leaving both running with no handle to stop them. Repeated backend failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders on a 30s ticker for the life of the process. Adds invariant tests for both halves of the ownership rule: a mounted volume's snapshot create and copy reuse the live engine and log zero volume opens, an unmounted volume opens exactly one, and a failed open leaves no uploader behind. The open counts are read from the same log record a production alarm watches. * fix(ebsprovider): route requests to the node that owns the mount Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one spinifex-workers group, so NATS delivered each request to a random node. Only the node holding the mount can reuse the live engine through findMountedVolume; any other node constructed a second viperblock engine, with a live 30s chunk uploader, against a volume nbdkit was serving on a different host. Measured on a three-node cluster, this was roughly two snapshots in three. Rather than resolve the owner, let the owner answer. A node subscribes ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts a volume and drops it on unmount, mirroring the per-volume ebs.config subscription. Clients try that subject first and fall back to the queue group only on ErrNoResponders, which means nothing is mounted and opening an engine is correct. A timeout deliberately does not fall back, since the owner may still be running the operation. Applies to snapshot create and copy, expand and describe, the four verbs that reuse a mounted engine. Delete keeps queue dispatch. * fix: rebuild viperblockd mounted-volume registry after a restart Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a viperblockd restart while a VM keeps running leaves nbdkit serving a volume the daemon has forgotten. Owner routing then infers from ErrNoResponders that nothing is mounted and falls back to the queue group, which opens a second viperblock engine, with a live chunk uploader, on another node. The same empty registry also defeats handlePublishVolume's double-mount guard, so a republish could start a second nbdkit. Discover surviving nbdkit processes from /proc at startup and rebuild the registry from them, following orphan_scan.go's precedent: match comm exactly, parse the argv for volume=<id> and --unix/-p, and adopt a PID only when nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or killed; anything that fails corroboration is logged and skipped. Recovery runs before the first subscription so no handler can race it. Extract mountVolume's engine construction into constructMountedVB so mount and recovery cannot drift, and release the WAL syncer viperblock.New starts when construction fails after it. * fix: corroborate recovered nbdkit processes from argv, not a pidfile nbdkit is launched with -f (required for the Go plugin) and never writes the --pidfile it is given, so no such file exists on any node; the daemon's own /tmp is a PrivateTmp namespace besides. Corroborating against it rejected every genuine mount, making recovery a no-op in production. Corroborate against what the kernel actually has: the process must serve our plugin, name this daemon's base dir, and still hold its socket. Adopt a volume once, so two live nbdkits for it leave one registry entry and a warning rather than two conflicting ones. * feat: add a second EBS provider backed by qcow2 and qemu-nbd The decoupling was justified by swappability that nothing demonstrated: the conformance suite ran against MemoryProvider and against a NATSProvider wired to a MemoryProvider, so both arms were the same implementation behind different transports. A fake is not a second implementation. Add three pieces. natsserve serves the ebs.provider.v1.* contract from any EBSProvider, depending on nothing but the interface, and replaces the test-only server twin so conformance now exercises production code. The conformance suite moves to its own package and is exported so implementations outside ebsprovider can run it, with no assertion changed. qemunbdd is the second implementation: one qcow2 file per volume, qemu-nbd for publish, real snapshot copies instead of chunk references, and every exec behind a runner seam so its tests need no qemu. Owner-subject routing is deliberately not served neutrally: it needs a per-node mounted-volume registry the interface does not model, so it stays in viperblockd and a neutral server's clients correctly fall back to the queue group. * test: run the conformance suite against the qcow2 provider The suite passes unmodified against an implementation sharing no code with viperblock, both in-process and through natsserve over the real wire contract, which is the swappability claim the provider boundary was built to support. Two real defects it caught. ExpandVolume and GetVolume inspected the image before consulting the publication map, so qemu-img could not open a volume qemu-nbd held the write lock on and the refusal surfaced as an internal error instead of ErrVolumeInUse; qemu-img info now runs with --force-share, so describing a volume no longer depends on whether it is attached. Publish also failed with qemu-nbd's raw exit status when the socket path exceeded the 108-byte AF_UNIX limit, and now reports that reason itself. * feat: make the EBS provider selectable by config Add the qemunbd selector alongside viperblockd, a spx service command that serves the qcow2 provider over natsserve, and service registration. Config validation accepts the new value everywhere viperblockd is accepted, while embedded stays rejected by name. The daemon logged a hard-coded viperblockd constant as the active provider rather than the resolved one, in configureEBSProvider and in both of probeEBSProvider's lines; all three now report what is actually configured. Both daemons answer the same subjects, so exactly one may run on a cluster. * test: add a live lifecycle check for the qcow2 provider Drives create, publish, write, snapshot, restore and read-back entirely through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a cluster node, so the client under test is the same shape the control plane uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE is set, so it stays out of preflight. * fix: publish NBD URIs in the form standard clients accept PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever consume what the EBS provider boundary hands back. Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit all accept. ParseNBDURI now takes both that and the TCP form, and still accepts the legacy nbd:unix: syntax for state written by older daemons, so a conformant third-party provider's URI is no longer rejected on the way in either. Verified against qemu 10.0.11: qemu-img and a real -drive option string both open the URI form, and qemu reports both forms as nbd+unix://?socket=, so it already treats the URI as canonical. * test: validate published exports with libnbd's client tools Every other assertion about the EBS provider goes through our own Go client, so nothing has ever checked whether the export we publish is usable by an NBD client that knows nothing about this codebase. That is the layer neither the conformance suite nor a CSI-shaped contract can reach. RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume: the URI connects, the export advertises its size and block-size constraints, data written externally reads back byte for byte, a ReadOnly publish sets the read-only transmission flag, and UnpublishVolume tears the export down. It skips when libnbd-bin is absent rather than passing vacuously. Wired to qemunbdd, where all five failed before the URI fix. The publish assertion in the main suite now requires a dialable URI instead of merely a non-empty string, which is what let the wrong form through. * test: run the external NBD client suite against a live viperblockd The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit export is the comparison worth having: one external client, two providers it cannot tell apart, and every difference is a candidate leak in the boundary. RunNBDClientSuiteWithConfig takes the node name and volume prefix a live cluster needs, and publishForNBD now unpublishes and deletes what it created so a run against a real cluster leaves nothing behind. Subtests dial a normalised URI so a provider emitting the legacy syntax still gets its export tested rather than every subtest failing on the same string; whether the URI itself is standard is now its own subtest. Adds a reconnect check, since publication holds the volume rather than any one connection. It retries the real operation because a cheaper liveness probe is itself a connection and re-triggers whatever refused it. * feat: make the conformance suite capability-driven The suite asserted a fixed Capabilities literal, so it could only ever pass against a provider advertising exactly that set. Every implementation had to match the suite rather than the suite testing the contract. RunSuite now reads GetCapabilities from the provider under test and gates optional behaviour on what it advertises, and the fixed literal is demoted to ReferenceCapabilities, a convenience for callers constructing a MemoryProvider. Add ReadOnlyPublish to Capabilities. A provider that leaves it false must refuse a ReadOnly publish rather than hand back a writable export, which is what viperblockd already does; it now has a way to say so up front. Add ErrUnsupportedCapability and the unsupported_capability error code. A request for optional behaviour a provider does not advertise is well formed, so classifying it as invalid_argument told the caller to fix a request that was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly. Declared capabilities now match observed behaviour: qemunbdd advertises SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises neither: the nbdkit Go binding exposes no extents callback, and the mount path never passes -r. The external NBD client suite gains a sparse-extent subtest that checks the export's advertised contexts against the capability, so SparseExtentReporting is falsifiable from outside rather than taken on trust, and its read-only subtest now branches on the capability instead of assuming refusal. Cover the new branches with a MemoryProvider advertising no optional capabilities and one advertising OnlineExpansion, the branch no provider we ship exercises. * refactor: rename SourceVolumeID to SourceSnapshotVolumeID CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd required it whenever SourceSnapshotID was set and used it to resolve the snapshot's chunk references, so there it named the snapshot's origin volume. MemoryProvider ignored it entirely. qemunbdd implemented it as an independent volume-to-volume clone, which is what the name and doc comment implied. The conformance suite could not catch the divergence because the reference provider had no branch for the field at all. The contract already has a field meaning "the volume this snapshot came from": Snapshot.SourceVolumeID, which every provider populates. So the request field is the same fact travelling back the other way, and its name was the problem. Rename it to SourceSnapshotVolumeID, document it as the source snapshot's origin volume, and state that it is required with SourceSnapshotID and meaningless without it. All three providers now enforce one rule: a source snapshot must be accompanied by its origin volume, or the request is invalid_argument. MemoryProvider additionally rejects an origin that does not match the snapshot's own SourceVolumeID, so a wrong name fails instead of resolving blocks against a volume the snapshot never came from. qemunbdd loses its volume-to-volume clone branch, which implemented behaviour the contract no longer offers and no caller ever requested. The suite pins both the requirement and a full create-snapshot-restore round trip, so the field is exercised end to end rather than only on the error path. * feat: make the conformance suite runnable against a live provider The suite had only ever run against MemoryProvider and qemunbdd, both of which are discarded with the test binary. viperblockd, the provider production actually uses, was the one implementation the suite never judged, because the suite assumed each subtest got fresh state: hard-coded volume IDs collide with an earlier run's leftovers on a store that keeps them, and a hard-coded node ID addresses a node that does not exist. SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records every volume, snapshot and publication the suite creates and tears them down in dependency order, logging rather than failing so a teardown problem cannot mask the assertion a subtest was making. Two new tests cover this by running the whole suite twice against one shared provider instance, which passes only because cleanup works, and once per prefix to cover concurrent runs sharing a daemon. TestLive_ViperblockdConformance wires this to a real cluster behind SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite. * fix(viperblockd): close the four contract gaps the live conformance run found Unpublishing a volume this node has no mount for now succeeds instead of returning not_found, matching the legacy unmount path: nothing mounted here is already the state the request asks for. CreateSnapshot refuses a snapshot ID that already exists against a different source volume with already_exists, rather than overwriting it. DeleteVolume refuses a published volume with volume_in_use instead of tearing the mount down, on this node or on whichever node answers the volume's owner subject. The export and the guest writing through it survive the metadata deletion, so the old behaviour surfaced later as corruption. CreateVolume reads the source snapshot before writing the volume, returning not_found when it is absent and invalid_argument when it was taken from a different volume than the request names. Previously both produced a blank volume and a successful response. CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument now that it carries a sentinel. * feat(ebsprovider): pin the source snapshot's origin volume in the contract The suite only required source_snapshot_volume_id to be present, so a caller naming a volume the snapshot never came from was accepted everywhere. The snapshot records its own origin, so this is checkable: creating from a snapshot with a mismatched source volume is now invalid_argument for every provider, enforced by qemunbdd against its snapshot metadata and by viperblockd against the snapshot's persisted SourceVolumeName. * test(ebsprovider): run the contract suite against viperblockd in CI CI installed neither qemu nor nbdkit, so every conformance test that needs a real implementation skipped silently and MemoryProvider was the only provider the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails instead of skipping. TestViperblockdConformance stands the provider handlers up over an embedded NATS and a real predastore and runs the same suite against them, which is what caught CopySnapshot accepting a foreign volume's snapshot on unencrypted volumes. * feat(viperblockd): honour read-only publish and advertise block size constraints ReadOnlyPublish was advertised false and read_only=true was rejected, so the capability was true only by omission. nbdkit -r now starts the export when the request asks for it, which sets the NBD read-only transmission flag and makes the plugin refuse writes. Access mode is fixed when nbdkit starts, so a republish asking for the other mode is refused with VolumeInUse rather than handed the running export. Block size constraints were absent entirely: the nbdkit Go binding has no block-size callback, so the plugin cannot advertise them. The blocksize-policy filter can, without a binding callback. Minimum 1 because viperblock's WriteAt handles unaligned requests with a read-modify-write, preferred 4096 to match its block size, maximum 32M to match what qemu-nbd advertises. * test(conformance): judge sparse reporting by the allocation map, not the context The subtest asserted that base:allocation was offered. nbdkit offers that context whether or not the plugin can compute extents, answering 'all allocated' when it cannot, so the assertion passed for a server with no extent support at all — it could not fail. What separates the two is the answer: a volume with nothing written to it is entirely holes to a server that really tracks allocation. Read the map with nbdinfo --map --json and compare holes against the advertised capability. qemunbdd reports holes and advertises true; viperblockd reports none and advertises false. * test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime The predastore fixture is one daemon per test binary, but its cert was exported with t.Setenv, which reverts when the first test that called Start finishes. In-process clients did not notice because crypto/x509 caches the pool behind a sync.Once, loaded while the variable was still set. Child processes did: nbdkit spawned by a later test inherited an environment with no SSL_CERT_FILE and could not verify the daemon, so the whole NBD client suite failed when the package ran as a whole while passing when run alone. Also install libnbd-bin in CI, so the NBD client suite runs there instead of failing on missing tools. * ci: install nbdkit-plugin-dev so the viperblock plugin can build * test(volume): give the snapshot fixture an owner CreateVolume now fails closed on a snapshot whose owner it cannot match, including an unset owner_id. The fixture predates that and wrote none, so the provider test failed with InvalidSnapshot.NotFound before ever reaching the provider. * test(integration): give the volume harness a real provider The control plane no longer builds volumes itself, so CreateVolume in the integration harness failed with "no EBS provider configured": the harness wired a VolumeServiceImpl with nothing behind it. It now serves viperblockd's provider subjects against the same predastore fixture, so the blocks are still written by real viperblock to a real daemon. Three persisted-state assertions moved with it. TenantID, VolumeType and AvailabilityZone are control-plane facts, held in the ebsmetadata document rather than viperblock's own state, so the storage oracle now asserts their absence and DescribeVolumes -- whose answer is account-scoped -- is what checks them. * refactor: route image import through the EBS provider and drop the legacy metadata path The control plane could still only run against viperblock: the AMI import and the ochre weights import both linked the storage library directly, and the ebsmetadata store kept a read fallback onto viperblock's config.json. Import now goes through the provider contract with no new verb: CreateVolume, PublishVolume, a bulk copy over the published NBD export with qemu-img, then UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS, whose request size the contract caps far below a disk image. The migration scaffolding goes with it. ebsmetadata documents are now the only thing the control plane reads: no legacy prefix scan, no config.json fallback, no backfill on daemon start. config.json belongs to the provider, which is the single writer of it. ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is right for DescribeImages, where one corrupt document must not hide the fleet, and wrong for a name-uniqueness check, which cannot rule out a name it failed to read. spinifex and cmd no longer import viperblock outside services/viperblockd and testutil/vbscan. * feat: add a volume enumeration verb so a lost document cannot strand blocks Making ebsmetadata documents the sole index of volumes left one failure with no recovery: a volume whose document is gone disappears from DescribeVolumes, from the leak reaper (which enumerates documents), and from the provider, whose contract had no verb that asks what it holds. ListVolumes closes that. It is capability-gated on VolumeEnumeration and paginated, following CSI's optional LIST_VOLUMES rather than inventing a shape. Refs are deliberately not Volumes: enumeration answers what exists, and a provider must not open every volume's engine to say so. Each implementation reads what actually exists rather than what it remembers. qemunbdd walks its volumes directory, not its in-memory map, which does not survive a restart; viperblockd walks the bucket's top-level prefixes. A volume the provider has forgotten is precisely what a caller is looking for. The consumer is a read-only orphan report: provider volumes minus documented ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is excused only when its base volume is documented, so a stranded pair is still reported. Listing documents stays strict, because reporting a volume as orphaned on the strength of a document that failed to decode would point an operator at live data. * fix: stop the orphan scan accusing storage of holding things it does not A live env19 run reported three orphans against a healthy cluster: an AMI's backing volume, a snapshot, and the cluster key store. All three were wrong, and a report that cries wolf on a clean cluster is worse than no report. Enumeration treated every top-level bucket prefix as a volume. Snapshots sit beside volumes at that level and have a config of their own, so presence of a config cannot separate them; the snap- name is what does, which is already how the AMI removal path reads them back. keys/ and spinifex/ are not volumes at all. An AMI's blocks are a genuine provider volume, so enumeration keeps reporting them. What was missing is on the reading side: an AMI is recorded as an image document, not a volume one, so the scan now treats a documented image as known. Listing images is strict for the same reason listing volumes is — unreadable and absent are different answers, and only one means orphaned. * fix: let an orphaned image's blocks be reclaimed The orphan report tells an operator that reclaiming a stranded volume is their decision, and images remove --force is the tool it points them at. On env19 that tool failed on exactly those volumes: removal drops the ebsmetadata document first, and predastore answers NoSuchKey for a document that is already gone, which is the definition of an orphan. Deleting the document stays first and stays fatal. That ordering is the barrier that stops a launch landing on blocks about to be deleted, and an AMI still advertised after its blocks are gone is the worse failure. Only an already-absent document is now tolerated: the barrier is satisfied, and the blocks are the sole thing left to reclaim. S3ObjectStore.DeleteObject normalises a missing key the way GetObject already does, so this is recognised rather than matched on error text. The obvious unit test passed without the fix, because the in-memory store is silently tolerant on delete and real S3 answers 204 — neither behaves like the backend this runs against. The test now drives a store that reports the miss, and fails without the fix. * feat: add per-verb OTel spans to the EBS provider contract The ten provider verbs cross a single NATS seam and nothing measured it. The only spans on the storage path were the legacy ebs.mount/ebs.unmount pair, so control-plane storage latency could not be attributed to a verb or compared across provider implementations. NATSProvider.request is the one client-side funnel every verb passes through, including both legs of an owner-subject fallback, so a single span there covers the whole contract and measures what the control plane experiences regardless of which provider answers. Span names come from a closed verb set rather than the subject. Owner and snapshot-create subjects embed a volume ID and the mount/unmount subjects embed a node ID; naming spans after them would give every volume a span name of its own. The IDs move to attributes and an unrecognised subject collapses to a single unknown verb. Trace context rides the message headers, so the serving provider joins the caller's trace instead of rooting one of its own. The carrier is a copy of the one in spinifex/utils rather than an import: that package pulls in the aws-sdk and the cluster key store, which the wire contract must not depend on. Responses carry no shared accessor for their error field, so both sides re-read just that field from the encoded reply, and only when the span is recording. Without it a failed verb looks like a successful round trip. viperblockd serves the contract with its own handlers rather than natsserve, so its handlers now take the per-message context and its publish/unpublish pair drops the subject-named consumer span it opened for itself. * feat: add null and hostile providers, and a per-verb bench suite The provider boundary made it possible to put something other than viperblock behind the control plane, and nothing yet did. nullprovider is the floor. It is not a new type: MemoryProvider already answers every verb from a map and natsserve already serves any EBSProvider, so the package names that composition and states what it measures. On loopback with no storage behind it a round trip costs around 195us against 74ns in process, which is the seam's own price and is not attributable to any storage engine. conformance.RunBenchSuite prices one round trip per verb and drives every implementation through the same calls in the same order, so two providers' numbers are comparable. It measures control operations only: guest I/O never crosses these verbs. hostile wraps any provider in one that misbehaves. Faults derive from the seed and a per-target sequence rather than a global counter, so concurrent callers on different volumes cannot reorder each other's faults and a failure found deep into a soak run replays exactly. Error and error-after-work are separate faults on purpose: the second reports failure after the work has already happened, which is how an orphan is made, and is what makes an enumeration scan the right oracle for a soak. Enumeration itself is never faulted, since an oracle that can be told what to say cannot falsify anything. The soak asserts that every volume the provider still holds is attributable to an injected fault. A leftover with nothing behind it is the control plane losing track of storage on its own. Two bugs were found by falsifying the new tests rather than by running them. The FNV-derived draw clustered badly enough that two fault bands were never reachable in 1200 draws, so the draw now goes through a SplitMix64 finalizer. And the first version of the lie test was worthless: MemoryProvider returns copies, so a lie could not corrupt it whether or not the injector copied, and the test now drives a provider that hands out pointers into its own state. * test: bench the provider contract against qemunbdd and a live viperblockd RunBenchSuite was only ever pointed at the null provider, so the numbers priced the seam and nothing else. Point it at the two real implementations. BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a provider that stores nothing and wrong against one that does, so qemunbdd and viperblockd are both measured at 64 MiB. liveProvider splits into a testing.TB flavour. Benchmarks need the same connection and the same node and cannot be handed a *testing.T. * fix(viperblockd): carry the trace context into every viperblock engine call Both S3 clients only emit a span when the request already carries one, so calling the no-context Init, LoadState, SaveState, LoadBlockState and Close left the whole engine open untraceable. Every provider verb traced as one flat server span, and the 342ms describe measured on env19 could not be attributed to anything. viperblock already exposes the Ctx variants and Backend already declares InitCtx, so this is threading, not new instrumentation. openVolumeVB, openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine, copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and mount/unmount keep the context their own span derives instead of dropping it. completeCreateSnapshot gets WithoutCancel: the background half of accept-then-publish has to outlive the request it was accepted by and still be attributable to it. The per-volume ebs.config handler opens a consumer span like the other legacy subjects, so a config update is no longer the one path with no trace at all. * perf: describe a volume without opening an engine on it describeVolumeEngine fell through to openVolumeVB for any volume not mounted on the node serving the request, which constructed a full viperblock engine to read one integer. On env19 that decomposed as a 22 ms reachability list, a 3 ms state read and a 16 ms PutObject of the volume's own config.json — the write coming from the SeqNum bootstrap a read-write open performs. Use viperblock's read-only state load instead. A describe is now the 3 ms read: no background goroutines, no reachability probe, no SeqNum window claimed, and no write issued by a worker that does not own the volume. Falsified: restoring the openVolumeVB call fails TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the list, with the fake backend answering each so neither assertion hides behind the other. * perf: share one HTTP client across every viperblock backend Four sites built the same S3Config by hand, each leaving HTTPClient unset, so every engine open and state read got its own connection pool and paid its own TLS handshake. That was invisible while Backend.Init issued a whole-bucket list — the list paid for the connection and the state read rode it — and it became the dominant cost once describe was reduced to a single request. Route all four through cfg.volumeS3Config, which builds the client once. Falsified: dropping the HTTPClient field takes TestDescribeVolume_ReusesOneConnection from one connection to three. * perf: wait for nbdkit to be ready instead of waiting a fixed second mountVolume spawned nbdkit and then ran a confirm loop until a one-second deadline passed. The loop only broke early when the process exited, so the fast-failure path worked and the success path always cost the full second. That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so the timings never contradicted the assumption that it was process spawn. Wait for the endpoint to accept a connection instead. That is also the question the caller needs answered, since it is about to hand the URI to something that will connect to it — where 'still alive after a second' only correlates with it. A nbdkit that exits during startup still fails the mount immediately, and one that never comes up now fails on a deadline rather than being assumed healthy. Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness timing assertion and the fast-failure one. * test: measure the data path, not just the control verbs The conformance benchmarks price create, publish and describe. Guest I/O never crosses those verbs, so none of those numbers say anything about what a guest experiences. This drives the published export with qemu-img bench, which sits on the same block layer virtio-blk does. It asserts only that the I/O completes and logs a table: what a given store is capable of is not a contract, and pinning a number here would only produce a flaky test. Read profiles stay inside the region the write profiles covered, or they would be timing how fast the export answers with zeroes, and one write profile flushes, because an unflushed write measures how fast the WAL accepts data rather than how fast the store takes it. Profiles are spaced apart because the plugin tears an engine down per disconnect and removes the volume's local directory as it goes, so a connection opened while the previous one is closing fails its handshake. * test: run data-path profiles back to back The spacing was a workaround for a volume lock that did not hold across close. With that fixed, consecutive profiles are worth more than the workaround was: each is its own NBD connection, so a run that completes has also shown that an arriving connection waits out the previous one's close rather than racing it. * fix: refuse a delete that cannot establish whether a volume is published volumeMountedElsewhere returned false on every failure path, including a request timeout, so a busy or partitioned owner was indistinguishable from no owner at all and the delete proceeded. Deleting a published volume leaves the nbdkit export and the guest writing through it alive over storage that is going away. Report the three outcomes separately. NATS distinguishes no-responders from no-answer, so only the former means unmounted; an unanswered probe is refused and the caller may retry once the owner is reachable. * feat: exclude a second viperblock engine with a JetStream volume lease Nothing in the stack stopped two viperblock engines opening one volume from different nodes. For an encrypted volume that is not just a consistency risk: the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte, and the SeqNum window claim is a read-then-write with no CAS, so two engines opening concurrently issue overlapping nonces under the same key. Every engine open now takes a lease first, keyed by volume in a JetStream KV bucket. A lease is a KV create, which JetStream serialises, so a second claimant loses instead of both winning. The revision that won is the generation: monotonic across the bucket, so a later opener always carries a higher one than the writer it displaced. Entries carry a TTL and a renewal loop, so a node that dies mid-mount does not wedge its volumes forever. Losing a lease is logged, not enforced. Until the object store can refuse a stale writer's PUT, failing live guest I/O on lease loss would add a failure mode without stopping the zombie it is aimed at. A daemon with no lease store refuses to open rather than opening blind, so nbdkit recovery binds the store before it adopts surviving exports. Repeat acquisitions on one node share a lease: the unmount seal opens a detached engine while the mount being torn down still holds it, and cross-node exclusion is what the lease is for. A restart reclaims this node's own stale entries, since the exports that outlived the daemon are still running. * refactor: consume masterkey from bluebottle in the test this branch added * feat(ebsprovider): classify refusals a caller may retry Every ErrorCode said what went wrong and none said whether the same request could succeed later, so a caller could not tell a permanent refusal from a transient one. Transient conditions shipped as internal, which tells a caller to give up when it should retry. Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(), which callers branch on instead of on the code itself. DeleteVolume's "could not establish whether the volume is published" refusal moves off internal onto it: the owner that did not answer may answer later. TestErrorCodeRetryable names every code, so adding one without deciding its retry meaning fails rather than silently defaulting to permanent. TestClassifyError pins the sentinel-to-wire-code mapping, which had no coverage at all; a sentinel missing from classifyError falls through to internal and loses both the reason and the retry signal. * fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for Four providers advertised the capability and nothing consumed it: there is no clone verb on the interface. CreateVolume's SourceSnapshotID restores from a snapshot and CopySnapshot duplicates snapshot metadata; neither is a volume clone, and the field documenting SourceSnapshotID says so explicitly. CSI expresses clone as a volume_content_source with a volume source, which this contract does not implement. Advertising it told a caller a feature was available and then gave them no way to use it. Removed rather than backed by a method: adding a clone verb is a feature, not a contract fix, and the capability can come back alongside it. The Capabilities doc comment now states the rule, and qemunbdd's stale comment about sparse extents is corrected while there. * feat(ebsprovider): add ListSnapshots so snapshots are reconcilable The contract could create, copy and delete snapshots but never say which ones it held. That left the control-plane metadata document as the only index of them, so losing one stranded the data it pointed at with no way to find it again. This is the same exposure ListVolumes closed for volumes, and it stayed invisible because ListSnapshots is optional in CSI too. ListSnapshots pages exactly as ListVolumes does, gated on a new SnapshotEnumeration capability. SnapshotRef is deliberately not a Snapshot: enumeration answers what exists, and reading a snapshot's size or state can cost far more than listing it. viperblockd walks the bucket's snap- prefixes, sharing the top-level walk with ListVolumes. It leaves SourceVolumeID empty because the link lives inside the snapshot's own encrypted state and reading it would mean a fetch per snapshot listed. qemunbdd walks its snapshots directory rather than its in-memory map, so a snapshot that outlived the process still appears; its SourceVolumeID is empty in that case for the same reason CreateSnapshot refuses to reuse the ID. Five conformance subtests cover it, and they run green against MemoryProvider, qemunbdd and viperblockd over a real predastore. * feat(ebsprovider): state the exclusion guarantee in the contract A caller could not tell which providers prevent two writers from holding one volume, at what scope, or for how long. The guarantee existed in viperblockd and was invisible from the outside, which is why the conformance suite had no concurrency dimension at all. Capabilities.Exclusion states it, and deliberately separates three questions the contract used to run together: how far exclusion reaches (none, node, cluster), whether a dead owner's claim ever expires, and whether a writer that lost its claim is actually stopped. That last one matters most and is the least flattering: exclusion refuses the second opener, fencing stops the first. viperblockd advertises cluster scope with a 45s TTL and FencesLostClaim false, because a partitioned node whose lease lapsed keeps writing until the object store can refuse it. Four new conformance subtests. Two required a second viperblockd node in the harness: cluster exclusion cannot be exercised by one node, because the second publish goes to a subject nobody serves and times out rather than being refused. Both nodes share the NATS the leases live in and the bucket the volumes live in, so a refusal comes from the lease. Standing that second node up immediately found a real gap, in a test that had been silently skipping for want of one. A publish refused by another node's lease came back as internal rather than volume_in_use, so a caller was told something had broken when the answer was that someone else held the volume. Mapped in handlePublishVolume. * build: pin viperblock to the release carrying the provider's dependencies The branch has been building against a local replace pointing at a viperblock worktree, so its committed pin never actually built: it named a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch, NewStateReader or s3.NewHTTPClient, all of which this branch calls. Points at the merged viperblock dev instead and drops the replace. GOWORK=off now builds clean, which is the deploy path ansible uses. * fix: let viperblockd's delete sweep tolerate an object another sweep removed deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it returned on any DeleteObject error. Two overlapping deletes for one volume left the loser reporting an internal failure once the winner had removed a chunk between the list and the delete. The control-plane copy that carried this fix is gone with the embedded engine path, so the provider path reintroduced the bug it had just fixed. * fix: stop CreateVolume leaving local state on whichever node served it A state read prefers this node's local copy over the backend, and provider RPCs are not owner-routed, so the node that happened to serve a create kept a copy that answered describe as if the volume still existed after another node deleted it. The two-node conformance harness caught it as DeleteVolume/success failing about one run in three. The seeded path already removed its local files after closing. The unseeded path saved state and left them, which was the only difference between the two. * fix: guard the mount path itself against a second export of one volume handlePublishVolume refused a volume this node already exported, but mountVolume did not, and the legacy ebs.<node>.mount subject is the route production takes for instance launch and hot-attach. A retried attach racing a fresh one started two real nbdkit against one volume, with no refusal and no warning: the volume lease only refuses a different node, and a same-node re-acquire finds the volume already held and increments its refcount. That also made the advertised exclusion inaccurate. viperblockd claims cluster-scoped exclusion on the grounds that a second opener is refused, while the conformance subtests reach it only through PublishVolume, so they passed with the legacy route unguarded. Moving the check into mountVolume gives both entry points the same behaviour and makes the claim true for the path production uses. * fix: tolerate a missing legacy AMI config on deregister DeregisterImage deletes the AMI document, then the pre-provider {imageID}/config.json. Predastore answers a delete of an absent key with NoSuchKey, and that was treated as fatal, so every AMI created since the metadata store landed failed to deregister after its document was already gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had just been told it could not delete. The regression test uses a store that answers a missing-key delete the way Predastore does; MemoryObjectStore silently succeeds, which is why the unit suite stayed green. * fix: reap ENIs left behind by an abandoned instance launch A launch creates the instance's ENI and then attaches it. When the daemon dies in between, the record survives with no instance and nothing ever looks at it again. The security group dependency check counts an ENI that lists the group, so the group cannot be deleted for as long as the record exists. The in-process launch path already rolls back on error; a process that dies cannot. This adds a cluster-wide sweep over records that are launch-created, never attached, and older than the launch RPC timeout by a wide margin. The description prefix is now a shared constant because it is the only thing separating an abandoned launch from a standalone ENI a caller asked for.	2026-08-12 16:54:19 +10:00
Julian Sommer	8f930e6275	refactor: consume the shared packages from bluebottle ... Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather than predastore, so the control plane no longer depends on the S3 server for shared auth and policy code.	2026-08-12 10:50:48 +10:00
brynmailer-mdc	a7a62addeb	test: rebuild the predastore fixture and service tests on the facade ... The fixture wrote a config the strict decoder rejects and built its cluster through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and serves under Run. Loading the real file means the fixture trips over the same validation an install does. Its data directory holds node-<id> directly, and it drains on cancellation rather than being abandoned at exit. The runtime tests drove buildRuntime, serve and the old on-disk layout, all of which predastore owns now, so they go. What replaces them covers what spinifex still decides: the flag merge and its precedence, a host id or gate node the config does not have, and the pid file behind Stop and Status. Two fixes to earlier commits in this series: the host id conversion moves behind its own guard, and the node status probes use WaitGroup.Go.	2026-08-11 17:23:42 +10:00
Josh Siviour	0a1d837f40	feat(bedrock): gate Ochre models behind per-account access grants (#741) ... * feat(bedrock): gate Ochre models behind per-account access grants Self-host models were invokable by any authenticated account: tieredCatalog returned them unconditionally and no runtime router checked the caller. GetFoundationModel also described models the caller could not use. Add a bedrock-model-access JetStream KV bucket holding presence-only grants, resolved through a new AccessResolver. Access is deny-by-default: a gateway with no store configured falls back to DenyAllAccessResolver, so an unconfigured deployment serves nothing rather than everything. The system account bypasses grants, matching how quotas exempt it. All four runtime routers share one grantedCatalogEntry gate so they cannot drift on who may invoke what. List and describe agree: an ungranted model is absent from ListFoundationModels and reported as ResourceNotFoundException by GetFoundationModel, so the error cannot confirm a model exists. Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on the spinifex gateway service for the UI, and `spx admin ochre access` for operators and provisioning, which opens the bucket over NATS directly rather than requiring a SigV4 client the CLI does not have. * feat(bedrock): seed the admin account's model grants at gateway start Deny-by-default left a fresh install with a catalog nobody could see, and the first cut filled that gap from ansible — which is a dev tool, so any install that never ran it came up unusable. Seed from the product instead: on start awsgw grants the platform admin account the full catalog, once, behind a marker key in the same bucket. The marker makes it a bootstrap step rather than a policy: tenant accounts still begin with no access, and an operator who revokes a grant does not have it restored by the next restart. Grants are written before the marker so a crash midway repeats the idempotent grants rather than leaving a half-seeded account, and the marker is a conditional create so only the first node to reach it need win. Seeding is best-effort: a KV failure logs and leaves the gateway serving, and the next start retries because the marker is written only on success. * test(bedrock): cover the model-access CLI and gateway admin actions Route the access CLI runners through the same loadConfigAndConnectFn / ochreExit seam the weights runners use, so grant, revoke and list can be driven against an embedded JetStream server, and add gateway tests for the grant admin actions including the non-admin denial.	2026-08-06 16:17:10 +10:00
Julian Sommer	1c4d41bffc	fix: reach the DB endpoint from any subnet of the VPC (#738)	2026-08-06 13:56:03 +10:00
Julian Sommer	8c07f80032	chore: remove the half line breaks	2026-08-06 11:10:15 +10:00
Josh Siviour	688ce5ccfd	feat: add AWS model conformance coverage (#736)	2026-08-05 18:46:37 +10:00
Julian Sommer	2a79bc8473	fix: record RDS instance creation events (#734)	2026-08-05 15:17:06 +10:00
Julian Sommer	f7d5d97e3f	fix: propagate RDS parameter group edits (#733)	2026-08-05 15:08:14 +10:00
Julian Sommer	68671cdf19	fix: prevent parameter groups bricking RDS instances (#731)	2026-08-05 14:16:31 +10:00
Julian Sommer	f240c33778	fix: wait for postgres to be up before running tofu workbook e2e	2026-08-05 11:48:58 +10:00
Julian Sommer	710169e5a6	fix: eliminate RDS Terraform drift (#729)	2026-08-05 10:34:51 +10:00
Julian Sommer	d2d0b357a9	fix: tofu e2e	2026-08-05 10:05:45 +10:00
Julian Sommer	871513dc6c	Reapply "feat: match new predastore config surface area (#591)" ... This reverts commit b4701c78de.	2026-08-04 16:13:28 +10:00
Julian Sommer	b4701c78de	Revert "feat: match new predastore config surface area (#591)" ... This reverts commit c2b80a9b93.	2026-08-04 15:02:39 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Julian Sommer	14e24243d8	fix: rds e2e assertion	2026-08-04 11:29:20 +10:00
Julian Sommer	faa0519cf1	fix: the RDS master user is no longer a PostgreSQL superuser (#687)	2026-08-04 10:59:45 +10:00
Julian Sommer	f3586ed32c	docs: add the RDS quickstart Terraform workbook (#686)	2026-08-04 09:54:41 +10:00
Julian Sommer	5e17169661	feat(e2e): complete the RDS end-to-end suite (#608)	2026-08-03 15:38:07 +10:00
Josh Siviour	261d5d505e	feat: Ochre catalog serving spec and a servable self-host model (#672) ... * feat: in-tree serving specs + KV weights override for ochre catalog Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a self-host model's serving requirements ship with the catalog rather than living only in an operator's head. Add WeightsStore, a deployment-scoped (not account-scoped) JetStream KV override in a new bedrock-weights bucket that resolves a model's staged weights snapshot ID, following the same shape as the existing CredentialStore. It is plumbed into tieredCatalog and GetFoundationModel through a package-level resolver (SetWeightsResolver/currentWeightsResolver), since both are called from gateway/bedrock.go's fixed-arity route table, which is out of scope here. A self-host model with no resolvable weights snapshot is not servable and must not be advertised: tieredCatalog filters it out and GetFoundationModel returns ResourceNotFoundException for it. Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which needs roughly 140 GB of VRAM in fp16 and has never actually run, with meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to 4096 to budget headroom for CUDA context, activations and KV cache on top of that. This is an estimate, not a measurement off wattle - .7.5's launcher should correct it against a real nvidia-smi reading once the model is actually running there. lookupCatalogEntry, used by the untiered invoke/converse routing path, is left unchanged and still ignores weights resolvability; gating that path is out of scope for this bead. * fix: separate weights resolve errors from not-found in ochre catalog An internal fault (e.g. a JetStream outage) reaching the KV weights resolver was being collapsed into the same "not servable" verdict as a clean not-found: tieredCatalog would silently unadvertise every self-host model, and GetFoundationModel would return ResourceNotFoundException for a model that does exist but whose resolver is temporarily broken. Both call sites now distinguish the two: a resolve error is logged and surfaced as its own error (mapped to ServerInternal at the AWS error boundary), and only a clean not-resolvable result yields ResourceNotFoundException. tieredCatalog's signature changes to return an error alongside the entries so ListFoundationModels can propagate a resolve fault instead of silently returning a thinned-out list. Also raises the self-host entry's gpu-memory-utilization from 0.5 to 0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to 5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV cache and activations at max-model-len=8192, likely too tight for any real concurrency. The two numbers are meant to agree: MinVRAMMiB is the admission-gate floor, gpu-memory-utilization is what tells vLLM to actually claim that much on an 8188 MiB device. Comments trimmed to the repo's 3-line limit and stripped of plan/bead references; the VRAM reasoning and model-choice rationale live in the commit body and the plan doc instead. Threading the weights resolver through as an explicit parameter (mirroring CredentialResolver) rather than a package-level global was considered per review. Doing so needs edits to gateway/bedrock.go (bedrockRouteHandler's signature, both route closures, the Bedrock_Request call site, a new bedrockWeightsResolver() method) and gateway/gateway.go (a new BedrockWeights field on GatewayConfig). Both are outside gateway/bedrock/ and are explicitly out of scope for this change, owned by a parallel workstream already modifying the same route table. The global resolver stays, now with the error-handling fix above. * fix: stop e2e GetFoundationModel regressing on the new weights gate TestGetFoundationModel's "known model" subtest asserted on meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now gated on a weights snapshot staged in the bedrock-weights KV bucket. On any real deployment that bucket starts empty, so the subtest would fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST. The "known model" assertion now targets the Anthropic entry, which resolves unconditionally and keeps testing what the subtest was actually there for: a known model round-trips, an unknown one is ResourceNotFoundException. The self-host assertion moves to a new TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1 alongside TestConverseSelfHost, where weights are expected staged. tests/integration/bedrock_test.go's TestListFoundationModels already wires a weights-resolver stub before asserting the self-host entry is present, so it does not regress the same way. * test: correct a stale assertion message on the ochre catalog weights gate	2026-08-03 12:34:07 +10:00
Bryn Mailer	c2b80a9b93	feat: match new predastore config surface area (#591) ... * match new predastore config surface area * fix(migrate): carry existing installs onto the predastore v5 topology The cluster topology refactor breaks an install three ways at once. The old config carries a top-level host key, which the new schema binds to the [[host]] table array, so TOML rejects the document and the daemon dies before it reads any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host data directory keyed by a node id that is now unique across roles. And because object metadata records which node holds each shard, the shard nodes had to keep their ids, so the state replicas took new ones and every replica's persisted raft configuration stopped listing it. predastore.toml gains a 4 to 5 migration covering all three: it rewrites the tables, relocates whichever node directories exist on this machine, and recovers each relocated state replica onto the new member set. The rewrite is line-oriented rather than a re-render from the template, so buckets, auth, rate limits and iam settings an operator edited survive. spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to host_id, which predastore-start.sh greps to select the host this machine runs. The predastore migration lives in its own package rather than in migrate, which every handler that stamps a KV bucket version imports; reaching the cluster runtime from there would pull raft, badger and the QUIC client into all of them. admin_upgrade.go blank-imports it, so only the binary that runs the upgrade links it, and a test pins both registrations. * test(predastore): cover the service runtime, gateway and shutdown paths The service wrapper's new seams had no unit tests: buildRuntime, serve and the errgroup Start runs the cluster and the S3 gateway under. CI's diff coverage gate measured the PR at 34.6% against a 60% threshold as a result. Cover every buildRuntime error wrap and its host selection, both ways the gateway can fail serve, and the full lifecycle: Start brings the nodes and the gateway up together, Shutdown drains both by cancelling the context they share. Topologies are written per test as real config, with node ids from a process-wide counter — the pipe transport's listener registry is global, and this package's integration tests hold a fixture daemon on the low ids for the life of the test binary.	2026-08-03 09:59:25 +10:00
Josh Siviour	3e1f9b1198	feat: segscan storage oracle for the real-predastore integration tier (#657) ... * feat: add segscan storage oracle for the real-predastore integration tier Adds spinifex/testutil/segscanoracle, which copies a fixture node data dir and execs the mulga umbrella repo's scripts/segscan (a separate Go module that cannot be imported) to assert live/dead-tombstoned/dead-orphan byte accounting against real predastore writes. Kept off the default PR path: a new segscanoracle build tag excludes the test from make test-integration, and a separate segscan-oracle.yml workflow (push to main/dev + manual dispatch only, mirroring e2e.yml's non-gating trigger) checks out scripts/segscan rather than adding that checkout cost to spinifex.yml. * test: cover segscanoracle's pure logic to satisfy diff coverage segscanoracle has no build tag and always compiles into the default build, but nothing in that build exercised it, so every new line counted as uncovered against the 60% diff-coverage gate. Adds unit tests for JSON decoding (including the malformed-payload path), the Locate resolution order, and the directory-copy routine, plus a throwaway stand-in module (not the real scripts/segscan) to exercise build+exec+decode end to end. Pulls decodeReport out of Run and resolveSegscanDir out of Locate so the pure logic is testable without a *testing.T or a real binary.	2026-08-03 00:40:45 +10:00
Josh Siviour	9a1fb44065	fix: CreateImage live drain + Alpine .raw sparsify (#647) ... * fix(ec2): drain the attached volume before CreateImage reads a live checkpoint CreateImage against a running instance (--no-reboot) read the volume's checkpoint directly from predastore without draining in-flight writes first, unlike CreateSnapshot. A checkpoint captured before a guest- triggered write (e.g. growpart rewriting the GPT after a root volume is launched larger than the base image) could be stale, producing an AMI that boots to a UEFI shell instead of the OS. Extracts the existing snapshot-package drain logic into a shared, reusable DrainVolume so both live-snapshot call sites route through the same drain-before-read guarantee instead of being able to diverge. * test(e2e): boot-verify CreateImage AMIs instead of trusting State=available describe-images reporting State=available is not evidence an AMI boots — a well-formed but unbootable AMI (stuck at the UEFI shell) still registers cleanly. Extend the CreateImage e2e test to launch a probe instance from each produced AMI (running-path and stopped-path) and require a real SSH login, with console-dump-on-failure wired up for diagnosis. * fix(images): sparsify the Alpine guestfish .raw before conversion The Alpine build path grows the qcow2 in place via qemu-img resize + guestfish resize2fs, which can leave metadata scattered across the newly-grown region. virt-sparsify zeroes the guest's unused blocks before the raw conversion so they collapse into real holes, matching the Ubuntu virt-resize path's sparseness.	2026-08-02 11:26:04 +10:00
Josh Siviour	a3fb4cccec	fix(eks): backoff and terminal state on delete re-drive, CSI volume reclaim reporting, independent e2e subtests (#642) ... * fix: bound EKS DELETING re-drive with backoff and a terminal give-up reapCluster re-drove purgeClusterInfra on every GC tick with no attempt tracking, so a permanently-failing teardown (e.g. an unretriable DependencyViolation) was retried forever. Track attempt count and last attempt time on ClusterMeta, back off exponentially between re-drives, and give up after maxDeleteReapAttempts with a single error-level log. Status stays DELETING on give-up so the cluster's infra remains tracked and billable pending operator intervention, preserving the ADR-0006 §6 billable-before-sweep invariant. * fix: create the EKS e2e node role independently in each subtest runEBSCSIVolume string-built the worker node role ARN without ever creating it, only passing because runIRSAPod's t.Cleanup discarded DeleteRole's error while the role was still attached to its instance profile, leaving it behind for EBSCSIVolume to reuse by naming convention. Extract ensureNodeRole and call it independently from both subtests, detaching the role from its instance profile before deleting it and asserting that error instead of discarding it. * fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete purgeClusterInfra tore down the control plane, nodegroups, and networking but never looked at EBS volumes, so any volume the in-cluster CSI driver had provisioned for a PVC was left behind untracked and unbilled for. Add a report-only reclaim step that scans for volumes tagged with the cluster's CSI owner tag and logs them for operator follow-up. Volumes are never auto-deleted: a PV's reclaim policy (Delete vs Retain) is a Kubernetes-side property the CSI driver does not tag onto the EC2 volume, so there is no safe way to tell from tags alone whether deleting a given volume is intended. This mirrors the volume-leak reaper's existing mark-and-alarm policy for the same reason. The cluster-owner tag requires the EBS CSI controller's --k8s-tag-cluster-id flag, added here to the controller manifest; without it a CSI volume carries no cluster-scoped tag and reclaim would have to guess across every cluster sharing the account. * fix: persist the last purge error on a wedged DELETING cluster DeleteReapExhausted told an operator a cluster's teardown had given up, but not why: nothing recorded the failing purge's error. Persist it on every failed re-drive, not only once the reaper exhausts its attempts, so it is visible on a still-retrying cluster too. Status stays DELETING throughout, preserving the billing invariant that untorn-down infra remains tracked. * fix: make the EKS e2e suite confirm its CSI volume is actually deleted The EBSCSIVolume subtest deleted its PVC on cleanup but never confirmed the underlying volume was reclaimed, so a CSI driver that silently failed to delete it would leak a volume with nobody noticing. The suite provisions and owns this volume, so unlike production teardown it can safely wait on its StorageClass's Delete reclaim policy: capture the CSI volumeHandle while the PVC is bound, then have cleanup poll EC2 for that exact volume to disappear before the addon/nodegroup that run the CSI controller are torn down.	2026-08-02 08:21:25 +10:00
Josh Siviour	9f5f877e50	fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637) ... * fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console.	2026-08-02 08:17:18 +10:00
Josh Siviour	6873875248	fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634) ... On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane.	2026-08-02 08:16:00 +10:00
Josh Siviour	d38db57912	fix(ec2): release post-launch ENIs on instance terminate (#623) ... * fix(ec2): release post-launch ENIs on instance terminate Terminate released only the ENI recorded at RunInstances time. Attachments made after boot — ECS awsvpc task ENIs, or any direct AttachNetworkInterface — are written to the spinifex-vpc-enis KV and never reach instance.ENIId, so both terminate paths walked straight past them. The survivors held their security group, blocking the SG, subnet and VPC deletes behind DependencyViolation. Both paths now sweep the KV for records naming the instance, independent of the scalar. The sweep is additive, so an instance with no secondary ENIs behaves exactly as before, and best-effort per ENI so one failure neither aborts the rest nor blocks convergence on re-run. DeleteOnTermination was advertised by DescribeNetworkInterfaces but backed by nothing; ENIRecord now carries it, defaulting to true, and false detaches without deleting. * test(ec2): cover ENI release branches Add table-driven and NATS-backed tests for the terminate-time ENI sweep: DetachENI/ForceDeleteInstanceENI failure tolerance, DeleteOnTermination skip, multi-ENI happy path, and ListInstanceENIs error mapping on both the daemon and instance service adapters.	2026-08-01 16:18:05 +10:00
Josh Siviour	5fc6a137ec	fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none (#612) ... * fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none An empty ipsec_encapsulation was treated as "bootstrapped without IPsec" and skipped. On ubuntu-24.04 openvswitch-ipsec.service is inactive at daemon start, so the enable guard correctly refuses to flip the flag and the cluster runs plaintext intra-AZ Geneve — but the test then skipped and the cell reported green, masking the gap on 2 of 3 multinode cells. Cross-check network.ipsec_enabled from the node config: an explicit false still skips, anything else fails with the plaintext condition named. * fix(e2e): read only the ipsec_enabled value, not the whole config line Detail() rendered "ipsec_enabled=ipsec_enabled = true" because the raw grep line was logged as the value. Take the first token after "=" so a trailing comment cannot be read as the value either, and compare exactly rather than by substring.	2026-07-30 23:28:42 +10:00
Josh Siviour	f973b306c4	fix: serialise DNS zone writes behind a per-zone lock (#610) ... Every daemon read-modify-wrote the same zone object in s3://northstar with no mutual exclusion. A NATS queue group balances load, it does not serialise distinct messages, so concurrent daemons lost each other's records and could leave the object unparseable as TOML. Once unparseable it stayed that way: both the writer and the drift reconciler must parse the zone before they can repair it, so DNS changes stopped applying until the object was edited by hand. The exposure did not need user activity. The drift backstop ran on every node, unelected, on a 60s ticker, so an idle three-node cluster raced its own zone once per tick. Five publisher families feed the same object, so ordinary concurrent API calls collided too. - Add a per-zone JetStream KV lock, following the CAS patterns already used by the vpcd reconcile leader and the vCPU quota store. One key per zone, so unrelated zones never block. The wait budget stays under requestTimeout so a producer's ack is not lost to contention, and a write aborts rather than land outside an expired lease. - Rebuild a zone whose stored bytes do not parse instead of failing every write forever. The reconciler reports a corrupt zone as absent, which suppresses pruning while still publishing the full desired set. Detection matches the decoder's error type so it works against the pinned northstar release. - Elect one node per drift cycle, removing the idle-cluster self-race. - Correct three comments claiming the queue group serialised zone writes; they are why the hazard went unnoticed. - Run the LB e2e internet-facing subtests in parallel. Their serialisation workaround exempted internal LBs on the false premise that those use a different DNS path; ELBChanges sets the same base-domain zone for both.	2026-07-30 18:47:07 +10:00
Josh Siviour	3e195506d3	feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. * refactor(images): extract eks-node initd logic into init-agnostic helpers The OpenRC runscripts carried their start_pre logic inline, so porting a service to systemd meant re-deriving that logic under a distro change rather than just rewriting the init wiring. Move it into /bin/sh helpers that either init system can drive, keeping eks-node on Alpine/OpenRC. k3s-prestart takes the restore-block guard, config.yaml derivation, konnectivity-agent and CoreDNS staging, and the token-webhook wait. konnectivity-server-prestart takes the CA wait and cert mint, plus the server-count derivation that used to run at parse time. Both keep every guard, bound and message as-is. Values flow back the way the GPU image already does it: the helper writes shell assignments to a /run file and the init system sources them, so command_args is built in start_pre where it used to be a top-level assignment. k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU manifest claimed its helpers were reused verbatim from eks-node while hand-copying this one; it now points at the shared file, so the two cannot drift. k3s-agent.initd still sources agent.env and exports itself rather than delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon inherits only what the runscript exports, so a helper subprocess sourcing it would not reach k3s. The helpers run under set -eu, which the runscripts did not. The awk over first-boot.env is guarded on the konnectivity manifest, not on the env file, so a missing env file exits awk nonzero; without || true that turned a warn-and-continue into a refusal to start k3s. Covered by a test that fails on rc=2 if the guard is dropped. * feat(images): build system images in a pinned mkosi container Move the image-build toolchain into a container so a runner needs only Docker. The toolchain cannot drift between runners, and the same build runs identically on a developer's machine. mkosi builds in a user namespace, which two of Docker's default confinements block: the seccomp profile masks CLONE_NEWUSER, and the docker-default AppArmor profile separately blocks unsharing the mount namespace. Both are relaxed; neither is a privilege grant, as capabilities stay dropped and no devices are exposed. mkosi also runs as an ordinary user rather than root, since as root it takes a path that needs real CAP_SYS_ADMIN. Verified end to end: an Ubuntu 26.04 Format=disk image builds from the Debian container and the artefact is a valid GPT disk, with the package cache on a named volume cutting a forced rebuild from ~2min to ~34s. * feat(images): publish system images from the build container Convert mkosi's raw output to a compressed qcow2, write the .sha256 sidecar the spx catalog verifier matches, and upload both to R2 using the same pinned toolchain as the build, so publishing needs nothing on the host either. The toolchain image build moves to a shared helper rather than being duplicated across the two entry points. Publishing needs no relaxed confinement, unlike the build: qemu-img and aws create no namespaces, so it keeps Docker's default seccomp and AppArmor profiles. The object path carries no version and the catalog resolves newest-wins, so an upload overwrites the live image with no way back. Copy the current object to a timestamped archive key first, making a bad publish a restore rather than an unrecoverable loss. * fix(images): build the image on the output's mount, not the cache volume mkosi assembles the image in its workspace and moves the result to the output directory. The workspace lived under the cache volume, which is a different mount from the output bind mount even when both sit on one filesystem, so the move failed EXDEV and mkosi fell back to copying — requiring the finished image to exist twice at once. That is invisible on a small image and fatal on a real one: a ~4G GPU image needed ~8G to land and died with ENOSPC after a successful build. Point the workspace at the output's own mount so the move is a rename again and the image only exists once. The package cache stays on the volume, which is the part worth persisting across runs. * feat(images): add mkosi base config and gpu-nvidia profile Establish images/ with a base mkosi.conf plus a gpu-nvidia profile, replacing the throwaway smoke-test config. The base builds a bootable UEFI disk; the profile adds the NVIDIA stack. Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version hunt, the do_bootloader=no kernel-img.conf hack and the manual update-initramfs all exist only because that pipeline chroots into a prebuilt cloud image that already has a kernel. mkosi installs the kernel and owns the bootloader, so the version is known rather than detected, and the postinst asserts exactly one kernel instead of searching for it. The static configuration (nouveau blacklist, unattended-upgrades blacklist, cloud-init docker group) becomes files under mkosi.extra/ rather than heredocs. linux-image-generic drags in every linux-firmware split package, none of it relevant to a headless VM guest; left at the default the modules initrd embeds all of it into the UKI, which grew to ~880MB and overflowed the 512M ESP. The initrd module set is trimmed to mkosi-initrd's curated default, which is a boot fix and not only a size one. Verified in the built artefact, not just the build log: nvidia.ko.zst and the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the nouveau blacklist, and the nvidia runtime in docker's daemon.json. * fix(images): unblock guest boot on wait-online and firstboot A guest built from these images stalled twice on the way to a login prompt, both times on a unit waiting for input that never arrives in a VM. systemd-networkd-wait-online holds network-online.target until every managed link is up, and docker.service pulls that target in with Wants=, so any image carrying docker inherits the wait whether or not it wants it. A guest whose links are slow or absent ran the unit to its full 120s default with multi-user.target queued behind it. Override it to --any with a short timeout: one link up is what a single-NIC guest actually needs. systemd-firstboot separately prompts on the console for anything the image does not already carry, before multi-user.target. With no timezone baked in, boot stopped dead at "Please enter the new timezone name" and waited for a keystroke. Setting locale, keymap and timezone at build time leaves firstboot with nothing to ask. Both are baked into the base rather than delivered as first-boot drop-ins: an image built from scratch has no cloud-init to write them. Measured on wattle with an A1000 passed through: login at 42.8s, down from ~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports "NVIDIA RTX A1000, 595.71.05, 8188 MiB". Also widen the images gitignore to output-*/ so a scratch build cannot leave multi-GB artefacts stageable. * feat(images): make the base image reachable — cloud-init, sshd, console An image built from scratch launched unreachable. The images this replaces start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a network renderer, dbus and a serial console from it; mkosi installs only what is listed, so none of it was there and nothing said so. Guests get their SSH key and user-data from IMDS via cloud-init, so without the package a launched instance has no authorized_keys at all — no error, just no way in. It also made any cloud-init-delivered config inert, including the gpu profile's docker-group drop-in. Each addition is load-bearing on its own: cloud-init consumes IMDS: keys, user-data, hostname, network openssh-server the key needs a sshd to be installed into netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is never configured, IMDS is never reached, and cloud-init falls back to DataSourceNone dbus systemd only Recommends it and mkosi installs no Recommends, so systemctl could not reach the system bus — and the EKS and ECS bootstraps drive guests with systemctl from user-data lsb-release cloud-init shells out to it; without it every boot reports "degraded" forever and the status stops being a usable signal The kernel also needs console=ttyS0 baked in. The launcher gives every guest a serial port and get-console-output reads it, but a kernel with no console= writes to none of it: the console stopped after the UEFI bootloader and every boot failure was invisible. ttyS0 goes last because Linux makes the final console= the controlling /dev/console. manage_etc_hosts is set because cloud-init renames the guest per launch but at its default never writes a matching hosts entry, so sudo warned on every call. datasource_list is deliberately NOT pinned: the guest identifies the platform from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on its own, and pinning per image is the customisation this migration deletes. Verified on wattle with an A1000 passed through: ssh in as ubuntu with spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s. * feat(images): let the guest grow its root into the volume it was given mkosi sizes the disk to its content, so the image ships with a root partition that fills the disk to its last sector and ~1.4G free. Nothing in it can expand that: cloud-init's growpart module is enabled, but the growpart binary lives in cloud-guest-utils, which is not installed and is not pulled in as a Recommends by anything in the package set. systemd-repart is not an alternative — Ubuntu's systemd package carries no repart binary, unit or definitions at all. The result was a node handed a larger volume that it could only ever use the first few GiB of, with the rest allocated and unreachable. That is fatal for a worker whose job is to pull container images. gdisk is deliberately left out despite being a cloud-init Recommends: growpart resizes GPT through sfdisk, which arrives via the hard fdisk dependency, and e2fsprogs brings the resize2fs that the resizefs module runs afterwards. Verified on wattle: the gpu-nvidia image launched with a 20G root reports 20G on df -h / (was 4.4G), via "'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with nvidia-smi still reporting the A1000 and no failed units. * feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles Adds the eks-common, eks-agent and ecs profiles, so both GPU node images build from the composed base rather than a hand-authored tree, and the three verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone. Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS node images run k3s or containerd and reach the GPU through CDI. Fusing the two was fatal for ECS rather than merely wasteful: gpu-nvidia installed containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs nvidia-ctk to generate its CDI spec. Closes three gaps the pilot left, each of which only shows up on a booted guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the modules on a headless node and CDI generation finds no GPU; mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on the toolkit packages, which the unattended-upgrades blacklist does not cover for a plain `apt upgrade`. mkosi-build.sh gains --image, naming an output and expanding it to an ordered profile list. The order is load-bearing: mkosi runs each profile's postinst in the order the profiles are passed, so docker's nvidia-ctk wiring must follow gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a healthy-looking image with no GPU wiring, so docker's postinst also fails loudly when gpu-nvidia is selected but nvidia-ctk is absent. Profiles reference build inputs where they already live via ExtraTrees= instead of copying them, keeping one copy of the helpers shared with the Alpine image. Go binaries are built on the host into a staging tree, since the builder container carries no Go on purpose: go.mod is the single pin for the toolchain and a second pin in the Dockerfile could disagree with it silently. * fix(images): install systemd-resolved so the guest has a DNS resolver mkosi installs only the packages named in Packages= plus their hard dependency closure, and systemd-resolved is merely a Recommends of systemd. So it was never installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated, because netplan renders to systemd-networkd and networkd alone does not write resolv.conf. glibc then fell back to localhost and every lookup failed with "[::1]:53 connection refused", which reads as a network fault rather than a missing package. A node that cannot resolve cannot pull a container image. The ECS GPU node booted, registered as a container instance, then failed every task at image pull. The EKS image carries the same defect but is masked by the cloud-init bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS launch path has no equivalent, so only ECS exposed it. * fix(images): name build output by image and never report a skipped build as success Two defects in the build wrapper that let the wrong bytes reach a boot test. Every composition wrote the same spinifex.raw, taken from the base config's ImageId, so an artefact carried no record of which profiles produced it and an EKS image was distinguishable from an ECS one only by which build ran last. Passing --image-id names the output after the image, and as a side effect the name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses its default input: --name is the published object and --image the local artefact, and with nothing tying the two together a defaulted input meant whatever raw happened to be in the output directory could be uploaded under the name the caller asked for. mkosi also skips the build whenever the output path merely exists. It does not compare inputs, so it never notices an edited mkosi.conf or a restaged binary, prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable from a fresh one. Builds are now always forced, and the run additionally asserts the expected raw was written during this invocation rather than trusting the exit status. The cache worth keeping is the package download volume, which --force does not touch. * feat: add gpu-amd mkosi profile for the ROCm AMI Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to stay alive for one remaining image. Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04 universe ships its own vendored rocminfo, which unpinned apt preferred over AMD's, mixing release trains within one image. Uses a systemd unit for render/video group membership instead of a cloud-init runcmd, because cloud-init's config merge takes a duplicated runcmd key wholesale from one fragment and silently discards the other, which would have dropped the docker profile's own usermod. Runtime GPU verification is deferred — no AMD hardware is available. * feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine systemd units, a templated etcd-snapshot service with daily and 15-minute timers, the addon/coredns/konnectivity assets, and the six server binaries. konnectivity-server is not an in-repo package, so it gets a pinned host-side clone-and-build rather than an entry in profile_binaries(). Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any attached volume and mulga-mgmt-net configures any multi-NIC system VM, so neither belongs behind an EKS profile that a database image would have to compose to reach them. Start-at-boot moves into preset files. mkosi runs preset-all after every postinst, so a postinst enable is discarded before the image is finished, and with no policy file the fallthrough is enable — which built an image that ran a control plane and a worker at once. Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the directory in reverse and the merge keeps the first source, so a 90- prefixed file lost the key silently. The postinst now asserts the merged result. * refactor(eks-node): make the role selector init-agnostic eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image the selector enabled nothing and the node never started k3s. Both now dispatch through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT overriding detection so either branch is testable on a host that has only one. Detection prefers rc-update over systemctl: the test suite stubs rc-update on PATH, and preferring systemctl would have silently rerouted those cases to a path they were not written for. Enable and start stay two separate passes. The server path starts mulga-eks-k3s-recovery before k3s so a recovery directive applies before the apiserver comes up, and collapsing to systemctl enable --now would interleave them. The dispatch is duplicated in both scripts rather than shared: they are single-file deliverables with no library-path convention between them. Tests cover both branches across all three roles, including that server-join still gets no k3s-first-boot, and that rc-update wins when both inits are present. * fix(eks-node): enqueue role-selected units instead of blocking on them eks-node-role.service declares Before=k3s.service, and mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a blocking systemctl start deadlocks: systemd will not run the queued job while the selector is still activating, and the selector cannot finish until that job runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie and the node wedges mid-boot with the role written but no control plane. On a booted guest the selector sat in activating for 261s until the VM was torn down. Start with --no-block on systemd. The ordering the blocking call was implicitly providing is already declared between the units themselves, so systemd sequences them correctly on its own. OpenRC runs the script directly rather than scheduling it, so that branch keeps blocking and keeps surfacing failures through set -e. * fix: escape $ in konnectivity-server ExecStart so server-count survives systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before the command runs, substituting an empty string for a name it does not know. KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes it to /run/konnectivity-server-args.env, which the wrapper shell sources at runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw it. konnectivity-server then died on flag parsing, which is why the unit logged Started and produced nothing despite --logtostderr=true and journal+console, and why the apiserver's cluster egress selector found no UDS for the life of the cluster. This is a port artefact. konnectivity-server.initd is flag-for-flag identical and shares the same prestart script, but under OpenRC the shell was the only thing reading ${KONN_SERVER_COUNT}, so the same text was correct. Escape it as $$ to defer expansion to the wrapper shell, and add lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of a profile's units. The linter joins line continuations first, since the offending reference sat mid-way through a wrapped multi-line ExecStart, and runs on the host before the container starts so a bad unit fails the build in seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd leaves it literal. Verified on env12 with a rebuilt image: TestEKS/IRSAPod and TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed out` through the egress tunnel, now pass, with zero UDS connect errors across the run. * fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw mkosi only emits the raw. A qcow2 next to it came from a previous run's hand conversion, and nothing about its name says which build it belongs to, so it survives a rebuild and silently ships the old image to a node while the raw beside it carries the new one. Observed on a rebuild that left the raw at 15:34 carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit. This is the same failure the existing freshness assertion guards against, one file over, so extend that block rather than add a second check. Refresh rather than delete: the qcow2 is what gets copied to a node and imported, so removing it would push the manual conversion, and this trap, back onto the caller. Convert via a temp name and rename so an interrupted run leaves the old file rather than a half-written one that looks current. * fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns, which systemd-networkd does not manage, so it shells out to a DHCP client. That call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not. `ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the failure surfaced as an opaque "network setup failed" with the task stuck in STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and ServiceWithELB both failed on the mkosi image, the latter as runningCount=0 because its target group targets task ENI IPs. Resolve the client at setup time, preferring udhcpc so Alpine behaviour is unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not sufficient to select it: Debian and Ubuntu's busybox ship the applet without /usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits nonzero having configured nothing. Same order and same reasoning as dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no daemon behind in the netns. Install dhcpcd-base in base, because neither client was present in any profile. That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved to nothing on Ubuntu and would have failed the first time a cross-host control-plane data NIC needed it — the S5 path the step-6 plan notes has never been exercised. The DHCP-client choice is injected so the tests assert a named platform's branch rather than whatever happens to be installed on the machine running them; without that the same test exercises different code on an Alpine CI box and an Ubuntu dev box. * refactor: retire the ec2-node profile, fold its helpers into the base image ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition had to remember to list it — eks-common even guarded on its presence and aborted the build without it. ECS did not list it, so the ECS image shipped with no by-id symlinks for hot-attached volumes at all. Neither helper is answerable at build time: any image may later be attached a volume, and any image may later be launched as a control plane. Both are inert when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its fw_cfg blob is absent), so the base image is where they belong. That deletes the composition rule, the guard, and the ECS gap in one move. Also drops ec2-node from the two eks image compositions in mkosi-build.sh. Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both carry the helpers and the udev rule, mulga-ebs-byid is enabled via multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants. * fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link mulga-vpc-mtu only won the boot race by ordering: netplan renders UseMTU=true (it does not inherit networkd's own false default), so a DHCP lease renewal at T1 could silently re-apply the advertised MTU to the link. usemtu_off() forces UseMTU=false via a runtime drop-in in the systemd-documented [DHCPv4] section for every rendered unit and reloads networkd, removing the dependency on boot ordering entirely. assert_pinned() then verifies the link and its routes actually match before exiting, so any regression is a loud boot-time failure instead of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has no systemd-networkd. * fix(images): state UseMTU=false under both the current section and the legacy alias netplan renders the DHCP settings under [DHCP] while systemd documents [DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the running systemd disagrees, which is the silent no-op the drop-in exists to prevent. * fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch credendpoint.go DNATs the ECS task credential endpoint via iptables; neither the mkosi ecs profile nor the legacy Alpine manifest installed it, so the endpoint silently failed to start and any task with a task role could not fetch credentials while the instance still looked healthy. Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only, which this platform's HTTPS AssumeRole path can never match) and adds a TaskRoleCredentials subtest that fetches real credentials from inside a task container, so a dead endpoint fails the suite instead of passing unnoticed. * fix(images): ship preset files for base, eks-common and ecs start-at-boot mkosi runs systemctl preset-all after every profile's postinst, discarding whatever systemctl enable a postinst performed. eks-server and eks-agent already carry preset files for this reason; base, eks-common and ecs still relied on the default-enable fallthrough, which happens to match today but is one upstream package change from reversing. Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net), 30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and 40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables cni-dhcp.service/socket, which containernetworking-plugins enables by default with nothing to serve under the baked host-local IPAM conflist). Removes the now-redundant systemctl enable calls from each postinst, matching the eks-server/eks-agent precedent. * fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal denied every Service clause other than the IMDS-only EC2 one. A role carrying the trust policy AWS documents for an ECS task role was unassumable by construction. Service attribution is now derived from the authenticated SigV4 caller: an assumed-role session of the account's ecsInstanceRole — the only credential the in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles in its own account. Nothing in the request body contributes, so a task holding its own task-role session cannot claim the service principal. * fix(images): remove baked SSH host keys, pin Seed= for reproducibility openssh-server's postinst bakes an identical host key pair into every build, making every guest from one build mutually MITM-able; sshd-keygen already regenerates keys on first boot (ConditionFirstBoot=yes), so the baked pair is pure non-determinism with no purpose. Also pin Seed= so systemd-repart stops assigning fresh partition UUIDs on every build. Verified on a built spinifex-ecs-node image: no ssh_host_* files ship, cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service is wired via ssh.service.wants/ (redundant regeneration path). A two-build comparison dropped content-differing files from 9 to 1 (only the UKI, whose zstd internals remain sensitive to unrelated input changes). --------- Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>	2026-07-30 12:57:43 +10:00
Julian Sommer	edf2113f54	feat: schedule RDS automated backups and sweep them past retention (#604)	2026-07-30 12:34:41 +10:00
Julian Sommer	d4d9db9e0b	feat: add DB subnet groups and DB parameter groups (#600)	2026-07-30 09:39:52 +10:00
Josh Siviour	7d2106bfb5	fix(test): reclaim fixture scratch dirs stranded in /tmp (#598) ... * fix(test): reclaim fixture scratch dirs stranded in /tmp The shared predastore fixture and the integration suite's JetStream store both outlive any individual test, so neither can use t.TempDir() - the first test to finish would delete files the daemon still has open. Both used os.MkdirTemp with nothing removing the directory afterwards, so every run stranded one, and a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and swapped tmpfs pages have no backing store to be dropped to. Add a shared sweep that reclaims a directory only when it matches both the prefix and an age no live run can reach. The NATS store also has a clean shutdown point, so TestMain now removes it outright and the sweep is left to cover the killed-run case. The predastore daemon runs until the process exits and has no such point, so the sweep is its only mechanism. * chore: repin viperblock to the dev tip carrying the scratch-dir fix Test-only upstream change, so the build output is unchanged - this keeps go.mod aligned with the submodule pointer rather than delivering a fix.	2026-07-29 14:57:50 +10:00
Julian Sommer	6eb76b439b	feat: RDS tag actions, --tags on create, and the ARN parser (#595)	2026-07-29 12:07:32 +10:00
Julian Sommer	2569601e3d	feat: implement RDS CreateDBInstance, reconciler and endpoint DNS (#594)	2026-07-29 11:32:59 +10:00
Julian Sommer	7e356b5b9b	test: cover DescribeInstanceStatus and the SDK instance-status-ok waiter (#588)	2026-07-27 13:46:16 +10:00
Julian Sommer	5b8f20b89f	test: assert ModifyVolume resize reaches the guest block device (#585)	2026-07-27 12:47:42 +10:00
Julian Sommer	95973eeb3c	refactor: migrate spinifex off legacy nats.JetStreamContext onto nats.go/jetstream (#584)	2026-07-24 16:40:57 +10:00
joshsiv-mulga	6e13a9d9f4	test: add continuous single-open churn e2e for viperblock 503/GC ... TestNPassOverwrite detaches after every pass, so viperblock's drain and GC only ever run at reopen — it cannot show whether GC reclaims superseded chunks while a volume stays open, and its serial write rate never nears the predastore rate limiter's burst. TestContinuousChurn attaches once and drives concurrent 4 MiB-aligned raw overwrites of per-writer regions for several minutes without detaching, so a sustained superseded-chunk backlog builds under one long open, spanning multiple GC ticks. The workload is bounded and self-contained: each dd writes one chunk and the clock is re-checked after every write, the loop runs as root under a single sudo bash with a process-group kill trap, and it is wrapped in a guest-side timeout. Root ownership is load-bearing — the writers open a raw device, so only a root-owned process group can reap them; a best-effort guest pkill runs before detach as a backstop so a stray writer cannot wedge the detach.	2026-07-23 14:49:01 +10:00
Julian Sommer	4803469037	fix: e2e test assertion	2026-07-23 13:56:01 +10:00
Josh Siviour	344a490a70	fix(vm): give relaunched data volumes named block nodes so detach can complete (#575) ... buildDrives emitted every non-boot data volume as a bare -drive with no id= and no node-name=, so a stop/start (or reboot/resize) relaunch left the volume anonymous: device_del/blockdev-del had nothing to address, QEMU kept the NBD socket open, and DetachVolume stalled through nbdkit's full kill grace before giving up. Add -blockdev support to the QEMU config builder and use it for non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id naming and virtio-blk-pci device arguments with AttachVolume's hot-plug path so cold-boot and hot-attach can no longer drift apart. A data volume with no persisted HotplugPort now gets one allocated and written back during buildDrives instead of landing on an unplugged bus. Also raise unmountSealTimeout past viperblockd's kill grace (it runs the seal after the SIGKILL, so matching the grace meant an unmount that used the whole grace could never be observed as successful), and bound the e2e harness's forced teardown detach by the teardown deadline so a slow detach is reported as slow rather than as an unexplained leak.	2026-07-22 20:38:44 +10:00