150 Commits

Author	SHA1	Message	Date
Josh Siviour	98ec781629	feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770) ... * feat: introduce the EBSProvider interface and control-plane metadata store Phase 1 of decoupling the control plane from viperblock. Adds a provider-neutral block-storage contract that spinifex owns end to end. ebsprovider defines EBSProvider, whose operation set and idempotency rules follow the CSI controller service, with NATS request/reply as the transport rather than CSI's gRPC. Every request and response carries an explicit schema version so wire skew fails loudly at the boundary instead of silently decoding a changed payload. The package deliberately exposes no viperblock types: provider-specific state is an opaque handle. Ships with a NATS client against the ebs.provider.v1.* subjects and an in-memory fake for tests. ebsmetadata gives spinifex its own volume and AMI documents under a spinifex key prefix, so the control plane can answer DescribeVolumes and DescribeImages without reading the storage engine's persisted state. The EC2 volume and snapshot handlers gain provider-backed paths for create, describe, modify, attach-state and delete. Provider injection is optional for now, so the existing engine path stays the default until the viperblockd side serves the new subjects and the metadata migration lands. vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol it used and is now a local helper. * feat: serve the EBS provider contract from viperblockd Gives the ebsprovider NATS client a server. Volume create, describe, expand and delete, plus snapshot create and delete, are now handled by the daemon that owns BaseDir and the mounted-volume registry, which is where viperblock engine construction belongs. Every handler prefers an already-mounted engine over opening a second one on the same volume, and expand refuses outright while the volume is mounted. Opening a second engine on a live volume is the double-writer hazard this decoupling exists to remove, so the boundary enforces it rather than trusting callers. Requests carrying an unrecognised schema version are rejected at the boundary instead of being decoded on a best-effort basis. Snapshot creation replies immediately with an operation ID and completes asynchronously on a per-snapshot subject, since a large volume will outrun the request timeout. Draining the volume stays with the caller: it coordinates with the guest and is not something the storage boundary can reach. The snapshot create subject gains an explicit create verb. Addressing it as ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject and left the wildcard unsubscribable. Publish and unpublish are not served yet; the mount and unmount logic they would front is still inline in launchService and needs extracting first. Also adds a conformance suite that runs one set of contract assertions against both the in-memory fake and the NATS client, so each side is checked against the contract rather than against the other's structs. * feat: let an operator select the EBS provider Adds an [ebs] provider key and wires the NATS provider into the EC2 volume, snapshot, image and instance services at the daemon's composition root. Until now nothing injected a provider, so the provider-backed code paths in those services were unreachable. The key is provider-neutral rather than nested under [viperblock], because it names the boundary rather than one provider's settings; a second provider will not need it renamed. An empty value resolves to "embedded", so deployed config files that predate the key keep today's behaviour, and an unrecognised value is a startup error rather than a silent fallback. Startup probes the provider's capability verb and warns when nothing answers. The daemon and viperblockd start independently, so an unanswered probe is not fatal, but without it a misconfigured node fails much later as an unexplained request timeout on the first EBS call. "viperblockd" stays experimental: under it volume metadata lives in the control plane's own document, and the leak reaper, GC and admin paths still read viperblock's config.json, so they cannot see tenant or tag data for volumes created that way. * fix: make every volume-metadata consumer provider-aware Several readers and writers of volume metadata were never given a provider branch, so under the viperblockd provider they silently used the wrong store. Each of these is a correctness bug rather than a missing feature, because the control plane had already stopped writing the document they read. Modifications had nowhere to live: ModifyVolume built its result in memory and dropped it, and DescribeVolumesModifications read it back out of viperblock's config.json, so a describe straight after a successful modify returned nothing. Volume metadata now carries its own modification record. The leak reaper read attachment and tags from config.json and wrote orphan markers to tags.json, neither of which exists for a provider-managed volume, so a leaked volume was invisible to it and any mark it did write could not be read back. Tag mutations had the same split: a tag applied after creation never reached the document DescribeVolumes reads. Snapshot deletion could not see a provider-managed clone, because a clone records its source only in the control-plane document. That let a delete strip chunks a live clone still reads, so it is fixed here rather than left for the migration. Volume enumeration and DescribeVolumeStatus were likewise scanning object prefixes that a provider-managed volume does not create. Also adds an encryption flag to the volume document, which DescribeVolumes had been reporting as false unconditionally and snapshots could not carry at all, and a list operation for AMIs so the image path can follow later. The legacy path is unchanged; its invariant tests pass unmodified. * feat: backfill ebsmetadata documents from legacy volume and AMI state DescribeVolumes under the viperblockd provider enumerates only ebsmetadata documents, so a volume created before the switch had no document and was invisible rather than merely slow to read. A per-object read fallback cannot fix enumeration, so both a backfill and a fallback are needed. Add an object-store migration kind to the migrate framework. Predastore has no conditional write, so an object-based version stamp would be last-writer-wins across nodes; RunObject stamps progress in a caller-supplied JetStream KV bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a read-then-write rather than a compare-and-swap, so correctness rests on every ObjectMigration.Run being safe to re-execute. The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the numbered config migrations: it needs viperblock's VBState schema to decode the legacy config.json, and migrate is imported by every handler that stamps a KV bucket version. It converts what it can reach and skips unreadable objects with a log, so one corrupt volume cannot hide the rest of the fleet. ebsmetadata gains the legacy read fallback for the window before the migration has run everywhere. It stays free of viperblock by taking the decode as an injected function value, wired at the composition root, and the fallback is retired by leaving it unset. A fallback read never writes the converted value back, which would race the migration. Prefix enumeration pages to exhaustion. Stopping at the first page would let the backfill stamp itself complete having converted only the volumes it happened to see, reintroducing the invisible-volume failure it exists to prevent. The migration runs from configureEBSProvider, so only when the provider is selected; backfilling while the embedded engine is still authoritative would produce documents that immediately go stale. Failure is fatal rather than advisory, unlike the capability probe beside it. This makes the switch to viperblockd one-way, which is now recorded on the config field. * feat: serve volume publish and unpublish from viperblockd PublishVolume and UnpublishVolume were the last unserved verbs on the provider contract, and they are the ones that attach a volume to a VM. Until they are served the provider path cannot boot an instance, so provider = "viperblockd" cannot be verified end to end. The mount and unmount logic was two large anonymous handler bodies inline in launchService. Rather than write a second mount path, extract each into a named function and have both the legacy ebs.mount/ebs.unmount handlers and the new provider handlers call it. Two divergent implementations of nbdkit spawning and the seal decision would be far worse than the inline bodies. The extraction relocates code without rewriting it; response publishing stays in each handler so the provider never publishes to ebs.mount.response. Publish is idempotent: a volume this node already has mounted returns its existing attachment rather than starting a second nbdkit against it, which is the double-writer hazard the boundary exists to prevent. Unpublish preserves the invariant that a failed seal leaves the volume mounted so a retry can re-attempt it, and reports an error rather than success. Both verbs are node-addressed, so they subscribe directly rather than through the spinifex-workers queue group, and are registered only when NodeName is set. There is no queue-group fallback: a subject that already names one node has nothing sensible to fall back to. Read-only attachment is refused rather than silently granted read-write, since nbdkit is started read-write or not at all. Also fixes a latent panic carried by the extracted mount path: viperblock.New returns a nil VB alongside its error, and SetCacheSize was called on it before the error was checked, so a failing New would have taken down the handler. * feat: read and write AMI metadata through ebsmetadata The backfill migration wrote ebsmetadata AMI documents that nothing read: every AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at {imageID}/config.json. Give the image service provider branches for the AMI read, write and enumerate paths, with enumeration going through ListAMIs rather than an object-prefix scan, as DescribeVolumes already does. AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a viperblock type on an interface boundary in the instance package. It now returns ebsmetadata.AMI on both paths, the embedded one converting before it returns, so the boundary names no viperblock type whichever provider is selected. The legacy-to-document mapping had accumulated three copies. Consolidate them in ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata itself stays free of viperblock, and cover it with a reflective test that fails when a field is added to either schema without being wired through. Sixteen fields copied by hand in three places would have dropped one silently. The admin image paths read whichever representation exists and write back to that same one, so a provider-managed AMI never gains a spurious config.json. FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan: missing a dependent there deletes an AMI a live volume still needs, the same class of bug as the snapshot clone path fixed earlier on this branch. That union is built by hand rather than by wiring Store.ListVolumes' fallback, which swallows per-ID read errors. This path deliberately fails closed on a transport error so a delete cannot proceed on a partial picture, and the fallback would have quietly turned that into failing open. The CLI image import now also writes the document, best-effort, so an AMI imported after the migration is not left reachable only through the fallback. * fix(ec2): resolve volume config through ebsmetadata when a provider is configured A provider-created volume has no vol-*/config.json — its metadata lives in the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy key unconditionally, so AttachVolume reported a volume that DescribeVolumes had just listed as available as InvalidVolume.NotFound. Routing through ebsmetadata.Store also puts this path behind the legacy fallback, so a volume created before the provider switch stays attachable. The NoSuchKey remap now names the key it missed; its silence made the original failure invisible in the daemon logs on every node. * test(bench): add an EC2 control-plane latency benchmark Measures the EBS control-plane operations either side of the provider boundary, so the cost of the NATS hop can be stated rather than guessed. Reports median and p95 with a bootstrap CI on the median difference, and labels a delta indicative unless that CI excludes zero. * fix(ec2): drain the volume before a provider snapshot The legacy CreateSnapshot branch drains first so the snapshot reads a current checkpoint; the provider branch did not, and viperblockd documents the drain as the caller's responsibility. Both sides deferred to the other, so an attached volume snapshotted stale data with no error raised. DrainVolume and volumeAttachment now take the fallback state and instance as strings rather than a viperblock.VolumeMetadata, removing a viperblock type from an exported signature. * feat(ec2): route the instance root volume through the EBS provider RunInstances built its own viperblock engine to clone the AMI snapshot into the root volume, which made every launch a second writer against a volume viperblockd already serves. The launch now calls the provider's CreateVolume with the AMI's snapshot and source volume, and records the result in ebsmetadata so DescribeVolumes and attach can see it. A snapshot clone needs both the snapshot ID and its source volume ID to resolve base blocks, so CreateVolume in the volume handler now sends SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected every create-volume --snapshot-id outright. The gp3 IOPS and throughput constants move to spinifex/types so the instance and volume handlers share one definition rather than duplicating literals. * feat(ebsprovider): add a bounded volume seed to CreateVolume The EFI variable store must be initialised from the launching node's firmware VARS template, byte for byte. Passing a host path would not work: CreateVolume is queue-subscribed, so any node can service it, and a foreign node's OVMF package would bind the guest's pflash region to the wrong firmware build. CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd writes them at offset 0 in the engine it already builds. The seed applies only when the create actually allocates the volume, so a relaunch never overwrites a BootOrder the guest has since written. JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes bounds the seed well inside the NATS max_payload the cluster runs with and both ends reject an oversized seed with a clear error rather than letting the publish fail. VolumeSeeding advertises the behaviour so a provider that cannot seed says so instead of silently dropping the bytes. * feat(ec2): route the EFI variable store through the EBS provider prepareEFIVolume built a control-plane viperblock engine to seed the variable store from the firmware VARS template, making every UEFI launch a second writer against a volume viperblockd already serves. It now calls the provider's CreateVolume with the template as seed bytes, so the store is allocated and written by the daemon that owns the volume. The template is still read on the launching node and shipped with the request. CreateVolume is queue-subscribed, so any node can service it, and a VARS region seeded from a foreign node's OVMF build would bind the guest's pflash to the wrong firmware. pflash refuses a VARS region that is not byte-exact, so the returned capacity is checked against the template length; a provider that rounded up to a GiB boundary would otherwise yield a guest that cannot boot. The engine path stays behind a nil-provider guard until the legacy branch goes. * feat(ec2): route image snapshots through the EBS provider snapshotRunningVolume and snapshotStoppedVolume each built a control-plane viperblock engine to take the snapshot backing a new AMI, making CreateImage a second writer against a volume viperblockd already serves. Both now call the provider's CreateSnapshot with the volume's provider handle. The running path keeps its drain, which is what makes the snapshot current rather than one predating a guest-triggered write. The stopped path needs no drain, and the daemon already prefers a live mounted engine over opening a second one, so it handles both cases itself. getVolumeConfig read the legacy config.json, which a provider-created volume does not have, so every CreateImage against one failed. It now reads the ebsmetadata document under a provider, the same way attach does. * fix(daemon): only the owning node creates an image for a stopped instance handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which only the node holding the VM passes, so a running instance is imaged once. A stopped instance lives in the cluster-replicated KV instead, which every node reads successfully, so all of them ran the full snapshot and AMI creation. The caller saw the first reply and the rest leaked: on a three-node cluster one create-image produced three AMIs and three snapshots. A stopped instance records its owner in LastNode, so a node that is not the owner now declines. Gather tallies error frames and keeps waiting, returning only on a success, so declining cannot preempt the owner's reply. An instance stopped by a build that never set LastNode elects an owner through the KV's compare-and-swap, since having every node decline would make create-image fail permanently for those records. * perf(ec2): serve DescribeVolumes by ID from the volume documents The provider branch called ListVolumes unconditionally and filtered in memory, so asking for one volume read and decoded every volume document in the bucket. The non-provider branch already had the equivalent fast path. Explicit IDs now fetch their documents directly. The tenant check moves with them: a document belonging to another account reports not-found rather than being returned, matching what the enumeration filter did. Measured against a memory object store, the fast path stays flat near 20us regardless of bucket size while the enumeration grows linearly, reaching 4.6ms at 500 volumes. * chore(lint): ban the viperblock import from the EC2 control plane The control plane reaches block storage through ebsprovider now, but nothing stopped a handler from importing the engine again. That is not hypothetical: the instance handler first imported viperblock for a config struct alone, and the engine construction followed thirteen days later. depguard denies the import across handlers/, which matters because viperblock.New starts a chunk uploader — every construction is a writer against a volume viperblockd already serves. A types-only import is denied too, since that is exactly how the last one started. The four files still carrying the pre-provider engine path behind a nil-provider guard are exempted by name. The exemption list is the remaining work: it shrinks to nothing when that path goes. * fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot CopySnapshot wrote a control-plane snapshot document under the new ID and nothing else. The copy described correctly and CreateVolume against it succeeded, so it looked like it worked, but the new ID had no viperblock snapshot metadata behind it and the volume failed when the guest attached it. Producing the metadata means re-sealing it under the destination ID. On an encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the sealing helpers are unexported, so spinifex cannot write it. Copying only the unencrypted form would be dead code on any cluster with encryption enabled. Refusing with UnsupportedOperation fails at the call the caller can act on rather than at attach time. Source lookup and the ownership check still run first, so a bad ID or another tenant's snapshot reports that instead. * test(ec2): give handler test fixtures their own legacy config.json shape Handler tests built the legacy config.json fixture by importing viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly. Mirror that shape in ebsmetadata/vblegacy, next to the production code that already reads it (ebsmetadatabackfill), and switch the fixture-only usages across the volume/snapshot/image handler tests to it. A byte-for-byte equivalence test against the real viperblock types guards the mirror from drifting out of sync. Call sites that hand a viperblock type into putVolumeConfig, mergeVolumeConfig, or buildVBConfig are untouched, since those functions still name the real type in their signature. * feat(ebsprovider): add CopySnapshot to the EBS provider contract Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy NATS subject, MemoryProvider and NATSProvider implementations, conformance coverage) and a viperblockd handler that duplicates a snapshot's metadata via viperblock.CopySnapshotMeta on the source volume's VB. * feat(ec2): serve CopySnapshot through the EBS provider CopySnapshot has been refusing with UnsupportedOperation because a copy needs its own snapshot metadata on the backend under the new ID, and on an encrypted volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the control plane cannot re-seal it. The provider contract now carries a copy operation, so the storage engine does it. The provider path generates a new snapshot ID, has the provider duplicate the backend metadata under it, then writes the control-plane document. Rollback mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot rather than leaving one nothing references. Ownership is still checked against the source before anything is created. The no-provider path keeps refusing. It has no way to produce the backend metadata, and the copy it used to write described fine and then failed at attach — a resource that fails as far as possible from the call that made it. Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot reached the provider rather than existing only as a control-plane document. That assertion is what catches the old behaviour. * refactor: remove the embedded EBS engine path from the EC2 control plane viperblockd is now the only EBS provider. The pre-provider path that constructed a viperblock engine in-process is deleted, and with it every github.com/mulgadc/viperblock import under spinifex/handlers/ (tests included). The depguard ec2-control-plane rule drops all four file exemptions and the !$test carve-out, so the rule now reads simply as "nothing under spinifex/handlers/** may import viperblock". viperblock.VolumeConfig was also a data carrier on the provider path. It is replaced by the spinifex-native ebsmetadata.Volume rather than the vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is decoding old objects, while ebsmetadata.Volume already carries every field the call sites read, so the conversion step disappears instead of being reimplemented against a legacy shape. The embedded-only helpers (volumeConfigWrapper, putVolumeConfig, mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached, deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json prefix scan) had exactly one production caller chain each, all through a nil-provider branch, and are removed. The snapshot-dependency guard on DeleteVolume is provider-agnostic and is kept, now running on the provider path where it previously did not. config: viperblockd is the default and an empty Provider resolves to it. An explicit provider = "embedded" is rejected at config load rather than silently upgraded, because starting anyway would migrate a node's volumes to ebsmetadata one-way without the operator asking. The generated spinifex.toml template emits viperblockd. * fix(images): write the imported AMI document from the config the import wrote ImportDiskImage creates the AMI's snapshot and records it on its own *viperblock.VB, which viperblock.New allocates fresh, so the value lands only in {imageID}/config.json. The import command was persisting the ebsmetadata document from its pre-import manifest copy, whose SnapshotID is still empty. Store.GetAMI only consults the legacy config.json fallback when the document is missing, so that empty-SnapshotID document permanently shadowed the real value and every launch from an imported AMI failed the zero-copy clone check with ServerInternal. Rebuild the document by reading back the config.json the import just wrote. A missing or unreadable config is skipped rather than written from the stale manifest: no document still resolves correctly through the legacy fallback. * test(e2e): read NODE from spx get vms instead of VCPU in the spread check spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to field 8. The spread placement check still read field 6, so every instance reported its vCPU count instead of its node, the unique-node count always collapsed to 1, and the check failed on clusters that had spread correctly. Read field 8 and strip the ANSI colour codes wrapping each cell. * feat(viperblockd): add a --debug flag so viperblock debug logging is reachable viperblockd hardcoded Info, and viperblock's VB.logger() falls back to slog.Default() because vb.log is only ever set by SetDebug. That left the engine's debug records, including the chunk uploader's start and disabled lines emitted during New(), unreachable in a deployed environment. Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG rather than the bare debug key, which several commands share. Also sets the Config.Debug field, which existed but was set only by tests. * fix(viperblockd): release the chunk uploader when openVolumeVB fails viperblock.New starts a chunk uploader and a WAL syncer before the backend is touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the caller while leaving both running with no handle to stop them. Repeated backend failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders on a 30s ticker for the life of the process. Adds invariant tests for both halves of the ownership rule: a mounted volume's snapshot create and copy reuse the live engine and log zero volume opens, an unmounted volume opens exactly one, and a failed open leaves no uploader behind. The open counts are read from the same log record a production alarm watches. * fix(ebsprovider): route requests to the node that owns the mount Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one spinifex-workers group, so NATS delivered each request to a random node. Only the node holding the mount can reuse the live engine through findMountedVolume; any other node constructed a second viperblock engine, with a live 30s chunk uploader, against a volume nbdkit was serving on a different host. Measured on a three-node cluster, this was roughly two snapshots in three. Rather than resolve the owner, let the owner answer. A node subscribes ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts a volume and drops it on unmount, mirroring the per-volume ebs.config subscription. Clients try that subject first and fall back to the queue group only on ErrNoResponders, which means nothing is mounted and opening an engine is correct. A timeout deliberately does not fall back, since the owner may still be running the operation. Applies to snapshot create and copy, expand and describe, the four verbs that reuse a mounted engine. Delete keeps queue dispatch. * fix: rebuild viperblockd mounted-volume registry after a restart Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a viperblockd restart while a VM keeps running leaves nbdkit serving a volume the daemon has forgotten. Owner routing then infers from ErrNoResponders that nothing is mounted and falls back to the queue group, which opens a second viperblock engine, with a live chunk uploader, on another node. The same empty registry also defeats handlePublishVolume's double-mount guard, so a republish could start a second nbdkit. Discover surviving nbdkit processes from /proc at startup and rebuild the registry from them, following orphan_scan.go's precedent: match comm exactly, parse the argv for volume=<id> and --unix/-p, and adopt a PID only when nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or killed; anything that fails corroboration is logged and skipped. Recovery runs before the first subscription so no handler can race it. Extract mountVolume's engine construction into constructMountedVB so mount and recovery cannot drift, and release the WAL syncer viperblock.New starts when construction fails after it. * fix: corroborate recovered nbdkit processes from argv, not a pidfile nbdkit is launched with -f (required for the Go plugin) and never writes the --pidfile it is given, so no such file exists on any node; the daemon's own /tmp is a PrivateTmp namespace besides. Corroborating against it rejected every genuine mount, making recovery a no-op in production. Corroborate against what the kernel actually has: the process must serve our plugin, name this daemon's base dir, and still hold its socket. Adopt a volume once, so two live nbdkits for it leave one registry entry and a warning rather than two conflicting ones. * feat: add a second EBS provider backed by qcow2 and qemu-nbd The decoupling was justified by swappability that nothing demonstrated: the conformance suite ran against MemoryProvider and against a NATSProvider wired to a MemoryProvider, so both arms were the same implementation behind different transports. A fake is not a second implementation. Add three pieces. natsserve serves the ebs.provider.v1.* contract from any EBSProvider, depending on nothing but the interface, and replaces the test-only server twin so conformance now exercises production code. The conformance suite moves to its own package and is exported so implementations outside ebsprovider can run it, with no assertion changed. qemunbdd is the second implementation: one qcow2 file per volume, qemu-nbd for publish, real snapshot copies instead of chunk references, and every exec behind a runner seam so its tests need no qemu. Owner-subject routing is deliberately not served neutrally: it needs a per-node mounted-volume registry the interface does not model, so it stays in viperblockd and a neutral server's clients correctly fall back to the queue group. * test: run the conformance suite against the qcow2 provider The suite passes unmodified against an implementation sharing no code with viperblock, both in-process and through natsserve over the real wire contract, which is the swappability claim the provider boundary was built to support. Two real defects it caught. ExpandVolume and GetVolume inspected the image before consulting the publication map, so qemu-img could not open a volume qemu-nbd held the write lock on and the refusal surfaced as an internal error instead of ErrVolumeInUse; qemu-img info now runs with --force-share, so describing a volume no longer depends on whether it is attached. Publish also failed with qemu-nbd's raw exit status when the socket path exceeded the 108-byte AF_UNIX limit, and now reports that reason itself. * feat: make the EBS provider selectable by config Add the qemunbd selector alongside viperblockd, a spx service command that serves the qcow2 provider over natsserve, and service registration. Config validation accepts the new value everywhere viperblockd is accepted, while embedded stays rejected by name. The daemon logged a hard-coded viperblockd constant as the active provider rather than the resolved one, in configureEBSProvider and in both of probeEBSProvider's lines; all three now report what is actually configured. Both daemons answer the same subjects, so exactly one may run on a cluster. * test: add a live lifecycle check for the qcow2 provider Drives create, publish, write, snapshot, restore and read-back entirely through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a cluster node, so the client under test is the same shape the control plane uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE is set, so it stays out of preflight. * fix: publish NBD URIs in the form standard clients accept PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever consume what the EBS provider boundary hands back. Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit all accept. ParseNBDURI now takes both that and the TCP form, and still accepts the legacy nbd:unix: syntax for state written by older daemons, so a conformant third-party provider's URI is no longer rejected on the way in either. Verified against qemu 10.0.11: qemu-img and a real -drive option string both open the URI form, and qemu reports both forms as nbd+unix://?socket=, so it already treats the URI as canonical. * test: validate published exports with libnbd's client tools Every other assertion about the EBS provider goes through our own Go client, so nothing has ever checked whether the export we publish is usable by an NBD client that knows nothing about this codebase. That is the layer neither the conformance suite nor a CSI-shaped contract can reach. RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume: the URI connects, the export advertises its size and block-size constraints, data written externally reads back byte for byte, a ReadOnly publish sets the read-only transmission flag, and UnpublishVolume tears the export down. It skips when libnbd-bin is absent rather than passing vacuously. Wired to qemunbdd, where all five failed before the URI fix. The publish assertion in the main suite now requires a dialable URI instead of merely a non-empty string, which is what let the wrong form through. * test: run the external NBD client suite against a live viperblockd The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit export is the comparison worth having: one external client, two providers it cannot tell apart, and every difference is a candidate leak in the boundary. RunNBDClientSuiteWithConfig takes the node name and volume prefix a live cluster needs, and publishForNBD now unpublishes and deletes what it created so a run against a real cluster leaves nothing behind. Subtests dial a normalised URI so a provider emitting the legacy syntax still gets its export tested rather than every subtest failing on the same string; whether the URI itself is standard is now its own subtest. Adds a reconnect check, since publication holds the volume rather than any one connection. It retries the real operation because a cheaper liveness probe is itself a connection and re-triggers whatever refused it. * feat: make the conformance suite capability-driven The suite asserted a fixed Capabilities literal, so it could only ever pass against a provider advertising exactly that set. Every implementation had to match the suite rather than the suite testing the contract. RunSuite now reads GetCapabilities from the provider under test and gates optional behaviour on what it advertises, and the fixed literal is demoted to ReferenceCapabilities, a convenience for callers constructing a MemoryProvider. Add ReadOnlyPublish to Capabilities. A provider that leaves it false must refuse a ReadOnly publish rather than hand back a writable export, which is what viperblockd already does; it now has a way to say so up front. Add ErrUnsupportedCapability and the unsupported_capability error code. A request for optional behaviour a provider does not advertise is well formed, so classifying it as invalid_argument told the caller to fix a request that was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly. Declared capabilities now match observed behaviour: qemunbdd advertises SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises neither: the nbdkit Go binding exposes no extents callback, and the mount path never passes -r. The external NBD client suite gains a sparse-extent subtest that checks the export's advertised contexts against the capability, so SparseExtentReporting is falsifiable from outside rather than taken on trust, and its read-only subtest now branches on the capability instead of assuming refusal. Cover the new branches with a MemoryProvider advertising no optional capabilities and one advertising OnlineExpansion, the branch no provider we ship exercises. * refactor: rename SourceVolumeID to SourceSnapshotVolumeID CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd required it whenever SourceSnapshotID was set and used it to resolve the snapshot's chunk references, so there it named the snapshot's origin volume. MemoryProvider ignored it entirely. qemunbdd implemented it as an independent volume-to-volume clone, which is what the name and doc comment implied. The conformance suite could not catch the divergence because the reference provider had no branch for the field at all. The contract already has a field meaning "the volume this snapshot came from": Snapshot.SourceVolumeID, which every provider populates. So the request field is the same fact travelling back the other way, and its name was the problem. Rename it to SourceSnapshotVolumeID, document it as the source snapshot's origin volume, and state that it is required with SourceSnapshotID and meaningless without it. All three providers now enforce one rule: a source snapshot must be accompanied by its origin volume, or the request is invalid_argument. MemoryProvider additionally rejects an origin that does not match the snapshot's own SourceVolumeID, so a wrong name fails instead of resolving blocks against a volume the snapshot never came from. qemunbdd loses its volume-to-volume clone branch, which implemented behaviour the contract no longer offers and no caller ever requested. The suite pins both the requirement and a full create-snapshot-restore round trip, so the field is exercised end to end rather than only on the error path. * feat: make the conformance suite runnable against a live provider The suite had only ever run against MemoryProvider and qemunbdd, both of which are discarded with the test binary. viperblockd, the provider production actually uses, was the one implementation the suite never judged, because the suite assumed each subtest got fresh state: hard-coded volume IDs collide with an earlier run's leftovers on a store that keeps them, and a hard-coded node ID addresses a node that does not exist. SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records every volume, snapshot and publication the suite creates and tears them down in dependency order, logging rather than failing so a teardown problem cannot mask the assertion a subtest was making. Two new tests cover this by running the whole suite twice against one shared provider instance, which passes only because cleanup works, and once per prefix to cover concurrent runs sharing a daemon. TestLive_ViperblockdConformance wires this to a real cluster behind SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite. * fix(viperblockd): close the four contract gaps the live conformance run found Unpublishing a volume this node has no mount for now succeeds instead of returning not_found, matching the legacy unmount path: nothing mounted here is already the state the request asks for. CreateSnapshot refuses a snapshot ID that already exists against a different source volume with already_exists, rather than overwriting it. DeleteVolume refuses a published volume with volume_in_use instead of tearing the mount down, on this node or on whichever node answers the volume's owner subject. The export and the guest writing through it survive the metadata deletion, so the old behaviour surfaced later as corruption. CreateVolume reads the source snapshot before writing the volume, returning not_found when it is absent and invalid_argument when it was taken from a different volume than the request names. Previously both produced a blank volume and a successful response. CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument now that it carries a sentinel. * feat(ebsprovider): pin the source snapshot's origin volume in the contract The suite only required source_snapshot_volume_id to be present, so a caller naming a volume the snapshot never came from was accepted everywhere. The snapshot records its own origin, so this is checkable: creating from a snapshot with a mismatched source volume is now invalid_argument for every provider, enforced by qemunbdd against its snapshot metadata and by viperblockd against the snapshot's persisted SourceVolumeName. * test(ebsprovider): run the contract suite against viperblockd in CI CI installed neither qemu nor nbdkit, so every conformance test that needs a real implementation skipped silently and MemoryProvider was the only provider the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails instead of skipping. TestViperblockdConformance stands the provider handlers up over an embedded NATS and a real predastore and runs the same suite against them, which is what caught CopySnapshot accepting a foreign volume's snapshot on unencrypted volumes. * feat(viperblockd): honour read-only publish and advertise block size constraints ReadOnlyPublish was advertised false and read_only=true was rejected, so the capability was true only by omission. nbdkit -r now starts the export when the request asks for it, which sets the NBD read-only transmission flag and makes the plugin refuse writes. Access mode is fixed when nbdkit starts, so a republish asking for the other mode is refused with VolumeInUse rather than handed the running export. Block size constraints were absent entirely: the nbdkit Go binding has no block-size callback, so the plugin cannot advertise them. The blocksize-policy filter can, without a binding callback. Minimum 1 because viperblock's WriteAt handles unaligned requests with a read-modify-write, preferred 4096 to match its block size, maximum 32M to match what qemu-nbd advertises. * test(conformance): judge sparse reporting by the allocation map, not the context The subtest asserted that base:allocation was offered. nbdkit offers that context whether or not the plugin can compute extents, answering 'all allocated' when it cannot, so the assertion passed for a server with no extent support at all — it could not fail. What separates the two is the answer: a volume with nothing written to it is entirely holes to a server that really tracks allocation. Read the map with nbdinfo --map --json and compare holes against the advertised capability. qemunbdd reports holes and advertises true; viperblockd reports none and advertises false. * test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime The predastore fixture is one daemon per test binary, but its cert was exported with t.Setenv, which reverts when the first test that called Start finishes. In-process clients did not notice because crypto/x509 caches the pool behind a sync.Once, loaded while the variable was still set. Child processes did: nbdkit spawned by a later test inherited an environment with no SSL_CERT_FILE and could not verify the daemon, so the whole NBD client suite failed when the package ran as a whole while passing when run alone. Also install libnbd-bin in CI, so the NBD client suite runs there instead of failing on missing tools. * ci: install nbdkit-plugin-dev so the viperblock plugin can build * test(volume): give the snapshot fixture an owner CreateVolume now fails closed on a snapshot whose owner it cannot match, including an unset owner_id. The fixture predates that and wrote none, so the provider test failed with InvalidSnapshot.NotFound before ever reaching the provider. * test(integration): give the volume harness a real provider The control plane no longer builds volumes itself, so CreateVolume in the integration harness failed with "no EBS provider configured": the harness wired a VolumeServiceImpl with nothing behind it. It now serves viperblockd's provider subjects against the same predastore fixture, so the blocks are still written by real viperblock to a real daemon. Three persisted-state assertions moved with it. TenantID, VolumeType and AvailabilityZone are control-plane facts, held in the ebsmetadata document rather than viperblock's own state, so the storage oracle now asserts their absence and DescribeVolumes -- whose answer is account-scoped -- is what checks them. * refactor: route image import through the EBS provider and drop the legacy metadata path The control plane could still only run against viperblock: the AMI import and the ochre weights import both linked the storage library directly, and the ebsmetadata store kept a read fallback onto viperblock's config.json. Import now goes through the provider contract with no new verb: CreateVolume, PublishVolume, a bulk copy over the published NBD export with qemu-img, then UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS, whose request size the contract caps far below a disk image. The migration scaffolding goes with it. ebsmetadata documents are now the only thing the control plane reads: no legacy prefix scan, no config.json fallback, no backfill on daemon start. config.json belongs to the provider, which is the single writer of it. ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is right for DescribeImages, where one corrupt document must not hide the fleet, and wrong for a name-uniqueness check, which cannot rule out a name it failed to read. spinifex and cmd no longer import viperblock outside services/viperblockd and testutil/vbscan. * feat: add a volume enumeration verb so a lost document cannot strand blocks Making ebsmetadata documents the sole index of volumes left one failure with no recovery: a volume whose document is gone disappears from DescribeVolumes, from the leak reaper (which enumerates documents), and from the provider, whose contract had no verb that asks what it holds. ListVolumes closes that. It is capability-gated on VolumeEnumeration and paginated, following CSI's optional LIST_VOLUMES rather than inventing a shape. Refs are deliberately not Volumes: enumeration answers what exists, and a provider must not open every volume's engine to say so. Each implementation reads what actually exists rather than what it remembers. qemunbdd walks its volumes directory, not its in-memory map, which does not survive a restart; viperblockd walks the bucket's top-level prefixes. A volume the provider has forgotten is precisely what a caller is looking for. The consumer is a read-only orphan report: provider volumes minus documented ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is excused only when its base volume is documented, so a stranded pair is still reported. Listing documents stays strict, because reporting a volume as orphaned on the strength of a document that failed to decode would point an operator at live data. * fix: stop the orphan scan accusing storage of holding things it does not A live env19 run reported three orphans against a healthy cluster: an AMI's backing volume, a snapshot, and the cluster key store. All three were wrong, and a report that cries wolf on a clean cluster is worse than no report. Enumeration treated every top-level bucket prefix as a volume. Snapshots sit beside volumes at that level and have a config of their own, so presence of a config cannot separate them; the snap- name is what does, which is already how the AMI removal path reads them back. keys/ and spinifex/ are not volumes at all. An AMI's blocks are a genuine provider volume, so enumeration keeps reporting them. What was missing is on the reading side: an AMI is recorded as an image document, not a volume one, so the scan now treats a documented image as known. Listing images is strict for the same reason listing volumes is — unreadable and absent are different answers, and only one means orphaned. * fix: let an orphaned image's blocks be reclaimed The orphan report tells an operator that reclaiming a stranded volume is their decision, and images remove --force is the tool it points them at. On env19 that tool failed on exactly those volumes: removal drops the ebsmetadata document first, and predastore answers NoSuchKey for a document that is already gone, which is the definition of an orphan. Deleting the document stays first and stays fatal. That ordering is the barrier that stops a launch landing on blocks about to be deleted, and an AMI still advertised after its blocks are gone is the worse failure. Only an already-absent document is now tolerated: the barrier is satisfied, and the blocks are the sole thing left to reclaim. S3ObjectStore.DeleteObject normalises a missing key the way GetObject already does, so this is recognised rather than matched on error text. The obvious unit test passed without the fix, because the in-memory store is silently tolerant on delete and real S3 answers 204 — neither behaves like the backend this runs against. The test now drives a store that reports the miss, and fails without the fix. * feat: add per-verb OTel spans to the EBS provider contract The ten provider verbs cross a single NATS seam and nothing measured it. The only spans on the storage path were the legacy ebs.mount/ebs.unmount pair, so control-plane storage latency could not be attributed to a verb or compared across provider implementations. NATSProvider.request is the one client-side funnel every verb passes through, including both legs of an owner-subject fallback, so a single span there covers the whole contract and measures what the control plane experiences regardless of which provider answers. Span names come from a closed verb set rather than the subject. Owner and snapshot-create subjects embed a volume ID and the mount/unmount subjects embed a node ID; naming spans after them would give every volume a span name of its own. The IDs move to attributes and an unrecognised subject collapses to a single unknown verb. Trace context rides the message headers, so the serving provider joins the caller's trace instead of rooting one of its own. The carrier is a copy of the one in spinifex/utils rather than an import: that package pulls in the aws-sdk and the cluster key store, which the wire contract must not depend on. Responses carry no shared accessor for their error field, so both sides re-read just that field from the encoded reply, and only when the span is recording. Without it a failed verb looks like a successful round trip. viperblockd serves the contract with its own handlers rather than natsserve, so its handlers now take the per-message context and its publish/unpublish pair drops the subject-named consumer span it opened for itself. * feat: add null and hostile providers, and a per-verb bench suite The provider boundary made it possible to put something other than viperblock behind the control plane, and nothing yet did. nullprovider is the floor. It is not a new type: MemoryProvider already answers every verb from a map and natsserve already serves any EBSProvider, so the package names that composition and states what it measures. On loopback with no storage behind it a round trip costs around 195us against 74ns in process, which is the seam's own price and is not attributable to any storage engine. conformance.RunBenchSuite prices one round trip per verb and drives every implementation through the same calls in the same order, so two providers' numbers are comparable. It measures control operations only: guest I/O never crosses these verbs. hostile wraps any provider in one that misbehaves. Faults derive from the seed and a per-target sequence rather than a global counter, so concurrent callers on different volumes cannot reorder each other's faults and a failure found deep into a soak run replays exactly. Error and error-after-work are separate faults on purpose: the second reports failure after the work has already happened, which is how an orphan is made, and is what makes an enumeration scan the right oracle for a soak. Enumeration itself is never faulted, since an oracle that can be told what to say cannot falsify anything. The soak asserts that every volume the provider still holds is attributable to an injected fault. A leftover with nothing behind it is the control plane losing track of storage on its own. Two bugs were found by falsifying the new tests rather than by running them. The FNV-derived draw clustered badly enough that two fault bands were never reachable in 1200 draws, so the draw now goes through a SplitMix64 finalizer. And the first version of the lie test was worthless: MemoryProvider returns copies, so a lie could not corrupt it whether or not the injector copied, and the test now drives a provider that hands out pointers into its own state. * test: bench the provider contract against qemunbdd and a live viperblockd RunBenchSuite was only ever pointed at the null provider, so the numbers priced the seam and nothing else. Point it at the two real implementations. BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a provider that stores nothing and wrong against one that does, so qemunbdd and viperblockd are both measured at 64 MiB. liveProvider splits into a testing.TB flavour. Benchmarks need the same connection and the same node and cannot be handed a *testing.T. * fix(viperblockd): carry the trace context into every viperblock engine call Both S3 clients only emit a span when the request already carries one, so calling the no-context Init, LoadState, SaveState, LoadBlockState and Close left the whole engine open untraceable. Every provider verb traced as one flat server span, and the 342ms describe measured on env19 could not be attributed to anything. viperblock already exposes the Ctx variants and Backend already declares InitCtx, so this is threading, not new instrumentation. openVolumeVB, openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine, copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and mount/unmount keep the context their own span derives instead of dropping it. completeCreateSnapshot gets WithoutCancel: the background half of accept-then-publish has to outlive the request it was accepted by and still be attributable to it. The per-volume ebs.config handler opens a consumer span like the other legacy subjects, so a config update is no longer the one path with no trace at all. * perf: describe a volume without opening an engine on it describeVolumeEngine fell through to openVolumeVB for any volume not mounted on the node serving the request, which constructed a full viperblock engine to read one integer. On env19 that decomposed as a 22 ms reachability list, a 3 ms state read and a 16 ms PutObject of the volume's own config.json — the write coming from the SeqNum bootstrap a read-write open performs. Use viperblock's read-only state load instead. A describe is now the 3 ms read: no background goroutines, no reachability probe, no SeqNum window claimed, and no write issued by a worker that does not own the volume. Falsified: restoring the openVolumeVB call fails TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the list, with the fake backend answering each so neither assertion hides behind the other. * perf: share one HTTP client across every viperblock backend Four sites built the same S3Config by hand, each leaving HTTPClient unset, so every engine open and state read got its own connection pool and paid its own TLS handshake. That was invisible while Backend.Init issued a whole-bucket list — the list paid for the connection and the state read rode it — and it became the dominant cost once describe was reduced to a single request. Route all four through cfg.volumeS3Config, which builds the client once. Falsified: dropping the HTTPClient field takes TestDescribeVolume_ReusesOneConnection from one connection to three. * perf: wait for nbdkit to be ready instead of waiting a fixed second mountVolume spawned nbdkit and then ran a confirm loop until a one-second deadline passed. The loop only broke early when the process exited, so the fast-failure path worked and the success path always cost the full second. That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so the timings never contradicted the assumption that it was process spawn. Wait for the endpoint to accept a connection instead. That is also the question the caller needs answered, since it is about to hand the URI to something that will connect to it — where 'still alive after a second' only correlates with it. A nbdkit that exits during startup still fails the mount immediately, and one that never comes up now fails on a deadline rather than being assumed healthy. Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness timing assertion and the fast-failure one. * test: measure the data path, not just the control verbs The conformance benchmarks price create, publish and describe. Guest I/O never crosses those verbs, so none of those numbers say anything about what a guest experiences. This drives the published export with qemu-img bench, which sits on the same block layer virtio-blk does. It asserts only that the I/O completes and logs a table: what a given store is capable of is not a contract, and pinning a number here would only produce a flaky test. Read profiles stay inside the region the write profiles covered, or they would be timing how fast the export answers with zeroes, and one write profile flushes, because an unflushed write measures how fast the WAL accepts data rather than how fast the store takes it. Profiles are spaced apart because the plugin tears an engine down per disconnect and removes the volume's local directory as it goes, so a connection opened while the previous one is closing fails its handshake. * test: run data-path profiles back to back The spacing was a workaround for a volume lock that did not hold across close. With that fixed, consecutive profiles are worth more than the workaround was: each is its own NBD connection, so a run that completes has also shown that an arriving connection waits out the previous one's close rather than racing it. * fix: refuse a delete that cannot establish whether a volume is published volumeMountedElsewhere returned false on every failure path, including a request timeout, so a busy or partitioned owner was indistinguishable from no owner at all and the delete proceeded. Deleting a published volume leaves the nbdkit export and the guest writing through it alive over storage that is going away. Report the three outcomes separately. NATS distinguishes no-responders from no-answer, so only the former means unmounted; an unanswered probe is refused and the caller may retry once the owner is reachable. * feat: exclude a second viperblock engine with a JetStream volume lease Nothing in the stack stopped two viperblock engines opening one volume from different nodes. For an encrypted volume that is not just a consistency risk: the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte, and the SeqNum window claim is a read-then-write with no CAS, so two engines opening concurrently issue overlapping nonces under the same key. Every engine open now takes a lease first, keyed by volume in a JetStream KV bucket. A lease is a KV create, which JetStream serialises, so a second claimant loses instead of both winning. The revision that won is the generation: monotonic across the bucket, so a later opener always carries a higher one than the writer it displaced. Entries carry a TTL and a renewal loop, so a node that dies mid-mount does not wedge its volumes forever. Losing a lease is logged, not enforced. Until the object store can refuse a stale writer's PUT, failing live guest I/O on lease loss would add a failure mode without stopping the zombie it is aimed at. A daemon with no lease store refuses to open rather than opening blind, so nbdkit recovery binds the store before it adopts surviving exports. Repeat acquisitions on one node share a lease: the unmount seal opens a detached engine while the mount being torn down still holds it, and cross-node exclusion is what the lease is for. A restart reclaims this node's own stale entries, since the exports that outlived the daemon are still running. * refactor: consume masterkey from bluebottle in the test this branch added * feat(ebsprovider): classify refusals a caller may retry Every ErrorCode said what went wrong and none said whether the same request could succeed later, so a caller could not tell a permanent refusal from a transient one. Transient conditions shipped as internal, which tells a caller to give up when it should retry. Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(), which callers branch on instead of on the code itself. DeleteVolume's "could not establish whether the volume is published" refusal moves off internal onto it: the owner that did not answer may answer later. TestErrorCodeRetryable names every code, so adding one without deciding its retry meaning fails rather than silently defaulting to permanent. TestClassifyError pins the sentinel-to-wire-code mapping, which had no coverage at all; a sentinel missing from classifyError falls through to internal and loses both the reason and the retry signal. * fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for Four providers advertised the capability and nothing consumed it: there is no clone verb on the interface. CreateVolume's SourceSnapshotID restores from a snapshot and CopySnapshot duplicates snapshot metadata; neither is a volume clone, and the field documenting SourceSnapshotID says so explicitly. CSI expresses clone as a volume_content_source with a volume source, which this contract does not implement. Advertising it told a caller a feature was available and then gave them no way to use it. Removed rather than backed by a method: adding a clone verb is a feature, not a contract fix, and the capability can come back alongside it. The Capabilities doc comment now states the rule, and qemunbdd's stale comment about sparse extents is corrected while there. * feat(ebsprovider): add ListSnapshots so snapshots are reconcilable The contract could create, copy and delete snapshots but never say which ones it held. That left the control-plane metadata document as the only index of them, so losing one stranded the data it pointed at with no way to find it again. This is the same exposure ListVolumes closed for volumes, and it stayed invisible because ListSnapshots is optional in CSI too. ListSnapshots pages exactly as ListVolumes does, gated on a new SnapshotEnumeration capability. SnapshotRef is deliberately not a Snapshot: enumeration answers what exists, and reading a snapshot's size or state can cost far more than listing it. viperblockd walks the bucket's snap- prefixes, sharing the top-level walk with ListVolumes. It leaves SourceVolumeID empty because the link lives inside the snapshot's own encrypted state and reading it would mean a fetch per snapshot listed. qemunbdd walks its snapshots directory rather than its in-memory map, so a snapshot that outlived the process still appears; its SourceVolumeID is empty in that case for the same reason CreateSnapshot refuses to reuse the ID. Five conformance subtests cover it, and they run green against MemoryProvider, qemunbdd and viperblockd over a real predastore. * feat(ebsprovider): state the exclusion guarantee in the contract A caller could not tell which providers prevent two writers from holding one volume, at what scope, or for how long. The guarantee existed in viperblockd and was invisible from the outside, which is why the conformance suite had no concurrency dimension at all. Capabilities.Exclusion states it, and deliberately separates three questions the contract used to run together: how far exclusion reaches (none, node, cluster), whether a dead owner's claim ever expires, and whether a writer that lost its claim is actually stopped. That last one matters most and is the least flattering: exclusion refuses the second opener, fencing stops the first. viperblockd advertises cluster scope with a 45s TTL and FencesLostClaim false, because a partitioned node whose lease lapsed keeps writing until the object store can refuse it. Four new conformance subtests. Two required a second viperblockd node in the harness: cluster exclusion cannot be exercised by one node, because the second publish goes to a subject nobody serves and times out rather than being refused. Both nodes share the NATS the leases live in and the bucket the volumes live in, so a refusal comes from the lease. Standing that second node up immediately found a real gap, in a test that had been silently skipping for want of one. A publish refused by another node's lease came back as internal rather than volume_in_use, so a caller was told something had broken when the answer was that someone else held the volume. Mapped in handlePublishVolume. * build: pin viperblock to the release carrying the provider's dependencies The branch has been building against a local replace pointing at a viperblock worktree, so its committed pin never actually built: it named a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch, NewStateReader or s3.NewHTTPClient, all of which this branch calls. Points at the merged viperblock dev instead and drops the replace. GOWORK=off now builds clean, which is the deploy path ansible uses. * fix: let viperblockd's delete sweep tolerate an object another sweep removed deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it returned on any DeleteObject error. Two overlapping deletes for one volume left the loser reporting an internal failure once the winner had removed a chunk between the list and the delete. The control-plane copy that carried this fix is gone with the embedded engine path, so the provider path reintroduced the bug it had just fixed. * fix: stop CreateVolume leaving local state on whichever node served it A state read prefers this node's local copy over the backend, and provider RPCs are not owner-routed, so the node that happened to serve a create kept a copy that answered describe as if the volume still existed after another node deleted it. The two-node conformance harness caught it as DeleteVolume/success failing about one run in three. The seeded path already removed its local files after closing. The unseeded path saved state and left them, which was the only difference between the two. * fix: guard the mount path itself against a second export of one volume handlePublishVolume refused a volume this node already exported, but mountVolume did not, and the legacy ebs.<node>.mount subject is the route production takes for instance launch and hot-attach. A retried attach racing a fresh one started two real nbdkit against one volume, with no refusal and no warning: the volume lease only refuses a different node, and a same-node re-acquire finds the volume already held and increments its refcount. That also made the advertised exclusion inaccurate. viperblockd claims cluster-scoped exclusion on the grounds that a second opener is refused, while the conformance subtests reach it only through PublishVolume, so they passed with the legacy route unguarded. Moving the check into mountVolume gives both entry points the same behaviour and makes the claim true for the path production uses. * fix: tolerate a missing legacy AMI config on deregister DeregisterImage deletes the AMI document, then the pre-provider {imageID}/config.json. Predastore answers a delete of an absent key with NoSuchKey, and that was treated as fatal, so every AMI created since the metadata store landed failed to deregister after its document was already gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had just been told it could not delete. The regression test uses a store that answers a missing-key delete the way Predastore does; MemoryObjectStore silently succeeds, which is why the unit suite stayed green. * fix: reap ENIs left behind by an abandoned instance launch A launch creates the instance's ENI and then attaches it. When the daemon dies in between, the record survives with no instance and nothing ever looks at it again. The security group dependency check counts an ENI that lists the group, so the group cannot be deleted for as long as the record exists. The in-process launch path already rolls back on error; a process that dies cannot. This adds a cluster-wide sweep over records that are launch-created, never attached, and older than the launch RPC timeout by a wide margin. The description prefix is now a shared constant because it is the only thing separating an abandoned launch from a standalone ENI a caller asked for.	2026-08-12 16:54:19 +10:00
Julian Sommer	2a3d6de7d5	fix: pin bluebottle, predastore, viperblock and northstar to dev ... The bare-checkout build on the CI runner resolves these from go.mod alone, where the tagged v1.16.0 releases still carry the pre-bluebottle masterkey and otelsetup layouts.	2026-08-12 11:29:37 +10:00
Julian Sommer	8f930e6275	refactor: consume the shared packages from bluebottle ... Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather than predastore, so the control plane no longer depends on the S3 server for shared auth and policy code.	2026-08-12 10:50:48 +10:00
Julian Sommer	020344ec46	release: update go.mod for v1.16.0	2026-08-12 09:14:29 +10:00
Ben Duncan	c92294e7d5	feat: render predastore templates with the cluster and S3 planes split (#763) ... * feat: render predastore templates with the cluster and S3 planes split The templates set one bind_addr per host, which put raft and blob wherever S3 was published. A host now binds the cluster plane on the address peers dial and the gate carries its own bind_addr for the S3 API, so the installer can hand S3 the WAN without exposing replication with it. The single-node template also drops to one node of each role. Extra blob nodes only hash-partition keys across more badger indexes on the same drive, and extra meta replicas add fsyncs for a quorum that dies with the machine. * docs: regenerate the AWS model operation coverage table * fix: repin predastore to the merged dev commit	2026-08-11 16:35:42 -05:00
brynmailer-mdc	d40747acae	fix: repin predastore to the merged dev commit	2026-08-11 17:23:42 +10:00
brynmailer-mdc	197ae5ccf1	fix: repin predastore to the commit carrying the pkg/ layout ... The branch imports predastore/pkg/auth, predastore/pkg/ratelimit and the root predastore package, none of which exist in the pinned v1.15.0. Every build that goes through the go.work workspace resolved those from the sibling checkout and passed; the e2e binary build on the CI runner uses a bare checkout governed by go.mod alone, and failed to resolve them.	2026-08-11 17:23:42 +10:00
Julian Sommer	5301af4c7b	chore: repin northstar to the otel v1.45.0 migration ... CI builds spinifex in a workspace over viperblock and predastore only, so northstar resolves from the proxy. Its v1.15.0 log sanitiser predates the attribute.Value migration and does not compile against otel/log v0.21.0.	2026-08-11 09:45:02 +10:00
Julian Sommer	19024f79a4	chore: bump otel to v1.45.0 and the log family to v0.21.0 ... The log API moved bodies and attributes onto attribute.Value in v0.21.0, so the otelslog bridge and the sibling log sanitisers have to move in the same window. No spinifex source change is needed; the workspace resolves the migrated predastore and viperblock.	2026-08-11 09:38:31 +10:00
Julian Sommer	ef8bdce9e9	release: update go.mod for v1.15.0	2026-08-04 15:26:46 +10:00
dependabot[bot]	25081fa349	chore: bump the go-dependencies group with 6 updates (#670)	2026-08-03 13:41:12 +10:00
Josh Siviour	68a3518bf0	fix(deps): bump viperblock to the merged clusterrun-compatible dev (#669) ... Viperblock's ordered extent index pulls in tidwall/btree, so the bump needs the matching go.sum entry or the build fails resolving a transitive import.	2026-08-03 11:19:31 +10:00
Josh Siviour	f77a7e5976	fix(deps): bump predastore to the commit that adds clusterrun (#668) ... Spinifex imports github.com/mulgadc/predastore/clusterrun, but go.mod still pinned the pre-merge pseudo-version 608939ef2bb4, which predates that package. Off-workspace builds (CI, which has no go.work) fell back to resolving the import at @latest — v1.14.0 — and failed.	2026-08-03 11:01:23 +10:00
Josh Siviour	972f5837ea	fix: bump goleak past the elided-frames stack-parse panic (#660) ... * fix: bump goleak past the elided-frames stack-parse panic goleak v1.3.0's stack parser treats a "...N frames elided..." line as a function name and panics when it can't parse one. Go elides frames on deep/recursive stacks, which is rare but load-dependent, so this surfaced as an intermittent panic inside goleak itself (not a real leak) during TestDrainVolume_DispatchedGoroutineDoesNotLeak under a full parallel preflight run. Upstream fixed this in goleak@b62053b, unreleased as a tagged version, so this pins to the latest commit. * test: guard goleak against panicking on elided-frame stacks * style: trim goleak regression test comments to the 3-line max	2026-08-03 00:49:50 +10:00
Josh Siviour	8ae8d2fec4	feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes (#658) ... * feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on CreateVolume, persists it to VolumeMetadata, and surfaces it via DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped in mulga-siv-380. Pins viperblock to the commit that adds the VolumeMetadata.Throughput field. Zero Throughput on a read volume means it predates this field; it is omitted from the response rather than surfaced as a misleading 0 MiB/s, matching the existing IOPS convention. * chore: pin viperblock to the merged gp3 Throughput commit	2026-08-03 00:25:49 +10:00
Josh Siviour	22171c44ab	fix: close the storage-exhaustion recovery gaps (durable terminate, per-service memory caps, local volume leak) (#653) ... * fix(vm): durably record terminate even when local state persistence fails finalizeTerminated bailed out of WriteTerminatedInstance whenever the transition to terminated hit a persistence-only failure (e.g. ENOSPC writing local state), even though the in-memory status had already reached terminated. That left the instance owned by nothing: StuckTerminateReaper skips it (no longer shutting-down) and TerminatedTeardownReaper never sees it (never reached the terminated KV bucket). Only an operator daemon restart recovered it. WriteTerminatedInstance is a JetStream KV write, independent of local disk space, so keep going into it whenever the in-memory status did reach terminated, distinguishing that case from a genuine invalid/raced transition. Once the disk has space again, teardown now completes on its own via the existing idempotent TerminatedTeardownReaper. * fix(systemd): cap per-service memory under spinifex-system.slice Every service under spinifex.slice ran with MemoryMax=infinity / MemoryHigh=infinity, so one runaway service could exhaust the whole slice budget and the slice-level OOM would then reap an arbitrary victim instead of the offender. Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the same units scale from a dev box to bare metal). spinifex-daemon.service still hosts every guest QEMU process in its own cgroup (no per-instance slice yet), so it gets by far the largest allowance, just under the slice's own 80%/90%, to avoid starving VM launches. predastore is the legitimately memory-hungry service and gets generous headroom above its idle footprint. The remaining infra services are small, single-purpose Go processes and get a shared conservative bound. * fix(viperblockd): remove local volume directory on ebs.delete DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete handler only stopped the volume's goroutines, killed nbdkit, and removed the NBD socket — it never removed the local WAL/checkpoint cache directory under BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only ran from the unmount handler's seal step, which -efi volumes never reach (they carry no durable data to seal) and which a main volume also skips whenever a prior seal failed (e.g. disk full, deliberately keeping local files for recovery). Since terminate always unmounts before deleting, and delete is permanent, retention had no remaining purpose by the time ebs.delete ran. Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless of mount-tracking state, closing the leak for both the root volume and its -efi companion. * fix(viperblockd): validate volume names before removing local state ebs.delete unmarshalled the volume name straight off the wire and handed it to os.RemoveAll: an empty name collapsed to BaseDir itself, and ".." components could escape it entirely. localVolumeDir now rejects an empty/invalid name or one that resolves outside BaseDir before any removal, and ebs.delete returns Success: false with an error instead of silently skipping. Applied the same gate to ebs.mount and ebs.config, the other two handlers that turn a wire-supplied volume name into a BaseDir-relative path. * chore: pin viperblock to the merged concurrent-open corruption fix The ansible build role builds with GOWORK=off, so the deployed binary links whatever go.mod pins rather than the go.work overlay. Move the pin to the revision carrying the per-volume flock and the WAL/state write fixes.	2026-08-02 18:58:34 +10:00
Josh Siviour	34aa6d2230	fix: stop passing nbdkit S3 credentials as literal argv (#644) ... * fix: stop passing nbdkit S3 credentials as literal argv access_key/secret_key were interpolated into the nbdkit plugin argv, which is world-readable via /proc/<pid>/cmdline for the life of the process. Credentials now travel via cmd.Env (VB_ACCESS_KEY/ VB_SECRET_KEY) instead. Requires a viperblock nbdkit plugin build that reads these environment variables; inert against an older plugin until that pin moves. * chore: pin viperblock to the VB_ACCESS_KEY/VB_SECRET_KEY fallback The spinifex half sets these env vars on the nbdkit child, but the deploy builds with GOWORK=off, so viperblock's matching read of them only ships once go.mod points at 8a4787a.	2026-08-02 09:07:39 +10:00
Josh Siviour	0628b5f20d	fix(ec2): project AMIMetadata.State into DescribeImages instead of hardcoding available (#627) ... DescribeImages hardcoded every AMI as "available" regardless of import progress, so a failed backend import that never reached the flush was indistinguishable from a complete one. Project the AMI's own State, treating empty as available so pre-existing AMIs registered before the field existed stay visible. CreateImageFromInstance, RegisterImage, and CopyImage now set State explicitly since each completes synchronously before the config write.	2026-08-01 22:24:43 +10:00
Josh Siviour	67a8b4409f	chore(deps): bump viperblock for the nearfull throttle fix (#622) ... viperblock#64 stops WriteAtCtx returning ErrNoSpace on predastore's nearfull pressure signal, which rides on a successful write and means slow down rather than stop. Without this, spx refused every write once the store fell below 15% free, 10 points above the watermark at which predastore actually rejects. Needed by spx itself: image import runs in-process against the module-cache build, so the plugin-side fix alone left the control plane on the old behaviour.	2026-08-01 12:16:36 +10:00
Josh Siviour	e4cd9ed527	chore(deps): bump viperblock for the durability batch (#621) ... Six merged viperblock changes behind one repin: local files kept when the WAL cannot be uploaded, stranded WAL generations retried after a failed chunk upload, the flush-covers-acked-writes invariant pinned, the nbdkit plugin's shutdown drain bounded at 20s, seal receipts written after a clean seal, and FLUSH fsyncing the WAL before it reports success. The seal receipts pair with the consumer already on dev, and the 20s drain bound pairs with TimeoutStopSec=30 on spinifex-viperblock.service.	2026-07-31 23:25:47 +10:00
Josh Siviour	034c786916	chore(deps): bump viperblock for the nbdkit plugin stderr fix (#618) ... Picks up viperblock #57, which routes the JSON slog handler to stderr so the nbdkit plugin's logs reach journald — nbdkit repoints a plugin's fd 1 at /dev/null, so the whole volume seal path was discarded. Also picks up #56, removing a per-call allocation from the per-block cache instrument. No spinifex code depends on either change: spx does not import viperblock/telemetry. The bump is needed because the build role compares the viperblock source behind spx (module cache, GOWORK=off) against the one behind the nbdkit plugin (the submodule tree) and fails on a mismatch.	2026-07-31 14:55:51 +10:00
Josh Siviour	b6d43d8b51	feat(gateway): log the canonical request and action on signature mismatch (#615) ... * feat(gateway): log the canonical request and action on signature mismatch A SignatureDoesNotMatch told us nothing beyond err="signature mismatch": the canonical request the server reconstructed was never logged, and the APM trace is a bare "POST /" because auth fails before the dispatcher parses the action. An intermittent mismatch was therefore undiagnosable. Logs the server-side canonical request, with the session token masked, plus the recovered action. Warn rather than Debug because the failures worth catching never reproduce on demand; the path is already rate-limited. * chore(deps): bump predastore for SignedRequest.CanonicalRequest Picks up the exported canonical request the mismatch logging needs.	2026-07-31 08:30:19 +10:00
Josh Siviour	7d2106bfb5	fix(test): reclaim fixture scratch dirs stranded in /tmp (#598) ... * fix(test): reclaim fixture scratch dirs stranded in /tmp The shared predastore fixture and the integration suite's JetStream store both outlive any individual test, so neither can use t.TempDir() - the first test to finish would delete files the daemon still has open. Both used os.MkdirTemp with nothing removing the directory afterwards, so every run stranded one, and a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and swapped tmpfs pages have no backing store to be dropped to. Add a shared sweep that reclaims a directory only when it matches both the prefix and an age no live run can reach. The NATS store also has a clean shutdown point, so TestMain now removes it outright and the sweep is left to cover the killed-run case. The predastore daemon runs until the process exits and has no such point, so the sweep is its only mechanism. * chore: repin viperblock to the dev tip carrying the scratch-dir fix Test-only upstream change, so the build output is unchanged - this keeps go.mod aligned with the submodule pointer rather than delivering a fix.	2026-07-29 14:57:50 +10:00
tomnewton-mulga	a1331b3785	release: update go.mod for v1.14.0	2026-07-28 16:14:13 +10:00
dependabot[bot]	1d1b1f6a51	chore: bump the go-dependencies group with 3 updates (#586)	2026-07-27 12:48:17 +10:00
Julian Sommer	d96bc3c095	fix: bump grpc	2026-07-27 09:15:36 +10:00
Julian Sommer	12085f029c	feat: render import progress bars in human-readable units (#581)	2026-07-23 14:40:37 +10:00
joshsiv-mulga	eae776b2f8	chore: bump viperblock to dev 8a2df56 (503-not-ErrNoSpace + latch logging)	2026-07-23 13:14:04 +10:00
joshsiv-mulga	e26a480e54	chore: repin viperblock to dev with chunk GC on its own goroutine ... Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a dedicated goroutine instead of sharing the chunk-uploader select, so sustained overwrite churn no longer starves GC and wedges the volume at the internal space cap.	2026-07-23 11:41:18 +10:00
Julian Sommer	b505887b36	chore: add nilaway	2026-07-23 11:28:02 +10:00
Josh Siviour	8989fc9ba8	fix: make the seal-failure unmount test deterministic (#578) ... * fix: make the seal-failure unmount test deterministic The test forced the seal to fail by pointing S3Host at an unreachable endpoint, so the handler only returned once the S3 client had exhausted its jittered backoff. Measured handler time spanned 1.8s to 5.5s against a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and worse on a loaded CI runner. Config gains a sealVolume seam, nil meaning the real seal, and the test injects an immediate failure instead. It now runs in 0.53s, 50 for 50, and still asserts what it was written to assert -- plus that the handler actually attempted the seal, which the old form could not distinguish. * chore: bump viperblock to dev tip (GC observability + scratch sweep) Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info) and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).	2026-07-23 10:02:26 +10:00
Josh Siviour	7e8cbeef6e	feat(config): enable viperblock chunk GC in rendered node configs (#563) ... * feat(config): enable viperblock chunk GC in rendered node configs The gc_enabled toggle was plumbed through every VB construction site but nothing ever set it. GCEnabled is a *bool that every consumer treats as off when nil, and the config template rendered shardwal while omitting gc_enabled, so no deployed node ever carried the key and chunk GC was off everywhere by construction. Superseded chunks were never reclaimed under overwrite churn. Render gc_enabled = true alongside shardwal, and cover it with a test that loads the rendered file through config.LoadConfig rather than matching the template text. A string match would still pass if the value failed to parse into a non-nil pointer, which is the exact way the setting reaches its consumers. Existing deployments keep the old rendering until their config is regenerated; the template governs install and bootstrap only. * chore: pin viperblock to the build with the cross-process GC snapshot marker Enabling chunk GC in rendered node configs is only safe once the sweeping engine can see a snapshot taken by another process. viperblock 3b56d61 adds the per-volume snapshot marker that closes that window, and 2a34add fixes the concurrent-stop panic in the background goroutine lifecycle that GC's own ticker would otherwise widen. Pin both before turning GC on.	2026-07-22 22:12:13 +10:00
Josh Siviour	4793b5ac09	fix(eks): node route MTU blackholing image pulls, and teardown wedged in DELETING (#562) ... * fix(eks-node): stop the DHCP MTU overriding the pinned link MTU Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which honours DHCP option 26 and stamps the advertised MTU onto every route it installs: default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \ metric 1002 mtu 1442 Linux prefers a route's MTU metric over the device MTU, so the node kept advertising an MSS derived from 1442 while the real path MTU through the NAT gateway is 1408. Small packets pass and full-MSS inbound segments — a TLS ServerHello with its certificate chain, image layers — are silently dropped. It presents as `net/http: TLS handshake timeout` on image pulls, not as anything MTU-shaped, and it only affects the host netns, so containerd pulls fail while pod traffic sized off cni0 is fine. Per-destination PMTU exceptions expiring in and out of the route cache made it look intermittent and host-specific. Three changes, innermost first: - dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a route MTU. mulga-vpc-mtu is the single source of truth for the MTU of these nodes. - mulga-vpc-mtu re-stamps any route that already carries a different MTU, covering an image built before this fix or a dhcpcd already running. Routes with no MTU metric inherit the device MTU and are left alone. - the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and ignores the IPsec ESP overhead on the NAT egress path, so it overstates the usable MTU for every guest, not just these nodes. Also corrects the header comment, which claimed Alpine's udhcpc ignores option 26 — this image runs dhcpcd, which honours it. That wrong premise is why the route metric was never considered. Adds a stubbed-`ip` test covering the rewrite, the dev token that `ip route show dev X` omits, idempotency, and routes without an MTU metric. * fix(eks): sweep the CP VPC by identity so teardown can finish DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs by role tag. A create that fails partway — the first control-plane server failing to launch, say — leaves resources behind that never got tagged, so the sweep cannot see them. DeleteVpc rejects while any subnet or non-default SG remains, and teardown is re-driven on a timer, so the cluster sits in DELETING forever retrying the same doomed delete every couple of minutes: eks-deleting: re-driving wedged DELETING teardown cluster=toc eks-deleting: re-drive teardown failed err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation" Nothing converges: the cluster name cannot be reused and terraform's delete wait can never pass. Observed on env19, wedged for ~11 hours until this change went in, at which point the next re-drive cleared it. After the tagged sweep, clear anything still in the VPC scoped by vpc-id: subnets, and non-default SGs (the default SG goes with the VPC cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG provisioner — nil is tolerated, since only teardown uses it. * chore: pin viperblock + predastore to the merged storage fixes viperblock f8cfd77 (#42) makes read-modify-write atomic per block, removing the lost update that silently reverted a 4096-byte block to a previous generation of itself. Confirmed as the env19 corruption cause by an A/B: the pre-fix build corrupted image layers within ~15 minutes (failed to extract layer / invalid tar header / unpigz: corrupted) while the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW conflicts on the worker root disks. predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore restart on a store whose metadata set outgrew one badger transaction fails with "Txn is too big to fit into one request" on every node at once, taking the metadata plane — and with it the AMI catalogue and cluster creation — down permanently. Both were verified in the built artifact rather than from go.mod alone: go version -m reports the new pseudo-versions, and the rmw_conflicts counter plus the batched-restore error strings are present in the binary. Pinning by go.mod while the submodule pointer says otherwise is how an earlier analysis ended up reading dead code. * test: record the corrected DHCP MTU in the kitchen-sink golden The NB golden pins the DHCP options published per subnet, including the advertised MTU. Dropping the subnet option from 1442 to 1408 changes what the reconciler writes, so the golden diverged and the live scenario test failed — the test doing exactly its job. Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408 on the two subnet DHCP records; nothing else in the snapshot moves.	2026-07-22 12:57:36 +10:00
joshsiv-mulga	220141b145	release: update go.mod for v1.13.0	2026-07-21 21:46:17 +10:00
dependabot[bot]	f02d9a6c02	chore: bump the go-dependencies group with 2 updates (#553) ... Bumps the go-dependencies group with 2 updates: [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2). Updates `github.com/aws/aws-sdk-go-v2/config` from 1.32.29 to 1.32.30 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.29...config/v1.32.30) Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.19.28 to 1.19.29 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.28...credentials/v1.19.29) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.32.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.19.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-07-21 15:49:57 +10:00
Josh Siviour	a01e58d90c	feat: deliver backend ENOSPC to the guest and enable chunk GC in production (#549) ... * feat: plumb GCEnabled config through to every VB construction site viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never set it, so every deployed node ran with GC off by construction regardless of which viperblock binary shipped. Add a [viperblock] gc_enabled config key (default false, mirrors shardwal) and thread it through to the three places spinifex builds a VB: the nbdkit plugin it spawns per mounted volume (via a new gc_enabled plugin param), viperblockd's detached-volume opens, and the volume service's CreateVolume. * chore: repin viperblock to GC-enable lineage The gc-enable plumbing references viperblock.VB.GCEnabled, which is absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable commit so a standalone (GOWORK=off) build — the path the ansible deploy uses — resolves the field, and spx plus the nbdkit plugin build from the same GC-bearing viperblock lineage. * fix(storagegrowth): stop launching a nano for image-pull repro ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the synthetic npass suite. A 512MB guest OOM-thrashes containerd before any disk-driven ENOSPC and conflates the storage leak with guest-memory pressure. Add a package-local imagePullInstanceType helper: an exact override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default 16384 MiB). npass keeps using the nano selector unchanged. * feat: report guest block write errors so backend ENOSPC reaches the guest Set werror=report and rerror=report on the guest boot drive so a viperblock ENOSPC (or any other backend write failure) is surfaced to the guest as a real I/O error instead of QEMU silently pausing the VM. Without this the guest kernel never learns its disk is out of space, so it can't fail writes cleanly or let an operator react. * chore: repin predastore and viperblock to the storage-capacity-safety lineage Drop the local-path replace for predastore (a test shortcut) and pin both predastore and viperblock to their feat/storage-capacity-safety commits, so a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC classification, and backpressure plumbing that spinifex's werror/rerror change depends on. * chore: bump viperblock to 054a44a3f0f5 Realigns spx's linked viperblock with the submodule working tree, which the build role's source-drift gate compares against. * chore: repin predastore and viperblock to their dev lineage Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work has now landed on dev in each repo, and both PRs were squash-merged, so the branch commits those pins named are unreachable from any branch. Repin to the dev commits carrying the same code: predastore 5699329 and viperblock 35dd9f2.	2026-07-21 15:39:14 +10:00
Ben Duncan	28a876d17d	feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532)	2026-07-17 12:02:59 +10:00
Julian Sommer	3b7b781bdf	release: update go.mod for v1.12.1	2026-07-15 09:29:22 +10:00
Julian Sommer	4d195eb0d9	release: update go.mod for v1.12.0	2026-07-14 13:43:02 +10:00
dependabot[bot]	f29c6585c2	chore: bump the go-dependencies group with 8 updates (#515)	2026-07-14 09:01:41 +10:00
Josh Siviour	2dbd1e716c	chore(deps): bump viperblock to dev (parallel chunk upload + backpressure) (#517) ... Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8 (dev tip through viperblock #34): bounded parallel chunk upload + coalesced checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and picking up the #33 write backpressure on the way. The nbdkit plugin the VM write plane uses is built from the submodule source; this bump aligns the viperblockd library that spx links.	2026-07-13 12:55:26 +10:00
Josh Siviour	98fc01764b	feat(ecs): GPU passthrough to task containers (Epic C) (#512) ... * feat(ecs): thread resourceRequirements GPU count through conv, ContainerDef, AssignContainer and RunSpec * feat(ecs): add GPU placement dimension — InstanceRecord capacity, fits/reserve/release, STRINGSET resource reporting * feat(ecs): agent GPU discovery via nvidia-smi, pinning ledger, device-UUID report-back and DescribeTasks gpuIds * feat(ecs): inject assigned GPU devices into containerd OCI spec via CDI (oci.WithCDIDevices) * test(ecs): add GPU task e2e (nvidia-smi container, gpuIds assertion) to gpu e2e suite * test(gpu): decouple base-AMI requirement from GPU fixture so ECS e2e runs without the ubuntu EC2 image * test(gpu): pick smallest advertised GPU instance type to fit a single host * fix(ecs): preserve pinned GPU UUIDs across STOPPED state change so DescribeTasks reports gpuIds for one-shot tasks	2026-07-13 10:54:55 +10:00
joshsiv-mulga	bdfefc48ae	chore: bump viperblock pseudo-version (io.ops namespace + ES|QL-queryable duration sums)	2026-07-12 10:04:15 +10:00
joshsiv-mulga	47a510f08c	chore: bump viperblock + predastore to dev pseudo-versions	2026-07-11 23:05:12 +10:00
Josh Siviour	beee92b468	feat(obs): bridge slog to OTLP LoggerProvider (#490) ... Init now installs an sdk/log LoggerProvider inside the existing exportEnabled() gate, sharing the same resource as the tracer/meter (incl. ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the returned shutdown func. The process default slog handler is fanned out to both the existing stdout/journald handler and an otelslog bridge, so app logs export over OTLP with trace correlation while journald output is unchanged. With no OTLP endpoint configured Init remains a no-op: no LoggerProvider is installed and the default logger is left untouched.	2026-07-09 22:20:30 +10:00
Julian Sommer	88de183f46	release: update go.mod for v1.11.0	2026-07-09 10:02:22 +10:00
dependabot[bot]	e7306106e8	chore: bump the go-dependencies group across 1 directory with 5 updates (#476)	2026-07-09 09:43:32 +10:00
Josh Siviour	a7b689fec3	feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479) ... * feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits	2026-07-09 09:43:32 +10:00
Josh Siviour	e47baf321c	feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480) ... * feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight.	2026-07-09 09:43:32 +10:00
tomnewton-mulga	37ee3e33ee	fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites (#478) ... * fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites viperblock.New() unconditionally starts a background chunk-uploader goroutine (30 s interval) that only VB.Close() stops. Seven call sites created short-lived or state-only VB instances without calling Close(), leaking one uploader goroutine per instance. On a single node the leaked goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle post-single-suite), causing liveMemGate to reject launches and the lb suite to fail. The leaked uploaders are also the primary cause of the AEAD auth failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the clone-point BlockLookup and overwrites the correct live checkpoint in S3 every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext. Fix all seven sites: - instance/service_impl.go prepareRootVolume: defer StopChunkUploader + StopWALSyncer after successful viperblock.New (mirrors the EFI path's intent; full Close is unsafe here because cloneAMIToVolume already called SaveBlockState + RemoveLocalFiles) - viperblockd.go ebs.delete handler: add StopChunkUploader alongside existing StopWALSyncer - viperblockd.go ebs.unmount handler: same - viperblockd.go shutdownVolumes: same - snapshot/service_impl.go snapshotVolume: defer both stops after New - volume/service_impl.go CreateVolume: defer both stops after New - image/service_impl.go snapshotRunningVolume: defer both stops after New - image/service_impl.go snapshotStoppedVolume: defer both stops after New Closes mulga-0zzci Closes mulga-ji5fi * fix(deps): bump viperblock to 01a7e686 (AEAD regression fix) The prior pin (6af89855) carried the snapshot-AMI AEAD regression that corrupts block storage on restore. Point at the merged viperblock dev fix. --------- Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>	2026-07-09 09:43:31 +10:00