221 Commits

Author	SHA1	Message	Date
Ben Duncan	8ab96b29a1	fix: replicate every JetStream KV bucket across the cluster (#1117) ... * fix: replicate every JetStream KV bucket across the cluster A KV bucket created without a replica count gets one replica, and one replica lives on one JetStream-chosen server recorded in no config. Losing, restarting or partitioning that server makes the bucket return "no responders available" from every node at once, so a single-node event becomes a cluster-wide outage. The reconciler leader-lease buckets were the worst case: acquiring a lease is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed. The cluster's node count is now declared once, in config.LoadConfig, and read by the one helper every bucket creation goes through. Every process that creates a bucket loads the cluster config first, so no service can forget to pass it and no signature has to carry it. An undeclared size is an error at creation rather than a silent fall back to one replica. Existing buckets are raised rather than left behind: a service opening one raises it, the daemon sweeps them all after formation, and "spx admin kv replicas --repair" does it on demand. Raising reads the live stream config and changes only the replica count, and never lowers one. Removes the ~30 replicas/clusterSize parameters that only forwarded a number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no clamp and would have failed outright on a cluster of six or more. Proved by a repo-wide invariant test pinning bucket creation to the one helper, a whole-cluster sweep in every multi-node nightly cell, and a new kvquorum suite (cell 33) that takes away the node leading the most buckets and requires the survivors to keep answering for all of them. * chore: name cell 33 in the nightly dispatch input * fix: a KV bucket that cannot be raised must still open The first version of the replica fix treated a raise it could not place as a hard error, and on a real three-node cluster that turned the fix into the outage it prevents. A node whose config named a fourth server wanted four replicas, NATS answered "no suitable peers for placement" for three, and the daemon retried for its full five-minute budget without finishing bootstrap. The bucket was present, quorate and serving on three nodes throughout. A config naming a node before that node serves is the ordinary middle of growing a cluster, so that is a state to tolerate rather than refuse. The line is now drawn between the two cases instead of around both. Creating a bucket below the cluster's replica count still fails, because that is the defect and a fallback there reintroduces it where it is hardest to notice. Failing to raise an existing bucket warns and hands the bucket back, and the raise is retried on every open besides the daemon sweep and --repair. This matters most in kvlease, which runs on every reconcile tick: refusing there would stop every reconciler sharing a lease bucket that was working. getOrCreateBucket now attaches before it creates. A create asking for more replicas than the cluster can place is refused before the server looks at the name, so with create-first "this bucket already exists" was an inference from which error came back, and the wrong one comes back on a single-node server. RaiseAllBucketReplicas attempts every bucket and joins the failures, and --repair keeps going too. Stopping at the first would leave the rest of a cluster short because one bucket could not be placed. Also adds nats.ServerName/NodeFromServerName beside the template that renders server_name, with a test tying them together. The kvquorum suite had mapped stream peers to nodes by comparing them to node names, which they are not: they are the node name behind a "spinifex-nats-" prefix, so nothing matched and the node-loss test could not choose a victim. * test: wait for the right NATS peer count after a node rejoins /routez reports a node's routes to its peers, not its own membership, so a fully formed three-node cluster is two peers per node. The kvquorum suite waited for three and spent its whole five-minute rejoin budget on a cluster that had already reformed. Every other caller passes len(Nodes)-1; the helper's doc comment now says so. The assertion the test exists for had already passed: node1 was leading 18 of 47 buckets when it was taken away, and node2 and node3 each still answered for all 47. * fix: replicate KV buckets across the largest odd count a cluster can place A bucket was replicated across min(nodes, 5) servers, which stores a fourth full copy on a four-node cluster and still survives exactly one loss — the same tolerance R3 gives for three quarters of the write amplification. A Raft majority needs an odd count, so the rule becomes the largest odd count the cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two entry points, so no caller can hold a stale copy of the policy and keep passing after the policy changes. Three retry loops could not tell a configuration fault from a cluster that had yet to form. An undeclared cluster size never becomes declared by waiting, so each of them spent its whole budget on one and then reported quorum as the cause. They now stop on a permanent error and say what is actually wrong. This is what the integration suite was hitting: it builds its gateway by hand and never loads a cluster config, so the size was never declared and the IAM retry sat in its sleep until the test timed out. Both ends are fixed. The audit reported configuration alone. A bucket at three replicas with one peer offline and one lagging is one failure from unavailable and read as ok. BucketReport now carries the leader and a count of the peers JetStream itself calls current and reachable, quorum is measured against that count, and spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem --repair fixes from one needing a human. Declaration is write-once: daemon.Start refuses a conflicting redeclaration because it is about to create buckets, and it declares before JetStream init rather than after it. config.LoadConfig warns instead, because it also reads config files that are not this process's own cluster. spx admin init --nodes 2 is refused — no replica count on two nodes both survives a loss and accepts writes — and single-node replication is asserted in the single e2e suite, so it runs in every nightly host permutation. * fix: withhold the vLLM serving image until its artifact is published The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names 404s on iso.mulgadc.com, so spx admin images import would fail for every operator who tried it. Its GPU siblings are published; this one never was. The entry moves to WithheldImages rather than being deleted, because the entry is the definition — the URL, the checksum and the two tags resolveServingAMI filters on. Publishing the artifact is then a move back, not a retype. Bedrock already handles the image not being registered: resolveServingAMI returns ErrServingAMINotFound, which is the honest answer while there is nothing to launch. The bedrock tag contract is still pinned, now against the map the definition lives in, and it additionally asserts the entry is not offered for import. The e2e reachability walk covers AvailableImages alone, so a withheld entry is out of it by construction. * fix: drop the cluster size from the call sites dev added Two constructors on dev still pass a replica count that no longer exists as a parameter: the cluster size is a declared process fact, so EKSServiceDeps has no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is textually clean and breaks the build, which is the shape of collision a signature removal always has. * docs: explain what an operator sees when a cluster server fails The replication section said how many copies a bucket has and stopped there, which leaves the question an operator actually has during an incident: one server is gone, what still works. It now covers the majority side carrying on, an isolated server refusing changes on purpose and why that is the behaviour you want, guests continuing to run while anything that has to be recorded does not, the risk of reading state from a server that is cut off, and a returning server catching up by itself. The surrounding prose reads from the operator's side rather than the implementation's: what you see and what to do, not which consensus protocol is underneath.	2026-09-28 12:34:55 -05:00
Ben Duncan	021f439bab	feat: recover instances automatically from a node that stopped heartbeating (#1111) ... * feat: recover instances automatically from a node that stopped heartbeating A node that dies hard takes its guests with it, and nothing brings them back until an operator brings that node back. Every daemon now runs the same small reconciler: it reads the instance records, selects desired-running instances whose owner is stale, and relaunches them here. The claim is a CAS on Status.LastNode and that is the whole of the coordination. Several survivors race one instance, exactly one revision wins, and the losers find a record naming a live owner and stop considering it. A claimant that dies mid-launch leaves the record naming a node that is itself stale, which is the same candidate the next pass derives again, so there is no lease to expire and nothing to reconcile by hand. Ownership is not permission to write. The volume leases still decide whether the guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt that loses the volume race has touched no tap, no OVS port and no OVN binding. A storage failure is not a host failure and wants the opposite response. A guest paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is refused: the volumes are the same objects in the same store from anywhere, so relaunching moves the pause rather than ending it. A node whose own object store is not answering stands down entirely, which separates a store broken here from a store broken everywhere without either having to be diagnosed. A launch that does not complete backs this node off that instance, doubling to a cap, because nothing on the mount path can yet say which of the two it hit. Recovery is off by default and inert below three nodes. It also waits until it has seen its peers live, or five minutes, so the first node back from a coordinated shutdown does not recover guests from nodes that are still booting. * test: prove instance recovery moves a host failure and refuses a storage failure The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves. Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live. harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down. * fix: make automatic instance recovery a cluster-wide setting Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file. Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it. * test: drive real I/O into the storage-fault case, and freeze the store rather than stop it Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted. The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them. The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three. * test: give instance recovery its own suite and its own nightly cell Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five. Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set. * fix: give a guest long enough to open a volume the object store is slow to serve QEMU opens every drive before its monitor answers a greeting, so the QMP greeting deadline is really a deadline on opening the guest's volumes. At 30s it was shorter than a cold viperblock open costs on a cluster that has lost a node, where every object read is served degraded: measured at 35s with one node of three away. The launch killed the guest five seconds short of ready and then did it again on every retry, which is a recovery that can never complete. A network-backed drive now gets its own deadline for the same reason VFIO already had one — synchronous work before the monitor answers — and the instancerecovery suite is what found it. Also make the suite's predastore thaw best effort per node. It runs after a node has been taken away, where there is no process to signal, and a node it cannot reach must not stop it restoring the rest. * docs: name the two newest cells in the dispatch input The cells input is the operator's index of what a number means, and it stopped at 29 while 30 and 31 were added. * test: prove the guest is writing before freezing the store in the recovery suite The storage-fault test froze the object store as soon as RunInstances reported running, which is when QEMU has launched rather than when the guest has booted. On the nightly cluster the freeze landed while the guest was still in UEFI, so the only drive it could reach was the EFI varstore — and pflash has no werror=stop to pause on, so the guest never paused and the test reported a product failure it had caused itself. The load driver now prints to the serial console after each pass it has written and synced, and the test waits for that before freezing. The console is read through GetConsoleOutput, so this still needs no login and still runs on cells whose guests have no public address. * feat: recover instances automatically with nothing to configure The switch is gone. An instance whose host has failed is down whether or not recovery is enabled, so the only outcome a disabled reconciler can produce is leaving it down — which is the state recovery exists to end, not a safer one to fall back to. Against that, the switch cost a cluster that can be configured into never recovering anything, with no symptom until a node dies. This feature already hit that once: a top-level table parsed into nothing and every node reported recovery off with the setting plainly in the file. What makes recovery inert is now only facts about the cluster — no cluster config, no JetStream, or fewer than three nodes — and each says in the log which one applied. The e2e no longer writes config or restarts daemons: it waits for every node to report the reconciler active on a cluster nothing configured, so reintroducing a switch fails there as well as in the unit tests. * feat: stop retrying a recovery that is never going to work Every failed attempt used to go to the same exponential backoff and stay there. An instance nothing could launch was retried every fifteen minutes forever, still observed-running on a node that is gone, with no account of it anywhere the customer or an operator would look. A failure is now classified and counted. Two classes are read from what the launch actually reports: the node it was taken from still holds a volume lease, which clears on that lease's clock and is retried on a faster one, and this node has no room, which another survivor may. There is no terminal class, because nothing on this path can prove one — the launch collapses everything else into ServerInternal, so a verdict of "never going to work" would be a guess, and the guess that abandons a recoverable instance is the expensive one. The budget bounds the unknown case instead. Every class counts against it, since a class that did not would be the old unbounded retry under a narrower condition. Exhausting it leaves the instance where AWS leaves one whose host failed and could not be brought back: stopped, carrying a StateReason that names the cause and the last error, one StartInstances from running again. Writing the desired state down is also what makes one node's decision cluster-wide, because selection reads desired state — the same property the claim already has. A record another node has since claimed is never touched, so a node exhausting its attempts cannot stop a guest that is now running elsewhere. * fix: give a guest the time for every cost it pays before its monitor answers A GPU instance booting from a network-backed root volume pays both costs: QEMU opens every drive across the object store and then pins the whole of guest RAM, one after the other, all before the monitor answers. The deadline took whichever was larger, so such a guest was given less time than it needs. That the two floors are both 180s is why nothing has been seen yet — the VFIO branch has always returned at least the network-drive deadline. It is the same defect either way, and the symptom when it lands is a GPU instance SIGKILLed on every attempt while its plain sibling launches, which reads as a GPU fault rather than as a deadline. The costs are now added, the plain QEMU startup is counted once rather than replaced, and the cap covers both floors together so it cannot clip a guest before either cost has been waited out. * test: assert the AWS-visible arc of a host failure end to end A customer cannot see which host an instance is on, and AWS never tells them. What they can see is DescribeInstanceStatus, so that is where a recovery has to be provable, and the suite never looked at it. Three frames are now required, in order: running with system-status ok before anything is broken, system-status impaired while the owning node is gone, and ok again once a survivor is running it. The middle frame is the one worth having — the window opens when the owner's heartbeat goes stale, which is strictly before any survivor may claim, and closes when the relaunch registers, so it is bounded rather than lucky. Reporting ok throughout would be the more comfortable lie. The availability zone is compared across the first and last frames as well. A recovery that crossed one would break every placement promise made to the customer, and nothing else in the suite would have noticed. * test: prove a half-healthy node fences itself and is never a second writer The two existing recovery cases take a node away by stopping its services, which is the failure this feature handles least interestingly: a node that is gone has no opinion about who owns its volumes. The one that matters is the node that is still up, still running QEMU, still answering its own local clients, and simply cannot reach consensus. That is the state in which two nodes can each believe they own a volume, and the only thing between that and a corrupted disk is that the partitioned node gives up first, on a clock it can evaluate without reaching anything. So this drops NATS on 4222 and 4248 between one node and its peers and nothing else, then asserts the ordering the safety argument rests on rather than the outcome it produces. A watch samples every node every three seconds for the whole run, including the heal, and fails on any single sample naming two hosts. Two guests writing one volume through one object store is data loss already under way, not a race that resolves, and no point-in-time assertion can see one that appeared between its own samples. The injector takes its own nft table rather than touching inet spinifex_filter, because a test that flushed the node's real firewall would leave it either unfirewalled or misfirewalled with no sign of which, and a separate table at priority -300 heals in one atomic delete. A transient systemd timer removes it independently, so a cancelled workflow does not leave a node cut off from its cluster until somebody notices. It proves SSH still works and proves a peer port is genuinely unreachable before anything is asserted, since a rule set that installed cleanly and dropped nothing would read as the system tolerating a partition. The suite budget goes to 90m and cell 31 to 105m to fit the third case. * fix: move an instance's boot state, volumes, logical port and address with it A record-driven launch assumes the node reading the record is the node that wrote it. On a daemon restart that holds. On a recovery onto a survivor it does not, and five things follow from it — one that stops a moved instance booting at all, and four that stop it running anywhere. A load balancer never comes back from a host failure. Its boot configuration is a set of files under the runtime directory and the record holds their paths, so the survivor restores paths that exist only on the node that died. The hook that rebuilds them was reachable from Restore alone, and the cross-node path goes straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert that are not there. The gateway URL baked into that env names the creating node's management bridge as well, which the rebuild corrects because it rebuilds from whichever node is running it. Firing the hook on both paths fixes both, and a rebuild that fails now refuses the launch: a guest that boots without its network configuration is worse than one that does not boot, because the records would say it recovered. The export outlives the guest and holds the volume lease. A returning node is healthy, so it renews that lease indefinitely and the new owner never acquires: the guest is stopped here and refused there, recovery spends its attempt budget on a volume that was never going to be free, and gives up on an instance that was never unrecoverable. So the export is given up too, over a new node-addressed request that tears it down exactly as the fence does. Deliberately not an unmount — an unmount seals, this node's block map is the stale one, and sealing publishes it over the winner's. Unlike the fence it deletes the lease key rather than leaving it to expire, because the lease is still this node's to hand back. The tap outlives the guest, and the tap is what claims the instance's OVN logical port. The binding names one chassis, so two nodes offering the same iface-id do not split the traffic — they contend, and the address answers from whichever won last. It also blocks the layer above: the external IP's host ingress is pruned on the evidence that the owning guest sits on another chassis, and that stays false while this node is still claiming the port. The management address outlives the guest as a cluster-wide reservation, and the release was unconditional. br-mgmt is one flat L2 segment, so the node an instance left was freeing an address still in use and offering it to the next instance anywhere. Only the node an entry names may release it now, with the takeover recorded on arrival so the node actually running the instance can still give the address back. An entry written before the node field existed names nobody and stays releasable, because there is no other holder to collide with. The fence path already cleaned up the tap, so the release fix repairs that path too. * test: prove a load balancer keeps its name, its address and its traffic when its host dies Takes away the node running an internet-facing ALB and asserts the four things a customer of one actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again without anybody being called, and the backends are healthy again from the recovered load balancer's point of view. The backends are placed around the load balancer rather than beside it. A load balancer lands wherever a node answers its queue group, so its node is discovered and the backends are relaunched until both are elsewhere; one on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve through the same server, so the claim under test is what this cluster answers for a name rather than what the machine running the tests was configured to forward to. A watch samples every surviving node every five seconds for the whole run, because an answer that went missing or pointed elsewhere for one reconcile pass is the outage, and no single sample reaches it. The returning node then has to give up what it no longer owns: no second copy of the guest, and no tap still claiming the load balancer's logical port, since OVN binds a port to one chassis and a second claimant contends for it rather than splitting the traffic. The last assertion is that the name still serves with that node back in the cluster. * docs: name the new cell in the nightly's cell list * test: stop the disk-load guest from filling the console it is judged by The load generator announced every write, once a second, for as long as the test ran. The console is a ring buffer this suite reads back, so an hour of that pushes out everything explaining a failure, including the kernel's own account of the I/O error the test exists to produce. Announce twenty times, then keep writing quietly. * docs: say what the OCI guide has been proven on, and what a host failure does The guide read as a support statement for OCI rather than as an account of one tested path, promised seven steps and then gave eight, and said nothing about what happens to a guest or its public address when the node under it fails. State the evidence, state that the nightly guards the datapath and not the cloud, and give the failure behaviour as a table with the reason codes a customer will actually see. Also say plainly what the volume lease is: one running guest and a bounded, logged window, not a write barrier the object store enforces. Someone sizing a workload against this should read that before they read the table. * docs: say why the mount classifier refuses to retry a lease another node holds It reads as an omission — a lease held elsewhere is the most transient refusal there is. It is not. This classifier drives the restore path, where the holder is a node that is running the guest, so retrying would relaunch against a live owner. Recovery wants the opposite and reads the refusal itself. * feat: claim a recovered guest's public address in line, and verify it arrived A recovered instance was reported running before anything moved its public address off the node that died, so for up to fifteen seconds the records said recovered and the address answered nothing. The recovery path now runs the affinity pass itself, which is idempotent and is the same pass the ticker runs. Two things the pass could not do before. It abandoned every remaining address on the first API failure, though a throttle or a denial is per-object and the guests behind the others are just as dark. And it trusted the move's own response, which is OCI agreeing to the request rather than OCI having done it, so an accepted move that never landed read as success. It now re-reads what OCI delivers here and reports the difference. An address that will not arrive makes the guest a system-status failure with the time it went dark, the same shape AWS uses for a host-level fault. The guest passes every check it can be asked about itself, which is exactly why reporting it healthy was the worst available answer. * feat: give a recovery attempt one trace across all three of its phases The volumes opening, QEMU starting and its monitor answering already trace themselves, but nothing tied them to the attempt that asked for them: three orphan traces per recovery, no way to ask which phase a slow one spent its time in, and nothing naming the node the guest came from. A span over the attempt makes them its children, and the reconciler's own log lines carry its trace ID because the handler stamps it from the context. A failed attempt records the fault and the attempt count on the span, so the budget running out is readable as a sequence rather than as twelve unrelated warnings. * fix: finish a fence once its writer finally exits A kill that did not take effect was logged, counted and dropped. The export stayed registered and the lease stayed held, which is correct while the writer lives and wrong forever after it dies: a lease a healthy node keeps renewing never expires, so the volume could not start anywhere. The failure is now re-reported every 30 s until it resolves, and the teardown completes as soon as the writer is confirmed gone. Only a delivered kill is waited on — a refused one says nothing about what became of the writer. * test: stop judging a detached stop or terminate on a transient state Both tests read the instance state immediately after the ack, but the transition runs on its own goroutine, so the read races it in both directions: too early sees running, too late sees the final state. The race gate caught the first of those on a loaded runner. * fix: replicate the volume lease and dirty buckets across the cluster Both were created with the default single replica, so both lived on one node. That node holds the answer to "who may write this volume" and the marker every mount reads, and losing it takes volume leasing and mounting down cluster-wide. A partitioned node was fenced correctly and then no survivor could relaunch its guest: every attempt failed on "read dirty mark: no responders available", for ten minutes, until the budget ran out. The replica count is now the cluster's node count, as the gateway's own buckets already do. Existing buckets are raised in place rather than requiring a wipe. The e2e diagnostics that should have said this are fixed too. The fence and the surrender are logged by the storage service, not the daemon, so the assertions read the wrong journal; and Eventuallyf evaluates its message at the call site, so a ten-minute wait was explained by a log that stopped before it began.	2026-09-27 15:56:09 -05:00
Ben Duncan	7163f7d62d	feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) ... * feat: ship uninstall-spx.sh and fix the OCI bring-up path Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the software where node-reset.sh clears state and leaves the node installed. An operator who installed with `curl ... | bash` had no teardown path at all. State teardown is delegated to node-reset.sh rather than reimplemented, and setup.sh now ships both to /usr/local/share/spinifex so they are reachable without a checkout. It refuses on a multi-node cluster and --yes does not override that: a flag meaning "don't ask me" must not also mean "ignore a safety property". /var/lib/spinifex and every apt package are kept unless asked for, and both opt-ins confirm separately even under --yes. Proved by a full uninstall/reinstall round trip on dev-oci, which found two bugs neither prod nor dev-prod can reach, because both are ISO-installed: node-reset.sh exited 2, silently, at its last step and after the wipe. setup.sh creates the spinifex *group* and the spinifex-* service users but never a *user* named spinifex, so `getent passwd spinifex` exits 2 and pipefail carried it into set -e. The uninstaller broke openvswitch-switch on its first run by removing the socket-perms helper while leaving the drop-in referencing it, failing the unit at step EXEC with status 203. Fixed on both sides: the uninstaller removes drop-ins it wrote into other units' namespaces and restarts anything it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can never stop another package's daemon. setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux bridge it links it to OVS with a veth pair instead of enslaving the NIC, so the VNIC keeps its MAC identity and netplan keeps owning the addresses. It now fails on OVS datapath errors instead of reporting success, and the --management demotion guard keys on /etc/default/ovn-central rather than on database presence, which the ovn-central package creates on every node. Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts along with shellcheck over the four scripts that ship to every node. * feat: allocate external IPs from the Oracle Cloud API An OCI VNIC drops any packet whose source is not a registered private IP object on it, so an external pool cannot be range math there — the address has to be created through the provider before it exists at all. Add source = "oci" and the allocator behind it. One allocation is two OCI objects: a secondary private IP that rides the wire, and a RESERVED public IP attached to it that the internet reaches. Allocate returns the public address because that is what AWS semantics are about, and records the pair so the host plumbing and release path can find the private half. The OCI objects are created before the binding is written, and dropped after it on release — both orders chosen so a crash leaves a discoverable leak rather than a record that lies. Reconcile squares the two on startup and is required, not optional: a leaked reserved public IP bills and holds one of the 50 per-region slots with nothing referencing it. It never touches an object without our display-name prefix, because the reference host carries operator-created secondaries on the same VNIC. Resolve the VNIC from an interface name by MAC through IMDS rather than by Oracle's VNIC ordering, which is not a documented contract. Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring is not a second single-purpose installer. * fix: refuse source = "oci" outside routed mode, and cover the SDK path An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound frame to it regardless of which of its addresses the packet is for. Measured on the reference host: an internet source reaches 10.200.0.144 as router-MAC to VNIC-MAC, the public address never appears on the wire at all, and every off-subnet destination resolves to the same next hop. Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on the uplink and ARPs for the address, which policy.go:50 justifies with "the Linux bridge learns the per-rule external MAC on the veth port like any other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode guest's egress is dropped and its ingress never reaches OVN. Routed mode keeps the address on the host, which already wears the VNIC's MAC. Pinning every LRP to that MAC does not rescue pool mode: they share one external logical switch and OVN resolves inbound frames by an L2 lookup on it, so duplicate MACs work for one VPC and silently misroute past that. The refusal names the MAC reason, because a refusal that only says "use nat mode" reads as arbitrary and gets worked around. Also test the SDK request shapes against a local endpoint. Two of them are wrong in ways reading does not catch: pagination stopping early reports a partial VNIC to reconcile, which then drops live bindings, and a nil privateIpId on detach is omitted from the body entirely, so OCI reads the update as a no-op, returns 200, and leaves the address attached and billing. * fix: install host firewall rules where a distro's catch-all reject cannot precede them Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD. Two Spinifex datapaths were silently dead behind it on any such host. EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules landed after the REJECT and never matched: 24113 packets on the reject, 0 on both accepts, and guests had no egress. The comment named the case it did not handle, anticipating a FORWARD policy of DROP — which an append does survive — rather than an explicit REJECT rule, which it cannot. Insert at the head, and drain existing copies first, because a rule an earlier build appended still satisfies -C and would otherwise stay stranded across an upgrade. Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254 and .253 per instance on the ime- devices, so guest requests are locally delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init spent 490s timing out and reported the endpoint as disabled. With one wildcarded accept the same image boots in 18.6s and finds its datasource. The natuplink e2e check had the same blind spot the code did, asserting -C and therefore presence but never position. It now asserts the accepts precede the first reject. * feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata An OCI external address is two objects, not one: a RESERVED public IP that OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP on the VNIC that the wire actually carries. Every AWS-facing record holds the public half; every datapath object must hold the private one. The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP, PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile rebuilds the rule set from AWS-facing intent on every pass — so a correctly mapped rule installed further upstream is swept on the next reconcile and replaced with one matching an address that never arrives. The seam is a DatapathResolver defaulting to identity, so every non-OCI deployment is unchanged. A resolve failure is fatal: falling back to the public address would install a rule that reads correctly in ovn-nbctl and blackholes. ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no credentials, because vpcd is a different process from the daemon that owns the allocator and the bindings record is all either would consult. Two live bugs the in-memory test store was hiding, both found by making the fake report ErrNotFound the way the real store does: - KVStore.Mutate could not create, so every pool failed its own first allocation after OCI had already created and billed the pair. - Reconcile treated a missing record as a failure, aborting the one pass that exists to find an object created before its binding was written. Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's own metadata service lives — the first guest launch made it a local address and took the host's metadata and DNS with it. The datapath is OpenFlow steered, so the guest-facing addresses stay put and only the host-side endpoint address moves, with a PREROUTING DNAT on ime-+ translating and conntrack un-NATing the reply. Off by default; identity when unset. The IMDS firewall rules move to vpcd. They are wildcarded over every endpoint rather than per-launch work, and spinifex-daemon holds no iptables grant and no CAP_NET_ADMIN by design, so attempting them aborted every launch. * fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package The error string pointed at DefaultHostBindAddrs, which does not exist — a caller following it would have looked for a symbol that was never written. nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries the //test:in-package marker the package check asks for. * docs: index what each e2e suite proves Written because a datapath comment was read as an end-to-end behavioural claim and the suite that already answered it was found only after the wrong answer had been given. TestVPCEgressPaths/NATBaseline is the authority on private subnet egress, and nothing pointed at it. * fix: source-route an EIP's reply out the VNIC that holds it An EIP's reply carries the EIP as its source. On a host that pins its uplink with a policy rule, the reply took the main table instead and left by the wrong interface — which a cloud VNIC drops, because the address is registered on a different one. Ingress reached the guest and nothing came back. Every bind now mirrors the uplink's rule onto the EIP. The selector is the rule rather than the interface: both OCI VNICs sit in one subnet, so "the interface owning the EIP's subnet" names two of them, and two candidate rules is a refusal rather than a guess. A host with no source routing gets no rule and no change. * docs: name the setup-ovn recipe routed mode actually needs on OCI The OCI guide told operators to run `--wan-bridge=br-wan`, which is mutually exclusive with `--nat-uplink` and deletes the spx-nat transit veth routed mode runs on. The node then refuses to bring the uplink up, with an error naming the flag the guide should have used. Several messages also claimed routed mode is outbound-only with no public IPs. That stopped being true when the routed EIP path landed, and it reads as "what you are about to do cannot work" during the one init the OCI path documents. * feat: let routed NAT form a multi-node cluster, and carry every pool through formation Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no technical justification -- it was a v1 scope decision -- but routed NAT is the only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become "Spinifex on a cloud host is single-node". It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms, all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only the gateway chassis reaches its own guests. init now says so on stderr rather than refusing, because the refusal made the defects unreachable and therefore unfixable. Distributing the EIP datapath is the actual fix. Lifting it exposed a second defect that stops formation outright. NetworkConfig held nine flattened Pool* scalars and the init node set PoolName = "wan" unconditionally; routed NAT renders two pools and the one that matters -- nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a single pool named "wan" holding the transit gateway and the detected WAN prefix, and the leader re-rendered its own config from the same struct, so all three nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData gains json tags because it is on the wire now rather than a template input. * feat(install-node): add --external-mode so a cluster can be formed in routed NAT install-node.sh could not express routed NAT at all. It required --external-pool/--external-gateway/--external-prefix-len and passed --external-mode=pool to every spx admin init unconditionally, so the only supported way to form a routed-NAT cluster was by hand. --external-mode nat refuses the three pool flags rather than ignoring them, and threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check rejects a physical NIC during formation, which is correct in pool mode and wrong in routed mode, where nothing is bridged. * fix(ocinet): stop each node deleting its peers' OCI addresses on startup Reconcile compares the binding record against the private IPs on one VNIC. The record is one key per pool and therefore cluster-wide, so on more than one node every peer's binding looks exactly like a binding whose objects OCI has lost. Each node dropped them, and direction 1 on the owning node then collected the now-unclaimed objects as leaks -- deleting the reserved public IP and private IP of a running instance. Observed on the three-node OCI cluster: node2's startup pass logged "collected a leaked address" for an address it had allocated ten minutes earlier and "dropped a stale binding" for one living on node3's VNIC. The damage did not stop at the OCI objects. With the binding gone, Lookup.DatapathIP returns the public address unchanged, so the OVN dnat_and_snat row carries the public address instead of the private half. Guest egress then leaves the VNIC sourced from a reserved public IP, which OCI drops because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute no-ops too, since a public address shares no subnet with any host address. One destroyed binding takes out both directions. Binding.VNICID already exists for exactly this ("how a node that comes back from the dead tells mine from moved on without me") and Reconcile never consulted it. A binding is now only dropped when it names this node's VNIC; one naming no VNIC cannot be attributed and is left alone rather than guessed at. * fix(host): find the EIP's source-routing table when the uplink is held as a /32 uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding a host address configured with a prefix that contains it. On OCI both VNICs sit in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts the subnet's on-link route in the main table at metric 0 and takes the whole subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was ever found, no source rule was installed, and the reply left by the primary VNIC where OCI dropped it for carrying an unregistered source. The address is not the only record of who owns the subnet. The rule names a table, and that table holds the on-link route. Consulted as a fallback, only when the address prefixes matched nobody, so a host that already resolved this the direct way keeps its existing answer -- prod and dev-prod are unaffected. Only scope-link routes count. A default route covers every address and would make every source-routing table an equally good answer. * docs(oci): correct the multi-node story and name the oci_vnic_id constraint The nat-mode warning said EIPs and default egress were both broken on more than one node. Default egress is fine: delete a guest's dnat_and_snat row and it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the public-IP path was broken, and the three defects behind it are now fixed. What remains is not a bug but the shape of the datapath: routed NAT centralises the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other nodes cannot reach it, and OCI drops any frame whose source is not registered on the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator writing the pool will read it, and in §9, where the cost is that a gateway change kills every public IP until the pools are repointed. The single-node cap itself is gone from the doc: a three-node cluster has been formed and measured, and the warning proceeds rather than refusing. * feat(nat): distribute the routed-NAT EIP path so it works on more than one node Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is a localnet, so the transit segment a /32 route pointed into existed only on that one node. Measured on three nodes: all three claimed all three EIPs, the nodes disagreed on the gateway LRP, and only guests on the gateway chassis were reachable. The mechanism to fix it is not new. Pool mode already distributes EIPs by putting external_mac and logical_port on the dnat_and_snat row and letting OVN's ARP responder answer for the external IP on the external segment. Routed mode has an external segment too, the spx-nat veth pair, and it being per node is the distribution: only the node running the instance can answer ARP on it. So a routed rule is now distributed on the same terms as a pool one, and its host route becomes on-link on the transit veth instead of via a gateway LRP that may be on another chassis. bindHostEIP stops demanding that address for a distributed rule, which is what previously failed the bind on every node but one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS rather than the Southbound DB because the question is about this host. The gateway LRP keeps its transit-pool address. Default egress routes through it and works across three nodes today, so making it link-local as pool mode does would break a working path to fix one distribution has already fixed. Pool and static are unmoved, and a test asserts that against the same input rather than leaving it to inspection. A rule with no MAC stays centralised, so a cluster part-way through an upgrade is never in a state neither shape handles. Beads: mulga-opyl7, mulga-la0pe * fix(nat): make routed-EIP host plumbing work on every node Three guests on three OCI nodes, one guest per node: only one was reachable. Two defects, both invisible on a single node. The transit veth took a random MAC per node, but OVN holds one MAC binding for the transit nexthop across the whole cluster — so every chassis addressed its egress to whichever node seeded the binding, and the other kernels dropped it at L2. The pair is node-local, so one locally-administered address everywhere is unambiguous; setup-ovn.sh pins it at creation and in the netdev unit, and EnsureUplinkPort repairs a veth made before that. Host routes and proxy-ARP also rode the leader-gated reconcile pass, so a node that never won the lease never plumbed its own guests' EIPs. BindHostEIPs does that half on its own loop on every node; Bind already leaves alone an EIP whose ENI is elsewhere, so two nodes never contend. An unresolvable address skips the sweep rather than shrinking it. Pool and static are untouched: the new pass is a no-op outside routed mode, and a test asserts it. Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on a node that is already tangled. It deliberately spares the NB/SB databases, /etc/spinifex, the system-id and the packages, and refuses to del-br a Linux bridge. * fix(nat): assert the transit MAC where vpcd actually runs it EnsureUplinkPort is never called in production — host.Wiring is constructed nowhere — so the repair landed in a path the deployed daemon does not take, and three nodes came back with three different MACs. vpcd re-asserts it beside the NAT egress rules instead, which is the routed-NAT branch that does run on every start. * fix(vpcd): make uplink-mode detection say what it saw, and check it detectBridgeMode logged only the branch it took, so a node detected as veth when it was wired for nat looked identical to one that was right. It now reports both probes every time, errors when a node carries both uplink veths (set up twice, and --teardown is the way out), and errors when external_mode and the wiring disagree — the datapath follows the wiring, so the config silently loses that argument. An explicitly configured bridge_mode gets the same cross-check, since nothing else ever compared it against the wiring. The vpc-networking doc now opens nat with "start with pool unless one of these applies" and gains a section on nat across more than one node; the admin warning for --external-mode=nat on two or more nodes says what was verified by hand and what is still uncovered. * test(nat): prove routed NAT delivers each node's own guests, on a cluster natuplink ran single-node only, which is exactly where the two defects that broke three nodes are invisible. It now admits ModeMultinode, and phase 9 takes a second shape there: a guest per node in a spread placement group, and for each one — its public IP plumbed by the node running it and by no other node, the route on-link with proxy-ARP answered there, a dnat_and_snat row carrying external_mac and logical_port, a TCP handshake in from another node, and the guest's own console verdict for egress. Every node's transit veth is checked against the one cluster-wide MAC before any guest boots. Single-node phase 9 gains the same row and on-link assertions, so the shape is held on both topologies. Two gates come from what three OCI nodes actually do: every node holds a gateway_chassis row for every VPC (the active one is the highest priority) and every node installs the VPC ingress route, so only phase 7 needs gating — it runs where the gateway port and the guest are both local, and logs which one is missing otherwise. Cell 20 is real-multi x nat with its own nightly step, leaving the permutation cells' suite set alone. * test(nat): run the VPC egress paths in nat mode, not pool mode only The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on external_mode = "pool", with a comment saying nat clusters have no EIP/NAT-GW support. They have had EIPs since the routed path landed, so the gate now asks whether the cluster can allocate public addresses at all — pool, or nat with a non-transit pool configured. The nat-single cell runs TestVPCEgressPaths beside natuplink, which is the only nightly coverage of a private subnet's isolation and its NAT gateway in routed mode. natuplink never creates a private subnet. * docs(config): list nat among external_mode values * fix(ci): renumber the multi-node nat cell off the bare-metal job Cell 20 is already the bare-metal single-node job, gated on the same cells input but declared in the workflow rather than e2e-cells.json. A matrix entry with the same number runs two unrelated jobs under one name, and naming 20 in a dispatch selects both. * feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests Oracle publishes its qcow2 checksums inline on an HTML page and ships no sums file, so the four new entries carry the expected digest in the catalog rather than a URL to fetch one from. Images gains ChecksumDigest for that, and the catalog import path verifies against it and says which of the two sources it used, so a recorded digest never reads as a fetched sums file. Pinning is safe here and only here: each URL names an immutable build (b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build alongside the plain -kvm- one; the cloud build is the cloud-init image and is the one that matches x86_64's -kvm-. TestImageCatalogReachable HEADs every URL in the catalog from the general single-node suite. These artifacts live on hosts nobody here controls, so an upstream that retires a build breaks `spx admin images import` for a customer and nothing else in the suite would notice. It also asserts that every entry sets exactly one checksum source, and that no pinned entry points at a "latest" URL whose bytes would change under it. * fix(formation): carry bridge_mode onto the leader and every joiner applyNetworkConfig copied external_mode, external_interface and the pools out of the formation NetworkConfig but never bridge_mode, so a multi-node cluster wrote a spinifex.toml without the key while single-node init wrote it. Every node then fell back to autodetecting a mode that init had pinned. Nightly cell 30 caught it: TestNATUplink failed at the config phase on a spinifex.toml that did not contain bridge_mode = "nat". Deriving it from the mode rather than adding a NetworkConfig field keeps the leader and the joiners on the same rule single-node init already uses. * feat(ui): show how to ssh into an instance, and which user to use An instance detail page gave the addresses and the key pair name but never the login user, which is the one part of the command nobody can derive. It belongs to the AMI, not to Spinifex: nothing in the launch path creates or renames an account, so stock cloud-init applies the distro's own default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is ubuntu, and guessing from the distro name gets two of those three wrong. The card resolves the user from the AMI's name and description, and an unrecognised image yields a placeholder rather than a guess, because a wrong username in a copy-paste command is worse than an honest gap. A private-only instance is called out as reachable from inside the VPC only, rather than presented as a command that will simply time out. * fix(nat): plumb host state for a NAT gateway's public address AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT node the host is what delivers a public address, so an egressing private guest left with a source the host held no route back to, and the masquerade rule — which matches the transit 100.127.0.0/24 alone — never saw the packet. The gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD accepts an EIP gets. hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs deliberately: pruneOrphanEIPs and addLive read that set as the complete list of addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so widening it would stop the sweep reclaiming a stale row on an address later reused as a gateway's. AddNATGateway also binds directly, on both the create and idempotent-skip paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP pass — and so a reboot that keeps the OVN row still replays the volatile half. Documents NAT gateways in nat mode in vpc-networking, and says at the head of that section that nat is a supported mode covered by nightly cells 19 and 30. * docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user Four catalog entries, all UEFI. The login user is the one that catches people: opc is the default on the images Oracle publishes into OCI itself, while these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle and ec2-user are all refused. Also records why these four pin a checksum digest inline (Oracle ships no sums file, and each URL names an immutable build), why arm64 takes the -kvm-cloud- build, and that SELinux ships enforcing. * fix(reconcile): stop reading an unreadable intent as an empty one A wedged JetStream KV returned no keys for the ENI bucket while writes were still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts read every guest LSP as an orphan against it, and every guest on the node lost DHCP and IMDS while the API kept answering and the instances stayed "running". The existing re-read guard did not help: it re-reads through the same path. Three defences, because no single one covers every shape of the fault. A failed kv.Get now fails the pass instead of dropping the record. Dropping one VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads as permission to delete every guest port in that VPC — one failed read, same outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that is a poison record rather than an unhealthy store, and failing on it would wedge reconcile forever. The port sweep now asks local OVS which taps are plugged in here and never deletes a port that has one. A running guest's port cannot be an orphan whatever intent says, and OVS is the one liveness signal that does not come through the KV read path the sweep is otherwise entirely driven by. Wired from vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the sweep rather than deciding on intent alone. Each sweep also refuses outright when the keep set it would delete against holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That is the case local taps cannot cover, because the drift sweep is leader-gated and lists NB cluster-wide, so the leader holds no tap for another node's guest. The cost is real: a genuinely orphaned row now survives until one live record of its class exists. Leaking a stale row is the recoverable half. Three existing prune tests drove their fixtures to a wholly empty intent, which is now the refusal case; they keep one live record of the class so they still prove the prune. The Get-failure path has no unit test — there is no jetstream.JetStream fake in the tree and an embedded server cannot be made to list a key it will then refuse. * docs(oci): separate the two address quotas, and the principal that can read them The regional reserved-public-IP ceiling is tenancy-wide and does not grow with node count, so a cluster hits it before the per-VNIC 64, which does. Records the commands that read both, and that a detached reserved public IP still bills. Reading limits or usage needs a tenancy-level grant, which cannot be compartment-scoped the way the node's policy is. That is a second, read-only principal kept off every node, not an addition to SpinifexOperators. * docs(oci): give quotas their own troubleshooting section, and say we enforce none Spinifex hardcodes no address ceiling: it attempts the allocation and maps OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with Oracle and nothing here is reconfigured or restarted — which is what an operator needs to know before it bites, not after. Separates the two quotas by how they scale: the regional reserved-public-IP default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and is not. * fix(oci): follow a guest to its new node when its EIP moves OCI delivers a public address to whichever VNIC holds the private IP behind it, and nothing ever moved that private IP. Binding.VNICID was written once at allocate time, as the VNIC of whichever queue-group daemon happened to serve the AWS request, and never updated — so an address landed on the right node only by chance, and a guest that stopped and started elsewhere went dark while every OVN row, host route and firewall rule on the new node was correct. ClaimLocalAddresses closes the gap from the node that should hold the address. Local OVS decides ownership, the same authority the host EIP binder already trusts for the same question, and UpdatePrivateIp with a new VnicId does the move in one server-side call. The private IP OCID survives it, so the reserved public IP attached to it follows without being touched and the customer's address never changes. A loop per node, not leader-gated: the question is about this host's guests, and a node that never wins the reconcile lease would otherwise never claim the addresses of the instances it is running. * fix(oci): record which ENI an EIP is associated with An EIP is allocated bare and attached by a later AssociateAddress, so the OCI binding written at allocate time named no ENI and nothing ever filled it in. The address then stayed on whichever node happened to serve the allocation: the affinity pass could follow an auto-assigned address, which is allocated with its ENI, but never an EIP — the case customers actually use. BindOwner is on an optional OwnerTracker interface rather than Allocator, because only OCI has a notion of an address being delivered to one node. A static or DHCP address is on the wire everywhere its node can reach, those allocators do not implement it, and ExternalIPAM treats a pool that does not as a success with nothing to do. The same gap made the release owner guard inert for EIPs: it compares against a binding ENI that was always empty, so a stale teardown could free an address already handed to another instance. * fix(ec2): report the address an instance is actually reachable on Three faults kept DescribeInstances answering with the address launch assigned, whatever happened afterwards. The launch-time stamp lives on the stored ec2.Instance, and the stopped path only declined to add an address rather than removing one, so a stopped instance went on advertising an address it had released and that may already belong to someone else. The projection now clears it, and the NIC association with it, working on its copy so a restart can still report the address. The running path filled the public address only when it was nil, which the stamp never is, so an EIP could not be reported however correctly it was recorded. The runtime address now wins. And nothing told the rest of the system about an association: AssociateAddress wrote neither the ENI record, which is what IMDS and the network reconcile read, nor the instance record, which is what a describe answers from. The ENI write is direct; the instance write is a fan-out, because the queue-group worker that serves AssociateAddress is rarely the node running the instance. * fix(net): take the host EIP plumbing off a node its guest has left Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept anything still wanted cluster-wide. A guest that moved therefore left the old node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with nothing that would ever remove them. On a shared segment that node goes on answering ARP for an address another node is now serving. The two halves now ask the same question by the same authority, in the loop that already runs per node. An unreadable local OVS is not a yes: tearing down a live address on a guess would fire everywhere at once, so it keeps the binding and the next pass tries again. A binder with no chassis test behaves exactly as before, and a centralised EIP is exempt because it hairpins through the gateway chassis and every node plumbs it legitimately. * fix(net): refuse to free an Elastic IP on behalf of a departing interface Releasing an instance's public address on teardown releases it by address, scoped to the ENI holding it. Since an association now writes the EIP onto the instance record, that address is the EIP — so terminating an instance gave a customer's Elastic IP back to the pool while describe-addresses still reported it as theirs, free to be handed to another guest. The existing ownership guard cannot catch this. It only fires when the stored allocation already names an ENI, and an EIP is allocated bare and associated later, so the stored ENI is empty and the guard passes straight through. The distinction that holds is who is asking. ReleaseAddress is the address's own owner and names no interface; instance teardown names the interface it is tearing down. So a release that names an owner may free an auto-assigned address and never an EIP, which is enforced in each of the three allocators. DHCP needed the scope carried on the wire, since it releases by address and resolves the lease afterwards. Absent on an older caller, which reads as false and keeps the previous behaviour. Worst on the two providers where it is not recoverable: on DHCP the lease is the only thing holding the address and the server promises nothing about handing it back, and on OCI the address is a reserved public IP object that the release destroys outright. * test(e2e): prove an Elastic IP survives a stop and start Instance teardown releases the public address by value, and since the association writes the EIP onto the instance record, that release aims straight at the customer's Elastic IP. The allocator guards now refuse it, but nothing end to end proved the address comes back on a restart. The step goes inside EIPFlip, which already runs last on the shared public guest for the same reason a stop is safe there: the address the guest returns on is the allocation's, not the pool's. It runs on the static cells and the DHCP ones alike, and on DHCP it doubles as the only outside view of the lease still renewing while the guest is down. * feat: release an auto-assigned public address on stop, take a new one on start AWS lends an instance a public address for as long as it runs and gives it a different one on the next start. Spinifex kept the address across a stop, which holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a lease nothing is using. A stop now offers the address back through the instance cleaner, which refuses anything an Elastic IP is behind — the EIP association record answers that, the same question an ENI delete already asks — and records on the VM that the start owes it a new one. A host drain keeps the address: Restore relaunches the guest on the next boot, and changing its address under it is an outage nobody asked for. The start allocates before the launch, so an exhausted pool refuses the start rather than booting a guest the customer cannot reach, adopts an Elastic IP associated while the instance was stopped instead of assigning a second address beside it, and hands the address back if the launch then fails. e2e: a StopStartAddress stage on the public guest asserts a stopped instance reports no address and comes back on a different one from a static pool, and a StopStartKeepsTheEIP step asserts the opposite for an Elastic IP. * docs: restructure the OCI guide around two deployment paths, with diagrams The guide had grown into a numbered wall with no shape: no frontmatter, no contents, and a reader with an OCI account had no way to tell where to start. It now follows the multi-node install template — overview, prerequisites, then the deployment path — and splits that path in two: Terraform for one node or three, which is the preferred route, and a manual single-node walkthrough for an instance that already exists. Both meet at one shared configuration section rather than repeating it. Two ASCII diagrams carry what the prose could not: one OCI instance with the whole AWS surface inside it, and three nodes with the same surface distributed, showing which plane each VNIC carries and how an address follows a guest that moves. The limits section was materially out of date and said so in the places that mattered most. Multi-node is no longer experimental, public addresses are no longer pinned to one node — each allocates on its own VNIC and the affinity pass moves the private half to wherever the guest is — and an auto-assigned address is now returned to OCI on stop, so a stopped fleet stops paying for addresses. What changed is recorded rather than quietly edited. The Terraform configuration moves in from the monorepo as scripts/terraform/oci-spx, so the guide points at something in this repository. Also marks the two new lifecycle test files in-package, with the reason each one needs to be: both substitute unexported dependencies. * docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan Replaces the two ASCII diagrams with SVGs in the house style, one for a single node and one for three. Renames the two deployment sections to Terraform deployment and Manual deployment, documents what the Terraform builds per node and every variable worth setting, and explains that the data volume is an iSCSI attachment rather than a disk. The manual path now covers the iSCSI login, the fstab entry and the netplan source routing that Terraform writes, and states what br-wan is for on a routed-NAT node: it names the VNIC for the OCI allocator and holds the registered addresses, but is never bridged into OVS. Opens with what Spinifex is and why OCI is worth running it on. * fix: make gateway convergence work on a multi-node routed-NAT cluster Three defects each stopped a multi-node cluster from ever attaching an internet gateway, and all three produced the same uninformative symptom: `terraform apply` sitting on `aws_internet_gateway.igw: Still creating...` forever while every service reported healthy. ovn-northd was given only its local OVSDB socket as a client connection. A RAFT follower does not forward writes -- it answers "not cluster leader, trying another server" -- so with one entry in the list, northd stops translating Northbound intent into Southbound flows the moment database leadership moves to another node. Measured on the OCI cluster: SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no chassisredirect port for any VPC created after leadership moved. setup-ovn.sh wrote the member list only when the caller passed --ovn-remote, which only install-node.sh did; it now derives the list from the live RAFT membership otherwise, and says so when it finds a single member, because that node needs a second pass once the others join. The gateway datapath probe ran wherever the reconcile lease landed rather than on the chassis owning the port. Both probes leave from the local uplink, and under routed NAT the transit subnet exists identically on every node, so a remote probe reported a healthy gateway as dead and aimed a repair at a datapath the node does not own. GatewayPortLocal gates it. On OCI the probe then ARPed the tenant-visible public IP, which the cloud NATs and which appears on no interface anywhere. The reconciler now maps the address through the same DatapathResolver the NAT rules are built from. Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed the destroy and stranded the VPC. The OCI guide gains a Terraform section recording which shipped workbooks were validated against a three-node OCI cluster and which are still open. * docs: record the validated Terraform workbook results for the three-node OCI cluster Six of the seven workbooks run against the OCI cluster pass, including the three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the clean eks-quickstart re-run, and names the two lifecycle gaps that bit during the run so an operator recognises them rather than reading them as a broken workbook. * fix: route the NAT gateway's SNAT source through the datapath resolver AddNATGateway was the one NAT path that did not map its external address to the one that rides the wire. Where the public half of an allocation is NAT'd by the cloud it is on no interface anywhere, so every private-subnet guest egressed with a source that was dropped, while the gateway reported available and DNS kept working because the VPC resolver is host-local. The unmapped address also failed the host EIP sweep, which works in resolved addresses, so the bind was pruned as foreign on the next pass and reinstalled on the one after. The public half now survives as an external ID instead. Also restructures the OCI guide around numbered steps through to running Terraform, and moves the cloud-init and metadata sections into troubleshooting where they are read when something is wrong rather than followed when it is not. * fix: place a NAT gateway's OCI address on the node that forwards for it OCI drops a packet whose source is not a registered private IP on the VNIC it leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis. Two independent placements decided those and nothing reconciled them: the address was created by whichever worker served AllocateAddress, while the chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster that made private-subnet egress a coin flip, and the same Terraform passed or died depending on which node answered. The affinity pass already solves this for guests, by asking local OVS where the tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which is exactly and only this address. The equivalent local fact is the VPC's chassisredirect binding, so ownership now has two answers rather than one, and an address that is neither a guest's nor a gateway's is still left alone. The VPC is recorded from the reconcile pass rather than from CreateNatGateway, because the answer is the gateway chassis and reconcile is what decides it. * fix: plumb a NAT gateway's host ingress only on its gateway chassis A NAT gateway's address has no ENI, so neither the bind's locality test nor the prune's could ever fire for it and every node installed its /32 route into OVN. Latent on bare metal, where pool addresses are disjoint from the node subnet; on a cloud that hands out the address from the nodes' own subnet it outranks the connected route and swallows node-to-node traffic. Give that address the same second authority the ocinet affinity pass has: the VPC's chassisredirect binding, read from the local ovn-controller. An unanswerable question still leaves the binding alone, so a single node whose gateway port is not yet claimed behaves exactly as before. The orphan EIP prune built its live set from ENI and user EIPs alone while also sweeping host ingress, so every drift pass tore down the ingress of every live NAT gateway and left it dark until the next host EIP tick. * fix: ask the gateway chassis only about a NAT gateway's address gatewayElsewhere skipped the NAT gateway marker it was named for and answered for every spec, so on a node that is not the VPC's gateway chassis every EIP — including a guest's, whose locality is its own port — was left unplumbed. Two of three nodes bound nothing. * test: assert the two private-subnet EKS workbooks in the tofu harness eks-https-ingress and eks-gitops-argocd put their workers in private subnets behind a NAT gateway, so a worker only registers Ready if its egress to the cluster endpoint and to ECR survives that path. That makes them the workbooks that catch a NAT gateway fault, and neither had an assertion, so neither could be run at all — run_workbook calls assert_<name> by construction. Both share eks-quickstart's readiness check, extracted unchanged. They also name the worker size node_instance_type rather than instance_type, and tofu rejects a -var for an undeclared variable outright, so the name is read from the config instead of kept in a list here. * test: count agents, not every node, when asserting EKS workers Ready The control plane is three K3s servers, so an unfiltered node count reaches a one-worker cluster's desired size before any worker joins — the assertion passed on exactly the failure it exists to catch, reporting "4/1 workers Ready". Filtering on the control-plane role label makes it 1/1, which is the number that says a node-group worker registered. Matters most for the two private-subnet workbooks, where the worker joining is the whole proof that egress through the NAT gateway works. * docs: record the two private-subnet EKS workbooks passing on OCI	2026-09-26 14:31:07 -05:00
Ben Duncan	deff3a1e1e	chore: dispatch the E2E Nightly from Cloudflare instead of a GitHub schedule (#1076) ... GitHub registers a schedule: only from a repository's default branch, so this cron could drive exactly one branch and always executed main's copy of this file whatever it tested. main was never exercised by the nightly, and a fix to the workflow itself could not be proved until it had already shipped — which is how cell-24's unguarded pkill stayed live on the schedule after its fix merged to dev. A Cloudflare Worker now dispatches this workflow once on dev and once on main. A dispatch takes both the workflow file and the source under test from its ref, so GIT_BRANCH is simply github.ref_name and every schedule-only expression goes with the cron.	2026-09-17 15:17:29 -05:00
Ben Duncan	2039c05c17	fix(ci): take the host benchmark lock before killing predastore nodes (#1065) ... cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d -config'`, which matches every predastore node on the runner and not only the ones this run left behind. The predastore repository runs the same harnesses on the same machine, so "still running" does not mean "stale" — on 2026-09-16 this step killed a predastore benchmark 32 seconds into its workload, and the run failed with 61 connection-refused errors that look nothing like their cause. Both kills now run under the host benchmark lock the harnesses themselves take, so anything alive while it is held really is a leftover. The second one needs it as much as the first: by the time cleanup runs the harness has released the lock, so another repository's benchmark can already be up.	2026-09-17 10:59:39 -05:00
Ben Duncan	8d7b0f0f47	ci: capture br-mgmt topology and peer reachability from every e2e node (#1042) ... Whether br-mgmt is a per-host island or one L2 segment spanning the cluster decides whether a system instance can reach the gateway address it was baked with. No artifact recorded which one a run built, and the environment is destroyed before it can be asked, so the decisive evidence for an LB agent heartbeat failure was gone by the time anyone looked. Records ports, address and fail_mode per node, then probes every node's br-mgmt address from every node. Islands reach only themselves and spanned segments reach everything; a mixed result is the failure this catches.	2026-09-15 13:13:05 -05:00
Ben Duncan	573a42a617	ci: run the e2e nightly every day, off the top of the hour (#1027) ... A weekend gap is two runs of evidence lost on a suite whose remaining failures are a rate rather than a bug. GitHub queues scheduled runs behind every other top-of-hour cron and sheds them under load, which is why 2026-09-14 never fired and why every run started 3h15m-3h45m late.	2026-09-14 15:40:25 -05:00
dependabot[bot]	b3a0a42c9a	chore: bump actions/download-artifact in the github-actions group (#1021)	2026-09-14 09:02:32 +10:00
Josh Siviour	b0faa72428	ci: benchmark Predastore across a real 3-node Spinifex env (#1015) ... * ci: benchmark Predastore across a real 3-node Spinifex env The nightly throughput job runs three s3d processes on loopback aliases on a single box against local NVMe, so nothing measures Predastore in the shape it ships in: three machines, with reads crossing a real NIC between hosts. This provisions a 3-node env the way the multi-node e2e job does, then hands its three gates to the existing harness through PERF_EXTERNAL_HOSTS. The harness skips its own start and stop and measures them directly, so the workloads and analysis are unchanged and the two numbers stay comparable. It runs on a systemd LAN runner rather than the ephemeral pool because a pool VM sits behind the geneve overlay at MTU 1408, which would put the runner's own network path inside the measurement. * ci: trigger the 3-node benchmark on its own branch for a first run A workflow_dispatch only resolves against the default branch, so this cannot be dispatched until it is merged. The push trigger is scoped to this branch and comes back out once the workflow is on dev. * fix: measure as the identity the provisioned cluster actually mints spx admin init generates a random access key pair at bootstrap and writes it to node1's ~/.aws/credentials, so the benchmark's built-in key is one this cluster has never heard of and every run died on the first CreateBucket. The pair is read and used inside a single step, the way e2e-ddil.yml does it: a step-level env: block is echoed verbatim into the run log, a shell variable is not. * ci: default the 3-node benchmark to the compare preset smoke was the right default while the pipeline was unproven. It is green now, and compare is what the nightly loopback run uses, so it is the preset whose numbers can be set beside it. * ci: make the cluster's ref a knob, pinned to v1.18.0 for one run The harness stays on this branch while the cluster is built from whatever ref is asked for, so an old release is measured with the same ruler as the current one. run-info.txt records the deployed ref rather than the harness checkout. The v1.18.0 default is temporary: a push event carries no dispatch inputs, and this cannot be dispatched until it is on dev. It comes back out after the run. * test: probe ranged GETs, which no workload here exercises Warp reads whole objects, so the throughput figures stay healthy while ranged GETs read the whole object per request. The probe reads the same boundary crossing range from a 64 MiB and a 256 MiB object: under read amplification the cost tracks object size, and otherwise the two match. Cluster ref is pinned to the fix branch for this run and comes back out after. * test: run the ranged GET probe against dev to prove it fails there A probe that has only ever been green proves nothing about the fix. Pointing the cluster at dev, which still has the defect, is the control that shows the probe detects it. The pin comes back out after. * ci: drop the branch push trigger and the cluster ref pin Both were scaffolding to run this before it existed on dev. The workflow is dispatch only again, and cluster_ref defaults to the branch under test.	2026-09-10 15:39:00 +10:00
Julian Sommer	368f4efa40	chore: trim comments from workflows	2026-09-10 11:48:14 +10:00
Julian Sommer	60509e7921	fix: remove currency block on e2e.yml	2026-09-10 11:39:43 +10:00
joshsiv-mulga	84565bc13b	fix: set ci.run_id on bare-metal dispatch runs ... A dispatch has no otel-attrs input, so the cell's telemetry reached the sink with no run id and could only be found by guessing a time window.	2026-09-08 07:46:00 +10:00
joshsiv-mulga	23d5b81cf0	fix: correct the eks-node build paths on the bare-metal cell ... spinifex is the workspace root there, not a subdirectory as in the nightly, so the copied paths resolved one level too deep.	2026-09-07 23:31:26 +10:00
joshsiv-mulga	01dd2a761a	feat: let the bare-metal cell run the eks suite ... The eks suite boots its control plane from the eks-node AMI, which only the nightly built. Bare metal had no EKS or mkosi step at all, so naming the suite there produced a run that could never pass.	2026-09-07 23:27:11 +10:00
Josh Siviour	8e11908dfb	fix: make the EKS control plane's disk latency observable, and give it room to run (#1009) ... * fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>	2026-09-07 21:53:41 +10:00
Ben Duncan	18853cef31	fix: close the four defects keeping the e2e nightly red (#1007) ... * fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main.	2026-09-06 16:53:32 -05:00
Ben Duncan	c2aad92044	fix: stop the detach seal, EIP teardown and key-pair races failing the nightly (#1003) ... * fix: stop the detach seal, EIP teardown and key-pair races failing the nightly Four independent faults the full-suite nightly surfaced, each of which had been masked while the heavy cells were skipped. unmountVolume SIGKILLed nbdkit the moment the request arrived. Removing the guest device has already started the plugin's own Close, which drains to the backend and leaves a seal receipt, so the kill landed 61ms into that flush and threw it away — the receipt path never fired once across a whole run. What was left was 74 orphaned WAL files for the fallback seal to replay, which ran past the caller's 180s deadline and returned DetachVolume ServerInternal with the volume still attached. It now terminates gracefully first, bounded by the plugin's own drain budget, and only escalates if that does not take. DeleteEIP returned early when the dnat_and_snat row was already absent, before the neighbour flush and the host unbind. An absent row is not a reassignment — nobody owns the address — so the /32 route and proxy-ARP entry were orphaned and a disassociate left the address still answering from the host. The reassignment cases keep their early return. EnsureKeyPair was not atomic: a failure after CreateKeyPair left the pair on the cluster with no cleanup registered, so every later caller failed with InvalidKeyPair.Duplicate in a test that never ran. The describe that follows a create also had no tolerance for the propagation window on a multi-node cluster; it now retries, and still reports a duplicate the moment it sees one. An admission refusal could not name its cause. The static vCPU/memory budget and the live MemAvailable clamp both surface as InsufficientInstanceCapacity, and they have different fixes; the subscription transition now records which gate closed. * fix: identify the default-deny by priority, not by being a drop DenyIsNotEthertypeScoped bucketed every drop ACL by direction, so the last one seen won each direction. The platform WAN egress block added at priority 1049 is also a drop and is legitimately ip4-scoped, so it displaced the default-deny in the map and the assertion fired on its qualifier instead: from-lport deny on sg_c94870c0d68288dc6 is IPv4-scoped: "inport == @sg_c94870c0d68288dc6 && ip4 && ip4.dst != {10.0.0.0/8, ...}" Select the two default-denies by their own priorities. What the test asserts is unchanged — the default-deny still must carry no ethertype qualifier, for the reason denyIngressACL states: OVN allows on no-match, so an ip4-scoped deny lets every other ethertype through. * fix: drain every EIP on an ENI at teardown, not just the first findByENI returned the first matching record and stopped, so an ENI holding more than one EIP had all but one left marked associated to an interface being deleted — with the host route still installed and no record left naming the ENI to find them. Teardown now clears them all. * ci: serialise e2e-nightly runs so two do not share the hypervisors Only iso-single and iso-multi were guarded, by the iso-qemu-rack group. Every other job could overlap a second run of the workflow, with both sets of cells building clusters on the same five hosts. Queue rather than cancel: a dispatch is usually a verification run whose result someone is waiting on.	2026-09-05 21:41:02 -05:00
Ben Duncan	ac5daa56ab	fix: make cell-24 able to pass — nvme work root and a realistic timeout (#1002) ... * fix: point cell-24 stress work root at the runner nvme The fault-injection harness ignores TMPDIR for large-object and multipart-upload, defaulting them to $HOME/.cache/predastore-e2e because TMPDIR is commonly tmpfs. On this runner $HOME is a 110G root disk with 14G free while TMPDIR is a 938G nvme, so the preflight guard passed on a filesystem the scenarios that need disk never touch. Set STRESS_WORK_ROOT alongside TMPDIR and check free space on both at a threshold that covers the 1GiBx8 multipart case, so a run that would skip a size for want of disk fails preflight instead of reporting a pass with less coverage. * fix: give cell-24 room for the full thirteen-scenario stress suite The scenarios run serially and are ~35 minutes of work on their own, measured from the per-scenario predastore matrix. The job allowed 30, so it could not have passed whatever the disk did.	2026-09-05 21:38:19 -05:00
Ben Duncan	2727f4f121	ci: build the nightly matrix from the selected cells, not filter inside them (#1004) ... The matrix job gated all 32 of its steps on a step output. A job whose every step is skipped still completes green, so a dispatch that ran one cell reported as a nineteen-job pass and was indistinguishable in the UI from a full one. The only tell was duration: a deselected cell finished in 11-12 seconds. Every green run in the recent history is such a dispatch, and every run that actually executed the suite has failed. There was no known-good baseline; there was a display bug. A job-level if cannot fix it — matrix is not available in that context, since it is evaluated before the matrix expands. So the selection moves ahead of the matrix: a select job filters the cell table and the matrix is built from what comes back. A cell that was not selected produces no job at all, which is plainer than skipped and cannot be read as a pass. The table moves to .github/e2e-cells.json so both the workflow and the filter read one copy.	2026-09-05 17:18:15 -05:00
Ben Duncan	828b5df4aa	fix: pause a guest on a backend I/O error instead of handing it EIO (#977) ... * fix: fail DRAIN when a volume seal failed shutdownAndUnmount logged the volume unmount error and returned nothing, so StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set. The coordinator then advanced to STORAGE and stopped viperblock while the seal was still in flight, tearing the VBState envelope and leaving the guest unable to mount its root volume on the next boot. The deploy reported success. shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup and stopOne carry it without aborting the rest of the teardown, since QEMU is already down by then, and StopAll aggregates the per-VM failures so handleShutdownDrain ACKs with Error set and never reports vms_stopped. volumeMounterAdapter.Unmount was the second swallow site: it logged each failed seal and returned nil unconditionally. It now attempts every volume and returns the aggregate, which is what the VolumeMounter contract already described. Terminate still tolerates a failed seal — the volumes are deleted next, so an unsealed block map loses nothing a caller can ask for. This does not fix the seal itself. It converts silent data loss into a refused upgrade that an operator can act on. * test: add the storagefault e2e suite for backend-outage guest survival Freezes predastore with SIGSTOP to hold connections open while answering nothing, which is the fault a timeout-free path cannot distinguish from a slow peer. On one node that is a total outage; on three it freezes every node but the guest's own, breaking the RS(2,1) shard floor. The suite writes and flushes a verifiable region before the fault and churns a separate unverified region during it, so an interrupted write is never mistaken for lost data. The root-filesystem load runs from the user's home rather than /tmp, which is tmpfs on this image and would move no bytes on the root device. * test: verify the root device is under write pressure, and size the outage The previous guard matched /^ *vda / against /proc/diskstats, where the device name is the third field, and read $7 which is ms_reading rather than sectors_written. It printed empty and asserted nothing, so a green run could not distinguish a surviving guest from an idle disk. It now samples sectors_written twice and fails unless it advances. The outage is sized from viperblock's absorption rather than guessed: a workload must fill DefaultMaxPendingBytes before backpressure engages at all, so a short freeze measures the retry budget rather than the fault. * test: add nbdkit fault injection to the storage-fault e2e suite The predastore freeze takes the whole storage cluster away from every guest on it, which is too broad to run beside anything else. Removing a single volume's NBD server is the same fault at one-volume scope: one host, one guest, nothing cluster-wide stopped. Both signals are here because they sit on opposite sides of the defect. SIGSTOP holds the socket open and answers nothing, so the guest blocks and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at the default reconnect-delay of 0 fails requests at once, which under werror=report aborts the journal. Signalling is guarded: a pid is matched on comm, our plugin and the volume argument together, and re-confirmed immediately before the signal, because these hosts run nbdkit for other volumes and pids are reused. * fix: pause a guest on a backend I/O error instead of handing it EIO Every virtio-blk device was created with werror=report,rerror=report, so a backend outage was delivered to the guest as EIO. ext4 aborts its journal on that and remounts read-only, and the write that failed is simply gone. Under werror=stop QEMU pauses the VM and holds the failed request, so a backend that comes back resumes the guest with nothing lost. The boot drive, the cold-boot data volume and the hot-plugged data volume all move to stop. Alongside it the NBD transport gets reconnect-delay=30, which pauses requests rather than failing them while a storage process restarts on the same socket; on the legacy -drive form it has to be file.reconnect-delay, because the bare name reaches the raw format driver, which has no such option. The QMP heartbeat now recognises run state io-error and issues cont. After ten attempts the instance is reported impaired and the cadence drops to one attempt per ten heartbeats, but it never stops: QEMU holds the request for as long as the guest exists, so a backend that returns an hour later can still be resumed. The storagefault suite gains the fault this was written for — kill the volume's own nbdkit, hold, then put it back on the same socket — and it now verifies a sha256 region across the outage rather than trusting the guest to be reachable. Three harness defects went with it: the socket-liveness poll ran unprivileged against a 0770 directory and could never succeed, the relaunched socket was left 0755 where QEMU needs group write, and the two variants shared one guest so a fault in the first was diagnosed in the second. * test: cover the io-error resume policy * test: stop two parallel tests racing on the terminate backoff variable * fix: report a guest paused on a backend I/O error as impaired A guest held by werror=stop kept reporting ok on both status checks. QMP stays responsive while the guest is paused -- cont is being sent and answered -- so the QMP failure gate that DescribeInstanceStatus keys on never trips, and the one state a customer most needs to see looked healthy. Instance status now reports impaired/failed while the resume budget is non-zero, timestamped from IOErrorSince so impairedSince is when the guest paused rather than when it was escalated. System status reports impaired too: a backend I/O error is an infrastructure fault, not a guest one, and that distinction is what the two checks exist to draw. The counter is cleared by the first poll that sees the guest running, so both clear on recovery. * test: reap the relaunched nbdkit when the fault test ends relaunchNbdkit detaches the server with setsid so it outlives the SSH session that started it, and viperblockd never learned it exists. Nothing reaped it, so every run left an nbdkit holding a socket and a cache for a volume that was later detached or deleted -- four were found running on dev-prod against two detached and two deleted volumes. The PID is resolved from the socket path rather than the volume id, so a legitimate viperblockd server for the same volume is never the one killed. * test: prove the backend is serving before a fault is called cleared Thawing the process a fault stopped is not the same as restoring the service. The outage test treated SIGCONT as the end of the fault and went straight to judging the guest, so it passed while the cluster it ran on stayed unable to serve a single object for the next 35 minutes, and the next test would have inherited that. thawPredastore now puts and gets an object through every gate, with a budget, and fails the test if the backend does not come back. * fix: do not hand back a mount entry whose export is gone A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit was gone. Every later ebs.mount then matched that entry and returned its socket path, so vmMgr waited for a socket nothing would ever create and the instance could not be started again — on dev-prod a guest stopped against a frozen backend was left permanently stopped, four start attempts failing identically on a socket the node had already removed. Drop the entry when there is positive evidence the export is dead: the socket is absent, or the recorded pid is not running. Neither can be true of a live export, so the double-writer hazard the guard exists to prevent is unaffected. Also make the durability test's pattern helpers report failures. The read dd sat in a pipeline, so its exit status was discarded and the harness hashed nothing and returned the empty-string sha256 as a checksum. * test: report the guest console when a restart never answers SSH A restart that reaches running but never comes back is indistinguishable from a dead root filesystem without the serial console, and the log is gone once the fixture terminates the instance. * fix: converge guest port datapaths concurrently, not one deadline after another Each guest port that misses its SB binding burns the full convergence deadline, and applyEIPs spent them in series. Three ports belonging to long-stopped instances therefore held a newly launched guest's EIP dark for the sum of them before its own probe started. AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and sb-cluster-state-reset is now serialised with a minimum interval so concurrent probes cannot each fire a cluster-wide resync. * fix: refuse a stale cross-node start after the owning node is gone The volume lease already refused a cross-node mount, but it is a liveness claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded writes stops renewing, and 45 seconds later nothing refuses a start elsewhere that loads a stale backend checkpoint. A failed seal now writes a durable marker naming the node holding the only current copy, cleared on a seal that reaches the backend and on delete. The lease covers a live holder, the marker a dead one. The refusal also reaches the caller instead of being flattened to ServerInternal, which told an operator to retry something that cannot succeed while the holder is down. * fix: keep the daemon's message when a start is refused Rebuilding the error from the code alone rendered the generic ErrorLookup text, so a refusal naming the node holding the volume reached the caller as boilerplate about attaching volumes. * fix(ec2): terminate detaches every DeleteOnTermination=false volume TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume on the assumption that Stop's Unmount had already cleared its attachment. Unmount only clears it when the seal succeeded, so a volume whose seal failed stayed in-use pointing at an instance that terminate then deleted, leaving it unreachable with no owner to release it. The running-instance path already detaches every DoT=false volume; the stopped path now matches it, and tolerates an already-gone volume the same way. DeleteVolume also keeps its provider error on the way out. The terminate reaper retries the delete every two minutes and logged a bare ServerInternal, which made a blob node being down indistinguishable from a transient fault. * fix(ec2): DetachVolume --force releases a volume whose instance is gone The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a volume left attached by a part-way teardown could never be detached: the instance no longer runs anywhere and nothing answers. That also made it undeletable, since DeleteVolume refuses an attached volume. ForceDetachVolume already existed for this deadlock but was reachable only from account teardown. Force now falls back to it when the instance has no responder, which is the one case where clearing the attachment cannot strand a live guest. * fix(viperblock): unsealed writes steer placement instead of pinning a volume The dirty marker refused any mount from a node other than the one holding unsealed writes. Instance start already forwards to the node that last ran the instance and falls back after a 30s window, so that refusal turned a working fallback into a volume that could run nowhere once its owner was gone. It traded a rare data-loss risk for a permanent availability failure. The marker is now a record and a warning, not a gate. A takeover logs the previous holder and why it could not seal, then adopts the marker so the old node is no longer treated as the better source. Live exclusion is unchanged and still enforced by the lease. It is also written on open rather than on seal failure. A node killed mid-write never reaches its seal, so the old trigger left no trace in exactly the case the marker exists to describe. Adds 'spx admin volumes unsealed' to see which volumes would cost something if started elsewhere. * fix(viperblock): fence a volume whose lease moved instead of logging it The lease was checked once, at mount, and never again. A node that lost it kept its export up and kept writing into a volume another node owned, so both copies advanced and the backend settled it by arrival order. Losing a renewal now decides: another holder means tear the export down, no holder means the entry merely aged out, so reclaim it and leave the guest running. Fencing never seals — this node's copy is the stale one — and leaves the dirty marker for the winner to take over. This is not an epoch fence. It narrows the window from the 45s TTL to one renewal interval at no write-path cost; the backend still cannot refuse a stale writer. * fix(network): advertise EIPs so a recycled address is reachable The gateway switch port only asked ovn-northd to publish the router's NAT addresses in centralised and routed modes, so on a distributed cluster nothing ever announced an elastic IP. OVN still answers ARP for it, which is why a first-time allocation works and hid this — but no gratuitous ARP is sent when the address moves to a new ENI, so every host on the segment keeps resolving it to the previous holder's MAC until its cache ages out. Reusing an address from the pool then lands on a dead MAC and the instance is unreachable from outside. Distributed mode needs the option as much as the others: northd scopes each entry with is_chassis_resident(<vif port>), so the announcement leaves from the chassis running the instance rather than the gateway, which is what distributed NAT wants. Converge on the already-attached path too. AttachIGW returns early once the gateway port exists and nothing else rewrites it, so without that an existing VPC would stay silent for its whole life. Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to carrying their correct external_mac, each scoped to its own ENI's port. * fix(viperblock): make the volume fence and its dirty marker ownership-safe Four corrections to the lease-loss fence, all found in review. Drop the reacquire path. Losing a lease and finding no successor was treated as a lapsed entry worth reclaiming, but leases.acquire returns the cached entry — already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease renewed nothing and the node went on writing a volume it could not prove it owned. The two errors are not symmetric: a false fence costs one guest a restart, a false reclaim leaves two nodes writing one volume. Fence in both cases, and report which happened through the outcome attribute. Surrender a lease this node cannot confirm. Renewal only noticed a loss when the server rejected an update, so a node partitioned from NATS kept its export up while the TTL expired and another node took the volume. Track the last confirmed renewal and give the volume up once that ages past the point where the server may admit another owner. Bound the renewal call too, so a hung JetStream request cannot hold the goroutine past that point. Do not act on a kill that did not take. Both the fence and ebs.unmount removed the mount entry, released the lease and announced the result before knowing nbdkit had exited — the fence then reported a writer stopped that was still running, and the unmount sealed into a directory a live process was writing. Wait for the exit, and on failure leave the volume mounted and say so. An already-exited process now counts as success, which is what the caller asked. Give the dirty marker a generation. It recorded an owner and nothing to order two nodes' claims, so a node returning after a takeover could clear or overwrite the winner's marker and point placement back at the stale copy. Stamp the lease generation, refuse a write from an older one, and only clear a marker this node still owns. A marker that cannot be written now fails the open rather than leaving writes nothing records. * test(viperblock): cover the stall surrender and the unmount kill guard The two ownership cases the fence work left untested. A holder partitioned from NATS while the winner keeps it never gets an update rejected, so revision checking cannot see it — it has to give the volume up on its own clock. Pins that validity sits below the server TTL with room for a check in between, that a lapsed holder reports the loss as a stall rather than as a takeover, and that releasing a surrendered lease leaves the entry for whoever holds it now. The unmount side of the kill guard: sealing under a live nbdkit writes a state torn between two writers, so a kill that did not take has to fail the unmount and leave the entry mounted for a retry. * fix(e2e): keep the guest private key for as long as the key pair EnsureKeyPair computed the PEM path from the caller's directory but wrote the file inside the ensureOnce closure, which runs only on the first call. Callers passed a per-test artifact dir, and ArtifactDir removes that dir when the test passes — so the first test's success deleted the only copy of the private key while the key pair stayed registered, and every later call returned a path nothing had ever been written to. Guests after the first were then launched against a public key whose private half no longer existed, so the harness could never authenticate and gave up after the full five-minute budget. The guests were healthy and reachable on tcp/22 throughout, which is why this read as a storage or network fault for so long. The directory parameter was the defect, so it is gone: the PEM now lives in a directory the fixture owns and removes with the key pair, and no caller can supply one that is shorter-lived. Also reorder the killed-nbdkit teardown. The reaper registered its own cleanup after the detach registered one, so LIFO killed the block server before the platform detached the disk and DetachVolume then failed with "Node ... is in use". The detach cleanup now reaps once it is done. * fix: stop leaking an nbdkit export when recovery cannot adopt it Restart recovery adopts a guest's boot volume, then routinely fails the -efi volume with "claim lease: context deadline exceeded" -- JetStream is still catching up at daemon start and the KV claim runs on the client's 5s default. Recovery gave up on the first error and never retried, so the export stayed live with no registry entry. At terminate the unmount found nothing to seal, returned NotFound and killed nothing, and the process outlived its guest holding the volume and socket open. EFI is always the second volume adopted, which is why it was always the one that leaked. Two changes, because either alone leaves a hole: - rebuildMountedVolume retries the build (4 attempts, 3s apart) so a JetStream that is merely slow no longer orphans an export for the life of the process. - unmountVolume reaps a corroborated export for the volume it was asked to unmount when the registry has no entry for it, so an export orphaned by any means is still cleaned up. The reap goes through reapOrphanedNbdkit, which declines anything a live process references: at unmount the guest's blockdev may not be torn down yet, and killing the server underneath it strands QEMU. * test: hold the backend-serving probe and assert predastore readiness The backend-outage test declared the backend restored on one successful round trip through each gate. A gate can answer the request that arrives just after a thaw and then lose its blob peers under the load that follows, which is a backend that never recovered reported as one that did -- and that is what happened: the probe reported all three gates serving minutes before the guest was still taking BLOCK_IO_ERROR on every resume. The round trip has to hold across three rounds now, and the count restarts rather than decays when one fails, so a gate that serves and then stops is the failure the hold is looking for rather than noise it averages out. It also checks /readyz on each host and reports any failed check by name. The round trip only proves this workstation can write through a gate; it says nothing about which blob peers that gate can reach, and a gate one node short of the write floor serves a small object and fails a guest. A host that serves no admin listener is logged and skipped, since the port is optional configuration. * fix: scope predastore's admin port to the cluster plane in the host policy predastore's admin listener binds the host's cluster bind_addr and serves /healthz and /readyz unauthenticated. Readiness now names which blob peers a gate cannot reach, which is exactly the kind of topology detail 4432 is peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the public one. * fix: keep resuming a paused guest at heartbeat rate after a long outage Past ten attempts the cont dropped to one per ten heartbeats, so a guest whose backend had come back could sit paused for another five minutes before anything retried it. Measured on dev-prod: the backend was proved serving through all three gates while the guest was still waiting on attempt 60, five minutes after attempt 50. The slowdown was there to avoid paying a failed I/O every thirty seconds on a backend still down. That is nearly free, and five minutes of downtime after recovery is not, so only the log line is throttled now. * test: fail the run when a key pair name resolves to more than one pair A create that lands twice leaves two key pairs with one name and different material. RunInstances resolves the name to one of them while the harness holds the private key of the other, and the only symptom is every guest in the run refusing the key at the SSH gate — five minutes of waiting per guest, four guests, and nothing anywhere naming the cause. Observed on dev-prod: two pairs created a millisecond apart, both named e2e-key-30de10ba. One DescribeKeyPairs after the create says so outright instead. * test: tolerate an already-released volume in the detach teardown Teardown runs DetachVolumeWait after a terminate that releases the volume itself, so the detach can arrive at a volume already on its way to available. That is the outcome it wanted, and treating the IncorrectState refusal as fatal failed a backend-outage run whose assertions had all passed — the guest came back, every acknowledged write verified, and the test still went red on the way out. VolumeInUse is still fatal: that one means the volume is attached somewhere it should not be. * test: carry the merged suites onto the reworked key-pair and account-scoped seams The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir this branch removed, and DeleteVolume's metadata lookup is account-scoped now, so the provider-failure test has to seed and delete under the same account. * refactor: make the fence outcome a type in one const block Four separate exported string constants with a paragraph of comment each, passed to RecordVolumeFence as a bare string. Group them, give them a FenceOutcome type, and take that type at both call sites so an arbitrary value can no longer reach the metric attribute. * refactor: move viperblockd's wire contract into a leaf vbwire package The daemon imported viperblockd for exactly two symbols -- the fenced volume subject and its event -- which put the storage service back on the daemon's dependency graph to name something it only listens for. vbwire holds the contract instead: the subject and event, the dirty bucket name and record shape, and the read-only listing an operator tool needs. It imports nothing from spinifex, so both the daemon and the admin CLI drop their edge to viperblockd entirely. cmd/service.go keeps its one, which is genuine: it constructs the service. * docs: drop a specific host name from the start-refusal comment Named a real dev-prod host in a comment and in test fixtures. Say what the name is instead. * ci: give storagefault a dedicated nightly cell The suite is eligible for both topologies and runs on the PR gate, but no nightly cell ran it: the permutation cells run a deliberately narrow subset and none of the dedicated suite cells named it. Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that environment would report the outage as its own failure. * ci: run storagefault nightly on three nodes, not one The previous commit put its dedicated cell on a single-node runner. That was the wrong topology and the wrong justification: single-node is RS(1,0), where the shard floor cannot be broken and a cross-node takeover after a failed seal does not exist, so the two assertions worth having a dedicated cell for were the two it could not make. Cell 29 is now real-multi on ci-multi. It also corrects the claim that the PR gate already covers the multi-node case. The suite is eligible for both topologies, but e2e.yml's default suite list never names it, so it has never run there. A full pass is ~63 minutes on dev-prod bare metal, almost all of it deliberate waiting -- one test holds predastore SIGSTOPped for seven minutes -- which is too long to add to a gate that currently finishes well inside its budget. Nightly is where it belongs. Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since 90 minutes of suite cannot run in a 50-minute job; the permutation steps gated on matrix.suite == '' so the dedicated cells do not also run them; and a Run Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts the suite against E2E_SUITES_MULTI before running it. * fix: stop a startup KV race leaving vpcd serving keyless guests vpcd opens five KV buckets while NATS is still coming up, each attempt carrying a ~5s JetStream API deadline, and applied three different policies to the same failure: spinifex-vpc-enis was fatal, while instance-state, security-groups, subnets and vpcs each degraded silently for the lifetime of the process. A five-second blip during a cold start therefore decided how a node served metadata until someone restarted it. On dev-prod all three nodes hit this after a deploy. Two were saved by accident: their ENI bucket also timed out, so they exited 1 and systemd restarted them cleanly. The third's ENI bucket opened in time, so it never crashed and ran for hours with no instance record loader. With the local state file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's crawl died on ami-id and fell back to DataSourceNone, and every guest launched on that node booted with no authorized_keys. kvstore.OpenWithRetry retries for a bounded window before any policy applies, logs the cause per attempt and the wait on a late success, wraps the last error with the bucket name, and returns at once on a cancelled context. All five opens use it. Failing to open the instance record space is now fatal, matching the ENI bucket: the dependency whose loss stops guests logging in should not be treated more leniently than the one whose loss stops ENI resolution. Also fixes the reason the fallback was load bearing at all. The local state file was written 0600 and vpcd reads it as a different user in the shared spinifex group, so the local-first path had never once worked and its mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so an inherited umask cannot undo it. Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an unresolvable instance as "no key pair" with nothing logged. * fix: scope the ime-* firewall accept to IMDS and VPC DNS only The per-tap OVS demux delivers any guest packet addressed to 169.254.169.254 or .253 onto an ime-* host port regardless of L4 port, and iifname "ime-*" accept then admitted all of it. Every wildcard-bound host service — sshd, the AWS gateway, northstar, the console — answered guest packets on those addresses; only asymmetric reply routing kept the handshake from completing, and a guest could still drive northstar to recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS (.253:53), letting everything else fall through to policy drop. * docs: document the instance-to-host plane and how to protect it without the ISO The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by host services and is invisible to a NIC-level or cloud firewall, since the traffic is intra-host on the OVS bridge. Explain that the ISO ships the scoping rule armed, that the binary installer ships the firewall off, and that a non-ISO install must enable it with setup.sh --firewall=on or firewall_enabled=true. Include verification from an instance and the host. * fix: withhold the IMDS meta-data listing when the instance does not resolve Every omission from the meta-data root is a positive claim about the instance — public-keys/ absent means "launched without a key pair", and cloud-init believes it, writes no authorized_keys and boots the guest keyless. So resolving the instance was best-effort for a listing whose whole meaning depends on it: a backend error or a miss shortened the listing instead of failing it, and nothing anywhere recorded that the answer was a guess. Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot retry a 200 that is merely wrong, so this converts a permanent silent failure into a transient loud one. servePublicKeys already had this policy for the leaf; the root is what contradicted it. A profile lookup that fails still serves the listing minus iam/, and now logs. The key pair is the field a guest cannot recover from losing, while a missing role surfaces through the credentials leaf, which reports the miss itself. Also stops the storagefault cross-node test failing on its own access path. It tolerates the pattern changing across a takeover, but readPattern needs sudo, and a volume opened from the backend checkpoint may predate cloud-init writing its sudoers drop-in — indistinguishable from an unreadable device at the dd call site. It now checks sudo first, and reads cloud-init's recorded datasource to tell a rolled-back root filesystem from IMDS never having answered, failing loudly on the latter rather than accepting it as storage loss. --------- Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>	2026-09-03 19:45:33 -05:00
Bryn Mailer	18aa67b0e9	feat: add the e2e guest disk-performance gate (#964) ... * feat: add the e2e guest disk-performance gate Nothing measured guest disk performance in CI, which is why a quadratic index scan shipped in viperblock 35dd9f2 and survived three subsequent fixes to adjacent code. It is also why the two availability clauses at the end of the ebs-performance plan's Stage 0 have never been evaluated: a sustained random-write workload took the guest, and then the host, down, and nothing since has demonstrated that it no longer does. tests/e2e/diskperf/ launches an instance, attaches a dedicated blank volume per repetition, and drives a pinned fio profile over guest SSH. Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit criterion names, and randrw-70-30-4k, the mixed workload the headline IOPS came from. The target is a raw device with direct=1 rather than a file on the boot disk, so guest page cache, filesystem allocation and journal traffic are out of the path. Two classes of assertion, kept apart because folding them together lets a run that left the host unreachable pass on a respectable IOPS figure. Availability is hard pass/fail and absolute, so it gates from the first run: the job completes inside its budget, a guest responsiveness probe answers throughout on a second SSH channel, and the host's own sshd answers throughout. The guest probe reads the root volume while the workload runs on a different one, so a pass is evidence of isolation between volumes rather than of the job merely being slow. The host probe is a TCP connect plus an SSH banner read, which needs no credential and still proves the kernel scheduled sshd. Throughput and latency are banded against a committed baseline: fail above 25% regression, warn above 10%, per-metric median across repetitions. baseline.json is committed and embedded, with every figure zeroed, and a zero reference records and warns rather than failing. Only the availability class gates until numbers are captured on the bare-metal cell; a throughput band nobody measured would be a fabricated gate. A fresh volume per repetition is the reset. Extent count grows monotonically, so reusing a volume makes the gate order-dependent and drifts its own numbers upward over time, slowly normalising the regression it exists to catch. Each job is its own guest invocation and fio's JSON is fetched by a separate command, so a wedged job loses only itself and reports as a budget breach rather than a parse failure. e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable workflow serves both the correctness cell and nightly cell 28, which queues behind cell 20 on the shared concurrency group. Cells 25-27 are the dedicated eks, ecs and rds cells. The per-suite Go test timeout is resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares its 180-minute budget beside rds's rather than in a second place. The gate's own arithmetic runs in preflight via test-harness: band edges, the unmeasured cases, per-stream metric selection, fio JSON parsing, and the probe accounting including a slow-but-successful answer counting as a breach. A gate whose comparison logic is untested is the failure mode the plan is otherwise full of. The extent-count ceiling named as a fourth availability assertion is not implemented: it needs viperblock.index.entries from S1.5, which does not exist. * feat: let a manual bare-metal run choose its e2e suites A dispatch run only sees inputs declared under workflow_dispatch, which had none, so suites and suite-env resolved empty and fell back to the correctness set. Capturing a diskperf baseline by hand was therefore unreachable without a workflow_call caller. * fix: build the bare-metal suites the run step will execute The build step packs pre-built binaries into the tarball and did not see the suite selection, so a selected suite was absent from the node unless it happened to be one of the three the build script hardcoded. * feat: commit the measured disk-performance baseline Medians of three repetitions on the bare-metal cell. The throughput and latency bands now gate instead of warning, so a 2x regression in either job fails the suite. randwrite-4k's write tail is two orders above the mixed job's. It is recorded as measured rather than zeroed: an ungated metric would miss the regression class the gate exists to catch.	2026-09-03 14:33:21 +10:00
Julian Sommer	4522494c17	fix: define the per-suite e2e test timeout once, in suite-sets.sh ... The timeout was written out separately in each workflow and drifted: rds ran 50m on the PR gate and 30m nightly, where it panicked and took six in-flight subtests with it. e2e-suites.yml had the same hardcoded 30m on its single path, which also runs rds. A new e2e_suite_timeout helper sits beside the eligibility lists it already guards, and all four call sites resolve through it. e2e.yml loops over suites on the remote VM, which has no checkout, so it resolves the pairs on the runner and looks them up in the loop. Also lift the nightly cell ceiling to 50 minutes so the rds cell is not cut short by the job before its own timeout applies.	2026-09-03 09:21:17 +10:00
Julian Sommer	be94beb52a	ci: run eks, ecs and rds nightly in dedicated cells (#984)	2026-09-02 16:16:05 +10:00
Julian Sommer	12408268be	chore: trim unused entries from the go build cache before saving ... The cache was a union that only grew: each run restored everything, added what it compiled and saved the result, so objects superseded by later commits stayed for Go's full five-day window. Deleting every entry and letting it rebuild took Build and Test from 3434 MB to 478 MB, restore from 75s to 13s and save from 64s to 7s. Dropping entries nothing has touched in four hours holds it near that. Go bumps an entry's mtime when a run uses it, so the working set survives; the step runs after the tests, so a pruned entry costs a later run a partial recompile at worst and never the current one.	2026-09-01 14:26:54 +10:00
Julian Sommer	da92c7a083	chore: only save the go build cache from dev and main ... Restore is what makes CI fast; save is what got expensive. A same-era A/B across the switch to this cache shows what restore buys: wall time went 6:50 to 3:52, Build Spinifex 63s to 10s, test-race 385s to 188s. Since then the cache has grown to its steady state and the work has not moved, so Build and Test now spends 75s restoring and 64s saving where it once spent 13s and 8s. A PR's save is read by nothing else. Splitting restore from save and gating the save on dev and main keeps every branch reading a cache a dev merge refreshes minutes later, and takes the upload off the 56% of runs that are pull requests.	2026-09-01 14:19:56 +10:00
Julian Sommer	91bab3b910	chore: merge integration job into lint	2026-09-01 13:58:03 +10:00
Julian Sommer	3cfa94d49a	chore: drop the go build cache age census ... It answered its question: of 18 GB on disk, 1.45 GB is untouched for a day and nothing at all survives three. Go's own trim and the rate CI runs already keep the cache live, so pruning before the save would shed under 4% of the bytes at a two-day cut and risk recompiles for it.	2026-09-01 13:54:32 +10:00
Julian Sommer	805dab275c	chore: measure go build cache staleness in CI ... The build cache key rolls per run, so each job restores the previous run's cache, adds to it and uploads the result. Nothing removes entries, and Go only trims at 5 days, so every restore and save carries whatever dead weight has accumulated. Build and Test moves 3.2 GB per run: 59s down, 56s up. Whether pruning that is worth doing depends on how much of the 3.2 GB is still live, which is not visible from outside the runner. This reports the age distribution and deletes nothing. It comes out once the number is known.	2026-09-01 12:57:36 +10:00
Julian Sommer	a5d942ec24	ci: skip the e2e suites on test-only pushes ... A unit test cannot change what e2e exercises, so a push touching only _test.go files no longer provisions the tofu VMs. tests/e2e/** is listed after the exclusion so the e2e tests themselves still trigger their own suites.	2026-09-01 11:58:27 +10:00
Julian Sommer	62c1378b79	fix: raise the Ubuntu minimum to 24.04 (mulga-ucvsq) ... Ubuntu 22.04 was accepted by detect_os but the runtime apt list has never resolved on jammy, which ships dhcpcd5 rather than dhcpcd-base. No CI cell or E2E matrix installs on it, and the install docs already claim Ubuntu 26.04 and Debian 13 only, so drop the claim instead of carrying a jammy-conditional package branch for an untested suite.	2026-08-27 10:49:46 +10:00
Julian Sommer	9637978177	fix: pin the ISO cells' spinifex checkout to GIT_BRANCH ... Cells 21 and 22 compile the checked-out spinifex tree, but their checkout had no ref, so a scheduled run built main against siblings pinned to dev.	2026-08-26 11:43:28 +10:00
Julian Sommer	e36c8f6ba8	chore: update to go 1.27	2026-08-20 10:37:52 +10:00
Julian Sommer	7eff416c25	fix: collect IPsec and OVN datapath logs in the nightly e2e bundle ... The nightly collected only spinifex-* journal units, so a cross-chassis black-hole could not be attributed to IPsec SA establishment versus SB port binding. ovn-controller, ovn-northd and ovs-vswitchd log to files rather than journald, so they were absent from the bundle entirely. Adds the openvswitch-ipsec.service journal per node, mirroring e2e.yml, and pulls /var/log/ovn and /var/log/openvswitch off each node.	2026-08-19 12:06:35 +10:00
Josh Siviour	0e08b9d256	feat: ship per-test JUnit results to the Elasticsearch sink (#829) ... * feat: ship per-test JUnit results to the Elasticsearch sink Every e2e workflow already converts its test log to JUnit XML and renders it into the job summary, then discards it with the artifact. mulga-ci-runs records which suite failed; nothing records which test failed, so root-causing a failure means reading raw Actions logs across historical runs. The junit-to-es action bulk-indexes one doc per leaf test case into mulga-ci-test-results: name, subtest, status, duration, failure text, run id, cell, suite, branch and sha. Parent t.Run wrappers are dropped so a failing subtest counts once, matching the job summary renderer. The doc id is deterministic, so a replayed post overwrites rather than double-counting. Called from every workflow that produces JUnit XML, and tolerated on failure: an unreachable sink must not turn a green test run red. * feat: classify unparsed JUnit cases as unknown, not error go-junit-report emits <error message="No test result found"> when a === RUN line has no matching result line, which means the log it was fed was truncated or filtered. Recording that as a failure would put 78 phantom failures per green bare-metal run into the pass rate.	2026-08-18 22:54:54 +10:00
Julian Sommer	333248568e	fix: use actions/cache@v6 + test-actions clean	2026-08-17 11:34:50 +10:00
Julian Sommer	ae0337d31b	test: run the unit and race tiers clean in CI ... Go stores test results in the build cache alongside compiled objects, so now that the cache persists across runs an unchanged package replays its previous result instead of executing. That is what makes a local run fast and exactly what a merge gate should not do: a green check would no longer mean the suite ran against that commit. Add a GOTESTFLAGS hook, empty by default so local runs keep reusing results, and pass -count=1 from the two CI steps that gate a PR. Compilation is still cached either way, so this costs no build time.	2026-08-17 11:28:59 +10:00
Julian Sommer	20af6e5a4a	fix: stop the Go build cache freezing at the last go.sum change ... setup-go only writes its cache on a key miss, and keyed it on spinifex/go.sum alone. Once a run saved that key every later run logged "Cache hit ... not saving cache", so the cache stayed frozen at whatever the first job to finish happened to have compiled and nothing built afterwards was ever stored. All four jobs also shared that one key, so the job that won the save race decided what the other three restored. The race job was handed artifacts built without -race, and the build job artifacts built without GOFIPS140, neither of which yields a single usable entry. Manage both caches directly instead: modules keyed on all five go.sum files so the sibling modules stop re-downloading, and build artifacts keyed per job with a per-run suffix so each flavour keeps its own entry and refreshes it every run.	2026-08-17 11:14:34 +10:00
Julian Sommer	c99b91bca5	fix: add publish system ami job to release	2026-08-17 09:56:30 +10:00
Ben Duncan	42d87d7a5a	feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799) ... * feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.	2026-08-15 14:22:49 -05:00
Josh Siviour	9102588e5a	chore: bump Go toolchain to 1.26.6 (#793)	2026-08-14 09:29:03 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Josh Siviour	98ec781629	feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770) ... * feat: introduce the EBSProvider interface and control-plane metadata store Phase 1 of decoupling the control plane from viperblock. Adds a provider-neutral block-storage contract that spinifex owns end to end. ebsprovider defines EBSProvider, whose operation set and idempotency rules follow the CSI controller service, with NATS request/reply as the transport rather than CSI's gRPC. Every request and response carries an explicit schema version so wire skew fails loudly at the boundary instead of silently decoding a changed payload. The package deliberately exposes no viperblock types: provider-specific state is an opaque handle. Ships with a NATS client against the ebs.provider.v1.* subjects and an in-memory fake for tests. ebsmetadata gives spinifex its own volume and AMI documents under a spinifex key prefix, so the control plane can answer DescribeVolumes and DescribeImages without reading the storage engine's persisted state. The EC2 volume and snapshot handlers gain provider-backed paths for create, describe, modify, attach-state and delete. Provider injection is optional for now, so the existing engine path stays the default until the viperblockd side serves the new subjects and the metadata migration lands. vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol it used and is now a local helper. * feat: serve the EBS provider contract from viperblockd Gives the ebsprovider NATS client a server. Volume create, describe, expand and delete, plus snapshot create and delete, are now handled by the daemon that owns BaseDir and the mounted-volume registry, which is where viperblock engine construction belongs. Every handler prefers an already-mounted engine over opening a second one on the same volume, and expand refuses outright while the volume is mounted. Opening a second engine on a live volume is the double-writer hazard this decoupling exists to remove, so the boundary enforces it rather than trusting callers. Requests carrying an unrecognised schema version are rejected at the boundary instead of being decoded on a best-effort basis. Snapshot creation replies immediately with an operation ID and completes asynchronously on a per-snapshot subject, since a large volume will outrun the request timeout. Draining the volume stays with the caller: it coordinates with the guest and is not something the storage boundary can reach. The snapshot create subject gains an explicit create verb. Addressing it as ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject and left the wildcard unsubscribable. Publish and unpublish are not served yet; the mount and unmount logic they would front is still inline in launchService and needs extracting first. Also adds a conformance suite that runs one set of contract assertions against both the in-memory fake and the NATS client, so each side is checked against the contract rather than against the other's structs. * feat: let an operator select the EBS provider Adds an [ebs] provider key and wires the NATS provider into the EC2 volume, snapshot, image and instance services at the daemon's composition root. Until now nothing injected a provider, so the provider-backed code paths in those services were unreachable. The key is provider-neutral rather than nested under [viperblock], because it names the boundary rather than one provider's settings; a second provider will not need it renamed. An empty value resolves to "embedded", so deployed config files that predate the key keep today's behaviour, and an unrecognised value is a startup error rather than a silent fallback. Startup probes the provider's capability verb and warns when nothing answers. The daemon and viperblockd start independently, so an unanswered probe is not fatal, but without it a misconfigured node fails much later as an unexplained request timeout on the first EBS call. "viperblockd" stays experimental: under it volume metadata lives in the control plane's own document, and the leak reaper, GC and admin paths still read viperblock's config.json, so they cannot see tenant or tag data for volumes created that way. * fix: make every volume-metadata consumer provider-aware Several readers and writers of volume metadata were never given a provider branch, so under the viperblockd provider they silently used the wrong store. Each of these is a correctness bug rather than a missing feature, because the control plane had already stopped writing the document they read. Modifications had nowhere to live: ModifyVolume built its result in memory and dropped it, and DescribeVolumesModifications read it back out of viperblock's config.json, so a describe straight after a successful modify returned nothing. Volume metadata now carries its own modification record. The leak reaper read attachment and tags from config.json and wrote orphan markers to tags.json, neither of which exists for a provider-managed volume, so a leaked volume was invisible to it and any mark it did write could not be read back. Tag mutations had the same split: a tag applied after creation never reached the document DescribeVolumes reads. Snapshot deletion could not see a provider-managed clone, because a clone records its source only in the control-plane document. That let a delete strip chunks a live clone still reads, so it is fixed here rather than left for the migration. Volume enumeration and DescribeVolumeStatus were likewise scanning object prefixes that a provider-managed volume does not create. Also adds an encryption flag to the volume document, which DescribeVolumes had been reporting as false unconditionally and snapshots could not carry at all, and a list operation for AMIs so the image path can follow later. The legacy path is unchanged; its invariant tests pass unmodified. * feat: backfill ebsmetadata documents from legacy volume and AMI state DescribeVolumes under the viperblockd provider enumerates only ebsmetadata documents, so a volume created before the switch had no document and was invisible rather than merely slow to read. A per-object read fallback cannot fix enumeration, so both a backfill and a fallback are needed. Add an object-store migration kind to the migrate framework. Predastore has no conditional write, so an object-based version stamp would be last-writer-wins across nodes; RunObject stamps progress in a caller-supplied JetStream KV bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a read-then-write rather than a compare-and-swap, so correctness rests on every ObjectMigration.Run being safe to re-execute. The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the numbered config migrations: it needs viperblock's VBState schema to decode the legacy config.json, and migrate is imported by every handler that stamps a KV bucket version. It converts what it can reach and skips unreadable objects with a log, so one corrupt volume cannot hide the rest of the fleet. ebsmetadata gains the legacy read fallback for the window before the migration has run everywhere. It stays free of viperblock by taking the decode as an injected function value, wired at the composition root, and the fallback is retired by leaving it unset. A fallback read never writes the converted value back, which would race the migration. Prefix enumeration pages to exhaustion. Stopping at the first page would let the backfill stamp itself complete having converted only the volumes it happened to see, reintroducing the invisible-volume failure it exists to prevent. The migration runs from configureEBSProvider, so only when the provider is selected; backfilling while the embedded engine is still authoritative would produce documents that immediately go stale. Failure is fatal rather than advisory, unlike the capability probe beside it. This makes the switch to viperblockd one-way, which is now recorded on the config field. * feat: serve volume publish and unpublish from viperblockd PublishVolume and UnpublishVolume were the last unserved verbs on the provider contract, and they are the ones that attach a volume to a VM. Until they are served the provider path cannot boot an instance, so provider = "viperblockd" cannot be verified end to end. The mount and unmount logic was two large anonymous handler bodies inline in launchService. Rather than write a second mount path, extract each into a named function and have both the legacy ebs.mount/ebs.unmount handlers and the new provider handlers call it. Two divergent implementations of nbdkit spawning and the seal decision would be far worse than the inline bodies. The extraction relocates code without rewriting it; response publishing stays in each handler so the provider never publishes to ebs.mount.response. Publish is idempotent: a volume this node already has mounted returns its existing attachment rather than starting a second nbdkit against it, which is the double-writer hazard the boundary exists to prevent. Unpublish preserves the invariant that a failed seal leaves the volume mounted so a retry can re-attempt it, and reports an error rather than success. Both verbs are node-addressed, so they subscribe directly rather than through the spinifex-workers queue group, and are registered only when NodeName is set. There is no queue-group fallback: a subject that already names one node has nothing sensible to fall back to. Read-only attachment is refused rather than silently granted read-write, since nbdkit is started read-write or not at all. Also fixes a latent panic carried by the extracted mount path: viperblock.New returns a nil VB alongside its error, and SetCacheSize was called on it before the error was checked, so a failing New would have taken down the handler. * feat: read and write AMI metadata through ebsmetadata The backfill migration wrote ebsmetadata AMI documents that nothing read: every AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at {imageID}/config.json. Give the image service provider branches for the AMI read, write and enumerate paths, with enumeration going through ListAMIs rather than an object-prefix scan, as DescribeVolumes already does. AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a viperblock type on an interface boundary in the instance package. It now returns ebsmetadata.AMI on both paths, the embedded one converting before it returns, so the boundary names no viperblock type whichever provider is selected. The legacy-to-document mapping had accumulated three copies. Consolidate them in ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata itself stays free of viperblock, and cover it with a reflective test that fails when a field is added to either schema without being wired through. Sixteen fields copied by hand in three places would have dropped one silently. The admin image paths read whichever representation exists and write back to that same one, so a provider-managed AMI never gains a spurious config.json. FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan: missing a dependent there deletes an AMI a live volume still needs, the same class of bug as the snapshot clone path fixed earlier on this branch. That union is built by hand rather than by wiring Store.ListVolumes' fallback, which swallows per-ID read errors. This path deliberately fails closed on a transport error so a delete cannot proceed on a partial picture, and the fallback would have quietly turned that into failing open. The CLI image import now also writes the document, best-effort, so an AMI imported after the migration is not left reachable only through the fallback. * fix(ec2): resolve volume config through ebsmetadata when a provider is configured A provider-created volume has no vol-*/config.json — its metadata lives in the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy key unconditionally, so AttachVolume reported a volume that DescribeVolumes had just listed as available as InvalidVolume.NotFound. Routing through ebsmetadata.Store also puts this path behind the legacy fallback, so a volume created before the provider switch stays attachable. The NoSuchKey remap now names the key it missed; its silence made the original failure invisible in the daemon logs on every node. * test(bench): add an EC2 control-plane latency benchmark Measures the EBS control-plane operations either side of the provider boundary, so the cost of the NATS hop can be stated rather than guessed. Reports median and p95 with a bootstrap CI on the median difference, and labels a delta indicative unless that CI excludes zero. * fix(ec2): drain the volume before a provider snapshot The legacy CreateSnapshot branch drains first so the snapshot reads a current checkpoint; the provider branch did not, and viperblockd documents the drain as the caller's responsibility. Both sides deferred to the other, so an attached volume snapshotted stale data with no error raised. DrainVolume and volumeAttachment now take the fallback state and instance as strings rather than a viperblock.VolumeMetadata, removing a viperblock type from an exported signature. * feat(ec2): route the instance root volume through the EBS provider RunInstances built its own viperblock engine to clone the AMI snapshot into the root volume, which made every launch a second writer against a volume viperblockd already serves. The launch now calls the provider's CreateVolume with the AMI's snapshot and source volume, and records the result in ebsmetadata so DescribeVolumes and attach can see it. A snapshot clone needs both the snapshot ID and its source volume ID to resolve base blocks, so CreateVolume in the volume handler now sends SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected every create-volume --snapshot-id outright. The gp3 IOPS and throughput constants move to spinifex/types so the instance and volume handlers share one definition rather than duplicating literals. * feat(ebsprovider): add a bounded volume seed to CreateVolume The EFI variable store must be initialised from the launching node's firmware VARS template, byte for byte. Passing a host path would not work: CreateVolume is queue-subscribed, so any node can service it, and a foreign node's OVMF package would bind the guest's pflash region to the wrong firmware build. CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd writes them at offset 0 in the engine it already builds. The seed applies only when the create actually allocates the volume, so a relaunch never overwrites a BootOrder the guest has since written. JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes bounds the seed well inside the NATS max_payload the cluster runs with and both ends reject an oversized seed with a clear error rather than letting the publish fail. VolumeSeeding advertises the behaviour so a provider that cannot seed says so instead of silently dropping the bytes. * feat(ec2): route the EFI variable store through the EBS provider prepareEFIVolume built a control-plane viperblock engine to seed the variable store from the firmware VARS template, making every UEFI launch a second writer against a volume viperblockd already serves. It now calls the provider's CreateVolume with the template as seed bytes, so the store is allocated and written by the daemon that owns the volume. The template is still read on the launching node and shipped with the request. CreateVolume is queue-subscribed, so any node can service it, and a VARS region seeded from a foreign node's OVMF build would bind the guest's pflash to the wrong firmware. pflash refuses a VARS region that is not byte-exact, so the returned capacity is checked against the template length; a provider that rounded up to a GiB boundary would otherwise yield a guest that cannot boot. The engine path stays behind a nil-provider guard until the legacy branch goes. * feat(ec2): route image snapshots through the EBS provider snapshotRunningVolume and snapshotStoppedVolume each built a control-plane viperblock engine to take the snapshot backing a new AMI, making CreateImage a second writer against a volume viperblockd already serves. Both now call the provider's CreateSnapshot with the volume's provider handle. The running path keeps its drain, which is what makes the snapshot current rather than one predating a guest-triggered write. The stopped path needs no drain, and the daemon already prefers a live mounted engine over opening a second one, so it handles both cases itself. getVolumeConfig read the legacy config.json, which a provider-created volume does not have, so every CreateImage against one failed. It now reads the ebsmetadata document under a provider, the same way attach does. * fix(daemon): only the owning node creates an image for a stopped instance handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which only the node holding the VM passes, so a running instance is imaged once. A stopped instance lives in the cluster-replicated KV instead, which every node reads successfully, so all of them ran the full snapshot and AMI creation. The caller saw the first reply and the rest leaked: on a three-node cluster one create-image produced three AMIs and three snapshots. A stopped instance records its owner in LastNode, so a node that is not the owner now declines. Gather tallies error frames and keeps waiting, returning only on a success, so declining cannot preempt the owner's reply. An instance stopped by a build that never set LastNode elects an owner through the KV's compare-and-swap, since having every node decline would make create-image fail permanently for those records. * perf(ec2): serve DescribeVolumes by ID from the volume documents The provider branch called ListVolumes unconditionally and filtered in memory, so asking for one volume read and decoded every volume document in the bucket. The non-provider branch already had the equivalent fast path. Explicit IDs now fetch their documents directly. The tenant check moves with them: a document belonging to another account reports not-found rather than being returned, matching what the enumeration filter did. Measured against a memory object store, the fast path stays flat near 20us regardless of bucket size while the enumeration grows linearly, reaching 4.6ms at 500 volumes. * chore(lint): ban the viperblock import from the EC2 control plane The control plane reaches block storage through ebsprovider now, but nothing stopped a handler from importing the engine again. That is not hypothetical: the instance handler first imported viperblock for a config struct alone, and the engine construction followed thirteen days later. depguard denies the import across handlers/, which matters because viperblock.New starts a chunk uploader — every construction is a writer against a volume viperblockd already serves. A types-only import is denied too, since that is exactly how the last one started. The four files still carrying the pre-provider engine path behind a nil-provider guard are exempted by name. The exemption list is the remaining work: it shrinks to nothing when that path goes. * fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot CopySnapshot wrote a control-plane snapshot document under the new ID and nothing else. The copy described correctly and CreateVolume against it succeeded, so it looked like it worked, but the new ID had no viperblock snapshot metadata behind it and the volume failed when the guest attached it. Producing the metadata means re-sealing it under the destination ID. On an encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the sealing helpers are unexported, so spinifex cannot write it. Copying only the unencrypted form would be dead code on any cluster with encryption enabled. Refusing with UnsupportedOperation fails at the call the caller can act on rather than at attach time. Source lookup and the ownership check still run first, so a bad ID or another tenant's snapshot reports that instead. * test(ec2): give handler test fixtures their own legacy config.json shape Handler tests built the legacy config.json fixture by importing viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly. Mirror that shape in ebsmetadata/vblegacy, next to the production code that already reads it (ebsmetadatabackfill), and switch the fixture-only usages across the volume/snapshot/image handler tests to it. A byte-for-byte equivalence test against the real viperblock types guards the mirror from drifting out of sync. Call sites that hand a viperblock type into putVolumeConfig, mergeVolumeConfig, or buildVBConfig are untouched, since those functions still name the real type in their signature. * feat(ebsprovider): add CopySnapshot to the EBS provider contract Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy NATS subject, MemoryProvider and NATSProvider implementations, conformance coverage) and a viperblockd handler that duplicates a snapshot's metadata via viperblock.CopySnapshotMeta on the source volume's VB. * feat(ec2): serve CopySnapshot through the EBS provider CopySnapshot has been refusing with UnsupportedOperation because a copy needs its own snapshot metadata on the backend under the new ID, and on an encrypted volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the control plane cannot re-seal it. The provider contract now carries a copy operation, so the storage engine does it. The provider path generates a new snapshot ID, has the provider duplicate the backend metadata under it, then writes the control-plane document. Rollback mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot rather than leaving one nothing references. Ownership is still checked against the source before anything is created. The no-provider path keeps refusing. It has no way to produce the backend metadata, and the copy it used to write described fine and then failed at attach — a resource that fails as far as possible from the call that made it. Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot reached the provider rather than existing only as a control-plane document. That assertion is what catches the old behaviour. * refactor: remove the embedded EBS engine path from the EC2 control plane viperblockd is now the only EBS provider. The pre-provider path that constructed a viperblock engine in-process is deleted, and with it every github.com/mulgadc/viperblock import under spinifex/handlers/ (tests included). The depguard ec2-control-plane rule drops all four file exemptions and the !$test carve-out, so the rule now reads simply as "nothing under spinifex/handlers/** may import viperblock". viperblock.VolumeConfig was also a data carrier on the provider path. It is replaced by the spinifex-native ebsmetadata.Volume rather than the vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is decoding old objects, while ebsmetadata.Volume already carries every field the call sites read, so the conversion step disappears instead of being reimplemented against a legacy shape. The embedded-only helpers (volumeConfigWrapper, putVolumeConfig, mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached, deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json prefix scan) had exactly one production caller chain each, all through a nil-provider branch, and are removed. The snapshot-dependency guard on DeleteVolume is provider-agnostic and is kept, now running on the provider path where it previously did not. config: viperblockd is the default and an empty Provider resolves to it. An explicit provider = "embedded" is rejected at config load rather than silently upgraded, because starting anyway would migrate a node's volumes to ebsmetadata one-way without the operator asking. The generated spinifex.toml template emits viperblockd. * fix(images): write the imported AMI document from the config the import wrote ImportDiskImage creates the AMI's snapshot and records it on its own *viperblock.VB, which viperblock.New allocates fresh, so the value lands only in {imageID}/config.json. The import command was persisting the ebsmetadata document from its pre-import manifest copy, whose SnapshotID is still empty. Store.GetAMI only consults the legacy config.json fallback when the document is missing, so that empty-SnapshotID document permanently shadowed the real value and every launch from an imported AMI failed the zero-copy clone check with ServerInternal. Rebuild the document by reading back the config.json the import just wrote. A missing or unreadable config is skipped rather than written from the stale manifest: no document still resolves correctly through the legacy fallback. * test(e2e): read NODE from spx get vms instead of VCPU in the spread check spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to field 8. The spread placement check still read field 6, so every instance reported its vCPU count instead of its node, the unique-node count always collapsed to 1, and the check failed on clusters that had spread correctly. Read field 8 and strip the ANSI colour codes wrapping each cell. * feat(viperblockd): add a --debug flag so viperblock debug logging is reachable viperblockd hardcoded Info, and viperblock's VB.logger() falls back to slog.Default() because vb.log is only ever set by SetDebug. That left the engine's debug records, including the chunk uploader's start and disabled lines emitted during New(), unreachable in a deployed environment. Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG rather than the bare debug key, which several commands share. Also sets the Config.Debug field, which existed but was set only by tests. * fix(viperblockd): release the chunk uploader when openVolumeVB fails viperblock.New starts a chunk uploader and a WAL syncer before the backend is touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the caller while leaving both running with no handle to stop them. Repeated backend failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders on a 30s ticker for the life of the process. Adds invariant tests for both halves of the ownership rule: a mounted volume's snapshot create and copy reuse the live engine and log zero volume opens, an unmounted volume opens exactly one, and a failed open leaves no uploader behind. The open counts are read from the same log record a production alarm watches. * fix(ebsprovider): route requests to the node that owns the mount Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one spinifex-workers group, so NATS delivered each request to a random node. Only the node holding the mount can reuse the live engine through findMountedVolume; any other node constructed a second viperblock engine, with a live 30s chunk uploader, against a volume nbdkit was serving on a different host. Measured on a three-node cluster, this was roughly two snapshots in three. Rather than resolve the owner, let the owner answer. A node subscribes ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts a volume and drops it on unmount, mirroring the per-volume ebs.config subscription. Clients try that subject first and fall back to the queue group only on ErrNoResponders, which means nothing is mounted and opening an engine is correct. A timeout deliberately does not fall back, since the owner may still be running the operation. Applies to snapshot create and copy, expand and describe, the four verbs that reuse a mounted engine. Delete keeps queue dispatch. * fix: rebuild viperblockd mounted-volume registry after a restart Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a viperblockd restart while a VM keeps running leaves nbdkit serving a volume the daemon has forgotten. Owner routing then infers from ErrNoResponders that nothing is mounted and falls back to the queue group, which opens a second viperblock engine, with a live chunk uploader, on another node. The same empty registry also defeats handlePublishVolume's double-mount guard, so a republish could start a second nbdkit. Discover surviving nbdkit processes from /proc at startup and rebuild the registry from them, following orphan_scan.go's precedent: match comm exactly, parse the argv for volume=<id> and --unix/-p, and adopt a PID only when nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or killed; anything that fails corroboration is logged and skipped. Recovery runs before the first subscription so no handler can race it. Extract mountVolume's engine construction into constructMountedVB so mount and recovery cannot drift, and release the WAL syncer viperblock.New starts when construction fails after it. * fix: corroborate recovered nbdkit processes from argv, not a pidfile nbdkit is launched with -f (required for the Go plugin) and never writes the --pidfile it is given, so no such file exists on any node; the daemon's own /tmp is a PrivateTmp namespace besides. Corroborating against it rejected every genuine mount, making recovery a no-op in production. Corroborate against what the kernel actually has: the process must serve our plugin, name this daemon's base dir, and still hold its socket. Adopt a volume once, so two live nbdkits for it leave one registry entry and a warning rather than two conflicting ones. * feat: add a second EBS provider backed by qcow2 and qemu-nbd The decoupling was justified by swappability that nothing demonstrated: the conformance suite ran against MemoryProvider and against a NATSProvider wired to a MemoryProvider, so both arms were the same implementation behind different transports. A fake is not a second implementation. Add three pieces. natsserve serves the ebs.provider.v1.* contract from any EBSProvider, depending on nothing but the interface, and replaces the test-only server twin so conformance now exercises production code. The conformance suite moves to its own package and is exported so implementations outside ebsprovider can run it, with no assertion changed. qemunbdd is the second implementation: one qcow2 file per volume, qemu-nbd for publish, real snapshot copies instead of chunk references, and every exec behind a runner seam so its tests need no qemu. Owner-subject routing is deliberately not served neutrally: it needs a per-node mounted-volume registry the interface does not model, so it stays in viperblockd and a neutral server's clients correctly fall back to the queue group. * test: run the conformance suite against the qcow2 provider The suite passes unmodified against an implementation sharing no code with viperblock, both in-process and through natsserve over the real wire contract, which is the swappability claim the provider boundary was built to support. Two real defects it caught. ExpandVolume and GetVolume inspected the image before consulting the publication map, so qemu-img could not open a volume qemu-nbd held the write lock on and the refusal surfaced as an internal error instead of ErrVolumeInUse; qemu-img info now runs with --force-share, so describing a volume no longer depends on whether it is attached. Publish also failed with qemu-nbd's raw exit status when the socket path exceeded the 108-byte AF_UNIX limit, and now reports that reason itself. * feat: make the EBS provider selectable by config Add the qemunbd selector alongside viperblockd, a spx service command that serves the qcow2 provider over natsserve, and service registration. Config validation accepts the new value everywhere viperblockd is accepted, while embedded stays rejected by name. The daemon logged a hard-coded viperblockd constant as the active provider rather than the resolved one, in configureEBSProvider and in both of probeEBSProvider's lines; all three now report what is actually configured. Both daemons answer the same subjects, so exactly one may run on a cluster. * test: add a live lifecycle check for the qcow2 provider Drives create, publish, write, snapshot, restore and read-back entirely through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a cluster node, so the client under test is the same shape the control plane uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE is set, so it stays out of preflight. * fix: publish NBD URIs in the form standard clients accept PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever consume what the EBS provider boundary hands back. Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit all accept. ParseNBDURI now takes both that and the TCP form, and still accepts the legacy nbd:unix: syntax for state written by older daemons, so a conformant third-party provider's URI is no longer rejected on the way in either. Verified against qemu 10.0.11: qemu-img and a real -drive option string both open the URI form, and qemu reports both forms as nbd+unix://?socket=, so it already treats the URI as canonical. * test: validate published exports with libnbd's client tools Every other assertion about the EBS provider goes through our own Go client, so nothing has ever checked whether the export we publish is usable by an NBD client that knows nothing about this codebase. That is the layer neither the conformance suite nor a CSI-shaped contract can reach. RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume: the URI connects, the export advertises its size and block-size constraints, data written externally reads back byte for byte, a ReadOnly publish sets the read-only transmission flag, and UnpublishVolume tears the export down. It skips when libnbd-bin is absent rather than passing vacuously. Wired to qemunbdd, where all five failed before the URI fix. The publish assertion in the main suite now requires a dialable URI instead of merely a non-empty string, which is what let the wrong form through. * test: run the external NBD client suite against a live viperblockd The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit export is the comparison worth having: one external client, two providers it cannot tell apart, and every difference is a candidate leak in the boundary. RunNBDClientSuiteWithConfig takes the node name and volume prefix a live cluster needs, and publishForNBD now unpublishes and deletes what it created so a run against a real cluster leaves nothing behind. Subtests dial a normalised URI so a provider emitting the legacy syntax still gets its export tested rather than every subtest failing on the same string; whether the URI itself is standard is now its own subtest. Adds a reconnect check, since publication holds the volume rather than any one connection. It retries the real operation because a cheaper liveness probe is itself a connection and re-triggers whatever refused it. * feat: make the conformance suite capability-driven The suite asserted a fixed Capabilities literal, so it could only ever pass against a provider advertising exactly that set. Every implementation had to match the suite rather than the suite testing the contract. RunSuite now reads GetCapabilities from the provider under test and gates optional behaviour on what it advertises, and the fixed literal is demoted to ReferenceCapabilities, a convenience for callers constructing a MemoryProvider. Add ReadOnlyPublish to Capabilities. A provider that leaves it false must refuse a ReadOnly publish rather than hand back a writable export, which is what viperblockd already does; it now has a way to say so up front. Add ErrUnsupportedCapability and the unsupported_capability error code. A request for optional behaviour a provider does not advertise is well formed, so classifying it as invalid_argument told the caller to fix a request that was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly. Declared capabilities now match observed behaviour: qemunbdd advertises SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises neither: the nbdkit Go binding exposes no extents callback, and the mount path never passes -r. The external NBD client suite gains a sparse-extent subtest that checks the export's advertised contexts against the capability, so SparseExtentReporting is falsifiable from outside rather than taken on trust, and its read-only subtest now branches on the capability instead of assuming refusal. Cover the new branches with a MemoryProvider advertising no optional capabilities and one advertising OnlineExpansion, the branch no provider we ship exercises. * refactor: rename SourceVolumeID to SourceSnapshotVolumeID CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd required it whenever SourceSnapshotID was set and used it to resolve the snapshot's chunk references, so there it named the snapshot's origin volume. MemoryProvider ignored it entirely. qemunbdd implemented it as an independent volume-to-volume clone, which is what the name and doc comment implied. The conformance suite could not catch the divergence because the reference provider had no branch for the field at all. The contract already has a field meaning "the volume this snapshot came from": Snapshot.SourceVolumeID, which every provider populates. So the request field is the same fact travelling back the other way, and its name was the problem. Rename it to SourceSnapshotVolumeID, document it as the source snapshot's origin volume, and state that it is required with SourceSnapshotID and meaningless without it. All three providers now enforce one rule: a source snapshot must be accompanied by its origin volume, or the request is invalid_argument. MemoryProvider additionally rejects an origin that does not match the snapshot's own SourceVolumeID, so a wrong name fails instead of resolving blocks against a volume the snapshot never came from. qemunbdd loses its volume-to-volume clone branch, which implemented behaviour the contract no longer offers and no caller ever requested. The suite pins both the requirement and a full create-snapshot-restore round trip, so the field is exercised end to end rather than only on the error path. * feat: make the conformance suite runnable against a live provider The suite had only ever run against MemoryProvider and qemunbdd, both of which are discarded with the test binary. viperblockd, the provider production actually uses, was the one implementation the suite never judged, because the suite assumed each subtest got fresh state: hard-coded volume IDs collide with an earlier run's leftovers on a store that keeps them, and a hard-coded node ID addresses a node that does not exist. SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records every volume, snapshot and publication the suite creates and tears them down in dependency order, logging rather than failing so a teardown problem cannot mask the assertion a subtest was making. Two new tests cover this by running the whole suite twice against one shared provider instance, which passes only because cleanup works, and once per prefix to cover concurrent runs sharing a daemon. TestLive_ViperblockdConformance wires this to a real cluster behind SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite. * fix(viperblockd): close the four contract gaps the live conformance run found Unpublishing a volume this node has no mount for now succeeds instead of returning not_found, matching the legacy unmount path: nothing mounted here is already the state the request asks for. CreateSnapshot refuses a snapshot ID that already exists against a different source volume with already_exists, rather than overwriting it. DeleteVolume refuses a published volume with volume_in_use instead of tearing the mount down, on this node or on whichever node answers the volume's owner subject. The export and the guest writing through it survive the metadata deletion, so the old behaviour surfaced later as corruption. CreateVolume reads the source snapshot before writing the volume, returning not_found when it is absent and invalid_argument when it was taken from a different volume than the request names. Previously both produced a blank volume and a successful response. CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument now that it carries a sentinel. * feat(ebsprovider): pin the source snapshot's origin volume in the contract The suite only required source_snapshot_volume_id to be present, so a caller naming a volume the snapshot never came from was accepted everywhere. The snapshot records its own origin, so this is checkable: creating from a snapshot with a mismatched source volume is now invalid_argument for every provider, enforced by qemunbdd against its snapshot metadata and by viperblockd against the snapshot's persisted SourceVolumeName. * test(ebsprovider): run the contract suite against viperblockd in CI CI installed neither qemu nor nbdkit, so every conformance test that needs a real implementation skipped silently and MemoryProvider was the only provider the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails instead of skipping. TestViperblockdConformance stands the provider handlers up over an embedded NATS and a real predastore and runs the same suite against them, which is what caught CopySnapshot accepting a foreign volume's snapshot on unencrypted volumes. * feat(viperblockd): honour read-only publish and advertise block size constraints ReadOnlyPublish was advertised false and read_only=true was rejected, so the capability was true only by omission. nbdkit -r now starts the export when the request asks for it, which sets the NBD read-only transmission flag and makes the plugin refuse writes. Access mode is fixed when nbdkit starts, so a republish asking for the other mode is refused with VolumeInUse rather than handed the running export. Block size constraints were absent entirely: the nbdkit Go binding has no block-size callback, so the plugin cannot advertise them. The blocksize-policy filter can, without a binding callback. Minimum 1 because viperblock's WriteAt handles unaligned requests with a read-modify-write, preferred 4096 to match its block size, maximum 32M to match what qemu-nbd advertises. * test(conformance): judge sparse reporting by the allocation map, not the context The subtest asserted that base:allocation was offered. nbdkit offers that context whether or not the plugin can compute extents, answering 'all allocated' when it cannot, so the assertion passed for a server with no extent support at all — it could not fail. What separates the two is the answer: a volume with nothing written to it is entirely holes to a server that really tracks allocation. Read the map with nbdinfo --map --json and compare holes against the advertised capability. qemunbdd reports holes and advertises true; viperblockd reports none and advertises false. * test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime The predastore fixture is one daemon per test binary, but its cert was exported with t.Setenv, which reverts when the first test that called Start finishes. In-process clients did not notice because crypto/x509 caches the pool behind a sync.Once, loaded while the variable was still set. Child processes did: nbdkit spawned by a later test inherited an environment with no SSL_CERT_FILE and could not verify the daemon, so the whole NBD client suite failed when the package ran as a whole while passing when run alone. Also install libnbd-bin in CI, so the NBD client suite runs there instead of failing on missing tools. * ci: install nbdkit-plugin-dev so the viperblock plugin can build * test(volume): give the snapshot fixture an owner CreateVolume now fails closed on a snapshot whose owner it cannot match, including an unset owner_id. The fixture predates that and wrote none, so the provider test failed with InvalidSnapshot.NotFound before ever reaching the provider. * test(integration): give the volume harness a real provider The control plane no longer builds volumes itself, so CreateVolume in the integration harness failed with "no EBS provider configured": the harness wired a VolumeServiceImpl with nothing behind it. It now serves viperblockd's provider subjects against the same predastore fixture, so the blocks are still written by real viperblock to a real daemon. Three persisted-state assertions moved with it. TenantID, VolumeType and AvailabilityZone are control-plane facts, held in the ebsmetadata document rather than viperblock's own state, so the storage oracle now asserts their absence and DescribeVolumes -- whose answer is account-scoped -- is what checks them. * refactor: route image import through the EBS provider and drop the legacy metadata path The control plane could still only run against viperblock: the AMI import and the ochre weights import both linked the storage library directly, and the ebsmetadata store kept a read fallback onto viperblock's config.json. Import now goes through the provider contract with no new verb: CreateVolume, PublishVolume, a bulk copy over the published NBD export with qemu-img, then UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS, whose request size the contract caps far below a disk image. The migration scaffolding goes with it. ebsmetadata documents are now the only thing the control plane reads: no legacy prefix scan, no config.json fallback, no backfill on daemon start. config.json belongs to the provider, which is the single writer of it. ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is right for DescribeImages, where one corrupt document must not hide the fleet, and wrong for a name-uniqueness check, which cannot rule out a name it failed to read. spinifex and cmd no longer import viperblock outside services/viperblockd and testutil/vbscan. * feat: add a volume enumeration verb so a lost document cannot strand blocks Making ebsmetadata documents the sole index of volumes left one failure with no recovery: a volume whose document is gone disappears from DescribeVolumes, from the leak reaper (which enumerates documents), and from the provider, whose contract had no verb that asks what it holds. ListVolumes closes that. It is capability-gated on VolumeEnumeration and paginated, following CSI's optional LIST_VOLUMES rather than inventing a shape. Refs are deliberately not Volumes: enumeration answers what exists, and a provider must not open every volume's engine to say so. Each implementation reads what actually exists rather than what it remembers. qemunbdd walks its volumes directory, not its in-memory map, which does not survive a restart; viperblockd walks the bucket's top-level prefixes. A volume the provider has forgotten is precisely what a caller is looking for. The consumer is a read-only orphan report: provider volumes minus documented ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is excused only when its base volume is documented, so a stranded pair is still reported. Listing documents stays strict, because reporting a volume as orphaned on the strength of a document that failed to decode would point an operator at live data. * fix: stop the orphan scan accusing storage of holding things it does not A live env19 run reported three orphans against a healthy cluster: an AMI's backing volume, a snapshot, and the cluster key store. All three were wrong, and a report that cries wolf on a clean cluster is worse than no report. Enumeration treated every top-level bucket prefix as a volume. Snapshots sit beside volumes at that level and have a config of their own, so presence of a config cannot separate them; the snap- name is what does, which is already how the AMI removal path reads them back. keys/ and spinifex/ are not volumes at all. An AMI's blocks are a genuine provider volume, so enumeration keeps reporting them. What was missing is on the reading side: an AMI is recorded as an image document, not a volume one, so the scan now treats a documented image as known. Listing images is strict for the same reason listing volumes is — unreadable and absent are different answers, and only one means orphaned. * fix: let an orphaned image's blocks be reclaimed The orphan report tells an operator that reclaiming a stranded volume is their decision, and images remove --force is the tool it points them at. On env19 that tool failed on exactly those volumes: removal drops the ebsmetadata document first, and predastore answers NoSuchKey for a document that is already gone, which is the definition of an orphan. Deleting the document stays first and stays fatal. That ordering is the barrier that stops a launch landing on blocks about to be deleted, and an AMI still advertised after its blocks are gone is the worse failure. Only an already-absent document is now tolerated: the barrier is satisfied, and the blocks are the sole thing left to reclaim. S3ObjectStore.DeleteObject normalises a missing key the way GetObject already does, so this is recognised rather than matched on error text. The obvious unit test passed without the fix, because the in-memory store is silently tolerant on delete and real S3 answers 204 — neither behaves like the backend this runs against. The test now drives a store that reports the miss, and fails without the fix. * feat: add per-verb OTel spans to the EBS provider contract The ten provider verbs cross a single NATS seam and nothing measured it. The only spans on the storage path were the legacy ebs.mount/ebs.unmount pair, so control-plane storage latency could not be attributed to a verb or compared across provider implementations. NATSProvider.request is the one client-side funnel every verb passes through, including both legs of an owner-subject fallback, so a single span there covers the whole contract and measures what the control plane experiences regardless of which provider answers. Span names come from a closed verb set rather than the subject. Owner and snapshot-create subjects embed a volume ID and the mount/unmount subjects embed a node ID; naming spans after them would give every volume a span name of its own. The IDs move to attributes and an unrecognised subject collapses to a single unknown verb. Trace context rides the message headers, so the serving provider joins the caller's trace instead of rooting one of its own. The carrier is a copy of the one in spinifex/utils rather than an import: that package pulls in the aws-sdk and the cluster key store, which the wire contract must not depend on. Responses carry no shared accessor for their error field, so both sides re-read just that field from the encoded reply, and only when the span is recording. Without it a failed verb looks like a successful round trip. viperblockd serves the contract with its own handlers rather than natsserve, so its handlers now take the per-message context and its publish/unpublish pair drops the subject-named consumer span it opened for itself. * feat: add null and hostile providers, and a per-verb bench suite The provider boundary made it possible to put something other than viperblock behind the control plane, and nothing yet did. nullprovider is the floor. It is not a new type: MemoryProvider already answers every verb from a map and natsserve already serves any EBSProvider, so the package names that composition and states what it measures. On loopback with no storage behind it a round trip costs around 195us against 74ns in process, which is the seam's own price and is not attributable to any storage engine. conformance.RunBenchSuite prices one round trip per verb and drives every implementation through the same calls in the same order, so two providers' numbers are comparable. It measures control operations only: guest I/O never crosses these verbs. hostile wraps any provider in one that misbehaves. Faults derive from the seed and a per-target sequence rather than a global counter, so concurrent callers on different volumes cannot reorder each other's faults and a failure found deep into a soak run replays exactly. Error and error-after-work are separate faults on purpose: the second reports failure after the work has already happened, which is how an orphan is made, and is what makes an enumeration scan the right oracle for a soak. Enumeration itself is never faulted, since an oracle that can be told what to say cannot falsify anything. The soak asserts that every volume the provider still holds is attributable to an injected fault. A leftover with nothing behind it is the control plane losing track of storage on its own. Two bugs were found by falsifying the new tests rather than by running them. The FNV-derived draw clustered badly enough that two fault bands were never reachable in 1200 draws, so the draw now goes through a SplitMix64 finalizer. And the first version of the lie test was worthless: MemoryProvider returns copies, so a lie could not corrupt it whether or not the injector copied, and the test now drives a provider that hands out pointers into its own state. * test: bench the provider contract against qemunbdd and a live viperblockd RunBenchSuite was only ever pointed at the null provider, so the numbers priced the seam and nothing else. Point it at the two real implementations. BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a provider that stores nothing and wrong against one that does, so qemunbdd and viperblockd are both measured at 64 MiB. liveProvider splits into a testing.TB flavour. Benchmarks need the same connection and the same node and cannot be handed a *testing.T. * fix(viperblockd): carry the trace context into every viperblock engine call Both S3 clients only emit a span when the request already carries one, so calling the no-context Init, LoadState, SaveState, LoadBlockState and Close left the whole engine open untraceable. Every provider verb traced as one flat server span, and the 342ms describe measured on env19 could not be attributed to anything. viperblock already exposes the Ctx variants and Backend already declares InitCtx, so this is threading, not new instrumentation. openVolumeVB, openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine, copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and mount/unmount keep the context their own span derives instead of dropping it. completeCreateSnapshot gets WithoutCancel: the background half of accept-then-publish has to outlive the request it was accepted by and still be attributable to it. The per-volume ebs.config handler opens a consumer span like the other legacy subjects, so a config update is no longer the one path with no trace at all. * perf: describe a volume without opening an engine on it describeVolumeEngine fell through to openVolumeVB for any volume not mounted on the node serving the request, which constructed a full viperblock engine to read one integer. On env19 that decomposed as a 22 ms reachability list, a 3 ms state read and a 16 ms PutObject of the volume's own config.json — the write coming from the SeqNum bootstrap a read-write open performs. Use viperblock's read-only state load instead. A describe is now the 3 ms read: no background goroutines, no reachability probe, no SeqNum window claimed, and no write issued by a worker that does not own the volume. Falsified: restoring the openVolumeVB call fails TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the list, with the fake backend answering each so neither assertion hides behind the other. * perf: share one HTTP client across every viperblock backend Four sites built the same S3Config by hand, each leaving HTTPClient unset, so every engine open and state read got its own connection pool and paid its own TLS handshake. That was invisible while Backend.Init issued a whole-bucket list — the list paid for the connection and the state read rode it — and it became the dominant cost once describe was reduced to a single request. Route all four through cfg.volumeS3Config, which builds the client once. Falsified: dropping the HTTPClient field takes TestDescribeVolume_ReusesOneConnection from one connection to three. * perf: wait for nbdkit to be ready instead of waiting a fixed second mountVolume spawned nbdkit and then ran a confirm loop until a one-second deadline passed. The loop only broke early when the process exited, so the fast-failure path worked and the success path always cost the full second. That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so the timings never contradicted the assumption that it was process spawn. Wait for the endpoint to accept a connection instead. That is also the question the caller needs answered, since it is about to hand the URI to something that will connect to it — where 'still alive after a second' only correlates with it. A nbdkit that exits during startup still fails the mount immediately, and one that never comes up now fails on a deadline rather than being assumed healthy. Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness timing assertion and the fast-failure one. * test: measure the data path, not just the control verbs The conformance benchmarks price create, publish and describe. Guest I/O never crosses those verbs, so none of those numbers say anything about what a guest experiences. This drives the published export with qemu-img bench, which sits on the same block layer virtio-blk does. It asserts only that the I/O completes and logs a table: what a given store is capable of is not a contract, and pinning a number here would only produce a flaky test. Read profiles stay inside the region the write profiles covered, or they would be timing how fast the export answers with zeroes, and one write profile flushes, because an unflushed write measures how fast the WAL accepts data rather than how fast the store takes it. Profiles are spaced apart because the plugin tears an engine down per disconnect and removes the volume's local directory as it goes, so a connection opened while the previous one is closing fails its handshake. * test: run data-path profiles back to back The spacing was a workaround for a volume lock that did not hold across close. With that fixed, consecutive profiles are worth more than the workaround was: each is its own NBD connection, so a run that completes has also shown that an arriving connection waits out the previous one's close rather than racing it. * fix: refuse a delete that cannot establish whether a volume is published volumeMountedElsewhere returned false on every failure path, including a request timeout, so a busy or partitioned owner was indistinguishable from no owner at all and the delete proceeded. Deleting a published volume leaves the nbdkit export and the guest writing through it alive over storage that is going away. Report the three outcomes separately. NATS distinguishes no-responders from no-answer, so only the former means unmounted; an unanswered probe is refused and the caller may retry once the owner is reachable. * feat: exclude a second viperblock engine with a JetStream volume lease Nothing in the stack stopped two viperblock engines opening one volume from different nodes. For an encrypted volume that is not just a consistency risk: the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte, and the SeqNum window claim is a read-then-write with no CAS, so two engines opening concurrently issue overlapping nonces under the same key. Every engine open now takes a lease first, keyed by volume in a JetStream KV bucket. A lease is a KV create, which JetStream serialises, so a second claimant loses instead of both winning. The revision that won is the generation: monotonic across the bucket, so a later opener always carries a higher one than the writer it displaced. Entries carry a TTL and a renewal loop, so a node that dies mid-mount does not wedge its volumes forever. Losing a lease is logged, not enforced. Until the object store can refuse a stale writer's PUT, failing live guest I/O on lease loss would add a failure mode without stopping the zombie it is aimed at. A daemon with no lease store refuses to open rather than opening blind, so nbdkit recovery binds the store before it adopts surviving exports. Repeat acquisitions on one node share a lease: the unmount seal opens a detached engine while the mount being torn down still holds it, and cross-node exclusion is what the lease is for. A restart reclaims this node's own stale entries, since the exports that outlived the daemon are still running. * refactor: consume masterkey from bluebottle in the test this branch added * feat(ebsprovider): classify refusals a caller may retry Every ErrorCode said what went wrong and none said whether the same request could succeed later, so a caller could not tell a permanent refusal from a transient one. Transient conditions shipped as internal, which tells a caller to give up when it should retry. Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(), which callers branch on instead of on the code itself. DeleteVolume's "could not establish whether the volume is published" refusal moves off internal onto it: the owner that did not answer may answer later. TestErrorCodeRetryable names every code, so adding one without deciding its retry meaning fails rather than silently defaulting to permanent. TestClassifyError pins the sentinel-to-wire-code mapping, which had no coverage at all; a sentinel missing from classifyError falls through to internal and loses both the reason and the retry signal. * fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for Four providers advertised the capability and nothing consumed it: there is no clone verb on the interface. CreateVolume's SourceSnapshotID restores from a snapshot and CopySnapshot duplicates snapshot metadata; neither is a volume clone, and the field documenting SourceSnapshotID says so explicitly. CSI expresses clone as a volume_content_source with a volume source, which this contract does not implement. Advertising it told a caller a feature was available and then gave them no way to use it. Removed rather than backed by a method: adding a clone verb is a feature, not a contract fix, and the capability can come back alongside it. The Capabilities doc comment now states the rule, and qemunbdd's stale comment about sparse extents is corrected while there. * feat(ebsprovider): add ListSnapshots so snapshots are reconcilable The contract could create, copy and delete snapshots but never say which ones it held. That left the control-plane metadata document as the only index of them, so losing one stranded the data it pointed at with no way to find it again. This is the same exposure ListVolumes closed for volumes, and it stayed invisible because ListSnapshots is optional in CSI too. ListSnapshots pages exactly as ListVolumes does, gated on a new SnapshotEnumeration capability. SnapshotRef is deliberately not a Snapshot: enumeration answers what exists, and reading a snapshot's size or state can cost far more than listing it. viperblockd walks the bucket's snap- prefixes, sharing the top-level walk with ListVolumes. It leaves SourceVolumeID empty because the link lives inside the snapshot's own encrypted state and reading it would mean a fetch per snapshot listed. qemunbdd walks its snapshots directory rather than its in-memory map, so a snapshot that outlived the process still appears; its SourceVolumeID is empty in that case for the same reason CreateSnapshot refuses to reuse the ID. Five conformance subtests cover it, and they run green against MemoryProvider, qemunbdd and viperblockd over a real predastore. * feat(ebsprovider): state the exclusion guarantee in the contract A caller could not tell which providers prevent two writers from holding one volume, at what scope, or for how long. The guarantee existed in viperblockd and was invisible from the outside, which is why the conformance suite had no concurrency dimension at all. Capabilities.Exclusion states it, and deliberately separates three questions the contract used to run together: how far exclusion reaches (none, node, cluster), whether a dead owner's claim ever expires, and whether a writer that lost its claim is actually stopped. That last one matters most and is the least flattering: exclusion refuses the second opener, fencing stops the first. viperblockd advertises cluster scope with a 45s TTL and FencesLostClaim false, because a partitioned node whose lease lapsed keeps writing until the object store can refuse it. Four new conformance subtests. Two required a second viperblockd node in the harness: cluster exclusion cannot be exercised by one node, because the second publish goes to a subject nobody serves and times out rather than being refused. Both nodes share the NATS the leases live in and the bucket the volumes live in, so a refusal comes from the lease. Standing that second node up immediately found a real gap, in a test that had been silently skipping for want of one. A publish refused by another node's lease came back as internal rather than volume_in_use, so a caller was told something had broken when the answer was that someone else held the volume. Mapped in handlePublishVolume. * build: pin viperblock to the release carrying the provider's dependencies The branch has been building against a local replace pointing at a viperblock worktree, so its committed pin never actually built: it named a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch, NewStateReader or s3.NewHTTPClient, all of which this branch calls. Points at the merged viperblock dev instead and drops the replace. GOWORK=off now builds clean, which is the deploy path ansible uses. * fix: let viperblockd's delete sweep tolerate an object another sweep removed deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it returned on any DeleteObject error. Two overlapping deletes for one volume left the loser reporting an internal failure once the winner had removed a chunk between the list and the delete. The control-plane copy that carried this fix is gone with the embedded engine path, so the provider path reintroduced the bug it had just fixed. * fix: stop CreateVolume leaving local state on whichever node served it A state read prefers this node's local copy over the backend, and provider RPCs are not owner-routed, so the node that happened to serve a create kept a copy that answered describe as if the volume still existed after another node deleted it. The two-node conformance harness caught it as DeleteVolume/success failing about one run in three. The seeded path already removed its local files after closing. The unseeded path saved state and left them, which was the only difference between the two. * fix: guard the mount path itself against a second export of one volume handlePublishVolume refused a volume this node already exported, but mountVolume did not, and the legacy ebs.<node>.mount subject is the route production takes for instance launch and hot-attach. A retried attach racing a fresh one started two real nbdkit against one volume, with no refusal and no warning: the volume lease only refuses a different node, and a same-node re-acquire finds the volume already held and increments its refcount. That also made the advertised exclusion inaccurate. viperblockd claims cluster-scoped exclusion on the grounds that a second opener is refused, while the conformance subtests reach it only through PublishVolume, so they passed with the legacy route unguarded. Moving the check into mountVolume gives both entry points the same behaviour and makes the claim true for the path production uses. * fix: tolerate a missing legacy AMI config on deregister DeregisterImage deletes the AMI document, then the pre-provider {imageID}/config.json. Predastore answers a delete of an absent key with NoSuchKey, and that was treated as fatal, so every AMI created since the metadata store landed failed to deregister after its document was already gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had just been told it could not delete. The regression test uses a store that answers a missing-key delete the way Predastore does; MemoryObjectStore silently succeeds, which is why the unit suite stayed green. * fix: reap ENIs left behind by an abandoned instance launch A launch creates the instance's ENI and then attaches it. When the daemon dies in between, the record survives with no instance and nothing ever looks at it again. The security group dependency check counts an ENI that lists the group, so the group cannot be deleted for as long as the record exists. The in-process launch path already rolls back on error; a process that dies cannot. This adds a cluster-wide sweep over records that are launch-created, never attached, and older than the launch RPC timeout by a wide margin. The description prefix is now a shared constant because it is the only thing separating an abandoned launch from a standalone ENI a caller asked for.	2026-08-12 16:54:19 +10:00
Julian Sommer	7dfadd4576	fix: add bluebottle to scripts	2026-08-12 10:37:28 +10:00
Julian Sommer	5363d25ef5	fix: add northstar to scripts	2026-08-12 10:25:36 +10:00
Julian Sommer	e5f528f877	chore: add rds image to publish	2026-08-05 11:03:35 +10:00
Julian Sommer	d2d0b357a9	fix: tofu e2e	2026-08-05 10:05:45 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Julian Sommer	5e17169661	feat(e2e): complete the RDS end-to-end suite (#608)	2026-08-03 15:38:07 +10:00
joshsiv-mulga	42c7b9b663	Stop segscan workflow running on push	2026-08-03 12:47:02 +10:00
Josh Siviour	a2c451128d	fix: unblock the Ubuntu 26.04 installer (qemu-kvm, sudo-rs) (#665) ... setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two providers there, so apt refuses to resolve it; the token was redundant anyway since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs the qemu-system-* binaries directly. sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it implements no pam_session Defaults setting, and forbids a wildcard inside a command argument. pam_session is now emitted only for classic sudo, and the sysctl grant names a fixed-verb helper instead of a wildcard. The helper is narrower than the rule it replaces: it accepts one interface, a key restricted to rp_filter or accept_local, and a value restricted to 0 or 1, where the old grant permitted any net.ipv4.conf key at any value. Granting sysctl with an unconstrained argument was rejected as root-equivalent. Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime package list is resolved against each supported suite's real indices in CI.	2026-08-03 08:57:50 +10:00