316 Commits

Author	SHA1	Message	Date
Ben Duncan	828b5df4aa	fix: pause a guest on a backend I/O error instead of handing it EIO (#977) ... * fix: fail DRAIN when a volume seal failed shutdownAndUnmount logged the volume unmount error and returned nothing, so StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set. The coordinator then advanced to STORAGE and stopped viperblock while the seal was still in flight, tearing the VBState envelope and leaving the guest unable to mount its root volume on the next boot. The deploy reported success. shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup and stopOne carry it without aborting the rest of the teardown, since QEMU is already down by then, and StopAll aggregates the per-VM failures so handleShutdownDrain ACKs with Error set and never reports vms_stopped. volumeMounterAdapter.Unmount was the second swallow site: it logged each failed seal and returned nil unconditionally. It now attempts every volume and returns the aggregate, which is what the VolumeMounter contract already described. Terminate still tolerates a failed seal — the volumes are deleted next, so an unsealed block map loses nothing a caller can ask for. This does not fix the seal itself. It converts silent data loss into a refused upgrade that an operator can act on. * test: add the storagefault e2e suite for backend-outage guest survival Freezes predastore with SIGSTOP to hold connections open while answering nothing, which is the fault a timeout-free path cannot distinguish from a slow peer. On one node that is a total outage; on three it freezes every node but the guest's own, breaking the RS(2,1) shard floor. The suite writes and flushes a verifiable region before the fault and churns a separate unverified region during it, so an interrupted write is never mistaken for lost data. The root-filesystem load runs from the user's home rather than /tmp, which is tmpfs on this image and would move no bytes on the root device. * test: verify the root device is under write pressure, and size the outage The previous guard matched /^ *vda / against /proc/diskstats, where the device name is the third field, and read $7 which is ms_reading rather than sectors_written. It printed empty and asserted nothing, so a green run could not distinguish a surviving guest from an idle disk. It now samples sectors_written twice and fails unless it advances. The outage is sized from viperblock's absorption rather than guessed: a workload must fill DefaultMaxPendingBytes before backpressure engages at all, so a short freeze measures the retry budget rather than the fault. * test: add nbdkit fault injection to the storage-fault e2e suite The predastore freeze takes the whole storage cluster away from every guest on it, which is too broad to run beside anything else. Removing a single volume's NBD server is the same fault at one-volume scope: one host, one guest, nothing cluster-wide stopped. Both signals are here because they sit on opposite sides of the defect. SIGSTOP holds the socket open and answers nothing, so the guest blocks and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at the default reconnect-delay of 0 fails requests at once, which under werror=report aborts the journal. Signalling is guarded: a pid is matched on comm, our plugin and the volume argument together, and re-confirmed immediately before the signal, because these hosts run nbdkit for other volumes and pids are reused. * fix: pause a guest on a backend I/O error instead of handing it EIO Every virtio-blk device was created with werror=report,rerror=report, so a backend outage was delivered to the guest as EIO. ext4 aborts its journal on that and remounts read-only, and the write that failed is simply gone. Under werror=stop QEMU pauses the VM and holds the failed request, so a backend that comes back resumes the guest with nothing lost. The boot drive, the cold-boot data volume and the hot-plugged data volume all move to stop. Alongside it the NBD transport gets reconnect-delay=30, which pauses requests rather than failing them while a storage process restarts on the same socket; on the legacy -drive form it has to be file.reconnect-delay, because the bare name reaches the raw format driver, which has no such option. The QMP heartbeat now recognises run state io-error and issues cont. After ten attempts the instance is reported impaired and the cadence drops to one attempt per ten heartbeats, but it never stops: QEMU holds the request for as long as the guest exists, so a backend that returns an hour later can still be resumed. The storagefault suite gains the fault this was written for — kill the volume's own nbdkit, hold, then put it back on the same socket — and it now verifies a sha256 region across the outage rather than trusting the guest to be reachable. Three harness defects went with it: the socket-liveness poll ran unprivileged against a 0770 directory and could never succeed, the relaunched socket was left 0755 where QEMU needs group write, and the two variants shared one guest so a fault in the first was diagnosed in the second. * test: cover the io-error resume policy * test: stop two parallel tests racing on the terminate backoff variable * fix: report a guest paused on a backend I/O error as impaired A guest held by werror=stop kept reporting ok on both status checks. QMP stays responsive while the guest is paused -- cont is being sent and answered -- so the QMP failure gate that DescribeInstanceStatus keys on never trips, and the one state a customer most needs to see looked healthy. Instance status now reports impaired/failed while the resume budget is non-zero, timestamped from IOErrorSince so impairedSince is when the guest paused rather than when it was escalated. System status reports impaired too: a backend I/O error is an infrastructure fault, not a guest one, and that distinction is what the two checks exist to draw. The counter is cleared by the first poll that sees the guest running, so both clear on recovery. * test: reap the relaunched nbdkit when the fault test ends relaunchNbdkit detaches the server with setsid so it outlives the SSH session that started it, and viperblockd never learned it exists. Nothing reaped it, so every run left an nbdkit holding a socket and a cache for a volume that was later detached or deleted -- four were found running on dev-prod against two detached and two deleted volumes. The PID is resolved from the socket path rather than the volume id, so a legitimate viperblockd server for the same volume is never the one killed. * test: prove the backend is serving before a fault is called cleared Thawing the process a fault stopped is not the same as restoring the service. The outage test treated SIGCONT as the end of the fault and went straight to judging the guest, so it passed while the cluster it ran on stayed unable to serve a single object for the next 35 minutes, and the next test would have inherited that. thawPredastore now puts and gets an object through every gate, with a budget, and fails the test if the backend does not come back. * fix: do not hand back a mount entry whose export is gone A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit was gone. Every later ebs.mount then matched that entry and returned its socket path, so vmMgr waited for a socket nothing would ever create and the instance could not be started again — on dev-prod a guest stopped against a frozen backend was left permanently stopped, four start attempts failing identically on a socket the node had already removed. Drop the entry when there is positive evidence the export is dead: the socket is absent, or the recorded pid is not running. Neither can be true of a live export, so the double-writer hazard the guard exists to prevent is unaffected. Also make the durability test's pattern helpers report failures. The read dd sat in a pipeline, so its exit status was discarded and the harness hashed nothing and returned the empty-string sha256 as a checksum. * test: report the guest console when a restart never answers SSH A restart that reaches running but never comes back is indistinguishable from a dead root filesystem without the serial console, and the log is gone once the fixture terminates the instance. * fix: converge guest port datapaths concurrently, not one deadline after another Each guest port that misses its SB binding burns the full convergence deadline, and applyEIPs spent them in series. Three ports belonging to long-stopped instances therefore held a newly launched guest's EIP dark for the sum of them before its own probe started. AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and sb-cluster-state-reset is now serialised with a minimum interval so concurrent probes cannot each fire a cluster-wide resync. * fix: refuse a stale cross-node start after the owning node is gone The volume lease already refused a cross-node mount, but it is a liveness claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded writes stops renewing, and 45 seconds later nothing refuses a start elsewhere that loads a stale backend checkpoint. A failed seal now writes a durable marker naming the node holding the only current copy, cleared on a seal that reaches the backend and on delete. The lease covers a live holder, the marker a dead one. The refusal also reaches the caller instead of being flattened to ServerInternal, which told an operator to retry something that cannot succeed while the holder is down. * fix: keep the daemon's message when a start is refused Rebuilding the error from the code alone rendered the generic ErrorLookup text, so a refusal naming the node holding the volume reached the caller as boilerplate about attaching volumes. * fix(ec2): terminate detaches every DeleteOnTermination=false volume TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume on the assumption that Stop's Unmount had already cleared its attachment. Unmount only clears it when the seal succeeded, so a volume whose seal failed stayed in-use pointing at an instance that terminate then deleted, leaving it unreachable with no owner to release it. The running-instance path already detaches every DoT=false volume; the stopped path now matches it, and tolerates an already-gone volume the same way. DeleteVolume also keeps its provider error on the way out. The terminate reaper retries the delete every two minutes and logged a bare ServerInternal, which made a blob node being down indistinguishable from a transient fault. * fix(ec2): DetachVolume --force releases a volume whose instance is gone The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a volume left attached by a part-way teardown could never be detached: the instance no longer runs anywhere and nothing answers. That also made it undeletable, since DeleteVolume refuses an attached volume. ForceDetachVolume already existed for this deadlock but was reachable only from account teardown. Force now falls back to it when the instance has no responder, which is the one case where clearing the attachment cannot strand a live guest. * fix(viperblock): unsealed writes steer placement instead of pinning a volume The dirty marker refused any mount from a node other than the one holding unsealed writes. Instance start already forwards to the node that last ran the instance and falls back after a 30s window, so that refusal turned a working fallback into a volume that could run nowhere once its owner was gone. It traded a rare data-loss risk for a permanent availability failure. The marker is now a record and a warning, not a gate. A takeover logs the previous holder and why it could not seal, then adopts the marker so the old node is no longer treated as the better source. Live exclusion is unchanged and still enforced by the lease. It is also written on open rather than on seal failure. A node killed mid-write never reaches its seal, so the old trigger left no trace in exactly the case the marker exists to describe. Adds 'spx admin volumes unsealed' to see which volumes would cost something if started elsewhere. * fix(viperblock): fence a volume whose lease moved instead of logging it The lease was checked once, at mount, and never again. A node that lost it kept its export up and kept writing into a volume another node owned, so both copies advanced and the backend settled it by arrival order. Losing a renewal now decides: another holder means tear the export down, no holder means the entry merely aged out, so reclaim it and leave the guest running. Fencing never seals — this node's copy is the stale one — and leaves the dirty marker for the winner to take over. This is not an epoch fence. It narrows the window from the 45s TTL to one renewal interval at no write-path cost; the backend still cannot refuse a stale writer. * fix(network): advertise EIPs so a recycled address is reachable The gateway switch port only asked ovn-northd to publish the router's NAT addresses in centralised and routed modes, so on a distributed cluster nothing ever announced an elastic IP. OVN still answers ARP for it, which is why a first-time allocation works and hid this — but no gratuitous ARP is sent when the address moves to a new ENI, so every host on the segment keeps resolving it to the previous holder's MAC until its cache ages out. Reusing an address from the pool then lands on a dead MAC and the instance is unreachable from outside. Distributed mode needs the option as much as the others: northd scopes each entry with is_chassis_resident(<vif port>), so the announcement leaves from the chassis running the instance rather than the gateway, which is what distributed NAT wants. Converge on the already-attached path too. AttachIGW returns early once the gateway port exists and nothing else rewrites it, so without that an existing VPC would stay silent for its whole life. Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to carrying their correct external_mac, each scoped to its own ENI's port. * fix(viperblock): make the volume fence and its dirty marker ownership-safe Four corrections to the lease-loss fence, all found in review. Drop the reacquire path. Losing a lease and finding no successor was treated as a lapsed entry worth reclaiming, but leases.acquire returns the cached entry — already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease renewed nothing and the node went on writing a volume it could not prove it owned. The two errors are not symmetric: a false fence costs one guest a restart, a false reclaim leaves two nodes writing one volume. Fence in both cases, and report which happened through the outcome attribute. Surrender a lease this node cannot confirm. Renewal only noticed a loss when the server rejected an update, so a node partitioned from NATS kept its export up while the TTL expired and another node took the volume. Track the last confirmed renewal and give the volume up once that ages past the point where the server may admit another owner. Bound the renewal call too, so a hung JetStream request cannot hold the goroutine past that point. Do not act on a kill that did not take. Both the fence and ebs.unmount removed the mount entry, released the lease and announced the result before knowing nbdkit had exited — the fence then reported a writer stopped that was still running, and the unmount sealed into a directory a live process was writing. Wait for the exit, and on failure leave the volume mounted and say so. An already-exited process now counts as success, which is what the caller asked. Give the dirty marker a generation. It recorded an owner and nothing to order two nodes' claims, so a node returning after a takeover could clear or overwrite the winner's marker and point placement back at the stale copy. Stamp the lease generation, refuse a write from an older one, and only clear a marker this node still owns. A marker that cannot be written now fails the open rather than leaving writes nothing records. * test(viperblock): cover the stall surrender and the unmount kill guard The two ownership cases the fence work left untested. A holder partitioned from NATS while the winner keeps it never gets an update rejected, so revision checking cannot see it — it has to give the volume up on its own clock. Pins that validity sits below the server TTL with room for a check in between, that a lapsed holder reports the loss as a stall rather than as a takeover, and that releasing a surrendered lease leaves the entry for whoever holds it now. The unmount side of the kill guard: sealing under a live nbdkit writes a state torn between two writers, so a kill that did not take has to fail the unmount and leave the entry mounted for a retry. * fix(e2e): keep the guest private key for as long as the key pair EnsureKeyPair computed the PEM path from the caller's directory but wrote the file inside the ensureOnce closure, which runs only on the first call. Callers passed a per-test artifact dir, and ArtifactDir removes that dir when the test passes — so the first test's success deleted the only copy of the private key while the key pair stayed registered, and every later call returned a path nothing had ever been written to. Guests after the first were then launched against a public key whose private half no longer existed, so the harness could never authenticate and gave up after the full five-minute budget. The guests were healthy and reachable on tcp/22 throughout, which is why this read as a storage or network fault for so long. The directory parameter was the defect, so it is gone: the PEM now lives in a directory the fixture owns and removes with the key pair, and no caller can supply one that is shorter-lived. Also reorder the killed-nbdkit teardown. The reaper registered its own cleanup after the detach registered one, so LIFO killed the block server before the platform detached the disk and DetachVolume then failed with "Node ... is in use". The detach cleanup now reaps once it is done. * fix: stop leaking an nbdkit export when recovery cannot adopt it Restart recovery adopts a guest's boot volume, then routinely fails the -efi volume with "claim lease: context deadline exceeded" -- JetStream is still catching up at daemon start and the KV claim runs on the client's 5s default. Recovery gave up on the first error and never retried, so the export stayed live with no registry entry. At terminate the unmount found nothing to seal, returned NotFound and killed nothing, and the process outlived its guest holding the volume and socket open. EFI is always the second volume adopted, which is why it was always the one that leaked. Two changes, because either alone leaves a hole: - rebuildMountedVolume retries the build (4 attempts, 3s apart) so a JetStream that is merely slow no longer orphans an export for the life of the process. - unmountVolume reaps a corroborated export for the volume it was asked to unmount when the registry has no entry for it, so an export orphaned by any means is still cleaned up. The reap goes through reapOrphanedNbdkit, which declines anything a live process references: at unmount the guest's blockdev may not be torn down yet, and killing the server underneath it strands QEMU. * test: hold the backend-serving probe and assert predastore readiness The backend-outage test declared the backend restored on one successful round trip through each gate. A gate can answer the request that arrives just after a thaw and then lose its blob peers under the load that follows, which is a backend that never recovered reported as one that did -- and that is what happened: the probe reported all three gates serving minutes before the guest was still taking BLOCK_IO_ERROR on every resume. The round trip has to hold across three rounds now, and the count restarts rather than decays when one fails, so a gate that serves and then stops is the failure the hold is looking for rather than noise it averages out. It also checks /readyz on each host and reports any failed check by name. The round trip only proves this workstation can write through a gate; it says nothing about which blob peers that gate can reach, and a gate one node short of the write floor serves a small object and fails a guest. A host that serves no admin listener is logged and skipped, since the port is optional configuration. * fix: scope predastore's admin port to the cluster plane in the host policy predastore's admin listener binds the host's cluster bind_addr and serves /healthz and /readyz unauthenticated. Readiness now names which blob peers a gate cannot reach, which is exactly the kind of topology detail 4432 is peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the public one. * fix: keep resuming a paused guest at heartbeat rate after a long outage Past ten attempts the cont dropped to one per ten heartbeats, so a guest whose backend had come back could sit paused for another five minutes before anything retried it. Measured on dev-prod: the backend was proved serving through all three gates while the guest was still waiting on attempt 60, five minutes after attempt 50. The slowdown was there to avoid paying a failed I/O every thirty seconds on a backend still down. That is nearly free, and five minutes of downtime after recovery is not, so only the log line is throttled now. * test: fail the run when a key pair name resolves to more than one pair A create that lands twice leaves two key pairs with one name and different material. RunInstances resolves the name to one of them while the harness holds the private key of the other, and the only symptom is every guest in the run refusing the key at the SSH gate — five minutes of waiting per guest, four guests, and nothing anywhere naming the cause. Observed on dev-prod: two pairs created a millisecond apart, both named e2e-key-30de10ba. One DescribeKeyPairs after the create says so outright instead. * test: tolerate an already-released volume in the detach teardown Teardown runs DetachVolumeWait after a terminate that releases the volume itself, so the detach can arrive at a volume already on its way to available. That is the outcome it wanted, and treating the IncorrectState refusal as fatal failed a backend-outage run whose assertions had all passed — the guest came back, every acknowledged write verified, and the test still went red on the way out. VolumeInUse is still fatal: that one means the volume is attached somewhere it should not be. * test: carry the merged suites onto the reworked key-pair and account-scoped seams The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir this branch removed, and DeleteVolume's metadata lookup is account-scoped now, so the provider-failure test has to seed and delete under the same account. * refactor: make the fence outcome a type in one const block Four separate exported string constants with a paragraph of comment each, passed to RecordVolumeFence as a bare string. Group them, give them a FenceOutcome type, and take that type at both call sites so an arbitrary value can no longer reach the metric attribute. * refactor: move viperblockd's wire contract into a leaf vbwire package The daemon imported viperblockd for exactly two symbols -- the fenced volume subject and its event -- which put the storage service back on the daemon's dependency graph to name something it only listens for. vbwire holds the contract instead: the subject and event, the dirty bucket name and record shape, and the read-only listing an operator tool needs. It imports nothing from spinifex, so both the daemon and the admin CLI drop their edge to viperblockd entirely. cmd/service.go keeps its one, which is genuine: it constructs the service. * docs: drop a specific host name from the start-refusal comment Named a real dev-prod host in a comment and in test fixtures. Say what the name is instead. * ci: give storagefault a dedicated nightly cell The suite is eligible for both topologies and runs on the PR gate, but no nightly cell ran it: the permutation cells run a deliberately narrow subset and none of the dedicated suite cells named it. Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that environment would report the outage as its own failure. * ci: run storagefault nightly on three nodes, not one The previous commit put its dedicated cell on a single-node runner. That was the wrong topology and the wrong justification: single-node is RS(1,0), where the shard floor cannot be broken and a cross-node takeover after a failed seal does not exist, so the two assertions worth having a dedicated cell for were the two it could not make. Cell 29 is now real-multi on ci-multi. It also corrects the claim that the PR gate already covers the multi-node case. The suite is eligible for both topologies, but e2e.yml's default suite list never names it, so it has never run there. A full pass is ~63 minutes on dev-prod bare metal, almost all of it deliberate waiting -- one test holds predastore SIGSTOPped for seven minutes -- which is too long to add to a gate that currently finishes well inside its budget. Nightly is where it belongs. Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since 90 minutes of suite cannot run in a 50-minute job; the permutation steps gated on matrix.suite == '' so the dedicated cells do not also run them; and a Run Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts the suite against E2E_SUITES_MULTI before running it. * fix: stop a startup KV race leaving vpcd serving keyless guests vpcd opens five KV buckets while NATS is still coming up, each attempt carrying a ~5s JetStream API deadline, and applied three different policies to the same failure: spinifex-vpc-enis was fatal, while instance-state, security-groups, subnets and vpcs each degraded silently for the lifetime of the process. A five-second blip during a cold start therefore decided how a node served metadata until someone restarted it. On dev-prod all three nodes hit this after a deploy. Two were saved by accident: their ENI bucket also timed out, so they exited 1 and systemd restarted them cleanly. The third's ENI bucket opened in time, so it never crashed and ran for hours with no instance record loader. With the local state file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's crawl died on ami-id and fell back to DataSourceNone, and every guest launched on that node booted with no authorized_keys. kvstore.OpenWithRetry retries for a bounded window before any policy applies, logs the cause per attempt and the wait on a late success, wraps the last error with the bucket name, and returns at once on a cancelled context. All five opens use it. Failing to open the instance record space is now fatal, matching the ENI bucket: the dependency whose loss stops guests logging in should not be treated more leniently than the one whose loss stops ENI resolution. Also fixes the reason the fallback was load bearing at all. The local state file was written 0600 and vpcd reads it as a different user in the shared spinifex group, so the local-first path had never once worked and its mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so an inherited umask cannot undo it. Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an unresolvable instance as "no key pair" with nothing logged. * fix: scope the ime-* firewall accept to IMDS and VPC DNS only The per-tap OVS demux delivers any guest packet addressed to 169.254.169.254 or .253 onto an ime-* host port regardless of L4 port, and iifname "ime-*" accept then admitted all of it. Every wildcard-bound host service — sshd, the AWS gateway, northstar, the console — answered guest packets on those addresses; only asymmetric reply routing kept the handshake from completing, and a guest could still drive northstar to recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS (.253:53), letting everything else fall through to policy drop. * docs: document the instance-to-host plane and how to protect it without the ISO The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by host services and is invisible to a NIC-level or cloud firewall, since the traffic is intra-host on the OVS bridge. Explain that the ISO ships the scoping rule armed, that the binary installer ships the firewall off, and that a non-ISO install must enable it with setup.sh --firewall=on or firewall_enabled=true. Include verification from an instance and the host. * fix: withhold the IMDS meta-data listing when the instance does not resolve Every omission from the meta-data root is a positive claim about the instance — public-keys/ absent means "launched without a key pair", and cloud-init believes it, writes no authorized_keys and boots the guest keyless. So resolving the instance was best-effort for a listing whose whole meaning depends on it: a backend error or a miss shortened the listing instead of failing it, and nothing anywhere recorded that the answer was a guess. Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot retry a 200 that is merely wrong, so this converts a permanent silent failure into a transient loud one. servePublicKeys already had this policy for the leaf; the root is what contradicted it. A profile lookup that fails still serves the listing minus iam/, and now logs. The key pair is the field a guest cannot recover from losing, while a missing role surfaces through the credentials leaf, which reports the miss itself. Also stops the storagefault cross-node test failing on its own access path. It tolerates the pattern changing across a takeover, but readPattern needs sudo, and a volume opened from the backend checkpoint may predate cloud-init writing its sudoers drop-in — indistinguishable from an unreadable device at the dd call site. It now checks sudo first, and reads cloud-init's recorded datasource to tell a rolled-back root filesystem from IMDS never having answered, failing loudly on the latter rather than accepting it as storage loss. --------- Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>	2026-09-03 19:45:33 -05:00
Julian Sommer	fae6b08acc	fix: conform DescribeImages BootMode and IAM list required members ... DescribeImages projected BootMode through aws.String unconditionally, so an AMI recording none emitted "", which is outside the model enum. RegisterImage takes BootMode as an optional input, so this is the normal path for an AMI registered without one, not stale data. Set the member only when present. ListAccessKeys and ListGroups built their results from a nil slice when nothing matched. xmlutil drops a nil slice entirely rather than emitting an empty element, so the response lost a member the model marks required. Start both slices empty and non-nil. Integration conformance drops from 18 violations to 15; the remainder are the EC2 AccessDenied error-code findings, which are unrelated.	2026-08-27 11:46:39 +10:00
Julian Sommer	3bf0379daa	fix: gate sts:AssumeRole on the caller's identity policy (#889)	2026-08-27 09:04:25 +10:00
Josh Siviour	e54b79b7bf	fix: harden ochre ingest progress, job cancel, and retrieve error typing (#893) ... Persist ingestion progress and bound per-document work with a timeout so a job on a dead backend fails fast and visibly instead of sitting silently IN_PROGRESS; add StopIngestionJob so a stuck job can be cancelled; map a backend-down Retrieve/RetrieveAndGenerate error to 503 instead of an opaque 500.	2026-08-26 23:01:57 +10:00
Ben Duncan	8b3c706ea3	docs: rework the multi-node install guide, fix cross-doc links ... Multi-node install - Lead with the fact that the guide builds a three-node cluster, and say how to go beyond three. The callout sits inside ## Overview because the docs site discards everything above the first section heading. - Replace the OVN-only sizing prose with a table covering all three distributed layers, and move the OVN control-plane detail into the VPC networking guide, linked from here. - Add minimum and recommended hardware. - Move the object storage layout to the end as background; it was the second thing a reader hit. - Collapse the ISO conversion notes from a section into one skippable box, now that --force is unconditional in Step 4 and the formation port needs no manual intervention. - Expand verification with spx get nodes and spx top nodes, including what to look for in each. - Drop the install-node.sh shortcut (internal tooling) and the "adding a node later" section (no good story yet). Corrections found while writing - Servers 4+ were described as running no database. They run predastore and viperblock like every other node; only the OVN database cluster is capped at three. Fixed in four places. - Step 4 never needed the firewall down: spx admin init opens the formation port itself for the formation window. Only Step 3's peer-scoped OVN ports are affected. - `nft list table inet spinifex_filter | grep spinifex_peers` could never match — the peer list is an nft define, expanded at load time. Anyone following it would have seen empty output and assumed the worst. - Disambiguate "database": with predastore, NATS and OVN all in play it needed qualifying throughout. - Align the example output region with the AWS_REGION the guide exports. Cross-doc links - Rewrite relative and category-prefixed links to the canonical /docs/<slug> form. The site publishes docs flat, so ../foo/README.md and /docs/category/foo both 404. - Point the six links to unpublished docs at the source repo instead. - Fix four fragments written against GitHub's anchor algorithm.	2026-08-21 14:14:28 -05:00
Ben Duncan	b834910e64	UCS tweaks	2026-08-21 09:53:04 -05:00
Josh Siviour	b482b41e96	feat: implement EC2 MonitorInstances and UnmonitorInstances (#853) ... * feat: implement EC2 MonitorInstances and UnmonitorInstances The telemetry collector already supports both EC2 monitoring tiers and already switches a running poller between them when its discovery file changes, but nothing could make that change: Monitoring.Enabled was only ever written at launch, and both API actions returned InvalidAction. Route through the owning daemon over ec2.cmd.<id> rather than the spinifex-workers queue group, so an instance running on any node is reachable; no responders falls back to the shared stopped store, where the write is a CAS so a racing start-claim cannot resurrect a deleted record. Also report Monitoring on DescribeInstances. It is derived in ProjectInstance from the same field the collector's period reads, so a describe cannot disagree with the tier actually being collected and instances launched before this change report disabled without a backfill. * fix: wake the telemetry poller when its monitoring tier changes poller.run re-read its period inside the tick loop, so a reset waited out the interval being replaced: moving a 300s instance to the detailed tier took up to five minutes to have any effect, which reads as the API call not working. Found verifying monitor-instances on env13. updateMeta now signals a buffered retier channel when PeriodSeconds moves and run selects on it alongside the ticker. The signal only resets the ticker rather than forcing a tick, so a published delta still spans the period its batch declares. The send is non-blocking and the channel coalesces, so a slow poller cannot stall the reconcile loop.	2026-08-21 15:36:40 +10:00
tomnewton-mulga	e22db02d24	fix: Rewrite Cisco and OnLogic docs	2026-08-21 14:06:00 +10:00
Josh Siviour	f9c84919ef	feat: co-served GPU model bundles (vLLM + TEI) with family-aware reaper (#845) ... * feat: co-served GPU model bundles (catalog, lifecycle, embed/rerank routing) Serve the generative LLM, embedder and reranker as one co-resident GPU bundle: EMBEDDING/RERANK catalog modalities + co-serve group, bundle-aware admission (one GPU claim sized to summed member VRAM), single-flight one-VM launch with a member->port EndpointRecord, DynamicEndpointResolver member routing. Ochrevector embed/rerank/token-introspection resolve through the resolver (static single-endpoint map retired); rerank degrades to plain KNN on absent/failure. Public InvokeModel-embeddings and top-level Rerank deferred. * feat: co-serve vLLM+TEI bundle image with per-engine readiness Replace the single-model vllm-serve unit with a generic bedrock-serve@ systemd template driven by per-member BEDROCK_* env files, dispatching to vLLM or text-embeddings-router by engine and binding each member to its own port and weights device. TEI serves directly on its serve port; readiness probes each member on the route its engine answers (vLLM /v1/models, TEI /health) with the engine family carried end-to-end from the catalog. * fix: serve nomic embedder correctly in the co-serve bundle Bound embeddings requests to the TEI client batch limit so a large document's chunks are split across requests instead of 422ing. Serve nomic with explicit mean pooling, since the flat weights layout omits 1_Pooling/config.json and TEI would otherwise default to CLS. Add bge-base-en-v1.5 as a standalone alternative embedder. * fix: make the bedrock reaper family-aware and pin-safe A pinned endpoint is now exempt from terminate-and-relaunch on scrape failure, mirroring how it is already exempt from idle reclaim, so a transient unreachability can no longer tear down operator- or commitment-managed serving. A bundle with no generative vLLM member exposes no vLLM load metrics, so it is liveness-probed on /health and kept warm rather than scraped for idleness on a route it does not serve and reaped in a loop. * test: lock in co-serve group-member convergence to one VM * fix: pin self-host HF repo/revision in catalog and stop nested config flattening Carry HFRepo/HFRevision per self-host catalog entry; weights pull defaults --hf-repo and --revision from them so a bare pull cannot silently grab a regressed upstream main (the nomic v5 config.json duplicate-field break). Restrict fixed config filenames to the repo root so a nested config.json (sentence-transformers 1_Pooling/) is dropped instead of flattening onto the model config, and refuse any two source paths that collide at one dest key. * fix: let endpoint delete clear an account-scoped pinned record and report honestly A bare 'endpoint delete' resolved the Global key, which never held a pinned, account-scoped record, then printed 'torn down' over the idempotent no-op while the record survived. Add --account to the delete command so an operator can target the account 'endpoint list' shows, and return DeleteEndpointOutput.Removed so the CLI names a real teardown versus a lookup that found nothing and points at --account. A gone VM already tolerates NotFound, so a scoped delete clears a record whose instance was terminated out of band.	2026-08-20 16:08:55 +10:00
Julian Sommer	e36c8f6ba8	chore: update to go 1.27	2026-08-20 10:37:52 +10:00
Ben Duncan	ae2a1830ef	Merge remote-tracking branch 'origin/dev' into account-quota	2026-08-20 04:01:14 +10:00
Ben Duncan	0f8471062f	feat: add an admin surface for per-account quota overrides	2026-08-20 02:05:17 +10:00
Josh Siviour	67425c41a0	docs: document the frontmatter length rules for the docs site (#832) ... * docs: bring doc titles and descriptions into SEO length limits Rewrite the frontmatter title and description of every published doc so the rendered <title> lands in 50-60 characters (including the site suffix) and the meta description lands in 150-160 characters. * docs: move SEO titles to a seoTitle field and restore display titles The frontmatter title renders as the page H1 and the sidebar label, so tuning it for the title tag shortened nothing and lengthened the visible navigation. Put the search-facing string in seoTitle and give title back its original value. * docs: document the frontmatter length rules for the docs site	2026-08-19 12:24:15 +10:00
Josh Siviour	e84753f9b9	docs: bring doc titles and descriptions into SEO length limits (#831) ... * docs: bring doc titles and descriptions into SEO length limits Rewrite the frontmatter title and description of every published doc so the rendered <title> lands in 50-60 characters (including the site suffix) and the meta description lands in 150-160 characters. * docs: move SEO titles to a seoTitle field and restore display titles The frontmatter title renders as the page H1 and the sidebar label, so tuning it for the title tag shortened nothing and lengthened the visible navigation. Put the search-facing string in seoTitle and give title back its original value.	2026-08-19 12:05:59 +10:00
Julian Sommer	bdc79108e9	fix: make the DB endpoint the only path to the engine (#830)	2026-08-19 11:04:21 +10:00
Ben Duncan	225bf65a8e	feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800) ... Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut.	2026-08-17 17:42:27 -05:00
tomnewton-mulga	0eb868198a	chore: OnLogic doc	2026-08-17 17:13:00 +10:00
Julian Sommer	95911cbfae	feat: DescribeDBEngineVersions and DescribeOrderableDBInstanceOptions (#807)	2026-08-17 15:04:12 +10:00
Julian Sommer	eac0f175ba	feat: require TLS on RDS connections by default, on both engines (#790)	2026-08-17 09:01:42 +10:00
Ben Duncan	42d87d7a5a	feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799) ... * feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.	2026-08-15 14:22:49 -05:00
Josh Siviour	bc8ad05a0f	feat: Ochre Phase 5 knowledge bases (bedrock-agent KB/data-source + RAG) (#792) ... * feat: ochre-vector list-jobs subject Adds ochre.vector.listJobs to the daemon-side VectorService surface (both the in-process and NATS-forwarding implementations) so a caller can list every ingestion job for an account rather than looking one up by id. Widens jobGetter to expose JobStore.List, registers the subject on the daemon's post-connect subscription table, and updates the CLI test double to satisfy the widened VectorService interface. Also carries the jackc/puddle/v2 indirect dependency go.sum was missing. * feat: bedrock-agent knowledge-base + data-source CRUD Adds the emulated AWS bedrock-agent control-plane gateway service: KnowledgeBase/DataSource CRUD, StartIngestionJob/GetIngestionJob/ ListIngestionJobs, backed by new gateway-owned KBStore/DataSourceStore JetStream KV persistence (mirroring registry.go's CAS idiom) and the existing ochre-vector daemon surface for the bound vector index. One knowledge base binds exactly one vector index. embeddingModelArn maps to EmbeddingModel/Dimension; storageConfiguration and roleArn are accepted and stored verbatim but never acted on. CreateDataSource maps S3 + fixed-size chunking config onto a SourceSpec, converting AWS's maxTokens/overlapPercentage into internal ChunkSize/ChunkOverlap. DeleteKnowledgeBase cascades to every data source under it before dropping the bound index. Wires BedrockAgentKB/BedrockAgentDataSources/ BedrockAgentVector into the gateway and registers bedrock-agent in the supported-services and dispatch tables. Runtime (Retrieve/RetrieveAndGenerate) is out of scope here. * test: cover bedrock-agent routing and KB/DataSource store edges Adds end-to-end HTTP tests for BedrockAgent_Request's route table and per-route handler closures (all eleven operations, plus unknown-route/ missing-account/nil-dependency error paths), KBStore/DataSourceStore PurgeAll and no-JetStream-client error paths, and ListJobs coverage (vectorService, NATS round trip, per-account scoping) in the existing ochre-vector API test suite. * feat: bedrock-agent-runtime retrieve + retrieve-and-generate Adds the emulated bedrock-agent-runtime data plane: Retrieve resolves a knowledge base to its bound vector index and runs a similarity query; RetrieveAndGenerate runs the same retrieval, renders a RAG prompt (default or caller-supplied template), and calls Converse in-process for generation, returning citations built from the retrieved chunks and a session id. Recovers the full RetrievalFilter operator set from the raw request body since the vendored aws-sdk-go RetrievalFilter type only carries andAll/orAll, and fixes GetService never routing real signed bedrock-agent traffic (only bedrock-runtime and bedrock had discriminator rules; bedrock-agent's own knowledgebases paths had none). * fix: bedrock-agent KB delete idempotency + ingestion-job ownership CreateKnowledgeBase: a rollback DeleteIndex failure after a kb.Create claim failure is now logged at Error level with the orphaned index id instead of a bare Warn, while the caller still sees kb.Create's own error unchanged. DeleteKnowledgeBase: a bound index that DeleteIndex reports already gone (ErrIndexNotFound) no longer aborts before the KB record itself is removed, so a KB never dangles pointing at an index that no longer exists. Any other DeleteIndex error still aborts the cascade. GetIngestionJob: now verifies the addressed dataSourceId owns the job (Bucket+Prefix match against the data source's own SourceSpec, same best-effort check ListIngestionJobs already used), not just the knowledge base's bound index -- closing a path where a job could be read through the wrong data source id. * fix: default bedrock-agent-runtime retrieve to 5 results (AWS parity) * fix: thread DataSourceID through ingestion jobs JobRecord and IngestRequest now carry an exact DataSourceID instead of relying on Bucket+Prefix as a best-effort proxy. StartIngestionJob stamps it from the DataSource record; GetIngestionJob and ListIngestionJobs filter on it exactly, treating a job with no DataSourceID as belonging to no bedrock-agent data source. * fix: purge KB/DataSource records on appliance teardown Appliance.Teardown purged the index registry and job store but left the gateway-owned KB/DataSource KV records dangling, pointing at a deleted index. WithKBStores attaches those stores the same way WithStores does for registry/jobs, built before Connect so teardown still works on a broken appliance, and Teardown purges them too.	2026-08-15 12:18:59 +10:00
Josh Siviour	d52245001e	feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788) ... * feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate	2026-08-13 15:54:48 +10:00
Josh Siviour	b93f6c258a	fix(e2e): resolve the listener inventory without a Go toolchain on the target node (#787) ... * fix(e2e): resolve the listener inventory without a Go toolchain on the target node tests/e2e/multinode/listener_invariant_test.go resolved docs/security/network-connections/README.md at runtime via `go env GOMOD`, which works under `go test` but not inside a compiled .test binary shipped to a node with no Go toolchain. Embed the doc in a new docs/security/network-connections package instead, so the content is compiled into the binary and every caller parses bytes rather than a resolved path. RepoRoot and DocPath are now unused and removed. * docs: trim the embed package comment to the three-line standard	2026-08-13 14:18:02 +10:00
Josh Siviour	57bfbc20df	test: listener invariant static guard + e2e runtime check (#782) ... * test: add the listener invariant static guard and e2e runtime check Adds both halves the plan called for: a static test in spinifex/network/invariants that scans install scripts and config templates for a bare OVSDB ptcp remote, a wildcard bind on a Cluster/Encap port, or a bind port absent from the inventory, and an e2e test in tests/e2e/multinode that asserts the same on live ss -tulnp output. Both read docs/security/network-connections/README.md "## 1. Inbound Listeners" as their fixture through a shared, non-test parser package (spinifex/network/listenerinventory), so a wildcard exception has to be declared in the doc's own prose rather than hardcoded in either test. * fix: make the listener wildcard exception fail closed on an explicit marker The prior heuristic granted an exception for any unnegated "wildcard" or "0.0.0.0" mention within 40 characters of the port's row. That fails open: a future row phrased as "avoids the wildcard" or with a negation further upstream than the window would silently exempt itself. Replace it with a single exact literal phrase, "binds the wildcard by design", that a row must contain to grant the exception — no other wording counts, so an unrecognized mention denies rather than grants. Update the two rows that need it (5300, 500/4500) and document the marker's role in the inventory's own prose. * docs: record that the Geneve tunnel socket binds the wildcard by design	2026-08-13 12:29:26 +10:00
Josh Siviour	ee1ee4f033	fix(preflight): report services still executing a replaced binary (#781) ... Overwriting /usr/local/bin/spx while spinifex.target is running does not move the services onto it. Each keeps executing the replaced inode until it restarts, so a node serves a mixture of two builds, and because the handlers are NATS queue-group workers the skew shows up as roughly one request in three behaving like the old build — an intermittent fault rather than a broken node. The ansible deploy path already detects and remediates this, but only there: an out-of-band binary copy leaves nothing to report it, and on env19 that state went unnoticed for a full day. spx admin preflight now reports it, so an operator has a command to run and a script has a non-zero exit to gate on. Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a target path that no longer resolves, or a path that resolves to a different inode than the one being executed.	2026-08-13 11:50:34 +10:00
Josh Siviour	2fe885441b	fix(shutdown): scope a local drain to its own node and stop counting failed phases as done (#780) ... * fix(shutdown): scope a local drain to its own node and stop counting failed phases as done A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes to as a fan-out. The node filter only decided which ACK the stopping node waited for, so by the time it applied, every other node had already stopped its gateway, UI and VPC daemon and powered down its guests. Stopping one node took down every guest in the cluster. Requests now carry the node they apply to, and a daemon drops one addressed elsewhere without ACKing. A cluster shutdown leaves the field empty, so it still reaches every node. The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on the host: another node's backends on a shared host, and a live guest's backend that survived a daemon restart under KillMode=process. Cleanup now sweeps this node's own nbdkit-vol-*.pid files and signals a PID only while /proc still agrees it is nbdkit, so a recycled PID is never hit. Finally, the coordinator counted errored ACKs toward its per-phase quorum, so a node that failed to drain reported N/N and the run advanced into STORAGE and PERSIST with guests still running. Only clean ACKs count now, and any errored one aborts the phase unless --force. * test(shutdown): cover the phase gate, every handler's target check and the real proc paths Diff coverage came in at 52.9% against a 60% threshold. The gaps were the per-phase target checks beyond GATE, the real procComm and signalProcess bodies that every cleanup test stubs out, and the coordinator's abort branch, which sat inside runClusterShutdown where no unit test could reach it. The abort decision moves into shutdownPhaseOutcome, a pure function returning the completed count and the message to print, so the rule that an errored node does not count toward the quorum is testable on its own rather than only reachable through a full cluster shutdown.	2026-08-13 11:50:13 +10:00
Josh Siviour	58e0f2879f	feat: serve the deployment CA from IMDS instead of baking it into images (#784) ... Image import shelled out to virt-customize to install the deployment CA into an imported image's trust store. The step was best-effort and skipped whenever virt-customize was absent, which is the default: nothing in provisioning installs guestfs-tools on a node. When it did run it mutated the image after its checksum had been verified, it could not handle Windows guests, and it tied CA rotation to re-importing every affected AMI. Nothing the stack launches depended on it. ECS container instances, EKS k3s servers and BootDirect microVMs all receive the gateway CA at launch via user-data or fw_cfg, and boot itself needs no CA because IMDS is plain HTTP. The only uncovered case is a customer instance whose own workload calls a Spinifex HTTPS endpoint, which needs a pull mechanism rather than an image mutation. The IMDS responder now serves the CA at /spinifex/ca.pem, outside the /latest tree so the AWS-compatible surface stays byte-identical to EC2's. The route is token-free because a CA certificate is public material and the console already serves it unauthenticated; the X-Forwarded-For SSRF guard still applies. The PEM is cached and re-read on modtime change, so rotating ca.pem on a node reaches guests without restarting vpcd.	2026-08-13 10:55:31 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Josh Siviour	6f4aa495e6	feat: reconcile systemd unit files on upgrade (#771) ... * feat: reconcile systemd unit files on upgrade Unit files were written once at install time by install_systemd() in scripts/setup.sh and never re-asserted. A node upgraded by swapping the spx binary kept whatever units it was installed with, so any fix that lives in a unit rather than in code never reached it. This is not theoretical. A node whose spinifex-viperblock.service predates KillMode=process defaults to KillMode=control-group, so on a target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs every nbdkit while guest QEMU survives in the daemon's own cgroup: the guest keeps running with no storage path. Units are now stamped with a spinifex-unit-version marker, embedded in the binary via a generated file, and reconciled by spx admin upgrade. Missing or version-stale units are replaced after a timestamped backup; a unit at the current version whose content differs is reported as operator-modified and never overwritten. Drop-ins under <unit>.d are untouched, so systemctl edit overrides survive. Nothing is restarted. A daemon-reload is enough for KillMode and TimeoutStopSec to apply to the next stop, so drift is corrected without disturbing a running guest. Adds --dry-run, --units-only and --skip-units to spx admin upgrade. * test: cover systemd reconcile error paths and the admin upgrade CLI flow Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate. Adds tests for reconcile.go's error/edge paths (unreadable unit, backup and write failures, daemon-reload failure, malformed version markers, downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly via an injectable exit and a decision matrix extracted into validateUpgradeFlags/upgradeSummary for testability.	2026-08-12 22:29:48 +10:00
Josh Siviour	1e1d00cda1	docs: record the AMI-metadata upgrade cliff and binary-swap skew (#773) ... AMI metadata moved to ebsmetadata documents and the legacy ami-<id>/config.json read path was removed rather than migrated, so an AMI imported before the change is invisible to the control plane after upgrading. Nothing reports it as pending, because the gap is in stored data rather than in a config file. Warn about it alongside the existing object-storage exemption, and say how to check and recover. Also document the binary-swap trap: replacing spx under a running target leaves services on the replaced inode, and because handlers are queue-group workers a skewed node answers a fraction of requests with the old behaviour, which reads as an intermittent fault.	2026-08-12 21:45:46 +10:00
tomnewton-mulga	a851d25cf1	chore: fix CISCO docs	2026-08-12 16:50:57 +10:00
tomnewton-mulga	5b1199a65c	chore: 3 new CISCO docs	2026-08-12 16:02:29 +10:00
Josh Siviour	93a15c1a8f	docs: reconcile the listener inventory with the listeners that exist (#767) ... The inventory was missing northstar (53 on the advertise address, 5300 on the wildcard and dialled cross-node by every node's per-instance DNS shim), charon (500/4500 plus ESP), Geneve 6081, the per-instance IMDS and VPC-DNS listeners, and chronyd. Section 6 asks the operator to compare the table against ss -tulnp, which was not possible while six listeners were absent. Two classifications were wrong. 8443 was Cluster, but predastore binds the S3 gate to 0.0.0.0 deliberately because S3 is a public plane. 4432 was External, but it binds the node's --bind address and is transient. The scope vocabulary predated the three-plane model, so it could not express the encap plane or record that a node with fewer NICs collapses the planes onto its public address. Cluster and Encap now state intent rather than a guarantee. Replaces the reference nftables ruleset. The previous one had no conntrack, loopback, ICMP or ime-* rules and ended in a blanket drop, so an operator who applied it lost guest metadata, guest DNS, all egress return traffic and their own SSH session. The replacement is input-only in a dedicated table, because nftables verdicts are per-table and a default-deny forward chain would override the accepts vpcd writes to ip filter and break every routed-NAT instance.	2026-08-12 12:47:47 +10:00
Josh Siviour	4dca2c7b59	fix: drain guests on a plain systemctl stop spinifex.target (#766) ... * fix: drain guests on a plain target stop, not just restart `systemctl is-system-running` cannot tell a plain `systemctl stop spinifex.target` apart from the stop half of a restart -- both report `running`, and the old `--only-if-host-stopping` gate skipped the drain for both. That left guests running with no QMP shutdown and no volume unmount while predastore/viperblock stopped underneath them, so a real stop discarded any writes still in flight. Add a second signal: `systemctl list-jobs` shows a pending START job for spinifex.target during a restart (the target's stop leg resolves instantly, since it owns no processes) or a pending RESTART job for spinifex-shutdown.service itself when that unit is restarted directly (PartOf propagates target-to-member only). Renamed the flag to `--unless-restarting` to match: drain on a real shutdown or a plain stop, skip only when one of those jobs shows a restart is already queued. Any step systemctl can't answer fails toward draining, and a skipped drain now warns loudly (naming each instance still running) so an operator sees the risk in the journal instead of discovering it via `ps`. `--only-if-host-stopping` stays as a deprecated, still-functional alias so a partial upgrade between unit file and binary doesn't break. * fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss A drain that fails at GATE is indistinguishable from a skipped drain from the guest's perspective: it keeps running while storage goes away underneath it. Stopping spinifex.target shortly after a restart hit exactly this: the daemon had not finished re-subscribing to NATS, so the ACK request came back with zero subscribers and NATS bounced it back near- instantly ("no responders"), which is why the failure logged in about a second despite --timeout=120s. collectLocalShutdownACK now retries with backoff bounded by the phase timeout instead of failing on the first bounce, and an exhausted retry budget logs at ERROR and names the guests still running via a severity-aware refactor of warnIfGuestsLeftRunning, so a failed drain is unmistakable in the journal rather than silent. * fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but shutdownVolumes reaped them one at a time and TimeoutStopSec was never re-derived for that change. With several mounted volumes the serial loop blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and whichever nbdkit hadn't been reaped yet was orphaned. Fan the per-volume utils.KillProcess calls out concurrently so the reap costs one worst-case nbdkit, not N of them summed, and raise TimeoutStopSec to 150s to strictly exceed that single-volume worst case (120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an invariant asserting the two stay in that relationship, and a test proving the reap is actually concurrent (fails against the old serial loop). * docs: document the host and guest lifecycle contract States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.	2026-08-12 12:39:38 +10:00
Julian Sommer	41fcb20161	feat: add note about potential drive wipes	2026-08-12 12:03:46 +10:00
Josh Siviour	80faf59a43	fix(ovn): bind the NB/SB client listeners to loopback and the lan plane (#765) ... setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB control plane was reachable from the public interface. Write access to the Southbound database is control of the whole tenant network fabric. ovsdb-server takes remotes from two additive sources: the Connection table, which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That split is what the fix uses. Loopback goes in the replicated row, because 127.0.0.1 is correct on every node, so replication is harmless and local clients keep working. The lan address goes on the per-node command line via --db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so each node binds its own address without pushing it to its peers. A new --lan-addr flag carries that address, falling back to --db-cluster-local-addr so clustered nodes stay correct without caller changes. The standalone branch now writes /etc/default/ovn-central too, since a two-node install still has a remote ovn-controller that needs the lan listener. The guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting create-insecure-remote without a resolved address would recreate the bug. install-node.sh passes the lan address on both branches and warns when the lan plane has collapsed onto the public address, since the listener then lands on the public interface and a firewall is the only remaining control.	2026-08-12 10:37:01 +10:00
Ben Duncan	c92294e7d5	feat: render predastore templates with the cluster and S3 planes split (#763) ... * feat: render predastore templates with the cluster and S3 planes split The templates set one bind_addr per host, which put raft and blob wherever S3 was published. A host now binds the cluster plane on the address peers dial and the gate carries its own bind_addr for the S3 API, so the installer can hand S3 the WAN without exposing replication with it. The single-node template also drops to one node of each role. Extra blob nodes only hash-partition keys across more badger indexes on the same drive, and extra meta replicas add fsyncs for a quorum that dies with the machine. * docs: regenerate the AWS model operation coverage table * fix: repin predastore to the merged dev commit	2026-08-11 16:35:42 -05:00
joshsiv-mulga	203b805cf0	docs: lead IAM pages with the AWS concept terms users search	2026-08-11 19:35:30 +10:00
joshsiv-mulga	8006cc2f17	docs: SEO pass on the IAM docs cluster ... Rework the frontmatter titles and descriptions for the five Identity docs so each leads with the AWS concept term users actually search, and add contextual in-prose links between them. Only the roles and instance profiles page currently earns search impressions, so it now links to all four siblings and each sibling links back into the cluster.	2026-08-11 17:58:27 +10:00
brynmailer-mdc	fbeb2133c3	docs: note the release range the object storage exemption applies to	2026-08-11 17:23:42 +10:00
brynmailer-mdc	1394fbf488	docs: describe the predastore host and node model ... The port tables were wrong in both directions: inter-node traffic is QUIC over UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are unique within a host rather than across the cluster, so every machine uses the same three, and a single-node install opens only the S3 port because colocated nodes talk over an in-process pipe. The reference architectures scripted a move of node data out of a directory layout that no longer exists, which would have broken any install following them. The update guide now says object storage has no migration path.	2026-08-11 17:23:42 +10:00
brynmailer-mdc	a7a62addeb	test: rebuild the predastore fixture and service tests on the facade ... The fixture wrote a config the strict decoder rejects and built its cluster through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and serves under Run. Loading the real file means the fixture trips over the same validation an install does. Its data directory holds node-<id> directly, and it drains on cancellation rather than being abandoned at exit. The runtime tests drove buildRuntime, serve and the old on-disk layout, all of which predastore owns now, so they go. What replaces them covers what spinifex still decides: the flag merge and its precedence, a host id or gate node the config does not have, and the pid file behind Stop and Status. Two fixes to earlier commits in this series: the host id conversion moves behind its own guard, and the node status probes use WaitGroup.Go.	2026-08-11 17:23:42 +10:00
Josh Siviour	9b2334b964	fix(ochre): self-heal weights stage when the recorded snapshot is gone (#753) ... * fix(ochre): self-heal weights stage when the recorded snapshot is gone runStageWeights's idempotency check trusted a matching source URI without verifying the snapshot it names still exists, so a host rebuild or volume GC that lost the snapshot left stage permanently short-circuited on a useless KV record until an operator ran 'weights remove' first. Inject a weightsSnapshotChecker, verify liveness before the no-op return, and fall through to re-materialise when the snapshot is gone. 'weights list' uses the same checker to mark dangling entries so an operator can see which staged models can actually serve without waiting for endpoint launch to fail. * docs: note weights stage self-heal and list STATUS column	2026-08-10 12:10:56 +10:00
Josh Siviour	272b4b22e8	feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740) ... * feat(images): add a vllm-serving AMI as a stacked mkosi profile LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and Docker but no Python, no vLLM and no serving unit, so tagging it would have moved the failure from admission to first boot. Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side of the launcher contract: the runtime, vllm-serve.service, the weights mount and the tags. vLLM is baked into a uv-managed venv rather than pulled as a container at boot — a runtime docker pull of vllm/vllm-openai is known to stall this platform's write plane, and mkosi's unprivileged build has no daemon to bake an image from anyway. Also bring images/mkosi.profiles/ under make test-images. Its scripts have no .sh suffix, so the existing name glob skipped exactly the files worth linting; sources are found by shebang instead. * fix(images): make the vllm postinst survive the build sandbox Two failures found by actually running the build. uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so `command -v uv` reported a good install as missing. Call it by absolute path, and check it the same way. `vllm --version` initialises device config and dies with "Failed to infer device type" on a builder with no GPU, so it could never have verified anything. Assert the pinned version through package metadata instead, which also catches a version mismatch rather than merely a broken entrypoint. * fix(images): pin the serving AMI datasource and mirror cloud-init to the console A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as the EKS control plane. That path carries a cloud-init datasource pin; the vllm profile never got one, and the base config deliberately leaves datasource_list unpinned, so the serving image was the only system VM booting without it. Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing: read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_' sorts above '-') and the pin is discarded in silence. The postinst asserts the MERGED datasource_list via cloud-init's own read_conf_with_confd rather than checking the file exists, because it existed at 90- too and did nothing there. Also mirror every cloud-init stage to the serial console. A serving VM is headless, and a guest that wedges before handover never surrenders /var/log/cloud-init-output.log, so the console get-console-output reads is the only channel an operator has. The images_test.go case covers the two tags resolveServingAMI filters on; it matches on those alone and never on name, so a typo hides the image entirely. * fix(images): make the dhcpcd data-NIC lease genuinely one-shot dhcp_oneshot() promised to configure the link and exit "leaving no daemon to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd daemon survived on the data NIC holding the control socket. cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same NIC. Instead of spawning, dhcpcd forwards the invocation to the resident daemon as a control command and exits 0, and --dumplease returns the resident daemon's lease, so discovery looks successful. Cleanup then waits for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote, spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s on every system-instance boot. Adding -1 exits once the interface is configured and -p keeps the address and routes across that exit. Both are needed; -1 alone tears the link back down. The test now models the real clients — the stub stays resident unless given the flag that makes its client exit, and the dhcpcd branch is forced by dropping udhcpc from PATH rather than left to whether the host happens to have /usr/share/udhcpc/default.script. * fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles that kernel on first use. The serving image ships the driver and the container toolkit but no nvcc, so the sampler warmup died with "Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes into a GPU cold start, after the model had already loaded. VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler, which is the same operation with no CUDA toolchain at runtime. Everything else on the serving path is already precompiled in the wheels: the attention backend resolves to FLASH_ATTN and CUDA graph capture completes, both without nvcc. Baking FlashInfer's AOT modules is the alternative if the fused kernel is ever wanted, and it keeps the toolchain in the builder rather than the appliance. The unit had Restart=on-failure with no bound, so an unfixable start failure restarted forever and buried its own first traceback under identical copies on the serial console — the only view an operator has of a headless guest. StartLimitIntervalSec/StartLimitBurst make it report instead. The postinst asserts the installed unit carries the sampler setting, for the same reason the datasource pin is asserted there: the alternative is finding out at runtime, on a GPU, five minutes in. Also corrects the shared datasource pin's header, which still claimed it installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm profile precisely because a 90- name loses the reverse-sorted, first-source- wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg. * fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes The 5 minute bound was chosen before any cold start had been measured, and its own comment asked to be corrected once a model actually ran on wattle. It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup complete" 257.6s after launch, leaving ~42s of margin on the smallest model this platform can serve, with a cold torch.compile accounting for 26.3s of it. A bigger card and a bigger model are unmeasured. The two failure modes are not symmetric. Too low tears down an endpoint that was about to come up and reports it as a failed launch; too high only makes a genuinely stuck launch take longer to give up. So the bound is set well clear of the single data point rather than close to it. The CLI's own wait moves to 20 minutes to stay above it. The daemon starts its clock when the VM is launched, up to a minute after the CLI starts its own, so leaving both at 15 would expire the client first and report a bare "timed out" in place of the daemon's abort, which carries the reason. * fix: give the Ochre daemon a routed port in the Bedrock system VPC The readiness probe and every InvokeModel call dial a serving VM's system-VPC address from inside the daemon, but that subnet lives in OVN on br-int and the host has no route to it, so a healthy VM never reached READY. Each daemon now owns an ENI in the system VPC's private subnet, bound to an OVS internal port on br-int by iface-id, exactly as ovn-controller binds a guest tap. The ENI address is installed at the subnet's prefix length, which is what supplies the connected route. BaseURL is unchanged: the address it already carried is now reachable. Describe-or-create keyed on a per-node description, since ENIs carry no tag filter, so a restart or a second endpoint reuses one address instead of leaking one per launch. The port is ensured before anything else the launch creates: a VM the daemon cannot dial holds a GPU for the whole readiness window before unwinding.	2026-08-06 15:26:05 +10:00
Julian Sommer	1c4d41bffc	fix: reach the DB endpoint from any subnet of the VPC (#738)	2026-08-06 13:56:03 +10:00
Julian Sommer	456a60717b	feat: add RDS to awsmodel coverage	2026-08-06 12:00:02 +10:00
Julian Sommer	8c07f80032	chore: remove the half line breaks	2026-08-06 11:10:15 +10:00
Josh Siviour	688ce5ccfd	feat: add AWS model conformance coverage (#736)	2026-08-05 18:46:37 +10:00
Julian Sommer	f7d5d97e3f	fix: propagate RDS parameter group edits (#733)	2026-08-05 15:08:14 +10:00
Julian Sommer	d2d0b357a9	fix: tofu e2e	2026-08-05 10:05:45 +10:00