mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
01cf08c22bcdd9ef52eb7afefbc3c081536fdf77
376 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
e30ec82f6a |
feat: feature-flag the Ochre console (off by default until release) (#950)
* feat: gate the Ochre console behind a runtime flag Add OchreEnabled to the spinifexui service config and /api/config body, defaulting to false so the console and every /bedrock/* route stay hidden until a node explicitly turns it on. * feat: wire the Ochre flag through to the spinifex-ui service unit Add a spinifex-ui-ochre-enabled flag (SPINIFEX_UI_OCHRE_ENABLED, off by default) so the console flag can actually be set on a deployed node, and bump the service unit so the reconciler picks it up. |
||
|
Josh Siviour
|
3680e5b5bc |
feat: render predastore admin_port and gate deploys on /readyz (#936)
* feat: render predastore admin_port and consult /readyz in health probe Spinifex now allocates predastore's admin listener port (8660, after the existing gate/blob/meta bases) and renders admin_port into both the single-node and multi-node predastore.toml templates. The daemon's health probe consults /readyz over that port on the two weak paths (single-host gate fallback and hosts running no meta node), falling back to the prior behavior when no admin port is configured or the listener refuses. * build: pin predastore to the release carrying Host.AdminPort The admin port field this branch reads is new in predastore, so CI could not compile against the pinned version: go.work hid the gap locally by resolving predastore to the checkout instead of the module. |
||
|
Josh Siviour
|
abb817c7d5 |
feat: add admin Ochre catalog read and drop provider-direct model (#917)
Expose per-model availability and reason on an operator-only ListOchreCatalog action for the console, keeping the tenant ListFoundationModels omission intact. Remove the unbuilt provider-direct entry from the v1 catalog. |
||
|
Julian Sommer
|
c81abd1836 | fix: add fipsboot guard to missing services | ||
|
Julian Sommer
|
8c7e3dfa7b | refactor: use bluebottle/pkg/safecast for the numeric conversions (#913) | ||
|
Julian Sommer
|
95afec6a76 | refactor: use bluebottle/pkg/fipsboot for the FIPS 140-3 startup guard (#911) | ||
|
Julian Sommer
|
97acfbbd80 | refactor: use bluebottle/pkg/tlsconfig for the TLS curve allowlist (#910) | ||
|
Julian Sommer
|
d82ee71b54 | refactor: drop duplicated bluebottle logic and hand-rolled token buckets (#905) | ||
|
Josh Siviour
|
e54b79b7bf |
fix: harden ochre ingest progress, job cancel, and retrieve error typing (#893)
Persist ingestion progress and bound per-document work with a timeout so a job on a dead backend fails fast and visibly instead of sitting silently IN_PROGRESS; add StopIngestionJob so a stuck job can be cancelled; map a backend-down Retrieve/RetrieveAndGenerate error to 503 instead of an opaque 500. |
||
|
Josh Siviour
|
f733fed177 |
fix: preserve ochre teardown metadata for auto-reconcile (#863)
Teardown no longer purges the index registry or KB/DataSource records by default, mirroring how they already survive a restart; a --purge-metadata flag restores the old full wipe for an intentional destroy. On the next appliance connect, a new reconcile pass re-creates any registry index whose pg table is missing and re-enqueues ingest from its persisted SourceSpecs, so a torn-down appliance heals without operator action and keeps its KB id. |
||
|
Julian Sommer
|
c8aafa5de8 | chore: delete stale migrations since we forced clean re init anyway | ||
|
Julian Sommer
|
da0c5cf27f | chore: delete useless mocks | ||
|
Julian Sommer
|
56d8792fa0 | fix: more test cleanup | ||
|
Josh Siviour
|
c918f6834a |
feat: back up the ochre pgvector schema to object storage (#847)
* feat: back up the ochre pgvector schema to object storage Adds account-scoped pg_dump/restore of the per-account kb_<accountID> schema to predastore, gzip-compressed and keyed under the account's own prefix. Backup shells out to pg_dump (--no-owner --no-privileges --clean --if-exists) behind a PgDumper seam; restore runs psql then repairs the account role's grants via EnsureAccount/RegrantAccount since the dump carries no ownership. Exposed as `spx admin ochre vector backup/restore --account <id>`. * test: cover ochre vector backup dump/restore and error paths |
||
|
Josh Siviour
|
f9c84919ef |
feat: co-served GPU model bundles (vLLM + TEI) with family-aware reaper (#845)
* feat: co-served GPU model bundles (catalog, lifecycle, embed/rerank routing) Serve the generative LLM, embedder and reranker as one co-resident GPU bundle: EMBEDDING/RERANK catalog modalities + co-serve group, bundle-aware admission (one GPU claim sized to summed member VRAM), single-flight one-VM launch with a member->port EndpointRecord, DynamicEndpointResolver member routing. Ochrevector embed/rerank/token-introspection resolve through the resolver (static single-endpoint map retired); rerank degrades to plain KNN on absent/failure. Public InvokeModel-embeddings and top-level Rerank deferred. * feat: co-serve vLLM+TEI bundle image with per-engine readiness Replace the single-model vllm-serve unit with a generic bedrock-serve@ systemd template driven by per-member BEDROCK_* env files, dispatching to vLLM or text-embeddings-router by engine and binding each member to its own port and weights device. TEI serves directly on its serve port; readiness probes each member on the route its engine answers (vLLM /v1/models, TEI /health) with the engine family carried end-to-end from the catalog. * fix: serve nomic embedder correctly in the co-serve bundle Bound embeddings requests to the TEI client batch limit so a large document's chunks are split across requests instead of 422ing. Serve nomic with explicit mean pooling, since the flat weights layout omits 1_Pooling/config.json and TEI would otherwise default to CLS. Add bge-base-en-v1.5 as a standalone alternative embedder. * fix: make the bedrock reaper family-aware and pin-safe A pinned endpoint is now exempt from terminate-and-relaunch on scrape failure, mirroring how it is already exempt from idle reclaim, so a transient unreachability can no longer tear down operator- or commitment-managed serving. A bundle with no generative vLLM member exposes no vLLM load metrics, so it is liveness-probed on /health and kept warm rather than scraped for idleness on a route it does not serve and reaped in a loop. * test: lock in co-serve group-member convergence to one VM * fix: pin self-host HF repo/revision in catalog and stop nested config flattening Carry HFRepo/HFRevision per self-host catalog entry; weights pull defaults --hf-repo and --revision from them so a bare pull cannot silently grab a regressed upstream main (the nomic v5 config.json duplicate-field break). Restrict fixed config filenames to the repo root so a nested config.json (sentence-transformers 1_Pooling/) is dropped instead of flattening onto the model config, and refuse any two source paths that collide at one dest key. * fix: let endpoint delete clear an account-scoped pinned record and report honestly A bare 'endpoint delete' resolved the Global key, which never held a pinned, account-scoped record, then printed 'torn down' over the idempotent no-op while the record survived. Add --account to the delete command so an operator can target the account 'endpoint list' shows, and return DeleteEndpointOutput.Removed so the CLI names a real teardown versus a lookup that found nothing and points at --account. A gone VM already tolerates NotFound, so a scoped delete clears a record whose instance was terminated out of band. |
||
|
Julian Sommer
|
7b1890eb8a | refactor: use the stdlib uuid package instead of github.com/google/uuid (#841) | ||
|
Julian Sommer
|
8e01e76dc0 | fix: disable embedlit | ||
|
Ben Duncan
|
ae2a1830ef | Merge remote-tracking branch 'origin/dev' into account-quota | ||
|
Ben Duncan
|
f41f5530a0 | test: add an e2e suite proving quotas gate every capped resource | ||
|
Ben Duncan
|
0f8471062f | feat: add an admin surface for per-account quota overrides | ||
|
Ben Duncan
|
dca7a38356 | feat: resolve service quotas per account from a KV override record | ||
|
Julian Sommer
|
60c9dfa89d | fix: remove rds panics | ||
|
Julian Sommer
|
28330cb318 | fix: dont panic on missing flags | ||
|
Julian Sommer
|
bdc79108e9 | fix: make the DB endpoint the only path to the engine (#830) | ||
|
Josh Siviour
|
ea4e7e3fb0 |
fix: log every duration as integer milliseconds under a _ms key (#827)
slog renders a bare time.Duration as an int64 nanosecond count and names the unit nowhere, so every duration field in the fleet read like "elapsed":135000000000. The repo idiom of .Round(time.Second) changed the value without changing the shape and truncated every sub-second measurement to zero. otelsetup.Millis converts a duration for logging under a snake_case key ending in _ms, and TestSlogDurationConvention type-checks ./spinifex/... and ./cmd/... to fail the build when a raw Duration reaches an slog call or a Millis value is logged under a key without the suffix. It found 55 sites across 34 files, well past the sites a grep for time.Since shows, because most durations come from a config field or a package constant. A handler-level ReplaceAttr would fix every site at once but can only emit d.String(), which Elasticsearch maps as a keyword and nothing can aggregate over. |
||
|
Ben Duncan
|
225bf65a8e |
feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800)
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut. |
||
|
Josh Siviour
|
57d0a3b813 |
fix: authenticate ECS ECR pulls from the co-located gateway registry (#814)
The ecs-agent classified only *.dkr.ecr.* hosts as ECR, so a registry advertised as the gateway's bare host/IP pulled anonymously and 401'd. Recognise the agent's own gateway host as ECR and mint a token for it. The instance role — the pull's fallback credential when a task carries no execution role — granted ecs:* only; add ECR read so the pull authorizes. |
||
|
Julian Sommer
|
8bf386617b |
fix: pin the spinifex-ui start test to its own config
The case seeded no config, so loadLocalConfig fell back to the host's installed spinifex.toml. It passed on a provisioned dev box and failed on a bare runner, where the region pre-flight fires before svc.Start(). |
||
|
Julian Sommer
|
eac0f175ba | feat: require TLS on RDS connections by default, on both engines (#790) | ||
|
Ben Duncan
|
42d87d7a5a |
feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799)
* feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down. |
||
|
Josh Siviour
|
f647b635b2 |
feat: pull Ochre model weights from a hub into predastore (#798)
* feat: pull Ochre model weights from a hub into predastore Add `spx admin ochre weights pull`: fetch a self-host model's weights from Hugging Face directly into predastore, producing a prefix `weights stage` consumes. Streams safetensors-only artefacts (plus config/tokenizer and the shard index), refusing pickle/.bin. Pins an immutable commit SHA, writes an ochre-pull.json manifest that stage records as SourceRevision, and aborts clean (removing any written object) on a gated or mid-stream failure. Token resolves from --hf-token, HF_TOKEN, or a stored platform credential; never logged. Also add `spx admin ochre credentials set` to store an encrypted provider credential from stdin, and SourceRevision on the weights KV record + list. * fix: stream hub downloads without a whole-request timeout http.Client.Timeout bounds the entire request including the body read, so a multi-GB safetensors shard was guillotined mid-download at 30s. Bound dial and response-header latency via the transport instead, leaving the body free to stream for as long as the server keeps sending. |
||
|
Josh Siviour
|
bc8ad05a0f |
feat: Ochre Phase 5 knowledge bases (bedrock-agent KB/data-source + RAG) (#792)
* feat: ochre-vector list-jobs subject Adds ochre.vector.listJobs to the daemon-side VectorService surface (both the in-process and NATS-forwarding implementations) so a caller can list every ingestion job for an account rather than looking one up by id. Widens jobGetter to expose JobStore.List, registers the subject on the daemon's post-connect subscription table, and updates the CLI test double to satisfy the widened VectorService interface. Also carries the jackc/puddle/v2 indirect dependency go.sum was missing. * feat: bedrock-agent knowledge-base + data-source CRUD Adds the emulated AWS bedrock-agent control-plane gateway service: KnowledgeBase/DataSource CRUD, StartIngestionJob/GetIngestionJob/ ListIngestionJobs, backed by new gateway-owned KBStore/DataSourceStore JetStream KV persistence (mirroring registry.go's CAS idiom) and the existing ochre-vector daemon surface for the bound vector index. One knowledge base binds exactly one vector index. embeddingModelArn maps to EmbeddingModel/Dimension; storageConfiguration and roleArn are accepted and stored verbatim but never acted on. CreateDataSource maps S3 + fixed-size chunking config onto a SourceSpec, converting AWS's maxTokens/overlapPercentage into internal ChunkSize/ChunkOverlap. DeleteKnowledgeBase cascades to every data source under it before dropping the bound index. Wires BedrockAgentKB/BedrockAgentDataSources/ BedrockAgentVector into the gateway and registers bedrock-agent in the supported-services and dispatch tables. Runtime (Retrieve/RetrieveAndGenerate) is out of scope here. * test: cover bedrock-agent routing and KB/DataSource store edges Adds end-to-end HTTP tests for BedrockAgent_Request's route table and per-route handler closures (all eleven operations, plus unknown-route/ missing-account/nil-dependency error paths), KBStore/DataSourceStore PurgeAll and no-JetStream-client error paths, and ListJobs coverage (vectorService, NATS round trip, per-account scoping) in the existing ochre-vector API test suite. * feat: bedrock-agent-runtime retrieve + retrieve-and-generate Adds the emulated bedrock-agent-runtime data plane: Retrieve resolves a knowledge base to its bound vector index and runs a similarity query; RetrieveAndGenerate runs the same retrieval, renders a RAG prompt (default or caller-supplied template), and calls Converse in-process for generation, returning citations built from the retrieved chunks and a session id. Recovers the full RetrievalFilter operator set from the raw request body since the vendored aws-sdk-go RetrievalFilter type only carries andAll/orAll, and fixes GetService never routing real signed bedrock-agent traffic (only bedrock-runtime and bedrock had discriminator rules; bedrock-agent's own knowledgebases paths had none). * fix: bedrock-agent KB delete idempotency + ingestion-job ownership CreateKnowledgeBase: a rollback DeleteIndex failure after a kb.Create claim failure is now logged at Error level with the orphaned index id instead of a bare Warn, while the caller still sees kb.Create's own error unchanged. DeleteKnowledgeBase: a bound index that DeleteIndex reports already gone (ErrIndexNotFound) no longer aborts before the KB record itself is removed, so a KB never dangles pointing at an index that no longer exists. Any other DeleteIndex error still aborts the cascade. GetIngestionJob: now verifies the addressed dataSourceId owns the job (Bucket+Prefix match against the data source's own SourceSpec, same best-effort check ListIngestionJobs already used), not just the knowledge base's bound index -- closing a path where a job could be read through the wrong data source id. * fix: default bedrock-agent-runtime retrieve to 5 results (AWS parity) * fix: thread DataSourceID through ingestion jobs JobRecord and IngestRequest now carry an exact DataSourceID instead of relying on Bucket+Prefix as a best-effort proxy. StartIngestionJob stamps it from the DataSource record; GetIngestionJob and ListIngestionJobs filter on it exactly, treating a job with no DataSourceID as belonging to no bedrock-agent data source. * fix: purge KB/DataSource records on appliance teardown Appliance.Teardown purged the index registry and job store but left the gateway-owned KB/DataSource KV records dangling, pointing at a deleted index. WithKBStores attaches those stores the same way WithStores does for registry/jobs, built before Connect so teardown still works on a broken appliance, and Teardown purges them too. |
||
|
Josh Siviour
|
c6bc2b4d29 |
fix: only restart NATS when JetStream has actually failed (#795)
The watchdog's is-active guard cannot fire against spinifex-nats.service (Type=simple reports active the instant the process forks, long before JetStream replays its state), so it restarted a NATS that was still recovering. Separately, any non-zero exit from spx js-probe was read as a JetStream write failure, so a broken spx binary read as a cluster-wide outage instead of an unrunnable tool. Replaces the is-active guard with an age gate on ExecMainStartTimestampMonotonic, adds a post-restart cooldown and a restart-count escalation that gives up rather than looping, and teaches the watchdog to treat js-probe exit 2 (and any exit code that isn't 0 or 1) as inconclusive rather than as proof of failure. Bounds the unit's own start with TimeoutStartSec=120, matching the existing TimeoutStopSec derivation. |
||
|
Josh Siviour
|
ab48ead095 |
fix: exit non-zero when a spinifex service fails to start (#791)
* fix: exit non-zero when a spinifex service fails to start Every service start/stop subcommand in cmd/spinifex/cmd/service.go used Run funcs that printed an error and returned, which exits 0. Systemd's Restart=on-failure never fires on a zero exit, so a fatal startup error (seen live: viperblockd's volume lease bucket lookup timing out during a cold JetStream start) left the node permanently down until a manual systemctl restart. Converts every fatal error path across predastore, viperblock, qemunbd, nats, spinifex, awsgw, spinifex-ui, vpcd, northstar and qmp-collector to RunE returning an error, following the RunE + SilenceErrors/SilenceUsage pattern the northstar start command already used in this file. root.go's Execute() already prints the returned error to stderr and calls os.Exit(1), so this centralizes the exit-code decision instead of repeating fmt.Fprintln+os.Exit at each call site. Pure status subcommands with no fallible operations are left as Run. Adds a subprocess re-exec test proving a fatal predastore start now exits 1 end to end, plus in-process RunE tests covering each service's start command. * test: cover the service startup validation branches |
||
|
Josh Siviour
|
d52245001e |
feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788)
* feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate |
||
|
Josh Siviour
|
ee1ee4f033 |
fix(preflight): report services still executing a replaced binary (#781)
Overwriting /usr/local/bin/spx while spinifex.target is running does not move the services onto it. Each keeps executing the replaced inode until it restarts, so a node serves a mixture of two builds, and because the handlers are NATS queue-group workers the skew shows up as roughly one request in three behaving like the old build — an intermittent fault rather than a broken node. The ansible deploy path already detects and remediates this, but only there: an out-of-band binary copy leaves nothing to report it, and on env19 that state went unnoticed for a full day. spx admin preflight now reports it, so an operator has a command to run and a script has a non-zero exit to gate on. Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a target path that no longer resolves, or a path that resolves to a different inode than the one being executed. |
||
|
Josh Siviour
|
2fe885441b |
fix(shutdown): scope a local drain to its own node and stop counting failed phases as done (#780)
* fix(shutdown): scope a local drain to its own node and stop counting failed phases as done A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes to as a fan-out. The node filter only decided which ACK the stopping node waited for, so by the time it applied, every other node had already stopped its gateway, UI and VPC daemon and powered down its guests. Stopping one node took down every guest in the cluster. Requests now carry the node they apply to, and a daemon drops one addressed elsewhere without ACKing. A cluster shutdown leaves the field empty, so it still reaches every node. The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on the host: another node's backends on a shared host, and a live guest's backend that survived a daemon restart under KillMode=process. Cleanup now sweeps this node's own nbdkit-vol-*.pid files and signals a PID only while /proc still agrees it is nbdkit, so a recycled PID is never hit. Finally, the coordinator counted errored ACKs toward its per-phase quorum, so a node that failed to drain reported N/N and the run advanced into STORAGE and PERSIST with guests still running. Only clean ACKs count now, and any errored one aborts the phase unless --force. * test(shutdown): cover the phase gate, every handler's target check and the real proc paths Diff coverage came in at 52.9% against a 60% threshold. The gaps were the per-phase target checks beyond GATE, the real procComm and signalProcess bodies that every cleanup test stubs out, and the coordinator's abort branch, which sat inside runClusterShutdown where no unit test could reach it. The abort decision moves into shutdownPhaseOutcome, a pure function returning the completed count and the message to print, so the rule that an errored node does not count toward the quorum is testable on its own rather than only reachable through a full cluster shutdown. |
||
|
Julian Sommer
|
660dfe3925 | fix: os.exit on viperblock failure so it restarts properly | ||
|
Josh Siviour
|
58e0f2879f |
feat: serve the deployment CA from IMDS instead of baking it into images (#784)
Image import shelled out to virt-customize to install the deployment CA into an imported image's trust store. The step was best-effort and skipped whenever virt-customize was absent, which is the default: nothing in provisioning installs guestfs-tools on a node. When it did run it mutated the image after its checksum had been verified, it could not handle Windows guests, and it tied CA rotation to re-importing every affected AMI. Nothing the stack launches depended on it. ECS container instances, EKS k3s servers and BootDirect microVMs all receive the gateway CA at launch via user-data or fw_cfg, and boot itself needs no CA because IMDS is plain HTTP. The only uncovered case is a customer instance whose own workload calls a Spinifex HTTPS endpoint, which needs a pull mechanism rather than an image mutation. The IMDS responder now serves the CA at /spinifex/ca.pem, outside the /latest tree so the AWS-compatible surface stays byte-identical to EC2's. The route is token-free because a CA certificate is public material and the console already serves it unauthenticated; the X-Forwarded-For SSRF guard still applies. The PEM is cached and re-read on modtime change, so rotating ca.pem on a node reaches guests without restarting vpcd. |
||
|
Julian Sommer
|
65d2501e6b | feat: add MariaDB as the second RDS engine (#762) | ||
|
Josh Siviour
|
9c995ad114 |
fix(test): restore in-package tests that reach unexported symbols (#779)
* fix(test): restore in-package tests that reach unexported symbols admin_upgrade_test.go and units_invariants_test.go were moved to external test packages while, in parallel, validateUpgradeFlags and unitVersionRe grew references in those same files. Both sides merged cleanly and the result did not compile, which silenced the systemd unit guards entirely. Also regenerates units_gen.go, which had drifted from build/systemd: it was missing spinifex-firewall.service and carried a stale spinifex-daemon.service. The drift landed unnoticed because those guards were the tests that could not build. Note that `go generate ./...` does not regenerate this file. The directive lives in generate.go, which is //go:build ignore, so go generate never scans it; the generator has to be run directly from spinifex/systemd. * fix(systemd): bump the daemon unit version so the firewall path reaches hosts The nftables firewall work added `-/etc/spinifex/firewall` to the daemon unit's ReadWritePaths but left the version marker at 1. Reconcile replaces a unit only when the installed version is lower than the embedded one; same version with different content is treated as operator-modified and left alone. An upgraded host would therefore keep its old unit and the daemon would never be able to write peers.nft. Bump the daemon unit to version 2 and give the firewall unit a marker of its own — it had none, so it read as version 0. Pin both body hashes. * fix(systemd): make go generate actually regenerate units_gen.go The directive sat in generate.go, which is //go:build ignore, so go generate never scanned the file and never ran it. Every drift failure told the reader to run `go generate ./...`, which silently did nothing for this file, and the units_gen.go drift that hid the daemon unit version bug landed that way. Move the directive to reconcile.go, which is part of the package and is scanned, and drop the dead one. |
||
|
Josh Siviour
|
fd8be8a814 |
refactor(test): route duplicated NATS bootstraps through testutil (#776)
* refactor(test): route duplicated NATS bootstraps through testutil Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that spinifex/testutil already provides, and one seedKV clone duplicated testutil.SeedKV. Route them through the shared helpers. Sites whose options genuinely diverge are left alone: fixed ports needed for reconnect and restart scenarios, custom TLS and token auth, a custom MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup. The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than CreateKeyValue, so their failure behaviour differs and they stay as they are. Also moves eight test files to external test packages where they declare no helper used by a sibling in the same package. Coverage is unaffected: Go compiles internal and external test packages into one binary per package, so attribution is identical either way. * chore(test): require new test files to use an external test package Adds a preflight check that fails when a newly added test file declares an in-package test. External test packages bind tests to the exported API and let packages share test doubles, which an in-package double cannot be. Only added files are checked, so the existing in-package tests are untouched. e2e and integration suites are exempt: both share a package-level fixture built in main_test.go, so their files cannot move individually. A file that needs unexported access can opt out with a //test:in-package comment. |
||
|
Josh Siviour
|
6f4aa495e6 |
feat: reconcile systemd unit files on upgrade (#771)
* feat: reconcile systemd unit files on upgrade Unit files were written once at install time by install_systemd() in scripts/setup.sh and never re-asserted. A node upgraded by swapping the spx binary kept whatever units it was installed with, so any fix that lives in a unit rather than in code never reached it. This is not theoretical. A node whose spinifex-viperblock.service predates KillMode=process defaults to KillMode=control-group, so on a target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs every nbdkit while guest QEMU survives in the daemon's own cgroup: the guest keeps running with no storage path. Units are now stamped with a spinifex-unit-version marker, embedded in the binary via a generated file, and reconciled by spx admin upgrade. Missing or version-stale units are replaced after a timestamped backup; a unit at the current version whose content differs is reported as operator-modified and never overwritten. Drop-ins under <unit>.d are untouched, so systemctl edit overrides survive. Nothing is restarted. A daemon-reload is enough for KillMode and TimeoutStopSec to apply to the next stop, so drift is corrected without disturbing a running guest. Adds --dry-run, --units-only and --skip-units to spx admin upgrade. * test: cover systemd reconcile error paths and the admin upgrade CLI flow Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate. Adds tests for reconcile.go's error/edge paths (unreadable unit, backup and write failures, daemon-reload failure, malformed version markers, downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly via an injectable exit and a decision matrix extracted into validateUpgradeFlags/upgradeSummary for testability. |
||
|
Josh Siviour
|
f14dd2ae1a |
fix: register EC2 snapshot metadata for provider-backed AMI imports (#774)
The catalog image-import command wrote a volume's blocks through the
EBS provider but never wrote the EC2 control plane's
{snapshotID}/metadata.json, so the imported AMI's snapshot was
invisible to DescribeSnapshots and any resolution path without the
system-AMI fallback. Register it the same way the ochre weights
import already does, and log when the system-AMI fallback masks a
missing document instead of silently succeeding.
|
||
|
Josh Siviour
|
98ec781629 |
feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770)
* feat: introduce the EBSProvider interface and control-plane metadata store
Phase 1 of decoupling the control plane from viperblock. Adds a
provider-neutral block-storage contract that spinifex owns end to end.
ebsprovider defines EBSProvider, whose operation set and idempotency rules
follow the CSI controller service, with NATS request/reply as the transport
rather than CSI's gRPC. Every request and response carries an explicit schema
version so wire skew fails loudly at the boundary instead of silently decoding
a changed payload. The package deliberately exposes no viperblock types:
provider-specific state is an opaque handle. Ships with a NATS client against
the ebs.provider.v1.* subjects and an in-memory fake for tests.
ebsmetadata gives spinifex its own volume and AMI documents under a spinifex
key prefix, so the control plane can answer DescribeVolumes and DescribeImages
without reading the storage engine's persisted state.
The EC2 volume and snapshot handlers gain provider-backed paths for create,
describe, modify, attach-state and delete. Provider injection is optional for
now, so the existing engine path stays the default until the viperblockd side
serves the new subjects and the metadata migration lands.
vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol
it used and is now a local helper.
* feat: serve the EBS provider contract from viperblockd
Gives the ebsprovider NATS client a server. Volume create, describe, expand
and delete, plus snapshot create and delete, are now handled by the daemon
that owns BaseDir and the mounted-volume registry, which is where viperblock
engine construction belongs.
Every handler prefers an already-mounted engine over opening a second one on
the same volume, and expand refuses outright while the volume is mounted.
Opening a second engine on a live volume is the double-writer hazard this
decoupling exists to remove, so the boundary enforces it rather than trusting
callers. Requests carrying an unrecognised schema version are rejected at the
boundary instead of being decoded on a best-effort basis.
Snapshot creation replies immediately with an operation ID and completes
asynchronously on a per-snapshot subject, since a large volume will outrun the
request timeout. Draining the volume stays with the caller: it coordinates
with the guest and is not something the storage boundary can reach.
The snapshot create subject gains an explicit create verb. Addressing it as
ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject
and left the wildcard unsubscribable.
Publish and unpublish are not served yet; the mount and unmount logic they
would front is still inline in launchService and needs extracting first.
Also adds a conformance suite that runs one set of contract assertions against
both the in-memory fake and the NATS client, so each side is checked against
the contract rather than against the other's structs.
* feat: let an operator select the EBS provider
Adds an [ebs] provider key and wires the NATS provider into the EC2 volume,
snapshot, image and instance services at the daemon's composition root. Until
now nothing injected a provider, so the provider-backed code paths in those
services were unreachable.
The key is provider-neutral rather than nested under [viperblock], because it
names the boundary rather than one provider's settings; a second provider will
not need it renamed. An empty value resolves to "embedded", so deployed config
files that predate the key keep today's behaviour, and an unrecognised value
is a startup error rather than a silent fallback.
Startup probes the provider's capability verb and warns when nothing answers.
The daemon and viperblockd start independently, so an unanswered probe is not
fatal, but without it a misconfigured node fails much later as an unexplained
request timeout on the first EBS call.
"viperblockd" stays experimental: under it volume metadata lives in the
control plane's own document, and the leak reaper, GC and admin paths still
read viperblock's config.json, so they cannot see tenant or tag data for
volumes created that way.
* fix: make every volume-metadata consumer provider-aware
Several readers and writers of volume metadata were never given a provider
branch, so under the viperblockd provider they silently used the wrong store.
Each of these is a correctness bug rather than a missing feature, because the
control plane had already stopped writing the document they read.
Modifications had nowhere to live: ModifyVolume built its result in memory and
dropped it, and DescribeVolumesModifications read it back out of viperblock's
config.json, so a describe straight after a successful modify returned nothing.
Volume metadata now carries its own modification record.
The leak reaper read attachment and tags from config.json and wrote orphan
markers to tags.json, neither of which exists for a provider-managed volume, so
a leaked volume was invisible to it and any mark it did write could not be read
back. Tag mutations had the same split: a tag applied after creation never
reached the document DescribeVolumes reads.
Snapshot deletion could not see a provider-managed clone, because a clone
records its source only in the control-plane document. That let a delete strip
chunks a live clone still reads, so it is fixed here rather than left for the
migration.
Volume enumeration and DescribeVolumeStatus were likewise scanning object
prefixes that a provider-managed volume does not create.
Also adds an encryption flag to the volume document, which DescribeVolumes had
been reporting as false unconditionally and snapshots could not carry at all,
and a list operation for AMIs so the image path can follow later.
The legacy path is unchanged; its invariant tests pass unmodified.
* feat: backfill ebsmetadata documents from legacy volume and AMI state
DescribeVolumes under the viperblockd provider enumerates only ebsmetadata
documents, so a volume created before the switch had no document and was
invisible rather than merely slow to read. A per-object read fallback cannot
fix enumeration, so both a backfill and a fallback are needed.
Add an object-store migration kind to the migrate framework. Predastore has no
conditional write, so an object-based version stamp would be last-writer-wins
across nodes; RunObject stamps progress in a caller-supplied JetStream KV
bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a
read-then-write rather than a compare-and-swap, so correctness rests on every
ObjectMigration.Run being safe to re-execute.
The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the
numbered config migrations: it needs viperblock's VBState schema to decode the
legacy config.json, and migrate is imported by every handler that stamps a KV
bucket version. It converts what it can reach and skips unreadable objects with
a log, so one corrupt volume cannot hide the rest of the fleet.
ebsmetadata gains the legacy read fallback for the window before the migration
has run everywhere. It stays free of viperblock by taking the decode as an
injected function value, wired at the composition root, and the fallback is
retired by leaving it unset. A fallback read never writes the converted value
back, which would race the migration.
Prefix enumeration pages to exhaustion. Stopping at the first page would let
the backfill stamp itself complete having converted only the volumes it
happened to see, reintroducing the invisible-volume failure it exists to
prevent.
The migration runs from configureEBSProvider, so only when the provider is
selected; backfilling while the embedded engine is still authoritative would
produce documents that immediately go stale. Failure is fatal rather than
advisory, unlike the capability probe beside it. This makes the switch to
viperblockd one-way, which is now recorded on the config field.
* feat: serve volume publish and unpublish from viperblockd
PublishVolume and UnpublishVolume were the last unserved verbs on the provider
contract, and they are the ones that attach a volume to a VM. Until they are
served the provider path cannot boot an instance, so provider = "viperblockd"
cannot be verified end to end.
The mount and unmount logic was two large anonymous handler bodies inline in
launchService. Rather than write a second mount path, extract each into a named
function and have both the legacy ebs.mount/ebs.unmount handlers and the new
provider handlers call it. Two divergent implementations of nbdkit spawning and
the seal decision would be far worse than the inline bodies. The extraction
relocates code without rewriting it; response publishing stays in each handler
so the provider never publishes to ebs.mount.response.
Publish is idempotent: a volume this node already has mounted returns its
existing attachment rather than starting a second nbdkit against it, which is
the double-writer hazard the boundary exists to prevent. Unpublish preserves
the invariant that a failed seal leaves the volume mounted so a retry can
re-attempt it, and reports an error rather than success.
Both verbs are node-addressed, so they subscribe directly rather than through
the spinifex-workers queue group, and are registered only when NodeName is set.
There is no queue-group fallback: a subject that already names one node has
nothing sensible to fall back to.
Read-only attachment is refused rather than silently granted read-write, since
nbdkit is started read-write or not at all.
Also fixes a latent panic carried by the extracted mount path: viperblock.New
returns a nil VB alongside its error, and SetCacheSize was called on it before
the error was checked, so a failing New would have taken down the handler.
* feat: read and write AMI metadata through ebsmetadata
The backfill migration wrote ebsmetadata AMI documents that nothing read: every
AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at
{imageID}/config.json. Give the image service provider branches for the AMI
read, write and enumerate paths, with enumeration going through ListAMIs rather
than an object-prefix scan, as DescribeVolumes already does.
AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a
viperblock type on an interface boundary in the instance package. It now returns
ebsmetadata.AMI on both paths, the embedded one converting before it returns, so
the boundary names no viperblock type whichever provider is selected.
The legacy-to-document mapping had accumulated three copies. Consolidate them in
ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata
itself stays free of viperblock, and cover it with a reflective test that fails
when a field is added to either schema without being wired through. Sixteen
fields copied by hand in three places would have dropped one silently.
The admin image paths read whichever representation exists and write back to
that same one, so a provider-managed AMI never gains a spurious config.json.
FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan:
missing a dependent there deletes an AMI a live volume still needs, the same
class of bug as the snapshot clone path fixed earlier on this branch.
That union is built by hand rather than by wiring Store.ListVolumes' fallback,
which swallows per-ID read errors. This path deliberately fails closed on a
transport error so a delete cannot proceed on a partial picture, and the
fallback would have quietly turned that into failing open.
The CLI image import now also writes the document, best-effort, so an AMI
imported after the migration is not left reachable only through the fallback.
* fix(ec2): resolve volume config through ebsmetadata when a provider is configured
A provider-created volume has no vol-*/config.json — its metadata lives in
the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy
key unconditionally, so AttachVolume reported a volume that DescribeVolumes
had just listed as available as InvalidVolume.NotFound.
Routing through ebsmetadata.Store also puts this path behind the legacy
fallback, so a volume created before the provider switch stays attachable.
The NoSuchKey remap now names the key it missed; its silence made the
original failure invisible in the daemon logs on every node.
* test(bench): add an EC2 control-plane latency benchmark
Measures the EBS control-plane operations either side of the provider
boundary, so the cost of the NATS hop can be stated rather than guessed.
Reports median and p95 with a bootstrap CI on the median difference, and
labels a delta indicative unless that CI excludes zero.
* fix(ec2): drain the volume before a provider snapshot
The legacy CreateSnapshot branch drains first so the snapshot reads a
current checkpoint; the provider branch did not, and viperblockd documents
the drain as the caller's responsibility. Both sides deferred to the other,
so an attached volume snapshotted stale data with no error raised.
DrainVolume and volumeAttachment now take the fallback state and instance
as strings rather than a viperblock.VolumeMetadata, removing a viperblock
type from an exported signature.
* feat(ec2): route the instance root volume through the EBS provider
RunInstances built its own viperblock engine to clone the AMI snapshot into
the root volume, which made every launch a second writer against a volume
viperblockd already serves. The launch now calls the provider's CreateVolume
with the AMI's snapshot and source volume, and records the result in
ebsmetadata so DescribeVolumes and attach can see it.
A snapshot clone needs both the snapshot ID and its source volume ID to
resolve base blocks, so CreateVolume in the volume handler now sends
SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected
every create-volume --snapshot-id outright.
The gp3 IOPS and throughput constants move to spinifex/types so the instance
and volume handlers share one definition rather than duplicating literals.
* feat(ebsprovider): add a bounded volume seed to CreateVolume
The EFI variable store must be initialised from the launching node's firmware
VARS template, byte for byte. Passing a host path would not work: CreateVolume
is queue-subscribed, so any node can service it, and a foreign node's OVMF
package would bind the guest's pflash region to the wrong firmware build.
CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd
writes them at offset 0 in the engine it already builds. The seed applies only
when the create actually allocates the volume, so a relaunch never overwrites
a BootOrder the guest has since written.
JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes
bounds the seed well inside the NATS max_payload the cluster runs with and
both ends reject an oversized seed with a clear error rather than letting the
publish fail. VolumeSeeding advertises the behaviour so a provider that cannot
seed says so instead of silently dropping the bytes.
* feat(ec2): route the EFI variable store through the EBS provider
prepareEFIVolume built a control-plane viperblock engine to seed the variable
store from the firmware VARS template, making every UEFI launch a second
writer against a volume viperblockd already serves. It now calls the
provider's CreateVolume with the template as seed bytes, so the store is
allocated and written by the daemon that owns the volume.
The template is still read on the launching node and shipped with the request.
CreateVolume is queue-subscribed, so any node can service it, and a VARS
region seeded from a foreign node's OVMF build would bind the guest's pflash
to the wrong firmware.
pflash refuses a VARS region that is not byte-exact, so the returned capacity
is checked against the template length; a provider that rounded up to a GiB
boundary would otherwise yield a guest that cannot boot.
The engine path stays behind a nil-provider guard until the legacy branch goes.
* feat(ec2): route image snapshots through the EBS provider
snapshotRunningVolume and snapshotStoppedVolume each built a control-plane
viperblock engine to take the snapshot backing a new AMI, making CreateImage
a second writer against a volume viperblockd already serves. Both now call
the provider's CreateSnapshot with the volume's provider handle.
The running path keeps its drain, which is what makes the snapshot current
rather than one predating a guest-triggered write. The stopped path needs no
drain, and the daemon already prefers a live mounted engine over opening a
second one, so it handles both cases itself.
getVolumeConfig read the legacy config.json, which a provider-created volume
does not have, so every CreateImage against one failed. It now reads the
ebsmetadata document under a provider, the same way attach does.
* fix(daemon): only the owning node creates an image for a stopped instance
handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which
only the node holding the VM passes, so a running instance is imaged once.
A stopped instance lives in the cluster-replicated KV instead, which every
node reads successfully, so all of them ran the full snapshot and AMI
creation. The caller saw the first reply and the rest leaked: on a three-node
cluster one create-image produced three AMIs and three snapshots.
A stopped instance records its owner in LastNode, so a node that is not the
owner now declines. Gather tallies error frames and keeps waiting, returning
only on a success, so declining cannot preempt the owner's reply.
An instance stopped by a build that never set LastNode elects an owner
through the KV's compare-and-swap, since having every node decline would make
create-image fail permanently for those records.
* perf(ec2): serve DescribeVolumes by ID from the volume documents
The provider branch called ListVolumes unconditionally and filtered in
memory, so asking for one volume read and decoded every volume document in
the bucket. The non-provider branch already had the equivalent fast path.
Explicit IDs now fetch their documents directly. The tenant check moves with
them: a document belonging to another account reports not-found rather than
being returned, matching what the enumeration filter did.
Measured against a memory object store, the fast path stays flat near 20us
regardless of bucket size while the enumeration grows linearly, reaching
4.6ms at 500 volumes.
* chore(lint): ban the viperblock import from the EC2 control plane
The control plane reaches block storage through ebsprovider now, but nothing
stopped a handler from importing the engine again. That is not hypothetical:
the instance handler first imported viperblock for a config struct alone, and
the engine construction followed thirteen days later.
depguard denies the import across handlers/, which matters because
viperblock.New starts a chunk uploader — every construction is a writer
against a volume viperblockd already serves. A types-only import is denied
too, since that is exactly how the last one started.
The four files still carrying the pre-provider engine path behind a
nil-provider guard are exempted by name. The exemption list is the remaining
work: it shrinks to nothing when that path goes.
* fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot
CopySnapshot wrote a control-plane snapshot document under the new ID and
nothing else. The copy described correctly and CreateVolume against it
succeeded, so it looked like it worked, but the new ID had no viperblock
snapshot metadata behind it and the volume failed when the guest attached it.
Producing the metadata means re-sealing it under the destination ID. On an
encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the
sealing helpers are unexported, so spinifex cannot write it. Copying only the
unencrypted form would be dead code on any cluster with encryption enabled.
Refusing with UnsupportedOperation fails at the call the caller can act on
rather than at attach time. Source lookup and the ownership check still run
first, so a bad ID or another tenant's snapshot reports that instead.
* test(ec2): give handler test fixtures their own legacy config.json shape
Handler tests built the legacy config.json fixture by importing
viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly.
Mirror that shape in ebsmetadata/vblegacy, next to the production code
that already reads it (ebsmetadatabackfill), and switch the
fixture-only usages across the volume/snapshot/image handler tests to
it. A byte-for-byte equivalence test against the real viperblock types
guards the mirror from drifting out of sync.
Call sites that hand a viperblock type into putVolumeConfig,
mergeVolumeConfig, or buildVBConfig are untouched, since those
functions still name the real type in their signature.
* feat(ebsprovider): add CopySnapshot to the EBS provider contract
Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy
NATS subject, MemoryProvider and NATSProvider implementations, conformance
coverage) and a viperblockd handler that duplicates a snapshot's metadata via
viperblock.CopySnapshotMeta on the source volume's VB.
* feat(ec2): serve CopySnapshot through the EBS provider
CopySnapshot has been refusing with UnsupportedOperation because a copy needs
its own snapshot metadata on the backend under the new ID, and on an encrypted
volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the
control plane cannot re-seal it. The provider contract now carries a copy
operation, so the storage engine does it.
The provider path generates a new snapshot ID, has the provider duplicate the
backend metadata under it, then writes the control-plane document. Rollback
mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot
rather than leaving one nothing references. Ownership is still checked against
the source before anything is created.
The no-provider path keeps refusing. It has no way to produce the backend
metadata, and the copy it used to write described fine and then failed at
attach — a resource that fails as far as possible from the call that made it.
Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot
reached the provider rather than existing only as a control-plane document.
That assertion is what catches the old behaviour.
* refactor: remove the embedded EBS engine path from the EC2 control plane
viperblockd is now the only EBS provider. The pre-provider path that
constructed a viperblock engine in-process is deleted, and with it every
github.com/mulgadc/viperblock import under spinifex/handlers/ (tests
included). The depguard ec2-control-plane rule drops all four file
exemptions and the !$test carve-out, so the rule now reads simply as
"nothing under spinifex/handlers/** may import viperblock".
viperblock.VolumeConfig was also a data carrier on the provider path. It
is replaced by the spinifex-native ebsmetadata.Volume rather than the
vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is
decoding old objects, while ebsmetadata.Volume already carries every field
the call sites read, so the conversion step disappears instead of being
reimplemented against a legacy shape.
The embedded-only helpers (volumeConfigWrapper, putVolumeConfig,
mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached,
deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json
prefix scan) had exactly one production caller chain each, all through a
nil-provider branch, and are removed. The snapshot-dependency guard on
DeleteVolume is provider-agnostic and is kept, now running on the provider
path where it previously did not.
config: viperblockd is the default and an empty Provider resolves to it.
An explicit provider = "embedded" is rejected at config load rather than
silently upgraded, because starting anyway would migrate a node's volumes
to ebsmetadata one-way without the operator asking. The generated
spinifex.toml template emits viperblockd.
* fix(images): write the imported AMI document from the config the import wrote
ImportDiskImage creates the AMI's snapshot and records it on its own
*viperblock.VB, which viperblock.New allocates fresh, so the value lands only
in {imageID}/config.json. The import command was persisting the ebsmetadata
document from its pre-import manifest copy, whose SnapshotID is still empty.
Store.GetAMI only consults the legacy config.json fallback when the document
is missing, so that empty-SnapshotID document permanently shadowed the real
value and every launch from an imported AMI failed the zero-copy clone check
with ServerInternal.
Rebuild the document by reading back the config.json the import just wrote. A
missing or unreadable config is skipped rather than written from the stale
manifest: no document still resolves correctly through the legacy fallback.
* test(e2e): read NODE from spx get vms instead of VCPU in the spread check
spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to
field 8. The spread placement check still read field 6, so every instance
reported its vCPU count instead of its node, the unique-node count always
collapsed to 1, and the check failed on clusters that had spread correctly.
Read field 8 and strip the ANSI colour codes wrapping each cell.
* feat(viperblockd): add a --debug flag so viperblock debug logging is reachable
viperblockd hardcoded Info, and viperblock's VB.logger() falls back to
slog.Default() because vb.log is only ever set by SetDebug. That left the
engine's debug records, including the chunk uploader's start and disabled
lines emitted during New(), unreachable in a deployed environment.
Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG
rather than the bare debug key, which several commands share. Also sets the
Config.Debug field, which existed but was set only by tests.
* fix(viperblockd): release the chunk uploader when openVolumeVB fails
viperblock.New starts a chunk uploader and a WAL syncer before the backend is
touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the
caller while leaving both running with no handle to stop them. Repeated backend
failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders
on a 30s ticker for the life of the process.
Adds invariant tests for both halves of the ownership rule: a mounted volume's
snapshot create and copy reuse the live engine and log zero volume opens, an
unmounted volume opens exactly one, and a failed open leaves no uploader behind.
The open counts are read from the same log record a production alarm watches.
* fix(ebsprovider): route requests to the node that owns the mount
Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one
spinifex-workers group, so NATS delivered each request to a random node. Only
the node holding the mount can reuse the live engine through findMountedVolume;
any other node constructed a second viperblock engine, with a live 30s chunk
uploader, against a volume nbdkit was serving on a different host. Measured on
a three-node cluster, this was roughly two snapshots in three.
Rather than resolve the owner, let the owner answer. A node subscribes
ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts
a volume and drops it on unmount, mirroring the per-volume ebs.config
subscription. Clients try that subject first and fall back to the queue group
only on ErrNoResponders, which means nothing is mounted and opening an engine
is correct. A timeout deliberately does not fall back, since the owner may
still be running the operation.
Applies to snapshot create and copy, expand and describe, the four verbs that
reuse a mounted engine. Delete keeps queue dispatch.
* fix: rebuild viperblockd mounted-volume registry after a restart
Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a
viperblockd restart while a VM keeps running leaves nbdkit serving a volume
the daemon has forgotten. Owner routing then infers from ErrNoResponders that
nothing is mounted and falls back to the queue group, which opens a second
viperblock engine, with a live chunk uploader, on another node. The same empty
registry also defeats handlePublishVolume's double-mount guard, so a republish
could start a second nbdkit.
Discover surviving nbdkit processes from /proc at startup and rebuild the
registry from them, following orphan_scan.go's precedent: match comm exactly,
parse the argv for volume=<id> and --unix/-p, and adopt a PID only when
nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or
killed; anything that fails corroboration is logged and skipped. Recovery runs
before the first subscription so no handler can race it.
Extract mountVolume's engine construction into constructMountedVB so mount and
recovery cannot drift, and release the WAL syncer viperblock.New starts when
construction fails after it.
* fix: corroborate recovered nbdkit processes from argv, not a pidfile
nbdkit is launched with -f (required for the Go plugin) and never writes the
--pidfile it is given, so no such file exists on any node; the daemon's own
/tmp is a PrivateTmp namespace besides. Corroborating against it rejected
every genuine mount, making recovery a no-op in production.
Corroborate against what the kernel actually has: the process must serve our
plugin, name this daemon's base dir, and still hold its socket. Adopt a volume
once, so two live nbdkits for it leave one registry entry and a warning rather
than two conflicting ones.
* feat: add a second EBS provider backed by qcow2 and qemu-nbd
The decoupling was justified by swappability that nothing demonstrated: the
conformance suite ran against MemoryProvider and against a NATSProvider wired
to a MemoryProvider, so both arms were the same implementation behind
different transports. A fake is not a second implementation.
Add three pieces. natsserve serves the ebs.provider.v1.* contract from any
EBSProvider, depending on nothing but the interface, and replaces the
test-only server twin so conformance now exercises production code. The
conformance suite moves to its own package and is exported so implementations
outside ebsprovider can run it, with no assertion changed. qemunbdd is the
second implementation: one qcow2 file per volume, qemu-nbd for publish, real
snapshot copies instead of chunk references, and every exec behind a runner
seam so its tests need no qemu.
Owner-subject routing is deliberately not served neutrally: it needs a
per-node mounted-volume registry the interface does not model, so it stays in
viperblockd and a neutral server's clients correctly fall back to the queue
group.
* test: run the conformance suite against the qcow2 provider
The suite passes unmodified against an implementation sharing no code with
viperblock, both in-process and through natsserve over the real wire
contract, which is the swappability claim the provider boundary was built to
support.
Two real defects it caught. ExpandVolume and GetVolume inspected the image
before consulting the publication map, so qemu-img could not open a volume
qemu-nbd held the write lock on and the refusal surfaced as an internal
error instead of ErrVolumeInUse; qemu-img info now runs with --force-share,
so describing a volume no longer depends on whether it is attached. Publish
also failed with qemu-nbd's raw exit status when the socket path exceeded the
108-byte AF_UNIX limit, and now reports that reason itself.
* feat: make the EBS provider selectable by config
Add the qemunbd selector alongside viperblockd, a spx service command that
serves the qcow2 provider over natsserve, and service registration. Config
validation accepts the new value everywhere viperblockd is accepted, while
embedded stays rejected by name.
The daemon logged a hard-coded viperblockd constant as the active provider
rather than the resolved one, in configureEBSProvider and in both of
probeEBSProvider's lines; all three now report what is actually configured.
Both daemons answer the same subjects, so exactly one may run on a cluster.
* test: add a live lifecycle check for the qcow2 provider
Drives create, publish, write, snapshot, restore and read-back entirely
through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a
cluster node, so the client under test is the same shape the control plane
uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE
is set, so it stays out of preflight.
* fix: publish NBD URIs in the form standard clients accept
PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy
pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright
with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever
consume what the EBS provider boundary hands back.
Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit
all accept. ParseNBDURI now takes both that and the TCP form, and still
accepts the legacy nbd:unix: syntax for state written by older daemons, so
a conformant third-party provider's URI is no longer rejected on the way
in either.
Verified against qemu 10.0.11: qemu-img and a real -drive option string
both open the URI form, and qemu reports both forms as nbd+unix://?socket=,
so it already treats the URI as canonical.
* test: validate published exports with libnbd's client tools
Every other assertion about the EBS provider goes through our own Go
client, so nothing has ever checked whether the export we publish is usable
by an NBD client that knows nothing about this codebase. That is the layer
neither the conformance suite nor a CSI-shaped contract can reach.
RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume:
the URI connects, the export advertises its size and block-size
constraints, data written externally reads back byte for byte, a ReadOnly
publish sets the read-only transmission flag, and UnpublishVolume tears the
export down. It skips when libnbd-bin is absent rather than passing
vacuously.
Wired to qemunbdd, where all five failed before the URI fix. The publish
assertion in the main suite now requires a dialable URI instead of merely
a non-empty string, which is what let the wrong form through.
* test: run the external NBD client suite against a live viperblockd
The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit
export is the comparison worth having: one external client, two providers it
cannot tell apart, and every difference is a candidate leak in the boundary.
RunNBDClientSuiteWithConfig takes the node name and volume prefix a live
cluster needs, and publishForNBD now unpublishes and deletes what it created
so a run against a real cluster leaves nothing behind. Subtests dial a
normalised URI so a provider emitting the legacy syntax still gets its export
tested rather than every subtest failing on the same string; whether the URI
itself is standard is now its own subtest.
Adds a reconnect check, since publication holds the volume rather than any one
connection. It retries the real operation because a cheaper liveness probe is
itself a connection and re-triggers whatever refused it.
* feat: make the conformance suite capability-driven
The suite asserted a fixed Capabilities literal, so it could only ever pass
against a provider advertising exactly that set. Every implementation had to
match the suite rather than the suite testing the contract. RunSuite now reads
GetCapabilities from the provider under test and gates optional behaviour on
what it advertises, and the fixed literal is demoted to ReferenceCapabilities,
a convenience for callers constructing a MemoryProvider.
Add ReadOnlyPublish to Capabilities. A provider that leaves it false must
refuse a ReadOnly publish rather than hand back a writable export, which is
what viperblockd already does; it now has a way to say so up front.
Add ErrUnsupportedCapability and the unsupported_capability error code. A
request for optional behaviour a provider does not advertise is well formed,
so classifying it as invalid_argument told the caller to fix a request that
was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly.
Declared capabilities now match observed behaviour: qemunbdd advertises
SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with
nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises
neither: the nbdkit Go binding exposes no extents callback, and the mount path
never passes -r.
The external NBD client suite gains a sparse-extent subtest that checks the
export's advertised contexts against the capability, so SparseExtentReporting
is falsifiable from outside rather than taken on trust, and its read-only
subtest now branches on the capability instead of assuming refusal.
Cover the new branches with a MemoryProvider advertising no optional
capabilities and one advertising OnlineExpansion, the branch no provider we
ship exercises.
* refactor: rename SourceVolumeID to SourceSnapshotVolumeID
CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd
required it whenever SourceSnapshotID was set and used it to resolve the
snapshot's chunk references, so there it named the snapshot's origin volume.
MemoryProvider ignored it entirely. qemunbdd implemented it as an independent
volume-to-volume clone, which is what the name and doc comment implied. The
conformance suite could not catch the divergence because the reference
provider had no branch for the field at all.
The contract already has a field meaning "the volume this snapshot came from":
Snapshot.SourceVolumeID, which every provider populates. So the request field
is the same fact travelling back the other way, and its name was the problem.
Rename it to SourceSnapshotVolumeID, document it as the source snapshot's
origin volume, and state that it is required with SourceSnapshotID and
meaningless without it.
All three providers now enforce one rule: a source snapshot must be
accompanied by its origin volume, or the request is invalid_argument.
MemoryProvider additionally rejects an origin that does not match the
snapshot's own SourceVolumeID, so a wrong name fails instead of resolving
blocks against a volume the snapshot never came from. qemunbdd loses its
volume-to-volume clone branch, which implemented behaviour the contract no
longer offers and no caller ever requested.
The suite pins both the requirement and a full create-snapshot-restore round
trip, so the field is exercised end to end rather than only on the error path.
* feat: make the conformance suite runnable against a live provider
The suite had only ever run against MemoryProvider and qemunbdd, both of which
are discarded with the test binary. viperblockd, the provider production
actually uses, was the one implementation the suite never judged, because the
suite assumed each subtest got fresh state: hard-coded volume IDs collide with
an earlier run's leftovers on a store that keeps them, and a hard-coded node ID
addresses a node that does not exist.
SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records
every volume, snapshot and publication the suite creates and tears them down in
dependency order, logging rather than failing so a teardown problem cannot mask
the assertion a subtest was making. Two new tests cover this by running the
whole suite twice against one shared provider instance, which passes only
because cleanup works, and once per prefix to cover concurrent runs sharing a
daemon.
TestLive_ViperblockdConformance wires this to a real cluster behind
SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite.
* fix(viperblockd): close the four contract gaps the live conformance run found
Unpublishing a volume this node has no mount for now succeeds instead of
returning not_found, matching the legacy unmount path: nothing mounted here is
already the state the request asks for.
CreateSnapshot refuses a snapshot ID that already exists against a different
source volume with already_exists, rather than overwriting it.
DeleteVolume refuses a published volume with volume_in_use instead of tearing
the mount down, on this node or on whichever node answers the volume's owner
subject. The export and the guest writing through it survive the metadata
deletion, so the old behaviour surfaced later as corruption.
CreateVolume reads the source snapshot before writing the volume, returning
not_found when it is absent and invalid_argument when it was taken from a
different volume than the request names. Previously both produced a blank
volume and a successful response.
CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument
now that it carries a sentinel.
* feat(ebsprovider): pin the source snapshot's origin volume in the contract
The suite only required source_snapshot_volume_id to be present, so a caller
naming a volume the snapshot never came from was accepted everywhere. The
snapshot records its own origin, so this is checkable: creating from a snapshot
with a mismatched source volume is now invalid_argument for every provider,
enforced by qemunbdd against its snapshot metadata and by viperblockd against
the snapshot's persisted SourceVolumeName.
* test(ebsprovider): run the contract suite against viperblockd in CI
CI installed neither qemu nor nbdkit, so every conformance test that needs a
real implementation skipped silently and MemoryProvider was the only provider
the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and
sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails
instead of skipping.
TestViperblockdConformance stands the provider handlers up over an embedded
NATS and a real predastore and runs the same suite against them, which is what
caught CopySnapshot accepting a foreign volume's snapshot on unencrypted
volumes.
* feat(viperblockd): honour read-only publish and advertise block size constraints
ReadOnlyPublish was advertised false and read_only=true was rejected,
so the capability was true only by omission. nbdkit -r now starts the
export when the request asks for it, which sets the NBD read-only
transmission flag and makes the plugin refuse writes. Access mode is
fixed when nbdkit starts, so a republish asking for the other mode is
refused with VolumeInUse rather than handed the running export.
Block size constraints were absent entirely: the nbdkit Go binding has
no block-size callback, so the plugin cannot advertise them. The
blocksize-policy filter can, without a binding callback. Minimum 1
because viperblock's WriteAt handles unaligned requests with a
read-modify-write, preferred 4096 to match its block size, maximum 32M
to match what qemu-nbd advertises.
* test(conformance): judge sparse reporting by the allocation map, not the context
The subtest asserted that base:allocation was offered. nbdkit offers
that context whether or not the plugin can compute extents, answering
'all allocated' when it cannot, so the assertion passed for a server
with no extent support at all — it could not fail. What separates the
two is the answer: a volume with nothing written to it is entirely
holes to a server that really tracks allocation.
Read the map with nbdinfo --map --json and compare holes against the
advertised capability. qemunbdd reports holes and advertises true;
viperblockd reports none and advertises false.
* test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime
The predastore fixture is one daemon per test binary, but its cert was
exported with t.Setenv, which reverts when the first test that called
Start finishes. In-process clients did not notice because crypto/x509
caches the pool behind a sync.Once, loaded while the variable was still
set. Child processes did: nbdkit spawned by a later test inherited an
environment with no SSL_CERT_FILE and could not verify the daemon, so
the whole NBD client suite failed when the package ran as a whole while
passing when run alone.
Also install libnbd-bin in CI, so the NBD client suite runs there
instead of failing on missing tools.
* ci: install nbdkit-plugin-dev so the viperblock plugin can build
* test(volume): give the snapshot fixture an owner
CreateVolume now fails closed on a snapshot whose owner it cannot match,
including an unset owner_id. The fixture predates that and wrote none,
so the provider test failed with InvalidSnapshot.NotFound before ever
reaching the provider.
* test(integration): give the volume harness a real provider
The control plane no longer builds volumes itself, so CreateVolume in
the integration harness failed with "no EBS provider configured": the
harness wired a VolumeServiceImpl with nothing behind it. It now serves
viperblockd's provider subjects against the same predastore fixture, so
the blocks are still written by real viperblock to a real daemon.
Three persisted-state assertions moved with it. TenantID, VolumeType
and AvailabilityZone are control-plane facts, held in the ebsmetadata
document rather than viperblock's own state, so the storage oracle now
asserts their absence and DescribeVolumes -- whose answer is
account-scoped -- is what checks them.
* refactor: route image import through the EBS provider and drop the legacy metadata path
The control plane could still only run against viperblock: the AMI import and
the ochre weights import both linked the storage library directly, and the
ebsmetadata store kept a read fallback onto viperblock's config.json.
Import now goes through the provider contract with no new verb: CreateVolume,
PublishVolume, a bulk copy over the published NBD export with qemu-img, then
UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS,
whose request size the contract caps far below a disk image.
The migration scaffolding goes with it. ebsmetadata documents are now the only
thing the control plane reads: no legacy prefix scan, no config.json fallback,
no backfill on daemon start. config.json belongs to the provider, which is the
single writer of it.
ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is
right for DescribeImages, where one corrupt document must not hide the fleet,
and wrong for a name-uniqueness check, which cannot rule out a name it failed
to read.
spinifex and cmd no longer import viperblock outside services/viperblockd and
testutil/vbscan.
* feat: add a volume enumeration verb so a lost document cannot strand blocks
Making ebsmetadata documents the sole index of volumes left one failure with
no recovery: a volume whose document is gone disappears from DescribeVolumes,
from the leak reaper (which enumerates documents), and from the provider,
whose contract had no verb that asks what it holds.
ListVolumes closes that. It is capability-gated on VolumeEnumeration and
paginated, following CSI's optional LIST_VOLUMES rather than inventing a
shape. Refs are deliberately not Volumes: enumeration answers what exists,
and a provider must not open every volume's engine to say so.
Each implementation reads what actually exists rather than what it remembers.
qemunbdd walks its volumes directory, not its in-memory map, which does not
survive a restart; viperblockd walks the bucket's top-level prefixes. A
volume the provider has forgotten is precisely what a caller is looking for.
The consumer is a read-only orphan report: provider volumes minus documented
ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is
excused only when its base volume is documented, so a stranded pair is still
reported. Listing documents stays strict, because reporting a volume as
orphaned on the strength of a document that failed to decode would point an
operator at live data.
* fix: stop the orphan scan accusing storage of holding things it does not
A live env19 run reported three orphans against a healthy cluster: an AMI's
backing volume, a snapshot, and the cluster key store. All three were wrong,
and a report that cries wolf on a clean cluster is worse than no report.
Enumeration treated every top-level bucket prefix as a volume. Snapshots sit
beside volumes at that level and have a config of their own, so presence of a
config cannot separate them; the snap- name is what does, which is already how
the AMI removal path reads them back. keys/ and spinifex/ are not volumes at
all.
An AMI's blocks are a genuine provider volume, so enumeration keeps reporting
them. What was missing is on the reading side: an AMI is recorded as an image
document, not a volume one, so the scan now treats a documented image as
known. Listing images is strict for the same reason listing volumes is —
unreadable and absent are different answers, and only one means orphaned.
* fix: let an orphaned image's blocks be reclaimed
The orphan report tells an operator that reclaiming a stranded volume is their
decision, and images remove --force is the tool it points them at. On env19
that tool failed on exactly those volumes: removal drops the ebsmetadata
document first, and predastore answers NoSuchKey for a document that is
already gone, which is the definition of an orphan.
Deleting the document stays first and stays fatal. That ordering is the
barrier that stops a launch landing on blocks about to be deleted, and an AMI
still advertised after its blocks are gone is the worse failure. Only an
already-absent document is now tolerated: the barrier is satisfied, and the
blocks are the sole thing left to reclaim.
S3ObjectStore.DeleteObject normalises a missing key the way GetObject already
does, so this is recognised rather than matched on error text.
The obvious unit test passed without the fix, because the in-memory store is
silently tolerant on delete and real S3 answers 204 — neither behaves like the
backend this runs against. The test now drives a store that reports the miss,
and fails without the fix.
* feat: add per-verb OTel spans to the EBS provider contract
The ten provider verbs cross a single NATS seam and nothing measured it.
The only spans on the storage path were the legacy ebs.mount/ebs.unmount
pair, so control-plane storage latency could not be attributed to a verb
or compared across provider implementations.
NATSProvider.request is the one client-side funnel every verb passes
through, including both legs of an owner-subject fallback, so a single
span there covers the whole contract and measures what the control plane
experiences regardless of which provider answers.
Span names come from a closed verb set rather than the subject. Owner
and snapshot-create subjects embed a volume ID and the mount/unmount
subjects embed a node ID; naming spans after them would give every
volume a span name of its own. The IDs move to attributes and an
unrecognised subject collapses to a single unknown verb.
Trace context rides the message headers, so the serving provider joins
the caller's trace instead of rooting one of its own. The carrier is a
copy of the one in spinifex/utils rather than an import: that package
pulls in the aws-sdk and the cluster key store, which the wire contract
must not depend on.
Responses carry no shared accessor for their error field, so both sides
re-read just that field from the encoded reply, and only when the span
is recording. Without it a failed verb looks like a successful round
trip.
viperblockd serves the contract with its own handlers rather than
natsserve, so its handlers now take the per-message context and its
publish/unpublish pair drops the subject-named consumer span it opened
for itself.
* feat: add null and hostile providers, and a per-verb bench suite
The provider boundary made it possible to put something other than
viperblock behind the control plane, and nothing yet did.
nullprovider is the floor. It is not a new type: MemoryProvider already
answers every verb from a map and natsserve already serves any
EBSProvider, so the package names that composition and states what it
measures. On loopback with no storage behind it a round trip costs
around 195us against 74ns in process, which is the seam's own price and
is not attributable to any storage engine.
conformance.RunBenchSuite prices one round trip per verb and drives
every implementation through the same calls in the same order, so two
providers' numbers are comparable. It measures control operations only:
guest I/O never crosses these verbs.
hostile wraps any provider in one that misbehaves. Faults derive from
the seed and a per-target sequence rather than a global counter, so
concurrent callers on different volumes cannot reorder each other's
faults and a failure found deep into a soak run replays exactly. Error
and error-after-work are separate faults on purpose: the second reports
failure after the work has already happened, which is how an orphan is
made, and is what makes an enumeration scan the right oracle for a soak.
Enumeration itself is never faulted, since an oracle that can be told
what to say cannot falsify anything.
The soak asserts that every volume the provider still holds is
attributable to an injected fault. A leftover with nothing behind it is
the control plane losing track of storage on its own.
Two bugs were found by falsifying the new tests rather than by running
them. The FNV-derived draw clustered badly enough that two fault bands
were never reachable in 1200 draws, so the draw now goes through a
SplitMix64 finalizer. And the first version of the lie test was
worthless: MemoryProvider returns copies, so a lie could not corrupt it
whether or not the injector copied, and the test now drives a provider
that hands out pointers into its own state.
* test: bench the provider contract against qemunbdd and a live viperblockd
RunBenchSuite was only ever pointed at the null provider, so the numbers
priced the seam and nothing else. Point it at the two real implementations.
BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a
provider that stores nothing and wrong against one that does, so qemunbdd and
viperblockd are both measured at 64 MiB.
liveProvider splits into a testing.TB flavour. Benchmarks need the same
connection and the same node and cannot be handed a *testing.T.
* fix(viperblockd): carry the trace context into every viperblock engine call
Both S3 clients only emit a span when the request already carries one, so
calling the no-context Init, LoadState, SaveState, LoadBlockState and Close
left the whole engine open untraceable. Every provider verb traced as one flat
server span, and the 342ms describe measured on env19 could not be attributed
to anything.
viperblock already exposes the Ctx variants and Backend already declares
InitCtx, so this is threading, not new instrumentation. openVolumeVB,
openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine,
copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and
mount/unmount keep the context their own span derives instead of dropping it.
completeCreateSnapshot gets WithoutCancel: the background half of
accept-then-publish has to outlive the request it was accepted by and still be
attributable to it.
The per-volume ebs.config handler opens a consumer span like the other legacy
subjects, so a config update is no longer the one path with no trace at all.
* perf: describe a volume without opening an engine on it
describeVolumeEngine fell through to openVolumeVB for any volume not mounted
on the node serving the request, which constructed a full viperblock engine
to read one integer. On env19 that decomposed as a 22 ms reachability list, a
3 ms state read and a 16 ms PutObject of the volume's own config.json — the
write coming from the SeqNum bootstrap a read-write open performs.
Use viperblock's read-only state load instead. A describe is now the 3 ms
read: no background goroutines, no reachability probe, no SeqNum window
claimed, and no write issued by a worker that does not own the volume.
Falsified: restoring the openVolumeVB call fails
TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the
list, with the fake backend answering each so neither assertion hides behind
the other.
* perf: share one HTTP client across every viperblock backend
Four sites built the same S3Config by hand, each leaving HTTPClient unset, so
every engine open and state read got its own connection pool and paid its own
TLS handshake. That was invisible while Backend.Init issued a whole-bucket
list — the list paid for the connection and the state read rode it — and it
became the dominant cost once describe was reduced to a single request.
Route all four through cfg.volumeS3Config, which builds the client once.
Falsified: dropping the HTTPClient field takes
TestDescribeVolume_ReusesOneConnection from one connection to three.
* perf: wait for nbdkit to be ready instead of waiting a fixed second
mountVolume spawned nbdkit and then ran a confirm loop until a one-second
deadline passed. The loop only broke early when the process exited, so the
fast-failure path worked and the success path always cost the full second.
That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so
the timings never contradicted the assumption that it was process spawn.
Wait for the endpoint to accept a connection instead. That is also the
question the caller needs answered, since it is about to hand the URI to
something that will connect to it — where 'still alive after a second' only
correlates with it. A nbdkit that exits during startup still fails the mount
immediately, and one that never comes up now fails on a deadline rather than
being assumed healthy.
Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness
timing assertion and the fast-failure one.
* test: measure the data path, not just the control verbs
The conformance benchmarks price create, publish and describe. Guest
I/O never crosses those verbs, so none of those numbers say anything
about what a guest experiences. This drives the published export with
qemu-img bench, which sits on the same block layer virtio-blk does.
It asserts only that the I/O completes and logs a table: what a given
store is capable of is not a contract, and pinning a number here would
only produce a flaky test.
Read profiles stay inside the region the write profiles covered, or
they would be timing how fast the export answers with zeroes, and one
write profile flushes, because an unflushed write measures how fast the
WAL accepts data rather than how fast the store takes it.
Profiles are spaced apart because the plugin tears an engine down per
disconnect and removes the volume's local directory as it goes, so a
connection opened while the previous one is closing fails its
handshake.
* test: run data-path profiles back to back
The spacing was a workaround for a volume lock that did not hold across
close. With that fixed, consecutive profiles are worth more than the
workaround was: each is its own NBD connection, so a run that completes
has also shown that an arriving connection waits out the previous one's
close rather than racing it.
* fix: refuse a delete that cannot establish whether a volume is published
volumeMountedElsewhere returned false on every failure path, including a
request timeout, so a busy or partitioned owner was indistinguishable
from no owner at all and the delete proceeded. Deleting a published
volume leaves the nbdkit export and the guest writing through it alive
over storage that is going away.
Report the three outcomes separately. NATS distinguishes no-responders
from no-answer, so only the former means unmounted; an unanswered probe
is refused and the caller may retry once the owner is reachable.
* feat: exclude a second viperblock engine with a JetStream volume lease
Nothing in the stack stopped two viperblock engines opening one volume from
different nodes. For an encrypted volume that is not just a consistency risk:
the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte,
and the SeqNum window claim is a read-then-write with no CAS, so two engines
opening concurrently issue overlapping nonces under the same key.
Every engine open now takes a lease first, keyed by volume in a JetStream KV
bucket. A lease is a KV create, which JetStream serialises, so a second
claimant loses instead of both winning. The revision that won is the
generation: monotonic across the bucket, so a later opener always carries a
higher one than the writer it displaced. Entries carry a TTL and a renewal
loop, so a node that dies mid-mount does not wedge its volumes forever.
Losing a lease is logged, not enforced. Until the object store can refuse a
stale writer's PUT, failing live guest I/O on lease loss would add a failure
mode without stopping the zombie it is aimed at.
A daemon with no lease store refuses to open rather than opening blind, so
nbdkit recovery binds the store before it adopts surviving exports. Repeat
acquisitions on one node share a lease: the unmount seal opens a detached
engine while the mount being torn down still holds it, and cross-node
exclusion is what the lease is for. A restart reclaims this node's own stale
entries, since the exports that outlived the daemon are still running.
* refactor: consume masterkey from bluebottle in the test this branch added
* feat(ebsprovider): classify refusals a caller may retry
Every ErrorCode said what went wrong and none said whether the same
request could succeed later, so a caller could not tell a permanent
refusal from a transient one. Transient conditions shipped as internal,
which tells a caller to give up when it should retry.
Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(),
which callers branch on instead of on the code itself. DeleteVolume's
"could not establish whether the volume is published" refusal moves off
internal onto it: the owner that did not answer may answer later.
TestErrorCodeRetryable names every code, so adding one without deciding
its retry meaning fails rather than silently defaulting to permanent.
TestClassifyError pins the sentinel-to-wire-code mapping, which had no
coverage at all; a sentinel missing from classifyError falls through to
internal and loses both the reason and the retry signal.
* fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for
Four providers advertised the capability and nothing consumed it: there
is no clone verb on the interface. CreateVolume's SourceSnapshotID
restores from a snapshot and CopySnapshot duplicates snapshot metadata;
neither is a volume clone, and the field documenting SourceSnapshotID
says so explicitly. CSI expresses clone as a volume_content_source with
a volume source, which this contract does not implement.
Advertising it told a caller a feature was available and then gave them
no way to use it. Removed rather than backed by a method: adding a clone
verb is a feature, not a contract fix, and the capability can come back
alongside it. The Capabilities doc comment now states the rule, and
qemunbdd's stale comment about sparse extents is corrected while there.
* feat(ebsprovider): add ListSnapshots so snapshots are reconcilable
The contract could create, copy and delete snapshots but never say which
ones it held. That left the control-plane metadata document as the only
index of them, so losing one stranded the data it pointed at with no way
to find it again. This is the same exposure ListVolumes closed for
volumes, and it stayed invisible because ListSnapshots is optional in
CSI too.
ListSnapshots pages exactly as ListVolumes does, gated on a new
SnapshotEnumeration capability. SnapshotRef is deliberately not a
Snapshot: enumeration answers what exists, and reading a snapshot's size
or state can cost far more than listing it.
viperblockd walks the bucket's snap- prefixes, sharing the top-level
walk with ListVolumes. It leaves SourceVolumeID empty because the link
lives inside the snapshot's own encrypted state and reading it would
mean a fetch per snapshot listed. qemunbdd walks its snapshots
directory rather than its in-memory map, so a snapshot that outlived the
process still appears; its SourceVolumeID is empty in that case for the
same reason CreateSnapshot refuses to reuse the ID.
Five conformance subtests cover it, and they run green against
MemoryProvider, qemunbdd and viperblockd over a real predastore.
* feat(ebsprovider): state the exclusion guarantee in the contract
A caller could not tell which providers prevent two writers from holding
one volume, at what scope, or for how long. The guarantee existed in
viperblockd and was invisible from the outside, which is why the
conformance suite had no concurrency dimension at all.
Capabilities.Exclusion states it, and deliberately separates three
questions the contract used to run together: how far exclusion reaches
(none, node, cluster), whether a dead owner's claim ever expires, and
whether a writer that lost its claim is actually stopped. That last one
matters most and is the least flattering: exclusion refuses the second
opener, fencing stops the first. viperblockd advertises cluster scope
with a 45s TTL and FencesLostClaim false, because a partitioned node
whose lease lapsed keeps writing until the object store can refuse it.
Four new conformance subtests. Two required a second viperblockd node in
the harness: cluster exclusion cannot be exercised by one node, because
the second publish goes to a subject nobody serves and times out rather
than being refused. Both nodes share the NATS the leases live in and the
bucket the volumes live in, so a refusal comes from the lease.
Standing that second node up immediately found a real gap, in a test
that had been silently skipping for want of one. A publish refused by
another node's lease came back as internal rather than volume_in_use, so
a caller was told something had broken when the answer was that someone
else held the volume. Mapped in handlePublishVolume.
* build: pin viperblock to the release carrying the provider's dependencies
The branch has been building against a local replace pointing at a
viperblock worktree, so its committed pin never actually built: it named
a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch,
NewStateReader or s3.NewHTTPClient, all of which this branch calls.
Points at the merged viperblock dev instead and drops the replace.
GOWORK=off now builds clean, which is the deploy path ansible uses.
* fix: let viperblockd's delete sweep tolerate an object another sweep removed
deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and
give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it
returned on any DeleteObject error. Two overlapping deletes for one volume left
the loser reporting an internal failure once the winner had removed a chunk
between the list and the delete.
The control-plane copy that carried this fix is gone with the embedded engine
path, so the provider path reintroduced the bug it had just fixed.
* fix: stop CreateVolume leaving local state on whichever node served it
A state read prefers this node's local copy over the backend, and provider RPCs
are not owner-routed, so the node that happened to serve a create kept a copy
that answered describe as if the volume still existed after another node
deleted it. The two-node conformance harness caught it as DeleteVolume/success
failing about one run in three.
The seeded path already removed its local files after closing. The unseeded
path saved state and left them, which was the only difference between the two.
* fix: guard the mount path itself against a second export of one volume
handlePublishVolume refused a volume this node already exported, but
mountVolume did not, and the legacy ebs.<node>.mount subject is the route
production takes for instance launch and hot-attach. A retried attach racing a
fresh one started two real nbdkit against one volume, with no refusal and no
warning: the volume lease only refuses a different node, and a same-node
re-acquire finds the volume already held and increments its refcount.
That also made the advertised exclusion inaccurate. viperblockd claims
cluster-scoped exclusion on the grounds that a second opener is refused, while
the conformance subtests reach it only through PublishVolume, so they passed
with the legacy route unguarded. Moving the check into mountVolume gives both
entry points the same behaviour and makes the claim true for the path
production uses.
* fix: tolerate a missing legacy AMI config on deregister
DeregisterImage deletes the AMI document, then the pre-provider
{imageID}/config.json. Predastore answers a delete of an absent key with
NoSuchKey, and that was treated as fatal, so every AMI created since the
metadata store landed failed to deregister after its document was already
gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had
just been told it could not delete.
The regression test uses a store that answers a missing-key delete the way
Predastore does; MemoryObjectStore silently succeeds, which is why the unit
suite stayed green.
* fix: reap ENIs left behind by an abandoned instance launch
A launch creates the instance's ENI and then attaches it. When the daemon
dies in between, the record survives with no instance and nothing ever looks
at it again. The security group dependency check counts an ENI that lists the
group, so the group cannot be deleted for as long as the record exists.
The in-process launch path already rolls back on error; a process that dies
cannot. This adds a cluster-wide sweep over records that are launch-created,
never attached, and older than the launch RPC timeout by a wide margin. The
description prefix is now a shared constant because it is the only thing
separating an abandoned launch from a standalone ENI a caller asked for.
|
||
|
Josh Siviour
|
4dca2c7b59 |
fix: drain guests on a plain systemctl stop spinifex.target (#766)
* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
|
||
|
Julian Sommer
|
c9e330d8ae |
refactor: consume bluebottle otelsetup, drop the local bootstrap
otelsetup/otelsetup.go and otelsetup/slog.go were a near-copy of the shared bootstrap. Entrypoints now call bluebottle directly, passing WithEnvironmentFallback for SPINIFEX_CI_ENV; the package keeps only the HTTP middleware and request metrics. Init reattaches the OTLP bridge to a default logger installed before it ran, so the local addFanoutHandler is no longer needed. The bridge is now level-gated, so services no longer ship Debug records to the sink while running at Info. |
||
|
Julian Sommer
|
8f930e6275 |
refactor: consume the shared packages from bluebottle
Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather than predastore, so the control plane no longer depends on the S3 server for shared auth and policy code. |
||
|
Ben Duncan
|
c92294e7d5 |
feat: render predastore templates with the cluster and S3 planes split (#763)
* feat: render predastore templates with the cluster and S3 planes split The templates set one bind_addr per host, which put raft and blob wherever S3 was published. A host now binds the cluster plane on the address peers dial and the gate carries its own bind_addr for the S3 API, so the installer can hand S3 the WAN without exposing replication with it. The single-node template also drops to one node of each role. Extra blob nodes only hash-partition keys across more badger indexes on the same drive, and extra meta replicas add fsyncs for a quorum that dies with the machine. * docs: regenerate the AWS model operation coverage table * fix: repin predastore to the merged dev commit |
||
|
brynmailer-mdc
|
1394fbf488 |
docs: describe the predastore host and node model
The port tables were wrong in both directions: inter-node traffic is QUIC over UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are unique within a host rather than across the cluster, so every machine uses the same three, and a single-node install opens only the S3 port because colocated nodes talk over an in-process pipe. The reference architectures scripted a move of node data out of a directory layout that no longer exists, which would have broken any install following them. The update guide now says object storage has no migration path. |