mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
225bf65a8eda1f090e67eba031736ebf82012149
1882 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Duncan
|
225bf65a8e |
feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800)
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut. |
||
|
Josh Siviour
|
2ced9154bd |
fix: count distinct auth attempts, not repeated ones, toward IP lockout (#819)
The lockout fires after 10 auth failures from an address in 60 seconds and then answers RequestLimitExceeded to everything from that address. A client repeating one hopeless request reaches the threshold as fast as an attacker working through a keyspace, and 503 tells it to retry later - advice it can never act on, because the fault is a credential that will never resolve. On wattle four guests holding deleted STS session credentials produced 53,392 rejected requests and 69,820 503s in a day, and could not recover by fixing their credentials because the lockout is on the address, not the principal. RecordFailure now takes a fingerprint identifying the attempt, and the window holds one entry per distinct fingerprint: a repeat refreshes its timestamp instead of adding to the count. Whatever an attacker must vary is what the fingerprint carries - the key id, the presented signature, a digest of the session token - so guessing is throttled exactly as before while one dead credential presented 14,000 times stays one fault. A request rejected before its key id is parsed has no identity to protect, so malformed envelopes and skewed timestamps still count every occurrence. |
||
|
Josh Siviour
|
fc04fb9da7 |
fix: log why IMDS cannot serve an instance role (#818)
An unresolvable instance role answers an empty 200 on the security-credentials list and a 404 on the credential itself. An SDK reads that as "no EC2 IMDS role found" and keeps signing with whatever it already holds, so a guest whose credentials have expired retries forever and never recovers. On wattle four guests did this for five days: 42,517 role-list requests, no credential fetches, and not one line anywhere saying why. The absence was invisible because three layers returned a silent nil - DescribeInstances missing the instance, no instance profile ARN on the record, and a profile IAM no longer has. profileFor now reports which of those it hit, and the two credential paths warn on it, throttled to one line per instance and reason every five minutes so a stuck fleet stays legible at that request volume. A role-name mismatch is the guest's own doing and stays silent. |
||
|
Josh Siviour
|
746f1e1de5 |
fix: record an outcome on daemon NATS request metrics (#817)
96.7% of spinifex-daemon request points carried no outcome. natsMetricsHandler
recorded around a plain nats.MsgHandler, which writes its own reply and returns
nothing, so the wrapper had nothing to observe and passed an empty outcome. Only
the /health HTTP path, which goes through the OTel HTTP middleware, ever
reported one.
The handler type now reports how it answered:
type natsHandler func(*nats.Msg) string
Making it a distinct type means the compiler enumerates every subscription and
every handler, rather than a correlation map guessing at responders that reply
after the wrapper returns. Four outcomes are defined: success, error, skipped
for a broadcast handler that declines because the request names another node,
and a deferred sentinel for handlers that answer from a goroutine and record
their own point once the work finishes — timing the wrapper would have reported
a five-minute system launch as an instant success.
utils.ServeNATSRequest and ServeNATSRequestCtx now return whether they replied
with a result, so handlers built on them report an outcome without restating the
error handling. Adding the return value leaves existing callers untouched.
Also measures the per-reservation launch subject, which was subscribed raw and
emitted no request points at all. It records under the subject root because the
reservation id is unbounded.
|
||
|
Josh Siviour
|
7049e72eb2 |
feat: put the caller and the auth verdict on gateway request telemetry (#816)
awsgw request telemetry said POST / and nothing else: no AWS action, no caller, no reason. On wattle that left 69,820 responses with status 503 in a day unattributable from the request docs — the cause was only findable by leaving them for separate auth-failure log lines, which do carry sourceIP and accessKeyID. The request logger is the outermost middleware, so a context derived downstream never reaches it. A small mutable record now rides the request: the auth path writes identity as it resolves it, writeSigV4Error writes the verdict, and the action enricher writes the action. The logger and the server span both read it, so a trace search for failing requests answers from where, doing what and rejected why without a join. The record holds an access key id and an error code and nothing else that could hold a credential; a test asserts the field set so that stays true. |
||
|
Josh Siviour
|
9b6fc239a2 |
feat: name vpcd and spinifex-ui request actions, split client errors from server errors (#815)
vpcd and spinifex-ui never called otelsetup.SetRequestAction, so rpc_method on their request metrics stayed at the bare HTTP verb the middleware sets. Over 24h vpcd had exactly two action values, GET and PUT, which left IMDS countable in aggregate but made /latest/api/token indistinguishable from a credential fetch at /latest/meta-data/iam/security-credentials/<role>. Guests depend on IMDS for credentials, so a failure there is user-visible and was not attributable. Both namers map to a closed set with a single catch-all, because a guest controls the request path and an unbounded name would put one metric series behind every invented path. The role name, the interface MAC and the public-key index are stripped for the same reason, and spinifex-ui names the static and SPA branches rather than the content-hashed asset filename. The middleware also treated every 4xx as a success, which is why vpcd and spinifex-ui reported zero errors over 24h despite IMDS answering 401 for a missing IMDSv2 token and 404 for an unresolvable ENI. Those now record client_error. error keeps meaning 5xx so existing error-rate panels are unchanged, and a 4xx still does not mark the span failed. |
||
|
tomnewton-mulga
|
0eb868198a | chore: OnLogic doc | ||
|
Josh Siviour
|
57d0a3b813 |
fix: authenticate ECS ECR pulls from the co-located gateway registry (#814)
The ecs-agent classified only *.dkr.ecr.* hosts as ECR, so a registry advertised as the gateway's bare host/IP pulled anonymously and 401'd. Recognise the agent's own gateway host as ECR and mint a token for it. The instance role — the pull's fallback credential when a task carries no execution role — granted ecs:* only; add ECR read so the pull authorizes. |
||
|
Josh Siviour
|
5ed7184cab |
fix: retry instance recovery when the backing store is not yet ready (#813)
* fix: retry recovery relaunch when the backing store is not yet ready * fix: treat a missing EBS mount responder as a retryable recovery failure |
||
|
Josh Siviour
|
fea20b24e0 |
docs: repoint the network invariants doc comment at the proposal path (#812)
* docs: repoint network invariants doc comment at docs/adr/ The region-local ADR set moved from docs/development/proposals/ to docs/adr/ on acceptance, and 0006 kept its filename but the directory changed. * docs: repoint network invariants doc comment at the proposal path The region-local ADR set was never accepted, so it lives under docs/development/proposals/region-local-architecture/ rather than docs/adr/. --------- Co-authored-by: joshsiv-mulga <engineering@mulgadc.com> |
||
|
Josh Siviour
|
61b6810911 |
fix: stop re-driving a public IP release that can never converge (#811)
A release for an IP with no tracked lease was answered with a plain error. The comment at the responder assumed callers log-and-continue, so failing loudly would surface the strand without wedging teardown. That assumption does not hold: TerminatedTeardownReaper marks TeardownNAT failed, TeardownComplete stays false, and the record is re-driven every sweep. No retry can make a deleted lease reappear, so it never converges. It also disables its own backstop. The terminated bucket has a 1h TTL meant to bound the record regardless, but each sweep persists the advanced marks and that write refreshes the TTL, so the record outlives the very thing meant to catch it. Measured across four hosts: seventeen stranded addresses, each retried on a fixed 120s cadence, indefinitely. Keep the loud failure and make it terminal. The release reply gains a machine-readable code, the client maps it to ErrLeaseNotTracked, and ReleasePublicIP treats that as done after recording the leak, since the local pool slot is already free and the upstream lease cannot be released without a lease record. Teardown then completes and the record purges. The strand is no longer silent: mulga.resource.leaked counts it by resource class and the address is logged at error level, so a leaked address is queryable rather than inferred from a repeating warning. |
||
|
Julian Sommer
|
95911cbfae | feat: DescribeDBEngineVersions and DescribeOrderableDBInstanceOptions (#807) | ||
|
Julian Sommer
|
dd398b3a93 | test: run the rds suite in parallel (#806) | ||
|
Julian Sommer
|
333248568e | fix: use actions/cache@v6 + test-actions clean | ||
|
Julian Sommer
|
ae0337d31b |
test: run the unit and race tiers clean in CI
Go stores test results in the build cache alongside compiled objects, so now that the cache persists across runs an unchanged package replays its previous result instead of executing. That is what makes a local run fast and exactly what a merge gate should not do: a green check would no longer mean the suite ran against that commit. Add a GOTESTFLAGS hook, empty by default so local runs keep reusing results, and pass -count=1 from the two CI steps that gate a PR. Compilation is still cached either way, so this costs no build time. |
||
|
Julian Sommer
|
20af6e5a4a |
fix: stop the Go build cache freezing at the last go.sum change
setup-go only writes its cache on a key miss, and keyed it on spinifex/go.sum alone. Once a run saved that key every later run logged "Cache hit ... not saving cache", so the cache stayed frozen at whatever the first job to finish happened to have compiled and nothing built afterwards was ever stored. All four jobs also shared that one key, so the job that won the save race decided what the other three restored. The race job was handed artifacts built without -race, and the build job artifacts built without GOFIPS140, neither of which yields a single usable entry. Manage both caches directly instead: modules keyed on all five go.sum files so the sibling modules stop re-downloading, and build artifacts keyed per job with a per-run suffix so each flavour keeps its own entry and refreshes it every run. |
||
|
Julian Sommer
|
20a540cb59 | chore: update manifest lint | ||
|
Julian Sommer
|
cba51b59e3 |
fix: stop GOFIPS140 splitting the Go build cache in two
GOFIPS140 is a global build-config change: it alters every package's action ID, stdlib included. A target that omits it therefore shares no build-cache entry with one that sets it, and the two flavours cost roughly a full cold build to switch between. Seven targets set it inline while eight go invocations did not, so test-integration, test-harness, test-actions, test-segscan-oracle, bench and the go run tools all built the non-FIPS flavour. In CI this is load-bearing: the integration job wins the shared setup-go cache-save race, stores a non-FIPS cache, and every FIPS build in the other jobs then misses on every lookup. Export it once so the two cannot drift apart again. |
||
|
Julian Sommer
|
df659177e0 | test: stop test suites waiting out timeouts they never needed (#805) | ||
|
Julian Sommer
|
67b37176a4 | fix: stop embedded NATS test servers sleeping 25ms on startup (#804) | ||
|
Julian Sommer
|
c99b91bca5 | fix: add publish system ami job to release | ||
|
Julian Sommer
|
8bf386617b |
fix: pin the spinifex-ui start test to its own config
The case seeded no config, so loadLocalConfig fell back to the host's installed spinifex.toml. It passed on a provisioned dev box and failed on a bare runner, where the region pre-flight fires before svc.Start(). |
||
|
Julian Sommer
|
4716ea306f | chore: bump insomniacslk/dhcp to fix panic | ||
|
Julian Sommer
|
eaedcb9c06 | chore: rebuild ui | ||
|
dependabot[bot]
|
9bc2b227c2 | chore: bump the npm-dependencies group (#802) | ||
|
dependabot[bot]
|
8fe299ba40 | chore: bump the go-dependencies group with 8 updates (#801) | ||
|
Julian Sommer
|
eac0f175ba | feat: require TLS on RDS connections by default, on both engines (#790) | ||
|
Ben Duncan
|
42d87d7a5a |
feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799)
* feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down. |
||
|
Josh Siviour
|
f647b635b2 |
feat: pull Ochre model weights from a hub into predastore (#798)
* feat: pull Ochre model weights from a hub into predastore Add `spx admin ochre weights pull`: fetch a self-host model's weights from Hugging Face directly into predastore, producing a prefix `weights stage` consumes. Streams safetensors-only artefacts (plus config/tokenizer and the shard index), refusing pickle/.bin. Pins an immutable commit SHA, writes an ochre-pull.json manifest that stage records as SourceRevision, and aborts clean (removing any written object) on a gated or mid-stream failure. Token resolves from --hf-token, HF_TOKEN, or a stored platform credential; never logged. Also add `spx admin ochre credentials set` to store an encrypted provider credential from stdin, and SourceRevision on the weights KV record + list. * fix: stream hub downloads without a whole-request timeout http.Client.Timeout bounds the entire request including the body read, so a multi-GB safetensors shard was guillotined mid-download at 30s. Bound dial and response-header latency via the transport instead, leaving the body free to stream for as long as the server keeps sending. |
||
|
Josh Siviour
|
23452ba71a |
fix: fail closed when guardrail OUTPUT extraction fails (#797)
The InvokeModel OUTPUT guardrail hook logged and forwarded the model response unguarded when the per-family completion extractor could not parse the body, so output enforcement failed open while the INPUT path already failed closed. Both the non-stream OUTPUT hook and the buffered InvokeModelWithResponseStream source now fail closed: on an unparseable completion (non-stream) or an undecodable assistant chunk (stream) they load the guardrail view and emit a blocked response carrying BlockedOutputsMessaging. A genuinely empty completion (valid chunks, no assistant text) still passes through. Shared resolve/load prologue extracted into loadGuardrailView. |
||
|
Josh Siviour
|
61a51eb34b |
fix: dispatch self-host invoke by model family and keep not-ready codes (#796)
Self-host InvokeModel/InvokeModelWithResponseStream picked the adapter by tier and always used the Llama adapter, silently mis-serving a non-Meta model through Llama's native request schema. Dispatch now selects by catalog Family; an unhandled self-host family is refused with ValidationException, not mis-served. Endpoint-resolution failures hardcoded ServiceUnavailableException, discarding the AWS code the daemon put on the wire. All four self-host resolve sites now resolve the carried code (ModelNotReadyException, ThrottlingException, ...) and fall back to ServiceUnavailable only for an un-coded transport failure. |
||
|
Josh Siviour
|
bc8ad05a0f |
feat: Ochre Phase 5 knowledge bases (bedrock-agent KB/data-source + RAG) (#792)
* feat: ochre-vector list-jobs subject Adds ochre.vector.listJobs to the daemon-side VectorService surface (both the in-process and NATS-forwarding implementations) so a caller can list every ingestion job for an account rather than looking one up by id. Widens jobGetter to expose JobStore.List, registers the subject on the daemon's post-connect subscription table, and updates the CLI test double to satisfy the widened VectorService interface. Also carries the jackc/puddle/v2 indirect dependency go.sum was missing. * feat: bedrock-agent knowledge-base + data-source CRUD Adds the emulated AWS bedrock-agent control-plane gateway service: KnowledgeBase/DataSource CRUD, StartIngestionJob/GetIngestionJob/ ListIngestionJobs, backed by new gateway-owned KBStore/DataSourceStore JetStream KV persistence (mirroring registry.go's CAS idiom) and the existing ochre-vector daemon surface for the bound vector index. One knowledge base binds exactly one vector index. embeddingModelArn maps to EmbeddingModel/Dimension; storageConfiguration and roleArn are accepted and stored verbatim but never acted on. CreateDataSource maps S3 + fixed-size chunking config onto a SourceSpec, converting AWS's maxTokens/overlapPercentage into internal ChunkSize/ChunkOverlap. DeleteKnowledgeBase cascades to every data source under it before dropping the bound index. Wires BedrockAgentKB/BedrockAgentDataSources/ BedrockAgentVector into the gateway and registers bedrock-agent in the supported-services and dispatch tables. Runtime (Retrieve/RetrieveAndGenerate) is out of scope here. * test: cover bedrock-agent routing and KB/DataSource store edges Adds end-to-end HTTP tests for BedrockAgent_Request's route table and per-route handler closures (all eleven operations, plus unknown-route/ missing-account/nil-dependency error paths), KBStore/DataSourceStore PurgeAll and no-JetStream-client error paths, and ListJobs coverage (vectorService, NATS round trip, per-account scoping) in the existing ochre-vector API test suite. * feat: bedrock-agent-runtime retrieve + retrieve-and-generate Adds the emulated bedrock-agent-runtime data plane: Retrieve resolves a knowledge base to its bound vector index and runs a similarity query; RetrieveAndGenerate runs the same retrieval, renders a RAG prompt (default or caller-supplied template), and calls Converse in-process for generation, returning citations built from the retrieved chunks and a session id. Recovers the full RetrievalFilter operator set from the raw request body since the vendored aws-sdk-go RetrievalFilter type only carries andAll/orAll, and fixes GetService never routing real signed bedrock-agent traffic (only bedrock-runtime and bedrock had discriminator rules; bedrock-agent's own knowledgebases paths had none). * fix: bedrock-agent KB delete idempotency + ingestion-job ownership CreateKnowledgeBase: a rollback DeleteIndex failure after a kb.Create claim failure is now logged at Error level with the orphaned index id instead of a bare Warn, while the caller still sees kb.Create's own error unchanged. DeleteKnowledgeBase: a bound index that DeleteIndex reports already gone (ErrIndexNotFound) no longer aborts before the KB record itself is removed, so a KB never dangles pointing at an index that no longer exists. Any other DeleteIndex error still aborts the cascade. GetIngestionJob: now verifies the addressed dataSourceId owns the job (Bucket+Prefix match against the data source's own SourceSpec, same best-effort check ListIngestionJobs already used), not just the knowledge base's bound index -- closing a path where a job could be read through the wrong data source id. * fix: default bedrock-agent-runtime retrieve to 5 results (AWS parity) * fix: thread DataSourceID through ingestion jobs JobRecord and IngestRequest now carry an exact DataSourceID instead of relying on Bucket+Prefix as a best-effort proxy. StartIngestionJob stamps it from the DataSource record; GetIngestionJob and ListIngestionJobs filter on it exactly, treating a job with no DataSourceID as belonging to no bedrock-agent data source. * fix: purge KB/DataSource records on appliance teardown Appliance.Teardown purged the index registry and job store but left the gateway-owned KB/DataSource KV records dangling, pointing at a deleted index. WithKBStores attaches those stores the same way WithStores does for registry/jobs, built before Connect so teardown still works on a broken appliance, and Teardown purges them too. |
||
|
Josh Siviour
|
c6bc2b4d29 |
fix: only restart NATS when JetStream has actually failed (#795)
The watchdog's is-active guard cannot fire against spinifex-nats.service (Type=simple reports active the instant the process forks, long before JetStream replays its state), so it restarted a NATS that was still recovering. Separately, any non-zero exit from spx js-probe was read as a JetStream write failure, so a broken spx binary read as a cluster-wide outage instead of an unrunnable tool. Replaces the is-active guard with an age gate on ExecMainStartTimestampMonotonic, adds a post-restart cooldown and a restart-count escalation that gives up rather than looping, and teaches the watchdog to treat js-probe exit 2 (and any exit code that isn't 0 or 1) as inconclusive rather than as proof of failure. Bounds the unit's own start with TimeoutStartSec=120, matching the existing TimeoutStopSec derivation. |
||
|
Josh Siviour
|
ab1046856b |
fix(health): probe the local gate when predastore is single-host (#794)
The predastore health probe QUIC-dials the raft meta node, but a single-host install binds no cluster-plane socket — colocated nodes talk over an in-process pipe. The daemon runs in a separate process from predastore, so it can never reach that pipe and the dial is always refused, reporting a permanent false unreachable that gates ansible deploy readiness. On a single-host install, probe predastore's one cross-process listener — the S3 gate — instead. A gate that accepts a connection is a predastore process serving, and on a single voter raft is trivially leader. Multi-host installs keep the raft QUIC probe unchanged. |
||
|
Josh Siviour
|
9102588e5a | chore: bump Go toolchain to 1.26.6 (#793) | ||
|
Josh Siviour
|
ab48ead095 |
fix: exit non-zero when a spinifex service fails to start (#791)
* fix: exit non-zero when a spinifex service fails to start Every service start/stop subcommand in cmd/spinifex/cmd/service.go used Run funcs that printed an error and returned, which exits 0. Systemd's Restart=on-failure never fires on a zero exit, so a fatal startup error (seen live: viperblockd's volume lease bucket lookup timing out during a cold JetStream start) left the node permanently down until a manual systemctl restart. Converts every fatal error path across predastore, viperblock, qemunbd, nats, spinifex, awsgw, spinifex-ui, vpcd, northstar and qmp-collector to RunE returning an error, following the RunE + SilenceErrors/SilenceUsage pattern the northstar start command already used in this file. root.go's Execute() already prints the returned error to stderr and calls os.Exit(1), so this centralizes the exit-code decision instead of repeating fmt.Fprintln+os.Exit at each call site. Pure status subcommands with no fallible operations are left as Run. Adds a subprocess re-exec test proving a fatal predastore start now exits 1 end to end, plus in-process RunE tests covering each service's start command. * test: cover the service startup validation branches |
||
|
joshsiv-mulga
|
1e00425bb3 |
fix: add missing puddle/v2 dependency for pgxpool
The Ochre vector store change imported github.com/jackc/pgx/v5/pgxpool without running go mod tidy, so github.com/jackc/puddle/v2 was absent from go.mod and go.sum. Any build under -mod=readonly failed, breaking the multi-node E2E job at 'go test -c'. Also promotes pgx/v5 from the indirect block, since it is imported directly. |
||
|
Josh Siviour
|
d52245001e |
feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788)
* feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate |
||
|
Josh Siviour
|
560b437f3b |
fix(health): probe predastore raft state instead of echoing the config (#786)
/health reported service_health.predastore == "ok" for every node simply because predastore was named in spinifex.toml, without contacting it — on env19 all three nodes stayed green for the whole time raft had no leader and volume creates were failing. A probe registry now backs /health's predastore entry with a real raft status dial (ok/no_leader/unreachable, cached briefly and bounded by a short timeout), while every other service keeps its unprobed "ok" default unchanged. spx admin storage status had the same root problem from the other side: it dialed HTTPS /health and /status endpoints predastore's meta nodes never served (they speak an opcode-multiplexed RPC), so every meta node always reported unhealthy with blank raft fields. It now dials the same raft status RPC predastore#82 exposes, verified against the cluster CA, with the JSON output shape unchanged. |
||
|
Josh Siviour
|
f7a2d85691 |
fix(e2e): stop flagging a Cluster bind to a single-homed node's own address (#789)
checkSocketScope compared a socket's bind address to node.Addr and called it the node's WAN address, but node.Addr is the cluster peer address (harness.Node's own doc). On CI's single-homed tofu nodes this is the node's only address, so every Cluster-scope service failed no matter what it bound. Compare against a positively-known SPINIFEX_WAN_IP instead, distinct from the peer address, and record via harness.Detail when no such address is available so the skipped check is visible rather than silently passing. |
||
|
Josh Siviour
|
b93f6c258a |
fix(e2e): resolve the listener inventory without a Go toolchain on the target node (#787)
* fix(e2e): resolve the listener inventory without a Go toolchain on the target node tests/e2e/multinode/listener_invariant_test.go resolved docs/security/network-connections/README.md at runtime via `go env GOMOD`, which works under `go test` but not inside a compiled .test binary shipped to a node with no Go toolchain. Embed the doc in a new docs/security/network-connections package instead, so the content is compiled into the binary and every caller parses bytes rather than a resolved path. RepoRoot and DocPath are now unused and removed. * docs: trim the embed package comment to the three-line standard |
||
|
Josh Siviour
|
7d99705eff |
fix(e2e): kill the node-failure phase abruptly instead of draining it (#785)
* fix(e2e): kill the node-failure phase abruptly instead of draining it Phase 8 used `systemctl stop spinifex.target`, which now drains guests gracefully since the target-stop drain fix landed, so it tested an orderly shutdown rather than a failure. Since a target unit has no process of its own and cannot be signalled, and PartOf= only propagates stop/restart jobs rather than kill, SIGKILL each core service's main process directly instead, leaving orphaned guest qemu/nbdkit processes for recovery to reattach. Also fix a stale comment on spinifex-predastore.service's TimeoutStopSec: it claimed to match viperblock's drain budget, which is now 150s rather than 30s. The value was already correct; only the rationale was outdated. * fix(e2e): hold node2's outage open through Phase 8's assertions Restart=on-failure was respawning each killed unit within ~5s, so every Phase 8 assertion ran against an already-recovered node2 and DEGRADED_PEERS never actually dropped. Stop the same units right after the kill to cancel the queued restart and keep them down for the rest of the phase, and make the NATS degraded-peer count a real assertion (sized off NODE_COUNT) now that it can genuinely fail. |
||
|
Josh Siviour
|
57bfbc20df |
test: listener invariant static guard + e2e runtime check (#782)
* test: add the listener invariant static guard and e2e runtime check Adds both halves the plan called for: a static test in spinifex/network/invariants that scans install scripts and config templates for a bare OVSDB ptcp remote, a wildcard bind on a Cluster/Encap port, or a bind port absent from the inventory, and an e2e test in tests/e2e/multinode that asserts the same on live ss -tulnp output. Both read docs/security/network-connections/README.md "## 1. Inbound Listeners" as their fixture through a shared, non-test parser package (spinifex/network/listenerinventory), so a wildcard exception has to be declared in the doc's own prose rather than hardcoded in either test. * fix: make the listener wildcard exception fail closed on an explicit marker The prior heuristic granted an exception for any unnegated "wildcard" or "0.0.0.0" mention within 40 characters of the port's row. That fails open: a future row phrased as "avoids the wildcard" or with a negation further upstream than the window would silently exempt itself. Replace it with a single exact literal phrase, "binds the wildcard by design", that a row must contain to grant the exception — no other wording counts, so an unrecognized mention denies rather than grants. Update the two rows that need it (5300, 500/4500) and document the marker's role in the inventory's own prose. * docs: record that the Geneve tunnel socket binds the wildcard by design |
||
|
Josh Siviour
|
eb53dc3396 |
fix: reap orphaned nbdkit processes recovery cannot adopt (#783)
* fix: reap orphaned nbdkit processes recovery cannot adopt recoverMountedVolumes already rebuilds MountedVolumes from surviving nbdkit processes, but an unadoptable backend (buildVB fails, most commonly a deleted volume) or a duplicate corroborated process for an already-adopted volume was logged and left running forever, with no later pass reconsidering it. Reap both classes, but only when a scan of every other process's /proc/<pid>/cmdline on the host finds no reference to the backend's NBD endpoint -- its socket path, or its 127.0.0.1:<port> pair, checked both joined and split across server.host=/server.port=. A referencing process, or a scan that could not complete, both leave the process running and log loudly instead, since signalling the wrong process destroys a running guest's data path. * fix: treat an unreadable /proc candidate as inconclusive, not absent endpointReferencer skipped any os.ReadFile failure on a candidate's cmdline, which silently counted a read it could not perform as evidence the candidate does not reference the endpoint. Under a /proc mounted with hidepid, a guest QEMU owned by another uid is invisible this way, so an in-use backend could be reaped. Only os.IsNotExist (the candidate exited between ReadDir and the read) continues the scan. Any other error now returns non-nil from endpointReferencer, which reapOrphanedNbdkit already treats as an inconclusive scan: leave the process running, log at WARN. * fix: stop orphan nbdkit reaping from killing live backends under hidepid=invisible hidepid=invisible hides another uid's /proc/<pid> directory entry outright, not just its contents, so a live QEMU referencer (owned by spinifex-daemon) never appeared in viperblockd's scan at all -- a clean "no referencer" result that was actually just a hidden one. reapOrphanedNbdkit now detects the hazardous mount via /proc/self/mountinfo and declines to reap rather than trust a scan that could be blind to a live guest. spinifex-viperblock.service drops ProtectProc=invisible (now default, same tradeoff already accepted by spinifex-qmp-collector.service) so the scan can actually see spinifex-daemon's processes again once it's safe to trust. |
||
|
Julian Sommer
|
29d4a6221e | fix: add nil check | ||
|
Josh Siviour
|
ee1ee4f033 |
fix(preflight): report services still executing a replaced binary (#781)
Overwriting /usr/local/bin/spx while spinifex.target is running does not move the services onto it. Each keeps executing the replaced inode until it restarts, so a node serves a mixture of two builds, and because the handlers are NATS queue-group workers the skew shows up as roughly one request in three behaving like the old build — an intermittent fault rather than a broken node. The ansible deploy path already detects and remediates this, but only there: an out-of-band binary copy leaves nothing to report it, and on env19 that state went unnoticed for a full day. spx admin preflight now reports it, so an operator has a command to run and a script has a non-zero exit to gate on. Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a target path that no longer resolves, or a path that resolves to a different inode than the one being executed. |
||
|
Josh Siviour
|
2fe885441b |
fix(shutdown): scope a local drain to its own node and stop counting failed phases as done (#780)
* fix(shutdown): scope a local drain to its own node and stop counting failed phases as done A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes to as a fan-out. The node filter only decided which ACK the stopping node waited for, so by the time it applied, every other node had already stopped its gateway, UI and VPC daemon and powered down its guests. Stopping one node took down every guest in the cluster. Requests now carry the node they apply to, and a daemon drops one addressed elsewhere without ACKing. A cluster shutdown leaves the field empty, so it still reaches every node. The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on the host: another node's backends on a shared host, and a live guest's backend that survived a daemon restart under KillMode=process. Cleanup now sweeps this node's own nbdkit-vol-*.pid files and signals a PID only while /proc still agrees it is nbdkit, so a recycled PID is never hit. Finally, the coordinator counted errored ACKs toward its per-phase quorum, so a node that failed to drain reported N/N and the run advanced into STORAGE and PERSIST with guests still running. Only clean ACKs count now, and any errored one aborts the phase unless --force. * test(shutdown): cover the phase gate, every handler's target check and the real proc paths Diff coverage came in at 52.9% against a 60% threshold. The gaps were the per-phase target checks beyond GATE, the real procComm and signalProcess bodies that every cleanup test stubs out, and the coordinator's abort branch, which sat inside runClusterShutdown where no unit test could reach it. The abort decision moves into shutdownPhaseOutcome, a pure function returning the completed count and the message to print, so the rule that an errored node does not count toward the quorum is testable on its own rather than only reachable through a full cluster shutdown. |
||
|
Julian Sommer
|
6b71fb876d | fix: use stdlib sort | ||
|
Julian Sommer
|
3abf5425ae | refactor: fold GPU compute and MIG flags into modelInfo | ||
|
Julian Sommer
|
1fb48b589e | refactor: call utils.NATSRequest directly from the ACM gateway handlers |