mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
2ba8f5e4abc8a4ebb7cb274105eaa494efc78e55
104 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
9bac0e2b45 | chore: update COMMANDS.md | ||
|
Ben Duncan
|
455c97a2e7 |
fix: prod review of 14 September — IPsec follower reads, stopped-instance EIP probes, audit client IP and JetStream formation divergence (#1029)
* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly * fix: stop probing and nudging the EIP datapath of a stopped instance's port * fix: log the real client IP, request ID and duration_ms in the awsgw access line * test: mark the UI proxy header test as in-package * fix: discard pre-formation JetStream data when forming a multi-node cluster A node that ran as its own single-node cluster carries KV streams into formation. nats-server adopts a local stream with the same name as the clustered one, and raft catch-up only fetches sequences beyond a follower's last, so the conflicting prefix is never repaired and replicas diverge permanently. - spx admin join removes the local JetStream store once formation succeeds. A store holding streams now needs --force, and the identity-discard warning names it. - spx admin init --nodes >= 2 refuses a store that holds streams unless --discard-jetstream is given, and removes it once every node has joined. Keeping it as the seed is unsafe: stream leadership ignores who holds data, and unassigned local streams are deleted as orphans. - Neither removes a store while `spx service nats` runs, --force included. - install-node.sh passes --discard-jetstream and waits for every `spx service` process to exit before formation. * feat: add spx admin kv digest for per-replica JetStream consistency checks Replicas of a stream can hold different content while raft reports them current, and nothing in the cluster detects it. A direct get over NATS is no way to inspect one replica: a client's request goes to a random member of the direct-get queue group, routed replicas included. kv digest copies this node's $G streams tree (without consumer state) to a temporary directory, opens the copy in a private in-process server with no listener, and prints first/last sequence, message count and a SHA-256 over every stored message per stream. Running it on each node and diffing the output shows which streams diverge; --seqs locates the first differing sequence. The live store is only ever read. * fix: let spx admin kv digest read replicated streams A replicated stream's meta.inf carries num_replicas > 1, which a standalone server refuses to recover, so the digest could not read any production store. Rewrite the copy's config to one replica with no expiry and a recomputed checksum, and test it against a three-node cluster and an expired bucket. * test: use assert.ErrorIs for the removed JetStream store * test: mark the JetStream store and digest tests as in-package * fix: stop the process-group test misreading a reaped child as alive The test polled until the child was gone, then checked once more. A task being reaped shows state X for an instant, which processRunning counted as running, so that second look failed about 1 run in 50 under -race. * fix: wait for the recovery retry loop before removing a test's data dir restoreInstances starts the retry loop untracked, and it rewrites the local state file when it ends. A write landing during t.TempDir cleanup failed TestRestoreInstances_* with RemoveAll: directory not empty. |
||
|
Julian Sommer
|
19a047bccb | fix: footnotes instead of notes for compatibility | ||
|
Julian Sommer
|
e4f0050a1a | fix: update COMMANDS.md with systemctl commands | ||
|
Julian Sommer
|
8d8f020ed3 | fix: update COMMANDS.md | ||
|
Josh Siviour
|
b482b41e96 |
feat: implement EC2 MonitorInstances and UnmonitorInstances (#853)
* feat: implement EC2 MonitorInstances and UnmonitorInstances The telemetry collector already supports both EC2 monitoring tiers and already switches a running poller between them when its discovery file changes, but nothing could make that change: Monitoring.Enabled was only ever written at launch, and both API actions returned InvalidAction. Route through the owning daemon over ec2.cmd.<id> rather than the spinifex-workers queue group, so an instance running on any node is reachable; no responders falls back to the shared stopped store, where the write is a CAS so a racing start-claim cannot resurrect a deleted record. Also report Monitoring on DescribeInstances. It is derived in ProjectInstance from the same field the collector's period reads, so a describe cannot disagree with the tier actually being collected and instances launched before this change report disabled without a backfill. * fix: wake the telemetry poller when its monitoring tier changes poller.run re-read its period inside the tick loop, so a reset waited out the interval being replaced: moving a 300s instance to the detailed tier took up to five minutes to have any effect, which reads as the API call not working. Found verifying monitor-instances on env13. updateMeta now signals a buffered retier channel when PeriodSeconds moves and run selects on it alongside the ticker. The signal only resets the ticker rather than forcing a tick, so a published delta still spans the period its batch declares. The send is non-blocking and the channel coalesces, so a slow poller cannot stall the reconcile loop. |
||
|
Josh Siviour
|
f9c84919ef |
feat: co-served GPU model bundles (vLLM + TEI) with family-aware reaper (#845)
* feat: co-served GPU model bundles (catalog, lifecycle, embed/rerank routing) Serve the generative LLM, embedder and reranker as one co-resident GPU bundle: EMBEDDING/RERANK catalog modalities + co-serve group, bundle-aware admission (one GPU claim sized to summed member VRAM), single-flight one-VM launch with a member->port EndpointRecord, DynamicEndpointResolver member routing. Ochrevector embed/rerank/token-introspection resolve through the resolver (static single-endpoint map retired); rerank degrades to plain KNN on absent/failure. Public InvokeModel-embeddings and top-level Rerank deferred. * feat: co-serve vLLM+TEI bundle image with per-engine readiness Replace the single-model vllm-serve unit with a generic bedrock-serve@ systemd template driven by per-member BEDROCK_* env files, dispatching to vLLM or text-embeddings-router by engine and binding each member to its own port and weights device. TEI serves directly on its serve port; readiness probes each member on the route its engine answers (vLLM /v1/models, TEI /health) with the engine family carried end-to-end from the catalog. * fix: serve nomic embedder correctly in the co-serve bundle Bound embeddings requests to the TEI client batch limit so a large document's chunks are split across requests instead of 422ing. Serve nomic with explicit mean pooling, since the flat weights layout omits 1_Pooling/config.json and TEI would otherwise default to CLS. Add bge-base-en-v1.5 as a standalone alternative embedder. * fix: make the bedrock reaper family-aware and pin-safe A pinned endpoint is now exempt from terminate-and-relaunch on scrape failure, mirroring how it is already exempt from idle reclaim, so a transient unreachability can no longer tear down operator- or commitment-managed serving. A bundle with no generative vLLM member exposes no vLLM load metrics, so it is liveness-probed on /health and kept warm rather than scraped for idleness on a route it does not serve and reaped in a loop. * test: lock in co-serve group-member convergence to one VM * fix: pin self-host HF repo/revision in catalog and stop nested config flattening Carry HFRepo/HFRevision per self-host catalog entry; weights pull defaults --hf-repo and --revision from them so a bare pull cannot silently grab a regressed upstream main (the nomic v5 config.json duplicate-field break). Restrict fixed config filenames to the repo root so a nested config.json (sentence-transformers 1_Pooling/) is dropped instead of flattening onto the model config, and refuse any two source paths that collide at one dest key. * fix: let endpoint delete clear an account-scoped pinned record and report honestly A bare 'endpoint delete' resolved the Global key, which never held a pinned, account-scoped record, then printed 'torn down' over the idempotent no-op while the record survived. Add --account to the delete command so an operator can target the account 'endpoint list' shows, and return DeleteEndpointOutput.Removed so the CLI names a real teardown versus a lookup that found nothing and points at --account. A gone VM already tolerates NotFound, so a scoped delete clears a record whose instance was terminated out of band. |
||
|
Ben Duncan
|
ae2a1830ef | Merge remote-tracking branch 'origin/dev' into account-quota | ||
|
Ben Duncan
|
0f8471062f | feat: add an admin surface for per-account quota overrides | ||
|
Julian Sommer
|
bdc79108e9 | fix: make the DB endpoint the only path to the engine (#830) | ||
|
Ben Duncan
|
225bf65a8e |
feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800)
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut. |
||
|
Julian Sommer
|
95911cbfae | feat: DescribeDBEngineVersions and DescribeOrderableDBInstanceOptions (#807) | ||
|
Julian Sommer
|
eac0f175ba | feat: require TLS on RDS connections by default, on both engines (#790) | ||
|
Ben Duncan
|
42d87d7a5a |
feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799)
* feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down. |
||
|
Josh Siviour
|
58e0f2879f |
feat: serve the deployment CA from IMDS instead of baking it into images (#784)
Image import shelled out to virt-customize to install the deployment CA into an imported image's trust store. The step was best-effort and skipped whenever virt-customize was absent, which is the default: nothing in provisioning installs guestfs-tools on a node. When it did run it mutated the image after its checksum had been verified, it could not handle Windows guests, and it tied CA rotation to re-importing every affected AMI. Nothing the stack launches depended on it. ECS container instances, EKS k3s servers and BootDirect microVMs all receive the gateway CA at launch via user-data or fw_cfg, and boot itself needs no CA because IMDS is plain HTTP. The only uncovered case is a customer instance whose own workload calls a Spinifex HTTPS endpoint, which needs a pull mechanism rather than an image mutation. The IMDS responder now serves the CA at /spinifex/ca.pem, outside the /latest tree so the AWS-compatible surface stays byte-identical to EC2's. The route is token-free because a CA certificate is public material and the console already serves it unauthenticated; the X-Forwarded-For SSRF guard still applies. The PEM is cached and re-read on modtime change, so rotating ca.pem on a node reaches guests without restarting vpcd. |
||
|
Julian Sommer
|
65d2501e6b | feat: add MariaDB as the second RDS engine (#762) | ||
|
Josh Siviour
|
6f4aa495e6 |
feat: reconcile systemd unit files on upgrade (#771)
* feat: reconcile systemd unit files on upgrade Unit files were written once at install time by install_systemd() in scripts/setup.sh and never re-asserted. A node upgraded by swapping the spx binary kept whatever units it was installed with, so any fix that lives in a unit rather than in code never reached it. This is not theoretical. A node whose spinifex-viperblock.service predates KillMode=process defaults to KillMode=control-group, so on a target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs every nbdkit while guest QEMU survives in the daemon's own cgroup: the guest keeps running with no storage path. Units are now stamped with a spinifex-unit-version marker, embedded in the binary via a generated file, and reconciled by spx admin upgrade. Missing or version-stale units are replaced after a timestamped backup; a unit at the current version whose content differs is reported as operator-modified and never overwritten. Drop-ins under <unit>.d are untouched, so systemctl edit overrides survive. Nothing is restarted. A daemon-reload is enough for KillMode and TimeoutStopSec to apply to the next stop, so drift is corrected without disturbing a running guest. Adds --dry-run, --units-only and --skip-units to spx admin upgrade. * test: cover systemd reconcile error paths and the admin upgrade CLI flow Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate. Adds tests for reconcile.go's error/edge paths (unreadable unit, backup and write failures, daemon-reload failure, malformed version markers, downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly via an injectable exit and a decision matrix extracted into validateUpgradeFlags/upgradeSummary for testability. |
||
|
brynmailer-mdc
|
1394fbf488 |
docs: describe the predastore host and node model
The port tables were wrong in both directions: inter-node traffic is QUIC over UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are unique within a host rather than across the cluster, so every machine uses the same three, and a single-node install opens only the S3 port because colocated nodes talk over an in-process pipe. The reference architectures scripted a move of node data out of a directory layout that no longer exists, which would have broken any install following them. The update guide now says object storage has no migration path. |
||
|
brynmailer-mdc
|
a7a62addeb |
test: rebuild the predastore fixture and service tests on the facade
The fixture wrote a config the strict decoder rejects and built its cluster through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and serves under Run. Loading the real file means the fixture trips over the same validation an install does. Its data directory holds node-<id> directly, and it drains on cancellation rather than being abandoned at exit. The runtime tests drove buildRuntime, serve and the old on-disk layout, all of which predastore owns now, so they go. What replaces them covers what spinifex still decides: the flag merge and its precedence, a host id or gate node the config does not have, and the pid file behind Stop and Status. Two fixes to earlier commits in this series: the host id conversion moves behind its own guard, and the node status probes use WaitGroup.Go. |
||
|
Josh Siviour
|
9b2334b964 |
fix(ochre): self-heal weights stage when the recorded snapshot is gone (#753)
* fix(ochre): self-heal weights stage when the recorded snapshot is gone runStageWeights's idempotency check trusted a matching source URI without verifying the snapshot it names still exists, so a host rebuild or volume GC that lost the snapshot left stage permanently short-circuited on a useless KV record until an operator ran 'weights remove' first. Inject a weightsSnapshotChecker, verify liveness before the no-op return, and fall through to re-materialise when the snapshot is gone. 'weights list' uses the same checker to mark dangling entries so an operator can see which staged models can actually serve without waiting for endpoint launch to fail. * docs: note weights stage self-heal and list STATUS column |
||
|
Josh Siviour
|
272b4b22e8 |
feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740)
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
|
||
|
Julian Sommer
|
1c4d41bffc | fix: reach the DB endpoint from any subnet of the VPC (#738) | ||
|
Julian Sommer
|
456a60717b | feat: add RDS to awsmodel coverage | ||
|
Julian Sommer
|
8c07f80032 | chore: remove the half line breaks | ||
|
Julian Sommer
|
f7d5d97e3f | fix: propagate RDS parameter group edits (#733) | ||
|
Julian Sommer
|
871513dc6c |
Reapply "feat: match new predastore config surface area (#591)"
This reverts commit
|
||
|
Julian Sommer
|
b4701c78de |
Revert "feat: match new predastore config surface area (#591)"
This reverts commit
|
||
|
Josh Siviour
|
43975b01e6 |
feat: add spx admin ochre endpoint ensure/describe/list/delete (#691)
* feat: add spx admin ochre endpoint ensure/describe/list/delete The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in production calls their NATS client, so the serving-endpoint lifecycle could not be exercised at all in a running system. This gives an operator a direct way to drive it. ensure --wait polls describe until READY and reports the elapsed time, since the daemon replies STARTING as soon as it claims the model and the launch outlives the request. That is the measured cold start. Also corrects the weights stage tokenizer rule in COMMANDS.md, which still described both tokenizer files as required rather than either. * test: cover the ochre endpoint Run wrappers' connect and exit paths * fix: treat an empty endpoint bucket as no endpoints, not an error ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound, so 'ochre endpoint list' failed on a cluster that had never launched one. |
||
|
Julian Sommer
|
5a3f5e60b2 | fix: cleanup COMMANDS.md | ||
|
Julian Sommer
|
faa0519cf1 | fix: the RDS master user is no longer a PostgreSQL superuser (#687) | ||
|
Josh Siviour
|
f3dd3c251e |
feat: add spx admin ochre weights stage/list/remove (#680)
* feat: add spx admin ochre weights stage/list/remove Give operators a supported path to stage a self-hosted model's weights: point at a predastore S3 prefix holding ordinary Hugging Face files, and stage materialises them into a viperblock volume, snapshots it, and records the source URI and snapshot ID in the bedrock-weights KV bucket. stage refuses before downloading anything if the model is not a self-host catalog entry, or if the prefix is missing any required file. Re-staging the same source is a no-op; a different source replaces the entry and reports the previous snapshot for separate reclamation. WeightsStore gains GetWeights and a source-URI field on its record to support this. LookupServingSpec (catalog.go) is reused unchanged to validate --model-id. * fix: unify ServingSpec after rebase onto dev's daemon-launcher spec Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec and LookupServingSpec, textually duplicating the ochre weights stage CLI's own versions in catalog.go. Collapse both into one type carrying ModelID and VLLMArgs, and one three-value LookupServingSpec so callers distinguish an unknown model (found=false) from a provider-tier one (selfHost=false). Update the daemon launcher's call site accordingly. * test: cover ochre weights stage/list/remove command behaviour Splits runOchreWeightsStage/List/Remove into testable decision functions (runStageWeights, listWeightsOutput, removeWeights) separate from cobra and NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps swappable so the full command flow can be driven against an embedded JetStream server instead of live infrastructure. Also relaxes the tokenizer file requirement to accept either tokenizer.json or tokenizer.model, since canonical Llama 3.x repos ship only the former at the prefix root. * fix: resolve mkfs.ext4 from sbin when it is absent from PATH e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so weights staging failed on an unprivileged operator's otherwise fine host. * fix: size the weights volume to a whole GiB so the tail block fits viperblock rejects a write whose block overruns VolumeSize, so importing an image of unaligned length failed on its last block. Rounding up also makes the volume's byte size and its manifest SizeGiB describe the same volume. Name the volume in the import error so a part-written import is traceable. |
||
|
Julian Sommer
|
6605d049bc | chore: fix COMMANDS.md | ||
|
Julian Sommer
|
53960e0bc4 | fix: update README.md to mark RDS as Available | ||
|
Bryn Mailer
|
c2b80a9b93 |
feat: match new predastore config surface area (#591)
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
|
||
|
Josh Siviour
|
8ae8d2fec4 |
feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes (#658)
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on CreateVolume, persists it to VolumeMetadata, and surfaces it via DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped in mulga-siv-380. Pins viperblock to the commit that adds the VolumeMetadata.Throughput field. Zero Throughput on a read volume means it predates this field; it is omitted from the response rather than surfaced as a misleading 0 MiB/s, matching the existing IOPS convention. * chore: pin viperblock to the merged gp3 Throughput commit |
||
|
Josh Siviour
|
5de598e85a |
ci: trim unused E2E builds, move IMDS to the SDK, derive predastore bind in spx (#656)
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs These two jobs immediately rebuild suite-aware binaries after Build Install Artifacts and never consume the tar's _bin contents, so the cross-compiled suite build was pure waste. build-install-artifacts.sh gains an optional --skip-e2e-build flag (positional contract unchanged); e2e-nightly.yml and the other callers keep building the full set. * refactor: replace hand-rolled IMDS client with AWS SDK v2 internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials provider used a hand-written IMDSv2 token/GET flow with manual RFC3339 expiry parsing. A malformed Expiration silently fell back to "never expires" instead of failing. Replace it with feature/ec2/imds, credentials/ec2rolecreds and aws.CredentialsCache, following the internal/gwsign/signer.go precedent. An unparseable Expiration is now a hard error from Fetch, covered by TestFetch_BadExpirationErrors. Output shapes to callers are unchanged, and IMDS base URLs ending in /latest keep working. * refactor: derive predastore bind config from spinifex.toml in spx itself predastore-start.sh awk-parsed spinifex.toml for the bind host/port and node_id, exported them as env vars, then exec'd spx. Move that derivation into predastoreStartCmd via derivePredastoreBind, which reads the raw (non-DIAL-normalized) nodes.<node>.predastore section, defaulting node_id to -1 (co-located) when the key is absent, same as the script. An explicit --host/--port/--node-id flag or SPINIFEX_PREDASTORE_* env var still overrides the derived value. Delete the now-unused script and point the systemd unit's ExecStart directly at spx service predastore start, matching the spinifex-viperblock.service precedent. |
||
|
Josh Siviour
|
2db1034436 |
feat(acm): managed certificate issuance and renewal from a tenant Private CA (#606)
* fix: fan out certificate re-import to referencing load balancers Re-importing new material under an existing ACM ARN previously updated only the stored CertRecord; no load balancer referencing that ARN ever re-rendered its config, so HAProxy kept serving the old leaf until it expired while the API reported success. Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff around every listener create/modify/delete and cert attach/detach in handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook (wired in daemon.go to avoid an import cycle) that re-runs updateStoredConfig for every load balancer in the ARN's InUseBy set, mirroring the existing target-group fan-out. lbagent is unchanged: it already polls ConfigHash and reloads on change. * fix: make InUseBy index updates CAS-safe under concurrent writers AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which loses concurrent writers silently: two listeners on different load balancers attaching the same certificate at once could drop one entry with no error. Because InUseBy is the sole mechanism that fans a renewed certificate out to the data plane, a lost entry means that load balancer never re-renders and its certificate silently expires in HAProxy while ACM reports it as healthy. Switch both to a JetStream KV revision-based compare-and-swap loop with jittered backoff, retrying on a concurrent writer instead of overwriting it. Replace the old concurrency test, which only asserted no errors/panics and so could not detect a lost index entry, with one that asserts exact final state across N concurrent writers. * fix: encrypt ACM certificate private keys at rest CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key. Store.PutCert encrypts when a master key is set; GetCert/ListCerts decrypt, falling back to a narrowly-bounded legacy-plaintext read (only when the stored value is unambiguously a PEM private key block) so pre-existing records keep working and are re-encrypted on next write. A Store never given a key (ELBv2's independent read-only acmStore) behaves exactly as before — encryption is opt-in per Store instance via the new SetMasterKey method, so no other package needs to change. NewACMServiceImplWithNATS now requires a master key and fails construction without one; daemon.go loads master.key inside the initServiceWithRetry closure so a missing key fails daemon startup after the retry window instead of leaving the service permanently degraded. * fix: wire ELBv2's ACM store with the deployment master key ELBv2 constructed its own handlers_acm.Store without a master key, so once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could no longer resolve listener certificates: it read back ciphertext where it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a key so an unkeyed Store can no longer be constructed, thread the deployment master key through NewELBv2ServiceImplWithNATS and the daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM store construction failure fatal instead of silently degrading every HTTPS listener. * feat(acm): tenant Private CA with x509 name constraints Adds an independent tenant root CA (separate from the platform CA) that signs leaf certificates only within its baked-in permitted domains. Name constraints (PermittedDNSDomains, excluded IP ranges, excluded email/URI) are frozen at creation and drift is rejected rather than silently regenerated, so a device's trust store is never invalidated without an explicit operator action. * docs(acm): record OpenSSL verification of the empty email/URI exclusions Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not just Go's own verifier, since this CA is meant for device trust stores. * feat: implement RequestCertificate with PRIVATE_CA managed issuance Adds the ACM RequestCertificate API surface: mode derivation (PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for the deferred modes. Extends CertRecord with issuance/validation/renewal state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md. * feat: wire tenant CA into the running spinifex daemon Add LoadTenantCA as a load-only entry point (never creates), refactor LoadOrCreateTenantCA to share the load path. Add 'spx admin cert create-tenant-ca' to create the tenant CA with explicit permitted domains, idempotent unless --regenerate is passed. Wire tenant CA loading into daemon startup: missing tenant CA logs a warning and degrades PRIVATE_CA issuance only, it does not block startup like the master key does. Fold tenant CA paths into admin's CA path helpers, keeping filenames distinct from the platform CA (tenant-ca.pem/.key vs ca.pem/.key) so the two roots can never be confused. Also give ACM's DeleteCertificate in-use refusal its own awserrors alias instead of reusing the EKS-labelled one directly, and correct COMMANDS.md to reflect that the tagging endpoints are implemented. * fix: derive ACM validation mode from the requested domain, not a global flag Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that northstar is configured, never whether it is authoritative for the domain being requested. Any deployment with a default domain set therefore returned MANUAL_TXT for every request, including domains hosted nowhere, with a TXT record the caller could not publish. Resolve the requested domain against northstar's actual hosted zones, and put MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly delegated so a public CA can resolve the challenge - which is never true of an internal northstar zone. A tenant CA covering every requested domain now wins, and a configured DNS provider still takes precedence over the manual path so operators with a real domain are not downgraded to a private certificate. * feat: renew PRIVATE_CA certificates automatically before they expire Managed issuance signed a leaf once and never renewed it. With 90-day leaves every certificate silently expired a quarter after the first apply, which is the treadmill managed issuance exists to remove. Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully offline, so this needs no ACME client and no network access. The window is proportional - two thirds of the certificate's own lifetime, floored at a minimum - so it self-tunes as issuance lifetimes change rather than degenerating into 'renew immediately, forever' the way a fixed window would. A deterministic per-ARN jitter keeps certificates issued in one apply from all renewing in the same minute. Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes through the ELBv2 fan-out. The push is mandatory rather than incidental: GetLBConfig returns stored configuration and never re-resolves certificate material, so a renewal that did not push would leave HAProxy serving the old leaf until something unrelated changed the config hash. A per-certificate lease keyed on the ARN means one node drives a given renewal without serialising all renewals behind a single leader. Certificates whose domains have fallen outside the tenant CA's name constraints record a failure reason instead of retrying in a tight loop. The renewal scan reads metadata only and never decrypts stored private keys; the lease is a raw compare-and-swap that leaves the encrypted key untouched. * fix: carry handler error messages over the daemon NATS transport handleNATSRequest responded with the sanitized error code alone, so any handler error without a registered code collapsed to a bare ServerInternal and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx has always preserved the message; the two implementations had drifted. Adds respondWithServiceError and routes the 16 service-error sites through it, plus transport-level tests pinning that the message survives for an uncoded error, that a wrapped coded error keeps its real code, and that a bare code carries no message. * feat: implement ACM GetCertificate and stop decrypting keys on read paths GetCertificate was unimplemented end to end: it now returns the PEM certificate body and chain for an owned ARN, ResourceNotFound for an unknown or cross-account ARN, and RequestInProgressException for a certificate that has a record but no body yet, wired through the service interface, the NATS client, the daemon subscription and the gateway action map. Store gains GetCertMetadata alongside GetCert, and ListCerts becomes ListCertMetadata. Every read that answers a question about a certificate rather than serving it - ownership checks, Describe, Delete, tag reads, GetCertificate and listing - now leaves PrivateKey empty instead of running an AES-GCM open per record to produce plaintext key material no caller reads. * fix: reconcile the ACM master-key wiring with dev's new tests The InUseBy and cert fan-out tests that landed on dev construct the ACM and ELBv2 services directly, and both constructors now take the deployment master key. Pass one shared key in the fan-out test - two services over the same bucket disagreeing on the key is the exact failure the wiring exists to stop. A Store is never unkeyed now, so the two test helpers collapse into one. * fix: rebuild the ACM InUseBy index at daemon startup The index that drives certificate fan-out is written only by listener mutations, so a listener whose certificates have not been touched since it was created has no entry. A certificate with an empty index then neither fans out on renewal nor trips DeleteCertificate's in-use guard, and both failures are silent: the renewal reports success while the load balancer keeps serving the old leaf, and a certificate a live listener depends on can be deleted. Reconcile the index from the persisted listener records once at startup. Walking every certificate rather than only the referenced ones makes it a rebuild rather than a backfill, so it also drops entries naming a load balancer that no longer exists. Writing only differences keeps it idempotent, and the index writes are already CAS-guarded, so the nodes of a cluster reconciling the same bucket converge. * fix: deliver listener SNI certificates to HAProxy Add/RemoveListenerCertificates persisted the listener record and then returned, so the config was never re-rendered and lbagent had nothing to poll for. Re-render through reloadListenerLB, the same helper the rule mutations use. That alone changed nothing, because the renderer dropped SNI certificates: resolveFrontendCert returned only the default certificate and the bind line emitted a single crt, with the PEM path keyed on the listener so a second certificate had nowhere to live. Render every attached certificate instead, default first so it stays the no-SNI fallback, each with its own path. HAProxy accepts crt more than once on a bind line and selects by SNI. The default certificate keeps its existing filename so an upgrade does not move a deployed PEM, and the nginx renderer keeps the single-certificate resolver it still relies on. * fix: prune stale TLS certs from the lb-agent cert dir A delivered PEM is cert + chain + private key. writeCertFiles only ever wrote, so every certificate detached from a listener left its key on the load balancer VM's disk indefinitely. Prune after a successful reload, scoped to .pem files in the cert dir's immediate entries. * fix: serve every attached certificate on the nginx NLB engine nginx allows one ssl_certificate per key type in a server block, and the NLB config is a stream block where server_name does not exist. Select by SNI instead: give ssl_certificate a variable and map $ssl_server_name to each certificate's path, keyed on the leaf's DNS SANs. A single-certificate listener keeps a literal path, since a variable costs a per-handshake read. The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so they now take the dir and both engines share one multi-cert resolver. * fix: emit ValidationMethod in DomainValidationOptions DescribeCertificate returned each domain validation entry without a ValidationMethod, so the AWS provider had nothing to compare against aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes. PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits the method on a private certificate, where the provider reads it as NONE whatever the request asked for. |
||
|
Julian Sommer
|
e79ca59d4b | fix: configure host DNS from spx admin init/join (#582) | ||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Julian Sommer
|
b40d538c61 | refactor(admin): remove dead --no-external flag from spx admin init (#537) | ||
|
Julian Sommer
|
744cb1929d | fix: admin init --force preserves keys, credentials and CA (#536) | ||
|
Josh Siviour
|
183a08353e |
feat(network): harden routed-NAT v1 (VPC cap, e2e gate, docs) (#504)
Three related nat-mode fixes: - ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template so operators can widen any pool. - 724mm: detectPoolMode returns true only for external_mode=pool. nat and disabled clusters now skip Phase 8b/8d, which otherwise failed on AllocateAddress UnsupportedOperation. - pnz2i: document that system instances (ECS/EKS/LB agents) are unsupported in nat v1 (require pool mode) in the vpc-networking README and COMMANDS.md; document the gw_lrp_range_* fields. |
||
|
Josh Siviour
|
d663760dec |
feat(eks): control-plane disaster recovery — etcd snapshot + restore-snapshot (#503)
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for
|
||
|
Julian Sommer
|
8fe8a9436c | feat(ec2): EC2 Launch Templates (#498) | ||
|
Julian Sommer
|
5d199361b6 | feat: implement IAM GetAccountSummary (#484) | ||
|
Josh Siviour
|
e1c4f85a36 |
feat: routed-NAT external mode with two-tier ingress (#475)
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets. |
||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
tomnewton-mulga
|
4f5fd88c86 |
feat: snapshot ami (#279)
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation - Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe) - Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels - Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type) - Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles - GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors * fix(daemon): enforce CPU/memory limits for MIG instance types Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate). MIG types now fall through to canAllocateCount so host CPU and RAM are checked before launch — prevents over-committing a host where one physical GPU backs multiple concurrent slices, each with non-trivial vCPU/memory allocation. * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * fixed testing * more tests * fix: added new graceful process killer in utils, bumped NATS timeout for testing * fix: complete at-rest encryption launch + volume lifecycle (#361) * chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged) * fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) * feat: cloud-init EC2 datasource (#411) * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * merge changes * chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never set post-rebase. Removes the field from MountedVolume and the two nil-checked unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig. * fix: post-rebase gofmt fixes and remove dead snapshot handler code Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go, viperblockd.go, volumes.go, and volume_durability_test.go. Removes the dead unix-socket snapshot handler tests from viperblockd_handlers_test.go. * test: add viperblock snapshot S3 coverage tests Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume past the nil-config guard using an httptest server that returns 403. The AWS SDK credential check fails synchronously without network I/O, keeping the tests fast while covering the S3Config construction, master key load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%. * fix: cloneAMIToVolume reads config via format-agnostic path LoadStateRequest expects a viperblock binary envelope (encrypted), but CreateImageFromInstance writes the AMI config as plain JSON via putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig which uses viperblock.StateBody to handle both formats, matching the path used by ListImages and all other AMI readers. * fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint. CI builds without go.work so it resolved to the published version and failed. Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged. * fix: replace indexOf with slices.Index, revert e2e runner and viperblock version - nbd_test.go: remove local indexOf helper, use slices.Index from stdlib - e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting - go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges) * fix: remove pseudo-version pin from go.sum * feat: drain viperblock via snapshot socket before CreateSnapshot Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock and wait for DrainToBackend() to complete (30s timeout). This ensures WAL chunks and the live checkpoint are current in S3 before the snapshot VB reads them. If the socket is absent the instance is stopped and Close() already drained. fix: socket path used s.config.WalDir which is empty in the daemon config; changed to s.config.DataDir + "/viperblock" to match the path the NBD plugin uses when creating the socket. * feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint - Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot) - Deregister from image detail (DeregisterImage with navigate-away on success) - Promote to System from image detail (admin-only, irreversible via UI) - NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage - signedFetch extended with optional params to support action-keyed extra fields * refactor: address PR review on PromoteImage types and subscription ordering - Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult (add JSON tags; drop the duplicate pair) - Inline the input variable in gateway/spx/promote_image.go - Move spinifex.image.promote subscription to the spinifex.* group in daemon.go * test: cover snapshotVolume/Running/Stopped error paths for diff coverage Add encryption-key-load-error and LoadState-failure tests for the three new viperblock snapshot paths (snapshotVolume in snapshot service, snapshotRunningVolume and snapshotStoppedVolume in image service). Diff coverage: 58.8% → 65.5%. --------- Co-authored-by: Josh Siviour <mrk@mulgadc.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Julian Sommer
|
81212fe91b | feat: EC2 central-record resource tagging (#469) | ||
|
Julian Sommer
|
b756798ed6 | feat: IAM resource tagging (#467) |