mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
d178ef5992eb0bbd4f96b00ffdc587c8956fe518
60 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Duncan
|
b76aab8b50 | feat(installer): three-plane networking, ZFS root, distributed NAT and multi-node install (#663) | ||
|
Josh Siviour
|
5de598e85a |
ci: trim unused E2E builds, move IMDS to the SDK, derive predastore bind in spx (#656)
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs These two jobs immediately rebuild suite-aware binaries after Build Install Artifacts and never consume the tar's _bin contents, so the cross-compiled suite build was pure waste. build-install-artifacts.sh gains an optional --skip-e2e-build flag (positional contract unchanged); e2e-nightly.yml and the other callers keep building the full set. * refactor: replace hand-rolled IMDS client with AWS SDK v2 internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials provider used a hand-written IMDSv2 token/GET flow with manual RFC3339 expiry parsing. A malformed Expiration silently fell back to "never expires" instead of failing. Replace it with feature/ec2/imds, credentials/ec2rolecreds and aws.CredentialsCache, following the internal/gwsign/signer.go precedent. An unparseable Expiration is now a hard error from Fetch, covered by TestFetch_BadExpirationErrors. Output shapes to callers are unchanged, and IMDS base URLs ending in /latest keep working. * refactor: derive predastore bind config from spinifex.toml in spx itself predastore-start.sh awk-parsed spinifex.toml for the bind host/port and node_id, exported them as env vars, then exec'd spx. Move that derivation into predastoreStartCmd via derivePredastoreBind, which reads the raw (non-DIAL-normalized) nodes.<node>.predastore section, defaulting node_id to -1 (co-located) when the key is absent, same as the script. An explicit --host/--port/--node-id flag or SPINIFEX_PREDASTORE_* env var still overrides the derived value. Delete the now-unused script and point the systemd unit's ExecStart directly at spx service predastore start, matching the spinifex-viperblock.service precedent. |
||
|
Josh Siviour
|
031f0a4cdd |
fix: preserve running guests across target restart, harden state write (#654)
* fix: preserve running guests across target restart, harden state write spinifex-shutdown.service's ExecStop fired on any spinifex.target stop, including the stop half of `systemctl restart spinifex.target`, draining every guest via QMP even though the host never went down. Gate the drain behind `systemctl is-system-running` == "stopping", which only holds once systemd is genuinely unwinding into shutdown.target (reboot/poweroff), so a target restart or plain `systemctl stop` now leaves QEMU/nbdkit running (KillMode=process) for the daemon to reconnect to on the way back up. Daemon.WriteState returned as soon as the local state file write failed, skipping the independent JetStream KV write entirely. A crash in that window left a live VM with no durable record anywhere. Attempt the KV write regardless of local disk health, then still surface the local write error to callers, mirroring finalizeTerminated's handling of the same fault class on the terminate path. * fix: gate the shutdown drain via an spx flag, not a shell script Replace the shutdown-drain-gate.sh gate with a --only-if-host-stopping flag on `spx admin node drain`. The unit now points at /usr/local/bin/spx directly (already required for the node to function) instead of a script installed under the data directory, which a partial rebuild of that directory could silently drop without ever failing ExecStop. The gate logic also becomes a plain Go function with injectable state lookup, so it gets real table-driven coverage instead of only a live reboot. hostIsStopping() drains only on a literal "stopping" from `systemctl is-system-running`. Every other recognized state (running/degraded/...) and any unrecognized output skip the drain; a fully unreachable systemctl fails toward draining, since a skipped drain on a real shutdown hard-kills guests and loses unflushed writes, while a spurious drain on a restart only costs an extra graceful stop/relaunch cycle. Run by hand without the flag, the command still drains unconditionally as it always has. * refactor: split the shutdown-gate comments to sit with the code they explain |
||
|
Josh Siviour
|
22171c44ab |
fix: close the storage-exhaustion recovery gaps (durable terminate, per-service memory caps, local volume leak) (#653)
* fix(vm): durably record terminate even when local state persistence fails finalizeTerminated bailed out of WriteTerminatedInstance whenever the transition to terminated hit a persistence-only failure (e.g. ENOSPC writing local state), even though the in-memory status had already reached terminated. That left the instance owned by nothing: StuckTerminateReaper skips it (no longer shutting-down) and TerminatedTeardownReaper never sees it (never reached the terminated KV bucket). Only an operator daemon restart recovered it. WriteTerminatedInstance is a JetStream KV write, independent of local disk space, so keep going into it whenever the in-memory status did reach terminated, distinguishing that case from a genuine invalid/raced transition. Once the disk has space again, teardown now completes on its own via the existing idempotent TerminatedTeardownReaper. * fix(systemd): cap per-service memory under spinifex-system.slice Every service under spinifex.slice ran with MemoryMax=infinity / MemoryHigh=infinity, so one runaway service could exhaust the whole slice budget and the slice-level OOM would then reap an arbitrary victim instead of the offender. Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the same units scale from a dev box to bare metal). spinifex-daemon.service still hosts every guest QEMU process in its own cgroup (no per-instance slice yet), so it gets by far the largest allowance, just under the slice's own 80%/90%, to avoid starving VM launches. predastore is the legitimately memory-hungry service and gets generous headroom above its idle footprint. The remaining infra services are small, single-purpose Go processes and get a shared conservative bound. * fix(viperblockd): remove local volume directory on ebs.delete DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete handler only stopped the volume's goroutines, killed nbdkit, and removed the NBD socket — it never removed the local WAL/checkpoint cache directory under BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only ran from the unmount handler's seal step, which -efi volumes never reach (they carry no durable data to seal) and which a main volume also skips whenever a prior seal failed (e.g. disk full, deliberately keeping local files for recovery). Since terminate always unmounts before deleting, and delete is permanent, retention had no remaining purpose by the time ebs.delete ran. Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless of mount-tracking state, closing the leak for both the root volume and its -efi companion. * fix(viperblockd): validate volume names before removing local state ebs.delete unmarshalled the volume name straight off the wire and handed it to os.RemoveAll: an empty name collapsed to BaseDir itself, and ".." components could escape it entirely. localVolumeDir now rejects an empty/invalid name or one that resolves outside BaseDir before any removal, and ebs.delete returns Success: false with an error instead of silently skipping. Applied the same gate to ebs.mount and ebs.config, the other two handlers that turn a wire-supplied volume name into a BaseDir-relative path. * chore: pin viperblock to the merged concurrent-open corruption fix The ansible build role builds with GOWORK=off, so the deployed binary links whatever go.mod pins rather than the go.work overlay. Move the pin to the revision carrying the per-volume flock and the WAL/state write fixes. |
||
|
Josh Siviour
|
6b8e3d870e |
fix(logrotate): declare su so spinifex logs actually rotate (#650)
/var/log/spinifex is 0775 root:spinifex so each service user can write its own log. logrotate refuses to rotate inside a group-writable directory unless the config says which user to rotate as, so every entry was skipped and the logs grew without bound. Verified on env13: before, logrotate -d reported 'skipping /var/log/spinifex/nats.log because parent directory has insecure permissions' and nothing had ever rotated. After, the debug run is clean and forced rotations produce nats.log.1 and a compressed nats.log.2.gz. |
||
|
Josh Siviour
|
a4274f95b5 |
fix(systemd): keep in-use nbdkit alive across a viperblock restart (#648)
shutdownVolumes deliberately spares nbdkit that a guest is still writing through, because tearing one out mid-write corrupts that guest's filesystem. The unit left KillMode at the default, so the cgroup SIGTERMed those children anyway and SIGKILLed them at TimeoutStopSec -- causing both a 30s stop stall and the corruption the skip exists to prevent. KillMode=process matches spinifex-daemon, where guest QEMU already survives a restart for DDIL reattach. Idle nbdkit is still reaped explicitly. Measured on env12 with a guest attached: control-group stopped in 30.08s with 'Failed with result timeout' and SIGKILLed nbdkit, leaving QEMU running with zero ESTAB nbd sockets. KillMode=process stops sub-second, both nbdkit survive, and the guest keeps its device. |
||
|
Josh Siviour
|
57a93fdee3 |
fix(nats): detect the JetStream write latch the health endpoint cannot see (#645)
The watchdog probed only healthz?js-enabled-only=true, which reports the server-wide JetStream state. A per-message-block write error is recorded on the block and returned on every later write, and nothing clears it short of a restart -- the server-wide state is never touched, so healthz stays 200 while writes fail indefinitely after the disk recovers. spx admin node js-probe round-trips a canary key through the cluster-state bucket, which is the only operation that observes the latch. It probes that bucket rather than one of its own because a dedicated bucket is a separate stream that can be placed elsewhere and stay writable. |
||
|
Josh Siviour
|
7d5ebabecc |
fix: explicit stop timeouts, graceful-shutdown exit code, and QEMU orphan classification (#640)
* fix(systemd): give every unit an explicit TimeoutStopSec 9 of 12 spinifex systemd units had no TimeoutStopSec and inherited systemd's 90s default, which is either far more or far less time than their own SIGTERM handling needs. Size each one to what its shutdown path actually does: predastore matches viperblock's 30s WAL-drain budget since viperblock flushes through it (After= already sequences viperblock's stop first; this is the backstop if that's ever bypassed), spinifex-ui covers its 30s internal shutdown context plus margin for long-lived streaming proxy connections, nats covers its own JetStream-shutdown/raft-stepdown/client-drain signal handler, and services with no SIGTERM handler at all (awsgw) get a short bound since they exit immediately regardless. Extends the existing unit-file invariant test with a check that every .service file declares TimeoutStopSec, so a newly added unit that forgets one fails CI instead of silently falling back to 90s. * fix(vm): reconcile QEMU processes with no instance record on restore classifyRestoredInstances only walks Manager.Snapshot(), so a qemu-system process with no instance record at all -- a crash mid-launch, a lost KV write, a stale process surviving a wiped state file -- was never handed to killOrphanedQEMU and stayed permanently invisible to recovery. Add a host-process scan that diffs live qemu-system PIDs (matched via /proc/<pid>/comm) against the PIDs claimed by known instances' own -pidfile files. It only reports what it finds, never kills: this daemon cannot distinguish "record not loaded yet" or "owned by another daemon/tenant on this host" from a genuine orphan, and killing a running customer VM to fix an invisibility gap is the wrong trade. Runs at the end of Restore, once every known instance has been classified and relaunched, so a live process still unaccounted for at that point genuinely has no record anywhere. * fix(vm): distinguish recordless QEMU orphans by pidfile ownership A recordless qemu-system process with a stale spinifex pidfile is positively this daemon's own orphan; one with no pidfile at all could belong to another tenant on a shared host. Log the two cases distinctly instead of one blanket warning, so an operator can tell them apart. Also sandbox XDG_RUNTIME_DIR in the shared daemon test harness so Restore()-driven tests never read this box's real runtime directory. * fix(spinifexui): treat ErrServerClosed as a clean shutdown, not a failure Serve returns http.ErrServerClosed after Shutdown is called deliberately (the SIGTERM handler above does exactly that), but Start propagated it as an error, making systemd log a graceful stop as exit code 1 and Result: exit-code -- indistinguishable from a real crash. |
||
|
Josh Siviour
|
9f5f877e50 |
fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637)
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console. |
||
|
Josh Siviour
|
70dda56fc3 |
fix(systemd): backstop viperblock stop at 30s instead of the 90s default (#619)
In-use nbdkit children are SIGTERMed with this cgroup and run the plugin's shutdown drain. That drain is now bounded at 20s in the plugin itself, so 30s here lets the plugin's own bound fire first and keeps SIGKILL exceptional. At the 90s default a slow or unreachable backend stalled the whole target-stop transaction. |
||
|
Josh Siviour
|
628fbbc0c3 |
fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609)
* fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts. |
||
|
Josh Siviour
|
08fc5e4c58 |
fix(northstar): initialise OTel SDK and source the telemetry environment (#555)
Northstar carried working northstar.dns.* instruments but shipped no telemetry, so its dashboard was empty and the only northstar data in Elasticsearch was journald scraped under the systemd unit name. Two gaps, both outside the instrumentation. northstarStartCmd never called initTelemetry, unlike every other service command, so the instruments recorded into a no-op provider. The unit also did not source /etc/spinifex/telemetry.env, which every other application unit reads for OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV and MULGA_SOURCE — so even an initialised SDK would have had no endpoint and no env label. Add an invariant covering the second gap: any unit launching an spx service must source the telemetry drop-in. A service missing it starts healthy and silently reports nothing, which only surfaces when someone notices an empty dashboard. Verified the test fails against the unpatched unit. |
||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
Josh Siviour
|
ab1b1bacf6 |
fix: auto-restart NATS when JetStream is disabled while the process lives (#445)
When NATS disables JetStream at runtime (e.g. a filesystem permission error flushing stream state) it keeps the process alive and serves KV as read-only. That is not a process failure, so spinifex-nats.service's Restart=on-failure never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes until an operator restarts NATS by hand. Add a JetStream liveness watchdog: - build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint (http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service is active and JS fails every sample of a short multi-sample window (so transient startup/quorum 503s are ignored) does it systemctl try-restart spinifex-nats. - spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a 60s boot delay so cluster quorum settles first. - spinifex.target Wants= the timer so it starts with the platform. The NATS-internal permission race itself is a server quirk outside spinifex's control; this restores service automatically when it recurs. |
||
|
Josh Siviour
|
9e333c8a6c |
feat: graceful guest drain on systemctl stop / host shutdown (#412)
* fix: drop removed predastore s3.Config.Compaction from admin_test The predastore submodule bump removed the Compaction field from s3.Config. Parse the rendered [compaction] block into a local struct so the template round-trip test no longer references the deleted field. * feat: graceful guest drain on systemctl stop / host shutdown systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU was still writing, corrupting guest filesystems. The ordered drain existed only over NATS via cluster shutdown, never on the systemd/host path. Add spx admin node drain --local (GATE+DRAIN against the local node, leaving STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new spinifex-shutdown.service oneshot ordered After= all services so it stops first and drains guests while every service is still up. DDIL preserved: the oneshot fires only on target/host shutdown, not on a daemon restart. Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an nbdkit a guest is still attached to. Add a units ordering invariant. * fix: ExecStop drain timeout needs duration unit (120 -> 120s) The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin node drain's --timeout is a cobra Duration flag and rejects a bare integer (time: missing unit in duration "120"). The drain exited 1 without running, so a systemctl stop tore nbdkit out from under the running guest and corrupted its ext4 (the exact failure this feature prevents). Pass 120s. * fix: nbdkitInUse read ss state from correct column + cover it ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0] (field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit read as idle and the SIGTERM path would reap it under a live guest. Add tests exercising a real connected unix socket, an absent socket, and TCP transport. |
||
|
Julian Sommer
|
e11437cec9 | feat(imds): per-tap responder datapath (#408) | ||
|
Josh Siviour
|
1654f98e18 |
feat(resource-governance): RG-3/4/6/8/9/11 — admission charge, OOM ladder, tier confinement (#327)
* feat(daemon): live MemAvailable admission gate + fixed-services reserve scope Admission previously charged only guest -m against a flat reserve, blind to the co-located storage VMs, nbdkit, and daemon stack — so it greenlit launches that overcommit real RAM and OOM-kill guests on memory-tight hosts. Add a second admission gate that clamps the accounting-derived launch count by live /proc/meminfo MemAvailable (which already nets reclaimable cache and sees every untracked consumer): admitted = min(accounting, (MemAvailable-reserve)/ guestMem). Reads live per admission, fails open on a /proc read error so a transient fault can't wedge launches, and is disabled by SPINIFEX_ADMISSION_LIVE_MEM=0. Re-scope the host reserve to the fixed infrastructure tier only; per-instance nbdkit is no longer expected to fit inside it (explicit per-instance charge is a follow-up; the live gate covers it implicitly in the meantime). Pure helpers (liveMemCount, parseMemAvailableKB, liveMemAdmissionEnabled) are unit-tested; liveMemGate covered via an injected reader. * feat(daemon): charge nbdkit per-instance in admission accounting An instance's admission memory charge is now its guest -m plus the nbdkit processes backing its volumes (one per attached volume), not guest -m alone. The static reserve no longer has to absorb per-instance nbdkit, which was unsizable and the root of the host-memory-overcommit OOM. Per-volume charge is measured per class (main 768 MiB: 128 cache + plugin working set + in-flight I/O; aux 96 MiB: process floor, 0 cache), env-tunable via SPINIFEX_NBDKIT_MAIN_MIB / SPINIFEX_NBDKIT_AUX_MIB. Default layout is one main + two aux volumes (root + cloudinit + efi) ~= 0.94 GB/instance. Applied at every accounting site through instanceMemChargeMiB so charge and release never drift; the live MemAvailable gate backstops block-device-mapping volumes the static count misses. * feat(vm): three-tier guest OOM ladder + spinifex-guests.slice Replace the flat oom_score_adj=500 on every guest QEMU with a ManagedBy-derived ladder: user guests (ManagedBy=='') stay +500 and are reaped first under host memory pressure; system instances (ELBv2 LB-VMs, EKS control-plane nodes, ManagedBy!='') score 0, ranking above user guests but below the infra tier (-500). Killing a system instance degrades the data path or cluster control plane for many user VMs, so it must outlive them. Add spinifex-guests.slice as a sibling of spinifex-system under spinifex.slice — the parent tier for the per-instance leaves that land with RG-7. Guest QEMU placement into the slice is deferred to the per-instance work. * refactor(systemd): lean unit comments to RG-n tags + RG-9/RG-11 invariants Strip the multi-paragraph rationale novels from spinifex-daemon.service (48->7 comment lines), spinifex-vpcd.service (29->8) and spinifex.slice (15->5), replacing them with settings + terse # RG-n clause tags. Remove plan-doc, bead and CI-run references from comments (project policy). Directives are byte-for-byte unchanged — comment-only. Add RG-9/RG-11 invariant tests asserting per-tier confinement (locked-down storage/control tier drops all caps + NoNewPrivileges=yes; daemon keeps exactly CAP_SYS_ADMIN+CAP_DAC_OVERRIDE and the four-entry DeviceAllow; vpcd is the documented network-tier exception) and the lean-comment budget + no-plan-ref rule. |
||
|
Josh Siviour
|
2ea3e22db6 |
feat(systemd): cgroup-confine spinifex to a resource-limited slice + host swap (#324)
* feat(systemd): bound spinifex to a resource-limited slice Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%, MemorySwapMax=0) with a spinifex-system.slice child holding the seven control/infra services, so a runaway hits the slice instead of the host. A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU for sshd/journald/the operator, keeping the box reachable when spinifex is maxed. The drop-in is generated in setup.sh install_systemd() rather than shipped as a staged file, since the packaging globs flatten the systemd/ dir; this mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow through every staging path unchanged. Within the single effective cgroup, a slice MemoryMax OOM picks the highest oom_score victim — guest QEMU (+500) before any service (-500) — so a slice OOM degrades by killing one guest while the daemon, storage tier, NATS, and the host all survive. * feat(setup): provision 8G host swapfile + vm.swappiness=10 Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it to fstab, and writes vm.swappiness=10 so the kernel only swaps under real pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap for graceful degradation instead of OOMing a guest. Idempotent; skipped under ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0 assumption. |
||
|
Josh Siviour
|
5cf5cba7ac |
fix(microvm): load virtio_net dependency chain explicitly in init (#311)
The LB system microVM stripped its module tree to virtio + fw_cfg +
failover, but init.sh only loaded qemu_fw_cfg and virtio_net. virtio_net
depends on net_failover -> failover; modprobe resolves that chain via
modules.dep, and load_mod()'s insmod-by-path fallback loads a single .ko
with no dependency resolution.
When the image is built on a kmod-less host (depmod skipped, no
modules.dep) the chain can't be resolved either way, so virtio_net fails
to load, no NIC appears ('no interface found for MAC'), the lb-agent has
no network, heartbeats never reach AWSGW and every LB hangs in
provisioning until the 5m e2e timeout. Baremetal runners ship kmod so
modules.dep is generated and the chain resolves — which is why the LB
suite passed there but failed on normal CI.
Load failover -> net_failover -> virtio_net explicitly bottom-up so the
NIC comes up regardless of whether modules.dep exists.
|
||
|
Josh Siviour
|
527308c342 |
fix(baremetal): empty modules.dep + capacity-aware single-node e2e (#305)
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug Add diagnostics so a stuck-in-provisioning internal LB can be pinned to one of three points on the heartbeat path: - lb-agent failure log now carries the gateway target + region, so the microVM serial console shows whether the heartbeat hit a transport error (never reached AWSGW) or an HTTP-level rejection. - AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the previously-silent not-found/account-mismatch reject path, so the journal distinguishes 'heartbeat arrived but cannot transition' from 'heartbeat never arrived'. * test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot - lb suite: gate the remaining internal subtests on Internal_ALB. The internal LBs share one mgmt-return datapath and fixture, so if the first never reaches active the rest time out identically — skip them instead of burning ~5m per subtest (~20m saved on a known-bad run). - microvm init.sh: widen the fw_cfg netcfg poll window and, on the not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name listing / dmesg fw_cfg to the serial console before init exits, so a boot panic is diagnosable from the captured console log. * fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep Baremetal-built microVM images shipped an unusable modules.dep (the build host's depmod errors were swallowed), so busybox modprobe in the guest init silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked before lb-agent ran — leaving internal LBs stuck in provisioning. init.sh: load_mod() tries modprobe, then falls back to insmod by explicit path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless of which host built the image. build-microvm-image.sh: require depmod, drop the error swallow, and assert modules.dep is non-empty after depmod so a boot-broken image fails the build loudly instead of silently. * fix(microvm): decompress kernel modules before depmod Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI was unaffected). Decompress kept modules to plain .ko after the keep-list strip, before depmod. Host depmod then produces a usable map regardless of its gzip support, and the guest loads modules without a decompressor. * feat(e2e): capacity-aware scheduling for dense single-node runs Denominate daemon schedulable capacity in physical cores instead of logical threads, and auto-serialise e2e launches when the cluster is at capacity, so a single hyperthreaded node stops over-packing guests. Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount() (distinct physical-id:core-id pairs in /proc/cpuinfo) rather than runtime.NumCPU(). Falls back to logical CPUs when topology is absent (non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded box this halves the over-pack: with the unchanged 2 vCPU reserve the i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the boot-spike CPU within real cores. Harness: capacityRetryEC2 decorates the EC2 client and, on InsufficientInstanceCapacity, waits and retries RunInstances instead of failing — turning the daemon admission gate into back-pressure so concurrent subtests queue. AWSClient.EC2 is now the wrapped ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access. Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/ fallback) and the retry decorator (retry-then-succeed, non-capacity passthrough, deadline). * fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start Denominating capacity by physical cores starves tiny hosts: a 2-physical- core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so NewResourceManager refuses to start and every daemon-constructing unit test fails. The previous runtime.NumCPU() count (4 logical on the same runner) incidentally cleared the reserve floor. Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon TestMain so unit tests are deterministic regardless of runner topology. Reserve stays at the default 2. |
||
|
Julian Sommer
|
bcb49cd771 | feat: IMDS v1 (#266) | ||
|
Josh Siviour
|
fd3dc6b680 |
feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255)
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
|
||
|
tomnewton-mulga
|
01d9fa858f | feat: iso release flow (#229) | ||
|
Josh Siviour
|
4ae2a4ac53 |
feat(elbv2): QEMU microvm direct-boot for LB system instances (#228)
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline
vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go
config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput
image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts
* feat(daemon): direct-boot launch path for microvm system instances
When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration
PC-machine path (DirectBoot=false) unchanged.
Closes mulga-siv-65
* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold
* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed
* fix(microvm): end-to-end fixes for direct-boot LB path
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
* fix(build): auto-detect docker/podman when apk unavailable for microvm image build
* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
* fix(microvm): fix mgmt NIC routing for all LB schemes
Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
* feat(microvm): Phase 2 kernel stripping + boot optimisation
- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches
* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip
The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.
Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.
* fix(build): keep failover modules in initramfs strip
virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.
Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.
* fix(daemon): restore slog default after viperblock.New mutates it
viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.
Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.
Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.
Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).
* perf(microvm): hit Phase 2 boot/memory gates
t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.
Root causes:
1. build-microvm-image.sh staged the entire chroot into the cpio
without dropping /boot, embedding a duplicate vmlinuz, Alpine's
stock initramfs-virt, System.map, and a "boot → ." symlink loop
(~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
the kernel is copied out, before the cpio. Initramfs drops from
31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
42 MiB → 22 MiB.
2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
no firmware, ROM, or block layer). Both extracted to
directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
qemuStartSettleWait to 50 ms.
3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
Added boot-time perf flags: quiet loglevel=3 (silence printk over
115200-baud isa-serial), mitigations=off (KVM guest on trusted
host, single-tenant LB workload), tsc=reliable, no_timer_check,
reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).
4. init.sh dropped modprobe virtio_mmio (built-in:
CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).
Phase 2 gates:
- t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms
- artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script)
- ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure,
unrelated to microvm path)
* build(microvm): wire build into deploy + distro + release pipeline
Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).
Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.
Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.
* build(ansible): include microvm artifacts in build role tarball
Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).
* feat(config): enable microvm direct-boot LB path by default
Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.
Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).
* fix(microvm): bootstrap apk.static when no host rootfs tool exists
Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.
When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.
No-op when apk, docker, or podman is already present.
Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244
* revert: drop apk.static bootstrap in build-microvm-image
Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.
* docs(workbook): drop lb-alpine import for microvm direct-boot LB
LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.
* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN
When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").
Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.
* fix(e2e): redirect lb probe chatter to stderr so stalls are visible
test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.
Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.
* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node
buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.
resolveMgmtRoute already encodes the correct policy:
- multi-node any scheme → explicit MgmtRoute{Gateway,Target}
- single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP)
- single-node internet-facing → empty (reach AWSGW via VPC SNAT)
Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.
Test covers all three branches.
* feat(ansible): dev-deploy playbook for in-place artifact swap
Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.
Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.
Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.
* chore(microvm): remove legacy LB Alpine image build pipeline
Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.
build-system-image stays — other manifests (e.g. GPU images) still
use it.
* refactor(microvm): remove legacy PC-machine LB boot path
Phase 3 — direct-boot becomes the only ELBv2 LB launch path.
- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
[daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
paths) and refresh stale comments that still mentioned lbVMUserData.
ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.
* docs: drop lb-alpine catalog entry from cluster setup guide
LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.
* feat(ui): remove LB image import banner + LB-AMI gating
Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:
- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
system-managed.ts (isSystemManagedImage stays — still used by the
EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
wizard pages
- The corresponding tests that asserted the banner appears
* revert: restore lb.conf + lb-setup.sh for mulga install pipeline
Phase 3 microvm removal (commit
|
||
|
Bryn Mailer
|
50eea38066 |
feat(predastore): aes 256 encryption at rest
Sibling changes in Spinifex to support changes made to Predastore in this PR. |
||
|
Josh Siviour
|
45555dde49 |
test(ddil): flip Scenarios A/B/C to real assertions (#205)
* test(ddil): flip Scenarios A/B/C to real assertions Replaces the t.Skip stubs landed in #204 with the assertions described in docs/development/improvements/ddil-e2e-test-harness.md §3: - A NATSKill — KillNATS on node3, assert ModeStandalone within 30s, /local/instances still serves on the isolated node, majority /health on node1+node2 stays up, witness counters on all three nodes keep advancing, snapshot survives reconnect. - B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert ModeStandalone within 30s of restart, recovered instances present with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post PID equality dropped — see file comment; recovery + a non-zero PID give the same regression coverage without coupling to the daemon unit's KillMode. - C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep serving /health and /local/instances, isolated node3 reports ModeStandalone with /local/* responsive, witness counter on the partitioned side keeps advancing, heal converges and snapshot is preserved. Supporting changes: - harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore, the deployed daemon ClusterManager binds [nodes.<id>.daemon].host. The Phase-1 stub used a placeholder that would have blocked any /local/* assertion against the real cluster. - harness/witness.go — export WitnessVM.Terminate so scenarios can clean up their counter VMs from t.Cleanup. - scenarios/main_test.go — requireLiveCluster + launchWitnesses helpers to keep the three flipped scenarios tight. - TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED; TestCoverageDrift continues to enforce alignment with the package. Validation: built with -tags=e2e, dry-run scenarios pass, coverage drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against tofu cluster is the next step. * fix(ddil): countDropRules — single iptables -S call + filter Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs across both chains, but iptables -S takes a single optional chain — the second -S errors to stderr, the combined SSH buffer poisons the grep input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState. Switch to one `iptables -S` (lists every chain in iptables-save format), filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin. Error path now also surfaces the raw SSH output for the next time this breaks somewhere unexpected. * ci(e2e-ddil): export spinifex AWS creds for witness SDK The witness EC2 client (harness/witness.go NewWitness) calls session.NewSession with no static credentials, so the AWS SDK falls through its default chain — env vars → shared profile → IAM role — and on the CI runner none of those exist. Result on the live cluster: "NoCredentialProviders: no valid providers in chain. Deprecated." launches every Scenario A/B/C attempt straight to t.Fatal at launchWitnesses. spx admin init writes [spinifex] aws_access_key_id / aws_secret_access_key to the operator's ~/.aws/credentials on node1 during bootstrap. Pull them over SSH and export as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step resolves before it gets to the empty profile/IAM checks. Hard-fail the step if either field is missing — better than silently watching every scenario t.Fatal on the same line. * fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually green on the tofu cluster. mulga-siv-56. 1. Witness auto-discovers the smallest registered instance type (DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded t2.micro default returned InvalidInstanceType on every Scenario A/B/C launch because the cluster doesn't register that type. DDIL_WITNESS_- INSTANCE_TYPE stays as an operator override. 2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/ SKIP lines instead of a JSON wall. results.json artifact preserved for future flake-rate dashboarding. 3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope <Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>. aws-sdk-go v1's ec2query handler rejects the latter with SerializationError and drops the code, leaving Go callers without an awserr.Code() to inspect. New SDK round-trip test in gateway_test.go guards the regression. * ci(e2e-ddil): install gotestsum from prebuilt tarball `go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose tokeninternal static-assert pegs a narrow Go window (~1.22). Both older and newer runner Go versions trip `invalid array length -delta * delta`. A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely without changing the gotestsum version pin. * ci(e2e-ddil): collect cluster logs on failure Witness `findHost` is currently failing with "could not locate QEMU host for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is visible on any node. Cluster gets destroyed during teardown, so logs have to be pulled before that runs. Adds a failure-only step that ssh's each node and bundles journalctl spinifex-*, process listings, and state dirs into the ddil-results artifact for offline triage. * fix(ddil): support tap-mode QEMU networking in witness harness findHost no longer extracts hostfwd port — production launches go through auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never hostfwd. Witness was timing out across all 3 nodes. Discover guest IP via DescribeInstances.PrivateIpAddress (populated by PrepareRunInstances on ENI auto-create) and tunnel SSH through the host node to guestIP:22. Host routes to the tap bridge subnet, so the relay works without runner-side bridge routing. WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed. Closes mulga-siv-57. * fix(ddil): SSH witness via public IP, drop unreachable tunnel path tofu-cluster bootstraps with --external-mode=pool which forces dev_networking=false → no QEMU hostfwd dev NIC and no host route to the ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to host kernel). Previous tunnel-via-host fix could find the host but couldn't dial the guest. Switch to the path shell e2e already uses: default subnet has MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external pool, and the runner reaches that IP via the same WAN subnet as the cluster nodes. No host relay required, survives Scenario C (partition only DROPs peer cluster IPs, not the runner's source IP). Workflow now scp's ~/.ssh/spinifex-key from the primary node (the counterpart to the public material bootstrap-install.sh imports) and exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against cloud-init's authorized_keys. WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed. Continues mulga-siv-57. * fix(ddil): tunnel guest SSH through host node EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet, which is OVN-routed — reachable from any cluster node, not from the off-cluster runner sitting on the DDIL_NODES WAN subnet. Restore the host-jump pattern: SSH the hosting node (already known from findHost), dial publicIP:22 over that connection, run a guest SSH handshake over the tunnel. Symmetric with how shell e2e reaches guests from inside a node. Continues mulga-siv-57. * fix(ddil): wait for witness sshd before baseline read Previous run reached the guest EIP via the host tunnel but hit 'Connection refused' — sshd hadn't bound yet. waitForRunning only checks EC2 state, which flips on QEMU start, not cloud-init completion. The OVN dnat_and_snat rule for the auto-allocated EIP is also written asynchronously, so the tunnel can land before either layer is ready. awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for up to 90s, retrying on transient transport errors (refused/reset/EOF/ timeout/no-route) and failing fast on auth errors so a bad key surfaces immediately. Called once after resolvePublicIP, before the baseline counter read. Continues mulga-siv-57. * fix(ddil): case-insensitive transient SSH error match awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh emits "ssh: rejected: connect failed (Connection refused)" with capital C — strings.Contains returned false and the helper bailed on first attempt instead of polling. Last run failed in 49s, not the 90s budget. Lowercase the error before matching, and add "connect failed" as the canonical signal for "jump host dial through tunnel failed". Continues mulga-siv-57. * fix(ddil): default guest SSH user to ec2-user Shell E2E SSHes the same ami-ubuntu-* images as ec2-user (tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu and hit 'unable to authenticate, attempted methods [none publickey]' — authorized_keys is provisioned under the AMI default account, which is ec2-user, not ubuntu. Continues mulga-siv-57. * fix(ddil): wait for witness-counter.service to seed /var/lib/counter awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables witness-counter.service runs in the final cloud-init stage — after sshd binds. Baseline read landed in the window where sshd accepts but ExecStartPre hasn't created /var/lib/counter: cat: /var/lib/counter: No such file or directory Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the full path (tunnel + SSH handshake + cat) and retries on the transport errors plus "no such file or directory". Single 2-minute budget covers cloud-init final stage on tofu-cluster hardware. Continues mulga-siv-57. * fix(systemd): drop Wants=spinifex-nats from daemon unit Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately killed spinifex-nats was getting pulled back up the moment the daemon restarted — masking the DDIL Tier 1 standalone-startup path and breaking the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact: spinifex.target's Wants= activates every spinifex unit on boot. After= remains for ordering and is non-blocking when NATS is absent. scripts/test-ddil-phase1.sh hit the same trap and worked around it with a runtime drop-in plus sentinel file; the fix here makes that workaround unnecessary for any caller that restarts the daemon while NATS is stopped. * feat(daemon): peer-health probe drives standalone mode on partition Mode() is now derived from two atomic signals — natsConnected (NATS client TCP link, flipped by the existing disconnect/reconnect callbacks) and peersReachable (any peer daemon's /health responded in the most recent probe cycle). Reports cluster iff both are up; standalone otherwise. The peer-health probe is the missing signal for DDIL Scenario C: an iptables DROP between peer IPs blocks NATS cluster-routing traffic but leaves each node's local NATS client TCP connection healthy, so nats.Conn.IsConnected() keeps returning true and onNATSDisconnect never fires. The probe dials each peer's /health over the cluster network with a 2s timeout on a 2s ticker; worst-case detection of total partition is ~6s on a 3-node cluster, well inside the 30s Scenario C budget. Single-node clusters keep peersReachable permanently true (nothing to probe) so the derived mode flips to cluster as soon as the local NATS connect succeeds. The probe runs from startLocal so the partition signal exists even before (or in the absence of) a successful NATS connect. It does not touch NATS or JetStream, so the §1e Tier 1 invariant asserted by assertNoClusterServicesInitialised is unaffected. Tests: - TestMode_RequiresBothSignals — every nats×peers combination. - TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test. - TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses. - TestMonitorPeerReachability_EndToEnd — ticker-driven false→true. Closes mulga-siv-58. * fix(daemon): peer probe falls back to local daemon port `spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml; remote-node blocks carry just `host = "<ip>"` with no daemon sub-table. So `peer.Daemon.Host` was always empty in a deployed cluster and the probe failed closed, leaving peersReachable permanently false. Tofu-cluster run 25665158964 surfaced this as a Scenario A regression: after StartNATS, natsConnected flipped true on reconnect but Mode() stayed standalone because the derived signal required peersReachable. `a_nats_kill_test.go:75: node3 did not return to cluster mode … (still reporting another mode)`. Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port when present and otherwise reuses the local daemon's port (clusters are symmetric — every node binds the same port the template renders). New TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port- available case; TestProbePeerHealth_RejectsBadAddr is updated to clear self config so it still asserts the fail-closed path. * fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit DDIL Scenario B asserts that `systemctl restart spinifex-daemon` preserves every guest VM so the new daemon can reattach via the persisted local state file. The previous setupShutdown SIGTERM handler called vmMgr.StopAll(), gracefully QMP-powering-down every guest before the daemon exited — ~90s wall-clock and zero surviving QEMU on the other side. CI run 25666369604 surfaced this as `recovered instance i-... has no live QEMU PID`. Two changes restore the Tier 1 contract: - setupShutdown no longer calls StopAll. It still flips d.shuttingDown so crash/restart handlers bail out, persists state, unsubscribes from NATS, and exits. VMs are only stopped when coordinated cluster shutdown drains them — handleShutdownDrain still calls StopAll and flips the flag before the SIGTERM path runs. Operator-visible behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs running. To stop VMs, use the cluster-shutdown topics or terminate via the AWS API. - The systemd unit gains `KillMode=process` so systemd only signals the daemon's main PID. With the default control-group mode, QEMU children were killed when systemd tore down the daemon's cgroup — defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the new fast-exit grace window. Also nil-guards natsConn.Close in the shutdown path: when a daemon restarts while NATS is unreachable (Scenario B), natsConn is nil and the close would panic. Mulga-siv-58 (reopened). * fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart) QEMU stdout/stderr were piped to daemon-side scanner goroutines. When the daemon exits (e.g. `systemctl restart spinifex-daemon`), the pipe read ends close; QEMU's next write triggers SIGPIPE and the guest dies — defeating DDIL Scenario B's "daemon restart preserves running instances" guarantee. Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr instead. The FD is dup'd into QEMU at fork, so closing it on the daemon side after Start() leaves QEMU writing into the file indefinitely. POSIX O_APPEND makes concurrent QEMU writes interleave atomically per line, so a single shared log is safe and avoids per-instance clutter in RuntimeDir. Trade-off: QEMU lines no longer land in journalctl -u spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly (path logged at instance start under the qemu_log field). mulga-siv-58 (DDIL Scenario B iteration 4) * fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's `PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()` falling through to `os.TempDir()` (=/tmp): - spinifex-daemon HOME=/var/lib/spinifex/spinifex, so `$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not exist and pidPath() returns /tmp. - Every daemon start gets a fresh private /tmp namespace. QEMU stays in the OLD namespace after the daemon dies (mount namespaces aren't changed by reparenting). The new daemon's /tmp is empty. - `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace the file doesn't exist, so PID=0. Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d, already in ReadWritePaths) and survives daemon restart. pidPath() short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and RuntimeDir() all resolve to the stable path. Both daemon incarnations and QEMU children see the same files. Revert iteration 4's pipe→file QEMU log change (commit |
||
|
Josh Siviour
|
6c54a6790c |
feat(daemon): DDIL Tier 1 — daemon-local-autonomy phase 1 (#204)
* refactor(daemon): extract Describe instance handlers to service (1b-pre phase 1)
Collapse the Describe family in daemon_handlers_instance.go to thin Pattern A
wrappers over a transport-neutral InstanceServiceImpl. This brings the read
path in line with the rest of the codebase and is the prerequisite for 1b's
local HTTP API (mulga-siv-14, daemon-local-autonomy plan).
Extracted: DescribeInstances, DescribeInstanceTypes, DescribeInstanceAttribute,
DescribeStoppedInstances, DescribeTerminatedInstances. Filter helpers
(instanceMatchesFilters, matchTagKey/Value, DescribeInstancesValidFilters,
IsInstanceVisible) move to the instance package alongside the service methods.
InstanceService gains the new methods and two dep interfaces
(ResourceCapacityProvider, StoppedInstanceStore) so the daemon's ResourceManager
and JetStreamManager can plug in without leaking concrete types. NATSInstanceService
returns a clear error for the fan-out describes (DescribeInstances,
DescribeInstanceTypes) since cluster-wide aggregation lives in the gateway's
scatter-gather helpers, not a single-node NATS request.
Behaviour changes: malformed JSON on DescribeInstanceAttribute now responds
with ValidationError (matching every other Pattern A handler) instead of
ServerInternal. Test updated.
Mutators (RunInstances, ModifyInstanceAttribute, StartStoppedInstance,
TerminateStoppedInstance, sub-handlers) remain Pattern B for now; follow-up
phases of mulga-siv-14 will collapse them.
* feat(daemon): local-first instance state persistence (1a)
Make /var/lib/spinifex/state/instance-state.json the source of truth for a
node's VMs; JetStream KV becomes a best-effort cluster cache. The daemon now
survives KV unavailability without losing instance state.
- New local_state.go: atomic write (tmp+fsync+rename), schema_version gate,
missing-file = fresh install, corrupt = refuse start.
- Daemon.WriteState writes the local file under Instances.Mu, then calls
jsManager.WriteStateBestEffort with a 1s timeout. KV failures log and
continue; clients never see "KV down" during partition.
- Daemon.LoadState reads from the local file only — no KV fallback (a fresh
node has empty KV anyway, so silent fallback would mask bugs).
- restoreInstances now returns error and Start() aborts on corrupt local
state rather than silently starting empty.
- jsManager.WriteState no longer self-locks (caller holds Mu); added
WriteStateBestEffort and shared marshalInstanceState helper.
Reconnect-driven KV resync deferred to siv-16 (1c owns the callback wiring).
Bead: mulga-siv-15
Plan: docs/development/feature/daemon-local-state-persistence.md
* fix(daemon): root local state at DataDir not BaseDir
LocalStatePath was falling back to BaseDir when DataDir was empty, putting
the file at /var/lib/spinifex/spinifex/state/instance-state.json (under the
daemon's own subdir) instead of the documented
/var/lib/spinifex/state/instance-state.json (the platform-shared root).
Drop the BaseDir fallback. Empty DataDir resolves to DefaultDataDir
(/var/lib/spinifex). Tests now set DataDir alongside BaseDir on their config
fixtures.
* feat(daemon): graceful NATS disconnect/reconnect handling (1c)
Daemon now flips a `mode` field (cluster|standalone) on NATS
disconnect/reconnect via optional callback hooks added to
utils.ConnectNATS. Reconnect bumps natsRetryCount and triggers 1a's
full local-state push to KV. NATSRequest/NATSScatterGather fail fast
with ErrClusterUnavailable when the connection is down. Gateway
short-circuits NATS-bound requests with a 503 ServiceUnavailable
carrying "cluster unavailable: NATS disconnected — check daemon
/local/status" so AWS clients see the failure mode immediately
instead of hanging on per-call timeouts.
Plan: docs/development/feature/daemon-autonomy-1c-disconnect.md
* fix(gateway): SigV4 middleware short-circuits on NATS disconnect
The 1c gateway.Request() IsConnected check (gateway.go:234) was
unreachable for any authenticated path because SigV4 auth middleware
runs first and calls IAMService.LookupAccessKey over JetStream KV. With
NATS down the lookup hangs ~5s on context.DeadlineExceeded and the
gateway returns 500 InternalError. AWS SDK retries it twice for ~18s of
user-visible latency instead of the <100ms 503 promised by the 1c plan.
Mirror the IsConnected check at the top of SigV4AuthMiddleware (after
credential parse so malformed auth headers still get the right SigV4
error code, before the IAM lookup). The catch-all in gateway.Request()
stays as defence-in-depth for any future unauthed paths.
Coverage debt picked up alongside: 1c utils.NATSRequest /
NATSScatterGather fail-fast paths, daemon Mode/onNATSDisconnect/
onNATSReconnect/localStatePath/Write/LoadState branches, and 1b-pre
InstanceServiceImpl Describe* methods (partial siv-22).
Bead: mulga-siv-23
* feat(daemon): read-only /local/* HTTP API (1b)
Wires GET /local/instances, GET /local/instances/{id}, GET /local/status
onto the existing ClusterManager TLS listener so observers can read this
node's instance state and connectivity mode while NATS is unreachable.
Also fixes a pre-existing race in WriteState: json.Marshal walked shared
*vm.VM pointers from SnapshotMap while concurrent TransitionState calls
mutated VM fields under the manager lock. Marshal now runs inside
vmMgr.View; WriteLocalStateBytes / WriteStateBytesBestEffort accept the
pre-marshalled JSON and commit lock-free. TestTransitionState_Concurrent
Transitions is race-clean across 30 iterations.
Adds Daemon.stateRevision (atomic.Uint64) bumped on every successful
local-state write, surfaced as /local/status.revision so observers can
detect changes without diffing payloads.
* feat(daemon): two-phase startup — survive NATS down (1d)
Split daemon Start() into startLocal() (HTTPS, mgmt bridge, OVS plumber,
OOM, local state load — no NATS) and startCluster() (background, infinite
retry exp backoff cap 60s for NATS connect, then JetStream + services +
restoreInstances + subscribeAll + heartbeat). Daemon never process-exits
on NATS failure; instead it stays up in standalone mode and reconnects in
the background.
Drop ExecStartPre=wait-for-nats.sh from spinifex-daemon.service — local
bootstrap no longer requires NATS to be reachable.
ConnectNATSWithRetry now supports infinite retry (WithMaxWait(0)),
context cancellation (WithContext), per-attempt error callback for the
nats_retry_count operator signal on /local/status, and a configurable
backoff cap (WithMaxRetryDelay).
bead: mulga-siv-17 (closes)
* feat(daemon): Tier 1 invariant guard for cluster-scoped service init (1e-audit)
Make the DDIL §1e-audit invariant explicit: startLocal must not initialise any
NATS-dependent or JetStream-KV-backed handle. Adds assertNoClusterServicesInitialised
as a runtime guard at the end of startLocal that fails fast if a future edit
hoists a cluster-scoped service into Phase 1, plus comment blocks naming the
invariant on both startLocal and startCluster.
Code audit confirms the existing 1d split already keeps every cluster-scoped
bucket out of Phase 1; this change pins the property so it cannot regress
silently. Tier table for the 25 daemon-process KV buckets is finalised in
docs/development/feature/daemon-local-autonomy.md §1e-audit.
bead: mulga-siv-19 (closes)
* feat(daemon): NATS observability hardening (1d-obs)
Closes the residual visibility gap left by the 1d non-blocking startup:
- Extend /local/status with kv_sync_failures, last_kv_sync_at,
last_kv_sync_error so an operator polling the local API sees prolonged
divergence between the local file and JetStream KV.
- Add a KVSyncObserver interface on JetStreamManager so the daemon can
count best-effort write outcomes without coupling the manager to
daemon state.
- Escalate utils.ConnectNATSWithRetry per-attempt log to slog.Error
(rate-limited to once per minute, with disconnected_for context) once
attempt count crosses the natsRetryEscalateAttempt threshold (~30
attempts ≈ ~30 min disconnected at the 60s backoff cap), so a stuck
NATS config surfaces in error logs instead of routine warn noise.
Prometheus metrics export deferred until a system-wide telemetry plan
(likely OTel) lands; the structured /local/status fields plus the
escalated retry log cover the Tier 1 visibility gap.
* feat(daemon): opt-in SPINIFEX_REQUIRE_NATS strict-startup mode (1d-strict)
When SPINIFEX_REQUIRE_NATS=1, startCluster's first NATS connect uses a
bounded ~30s retry and the daemon aborts on timeout. Default behaviour
(env unset) keeps the Tier 1 infinite-retry path so prod / edge nodes
still survive a missing NATS at boot. Restores pre-DDIL fail-fast UX
for dev / test / single-node deploys without flipping the prod default.
mulga-siv-44.
* test(scripts): systemd-level smoke for DDIL phase 1
Bash harness exercising daemon-local-autonomy on a real `make deploy` node:
baseline cluster mode → kill spinifex-nats → assert standalone + retry
counter advance + /local/instances served → restart NATS → assert mode flip
+ KV resync stamp → SPINIFEX_REQUIRE_NATS=1 bounded-exit. Cleanup trap
restores both services. Complements existing unit coverage by validating
real systemd wiring on port 8443.
* fix(test-ddil): daemon ClusterManager binds 4432 not 8443
8443 is predastore. Daemon's HTTPS ClusterManager (where /local/* lives)
binds [nodes.node1.daemon].host = 0.0.0.0:4432 in the deployed config.
Plan doc daemon-local-autonomy.md:112 incorrectly said 8443; actual code
uses d.config.Daemon.Host.
* fix(test-ddil): correct retry-counter site + bypass NATS Wants= for strict test
Two bugs in the v1 script:
1) nats_retry_count assertion ran during disconnect — but the counter
only bumps on disconnect→reconnect *cycles* (daemon.go:321,
onNATSReconnect). Moved assertion to step 3 (restart NATS) where
the reconnect callback fires.
2) Step 4 stopped spinifex-nats then ran 'systemctl restart
spinifex-daemon' — but the daemon unit's Wants=spinifex-nats.service
re-pulled NATS up via the Wants= chain, so the strict-mode 30s
timeout never fired. systemctl mask refuses on /etc-installed
units, and --runtime mask is shadowed by /etc. Block via a runtime
drop-in with ConditionPathExists=! pointing at a sentinel file.
Verified: all 5 checks pass on a single-node make deploy box; daemon
exits 32s after restart with SPINIFEX_REQUIRE_NATS=1 + NATS blocked.
* fix(admin): render data_dir into spinifex.toml so non-root nodes write local state
Pseudo-multinode E2E (rootless tf-user, ~/nodeN data roots) hit
"mkdir /var/lib/spinifex/state: permission denied" on every RunInstances:
Config.DataDir was empty because the template never emitted a data_dir
key, and LocalStatePath fell back to DefaultDataDir (/var/lib/spinifex).
Surfaced after
|
||
|
tomnewton-mulga
|
28ca689130 |
feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Bryn Mailer
|
3c2f57101f |
refactor: lock free write path (#179)
* fix: remove predastore filesystem backend references * fix: remove unused import * chore: bump predastore version * fix: create test buckets explicitly |
||
|
Ben Duncan
|
94600d7b84 | OVN chassis issues, new reconsile function, reboot fixed (#165) | ||
|
joshsiv-mulga
|
187b641726 |
fix: order spinifex-daemon after ovn-controller too
Symmetric to the openvswitch-switch ordering: ensures ovn-controller shutdown waits for spinifex-daemon's instance teardown, the correct general invariant for VM cleanup since logical port programming flows through OVN. PR1a flagged this as optional follow-up. Bead: mulga-siv-20.1 Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a) |
||
|
joshsiv-mulga
|
58bdcd43f5 |
fix: order spinifex-daemon after openvswitch-switch on shutdown
systemd reverses After= on shutdown so daemon now runs CleanupTapDevice while ovsdb-server still answers. Eliminates orphan OVS port left in conf.db when host reboots — root trigger for "port already exists" on post-reboot RunInstances. spinifex-vpcd.service already had this; the daemon unit was missed. Also pull openvswitch-switch + ovn-controller into spinifex.target Wants so the platform brings them up explicitly rather than relying on implicit ordering from spinifex-vpcd alone. Bead: mulga-siv-20.1 Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a) |
||
|
joshsiv-mulga
|
c01207b342 |
merge: fix/dhcp-vpcd-issues (mulga-998.b Fixes 1+1b+2+3) into feat/iso-installer
Fixes DHCP/vpcd external-path issues discovered during reboot drill on josh-banksia: - Fix 1: persist veth-wan pair via networkd .netdev/.network - Fix 1b: admin-up veth-wan-ovs via second .network unit (peer stayed LOWERLAYERDOWN) - Fix 2: loud detectBridgeMode + verifyBridgeMode sanity check - Fix 3: retrofit nat-addresses=router + network_name=external on every vpcd startup via RetrofitAllExternalLocalnetOptions Bead: mulga-siv-13 (closed). Plan: docs/development/bugs/dhcp-vpcd-issues.md. Reboot-drill evidence: smoketest 9/9, Drill 2 retrofit verified post-reboot. # Conflicts: # scripts/reset-dev-env.sh # spinifex/handlers/ec2/vpc/external_ipam.go # spinifex/services/vpcd/vpcd.go |
||
|
joshsiv-mulga
|
abd93098fc |
fix: persist veth-wan pair across reboot (mulga-998.b Fix 1)
setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing
recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs
as a port on br-ext but the peer is gone — the external path silently
breaks and vpcd falls through to direct mode.
- setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network}
in the veth branch so systemd-networkd recreates the pair on boot. Add
a single pre-case guard that rips down stale veth persistence + the OVS
port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode
transitions are idempotent (D17).
- reset-dev-env.sh: delete the netdev/network units during cleanup so
`make reset` gives a truly clean slate.
- spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service
so vpcd's sanity checks see a populated kernel + OVSDB (D5).
|
||
|
Ben Duncan
|
c908662f69 | DHCP tweaks, inflight | ||
|
Ben Duncan
|
d7c95bda5e |
feat: wire ExternalIPAM to NATS DHCP + move AF_PACKET to vpcd (mulga-991)
Replaces the sudo-dhcpcd shell-out in ObtainDHCPLease / ReleaseDHCPLease with NATS request/reply against vpc.dhcp.acquire / vpc.dhcp.release. The daemon-side IPAM passes its nats.Conn into NewExternalIPAM and threads hostname (option 12) / vendor-class (option 60) through to vpcd; ExternalIPAllocation now records LeaseExpiresUnix, DHCPServerID and HWAddr on DHCP-sourced entries. allocateFromPool is restructured to acquire the lease once up-front and rely on vpcd's idempotent Manager for CAS-retry safety; a single release fires only if every CAS attempt fails. Drops the stdout parser, the per-bridge mutex, and the parse test. Systemd: spinifex-daemon loses AF_PACKET, ProtectKernelTunables and MemoryDenyWriteExecute are restored to =yes; spinifex-vpcd gains AF_PACKET in RestrictAddressFamilies, AmbientCapabilities=CAP_NET_RAW, and CAP_NET_ADMIN / CAP_NET_RAW / CAP_NET_BIND_SERVICE in the bounding set. Net privilege footprint narrows. See docs/development/bugs/dhcp-allocate-error.md. |
||
|
Ben Duncan
|
889ef361eb |
fix: interim -T retrofit for dhcpcd EIP allocation
Stop ObtainDHCPLease from erroring on hosts where dhcpcd is already running as a daemon managing br-wan (e.g. server 2). Add -T (test-only mode) so the invocation does not collide with the host daemon's control socket on the interface. Softens ProtectKernelTunables and MemoryDenyWriteExecute in the daemon unit to accommodate dhcpcd's runtime requirements; both revert when the in-process Go DHCP client rewrite lands (mulga-989 / mulga-990 / mulga-991, see docs/development/bugs/dhcp-allocate-error.md). Interim: leaves the phantom-lease and release-socket-clobber risks active until the rewrite replaces this path entirely. |
||
|
tomnewton-mulga
|
aa1fc546bd | Merge branch 'main' into feat/iso-installer | ||
|
Ben Duncan
|
a64ee2b2da | ISO tweaks | ||
|
Julian Sommer
|
ff9f9485d2 | feat: nats TLS (#144) | ||
|
tomnewton-mulga
|
24429ee8b5 |
fix: simplify firstboot — delegate setup to setup.sh with embedded tarball
Remove the duplicated chown/ownership/symlink block from the firstboot script. setup.sh (called with INSTALL_SPINIFEX_TARBALL) already handles service users, directory ownership, sudoers, systemd unit installation, and enabling spinifex.target — the same steps as the online curl|bash install. Firstboot now follows the documented install flow directly: 1. setup.sh --tarball (air-gapped equivalent of curl|bash) 2. setup-ovn.sh --management 3. spx admin init 4. systemctl start spinifex.target Also reduce wait-for-nats.sh timeout from 90s to 30s per reviewer feedback. |
||
|
tomnewton-mulga
|
01cd3b9ab1 |
Merge commit 'b7b639084b3c2a40b97c2f543b88ad498901b77d' into feat/iso-installer
# Conflicts: # Makefile # cmd/spinifex/cmd/admin.go # go.sum |
||
|
Julian Sommer
|
0547bdd1fe | feat: loadbalancer frontend (#136) | ||
|
tomnewton-mulga
|
e8e05d6898 | chore: merge main into feat/iso-installer, resolve Makefile .PHONY conflict | ||
|
tomnewton-mulga
|
10a2a8116f |
fix: resolve firstboot service failures and improve installer UX (mulga-859)
- chown /etc/spinifex and /var/lib/spinifex to spinifex:spinifex after spx admin init so services running as the spinifex user can read configs - add Environment=HOME=/root to firstboot unit to suppress $HOME warning - increase wait-for-nats.sh timeout from 30s to 90s for slow startup - banner now writes to both /dev/tty1 and /etc/motd (visible over SSH) - add systemd.show_status=1 to GRUB cmdline for per-service boot feedback - add --removable to grub-install EFI so fallback path EFI/BOOT/BOOTX64.EFI is written (required for UTM and bare metal without NVRAM boot entries) - add root password screen and manual NIC entry to installer TUI - tee spx admin init output to /root/spinifex-credentials.txt with 0600 perms |
||
|
Julian Sommer
|
26e590716e | feat: add systemd hardening directives to all service units (#120) | ||
|
Julian Sommer
|
e27d909e8d | feat: move NBD sockets to /run/spinifex/nbd/ and enable PrivateTmp (#119) | ||
|
Julian Sommer
|
ccb6ed287d | fix: move bootstrap.json to different folder to tighten perms (#118) | ||
|
Julian Sommer
|
9a7e914d28 | feat: awsgw directory (#117) |