* test(ddil): flip Scenarios A/B/C to real assertions
Replaces the t.Skip stubs landed in #204 with the assertions described in
docs/development/improvements/ddil-e2e-test-harness.md §3:
- A NATSKill — KillNATS on node3, assert ModeStandalone within 30s,
/local/instances still serves on the isolated node, majority /health
on node1+node2 stays up, witness counters on all three nodes keep
advancing, snapshot survives reconnect.
- B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert
ModeStandalone within 30s of restart, recovered instances present
with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post
PID equality dropped — see file comment; recovery + a non-zero PID
give the same regression coverage without coupling to the daemon
unit's KillMode.
- C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep
serving /health and /local/instances, isolated node3 reports
ModeStandalone with /local/* responsive, witness counter on the
partitioned side keeps advancing, heal converges and snapshot is
preserved.
Supporting changes:
- harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore,
the deployed daemon ClusterManager binds [nodes.<id>.daemon].host.
The Phase-1 stub used a placeholder that would have blocked any
/local/* assertion against the real cluster.
- harness/witness.go — export WitnessVM.Terminate so scenarios can
clean up their counter VMs from t.Cleanup.
- scenarios/main_test.go — requireLiveCluster + launchWitnesses
helpers to keep the three flipped scenarios tight.
- TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED;
TestCoverageDrift continues to enforce alignment with the package.
Validation: built with -tags=e2e, dry-run scenarios pass, coverage
drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against
tofu cluster is the next step.
* fix(ddil): countDropRules — single iptables -S call + filter
Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs
across both chains, but iptables -S takes a single optional chain — the
second -S errors to stderr, the combined SSH buffer poisons the grep
input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by
TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState.
Switch to one `iptables -S` (lists every chain in iptables-save format),
filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences
sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin.
Error path now also surfaces the raw SSH output for the next time this
breaks somewhere unexpected.
* ci(e2e-ddil): export spinifex AWS creds for witness SDK
The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.
spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.
* fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope
Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually
green on the tofu cluster. mulga-siv-56.
1. Witness auto-discovers the smallest registered instance type
(DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded
t2.micro default returned InvalidInstanceType on every Scenario A/B/C
launch because the cluster doesn't register that type. DDIL_WITNESS_-
INSTANCE_TYPE stays as an operator override.
2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/
SKIP lines instead of a JSON wall. results.json artifact preserved for
future flake-rate dashboarding.
3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope
<Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>.
aws-sdk-go v1's ec2query handler rejects the latter with
SerializationError and drops the code, leaving Go callers without an
awserr.Code() to inspect. New SDK round-trip test in gateway_test.go
guards the regression.
* ci(e2e-ddil): install gotestsum from prebuilt tarball
`go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted
runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose
tokeninternal static-assert pegs a narrow Go window (~1.22). Both older
and newer runner Go versions trip `invalid array length -delta * delta`.
A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely
without changing the gotestsum version pin.
* ci(e2e-ddil): collect cluster logs on failure
Witness `findHost` is currently failing with "could not locate QEMU host
for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is
visible on any node. Cluster gets destroyed during teardown, so logs
have to be pulled before that runs.
Adds a failure-only step that ssh's each node and bundles
journalctl spinifex-*, process listings, and state dirs into the
ddil-results artifact for offline triage.
* fix(ddil): support tap-mode QEMU networking in witness harness
findHost no longer extracts hostfwd port — production launches go through
auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never
hostfwd. Witness was timing out across all 3 nodes.
Discover guest IP via DescribeInstances.PrivateIpAddress (populated by
PrepareRunInstances on ENI auto-create) and tunnel SSH through the host
node to guestIP:22. Host routes to the tap bridge subnet, so the relay
works without runner-side bridge routing.
WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed.
Closes mulga-siv-57.
* fix(ddil): SSH witness via public IP, drop unreachable tunnel path
tofu-cluster bootstraps with --external-mode=pool which forces
dev_networking=false → no QEMU hostfwd dev NIC and no host route to the
ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to
host kernel). Previous tunnel-via-host fix could find the host but
couldn't dial the guest.
Switch to the path shell e2e already uses: default subnet has
MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external
pool, and the runner reaches that IP via the same WAN subnet as the
cluster nodes. No host relay required, survives Scenario C (partition
only DROPs peer cluster IPs, not the runner's source IP).
Workflow now scp's ~/.ssh/spinifex-key from the primary node (the
counterpart to the public material bootstrap-install.sh imports) and
exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against
cloud-init's authorized_keys.
WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed.
Continues mulga-siv-57.
* fix(ddil): tunnel guest SSH through host node
EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet,
which is OVN-routed — reachable from any cluster node, not from the
off-cluster runner sitting on the DDIL_NODES WAN subnet.
Restore the host-jump pattern: SSH the hosting node (already known from
findHost), dial publicIP:22 over that connection, run a guest SSH
handshake over the tunnel. Symmetric with how shell e2e reaches guests
from inside a node.
Continues mulga-siv-57.
* fix(ddil): wait for witness sshd before baseline read
Previous run reached the guest EIP via the host tunnel but hit
'Connection refused' — sshd hadn't bound yet. waitForRunning only
checks EC2 state, which flips on QEMU start, not cloud-init completion.
The OVN dnat_and_snat rule for the auto-allocated EIP is also written
asynchronously, so the tunnel can land before either layer is ready.
awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for
up to 90s, retrying on transient transport errors (refused/reset/EOF/
timeout/no-route) and failing fast on auth errors so a bad key surfaces
immediately. Called once after resolvePublicIP, before the baseline
counter read.
Continues mulga-siv-57.
* fix(ddil): case-insensitive transient SSH error match
awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh
emits "ssh: rejected: connect failed (Connection refused)" with capital
C — strings.Contains returned false and the helper bailed on first
attempt instead of polling. Last run failed in 49s, not the 90s budget.
Lowercase the error before matching, and add "connect failed" as the
canonical signal for "jump host dial through tunnel failed".
Continues mulga-siv-57.
* fix(ddil): default guest SSH user to ec2-user
Shell E2E SSHes the same ami-ubuntu-* images as ec2-user
(tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu
and hit 'unable to authenticate, attempted methods [none publickey]' —
authorized_keys is provisioned under the AMI default account, which is
ec2-user, not ubuntu.
Continues mulga-siv-57.
* fix(ddil): wait for witness-counter.service to seed /var/lib/counter
awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables
witness-counter.service runs in the final cloud-init stage — after sshd
binds. Baseline read landed in the window where sshd accepts but
ExecStartPre hasn't created /var/lib/counter:
cat: /var/lib/counter: No such file or directory
Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the
full path (tunnel + SSH handshake + cat) and retries on the transport
errors plus "no such file or directory". Single 2-minute budget covers
cloud-init final stage on tofu-cluster hardware.
Continues mulga-siv-57.
* fix(systemd): drop Wants=spinifex-nats from daemon unit
Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately
killed spinifex-nats was getting pulled back up the moment the daemon
restarted — masking the DDIL Tier 1 standalone-startup path and breaking
the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact:
spinifex.target's Wants= activates every spinifex unit on boot. After=
remains for ordering and is non-blocking when NATS is absent.
scripts/test-ddil-phase1.sh hit the same trap and worked around it with a
runtime drop-in plus sentinel file; the fix here makes that workaround
unnecessary for any caller that restarts the daemon while NATS is stopped.
* feat(daemon): peer-health probe drives standalone mode on partition
Mode() is now derived from two atomic signals — natsConnected (NATS client
TCP link, flipped by the existing disconnect/reconnect callbacks) and
peersReachable (any peer daemon's /health responded in the most recent
probe cycle). Reports cluster iff both are up; standalone otherwise.
The peer-health probe is the missing signal for DDIL Scenario C: an iptables
DROP between peer IPs blocks NATS cluster-routing traffic but leaves each
node's local NATS client TCP connection healthy, so nats.Conn.IsConnected()
keeps returning true and onNATSDisconnect never fires. The probe dials each
peer's /health over the cluster network with a 2s timeout on a 2s ticker;
worst-case detection of total partition is ~6s on a 3-node cluster, well
inside the 30s Scenario C budget. Single-node clusters keep peersReachable
permanently true (nothing to probe) so the derived mode flips to cluster as
soon as the local NATS connect succeeds.
The probe runs from startLocal so the partition signal exists even before
(or in the absence of) a successful NATS connect. It does not touch NATS or
JetStream, so the §1e Tier 1 invariant asserted by
assertNoClusterServicesInitialised is unaffected.
Tests:
- TestMode_RequiresBothSignals — every nats×peers combination.
- TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test.
- TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses.
- TestMonitorPeerReachability_EndToEnd — ticker-driven false→true.
Closes mulga-siv-58.
* fix(daemon): peer probe falls back to local daemon port
`spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml;
remote-node blocks carry just `host = "<ip>"` with no daemon sub-table.
So `peer.Daemon.Host` was always empty in a deployed cluster and the
probe failed closed, leaving peersReachable permanently false.
Tofu-cluster run 25665158964 surfaced this as a Scenario A regression:
after StartNATS, natsConnected flipped true on reconnect but Mode()
stayed standalone because the derived signal required peersReachable.
`a_nats_kill_test.go:75: node3 did not return to cluster mode … (still
reporting another mode)`.
Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port
when present and otherwise reuses the local daemon's port (clusters are
symmetric — every node binds the same port the template renders). New
TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port-
available case; TestProbePeerHealth_RejectsBadAddr is updated to clear
self config so it still asserts the fail-closed path.
* fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit
DDIL Scenario B asserts that `systemctl restart spinifex-daemon`
preserves every guest VM so the new daemon can reattach via the
persisted local state file. The previous setupShutdown SIGTERM handler
called vmMgr.StopAll(), gracefully QMP-powering-down every guest before
the daemon exited — ~90s wall-clock and zero surviving QEMU on the
other side. CI run 25666369604 surfaced this as
`recovered instance i-... has no live QEMU PID`.
Two changes restore the Tier 1 contract:
- setupShutdown no longer calls StopAll. It still flips d.shuttingDown
so crash/restart handlers bail out, persists state, unsubscribes from
NATS, and exits. VMs are only stopped when coordinated cluster
shutdown drains them — handleShutdownDrain still calls StopAll and
flips the flag before the SIGTERM path runs. Operator-visible
behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs
running. To stop VMs, use the cluster-shutdown topics or terminate
via the AWS API.
- The systemd unit gains `KillMode=process` so systemd only signals the
daemon's main PID. With the default control-group mode, QEMU
children were killed when systemd tore down the daemon's cgroup —
defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the
new fast-exit grace window.
Also nil-guards natsConn.Close in the shutdown path: when a daemon
restarts while NATS is unreachable (Scenario B), natsConn is nil and
the close would panic.
Mulga-siv-58 (reopened).
* fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart)
QEMU stdout/stderr were piped to daemon-side scanner goroutines.
When the daemon exits (e.g. `systemctl restart spinifex-daemon`),
the pipe read ends close; QEMU's next write triggers SIGPIPE and
the guest dies — defeating DDIL Scenario B's "daemon restart
preserves running instances" guarantee.
Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr
instead. The FD is dup'd into QEMU at fork, so closing it on the
daemon side after Start() leaves QEMU writing into the file
indefinitely. POSIX O_APPEND makes concurrent QEMU writes
interleave atomically per line, so a single shared log is safe and
avoids per-instance clutter in RuntimeDir.
Trade-off: QEMU lines no longer land in journalctl -u
spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly
(path logged at instance start under the qemu_log field).
mulga-siv-58 (DDIL Scenario B iteration 4)
* fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround
Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU
ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's
`PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()`
falling through to `os.TempDir()` (=/tmp):
- spinifex-daemon HOME=/var/lib/spinifex/spinifex, so
`$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not
exist and pidPath() returns /tmp.
- Every daemon start gets a fresh private /tmp namespace. QEMU stays
in the OLD namespace after the daemon dies (mount namespaces aren't
changed by reparenting). The new daemon's /tmp is empty.
- `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace
the file doesn't exist, so PID=0.
Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon
unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d,
already in ReadWritePaths) and survives daemon restart. pidPath()
short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and
RuntimeDir() all resolve to the stable path. Both daemon incarnations
and QEMU children see the same files.
Revert iteration 4's pipe→file QEMU log change (commit 8a9a6f9f).
SIGPIPE was never killing QEMU, and the file-redirect dropped QEMU
lines from `journalctl -u spinifex-daemon` for no gain.
mulga-siv-58 (DDIL Scenario B iteration 5)
* feat(daemon): siv-60 heal reconcile — phase 1 (probe-heal trigger) (#211)
* feat(daemon): wire reconcileOnHeal on peer-probe heal edge (siv-60 phase 1)
Scenario C heal triggered zero resync: the local NATS client stayed
connected to its loopback server throughout the partition, so
onNATSReconnect never fired. The peer-probe false→true transition was
the only heal signal but had no resync action wired to it.
Adds reconcileOnHeal, called from both edges and coalesced via a
reconciling atomic.Bool so the two callbacks landing on the same heal
don't double-push. For phase 1 the body is the existing WriteState
(re-asserts our per-node row in the instance-state KV). Phase 2 will
extend it to pull KV → merge → push for the stopped/terminated buckets
and re-fire per-node bootstrap.
Plan: docs/development/feature/ddil-heal-reconcile.md
* feat(daemon): siv-60 phase 2 — idempotent default-VPC bootstrap re-fire on heal
Calls ensureDefaultVPCInfrastructure() from reconcileOnHeal after the
WriteState push. Idempotent: each branch (default VPC lookup, IGW
attach, default-SG create, default route insert) checks for existing
infrastructure before creating. Covers the failure mode where
startCluster ran during partition and the initial creates returned
errors against an unreachable NATS — without a heal-time retry the
cluster would be stuck without a default VPC until the next daemon
restart.
Scope note: the bead's "pull KV → merge → push" assumed the daemon
cached peer-node running state, stopped instances, terminated
instances, and VPC/IGW/SG state locally. The codebase audit (see plan
doc) confirms it doesn't — every consumer reads its KV bucket fresh
per request, and vmMgr only tracks VMs running on this node. The
bootstrap re-fire is the one heal-time action that has a real
in-memory target. Watchdog sweep is unnecessary because
vm.Manager.CrashHandler is event-driven from cmd.Wait().
* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal (siv-60 follow-up) (#212)
* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal
DDIL Scenario C regressed at the witness-SSH step after siv-60
landed in #211. Root cause: the first peer-probe tick on multi-node
startup flips peersReachable false→true (its zero value is false), so
reconcileOnHeal fires concurrently with startCluster's own
ensureDefaultVPCInfrastructure call. The Describe→Create→Attach
sequence is not cluster-wide singleton, so multiple daemons + double
calls per daemon raced and produced duplicate IGW attachments,
corrupting default-VPC routing — which broke the witness VM's tunnel
SSH path once Scenario C's partition was installed.
Phase 1 (the heal trigger + coalescing) is the meaningful fix and
stays. Phase 2's bootstrap re-fire needs leader election or a
debounce against the startup-time path before being re-introduced;
leaving an inline NOTE so the next attempt has the context.
Failed run: actions/runs/25706585637 (Scenario C, attempts 1+2).
Last green DDIL run: 25671044714 (pre-merge, SHA 40e3d847).
* fix(daemon): init peersReachable=true to suppress startup heal edge
DDIL Scenario C kept flaking at attempt-1 even after dropping the
bootstrap re-fire. Remaining trigger: peersReachable starts as the
zero value (false) on multi-node clusters, and the first probePeersOnce
tick flips it false→true. That counts as a "heal" edge under siv-60
phase 1, so reconcileOnHeal fires at boot — concurrent with
startCluster's bootstrap — perturbing timing in a way that the
harness's Scenario C exposes (witness sshd handshake EOF mid-test).
Initialise peersReachable optimistically. Mode() also requires
natsConnected (starts false), so the optimistic peer init can never
falsely report cluster mode — but it does eliminate the spurious
startup edge. Real partition→heal transitions still fire the trigger
exactly as designed: true→false on partition, false→true on heal.
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* feat(viperblockd): retry LoadState on transient backend errors
ebs.mount used to call vb.LoadState once and treat any failure as fatal.
When predastore was still warming up after a coordinated daemon restart,
the backend Read came back with a network/timeout error, viperblock
folded it into "block size is 0 in both local and backend state", and
the daemon's recovery path then ran the cleanup branch — destroying
DeleteOnTermination=true volumes (mulga-siv-25).
Wrap LoadState in retryLoadState: 5 attempts, 200ms * 1.5^n backoff
(total worst-case ~3.7s, well inside the 30s ebs.mount NATS deadline).
Retry only on viperblock.ErrStateBackendUnavailable; ErrStateNotFound
and any unclassified error fail fast so genuine problems are still
visible.
Retry loop is parameterised over sleep and loadFn so the retry policy
itself is unit-tested without spinning up a real VB.
* chore: bump viperblock to v1.2.2-0.20260511064619-1aa54f0dc1f6
Pulls in the ErrStateNotFound / ErrStateBackendUnavailable sentinels
introduced by mulgadc/viperblock#11 (merged on dev). Required for the
retry wrapper added in the previous commit to compile against the
public viperblock module.
Stops the P0 data-loss path where `systemctl restart spinifex-daemon`
destroys user volumes flagged `DeleteOnTermination=true`.
Previously, when daemon recovery couldn't relaunch (Run error) or
reconnect (QMP attach failure) to a previously-running instance, the
code called `MarkFailed` — which transitions to `terminated`, runs the
full destructive cleanup (`terminateCleanup`), and deletes volumes per
`DeleteOnTermination`. A benign operator action (daemon restart) was
treated identically to a user-initiated `ec2.TerminateInstances`.
New `MarkRecoveryFailed` transitions to `StateError` with a
`Server.RecoveryFailed` StateReason, runs only the non-destructive
`stopCleanup` chain (graceful unmount + tap teardown + GPU release +
resource deallocation), and persists the running set. Volumes, public
IPs, ENIs, and placement-group membership are preserved so the operator
can either retry recovery or issue an explicit terminate (which will
then honour DoT as normal). The instance stays in the local map and
`OnInstanceDown` is NOT fired, keeping `ec2.cmd.<id>` live so operator
commands can still reach this node.
`classifyRestoredInstances` now skips relaunch + resource re-allocation
for instances already in `StateError`, so a recovery-failed instance
stays put across subsequent daemon restarts until the operator acts.
Medium-term (viperblock "block size 0" hard-vs-soft classification +
mount retry-with-backoff) and long-term (audit of every non-user-
initiated cleanup caller) tiers remain open on the siv-25 plan.
* refactor(daemon): extract ModifyInstanceAttribute to service (1b-pre phase 2a)
Collapse handleEC2ModifyInstanceAttribute to a thin Pattern A wrapper over
InstanceServiceImpl.ModifyInstanceAttribute, mirroring the Describe extraction
in 719ac63 (phase 1). The InstanceService interface and StoppedInstanceStore
each gain one method (ModifyInstanceAttribute, WriteStoppedInstance); the
daemon's JetStreamManager already implements WriteStoppedInstance, so no
production code outside the daemon handler changes.
Logic-level tests move from daemon/daemon_handlers_test.go to
handlers/ec2/instance/service_impl_test.go (12 cases against an in-memory
fakeStoppedStore). The daemon keeps one NATS round-trip wrapper smoke test.
The error-injection cases in daemon_handlers_instance_test.go remain in place
but now require daemonWithFakeStateStore to rewire d.instanceService with
the fake store — the previous `d.stateStore = store` swap no longer reaches
the new code path.
No behaviour change: SourceDestCheck stays a state-agnostic no-op, the
ownership/visibility check still gates after the stopped-state gate, and the
success response payload remains `{}`.
* refactor(daemon): extract TerminateStoppedInstance to service (1b-pre phase 2b)
Collapses ~140 lines of daemon termination logic into a one-liner
handleNATSRequest wrapper plus an InstanceServiceImpl method, matching
the Pattern A precedent from phase 1 (Describes) and phase 2a
(ModifyInstanceAttribute). Adds VolumeDeleter, ENIDeleter, and
PublicIPReleaser dep interfaces (wired via SetTerminationDeps) so the
service can clean up volumes, ENI, and EIP without depending on daemon
internals.
Hoists publishNATEvent off the daemon into utils.PublishNATEvent so the
service can drive vpc.delete-nat directly. Removes the
terminateStoppedInstanceRequest daemon-local payload in favour of the
exported TerminateStoppedInstanceInput on the service interface.
Tests: 13 service-level cases against an in-memory fake (missing ID,
nil store, load error, not found, not stopped, cross-tenant,
happy path, terminated-write failure aborts, retry-once delete,
user volume deleted, no-deleter graceful skip, internal volumes via
NATS, public IP release, ENI delete). The five JetStream-backed
daemon Terminate tests collapse to one wrapper smoke
(WritesToTerminatedKV); the six error-injection tests in
daemon_handlers_instance_test.go continue to drive the wrapper through
fakeStateStore for wire-format coverage.
Refs mulga-siv-21.
* refactor(daemon): extract StartStoppedInstance to service (1b-pre phase 2c)
Daemon's handleEC2StartStoppedInstance collapses to a Pattern A handleNATSRequest
wrapper over InstanceServiceImpl.StartStoppedInstance. Promotes ResourceCapacityProvider
to InstanceTypeAllocator with Allocate/Deallocate/CanAllocate/InstanceTypes, and adds
a GPUClaimer interface returning (pciAddress, xvgaEnabled, err) so the instance package
stays free of the gpu package. Daemon ResourceManager gains exported wrappers around
its internal allocator methods; a daemonGPUClaimer adapter wires gpu.Manager + the
GPU model overrides through SetGPUClaimer.
No behaviour change. New service-level tests cover the moved logic; existing
daemon integration tests continue to validate the end-to-end path through the
wrapper.
* ci(e2e-ddil): export spinifex AWS creds for witness SDK
The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.
spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.
* Revert "ci(e2e-ddil): export spinifex AWS creds for witness SDK"
This reverts commit 5b890162a6.
* refactor(daemon): extract RunInstances + ec2.cmd handlers to service (1b-pre phase 2d)
Final sub-phase of mulga-siv-21. Moves the remaining instance mutators
from spinifex/daemon/daemon_handlers_instance.go into InstanceServiceImpl:
- PrepareRunInstances: input validation, capacity allocation, per-instance
RunInstance + ENI auto-create + public IP assignment.
- LaunchRunInstances: volume preparation, GPU claim, vmMgr.Run loop.
- RunInstances: synchronous Prepare+Launch convenience wrapper used by
the InstanceService interface for non-daemon callers.
- RebootInstance, StartInstance, StopOrTerminateInstance: extract the
three ec2.cmd sub-handlers; daemon.handleEC2Events shrinks to a
dispatcher that looks up the VM and calls the matching service method.
The daemon's RunInstances handler is now a thin orchestrator that splits
the call into Prepare and Launch around the early NATS respond so the
original respond-then-launch timing is preserved (AWS clients see the
reservation before volumes prepare and VMs launch).
New service deps wired via SetRunInstancesDeps:
- AMIMetaLoader (handlers/ec2/image.ImageServiceImpl)
- KeyPairValidator (handlers/ec2/key.KeyServiceImpl)
- ENICreator + SubnetInfo (new daemonENICreator adapter over VPCServiceImpl
to avoid an instance↔vpc cyclic import)
- PublicIPAllocator (handlers/ec2/vpc.ExternalIPAM)
No behaviour change. Tests retarget the service directly for the new
error paths; existing daemon-side RunInstances handler tests continue to
pass via the wrapper.
* test(instance): add LaunchRunInstances + setter coverage for phase 2d
* refactor(daemon): extract Describe instance handlers to service (1b-pre phase 1)
Collapse the Describe family in daemon_handlers_instance.go to thin Pattern A
wrappers over a transport-neutral InstanceServiceImpl. This brings the read
path in line with the rest of the codebase and is the prerequisite for 1b's
local HTTP API (mulga-siv-14, daemon-local-autonomy plan).
Extracted: DescribeInstances, DescribeInstanceTypes, DescribeInstanceAttribute,
DescribeStoppedInstances, DescribeTerminatedInstances. Filter helpers
(instanceMatchesFilters, matchTagKey/Value, DescribeInstancesValidFilters,
IsInstanceVisible) move to the instance package alongside the service methods.
InstanceService gains the new methods and two dep interfaces
(ResourceCapacityProvider, StoppedInstanceStore) so the daemon's ResourceManager
and JetStreamManager can plug in without leaking concrete types. NATSInstanceService
returns a clear error for the fan-out describes (DescribeInstances,
DescribeInstanceTypes) since cluster-wide aggregation lives in the gateway's
scatter-gather helpers, not a single-node NATS request.
Behaviour changes: malformed JSON on DescribeInstanceAttribute now responds
with ValidationError (matching every other Pattern A handler) instead of
ServerInternal. Test updated.
Mutators (RunInstances, ModifyInstanceAttribute, StartStoppedInstance,
TerminateStoppedInstance, sub-handlers) remain Pattern B for now; follow-up
phases of mulga-siv-14 will collapse them.
* feat(daemon): local-first instance state persistence (1a)
Make /var/lib/spinifex/state/instance-state.json the source of truth for a
node's VMs; JetStream KV becomes a best-effort cluster cache. The daemon now
survives KV unavailability without losing instance state.
- New local_state.go: atomic write (tmp+fsync+rename), schema_version gate,
missing-file = fresh install, corrupt = refuse start.
- Daemon.WriteState writes the local file under Instances.Mu, then calls
jsManager.WriteStateBestEffort with a 1s timeout. KV failures log and
continue; clients never see "KV down" during partition.
- Daemon.LoadState reads from the local file only — no KV fallback (a fresh
node has empty KV anyway, so silent fallback would mask bugs).
- restoreInstances now returns error and Start() aborts on corrupt local
state rather than silently starting empty.
- jsManager.WriteState no longer self-locks (caller holds Mu); added
WriteStateBestEffort and shared marshalInstanceState helper.
Reconnect-driven KV resync deferred to siv-16 (1c owns the callback wiring).
Bead: mulga-siv-15
Plan: docs/development/feature/daemon-local-state-persistence.md
* fix(daemon): root local state at DataDir not BaseDir
LocalStatePath was falling back to BaseDir when DataDir was empty, putting
the file at /var/lib/spinifex/spinifex/state/instance-state.json (under the
daemon's own subdir) instead of the documented
/var/lib/spinifex/state/instance-state.json (the platform-shared root).
Drop the BaseDir fallback. Empty DataDir resolves to DefaultDataDir
(/var/lib/spinifex). Tests now set DataDir alongside BaseDir on their config
fixtures.
* feat(daemon): graceful NATS disconnect/reconnect handling (1c)
Daemon now flips a `mode` field (cluster|standalone) on NATS
disconnect/reconnect via optional callback hooks added to
utils.ConnectNATS. Reconnect bumps natsRetryCount and triggers 1a's
full local-state push to KV. NATSRequest/NATSScatterGather fail fast
with ErrClusterUnavailable when the connection is down. Gateway
short-circuits NATS-bound requests with a 503 ServiceUnavailable
carrying "cluster unavailable: NATS disconnected — check daemon
/local/status" so AWS clients see the failure mode immediately
instead of hanging on per-call timeouts.
Plan: docs/development/feature/daemon-autonomy-1c-disconnect.md
* fix(gateway): SigV4 middleware short-circuits on NATS disconnect
The 1c gateway.Request() IsConnected check (gateway.go:234) was
unreachable for any authenticated path because SigV4 auth middleware
runs first and calls IAMService.LookupAccessKey over JetStream KV. With
NATS down the lookup hangs ~5s on context.DeadlineExceeded and the
gateway returns 500 InternalError. AWS SDK retries it twice for ~18s of
user-visible latency instead of the <100ms 503 promised by the 1c plan.
Mirror the IsConnected check at the top of SigV4AuthMiddleware (after
credential parse so malformed auth headers still get the right SigV4
error code, before the IAM lookup). The catch-all in gateway.Request()
stays as defence-in-depth for any future unauthed paths.
Coverage debt picked up alongside: 1c utils.NATSRequest /
NATSScatterGather fail-fast paths, daemon Mode/onNATSDisconnect/
onNATSReconnect/localStatePath/Write/LoadState branches, and 1b-pre
InstanceServiceImpl Describe* methods (partial siv-22).
Bead: mulga-siv-23
* feat(daemon): read-only /local/* HTTP API (1b)
Wires GET /local/instances, GET /local/instances/{id}, GET /local/status
onto the existing ClusterManager TLS listener so observers can read this
node's instance state and connectivity mode while NATS is unreachable.
Also fixes a pre-existing race in WriteState: json.Marshal walked shared
*vm.VM pointers from SnapshotMap while concurrent TransitionState calls
mutated VM fields under the manager lock. Marshal now runs inside
vmMgr.View; WriteLocalStateBytes / WriteStateBytesBestEffort accept the
pre-marshalled JSON and commit lock-free. TestTransitionState_Concurrent
Transitions is race-clean across 30 iterations.
Adds Daemon.stateRevision (atomic.Uint64) bumped on every successful
local-state write, surfaced as /local/status.revision so observers can
detect changes without diffing payloads.
* feat(daemon): two-phase startup — survive NATS down (1d)
Split daemon Start() into startLocal() (HTTPS, mgmt bridge, OVS plumber,
OOM, local state load — no NATS) and startCluster() (background, infinite
retry exp backoff cap 60s for NATS connect, then JetStream + services +
restoreInstances + subscribeAll + heartbeat). Daemon never process-exits
on NATS failure; instead it stays up in standalone mode and reconnects in
the background.
Drop ExecStartPre=wait-for-nats.sh from spinifex-daemon.service — local
bootstrap no longer requires NATS to be reachable.
ConnectNATSWithRetry now supports infinite retry (WithMaxWait(0)),
context cancellation (WithContext), per-attempt error callback for the
nats_retry_count operator signal on /local/status, and a configurable
backoff cap (WithMaxRetryDelay).
bead: mulga-siv-17 (closes)
* feat(daemon): Tier 1 invariant guard for cluster-scoped service init (1e-audit)
Make the DDIL §1e-audit invariant explicit: startLocal must not initialise any
NATS-dependent or JetStream-KV-backed handle. Adds assertNoClusterServicesInitialised
as a runtime guard at the end of startLocal that fails fast if a future edit
hoists a cluster-scoped service into Phase 1, plus comment blocks naming the
invariant on both startLocal and startCluster.
Code audit confirms the existing 1d split already keeps every cluster-scoped
bucket out of Phase 1; this change pins the property so it cannot regress
silently. Tier table for the 25 daemon-process KV buckets is finalised in
docs/development/feature/daemon-local-autonomy.md §1e-audit.
bead: mulga-siv-19 (closes)
* feat(daemon): NATS observability hardening (1d-obs)
Closes the residual visibility gap left by the 1d non-blocking startup:
- Extend /local/status with kv_sync_failures, last_kv_sync_at,
last_kv_sync_error so an operator polling the local API sees prolonged
divergence between the local file and JetStream KV.
- Add a KVSyncObserver interface on JetStreamManager so the daemon can
count best-effort write outcomes without coupling the manager to
daemon state.
- Escalate utils.ConnectNATSWithRetry per-attempt log to slog.Error
(rate-limited to once per minute, with disconnected_for context) once
attempt count crosses the natsRetryEscalateAttempt threshold (~30
attempts ≈ ~30 min disconnected at the 60s backoff cap), so a stuck
NATS config surfaces in error logs instead of routine warn noise.
Prometheus metrics export deferred until a system-wide telemetry plan
(likely OTel) lands; the structured /local/status fields plus the
escalated retry log cover the Tier 1 visibility gap.
* feat(daemon): opt-in SPINIFEX_REQUIRE_NATS strict-startup mode (1d-strict)
When SPINIFEX_REQUIRE_NATS=1, startCluster's first NATS connect uses a
bounded ~30s retry and the daemon aborts on timeout. Default behaviour
(env unset) keeps the Tier 1 infinite-retry path so prod / edge nodes
still survive a missing NATS at boot. Restores pre-DDIL fail-fast UX
for dev / test / single-node deploys without flipping the prod default.
mulga-siv-44.
* test(scripts): systemd-level smoke for DDIL phase 1
Bash harness exercising daemon-local-autonomy on a real `make deploy` node:
baseline cluster mode → kill spinifex-nats → assert standalone + retry
counter advance + /local/instances served → restart NATS → assert mode flip
+ KV resync stamp → SPINIFEX_REQUIRE_NATS=1 bounded-exit. Cleanup trap
restores both services. Complements existing unit coverage by validating
real systemd wiring on port 8443.
* fix(test-ddil): daemon ClusterManager binds 4432 not 8443
8443 is predastore. Daemon's HTTPS ClusterManager (where /local/* lives)
binds [nodes.node1.daemon].host = 0.0.0.0:4432 in the deployed config.
Plan doc daemon-local-autonomy.md:112 incorrectly said 8443; actual code
uses d.config.Daemon.Host.
* fix(test-ddil): correct retry-counter site + bypass NATS Wants= for strict test
Two bugs in the v1 script:
1) nats_retry_count assertion ran during disconnect — but the counter
only bumps on disconnect→reconnect *cycles* (daemon.go:321,
onNATSReconnect). Moved assertion to step 3 (restart NATS) where
the reconnect callback fires.
2) Step 4 stopped spinifex-nats then ran 'systemctl restart
spinifex-daemon' — but the daemon unit's Wants=spinifex-nats.service
re-pulled NATS up via the Wants= chain, so the strict-mode 30s
timeout never fired. systemctl mask refuses on /etc-installed
units, and --runtime mask is shadowed by /etc. Block via a runtime
drop-in with ConditionPathExists=! pointing at a sentinel file.
Verified: all 5 checks pass on a single-node make deploy box; daemon
exits 32s after restart with SPINIFEX_REQUIRE_NATS=1 + NATS blocked.
* fix(admin): render data_dir into spinifex.toml so non-root nodes write local state
Pseudo-multinode E2E (rootless tf-user, ~/nodeN data roots) hit
"mkdir /var/lib/spinifex/state: permission denied" on every RunInstances:
Config.DataDir was empty because the template never emitted a data_dir
key, and LocalStatePath fell back to DefaultDataDir (/var/lib/spinifex).
Surfaced after 8eb9c5fa dropped the BaseDir fallback in LocalStatePath —
previously SPINIFEX_BASE_DIR=$DATA_DIR/spinifex/ (set by start-dev.sh)
masked the missing data_dir.
ConfigSettings.DataDir is already populated to spxRoot at admin
init/join; render it under [nodes.X] so it actually reaches the daemon.
reconcileIGW was non-idempotent and gated on "external switch missing",
so once a prior pass left the ext switch + LRP in place without the
localnet LSP, every subsequent vpcd start silently skipped the function
and the partial topology persisted forever (br-ext never got its patch
port; external traffic failed).
Convert each step to Get-then-Create idempotent and drop the cascade
rollbacks — partial state is now safe to leave for the next pass to
heal. Add FindStaticRoute so the default route is deduped instead of
duplicated on every retry. Drop the ext-switch gate in Reconcile and
ReconcileFromKV; rename IGWsCreated to IGWsReconciled to reflect the
new "called" semantics.
NAT rules are purely event-driven — only AssociateAddress creates them via
vpc.add-nat. No startup reconcile path means any OVN NB DB state drift
silently breaks EIP inbound traffic after a node reboot.
Add Step 5 to ReconcileFromKV that reads associated EIPs from the
spinifex-vpc-elastic-ips KV bucket and ensures each has a dnat_and_snat rule
in OVN NB DB. Correct rules are skipped (idempotent). Stale rules (wrong
LogicalIP) are replaced. Missing rules are created.
Also adds FindNATByExternalIP to OVNClient interface + LiveOVNClient/Mock
implementations, and three new tests covering: missing rule creation,
idempotent skip on correct rule, and stale rule replacement.
The internal-ALB approach was wrong on two counts:
- ALB_ENI_MAC is the OVN VPC ENI MAC (lives on br-int), never appears
in br-mgmt neighbour table, so the mgmt IP lookup always fails.
- Switching to internal masked the real bug: OVN EIP DNAT not restoring
properly after a host reboot, which is exactly what this test should catch.
Restore internet-facing ALB and EIP-based traffic bursts. The post-reboot
traffic failure (Phase 8.6) is the correct signal for the outstanding OVN
EIP DNAT persistence bug.
Keep the Phase 8.3 LaunchTime fix: two-hop SSH to VPC private IPs doesn't
work in single-node macvlan (host has no route to VPC subnets). LaunchTime
from describe-instances is fast and topology-independent.
Two changes to fix cell-18 (single/static/debian-13/reboot-single):
1. Switch ALB to --scheme internal.
Internet-facing ALBs work pre-reboot on single-node but post-reboot the
OVN EIP DNAT (nat-addresses=router GARP) is not re-established in time,
causing 100% curl timeouts. Internal ALB routes traffic via br-mgmt using
the mgmt IP discovered from the ALB ENI MAC in the br-mgmt neighbour table,
bypassing OVN external NAT entirely.
2. Replace Phase 8.3 guest SSH uptime check with LaunchTime assertion.
Two-hop SSH to VPC private IPs fails in single-node macvlan: the host has
no route to VPC subnets. LaunchTime from describe-instances is reset by the
daemon on QEMU relaunch, so LaunchTime >= REBOOT_START reliably confirms
recovery without network reachability to the guest.
Separate from normal e2e. Checks out spinifex, checks out mulga scripts
(sparse: iso-builder + baremetal), builds spx + test tar via
bm-build-artifacts.sh, builds PXE artifacts (squashfs/initrd/vmlinuz)
via iso-builder with persistent cache on banksia, then runs
run-baremetal-e2e.sh against the single bare-metal node.
In the PXE path SPINIFEX_INSTALL_CALLBACK arrives on the iPXE kernel
cmdline but was never threaded into the firstboot script, so firstboot
never curled the controller's /done endpoint. The flag stayed on disk,
the node re-entered the install loop on the next PXE boot.
Fix: read SPINIFEX_INSTALL_CALLBACK from the live-environment env in
toFirstbootConfig and embed a best-effort curl into the generated
firstboot script, fired once after the success marker is written.
Interactive and USB-stick installs are unaffected (env var absent
-> callbackBlock is empty, no curl injected).
NBDKit exiting within 1s was only treated as an error if exit code != 0.
A clean exit (code=0) fell through to Mounted=true, leaving QEMU with a
dead NBD socket and VMs that never boot post-reboot.
Treat any premature exit as failure regardless of exit code.
manager_deps_test.go: drop TestDeps_VolumeMounterSatisfiesInterface,
TestDeps_StateStoreSatisfiesInterface, TestDeps_StateStoreErrorPropagates,
TestSetDeps_OverridesDeps, TestManagerHooks_InitiallyNil — they assert a
fake recorded its own input, that a fake returns an error literally
constructed in the test, or Go zero-value/direct-assignment semantics.
Compile-time interface checks already prove satisfaction.
state_test.go: replace TestEC2StateCodes_AllStatesHaveMapping and
TestEC2StateCodes_CorrectValues — both compared the production map to
its own literal. Now assert against an independent AWS contract using
ec2.InstanceStateName* SDK enum constants paired with the
AWS-documented numeric codes; a typo in production no longer
replicates into the test.
watchdog_test.go: replace TestStartPendingWatchdog_CtxCancelStopsGoroutine's
time.Sleep(10ms) "assertion" with goleak.VerifyNone. The previous form
passed even if the goroutine never returned on cancel — the harness
reaped it on test exit.
* feat: add GPU discovery package (mulga-992)
Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.
* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)
Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.
* feat: add G5 instance type generator with GPU model support (mulga-994)
- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter
* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)
- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)
Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.
Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* feat: spx admin gpu status/enable/disable commands (mulga-1000)
- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
(replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable
* fixed admin
* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names
Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.
* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon
GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.
* fixing gpu bining permissions
* working state of gpu passthrough on phyiscal box
* installer grub image fixes
* made gpu-passthrough more user facing
* fixed banner
* fixed banner
* banner fix
* gpu passthrough docs
* preflight tweak
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* fixes after running /review-branch
* feat: re-wire GPU claim and device injection after vm.Manager refactor
GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:
- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
derived from instance.GPUXVGAEnabled when GPUPCIAddress is set
* fix: Create spinifex user ~/.ssh and keys in firstboot
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
Scheduled runs were using github.ref_name, which equals the default branch
(main) — so the nightly was testing main. Hardcode GIT_BRANCH=dev so the
build-install-artifacts.sh + tofu provisioning target dev's HEAD instead.
GitHub always loads scheduled-workflow files from the default branch, so
this change must stay in sync on main for the schedule to use it.
The post-reboot guest uptime check was a single-shot ssh with a 10s connect
timeout. After the host comes back, guest QEMU + cloud-init + sshd inside
each guest still need a few seconds to settle, so the check raced them.
Poll up to GUEST_SSH_SECS (default 180s) per guest, mirroring the host
TCP/22 poll. Run 25329248329 cell-18 reproduced this — ALB targets went
healthy ~2s after the second SSH attempt failed.
When a function returning non-zero is captured via command substitution
under `set -euo pipefail`, the assignment exits the script before the
following `if [ -z "$var" ]` diagnostic can run. CI failures showed up
as a sudden jump to the EXIT trap with no error message.
- run-multinode-e2e.sh: wrap get_remote_ssh_port in `if !`, bump retry
10 -> 30, add dump_guest_ssh_diagnostics on miss
- run-pseudo-multinode-e2e.sh, run-e2e.sh: wrap get_ssh_port in `if !`
- run-lb-e2e.sh: add `|| VAR=""` fallback on pick_peer
Add WAN bridge prerequisite callout to single-node and multi-node
install guides. Bootstrap installer assumes br-wan exists; running
on a host whose default route is on a bare NIC leaves install in a
non-working state. Auto-provisioning will land in next release.
Both services used one-shot utils.ConnectNATS in launchService. On ISO
firstboot, when nats.service is starting in parallel, the dial can race
ahead of NATS being ready and exit non-zero. systemd's
Restart=on-failure with default StartLimitBurst=5 in 10s burns through
the budget in ~25s and the unit enters failed permanently — DHCP
manager never starts, vpc.dhcp.acquire requests hit no-responders and
the install hangs nondeterministically.
Match daemon/awsgw, which already use ConnectNATSWithRetry (5min
exponential backoff). The newDHCPManagerWithRetry guard (mulga-siv-40)
only helps after the dial succeeds.
Conflict resolution:
- external_ipam_dhcp.go: take dev deletion (consolidated into dhcp package)
- external_ipam.go: take dev's dhcp.RequestAcquire call
- external_ipam_test.go: drop duplicate ObtainDHCPLease/ReleaseDHCPLease tests
- daemon.go: drop dead gwMAC block (GatewayMAC field not in dev's ExternalPoolConfig)
Playbooks for preflight / teardown / install / reset covering the same
ground as dev-install.sh, dev-setup.sh, reset-dev-env.sh. Existing bash
scripts untouched — decide which to keep after parity testing.
- roles/teardown + teardown catalog README track every known state
source; add a task + row when a leak is reported.
- roles/init gates smoketest on awsgw readiness (describe-regions poll).
- roles/smoketest is dup-tolerant, retry-wrapped, auto-terminates the
launched instance.
- dev-install.yml skips build/install/init when a healthy environment
is already present (4-signal health check), always runs smoketest.
- dev-reset.yml captures region/az/external-mode scalars from the
existing spinifex.toml and replays them into init.
Plan: docs/development/improvements/ansible-dev-lifecycle.md (mulga).
Bead: mulga-siv-9.
* fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33)
GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.
Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34)
ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.
Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.
The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33)
The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.
br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.
Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.
The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.
Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.
* fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33)
Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").
Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
their replies are L2 broadcast which the bridge always passes up.
Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.
* revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New
ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to
catch unicast OFFER replies that bypass the bridge netdev. On dev hosts
where upstream honors the BOOTP BROADCAST flag, the wrapper regressed
DORA — public-IP allocation failed where main works. Reverting until
the promisc path is validated against both broadcast- and unicast-OFFER
servers.
* fix(setup-ovn): mark br-int/br-ext Unmanaged for systemd-networkd (mulga-siv-37)
Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs
admin-down: networkd has no matching .network so it tags them as unmanaged
but never brings them up. OVS dataplane works but ovn-controller flow
programming and any tooling that probes link state misbehaves.
Drop /etc/systemd/network/05-spinifex-ovs-internal.network with
Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br`
already runs the netdev up at create time.
* fix(vpcd): allocate gateway LRP IP from pool in centralized NAT (mulga-siv-36)
In veth/macvlan modes the gateway LRP carries all egress, so it must hold
a WAN-subnet IP — otherwise the upstream router silently drops ARP for
the WAN nexthop (RFC 826) and the default route never resolves. The
hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed
centralized NAT (introduced under mulga-siv-26).
Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW
attach the gateway LRP gets a fresh IP from that range, persisted as
external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and
retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the
gw range; ValidatePoolConfig rejects overlap with the EIP range.
Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and
extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs
to allocated pool IPs at startup.
* fix(vpcd): drop link-local fallback, auto-derive gw_lrp_range (mulga-siv-36)
In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet
IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream
router silently drops egress. Hardware deployment confirmed the previous
fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not
explicitly configured.
Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16
.. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on
overlap with the per-VM EIP range; clamp inside the subnet; guard the
upstream gateway IP. IGW attach now errors out instead of falling back
to link-local when allocation fails — surfaces misconfiguration loudly.
Adds focused unit tests for gwLrpRange, ValidatePoolConfig and
nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and
PoolWithGatewayIP regression tests.
* feat(vpcd): DHCP-acquire gateway LRP IPs (mulga-siv-38)
Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire
when the pool is source="dhcp", instead of carving from a static gw_lrp_range.
The upstream DHCP server becomes the single arbiter of LAN address space; a
cluster grows by N nodes without operator-side range carving in spinifex.toml.
- Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as
RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM
share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path.
- TopologyHandler dispatches by pool source: DHCP pools call DHCPManager
via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools
keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and
no-pool deployments still use link-local.
- IGW detach issues vpc.dhcp.release so the upstream server reclaims the
IP. Best-effort — server-side expiry is the safety net.
- Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP
pool, distinct IPs, idempotent attach, release on detach),
IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP
errors), TestPrefixFromMask. Existing static / auto-derive tests
preserved.
* fix(vpcd): subscribe DHCPManager before reconcile to fix boot race (mulga-siv-38)
DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks,
so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and
burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a
dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed
by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN
until the next reconcile pass.
Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription
is live before any topology code can fan out an acquire.
* fix(vpcd): also subscribe DHCPManager before bootstrap reconcile pass
Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but
missed Pass 1 (the bootstrap-config Reconcile that runs even earlier).
On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier
pass still hit "no responders" and burned the 60-retry cold-boot budget
before the Manager came up. Move both reconcile passes after Subscribe.
* fix(vpcd): RFC 2131 §4.1 DORA exponential backoff (mulga-siv-39)
DHCPManager.Acquire previously did one DISCOVER and gave up after 15s
because the outer ctx.WithTimeout matched the per-attempt timeout —
nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer
routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec
mandates retransmission with exponential backoff, and a single lost
DISCOVER was killing legitimate boots.
Move retransmission into DHCPManager.acquireWithBackoff with a
4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client
becomes single-attempt; manager owns retry policy. Default
acquireTimeout lifts from 15s to 90s (sum of schedule + slack).
* fix(vpcd): retry NewDHCPManager on JetStream-not-ready (mulga-siv-40)
NewDHCPManager creates the spinifex-dhcp-leases KV bucket via
utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the
mulga-siv-38 launchService reorder, this runs before any other JetStream
user, so on multi-node cold-boot vpcd hits "JetStream system temporarily
unavailable" and exits before cluster quorum forms. Pseudo-multinode CI
calls start_node_services sequentially, so node2's vpcd failure aborts
the script before node3 ever launches and quorum then never forms.
Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap,
ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as
patiently as the daemon for the cluster to come up.
* fix(daemon): idempotent DetachVolume QMP path (mulga-siv-41)
DetachVolume's QMP sequence was not retry-safe. First call: device_del
succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD
client teardown / in-flight I/O still hold the block node. Handler bails
ServerInternal "leaving volume state intact" but the guest device is
actually gone. AWS-CLI retries: device_del returns DeviceNotFound,
handler bails again. After 2 retries aws cli exits 254 — Single-Node
Phase 8 Step 3 reproducibly fails this way.
Phase 1: treat DeviceNotFound on device_del as already-detached, log and
continue. Detach is now idempotent across retries.
Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in
use" errors so the I/O drain can complete instead of failing the whole
detach. Other QMP errors fall through immediately as before.
DetachVolume's QMP sequence was not retry-safe. First call: device_del
succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD
client teardown / in-flight I/O still hold the block node. Handler bails
ServerInternal "leaving volume state intact" but the guest device is
actually gone. AWS-CLI retries: device_del returns DeviceNotFound,
handler bails again. After 2 retries aws cli exits 254 — Single-Node
Phase 8 Step 3 reproducibly fails this way.
Phase 1: treat DeviceNotFound on device_del as already-detached, log and
continue. Detach is now idempotent across retries.
Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in
use" errors so the I/O drain can complete instead of failing the whole
detach. Other QMP errors fall through immediately as before.
NewDHCPManager creates the spinifex-dhcp-leases KV bucket via
utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the
mulga-siv-38 launchService reorder, this runs before any other JetStream
user, so on multi-node cold-boot vpcd hits "JetStream system temporarily
unavailable" and exits before cluster quorum forms. Pseudo-multinode CI
calls start_node_services sequentially, so node2's vpcd failure aborts
the script before node3 ever launches and quorum then never forms.
Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap,
ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as
patiently as the daemon for the cluster to come up.
DHCPManager.Acquire previously did one DISCOVER and gave up after 15s
because the outer ctx.WithTimeout matched the per-attempt timeout —
nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer
routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec
mandates retransmission with exponential backoff, and a single lost
DISCOVER was killing legitimate boots.
Move retransmission into DHCPManager.acquireWithBackoff with a
4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client
becomes single-attempt; manager owns retry policy. Default
acquireTimeout lifts from 15s to 90s (sum of schedule + slack).
Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but
missed Pass 1 (the bootstrap-config Reconcile that runs even earlier).
On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier
pass still hit "no responders" and burned the 60-retry cold-boot budget
before the Manager came up. Move both reconcile passes after Subscribe.
DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks,
so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and
burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a
dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed
by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN
until the next reconcile pass.
Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription
is live before any topology code can fan out an acquire.
Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire
when the pool is source="dhcp", instead of carving from a static gw_lrp_range.
The upstream DHCP server becomes the single arbiter of LAN address space; a
cluster grows by N nodes without operator-side range carving in spinifex.toml.
- Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as
RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM
share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path.
- TopologyHandler dispatches by pool source: DHCP pools call DHCPManager
via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools
keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and
no-pool deployments still use link-local.
- IGW detach issues vpc.dhcp.release so the upstream server reclaims the
IP. Best-effort — server-side expiry is the safety net.
- Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP
pool, distinct IPs, idempotent attach, release on detach),
IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP
errors), TestPrefixFromMask. Existing static / auto-derive tests
preserved.
In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet
IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream
router silently drops egress. Hardware deployment confirmed the previous
fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not
explicitly configured.
Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16
.. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on
overlap with the per-VM EIP range; clamp inside the subnet; guard the
upstream gateway IP. IGW attach now errors out instead of falling back
to link-local when allocation fails — surfaces misconfiguration loudly.
Adds focused unit tests for gwLrpRange, ValidatePoolConfig and
nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and
PoolWithGatewayIP regression tests.
In veth/macvlan modes the gateway LRP carries all egress, so it must hold
a WAN-subnet IP — otherwise the upstream router silently drops ARP for
the WAN nexthop (RFC 826) and the default route never resolves. The
hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed
centralized NAT (introduced under mulga-siv-26).
Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW
attach the gateway LRP gets a fresh IP from that range, persisted as
external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and
retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the
gw range; ValidatePoolConfig rejects overlap with the EIP range.
Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and
extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs
to allocated pool IPs at startup.
Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs
admin-down: networkd has no matching .network so it tags them as unmanaged
but never brings them up. OVS dataplane works but ovn-controller flow
programming and any tooling that probes link state misbehaves.
Drop /etc/systemd/network/05-spinifex-ovs-internal.network with
Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br`
already runs the netdev up at create time.
ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to
catch unicast OFFER replies that bypass the bridge netdev. On dev hosts
where upstream honors the BOOTP BROADCAST flag, the wrapper regressed
DORA — public-IP allocation failed where main works. Reverting until
the promisc path is validated against both broadcast- and unicast-OFFER
servers.
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml)
- workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch
so checkout@v6 uses the default branch implicitly
- dependabot: route gomod/github-actions/npm PRs to dev
- diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev
- iac/proxmox-template: default git_branch to dev for cluster builds
Refs docs/development/improvements/reintroduce-dev-branch.md
Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").
Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
their replies are L2 broadcast which the bridge always passes up.
Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.
The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.
br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.
Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.
The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.
Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.
ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.
Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.
The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.
Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
bc33c72 added the mask but placed it after the SKIP_APT early-return.
CI bootstrap runs setup.sh with INSTALL_SPINIFEX_SKIP_APT=1 against
runners that already have dhcpcd-base, so the mask never ran. On
Debian-13 the auto-enabled dhcpcd.service kept binding br-wan and
racing vpcd's nclient4, reproducing the mulga-siv-32 DORA timeout
on the public-IP allocation path. Lift the disable+mask out of the
apt branch so it runs unconditionally.
* preflight tweak
* fix: pin gateway LRP to link-local, end intra-cluster ARP collision
Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on
attach. Multi-VPC clusters all GARP'd the same .160 with three different
deterministic MACs on the shared br-ext localnet, flapping the upstream
ARP cache and dropping 100% of inbound traffic to dnat_and_snat external
IPs (mulga-siv-26).
handleIGWAttach and reconcileIGW now unconditionally set the gateway
router port to 169.254.0.1/30. The default route's OutputPort already
pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay
allocated only for per-VM dnat_and_snat external addresses, which are
unique by construction.
Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites
stale Networks in place via the new UpdateLogicalRouterPort path. Needed
because Reconcile gates IGW work on "ext switch missing", so a cluster
shipped with the buggy CIDR never re-enters reconcileIGW.
Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision
and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests
updated to assert link-local and OutputPort=&gwPortName. Log field
gateway_ip renamed to lrp_network to signal the semantic shift.
* fix: rewrite utils.HashMAC to SHA-256 / 46-bit output
The old 24-bit rolling hash with literal class prefix had birthday-paradox
collision probability of 1% at ~580 ids and 50% at 4096 — well within the
range of a single deployment's resource population (LRPs, ENIs, dev/mgmt
NICs). On a shared L2 segment, two clusters could trivially produce
duplicate MACs, breaking neighbour discovery and DHCP.
Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes
and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the
first octet. Output entropy goes from 24 bits to 46 bits — 1% collision
boundary moves from ~580 to ~1.2M ids.
Drop the redundant prefix arg: it was no longer the literal output prefix
under the new scheme, just folded into the hash input. Class disambiguation
moves to the caller, where it's explicit. Three of four callers had nothing
to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and
mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags.
Also consolidate vpcd.generateMAC, which was a local duplicate of the
broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation.
Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md
Bead: mulga-siv-27
* fix: pin HashMAC first octet to 0x02
Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast,
bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived.
Technically valid LAA unicast, but produced MACs like 4a:.. and fe:..
that visually encroach on vendor OUI space and tripped CI/ops checks
expecting all generated MACs to start with 02:.
Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output
entropy drops from 46 to 40 bits — birthday-paradox 1% collision at
~150k ids, still 256x the old 24-bit bound (~580).
* chore(e2e): dump cross-chassis OVN state on guest SSH failure
Adds chassis registrations, port_binding claims, gateway_chassis rows,
ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to
dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis
dataplane stalls (cr-gw chassis vs remote chassis with the VM bound).
* fix: gate vpcd startup reconcile behind NATS KV leader election
N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router
with no atomicity, producing duplicate rows that ovn-nbctl rejects with
"Multiple logical routers named '...'. Use a UUID."
Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key
in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile,
ReconcileFromKV, and the two retrofit passes — others Subscribe and rely
on the vpcd-workers queue group for runtime events. MaxAge=60s on the
bucket caps recovery time when a leader crashes mid-reconcile.
Falls through (elected=true) when KV is unavailable so first-boot does
not deadlock on a missing JetStream.
mulga-siv-29
* fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30)
Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s
because the lb-agent inside the ALB VM has no return path to AWSGW.
The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP
for single-node and deliberately skips adding the /32 mgmt host route — the
WAN-return-path concern in the comment is real but applies only to
internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that
gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when
the agent dials advertiseIP the reply targets the VM's private subnet IP from
the host's WAN with no route back.
Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the
existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and
falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only
internal LBs need the /32 route, so internet-facing single-node still gets
no bootcmd and the host-initiated WAN path stays clean.
Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the
fallback has the inputs it needs.
* fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31)
The test was calling real SetupComputeNode("chassis-test", ...) which shelled
out via sudoCommand to set external_ids:system-id=chassis-test on the host's
live OVS. On a dev box with OVS installed this overwrote the persistent UUID
(owned by openvswitch-switch and re-applied only on reboot), so vpcd's
discoverChassis filtered out the legitimate SBDB chassis row as 'stale local'
and crashlooped — taking out all VPC ops including ALB ENI provisioning.
Bind sudoCommand to a var so tests can swap it. Stub it in the test to return
exec.Command("/bin/false") and assert SetupComputeNode returns an error. The
test's intent (signature + error-path coverage) is preserved; nothing on the
host is touched.
* chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32)
Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8
DORA timeouts correlate with concurrent Acquire calls on the same
bridge. Tracks per-bridge in-flight count and wallclock duration of each
Acquire; logs WARN on failure with both fields, DEBUG on success.
No behavioural change.
* ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32)
run-multinode-e2e.sh dumps node journals via its own EXIT trap, but
run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts
fail the cluster is destroyed in Teardown without surfacing any daemon
logs, blocking root-cause analysis.
Add a workflow step that runs after the test scripts but before
Teardown when a previous step failed, ssh'ing to each node and pulling
the last 200 lines of every spinifex-* unit. Best-effort: skips
gracefully when inventory or nodes are unreachable.
Diagnostic-only; no behavioural change to the test path.
* fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32)
Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when
dhcpcd-base is installed. It binds br-wan and races vpcd's in-process
nclient4 client for upstream OFFERs, draining the lab DHCP pool with
duplicate leases and causing intermittent DORA timeouts in the e2e
public-IP path (worst on cell-8 real-multi/dhcp/debian-13).
The ISO installer already masks the unit at install time
(cmd/installer/install/install.go:316). The binary install path used by
e2e nightly missed it, so debian hosts installed via setup.sh ran with
both DHCP clients live. Mirror the ISO mask here.
Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable
dhcpcd.service on Noble).
* ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32)
Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the
dhcpcd.service journal alongside the spinifex-* units, plus
systemctl is-enabled / is-active and the DHCP client port (UDP 68)
listener so we can see directly whether dhcpcd is running and bound
to br-wan when an Acquire times out.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
initPool's first guard inverted: !errors.Is(err, ErrKeyNotFound) was
true when err==nil, so initPool returned immediately when a record
existed and never reached the drift-reconcile or already-initialized
branches. As a result, range_start/range_end/source changes in
spinifex.toml never propagated and nextAvailableExternalIP failed
with "invalid IP range: - " on dhcp -> static transitions.
Restructure to: bail on hard errors, reconcile or no-op when record
exists, fall through to create when ErrKeyNotFound.
Verified on dev cluster: dhcp -> static toml change now reconciles
KV bounds on restart and RunInstances allocates from the new range.
Stale Allocated entries and stale gateway_ip on source change remain
out of scope for this hotfix; tracked under mulga-siv-24 follow-ups.
Symmetric to the openvswitch-switch ordering: ensures ovn-controller
shutdown waits for spinifex-daemon's instance teardown, the correct
general invariant for VM cleanup since logical port programming
flows through OVN. PR1a flagged this as optional follow-up.
Bead: mulga-siv-20.1
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a)
Three minimal edits to close the orphan-OVS-port failure mode that
survives PR1a's systemd ordering fix (host crash, OOM-kill of ovsdb,
operator manually stopping OVS on a live host) and the
terminate-during-pending race that leaks taps when a user aborts a
slow launch.
1. SetupTapDevice (network.go): unconditional `ovs-vsctl --if-exists
del-port br-int <tap>` + separately-gated kernel `ip tuntap del`.
The previous kernel-side-gated cleanup missed orphan OVS ports
because OVS conf.db survives reboot while kernel taps don't.
Rejected `--may-exist add-port`: it would silently keep stale
external_ids:iface-id/attached-mac from a prior launch.
2. stopInstance mgmt-tap branch (daemon.go) + CleanupMgmtTapDevice
(network.go): derive the mgmt tap name from instance.ID rather
than reading instance.MgmtTap, since that field is set only after
SetupMgmtTapDevice returns and a terminate racing mid-launch
would skip cleanup. CleanupMgmtTapDevice gains a kernel-presence
short-circuit so unconditional invocation doesn't log spurious
"Device does not exist" errors for instances that never reached
SetupMgmtTapDevice.
3. LaunchInstance (daemon.go): `launchStillValid` helper re-checks
instance status before StartInstance and TransitionState, in
addition to the entry guard. On detection of shutting-down /
terminated, log Info and return nil — the concurrent terminate
goroutine owns cleanup; this is an expected race outcome, not an
error. Removes the spurious "Failed to transition instance to
running" ERROR that surfaced on every legitimate
terminate-during-pending. Plus the entry path now returns nil
instead of an error for the same reason.
Also lands `scripts/start-then-terminate-pending.sh` (added during
investigation) which reproduces the runtime race for manual
verification.
Bead: mulga-siv-20.2
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1b)
systemd reverses After= on shutdown so daemon now runs CleanupTapDevice
while ovsdb-server still answers. Eliminates orphan OVS port left in
conf.db when host reboots — root trigger for "port already exists" on
post-reboot RunInstances. spinifex-vpcd.service already had this; the
daemon unit was missed.
Also pull openvswitch-switch + ovn-controller into spinifex.target Wants
so the platform brings them up explicitly rather than relying on
implicit ordering from spinifex-vpcd alone.
Bead: mulga-siv-20.1
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a)
Follow-up to d509df7. The sudo PAM audit bug exists in two more vpcd call
sites beyond portToBr: localSystemID() and discoverChassis().
When vpcd.service's AmbientCapabilities trip sudo's PAM, stderr carries
"sudo: unable to send audit message: Operation not permitted". CombinedOutput
merges that into stdout and corrupts the parsed value:
localSystemID → localID="sudo: ...\n\"add0104e-..."
discoverChassis → tries to parse the stderr banner as chassis name/hostname
Downstream effect: discoverChassis skips the real local chassis as "stale",
vpcd falls back to cfg.ChassisNames (e.g. "chassis-josh-banksia") which no
chassis in SBDB actually uses, so every gateway_chassis row points at a
phantom chassis. cr-gw* chassisredirect ports never bind → no proxy-ARP
for EIPs → external ping fails with ARP timeout.
Switch both to Output() + exec.ExitError.Stderr, matching d509df7's pattern.
Found during mulga-998.b reboot drill while verifying external reachability:
chassis discovery log showed `localID="sudo: unable to send audit message...`,
`cr-gw-vpc-* chassis=[]`, and host ping to EIP returned Destination Host
Unreachable. With this fix applied, discoverChassis returns the real system-id
and (after stale gateway_chassis rows are cleared) cr-gw binds, proxy-ARP
responds, and EIP ping succeeds end-to-end.
Note: existing gateway_chassis rows written pre-fix still carry the phantom
chassis_name. Cleanup is one-shot — delete those rows and trigger an IGW
detach+attach, or run reset-dev-env.sh. vpcd does not reconcile existing
rows on startup (potential future retrofit, similar to Fix 3's
RetrofitAllExternalLocalnetOptions).
vpcd.service sets AmbientCapabilities which trips sudo's pam_limits into
emitting 'sudo: unable to send audit message: Operation not permitted'
on stderr. CombinedOutput merged that into stdout and poisoned the
port-to-br result, causing verifyBridgeMode to fail with a bogus bridge
name on service start after reinstall.
Switch to Output() (stdout only) + exec.ExitError.Stderr on error.
ensureLocalnetOptions was only invoked inside reconcileIGW, which is gated
on the external logical switch being absent — so a port whose options were
cleared or drifted from the current bridge mode would never be corrected
on restart. Drill 2 (stale-options recovery) exposed this: clearing
ext-port options and restarting vpcd left the port with options={}.
Changes:
- Extend ensureLocalnetOptions to also restore network_name=external (not
just nat-addresses). A manual `ovn-nbctl clear` wipes both, and the port
is non-functional without network_name.
- Add RetrofitAllExternalLocalnetOptions: walks every OVN logical switch
tagged spinifex:role=external and calls ensureLocalnetOptions on its
ext-port-<vpcId>. OVN is the source of truth; NATS KV records may be
missing or stale.
- Call RetrofitAllExternalLocalnetOptions after ReconcileFromKV during
vpcd startup.
- Tests: network_name restoration in both centralized and direct modes;
RetrofitAll across multiple switches with mixed stale / correct / non-
external state.
Fix 1 persisted the veth pair across reboot via
15-spinifex-veth-wan.{netdev,network}, but only configured
veth-wan-br. Its peer veth-wan-ovs had no .network unit, so
networkd left it unmanaged and admin-DOWN after reboot. That
cascades: peer goes LOWERLAYERDOWN, br-wan loses carrier,
external DHCP/NAT path broken — despite vpcd correctly
detecting mode=veth and the sanity check passing.
OVS does not admin-up external (non-internal) ports — that's
networkd's job. Pre-reboot the setup-ovn.sh 'ip link set
veth-wan-ovs up' made it work, but that doesn't persist.
Write a second unit 16-spinifex-veth-wan-ovs.network with
ConfigureWithoutCarrier=yes so networkd flips admin state
without taking ownership of bridge/address assignment (OVS
retains that). Cleanup paths in the non-veth branch of
setup-ovn.sh and reset-dev-env.sh updated to remove the new
unit too.
Verified live via 'networkctl reload': veth-wan-ovs and
veth-wan-br both UP, br-wan LOWER_UP. Reboot drill per
docs/development/bugs/dhcp-vpcd-issues.md 'Live drill 1'
still required to confirm persistence.
Add four branch tests for verifyBridgeMode that were missing from the
initial Fix 2 commit:
- direct mode rejects empty external_interface
- direct mode rejects empty dhcp_bind_bridge
- veth mode rejects empty dhcp_bind_bridge
- veth mode fails when veth-wan-br link missing
Lifts diff coverage from 59.5% to 65.6%, meeting the 65% gate.
Make bridge-mode detection loud and fail-start when runtime state does
not match config. Closes D4/D12/D18 of dhcp-vpcd-issues.md.
Changes:
- detectBridgeMode: slog.Info at each decision, slog.Warn on fall-through
(was silent Debug — let the veth persistence bug hide for weeks).
- Extract resolveBridgeConfig so mode/bind-bridge resolution is unit
testable. Empty mode now stays empty so verifyBridgeMode rejects it
with an explicit list of supported values.
- Add verifyBridgeMode: refuses to start when external iface missing
(direct), when external iface not on the OVS bridge (direct), when
veth-wan-ovs missing or attached to the wrong OVS bridge, or when
veth-wan-br missing or on the wrong Linux bridge. References
OvnExternalBridge (= "br-ext") so the sanity check tracks the contract,
not a hardcoded string.
- Wire verifyBridgeMode into launchService. Error is loud and includes
operator guidance ("re-run scripts/setup-ovn.sh").
- Injectable probes (ifaceIsMacvlan, ifaceExists, portToBr,
readLinkMaster) so tests can exercise every branch without touching
real interfaces.
- Tests: 16 new unit tests covering detectBridgeMode, resolveBridgeConfig,
and all verifyBridgeMode branches.
CreateLogicalSwitchPort only seeds options:nat-addresses at first
creation. If a VPC's localnet port was created during a vpcd run that
saw direct mode (e.g. pre-Fix 1 reboot without the veth pair), the port
keeps stale options forever — OVN never GARPs for the external IP and
centralized NAT stays broken.
Add TopologyHandler.ensureLocalnetOptions: read the LSP, diff
options["nat-addresses"] against useCentralizedNAT(), update only on
mismatch. Called from reconcileIGW after CreateLogicalSwitchPort so the
retrofit runs on every reconcile pass (no-op when already correct).
Mock tracks UpdateLogicalSwitchPortCalls so the idempotency test
asserts no write was issued.
setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing
recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs
as a port on br-ext but the peer is gone — the external path silently
breaks and vpcd falls through to direct mode.
- setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network}
in the veth branch so systemd-networkd recreates the pair on boot. Add
a single pre-case guard that rips down stale veth persistence + the OVS
port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode
transitions are idempotent (D17).
- reset-dev-env.sh: delete the netdev/network units during cleanup so
`make reset` gives a truly clean slate.
- spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service
so vpcd's sanity checks see a populated kernel + OVSDB (D5).
The old wan_bridge name misled operators into pointing the field at
the OVN-side bridge 'br-ext', which never sees LAN DHCP traffic. On
consumer-router LANs (WAN NIC enslaved to a Linux bridge) the vpcd
DHCP client's AF_PACKET bind failed silently, blocking the default
install path.
- Rename Go field WanBridge -> DhcpBindBridge and TOML key wan_bridge
-> dhcp_bind_bridge across config, vpcd service, external IPAM,
daemon pool wiring, admin settings, and spinifex.toml template.
- Rewrite detectedWanBridge() -> detectedDhcpBindBridge(): the
Linux-bridge branch now returns the bridge name verbatim (e.g.
"br-wan") instead of synthesising "br-ext". OVS-vs-Linux probe via
ovs-vsctl is no longer needed for this field.
- Add checkLegacyWanBridgeKey() in cmd/service.go: vpcd fail-starts
if the deprecated 'wan_bridge' TOML key or SPINIFEX_VPCD_WAN_BRIDGE
env var is set. No silent alias, no auto-rewrite (per plan D3/D13).
- Template carries an operator-facing comment warning never to set
the field to 'br-ext'.
- 7 new unit tests for the detector branches and fail-start paths.
Smoke on ben-bottlebrush: spx admin init --force writes
dhcp_bind_bridge matching the live br-wan Linux bridge; VM launched
with --map-public-ip-on-launch reached from LAN; restoring a
legacy-key backup causes vpcd to enter failed state with the
deprecated-key guidance log.
Plan: docs/development/bugs/dhcp-vpcd-issues.md (Fix 4)
Bead: mulga-998.a (unblocker for .b / .c)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)
Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:
- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
/local/instances (placeholder types until daemon-local-autonomy lands),
plus /health for scaffolding validation
Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.
* feat: add DDIL harness fault injection and link profiles
Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:
- harness/fault.go — PartitionNode (per-peer iptables DROP with
orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
RestartDaemonOnly, WaitForMode (polls /local/status for typed
DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
SATCOM, HFData, Flapping constants (unvalidated approximations per
plan §2 table; validation tracking lives in TEST_COVERAGE.md)
All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.
* feat: add DDIL harness witness VM helpers
LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.
* feat: add DDIL harness snapshot and state assertions
TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.
* feat: add DDIL harness reset, retry, and quarantine
ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.
AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.
Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.
bd: mulga-siv-5.5
* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)
Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.
* feat: add DDIL E2E CI workflow (workflow_dispatch only)
Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
siv-8.5 migrated daemon + awsgw but missed the sidecar services and
the spx CLI. vpcd and viperblockd read cfg.NatsHost / cfg.S3Host raw
from spinifex.toml; when BindIP=0.0.0.0 the TLS handshake fails SAN
verification because 0.0.0.0 is not a cert SAN. Same for the spx get
CLI path.
Call sites:
spinifex/services/viperblockd/viperblockd.go:173 (NATS)
spinifex/services/viperblockd/viperblockd.go:407 (S3 s3.S3Config)
spinifex/services/viperblockd/viperblockd.go:533 (S3 nbd.NBDKitConfig)
spinifex/services/vpcd/vpcd.go:277 (NATS)
cmd/spinifex/cmd/get.go:93 (CLI NATS)
admin.DialTarget rewrites 0.0.0.0:N to 127.0.0.1:N — loopback is always
a SAN and in-process sidecars share the host with NATS/predastore.
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway
Convert br-mgmt from an externally-provisioned Linux bridge to an OVS
bridge owned by scripts/setup-ovn.sh, giving every deployment type
(tofu-cluster, production single-node, production multi-node) a single
provisioning path.
setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface /
--no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone
(excluded from ovn-bridge-mappings so ovn-controller ignores it) and
assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot.
SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master`
to `ovs-vsctl add-port` / `del-port` to match the new bridge type.
Add a loopback guard to wireLBAgentConfig's fallback branch: on a
production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge,
the fallback previously injected 127.0.0.1 into LB VM cloud-init —
silently unreachable because the VM's loopback is its own. The guard
leaves GatewayURL empty and the existing "LB agent gateway URL not
configured" error fires instead. Regression covered in
daemon_wire_lb_test.go.
* fix: auto-detect primary IP for admin init --bind default
Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback
only, so system VMs on br-mgmt could never reach it. Auto-detect the
default-route source IPv4 instead, which keeps TLS cert SANs specific
(still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1
explicitly to preserve loopback-only behavior on laptops.
Closes mulga-siv-7.
* test: refresh wire lb loopback-bind comment for siv-7
* fix: dev-install scripts should let admin init auto-detect --bind
Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while
setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with
no listener. Drop the explicit --bind so auto-detect picks the primary IP
and the siv-6 mgmtRouteVia path routes LB traffic to a real listener.
* fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
PR #144 (nats TLS) made every service listen on --bind's specific IP. For
awsgw and predastore this broke localhost-reachability that the UI proxy
(spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on —
latent since #144, surfaced by siv-7's auto-detect default on dev installs
(502 from the UI).
Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen
wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host
daemon → predastore) keep working, LAN clients still reach them via the
LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}}
because peer-advertised values must not resolve to 0.0.0.0/loopback.
predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1
for local-node consumers, so existing callers see a usable connect address.
* revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8)
Clears the deck before layering the listen-vs-advertise split
(mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node
default and introduces an explicit --advertise flag for off-host dial
targets; the reverted approach (auto-detect a specific IP at init,
hardcode 0.0.0.0 in the template, guard against loopback in
wireLBAgentConfig) becomes redundant once that lands.
Reverts:
- bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
- f20a47b fix: dev-install scripts should let admin init auto-detect --bind
- 49b0b4e test: refresh wire lb loopback-bind comment for siv-7
- b88dfdf fix: auto-detect primary IP for admin init --bind default
Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig
fallback and its regression test — subsumed by the GatewayURL
precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error,
no silent empty-string fallthrough). The br-mgmt OVS provisioning
from 74b6e2d is retained.
Plan: docs/development/bugs/spx-admin-init-bind.md
* feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2)
Introduce the two small address-resolution helpers the listen-vs-
advertise split needs, with full unit coverage. No callers yet —
dial-site migration lands in mulga-siv-8.5.
- DialTarget(listenAddr): rewrites wildcard listen addresses
(0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process
clients dial a target the daemon's cert already has in its SANs.
Specific IPs, loopback, hostnames, and malformed input pass through.
- AdvertiseHost(listenAddr, advertiseIP): picks the host off-host
clients should dial. Wildcard listen → advertiseIP; specific host →
that host. Port is dropped so callers append the protocol-appropriate
port.
Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1
* feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3)
Pure data-model change. Add AdvertiseIP to:
- admin.ConfigSettings (local node, populated by init).
- config.Config (per-node; mapstructure "advertise").
- formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake
carries it between peers.
buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so
joined nodes from older clusters / pre-siv-8 joiners keep working.
No behaviour change yet — templates, CLI flag, and dial-site migration
land in siv-8.4 and siv-8.5.
* feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4)
Introduce the listen-vs-advertise split at the CLI boundary.
admin init/join:
- --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert).
- New --advertise <IP> flag on both init and join; optional, auto-detected
from DetectNetwork().WAN.IP when --bind is wildcard or loopback.
- resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements
precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 +
warning.
- Join resolves advertise before building the JoinRequest so peers record
a reachable dial target (not 0.0.0.0).
- isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator
can now run "init --nodes 3" without typing a specific --bind.
- Init success message prints bind / advertise / loopback three-way split.
Templates:
- spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered
only when AdvertiseIP is set.
- predastore-multinode.toml: iam.nats_url host wrapped in the same
{{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that
predastore.toml already uses, so co-resident predastore dials its local
NATS via loopback instead of 0.0.0.0.
Tests cover the resolveAdvertiseIP matrix, the template advertise field
(present + omitted), and the predastore nats_url loopback shim.
Dial-site migration and GatewayURL rework land in siv-8.5.
* feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5)
Switch every in-process dial site to admin.DialTarget so the wildcard
listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon
→ NATS and → predastore (boot, reconnect, status probe, S3 store, service
manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so
loopback-dial a wildcard-listen service is always valid; the old
0.0.0.0 literal was tripping TLS SAN verification.
Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go):
1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged)
2. br-mgmt present, AWSGW on 0.0.0.0 → br-mgmt IP (unchanged)
3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW)
4. DevNetworking shim → 10.0.2.2 (unchanged)
5. AWSGW on specific IP, no bridge → that IP (unchanged)
6. Else → log error, skip assignment
Kills the pre-siv-8 silent empty-URL failure mode that made
CreateLoadBalancer fail with 'missing system credentials: gatewayURL='
on every fresh single-node install. Supersedes siv-6's loopback guard.
Extend SetupAWSCredentials with a wanIP parameter (reserved for a
future --operator-endpoint flag) and thread AdvertiseIP through
finalizeNodeSetup's three call sites (single-node init, multi-node
init, join). Print 'Bind / Advertise / Loopback' on multi-node init
and join completion to match the single-node summary.
Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
Replaces the sudo-dhcpcd shell-out in ObtainDHCPLease / ReleaseDHCPLease
with NATS request/reply against vpc.dhcp.acquire / vpc.dhcp.release.
The daemon-side IPAM passes its nats.Conn into NewExternalIPAM and
threads hostname (option 12) / vendor-class (option 60) through to
vpcd; ExternalIPAllocation now records LeaseExpiresUnix, DHCPServerID
and HWAddr on DHCP-sourced entries. allocateFromPool is restructured
to acquire the lease once up-front and rely on vpcd's idempotent
Manager for CAS-retry safety; a single release fires only if every
CAS attempt fails. Drops the stdout parser, the per-bridge mutex, and
the parse test.
Systemd: spinifex-daemon loses AF_PACKET, ProtectKernelTunables and
MemoryDenyWriteExecute are restored to =yes; spinifex-vpcd gains
AF_PACKET in RestrictAddressFamilies, AmbientCapabilities=CAP_NET_RAW,
and CAP_NET_ADMIN / CAP_NET_RAW / CAP_NET_BIND_SERVICE in the bounding
set. Net privilege footprint narrows.
See docs/development/bugs/dhcp-allocate-error.md.
Introduces the spinifex/services/vpcd/dhcp package with a Client
interface, typed Lease struct, nclient4-backed implementation, and a
Fake for unit tests. Adds DHCPManager in the vpcd package: subscribes
to vpc.dhcp.acquire / vpc.dhcp.release NATS requests, runs a renewal
goroutine per lease with ±10% jitter on T1/T2, persists leases in the
new spinifex-dhcp-leases KV bucket, and fans out vpc.dhcp.lease-expired
when a lease permanently fails to renew. Wires the Manager into
launchService so it starts on vpcd boot and rehydrates persisted leases
via Bootstrap.
Handler-side wiring (ExternalIPAM → NATS) and the systemd capability
move land separately in mulga-991.
See docs/development/bugs/dhcp-allocate-error.md.
Stop ObtainDHCPLease from erroring on hosts where dhcpcd is already
running as a daemon managing br-wan (e.g. server 2). Add -T (test-only
mode) so the invocation does not collide with the host daemon's
control socket on the interface.
Softens ProtectKernelTunables and MemoryDenyWriteExecute in the daemon
unit to accommodate dhcpcd's runtime requirements; both revert when
the in-process Go DHCP client rewrite lands (mulga-989 / mulga-990 /
mulga-991, see docs/development/bugs/dhcp-allocate-error.md).
Interim: leaves the phantom-lease and release-socket-clobber risks
active until the rewrite replaces this path entirely.
Internet-facing ALBs allocated their public IP by calling
externalIPAM.AllocateIP() directly from LaunchSystemInstance, bypassing
the EIP service entirely. No EIPRecord landed in the eips KV bucket, the
tag was "auto_assign" (EIP service uses "elastic_ip"), and
DescribeAddresses never returned the IP — so any OpenTofu flow that
expects an AllocateAddress-observable record hung waiting.
LaunchSystemInstance now calls eipService.AllocateAddress +
AssociateAddress when the EIP service is wired (falls back to the old
direct IPAM path otherwise). TerminateSystemInstance releases through
the service so the KV record is cleaned up and vpc.delete-nat is
published. The generic stopInstance teardown is left untouched so
customer RunInstances flows behave identically.
Also refreshes the nginx-alb terraform example to use public subnets
with auto-assigned public IPs (the configuration that actually exercises
this path end-to-end) and tightens the awsgwBindIP loopback guard in
wireLBAgentConfig so the gateway URL is usable by LB agents.
* fix: replace stringly-typed TLS error detection with sentinel errors
Export ErrCACertRead and ErrCACertParse from utils/nats.go so callers can
detect permanent TLS configuration failures with errors.Is instead of
substring matching on error messages. Update daemon.go and awsgw.go retry
loops accordingly, and update tests to use assert.ErrorIs.
Refs: mulga-siv-1
* refactor: extract shared NATS connect-with-retry into utils
daemon.connectNATS() and awsgw.connectNATS() were near-identical retry
loops. Extract to utils.ConnectNATSWithRetry() with functional options
for test configurability.
Reset now targets /etc/spinifex, /var/lib/spinifex, /var/log/spinifex,
/run/spinifex (the privilege-separated layout installed by setup.sh) and
drives re-install from a source build via dev-install.sh instead of the
legacy in-tree $HOME/spinifex + start-dev.sh flow.
Capture region, AZ, external_mode, pool range, gateway, and prefix from
the existing spinifex.toml before the wipe so the reset restores the
current topology rather than applying defaults. Fresh boxes fall back to
ap-southeast-2 and a pool derived from the WAN subnet.
Drop the macvlan branch entirely — bridge-only from here on. Operators
on single-NIC hosts must run reset from the console (documented in the
script header).
dev-install.sh gains DEV_INSTALL_SKIP_INIT so reset can own the post-
setup.sh steps (setup-ovn, spx admin init with preserved args, CA
trust, service start, LB image import). Also honors SPINIFEX_REGION /
SPINIFEX_AZ for non-reset callers, and scopes systemctl reset-failed
to spinifex-* so unrelated failed units aren't silently cleared.
Closes mulga-983.
Extract all inline systemd unit and drop-in content from install.go and
firstboot.go into a dedicated cmd/installer/systemd package for editability.
Bash scripts remain inline in firstboot.go.
Also copies the GRUB splash image from the live ISO into the installed
system's /boot/grub/ and sets GRUB_BACKGROUND so the branded background
appears in the post-install GRUB, not just the installer GRUB.
Remove the duplicated chown/ownership/symlink block from the firstboot
script. setup.sh (called with INSTALL_SPINIFEX_TARBALL) already handles
service users, directory ownership, sudoers, systemd unit installation,
and enabling spinifex.target — the same steps as the online curl|bash
install.
Firstboot now follows the documented install flow directly:
1. setup.sh --tarball (air-gapped equivalent of curl|bash)
2. setup-ovn.sh --management
3. spx admin init
4. systemctl start spinifex.target
Also reduce wait-for-nats.sh timeout from 90s to 30s per reviewer feedback.
- partUUID: error on empty blkid output instead of writing UUID= to fstab
- installBootloader: fail if both EFI and BIOS targets fail; track efiErr
- machine-id fallback: write at mode 0600 so systemd can commit on first boot
- Run: defer unmount so partitions are always cleaned up on failure
- copyRootfs: verify critical paths exist after rsync before continuing
- resolveNIC: skip virtual/bridge interfaces (docker0, virbr0, ovs-system)
- EjectAndReboot: use syscall.Reboot instead of exec(reboot) which needs D-Bus
- ui.Run: return error on TTY open failure instead of silent fallback
- firstboot script: trap EXIT to always self-disable, preventing retry loops
- enableUnit/enableBannerUnit: deduplicate into enableSystemdUnit; check Remove
- runAdminBanner: promote service restart failure to slog.Error
- parseNodeConf: log non-ENOENT read failures
Replace fragile image-name substring match (-lb-/-alb-) with a
DescribeImages tag filter on spinifex:managed-by=elbv2. Remove the
silent fallback that picks the first image in the store when no LB
image exists — CreateLoadBalancer now returns ServerInternal with a
remediation hint instead of persisting a record in StateFailed.
- daemon: tag filter + error return from the resolver closure
- service_impl: resolver signature is func() (string, error); caches
only successful results so the next call retries after an error
- CreateLoadBalancer gate now requires launcher + networking; AMI is
resolved inside the gate and failures short-circuit before persist
- admin images import: add --name+--file combo (catalog metadata with
a local file source) and a --tag key=value flag for explicit tagging
of file-only imports
- build-system-image.sh + lb.conf: propagate SYSTEM_TAG via --tag so
locally built system AMIs get the right tag even when the catalog
is bypassed
Closes mulga-969
chmod 0770 without -R only affected the top directory; pre-existing files
(e.g. images imported while owned by another user at mode 0600) stayed
unreadable to the spinifex group, breaking image import on upgrade paths.
- autoinstall: replace file scanner with env var reader (SPINIFEX_AUTO,
SPINIFEX_HOSTNAME, etc.) set by spinifex-init.sh from kernel cmdline
- autoinstall: disk selection modes — auto (single or fail), largest,
smallest, nvme, or exact path; clear error listing available disks
when auto/nvme finds multiple candidates
- admin banner: add Login row showing spinifex account name
- admin banner: write to /etc/issue (pre-login console) as well as
/etc/motd (post-login SSH); fix SSH line root@ → spinifex@
Scans the USB EFI partition at boot for autoinstall.toml. When
enabled = true, the installer runs fully unattended using values from
the config file then ejects the USB and reboots — no user interaction
needed.
- New autoinstall package: TOML config struct, boot-media scanner,
install.Config converter, EjectAndReboot
- Scanner skips iso9660 mounts so the bundled reference copy of the
template never accidentally triggers headless mode
- "auto" disk target selects the largest non-removable disk
- "auto" NIC target picks the first physical interface
- Validates all required fields before touching any disk
- main.go: check for autoinstall config before launching TUI
Picks up multipart upload wedge fixes from predastore v1.1.5:
- 76fdbf1 fix: pool cleanup must not reap connections with in-flight streams (Bug A)
- 7df9645 fix: add mutex to bufferedWALFile to prevent bufio race (Bug B)
- 9dcd3dd fix: drain shard body before taking wal.mu in handlePUTShard (Bug C)
- 43e20f6 fix: widen QUIC flow-control windows to mitigate multipart upload deadlock
plus regression tests for all three bugs (predastore PR #8).
- Add spinifex login user password via chpasswd in install.go (same
password as root, collected once in TUI)
- Copy /root/.aws/ to /home/spinifex/.aws/ in firstboot so the operator
can use the AWS CLI without sudo after first boot
- Fix getty drop-ins to be instance-specific (tty1/ttyS0 only), leaving
tty2+ immediately available as rescue terminals
- Remove autologin — node now requires password at login prompt
- Pre-start OVS/OVN in firstboot and poll NB DB up to 120s to eliminate
setup-ovn.sh timeout races on slow physical hardware
- Tighten per-service ownership in firstboot (each service user owns only
its own /var/lib/spinifex subdirectory)
- Add LAN bridge as a non-critical systemd unit so a missing LAN cable
can never stall network-online.target or firstboot
- Write dhcpcd.conf with short br-lan timeout; add networking.service
drop-in to treat exit 1 as success for optional interface failures
- Disable IPv6 on management bridges to silence dhcpcd journal noise
- Fix WAN/LAN enter-key and ESC navigation logic in TUI
- Fix LAN NIC cursor to skip WAN NIC correctly
* feat: add attributes map to LB and TG records with defaults
Replace CrossZoneEnabled bool with Attributes map[string]string on
LoadBalancerRecord and TargetGroupRecord. Add DefaultLoadBalancerAttributes()
and DefaultTargetGroupAttributes() functions. ALBs seed cross-zone enabled
at create time.
* feat: add 4 attributes methods to ELBv2Service interface
* feat: implement ELBv2 attributes service methods and NATS wrappers
Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
to service_impl.go with proper ARN validation and default merging.
Add corresponding NATS request wrappers in service_nats.go.
* feat: add gateway handlers for ELBv2 attributes API endpoints
Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
gateway handlers with input validation and action map registration.
* feat: add daemon handlers & NATS subscriptions for elbv2 attributes
Add 4 handler functions in daemon_handlers_elbv2.go and 4 NATS
subscriptions in daemon.go for ModifyTargetGroupAttributes,
DescribeTargetGroupAttributes, ModifyLoadBalancerAttributes, and
DescribeLoadBalancerAttributes.
* fix: add missing accountID authorization check to ELBv2 attribute handlers
* fix: guard nil attributes and log skips in ELBv2 Modify*Attributes
ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes silently
dropped attributes with nil Key or Value, and would panic on a nil
slice element when dereferencing attr.Key. Add an explicit nil-element
guard and log a warning whenever an attribute is skipped, so callers
and operators can see divergence between the submitted payload and the
persisted state.
* fix: sort ELBv2 attribute responses by key for deterministic output
DescribeTargetGroupAttributes and DescribeLoadBalancerAttributes iterated
a map and appended to a slice, producing random attribute order on each
call. Sort keys before building the response so Terraform plans stay
stable and snapshot tests can rely on consistent ordering.
Closes mulga-910
* feat: add ELBv2 DescribeTags handler (mulga-919)
Required by the Terraform AWS provider's post-create refresh on
aws_lb, aws_lb_target_group, and aws_lb_listener. Without it, the
gateway returns InvalidAction and apply fails after the resources
are already created.
Dispatches by ARN resource segment (loadbalancer/targetgroup/listener)
to the existing record stores. Enforces account scoping via per-type
not-found errors to avoid cross-account existence leaks. Listeners
return an empty Tags slice since ListenerRecord doesn't store tags yet
— that's the exact case the provider hits on every untagged listener
refresh. Output is sorted by key for deterministic responses.
* refactor: parameterize DefaultLoadBalancerAttributes by lbType (mulga-915)
Cross-zone default was split across three places: CreateLoadBalancer seeded
{load_balancing.cross_zone.enabled: true} for ALBs, DefaultLoadBalancerAttributes
returned false unconditionally, and tests asserted both sides. Make
DefaultLoadBalancerAttributes(lbType) the single source of truth: ALBs get
"true", NLBs (and any other type) get "false". CreateLoadBalancer no longer
seeds attributes — DescribeLoadBalancerAttributes derives per-type defaults
from lb.Type on read. Updated TestCreateLoadBalancer_CrossZoneAttributes to
assert via the Describe handler instead of poking stored record state.
* perf: direct KV lookup for ELBv2 *ByArn finders (mulga-913)
GetLoadBalancerByArn and GetTargetGroupByArn used to call ListLoadBalancers /
ListTargetGroups and linearly scan, doing one NATS round-trip per record in
the bucket. Terraform hits Describe*Attributes on every plan/refresh — a
20-ALB / 40-TG workspace was doing ~60 calls × ~60 records = ~3600 KV gets
per refresh where ~60 would suffice. The LB/TG ARN's final path segment IS
the KV key used by GetLoadBalancer(lbID)/GetTargetGroup(tgID), so parse the
short ID from the ARN and do an O(1) lookup, verifying the record's stored
ARN matches as defence-in-depth against any stale-key weirdness.
* fix: reject ELBv2 ModifyAttributes calls where every entry is invalid (mulga-911)
When every submitted attribute was rejected by the nil-guard (nil element,
nil Key, or nil Value), Modify{TargetGroup,LoadBalancer}Attributes still
called PutTargetGroup/PutLoadBalancer with an unchanged record and returned
a successful ModifyOutput with empty Attributes. Only a slog.Warn marked
the skip — the caller saw 200 OK and moved on. Return
ErrorInvalidParameterValue when len(input.Attributes) > 0 and every entry
was rejected, so the caller learns that nothing landed.
* fix: reject unknown ELBv2 ModifyAttributes keys with ValidationError (mulga-912)
ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes had no whitelist
check — any caller-supplied key (typo like 'stickness.enabled', deprecated
keys, or cross-product keys sent to the wrong resource) was persisted verbatim
and merrily echoed back by Describe. AWS rejects unknown keys with
ValidationError; match that so Terraform surfaces the typo at plan time.
Validate each key against DefaultTargetGroupAttributes() /
DefaultLoadBalancerAttributes(lb.Type) and return ErrorValidationError on the
first unknown key, logging both the arn and the offending key name.
* perf: skip no-op Put in ELBv2 ModifyAttributes (mulga-914)
The unconditional PutTargetGroup/PutLoadBalancer fired on every Modify call,
even when the caller sent an empty attribute list or when every submitted
value already matched the stored value. Terraform re-applies the same
attribute set on every drift check, so the steady-state path was doing a
pointless NATS/KV round-trip per call and unnecessarily expanding the
read-modify-write race window.
Track a dirty flag per Modify call: compare each submitted key's value
against tg.Attributes/lb.Attributes and only write to the map (and allocate
the map lazily) when it actually differs. Skip the Put entirely if nothing
changed. Added regression tests that capture the KV entry revision before
and after a no-op modify to prove the write was skipped.
* test: assert exact ModifyAttributes echoed key/value pairs (mulga-917)
Both round-trip tests used assert.Len(t, modOut.Attributes, 2) and never
checked that the returned Attributes actually contained the submitted
keys/values. A regression that pointed at the wrong source, dropped Value,
or returned a copy of defaults would have passed the length check
unchallenged. Build a modMap from the echoed response and assert each
expected Key/Value pair explicitly.
* test: cover sequential ModifyAttributes merge and same-key overwrite (mulga-916)
All existing round-trip tests did one Modify then Describe. No test covered:
Modify {A}, Modify {B}, Describe, assert BOTH keys present; or Modify
overwriting the same key on a second call. A future refactor that replaced
tg.Attributes instead of mutating in place would pass every existing test
but silently wipe previous attributes on each Modify — the most likely
real-world ALB/TG bug. Add sequential-merge tests for both TG and LB
handlers that exercise multi-key merge, same-key overwrite, and
cross-survival of unrelated keys.
The ALB reaches backends over the private VPC network, so the Nginx
instances no longer need public IPs. Drop associate_public_ip_address
and map_public_ip_on_launch, and swap the instance public-IP outputs
for private-IP ones.
The *.elb.spinifex.local DNS name does not resolve from the operator's
host, so document an AWS CLI one-liner (describe-load-balancers with
an AvailabilityZones[].LoadBalancerAddresses[].IpAddress query) that
retrieves the ALB's public IP for curl/browser testing.
* feat: add ALB target health reporting via HAProxy stats
Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).
Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.
E2E: remove health check skip — targets must now become healthy.
* feat: allocate public IP for internet-facing ALB VMs in daemon
Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.
* fix: release auto-assigned public IPs on DeleteNetworkInterface
DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.
Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.
* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)
In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.
* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)
Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs
Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing
* feat: add custom AMI for ALB to images list
* feat: add internet-facing ALB data plane E2E test (mulga-772)
New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.
* fix: set ENDPOINT to runner node IP in ALB E2E workflow
* fix: correct paths filter to reference alb-e2e.yml
* fix: alb ami name and workflow step to import the image
* fix: spx path
* fix: remove ALB AMI import from workflow and added to bootstrap.sh
* fix: use two subnets in different AZs for internet-facing ALB E2E
Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.
* refactor: remove multi-AZ enforcement for internet-facing ALBs
Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.
* fix: use correct account ID for ALB ENI public IP assignment
The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.
* fix: add debug logging for ALB public IP assignment failure
* fix: expand debug logging to include peer node and more keywords
* fix: lazy system AMI discovery for ALB VMs
The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.
* fix: resolve ALB VM instance type dynamically from resource manager
The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.
* push interim changes
* refactor: rewrite ALB agent from NATS client to HTTP server
Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.
- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)
* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms
Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.
- service_impl: pushConfig uses http.Post, waitForAgentReady uses
http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature
* refactor: rewrite ALB health checker from NATS subscription to HTTP polling
Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.
- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths
* refactor: remove NATS wiring from ALB image and make hostfwd dev-only
- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
placeholder HAProxy frontend on 8405 (agent owns the port now);
update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
is enabled — production ALBs are reached via VPC IP directly
* feat: system credentials file + daemon wiring for ALB agent SigV4 auth
Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.
* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)
Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.
- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig
* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)
Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.
* feat: add VPC service route for ALB agent gateway access
ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.
Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.
* Revert "feat: add VPC service route for ALB agent gateway access"
This reverts commit d4220d8806.
* feat: add management NIC infrastructure for system instance control plane
Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.
- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields
* feat: wire management NIC into QEMU launch and cloud-init config
Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.
* test: add coverage for ELBv2 service lifecycle and setter methods
* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP
* fix: generate per-interface cloud-init config when eniMAC present without devMAC
The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.
* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image
- ENI DescribeNetworkInterfaces description filter did exact string match,
but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
"ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
module couldn't add the Spinifex CA to the system trust store. Agent
TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
where async TerminateSystemInstance goroutine wrote to mock fields
while the test read them.
* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance
DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.
Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.
E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.
* fix: detach then delete ENIs in DeleteLoadBalancer
ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.
* test: add internal ALB dataplane E2E test
Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.
The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion
Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.
* feat: add NLB constants and data model changes (mulga-838)
* feat: type-aware validation in CreateLoadBalancer (mulga-839)
Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.
* feat: type-aware validation in CreateTargetGroup (mulga-840)
Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.
* feat: type-aware validation in CreateListener (mulga-841)
* feat: add NLB listener and protocol compatibility unit tests
Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.
* feat: add TCP HAProxy config generation for NLB
Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.
* refactor: rename albagent to lbagent for NLB generalization
Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.
- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest
* feat: add sys.micro system instance type for LB VMs (mulga-845)
Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.
* feat: LB image shrink and rename (mulga-846)
Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.
* chore: rename file
* feat: NLB E2E test and bug fixes (mulga-847)
Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.
Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)
* test: validate NLB internal scheme (mulga-857)
NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:
- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)
* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)
Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).
- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node
* chore: disable LB E2E tests in CI until dev env issue resolved
* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json
The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.
* feat: automate LB image build and import (mulga-861)
- dev-install.sh: build lb-agent and run build-system-image.sh --import
after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming
* update go version
* add LB e2e tests
* fix: clean LB image build dir and wait for nbd device readiness
Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.
* fix: serialize nbd access with flock for concurrent CI builds
Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.
* fix: move flock to cover entire image build, not just nbd section
Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.
* fix: skip LB image build if output is fresh from concurrent CI job
When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.
* fix: filter AMI discovery by name in E2E tests
With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.
* fix: filter AMI discovery by name in multi-node E2E tests
Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.
* fix: use awsgw bind IP as LB agent gateway URL fallback
When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.
Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.
* fix: capture StateReason and daemon logs when LB app instances fail
Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.
* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning
Three fixes for CI E2E failures:
1. run-e2e.sh: Remove custom AMI from predastore after deleting its
backing snapshot. Previously the AMI remained discoverable but
unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
on the deleted snapshot.
2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
of picking any non-alpine image. Also improve debug log dump to use
journalctl (binary-install CI uses systemd, not log files).
3. CreateLoadBalancer: Only set provisioning state when a system VM
was actually launched. Previously a failed launch still set
provisioning, leaving the LB stuck forever with no agent to
send heartbeats.
* fix: use AWSGW bind IP as LB agent gateway URL in multi-node
In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.
Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.
* fix: add mgmt route via bootcmd for multi-node internal LBs
Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.
Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.
Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.
Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).
* add LB enable flag to e2e
* test: add elbv2 tests to meet diff coverage threshold
Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).
* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat
Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.
* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)
Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.
* fix: data race in getSystemAMI/getSystemInstanceType lazy caching
Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.
Also fix pre-existing nosprintfhostport lint in daemon.go.
* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned
publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).
isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.
* fix: narrow health checker to only scan target groups for reporting LB
handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.
* fix: pass region through to LB agent SigV4 signing (mulga-871)
LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.
* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig
Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.
* fix: upgrade silent failure paths from Debug to Warn/Error
- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding
* refactor: deduplicate MAC hash function into utils.HashMAC
Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.
* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps
Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.
* fix: fail ALB launch when public IP allocation fails for internet-facing scheme
Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.
* fix: retry system AMI resolution when image not yet imported
sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.
Same fix applied to getSystemInstanceType for consistency.
* fix: propagate updateStoredConfig errors to API callers (mulga-884)
updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.
* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)
* fix: parse HAProxy templates once at package level (mulga-875)
generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.
* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)
When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.
* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)
The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.
* refactor: use aws.StringValue in toHealthReport (mulga-878)
Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.
* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)
Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.
* fix: address PR #122 review comments (mulga-887)
- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh
* fix: check specific error type for ENI deletion on termination (mulga-888)
ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.
* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)
GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.
* fix: remove backend address from proxy error response (mulga-892)
The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.
* fix: release mutex before KV store I/O in health checker (mulga-893)
handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.
* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)
fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)
* fix: detect veth bridge mode and use centralized NAT for it
The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.
Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).
* fix: add retry loop for peer NLB reachability test
With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.
* fix: use request-reply for vpc.add-nat to prevent ARP propagation races
publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.
Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.
* fix: clean up stale NAT rules before adding new ones for reused IPs
When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.
This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.
* fix: clean up stale NAT rules across all VPC routers on IP reuse
The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.
DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.
Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.
* fix: run LB inet tests from actual hosting node, not hardcoded node1
The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.
Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
different node
Also updates e2e.yml to pass --nodes instead of --peer.
- Replace single network screen with separate WAN and LAN screens
- WAN/LAN IPs now live on Linux bridges (br-wan, br-lan) over the physical
NICs, not on the NICs directly — makes OVS/OVN veth attachment SSH-safe
- Add DHCP/static toggle on both screens (DHCP is the default)
- Add DNS nameserver input for static mode (comma-separated, default hint
1.1.1.1 / 8.8.8.8 shown as placeholder)
- Detect WiFi NICs via /sys/class/net/<name>/wireless; show SSID and
password fields when a WiFi adapter is selected
- LAN screen shows WAN NIC greyed/unavailable; cursor skips over it
- LAN screen only shown when 2+ NICs are detected
- Geneve encap IP uses br-lan address when LAN NIC present, else br-wan
- Remove --macvlan/--wan-iface from firstboot; setup-ovn.sh auto-detects
br-wan as a Linux bridge and uses veth mode
- Pin both WAN and LAN NIC names to MAC via a single udev rules file
- node.conf writes MANAGEMENT_IFACE=br-wan (bridge, not physical NIC)
- banner runs After=spinifex.target so services are settled before banner writes
- switch systemctl restart → try-restart (safe if target not yet active)
- append banner to /etc/motd with sentinel instead of overwriting, preserving
the Debian disclaimer; sentinel ensures re-runs replace cleanly
- remove /etc/issue write — getty shows it pre-login which is wrong
- add getty@.service.d drop-in so all getty instances wait for banner before
showing login prompt; prevents service messages scrolling during login
- fix Makefile to prefer locally built spx/installer binaries over system PATH
- add `spx admin banner --boot-check` subcommand to spx CLI
- reads management IP from the NIC at runtime (not from node.conf)
- if IP has changed, updates /etc/spinifex/node.conf and restarts spinifex.target
- writes banner to /etc/issue (pre-login console) and /etc/motd (SSH MOTD)
- banner instructs user to cat ~/.aws/credentials (standard AWS CLI location)
- update firstboot.go to use spx-based banner unit; remove writeBannerScript
and old spinifex-banner.sh entirely
- new banner unit runs Before=spinifex.target so IP is current before services start
- chown /etc/spinifex and /var/lib/spinifex to spinifex:spinifex after
spx admin init so services running as the spinifex user can read configs
- add Environment=HOME=/root to firstboot unit to suppress $HOME warning
- increase wait-for-nats.sh timeout from 30s to 90s for slow startup
- banner now writes to both /dev/tty1 and /etc/motd (visible over SSH)
- add systemd.show_status=1 to GRUB cmdline for per-service boot feedback
- add --removable to grub-install EFI so fallback path EFI/BOOT/BOOTX64.EFI
is written (required for UTM and bare metal without NVRAM boot entries)
- add root password screen and manual NIC entry to installer TUI
- tee spx admin init output to /root/spinifex-credentials.txt with 0600 perms
Replaces charmbracelet/huh with a custom bubbletea wizard for the
installer TUI. This gives full control over terminal I/O so the
installer attaches directly to the correct console device (ttyS0 for
serial, tty1 for VGA) as directed by the new spinifex-init boot script.
- branding/branding.go: centralised logo (ASCII placeholder), color
constants, and publisher string ready for Mulga asset swap
- ui/ui.go: multi-screen bubbletea model (welcome → disk → network →
identity → join → CA cert → confirm) styled with lipgloss
- main.go: reads SPINIFEX_CONSOLE from /proc/cmdline to open the right
tty before launching tea.Program
- go.mod: bubbletea/lipgloss/bubbles promoted to direct deps; huh removed
Adds cmd/installer with three packages:
- ui: huh-based TUI collecting disk, network, node, and cluster config
- install: sequential install steps (partition, debootstrap, chroot, bootloader)
- firstboot: oneshot systemd service that runs setup-ovn and spx admin init/join on first real boot
Supports both single-node init and multi-node join flows.
Access key and secret key values were printed to stdout when using
--access-key and --secret-key flags. These are captured by systemd
journal and log aggregation, exposing secrets. Log that keys were
overridden without printing the actual values.
Addresses H-5 from security audit.
Phase 11 in run-multinode-e2e.sh:
- Create VPC with public + private subnets + IGW
- Launch bastion in public subnet, verify SSH
- Create spread placement group, launch 3 VMs in private subnet
- Validate spread distribution via spx get vms (3 nodes)
- Verify no internet pre-NAT (ping 8.8.8.8 fails from all 3)
- Create NAT GW + EIP + route, poll for OVN SNAT rule
- Verify internet via NAT GW from all 3 nodes (different physical hosts)
- Full cleanup: NAT GW, instances, placement group, VPC resources
Sends anonymous telemetry to install.mulgadc.com/install during
spx admin init and join. Reports machine_id (/etc/machine-id),
region, az, node count, version, and arch. Fire-and-forget with
5-second timeout — never blocks the user.
Opt out with --no-telemetry flag or SPX_NO_TELEMETRY=1 env var.
In single-node mode, cert E2E discovered all host IPs (WAN, mgmt, VPC)
and tried openssl s_client on each. But awsgw only binds to the
configured host IP, not all interfaces. Management (10.11.x) and VPC
(10.x) IPs have no listener on port 9999, causing false failures.
Split NODE_IPS (all IPs for SAN validation) from SERVICE_IPS (IPs
where awsgw listens) parsed from spinifex.toml. Connection-based
tests (3, 4c, 5, 8) now use SERVICE_IPS.
The NAT GW E2E tests used 2-3 second sleeps before verifying SNAT
rule creation, outbound connectivity, and rule removal. This caused
flaky CI failures when OVN took longer to propagate flows to the
datapath (varies by host load and OVN version).
Replace with retry loops:
- SNAT rule check: poll OVN NB every 1s for up to 30s
- Outbound ping: retry up to 10 attempts with 5s backoff (~50s max)
- SNAT removal: poll up to 15 attempts with 2s backoff (~30s max)
Also dump OVN routes alongside NAT rules on failure for debugging.
Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.
Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.
Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
Previously hardcoded CHASSIS_ID="chassis-node1" which could mismatch
the OVS system-id from prior setup-ovn.sh runs, creating stale chassis
entries in the SBDB. Now defers to setup-ovn.sh's auto-detect
(chassis-$(hostname -s)) so the name is always consistent.
Two stacked bugs prevented SSH to VPC instances:
1. The default route (0.0.0.0/0) was added without OutputPort. For
DHCP-sourced pools the gateway router port uses link-local addressing
(169.254.0.1/30) so OVN northd cannot resolve the nexthop (e.g.
192.168.1.1) to an output port — it silently drops the route from
the southbound DB.
2. discoverChassis returned all chassis from the SBDB including stale
entries from previous OVN setups on the same host. When a stale
chassis got highest gateway priority, the chassisredirect port
stayed unbound (up: false), so no ARP/NAT traffic was processed.
Fix: add OutputPort to static routes in both handleIGWAttach and
reconcileIGW, and filter stale chassis by cross-referencing the local
OVS system-id with SBDB hostnames.
Root cause: when terminating instances, the vpc.delete-nat event was
published with an empty logicalIP. vpcd's DeleteNAT looked for a
dnat_and_snat rule with logicalIP="" — found nothing — and the real
rule was left as a stale entry in OVN.
When the IPAM later recycled that public IP for an EIP/NAT Gateway,
OVN had conflicting rules: stale dnat_and_snat AND new snat, both
using the same external IP.
Fix: extract PrivateIpAddress from instance.Instance before publishing
vpc.delete-nat, in both running and stopped termination paths.
Also adds 14 unit tests for NAT Gateway service.
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.
Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.
reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
The bastion SSH hop failed because the private instance's cloud-init
hadn't completed (SSH daemon not ready). The instance was "running"
from the hypervisor perspective but cloud-init takes ~30-40s.
- Replace 2-attempt retry with 30-attempt poll (5s intervals, 150s max)
- Hard failure with OVN debug output instead of silent skip
- Matches how wait_for_ssh works for public instances
- getRouteTable: distinguish ErrKeyNotFound (→ not found) from other
errors (→ log + internal error) instead of mapping all to "not found"
- putRouteTable: log underlying error on marshal/write failures
- KV scan loops (allRouteTablesForVPC, DescribeRouteTables,
DisassociateRouteTable, ReplaceRouteTableAssociation): fail on
transient NATS errors instead of silently skipping entries. Only
skip ErrKeyNotFound (deleted between Keys() and Get())
- ReplaceRouteTableAssociation: compensating write if second KV put
fails — restores association to old table to prevent data loss
- createMainRouteTable: returns error instead of swallowing it
- ensureDefaultIGWRoute: log actual errors at Warn level, not Debug
- E2E test: fail on missing default VPC instead of skipping
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.
- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
After enabling tap networking (public subnet), QEMU no longer uses
hostfwd bindings. The pseudo multi-node test was still grepping for
hostfwd to detect instance distribution and SSH endpoints.
- check_instance_distribution: count nbdkit root-volume processes per
node IP instead of hostfwd-bound QEMU processes
- get_instance_node: match QEMU nbd socket → nbdkit host= IP to
determine which simulated node hosts each instance
- count_instances_per_node: use get_instance_node instead of hostfwd
- SSH connectivity: check PublicIpAddress first (port 22), fall back
to hostfwd — same pattern as single-node and multi-node E2E
admin init builds a NetworkConfig from external networking flags and
passes it to the formation server. When joining nodes poll /formation/status,
they receive the cluster-wide [network] section (external mode, pools,
gateway, DNS, bootstrap VPC IDs). Each joining node auto-detects its
own external_interface and wan_bridge locally.
Fixes multi-node clusters having no [network] section in spinifex.toml
on secondary nodes, which caused no public IP assignment and broke
guest SSH in E2E tests.
The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
GenerateVolumes did not set TenantID on the VolumeMetadata for root
volumes cloned from AMIs. After the pre-Phase4 tenant fallback was
removed (ae316db), describe-volumes filters volumes by exact tenantID
match — root volumes with empty TenantID became invisible to the
owning account, breaking E2E Phase 5b volume lifecycle tests.
When external_mode is set, dev_networking=false (no QEMU hostfwd NIC).
E2E tests now detect the instance's public IP from describe-instances
and SSH via port 22 on the public IP. Falls back to hostfwd only when
no public IP is available (overlay-only mode).
Applies to Phase 5a-ii, 7b, 7c-pre, and 8b SSH tests.
Verify full networking path: host → public IP → OVN DNAT → VM.
Test outbound internet from VM via OVN SNAT (ping 8.8.8.8).
Falls back to API-only verification if public IP unreachable.
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
and dnat_and_snat rule; private subnet instance gets neither; no blanket
VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
DHCP pool config validation to meet 60% diff coverage threshold
Replace all ListObjects (V1) usage with ListObjectsV2 across the
objectstore interface, both implementations, all 10 call sites, and
tests. V2 uses ContinuationToken instead of Marker for pagination
and adds KeyCount to responses. The frontend and Predastore already
speak V2 — this aligns the Go backend.
DHCP pool mode requires dhcpcd-base (/usr/sbin/dhcpcd) to obtain
leases from the router. Added to Makefile install-system, setup.sh
binary installer, and source-install docs.
Closes: mulga-753
handleIGWAttach and reconcileIGW created a blanket snat rule for the
entire VPC CIDR, allowing private subnet instances (no public IP) to
route outbound through the IGW. AWS behavior: only instances with
public IPs route through IGW via per-VM dnat_and_snat rules. Private
subnets require a NAT Gateway for outbound internet.
Also fixes pre-existing lint issues in external_ipam_dhcp.go (gofmt,
modernize, nolintlint).
Closes: mulga-754
Adds source="dhcp" option to external pool config. When set, ExternalIPAM
obtains public IPs from the upstream router's DHCP server (via dhcpcd on
br-ext) instead of allocating from a static range_start..range_end.
The VM behavior is identical to static pool mode: private VPC IP from OVN
DHCP, public IP via DNAT+SNAT, full DescribeInstances support. The only
difference is where the public IP comes from.
Two separate DHCP conversations:
- Host ↔ Router DHCP (br-ext): "give me a public IP for this VM's NAT"
- VM ↔ OVN DHCP (br-int): "give me my private VPC IP"
Tested: DHCP pool mode on single-node (ben-banksia). VM got public IP
192.168.1.77 from router DHCP, SSH works, outbound internet works.
Explains static vs DHCP IP sources in pool mode, the two-DHCP-conversations
architecture, when to use each, and adds a DHCP homelab deployment example.
Also updates bridge setup section for veth pair approach.
setup-ovn.sh now detects whether the default route is on a Linux bridge
or OVS bridge. When it's a Linux bridge (e.g. br-wan from cloud-init),
instead of destructively converting it, the script creates an OVS bridge
(br-ext) and links them with a veth pair. The Linux bridge keeps its IP,
routes, and ports untouched — no WAN interruption.
Also fixes all pre-existing lint issues (gofmt, ineffassign, errchkjson,
gosec, modernize, whitespace) to bring golangci-lint to 0 issues.
Tested: pool mode with static IP range on single-node (ben-banksia).
VM gets public IP 192.168.1.201, SSH works, outbound internet works.
Simplify all 6 tenant-check sites in volume handlers from
`if accountID != "" && tenantID != "" && tenantID != accountID`
to `if tenantID != accountID`. No pre-Phase4 volumes with empty
TenantID exist in any environment — this code was unreachable.
Remove vol-legacy test fixture and empty-accountID-sees-all assertion.
The second admin init --force was called without external networking
flags, overwriting the pool/nat config from the first call with
auto-detected NAT+DHCP defaults.
In NAT mode, all VMs share a single gateway IP for SNAT. No per-VM
public IPs are allocated. Previously, external IPAM was initialized
for any external_mode != "", causing noisy AllocateIP failures (empty
pool range) on every instance launch.
Now IPAM is only initialized when external_mode == "pool". In NAT mode,
d.externalIPAM stays nil, cleanly skipping the auto-assign public IP
path. SNAT topology is still created by vpcd via the pool's gateway_ip.
Part of mulga-737 (Phase 16).
Replace all br-external references with br-wan. Update setup-ovn.sh
command examples to use --wan-bridge/--wan-iface/--macvlan instead of
--external-bridge/--external-iface/--direct.
Part of mulga-742.
- All guides: OVN setup before admin init (consistent ordering)
- Single-node + source-install: Replace --external-bridge/--external-iface
with auto-detection + --wan-bridge/--wan-iface/--macvlan options
- Multi-node: Simplify examples (auto-detect WAN bridge), show override
options in prose
- setup.sh: Update "next steps" to match single-node guide, add
--wan-bridge/--macvlan guidance for physical NIC setups
Part of mulga-742.
- config.go: Add WanBridge field to VPCDConfig (default "br-wan")
- spinifex.toml template: Add wan_bridge under [nodes.X.vpcd]
- admin.go: Auto-detect WAN bridge name from default route interface,
add detectedWanBridge helper
- service.go: Wire WanBridge from cluster config to vpcd Config
- vpcd.go: Add WanBridge to Config, log it on startup, update comments
- network_detect.go: Fix isVirtualInterface to accept the default route
device as a parameter instead of hardcoding br-external. Bridges like
br-wan are now correctly treated as physical when they carry the
default route.
Part of mulga-741.
reset-dev-env.sh: Replace --external-bridge/--external-iface/--direct
with --wan-bridge/--wan-iface/--macvlan. Auto-detect when WAN interface
is already a bridge (tofu-cluster) vs physical NIC.
stop-dev.sh: Replace hardcoded br-external check with dynamic scan of
all OVS bridges — any non-br-int bridge with an IP means OVS is
infrastructure and should not be stopped.
Part of mulga-740.
Replace --external-bridge/--external-iface/--direct with:
- --wan-bridge=NAME: OVS bridge for WAN traffic (auto-detected from
default route if it's already a bridge)
- --wan-iface=NAME: physical NIC to add to the WAN bridge
- --macvlan: SSH-safe macvlan mode for single-NIC hosts
Auto-detection logic:
- If default route goes through a bridge (br-wan) → use it directly
- If default route is a physical NIC → stop with guidance (never
auto-move a NIC into OVS — may drop SSH)
Bridge-mappings updated: external:$WAN_BRIDGE (was external:br-external).
OVN logical network name stays "external" (no topology.go change needed).
Encap IP now prefers br-vpc IP when available (dedicated VPC data plane).
Part of mulga-739. Blocks VPC E2E work.
Phase 12b: vpcd now reconciles OVN topology from NATS KV on startup
(Pass 2). After subscribing to topics, it reads all VPCs, subnets,
IGW attachments, and ENIs from KV and ensures matching OVN objects
exist. Handles reboots, OVN DB loss, and missed events. All operations
are idempotent.
Phase 16 prep: Add --gateway-ip flag to admin init for NAT mode
(specifies the OVN gateway router's external IP without DHCP).
Update reset-dev-env.sh to support EXTERNAL_MODE=nat env var for
NAT/DHCP mode testing alongside the default pool mode.
Auto-detect bridge topology: if a dedicated WAN NIC exists (not used for
SSH), add it directly to br-external instead of creating a macvlan.
Direct bridge enables distributed NAT (DNAT on VM's own chassis) and
eliminates macvlan workarounds (MAC alignment, gratuitous ARPs,
centralized NAT bottleneck).
- topology.go: bridge-mode-aware NAT and localnet port options
- vpcd.go: auto-detect bridge mode, skip MAC alignment for direct bridge
- setup-ovn.sh: --direct flag for physical NIC on br-external
- reset-dev-env.sh: SSH NIC detection via SSH_CONNECTION
Direct bridge (WAN NIC added directly to OVS) is preferred when a
dedicated WAN NIC exists. Enables distributed NAT, no MAC workarounds,
host can reach VMs. Macvlan is fallback for single-NIC hosts only.
Three blockers fixed for VPC public subnet E2E on single-node:
1. OVN SB stale data: reset-dev-env.sh now deletes both NB+SB DB files
instead of individual NB cleanup. Eliminates ovn-controller commit
failure loop from stale chassis/port binding entries.
2. Chassis name mismatch: reset-dev-env.sh passes --chassis-id=chassis-node1
to setup-ovn.sh, aligning OVS system-id with vpcd's chassis-{nodeName}.
3. Macvlan unicast filtering: switched to centralized NAT (no external_mac
on DNAT rules), set nat-addresses=router on localnet ports, and vpcd
aligns macvlan MAC with OVN gateway router MAC on startup. Macvlan only
delivers unicast matching its own MAC — this ensures inbound traffic
from the LAN reaches OVS/OVN.
Also fixed: DNS for VMs. admin init auto-detects host DNS servers via
resolvectl, stores them in spinifex.toml dns_servers field, and vpcd
propagates them to OVN DHCP options. Fallback: 8.8.8.8 + 1.1.1.1.
Validated E2E: inbound ping, SSH, outbound ping, DNS, curl, apt update.
- admin init now pre-generates VPC/subnet/IGW IDs and writes [bootstrap]
to spinifex.toml before services ever start. No race condition.
- EnsureDefaultVPC accepts optional BootstrapIDs to use pre-determined
IDs instead of generating random ones.
- CreateInternetGatewayWithID allows daemon to use the bootstrap IGW ID.
- Removed writeBootstrapConfig from daemon (no longer needed).
- Fixed bootstrap account_id: uses admin account (000000000001) not
system account (000000000000).
- reconcileIGW now sets spinifex:igw_id in OVN external IDs.
Phase 11b: RunInstances now auto-resolves the default subnet when no
SubnetId is specified, matching AWS behavior. Previously fell back to
SLIRP networking.
Phase 12: vpcd reconciles OVN topology from [bootstrap] config on startup.
The daemon writes a [bootstrap] section to spinifex.toml with the default
VPC/subnet IDs on first run. vpcd reads this on startup and creates any
missing OVN objects (router, switch, DHCP, IGW topology, SNAT, gateway
chassis). All operations are idempotent — safe on every restart.
This solves the bootstrap ordering problem where admin init creates VPC
records before vpcd is running, so NATS events are missed.
EnsureDefaultVPC now returns DefaultVPCInfo with VPC/subnet IDs.
The single-nic IP migration path was fragile and caused connectivity
issues during setup. macvlan is now the only bridge strategy — always
creates spx-ext-{nic} on the WAN NIC. Removed --single-nic flag,
SingleNIC config/detection fields, and updated all docs.
Phase 8a: Wire all 5 EIP gateway actions (AllocateAddress, ReleaseAddress,
AssociateAddress, DisassociateAddress, DescribeAddresses) end-to-end through
gateway → NATS → daemon → EIP service.
Phase 8b: Wire all 7 SG gateway actions (Create/Delete/Describe SecurityGroup,
Authorize/Revoke Ingress/Egress) through the same pipeline.
Critical fix: handleIGWAttach was not calling SetGatewayChassis on the gateway
router port. Without this, OVN has no chassis binding for external traffic —
SNAT/default route exist logically but packets have nowhere to go physically.
This was the root cause of outbound routing failure (can't ping 8.8.8.8).
Default VPC now matches AWS behavior: auto-creates IGW (attached),
MapPublicIpOnLaunch=true on default subnet, and a default security group
with intra-SG inbound allow + all outbound allow.
Comprehensive VPC networking docs rewrite covering modes, bridge setup,
configuration, security groups, EIPs, OVN reference, and troubleshooting.
EnsureDefaultVPC was only called for GlobalAccountID (000000000000).
The admin account (000000000001) from admin init never got a default
VPC because seedAdminAccount didn't publish iam.account.created.
Fix: daemon startup now ensures default VPC for both system and admin
accounts. Also publish iam.account.created from seedAdminAccount so
any future account-creation listeners fire correctly.
All 3 install guides (binary, multi-node, source) now end at the same
verification point: aws ec2 describe-instance-types. Installation is
complete when that command returns instance types.
New docs/setting-up-your-cluster/README.md is the common guide all 3
install paths point to. Covers: AMI import, SSH key creation, VPC with
public subnet (IGW + modify-subnet-attribute), instance launch with
public IP, SSH via public IP, instance lifecycle management.
Removed SSH-via-hostfwd from all guides — public subnet is the default
connectivity path. Private subnet documented as an alternative.
When setup-ovn.sh migrates the WAN IP from a physical NIC to
br-external, systemd-resolved loses its DNS association (DNS servers
are per-interface). This breaks hostname resolution for everything
on the host — git, apt, curl, SSH all fail.
Now captures DNS config from the original interface before migration
and re-associates it with br-external via resolvectl. Falls back to
8.8.8.8 if no DNS was configured on the original interface.
When setup-ovn.sh migrates the host's WAN IP to br-external, stopping
openvswitch-switch destroys the bridge and the host loses all network
connectivity — unreachable via SSH, requires console reboot.
Now checks if br-external has an IP before stopping OVS. If it does,
skip the OVS stop and only stop ovn-controller + ovn-central. OVS is
infrastructure — bridges must survive service stop/start cycles.
Discovered during 3-node E2E testing: stop-dev.sh killed all 3 nodes'
WAN connectivity simultaneously.
Admin init output was duplicating install guide steps. Simplified to
just show completion message. Install guides already have all steps.
Set dev_networking=false when external_mode is configured — the hostfwd
NIC adds a confusing second interface inside VMs and is not needed when
public subnets provide WAN-routable IPs.
Wire public IP auto-assignment end-to-end:
- RunInstances checks MapPublicIpOnLaunch on subnet, allocates external
IP from ExternalIPAM, updates ENI record, publishes vpc.add-nat for
dnat_and_snat rule, returns PublicIpAddress in response
- TerminateInstances releases public IP back to pool, publishes
vpc.delete-nat to remove NAT rule before ENI deletion
- DescribeInstances populates PublicIpAddress from VM struct
Add ExternalIPAM initialization in daemon startup from cluster
NetworkConfig. Add GetSubnet and UpdateENIPublicIP helpers. Add
PublicIP/PublicIPPool fields to vm.VM struct. Fix network auto-detect
to recognize br-external as WAN interface.
spx admin init now works with zero external networking flags:
- Auto-detects WAN interface, gateway, prefix from ip route
- Auto-detects single-NIC vs multi-NIC mode
- Defaults to nat mode with DHCP for gateway IP
- Prints detected topology with interface roles
setup-ovn.sh gains --dhcp flag: obtains a DHCP lease on the
macvlan/bridge interface for the OVN gateway SNAT address, then
writes gateway_ip into spinifex.toml. Tested on VM: macvlan gets
separate IP from router DHCP, host keeps its original IP.
Three tiers of external networking:
- Zero flags: auto nat+DHCP (VMs get outbound internet)
- --external-pool=start-end: pool mode (VMs get public IPs)
- --no-external: overlay only (no internet for VMs)
Add --single-nic flag to setup-ovn.sh. When used with --external-bridge,
creates a macvlan sub-interface (spx-ext-{iface}) in bridge mode off the
host's NIC instead of adding the NIC directly to br-external. The host
keeps its IP — no migration, SSH-safe.
Strategy selection:
- Dedicated NIC: --external-iface=eth1 (add directly)
- Single NIC: --external-iface=eth0 --single-nic (macvlan)
- IP migration: --external-iface=eth0 (legacy, no --single-nic)
Add single_nic bool field to VPCDConfig for per-node config. Different
nodes in a cluster can use different strategies. Unit test verifies
single_nic parsing alongside external_interface.
Add security group CRUD handlers: CreateSecurityGroup, DeleteSecurityGroup,
DescribeSecurityGroups, AuthorizeSecurityGroupIngress/Egress,
RevokeSecurityGroupIngress/Egress. SG records stored in NATS KV.
Add ACL match expression compiler (acl.go) translating AWS SG rules to
OVN ACL match strings for TCP/UDP port/ranges, ICMP, all-protocols, CIDR
filters, and intra-SG references.
Add vpcd security.go handlers for OVN Port Group lifecycle and ACL
management via vpc.create-sg/delete-sg/update-sg NATS topics. Extend
OVNClient interface with CreatePortGroup, DeletePortGroup,
SetPortGroupPorts, AddACL, ClearACLs. Add PortGroup and ACL to nbdb
model.
Add EIP service with AllocateAddress, ReleaseAddress, AssociateAddress,
DisassociateAddress, and DescribeAddresses. Uses ExternalIPAM for IP
allocation from admin-defined pools. Publishes vpc.add-nat/delete-nat
events for OVN dnat_and_snat rules.
EIP records stored in spinifex-vpc-elastic-ips NATS KV bucket with
account-scoped keys. AssociateAddress replaces auto-assigned public IPs
when present.
Add MapPublicIpOnLaunch field to SubnetRecord with ModifySubnetAttribute
API handler. Add PublicIpAddress/PublicIpPool fields to ENIRecord for
tracking auto-assigned public IPs.
Add vpc.add-nat / vpc.delete-nat NATS topics and handlers in vpcd
topology for creating distributed dnat_and_snat NAT rules (with
LogicalPort + ExternalMAC for chassis-local NAT processing).
Wire ModifySubnetAttribute through gateway → daemon → VPC service.
Enable PR-triggered CI so required status checks can gate merges.
Spinifex runs on all PRs (no path filter) to avoid blocking non-Go PRs.
UI workflow keeps path filters as it's not a required check.
Add external IPAM module (CAS-based IP allocation from admin-defined
pools in NATS KV). Supports multi-pool with region/AZ scoping, pool
fallback logic, gateway IP reservation, and concurrent allocation via
CAS retries. 12 unit tests covering allocation, release, exhaustion,
CAS conflict, multi-pool, fallback, validation, and idempotent init.
Add --external-bridge flag to setup-ovn.sh: creates OVS bridge
br-external, adds WAN NIC, migrates host IP to bridge, and sets
ovn-bridge-mappings=external:br-external for OVN localnet ports.
Add commented [network] section and external_interface to spinifex.toml
config template for discoverability.
Redistribute E2E tests between single-node and multi-node suites, slim
down pseudo multi-node suite, and remove directory creation from
dev-setup.sh so spx admin init is the single source of truth for
~/spinifex/ setup.
Add cluster-wide [network] config with external_mode and multi-pool
support (ExternalPool with region/AZ scoping). Per-node vpcd gets
external_interface for WAN NIC name.
Fix handleIGWAttach to use real external IPs from the configured pool
instead of link-local 169.254.0.1/30. Gateway router port now gets
the pool's gateway IP with correct prefix length, SNAT uses the real
external IP, and the default route points to the actual WAN gateway.
Includes pool selection logic (AZ-scoped → region-scoped → unscoped
fallback) and unit tests for config parsing, IGW with external pool,
explicit gateway IP, and pool selection.
<textx="32"y="90"font-family="'Segoe UI', sans-serif"font-size="10"fill="#a1a1aa">Users interact with Spinifex using standard AWS SDKs and CLIs against a custom endpoint URL</text>
<textx="24"y="28"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">BASTION · only entry point into a private subnet</text>
<textx="110"y="46"font-family="'Segoe UI', sans-serif"font-size="10"fill="#d8b4fe"text-anchor="middle">private subnet · no internet</text>
</g>
<textx="450"y="156"font-family="'Segoe UI', sans-serif"font-size="10"fill="#71717a"text-anchor="middle"font-style="italic">App server has no route to the internet — all access hops through the bastion.</text>
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
<textx="24"y="22"font-family="'Segoe UI', system-ui, sans-serif"font-size="10"fill="#a1a1aa"letter-spacing="2"font-weight="600">DATA PATH · VM to WAN via OVN localnet</text>
<textx="44"y="181"font-family="'Segoe UI', sans-serif"font-size="10"fill="#71717a"font-style="italic">Only used in DHCP source mode. Static mode picks from range.</text>
<textx="670"y="130"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#F87171"font-weight="600">distributed NAT · any MAC · all frames</text>
<textx="876"y="30"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#71717a"text-anchor="end">WAN NIC == SSH NIC · host IP preserved</text>
<textx="24"y="548"font-family="'Segoe UI', sans-serif"font-size="10"fill="#71717a">Cross-host traffic uses Geneve tunnels (UDP 6081) over the overlay NIC · ovn-controller programs OpenFlow rules on br-int</text>
<textx="136"y="18"font-family="'SF Mono', ui-monospace, monospace"font-size="11"fill="#8ab4f8"text-anchor="middle"font-weight="700">arrives at 10.0.1.5</text>
<textx="136"y="32"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#a1a1aa"text-anchor="middle">dest IP rewritten</text>
</g>
<!-- Footnote -->
<textx="24"y="158"font-family="'Segoe UI', sans-serif"font-size="9"fill="#71717a">The instance OS only sees its private IP (10.0.1.5) — the public address is managed entirely by OVN's dnat_and_snat rule.</text>
<textx="450"y="58"font-family="'Segoe UI', system-ui, sans-serif"font-size="20"font-weight="600"fill="#f5f5f5"text-anchor="middle">AWS-style workflows, on your own stack</text>
(Developer Preview) - Hive developed by [Mulga Defense Corporation](https://mulgadc.com/) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Hive gives you AWS-style workflows on your own hardware.
Hive replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
---
It’s designed for developers and operators who need:
# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
## What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It's designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Hive on a few servers in a rack, a field site, or anywhere centralized cloud services aren’t feasible.
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
## The Platform
<p align="center">
<img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
## Core Components
### Hive (Compute Service – EC2 Alternative)
### Spinifex (Compute Service – EC2 Alternative)
Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU’s QMP interface. Designed to be straightforward and scriptable, Hive lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Hive with underlying instances.
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
@@ -27,7 +56,7 @@ Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs sim
### Viperblock (Block Storage – EBS Alternative)
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It’s built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
@@ -43,54 +72,98 @@ Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs sim
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that’s in a datacenter, remote site, or sensitive environment.
- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under Apache 2.0. Fork it, extend it, or deploy it as-is.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
## Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at [https://docs.mulgadc.com](https://docs.mulgadc.com/) for maintaining and installing Hive.
Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
### Single node install
### Bare Metal ISO
The installation is straightforward to set up and running on a single node for testing purposes.
The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
```bash
exportAWS_PROFILE=hive
exportAWS_REGION=ap-southeast-2-sc
curl -fLO https://iso.mulgadc.com/spinifex.iso
```
```bash
curl https://install.mulgadc.com/ | bash
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
hive admin init --region $AWS_REGION
### Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
>*Prerequisite:* Linux bridge for networking.
Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
Each component can be developed independently. See component-specific documentation for focused development guides.
## Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
<p align="center">
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console — dashboard view" width="900">
</p>
The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
```bash
open https://YOUR_NODE_IP:3000
```
- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
## Development Philosophy
### Built by Engineers, For Engineers
Hive is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
**Real-World Experience:**
@@ -101,7 +174,7 @@ Hive is developed by experienced infrastructure engineers with deep AWS expertis
### AI-Assisted Development
While Hive is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
While Spinifex is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
**How We Use Claude Code:**
@@ -117,8 +190,8 @@ While Hive is architected and implemented by experienced engineers, we leverage
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
This hybrid approach ensures Hive benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## License
Hive is open source under the Apache 2.0 License. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy itanywhere you need reliable infrastructure without depending on centralized cloud platforms.
# API request throttling (per-account, per-action token bucket)
[ratelimit]
enabled=false
rate=20# Default sustained requests per second per account per action
burst=100# Default maximum burst capacity
[ratelimit.action.RunInstances]
rate=2
burst=40
[ratelimit.action.TerminateInstances]
rate=2
burst=40
[ratelimit.action.StartInstances]
rate=2
burst=40
[ratelimit.action.StopInstances]
rate=2
burst=40
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.