* test(ddil): flip Scenarios A/B/C to real assertions
Replaces the t.Skip stubs landed in #204 with the assertions described in
docs/development/improvements/ddil-e2e-test-harness.md §3:
- A NATSKill — KillNATS on node3, assert ModeStandalone within 30s,
/local/instances still serves on the isolated node, majority /health
on node1+node2 stays up, witness counters on all three nodes keep
advancing, snapshot survives reconnect.
- B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert
ModeStandalone within 30s of restart, recovered instances present
with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post
PID equality dropped — see file comment; recovery + a non-zero PID
give the same regression coverage without coupling to the daemon
unit's KillMode.
- C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep
serving /health and /local/instances, isolated node3 reports
ModeStandalone with /local/* responsive, witness counter on the
partitioned side keeps advancing, heal converges and snapshot is
preserved.
Supporting changes:
- harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore,
the deployed daemon ClusterManager binds [nodes.<id>.daemon].host.
The Phase-1 stub used a placeholder that would have blocked any
/local/* assertion against the real cluster.
- harness/witness.go — export WitnessVM.Terminate so scenarios can
clean up their counter VMs from t.Cleanup.
- scenarios/main_test.go — requireLiveCluster + launchWitnesses
helpers to keep the three flipped scenarios tight.
- TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED;
TestCoverageDrift continues to enforce alignment with the package.
Validation: built with -tags=e2e, dry-run scenarios pass, coverage
drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against
tofu cluster is the next step.
* fix(ddil): countDropRules — single iptables -S call + filter
Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs
across both chains, but iptables -S takes a single optional chain — the
second -S errors to stderr, the combined SSH buffer poisons the grep
input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by
TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState.
Switch to one `iptables -S` (lists every chain in iptables-save format),
filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences
sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin.
Error path now also surfaces the raw SSH output for the next time this
breaks somewhere unexpected.
* ci(e2e-ddil): export spinifex AWS creds for witness SDK
The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.
spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.
* fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope
Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually
green on the tofu cluster. mulga-siv-56.
1. Witness auto-discovers the smallest registered instance type
(DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded
t2.micro default returned InvalidInstanceType on every Scenario A/B/C
launch because the cluster doesn't register that type. DDIL_WITNESS_-
INSTANCE_TYPE stays as an operator override.
2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/
SKIP lines instead of a JSON wall. results.json artifact preserved for
future flake-rate dashboarding.
3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope
<Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>.
aws-sdk-go v1's ec2query handler rejects the latter with
SerializationError and drops the code, leaving Go callers without an
awserr.Code() to inspect. New SDK round-trip test in gateway_test.go
guards the regression.
* ci(e2e-ddil): install gotestsum from prebuilt tarball
`go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted
runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose
tokeninternal static-assert pegs a narrow Go window (~1.22). Both older
and newer runner Go versions trip `invalid array length -delta * delta`.
A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely
without changing the gotestsum version pin.
* ci(e2e-ddil): collect cluster logs on failure
Witness `findHost` is currently failing with "could not locate QEMU host
for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is
visible on any node. Cluster gets destroyed during teardown, so logs
have to be pulled before that runs.
Adds a failure-only step that ssh's each node and bundles
journalctl spinifex-*, process listings, and state dirs into the
ddil-results artifact for offline triage.
* fix(ddil): support tap-mode QEMU networking in witness harness
findHost no longer extracts hostfwd port — production launches go through
auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never
hostfwd. Witness was timing out across all 3 nodes.
Discover guest IP via DescribeInstances.PrivateIpAddress (populated by
PrepareRunInstances on ENI auto-create) and tunnel SSH through the host
node to guestIP:22. Host routes to the tap bridge subnet, so the relay
works without runner-side bridge routing.
WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed.
Closes mulga-siv-57.
* fix(ddil): SSH witness via public IP, drop unreachable tunnel path
tofu-cluster bootstraps with --external-mode=pool which forces
dev_networking=false → no QEMU hostfwd dev NIC and no host route to the
ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to
host kernel). Previous tunnel-via-host fix could find the host but
couldn't dial the guest.
Switch to the path shell e2e already uses: default subnet has
MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external
pool, and the runner reaches that IP via the same WAN subnet as the
cluster nodes. No host relay required, survives Scenario C (partition
only DROPs peer cluster IPs, not the runner's source IP).
Workflow now scp's ~/.ssh/spinifex-key from the primary node (the
counterpart to the public material bootstrap-install.sh imports) and
exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against
cloud-init's authorized_keys.
WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed.
Continues mulga-siv-57.
* fix(ddil): tunnel guest SSH through host node
EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet,
which is OVN-routed — reachable from any cluster node, not from the
off-cluster runner sitting on the DDIL_NODES WAN subnet.
Restore the host-jump pattern: SSH the hosting node (already known from
findHost), dial publicIP:22 over that connection, run a guest SSH
handshake over the tunnel. Symmetric with how shell e2e reaches guests
from inside a node.
Continues mulga-siv-57.
* fix(ddil): wait for witness sshd before baseline read
Previous run reached the guest EIP via the host tunnel but hit
'Connection refused' — sshd hadn't bound yet. waitForRunning only
checks EC2 state, which flips on QEMU start, not cloud-init completion.
The OVN dnat_and_snat rule for the auto-allocated EIP is also written
asynchronously, so the tunnel can land before either layer is ready.
awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for
up to 90s, retrying on transient transport errors (refused/reset/EOF/
timeout/no-route) and failing fast on auth errors so a bad key surfaces
immediately. Called once after resolvePublicIP, before the baseline
counter read.
Continues mulga-siv-57.
* fix(ddil): case-insensitive transient SSH error match
awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh
emits "ssh: rejected: connect failed (Connection refused)" with capital
C — strings.Contains returned false and the helper bailed on first
attempt instead of polling. Last run failed in 49s, not the 90s budget.
Lowercase the error before matching, and add "connect failed" as the
canonical signal for "jump host dial through tunnel failed".
Continues mulga-siv-57.
* fix(ddil): default guest SSH user to ec2-user
Shell E2E SSHes the same ami-ubuntu-* images as ec2-user
(tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu
and hit 'unable to authenticate, attempted methods [none publickey]' —
authorized_keys is provisioned under the AMI default account, which is
ec2-user, not ubuntu.
Continues mulga-siv-57.
* fix(ddil): wait for witness-counter.service to seed /var/lib/counter
awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables
witness-counter.service runs in the final cloud-init stage — after sshd
binds. Baseline read landed in the window where sshd accepts but
ExecStartPre hasn't created /var/lib/counter:
cat: /var/lib/counter: No such file or directory
Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the
full path (tunnel + SSH handshake + cat) and retries on the transport
errors plus "no such file or directory". Single 2-minute budget covers
cloud-init final stage on tofu-cluster hardware.
Continues mulga-siv-57.
* fix(systemd): drop Wants=spinifex-nats from daemon unit
Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately
killed spinifex-nats was getting pulled back up the moment the daemon
restarted — masking the DDIL Tier 1 standalone-startup path and breaking
the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact:
spinifex.target's Wants= activates every spinifex unit on boot. After=
remains for ordering and is non-blocking when NATS is absent.
scripts/test-ddil-phase1.sh hit the same trap and worked around it with a
runtime drop-in plus sentinel file; the fix here makes that workaround
unnecessary for any caller that restarts the daemon while NATS is stopped.
* feat(daemon): peer-health probe drives standalone mode on partition
Mode() is now derived from two atomic signals — natsConnected (NATS client
TCP link, flipped by the existing disconnect/reconnect callbacks) and
peersReachable (any peer daemon's /health responded in the most recent
probe cycle). Reports cluster iff both are up; standalone otherwise.
The peer-health probe is the missing signal for DDIL Scenario C: an iptables
DROP between peer IPs blocks NATS cluster-routing traffic but leaves each
node's local NATS client TCP connection healthy, so nats.Conn.IsConnected()
keeps returning true and onNATSDisconnect never fires. The probe dials each
peer's /health over the cluster network with a 2s timeout on a 2s ticker;
worst-case detection of total partition is ~6s on a 3-node cluster, well
inside the 30s Scenario C budget. Single-node clusters keep peersReachable
permanently true (nothing to probe) so the derived mode flips to cluster as
soon as the local NATS connect succeeds.
The probe runs from startLocal so the partition signal exists even before
(or in the absence of) a successful NATS connect. It does not touch NATS or
JetStream, so the §1e Tier 1 invariant asserted by
assertNoClusterServicesInitialised is unaffected.
Tests:
- TestMode_RequiresBothSignals — every nats×peers combination.
- TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test.
- TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses.
- TestMonitorPeerReachability_EndToEnd — ticker-driven false→true.
Closes mulga-siv-58.
* fix(daemon): peer probe falls back to local daemon port
`spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml;
remote-node blocks carry just `host = "<ip>"` with no daemon sub-table.
So `peer.Daemon.Host` was always empty in a deployed cluster and the
probe failed closed, leaving peersReachable permanently false.
Tofu-cluster run 25665158964 surfaced this as a Scenario A regression:
after StartNATS, natsConnected flipped true on reconnect but Mode()
stayed standalone because the derived signal required peersReachable.
`a_nats_kill_test.go:75: node3 did not return to cluster mode … (still
reporting another mode)`.
Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port
when present and otherwise reuses the local daemon's port (clusters are
symmetric — every node binds the same port the template renders). New
TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port-
available case; TestProbePeerHealth_RejectsBadAddr is updated to clear
self config so it still asserts the fail-closed path.
* fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit
DDIL Scenario B asserts that `systemctl restart spinifex-daemon`
preserves every guest VM so the new daemon can reattach via the
persisted local state file. The previous setupShutdown SIGTERM handler
called vmMgr.StopAll(), gracefully QMP-powering-down every guest before
the daemon exited — ~90s wall-clock and zero surviving QEMU on the
other side. CI run 25666369604 surfaced this as
`recovered instance i-... has no live QEMU PID`.
Two changes restore the Tier 1 contract:
- setupShutdown no longer calls StopAll. It still flips d.shuttingDown
so crash/restart handlers bail out, persists state, unsubscribes from
NATS, and exits. VMs are only stopped when coordinated cluster
shutdown drains them — handleShutdownDrain still calls StopAll and
flips the flag before the SIGTERM path runs. Operator-visible
behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs
running. To stop VMs, use the cluster-shutdown topics or terminate
via the AWS API.
- The systemd unit gains `KillMode=process` so systemd only signals the
daemon's main PID. With the default control-group mode, QEMU
children were killed when systemd tore down the daemon's cgroup —
defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the
new fast-exit grace window.
Also nil-guards natsConn.Close in the shutdown path: when a daemon
restarts while NATS is unreachable (Scenario B), natsConn is nil and
the close would panic.
Mulga-siv-58 (reopened).
* fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart)
QEMU stdout/stderr were piped to daemon-side scanner goroutines.
When the daemon exits (e.g. `systemctl restart spinifex-daemon`),
the pipe read ends close; QEMU's next write triggers SIGPIPE and
the guest dies — defeating DDIL Scenario B's "daemon restart
preserves running instances" guarantee.
Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr
instead. The FD is dup'd into QEMU at fork, so closing it on the
daemon side after Start() leaves QEMU writing into the file
indefinitely. POSIX O_APPEND makes concurrent QEMU writes
interleave atomically per line, so a single shared log is safe and
avoids per-instance clutter in RuntimeDir.
Trade-off: QEMU lines no longer land in journalctl -u
spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly
(path logged at instance start under the qemu_log field).
mulga-siv-58 (DDIL Scenario B iteration 4)
* fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround
Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU
ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's
`PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()`
falling through to `os.TempDir()` (=/tmp):
- spinifex-daemon HOME=/var/lib/spinifex/spinifex, so
`$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not
exist and pidPath() returns /tmp.
- Every daemon start gets a fresh private /tmp namespace. QEMU stays
in the OLD namespace after the daemon dies (mount namespaces aren't
changed by reparenting). The new daemon's /tmp is empty.
- `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace
the file doesn't exist, so PID=0.
Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon
unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d,
already in ReadWritePaths) and survives daemon restart. pidPath()
short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and
RuntimeDir() all resolve to the stable path. Both daemon incarnations
and QEMU children see the same files.
Revert iteration 4's pipe→file QEMU log change (commit 8a9a6f9f).
SIGPIPE was never killing QEMU, and the file-redirect dropped QEMU
lines from `journalctl -u spinifex-daemon` for no gain.
mulga-siv-58 (DDIL Scenario B iteration 5)
* feat(daemon): siv-60 heal reconcile — phase 1 (probe-heal trigger) (#211)
* feat(daemon): wire reconcileOnHeal on peer-probe heal edge (siv-60 phase 1)
Scenario C heal triggered zero resync: the local NATS client stayed
connected to its loopback server throughout the partition, so
onNATSReconnect never fired. The peer-probe false→true transition was
the only heal signal but had no resync action wired to it.
Adds reconcileOnHeal, called from both edges and coalesced via a
reconciling atomic.Bool so the two callbacks landing on the same heal
don't double-push. For phase 1 the body is the existing WriteState
(re-asserts our per-node row in the instance-state KV). Phase 2 will
extend it to pull KV → merge → push for the stopped/terminated buckets
and re-fire per-node bootstrap.
Plan: docs/development/feature/ddil-heal-reconcile.md
* feat(daemon): siv-60 phase 2 — idempotent default-VPC bootstrap re-fire on heal
Calls ensureDefaultVPCInfrastructure() from reconcileOnHeal after the
WriteState push. Idempotent: each branch (default VPC lookup, IGW
attach, default-SG create, default route insert) checks for existing
infrastructure before creating. Covers the failure mode where
startCluster ran during partition and the initial creates returned
errors against an unreachable NATS — without a heal-time retry the
cluster would be stuck without a default VPC until the next daemon
restart.
Scope note: the bead's "pull KV → merge → push" assumed the daemon
cached peer-node running state, stopped instances, terminated
instances, and VPC/IGW/SG state locally. The codebase audit (see plan
doc) confirms it doesn't — every consumer reads its KV bucket fresh
per request, and vmMgr only tracks VMs running on this node. The
bootstrap re-fire is the one heal-time action that has a real
in-memory target. Watchdog sweep is unnecessary because
vm.Manager.CrashHandler is event-driven from cmd.Wait().
* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal (siv-60 follow-up) (#212)
* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal
DDIL Scenario C regressed at the witness-SSH step after siv-60
landed in #211. Root cause: the first peer-probe tick on multi-node
startup flips peersReachable false→true (its zero value is false), so
reconcileOnHeal fires concurrently with startCluster's own
ensureDefaultVPCInfrastructure call. The Describe→Create→Attach
sequence is not cluster-wide singleton, so multiple daemons + double
calls per daemon raced and produced duplicate IGW attachments,
corrupting default-VPC routing — which broke the witness VM's tunnel
SSH path once Scenario C's partition was installed.
Phase 1 (the heal trigger + coalescing) is the meaningful fix and
stays. Phase 2's bootstrap re-fire needs leader election or a
debounce against the startup-time path before being re-introduced;
leaving an inline NOTE so the next attempt has the context.
Failed run: actions/runs/25706585637 (Scenario C, attempts 1+2).
Last green DDIL run: 25671044714 (pre-merge, SHA 40e3d847).
* fix(daemon): init peersReachable=true to suppress startup heal edge
DDIL Scenario C kept flaking at attempt-1 even after dropping the
bootstrap re-fire. Remaining trigger: peersReachable starts as the
zero value (false) on multi-node clusters, and the first probePeersOnce
tick flips it false→true. That counts as a "heal" edge under siv-60
phase 1, so reconcileOnHeal fires at boot — concurrent with
startCluster's bootstrap — perturbing timing in a way that the
harness's Scenario C exposes (witness sshd handshake EOF mid-test).
Initialise peersReachable optimistically. Mode() also requires
natsConnected (starts false), so the optimistic peer init can never
falsely report cluster mode — but it does eliminate the spurious
startup edge. Real partition→heal transitions still fire the trigger
exactly as designed: true→false on partition, false→true on heal.
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* feat(viperblockd): retry LoadState on transient backend errors
ebs.mount used to call vb.LoadState once and treat any failure as fatal.
When predastore was still warming up after a coordinated daemon restart,
the backend Read came back with a network/timeout error, viperblock
folded it into "block size is 0 in both local and backend state", and
the daemon's recovery path then ran the cleanup branch — destroying
DeleteOnTermination=true volumes (mulga-siv-25).
Wrap LoadState in retryLoadState: 5 attempts, 200ms * 1.5^n backoff
(total worst-case ~3.7s, well inside the 30s ebs.mount NATS deadline).
Retry only on viperblock.ErrStateBackendUnavailable; ErrStateNotFound
and any unclassified error fail fast so genuine problems are still
visible.
Retry loop is parameterised over sleep and loadFn so the retry policy
itself is unit-tested without spinning up a real VB.
* chore: bump viperblock to v1.2.2-0.20260511064619-1aa54f0dc1f6
Pulls in the ErrStateNotFound / ErrStateBackendUnavailable sentinels
introduced by mulgadc/viperblock#11 (merged on dev). Required for the
retry wrapper added in the previous commit to compile against the
public viperblock module.
Stops the P0 data-loss path where `systemctl restart spinifex-daemon`
destroys user volumes flagged `DeleteOnTermination=true`.
Previously, when daemon recovery couldn't relaunch (Run error) or
reconnect (QMP attach failure) to a previously-running instance, the
code called `MarkFailed` — which transitions to `terminated`, runs the
full destructive cleanup (`terminateCleanup`), and deletes volumes per
`DeleteOnTermination`. A benign operator action (daemon restart) was
treated identically to a user-initiated `ec2.TerminateInstances`.
New `MarkRecoveryFailed` transitions to `StateError` with a
`Server.RecoveryFailed` StateReason, runs only the non-destructive
`stopCleanup` chain (graceful unmount + tap teardown + GPU release +
resource deallocation), and persists the running set. Volumes, public
IPs, ENIs, and placement-group membership are preserved so the operator
can either retry recovery or issue an explicit terminate (which will
then honour DoT as normal). The instance stays in the local map and
`OnInstanceDown` is NOT fired, keeping `ec2.cmd.<id>` live so operator
commands can still reach this node.
`classifyRestoredInstances` now skips relaunch + resource re-allocation
for instances already in `StateError`, so a recovery-failed instance
stays put across subsequent daemon restarts until the operator acts.
Medium-term (viperblock "block size 0" hard-vs-soft classification +
mount retry-with-backoff) and long-term (audit of every non-user-
initiated cleanup caller) tiers remain open on the siv-25 plan.
* refactor(daemon): extract ModifyInstanceAttribute to service (1b-pre phase 2a)
Collapse handleEC2ModifyInstanceAttribute to a thin Pattern A wrapper over
InstanceServiceImpl.ModifyInstanceAttribute, mirroring the Describe extraction
in 719ac63 (phase 1). The InstanceService interface and StoppedInstanceStore
each gain one method (ModifyInstanceAttribute, WriteStoppedInstance); the
daemon's JetStreamManager already implements WriteStoppedInstance, so no
production code outside the daemon handler changes.
Logic-level tests move from daemon/daemon_handlers_test.go to
handlers/ec2/instance/service_impl_test.go (12 cases against an in-memory
fakeStoppedStore). The daemon keeps one NATS round-trip wrapper smoke test.
The error-injection cases in daemon_handlers_instance_test.go remain in place
but now require daemonWithFakeStateStore to rewire d.instanceService with
the fake store — the previous `d.stateStore = store` swap no longer reaches
the new code path.
No behaviour change: SourceDestCheck stays a state-agnostic no-op, the
ownership/visibility check still gates after the stopped-state gate, and the
success response payload remains `{}`.
* refactor(daemon): extract TerminateStoppedInstance to service (1b-pre phase 2b)
Collapses ~140 lines of daemon termination logic into a one-liner
handleNATSRequest wrapper plus an InstanceServiceImpl method, matching
the Pattern A precedent from phase 1 (Describes) and phase 2a
(ModifyInstanceAttribute). Adds VolumeDeleter, ENIDeleter, and
PublicIPReleaser dep interfaces (wired via SetTerminationDeps) so the
service can clean up volumes, ENI, and EIP without depending on daemon
internals.
Hoists publishNATEvent off the daemon into utils.PublishNATEvent so the
service can drive vpc.delete-nat directly. Removes the
terminateStoppedInstanceRequest daemon-local payload in favour of the
exported TerminateStoppedInstanceInput on the service interface.
Tests: 13 service-level cases against an in-memory fake (missing ID,
nil store, load error, not found, not stopped, cross-tenant,
happy path, terminated-write failure aborts, retry-once delete,
user volume deleted, no-deleter graceful skip, internal volumes via
NATS, public IP release, ENI delete). The five JetStream-backed
daemon Terminate tests collapse to one wrapper smoke
(WritesToTerminatedKV); the six error-injection tests in
daemon_handlers_instance_test.go continue to drive the wrapper through
fakeStateStore for wire-format coverage.
Refs mulga-siv-21.
* refactor(daemon): extract StartStoppedInstance to service (1b-pre phase 2c)
Daemon's handleEC2StartStoppedInstance collapses to a Pattern A handleNATSRequest
wrapper over InstanceServiceImpl.StartStoppedInstance. Promotes ResourceCapacityProvider
to InstanceTypeAllocator with Allocate/Deallocate/CanAllocate/InstanceTypes, and adds
a GPUClaimer interface returning (pciAddress, xvgaEnabled, err) so the instance package
stays free of the gpu package. Daemon ResourceManager gains exported wrappers around
its internal allocator methods; a daemonGPUClaimer adapter wires gpu.Manager + the
GPU model overrides through SetGPUClaimer.
No behaviour change. New service-level tests cover the moved logic; existing
daemon integration tests continue to validate the end-to-end path through the
wrapper.
* ci(e2e-ddil): export spinifex AWS creds for witness SDK
The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.
spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.
* Revert "ci(e2e-ddil): export spinifex AWS creds for witness SDK"
This reverts commit 5b890162a6.
* refactor(daemon): extract RunInstances + ec2.cmd handlers to service (1b-pre phase 2d)
Final sub-phase of mulga-siv-21. Moves the remaining instance mutators
from spinifex/daemon/daemon_handlers_instance.go into InstanceServiceImpl:
- PrepareRunInstances: input validation, capacity allocation, per-instance
RunInstance + ENI auto-create + public IP assignment.
- LaunchRunInstances: volume preparation, GPU claim, vmMgr.Run loop.
- RunInstances: synchronous Prepare+Launch convenience wrapper used by
the InstanceService interface for non-daemon callers.
- RebootInstance, StartInstance, StopOrTerminateInstance: extract the
three ec2.cmd sub-handlers; daemon.handleEC2Events shrinks to a
dispatcher that looks up the VM and calls the matching service method.
The daemon's RunInstances handler is now a thin orchestrator that splits
the call into Prepare and Launch around the early NATS respond so the
original respond-then-launch timing is preserved (AWS clients see the
reservation before volumes prepare and VMs launch).
New service deps wired via SetRunInstancesDeps:
- AMIMetaLoader (handlers/ec2/image.ImageServiceImpl)
- KeyPairValidator (handlers/ec2/key.KeyServiceImpl)
- ENICreator + SubnetInfo (new daemonENICreator adapter over VPCServiceImpl
to avoid an instance↔vpc cyclic import)
- PublicIPAllocator (handlers/ec2/vpc.ExternalIPAM)
No behaviour change. Tests retarget the service directly for the new
error paths; existing daemon-side RunInstances handler tests continue to
pass via the wrapper.
* test(instance): add LaunchRunInstances + setter coverage for phase 2d
* refactor(daemon): extract Describe instance handlers to service (1b-pre phase 1)
Collapse the Describe family in daemon_handlers_instance.go to thin Pattern A
wrappers over a transport-neutral InstanceServiceImpl. This brings the read
path in line with the rest of the codebase and is the prerequisite for 1b's
local HTTP API (mulga-siv-14, daemon-local-autonomy plan).
Extracted: DescribeInstances, DescribeInstanceTypes, DescribeInstanceAttribute,
DescribeStoppedInstances, DescribeTerminatedInstances. Filter helpers
(instanceMatchesFilters, matchTagKey/Value, DescribeInstancesValidFilters,
IsInstanceVisible) move to the instance package alongside the service methods.
InstanceService gains the new methods and two dep interfaces
(ResourceCapacityProvider, StoppedInstanceStore) so the daemon's ResourceManager
and JetStreamManager can plug in without leaking concrete types. NATSInstanceService
returns a clear error for the fan-out describes (DescribeInstances,
DescribeInstanceTypes) since cluster-wide aggregation lives in the gateway's
scatter-gather helpers, not a single-node NATS request.
Behaviour changes: malformed JSON on DescribeInstanceAttribute now responds
with ValidationError (matching every other Pattern A handler) instead of
ServerInternal. Test updated.
Mutators (RunInstances, ModifyInstanceAttribute, StartStoppedInstance,
TerminateStoppedInstance, sub-handlers) remain Pattern B for now; follow-up
phases of mulga-siv-14 will collapse them.
* feat(daemon): local-first instance state persistence (1a)
Make /var/lib/spinifex/state/instance-state.json the source of truth for a
node's VMs; JetStream KV becomes a best-effort cluster cache. The daemon now
survives KV unavailability without losing instance state.
- New local_state.go: atomic write (tmp+fsync+rename), schema_version gate,
missing-file = fresh install, corrupt = refuse start.
- Daemon.WriteState writes the local file under Instances.Mu, then calls
jsManager.WriteStateBestEffort with a 1s timeout. KV failures log and
continue; clients never see "KV down" during partition.
- Daemon.LoadState reads from the local file only — no KV fallback (a fresh
node has empty KV anyway, so silent fallback would mask bugs).
- restoreInstances now returns error and Start() aborts on corrupt local
state rather than silently starting empty.
- jsManager.WriteState no longer self-locks (caller holds Mu); added
WriteStateBestEffort and shared marshalInstanceState helper.
Reconnect-driven KV resync deferred to siv-16 (1c owns the callback wiring).
Bead: mulga-siv-15
Plan: docs/development/feature/daemon-local-state-persistence.md
* fix(daemon): root local state at DataDir not BaseDir
LocalStatePath was falling back to BaseDir when DataDir was empty, putting
the file at /var/lib/spinifex/spinifex/state/instance-state.json (under the
daemon's own subdir) instead of the documented
/var/lib/spinifex/state/instance-state.json (the platform-shared root).
Drop the BaseDir fallback. Empty DataDir resolves to DefaultDataDir
(/var/lib/spinifex). Tests now set DataDir alongside BaseDir on their config
fixtures.
* feat(daemon): graceful NATS disconnect/reconnect handling (1c)
Daemon now flips a `mode` field (cluster|standalone) on NATS
disconnect/reconnect via optional callback hooks added to
utils.ConnectNATS. Reconnect bumps natsRetryCount and triggers 1a's
full local-state push to KV. NATSRequest/NATSScatterGather fail fast
with ErrClusterUnavailable when the connection is down. Gateway
short-circuits NATS-bound requests with a 503 ServiceUnavailable
carrying "cluster unavailable: NATS disconnected — check daemon
/local/status" so AWS clients see the failure mode immediately
instead of hanging on per-call timeouts.
Plan: docs/development/feature/daemon-autonomy-1c-disconnect.md
* fix(gateway): SigV4 middleware short-circuits on NATS disconnect
The 1c gateway.Request() IsConnected check (gateway.go:234) was
unreachable for any authenticated path because SigV4 auth middleware
runs first and calls IAMService.LookupAccessKey over JetStream KV. With
NATS down the lookup hangs ~5s on context.DeadlineExceeded and the
gateway returns 500 InternalError. AWS SDK retries it twice for ~18s of
user-visible latency instead of the <100ms 503 promised by the 1c plan.
Mirror the IsConnected check at the top of SigV4AuthMiddleware (after
credential parse so malformed auth headers still get the right SigV4
error code, before the IAM lookup). The catch-all in gateway.Request()
stays as defence-in-depth for any future unauthed paths.
Coverage debt picked up alongside: 1c utils.NATSRequest /
NATSScatterGather fail-fast paths, daemon Mode/onNATSDisconnect/
onNATSReconnect/localStatePath/Write/LoadState branches, and 1b-pre
InstanceServiceImpl Describe* methods (partial siv-22).
Bead: mulga-siv-23
* feat(daemon): read-only /local/* HTTP API (1b)
Wires GET /local/instances, GET /local/instances/{id}, GET /local/status
onto the existing ClusterManager TLS listener so observers can read this
node's instance state and connectivity mode while NATS is unreachable.
Also fixes a pre-existing race in WriteState: json.Marshal walked shared
*vm.VM pointers from SnapshotMap while concurrent TransitionState calls
mutated VM fields under the manager lock. Marshal now runs inside
vmMgr.View; WriteLocalStateBytes / WriteStateBytesBestEffort accept the
pre-marshalled JSON and commit lock-free. TestTransitionState_Concurrent
Transitions is race-clean across 30 iterations.
Adds Daemon.stateRevision (atomic.Uint64) bumped on every successful
local-state write, surfaced as /local/status.revision so observers can
detect changes without diffing payloads.
* feat(daemon): two-phase startup — survive NATS down (1d)
Split daemon Start() into startLocal() (HTTPS, mgmt bridge, OVS plumber,
OOM, local state load — no NATS) and startCluster() (background, infinite
retry exp backoff cap 60s for NATS connect, then JetStream + services +
restoreInstances + subscribeAll + heartbeat). Daemon never process-exits
on NATS failure; instead it stays up in standalone mode and reconnects in
the background.
Drop ExecStartPre=wait-for-nats.sh from spinifex-daemon.service — local
bootstrap no longer requires NATS to be reachable.
ConnectNATSWithRetry now supports infinite retry (WithMaxWait(0)),
context cancellation (WithContext), per-attempt error callback for the
nats_retry_count operator signal on /local/status, and a configurable
backoff cap (WithMaxRetryDelay).
bead: mulga-siv-17 (closes)
* feat(daemon): Tier 1 invariant guard for cluster-scoped service init (1e-audit)
Make the DDIL §1e-audit invariant explicit: startLocal must not initialise any
NATS-dependent or JetStream-KV-backed handle. Adds assertNoClusterServicesInitialised
as a runtime guard at the end of startLocal that fails fast if a future edit
hoists a cluster-scoped service into Phase 1, plus comment blocks naming the
invariant on both startLocal and startCluster.
Code audit confirms the existing 1d split already keeps every cluster-scoped
bucket out of Phase 1; this change pins the property so it cannot regress
silently. Tier table for the 25 daemon-process KV buckets is finalised in
docs/development/feature/daemon-local-autonomy.md §1e-audit.
bead: mulga-siv-19 (closes)
* feat(daemon): NATS observability hardening (1d-obs)
Closes the residual visibility gap left by the 1d non-blocking startup:
- Extend /local/status with kv_sync_failures, last_kv_sync_at,
last_kv_sync_error so an operator polling the local API sees prolonged
divergence between the local file and JetStream KV.
- Add a KVSyncObserver interface on JetStreamManager so the daemon can
count best-effort write outcomes without coupling the manager to
daemon state.
- Escalate utils.ConnectNATSWithRetry per-attempt log to slog.Error
(rate-limited to once per minute, with disconnected_for context) once
attempt count crosses the natsRetryEscalateAttempt threshold (~30
attempts ≈ ~30 min disconnected at the 60s backoff cap), so a stuck
NATS config surfaces in error logs instead of routine warn noise.
Prometheus metrics export deferred until a system-wide telemetry plan
(likely OTel) lands; the structured /local/status fields plus the
escalated retry log cover the Tier 1 visibility gap.
* feat(daemon): opt-in SPINIFEX_REQUIRE_NATS strict-startup mode (1d-strict)
When SPINIFEX_REQUIRE_NATS=1, startCluster's first NATS connect uses a
bounded ~30s retry and the daemon aborts on timeout. Default behaviour
(env unset) keeps the Tier 1 infinite-retry path so prod / edge nodes
still survive a missing NATS at boot. Restores pre-DDIL fail-fast UX
for dev / test / single-node deploys without flipping the prod default.
mulga-siv-44.
* test(scripts): systemd-level smoke for DDIL phase 1
Bash harness exercising daemon-local-autonomy on a real `make deploy` node:
baseline cluster mode → kill spinifex-nats → assert standalone + retry
counter advance + /local/instances served → restart NATS → assert mode flip
+ KV resync stamp → SPINIFEX_REQUIRE_NATS=1 bounded-exit. Cleanup trap
restores both services. Complements existing unit coverage by validating
real systemd wiring on port 8443.
* fix(test-ddil): daemon ClusterManager binds 4432 not 8443
8443 is predastore. Daemon's HTTPS ClusterManager (where /local/* lives)
binds [nodes.node1.daemon].host = 0.0.0.0:4432 in the deployed config.
Plan doc daemon-local-autonomy.md:112 incorrectly said 8443; actual code
uses d.config.Daemon.Host.
* fix(test-ddil): correct retry-counter site + bypass NATS Wants= for strict test
Two bugs in the v1 script:
1) nats_retry_count assertion ran during disconnect — but the counter
only bumps on disconnect→reconnect *cycles* (daemon.go:321,
onNATSReconnect). Moved assertion to step 3 (restart NATS) where
the reconnect callback fires.
2) Step 4 stopped spinifex-nats then ran 'systemctl restart
spinifex-daemon' — but the daemon unit's Wants=spinifex-nats.service
re-pulled NATS up via the Wants= chain, so the strict-mode 30s
timeout never fired. systemctl mask refuses on /etc-installed
units, and --runtime mask is shadowed by /etc. Block via a runtime
drop-in with ConditionPathExists=! pointing at a sentinel file.
Verified: all 5 checks pass on a single-node make deploy box; daemon
exits 32s after restart with SPINIFEX_REQUIRE_NATS=1 + NATS blocked.
* fix(admin): render data_dir into spinifex.toml so non-root nodes write local state
Pseudo-multinode E2E (rootless tf-user, ~/nodeN data roots) hit
"mkdir /var/lib/spinifex/state: permission denied" on every RunInstances:
Config.DataDir was empty because the template never emitted a data_dir
key, and LocalStatePath fell back to DefaultDataDir (/var/lib/spinifex).
Surfaced after 8eb9c5fa dropped the BaseDir fallback in LocalStatePath —
previously SPINIFEX_BASE_DIR=$DATA_DIR/spinifex/ (set by start-dev.sh)
masked the missing data_dir.
ConfigSettings.DataDir is already populated to spxRoot at admin
init/join; render it under [nodes.X] so it actually reaches the daemon.
reconcileIGW was non-idempotent and gated on "external switch missing",
so once a prior pass left the ext switch + LRP in place without the
localnet LSP, every subsequent vpcd start silently skipped the function
and the partial topology persisted forever (br-ext never got its patch
port; external traffic failed).
Convert each step to Get-then-Create idempotent and drop the cascade
rollbacks — partial state is now safe to leave for the next pass to
heal. Add FindStaticRoute so the default route is deduped instead of
duplicated on every retry. Drop the ext-switch gate in Reconcile and
ReconcileFromKV; rename IGWsCreated to IGWsReconciled to reflect the
new "called" semantics.
NAT rules are purely event-driven — only AssociateAddress creates them via
vpc.add-nat. No startup reconcile path means any OVN NB DB state drift
silently breaks EIP inbound traffic after a node reboot.
Add Step 5 to ReconcileFromKV that reads associated EIPs from the
spinifex-vpc-elastic-ips KV bucket and ensures each has a dnat_and_snat rule
in OVN NB DB. Correct rules are skipped (idempotent). Stale rules (wrong
LogicalIP) are replaced. Missing rules are created.
Also adds FindNATByExternalIP to OVNClient interface + LiveOVNClient/Mock
implementations, and three new tests covering: missing rule creation,
idempotent skip on correct rule, and stale rule replacement.
The internal-ALB approach was wrong on two counts:
- ALB_ENI_MAC is the OVN VPC ENI MAC (lives on br-int), never appears
in br-mgmt neighbour table, so the mgmt IP lookup always fails.
- Switching to internal masked the real bug: OVN EIP DNAT not restoring
properly after a host reboot, which is exactly what this test should catch.
Restore internet-facing ALB and EIP-based traffic bursts. The post-reboot
traffic failure (Phase 8.6) is the correct signal for the outstanding OVN
EIP DNAT persistence bug.
Keep the Phase 8.3 LaunchTime fix: two-hop SSH to VPC private IPs doesn't
work in single-node macvlan (host has no route to VPC subnets). LaunchTime
from describe-instances is fast and topology-independent.
Two changes to fix cell-18 (single/static/debian-13/reboot-single):
1. Switch ALB to --scheme internal.
Internet-facing ALBs work pre-reboot on single-node but post-reboot the
OVN EIP DNAT (nat-addresses=router GARP) is not re-established in time,
causing 100% curl timeouts. Internal ALB routes traffic via br-mgmt using
the mgmt IP discovered from the ALB ENI MAC in the br-mgmt neighbour table,
bypassing OVN external NAT entirely.
2. Replace Phase 8.3 guest SSH uptime check with LaunchTime assertion.
Two-hop SSH to VPC private IPs fails in single-node macvlan: the host has
no route to VPC subnets. LaunchTime from describe-instances is reset by the
daemon on QEMU relaunch, so LaunchTime >= REBOOT_START reliably confirms
recovery without network reachability to the guest.
Separate from normal e2e. Checks out spinifex, checks out mulga scripts
(sparse: iso-builder + baremetal), builds spx + test tar via
bm-build-artifacts.sh, builds PXE artifacts (squashfs/initrd/vmlinuz)
via iso-builder with persistent cache on banksia, then runs
run-baremetal-e2e.sh against the single bare-metal node.
In the PXE path SPINIFEX_INSTALL_CALLBACK arrives on the iPXE kernel
cmdline but was never threaded into the firstboot script, so firstboot
never curled the controller's /done endpoint. The flag stayed on disk,
the node re-entered the install loop on the next PXE boot.
Fix: read SPINIFEX_INSTALL_CALLBACK from the live-environment env in
toFirstbootConfig and embed a best-effort curl into the generated
firstboot script, fired once after the success marker is written.
Interactive and USB-stick installs are unaffected (env var absent
-> callbackBlock is empty, no curl injected).
NBDKit exiting within 1s was only treated as an error if exit code != 0.
A clean exit (code=0) fell through to Mounted=true, leaving QEMU with a
dead NBD socket and VMs that never boot post-reboot.
Treat any premature exit as failure regardless of exit code.
manager_deps_test.go: drop TestDeps_VolumeMounterSatisfiesInterface,
TestDeps_StateStoreSatisfiesInterface, TestDeps_StateStoreErrorPropagates,
TestSetDeps_OverridesDeps, TestManagerHooks_InitiallyNil — they assert a
fake recorded its own input, that a fake returns an error literally
constructed in the test, or Go zero-value/direct-assignment semantics.
Compile-time interface checks already prove satisfaction.
state_test.go: replace TestEC2StateCodes_AllStatesHaveMapping and
TestEC2StateCodes_CorrectValues — both compared the production map to
its own literal. Now assert against an independent AWS contract using
ec2.InstanceStateName* SDK enum constants paired with the
AWS-documented numeric codes; a typo in production no longer
replicates into the test.
watchdog_test.go: replace TestStartPendingWatchdog_CtxCancelStopsGoroutine's
time.Sleep(10ms) "assertion" with goleak.VerifyNone. The previous form
passed even if the goroutine never returned on cancel — the harness
reaped it on test exit.
Julian Sommer