Compare commits

..

532 Commits

Author SHA1 Message Date
Julian Sommer fe59017cbb release: update go.mod for v1.3.0 2026-05-12 16:12:38 +10:00
Bryn Mailer 50eea38066 feat(predastore): aes 256 encryption at rest
Sibling changes in Spinifex to support changes made to Predastore in this PR.
2026-05-12 16:06:16 +10:00
Josh Siviour 45555dde49 test(ddil): flip Scenarios A/B/C to real assertions (#205)
* test(ddil): flip Scenarios A/B/C to real assertions

Replaces the t.Skip stubs landed in #204 with the assertions described in
docs/development/improvements/ddil-e2e-test-harness.md §3:

- A NATSKill — KillNATS on node3, assert ModeStandalone within 30s,
  /local/instances still serves on the isolated node, majority /health
  on node1+node2 stays up, witness counters on all three nodes keep
  advancing, snapshot survives reconnect.
- B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert
  ModeStandalone within 30s of restart, recovered instances present
  with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post
  PID equality dropped — see file comment; recovery + a non-zero PID
  give the same regression coverage without coupling to the daemon
  unit's KillMode.
- C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep
  serving /health and /local/instances, isolated node3 reports
  ModeStandalone with /local/* responsive, witness counter on the
  partitioned side keeps advancing, heal converges and snapshot is
  preserved.

Supporting changes:
- harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore,
  the deployed daemon ClusterManager binds [nodes.<id>.daemon].host.
  The Phase-1 stub used a placeholder that would have blocked any
  /local/* assertion against the real cluster.
- harness/witness.go — export WitnessVM.Terminate so scenarios can
  clean up their counter VMs from t.Cleanup.
- scenarios/main_test.go — requireLiveCluster + launchWitnesses
  helpers to keep the three flipped scenarios tight.
- TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED;
  TestCoverageDrift continues to enforce alignment with the package.

Validation: built with -tags=e2e, dry-run scenarios pass, coverage
drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against
tofu cluster is the next step.

* fix(ddil): countDropRules — single iptables -S call + filter

Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs
across both chains, but iptables -S takes a single optional chain — the
second -S errors to stderr, the combined SSH buffer poisons the grep
input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by
TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState.

Switch to one `iptables -S` (lists every chain in iptables-save format),
filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences
sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin.
Error path now also surfaces the raw SSH output for the next time this
breaks somewhere unexpected.

* ci(e2e-ddil): export spinifex AWS creds for witness SDK

The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.

spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.

* fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope

Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually
green on the tofu cluster. mulga-siv-56.

1. Witness auto-discovers the smallest registered instance type
   (DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded
   t2.micro default returned InvalidInstanceType on every Scenario A/B/C
   launch because the cluster doesn't register that type. DDIL_WITNESS_-
   INSTANCE_TYPE stays as an operator override.

2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/
   SKIP lines instead of a JSON wall. results.json artifact preserved for
   future flake-rate dashboarding.

3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope
   <Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>.
   aws-sdk-go v1's ec2query handler rejects the latter with
   SerializationError and drops the code, leaving Go callers without an
   awserr.Code() to inspect. New SDK round-trip test in gateway_test.go
   guards the regression.

* ci(e2e-ddil): install gotestsum from prebuilt tarball

`go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted
runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose
tokeninternal static-assert pegs a narrow Go window (~1.22). Both older
and newer runner Go versions trip `invalid array length -delta * delta`.
A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely
without changing the gotestsum version pin.

* ci(e2e-ddil): collect cluster logs on failure

Witness `findHost` is currently failing with "could not locate QEMU host
for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is
visible on any node. Cluster gets destroyed during teardown, so logs
have to be pulled before that runs.

Adds a failure-only step that ssh's each node and bundles
journalctl spinifex-*, process listings, and state dirs into the
ddil-results artifact for offline triage.

* fix(ddil): support tap-mode QEMU networking in witness harness

findHost no longer extracts hostfwd port — production launches go through
auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never
hostfwd. Witness was timing out across all 3 nodes.

Discover guest IP via DescribeInstances.PrivateIpAddress (populated by
PrepareRunInstances on ENI auto-create) and tunnel SSH through the host
node to guestIP:22. Host routes to the tap bridge subnet, so the relay
works without runner-side bridge routing.

WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed.

Closes mulga-siv-57.

* fix(ddil): SSH witness via public IP, drop unreachable tunnel path

tofu-cluster bootstraps with --external-mode=pool which forces
dev_networking=false → no QEMU hostfwd dev NIC and no host route to the
ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to
host kernel). Previous tunnel-via-host fix could find the host but
couldn't dial the guest.

Switch to the path shell e2e already uses: default subnet has
MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external
pool, and the runner reaches that IP via the same WAN subnet as the
cluster nodes. No host relay required, survives Scenario C (partition
only DROPs peer cluster IPs, not the runner's source IP).

Workflow now scp's ~/.ssh/spinifex-key from the primary node (the
counterpart to the public material bootstrap-install.sh imports) and
exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against
cloud-init's authorized_keys.

WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed.

Continues mulga-siv-57.

* fix(ddil): tunnel guest SSH through host node

EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet,
which is OVN-routed — reachable from any cluster node, not from the
off-cluster runner sitting on the DDIL_NODES WAN subnet.

Restore the host-jump pattern: SSH the hosting node (already known from
findHost), dial publicIP:22 over that connection, run a guest SSH
handshake over the tunnel. Symmetric with how shell e2e reaches guests
from inside a node.

Continues mulga-siv-57.

* fix(ddil): wait for witness sshd before baseline read

Previous run reached the guest EIP via the host tunnel but hit
'Connection refused' — sshd hadn't bound yet. waitForRunning only
checks EC2 state, which flips on QEMU start, not cloud-init completion.
The OVN dnat_and_snat rule for the auto-allocated EIP is also written
asynchronously, so the tunnel can land before either layer is ready.

awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for
up to 90s, retrying on transient transport errors (refused/reset/EOF/
timeout/no-route) and failing fast on auth errors so a bad key surfaces
immediately. Called once after resolvePublicIP, before the baseline
counter read.

Continues mulga-siv-57.

* fix(ddil): case-insensitive transient SSH error match

awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh
emits "ssh: rejected: connect failed (Connection refused)" with capital
C — strings.Contains returned false and the helper bailed on first
attempt instead of polling. Last run failed in 49s, not the 90s budget.

Lowercase the error before matching, and add "connect failed" as the
canonical signal for "jump host dial through tunnel failed".

Continues mulga-siv-57.

* fix(ddil): default guest SSH user to ec2-user

Shell E2E SSHes the same ami-ubuntu-* images as ec2-user
(tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu
and hit 'unable to authenticate, attempted methods [none publickey]' —
authorized_keys is provisioned under the AMI default account, which is
ec2-user, not ubuntu.

Continues mulga-siv-57.

* fix(ddil): wait for witness-counter.service to seed /var/lib/counter

awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables
witness-counter.service runs in the final cloud-init stage — after sshd
binds. Baseline read landed in the window where sshd accepts but
ExecStartPre hasn't created /var/lib/counter:

  cat: /var/lib/counter: No such file or directory

Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the
full path (tunnel + SSH handshake + cat) and retries on the transport
errors plus "no such file or directory". Single 2-minute budget covers
cloud-init final stage on tofu-cluster hardware.

Continues mulga-siv-57.

* fix(systemd): drop Wants=spinifex-nats from daemon unit

Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately
killed spinifex-nats was getting pulled back up the moment the daemon
restarted — masking the DDIL Tier 1 standalone-startup path and breaking
the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact:
spinifex.target's Wants= activates every spinifex unit on boot. After=
remains for ordering and is non-blocking when NATS is absent.

scripts/test-ddil-phase1.sh hit the same trap and worked around it with a
runtime drop-in plus sentinel file; the fix here makes that workaround
unnecessary for any caller that restarts the daemon while NATS is stopped.

* feat(daemon): peer-health probe drives standalone mode on partition

Mode() is now derived from two atomic signals — natsConnected (NATS client
TCP link, flipped by the existing disconnect/reconnect callbacks) and
peersReachable (any peer daemon's /health responded in the most recent
probe cycle). Reports cluster iff both are up; standalone otherwise.

The peer-health probe is the missing signal for DDIL Scenario C: an iptables
DROP between peer IPs blocks NATS cluster-routing traffic but leaves each
node's local NATS client TCP connection healthy, so nats.Conn.IsConnected()
keeps returning true and onNATSDisconnect never fires. The probe dials each
peer's /health over the cluster network with a 2s timeout on a 2s ticker;
worst-case detection of total partition is ~6s on a 3-node cluster, well
inside the 30s Scenario C budget. Single-node clusters keep peersReachable
permanently true (nothing to probe) so the derived mode flips to cluster as
soon as the local NATS connect succeeds.

The probe runs from startLocal so the partition signal exists even before
(or in the absence of) a successful NATS connect. It does not touch NATS or
JetStream, so the §1e Tier 1 invariant asserted by
assertNoClusterServicesInitialised is unaffected.

Tests:
- TestMode_RequiresBothSignals — every nats×peers combination.
- TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test.
- TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses.
- TestMonitorPeerReachability_EndToEnd — ticker-driven false→true.

Closes mulga-siv-58.

* fix(daemon): peer probe falls back to local daemon port

`spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml;
remote-node blocks carry just `host = "<ip>"` with no daemon sub-table.
So `peer.Daemon.Host` was always empty in a deployed cluster and the
probe failed closed, leaving peersReachable permanently false.

Tofu-cluster run 25665158964 surfaced this as a Scenario A regression:
after StartNATS, natsConnected flipped true on reconnect but Mode()
stayed standalone because the derived signal required peersReachable.
`a_nats_kill_test.go:75: node3 did not return to cluster mode … (still
reporting another mode)`.

Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port
when present and otherwise reuses the local daemon's port (clusters are
symmetric — every node binds the same port the template renders). New
TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port-
available case; TestProbePeerHealth_RejectsBadAddr is updated to clear
self config so it still asserts the fail-closed path.

* fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit

DDIL Scenario B asserts that `systemctl restart spinifex-daemon`
preserves every guest VM so the new daemon can reattach via the
persisted local state file. The previous setupShutdown SIGTERM handler
called vmMgr.StopAll(), gracefully QMP-powering-down every guest before
the daemon exited — ~90s wall-clock and zero surviving QEMU on the
other side. CI run 25666369604 surfaced this as
`recovered instance i-... has no live QEMU PID`.

Two changes restore the Tier 1 contract:

- setupShutdown no longer calls StopAll. It still flips d.shuttingDown
  so crash/restart handlers bail out, persists state, unsubscribes from
  NATS, and exits. VMs are only stopped when coordinated cluster
  shutdown drains them — handleShutdownDrain still calls StopAll and
  flips the flag before the SIGTERM path runs. Operator-visible
  behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs
  running. To stop VMs, use the cluster-shutdown topics or terminate
  via the AWS API.

- The systemd unit gains `KillMode=process` so systemd only signals the
  daemon's main PID. With the default control-group mode, QEMU
  children were killed when systemd tore down the daemon's cgroup —
  defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the
  new fast-exit grace window.

Also nil-guards natsConn.Close in the shutdown path: when a daemon
restarts while NATS is unreachable (Scenario B), natsConn is nil and
the close would panic.

Mulga-siv-58 (reopened).

* fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart)

QEMU stdout/stderr were piped to daemon-side scanner goroutines.
When the daemon exits (e.g. `systemctl restart spinifex-daemon`),
the pipe read ends close; QEMU's next write triggers SIGPIPE and
the guest dies — defeating DDIL Scenario B's "daemon restart
preserves running instances" guarantee.

Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr
instead. The FD is dup'd into QEMU at fork, so closing it on the
daemon side after Start() leaves QEMU writing into the file
indefinitely. POSIX O_APPEND makes concurrent QEMU writes
interleave atomically per line, so a single shared log is safe and
avoids per-instance clutter in RuntimeDir.

Trade-off: QEMU lines no longer land in journalctl -u
spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly
(path logged at instance start under the qemu_log field).

mulga-siv-58 (DDIL Scenario B iteration 4)

* fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround

Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU
ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's
`PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()`
falling through to `os.TempDir()` (=/tmp):

- spinifex-daemon HOME=/var/lib/spinifex/spinifex, so
  `$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not
  exist and pidPath() returns /tmp.
- Every daemon start gets a fresh private /tmp namespace. QEMU stays
  in the OLD namespace after the daemon dies (mount namespaces aren't
  changed by reparenting). The new daemon's /tmp is empty.
- `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace
  the file doesn't exist, so PID=0.

Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon
unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d,
already in ReadWritePaths) and survives daemon restart. pidPath()
short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and
RuntimeDir() all resolve to the stable path. Both daemon incarnations
and QEMU children see the same files.

Revert iteration 4's pipe→file QEMU log change (commit 8a9a6f9f).
SIGPIPE was never killing QEMU, and the file-redirect dropped QEMU
lines from `journalctl -u spinifex-daemon` for no gain.

mulga-siv-58 (DDIL Scenario B iteration 5)

* feat(daemon): siv-60 heal reconcile — phase 1 (probe-heal trigger) (#211)

* feat(daemon): wire reconcileOnHeal on peer-probe heal edge (siv-60 phase 1)

Scenario C heal triggered zero resync: the local NATS client stayed
connected to its loopback server throughout the partition, so
onNATSReconnect never fired. The peer-probe false→true transition was
the only heal signal but had no resync action wired to it.

Adds reconcileOnHeal, called from both edges and coalesced via a
reconciling atomic.Bool so the two callbacks landing on the same heal
don't double-push. For phase 1 the body is the existing WriteState
(re-asserts our per-node row in the instance-state KV). Phase 2 will
extend it to pull KV → merge → push for the stopped/terminated buckets
and re-fire per-node bootstrap.

Plan: docs/development/feature/ddil-heal-reconcile.md

* feat(daemon): siv-60 phase 2 — idempotent default-VPC bootstrap re-fire on heal

Calls ensureDefaultVPCInfrastructure() from reconcileOnHeal after the
WriteState push. Idempotent: each branch (default VPC lookup, IGW
attach, default-SG create, default route insert) checks for existing
infrastructure before creating. Covers the failure mode where
startCluster ran during partition and the initial creates returned
errors against an unreachable NATS — without a heal-time retry the
cluster would be stuck without a default VPC until the next daemon
restart.

Scope note: the bead's "pull KV → merge → push" assumed the daemon
cached peer-node running state, stopped instances, terminated
instances, and VPC/IGW/SG state locally. The codebase audit (see plan
doc) confirms it doesn't — every consumer reads its KV bucket fresh
per request, and vmMgr only tracks VMs running on this node. The
bootstrap re-fire is the one heal-time action that has a real
in-memory target. Watchdog sweep is unnecessary because
vm.Manager.CrashHandler is event-driven from cmd.Wait().

* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal (siv-60 follow-up) (#212)

* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal

DDIL Scenario C regressed at the witness-SSH step after siv-60
landed in #211. Root cause: the first peer-probe tick on multi-node
startup flips peersReachable false→true (its zero value is false), so
reconcileOnHeal fires concurrently with startCluster's own
ensureDefaultVPCInfrastructure call. The Describe→Create→Attach
sequence is not cluster-wide singleton, so multiple daemons + double
calls per daemon raced and produced duplicate IGW attachments,
corrupting default-VPC routing — which broke the witness VM's tunnel
SSH path once Scenario C's partition was installed.

Phase 1 (the heal trigger + coalescing) is the meaningful fix and
stays. Phase 2's bootstrap re-fire needs leader election or a
debounce against the startup-time path before being re-introduced;
leaving an inline NOTE so the next attempt has the context.

Failed run: actions/runs/25706585637 (Scenario C, attempts 1+2).
Last green DDIL run: 25671044714 (pre-merge, SHA 40e3d847).

* fix(daemon): init peersReachable=true to suppress startup heal edge

DDIL Scenario C kept flaking at attempt-1 even after dropping the
bootstrap re-fire. Remaining trigger: peersReachable starts as the
zero value (false) on multi-node clusters, and the first probePeersOnce
tick flips it false→true. That counts as a "heal" edge under siv-60
phase 1, so reconcileOnHeal fires at boot — concurrent with
startCluster's bootstrap — perturbing timing in a way that the
harness's Scenario C exposes (witness sshd handshake EOF mid-test).

Initialise peersReachable optimistically. Mode() also requires
natsConnected (starts false), so the optimistic peer init can never
falsely report cluster mode — but it does eliminate the spurious
startup edge. Real partition→heal transitions still fire the trigger
exactly as designed: true→false on partition, false→true on heal.

---------

Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
2026-05-12 15:56:17 +10:00
Julian Sommer 23c26d8202 feat: enable no floating promises (#214) 2026-05-12 15:40:25 +10:00
Julian Sommer 069e6d411a fix: continue using ubuntu 24 image in workbooks 2026-05-12 15:09:53 +10:00
Julian Sommer dc3b48f17e feat: security groups enforcement (#201) 2026-05-12 14:16:50 +10:00
Julian Sommer bb08335831 fix: explicit workflow permission 2026-05-12 11:57:03 +10:00
Julian Sommer f9dddd3c9c fix: min package age 3 days 2026-05-12 11:23:58 +10:00
Julian Sommer 8b17ce4060 fix: allow esbuild build 2026-05-12 09:33:56 +10:00
Julian Sommer f49923ef10 chore: update pnpm 2026-05-12 09:22:04 +10:00
dependabot[bot] dbb153bd42 chore: bump the go-dependencies group across 1 directory with 2 updates (#206) 2026-05-12 08:58:12 +10:00
dependabot[bot] 13cf15b746 chore: bump the npm-dependencies group (#207) 2026-05-12 08:57:52 +10:00
Josh Siviour 48d7ce8a45 feat(viperblockd): retry LoadState on transient backend errors (siv-25 medium-term) (#210)
* feat(viperblockd): retry LoadState on transient backend errors

ebs.mount used to call vb.LoadState once and treat any failure as fatal.
When predastore was still warming up after a coordinated daemon restart,
the backend Read came back with a network/timeout error, viperblock
folded it into "block size is 0 in both local and backend state", and
the daemon's recovery path then ran the cleanup branch — destroying
DeleteOnTermination=true volumes (mulga-siv-25).

Wrap LoadState in retryLoadState: 5 attempts, 200ms * 1.5^n backoff
(total worst-case ~3.7s, well inside the 30s ebs.mount NATS deadline).
Retry only on viperblock.ErrStateBackendUnavailable; ErrStateNotFound
and any unclassified error fail fast so genuine problems are still
visible.

Retry loop is parameterised over sleep and loadFn so the retry policy
itself is unit-tested without spinning up a real VB.

* chore: bump viperblock to v1.2.2-0.20260511064619-1aa54f0dc1f6

Pulls in the ErrStateNotFound / ErrStateBackendUnavailable sentinels
introduced by mulgadc/viperblock#11 (merged on dev). Required for the
retry wrapper added in the previous commit to compile against the
public viperblock module.
2026-05-11 17:10:48 +10:00
Josh Siviour f6ddd602b9 fix(daemon): preserve volumes when recovery fails on restart (siv-25 short-term) (#209)
Stops the P0 data-loss path where `systemctl restart spinifex-daemon`
destroys user volumes flagged `DeleteOnTermination=true`.

Previously, when daemon recovery couldn't relaunch (Run error) or
reconnect (QMP attach failure) to a previously-running instance, the
code called `MarkFailed` — which transitions to `terminated`, runs the
full destructive cleanup (`terminateCleanup`), and deletes volumes per
`DeleteOnTermination`. A benign operator action (daemon restart) was
treated identically to a user-initiated `ec2.TerminateInstances`.

New `MarkRecoveryFailed` transitions to `StateError` with a
`Server.RecoveryFailed` StateReason, runs only the non-destructive
`stopCleanup` chain (graceful unmount + tap teardown + GPU release +
resource deallocation), and persists the running set. Volumes, public
IPs, ENIs, and placement-group membership are preserved so the operator
can either retry recovery or issue an explicit terminate (which will
then honour DoT as normal). The instance stays in the local map and
`OnInstanceDown` is NOT fired, keeping `ec2.cmd.<id>` live so operator
commands can still reach this node.

`classifyRestoredInstances` now skips relaunch + resource re-allocation
for instances already in `StateError`, so a recovery-failed instance
stays put across subsequent daemon restarts until the operator acts.

Medium-term (viperblock "block size 0" hard-vs-soft classification +
mount retry-with-backoff) and long-term (audit of every non-user-
initiated cleanup caller) tiers remain open on the siv-25 plan.
2026-05-11 16:47:24 +10:00
Josh Siviour 13d9fb494a refactor(daemon): extract instance mutators to service (siv-21, 1b-pre phase 2) (#208)
* refactor(daemon): extract ModifyInstanceAttribute to service (1b-pre phase 2a)

Collapse handleEC2ModifyInstanceAttribute to a thin Pattern A wrapper over
InstanceServiceImpl.ModifyInstanceAttribute, mirroring the Describe extraction
in 719ac63 (phase 1). The InstanceService interface and StoppedInstanceStore
each gain one method (ModifyInstanceAttribute, WriteStoppedInstance); the
daemon's JetStreamManager already implements WriteStoppedInstance, so no
production code outside the daemon handler changes.

Logic-level tests move from daemon/daemon_handlers_test.go to
handlers/ec2/instance/service_impl_test.go (12 cases against an in-memory
fakeStoppedStore). The daemon keeps one NATS round-trip wrapper smoke test.
The error-injection cases in daemon_handlers_instance_test.go remain in place
but now require daemonWithFakeStateStore to rewire d.instanceService with
the fake store — the previous `d.stateStore = store` swap no longer reaches
the new code path.

No behaviour change: SourceDestCheck stays a state-agnostic no-op, the
ownership/visibility check still gates after the stopped-state gate, and the
success response payload remains `{}`.

* refactor(daemon): extract TerminateStoppedInstance to service (1b-pre phase 2b)

Collapses ~140 lines of daemon termination logic into a one-liner
handleNATSRequest wrapper plus an InstanceServiceImpl method, matching
the Pattern A precedent from phase 1 (Describes) and phase 2a
(ModifyInstanceAttribute). Adds VolumeDeleter, ENIDeleter, and
PublicIPReleaser dep interfaces (wired via SetTerminationDeps) so the
service can clean up volumes, ENI, and EIP without depending on daemon
internals.

Hoists publishNATEvent off the daemon into utils.PublishNATEvent so the
service can drive vpc.delete-nat directly. Removes the
terminateStoppedInstanceRequest daemon-local payload in favour of the
exported TerminateStoppedInstanceInput on the service interface.

Tests: 13 service-level cases against an in-memory fake (missing ID,
nil store, load error, not found, not stopped, cross-tenant,
happy path, terminated-write failure aborts, retry-once delete,
user volume deleted, no-deleter graceful skip, internal volumes via
NATS, public IP release, ENI delete). The five JetStream-backed
daemon Terminate tests collapse to one wrapper smoke
(WritesToTerminatedKV); the six error-injection tests in
daemon_handlers_instance_test.go continue to drive the wrapper through
fakeStateStore for wire-format coverage.

Refs mulga-siv-21.

* refactor(daemon): extract StartStoppedInstance to service (1b-pre phase 2c)

Daemon's handleEC2StartStoppedInstance collapses to a Pattern A handleNATSRequest
wrapper over InstanceServiceImpl.StartStoppedInstance. Promotes ResourceCapacityProvider
to InstanceTypeAllocator with Allocate/Deallocate/CanAllocate/InstanceTypes, and adds
a GPUClaimer interface returning (pciAddress, xvgaEnabled, err) so the instance package
stays free of the gpu package. Daemon ResourceManager gains exported wrappers around
its internal allocator methods; a daemonGPUClaimer adapter wires gpu.Manager + the
GPU model overrides through SetGPUClaimer.

No behaviour change. New service-level tests cover the moved logic; existing
daemon integration tests continue to validate the end-to-end path through the
wrapper.

* ci(e2e-ddil): export spinifex AWS creds for witness SDK

The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.

spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.

* Revert "ci(e2e-ddil): export spinifex AWS creds for witness SDK"

This reverts commit 5b890162a6.

* refactor(daemon): extract RunInstances + ec2.cmd handlers to service (1b-pre phase 2d)

Final sub-phase of mulga-siv-21. Moves the remaining instance mutators
from spinifex/daemon/daemon_handlers_instance.go into InstanceServiceImpl:

- PrepareRunInstances: input validation, capacity allocation, per-instance
  RunInstance + ENI auto-create + public IP assignment.
- LaunchRunInstances: volume preparation, GPU claim, vmMgr.Run loop.
- RunInstances: synchronous Prepare+Launch convenience wrapper used by
  the InstanceService interface for non-daemon callers.
- RebootInstance, StartInstance, StopOrTerminateInstance: extract the
  three ec2.cmd sub-handlers; daemon.handleEC2Events shrinks to a
  dispatcher that looks up the VM and calls the matching service method.

The daemon's RunInstances handler is now a thin orchestrator that splits
the call into Prepare and Launch around the early NATS respond so the
original respond-then-launch timing is preserved (AWS clients see the
reservation before volumes prepare and VMs launch).

New service deps wired via SetRunInstancesDeps:
- AMIMetaLoader (handlers/ec2/image.ImageServiceImpl)
- KeyPairValidator (handlers/ec2/key.KeyServiceImpl)
- ENICreator + SubnetInfo (new daemonENICreator adapter over VPCServiceImpl
  to avoid an instance↔vpc cyclic import)
- PublicIPAllocator (handlers/ec2/vpc.ExternalIPAM)

No behaviour change. Tests retarget the service directly for the new
error paths; existing daemon-side RunInstances handler tests continue to
pass via the wrapper.

* test(instance): add LaunchRunInstances + setter coverage for phase 2d
2026-05-11 13:50:55 +10:00
Josh Siviour 6c54a6790c feat(daemon): DDIL Tier 1 — daemon-local-autonomy phase 1 (#204)
* refactor(daemon): extract Describe instance handlers to service (1b-pre phase 1)

Collapse the Describe family in daemon_handlers_instance.go to thin Pattern A
wrappers over a transport-neutral InstanceServiceImpl. This brings the read
path in line with the rest of the codebase and is the prerequisite for 1b's
local HTTP API (mulga-siv-14, daemon-local-autonomy plan).

Extracted: DescribeInstances, DescribeInstanceTypes, DescribeInstanceAttribute,
DescribeStoppedInstances, DescribeTerminatedInstances. Filter helpers
(instanceMatchesFilters, matchTagKey/Value, DescribeInstancesValidFilters,
IsInstanceVisible) move to the instance package alongside the service methods.

InstanceService gains the new methods and two dep interfaces
(ResourceCapacityProvider, StoppedInstanceStore) so the daemon's ResourceManager
and JetStreamManager can plug in without leaking concrete types. NATSInstanceService
returns a clear error for the fan-out describes (DescribeInstances,
DescribeInstanceTypes) since cluster-wide aggregation lives in the gateway's
scatter-gather helpers, not a single-node NATS request.

Behaviour changes: malformed JSON on DescribeInstanceAttribute now responds
with ValidationError (matching every other Pattern A handler) instead of
ServerInternal. Test updated.

Mutators (RunInstances, ModifyInstanceAttribute, StartStoppedInstance,
TerminateStoppedInstance, sub-handlers) remain Pattern B for now; follow-up
phases of mulga-siv-14 will collapse them.

* feat(daemon): local-first instance state persistence (1a)

Make /var/lib/spinifex/state/instance-state.json the source of truth for a
node's VMs; JetStream KV becomes a best-effort cluster cache. The daemon now
survives KV unavailability without losing instance state.

- New local_state.go: atomic write (tmp+fsync+rename), schema_version gate,
  missing-file = fresh install, corrupt = refuse start.
- Daemon.WriteState writes the local file under Instances.Mu, then calls
  jsManager.WriteStateBestEffort with a 1s timeout. KV failures log and
  continue; clients never see "KV down" during partition.
- Daemon.LoadState reads from the local file only — no KV fallback (a fresh
  node has empty KV anyway, so silent fallback would mask bugs).
- restoreInstances now returns error and Start() aborts on corrupt local
  state rather than silently starting empty.
- jsManager.WriteState no longer self-locks (caller holds Mu); added
  WriteStateBestEffort and shared marshalInstanceState helper.

Reconnect-driven KV resync deferred to siv-16 (1c owns the callback wiring).

Bead: mulga-siv-15
Plan: docs/development/feature/daemon-local-state-persistence.md

* fix(daemon): root local state at DataDir not BaseDir

LocalStatePath was falling back to BaseDir when DataDir was empty, putting
the file at /var/lib/spinifex/spinifex/state/instance-state.json (under the
daemon's own subdir) instead of the documented
/var/lib/spinifex/state/instance-state.json (the platform-shared root).

Drop the BaseDir fallback. Empty DataDir resolves to DefaultDataDir
(/var/lib/spinifex). Tests now set DataDir alongside BaseDir on their config
fixtures.

* feat(daemon): graceful NATS disconnect/reconnect handling (1c)

Daemon now flips a `mode` field (cluster|standalone) on NATS
disconnect/reconnect via optional callback hooks added to
utils.ConnectNATS. Reconnect bumps natsRetryCount and triggers 1a's
full local-state push to KV. NATSRequest/NATSScatterGather fail fast
with ErrClusterUnavailable when the connection is down. Gateway
short-circuits NATS-bound requests with a 503 ServiceUnavailable
carrying "cluster unavailable: NATS disconnected — check daemon
/local/status" so AWS clients see the failure mode immediately
instead of hanging on per-call timeouts.

Plan: docs/development/feature/daemon-autonomy-1c-disconnect.md

* fix(gateway): SigV4 middleware short-circuits on NATS disconnect

The 1c gateway.Request() IsConnected check (gateway.go:234) was
unreachable for any authenticated path because SigV4 auth middleware
runs first and calls IAMService.LookupAccessKey over JetStream KV. With
NATS down the lookup hangs ~5s on context.DeadlineExceeded and the
gateway returns 500 InternalError. AWS SDK retries it twice for ~18s of
user-visible latency instead of the <100ms 503 promised by the 1c plan.

Mirror the IsConnected check at the top of SigV4AuthMiddleware (after
credential parse so malformed auth headers still get the right SigV4
error code, before the IAM lookup). The catch-all in gateway.Request()
stays as defence-in-depth for any future unauthed paths.

Coverage debt picked up alongside: 1c utils.NATSRequest /
NATSScatterGather fail-fast paths, daemon Mode/onNATSDisconnect/
onNATSReconnect/localStatePath/Write/LoadState branches, and 1b-pre
InstanceServiceImpl Describe* methods (partial siv-22).

Bead: mulga-siv-23

* feat(daemon): read-only /local/* HTTP API (1b)

Wires GET /local/instances, GET /local/instances/{id}, GET /local/status
onto the existing ClusterManager TLS listener so observers can read this
node's instance state and connectivity mode while NATS is unreachable.

Also fixes a pre-existing race in WriteState: json.Marshal walked shared
*vm.VM pointers from SnapshotMap while concurrent TransitionState calls
mutated VM fields under the manager lock. Marshal now runs inside
vmMgr.View; WriteLocalStateBytes / WriteStateBytesBestEffort accept the
pre-marshalled JSON and commit lock-free. TestTransitionState_Concurrent
Transitions is race-clean across 30 iterations.

Adds Daemon.stateRevision (atomic.Uint64) bumped on every successful
local-state write, surfaced as /local/status.revision so observers can
detect changes without diffing payloads.

* feat(daemon): two-phase startup — survive NATS down (1d)

Split daemon Start() into startLocal() (HTTPS, mgmt bridge, OVS plumber,
OOM, local state load — no NATS) and startCluster() (background, infinite
retry exp backoff cap 60s for NATS connect, then JetStream + services +
restoreInstances + subscribeAll + heartbeat). Daemon never process-exits
on NATS failure; instead it stays up in standalone mode and reconnects in
the background.

Drop ExecStartPre=wait-for-nats.sh from spinifex-daemon.service — local
bootstrap no longer requires NATS to be reachable.

ConnectNATSWithRetry now supports infinite retry (WithMaxWait(0)),
context cancellation (WithContext), per-attempt error callback for the
nats_retry_count operator signal on /local/status, and a configurable
backoff cap (WithMaxRetryDelay).

bead: mulga-siv-17 (closes)

* feat(daemon): Tier 1 invariant guard for cluster-scoped service init (1e-audit)

Make the DDIL §1e-audit invariant explicit: startLocal must not initialise any
NATS-dependent or JetStream-KV-backed handle. Adds assertNoClusterServicesInitialised
as a runtime guard at the end of startLocal that fails fast if a future edit
hoists a cluster-scoped service into Phase 1, plus comment blocks naming the
invariant on both startLocal and startCluster.

Code audit confirms the existing 1d split already keeps every cluster-scoped
bucket out of Phase 1; this change pins the property so it cannot regress
silently. Tier table for the 25 daemon-process KV buckets is finalised in
docs/development/feature/daemon-local-autonomy.md §1e-audit.

bead: mulga-siv-19 (closes)

* feat(daemon): NATS observability hardening (1d-obs)

Closes the residual visibility gap left by the 1d non-blocking startup:

- Extend /local/status with kv_sync_failures, last_kv_sync_at,
  last_kv_sync_error so an operator polling the local API sees prolonged
  divergence between the local file and JetStream KV.
- Add a KVSyncObserver interface on JetStreamManager so the daemon can
  count best-effort write outcomes without coupling the manager to
  daemon state.
- Escalate utils.ConnectNATSWithRetry per-attempt log to slog.Error
  (rate-limited to once per minute, with disconnected_for context) once
  attempt count crosses the natsRetryEscalateAttempt threshold (~30
  attempts ≈ ~30 min disconnected at the 60s backoff cap), so a stuck
  NATS config surfaces in error logs instead of routine warn noise.

Prometheus metrics export deferred until a system-wide telemetry plan
(likely OTel) lands; the structured /local/status fields plus the
escalated retry log cover the Tier 1 visibility gap.

* feat(daemon): opt-in SPINIFEX_REQUIRE_NATS strict-startup mode (1d-strict)

When SPINIFEX_REQUIRE_NATS=1, startCluster's first NATS connect uses a
bounded ~30s retry and the daemon aborts on timeout. Default behaviour
(env unset) keeps the Tier 1 infinite-retry path so prod / edge nodes
still survive a missing NATS at boot. Restores pre-DDIL fail-fast UX
for dev / test / single-node deploys without flipping the prod default.

mulga-siv-44.

* test(scripts): systemd-level smoke for DDIL phase 1

Bash harness exercising daemon-local-autonomy on a real `make deploy` node:
baseline cluster mode → kill spinifex-nats → assert standalone + retry
counter advance + /local/instances served → restart NATS → assert mode flip
+ KV resync stamp → SPINIFEX_REQUIRE_NATS=1 bounded-exit. Cleanup trap
restores both services. Complements existing unit coverage by validating
real systemd wiring on port 8443.

* fix(test-ddil): daemon ClusterManager binds 4432 not 8443

8443 is predastore. Daemon's HTTPS ClusterManager (where /local/* lives)
binds [nodes.node1.daemon].host = 0.0.0.0:4432 in the deployed config.
Plan doc daemon-local-autonomy.md:112 incorrectly said 8443; actual code
uses d.config.Daemon.Host.

* fix(test-ddil): correct retry-counter site + bypass NATS Wants= for strict test

Two bugs in the v1 script:

1) nats_retry_count assertion ran during disconnect — but the counter
   only bumps on disconnect→reconnect *cycles* (daemon.go:321,
   onNATSReconnect). Moved assertion to step 3 (restart NATS) where
   the reconnect callback fires.

2) Step 4 stopped spinifex-nats then ran 'systemctl restart
   spinifex-daemon' — but the daemon unit's Wants=spinifex-nats.service
   re-pulled NATS up via the Wants= chain, so the strict-mode 30s
   timeout never fired. systemctl mask refuses on /etc-installed
   units, and --runtime mask is shadowed by /etc. Block via a runtime
   drop-in with ConditionPathExists=! pointing at a sentinel file.

Verified: all 5 checks pass on a single-node make deploy box; daemon
exits 32s after restart with SPINIFEX_REQUIRE_NATS=1 + NATS blocked.

* fix(admin): render data_dir into spinifex.toml so non-root nodes write local state

Pseudo-multinode E2E (rootless tf-user, ~/nodeN data roots) hit
"mkdir /var/lib/spinifex/state: permission denied" on every RunInstances:
Config.DataDir was empty because the template never emitted a data_dir
key, and LocalStatePath fell back to DefaultDataDir (/var/lib/spinifex).

Surfaced after 8eb9c5fa dropped the BaseDir fallback in LocalStatePath —
previously SPINIFEX_BASE_DIR=$DATA_DIR/spinifex/ (set by start-dev.sh)
masked the missing data_dir.

ConfigSettings.DataDir is already populated to spxRoot at admin
init/join; render it under [nodes.X] so it actually reaches the daemon.
2026-05-10 10:08:12 +10:00
joshsiv-mulga 7493f087d1 Update Ansible smoketest playbook to use debian 13 AMI 2026-05-08 15:56:16 +10:00
Julian Sommer 51f7afab4c fix(ui): github workflow 2026-05-08 12:42:02 +10:00
Julian Sommer 0aa53f78ea chore(ui): enable no-console 2026-05-08 12:39:55 +10:00
Julian Sommer fc0cc85731 chore(ui): enable vitest lint rules 2026-05-08 12:36:33 +10:00
Julian Sommer 6fa82c470d fix(vpcd): self-heal partial IGW topology on reconcile
reconcileIGW was non-idempotent and gated on "external switch missing",
so once a prior pass left the ext switch + LRP in place without the
localnet LSP, every subsequent vpcd start silently skipped the function
and the partial topology persisted forever (br-ext never got its patch
port; external traffic failed).

Convert each step to Get-then-Create idempotent and drop the cascade
rollbacks — partial state is now safe to leave for the next pass to
heal. Add FindStaticRoute so the default route is deduped instead of
duplicated on every retry. Drop the ext-switch gate in Reconcile and
ReconcileFromKV; rename IGWsCreated to IGWsReconciled to reflect the
new "called" semantics.
2026-05-08 12:09:05 +10:00
Julian Sommer 1c3583651d fix(ui): enable more rules 2026-05-08 11:55:18 +10:00
Julian Sommer f8ce8de566 fix(ui): require await 2026-05-08 11:39:33 +10:00
Julian Sommer b26bb8d35d fix(vpcd): bounded-retry reconcile-leader instead of fail-open (#202) 2026-05-08 10:47:22 +10:00
Josh Siviour 62ab74088e refactor(daemon): unify VPC + mgmt tap setup behind SetupTap(TapSpec) (#195) 2026-05-08 10:40:53 +10:00
Julian Sommer 6f3e901b9f chore: use go 1.26.3 2026-05-08 09:21:12 +10:00
joshsiv-mulga f44db4d84e fix(vpcd): reconcile EIP dnat_and_snat rules on startup
NAT rules are purely event-driven — only AssociateAddress creates them via
vpc.add-nat. No startup reconcile path means any OVN NB DB state drift
silently breaks EIP inbound traffic after a node reboot.

Add Step 5 to ReconcileFromKV that reads associated EIPs from the
spinifex-vpc-elastic-ips KV bucket and ensures each has a dnat_and_snat rule
in OVN NB DB. Correct rules are skipped (idempotent). Stale rules (wrong
LogicalIP) are replaced. Missing rules are created.

Also adds FindNATByExternalIP to OVNClient interface + LiveOVNClient/Mock
implementations, and three new tests covering: missing rule creation,
idempotent skip on correct rule, and stale rule replacement.
2026-05-07 23:41:41 +10:00
joshsiv-mulga 5ea7df6e7c fix(e2e): revert to internet-facing ALB; keep LaunchTime reboot check
The internal-ALB approach was wrong on two counts:
- ALB_ENI_MAC is the OVN VPC ENI MAC (lives on br-int), never appears
  in br-mgmt neighbour table, so the mgmt IP lookup always fails.
- Switching to internal masked the real bug: OVN EIP DNAT not restoring
  properly after a host reboot, which is exactly what this test should catch.

Restore internet-facing ALB and EIP-based traffic bursts. The post-reboot
traffic failure (Phase 8.6) is the correct signal for the outstanding OVN
EIP DNAT persistence bug.

Keep the Phase 8.3 LaunchTime fix: two-hop SSH to VPC private IPs doesn't
work in single-node macvlan (host has no route to VPC subnets). LaunchTime
from describe-instances is fast and topology-independent.
2026-05-07 21:52:37 +10:00
joshsiv-mulga 94405effec fix(e2e): cell-18 internal ALB + LaunchTime reboot check
Two changes to fix cell-18 (single/static/debian-13/reboot-single):

1. Switch ALB to --scheme internal.
   Internet-facing ALBs work pre-reboot on single-node but post-reboot the
   OVN EIP DNAT (nat-addresses=router GARP) is not re-established in time,
   causing 100% curl timeouts. Internal ALB routes traffic via br-mgmt using
   the mgmt IP discovered from the ALB ENI MAC in the br-mgmt neighbour table,
   bypassing OVN external NAT entirely.

2. Replace Phase 8.3 guest SSH uptime check with LaunchTime assertion.
   Two-hop SSH to VPC private IPs fails in single-node macvlan: the host has
   no route to VPC subnets. LaunchTime from describe-instances is reset by the
   daemon on QEMU relaunch, so LaunchTime >= REBOOT_START reliably confirms
   recovery without network reachability to the guest.
2026-05-07 21:35:23 +10:00
joshsiv-mulga 9ca8b84bea ci(baremetal): skip ePDU power-cycle until credentials are wired in 2026-05-07 16:39:21 +10:00
joshsiv-mulga e9aec9e784 ci(baremetal): restore bm-ci SSH key from secret at run start 2026-05-07 16:07:30 +10:00
joshsiv-mulga 8c6d0747e0 fix(ci): pass SPINIFEX_SRC/VIPERBLOCK_SRC to iso-builder, build viperblock .so 2026-05-07 15:50:30 +10:00
joshsiv-mulga 374f2ed7a3 feat(ci): add e2e-baremetal workflow — workflow_dispatch only
Separate from normal e2e. Checks out spinifex, checks out mulga scripts
(sparse: iso-builder + baremetal), builds spx + test tar via
bm-build-artifacts.sh, builds PXE artifacts (squashfs/initrd/vmlinuz)
via iso-builder with persistent cache on banksia, then runs
run-baremetal-e2e.sh against the single bare-metal node.
2026-05-07 15:38:05 +10:00
Julian Sommer 46d7a875ab fix: single node e2e direct s3 access 2026-05-07 14:52:58 +10:00
Julian Sommer 82d1c7aeee fix: add account id to predastore test 2026-05-07 14:42:41 +10:00
Julian Sommer 9640cca480 chore: bump predastore version 2026-05-07 14:29:57 +10:00
Julian Sommer 29a192766c configs 2026-05-07 13:51:03 +10:00
joshsiv-mulga 5f8ee0b5a2 feat(installer): persist SPINIFEX_INSTALL_CALLBACK through firstboot reboot
In the PXE path SPINIFEX_INSTALL_CALLBACK arrives on the iPXE kernel
cmdline but was never threaded into the firstboot script, so firstboot
never curled the controller's /done endpoint. The flag stayed on disk,
the node re-entered the install loop on the next PXE boot.

Fix: read SPINIFEX_INSTALL_CALLBACK from the live-environment env in
toFirstbootConfig and embed a best-effort curl into the generated
firstboot script, fired once after the success marker is written.

Interactive and USB-stick installs are unaffected (env var absent
-> callbackBlock is empty, no curl injected).
2026-05-07 12:28:09 +10:00
Julian Sommer a753ff99dd fix(UI): make mulga logo change with theme 2026-05-07 09:59:21 +10:00
Julian Sommer 7c860c99ca feat(UI): run instances block device mapping field (#200) 2026-05-07 09:54:46 +10:00
Julian Sommer 5e70ebd63f feat(UI): terminate instance confirmation (#199) 2026-05-07 09:21:33 +10:00
Julian Sommer a8591380cb test(daemon): add stopped/terminated handler tests against fake StateStore (#198) 2026-05-07 09:17:41 +10:00
Julian Sommer fefe506042 test(vm): improve vm tests (#197) 2026-05-06 16:42:14 +10:00
joshsiv-mulga 7ee21e0f25 fix(viperblockd): treat NBDKit code=0 premature exit as mount failure
NBDKit exiting within 1s was only treated as an error if exit code != 0.
A clean exit (code=0) fell through to Mounted=true, leaving QEMU with a
dead NBD socket and VMs that never boot post-reboot.

Treat any premature exit as failure regardless of exit code.
2026-05-06 15:55:41 +10:00
Julian Sommer 3eff6ccff2 test(vm): remove tautologies and asserts-nothing tests
manager_deps_test.go: drop TestDeps_VolumeMounterSatisfiesInterface,
TestDeps_StateStoreSatisfiesInterface, TestDeps_StateStoreErrorPropagates,
TestSetDeps_OverridesDeps, TestManagerHooks_InitiallyNil — they assert a
fake recorded its own input, that a fake returns an error literally
constructed in the test, or Go zero-value/direct-assignment semantics.
Compile-time interface checks already prove satisfaction.

state_test.go: replace TestEC2StateCodes_AllStatesHaveMapping and
TestEC2StateCodes_CorrectValues — both compared the production map to
its own literal. Now assert against an independent AWS contract using
ec2.InstanceStateName* SDK enum constants paired with the
AWS-documented numeric codes; a typo in production no longer
replicates into the test.

watchdog_test.go: replace TestStartPendingWatchdog_CtxCancelStopsGoroutine's
time.Sleep(10ms) "assertion" with goleak.VerifyNone. The previous form
passed even if the goroutine never returned on cancel — the harness
reaped it on test exit.
2026-05-06 15:19:51 +10:00
Julian Sommer 5b9a35914e feat(tests): missing vm tests (#196) 2026-05-06 15:13:32 +10:00
joshsiv-mulga 14b8fc234a fix: counts: unbound variable in bash 5.2 + set -u + empty declare -A 2026-05-06 15:01:40 +10:00
Julian Sommer 9c8f49ed71 fix: mark restored instance as failed on failed launch 2026-05-06 13:55:32 +10:00
Julian Sommer 8f7733eb64 fix: slog.Warn on instance kv migration failure 2026-05-06 13:51:34 +10:00
Julian Sommer f0eea9867b fix: cleanup instanceUp subscription if getConsoleOutput subscription fails 2026-05-06 13:45:35 +10:00
Julian Sommer c416a645f2 refactor(vm,daemon): post-extraction cleanup groups 1+2 (#194) 2026-05-06 13:30:06 +10:00
Julian Sommer 3aca1c9763 feat: require host and x-amz-date in SigV4 SignedHeaders (#193) 2026-05-06 11:31:33 +10:00
tomnewton-mulga 28ca689130 feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992)

Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.

* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)

Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.

* feat: add G5 instance type generator with GPU model support (mulga-994)

- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter

* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)

- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound

* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)

- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present

* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)

Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.

Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.

* Fixed GPU passthrough and added test script

* feat: always-probe GPU at startup, surface capability in NodeStatusResponse

- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
  regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
  groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
  and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
  so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
  passthrough not active, 'used/total' = passthrough active.

* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)

- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
  (preserves references held by instanceService) and updates NATS
  subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
  in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
  handles false→true (re-probe + activate) and true→false (refuse if
  instances running, otherwise deactivate) transitions

* feat: spx admin gpu status/enable/disable commands (mulga-1000)

- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
  (replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
  activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
  polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
  writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable

* fixed admin

* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names

Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.

* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon

GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.

* fixing gpu bining permissions

* working state of gpu passthrough on phyiscal box

* installer grub image fixes

* made gpu-passthrough more user facing

* fixed banner

* fixed banner

* banner fix

* gpu passthrough docs

* preflight tweak

* vpcd fixes to set br-wan to promisc for public IP DHCP processes

* fixed vpcd br-wan promisc and veth pair

* ssh keygen fix

* fixes after running /review-branch

* feat: re-wire GPU claim and device injection after vm.Manager refactor

GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:

- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
  RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
  derived from instance.GPUXVGAEnabled when GPUPCIAddress is set

* fix: Create spinifex user ~/.ssh and keys in firstboot

---------

Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
2026-05-06 09:52:23 +10:00
Julian Sommer 2c1c46b1f6 test(vm): migrate lifecycle coverage from daemon to vm package (#192) 2026-05-06 09:34:55 +10:00
Julian Sommer addf2bc86e refactor(vm): typed Status accessor, sentinel-aware Stop/Terminate (#191) 2026-05-05 16:37:21 +10:00
Julian Sommer 34c461be9c refactor(vm): move Restore/crash recovery/watchdog onto Manager (#190) 2026-05-05 16:13:20 +10:00
Julian Sommer c2be7f047c refactor(vm): move AttachVolume/DetachVolume/Reboot onto Manager (#189) 2026-05-05 14:27:59 +10:00
Julian Sommer cfe56c49d1 refactor(vm): move Stop/Terminate/MarkFailed onto Manager (#187) 2026-05-05 13:25:32 +10:00
Julian Sommer e3d2504b42 refactor(daemon): Move instance commands to vm manager (#186) 2026-05-05 11:38:53 +10:00
Julian Sommer d28ac40ce7 refactor(vm): introduce collaborator interfaces and daemon adapters (#185) 2026-05-05 10:48:38 +10:00
Julian Sommer 56cc83f723 chore: merge main into dev 2026-05-05 09:55:10 +10:00
Julian Sommer a216d6f596 chore(ci): point nightly e2e at dev branch
Scheduled runs were using github.ref_name, which equals the default branch
(main) — so the nightly was testing main. Hardcode GIT_BRANCH=dev so the
build-install-artifacts.sh + tofu provisioning target dev's HEAD instead.

GitHub always loads scheduled-workflow files from the default branch, so
this change must stay in sync on main for the schedule to use it.
2026-05-05 09:54:53 +10:00
Julian Sommer ce0bbde8dc fix(e2e): poll guest SSH after reboot in reboot-e2e cell
The post-reboot guest uptime check was a single-shot ssh with a 10s connect
timeout. After the host comes back, guest QEMU + cloud-init + sshd inside
each guest still need a few seconds to settle, so the check raced them.

Poll up to GUEST_SSH_SECS (default 180s) per guest, mirroring the host
TCP/22 poll. Run 25329248329 cell-18 reproduced this — ALB targets went
healthy ~2s after the second SSH attempt failed.
2026-05-05 09:54:47 +10:00
dependabot[bot] 78c3cce183 chore: bump the go-dependencies group with 3 updates (#183) 2026-05-05 09:38:50 +10:00
dependabot[bot] 968cc687a0 chore: bump the npm-dependencies group (#184) 2026-05-05 09:36:26 +10:00
Julian Sommer f3b00eb85a feat: ubuntu 26.04 image 2026-05-05 09:32:29 +10:00
Julian Sommer fa071795d0 fix: use debian 13 2026-05-05 09:10:59 +10:00
Julian Sommer 2489621831 chore: merge main into dev 2026-05-05 09:04:08 +10:00
joshsiv-mulga 9ed59aac79 fix(e2e): prevent silent set -e abort on var=$(func) patterns
When a function returning non-zero is captured via command substitution
under `set -euo pipefail`, the assignment exits the script before the
following `if [ -z "$var" ]` diagnostic can run. CI failures showed up
as a sudden jump to the EXIT trap with no error message.

- run-multinode-e2e.sh: wrap get_remote_ssh_port in `if !`, bump retry
  10 -> 30, add dump_guest_ssh_diagnostics on miss
- run-pseudo-multinode-e2e.sh, run-e2e.sh: wrap get_ssh_port in `if !`
- run-lb-e2e.sh: add `|| VAR=""` fallback on pick_peer
2026-05-04 13:44:49 +10:00
joshsiv-mulga 89aec5c709 docs: warn br-wan prerequisite for curl install
Add WAN bridge prerequisite callout to single-node and multi-node
install guides. Bootstrap installer assumes br-wan exists; running
on a host whose default route is on a bare NIC leaves install in a
non-working state. Auto-provisioning will land in next release.
2026-05-01 15:59:14 +10:00
Julian Sommer cbe335e903 chore: bump single node e2e timeout to 35 mins 2026-05-01 15:51:42 +10:00
Julian Sommer e9a335f433 chore: merge main into dev (ISO installer) 2026-05-01 15:22:41 +10:00
Ben Duncan 89318d8f6e ISO method for installer 2026-05-01 15:18:00 +10:00
Julian Sommer e1b3623599 refactor: introduce vm.Manager (phase 1 of lifecycle extraction) (#176) 2026-05-01 15:15:54 +10:00
Julian Sommer 57a11c0668 fix(vpc): validate SG rule CidrIp/SourceSG to block OVN ACL injection (#182) 2026-05-01 15:15:42 +10:00
Julian Sommer 050cc03ae2 fix: daemon_test.go allocation 2026-05-01 15:14:03 +10:00
Julian Sommer bab16c75cc fix(elbv2): validate HAProxy inputs (#181) 2026-05-01 12:17:42 +10:00
tomnewton-mulga 85184d45f9 Merge branch 'dev-test' into dev 2026-05-01 12:15:05 +10:00
tomnewton-mulga 9d665f9dde bringing local networking changes onto dev branch 2026-05-01 12:13:34 +10:00
joshsiv-mulga 97f57d3826 fix(vpcd,viperblockd): retry NATS connect for ISO firstboot resilience
Both services used one-shot utils.ConnectNATS in launchService. On ISO
firstboot, when nats.service is starting in parallel, the dial can race
ahead of NATS being ready and exit non-zero. systemd's
Restart=on-failure with default StartLimitBurst=5 in 10s burns through
the budget in ~25s and the unit enters failed permanently — DHCP
manager never starts, vpc.dhcp.acquire requests hit no-responders and
the install hangs nondeterministically.

Match daemon/awsgw, which already use ConnectNATSWithRetry (5min
exponential backoff). The newDHCPManagerWithRetry guard (mulga-siv-40)
only helps after the dial succeeds.
2026-05-01 12:10:59 +10:00
Bryn Mailer 3c2f57101f refactor: lock free write path (#179)
* fix: remove predastore filesystem backend references

* fix: remove unused import

* chore: bump predastore version

* fix: create test buckets explicitly
2026-05-01 11:46:33 +10:00
Julian Sommer 4542b22e87 chore: decrease daemon resource pre allocation 2026-05-01 11:42:32 +10:00
Julian Sommer ca965e48d6 fix(daemon): deny cross tenant attach on empty tenant id volumes 2026-05-01 11:15:58 +10:00
tomnewton-mulga 09638a874d chore: merge fix/iso-public-ip-boot-race into dev-test
Conflict resolution:
- external_ipam_dhcp.go: take dev deletion (consolidated into dhcp package)
- external_ipam.go: take dev's dhcp.RequestAcquire call
- external_ipam_test.go: drop duplicate ObtainDHCPLease/ReleaseDHCPLease tests
- daemon.go: drop dead gwMAC block (GatewayMAC field not in dev's ExternalPoolConfig)
2026-05-01 10:59:57 +10:00
Josh Siviour b7dea45ee0 feat: Ansible dev lifecycle (parallel to bash scripts) (#178)
Playbooks for preflight / teardown / install / reset covering the same
ground as dev-install.sh, dev-setup.sh, reset-dev-env.sh. Existing bash
scripts untouched — decide which to keep after parity testing.

- roles/teardown + teardown catalog README track every known state
  source; add a task + row when a leak is reported.
- roles/init gates smoketest on awsgw readiness (describe-regions poll).
- roles/smoketest is dup-tolerant, retry-wrapped, auto-terminates the
  launched instance.
- dev-install.yml skips build/install/init when a healthy environment
  is already present (4-signal health check), always runs smoketest.
- dev-reset.yml captures region/az/external-mode scalars from the
  existing spinifex.toml and replays them into init.

Plan: docs/development/improvements/ansible-dev-lifecycle.md (mulga).
Bead: mulga-siv-9.
2026-05-01 10:04:37 +10:00
Josh Siviour 69404b38ce fix(vpcd): public-IP boot race + DHCP DORA hardening (siv-33/34/36/37/38/39) (#177)
* fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33)

GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.

Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.

Plan: docs/development/bugs/iso-public-ip-boot-race.md

* fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34)

ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.

Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.

The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.

Plan: docs/development/bugs/iso-public-ip-boot-race.md

* fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33)

The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.

br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.

Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.

The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.

Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.

* fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33)

Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").

Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
  their replies are L2 broadcast which the bridge always passes up.

Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.

* revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New

ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to
catch unicast OFFER replies that bypass the bridge netdev. On dev hosts
where upstream honors the BOOTP BROADCAST flag, the wrapper regressed
DORA — public-IP allocation failed where main works. Reverting until
the promisc path is validated against both broadcast- and unicast-OFFER
servers.

* fix(setup-ovn): mark br-int/br-ext Unmanaged for systemd-networkd (mulga-siv-37)

Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs
admin-down: networkd has no matching .network so it tags them as unmanaged
but never brings them up. OVS dataplane works but ovn-controller flow
programming and any tooling that probes link state misbehaves.

Drop /etc/systemd/network/05-spinifex-ovs-internal.network with
Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br`
already runs the netdev up at create time.

* fix(vpcd): allocate gateway LRP IP from pool in centralized NAT (mulga-siv-36)

In veth/macvlan modes the gateway LRP carries all egress, so it must hold
a WAN-subnet IP — otherwise the upstream router silently drops ARP for
the WAN nexthop (RFC 826) and the default route never resolves. The
hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed
centralized NAT (introduced under mulga-siv-26).

Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW
attach the gateway LRP gets a fresh IP from that range, persisted as
external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and
retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the
gw range; ValidatePoolConfig rejects overlap with the EIP range.

Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and
extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs
to allocated pool IPs at startup.

* fix(vpcd): drop link-local fallback, auto-derive gw_lrp_range (mulga-siv-36)

In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet
IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream
router silently drops egress. Hardware deployment confirmed the previous
fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not
explicitly configured.

Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16
.. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on
overlap with the per-VM EIP range; clamp inside the subnet; guard the
upstream gateway IP. IGW attach now errors out instead of falling back
to link-local when allocation fails — surfaces misconfiguration loudly.

Adds focused unit tests for gwLrpRange, ValidatePoolConfig and
nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and
PoolWithGatewayIP regression tests.

* feat(vpcd): DHCP-acquire gateway LRP IPs (mulga-siv-38)

Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire
when the pool is source="dhcp", instead of carving from a static gw_lrp_range.
The upstream DHCP server becomes the single arbiter of LAN address space; a
cluster grows by N nodes without operator-side range carving in spinifex.toml.

- Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as
  RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM
  share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path.

- TopologyHandler dispatches by pool source: DHCP pools call DHCPManager
  via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools
  keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and
  no-pool deployments still use link-local.

- IGW detach issues vpc.dhcp.release so the upstream server reclaims the
  IP. Best-effort — server-side expiry is the safety net.

- Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP
  pool, distinct IPs, idempotent attach, release on detach),
  IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP
  errors), TestPrefixFromMask. Existing static / auto-derive tests
  preserved.

* fix(vpcd): subscribe DHCPManager before reconcile to fix boot race (mulga-siv-38)

DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks,
so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and
burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a
dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed
by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN
until the next reconcile pass.

Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription
is live before any topology code can fan out an acquire.

* fix(vpcd): also subscribe DHCPManager before bootstrap reconcile pass

Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but
missed Pass 1 (the bootstrap-config Reconcile that runs even earlier).
On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier
pass still hit "no responders" and burned the 60-retry cold-boot budget
before the Manager came up. Move both reconcile passes after Subscribe.

* fix(vpcd): RFC 2131 §4.1 DORA exponential backoff (mulga-siv-39)

DHCPManager.Acquire previously did one DISCOVER and gave up after 15s
because the outer ctx.WithTimeout matched the per-attempt timeout —
nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer
routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec
mandates retransmission with exponential backoff, and a single lost
DISCOVER was killing legitimate boots.

Move retransmission into DHCPManager.acquireWithBackoff with a
4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client
becomes single-attempt; manager owns retry policy. Default
acquireTimeout lifts from 15s to 90s (sum of schedule + slack).

* fix(vpcd): retry NewDHCPManager on JetStream-not-ready (mulga-siv-40)

NewDHCPManager creates the spinifex-dhcp-leases KV bucket via
utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the
mulga-siv-38 launchService reorder, this runs before any other JetStream
user, so on multi-node cold-boot vpcd hits "JetStream system temporarily
unavailable" and exits before cluster quorum forms. Pseudo-multinode CI
calls start_node_services sequentially, so node2's vpcd failure aborts
the script before node3 ever launches and quorum then never forms.

Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap,
ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as
patiently as the daemon for the cluster to come up.

* fix(daemon): idempotent DetachVolume QMP path (mulga-siv-41)

DetachVolume's QMP sequence was not retry-safe. First call: device_del
succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD
client teardown / in-flight I/O still hold the block node. Handler bails
ServerInternal "leaving volume state intact" but the guest device is
actually gone. AWS-CLI retries: device_del returns DeviceNotFound,
handler bails again. After 2 retries aws cli exits 254 — Single-Node
Phase 8 Step 3 reproducibly fails this way.

Phase 1: treat DeviceNotFound on device_del as already-detached, log and
continue. Detach is now idempotent across retries.

Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in
use" errors so the I/O drain can complete instead of failing the whole
detach. Other QMP errors fall through immediately as before.
2026-05-01 09:48:52 +10:00
tomnewton-mulga 5c14f56428 Merge branch 'fix/iso-public-ip-boot-race' of github.com:mulgadc/spinifex into fix/iso-public-ip-boot-race 2026-05-01 09:11:59 +10:00
joshsiv-mulga 11b3f37d06 fix(daemon): idempotent DetachVolume QMP path (mulga-siv-41)
DetachVolume's QMP sequence was not retry-safe. First call: device_del
succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD
client teardown / in-flight I/O still hold the block node. Handler bails
ServerInternal "leaving volume state intact" but the guest device is
actually gone. AWS-CLI retries: device_del returns DeviceNotFound,
handler bails again. After 2 retries aws cli exits 254 — Single-Node
Phase 8 Step 3 reproducibly fails this way.

Phase 1: treat DeviceNotFound on device_del as already-detached, log and
continue. Detach is now idempotent across retries.

Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in
use" errors so the I/O drain can complete instead of failing the whole
detach. Other QMP errors fall through immediately as before.
2026-05-01 08:52:49 +10:00
joshsiv-mulga c234cee60d fix(vpcd): retry NewDHCPManager on JetStream-not-ready (mulga-siv-40)
NewDHCPManager creates the spinifex-dhcp-leases KV bucket via
utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the
mulga-siv-38 launchService reorder, this runs before any other JetStream
user, so on multi-node cold-boot vpcd hits "JetStream system temporarily
unavailable" and exits before cluster quorum forms. Pseudo-multinode CI
calls start_node_services sequentially, so node2's vpcd failure aborts
the script before node3 ever launches and quorum then never forms.

Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap,
ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as
patiently as the daemon for the cluster to come up.
2026-05-01 08:52:41 +10:00
joshsiv-mulga efe054e4e4 fix(vpcd): RFC 2131 §4.1 DORA exponential backoff (mulga-siv-39)
DHCPManager.Acquire previously did one DISCOVER and gave up after 15s
because the outer ctx.WithTimeout matched the per-attempt timeout —
nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer
routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec
mandates retransmission with exponential backoff, and a single lost
DISCOVER was killing legitimate boots.

Move retransmission into DHCPManager.acquireWithBackoff with a
4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client
becomes single-attempt; manager owns retry policy. Default
acquireTimeout lifts from 15s to 90s (sum of schedule + slack).
2026-04-30 23:01:05 +10:00
joshsiv-mulga 714898022e fix(vpcd): also subscribe DHCPManager before bootstrap reconcile pass
Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but
missed Pass 1 (the bootstrap-config Reconcile that runs even earlier).
On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier
pass still hit "no responders" and burned the 60-retry cold-boot budget
before the Manager came up. Move both reconcile passes after Subscribe.
2026-04-30 21:58:15 +10:00
joshsiv-mulga a6ef8e575b fix(vpcd): subscribe DHCPManager before reconcile to fix boot race (mulga-siv-38)
DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks,
so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and
burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a
dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed
by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN
until the next reconcile pass.

Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription
is live before any topology code can fan out an acquire.
2026-04-30 21:49:25 +10:00
joshsiv-mulga f5c9ad59cf feat(vpcd): DHCP-acquire gateway LRP IPs (mulga-siv-38)
Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire
when the pool is source="dhcp", instead of carving from a static gw_lrp_range.
The upstream DHCP server becomes the single arbiter of LAN address space; a
cluster grows by N nodes without operator-side range carving in spinifex.toml.

- Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as
  RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM
  share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path.

- TopologyHandler dispatches by pool source: DHCP pools call DHCPManager
  via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools
  keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and
  no-pool deployments still use link-local.

- IGW detach issues vpc.dhcp.release so the upstream server reclaims the
  IP. Best-effort — server-side expiry is the safety net.

- Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP
  pool, distinct IPs, idempotent attach, release on detach),
  IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP
  errors), TestPrefixFromMask. Existing static / auto-derive tests
  preserved.
2026-04-30 21:26:06 +10:00
joshsiv-mulga c73e172fac fix(vpcd): drop link-local fallback, auto-derive gw_lrp_range (mulga-siv-36)
In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet
IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream
router silently drops egress. Hardware deployment confirmed the previous
fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not
explicitly configured.

Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16
.. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on
overlap with the per-VM EIP range; clamp inside the subnet; guard the
upstream gateway IP. IGW attach now errors out instead of falling back
to link-local when allocation fails — surfaces misconfiguration loudly.

Adds focused unit tests for gwLrpRange, ValidatePoolConfig and
nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and
PoolWithGatewayIP regression tests.
2026-04-30 17:38:02 +10:00
tomnewton-mulga 2dae3f00bd Merge branch 'fix/iso-public-ip-boot-race' of github.com:mulgadc/spinifex into fix/iso-public-ip-boot-race 2026-04-30 16:42:48 +10:00
joshsiv-mulga 71ef7236ca fix(vpcd): allocate gateway LRP IP from pool in centralized NAT (mulga-siv-36)
In veth/macvlan modes the gateway LRP carries all egress, so it must hold
a WAN-subnet IP — otherwise the upstream router silently drops ARP for
the WAN nexthop (RFC 826) and the default route never resolves. The
hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed
centralized NAT (introduced under mulga-siv-26).

Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW
attach the gateway LRP gets a fresh IP from that range, persisted as
external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and
retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the
gw range; ValidatePoolConfig rejects overlap with the EIP range.

Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and
extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs
to allocated pool IPs at startup.
2026-04-30 16:04:10 +10:00
joshsiv-mulga 87dd7d242a fix(setup-ovn): mark br-int/br-ext Unmanaged for systemd-networkd (mulga-siv-37)
Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs
admin-down: networkd has no matching .network so it tags them as unmanaged
but never brings them up. OVS dataplane works but ovn-controller flow
programming and any tooling that probes link state misbehaves.

Drop /etc/systemd/network/05-spinifex-ovs-internal.network with
Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br`
already runs the netdev up at create time.
2026-04-30 16:03:54 +10:00
Julian Sommer 6ec2f4bd08 fix: usb install doc formatting 2026-04-30 14:32:56 +10:00
joshsiv-mulga 82b0bdfd56 revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New
ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to
catch unicast OFFER replies that bypass the bridge netdev. On dev hosts
where upstream honors the BOOTP BROADCAST flag, the wrapper regressed
DORA — public-IP allocation failed where main works. Reverting until
the promisc path is validated against both broadcast- and unicast-OFFER
servers.
2026-04-30 14:26:43 +10:00
Julian Sommer 97977be286 fix: install-usb doc 2026-04-30 14:07:13 +10:00
tomnewton-mulga df2e652ca6 OVN GatewayMAC fixes 2026-04-30 12:17:17 +10:00
Julian Sommer 84278b794f chore: reintroduce dev branch in CI + scripts
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml)
- workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch
  so checkout@v6 uses the default branch implicitly
- dependabot: route gomod/github-actions/npm PRs to dev
- diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev
- iac/proxmox-template: default git_branch to dev for cluster builds

Refs docs/development/improvements/reintroduce-dev-branch.md
2026-04-30 11:50:13 +10:00
tomnewton-mulga 819a7a8cd3 Merge branch 'fix/iso-public-ip-boot-race' of github.com:mulgadc/spinifex into fix/iso-public-ip-boot-race 2026-04-30 11:14:10 +10:00
tomnewton-mulga 9f0b8c5d7d boradcast fixes 2026-04-30 11:14:03 +10:00
joshsiv-mulga ecfbd4fd19 fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33)
Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").

Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
  their replies are L2 broadcast which the bridge always passes up.

Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.
2026-04-30 10:53:59 +10:00
joshsiv-mulga 9bc15aa306 fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33)
The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.

br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.

Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.

The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.

Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.
2026-04-29 23:35:42 +10:00
joshsiv-mulga 619f944d80 fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34)
ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.

Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.

The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.

Plan: docs/development/bugs/iso-public-ip-boot-race.md
2026-04-29 21:54:23 +10:00
joshsiv-mulga 7b8bb92905 fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33)
GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.

Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.

Plan: docs/development/bugs/iso-public-ip-boot-race.md
2026-04-29 21:54:07 +10:00
joshsiv-mulga 9b9bcc5b8f fix(setup): mask dhcpcd.service even when apt install skipped
bc33c72 added the mask but placed it after the SKIP_APT early-return.
CI bootstrap runs setup.sh with INSTALL_SPINIFEX_SKIP_APT=1 against
runners that already have dhcpcd-base, so the mask never ran. On
Debian-13 the auto-enabled dhcpcd.service kept binding br-wan and
racing vpcd's nclient4, reproducing the mulga-siv-32 DORA timeout
on the public-IP allocation path. Lift the disable+mask out of the
apt branch so it runs unconditionally.
2026-04-29 16:10:59 +10:00
Julian Sommer 6538a6cb69 feat: DescribeVolumeModifications (#174) 2026-04-29 15:54:43 +10:00
Josh Siviour 04f63ad8c6 feat(iso-installer): DHCP race fix + vpcd reconcile gate + ELB mgmt route + HashMAC + diagnostics (#172)
* preflight tweak

* fix: pin gateway LRP to link-local, end intra-cluster ARP collision

Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on
attach. Multi-VPC clusters all GARP'd the same .160 with three different
deterministic MACs on the shared br-ext localnet, flapping the upstream
ARP cache and dropping 100% of inbound traffic to dnat_and_snat external
IPs (mulga-siv-26).

handleIGWAttach and reconcileIGW now unconditionally set the gateway
router port to 169.254.0.1/30. The default route's OutputPort already
pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay
allocated only for per-VM dnat_and_snat external addresses, which are
unique by construction.

Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites
stale Networks in place via the new UpdateLogicalRouterPort path. Needed
because Reconcile gates IGW work on "ext switch missing", so a cluster
shipped with the buggy CIDR never re-enters reconcileIGW.

Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision
and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests
updated to assert link-local and OutputPort=&gwPortName. Log field
gateway_ip renamed to lrp_network to signal the semantic shift.

* fix: rewrite utils.HashMAC to SHA-256 / 46-bit output

The old 24-bit rolling hash with literal class prefix had birthday-paradox
collision probability of 1% at ~580 ids and 50% at 4096 — well within the
range of a single deployment's resource population (LRPs, ENIs, dev/mgmt
NICs). On a shared L2 segment, two clusters could trivially produce
duplicate MACs, breaking neighbour discovery and DHCP.

Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes
and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the
first octet. Output entropy goes from 24 bits to 46 bits — 1% collision
boundary moves from ~580 to ~1.2M ids.

Drop the redundant prefix arg: it was no longer the literal output prefix
under the new scheme, just folded into the hash input. Class disambiguation
moves to the caller, where it's explicit. Three of four callers had nothing
to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and
mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags.

Also consolidate vpcd.generateMAC, which was a local duplicate of the
broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation.

Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md
Bead: mulga-siv-27

* fix: pin HashMAC first octet to 0x02

Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast,
bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived.
Technically valid LAA unicast, but produced MACs like 4a:.. and fe:..
that visually encroach on vendor OUI space and tripped CI/ops checks
expecting all generated MACs to start with 02:.

Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output
entropy drops from 46 to 40 bits — birthday-paradox 1% collision at
~150k ids, still 256x the old 24-bit bound (~580).

* chore(e2e): dump cross-chassis OVN state on guest SSH failure

Adds chassis registrations, port_binding claims, gateway_chassis rows,
ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to
dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis
dataplane stalls (cr-gw chassis vs remote chassis with the VM bound).

* fix: gate vpcd startup reconcile behind NATS KV leader election

N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router
with no atomicity, producing duplicate rows that ovn-nbctl rejects with
"Multiple logical routers named '...'. Use a UUID."

Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key
in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile,
ReconcileFromKV, and the two retrofit passes — others Subscribe and rely
on the vpcd-workers queue group for runtime events. MaxAge=60s on the
bucket caps recovery time when a leader crashes mid-reconcile.

Falls through (elected=true) when KV is unavailable so first-boot does
not deadlock on a missing JetStream.

mulga-siv-29

* fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30)

Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s
because the lb-agent inside the ALB VM has no return path to AWSGW.

The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP
for single-node and deliberately skips adding the /32 mgmt host route — the
WAN-return-path concern in the comment is real but applies only to
internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that
gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when
the agent dials advertiseIP the reply targets the VM's private subnet IP from
the host's WAN with no route back.

Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the
existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and
falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only
internal LBs need the /32 route, so internet-facing single-node still gets
no bootcmd and the host-initiated WAN path stays clean.

Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the
fallback has the inputs it needs.

* fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31)

The test was calling real SetupComputeNode("chassis-test", ...) which shelled
out via sudoCommand to set external_ids:system-id=chassis-test on the host's
live OVS. On a dev box with OVS installed this overwrote the persistent UUID
(owned by openvswitch-switch and re-applied only on reboot), so vpcd's
discoverChassis filtered out the legitimate SBDB chassis row as 'stale local'
and crashlooped — taking out all VPC ops including ALB ENI provisioning.

Bind sudoCommand to a var so tests can swap it. Stub it in the test to return
exec.Command("/bin/false") and assert SetupComputeNode returns an error. The
test's intent (signature + error-path coverage) is preserved; nothing on the
host is touched.

* chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32)

Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8
DORA timeouts correlate with concurrent Acquire calls on the same
bridge. Tracks per-bridge in-flight count and wallclock duration of each
Acquire; logs WARN on failure with both fields, DEBUG on success.

No behavioural change.

* ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32)

run-multinode-e2e.sh dumps node journals via its own EXIT trap, but
run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts
fail the cluster is destroyed in Teardown without surfacing any daemon
logs, blocking root-cause analysis.

Add a workflow step that runs after the test scripts but before
Teardown when a previous step failed, ssh'ing to each node and pulling
the last 200 lines of every spinifex-* unit. Best-effort: skips
gracefully when inventory or nodes are unreachable.

Diagnostic-only; no behavioural change to the test path.

* fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32)

Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when
dhcpcd-base is installed. It binds br-wan and races vpcd's in-process
nclient4 client for upstream OFFERs, draining the lab DHCP pool with
duplicate leases and causing intermittent DORA timeouts in the e2e
public-IP path (worst on cell-8 real-multi/dhcp/debian-13).

The ISO installer already masks the unit at install time
(cmd/installer/install/install.go:316). The binary install path used by
e2e nightly missed it, so debian hosts installed via setup.sh ran with
both DHCP clients live. Mirror the ISO mask here.

Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable
dhcpcd.service on Noble).

* ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32)

Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the
dhcpcd.service journal alongside the spinifex-* units, plus
systemctl is-enabled / is-active and the DHCP client port (UDP 68)
listener so we can see directly whether dhcpcd is running and bound
to br-wan when an Acquire times out.

---------

Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
2026-04-29 14:38:29 +10:00
Julian Sommer 3494ea4eb0 fix: don't log secret key 2026-04-29 14:31:39 +10:00
Julian Sommer 050054ad0f fix: enable alpine x86 image 2026-04-29 12:03:58 +10:00
Julian Sommer bc9f881034 feat: AWS input validation compliance (#171) 2026-04-29 11:19:44 +10:00
Julian Sommer 3dad452b09 feat: reserve compute for spinifex services (#170) 2026-04-28 16:11:49 +10:00
Julian Sommer 5911a878c6 fix: flaky daemon test 2026-04-28 14:40:37 +10:00
Julian Sommer 94735e7d19 chore: interface compliance comp time check 2026-04-28 14:27:33 +10:00
Julian Sommer b0d58ff0f6 chore: add missing tests 2026-04-28 14:18:56 +10:00
Julian Sommer a3d1d5f7b3 chore: delete zero-op tests (#169) 2026-04-28 14:10:46 +10:00
Julian Sommer a3994510e0 feat: e2e nightly reboot tests (#168) 2026-04-28 13:47:58 +10:00
Julian Sommer 3da6e3cfa9 docs: update DESIGN.md 2026-04-28 11:37:31 +10:00
Julian Sommer 138083f6c8 chore: only run e2e nightly mon to friday 2026-04-28 11:20:24 +10:00
Julian Sommer f2dcb2f33f fix: tofu nightly e2e 2026-04-28 10:58:05 +10:00
Julian Sommer 7128dc785c chore: dhcp e2e nightly 2026-04-28 10:54:19 +10:00
dependabot[bot] d663d143a4 chore: bump the npm-dependencies group across 1 directory with 22 updates (#164) 2026-04-28 10:37:24 +10:00
Josh Siviour dd30790d51 fix: reconcile external IPAM pool on config drift (mulga-siv-24) (#167)
initPool's first guard inverted: !errors.Is(err, ErrKeyNotFound) was
true when err==nil, so initPool returned immediately when a record
existed and never reached the drift-reconcile or already-initialized
branches. As a result, range_start/range_end/source changes in
spinifex.toml never propagated and nextAvailableExternalIP failed
with "invalid IP range:  - " on dhcp -> static transitions.

Restructure to: bail on hard errors, reconcile or no-op when record
exists, fall through to create when ErrKeyNotFound.

Verified on dev cluster: dhcp -> static toml change now reconciles
KV bounds on restart and RunInstances allocates from the new range.

Stale Allocated entries and stale gateway_ip on source change remain
out of scope for this hotfix; tracked under mulga-siv-24 follow-ups.
2026-04-28 10:20:51 +10:00
Julian Sommer a7f965526b fix: lint error 2026-04-28 09:59:28 +10:00
dependabot[bot] f3cfee3189 chore: bump actions/upload-artifact in the github-actions group (#163) 2026-04-28 09:16:05 +10:00
dependabot[bot] b2921dbad6 chore: bump golang.org/x/crypto (#162) 2026-04-28 09:11:29 +10:00
dependabot[bot] 1f5aee54d7 chore: bump postcss (#166) 2026-04-28 09:11:07 +10:00
Ben Duncan b18bad5b93 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-27 21:56:10 +10:00
Ben Duncan a2107bba8a Manual testing, spinifex.toml changes for dhcp/static, pool names, retry timeout for DHCP lease 2026-04-27 21:56:00 +10:00
joshsiv-mulga 05d5148611 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-27 21:18:40 +10:00
Ben Duncan 94600d7b84 OVN chassis issues, new reconsile function, reboot fixed (#165) 2026-04-27 15:48:47 +10:00
Ben Duncan a25bc6ef19 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-27 14:47:45 +10:00
joshsiv-mulga 8735275d84 Merge commit 'eeb7ebc5db77e06db11ed7a58b9a70cdcc4128fb' into feat/iso-installer 2026-04-27 13:09:51 +10:00
joshsiv-mulga 187b641726 fix: order spinifex-daemon after ovn-controller too
Symmetric to the openvswitch-switch ordering: ensures ovn-controller
shutdown waits for spinifex-daemon's instance teardown, the correct
general invariant for VM cleanup since logical port programming
flows through OVN. PR1a flagged this as optional follow-up.

Bead: mulga-siv-20.1
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a)
2026-04-27 12:39:58 +10:00
joshsiv-mulga 660fe5227e fix: idempotent tap setup + race-free terminate cleanup
Three minimal edits to close the orphan-OVS-port failure mode that
survives PR1a's systemd ordering fix (host crash, OOM-kill of ovsdb,
operator manually stopping OVS on a live host) and the
terminate-during-pending race that leaks taps when a user aborts a
slow launch.

1. SetupTapDevice (network.go): unconditional `ovs-vsctl --if-exists
   del-port br-int <tap>` + separately-gated kernel `ip tuntap del`.
   The previous kernel-side-gated cleanup missed orphan OVS ports
   because OVS conf.db survives reboot while kernel taps don't.
   Rejected `--may-exist add-port`: it would silently keep stale
   external_ids:iface-id/attached-mac from a prior launch.

2. stopInstance mgmt-tap branch (daemon.go) + CleanupMgmtTapDevice
   (network.go): derive the mgmt tap name from instance.ID rather
   than reading instance.MgmtTap, since that field is set only after
   SetupMgmtTapDevice returns and a terminate racing mid-launch
   would skip cleanup. CleanupMgmtTapDevice gains a kernel-presence
   short-circuit so unconditional invocation doesn't log spurious
   "Device does not exist" errors for instances that never reached
   SetupMgmtTapDevice.

3. LaunchInstance (daemon.go): `launchStillValid` helper re-checks
   instance status before StartInstance and TransitionState, in
   addition to the entry guard. On detection of shutting-down /
   terminated, log Info and return nil — the concurrent terminate
   goroutine owns cleanup; this is an expected race outcome, not an
   error. Removes the spurious "Failed to transition instance to
   running" ERROR that surfaced on every legitimate
   terminate-during-pending. Plus the entry path now returns nil
   instead of an error for the same reason.

Also lands `scripts/start-then-terminate-pending.sh` (added during
investigation) which reproduces the runtime race for manual
verification.

Bead: mulga-siv-20.2
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1b)
2026-04-27 12:33:54 +10:00
joshsiv-mulga 58bdcd43f5 fix: order spinifex-daemon after openvswitch-switch on shutdown
systemd reverses After= on shutdown so daemon now runs CleanupTapDevice
while ovsdb-server still answers. Eliminates orphan OVS port left in
conf.db when host reboots — root trigger for "port already exists" on
post-reboot RunInstances. spinifex-vpcd.service already had this; the
daemon unit was missed.

Also pull openvswitch-switch + ovn-controller into spinifex.target Wants
so the platform brings them up explicitly rather than relying on
implicit ordering from spinifex-vpcd alone.

Bead: mulga-siv-20.1
Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1a)
2026-04-27 12:19:40 +10:00
Julian Sommer eeb7ebc5db chore: enable nightly nginx-alb workbook (#161) 2026-04-24 17:14:57 +10:00
Ben Duncan e799af48e0 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-24 16:33:36 +10:00
joshsiv-mulga 1059412e28 fix: localSystemID + discoverChassis use Output() to skip sudo PAM noise
Follow-up to d509df7. The sudo PAM audit bug exists in two more vpcd call
sites beyond portToBr: localSystemID() and discoverChassis().

When vpcd.service's AmbientCapabilities trip sudo's PAM, stderr carries
"sudo: unable to send audit message: Operation not permitted". CombinedOutput
merges that into stdout and corrupts the parsed value:

  localSystemID → localID="sudo: ...\n\"add0104e-..."
  discoverChassis → tries to parse the stderr banner as chassis name/hostname

Downstream effect: discoverChassis skips the real local chassis as "stale",
vpcd falls back to cfg.ChassisNames (e.g. "chassis-josh-banksia") which no
chassis in SBDB actually uses, so every gateway_chassis row points at a
phantom chassis. cr-gw* chassisredirect ports never bind → no proxy-ARP
for EIPs → external ping fails with ARP timeout.

Switch both to Output() + exec.ExitError.Stderr, matching d509df7's pattern.

Found during mulga-998.b reboot drill while verifying external reachability:
chassis discovery log showed `localID="sudo: unable to send audit message...`,
`cr-gw-vpc-* chassis=[]`, and host ping to EIP returned Destination Host
Unreachable. With this fix applied, discoverChassis returns the real system-id
and (after stale gateway_chassis rows are cleared) cr-gw binds, proxy-ARP
responds, and EIP ping succeeds end-to-end.

Note: existing gateway_chassis rows written pre-fix still carry the phantom
chassis_name. Cleanup is one-shot — delete those rows and trigger an IGW
detach+attach, or run reset-dev-env.sh. vpcd does not reconcile existing
rows on startup (potential future retrofit, similar to Fix 3's
RetrofitAllExternalLocalnetOptions).
2026-04-24 16:31:19 +10:00
Ben Duncan 0434a823cc Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-24 16:14:39 +10:00
Ben Duncan 40a42cbb73 grub branding tweaks 2026-04-24 16:14:30 +10:00
joshsiv-mulga d509df731a fix: portToBr uses Output() to skip sudo PAM audit noise
vpcd.service sets AmbientCapabilities which trips sudo's pam_limits into
emitting 'sudo: unable to send audit message: Operation not permitted'
on stderr. CombinedOutput merged that into stdout and poisoned the
port-to-br result, causing verifyBridgeMode to fail with a bogus bridge
name on service start after reinstall.

Switch to Output() (stdout only) + exec.ExitError.Stderr on error.
2026-04-24 16:10:55 +10:00
joshsiv-mulga c01207b342 merge: fix/dhcp-vpcd-issues (mulga-998.b Fixes 1+1b+2+3) into feat/iso-installer
Fixes DHCP/vpcd external-path issues discovered during reboot drill on josh-banksia:
- Fix 1: persist veth-wan pair via networkd .netdev/.network
- Fix 1b: admin-up veth-wan-ovs via second .network unit (peer stayed LOWERLAYERDOWN)
- Fix 2: loud detectBridgeMode + verifyBridgeMode sanity check
- Fix 3: retrofit nat-addresses=router + network_name=external on every vpcd startup via RetrofitAllExternalLocalnetOptions

Bead: mulga-siv-13 (closed). Plan: docs/development/bugs/dhcp-vpcd-issues.md.
Reboot-drill evidence: smoketest 9/9, Drill 2 retrofit verified post-reboot.

# Conflicts:
#	scripts/reset-dev-env.sh
#	spinifex/handlers/ec2/vpc/external_ipam.go
#	spinifex/services/vpcd/vpcd.go
2026-04-24 15:59:57 +10:00
joshsiv-mulga a00550a200 fix: retrofit localnet options on every vpcd startup (mulga-998.b Fix 3)
ensureLocalnetOptions was only invoked inside reconcileIGW, which is gated
on the external logical switch being absent — so a port whose options were
cleared or drifted from the current bridge mode would never be corrected
on restart. Drill 2 (stale-options recovery) exposed this: clearing
ext-port options and restarting vpcd left the port with options={}.

Changes:
- Extend ensureLocalnetOptions to also restore network_name=external (not
  just nat-addresses). A manual `ovn-nbctl clear` wipes both, and the port
  is non-functional without network_name.
- Add RetrofitAllExternalLocalnetOptions: walks every OVN logical switch
  tagged spinifex:role=external and calls ensureLocalnetOptions on its
  ext-port-<vpcId>. OVN is the source of truth; NATS KV records may be
  missing or stale.
- Call RetrofitAllExternalLocalnetOptions after ReconcileFromKV during
  vpcd startup.
- Tests: network_name restoration in both centralized and direct modes;
  RetrofitAll across multiple switches with mixed stale / correct / non-
  external state.
2026-04-24 15:31:42 +10:00
joshsiv-mulga 6b75470b0f fix: admin-up veth-wan-ovs via networkd (mulga-998.b Fix 1 follow-up)
Fix 1 persisted the veth pair across reboot via
15-spinifex-veth-wan.{netdev,network}, but only configured
veth-wan-br. Its peer veth-wan-ovs had no .network unit, so
networkd left it unmanaged and admin-DOWN after reboot. That
cascades: peer goes LOWERLAYERDOWN, br-wan loses carrier,
external DHCP/NAT path broken — despite vpcd correctly
detecting mode=veth and the sanity check passing.

OVS does not admin-up external (non-internal) ports — that's
networkd's job. Pre-reboot the setup-ovn.sh 'ip link set
veth-wan-ovs up' made it work, but that doesn't persist.

Write a second unit 16-spinifex-veth-wan-ovs.network with
ConfigureWithoutCarrier=yes so networkd flips admin state
without taking ownership of bridge/address assignment (OVS
retains that). Cleanup paths in the non-veth branch of
setup-ovn.sh and reset-dev-env.sh updated to remove the new
unit too.

Verified live via 'networkctl reload': veth-wan-ovs and
veth-wan-br both UP, br-wan LOWER_UP. Reboot drill per
docs/development/bugs/dhcp-vpcd-issues.md 'Live drill 1'
still required to confirm persistence.
2026-04-24 15:03:05 +10:00
joshsiv-mulga 825f7202b4 test: cover verifyBridgeMode empty-parameter branches
Add four branch tests for verifyBridgeMode that were missing from the
initial Fix 2 commit:
- direct mode rejects empty external_interface
- direct mode rejects empty dhcp_bind_bridge
- veth mode rejects empty dhcp_bind_bridge
- veth mode fails when veth-wan-br link missing

Lifts diff coverage from 59.5% to 65.6%, meeting the 65% gate.
2026-04-24 14:18:00 +10:00
joshsiv-mulga 853da4cc2a fix: loud detectBridgeMode + verifyBridgeMode sanity check (mulga-998.b Fix 2)
Make bridge-mode detection loud and fail-start when runtime state does
not match config. Closes D4/D12/D18 of dhcp-vpcd-issues.md.

Changes:
- detectBridgeMode: slog.Info at each decision, slog.Warn on fall-through
  (was silent Debug — let the veth persistence bug hide for weeks).
- Extract resolveBridgeConfig so mode/bind-bridge resolution is unit
  testable. Empty mode now stays empty so verifyBridgeMode rejects it
  with an explicit list of supported values.
- Add verifyBridgeMode: refuses to start when external iface missing
  (direct), when external iface not on the OVS bridge (direct), when
  veth-wan-ovs missing or attached to the wrong OVS bridge, or when
  veth-wan-br missing or on the wrong Linux bridge. References
  OvnExternalBridge (= "br-ext") so the sanity check tracks the contract,
  not a hardcoded string.
- Wire verifyBridgeMode into launchService. Error is loud and includes
  operator guidance ("re-run scripts/setup-ovn.sh").
- Injectable probes (ifaceIsMacvlan, ifaceExists, portToBr,
  readLinkMaster) so tests can exercise every branch without touching
  real interfaces.
- Tests: 16 new unit tests covering detectBridgeMode, resolveBridgeConfig,
  and all verifyBridgeMode branches.
2026-04-24 13:08:04 +10:00
joshsiv-mulga 904071099a fix: retrofit nat-addresses=router on mode transition (mulga-998.b Fix 3)
CreateLogicalSwitchPort only seeds options:nat-addresses at first
creation. If a VPC's localnet port was created during a vpcd run that
saw direct mode (e.g. pre-Fix 1 reboot without the veth pair), the port
keeps stale options forever — OVN never GARPs for the external IP and
centralized NAT stays broken.

Add TopologyHandler.ensureLocalnetOptions: read the LSP, diff
options["nat-addresses"] against useCentralizedNAT(), update only on
mismatch. Called from reconcileIGW after CreateLogicalSwitchPort so the
retrofit runs on every reconcile pass (no-op when already correct).

Mock tracks UpdateLogicalSwitchPortCalls so the idempotency test
asserts no write was issued.
2026-04-24 12:40:48 +10:00
joshsiv-mulga abd93098fc fix: persist veth-wan pair across reboot (mulga-998.b Fix 1)
setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing
recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs
as a port on br-ext but the peer is gone — the external path silently
breaks and vpcd falls through to direct mode.

- setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network}
  in the veth branch so systemd-networkd recreates the pair on boot. Add
  a single pre-case guard that rips down stale veth persistence + the OVS
  port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode
  transitions are idempotent (D17).
- reset-dev-env.sh: delete the netdev/network units during cleanup so
  `make reset` gives a truly clean slate.
- spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service
  so vpcd's sanity checks see a populated kernel + OVSDB (D5).
2026-04-24 12:37:01 +10:00
Julian Sommer 5126a1df8d tweak: README.md link 2026-04-24 12:34:04 +10:00
joshsiv-mulga 112738198f fix: rename wan_bridge to dhcp_bind_bridge (mulga-998.a)
The old wan_bridge name misled operators into pointing the field at
the OVN-side bridge 'br-ext', which never sees LAN DHCP traffic. On
consumer-router LANs (WAN NIC enslaved to a Linux bridge) the vpcd
DHCP client's AF_PACKET bind failed silently, blocking the default
install path.

- Rename Go field WanBridge -> DhcpBindBridge and TOML key wan_bridge
  -> dhcp_bind_bridge across config, vpcd service, external IPAM,
  daemon pool wiring, admin settings, and spinifex.toml template.
- Rewrite detectedWanBridge() -> detectedDhcpBindBridge(): the
  Linux-bridge branch now returns the bridge name verbatim (e.g.
  "br-wan") instead of synthesising "br-ext". OVS-vs-Linux probe via
  ovs-vsctl is no longer needed for this field.
- Add checkLegacyWanBridgeKey() in cmd/service.go: vpcd fail-starts
  if the deprecated 'wan_bridge' TOML key or SPINIFEX_VPCD_WAN_BRIDGE
  env var is set. No silent alias, no auto-rewrite (per plan D3/D13).
- Template carries an operator-facing comment warning never to set
  the field to 'br-ext'.
- 7 new unit tests for the detector branches and fail-start paths.

Smoke on ben-bottlebrush: spx admin init --force writes
dhcp_bind_bridge matching the live br-wan Linux bridge; VM launched
with --map-public-ip-on-launch reached from LAN; restoring a
legacy-key backup causes vpcd to enter failed state with the
deprecated-key guidance log.

Plan: docs/development/bugs/dhcp-vpcd-issues.md (Fix 4)
Bead: mulga-998.a (unblocker for .b / .c)
2026-04-24 12:27:41 +10:00
Julian Sommer c7d4040dd3 feat: nightly e2e (#155) 2026-04-24 12:25:03 +10:00
Josh Siviour 068aef5589 feat: DDIL E2E test harness (mulga-siv-5) (#160)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)

Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:

- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
  interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
  /local/instances (placeholder types until daemon-local-autonomy lands),
  plus /health for scaffolding validation

Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.

* feat: add DDIL harness fault injection and link profiles

Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:

- harness/fault.go — PartitionNode (per-peer iptables DROP with
  orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
  RestartDaemonOnly, WaitForMode (polls /local/status for typed
  DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
  ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
  SATCOM, HFData, Flapping constants (unvalidated approximations per
  plan §2 table; validation tracking lives in TEST_COVERAGE.md)

All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.

* feat: add DDIL harness witness VM helpers

LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.

* feat: add DDIL harness snapshot and state assertions

TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.

* feat: add DDIL harness reset, retry, and quarantine

ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.

AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.

Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.

bd: mulga-siv-5.5

* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)

Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.

* feat: add DDIL E2E CI workflow (workflow_dispatch only)

Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
2026-04-23 22:38:29 +10:00
Ben Duncan c908662f69 DHCP tweaks, inflight 2026-04-23 21:37:30 +10:00
Ben Duncan 6f0821d02b Latest merge 2026-04-23 15:01:28 +10:00
Josh Siviour 604693ea0c fix: wrap missed NATS/S3 dial sites with admin.DialTarget (#159)
siv-8.5 migrated daemon + awsgw but missed the sidecar services and
the spx CLI. vpcd and viperblockd read cfg.NatsHost / cfg.S3Host raw
from spinifex.toml; when BindIP=0.0.0.0 the TLS handshake fails SAN
verification because 0.0.0.0 is not a cert SAN. Same for the spx get
CLI path.

Call sites:
  spinifex/services/viperblockd/viperblockd.go:173  (NATS)
  spinifex/services/viperblockd/viperblockd.go:407  (S3 s3.S3Config)
  spinifex/services/viperblockd/viperblockd.go:533  (S3 nbd.NBDKitConfig)
  spinifex/services/vpcd/vpcd.go:277                (NATS)
  cmd/spinifex/cmd/get.go:93                        (CLI NATS)

admin.DialTarget rewrites 0.0.0.0:N to 127.0.0.1:N — loopback is always
a SAN and in-process sidecars share the host with NATS/predastore.
2026-04-23 14:42:13 +10:00
Ben Duncan 1d38311e07 main merge 2026-04-23 14:10:34 +10:00
Josh Siviour 75fb15be2b feat: spx admin init/join listen-vs-advertise split + br-mgmt OVS provisioning (#158)
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway

Convert br-mgmt from an externally-provisioned Linux bridge to an OVS
bridge owned by scripts/setup-ovn.sh, giving every deployment type
(tofu-cluster, production single-node, production multi-node) a single
provisioning path.

setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface /
--no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone
(excluded from ovn-bridge-mappings so ovn-controller ignores it) and
assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot.

SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master`
to `ovs-vsctl add-port` / `del-port` to match the new bridge type.

Add a loopback guard to wireLBAgentConfig's fallback branch: on a
production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge,
the fallback previously injected 127.0.0.1 into LB VM cloud-init —
silently unreachable because the VM's loopback is its own. The guard
leaves GatewayURL empty and the existing "LB agent gateway URL not
configured" error fires instead. Regression covered in
daemon_wire_lb_test.go.

* fix: auto-detect primary IP for admin init --bind default

Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback
only, so system VMs on br-mgmt could never reach it. Auto-detect the
default-route source IPv4 instead, which keeps TLS cert SANs specific
(still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1
explicitly to preserve loopback-only behavior on laptops.

Closes mulga-siv-7.

* test: refresh wire lb loopback-bind comment for siv-7

* fix: dev-install scripts should let admin init auto-detect --bind

Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while
setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with
no listener. Drop the explicit --bind so auto-detect picks the primary IP
and the siv-6 mgmtRouteVia path routes LB traffic to a real listener.

* fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template

PR #144 (nats TLS) made every service listen on --bind's specific IP. For
awsgw and predastore this broke localhost-reachability that the UI proxy
(spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on —
latent since #144, surfaced by siv-7's auto-detect default on dev installs
(502 from the UI).

Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen
wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host
daemon → predastore) keep working, LAN clients still reach them via the
LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}}
because peer-advertised values must not resolve to 0.0.0.0/loopback.
predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1
for local-node consumers, so existing callers see a usable connect address.

* revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8)

Clears the deck before layering the listen-vs-advertise split
(mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node
default and introduces an explicit --advertise flag for off-host dial
targets; the reverted approach (auto-detect a specific IP at init,
hardcode 0.0.0.0 in the template, guard against loopback in
wireLBAgentConfig) becomes redundant once that lands.

Reverts:
- bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template
- f20a47b fix: dev-install scripts should let admin init auto-detect --bind
- 49b0b4e test: refresh wire lb loopback-bind comment for siv-7
- b88dfdf fix: auto-detect primary IP for admin init --bind default

Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig
fallback and its regression test — subsumed by the GatewayURL
precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error,
no silent empty-string fallthrough). The br-mgmt OVS provisioning
from 74b6e2d is retained.

Plan: docs/development/bugs/spx-admin-init-bind.md

* feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2)

Introduce the two small address-resolution helpers the listen-vs-
advertise split needs, with full unit coverage. No callers yet —
dial-site migration lands in mulga-siv-8.5.

- DialTarget(listenAddr): rewrites wildcard listen addresses
  (0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process
  clients dial a target the daemon's cert already has in its SANs.
  Specific IPs, loopback, hostnames, and malformed input pass through.
- AdvertiseHost(listenAddr, advertiseIP): picks the host off-host
  clients should dial. Wildcard listen → advertiseIP; specific host →
  that host. Port is dropped so callers append the protocol-appropriate
  port.

Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1

* feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3)

Pure data-model change. Add AdvertiseIP to:
- admin.ConfigSettings (local node, populated by init).
- config.Config (per-node; mapstructure "advertise").
- formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake
  carries it between peers.

buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so
joined nodes from older clusters / pre-siv-8 joiners keep working.

No behaviour change yet — templates, CLI flag, and dial-site migration
land in siv-8.4 and siv-8.5.

* feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4)

Introduce the listen-vs-advertise split at the CLI boundary.

admin init/join:
- --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert).
- New --advertise <IP> flag on both init and join; optional, auto-detected
  from DetectNetwork().WAN.IP when --bind is wildcard or loopback.
- resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements
  precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 +
  warning.
- Join resolves advertise before building the JoinRequest so peers record
  a reachable dial target (not 0.0.0.0).
- isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator
  can now run "init --nodes 3" without typing a specific --bind.
- Init success message prints bind / advertise / loopback three-way split.

Templates:
- spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered
  only when AdvertiseIP is set.
- predastore-multinode.toml: iam.nats_url host wrapped in the same
  {{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that
  predastore.toml already uses, so co-resident predastore dials its local
  NATS via loopback instead of 0.0.0.0.

Tests cover the resolveAdvertiseIP matrix, the template advertise field
(present + omitted), and the predastore nats_url loopback shim.

Dial-site migration and GatewayURL rework land in siv-8.5.

* feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5)

Switch every in-process dial site to admin.DialTarget so the wildcard
listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon
→ NATS and → predastore (boot, reconnect, status probe, S3 store, service
manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so
loopback-dial a wildcard-listen service is always valid; the old
0.0.0.0 literal was tripping TLS SAN verification.

Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go):

  1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged)
  2. br-mgmt present, AWSGW on 0.0.0.0     → br-mgmt IP (unchanged)
  3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW)
  4. DevNetworking shim                    → 10.0.2.2 (unchanged)
  5. AWSGW on specific IP, no bridge       → that IP (unchanged)
  6. Else                                  → log error, skip assignment

Kills the pre-siv-8 silent empty-URL failure mode that made
CreateLoadBalancer fail with 'missing system credentials: gatewayURL='
on every fresh single-node install. Supersedes siv-6's loopback guard.

Extend SetupAWSCredentials with a wanIP parameter (reserved for a
future --operator-endpoint flag) and thread AdvertiseIP through
finalizeNodeSetup's three call sites (single-node init, multi-node
init, join). Print 'Bind / Advertise / Loopback' on multi-node init
and join completion to match the single-node summary.

Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
2026-04-23 14:02:12 +10:00
Julian Sommer 3c5e09a937 chore: enable frontend lint rules 2026-04-23 12:43:53 +10:00
Julian Sommer fd17c35330 chore: disable frontend lint rules 2026-04-23 12:23:55 +10:00
Ben Duncan d7c95bda5e feat: wire ExternalIPAM to NATS DHCP + move AF_PACKET to vpcd (mulga-991)
Replaces the sudo-dhcpcd shell-out in ObtainDHCPLease / ReleaseDHCPLease
with NATS request/reply against vpc.dhcp.acquire / vpc.dhcp.release.
The daemon-side IPAM passes its nats.Conn into NewExternalIPAM and
threads hostname (option 12) / vendor-class (option 60) through to
vpcd; ExternalIPAllocation now records LeaseExpiresUnix, DHCPServerID
and HWAddr on DHCP-sourced entries. allocateFromPool is restructured
to acquire the lease once up-front and rely on vpcd's idempotent
Manager for CAS-retry safety; a single release fires only if every
CAS attempt fails. Drops the stdout parser, the per-bridge mutex, and
the parse test.

Systemd: spinifex-daemon loses AF_PACKET, ProtectKernelTunables and
MemoryDenyWriteExecute are restored to =yes; spinifex-vpcd gains
AF_PACKET in RestrictAddressFamilies, AmbientCapabilities=CAP_NET_RAW,
and CAP_NET_ADMIN / CAP_NET_RAW / CAP_NET_BIND_SERVICE in the bounding
set. Net privilege footprint narrows.

See docs/development/bugs/dhcp-allocate-error.md.
2026-04-23 12:21:55 +10:00
tomnewton-mulga 265556dc4e updated ISO doc 2026-04-23 11:27:26 +10:00
Julian Sommer 0f89db1893 chore: extract validation logic from gateway function 2026-04-23 10:46:16 +10:00
Julian Sommer 3b0376c3a6 fix: Makefile 2026-04-23 10:04:11 +10:00
tomnewton-mulga cc3c77fdb7 Merge commit '0a7832440504442f685e85048d545848a8f2b8f6' into feat/iso-installer 2026-04-23 09:07:08 +10:00
Ben Duncan f73e151c77 feat: in-process DHCP client and Manager in vpcd (mulga-990)
Introduces the spinifex/services/vpcd/dhcp package with a Client
interface, typed Lease struct, nclient4-backed implementation, and a
Fake for unit tests. Adds DHCPManager in the vpcd package: subscribes
to vpc.dhcp.acquire / vpc.dhcp.release NATS requests, runs a renewal
goroutine per lease with ±10% jitter on T1/T2, persists leases in the
new spinifex-dhcp-leases KV bucket, and fans out vpc.dhcp.lease-expired
when a lease permanently fails to renew. Wires the Manager into
launchService so it starts on vpcd boot and rehydrates persisted leases
via Bootstrap.

Handler-side wiring (ExternalIPAM → NATS) and the systemd capability
move land separately in mulga-991.

See docs/development/bugs/dhcp-allocate-error.md.
2026-04-22 16:45:30 +10:00
Ben Duncan 889ef361eb fix: interim -T retrofit for dhcpcd EIP allocation
Stop ObtainDHCPLease from erroring on hosts where dhcpcd is already
running as a daemon managing br-wan (e.g. server 2). Add -T (test-only
mode) so the invocation does not collide with the host daemon's
control socket on the interface.

Softens ProtectKernelTunables and MemoryDenyWriteExecute in the daemon
unit to accommodate dhcpcd's runtime requirements; both revert when
the in-process Go DHCP client rewrite lands (mulga-989 / mulga-990 /
mulga-991, see docs/development/bugs/dhcp-allocate-error.md).

Interim: leaves the phantom-lease and release-socket-clobber risks
active until the rewrite replaces this path entirely.
2026-04-22 16:30:38 +10:00
tomnewton-mulga 7fd1019efa tweaks to smoke-test.sh 2026-04-21 17:14:14 +10:00
Julian Sommer 0a78324405 feat: missing image commands (#156) 2026-04-21 15:10:09 +10:00
tomnewton-mulga 7cc90f554c daemon race fix 2026-04-21 14:54:00 +10:00
tomnewton-mulga f0d48c4274 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-21 14:33:46 +10:00
Ben Duncan b79460f9ff Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-21 14:31:54 +10:00
Ben Duncan 93a6704944 Merge remote-tracking branch 'origin/main' into feat/iso-installer 2026-04-21 14:29:16 +10:00
Josh Siviour 1e3d82f978 fix: route elbv2 ALB public IP through EIP service (#157)
Internet-facing ALBs allocated their public IP by calling
externalIPAM.AllocateIP() directly from LaunchSystemInstance, bypassing
the EIP service entirely. No EIPRecord landed in the eips KV bucket, the
tag was "auto_assign" (EIP service uses "elastic_ip"), and
DescribeAddresses never returned the IP — so any OpenTofu flow that
expects an AllocateAddress-observable record hung waiting.

LaunchSystemInstance now calls eipService.AllocateAddress +
AssociateAddress when the EIP service is wired (falls back to the old
direct IPAM path otherwise). TerminateSystemInstance releases through
the service so the KV record is cleaned up and vpc.delete-nat is
published. The generic stopInstance teardown is left untouched so
customer RunInstances flows behave identically.

Also refreshes the nginx-alb terraform example to use public subnets
with auto-assigned public IPs (the configuration that actually exercises
this path end-to-end) and tightens the awsgwBindIP loopback guard in
wireLBAgentConfig so the gateway URL is usable by LB agents.
2026-04-21 13:52:48 +10:00
Julian Sommer 1ca8621e16 fix: allow mulga feat branch 2026-04-21 11:14:29 +10:00
tomnewton-mulga 56a1ead706 extra pics for ISO docs 2026-04-21 11:05:38 +10:00
Julian Sommer acefd1cfd4 chore: make README have go 1.26 2026-04-21 10:19:04 +10:00
Julian Sommer 1bad55ce54 chore: update frontend build 2026-04-21 09:20:51 +10:00
dependabot[bot] 869184efb5 chore: bump the go-dependencies group with 2 updates (#153) 2026-04-21 09:01:49 +10:00
dependabot[bot] b46fe2531a chore: bump the npm-dependencies group (#154) 2026-04-21 09:01:29 +10:00
tomnewton-mulga c4b2b13c89 local tests for ISO 2026-04-20 17:11:18 +10:00
Ben Duncan 49acc12390 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-20 16:56:38 +10:00
Ben Duncan 3d1f8340be ISO tweaks, email prompt on installer, improved network detection 2026-04-20 16:56:29 +10:00
tomnewton-mulga 3a5d823fbd Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-20 15:16:34 +10:00
Ben Duncan 27d7fdd12b Latest changes re ISO and binary install path 2026-04-20 15:14:32 +10:00
joshsiv-mulga d6863f3d5e Change LB ISO download to Cloudflare host 2026-04-20 14:18:22 +10:00
tomnewton-mulga 3c6077ef58 ISO install doc tweaks 2026-04-20 13:10:11 +10:00
Ben Duncan 728cdd972b Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-20 12:13:25 +10:00
Ben Duncan c3242b6bc4 Improved reset-dev-env and validated 2026-04-20 12:09:40 +10:00
Josh Siviour f97ff772a3 fix: sentinel TLS errors + shared NATS retry (#152)
* fix: replace stringly-typed TLS error detection with sentinel errors

Export ErrCACertRead and ErrCACertParse from utils/nats.go so callers can
detect permanent TLS configuration failures with errors.Is instead of
substring matching on error messages. Update daemon.go and awsgw.go retry
loops accordingly, and update tests to use assert.ErrorIs.

Refs: mulga-siv-1

* refactor: extract shared NATS connect-with-retry into utils

daemon.connectNATS() and awsgw.connectNATS() were near-identical retry
loops. Extract to utils.ConnectNATSWithRetry() with functional options
for test configurability.
2026-04-20 12:04:40 +10:00
tomnewton-mulga f4e0b4a01b Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-20 11:41:40 +10:00
tomnewton-mulga 6736722533 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-20 11:40:18 +10:00
tomnewton-mulga 9d51800f58 fixed duplicate IP bug between physical and VM nodes 2026-04-20 11:40:10 +10:00
Ben Duncan 36425a48eb refactor: rewrite reset-dev-env.sh against production layout
Reset now targets /etc/spinifex, /var/lib/spinifex, /var/log/spinifex,
/run/spinifex (the privilege-separated layout installed by setup.sh) and
drives re-install from a source build via dev-install.sh instead of the
legacy in-tree $HOME/spinifex + start-dev.sh flow.

Capture region, AZ, external_mode, pool range, gateway, and prefix from
the existing spinifex.toml before the wipe so the reset restores the
current topology rather than applying defaults. Fresh boxes fall back to
ap-southeast-2 and a pool derived from the WAN subnet.

Drop the macvlan branch entirely — bridge-only from here on. Operators
on single-NIC hosts must run reset from the console (documented in the
script header).

dev-install.sh gains DEV_INSTALL_SKIP_INIT so reset can own the post-
setup.sh steps (setup-ovn, spx admin init with preserved args, CA
trust, service start, LB image import). Also honors SPINIFEX_REGION /
SPINIFEX_AZ for non-reset callers, and scopes systemctl reset-failed
to spinifex-* so unrelated failed units aren't silently cleared.

Closes mulga-983.
2026-04-20 11:38:10 +10:00
Ben Duncan a209805ffc Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-20 11:27:58 +10:00
Ben Duncan b591c04880 firstboot tweaks 2026-04-20 11:27:49 +10:00
tomnewton-mulga ad3762bdcc fixing duplicate IP issue 2026-04-20 10:29:50 +10:00
tomnewton-mulga 7b9fdeb60c new docs 2026-04-17 17:08:28 +10:00
tomnewton-mulga 9c98c023ce fixed GRUB splash and drafted ISO README 2026-04-17 16:20:25 +10:00
Julian Sommer f00a376d0b tweak: docs 2026-04-17 16:09:06 +10:00
Julian Sommer c469eecfa1 chore: doc tweaks 2026-04-17 15:27:14 +10:00
tomnewton-mulga aa1fc546bd Merge branch 'main' into feat/iso-installer 2026-04-17 14:48:37 +10:00
Julian Sommer 4f991095ee feat: physical security guide 2026-04-17 14:28:43 +10:00
Ben Duncan a64ee2b2da ISO tweaks 2026-04-17 14:22:34 +10:00
Julian Sommer 14efe44125 feat: flaw remediation policy doc (#151) 2026-04-17 14:15:44 +10:00
Julian Sommer e131caf9ec fix: doc links 2026-04-17 13:41:58 +10:00
Julian Sommer e46b427938 feat: network-connections docs (#150) 2026-04-17 12:44:48 +10:00
Julian Sommer 8d6f4df526 fix: setting up cluster image 2026-04-17 12:13:44 +10:00
Julian Sommer 1a41731ef5 chore: tweak malware-protection doc 2026-04-17 11:59:10 +10:00
Julian Sommer 6f58a3f190 feat(docs): malware protection guide (#149) 2026-04-17 11:17:16 +10:00
Julian Sommer 883fac980c fix: remove unused /join /config endpoints (#148) 2026-04-17 10:32:49 +10:00
tomnewton-mulga 72503cd7ae fix: prevented systemd blocking 2026-04-16 17:58:54 +10:00
Julian Sommer 084e93cfd0 feat: validate image checksums (#147) 2026-04-16 17:00:27 +10:00
Ben Duncan d8924a2ecc ISO tweaks and doc template 2026-04-16 15:38:56 +10:00
Julian Sommer 6f02f6bc22 feat(vpcd): log=true on default-deny ACLs (#145) 2026-04-16 14:45:21 +10:00
Julian Sommer 3986549783 fix: log failed attempts at warn level for compliance 2026-04-16 13:51:46 +10:00
Julian Sommer ff9f9485d2 feat: nats TLS (#144) 2026-04-16 13:46:12 +10:00
Julian Sommer 29ba511b38 feat: disable public nats monitoring (#143) 2026-04-16 11:31:28 +10:00
Julian Sommer 8e9aff3299 chore: update old doc 2026-04-16 10:15:10 +10:00
Julian Sommer 911b36a696 refactor: extract shared helpers from admin init/join (#142) 2026-04-16 10:00:22 +10:00
Ben Duncan 402a04e1b9 Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer 2026-04-16 09:29:15 +10:00
Ben Duncan 25d02707d0 ISO tweaks 2026-04-16 09:29:00 +10:00
Julian Sommer bb9cd8ea23 feat: formation join token (#141) 2026-04-15 16:55:55 +10:00
tomnewton-mulga 0e51a6b06a fix: enable AWS CLI install in firstboot (remove SKIP_AWS flag) 2026-04-15 16:21:56 +10:00
tomnewton-mulga d3e21d3360 fix: added remove usb prompt 2026-04-15 16:21:56 +10:00
Julian Sommer 32effaa16e Merge branch 'main' into feat/iso-installer 2026-04-15 15:50:37 +10:00
tomnewton-mulga 054485e930 refactor: move systemd unit files into installer/systemd package
Extract all inline systemd unit and drop-in content from install.go and
firstboot.go into a dedicated cmd/installer/systemd package for editability.
Bash scripts remain inline in firstboot.go.

Also copies the GRUB splash image from the live ISO into the installed
system's /boot/grub/ and sets GRUB_BACKGROUND so the branded background
appears in the post-install GRUB, not just the installer GRUB.
2026-04-15 15:34:40 +10:00
Ben Duncan c8429f98ae Update ALB doc 2026-04-15 15:25:48 +10:00
Julian Sommer 68ae17dcf1 chore: rename app-id to client-id 2026-04-15 15:03:16 +10:00
Julian Sommer 5b39efb078 fix: make e2e use env vars 2026-04-15 13:40:22 +10:00
tomnewton-mulga 24429ee8b5 fix: simplify firstboot — delegate setup to setup.sh with embedded tarball
Remove the duplicated chown/ownership/symlink block from the firstboot
script. setup.sh (called with INSTALL_SPINIFEX_TARBALL) already handles
service users, directory ownership, sudoers, systemd unit installation,
and enabling spinifex.target — the same steps as the online curl|bash
install.

Firstboot now follows the documented install flow directly:
  1. setup.sh --tarball (air-gapped equivalent of curl|bash)
  2. setup-ovn.sh --management
  3. spx admin init
  4. systemctl start spinifex.target

Also reduce wait-for-nats.sh timeout from 90s to 30s per reviewer feedback.
2026-04-15 13:36:07 +10:00
Julian Sommer 7d6e0780b5 feat: action rate limiting (#140) 2026-04-15 12:19:30 +10:00
tomnewton-mulga 83f1766081 fix: fixed TUI breaking at the end 2026-04-15 11:54:57 +10:00
tomnewton-mulga 811a0230fc fix: fall back to tty1 when requested TTY unavailable instead of aborting 2026-04-15 11:17:58 +10:00
tomnewton-mulga 69c8b20b78 fix: harden ISO installer error handling and safety checks
- partUUID: error on empty blkid output instead of writing UUID= to fstab
- installBootloader: fail if both EFI and BIOS targets fail; track efiErr
- machine-id fallback: write at mode 0600 so systemd can commit on first boot
- Run: defer unmount so partitions are always cleaned up on failure
- copyRootfs: verify critical paths exist after rsync before continuing
- resolveNIC: skip virtual/bridge interfaces (docker0, virbr0, ovs-system)
- EjectAndReboot: use syscall.Reboot instead of exec(reboot) which needs D-Bus
- ui.Run: return error on TTY open failure instead of silent fallback
- firstboot script: trap EXIT to always self-disable, preventing retry loops
- enableUnit/enableBannerUnit: deduplicate into enableSystemdUnit; check Remove
- runAdminBanner: promote service restart failure to slog.Error
- parseNodeConf: log non-ENOENT read failures
2026-04-15 10:59:51 +10:00
Julian Sommer f0bb064237 Merge branch 'main' into feat/iso-installer 2026-04-15 09:57:15 +10:00
Julian Sommer 1ad03bb518 feat(gateway): per-IP auth failure rate limiting (#138) 2026-04-15 09:56:04 +10:00
tomnewton-mulga c75f4a0160 fix: eject iso boot device 2026-04-15 09:46:53 +10:00
tomnewton-mulga 01cd3b9ab1 Merge commit 'b7b639084b3c2a40b97c2f543b88ad498901b77d' into feat/iso-installer
# Conflicts:
#	Makefile
#	cmd/spinifex/cmd/admin.go
#	go.sum
2026-04-14 16:11:49 +10:00
Julian Sommer b7b639084b chore: use ts config files 2026-04-14 14:28:00 +10:00
Julian Sommer 7dcad2ba1d chore: remove unused e2e 2026-04-14 13:59:29 +10:00
Julian Sommer 7cf0783122 fix: set dev tags as prerelease 2026-04-14 13:29:05 +10:00
Josh Siviour 13eae4d40b fix(elbv2): tag-based LB AMI lookup with hard error on miss (#137)
Replace fragile image-name substring match (-lb-/-alb-) with a
DescribeImages tag filter on spinifex:managed-by=elbv2. Remove the
silent fallback that picks the first image in the store when no LB
image exists — CreateLoadBalancer now returns ServerInternal with a
remediation hint instead of persisting a record in StateFailed.

- daemon: tag filter + error return from the resolver closure
- service_impl: resolver signature is func() (string, error); caches
  only successful results so the next call retries after an error
- CreateLoadBalancer gate now requires launcher + networking; AMI is
  resolved inside the gate and failures short-circuit before persist
- admin images import: add --name+--file combo (catalog metadata with
  a local file source) and a --tag key=value flag for explicit tagging
  of file-only imports
- build-system-image.sh + lb.conf: propagate SYSTEM_TAG via --tag so
  locally built system AMIs get the right tag even when the catalog
  is bypassed

Closes mulga-969
2026-04-14 12:52:28 +10:00
Julian Sommer b32111102d fix: README.md 2026-04-14 12:06:33 +10:00
Ben Duncan 71d0f31997 Merge branch 'main' of github.com:mulgadc/spinifex 2026-04-14 11:45:46 +10:00
Ben Duncan f99eb42ca7 Doc tweaks and README for release 2026-04-14 11:45:20 +10:00
Julian Sommer 56cdc158a6 chore: bump viperblock 2026-04-14 11:17:48 +10:00
Julian Sommer a4ae813e38 fix: recursive chmod on shared data dirs so existing image files are group-readable
chmod 0770 without -R only affected the top directory; pre-existing files
(e.g. images imported while owned by another user at mode 0600) stayed
unreadable to the spinifex group, breaking image import on upgrade paths.
2026-04-14 11:05:01 +10:00
Julian Sommer e0118975d1 fix: setup.sh 2026-04-14 10:17:01 +10:00
Julian Sommer 36e3481ca3 fix: enable spinifex.target so services autostart on boot 2026-04-14 10:14:09 +10:00
Julian Sommer 515c313e07 fix: setup.sh newgrp 2026-04-14 10:07:59 +10:00
Julian Sommer 0547bdd1fe feat: loadbalancer frontend (#136) 2026-04-14 09:24:14 +10:00
Julian Sommer e75953f5e2 chore: build ui with latest deps 2026-04-14 09:07:59 +10:00
dependabot[bot] e5a6f3ea60 chore: bump the npm-dependencies group (#134) 2026-04-14 08:59:32 +10:00
dependabot[bot] 36da673ecd chore: bump the github-actions group with 2 updates (#133) 2026-04-14 08:59:01 +10:00
tomnewton-mulga ca63011f25 feat: GRUB env var headless install, disk selection modes, banner
- autoinstall: replace file scanner with env var reader (SPINIFEX_AUTO,
  SPINIFEX_HOSTNAME, etc.) set by spinifex-init.sh from kernel cmdline
- autoinstall: disk selection modes — auto (single or fail), largest,
  smallest, nvme, or exact path; clear error listing available disks
  when auto/nvme finds multiple candidates
- admin banner: add Login row showing spinifex account name
- admin banner: write to /etc/issue (pre-login console) as well as
  /etc/motd (post-login SSH); fix SSH line root@ → spinifex@
2026-04-13 15:37:16 +10:00
Ben Duncan 96b5ad5740 Minor doc tweaks and permissions 2026-04-13 15:23:18 +10:00
tomnewton-mulga e8cb102819 feat: headless autoinstall mode via autoinstall.toml on boot media
Scans the USB EFI partition at boot for autoinstall.toml. When
enabled = true, the installer runs fully unattended using values from
the config file then ejects the USB and reboots — no user interaction
needed.

- New autoinstall package: TOML config struct, boot-media scanner,
  install.Config converter, EjectAndReboot
- Scanner skips iso9660 mounts so the bundled reference copy of the
  template never accidentally triggers headless mode
- "auto" disk target selects the largest non-removable disk
- "auto" NIC target picks the first physical interface
- Validates all required fields before touching any disk
- main.go: check for autoinstall config before launching TUI
2026-04-13 13:19:05 +10:00
Ben Duncan ebd0a64fe5 chore: bump predastore to v1.1.5 (#135)
Picks up multipart upload wedge fixes from predastore v1.1.5:
- 76fdbf1 fix: pool cleanup must not reap connections with in-flight streams (Bug A)
- 7df9645 fix: add mutex to bufferedWALFile to prevent bufio race (Bug B)
- 9dcd3dd fix: drain shard body before taking wal.mu in handlePUTShard (Bug C)
- 43e20f6 fix: widen QUIC flow-control windows to mitigate multipart upload deadlock

plus regression tests for all three bugs (predastore PR #8).
2026-04-13 13:00:25 +10:00
tomnewton-mulga 02aba39942 feat: spinifex login user, credential handoff, networking hardening (ISO)
- Add spinifex login user password via chpasswd in install.go (same
  password as root, collected once in TUI)
- Copy /root/.aws/ to /home/spinifex/.aws/ in firstboot so the operator
  can use the AWS CLI without sudo after first boot
- Fix getty drop-ins to be instance-specific (tty1/ttyS0 only), leaving
  tty2+ immediately available as rescue terminals
- Remove autologin — node now requires password at login prompt
- Pre-start OVS/OVN in firstboot and poll NB DB up to 120s to eliminate
  setup-ovn.sh timeout races on slow physical hardware
- Tighten per-service ownership in firstboot (each service user owns only
  its own /var/lib/spinifex subdirectory)
- Add LAN bridge as a non-critical systemd unit so a missing LAN cable
  can never stall network-online.target or firstboot
- Write dhcpcd.conf with short br-lan timeout; add networking.service
  drop-in to treat exit 1 as success for optional interface failures
- Disable IPv6 on management bridges to silence dhcpcd journal noise
- Fix WAN/LAN enter-key and ESC navigation logic in TUI
- Fix LAN NIC cursor to skip WAN NIC correctly
2026-04-13 12:53:07 +10:00
Josh Siviour db9e93351e fix: nginx-alb + route table NAT GW SNAT event bugs
Bundle of ALB and NAT GW fixes surfaced by the nginx-alb terraform example.

- fix: expose ALB ENI private IPs in DescribeLoadBalancers (mulga-928)
- fix: attach every ENI to multi-subnet ALB VMs (mulga-929)
- fix: nginx-alb example workers reach apt via NAT Gateway (mulga-930)
- fix: accept every ALB attribute key terraform sends (mulga-931)
- fix: publish NAT GW SNAT events on route table (dis)association (mulga-932)
- fix: publish NAT GW SNAT events on route table replace (mulga-933)
- refactor: extract multi-ENI NIC setup into testable helpers

E2E: https://github.com/mulgadc/spinifex/actions/runs/24318441048 (all green)
2026-04-13 09:29:47 +10:00
Josh Siviour c8ca746bf0 feat: ELBv2 attributes API + DescribeTags (#128)
* feat: add attributes map to LB and TG records with defaults

Replace CrossZoneEnabled bool with Attributes map[string]string on
LoadBalancerRecord and TargetGroupRecord. Add DefaultLoadBalancerAttributes()
and DefaultTargetGroupAttributes() functions. ALBs seed cross-zone enabled
at create time.

* feat: add 4 attributes methods to ELBv2Service interface

* feat: implement ELBv2 attributes service methods and NATS wrappers

Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
to service_impl.go with proper ARN validation and default merging.

Add corresponding NATS request wrappers in service_nats.go.

* feat: add gateway handlers for ELBv2 attributes API endpoints

Add ModifyTargetGroupAttributes, DescribeTargetGroupAttributes,
ModifyLoadBalancerAttributes, and DescribeLoadBalancerAttributes
gateway handlers with input validation and action map registration.

* feat: add daemon handlers & NATS subscriptions for elbv2 attributes

Add 4 handler functions in daemon_handlers_elbv2.go and 4 NATS
subscriptions in daemon.go for ModifyTargetGroupAttributes,
DescribeTargetGroupAttributes, ModifyLoadBalancerAttributes, and
DescribeLoadBalancerAttributes.

* fix: add missing accountID authorization check to ELBv2 attribute handlers

* fix: guard nil attributes and log skips in ELBv2 Modify*Attributes

ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes silently
dropped attributes with nil Key or Value, and would panic on a nil
slice element when dereferencing attr.Key. Add an explicit nil-element
guard and log a warning whenever an attribute is skipped, so callers
and operators can see divergence between the submitted payload and the
persisted state.

* fix: sort ELBv2 attribute responses by key for deterministic output

DescribeTargetGroupAttributes and DescribeLoadBalancerAttributes iterated
a map and appended to a slice, producing random attribute order on each
call. Sort keys before building the response so Terraform plans stay
stable and snapshot tests can rely on consistent ordering.

Closes mulga-910

* feat: add ELBv2 DescribeTags handler (mulga-919)

Required by the Terraform AWS provider's post-create refresh on
aws_lb, aws_lb_target_group, and aws_lb_listener. Without it, the
gateway returns InvalidAction and apply fails after the resources
are already created.

Dispatches by ARN resource segment (loadbalancer/targetgroup/listener)
to the existing record stores. Enforces account scoping via per-type
not-found errors to avoid cross-account existence leaks. Listeners
return an empty Tags slice since ListenerRecord doesn't store tags yet
— that's the exact case the provider hits on every untagged listener
refresh. Output is sorted by key for deterministic responses.

* refactor: parameterize DefaultLoadBalancerAttributes by lbType (mulga-915)

Cross-zone default was split across three places: CreateLoadBalancer seeded
{load_balancing.cross_zone.enabled: true} for ALBs, DefaultLoadBalancerAttributes
returned false unconditionally, and tests asserted both sides. Make
DefaultLoadBalancerAttributes(lbType) the single source of truth: ALBs get
"true", NLBs (and any other type) get "false". CreateLoadBalancer no longer
seeds attributes — DescribeLoadBalancerAttributes derives per-type defaults
from lb.Type on read. Updated TestCreateLoadBalancer_CrossZoneAttributes to
assert via the Describe handler instead of poking stored record state.

* perf: direct KV lookup for ELBv2 *ByArn finders (mulga-913)

GetLoadBalancerByArn and GetTargetGroupByArn used to call ListLoadBalancers /
ListTargetGroups and linearly scan, doing one NATS round-trip per record in
the bucket. Terraform hits Describe*Attributes on every plan/refresh — a
20-ALB / 40-TG workspace was doing ~60 calls × ~60 records = ~3600 KV gets
per refresh where ~60 would suffice. The LB/TG ARN's final path segment IS
the KV key used by GetLoadBalancer(lbID)/GetTargetGroup(tgID), so parse the
short ID from the ARN and do an O(1) lookup, verifying the record's stored
ARN matches as defence-in-depth against any stale-key weirdness.

* fix: reject ELBv2 ModifyAttributes calls where every entry is invalid (mulga-911)

When every submitted attribute was rejected by the nil-guard (nil element,
nil Key, or nil Value), Modify{TargetGroup,LoadBalancer}Attributes still
called PutTargetGroup/PutLoadBalancer with an unchanged record and returned
a successful ModifyOutput with empty Attributes. Only a slog.Warn marked
the skip — the caller saw 200 OK and moved on. Return
ErrorInvalidParameterValue when len(input.Attributes) > 0 and every entry
was rejected, so the caller learns that nothing landed.

* fix: reject unknown ELBv2 ModifyAttributes keys with ValidationError (mulga-912)

ModifyTargetGroupAttributes and ModifyLoadBalancerAttributes had no whitelist
check — any caller-supplied key (typo like 'stickness.enabled', deprecated
keys, or cross-product keys sent to the wrong resource) was persisted verbatim
and merrily echoed back by Describe. AWS rejects unknown keys with
ValidationError; match that so Terraform surfaces the typo at plan time.
Validate each key against DefaultTargetGroupAttributes() /
DefaultLoadBalancerAttributes(lb.Type) and return ErrorValidationError on the
first unknown key, logging both the arn and the offending key name.

* perf: skip no-op Put in ELBv2 ModifyAttributes (mulga-914)

The unconditional PutTargetGroup/PutLoadBalancer fired on every Modify call,
even when the caller sent an empty attribute list or when every submitted
value already matched the stored value. Terraform re-applies the same
attribute set on every drift check, so the steady-state path was doing a
pointless NATS/KV round-trip per call and unnecessarily expanding the
read-modify-write race window.

Track a dirty flag per Modify call: compare each submitted key's value
against tg.Attributes/lb.Attributes and only write to the map (and allocate
the map lazily) when it actually differs. Skip the Put entirely if nothing
changed. Added regression tests that capture the KV entry revision before
and after a no-op modify to prove the write was skipped.

* test: assert exact ModifyAttributes echoed key/value pairs (mulga-917)

Both round-trip tests used assert.Len(t, modOut.Attributes, 2) and never
checked that the returned Attributes actually contained the submitted
keys/values. A regression that pointed at the wrong source, dropped Value,
or returned a copy of defaults would have passed the length check
unchallenged. Build a modMap from the echoed response and assert each
expected Key/Value pair explicitly.

* test: cover sequential ModifyAttributes merge and same-key overwrite (mulga-916)

All existing round-trip tests did one Modify then Describe. No test covered:
Modify {A}, Modify {B}, Describe, assert BOTH keys present; or Modify
overwriting the same key on a second call. A future refactor that replaced
tg.Attributes instead of mutating in place would pass every existing test
but silently wipe previous attributes on each Modify — the most likely
real-world ALB/TG bug. Add sequential-merge tests for both TG and LB
handlers that exercise multi-key merge, same-key overwrite, and
cross-survival of unrelated keys.
2026-04-10 21:31:43 +10:00
Josh Siviour 67bdc9d4af fix: nginx-alb terraform example uses private IPs for instances (#131)
The ALB reaches backends over the private VPC network, so the Nginx
instances no longer need public IPs. Drop associate_public_ip_address
and map_public_ip_on_launch, and swap the instance public-IP outputs
for private-IP ones.

The *.elb.spinifex.local DNS name does not resolve from the operator's
host, so document an AWS CLI one-liner (describe-load-balancers with
an AvailabilityZones[].LoadBalancerAddresses[].IpAddress query) that
retrieves the ALB's public IP for curl/browser testing.
2026-04-10 18:54:12 +10:00
Josh Siviour 8d2830481d update alb terraform example (#130) 2026-04-10 16:52:52 +10:00
Josh Siviour 6238988a8f feat: add Terraform nginx-alb example (#129)
Deploys two Nginx EC2 instances behind an Application Load Balancer
on Spinifex, exercising VPC, public subnets, IGW, security group,
key pair + cloud-init, ALB, target group, and HTTP listener end-to-end.

Also gitignore generated *.pem files from tofu local_file outputs.
2026-04-10 15:56:50 +10:00
Julian Sommer 4365fb0958 feat: update guide 2026-04-10 15:05:46 +10:00
Julian Sommer 48c90b795a feat: migration framework (#127) 2026-04-10 14:34:15 +10:00
Julian Sommer 18684d5c8b fix: remove InsecureSkipVerify and normalize 0.0.0.0 connect addresses (#125) 2026-04-10 13:30:13 +10:00
Julian Sommer 708bb1d340 feat: enforce nats authentication (again) (#126) 2026-04-10 12:18:46 +10:00
Julian Sommer 893fbbc243 fix: remediate trivial security audit findings (#124) 2026-04-10 11:42:43 +10:00
Julian Sommer 7723ad4a89 fix: enable perfsprint 2026-04-10 11:36:51 +10:00
Josh Siviour ab95566795 feat: initial ELBv2 implementation (#122)
* feat: add ALB target health reporting via HAProxy stats

Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).

Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.

E2E: remove health check skip — targets must now become healthy.

* feat: allocate public IP for internet-facing ALB VMs in daemon

Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.

* fix: release auto-assigned public IPs on DeleteNetworkInterface

DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.

Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.

* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)

In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.

* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)

Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs

Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing

* feat: add custom AMI for ALB to images list

* feat: add internet-facing ALB data plane E2E test (mulga-772)

New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.

* fix: set ENDPOINT to runner node IP in ALB E2E workflow

* fix: correct paths filter to reference alb-e2e.yml

* fix: alb ami name and workflow step to import the image

* fix: spx path

* fix: remove ALB AMI import from workflow and added to bootstrap.sh

* fix: use two subnets in different AZs for internet-facing ALB E2E

Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.

* refactor: remove multi-AZ enforcement for internet-facing ALBs

Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.

* fix: use correct account ID for ALB ENI public IP assignment

The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.

* fix: add debug logging for ALB public IP assignment failure

* fix: expand debug logging to include peer node and more keywords

* fix: lazy system AMI discovery for ALB VMs

The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.

* fix: resolve ALB VM instance type dynamically from resource manager

The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.

* push interim changes

* refactor: rewrite ALB agent from NATS client to HTTP server

Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.

- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
  calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)

* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms

Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.

- service_impl: pushConfig uses http.Post, waitForAgentReady uses
  http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature

* refactor: rewrite ALB health checker from NATS subscription to HTTP polling

Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.

- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths

* refactor: remove NATS wiring from ALB image and make hostfwd dev-only

- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
  placeholder HAProxy frontend on 8405 (agent owns the port now);
  update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
  is enabled — production ALBs are reached via VPC IP directly

* feat: system credentials file + daemon wiring for ALB agent SigV4 auth

Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.

* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)

Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.

- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
  on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
  config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
  reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig

* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)

Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.

* feat: add VPC service route for ALB agent gateway access

ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.

Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.

* Revert "feat: add VPC service route for ALB agent gateway access"

This reverts commit d4220d8806.

* feat: add management NIC infrastructure for system instance control plane

Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.

- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields

* feat: wire management NIC into QEMU launch and cloud-init config

Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.

* test: add coverage for ELBv2 service lifecycle and setter methods

* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP

* fix: generate per-interface cloud-init config when eniMAC present without devMAC

The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.

* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image

- ENI DescribeNetworkInterfaces description filter did exact string match,
  but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
  "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
  module couldn't add the Spinifex CA to the system trust store. Agent
  TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
  where async TerminateSystemInstance goroutine wrote to mock fields
  while the test read them.

* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance

DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.

Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.

E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.

* fix: detach then delete ENIs in DeleteLoadBalancer

ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.

* test: add internal ALB dataplane E2E test

Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.

The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion

Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.

* feat: add NLB constants and data model changes (mulga-838)

* feat: type-aware validation in CreateLoadBalancer (mulga-839)

Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.

* feat: type-aware validation in CreateTargetGroup (mulga-840)

Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.

* feat: type-aware validation in CreateListener (mulga-841)

* feat: add NLB listener and protocol compatibility unit tests

Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.

* feat: add TCP HAProxy config generation for NLB

Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.

* refactor: rename albagent to lbagent for NLB generalization

Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.

- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest

* feat: add sys.micro system instance type for LB VMs (mulga-845)

Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.

* feat: LB image shrink and rename (mulga-846)

Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.

* chore: rename file

* feat: NLB E2E test and bug fixes (mulga-847)

Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.

Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
  prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
  back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)

* test: validate NLB internal scheme (mulga-857)

NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:

- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)

* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)

Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).

- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node

* chore: disable LB E2E tests in CI until dev env issue resolved

* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json

The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.

* feat: automate LB image build and import (mulga-861)

- dev-install.sh: build lb-agent and run build-system-image.sh --import
  after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
  remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming

* update go version

* add LB e2e tests

* fix: clean LB image build dir and wait for nbd device readiness

Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.

* fix: serialize nbd access with flock for concurrent CI builds

Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.

* fix: move flock to cover entire image build, not just nbd section

Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.

* fix: skip LB image build if output is fresh from concurrent CI job

When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.

* fix: filter AMI discovery by name in E2E tests

With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.

* fix: filter AMI discovery by name in multi-node E2E tests

Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.

* fix: use awsgw bind IP as LB agent gateway URL fallback

When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.

Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.

* fix: capture StateReason and daemon logs when LB app instances fail

Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.

* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning

Three fixes for CI E2E failures:

1. run-e2e.sh: Remove custom AMI from predastore after deleting its
   backing snapshot. Previously the AMI remained discoverable but
   unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
   on the deleted snapshot.

2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
   of picking any non-alpine image. Also improve debug log dump to use
   journalctl (binary-install CI uses systemd, not log files).

3. CreateLoadBalancer: Only set provisioning state when a system VM
   was actually launched. Previously a failed launch still set
   provisioning, leaving the LB stuck forever with no agent to
   send heartbeats.

* fix: use AWSGW bind IP as LB agent gateway URL in multi-node

In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.

Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.

* fix: add mgmt route via bootcmd for multi-node internal LBs

Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.

Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.

Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.

Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).

* add LB enable flag to e2e

* test: add elbv2 tests to meet diff coverage threshold

Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).

* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat

Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.

* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)

Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.

* fix: data race in getSystemAMI/getSystemInstanceType lazy caching

Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.

Also fix pre-existing nosprintfhostport lint in daemon.go.

* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned

publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).

isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.

* fix: narrow health checker to only scan target groups for reporting LB

handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.

* fix: pass region through to LB agent SigV4 signing (mulga-871)

LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.

* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig

Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.

* fix: upgrade silent failure paths from Debug to Warn/Error

- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding

* refactor: deduplicate MAC hash function into utils.HashMAC

Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.

* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps

Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.

* fix: fail ALB launch when public IP allocation fails for internet-facing scheme

Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.

* fix: retry system AMI resolution when image not yet imported

sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.

Same fix applied to getSystemInstanceType for consistency.

* fix: propagate updateStoredConfig errors to API callers (mulga-884)

updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.

* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)

* fix: parse HAProxy templates once at package level (mulga-875)

generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.

* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)

When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.

* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)

The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.

* refactor: use aws.StringValue in toHealthReport (mulga-878)

Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.

* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)

Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.

* fix: address PR #122 review comments (mulga-887)

- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh

* fix: check specific error type for ENI deletion on termination (mulga-888)

ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.

* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)

GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.

* fix: remove backend address from proxy error response (mulga-892)

The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.

* fix: release mutex before KV store I/O in health checker (mulga-893)

handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.

* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)

fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)

* fix: detect veth bridge mode and use centralized NAT for it

The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.

Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).

* fix: add retry loop for peer NLB reachability test

With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.

* fix: use request-reply for vpc.add-nat to prevent ARP propagation races

publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.

Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.

* fix: clean up stale NAT rules before adding new ones for reused IPs

When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.

This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.

* fix: clean up stale NAT rules across all VPC routers on IP reuse

The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.

DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.

Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.

* fix: run LB inet tests from actual hosting node, not hardcoded node1

The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.

Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
   for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
   different node

Also updates e2e.yml to pass --nodes instead of --peer.
2026-04-10 10:27:20 +10:00
tomnewton-mulga c5b6beb006 feat: multi-NIC WAN/LAN bridge network setup in ISO installer (mulga-897)
- Replace single network screen with separate WAN and LAN screens
- WAN/LAN IPs now live on Linux bridges (br-wan, br-lan) over the physical
  NICs, not on the NICs directly — makes OVS/OVN veth attachment SSH-safe
- Add DHCP/static toggle on both screens (DHCP is the default)
- Add DNS nameserver input for static mode (comma-separated, default hint
  1.1.1.1 / 8.8.8.8 shown as placeholder)
- Detect WiFi NICs via /sys/class/net/<name>/wireless; show SSID and
  password fields when a WiFi adapter is selected
- LAN screen shows WAN NIC greyed/unavailable; cursor skips over it
- LAN screen only shown when 2+ NICs are detected
- Geneve encap IP uses br-lan address when LAN NIC present, else br-wan
- Remove --macvlan/--wan-iface from firstboot; setup-ovn.sh auto-detects
  br-wan as a Linux bridge and uses veth mode
- Pin both WAN and LAN NIC names to MAC via a single udev rules file
- node.conf writes MANAGEMENT_IFACE=br-wan (bridge, not physical NIC)
2026-04-10 10:02:16 +10:00
tomnewton-mulga e8e05d6898 chore: merge main into feat/iso-installer, resolve Makefile .PHONY conflict 2026-04-10 09:40:16 +10:00
Julian Sommer b001c3929d Revert "feat: enforce nats authentication (#123)"
This reverts commit 553e7857df.
2026-04-09 16:06:03 +10:00
tomnewton-mulga 03601ca89b fix: banner service ordering, getty wait, motd append (mulga-866)
- banner runs After=spinifex.target so services are settled before banner writes
- switch systemctl restart → try-restart (safe if target not yet active)
- append banner to /etc/motd with sentinel instead of overwriting, preserving
  the Debian disclaimer; sentinel ensures re-runs replace cleanly
- remove /etc/issue write — getty shows it pre-login which is wrong
- add getty@.service.d drop-in so all getty instances wait for banner before
  showing login prompt; prevents service messages scrolling during login
- fix Makefile to prefer locally built spx/installer binaries over system PATH
2026-04-09 15:40:26 +10:00
Julian Sommer 1a2e490ae7 fix: policy matching case insensitive 2026-04-09 15:34:43 +10:00
Julian Sommer 553e7857df feat: enforce nats authentication (#123) 2026-04-09 15:27:00 +10:00
tomnewton-mulga cbb3e91ada feat: add spx admin banner --boot-check, replace bash banner script (mulga-866)
- add `spx admin banner --boot-check` subcommand to spx CLI
  - reads management IP from the NIC at runtime (not from node.conf)
  - if IP has changed, updates /etc/spinifex/node.conf and restarts spinifex.target
  - writes banner to /etc/issue (pre-login console) and /etc/motd (SSH MOTD)
  - banner instructs user to cat ~/.aws/credentials (standard AWS CLI location)
- update firstboot.go to use spx-based banner unit; remove writeBannerScript
  and old spinifex-banner.sh entirely
- new banner unit runs Before=spinifex.target so IP is current before services start
2026-04-09 14:39:56 +10:00
Julian Sommer 7449090c86 fix: enable more linters (#121) 2026-04-09 14:36:22 +10:00
tomnewton-mulga 10a2a8116f fix: resolve firstboot service failures and improve installer UX (mulga-859)
- chown /etc/spinifex and /var/lib/spinifex to spinifex:spinifex after
  spx admin init so services running as the spinifex user can read configs
- add Environment=HOME=/root to firstboot unit to suppress $HOME warning
- increase wait-for-nats.sh timeout from 30s to 90s for slow startup
- banner now writes to both /dev/tty1 and /etc/motd (visible over SSH)
- add systemd.show_status=1 to GRUB cmdline for per-service boot feedback
- add --removable to grub-install EFI so fallback path EFI/BOOT/BOOTX64.EFI
  is written (required for UTM and bare metal without NVRAM boot entries)
- add root password screen and manual NIC entry to installer TUI
- tee spx admin init output to /root/spinifex-credentials.txt with 0600 perms
2026-04-09 13:52:31 +10:00
Julian Sommer 26e590716e feat: add systemd hardening directives to all service units (#120) 2026-04-09 12:09:43 +10:00
Julian Sommer e27d909e8d feat: move NBD sockets to /run/spinifex/nbd/ and enable PrivateTmp (#119) 2026-04-09 11:11:54 +10:00
Julian Sommer ccb6ed287d fix: move bootstrap.json to different folder to tighten perms (#118) 2026-04-08 16:12:25 +10:00
Julian Sommer 9a7e914d28 feat: awsgw directory (#117) 2026-04-08 15:19:29 +10:00
Julian Sommer 6026d46ba1 fix: ovn logging 2026-04-08 13:00:06 +10:00
Julian Sommer f0eb28ffc2 fix: remove gateway csp 2026-04-08 12:45:36 +10:00
Julian Sommer d33e6f3d70 chore: go 1.26.2 2026-04-08 11:53:29 +10:00
Julian Sommer 45ea467fcc feat: webserver proxy (#116) 2026-04-08 11:46:43 +10:00
tomnewton-mulga 7c007ea6b5 feat: replace huh TUI with bubbletea + add branding package (mulga-859)
Replaces charmbracelet/huh with a custom bubbletea wizard for the
installer TUI. This gives full control over terminal I/O so the
installer attaches directly to the correct console device (ttyS0 for
serial, tty1 for VGA) as directed by the new spinifex-init boot script.

- branding/branding.go: centralised logo (ASCII placeholder), color
  constants, and publisher string ready for Mulga asset swap
- ui/ui.go: multi-screen bubbletea model (welcome → disk → network →
  identity → join → CA cert → confirm) styled with lipgloss
- main.go: reads SPINIFEX_CONSOLE from /proc/cmdline to open the right
  tty before launching tea.Program
- go.mod: bubbletea/lipgloss/bubbles promoted to direct deps; huh removed
2026-04-08 10:49:57 +10:00
Julian Sommer 820b605c0b fix: require sudo admin init 2026-04-08 09:32:25 +10:00
Julian Sommer a56a2baf1f feat: privilege separation (#114) 2026-04-07 16:22:19 +10:00
tomnewton-mulga 6dfb350be8 fix: use INSTALL_SPINIFEX_SKIP_APT/AWS env vars instead of non-existent --offline flag (mulga-859) 2026-04-07 16:01:17 +10:00
tomnewton-mulga 5d6f25a695 feat: add build-installer Makefile target (mulga-859) 2026-04-07 15:56:39 +10:00
tomnewton-mulga 1b39439cb8 feat: scaffold ISO installer TUI and install pipeline (mulga-859)
Adds cmd/installer with three packages:
- ui: huh-based TUI collecting disk, network, node, and cluster config
- install: sequential install steps (partition, debootstrap, chroot, bootloader)
- firstboot: oneshot systemd service that runs setup-ovn and spx admin init/join on first real boot

Supports both single-node init and multi-node join flows.
2026-04-07 15:50:40 +10:00
Julian Sommer 99f9949bdd fix: enable TLS on cluster manager API (#113) 2026-04-07 10:56:16 +10:00
dependabot[bot] 5598e523d5 chore: bump the npm-dependencies group (#112) 2026-04-07 08:57:04 +10:00
Julian Sommer dda4575888 fix: use binary install for single node e2e (#111) 2026-04-02 16:22:41 +11:00
Ben Duncan e5b6204875 Fix service helper sequence 2026-04-02 13:53:29 +10:00
Ben Duncan 21aa1296c9 systemd tweaks for binary install to correctly function 2026-04-02 13:35:52 +10:00
Julian Sommer ea6b5d770a fix: mask credentials in CLI override log output
Access key and secret key values were printed to stdout when using
--access-key and --secret-key flags. These are captured by systemd
journal and log aggregation, exposing secrets. Log that keys were
overridden without printing the actual values.

Addresses H-5 from security audit.
2026-04-02 12:25:27 +10:00
Julian Sommer f986cf9748 fix: ui cert path 2026-04-02 11:55:32 +10:00
Julian Sommer 11f7c99955 fix: ui cert path 2026-04-02 11:20:02 +10:00
Julian Sommer 1b45c7fc7d fix: nats ip in services 2026-04-02 10:58:39 +10:00
Julian Sommer 868a92bd55 fix: service 2026-04-02 10:35:00 +10:00
Julian Sommer af52d881b9 fix: chown 2026-04-02 10:26:56 +10:00
Julian Sommer c51ea0c29d fix: make download checksum verification mandatory 2026-04-02 09:56:57 +10:00
Julian Sommer 8cf0cc332f feat: remove pw login 2026-04-02 09:51:07 +10:00
Julian Sommer 1d6cefd9d2 feat: extract helpers (#109) 2026-04-02 10:25:01 +11:00
Julian Sommer 0c160d7f45 fix: add request body size limit 2026-04-01 15:36:06 +10:00
Julian Sommer 91e4efb0ef feat: extract tag helper 2026-04-01 15:27:43 +10:00
Julian Sommer f5ec8c3bae feat: formation server HTTPS (#108) 2026-04-01 16:20:28 +11:00
Julian Sommer 8795a7b926 fix: remove dev branch 2026-04-01 13:54:35 +10:00
Julian Sommer 39e9fa6fb8 feat: more filters (#106) 2026-04-01 14:23:04 +11:00
Julian Sommer 853b5bfd32 feat: filterutil migration (#105) 2026-04-01 14:05:21 +11:00
Julian Sommer d7c72e80af feat: filters (#104) 2026-04-01 12:20:56 +11:00
Julian Sommer 475e632404 chore: update COMMANDS.md 2026-03-31 15:53:02 +10:00
Julian Sommer 92e25a9653 chore: bump predastore 2026-03-31 14:11:47 +10:00
Julian Sommer ec525d3a6d add back -fsSL 2026-03-31 13:59:58 +10:00
Julian Sommer 94978f517b actual last 2026-03-31 13:55:48 +10:00
Julian Sommer 4d7239e645 last doc tweaks 2026-03-31 13:50:18 +10:00
Ben Duncan acc8dc1994 Remove macvlan 2026-03-31 13:36:42 +10:00
Julian Sommer a7ed93c6f3 docs: add prerequisite 2026-03-31 13:28:12 +10:00
Julian Sommer 67075c6600 docs: better urls 2026-03-31 13:25:23 +10:00
Ben Duncan ae223cdf15 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-31 13:19:48 +10:00
Ben Duncan b73a630f41 TF updates 2026-03-31 13:19:17 +10:00
Julian Sommer c283100fcf docs: rename source-install to source-installation 2026-03-31 13:16:42 +10:00
Julian Sommer 355decd406 docs: tweaks 2026-03-31 13:10:59 +10:00
Julian Sommer b76953bfb4 doc tweaks 2026-03-31 13:02:21 +10:00
Julian Sommer c9e83380df a 2026-03-31 11:59:07 +10:00
Julian Sommer e421840541 feat: placment group docs 2026-03-31 11:42:29 +10:00
Julian Sommer 406ba9431e fix 2026-03-31 11:39:10 +10:00
Julian Sommer 7882f77a36 add more docs 2026-03-31 11:36:09 +10:00
Julian Sommer f7b99f0c50 fix: COMMANDS.md 2026-03-31 11:29:28 +10:00
Julian Sommer f3ee50e9b8 fix: show deleted nat gateways 2026-03-31 10:49:36 +10:00
Julian Sommer 4f3b54abc2 fix: ModifyInstanceAttribute SourceDestCheck 2026-03-31 10:30:14 +10:00
Julian Sommer a7925d65f2 feat: DescribeAddressesAttribute (#103) 2026-03-31 11:14:27 +11:00
Julian Sommer 0d2e91da37 fix: downgrade node types to 24.12.0 2026-03-31 08:16:45 +10:00
Julian Sommer e2c7a64aff fix: set ui to use dark mode by default 2026-03-31 08:10:59 +10:00
dependabot[bot] 588141cf67 chore: bump the npm-dependencies group across 1 directory with 18 updates (#101) 2026-03-31 09:05:09 +11:00
dependabot[bot] 9e3076727e chore: bump github.com/nats-io/nats.go in the go-dependencies group (#99) 2026-03-31 09:03:19 +11:00
Ben Duncan c1df418c5c fix: parse NODE column correctly from pipe-delimited spx get vms output 2026-03-30 22:16:23 +10:00
Ben Duncan 5e0d819393 test: add frontend unit tests to meet 70% line coverage threshold
Add tests for: back-link, not-found, list-card, header,
delete-confirmation-dialog, use-initial-sidebar hook, admin queries,
and admin-dashboard-panel. Line coverage: 63.98% → 71.12%.
2026-03-30 22:09:14 +10:00
Ben Duncan 6d3b2f7e5d feat: add multi-node NAT GW + spread placement group E2E test
Phase 11 in run-multinode-e2e.sh:
- Create VPC with public + private subnets + IGW
- Launch bastion in public subnet, verify SSH
- Create spread placement group, launch 3 VMs in private subnet
- Validate spread distribution via spx get vms (3 nodes)
- Verify no internet pre-NAT (ping 8.8.8.8 fails from all 3)
- Create NAT GW + EIP + route, poll for OVN SNAT rule
- Verify internet via NAT GW from all 3 nodes (different physical hosts)
- Full cleanup: NAT GW, instances, placement group, VPC resources
2026-03-30 21:54:22 +10:00
Ben Duncan f3c5bce811 feat: admin dashboard, nodes page, S3 service metrics, sidebar branding
Admin Dashboard Panel:
- spinifex service namespace on AWS gateway (GetCallerIdentity,
  GetVersion, GetNodes, GetVMs, GetStorageStatus)
- Admin detection via SigV4-signed GetCallerIdentity
- Dashboard panel: version, license, cluster status, add-ons matrix
- Sidebar branding: Mulga logo + "Spinifex", SVG favicon
- Upgrade button (admin + open-source)
- Documentation link (admin only)

Nodes Page (/nodes, admin only):
- Cluster nodes with services, roles, region, uptime
- Elastic Compute Cloud (EC2) instances with age
- Resource usage per node + instance type capacity table

S3 Service Metrics (/s3/service-metrics, admin only):
- Daemon NATS handler parses predastore TOML (creds stripped)
- Gateway queries DB nodes /status + /health in parallel
- Predastore (S3) Overview: RS encoding, node counts, cluster health
- DB Consensus Nodes: Raft state, leader, term, commit/applied index
- Object Storage Shard Nodes: topology from config

Frontend infrastructure:
- signedFetch utility (SigV4 via @smithy/signature-v4)
- AdminContext + useAdmin hook
- React Query options for admin endpoints
- Full-width sidebar separator above sign out
2026-03-30 17:21:45 +10:00
Ben Duncan 9b8863a182 Revert "feat: add install call-home telemetry to admin init and join"
This reverts commit 6e807f88cf.
2026-03-30 16:35:47 +10:00
Ben Duncan 6e807f88cf feat: add install call-home telemetry to admin init and join
Sends anonymous telemetry to install.mulgadc.com/install during
spx admin init and join. Reports machine_id (/etc/machine-id),
region, az, node count, version, and arch. Fire-and-forget with
5-second timeout — never blocks the user.

Opt out with --no-telemetry flag or SPX_NO_TELEMETRY=1 env var.
2026-03-30 16:34:45 +10:00
Ben Duncan 32f5ec92c0 fix: exclude 127.0.0.1 from SERVICE_IPS when awsgw binds to specific IP 2026-03-30 11:14:49 +10:00
Ben Duncan 6a4b823de7 fix: only test TLS connections on awsgw bind IP, not all interfaces
In single-node mode, cert E2E discovered all host IPs (WAN, mgmt, VPC)
and tried openssl s_client on each. But awsgw only binds to the
configured host IP, not all interfaces. Management (10.11.x) and VPC
(10.x) IPs have no listener on port 9999, causing false failures.

Split NODE_IPS (all IPs for SAN validation) from SERVICE_IPS (IPs
where awsgw listens) parsed from spinifex.toml. Connection-based
tests (3, 4c, 5, 8) now use SERVICE_IPS.
2026-03-30 10:51:29 +10:00
Ben Duncan 91483d632e fix: replace fixed sleeps with retry loops in NAT GW E2E tests
The NAT GW E2E tests used 2-3 second sleeps before verifying SNAT
rule creation, outbound connectivity, and rule removal. This caused
flaky CI failures when OVN took longer to propagate flows to the
datapath (varies by host load and OVN version).

Replace with retry loops:
- SNAT rule check: poll OVN NB every 1s for up to 30s
- Outbound ping: retry up to 10 attempts with 5s backoff (~50s max)
- SNAT removal: poll up to 15 attempts with 2s backoff (~30s max)

Also dump OVN routes alongside NAT rules on failure for debugging.
2026-03-30 10:27:22 +10:00
Ben Duncan 5426dc1bb0 feat: improved certificate authority with auto-discovery, CA download, and cloud-init injection
Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.

Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.

Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
2026-03-28 14:29:07 +10:00
Julian Sommer 15f88caee7 fix: latest frontend build 2026-03-27 15:57:23 +10:00
Julian Sommer 5d194098ca feat: security groups frontend (#97) 2026-03-27 16:56:00 +11:00
Julian Sommer 94a80a3be7 fix: lint issues 2026-03-27 15:13:55 +10:00
Julian Sommer 3f642524a8 feat: vpc create wizard (#96) 2026-03-27 16:07:20 +11:00
Julian Sommer de1340a331 fix: let reset-dev-env.sh auto-detect chassis ID from hostname
Previously hardcoded CHASSIS_ID="chassis-node1" which could mismatch
the OVS system-id from prior setup-ovn.sh runs, creating stale chassis
entries in the SBDB. Now defers to setup-ovn.sh's auto-detect
(chassis-$(hostname -s)) so the name is always consistent.
2026-03-27 12:44:01 +10:00
Julian Sommer ae094cb936 fix: set OutputPort on default route and filter stale chassis
Two stacked bugs prevented SSH to VPC instances:

1. The default route (0.0.0.0/0) was added without OutputPort. For
   DHCP-sourced pools the gateway router port uses link-local addressing
   (169.254.0.1/30) so OVN northd cannot resolve the nexthop (e.g.
   192.168.1.1) to an output port — it silently drops the route from
   the southbound DB.

2. discoverChassis returned all chassis from the SBDB including stale
   entries from previous OVN setups on the same host. When a stale
   chassis got highest gateway priority, the chassisredirect port
   stayed unbound (up: false), so no ARP/NAT traffic was processed.

Fix: add OutputPort to static routes in both handleIGWAttach and
reconcileIGW, and filter stale chassis by cross-referencing the local
OVS system-id with SBDB hostnames.
2026-03-27 12:40:52 +10:00
Julian Sommer c2ab63fdef feat: basic terraform templates 2026-03-27 12:09:49 +10:00
Julian Sommer 04bea48a39 fix: dhcp default behaviour 2026-03-27 11:53:48 +10:00
Julian Sommer a92ea13798 fixes 2026-03-27 11:30:39 +10:00
Julian Sommer 46670ee8af test 2026-03-27 10:54:02 +10:00
Julian Sommer cbc04a2437 fix: chassis names mismatch 2026-03-27 10:26:16 +10:00
Julian Sommer 216842cc1b fix: enable ovn on boot 2026-03-27 09:41:23 +10:00
Julian Sommer 8ea40a1973 fix: test coverage 2026-03-27 09:04:21 +10:00
Julian Sommer 5bfc680a68 fix: elbv2 error handling 2026-03-27 08:45:32 +10:00
Julian Sommer 98d5ec92ed fix: add pre commit hook 2026-03-27 08:28:16 +10:00
Julian Sommer 2244ba56b0 fix: race condition test 2026-03-27 08:09:26 +10:00
Ben Duncan 9e8e944faa fix: pass private IP in vpc.delete-nat event to clean up OVN rules (mulga-764)
Root cause: when terminating instances, the vpc.delete-nat event was
published with an empty logicalIP. vpcd's DeleteNAT looked for a
dnat_and_snat rule with logicalIP="" — found nothing — and the real
rule was left as a stale entry in OVN.

When the IPAM later recycled that public IP for an EIP/NAT Gateway,
OVN had conflicting rules: stale dnat_and_snat AND new snat, both
using the same external IP.

Fix: extract PrivateIpAddress from instance.Instance before publishing
vpc.delete-nat, in both running and stopped termination paths.

Also adds 14 unit tests for NAT Gateway service.
2026-03-27 07:43:53 +10:00
joshsiv-mulga 3129c9eb70 feat: generic system image builder + lint fixes (mulga-766)
Extract build-alb-image.sh into manifest-driven build-system-image.sh:
- alb.conf manifest + alb-setup.sh for ALB-specific config
- build-alb-image.sh kept as thin wrapper for backward compat
- Makefile targets: build-system-image, build-alb-image
- Hardened error handling: curl --fail, e2fsck/resize2fs checks,
  static-link enforcement, rc-update validation, nbd device guard

Also fixes 12 lint issues: errcheck, errchkjson, gosec, gofmt, modernize.
2026-03-26 22:30:19 +10:00
joshsiv-mulga f744aedede feat: ALB/ELBv2 service 2026-03-26 21:38:14 +10:00
Ben Duncan e5801aa6ba fix: NAT Gateway E2E order + CA cert in reset-dev-env (mulga-763)
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.

Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.

reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
2026-03-26 16:23:38 +10:00
Julian Sommer 77a4a40ca6 fix: tf 2026-03-26 15:46:09 +10:00
Julian Sommer ab84905297 fix: test coverage (#93) 2026-03-26 16:30:44 +11:00
Ben Duncan 2a3ada5e93 fix: NAT Gateway E2E — poll for private instance SSH readiness (150s)
The bastion SSH hop failed because the private instance's cloud-init
hadn't completed (SSH daemon not ready). The instance was "running"
from the hypervisor perspective but cloud-init takes ~30-40s.

- Replace 2-attempt retry with 30-attempt poll (5s intervals, 150s max)
- Hard failure with OVN debug output instead of silent skip
- Matches how wait_for_ssh works for public instances
2026-03-26 15:29:48 +10:00
Ben Duncan adfd20dee5 test: NAT Gateway E2E — bastion SSH + ping validation (mulga-763)
Adds Phase 8d to run-e2e.sh with 5-step NAT Gateway validation:

1. Baseline: bastion → private VM → ping 8.8.8.8 (FAIL, no NAT GW)
2. Create NAT GW + EIP + route table + route + associate with private subnet
3. With NAT GW: bastion → private VM → ping 8.8.8.8 (PASS, SNAT active)
4. Delete NAT GW + route + cleanup
5. After deletion: bastion → private VM → ping 8.8.8.8 (FAIL, confirms NAT GW)

Reuses public + private instances from Phase 8b (deferred cleanup).
Bastion SSH approach: CI → public VM (pub IP) → private VM (VPC priv IP).
OVN SNAT rule verification when ovn-nbctl is available.
2026-03-26 14:43:01 +10:00
Ben Duncan 4ffc7785a5 feat: NAT Gateway — CRUD + vpcd SNAT rules + route table integration (mulga-750)
Implements AWS-compatible NAT Gateways for private subnet outbound internet:

API (3 actions):
- CreateNatGateway: validates subnet exists, EIP allocated, stores in KV
- DeleteNatGateway: removes record, publishes delete events for SNAT cleanup
- DescribeNatGateways: filter by ID, vpc-id, state

vpcd SNAT rule management:
- handleAddNATGateway: creates OVN snat rule (subnet CIDR → NAT GW EIP)
- handleDeleteNATGateway: removes OVN snat rule
- Subscribes to vpc.add-nat-gateway / vpc.delete-nat-gateway topics

Route table integration:
- CreateRoute now accepts NatGatewayId target (in addition to GatewayId)
- Validates NAT GW exists and belongs to same VPC
- Publishes vpc.add-nat-gateway events for associated subnets

OVN mechanics:
- Single snat rule per private subnet CIDR (shared by all VMs)
- No policy-based routing needed — SNAT rewrites before route lookup
- Existing 0.0.0.0/0 default route handles all egress
- Connection tracking handles return traffic automatically
2026-03-26 14:43:01 +10:00
Julian Sommer e91b67a7c6 feat: ModifyNetworkInterfaceAttribute 2026-03-26 14:08:28 +10:00
Julian Sommer 051ecbd4ce fix test 2026-03-26 13:54:29 +10:00
Julian Sommer ef68c5bffe feat: DescribeInstanceCreditSpecifications 2026-03-26 13:46:46 +10:00
Julian Sommer 64c7305b37 feat: DescribeInstanceAttribute (#92) 2026-03-26 14:39:45 +11:00
Ben Duncan d025cab50e fix: harden route table error handling (silent-failure-hunter review)
- getRouteTable: distinguish ErrKeyNotFound (→ not found) from other
  errors (→ log + internal error) instead of mapping all to "not found"
- putRouteTable: log underlying error on marshal/write failures
- KV scan loops (allRouteTablesForVPC, DescribeRouteTables,
  DisassociateRouteTable, ReplaceRouteTableAssociation): fail on
  transient NATS errors instead of silently skipping entries. Only
  skip ErrKeyNotFound (deleted between Keys() and Get())
- ReplaceRouteTableAssociation: compensating write if second KV put
  fails — restores association to old table to prevent data loss
- createMainRouteTable: returns error instead of swallowing it
- ensureDefaultIGWRoute: log actual errors at Warn level, not Debug
- E2E test: fail on missing default VPC instead of skipping
2026-03-26 13:33:22 +10:00
Julian Sommer 1c194da915 fix: change "attached" to "available" in igw to match aws + add describe instance types filtering for TF 2026-03-26 12:34:32 +10:00
Julian Sommer 4ad5843ff5 feat: ModifyVpcAttribute + DescribeVpcAttribute (#91) 2026-03-26 13:28:09 +11:00
Ben Duncan 7ea0353d2f test: E2E route table validation in single-node test (mulga-761)
Adds Phase 8c to run-e2e.sh with three steps:

1. Verify default VPC main route table has local + IGW routes
2. Custom route table full lifecycle (create → add route → associate
   → disassociate → delete route → delete table)
3. Error paths: reject local route deletion, reject main table deletion
2026-03-26 12:26:52 +10:00
Ben Duncan f8fc2b2cbb feat: route tables V1 — CRUD, associations, VPC integration (mulga-760)
Implements AWS-compatible route tables for VPC subnets:

Route Table CRUD:
- CreateRouteTable, DeleteRouteTable, DescribeRouteTables
- Main route table auto-created with every VPC (CreateVpc + EnsureDefaultVPC)
- Cannot delete main table or tables with subnet associations

Route CRUD:
- CreateRoute (IGW targets), DeleteRoute, ReplaceRoute
- Immutable local route (VPC CIDR → local) cannot be deleted
- IGW ownership validation (must be attached to same VPC)
- Duplicate destination CIDR rejected

Association CRUD:
- AssociateRouteTable, DisassociateRouteTable, ReplaceRouteTableAssociation
- One explicit association per subnet
- Disassociating reverts subnet to main table

VPC Integration:
- CreateVpc auto-creates main route table with local route
- EnsureDefaultVPC creates main route table for default VPC
- ensureDefaultVPCInfrastructure adds 0.0.0.0/0 → IGW route

DescribeRouteTables filters: vpc-id, route-table-id, association.main,
association.subnet-id, route.destination-cidr-block, route.gateway-id

20 unit tests covering all CRUD operations and error paths.
2026-03-26 12:26:52 +10:00
Julian Sommer 74671a05a5 aws basic 2026-03-26 12:00:38 +10:00
Julian Sommer 90a3da2cc2 fix: daemon nil pointer 2026-03-26 11:59:44 +10:00
Julian Sommer 06a8da8fb5 fix: dependabot 2026-03-26 11:05:04 +10:00
Julian Sommer 7c6f03f0d0 chore: update action versions 2026-03-26 10:45:02 +10:00
Julian Sommer e6b48ce170 fix: docs 2026-03-26 10:29:48 +10:00
Julian Sommer fca9d018fc fix: install guide 2026-03-26 10:02:04 +10:00
Julian Sommer 4a2ae33dd9 fix: add missing assets 2026-03-26 09:56:38 +10:00
Julian Sommer 9e53443f7a fix: decrease shutdown wait time 2026-03-26 08:55:55 +10:00
Julian Sommer bf6b25842f feat: add dependabot auto updates 2026-03-26 08:36:50 +10:00
Ben Duncan 5b442e8eef chore: increase pseudo multi-node E2E timeout to 35m
External networking adds overhead (DHCP, tap setup/teardown) and the
shutdown drain phase needs more time with tap networking cleanup.
2026-03-25 20:54:46 +10:00
Ben Duncan a3c4819c1c fix: enable external networking in pseudo multi-node E2E
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.

- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
2026-03-25 20:26:52 +10:00
Ben Duncan 70199f410a fix: pseudo multi-node E2E uses public IP + nbdkit for node detection
After enabling tap networking (public subnet), QEMU no longer uses
hostfwd bindings. The pseudo multi-node test was still grepping for
hostfwd to detect instance distribution and SSH endpoints.

- check_instance_distribution: count nbdkit root-volume processes per
  node IP instead of hostfwd-bound QEMU processes
- get_instance_node: match QEMU nbd socket → nbdkit host= IP to
  determine which simulated node hosts each instance
- count_instances_per_node: use get_instance_node instead of hostfwd
- SSH connectivity: check PublicIpAddress first (port 22), fall back
  to hostfwd — same pattern as single-node and multi-node E2E
2026-03-25 19:53:00 +10:00
Ben Duncan 7cdf630cf7 feat: propagate network config via formation server to joining nodes (mulga-759)
admin init builds a NetworkConfig from external networking flags and
passes it to the formation server. When joining nodes poll /formation/status,
they receive the cluster-wide [network] section (external mode, pools,
gateway, DNS, bootstrap VPC IDs). Each joining node auto-detects its
own external_interface and wan_bridge locally.

Fixes multi-node clusters having no [network] section in spinifex.toml
on secondary nodes, which caused no public IP assignment and broke
guest SSH in E2E tests.
2026-03-25 17:06:21 +10:00
Ben Duncan fb8c42e934 fix: poll for NAT rule cleanup instead of fixed sleep + hard fail
The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
2026-03-25 16:35:15 +10:00
Ben Duncan cc46cf746e fix: multi-node E2E SSH via public IP when external networking enabled
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
2026-03-25 16:05:54 +10:00
Ben Duncan e65f00c86a fix: set TenantID on root volume created by RunInstances (mulga-758)
GenerateVolumes did not set TenantID on the VolumeMetadata for root
volumes cloned from AMIs. After the pre-Phase4 tenant fallback was
removed (ae316db), describe-volumes filters volumes by exact tenantID
match — root volumes with empty TenantID became invisible to the
owning account, breaking E2E Phase 5b volume lifecycle tests.
2026-03-25 16:01:18 +10:00
Ben Duncan 85f51b8084 Merge branch 'vpc-public-subnet' into dev 2026-03-25 15:43:08 +10:00
Ben Duncan 4749a37725 fix: SSH via public IP instead of QEMU hostfwd when external networking enabled
When external_mode is set, dev_networking=false (no QEMU hostfwd NIC).
E2E tests now detect the instance's public IP from describe-instances
and SSH via port 22 on the public IP. Falls back to hostfwd only when
no public IP is available (overlay-only mode).

Applies to Phase 5a-ii, 7b, 7c-pre, and 8b SSH tests.
2026-03-25 15:34:33 +10:00
Ben Duncan 982a7b6b45 feat: SSH via public IP in E2E public subnet test
Verify full networking path: host → public IP → OVN DNAT → VM.
Test outbound internet from VM via OVN SNAT (ping 8.8.8.8).
Falls back to API-only verification if public IP unreachable.
2026-03-25 15:33:21 +10:00
Julian Sommer b81fdb5bc0 chore: frontend deps 2026-03-25 15:26:50 +10:00
Julian Sommer 71e82b22a7 chore: bump deps 2026-03-25 15:24:21 +10:00
Ben Duncan 21c4754336 feat: E2E tests for public/private subnet isolation + remove legacy nat mode
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
  and dnat_and_snat rule; private subnet instance gets neither; no blanket
  VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
  source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
  E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
  static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
  DHCP pool config validation to meet 60% diff coverage threshold
2026-03-25 15:22:18 +10:00
Julian Sommer ec29cc2745 Revert "fix: remove InsecureSkipVerify (#87)"
This reverts commit a66b1a3d48.
2026-03-25 15:19:48 +10:00
Julian Sommer 5d31f05dba refactor: migrate ObjectStore from ListObjects V1 to V2
Replace all ListObjects (V1) usage with ListObjectsV2 across the
objectstore interface, both implementations, all 10 call sites, and
tests. V2 uses ContinuationToken instead of Marker for pagination
and adds KeyCount to responses. The frontend and Predastore already
speak V2 — this aligns the Go backend.
2026-03-25 14:44:13 +10:00
Julian Sommer 73e5905092 chore: bump viperblock version 2026-03-25 14:17:15 +10:00
Ben Duncan edcc53b947 Merge commit '820f620e34006699de1105859301146aaf491ca5' into vpc-public-subnet
# Conflicts:
#	cmd/spinifex/cmd/admin.go
#	docs/installing-spinifex/README.md
#	scripts/setup.sh
#	spinifex/admin/admin.go
#	spinifex/admin/network_detect.go
#	spinifex/handlers/iam/service_impl.go
2026-03-25 14:16:47 +10:00
Julian Sommer 820f620e34 feat: aws tf template 2026-03-25 13:54:24 +10:00
Ben Duncan 3d73dd180b fix: add dhcpcd-base to system dependencies for DHCP pool mode
DHCP pool mode requires dhcpcd-base (/usr/sbin/dhcpcd) to obtain
leases from the router. Added to Makefile install-system, setup.sh
binary installer, and source-install docs.

Closes: mulga-753
2026-03-25 12:05:21 +10:00
Ben Duncan 7bfb032f90 fix: remove blanket VPC CIDR SNAT from IGW — private subnet isolation (AWS parity)
handleIGWAttach and reconcileIGW created a blanket snat rule for the
entire VPC CIDR, allowing private subnet instances (no public IP) to
route outbound through the IGW. AWS behavior: only instances with
public IPs route through IGW via per-VM dnat_and_snat rules. Private
subnets require a NAT Gateway for outbound internet.

Also fixes pre-existing lint issues in external_ipam_dhcp.go (gofmt,
modernize, nolintlint).

Closes: mulga-754
2026-03-25 12:05:15 +10:00
Julian Sommer 15d6b0aa20 fix: multi node e2e decrease ssh timeout 2026-03-25 11:53:52 +10:00
Julian Sommer a66b1a3d48 fix: remove InsecureSkipVerify (#87) 2026-03-25 12:51:34 +11:00
Ben Duncan cbbd81bc76 feat: DHCP pool mode — public IPs from router DHCP instead of static range
Adds source="dhcp" option to external pool config. When set, ExternalIPAM
obtains public IPs from the upstream router's DHCP server (via dhcpcd on
br-ext) instead of allocating from a static range_start..range_end.

The VM behavior is identical to static pool mode: private VPC IP from OVN
DHCP, public IP via DNAT+SNAT, full DescribeInstances support. The only
difference is where the public IP comes from.

Two separate DHCP conversations:
- Host ↔ Router DHCP (br-ext): "give me a public IP for this VM's NAT"
- VM ↔ OVN DHCP (br-int): "give me my private VPC IP"

Tested: DHCP pool mode on single-node (ben-banksia). VM got public IP
192.168.1.77 from router DHCP, SSH works, outbound internet works.
2026-03-25 10:53:29 +10:00
Julian Sommer 8c4adf08a3 chore: bump predastore to v1.1.2 2026-03-25 10:52:08 +10:00
Julian Sommer c0b89e6d8c fix: path 2026-03-25 10:35:30 +10:00
Ben Duncan dd86805514 docs: add DHCP pool mode to vpc-networking guide
Explains static vs DHCP IP sources in pool mode, the two-DHCP-conversations
architecture, when to use each, and adds a DHCP homelab deployment example.
Also updates bridge setup section for veth pair approach.
2026-03-25 10:30:42 +10:00
Julian Sommer ad1339505d fix: dump vpcd log 2026-03-25 10:09:39 +10:00
Julian Sommer ba7e6a5c67 fix: redirect image import to dev null 2026-03-25 10:04:16 +10:00
Ben Duncan 4d56565c3f feat: veth pair bridge — link Linux bridge to OVS without conversion
setup-ovn.sh now detects whether the default route is on a Linux bridge
or OVS bridge. When it's a Linux bridge (e.g. br-wan from cloud-init),
instead of destructively converting it, the script creates an OVS bridge
(br-ext) and links them with a veth pair. The Linux bridge keeps its IP,
routes, and ports untouched — no WAN interruption.

Also fixes all pre-existing lint issues (gofmt, ineffassign, errchkjson,
gosec, modernize, whitespace) to bring golangci-lint to 0 issues.

Tested: pool mode with static IP range on single-node (ben-banksia).
VM gets public IP 192.168.1.201, SSH works, outbound internet works.
2026-03-25 10:01:37 +10:00
Julian Sommer ae316dba47 fix: remove pre-Phase4 volume tenant fallback
Simplify all 6 tenant-check sites in volume handlers from
`if accountID != "" && tenantID != "" && tenantID != accountID`
to `if tenantID != accountID`. No pre-Phase4 volumes with empty
TenantID exist in any environment — this code was unreachable.
Remove vol-legacy test fixture and empty-accountID-sees-all assertion.
2026-03-25 09:59:42 +10:00
Julian Sommer bdf0522895 feat: admin init auto install cert 2026-03-25 09:06:50 +10:00
dependabot[bot] e347f0a75e chore: bump github.com/nats-io/nats-server/v2 (#86) 2026-03-25 08:14:15 +10:00
Ben Duncan c2992ebe5e fix: reset-dev-env.sh — pass ADMIN_INIT_ARGS to second admin init --force
The second admin init --force was called without external networking
flags, overwriting the pool/nat config from the first call with
auto-detected NAT+DHCP defaults.
2026-03-24 17:09:48 +10:00
Ben Duncan ffca7330c5 fix: skip external IPAM init in NAT mode — only pool mode needs per-VM allocation
In NAT mode, all VMs share a single gateway IP for SNAT. No per-VM
public IPs are allocated. Previously, external IPAM was initialized
for any external_mode != "", causing noisy AllocateIP failures (empty
pool range) on every instance launch.

Now IPAM is only initialized when external_mode == "pool". In NAT mode,
d.externalIPAM stays nil, cleanly skipping the auto-assign public IP
path. SNAT topology is still created by vpcd via the pool's gateway_ip.

Part of mulga-737 (Phase 16).
2026-03-24 16:27:28 +10:00
Ben Duncan a21a275776 docs: update vpc-networking README for br-wan + new setup-ovn.sh flags
Replace all br-external references with br-wan. Update setup-ovn.sh
command examples to use --wan-bridge/--wan-iface/--macvlan instead of
--external-bridge/--external-iface/--direct.

Part of mulga-742.
2026-03-24 16:19:19 +10:00
Ben Duncan 78bc0d989d docs: update install guides + setup.sh for new setup-ovn.sh flags
- All guides: OVN setup before admin init (consistent ordering)
- Single-node + source-install: Replace --external-bridge/--external-iface
  with auto-detection + --wan-bridge/--wan-iface/--macvlan options
- Multi-node: Simplify examples (auto-detect WAN bridge), show override
  options in prose
- setup.sh: Update "next steps" to match single-node guide, add
  --wan-bridge/--macvlan guidance for physical NIC setups

Part of mulga-742.
2026-03-24 16:17:35 +10:00
Ben Duncan 8c6f66670a feat: add wan_bridge config field, update vpcd detection + network_detect
- config.go: Add WanBridge field to VPCDConfig (default "br-wan")
- spinifex.toml template: Add wan_bridge under [nodes.X.vpcd]
- admin.go: Auto-detect WAN bridge name from default route interface,
  add detectedWanBridge helper
- service.go: Wire WanBridge from cluster config to vpcd Config
- vpcd.go: Add WanBridge to Config, log it on startup, update comments
- network_detect.go: Fix isVirtualInterface to accept the default route
  device as a parameter instead of hardcoding br-external. Bridges like
  br-wan are now correctly treated as physical when they carry the
  default route.

Part of mulga-741.
2026-03-24 16:02:22 +10:00
Ben Duncan 1fca43c9f4 refactor: reset-dev-env.sh + stop-dev.sh — use new setup-ovn.sh flags
reset-dev-env.sh: Replace --external-bridge/--external-iface/--direct
with --wan-bridge/--wan-iface/--macvlan. Auto-detect when WAN interface
is already a bridge (tofu-cluster) vs physical NIC.

stop-dev.sh: Replace hardcoded br-external check with dynamic scan of
all OVS bridges — any non-br-int bridge with an IP means OVS is
infrastructure and should not be stopped.

Part of mulga-740.
2026-03-24 15:58:49 +10:00
Ben Duncan 34b50b11fb refactor: setup-ovn.sh — replace br-external with configurable WAN bridge
Replace --external-bridge/--external-iface/--direct with:
- --wan-bridge=NAME: OVS bridge for WAN traffic (auto-detected from
  default route if it's already a bridge)
- --wan-iface=NAME: physical NIC to add to the WAN bridge
- --macvlan: SSH-safe macvlan mode for single-NIC hosts

Auto-detection logic:
- If default route goes through a bridge (br-wan) → use it directly
- If default route is a physical NIC → stop with guidance (never
  auto-move a NIC into OVS — may drop SSH)

Bridge-mappings updated: external:$WAN_BRIDGE (was external:br-external).
OVN logical network name stays "external" (no topology.go change needed).

Encap IP now prefers br-vpc IP when available (dedicated VPC data plane).

Part of mulga-739. Blocks VPC E2E work.
2026-03-24 15:46:49 +10:00
Julian Sommer 8f81ac29a4 chore: update license references 2026-03-24 15:43:11 +10:00
Julian Sommer eefed70658 docs: add iam 2026-03-24 15:16:33 +10:00
Ben Duncan 216ca9e710 feat: Phase 12b + Phase 16 — KV reconcile on startup + NAT mode support
Phase 12b: vpcd now reconciles OVN topology from NATS KV on startup
(Pass 2). After subscribing to topics, it reads all VPCs, subnets,
IGW attachments, and ENIs from KV and ensures matching OVN objects
exist. Handles reboots, OVN DB loss, and missed events. All operations
are idempotent.

Phase 16 prep: Add --gateway-ip flag to admin init for NAT mode
(specifies the OVN gateway router's external IP without DHCP).
Update reset-dev-env.sh to support EXTERNAL_MODE=nat env var for
NAT/DHCP mode testing alongside the default pool mode.
2026-03-24 15:03:04 +10:00
Julian Sommer fefd5ed9e8 chore: update to typescript 6 2026-03-24 14:29:55 +10:00
Julian Sommer d47f2ab4fd fix: a11y 2026-03-24 14:02:26 +10:00
Julian Sommer 58a9d5741f fix: missing ui tests 2026-03-24 13:45:48 +10:00
Julian Sommer 625a78c529 feat: placement groups frontend 2026-03-24 13:39:04 +10:00
Ben Duncan 88baf5e202 feat: direct bridge mode — prefer over macvlan when WAN NIC != SSH NIC
Auto-detect bridge topology: if a dedicated WAN NIC exists (not used for
SSH), add it directly to br-external instead of creating a macvlan.
Direct bridge enables distributed NAT (DNAT on VM's own chassis) and
eliminates macvlan workarounds (MAC alignment, gratuitous ARPs,
centralized NAT bottleneck).

- topology.go: bridge-mode-aware NAT and localnet port options
- vpcd.go: auto-detect bridge mode, skip MAC alignment for direct bridge
- setup-ovn.sh: --direct flag for physical NIC on br-external
- reset-dev-env.sh: SSH NIC detection via SSH_CONNECTION
2026-03-24 01:57:54 +00:00
Julian Sommer 6379a2098a fix: lint errors 2026-03-24 11:55:42 +10:00
Julian Sommer 638828f636 feat: add leaders to get nodes command 2026-03-24 11:50:28 +10:00
Julian Sommer 0fbf4537c0 chore: bump viperblock 2026-03-24 11:24:40 +10:00
Julian Sommer 1b5108d379 chore: bump frontend deps 2026-03-24 10:45:01 +10:00
Julian Sommer bab7b1dea3 fix: enable last rules 2026-03-24 10:28:20 +10:00
Julian Sommer 30b08ca984 fix: use nullish coalescing 2026-03-24 10:20:11 +10:00
Julian Sommer 3467e0cf22 feat: oxlint type checked rules 2026-03-24 10:14:31 +10:00
Julian Sommer 35d0df5b56 feat: placement groups (#85) 2026-03-24 10:52:00 +11:00
Julian Sommer 648b36d672 fix: add agent browser 2026-03-24 09:25:48 +10:00
Ben Duncan d6062ed550 docs: document direct bridge vs macvlan modes for br-external
Direct bridge (WAN NIC added directly to OVS) is preferred when a
dedicated WAN NIC exists. Enables distributed NAT, no MAC workarounds,
host can reach VMs. Macvlan is fallback for single-NIC hosts only.
2026-03-23 11:36:15 +00:00
Ben Duncan 93aaf0f991 fix: single-node E2E — OVN SB cleanup, chassis naming, macvlan NAT, DNS
Three blockers fixed for VPC public subnet E2E on single-node:

1. OVN SB stale data: reset-dev-env.sh now deletes both NB+SB DB files
   instead of individual NB cleanup. Eliminates ovn-controller commit
   failure loop from stale chassis/port binding entries.

2. Chassis name mismatch: reset-dev-env.sh passes --chassis-id=chassis-node1
   to setup-ovn.sh, aligning OVS system-id with vpcd's chassis-{nodeName}.

3. Macvlan unicast filtering: switched to centralized NAT (no external_mac
   on DNAT rules), set nat-addresses=router on localnet ports, and vpcd
   aligns macvlan MAC with OVN gateway router MAC on startup. Macvlan only
   delivers unicast matching its own MAC — this ensures inbound traffic
   from the LAN reaches OVS/OVN.

Also fixed: DNS for VMs. admin init auto-detects host DNS servers via
resolvectl, stores them in spinifex.toml dns_servers field, and vpcd
propagates them to OVN DHCP options. Fallback: 8.8.8.8 + 1.1.1.1.

Validated E2E: inbound ping, SSH, outbound ping, DNS, curl, apt update.
2026-03-23 10:59:08 +00:00
Ben Duncan f34452883c refactor: move bootstrap config to admin init, add IGW ID, eliminate race
- admin init now pre-generates VPC/subnet/IGW IDs and writes [bootstrap]
  to spinifex.toml before services ever start. No race condition.
- EnsureDefaultVPC accepts optional BootstrapIDs to use pre-determined
  IDs instead of generating random ones.
- CreateInternetGatewayWithID allows daemon to use the bootstrap IGW ID.
- Removed writeBootstrapConfig from daemon (no longer needed).
- Fixed bootstrap account_id: uses admin account (000000000001) not
  system account (000000000000).
- reconcileIGW now sets spinifex:igw_id in OVN external IDs.
2026-03-23 06:57:25 +00:00
Ben Duncan c21f3dc188 feat: vpcd OVN reconciliation on startup + default subnet resolution
Phase 11b: RunInstances now auto-resolves the default subnet when no
SubnetId is specified, matching AWS behavior. Previously fell back to
SLIRP networking.

Phase 12: vpcd reconciles OVN topology from [bootstrap] config on startup.
The daemon writes a [bootstrap] section to spinifex.toml with the default
VPC/subnet IDs on first run. vpcd reads this on startup and creates any
missing OVN objects (router, switch, DHCP, IGW topology, SNAT, gateway
chassis). All operations are idempotent — safe on every restart.

This solves the bootstrap ordering problem where admin init creates VPC
records before vpcd is running, so NATS events are missed.

EnsureDefaultVPC now returns DefaultVPCInfo with VPC/subnet IDs.
2026-03-23 06:40:19 +00:00
Ben Duncan daae6ca6b8 refactor: remove single-nic bridge mode, standardize on macvlan
The single-nic IP migration path was fragile and caused connectivity
issues during setup. macvlan is now the only bridge strategy — always
creates spx-ext-{nic} on the WAN NIC. Removed --single-nic flag,
SingleNIC config/detection fields, and updated all docs.
2026-03-23 01:12:22 +00:00
Ben Duncan 32276c7cc7 feat: wire EIP + SG gateway actions, fix gateway chassis scheduling, default VPC infra
Phase 8a: Wire all 5 EIP gateway actions (AllocateAddress, ReleaseAddress,
AssociateAddress, DisassociateAddress, DescribeAddresses) end-to-end through
gateway → NATS → daemon → EIP service.

Phase 8b: Wire all 7 SG gateway actions (Create/Delete/Describe SecurityGroup,
Authorize/Revoke Ingress/Egress) through the same pipeline.

Critical fix: handleIGWAttach was not calling SetGatewayChassis on the gateway
router port. Without this, OVN has no chassis binding for external traffic —
SNAT/default route exist logically but packets have nowhere to go physically.
This was the root cause of outbound routing failure (can't ping 8.8.8.8).

Default VPC now matches AWS behavior: auto-creates IGW (attached),
MapPublicIpOnLaunch=true on default subnet, and a default security group
with intra-SG inbound allow + all outbound allow.

Comprehensive VPC networking docs rewrite covering modes, bridge setup,
configuration, security groups, EIPs, OVN reference, and troubleshooting.
2026-03-21 10:26:59 +00:00
Ben Duncan 30394b735b Removed legacy dir 2026-03-21 09:36:28 +00:00
Ben Duncan a25a4cec34 Merge commit '9b64c1911fc7e1a2442c376fb2cba0f5ece9a913' into vpc-public-subnet 2026-03-20 07:28:15 +00:00
Ben Duncan f6c6660f15 fix: create default VPC for admin account, not just system account
EnsureDefaultVPC was only called for GlobalAccountID (000000000000).
The admin account (000000000001) from admin init never got a default
VPC because seedAdminAccount didn't publish iam.account.created.

Fix: daemon startup now ensures default VPC for both system and admin
accounts. Also publish iam.account.created from seedAdminAccount so
any future account-creation listeners fire correctly.
2026-03-20 07:19:05 +00:00
Ben Duncan 704874c0c7 docs: restructure install guides + new common setup guide
All 3 install guides (binary, multi-node, source) now end at the same
verification point: aws ec2 describe-instance-types. Installation is
complete when that command returns instance types.

New docs/setting-up-your-cluster/README.md is the common guide all 3
install paths point to. Covers: AMI import, SSH key creation, VPC with
public subnet (IGW + modify-subnet-attribute), instance launch with
public IP, SSH via public IP, instance lifecycle management.

Removed SSH-via-hostfwd from all guides — public subnet is the default
connectivity path. Private subnet documented as an alternative.
2026-03-20 07:10:33 +00:00
Ben Duncan 665093343c fix: preserve DNS resolution after IP migration to br-external
When setup-ovn.sh migrates the WAN IP from a physical NIC to
br-external, systemd-resolved loses its DNS association (DNS servers
are per-interface). This breaks hostname resolution for everything
on the host — git, apt, curl, SSH all fail.

Now captures DNS config from the original interface before migration
and re-associates it with br-external via resolvectl. Falls back to
8.8.8.8 if no DNS was configured on the original interface.
2026-03-20 06:57:18 +00:00
Ben Duncan bb9e8db386 fix: stop-dev.sh must not stop OVS when br-external has WAN IP
When setup-ovn.sh migrates the host's WAN IP to br-external, stopping
openvswitch-switch destroys the bridge and the host loses all network
connectivity — unreachable via SSH, requires console reboot.

Now checks if br-external has an IP before stopping OVS. If it does,
skip the OVS stop and only stop ovn-controller + ovn-central. OVS is
infrastructure — bridges must survive service stop/start cycles.

Discovered during 3-node E2E testing: stop-dev.sh killed all 3 nodes'
WAN connectivity simultaneously.
2026-03-20 06:55:23 +00:00
Ben Duncan 0d4a70ef28 fix: remove verbose next-steps from admin init, disable dev_networking with external mode
Admin init output was duplicating install guide steps. Simplified to
just show completion message. Install guides already have all steps.

Set dev_networking=false when external_mode is configured — the hostfwd
NIC adds a confusing second interface inside VMs and is not needed when
public subnets provide WAN-routable IPs.
2026-03-20 06:45:11 +00:00
Ben Duncan 946429d792 feat: Phase 8c — daemon auto-assign public IP on RunInstances
Wire public IP auto-assignment end-to-end:
- RunInstances checks MapPublicIpOnLaunch on subnet, allocates external
  IP from ExternalIPAM, updates ENI record, publishes vpc.add-nat for
  dnat_and_snat rule, returns PublicIpAddress in response
- TerminateInstances releases public IP back to pool, publishes
  vpc.delete-nat to remove NAT rule before ENI deletion
- DescribeInstances populates PublicIpAddress from VM struct

Add ExternalIPAM initialization in daemon startup from cluster
NetworkConfig. Add GetSubnet and UpdateENIPublicIP helpers. Add
PublicIP/PublicIPPool fields to vm.VM struct. Fix network auto-detect
to recognize br-external as WAN interface.
2026-03-20 06:41:32 +00:00
Ben Duncan a554fe4ecf docs: rewrite single + multi-node install guides
Single-node:
- Add auto-detected network topology + external networking section
- Replace hostfwd SSH with VPC overlay private IP SSH
- Add setup-ovn.sh step with --dhcp flag
- Fix undefined variables ($SPINIFEX_AMI, $SPINIFEX_SUBNET)
- Use bash variable capture for VPC/subnet/AMI IDs
- Add three-tier external networking table
- Fix step headings (## → ###)

Multi-node:
- Move CA certificate trust to Step 5 (before starting services)
- Add --external-bridge --dhcp to all OVN setup commands
- Add AMI import step (was missing entirely)
- Add VPC + subnet creation step with variable capture
- Add SSH via private IP step
- Fix sudo inconsistency (all formation commands use sudo)
- Add external networking section with pool upgrade path
- Add DHCP troubleshooting section
- Add port requirements to overview
2026-03-20 05:58:38 +00:00
Ben Duncan 2af3bf5287 feat: auto-detect network topology + DHCP gateway in admin init
spx admin init now works with zero external networking flags:
- Auto-detects WAN interface, gateway, prefix from ip route
- Auto-detects single-NIC vs multi-NIC mode
- Defaults to nat mode with DHCP for gateway IP
- Prints detected topology with interface roles

setup-ovn.sh gains --dhcp flag: obtains a DHCP lease on the
macvlan/bridge interface for the OVN gateway SNAT address, then
writes gateway_ip into spinifex.toml. Tested on VM: macvlan gets
separate IP from router DHCP, host keeps its original IP.

Three tiers of external networking:
- Zero flags: auto nat+DHCP (VMs get outbound internet)
- --external-pool=start-end: pool mode (VMs get public IPs)
- --no-external: overlay only (no internet for VMs)
2026-03-20 05:45:30 +00:00
Ben Duncan 1f38213d4b feat: add external networking flags to spx admin init
Add --external-mode, --external-iface, --single-nic, --external-pool,
--external-gateway, --external-prefix-len flags to admin init. Config
is now generated with the correct [network] section and per-node vpcd
external_interface/single_nic — no manual spinifex.toml editing needed.

Single-node single-NIC:
  spx admin init --external-mode=pool --external-iface=eth0 --single-nic \
    --external-pool=192.168.1.150-192.168.1.250 --external-gateway=192.168.1.1

Single-node two-NIC:
  spx admin init --external-mode=pool --external-iface=eth1 \
    --external-pool=192.168.1.150-192.168.1.250 --external-gateway=192.168.1.1

Next steps output now shows the correct setup-ovn.sh command with
matching flags.
2026-03-20 05:24:06 +00:00
Ben Duncan 1ee7012ba9 feat: Phase 7 — single-NIC macvlan bridge strategy
Add --single-nic flag to setup-ovn.sh. When used with --external-bridge,
creates a macvlan sub-interface (spx-ext-{iface}) in bridge mode off the
host's NIC instead of adding the NIC directly to br-external. The host
keeps its IP — no migration, SSH-safe.

Strategy selection:
- Dedicated NIC: --external-iface=eth1 (add directly)
- Single NIC:    --external-iface=eth0 --single-nic (macvlan)
- IP migration:  --external-iface=eth0 (legacy, no --single-nic)

Add single_nic bool field to VPCDConfig for per-node config. Different
nodes in a cluster can use different strategies. Unit test verifies
single_nic parsing alongside external_interface.
2026-03-20 05:15:35 +00:00
Ben Duncan b6e99947e1 feat: Phase 5 — unit tests for ACL, security groups, public subnet, EIP
Add 18 unit tests covering:
- ACL match expression compiler (8 tests): TCP port/range, ICMP, UDP
  range, all-protocols, SG-ref source, egress, no-source filter
- Security group Port Group lifecycle (3 tests): create, delete, update
  with ACL generation via NATS events + MockOVNClient
- ModifySubnetAttribute + MapPublicIpOnLaunch (2 tests): default false,
  toggle to true via ModifySubnetAttribute
- Elastic IP lifecycle (5 tests): allocate, allocate from specific pool,
  release, release-while-associated error, describe
2026-03-20 04:35:52 +00:00
Ben Duncan 5481ef4361 feat: Phase 4 — security groups (OVN ACLs + Port Groups)
Add security group CRUD handlers: CreateSecurityGroup, DeleteSecurityGroup,
DescribeSecurityGroups, AuthorizeSecurityGroupIngress/Egress,
RevokeSecurityGroupIngress/Egress. SG records stored in NATS KV.

Add ACL match expression compiler (acl.go) translating AWS SG rules to
OVN ACL match strings for TCP/UDP port/ranges, ICMP, all-protocols, CIDR
filters, and intra-SG references.

Add vpcd security.go handlers for OVN Port Group lifecycle and ACL
management via vpc.create-sg/delete-sg/update-sg NATS topics. Extend
OVNClient interface with CreatePortGroup, DeletePortGroup,
SetPortGroupPorts, AddACL, ClearACLs. Add PortGroup and ACL to nbdb
model.
2026-03-20 04:24:49 +00:00
Ben Duncan ee06105c56 feat: Phase 3 — Elastic IPs (AllocateAddress, AssociateAddress)
Add EIP service with AllocateAddress, ReleaseAddress, AssociateAddress,
DisassociateAddress, and DescribeAddresses. Uses ExternalIPAM for IP
allocation from admin-defined pools. Publishes vpc.add-nat/delete-nat
events for OVN dnat_and_snat rules.

EIP records stored in spinifex-vpc-elastic-ips NATS KV bucket with
account-scoped keys. AssociateAddress replaces auto-assigned public IPs
when present.
2026-03-20 04:21:46 +00:00
Ben Duncan 150bf246a4 feat: Phase 2 — public subnets + auto-assign public IPs
Add MapPublicIpOnLaunch field to SubnetRecord with ModifySubnetAttribute
API handler. Add PublicIpAddress/PublicIpPool fields to ENIRecord for
tracking auto-assigned public IPs.

Add vpc.add-nat / vpc.delete-nat NATS topics and handlers in vpcd
topology for creating distributed dnat_and_snat NAT rules (with
LogicalPort + ExternalMAC for chassis-local NAT processing).

Wire ModifySubnetAttribute through gateway → daemon → VPC service.
2026-03-20 04:19:28 +00:00
Julian Sommer 9b64c1911f ci: add pull_request triggers to CI workflows
Enable PR-triggered CI so required status checks can gate merges.
Spinifex runs on all PRs (no path filter) to avoid blocking non-Go PRs.
UI workflow keeps path filters as it's not a required check.
2026-03-20 14:08:12 +10:00
Ben Duncan c1cdfc07c3 feat: Phase 1 completion — external IPAM, br-external setup, config template
Add external IPAM module (CAS-based IP allocation from admin-defined
pools in NATS KV). Supports multi-pool with region/AZ scoping, pool
fallback logic, gateway IP reservation, and concurrent allocation via
CAS retries. 12 unit tests covering allocation, release, exhaustion,
CAS conflict, multi-pool, fallback, validation, and idempotent init.

Add --external-bridge flag to setup-ovn.sh: creates OVS bridge
br-external, adds WAN NIC, migrates host IP to bridge, and sets
ovn-bridge-mappings=external:br-external for OVN localnet ports.

Add commented [network] section and external_interface to spinifex.toml
config template for discoverability.
2026-03-20 03:59:47 +00:00
Ben Duncan c6af189638 Merge commit '02931567ac207bd074d8b2efda6e523ab36540d9' into vpc-public-subnet 2026-03-20 03:47:50 +00:00
Julian Sommer 02931567ac release: update go.mod for v0.4.2 2026-03-20 13:09:48 +10:00
dependabot[bot] c4890c80d3 build(deps): bump fast-xml-parser (#83) 2026-03-20 12:48:53 +11:00
Julian Sommer d2d3e52522 feat: migrate to golangci-lint (#82) 2026-03-20 12:14:42 +11:00
Julian Sommer 42cb6dbed0 fix: add better error message 2026-03-20 11:12:32 +10:00
Julian Sommer 6cdeea3aa8 chore: more tests 2026-03-20 09:27:21 +10:00
Julian Sommer 4884b68b51 feat: version bootstrap file 2026-03-20 09:12:42 +10:00
Julian Sommer 726528bd9d feat: version jetstream buckets 2026-03-20 09:07:40 +10:00
Julian Sommer 593920ce7d feat: E2E test redistribution and dev-setup cleanup
Redistribute E2E tests between single-node and multi-node suites, slim
down pseudo multi-node suite, and remove directory creation from
dev-setup.sh so spx admin init is the single source of truth for
~/spinifex/ setup.
2026-03-20 08:24:15 +10:00
Ben Duncan 5e30d12f76 feat: Phase 1 — external network config + IGW gateway router fix
Add cluster-wide [network] config with external_mode and multi-pool
support (ExternalPool with region/AZ scoping). Per-node vpcd gets
external_interface for WAN NIC name.

Fix handleIGWAttach to use real external IPs from the configured pool
instead of link-local 169.254.0.1/30. Gateway router port now gets
the pool's gateway IP with correct prefix length, SNAT uses the real
external IP, and the default route points to the actual WAN gateway.

Includes pool selection logic (AZ-scoped → region-scoped → unscoped
fallback) and unit tests for config parsing, IGW with external pool,
explicit gateway IP, and pool selection.
2026-03-19 11:34:30 +00:00
Julian Sommer ea8700df99 fix: increase sleep time 2026-03-19 15:02:50 +10:00
Julian Sommer 857cbd75ac fix: use spx service stop instead of shell PID kill in stop-dev.sh 2026-03-19 14:50:50 +10:00
Julian Sommer 0fa38f7986 fix: write credentials to user even when running as sudo 2026-03-19 14:05:48 +10:00
Julian Sommer 1a4d590bf3 fix: nbd log level 2026-03-19 13:45:27 +10:00
Julian Sommer b7678ab7c5 feat: fix silent failures across spinifex (P2 audit H-25 through H-43)
- PID file failures now prevent service start instead of logging and continuing
- Status() methods use PID-based health checks instead of returning empty string
- NewResourceManager returns error instead of silently defaulting to 8GB
- Crash recovery re-allocates resources before relaunch
- restoreInstances stops after failed resource allocation
- Cluster manager bind errors propagated to Start()
- NATS disconnect/reconnect logged at Warn level
- VPC CIDR fallback logs warning about overbroad rule
- vpcd respond() handles json.Marshal failure
- Network cleanup failures logged instead of silently discarded
- fmt.Println replaced with slog.Debug in NATS service
- Signature mismatch no longer logs expected signature
- extractDiskImagePath uses errors.Is for EOF
- validateDiskImagePath checks file command error
- os.Stat uncommented in admin image import
- net.SplitHostPort errors checked
- Gateway response write errors logged
- vpcd OVN NB address uses leader IP on non-primary nodes
- Release workflow parallelized with native arm64 runner
2026-03-19 13:37:41 +10:00
Julian Sommer 11730be685 fix: locate release assets by filename instead of hardcoded path 2026-03-19 12:13:28 +10:00
Julian Sommer a4bc8bee74 fix: pending instance terminate (#80) 2026-03-19 12:47:00 +11:00
Julian Sommer fafc104c30 feat: add release verification job with draft-then-promote flow 2026-03-19 10:52:39 +10:00
dependabot[bot] efa6257850 build(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.3 (#79) 2026-03-19 11:35:06 +11:00
Julian Sommer 7e46f0012b chore: bump more action versions 2026-03-19 10:23:52 +10:00
Julian Sommer 58e6817efe chore: bump docker build action ver 2026-03-19 10:19:02 +10:00
Julian Sommer 71ec2b7ee5 release: update go.mod for v0.4.0 2026-03-19 08:28:16 +10:00
Julian Sommer 9c430eea07 fix: release workflow 2026-03-19 08:17:30 +10:00
Julian Sommer 1be6c69e51 format 2026-03-19 08:03:23 +10:00
Ben Duncan c523f78403 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-18 05:08:30 +00:00
Ben Duncan 2918a02fac correct spx path 2026-03-18 05:08:15 +00:00
Julian Sommer 8e3210f023 fix 2026-03-18 14:36:12 +10:00
Julian Sommer 6ba9d3dbf0 remove trailing slash 2026-03-18 14:24:52 +10:00
Ben Duncan 27cdfc4653 Merge branch 'dev' of github.com:mulgadc/spinifex into dev 2026-03-18 04:14:22 +00:00
Ben Duncan adadb5ef4c Fixed noninteractive error on apt get 2026-03-18 04:13:53 +00:00
Julian Sommer d0fc2a2120 last fix 2026-03-18 13:40:27 +10:00
Julian Sommer a09a8efe3c build frontend with new branding 2026-03-18 13:38:47 +10:00
Julian Sommer 7d43d0f782 hopefully last rename 2026-03-18 13:24:45 +10:00
Julian Sommer ba0d8b67ff feat: e2e pipeline speedup (#78) 2026-03-18 13:50:45 +11:00
Ben Duncan 454f381826 Moved to GNU Affero General Public License v3 2026-03-18 02:04:41 +00:00
Ben Duncan b95ff42a2f feat: rename product from Hive to Spinifex
Trademark conflict — bulk rename across entire codebase:
- Go module: github.com/mulgadc/hive → github.com/mulgadc/spinifex
- Binary: hive → spx
- Directories: hive/ → spinifex/, hiveui → spinifexui
- Env vars: HIVE_* → SPINIFEX_*
- Config: hive.toml → spinifex.toml
- Systemd: hive-* → spinifex-*
- NATS: hive-workers → spinifex-workers
- All paths, docs, scripts, CI workflows updated
2026-03-18 01:42:53 +00:00
Ben Duncan 2064c338b3 chore: gofmt formatting after rename 2026-03-18 01:23:11 +00:00
Ben Duncan 4f00cf9e4c refactor: update Dockerfile and build artifacts for spinifex rename
- Dockerfile.distro: update all paths from hive → spinifex
- Binary output name: spx
- Logrotate: logrotate-spinifex
2026-03-18 01:20:27 +00:00
Ben Duncan 48012a4058 refactor: rename all documentation from hive to spinifex
- Rename .claude/rules/hive.md → spinifex.md
- Update CLAUDE.md component description
- Rename doc directories: hive-admin-cli → spinifex-admin-cli, etc.
- Rename doc files: hive-cluster-stats, hive-code-consolidation, etc.
- Bulk update all markdown files across docs/ tree
- Update internal docs: CLUSTER-INSTALL, TESTING, DEVNOTES, BRANCHES
2026-03-18 01:19:05 +00:00
Ben Duncan 55186d95f2 refactor: rename CI/CD workflows from hive to spinifex
- Rename hive.yml → spinifex.yml
- Update workflow names, path triggers, working directories
- Update release.yml tarball references
- Update e2e.yml test paths
- Update ui.yml frontend paths
2026-03-18 01:09:02 +00:00
Ben Duncan c11ac04892 refactor: rename frontend UI branding from hive to spinifex
- package.json: name → spinifex-ui
- index.html: page title → Spinifex Control Panel
- sidebar-layout: Mulga Hive → Mulga Spinifex
- auth.ts: localStorage key → spinifex:v1:aws-credentials
- main.tsx: theme storage key → spinifex-ui-theme
- spinifexui_test.go: update service name assertion
2026-03-18 01:06:55 +00:00
Ben Duncan 9e744e5be6 refactor: rename NATS queue group and topics from hive to spinifex
- hive-workers → spinifex-workers queue group (~16 files)
- hive.admin.* → spinifex.admin.* topics
- hive.nodes.* → spinifex.nodes.* topics
- hive.node.* → spinifex.node.* topics
- hive.cluster.* → spinifex.cluster.* topics
- spinifex-nats server_name already updated in Phase 5
2026-03-18 01:06:03 +00:00
Ben Duncan 5d93786995 refactor: rename hive to spinifex in all shell scripts
- Update scripts/: start-dev, stop-dev, dev-setup, setup, setup-ovn, etc.
- Update tests/e2e/: all E2E test scripts
- Update scripts/iac/: rename hive-test.sh → spinifex-test.sh, update TF files
- Update config/: rename hive.service and install script
- Binary refs use spx, env vars use SPINIFEX_, paths use spinifex
2026-03-18 01:04:12 +00:00
Ben Duncan b97a98e2b7 refactor: rename systemd units and logrotate from hive to spinifex
- Rename all build/systemd/hive-* → spinifex-*
- Rename build/logrotate/hive → spinifex
- Update unit contents: descriptions, dependencies, user/group, paths
- Binary path set to /usr/local/bin/spx
2026-03-18 00:53:34 +00:00
Ben Duncan ee0b92c192 refactor: rename config files, paths, and env vars from hive to spinifex
- Rename hive.toml template → spinifex.toml
- Update all default paths: /etc/hive → /etc/spinifex, ~/hive → ~/spinifex
- Rename all HIVE_* env vars → SPINIFEX_* (SetEnvPrefix, BindEnv calls)
- Update nats.conf template server_name
- Update admin.go init/join commands for new paths and names
- Update paths.go, paths_test.go, config.go, config_test.go
2026-03-18 00:47:01 +00:00
Ben Duncan 00f5b6590e refactor: rename CLI binary to spx and update all command names
- Makefile: GO_PROJECT_NAME → spx, update build/test/dist paths
- root.go: Use → spx, update branding to Spinifex
- service.go: hive/hive-ui commands → spinifex/spinifex-ui
- Service name constants: hive → spinifex, hive-ui → spinifex-ui
- Update service_test.go switch cases
2026-03-18 00:42:34 +00:00
Ben Duncan 06804f5775 refactor: rename directory structure hive → spinifex
- git mv hive/ → spinifex/
- git mv cmd/hive/ → cmd/spinifex/
- git mv services/hive/ → services/spinifex/
- git mv services/hiveui/ → services/spinifexui/
- Update package declarations and all internal import paths
- Update service.go switch cases for new service names
2026-03-18 00:37:30 +00:00
Ben Duncan 767ad0bf5c refactor: rename Go module path from hive to spinifex
Bulk rename github.com/mulgadc/hive → github.com/mulgadc/spinifex
across go.mod and all 178 Go files (~411 import occurrences).
2026-03-18 00:29:24 +00:00
Julian Sommer 4152e4422f chore: remove redundant x/net/http2 and automaxprocs deps
ForceAttemptHTTP2 + NextProtos ALPN already handle HTTP/2 on Go 1.22+.
Go 1.25+ runtime is container-aware for GOMAXPROCS by default.
2026-03-18 10:14:52 +10:00
Julian Sommer 48bbe423df fix: complete termination lifecycle for failed instances 2026-03-18 09:53:48 +10:00
Julian Sommer e6b5b3b135 chore: update vite react plugin 2026-03-18 08:54:57 +10:00
Julian Sommer f526972607 chore: go back to ultracite 2026-03-18 08:46:49 +10:00
Julian Sommer ddcd7f622f chore: bump deps 2026-03-18 08:20:18 +10:00
Ben Duncan 2459e5f7ac improved docs 2026-03-16 12:52:14 +00:00
Ben Duncan 120bfa8953 fix: generate go.work in Dockerfile instead of copying gitignored file 2026-03-16 12:36:56 +00:00
1234 changed files with 146429 additions and 40909 deletions
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -e
branch=$(git rev-parse --abbrev-ref HEAD)
# Only enforce preflight on dev and main
if [ "$branch" != "dev" ] && [ "$branch" != "main" ]; then
exit 0
fi
# Only run preflight if Go files were staged
if git diff --cached --name-only | grep -q '\.go$'; then
echo "Go files staged on $branch — running make preflight..."
make preflight
else
echo "No Go files staged, skipping preflight."
fi
+225
View File
@@ -0,0 +1,225 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#161b22"/>
<stop offset="45%" style="stop-color:#1f2028"/>
<stop offset="100%" style="stop-color:#161b22"/>
</linearGradient>
<linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262633"/>
<stop offset="100%" style="stop-color:#7F1D1D14"/>
</linearGradient>
<linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#F87171"/>
<stop offset="50%" style="stop-color:#DC2626"/>
<stop offset="100%" style="stop-color:#7F1D1D"/>
</linearGradient>
</defs>
<rect width="900" height="620" fill="url(#bg)" rx="12"/>
<!-- Title bar -->
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
<text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
<!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
<rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
<text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
<text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
<g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
<rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
<rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
<rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
<rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
<rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
<rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
<text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
</g>
<!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
<line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
<text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
<!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
<rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
<text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
<text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
<g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
<rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
<rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
<rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
<rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
<rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
<rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
</g>
<!-- ═══ NATS BUS ═══════════════════════════════════════ -->
<line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
<text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
<text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.* ebs.* s3.* vpc.* iam.*</text>
<text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
<!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
<g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
<line x1="100" y1="306" x2="100" y2="328"/>
<line x1="275" y1="306" x2="275" y2="328"/>
<line x1="450" y1="306" x2="450" y2="328"/>
<line x1="625" y1="306" x2="625" y2="328"/>
<line x1="800" y1="306" x2="800" y2="328"/>
</g>
<!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
<!-- EC2 -->
<g transform="translate(28, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
</g>
<!-- EBS -->
<g transform="translate(204, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
</g>
<!-- S3 -->
<g transform="translate(380, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
</g>
<!-- VPC -->
<g transform="translate(556, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
</g>
<!-- IAM -->
<g transform="translate(732, 328)">
<rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
<line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
<text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
<text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
</g>
<!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
<g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
<line x1="100" y1="410" x2="100" y2="440"/>
<line x1="276" y1="410" x2="276" y2="440"/>
<line x1="452" y1="410" x2="452" y2="440"/>
<line x1="628" y1="410" x2="628" y2="440"/>
<line x1="804" y1="410" x2="804" y2="440"/>
</g>
<!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
<rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
<text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
<!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
<!-- QEMU/KVM -->
<g transform="translate(28, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
</g>
<!-- Viperblock -->
<g transform="translate(204, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
</g>
<!-- Predastore -->
<g transform="translate(380, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
</g>
<!-- OVN/OVS -->
<g transform="translate(556, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
</g>
<!-- IAM store -->
<g transform="translate(732, 474)">
<rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
<text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
<text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
</g>
<!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
<g transform="translate(16, 558)">
<rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
<text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
<tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
<tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
</text>
<text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 17 KiB

+154
View File
@@ -0,0 +1,154 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="300" viewBox="0 0 900 300">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a;stop-opacity:1" />
<stop offset="50%" style="stop-color:#0d1117;stop-opacity:1" />
<stop offset="100%" style="stop-color:#0a0a0a;stop-opacity:1" />
</linearGradient>
<radialGradient id="vignette" cx="50%" cy="50%" r="70%">
<stop offset="0%" style="stop-color:#7F1D1D;stop-opacity:0.08"/>
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0"/>
</radialGradient>
<linearGradient id="mulga-grad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.95" />
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.95" />
</linearGradient>
<linearGradient id="rule" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC2626;stop-opacity:0" />
<stop offset="50%" style="stop-color:#F87171;stop-opacity:0.45" />
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0" />
</linearGradient>
<filter id="node-glow">
<feGaussianBlur in="SourceGraphic" stdDeviation="3" />
</filter>
</defs>
<!-- Background -->
<rect width="900" height="300" fill="url(#bg)" rx="12"/>
<rect width="900" height="300" fill="url(#vignette)" rx="12"/>
<!-- Mesh network background (mulgadc style) -->
<g opacity="0.35">
<!-- Connections (drawn first, under nodes) -->
<g stroke="#7F1D1D" stroke-width="0.6" fill="none">
<line x1="80" y1="60" x2="200" y2="40"/>
<line x1="80" y1="60" x2="150" y2="140"/>
<line x1="200" y1="40" x2="330" y2="90"/>
<line x1="150" y1="140" x2="260" y2="180"/>
<line x1="150" y1="140" x2="60" y2="220"/>
<line x1="330" y1="90" x2="260" y2="180"/>
<line x1="260" y1="180" x2="120" y2="260"/>
<line x1="60" y1="220" x2="120" y2="260"/>
<line x1="620" y1="50" x2="760" y2="90"/>
<line x1="620" y1="50" x2="700" y2="160"/>
<line x1="760" y1="90" x2="850" y2="160"/>
<line x1="700" y1="160" x2="820" y2="220"/>
<line x1="850" y1="160" x2="820" y2="220"/>
<line x1="700" y1="160" x2="600" y2="240"/>
<line x1="600" y1="240" x2="820" y2="220"/>
<line x1="760" y1="90" x2="700" y2="160"/>
</g>
<!-- Nodes -->
<g fill="#DC2626">
<circle cx="80" cy="60" r="2.5"/>
<circle cx="200" cy="40" r="2"/>
<circle cx="330" cy="90" r="2.5"/>
<circle cx="150" cy="140" r="3"/>
<circle cx="260" cy="180" r="2"/>
<circle cx="60" cy="220" r="2"/>
<circle cx="120" cy="260" r="2.5"/>
<circle cx="620" cy="50" r="2"/>
<circle cx="760" cy="90" r="3"/>
<circle cx="700" cy="160" r="2.5"/>
<circle cx="850" cy="160" r="2"/>
<circle cx="820" cy="220" r="2.5"/>
<circle cx="600" cy="240" r="2"/>
</g>
<!-- Pulse rings (static "about-to-pulse" look) -->
<g fill="none" stroke="#F87171" stroke-width="0.8" opacity="0.5">
<circle cx="150" cy="140" r="6"/>
<circle cx="760" cy="90" r="6"/>
<circle cx="700" cy="160" r="8"/>
</g>
</g>
<!-- Main emblem: cluster-node stack with NATS bus -->
<g transform="translate(200, 150)">
<!-- Outer glow ring -->
<circle cx="0" cy="0" r="64" fill="none" stroke="url(#mulga-grad)" stroke-width="1.5" opacity="0.35" filter="url(#node-glow)"/>
<!-- Top node -->
<rect x="-24" y="-52" width="48" height="22" rx="3" fill="none" stroke="#DC2626" stroke-width="1.8"/>
<circle cx="-16" cy="-41" r="1.6" fill="#F87171"/>
<circle cx="-8" cy="-41" r="1.6" fill="#F87171"/>
<line x1="0" y1="-41" x2="16" y2="-41" stroke="#F87171" stroke-width="0.9" opacity="0.6"/>
<!-- Middle node (active) -->
<rect x="-30" y="-14" width="60" height="28" rx="3" fill="url(#mulga-grad)" opacity="0.22" stroke="#DC2626" stroke-width="2"/>
<circle cx="-20" cy="0" r="2" fill="#F87171"/>
<circle cx="-10" cy="0" r="2" fill="#F87171"/>
<line x1="-2" y1="0" x2="22" y2="0" stroke="#F87171" stroke-width="1.2" opacity="0.8"/>
<!-- Bottom node -->
<rect x="-24" y="30" width="48" height="22" rx="3" fill="none" stroke="#7F1D1D" stroke-width="1.8"/>
<circle cx="-16" cy="41" r="1.6" fill="#DC2626"/>
<circle cx="-8" cy="41" r="1.6" fill="#DC2626"/>
<line x1="0" y1="41" x2="16" y2="41" stroke="#DC2626" stroke-width="0.9" opacity="0.6"/>
<!-- Radiating mesh ports -->
<g stroke-width="0.9" opacity="0.55">
<line x1="-30" y1="0" x2="-58" y2="-26" stroke="#DC2626"/>
<line x1="-30" y1="0" x2="-64" y2="0" stroke="#DC2626"/>
<line x1="-30" y1="0" x2="-58" y2="26" stroke="#DC2626"/>
<line x1="30" y1="0" x2="58" y2="-26" stroke="#7F1D1D"/>
<line x1="30" y1="0" x2="64" y2="0" stroke="#7F1D1D"/>
<line x1="30" y1="0" x2="58" y2="26" stroke="#7F1D1D"/>
</g>
<g fill="#F87171">
<circle cx="-58" cy="-26" r="2.5"/>
<circle cx="-64" cy="0" r="2.5"/>
<circle cx="-58" cy="26" r="2.5"/>
</g>
<g fill="#DC2626">
<circle cx="58" cy="-26" r="2.5"/>
<circle cx="64" cy="0" r="2.5"/>
<circle cx="58" cy="26" r="2.5"/>
</g>
</g>
<!-- Title -->
<text x="340" y="120" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="56" font-weight="700" fill="#f5f5f5" letter-spacing="-1.5">Spinifex</text>
<!-- Subtitle -->
<text x="340" y="155" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">Open-source AWS-compatible platform</text>
<text x="340" y="180" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">for secure edge deployments</text>
<!-- Rule -->
<rect x="340" y="200" width="440" height="1" fill="url(#rule)" rx="1"/>
<!-- Service pills -->
<g font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" font-weight="500">
<rect x="340" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="365" y="231" fill="#F87171" text-anchor="middle">EC2</text>
<rect x="398" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="423" y="231" fill="#F87171" text-anchor="middle">EBS</text>
<rect x="456" y="215" width="45" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="478" y="231" fill="#F87171" text-anchor="middle">S3</text>
<rect x="509" y="215" width="55" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="536" y="231" fill="#F87171" text-anchor="middle">VPC</text>
<rect x="572" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
<text x="597" y="231" fill="#F87171" text-anchor="middle">IAM</text>
<rect x="630" y="215" width="78" height="24" rx="12" fill="#18181b" stroke="#52525b" stroke-width="0.5" stroke-opacity="0.8"/>
<text x="669" y="231" fill="#a1a1aa" text-anchor="middle">Air-Gapped</text>
</g>
<!-- Tagline -->
<text x="340" y="265" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" fill="#71717a" letter-spacing="1.8">YOUR INFRASTRUCTURE · YOUR HARDWARE · YOUR CLOUD</text>
</svg>

After

Width:  |  Height:  |  Size: 7.5 KiB

+175
View File
@@ -0,0 +1,175 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="720" viewBox="0 0 900 720">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#161b22"/>
<stop offset="45%" style="stop-color:#1f2028"/>
<stop offset="100%" style="stop-color:#161b22"/>
</linearGradient>
<linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262633"/>
<stop offset="100%" style="stop-color:#7F1D1D14"/>
</linearGradient>
<linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#F87171"/>
<stop offset="50%" style="stop-color:#DC2626"/>
<stop offset="100%" style="stop-color:#7F1D1D"/>
</linearGradient>
<linearGradient id="daemon-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262622"/>
<stop offset="100%" style="stop-color:#7F1D1D0a"/>
</linearGradient>
</defs>
<rect width="900" height="720" fill="url(#bg)" rx="12"/>
<!-- Title -->
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">HIGH-LEVEL ARCHITECTURE</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">SDK → Gateway → NATS → Daemons → VMs</text>
<!-- ═══ AWS SDK / CLI ══════════════════════════════════ -->
<rect x="16" y="50" width="868" height="76" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
<text x="32" y="72" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="1">AWS SDK / CLI</text>
<text x="32" y="90" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Users interact with Spinifex using standard AWS SDKs and CLIs against a custom endpoint URL</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
<rect x="32" y="98" width="480" height="20" rx="4" fill="#0a0a0a" stroke="#3fb950" stroke-width="0.6" stroke-opacity="0.4"/>
<text x="42" y="112" fill="#7ee787">$ aws ec2 run-instances --endpoint-url https://localhost:9999 ...</text>
</g>
<!-- Connector: HTTPS + SigV4 -->
<line x1="450" y1="126" x2="450" y2="154" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="380" y="128" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
<text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">HTTPS · SigV4 Auth</text>
<!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
<rect x="16" y="154" width="868" height="104" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
<text x="32" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY · port 9999</text>
<text x="876" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a" text-anchor="end">spinifex/services/awsgw/awsgw.go</text>
<!-- Two-column feature bullets -->
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="40" y="200">● TLS termination</text>
<text x="40" y="220">● EC2 Query Protocol parsing</text>
<text x="40" y="240">● XML response generation</text>
<text x="460" y="200">● AWS SigV4 authentication</text>
<text x="460" y="220">● Routes to service handlers</text>
<text x="460" y="240">● NATS client connection</text>
</g>
<!-- Connector: JSON over NATS -->
<line x1="450" y1="258" x2="450" y2="286" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
<rect x="380" y="260" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
<text x="450" y="274" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">JSON over NATS</text>
<!-- ═══ NATS BROKER ═══════════════════════════════════ -->
<rect x="16" y="286" width="868" height="124" rx="8" fill="#1a0a0a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<rect x="16" y="286" width="868" height="28" rx="8" fill="url(#nats-grad)" opacity="0.9"/>
<text x="32" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS MESSAGE BROKER</text>
<text x="876" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">request/reply · queue groups · fan-out</text>
<!-- Two-column topics/queue groups -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="40" y="332" fill="#F87171" font-weight="600">Topics</text>
<text x="40" y="350" fill="#f5f5f5">ec2.RunInstances</text>
<text x="40" y="368" fill="#f5f5f5">ec2.DescribeInstances</text>
<text x="40" y="386" fill="#f5f5f5">ec2.cmd.{instanceID}</text>
<text x="40" y="404" fill="#f5f5f5">ebs.mount · ebs.unmount</text>
<text x="460" y="332" fill="#F87171" font-weight="600">Delivery mode</text>
<text x="460" y="350" fill="#a1a1aa">spinifex-workers <tspan fill="#71717a">(load-balanced)</tspan></text>
<text x="460" y="368" fill="#a1a1aa">no queue group <tspan fill="#71717a">(fan-out to all nodes)</tspan></text>
<text x="460" y="386" fill="#a1a1aa">per-instance <tspan fill="#71717a">(start / stop / terminate)</tspan></text>
<text x="460" y="404" fill="#a1a1aa">Viperblock request/reply</text>
</g>
<!-- Fan-out connectors to 3 daemons -->
<g stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6" fill="none">
<line x1="450" y1="410" x2="450" y2="430"/>
<line x1="450" y1="430" x2="155" y2="452"/>
<line x1="450" y1="430" x2="450" y2="452"/>
<line x1="450" y1="430" x2="745" y2="452"/>
</g>
<!-- ═══ DAEMON ROW ════════════════════════════════════ -->
<!-- Daemon 1 -->
<g transform="translate(32, 452)">
<rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
<text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 1</text>
<line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="20" y="52">● NATS subscriber</text>
<text x="20" y="70">● Resource manager</text>
<text x="20" y="88">● QEMU/KVM launcher</text>
<text x="20" y="106">● QMP monitoring</text>
</g>
<text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
</g>
<!-- Daemon 2 -->
<g transform="translate(327, 452)">
<rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
<text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 2</text>
<line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="20" y="52">● NATS subscriber</text>
<text x="20" y="70">● Resource manager</text>
<text x="20" y="88">● QEMU/KVM launcher</text>
<text x="20" y="106">● QMP monitoring</text>
</g>
<text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
</g>
<!-- Daemon N -->
<g transform="translate(622, 452)">
<rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
<text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node N</text>
<line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
<text x="20" y="52">● NATS subscriber</text>
<text x="20" y="70">● Resource manager</text>
<text x="20" y="88">● QEMU/KVM launcher</text>
<text x="20" y="106">● QMP monitoring</text>
</g>
<text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
</g>
<!-- Daemon → VM connectors -->
<g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
<line x1="155" y1="576" x2="155" y2="610"/>
<line x1="450" y1="576" x2="450" y2="610"/>
<line x1="745" y1="576" x2="745" y2="610"/>
</g>
<!-- ═══ VM BAR + ROW ═══════════════════════════════════ -->
<rect x="16" y="610" width="868" height="22" rx="5" fill="url(#bar-grad)"/>
<text x="450" y="625" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">WORKLOADS · QEMU/KVM virtual machines per node</text>
<!-- VM boxes -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="32" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="155" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
<text x="155" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
<text x="155" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 1 workloads</text>
<rect x="327" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="450" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
<text x="450" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
<text x="450" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 2 workloads</text>
<rect x="622" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
<text x="745" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
<text x="745" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
<text x="745" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node N workloads</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 12 KiB

+46
View File
@@ -0,0 +1,46 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="180" fill="url(#bg)" rx="10"/>
<text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">BASTION · only entry point into a private subnet</text>
<!-- WAN -->
<g transform="translate(40, 56)">
<rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
<text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">operator / laptop</text>
</g>
<!-- Arrow: SSH -->
<line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
<rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">SSH</text>
<!-- Bastion -->
<g transform="translate(338, 56)">
<rect width="220" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.3" stroke-opacity="0.6"/>
<text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Bastion</text>
<text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">public subnet</text>
</g>
<!-- Arrow: SSH (private IP) -->
<line x1="558" y1="86" x2="638" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="634,82 646,86 634,90" fill="#a855f7" opacity="0.8"/>
<rect x="552" y="62" width="140" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="622" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">SSH (private IP)</text>
<!-- App server -->
<g transform="translate(656, 56)">
<rect width="220" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
<text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">App Server</text>
<text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">private subnet · no internet</text>
</g>
<text x="450" y="156" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">App server has no route to the internet — all access hops through the bastion.</text>
</svg>

After

Width:  |  Height:  |  Size: 3.1 KiB

+98
View File
@@ -0,0 +1,98 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="480" viewBox="0 0 900 480">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="inet-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb9501f"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f71f"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
</defs>
<rect width="900" height="480" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · multi-AZ</text>
<!-- Internet -->
<rect x="350" y="54" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
<!-- connector to IGW -->
<line x1="450" y1="94" x2="450" y2="116" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="446,112 454,112 450,120" fill="#71717a" opacity="0.7"/>
<!-- IGW -->
<rect x="380" y="122" width="140" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">IGW</text>
<text x="450" y="158" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">nginx-alb-igw</text>
<!-- Fan-out to two public subnets -->
<line x1="450" y1="166" x2="450" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="230" y1="184" x2="670" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="230" y1="184" x2="230" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="670" y1="184" x2="670" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<!-- public_a -->
<g transform="translate(108, 202)">
<rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
<rect x="20" y="70" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="85" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">NAT GW</text>
</g>
<!-- public_b -->
<g transform="translate(548, 202)">
<rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="56" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="71" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
</g>
<!-- SNAT arrow from public_a NAT → private row -->
<line x1="230" y1="306" x2="230" y2="340" stroke="#F87171" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.65"/>
<polygon points="226,336 234,336 230,344" fill="#F87171" opacity="0.8"/>
<rect x="256" y="314" width="240" height="20" rx="10" fill="#0a0a0a" stroke="#F87171" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="376" y="328" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">SNAT for private subnets (egress)</text>
<!-- private_a -->
<g transform="translate(108, 350)">
<rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_1</text>
<text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
</g>
<!-- private_b -->
<g transform="translate(548, 350)">
<rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_2</text>
<text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 7.1 KiB

+44
View File
@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="180" fill="url(#bg)" rx="10"/>
<text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">S3 WEBAPP · Flask on EC2 with Predastore backend</text>
<!-- Browser -->
<g transform="translate(40, 56)">
<rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Browser</text>
<text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">end user</text>
</g>
<!-- Arrow: HTTP -->
<line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
<rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">HTTP</text>
<!-- EC2 Instance -->
<g transform="translate(338, 56)">
<rect width="240" height="60" rx="8" fill="#DC26261a" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.65"/>
<text x="120" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">EC2 Instance</text>
<text x="120" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">Flask webapp · port 80</text>
</g>
<!-- Arrow: S3 API (boto3) -->
<line x1="578" y1="86" x2="658" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="654,82 666,86 654,90" fill="#a855f7" opacity="0.8"/>
<rect x="570" y="62" width="144" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="642" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">S3 API (boto3)</text>
<!-- Predastore -->
<g transform="translate(676, 56)">
<rect width="200" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
<text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">Predastore</text>
<text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">S3 store · port 8443</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.9 KiB

@@ -0,0 +1,78 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="330" viewBox="0 0 900 330">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="cluster-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="pool-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#DC262622"/>
<stop offset="100%" style="stop-color:#7F1D1D08"/>
</linearGradient>
<linearGradient id="node-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#58a6ff22"/>
<stop offset="100%" style="stop-color:#58a6ff08"/>
</linearGradient>
</defs>
<rect width="900" height="330" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">CONFIGURATION LEVELS · spinifex.toml</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">three scopes: cluster · pool · node</text>
<!-- File badge -->
<rect x="24" y="50" width="140" height="24" rx="6" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
<text x="94" y="67" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#e4e4e7" text-anchor="middle">spinifex.toml</text>
<!-- spine -->
<line x1="52" y1="74" x2="52" y2="298" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<!-- ═══ Level 1: [network] cluster-wide ═══════════════════ -->
<line x1="52" y1="108" x2="92" y2="108" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<rect x="92" y="86" width="764" height="56" rx="8" fill="url(#cluster-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="108" y="108" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">[network]</text>
<text x="220" y="108" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Cluster-wide · which mode?</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="108" y="128" fill="#a1a1aa">└─</text>
<text x="128" y="128" fill="#7ee787">external_mode</text>
<text x="224" y="128" fill="#a1a1aa">=</text>
<text x="238" y="128" fill="#f5f5f5">"pool"</text>
</g>
<!-- ═══ Level 2: [[network.external_pools]] ═══════════════ -->
<line x1="52" y1="176" x2="92" y2="176" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<rect x="92" y="154" width="764" height="78" rx="8" fill="url(#pool-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="108" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">[[network.external_pools]]</text>
<text x="328" y="176" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">IP ranges · what IPs to use?</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="108" y="196" fill="#a1a1aa">├─</text>
<text x="128" y="196" fill="#FCA5A5">name</text>
<text x="158" y="196" fill="#71717a">·</text>
<text x="166" y="196" fill="#FCA5A5">range_start</text>
<text x="246" y="196" fill="#71717a">·</text>
<text x="254" y="196" fill="#FCA5A5">range_end</text>
<text x="108" y="212" fill="#a1a1aa">├─</text>
<text x="128" y="212" fill="#FCA5A5">gateway</text>
<text x="182" y="212" fill="#71717a">·</text>
<text x="190" y="212" fill="#FCA5A5">prefix_len</text>
<text x="108" y="228" fill="#a1a1aa">└─</text>
<text x="128" y="228" fill="#FCA5A5">region</text>
<text x="174" y="228" fill="#71717a">·</text>
<text x="182" y="228" fill="#FCA5A5">az</text>
<text x="202" y="228" fill="#71717a">(optional)</text>
</g>
<!-- ═══ Level 3: [nodes.NAME.vpcd] ════════════════════════ -->
<line x1="52" y1="268" x2="92" y2="268" stroke="#52525b" stroke-width="1" opacity="0.6"/>
<rect x="92" y="246" width="764" height="56" rx="8" fill="url(#node-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="108" y="268" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700">[nodes.NAME.vpcd]</text>
<text x="266" y="268" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Per-node · which NIC?</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
<text x="108" y="288" fill="#a1a1aa">└─</text>
<text x="128" y="288" fill="#8ab4f8">external_interface</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.8 KiB

@@ -0,0 +1,63 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="110" fill="url(#bg)" rx="10"/>
<text x="24" y="22" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DATA PATH · VM to WAN via OVN localnet</text>
<!-- VM TAP -->
<g transform="translate(24, 40)">
<rect width="118" height="48" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="59" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">VM TAP</text>
<text x="59" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">guest interface</text>
</g>
<polygon points="148,60 160,64 148,68" fill="#71717a" opacity="0.7"/>
<line x1="142" y1="64" x2="148" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- br-int -->
<g transform="translate(166, 40)">
<rect width="102" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="51" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-int</text>
<text x="51" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
</g>
<polygon points="274,60 286,64 274,68" fill="#71717a" opacity="0.7"/>
<line x1="268" y1="64" x2="274" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- OVN logical pipeline -->
<g transform="translate(292, 40)">
<rect width="192" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="96" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN logical pipeline</text>
<text x="96" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">switch → router → SNAT/DNAT</text>
</g>
<polygon points="490,60 502,64 490,68" fill="#71717a" opacity="0.7"/>
<line x1="484" y1="64" x2="490" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- localnet -->
<g transform="translate(508, 40)">
<rect width="120" height="48" rx="7" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
<text x="60" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">localnet</text>
<text x="60" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">OVN port type</text>
</g>
<polygon points="634,60 646,64 634,68" fill="#71717a" opacity="0.7"/>
<line x1="628" y1="64" x2="634" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- br-wan -->
<g transform="translate(652, 40)">
<rect width="110" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="55" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
<text x="55" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
</g>
<polygon points="768,60 780,64 768,68" fill="#71717a" opacity="0.7"/>
<line x1="762" y1="64" x2="768" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- WAN -->
<g transform="translate(786, 40)">
<rect width="90" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="45" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
<text x="45" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">uplink</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.1 KiB

@@ -0,0 +1,74 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="360" viewBox="0 0 900 360">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="host-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#58a6ff22"/>
<stop offset="100%" style="stop-color:#58a6ff08"/>
</linearGradient>
<linearGradient id="vm-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
</defs>
<rect width="900" height="360" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">TWO DHCP CONVERSATIONS · completely independent</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">pool mode · static or dhcp source</text>
<!-- ═══ HOST ↔ ROUTER DHCP ══════════════════════════════ -->
<rect x="24" y="54" width="852" height="138" rx="10" fill="url(#host-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="44" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700" letter-spacing="0.8">HOST ↔ ROUTER DHCP</text>
<text x="876" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="end">on br-ext · veth pair → br-wan</text>
<line x1="24" y1="88" x2="876" y2="88" stroke="#58a6ff" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<!-- step 1 -->
<circle cx="54" cy="108" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
<text x="54" y="112" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">1</text>
<text x="74" y="112" fill="#f5f5f5">Spinifex asks router:</text>
<text x="228" y="112" fill="#a1a1aa">"give me a public IP for this VM"</text>
<!-- step 2 -->
<circle cx="54" cy="132" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
<text x="54" y="136" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">2</text>
<text x="74" y="136" fill="#f5f5f5">Router responds:</text>
<text x="196" y="136" fill="#8ab4f8" font-weight="600">192.168.1.75</text>
<!-- step 3 -->
<circle cx="54" cy="156" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
<text x="54" y="160" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">3</text>
<text x="74" y="160" fill="#f5f5f5">Spinifex stores IP, tells OVN:</text>
<text x="274" y="160" fill="#F87171" font-weight="600">dnat_and_snat(192.168.1.75 ↔ 172.31.0.4)</text>
</g>
<text x="44" y="181" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">Only used in DHCP source mode. Static mode picks from range.</text>
<!-- separator -->
<line x1="24" y1="206" x2="876" y2="206" stroke="#27272a" stroke-width="0.8"/>
<text x="450" y="220" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#52525b" text-anchor="middle" letter-spacing="2">INDEPENDENT · no packets cross between</text>
<!-- ═══ VM ↔ OVN DHCP ═══════════════════════════════════ -->
<rect x="24" y="234" width="852" height="110" rx="10" fill="url(#vm-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="44" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="0.8">VM ↔ OVN DHCP</text>
<text x="876" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end">on tap → br-int → OVN logical switch</text>
<line x1="24" y1="268" x2="876" y2="268" stroke="#3fb950" stroke-opacity="0.25" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<circle cx="54" cy="288" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
<text x="54" y="292" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">1</text>
<text x="74" y="292" fill="#f5f5f5">cloud-init asks:</text>
<text x="190" y="292" fill="#a1a1aa">"what's my IP?"</text>
<circle cx="54" cy="312" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
<text x="54" y="316" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">2</text>
<text x="74" y="316" fill="#f5f5f5">OVN DHCP responds:</text>
<text x="226" y="316" fill="#a1a1aa">"you are</text>
<text x="284" y="316" fill="#7ee787" font-weight="600">172.31.0.4</text>
<text x="356" y="316" fill="#a1a1aa">"</text>
</g>
<text x="44" y="336" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">VM never sees 192.168.1.x or the router.</text>
</svg>

After

Width:  |  Height:  |  Size: 5.2 KiB

@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="170" fill="url(#bg)" rx="10"/>
<text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DIRECT BRIDGE MODE · dedicated WAN NIC</text>
<!-- Row 1: Management NIC (independent, no WAN bridge path) -->
<g transform="translate(24, 42)">
<rect width="852" height="44" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="20" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">Management NIC</text>
<text x="20" y="36" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">host IP · SSH · spinifex services · overlay (Geneve)</text>
<text x="840" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end" font-weight="600">independent path</text>
</g>
<!-- Row 2: WAN NIC → br-wan → localnet -->
<g transform="translate(24, 100)">
<rect width="160" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="80" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
<text x="80" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">dedicated physical</text>
</g>
<polygon points="198,118 210,122 198,126" fill="#71717a" opacity="0.7"/>
<line x1="186" y1="122" x2="198" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<g transform="translate(214, 100)">
<rect width="220" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="110" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
<text x="110" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS port (NIC added directly)</text>
</g>
<polygon points="448,118 460,122 448,126" fill="#71717a" opacity="0.7"/>
<line x1="434" y1="122" x2="448" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<g transform="translate(464, 100)">
<rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">OVN localnet</text>
<text x="90" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">external switch port</text>
</g>
<text x="670" y="130" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" font-weight="600">distributed NAT · any MAC · all frames</text>
</svg>

After

Width:  |  Height:  |  Size: 3.1 KiB

+91
View File
@@ -0,0 +1,91 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="440" viewBox="0 0 900 440">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="host-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#27272a88"/>
<stop offset="100%" style="stop-color:#27272a22"/>
</linearGradient>
<linearGradient id="br-int-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f71f"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
<linearGradient id="br-wan-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#58a6ff1f"/>
<stop offset="100%" style="stop-color:#58a6ff08"/>
</linearGradient>
</defs>
<rect width="900" height="440" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK FLOW · bare-metal host</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">br-int overlay · br-wan WAN uplink</text>
<!-- ═══ Bare-Metal Host container ═══════════════════════ -->
<rect x="40" y="60" width="820" height="284" rx="12" fill="url(#host-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.7"/>
<text x="58" y="84" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" font-weight="700" letter-spacing="1">BARE-METAL HOST</text>
<line x1="40" y1="96" x2="860" y2="96" stroke="#52525b" stroke-opacity="0.5" stroke-width="0.5"/>
<!-- br-int bridge (left) -->
<g transform="translate(80, 122)">
<rect width="260" height="176" rx="10" fill="url(#br-int-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" font-weight="700">br-int</text>
<text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="end">(overlay)</text>
<line x1="20" y1="34" x2="240" y2="34" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="30" y="52" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
<text x="130" y="70" fill="#d8b4fe" text-anchor="middle">tap-eni1</text>
<rect x="30" y="88" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
<text x="130" y="106" fill="#d8b4fe" text-anchor="middle">tap-eni2</text>
<rect x="30" y="124" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
<text x="130" y="142" fill="#d8b4fe" text-anchor="middle">tap-eni3</text>
</g>
</g>
<!-- OVN NAT pipeline arrow in middle -->
<g transform="translate(352, 200)">
<line x1="0" y1="10" x2="148" y2="10" stroke="#DC2626" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
<polygon points="148,5 160,10 148,15" fill="#DC2626" opacity="0.8"/>
<text x="78" y="0" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN</text>
<text x="78" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">NAT pipeline</text>
</g>
<!-- br-wan bridge (right) -->
<g transform="translate(520, 122)">
<rect width="260" height="176" rx="10" fill="url(#br-wan-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" font-weight="700">br-wan</text>
<text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#8ab4f8" text-anchor="end">(WAN)</text>
<line x1="20" y1="34" x2="240" y2="34" stroke="#58a6ff" stroke-opacity="0.3" stroke-width="0.5"/>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="30" y="52" width="200" height="60" rx="5" fill="#58a6ff1a" stroke="#58a6ff" stroke-opacity="0.45" stroke-width="0.8"/>
<text x="130" y="74" fill="#8ab4f8" text-anchor="middle" font-weight="600">spx-ext-{nic}</text>
<text x="130" y="92" fill="#a1a1aa" text-anchor="middle" font-size="10">(macvlan)</text>
</g>
</g>
<!-- macvlan bridge mode label -->
<text x="650" y="314" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">macvlan bridge mode</text>
<!-- WAN NIC inside host, bottom -->
<line x1="650" y1="298" x2="650" y2="320" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<g transform="translate(500, 320)">
<rect width="300" height="20" rx="5" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
<text x="150" y="14" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">WAN NIC · host IP unchanged</text>
</g>
<!-- connector out -->
<line x1="650" y1="344" x2="650" y2="380" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="646,376 654,376 650,384" fill="#71717a" opacity="0.7"/>
<!-- Physical WAN -->
<g transform="translate(500, 388)">
<rect width="300" height="40" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="150" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">Physical WAN</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 5.7 KiB

+57
View File
@@ -0,0 +1,57 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="320" viewBox="0 0 900 320">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="320" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">MACVLAN FALLBACK · single-NIC host</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">WAN NIC == SSH NIC · host IP preserved</text>
<!-- WAN NIC (parent) -->
<g transform="translate(280, 58)">
<rect width="340" height="54" rx="8" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.3" stroke-opacity="0.65"/>
<text x="170" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
<text x="170" y="42" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">host IP unchanged · parent interface</text>
</g>
<!-- Split point -->
<line x1="450" y1="112" x2="450" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<line x1="260" y1="132" x2="640" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<line x1="260" y1="132" x2="260" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<line x1="640" y1="132" x2="640" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- Child 1: host traffic -->
<g transform="translate(120, 148)">
<rect width="280" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">host traffic</text>
<text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SSH · management · overlay</text>
</g>
<!-- Child 2: macvlan -->
<g transform="translate(500, 148)">
<rect width="280" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">spx-ext-{nic}</text>
<text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">macvlan · bridge mode · no IP</text>
</g>
<!-- br-wan -->
<line x1="640" y1="196" x2="640" y2="216" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="636,212 644,212 640,220" fill="#71717a" opacity="0.7"/>
<g transform="translate(500, 220)">
<rect width="280" height="40" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="140" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
</g>
<!-- localnet -->
<line x1="640" y1="260" x2="640" y2="276" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="636,272 644,272 640,280" fill="#71717a" opacity="0.7"/>
<g transform="translate(500, 280)">
<rect width="280" height="30" rx="6" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
<text x="140" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="600">localnet → OVN ext switch</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 3.6 KiB

@@ -0,0 +1,131 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="wan-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="ovs-grad" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" style="stop-color:#58a6ff22"/>
<stop offset="100%" style="stop-color:#58a6ff0a"/>
</linearGradient>
<linearGradient id="router-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#DC262622"/>
<stop offset="100%" style="stop-color:#7F1D1D0a"/>
</linearGradient>
<linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f722"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
</defs>
<rect width="900" height="560" fill="url(#bg)" rx="12"/>
<!-- Title -->
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK PATH</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">OVN logical topology · bridge mapping · NAT</text>
<!-- WAN -->
<rect x="300" y="54" width="300" height="40" rx="8" fill="url(#wan-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.55"/>
<text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Physical Network (WAN)</text>
<!-- connector -->
<line x1="450" y1="94" x2="450" y2="112" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<!-- br-wan bridge -->
<rect x="340" y="112" width="220" height="52" rx="8" fill="url(#ovs-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="450" y="134" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
<text x="450" y="152" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">OVS bridge · physical uplink</text>
<!-- connector with bridge-mappings label -->
<line x1="450" y1="164" x2="450" y2="186" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<rect x="316" y="170" width="268" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.5"/>
<text x="450" y="183" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">ovn-bridge-mappings: external:br-wan</text>
<!-- VPC Logical Router (the big box) -->
<rect x="60" y="208" width="780" height="196" rx="10" fill="url(#router-grad)" stroke="#DC2626" stroke-width="1.4" stroke-opacity="0.75"/>
<text x="78" y="232" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" font-weight="700" letter-spacing="1">VPC LOGICAL ROUTER</text>
<line x1="60" y1="242" x2="840" y2="242" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
<!-- Router ports row -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
<rect x="82" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
<text x="142" y="270" fill="#F87171" text-anchor="middle" font-weight="600">Gateway port</text>
<text x="142" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">public IP</text>
<text x="215" y="282" fill="#71717a">─────</text>
<rect x="260" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
<text x="320" y="270" fill="#F87171" text-anchor="middle" font-weight="600">ext switch</text>
<text x="320" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">OVN logical</text>
<text x="393" y="282" fill="#71717a">─────</text>
<rect x="438" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
<text x="498" y="270" fill="#F87171" text-anchor="middle" font-weight="600">localnet port</text>
<text x="498" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">→ br-wan</text>
</g>
<!-- NAT rules + Routes, two columns on right -->
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
<text x="598" y="260" fill="#F87171" font-weight="600">NAT rules</text>
<text x="598" y="276" fill="#f5f5f5">SNAT:</text>
<text x="644" y="276" fill="#a1a1aa">VPC CIDR → gw IP</text>
<text x="598" y="292" fill="#f5f5f5">dnat_and_snat:</text>
<text x="704" y="292" fill="#a1a1aa">pub ↔ priv</text>
</g>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
<text x="82" y="330" fill="#F87171" font-weight="600">Routes</text>
<text x="82" y="346" fill="#f5f5f5">0.0.0.0/0</text>
<text x="154" y="346" fill="#a1a1aa">→ WAN gateway</text>
</g>
<!-- Subnet branches inside router -->
<line x1="150" y1="362" x2="150" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="150" y1="386" x2="305" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="305" y1="386" x2="305" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="150" y1="386" x2="515" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="515" y1="386" x2="515" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<!-- Subnet A (public) -->
<g transform="translate(215, 404)">
<rect width="180" height="124" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">subnet-a</text>
<text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">(public)</text>
<line x1="16" y1="46" x2="164" y2="46" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.1.0/24</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
<circle cx="40" cy="88" r="2.5" fill="#3fb950"/>
<text x="52" y="92">eni-1</text>
<circle cx="40" cy="108" r="2.5" fill="#3fb950"/>
<text x="52" y="112">eni-2</text>
</g>
</g>
<!-- Subnet B (private) -->
<g transform="translate(425, 404)">
<rect width="180" height="124" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">subnet-b</text>
<text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">(private)</text>
<line x1="16" y1="46" x2="164" y2="46" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.2.0/24</text>
<g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
<circle cx="40" cy="88" r="2.5" fill="#a855f7"/>
<text x="52" y="92">eni-3</text>
<circle cx="40" cy="108" r="2.5" fill="#a855f7"/>
<text x="52" y="112">eni-4</text>
</g>
</g>
<!-- Footnote -->
<text x="24" y="548" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a">Cross-host traffic uses Geneve tunnels (UDP 6081) over the overlay NIC · ovn-controller programs OpenFlow rules on br-int</text>
</svg>

After

Width:  |  Height:  |  Size: 8.2 KiB

@@ -0,0 +1,56 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="110" fill="url(#bg)" rx="10"/>
<text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PRIVATE SUBNET · no default route</text>
<!-- Instance -->
<g transform="translate(40, 42)">
<rect width="180" height="44" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">Instance</text>
<text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">10.0.2.5</text>
</g>
<!-- arrow -->
<g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
<line x1="220" y1="64" x2="260" y2="64" stroke-dasharray="3,3"/>
</g>
<polygon points="260,60 268,64 260,68" fill="#DC2626" opacity="0.6"/>
<!-- Router -->
<g transform="translate(275, 42)">
<rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">VPC Router</text>
<text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">lookup 0.0.0.0/0</text>
</g>
<!-- arrow -->
<g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
<line x1="455" y1="64" x2="495" y2="64" stroke-dasharray="3,3"/>
</g>
<polygon points="495,60 503,64 495,68" fill="#DC2626" opacity="0.6"/>
<!-- [no default route] -->
<g transform="translate(510, 42)">
<rect width="196" height="44" rx="7" fill="#0a0a0a" stroke="#71717a" stroke-width="1" stroke-opacity="0.7" stroke-dasharray="4,3"/>
<text x="98" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">[no default route]</text>
<text x="98" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle">no IGW attached</text>
</g>
<!-- arrow to dropped -->
<g stroke="#71717a" stroke-width="1.2" opacity="0.6" fill="none">
<line x1="706" y1="64" x2="746" y2="64" stroke-dasharray="3,3"/>
</g>
<polygon points="746,60 754,64 746,68" fill="#71717a" opacity="0.6"/>
<!-- dropped -->
<g transform="translate(760, 42)">
<rect width="116" height="44" rx="7" fill="#27272a" stroke="#52525b" stroke-width="1" stroke-opacity="0.7"/>
<text x="58" y="27" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">dropped</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 3.0 KiB

@@ -0,0 +1,74 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
</defs>
<rect width="900" height="170" fill="url(#bg)" rx="10"/>
<text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PUBLIC SUBNET · 1:1 NAT managed by OVN</text>
<!-- OUTBOUND row -->
<text x="24" y="52" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700">Outbound</text>
<g transform="translate(112, 38)">
<rect width="180" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">Instance</text>
<text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">10.0.1.5</text>
</g>
<g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="292" y1="58" x2="352" y2="58" stroke-dasharray="3,3"/>
</g>
<polygon points="352,54 360,58 352,62" fill="#3fb950" opacity="0.7"/>
<g transform="translate(368, 38)">
<rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">SNAT</text>
</g>
<g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="498" y1="58" x2="558" y2="58" stroke-dasharray="3,3"/>
</g>
<polygon points="558,54 566,58 558,62" fill="#3fb950" opacity="0.7"/>
<g transform="translate(574, 38)">
<rect width="272" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">exits as 203.0.113.10</text>
<text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">source IP rewritten</text>
</g>
<!-- INBOUND row -->
<text x="24" y="110" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" font-weight="700">Inbound</text>
<g transform="translate(112, 96)">
<rect width="180" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">WAN client</text>
<text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">→ 203.0.113.10</text>
</g>
<g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="292" y1="116" x2="352" y2="116" stroke-dasharray="3,3"/>
</g>
<polygon points="352,112 360,116 352,120" fill="#58a6ff" opacity="0.7"/>
<g transform="translate(368, 96)">
<rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
<text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">DNAT</text>
</g>
<g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
<line x1="498" y1="116" x2="558" y2="116" stroke-dasharray="3,3"/>
</g>
<polygon points="558,112 566,116 558,120" fill="#58a6ff" opacity="0.7"/>
<g transform="translate(574, 96)">
<rect width="272" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">arrives at 10.0.1.5</text>
<text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">dest IP rewritten</text>
</g>
<!-- Footnote -->
<text x="24" y="158" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a">The instance OS only sees its private IP (10.0.1.5) — the public address is managed entirely by OVN's dnat_and_snat rule.</text>
</svg>

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 437 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 378 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 396 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 525 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 490 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 754 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 406 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 530 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 544 KiB

+144
View File
@@ -0,0 +1,144 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
<defs>
<linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
<stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
</linearGradient>
<linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
<stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
</linearGradient>
<linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
<stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
</linearGradient>
<linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
<stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
</linearGradient>
</defs>
<rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
<!-- Subtle mesh dots background -->
<g opacity="0.12" fill="#7F1D1D">
<circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
<circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
<circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
<circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
<circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
<circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
</g>
<!-- Title -->
<text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
<text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
<!-- Layer 4: Applications (top) -->
<g transform="translate(60, 95)">
<rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
<text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
<g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
<rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
<rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
<rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
<rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
<rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
<rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
<text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
</g>
</g>
<!-- Down arrow -->
<g transform="translate(450, 188)" opacity="0.45">
<line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
<polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
</g>
<!-- Layer 3: Spinifex (highlighted) -->
<g transform="translate(60, 215)">
<rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
<text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
<text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
<g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
<!-- EC2 -->
<rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
<text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
<text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
<text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
<!-- EBS -->
<rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
<text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
<text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
<text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
<!-- S3 -->
<rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
<text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
<text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
<text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
<!-- VPC -->
<rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
<text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
<text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
<text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
<!-- IAM -->
<rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
<text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
<text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
<text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
<!-- Control -->
<rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
<text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
<text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
<text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
</g>
</g>
<!-- Down arrow -->
<g transform="translate(450, 360)" opacity="0.45">
<line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
<polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
</g>
<!-- Layer 2: OS -->
<g transform="translate(60, 387)">
<rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
<text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
<text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
</g>
<!-- Down arrow -->
<g transform="translate(450, 454)" opacity="0.45">
<line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
<polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
</g>
<!-- Layer 1: Hardware -->
<g transform="translate(60, 481)">
<rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
<text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 9.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 151 KiB

+52
View File
@@ -0,0 +1,52 @@
version: 2
updates:
- package-ecosystem: gomod
target-branch: dev
directory: /
schedule:
interval: weekly
day: monday
timezone: Australia/Sydney
cooldown:
default-days: 3
groups:
go-dependencies:
patterns:
- "*"
- package-ecosystem: github-actions
target-branch: dev
directory: /
schedule:
interval: weekly
day: monday
timezone: Australia/Sydney
cooldown:
default-days: 3
ignore:
- dependency-name: "*"
update-types:
- "version-update:semver-minor"
- "version-update:semver-patch"
groups:
github-actions:
patterns:
- "*"
- package-ecosystem: npm
target-branch: dev
directory: /spinifex/services/spinifexui/frontend
schedule:
interval: weekly
day: monday
timezone: Australia/Sydney
cooldown:
default-days: 3
ignore:
- dependency-name: "@types/node"
update-types:
- "version-update:semver-major"
groups:
npm-dependencies:
patterns:
- "*"
+80
View File
@@ -0,0 +1,80 @@
name: E2E Bare-Metal
on:
workflow_dispatch:
permissions:
contents: read
statuses: write
env:
GIT_BRANCH: ${{ github.ref_name }}
jobs:
e2e_baremetal:
name: Bare-Metal Single-Node E2E
runs-on: [ self-hosted, ci-baremetal ]
timeout-minutes: 60
concurrency:
group: e2e-baremetal-cluster
cancel-in-progress: false
env:
BM_ISO_CACHE: /var/cache/spinifex-iso-builder
steps:
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/checkout@v6
- uses: actions/create-github-app-token@v3
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v6
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: |
scripts/iso-builder
scripts/baremetal
path: mulga
- name: Install SSH Key
run: |
install -m 0700 -d ~/.ssh
echo "${{ secrets.BM_SSH_PRIVATE_KEY }}" > ~/.ssh/bm-ci
chmod 0600 ~/.ssh/bm-ci
- name: Build Artifacts
run: |
mulga/scripts/baremetal/bm-build-artifacts.sh \
"$GITHUB_WORKSPACE" \
"${RUNNER_TEMP}/spinifex-tests.tar" \
"${RUNNER_TEMP}/bm-bins"
- name: Build PXE Artifacts
run: |
sudo make -C mulga/scripts/iso-builder \
build-rootfs build-squashfs build-initramfs \
CACHE="${BM_ISO_CACHE}" \
SPINIFEX_BIN="${RUNNER_TEMP}/bm-bins/spx" \
INSTALLER_BIN="${RUNNER_TEMP}/bm-bins/spinifex-installer" \
SPINIFEX_SRC="${GITHUB_WORKSPACE}" \
VIPERBLOCK_SRC="${RUNNER_TEMP}/bm-bins/viperblock" \
VERSION="$(cd "$GITHUB_WORKSPACE" && git describe --tags --always 2>/dev/null || echo dev)"
- name: Run Bare-Metal E2E
env:
BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
BM_SSH_KEY: /home/tf-user/.ssh/bm-ci
BM_SKIP_EPDU: "1"
SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
run: |
sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
--vmlinuz "${BM_ISO_CACHE}/vmlinuz" \
--initrd "${BM_ISO_CACHE}/initrd.img" \
--squashfs "${BM_ISO_CACHE}/spinifex.squashfs" \
--tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+203
View File
@@ -0,0 +1,203 @@
name: E2E DDIL
# Phase 1 trigger: workflow_dispatch only. Nightly cron + on-main-merge
# land in mulga-siv-5.8 once ≥4 scenarios are asserting. See
# docs/development/improvements/ddil-e2e-test-harness.md §4 / §Dependencies.
on:
workflow_dispatch:
inputs:
quarantined:
description: "Comma-separated scenario letters to quarantine (A..F)"
required: false
default: ""
permissions:
contents: read
env:
GIT_BRANCH: ${{ github.ref_name }}
jobs:
ddil:
name: DDIL Real Multi-Node
runs-on: [ self-hosted, ci-multi ]
# Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
# In Phase 1 the scenarios all t.Skip, so the real-world runtime is
# dominated by provisioning; the 60m ceiling leaves headroom for
# Phase 2/3 when assertions come online.
timeout-minutes: 60
env:
TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
steps:
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
# Full spinifex checkout: the Go harness must run on the runner
# (outside the cluster) so its SSH path survives scenario partitions
# that drop peer-to-peer cluster traffic. Running inside the cluster
# would self-sever when Scenario C isolates a node.
- uses: actions/checkout@v6
name: Checkout Spinifex
with:
path: spinifex
- uses: actions/checkout@v6
name: Checkout Mulga (tofu-cluster)
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Install gotestsum
env:
GOTESTSUM_VERSION: "1.12.0"
run: |
# Prebuilt tarball, not `go install`: gotestsum v1.12.0 pulls
# x/tools v0.19.0 whose tokeninternal static-assert pegs a
# narrow Go window (~1.22). The self-hosted runner's Go falls
# outside that window, so source-install fails with
# `invalid array length -delta * delta`. A prebuilt binary
# sidesteps toolchain coupling entirely.
BIN_DIR="${RUNNER_TEMP}/gotestsum-bin"
mkdir -p "${BIN_DIR}"
if [ ! -x "${BIN_DIR}/gotestsum" ]; then
curl -fsSL \
"https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz" \
| tar -xz -C "${BIN_DIR}" gotestsum
fi
echo "${BIN_DIR}" >> "${GITHUB_PATH}"
"${BIN_DIR}/gotestsum" --version
- name: Build Install Artifacts
working-directory: mulga/scripts/tofu-cluster
run: |
./build-install-artifacts.sh "${GIT_BRANCH}" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-tests.tar"
- name: Provision VMs
working-directory: mulga/scripts/tofu-cluster
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: |
SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
- name: Run DDIL Scenarios
working-directory: mulga/scripts/tofu-cluster
run: |
set -o pipefail
INVENTORY=$(tofu output -json inventory)
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
REGION=$(echo "$INVENTORY" | jq -r '.region')
# DDIL_NODES must be the cluster-traffic IPs: PartitionNode
# installs iptables DROP rules for each peer, so the runner's
# SSH source IP must differ from these. On single-NIC
# tofu-cluster, wan_ip == cluster IP and the runner's outbound
# IP is distinct — see e2e-ddil-test-harness.md §3 Scenario C.
NODE_IPS=$(echo "$INVENTORY" | jq -r '[.nodes[].wan_ip] | join(",")')
PRIMARY_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
echo "DDIL_NODES=${NODE_IPS}"
# Witness VMs talk to the cluster's AWS gateway via aws-sdk-go,
# which resolves credentials from the standard chain. spx admin
# init writes [spinifex] to ${SSH_USER}'s ~/.aws/credentials on
# node1 — pull them over SSH and export as the SDK's env-var
# entries so the harness can launch instances without depending
# on a cluster-bootstrapped profile being present on the runner.
CREDS=$(ssh -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"${SSH_USER}@${PRIMARY_IP}" 'cat ~/.aws/credentials')
AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
echo "::error::Failed to extract spinifex AWS credentials from ${PRIMARY_IP}"
exit 1
fi
# Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
# the matching private material was generated on the primary node at
# ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
# the witness via its MapPublicIpOnLaunch-assigned EIP.
GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
chmod 600 "${GUEST_KEY}"
mkdir -p "${RUNNER_TEMP}/ddil-results"
cd "${GITHUB_WORKSPACE}/spinifex"
# gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout
# while still emitting the machine-readable jsonfile artifact.
DDIL_NODES="${NODE_IPS}" \
DDIL_SSH_USER="${SSH_USER}" \
DDIL_SSH_KEY="${SSH_KEY}" \
DDIL_GUEST_SSH_KEY="${GUEST_KEY}" \
DDIL_QUARANTINED="${{ inputs.quarantined }}" \
AWS_REGION="${REGION}" \
AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
--format testname \
-- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
- name: Collect Cluster Logs
if: failure()
working-directory: mulga/scripts/tofu-cluster
run: |
# Pull spinifex-* journals + QEMU process listings from every
# node into the artifact bundle so failures can be triaged
# without re-provisioning the cluster. Best-effort — a missing
# node should not mask the original test failure.
set +e
INVENTORY=$(tofu output -json inventory)
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
mkdir -p "${LOG_DIR}"
mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
for ip in "${NODE_IPS[@]}"; do
echo "=== collecting ${ip} ==="
SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
done
- name: Upload DDIL Results
if: always()
uses: actions/upload-artifact@v7
with:
name: ddil-results-${{ github.run_id }}
path: ${{ runner.temp }}/ddil-results
retention-days: 30
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+219
View File
@@ -0,0 +1,219 @@
name: E2E Nightly
on:
schedule:
- cron: "0 14 * * 0-4" # 00:00 Mon-Fri AEST = 14:00 Sun-Thu UTC
workflow_dispatch:
inputs:
cells:
description: "Comma-separated cell numbers to run (blank = all)"
required: false
permissions:
contents: read
# Scheduled runs target dev (active development trunk). GitHub loads scheduled
# workflow files from the repo's default branch, so this file must be kept in
# sync on main for the schedule to use it.
env:
GIT_BRANCH: dev
UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-24.04.qcow2
jobs:
matrix:
name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{
matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
strategy:
fail-fast: false
max-parallel: 3
matrix:
include:
- { cell: 1, install: binary, topo: single, net: dhcp, host: debian-13, arch: x86_64, runner: ci-single, extra: "" }
- { cell: 2, install: binary, topo: single, net: static, host: ubuntu-24.04, arch: x86_64, runner: ci-single, extra: "" }
- { cell: 3, install: binary, topo: single, net: dhcp, host: ubuntu-24.04, arch: x86_64, runner: ci-single, extra: "" }
- { cell: 8, install: binary, topo: real-multi, net: dhcp, host: debian-13, arch: x86_64, runner: ci-multi, extra: "" }
- { cell: 9, install: binary, topo: real-multi, net: static, host: ubuntu-24.04, arch: x86_64, runner: ci-multi, extra: "" }
- { cell: 10, install: binary, topo: real-multi, net: dhcp, host: ubuntu-24.04, arch: x86_64, runner: ci-multi, extra: "" }
- { cell: 17, install: binary, topo: single, net: static, host: debian-13, arch: x86_64, runner: ci-pseudo, extra: "tofu-examples" }
- { cell: 18, install: binary, topo: single, net: static, host: debian-13, arch: x86_64, runner: ci-single, extra: "reboot-single" }
runs-on: [ self-hosted, "${{ matrix.runner }}" ]
timeout-minutes: 35
env:
TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
steps:
# blank or scheduled run = full matrix.
# Input must be comma-separated with no whitespace (e.g. '1,2,17', not '1, 2').
- name: Gate on Dispatch Input
id: gate
env:
CELL: ${{ matrix.cell }}
CELLS: ${{ github.event_name == 'workflow_dispatch' && inputs.cells || '' }}
run: |
if [ -z "$CELLS" ] || [[ ",${CELLS}," == *",${CELL},"* ]]; then
echo "selected=true" >> "$GITHUB_OUTPUT"
echo "cell ${CELL} selected"
else
echo "selected=false" >> "$GITHUB_OUTPUT"
echo "cell ${CELL} not in '${CELLS}' — subsequent steps will skip"
fi
- name: Clean Workspace
if: steps.gate.outputs.selected == 'true'
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
if: steps.gate.outputs.selected == 'true'
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v6
if: steps.gate.outputs.selected == 'true'
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Build Install Artifacts
if: steps.gate.outputs.selected == 'true'
working-directory: mulga/scripts/tofu-cluster
run: |
./build-install-artifacts.sh "${GIT_BRANCH}" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-tests.tar"
- name: Provision VMs
if: steps.gate.outputs.selected == 'true'
working-directory: mulga/scripts/tofu-cluster
env:
MATRIX_HOST: ${{ matrix.host }}
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
TOFU_VAR_ARGS=""
if [ "$MATRIX_HOST" = "ubuntu-24.04" ]; then
TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
fi
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap via Binary Install
if: steps.gate.outputs.selected == 'true'
working-directory: mulga/scripts/tofu-cluster
env:
MATRIX_NET: ${{ matrix.net }}
run: |
if [ "$MATRIX_NET" = "dhcp" ]; then
unset SPX_EXTERNAL_POOL
export SPX_EXTERNAL_SOURCE=dhcp
else
export SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}"
fi
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
- name: Run Single Node E2E
if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory)
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
"cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
- name: Run Multi Node E2E
if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory)
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
NODE_IPS=""
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
for i in $(seq 0 $((NODE_COUNT - 1))); do
NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
done
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
"cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
- name: Run Tofu Workbooks
if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory)
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
"cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
- name: Run Reboot Resilience E2E
if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
working-directory: mulga/scripts/tofu-cluster
# Driver runs from the runner (NOT via ssh-and-run) so it survives the
# systemctl-reboot-induced ssh disconnect mid-run.
run: |
INVENTORY=$(tofu output -json inventory)
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
mkdir -p "$REBOOT_DIR"
tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
- name: Dump Cluster Journals on Failure
if: failure() && steps.gate.outputs.selected == 'true'
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
for i in $(seq 0 $((NODE_COUNT - 1))); do
IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
echo "===== node$((i+1)) ($IP) ====="
for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
echo "--- $svc ---"
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
"sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
done
echo "--- DHCP client state ---"
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
"systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
done
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
env:
MATRIX_HOST: ${{ matrix.host }}
run: |
TOFU_VAR_ARGS=""
if [ "$MATRIX_HOST" = "ubuntu-24.04" ]; then
TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
fi
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+114 -45
View File
@@ -1,4 +1,4 @@
name: Hive E2E Tests
name: E2E
on:
push:
@@ -13,6 +13,14 @@ on:
- tests/e2e/**
- .github/workflows/e2e.yml
workflow_dispatch:
inputs:
sha:
description: "PR head SHA to report status on"
required: false
permissions:
contents: read
statuses: write
env:
GIT_BRANCH: ${{ github.ref_name }}
@@ -20,18 +28,18 @@ env:
jobs:
e2e_single:
name: Single-Node E2E
runs-on: [self-hosted, ci-single]
timeout-minutes: 25
runs-on: [ self-hosted, ci-single ]
timeout-minutes: 35
env:
TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
steps:
- name: Clean workspace
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v2
- uses: actions/create-github-app-token@v3
id: app-token
with:
app-id: ${{ secrets.APP_ID }}
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
@@ -42,19 +50,32 @@ jobs:
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Build Install Artifacts
working-directory: mulga/scripts/tofu-cluster
run: |
./build-install-artifacts.sh "${GIT_BRANCH}" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-tests.tar"
- name: Provision VM
working-directory: mulga/scripts/tofu-cluster
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap Cluster
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: ./bootstrap.sh env3 >> "$TOFU_LOG" 2>&1
run: |
SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
- name: Run E2E Tests
working-directory: mulga/scripts/tofu-cluster
@@ -64,30 +85,30 @@ jobs:
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
'cd ~/Development/mulga/hive && ./tests/e2e/run-e2e.sh'
"cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
e2e_pseudo_multi:
name: Pseudo Multi-Node E2E
runs-on: [self-hosted, ci-pseudo]
runs-on: [ self-hosted, ci-pseudo ]
timeout-minutes: 25
env:
TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
steps:
- name: Clean workspace
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v2
- uses: actions/create-github-app-token@v3
id: app-token
with:
app-id: ${{ secrets.APP_ID }}
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
@@ -96,22 +117,29 @@ jobs:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Clone and Build
working-directory: mulga/scripts/tofu-cluster
run: ./build-artifacts.sh "${GIT_BRANCH}"
"${RUNNER_TEMP}/spinifex-artifacts.tar"
- name: Provision VM
working-directory: mulga/scripts/tofu-cluster
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap Cluster
working-directory: mulga/scripts/tofu-cluster
run: ./bootstrap.sh env4 >> "$TOFU_LOG" 2>&1
run: |
SPX_PREBUILT=true \
SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
./bootstrap.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
- name: Run E2E Tests
working-directory: mulga/scripts/tofu-cluster
@@ -121,30 +149,30 @@ jobs:
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
'cd ~/Development/mulga/hive && ./tests/e2e/run-pseudo-multinode-e2e.sh'
"cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL=\"${SPINIFEX_CI_POOL}\" ./tests/e2e/run-pseudo-multinode-e2e.sh"
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
e2e_real_multi:
name: Real Multi-Node E2E
runs-on: [self-hosted, ci-multi]
timeout-minutes: 25
runs-on: [ self-hosted, ci-multi ]
timeout-minutes: 30
env:
TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
steps:
- name: Clean workspace
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v2
- uses: actions/create-github-app-token@v3
id: app-token
with:
app-id: ${{ secrets.APP_ID }}
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
@@ -155,19 +183,32 @@ jobs:
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Build Install Artifacts
working-directory: mulga/scripts/tofu-cluster
run: |
./build-install-artifacts.sh "${GIT_BRANCH}" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-tests.tar"
- name: Provision VMs
working-directory: mulga/scripts/tofu-cluster
run: |
export TF_VAR_git_branch="${GIT_BRANCH}"
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap Cluster
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: ./bootstrap.sh env2 >> "$TOFU_LOG" 2>&1
run: |
SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
"${RUNNER_TEMP}/setup.sh" \
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
- name: Run E2E Tests
working-directory: mulga/scripts/tofu-cluster
@@ -184,13 +225,41 @@ jobs:
NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
done
NODE2_IP=$(echo "$INVENTORY" | jq -r '.nodes[1].wan_ip')
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
"cd ~/Development/mulga/hive && ./tests/e2e/run-multinode-e2e.sh ${NODE_IPS}"
"cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
e2e_status:
name: Report E2E Status
if: always() && inputs.sha
needs: [ e2e_single, e2e_pseudo_multi, e2e_real_multi ]
runs-on: [ self-hosted, ci-single ]
timeout-minutes: 5
steps:
- name: Report Status to PR
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "${{ needs.e2e_single.result }}" == "success" && \
"${{ needs.e2e_pseudo_multi.result }}" == "success" && \
"${{ needs.e2e_real_multi.result }}" == "success" ]]; then
STATE="success"
DESC="All E2E suites passed"
else
STATE="failure"
DESC="E2E failed: single=${{ needs.e2e_single.result }} pseudo=${{ needs.e2e_pseudo_multi.result }} multi=${{ needs.e2e_real_multi.result }}"
fi
gh api "repos/${{ github.repository }}/statuses/${{ inputs.sha }}" \
-f state="$STATE" \
-f context=e2e \
-f description="$DESC" \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+168 -18
View File
@@ -2,43 +2,193 @@ name: Release
on:
push:
tags: ['v*']
tags: [ "v*" ]
permissions:
contents: write
jobs:
distro:
runs-on: ubuntu-latest
distro-amd64:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: hive
path: spinifex
- uses: actions/checkout@v4
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- uses: actions/checkout@v4
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Set up QEMU (for arm64 cross-build)
uses: docker/setup-qemu-action@v3
- name: Build amd64 Tarball
env:
VERSION: ${{ github.ref_name }}
run: cd spinifex && make distro-amd64 VERSION="$VERSION"
- name: Build distribution tarballs
run: cd hive && make distro VERSION=${{ github.ref_name }}
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
- name: Upload amd64 Artifacts
uses: actions/upload-artifact@v7
with:
name: distro-amd64
path: |
spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz
spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz.sha256
spinifex/scripts/setup.sh
distro-arm64:
runs-on: ubuntu-24.04-arm
steps:
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: spinifex
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Build arm64 Tarball
env:
VERSION: ${{ github.ref_name }}
run: cd spinifex && make distro-arm64 VERSION="$VERSION"
- name: Upload arm64 Artifacts
uses: actions/upload-artifact@v7
with:
name: distro-arm64
path: |
spinifex/dist/spinifex-${{ github.ref_name }}-linux-arm64.tar.gz
spinifex/dist/spinifex-${{ github.ref_name }}-linux-arm64.tar.gz.sha256
release:
needs: [ distro-amd64, distro-arm64 ]
runs-on: ubuntu-24.04
steps:
- name: Download amd64 Artifacts
uses: actions/download-artifact@v8
with:
name: distro-amd64
path: artifacts
- name: Download arm64 Artifacts
uses: actions/download-artifact@v8
with:
name: distro-arm64
path: artifacts
- name: Create Draft GitHub Release
uses: softprops/action-gh-release@v3
with:
draft: true
files: |
hive/dist/*.tar.gz
hive/dist/*.sha256
hive/scripts/setup.sh
artifacts/**/*.tar.gz
artifacts/**/*.sha256
artifacts/**/setup.sh
generate_release_notes: true
- name: Upload Tarball for Verify Job
uses: actions/upload-artifact@v7
with:
name: release-assets
path: artifacts/
verify:
needs: release
runs-on: [ self-hosted, ci-single ]
timeout-minutes: 25
env:
TOFU_LOG: /tmp/release-verify-${{ github.run_id }}.log
TAG: ${{ github.ref_name }}
REPO: ${{ github.repository }}
steps:
- name: Clean Workspace
run: rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v6
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- name: Download Release Assets
uses: actions/download-artifact@v8
with:
name: release-assets
path: release-assets
- name: Provision VM
working-directory: mulga/scripts/tofu-cluster
run: |
tofu init -input=false >> "$TOFU_LOG" 2>&1
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
- name: Bootstrap via Binary Install
working-directory: mulga/scripts/tofu-cluster
run: |
SETUP_SH=$(find "${GITHUB_WORKSPACE}/release-assets" -name "setup.sh" | head -1)
TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" "$SETUP_SH" "$TARBALL"
- name: Promote Release
if: success()
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "$TAG" =~ -(dev|rc|alpha|beta) ]]; then
gh release edit "$TAG" --repo "$REPO" --draft=false --prerelease
echo "Release ${TAG} promoted as prerelease"
else
gh release edit "$TAG" --repo "$REPO" --draft=false
echo "Release ${TAG} promoted to latest"
fi
- name: Mark Release Failed
if: failure()
env:
GH_TOKEN: ${{ github.token }}
run: |
EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
NOTES_FILE=$(mktemp)
printf 'Verification failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
rm -f "$NOTES_FILE"
- name: Teardown
if: always()
working-directory: mulga/scripts/tofu-cluster
run: |
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
@@ -1,4 +1,4 @@
name: Hive Build and Test
name: Build and Test
on:
push:
@@ -10,19 +10,26 @@ on:
- go.mod
- go.sum
- Makefile
- .github/workflows/hive.yml
- .github/workflows/spinifex.yml
pull_request:
branches:
- main
- dev
workflow_dispatch:
permissions:
contents: read
jobs:
build_and_test:
name: Hive Build and Test
name: Build and Test
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: hive
path: spinifex
fetch-depth: 2
- name: Checkout Viperblock
@@ -40,30 +47,30 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
run: go work init ./spinifex ./viperblock ./predastore
- name: Build Hive
run: make -C hive build
- name: Build Spinifex
run: make -C spinifex build
- name: Run Tests with Coverage
run: make -C hive test-cover
run: make -C spinifex test-cover
- name: Check Diff Coverage
run: cd hive && scripts/diff-coverage.sh coverage.out --base HEAD~1
run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
race_detection:
name: Hive Race Detection
name: Race Detection
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: hive
path: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v6
@@ -80,24 +87,24 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
run: go work init ./spinifex ./viperblock ./predastore
- name: Run Tests with Race Detector
run: make -C hive test-race
run: make -C spinifex test-race
lint_and_security:
name: Hive Lint and Security
name: Lint and Security
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
- name: Checkout Spinifex
uses: actions/checkout@v6
with:
path: hive
path: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v6
@@ -114,20 +121,17 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
run: go work init ./spinifex ./viperblock ./predastore
- name: Check Format
run: make -C hive check-format
- name: Lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.11.3
working-directory: spinifex
- name: Check Modernize
run: make -C hive check-modernize
- name: Go Vet
run: make -C hive vet
- name: Security Checks
run: make -C hive security-check
- name: Govulncheck
run: make -C spinifex govulncheck
+19 -9
View File
@@ -6,41 +6,51 @@ on:
- main
- dev
paths:
- hive/services/hiveui/frontend/**
- spinifex/services/spinifexui/frontend/**
- .github/workflows/ui.yml
pull_request:
branches:
- main
- dev
paths:
- spinifex/services/spinifexui/frontend/**
- .github/workflows/ui.yml
workflow_dispatch:
permissions:
contents: read
jobs:
ui_build_and_test:
name: UI Build and Test
runs-on: ubuntu-24.04
defaults:
run:
working-directory: hive/services/hiveui/frontend
working-directory: spinifex/services/spinifexui/frontend
steps:
- name: Checkout Code
uses: actions/checkout@v6
with:
sparse-checkout: hive/services/hiveui/frontend
sparse-checkout: spinifex/services/spinifexui/frontend
- name: Setup pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v6
with:
package_json_file: hive/services/hiveui/frontend/package.json
package_json_file: spinifex/services/spinifexui/frontend/package.json
- name: Setup Node.js
uses: actions/setup-node@v6
with:
cache: "pnpm"
cache-dependency-path: hive/services/hiveui/frontend/pnpm-lock.yaml
node-version-file: hive/services/hiveui/frontend/.nvmrc
cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml
node-version-file: spinifex/services/spinifexui/frontend/.nvmrc
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Run Biome Check
run: pnpm check
- name: Run Lint
run: pnpm lint
- name: Run Tests with Coverage
run: pnpm test:coverage
+6 -1
View File
@@ -7,6 +7,9 @@
!example.tfvars
scripts/iac/proxmox/clusters/
# Generated SSH private keys from terraform examples (e.g. local_file.nginx_pem)
*.pem
# Go workspace
go.work
go.work.sum
@@ -14,6 +17,7 @@ go.work.sum
# Build outputs
bin/
dist/
!spinifex/services/spinifexui/frontend/dist/
tmp/
*.test
*.out
@@ -27,6 +31,7 @@ Thumbs.db
# Environment
.env
.venv/
# Test reports
*-report.txt
@@ -35,8 +40,8 @@ Thumbs.db
docs/INTERNAL.md
docs/PRIVATE.md
docs/DEVNOTES.md
docs/HIVE_DEVELOPMENT_PLAN.md
docs/PLAN.md
docs/TODO.md
docs/development/
docs/internal/
scripts/iac/aws/spinifex-test.pem
+136
View File
@@ -0,0 +1,136 @@
version: "2"
formatters:
enable:
- gofmt
settings:
gofmt:
simplify: false # modernize linter covers gofmt -s simplifications (Go 1.23+)
linters:
enable:
- asasalint # catches pass []any as any in variadic func(...any)
- asciicheck # checks that code does not contain non-ASCII identifiers
- bidichk # checks for dangerous unicode bidi character sequences
- bodyclose # checks whether HTTP response body is closed successfully
- canonicalheader # checks whether net/http.Header uses canonical header keys
- decorder # checks declaration order of types, consts, vars, and funcs
- durationcheck # checks for two durations multiplied together
- embeddedstructfieldcheck # checks embedded sync.Mutex/RWMutex in structs
- errcheck # checks for unchecked errors in type assertions and I/O
- errchkjson # checks types passed to json encoding functions, flags omittable error checks
- errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
- exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
- fatcontext # detects nested contexts in loops
- forcetypeassert # flags unchecked type assertions that can panic
- gocheckcompilerdirectives # validates go compiler directive comments (//go:)
- goprintffuncname # checks that printf-like functions are named with f at the end
- gosec # inspects source code for security problems
- govet # reports suspicious constructs (Printf alignment, shadow, etc.)
- ineffassign # detects assignments to existing variables that are not used
- makezero # finds slice declarations with non-zero initial length followed by append
- mirror # catches wrong mirror patterns in bytes/strings usage
- musttag # checks structs passed to marshal/unmarshal have proper tags
- modernize # modernizes code to use newer Go idioms (go fix analyzers)
- nakedret # finds naked returns in functions
- nilerr # catches code that returns nil even after checking err != nil
- nilnesserr # catches returning a different nil error variable after err != nil check
- nolintlint # reports ill-formed or insufficient nolint directives
- nosprintfhostport # flags fmt.Sprintf host:port — use net.JoinHostPort for IPv6 safety
- perfsprint # detects fmt.Sprintf calls replaceable with faster alternatives
- predeclared # catches shadowing of Go builtins (len, cap, error, etc.)
- reassign # checks that package variables are not reassigned
- recvcheck # checks for receiver type consistency (pointer vs value)
- sloglint # enforces consistent log/slog usage (key-value pairs, no globals)
- staticcheck # extensive static analysis checks
- unconvert # removes unnecessary type conversions
- usestdlibvars # detects hardcoded values that should use stdlib constants
- usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
- wastedassign # finds assignments to variables that are never subsequently read
- whitespace # detects leading and trailing whitespace in functions
settings:
embeddedstructfieldcheck:
forbid-mutex: true
errcheck:
check-type-assertions: true
exclude-functions:
- (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
- (*github.com/kdomanski/iso9660.ImageWriter).Cleanup # deferred cleanup, errors not actionable
govet:
enable-all: true
disable:
- fieldalignment
- shadow # too noisy to retrofit — all 50 findings are err re-declarations
- unusedwrite # false positives on struct field assignments before marshal/send
staticcheck:
checks:
- all
- -ST1000 # package comment
- -ST1003 # naming conventions (Id→ID etc.) — too noisy to retrofit
- -ST1012 # error var naming — errname covers this + error type suffixes
- -ST1013 # HTTP status constants — usestdlibvars is broader
- -ST1016 # receiver name consistency (too noisy to retrofit)
- -ST1020 # godoc on exported types — not enforcing yet
- -ST1021 # godoc on exported type groups — not enforcing yet
- -ST1022 # godoc on exported vars/consts — not enforcing yet
- -SA1019 # deprecated usage (handle separately)
- -SA4006 # value overwritten — ineffassign + wastedassign cover this better
- -SA9005 # ineffective break (staticcheck false positives)
gosec:
excludes:
- G104 # unchecked errors — errcheck is stricter
- G116 # bidi chars — bidichk is more thorough
- G117 # use of net/http serve functions (we handle TLS separately)
- G703 # path traversal taint analysis — false positives on config file loading
- G705 # XSS taint analysis — false positives on internal error responses
- G706 # log injection taint analysis — false positives on structured slog usage
modernize:
disable:
- newexpr # don't replace aws.String() with new() — AWS SDK idiom
- stringsbuilder # don't replace small string concatenations with strings.Builder
perfsprint:
concat-loop: false
err-error: false # don't flag err.Error()
errorf: false # don't convert fmt.Errorf to errors.New
strconcat: false
nakedret:
max-func-lines: 0
nolintlint:
require-explanation: true
require-specific: true
reassign:
patterns:
- ".*"
exclusions:
presets:
- std-error-handling
- common-false-positives
rules:
- path: '_test\.go'
linters:
- bodyclose
- errcheck
- errchkjson
- gosec
# spinifex-specific: gosec exclusions
- path: "cmd/"
linters:
- errcheck # CLI glue code — viper/cobra/pterm errors not actionable
- gosec # matches current -exclude-dir=cmd
- text: "G204"
linters: [gosec] # subprocess execution — legitimate in VM orchestration
- text: "G304"
linters: [gosec] # file path from variable — necessary for config loading
+2 -2
View File
@@ -1,3 +1,3 @@
# Hive Installation
# Spinifex Installation
For documentation on using and installing Hive see [https://docs.mulgadc.com](https://docs.mulgadc.com/)
For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)
+629 -169
View File
@@ -1,201 +1,661 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
1. Definitions.
Preamble
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
The precise terms and conditions for copying, distribution and
modification follow.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
TERMS AND CONDITIONS
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
0. Definitions.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
"This License" refers to version 3 of the GNU Affero General Public License.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
A "covered work" means either the unmodified Program or a work based
on the Program.
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
1. Source Code.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
END OF TERMS AND CONDITIONS
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
APPENDIX: How to apply the Apache License to your work.
The Corresponding Source for a work in source code form is that
same work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
2. Basic Permissions.
Copyright [yyyy] [name of copyright owner]
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
http://www.apache.org/licenses/LICENSE-2.0
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
+109 -97
View File
@@ -1,4 +1,4 @@
GO_PROJECT_NAME := hive
GO_PROJECT_NAME := spx
SHELL := /bin/bash
# Detect architecture for cross-platform support
@@ -34,63 +34,80 @@ else
endif
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
# propagates through the filter — otherwise a test failure is swallowed by
# grep's own (success) exit code and preflight prints "passed" on red.
ifdef QUIET
_Q = @
_COVQ = 2>&1 | grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
_SECQ = >
else
_Q =
_COVQ = || true
_COVQ =
_RACEQ =
_SECQ = 2>&1 | tee
endif
build:
$(MAKE) go_build
build: go_build build-installer build-lb-agent
# Build hive-ui frontend (requires pnpm)
# Build spinifex-ui frontend (requires pnpm)
build-ui:
@echo -e "\n....Building hive-ui frontend...."
cd hive/services/hiveui/frontend && pnpm build
@echo -e "\n....Building spinifex-ui frontend...."
cd spinifex/services/spinifexui/frontend && pnpm build
# GO commands
VERSION ?= $(shell git describe --tags --always --dirty)
COMMIT ?= $(shell git rev-parse --short HEAD)
LDFLAGS := -s -w -X github.com/mulgadc/hive/cmd/hive/cmd.Version=$(VERSION) -X github.com/mulgadc/hive/cmd/hive/cmd.Commit=$(COMMIT)
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
go_build:
@echo -e "\n....Building $(GO_PROJECT_NAME)"
go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/hive/main.go
go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
build-installer:
@echo -e "\n....Building spinifex-installer"
go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
build-lb-agent:
@echo -e "\n....Building lb-agent (static)"
CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
build-system-image: ## Build a system image from manifest (use IMAGE=lb)
ifndef IMAGE
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
endif
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
build-lb-image: ## Build LB Alpine image
$(MAKE) build-system-image IMAGE=lb
go_run:
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
$(GOPATH)/bin/$(GO_PROJECT_NAME)
# Preflight — runs the same checks as GitHub Actions (format + lint + security + tests).
# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
# Use this before committing to catch CI failures locally.
preflight:
@$(MAKE) --no-print-directory QUIET=1 check-format check-modernize vet security-check test-cover diff-coverage test-race
@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
@echo -e "\n ✅ Preflight passed — safe to commit."
# Run unit tests
test:
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -timeout 120s ./hive/...
LOG_IGNORE=1 go test -timeout 120s ./spinifex/...
# Run unit tests with coverage profile
# Note: go test may exit non-zero due to Go version mismatch in coverage instrumentation
# for packages without test files. We check actual test results + coverage threshold instead.
COVERPROFILE ?= coverage.out
test-cover:
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./hive/... $(_COVQ)
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... $(_COVQ)
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
# Run unit tests with race detector
test-race:
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./hive/... $(_RACEQ)
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
# Check that new/changed code meets coverage threshold (runs tests first)
diff-coverage: test-cover
@@ -105,23 +122,36 @@ run:
$(MAKE) go_build
$(MAKE) go_run
# Fast iteration: build + install binary + restart all services
deploy: build
sudo install -m 755 bin/spx /usr/local/bin/spx
sudo systemctl daemon-reload
sudo systemctl restart spinifex.target
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
# Not needed for code-only changes — use deploy for those.
reinstall:
scripts/dev-install.sh
clean:
rm -f ./bin/$(GO_PROJECT_NAME)
rm -rf hive/services/hiveui/frontend/dist
rm -rf spinifex/services/spinifexui/frontend/dist
install-system:
@echo -e "\n....Installing system dependencies for $(ARCH)...."
@echo "QEMU packages: $(QEMU_PACKAGES)"
apt-get update && sudo apt-get install -y \
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch
ovn-central ovn-host openvswitch-switch dhcpcd-base
install-go:
@echo -e "\n....Installing Go 1.26.1 for $(ARCH) ($(GO_ARCH))...."
@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
@if [ ! -d "/usr/local/go" ]; then \
curl -L https://go.dev/dl/go1.26.1.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
else \
echo "Go already installed in /usr/local/go"; \
fi
@@ -143,92 +173,74 @@ quickinstall: install-system install-go install-aws
@echo " Please ensure /usr/local/go/bin is in your PATH."
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
# Format all Go files in place
format:
gofmt -w .
# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
lint:
@echo "Running golangci-lint..."
$(_Q)golangci-lint run ./...
@echo " golangci-lint ok"
# Check that all Go files are formatted (CI-compatible, fails on diff)
check-format:
@echo "Checking gofmt..."
@UNFORMATTED=$$(gofmt -l .); \
if [ -n "$$UNFORMATTED" ]; then \
echo "Files not formatted:"; \
echo "$$UNFORMATTED"; \
echo "Run 'make format' to fix."; \
exit 1; \
fi
@echo " gofmt ok"
# Auto-fix all linter issues that have fixers
fix:
golangci-lint run --fix ./...
# Go vet (fails on issues, matches CI)
vet:
@echo "Running go vet..."
$(_Q)go vet ./...
@echo " go vet ok"
# Excluded: newexpr (replaces aws.String with new, not idiomatic for AWS SDK)
# Excluded: stringsbuilder (replaces string += in loops with strings.Builder, not worth the complexity for small loops)
GOFIX_EXCLUDE := -newexpr=false -stringsbuilder=false
# Apply go fix modernizations
modernize:
@echo "Applying go fix modernizations..."
go fix $(GOFIX_EXCLUDE) ./...
@echo " go fix applied"
# Check that code is modernized (CI-compatible, fails on diff)
check-modernize:
@echo "Checking go fix modernizations..."
@DIFF=$$(go fix $(GOFIX_EXCLUDE) -diff ./...); \
if [ -n "$$DIFF" ]; then \
echo "$$DIFF"; \
echo "Run 'make modernize' to fix."; \
exit 1; \
fi
@echo " go fix ok"
# Security checks — each tool fails the build on findings (matches CI).
# Reports are also saved to tests/ for review.
security-check:
@echo -e "\n....Running security checks for $(GO_PROJECT_NAME)...."
$(_Q)set -o pipefail && go tool govulncheck ./... $(_SECQ) tests/govulncheck-report.txt $(if $(QUIET),|| { cat tests/govulncheck-report.txt; exit 1; })
# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
govulncheck:
@echo "Running govulncheck..."
$(_Q)go tool govulncheck ./...
@echo " govulncheck ok"
$(_Q)set -o pipefail && go tool gosec -quiet -exclude=G204,G304,G402,G117,G703,G705,G706 -exclude-generated -exclude-dir=cmd ./... $(_SECQ) tests/gosec-report.txt $(if $(QUIET),|| { cat tests/gosec-report.txt; exit 1; })
@echo " gosec ok"
$(_Q)set -o pipefail && go tool staticcheck -checks="all,-ST1000,-ST1003,-ST1016,-ST1020,-ST1021,-ST1022,-SA1019,-SA9005" ./... $(_SECQ) tests/staticcheck-report.txt $(if $(QUIET),|| { cat tests/staticcheck-report.txt; exit 1; })
@echo " staticcheck ok"
# Build release tarballs for both architectures via Docker
distro:
@echo "Building hive $(VERSION) distribution tarballs..."
@mkdir -p dist/
@echo "Building linux/amd64..."
docker buildx build \
--platform linux/amd64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/amd64/ \
../
tar -czf dist/hive-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/hive-$(VERSION)-linux-amd64.tar.gz > dist/hive-$(VERSION)-linux-amd64.tar.gz.sha256
@echo "Building linux/arm64..."
docker buildx build \
--platform linux/arm64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/arm64/ \
../
tar -czf dist/hive-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
sha256sum dist/hive-$(VERSION)-linux-arm64.tar.gz > dist/hive-$(VERSION)-linux-arm64.tar.gz.sha256
# Build release tarballs — use distro-ARCH for single arch, distro for both
distro: distro-amd64 distro-arm64
@echo ""
@echo "Distribution tarballs:"
@ls -lh dist/*.tar.gz
@echo ""
@cat dist/*.sha256
distro-amd64:
@echo "Building spinifex $(VERSION) linux/amd64..."
@mkdir -p dist/
docker buildx build \
--platform linux/amd64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/amd64/ \
../
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
distro-arm64:
@echo "Building spinifex $(VERSION) linux/arm64..."
@mkdir -p dist/
docker buildx build \
--platform linux/arm64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/arm64/ \
../
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
distro-clean:
rm -rf dist/
.PHONY: build build-ui go_build go_run preflight test test-cover test-race diff-coverage bench run clean \
# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
ansible-dev-preflight:
cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml
ansible-dev-teardown:
cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml
ansible-dev-install:
cd scripts/ansible && ansible-playbook playbooks/dev-install.yml
ansible-dev-reset:
cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-lb-image go_build go_run preflight test test-cover test-race diff-coverage bench run \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
format check-format modernize check-modernize vet security-check \
distro distro-clean
lint fix govulncheck \
distro distro-amd64 distro-arm64 distro-clean \
ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset
+98 -25
View File
@@ -1,25 +1,54 @@
# Hive: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
<p align="center">
<img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
</p>
(Developer Preview) - Hive developed by [Mulga Defense Corporation](https://mulgadc.com/) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Hive gives you AWS-style workflows on your own hardware.
<p align="center">
<a href="https://go.dev"><img src="https://img.shields.io/badge/Go-1.26+-00ADD8?style=flat-square&logo=go&logoColor=white" alt="Go"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-3fb950?style=flat-square" alt="License"></a>
<a href="https://docs.mulgadc.com"><img src="https://img.shields.io/badge/Docs-docs.mulgadc.com-DC2626?style=flat-square" alt="Docs"></a>
</p>
## What is Hive?
<p align="center">
<a href="#what-is-spinifex">What is Spinifex?</a> ·
<a href="#the-platform">Platform</a> ·
<a href="#core-components">Components</a> ·
<a href="#architecture-at-a-glance">Architecture</a> ·
<a href="#installation">Installation</a> ·
<a href="https://docs.mulgadc.com">Docs</a>
</p>
Hive replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
---
It’s designed for developers and operators who need:
# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
## What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It's designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Hive on a few servers in a rack, a field site, or anywhere centralized cloud services aren’t feasible.
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
## The Platform
<p align="center">
<img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
## Core Components
### Hive (Compute Service – EC2 Alternative)
### Spinifex (Compute Service – EC2 Alternative)
Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU’s QMP interface. Designed to be straightforward and scriptable, Hive lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Hive with underlying instances.
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
@@ -27,7 +56,7 @@ Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs sim
### Viperblock (Block Storage – EBS Alternative)
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It’s built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
@@ -43,54 +72,98 @@ Hive is a minimal VM orchestration layer built on top of QEMU, exposing APIs sim
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
## Architecture at a Glance
<p align="center">
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that’s in a datacenter, remote site, or sensitive environment.
- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under Apache 2.0. Fork it, extend it, or deploy it as-is.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
## Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at [https://docs.mulgadc.com](https://docs.mulgadc.com/) for maintaining and installing Hive.
Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
### Single node install
### Bare Metal ISO
The installation is straightforward to set up and running on a single node for testing purposes.
The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
```bash
export AWS_PROFILE=hive
export AWS_REGION=ap-southeast-2-sc
curl -fLO https://iso.mulgadc.com/spinifex.iso
```
```bash
curl https://install.mulgadc.com/ | bash
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
hive admin init --region $AWS_REGION
### Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
>*Prerequisite:* Linux bridge for networking.
Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
```bash
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
### Development Setup
For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/source-install/) documentation
For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation
### Component Repositories
Hive coordinates these independent components:
Spinifex coordinates these independent components:
- **[Predastore](https://github.com/mulgadc/predastore)** - S3-compatible object storage
- **[Viperblock](https://github.com/mulgadc/viperblock)** - EBS-compatible block storage
Each component can be developed independently. See component-specific documentation for focused development guides.
## Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
<p align="center">
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console — dashboard view" width="900">
</p>
The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
```bash
open https://YOUR_NODE_IP:3000
```
- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
## Development Philosophy
### Built by Engineers, For Engineers
Hive is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
**Real-World Experience:**
@@ -101,7 +174,7 @@ Hive is developed by experienced infrastructure engineers with deep AWS expertis
### AI-Assisted Development
While Hive is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
While Spinifex is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
**How We Use Claude Code:**
@@ -117,8 +190,8 @@ While Hive is architected and implemented by experienced engineers, we leverage
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
This hybrid approach ensures Hive benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## License
Hive is open source under the Apache 2.0 License. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
+17 -16
View File
@@ -10,39 +10,40 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
WORKDIR /build
# Copy go.work first (lives in hive/)
COPY hive/go.work ./hive/
# Copy module files for dependency caching
COPY hive/go.mod hive/go.sum ./hive/
COPY spinifex/go.mod spinifex/go.sum ./spinifex/
COPY predastore/go.mod predastore/go.sum ./predastore/
COPY viperblock/go.mod viperblock/go.sum ./viperblock/
COPY viperblock/nbd/go.mod viperblock/nbd/go.sum ./viperblock/nbd/
COPY viperblock/nbd/libguestfs.org/nbdkit/go.mod ./viperblock/nbd/libguestfs.org/nbdkit/
# Generate go.work (gitignored, so not in checkout)
RUN cd spinifex && go work init . ../predastore ../viperblock ../viperblock/nbd ../viperblock/nbd/libguestfs.org/nbdkit
# Download dependencies (cached layer)
RUN cd hive && go mod download
RUN cd spinifex && go mod download
# Copy full source
COPY hive/ ./hive/
COPY spinifex/ ./spinifex/
COPY predastore/ ./predastore/
COPY viperblock/ ./viperblock/
# Build hive binary
RUN COMMIT=$(cd hive && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
cd hive && go build -ldflags "-s -w \
-X github.com/mulgadc/hive/cmd/hive/cmd.Version=${VERSION} \
-X github.com/mulgadc/hive/cmd/hive/cmd.Commit=${COMMIT}" \
-o /out/hive cmd/hive/main.go
# Build spinifex binary (spx)
RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
cd spinifex && go build -ldflags "-s -w \
-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
-o /out/spx cmd/spinifex/main.go
# Build nbdkit viperblock plugin (.so)
RUN cd viperblock && go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
# Copy setup scripts and systemd units
RUN cp hive/scripts/setup.sh /out/setup.sh 2>/dev/null || true
RUN cp hive/scripts/setup-ovn.sh /out/setup-ovn.sh
RUN mkdir -p /out/systemd && cp hive/build/systemd/* /out/systemd/
RUN cp hive/build/logrotate/hive /out/logrotate-hive
RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex
FROM scratch AS export
COPY --from=builder /out/ /
@@ -1,4 +1,4 @@
/var/log/hive/*.log {
/var/log/spinifex/*.log {
daily
rotate 14
compress
+30
View File
@@ -0,0 +1,30 @@
#!/bin/sh
# Helper script for spinifex-predastore.service
# Detects host, port, backend, and node-id from spinifex.toml
# then execs spx. Falls back to single-node defaults if no config found.
CONF=/etc/spinifex/spinifex.toml
BIND="0.0.0.0:8443"
# -1 = co-located mode (all [[db]] peers in one process). Single-node spinifex
# templates list 3 DB peers on 127.0.0.1; only -1 can serve that topology.
# Multi-node spinifex.toml emits node_id = N (N >= 1) which overrides below.
# predastore rejects NODE_ID=0 since d3a02f0 (strconv.Atoi-of-garbage guard).
NODE_ID="-1"
if [ -f "$CONF" ]; then
H=$(awk -F'"' '/\[nodes\..*\.predastore\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
if [ -n "$H" ]; then
BIND="$H"
fi
N=$(awk -F'= *' '/node_id/{gsub(/ /,"",$2);print $2;exit}' "$CONF")
if [ -n "$N" ]; then
NODE_ID="$N"
fi
fi
export SPINIFEX_PREDASTORE_HOST="${BIND%%:*}"
export SPINIFEX_PREDASTORE_PORT="${BIND##*:}"
export SPINIFEX_PREDASTORE_NODE_ID="$NODE_ID"
exec /usr/local/bin/spx service predastore start
+37
View File
@@ -0,0 +1,37 @@
#!/bin/sh
# Wait for NATS to be reachable before starting a dependent service.
# Reads the NATS address from spinifex.toml [nodes.*.nats] section,
# falls back to nats.conf, then to 127.0.0.1:4222.
CONF=/etc/spinifex/spinifex.toml
NATS_CONF=/etc/spinifex/nats/nats.conf
NATS_HOST=""
NATS_PORT="4222"
TIMEOUT=30
# Try spinifex.toml first
if [ -f "$CONF" ]; then
NATS_BIND=$(awk -F'"' '/\[nodes\..*\.nats\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
if [ -n "$NATS_BIND" ]; then
NATS_HOST="${NATS_BIND%%:*}"
NATS_PORT="${NATS_BIND##*:}"
fi
fi
# Fall back to nats.conf
if [ -z "$NATS_HOST" ] && [ -f "$NATS_CONF" ]; then
NATS_HOST=$(grep "^listen:" "$NATS_CONF" | sed 's/listen: *//;s/:.*//')
fi
# Final fallback
NATS_HOST="${NATS_HOST:-127.0.0.1}"
for i in $(seq 1 "$TIMEOUT"); do
if nc -z "$NATS_HOST" "$NATS_PORT" 2>/dev/null; then
exit 0
fi
sleep 1
done
echo "Timed out waiting for NATS at $NATS_HOST:$NATS_PORT" >&2
exit 1
-22
View File
@@ -1,22 +0,0 @@
[Unit]
Description=Hive AWS Gateway
PartOf=hive.target
After=hive-nats.service hive-daemon.service
Requires=hive-nats.service
[Service]
Type=simple
User=hive
Group=hive
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
ExecStart=/usr/local/bin/hive service awsgw start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=HIVE_CONFIG_PATH=/etc/hive/hive.toml
Environment=HIVE_BASE_DIR=/var/lib/hive/
Environment=HIVE_AWSGW_TLS_CERT=/etc/hive/server.pem
Environment=HIVE_AWSGW_TLS_KEY=/etc/hive/server.key
[Install]
WantedBy=hive.target
-21
View File
@@ -1,21 +0,0 @@
[Unit]
Description=Hive Daemon (VM Orchestration)
PartOf=hive.target
After=hive-nats.service hive-viperblock.service
Requires=hive-nats.service
[Service]
Type=simple
User=hive
Group=hive
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
ExecStart=/usr/local/bin/hive service hive start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=HIVE_CONFIG_PATH=/etc/hive/hive.toml
Environment=HIVE_BASE_DIR=/var/lib/hive/hive/
Environment=HIVE_WAL_DIR=/var/lib/hive/hive/
[Install]
WantedBy=hive.target
-17
View File
@@ -1,17 +0,0 @@
[Unit]
Description=Hive NATS Message Bus
PartOf=hive.target
Before=hive-predastore.service hive-daemon.service hive-awsgw.service hive-vpcd.service
[Service]
Type=simple
User=hive
Group=hive
ExecStart=/usr/local/bin/hive service nats start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=HIVE_CONFIG_PATH=/etc/hive/nats/nats.conf
[Install]
WantedBy=hive.target
-22
View File
@@ -1,22 +0,0 @@
[Unit]
Description=Hive Predastore (S3-compatible Object Storage)
PartOf=hive.target
After=hive-nats.service
Requires=hive-nats.service
[Service]
Type=simple
User=hive
Group=hive
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
ExecStart=/usr/local/bin/hive service predastore start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=HIVE_PREDASTORE_CONFIG_PATH=/etc/hive/predastore/predastore.toml
Environment=HIVE_PREDASTORE_BASE_PATH=/var/lib/hive/predastore/
Environment=HIVE_PREDASTORE_TLS_CERT=/etc/hive/server.pem
Environment=HIVE_PREDASTORE_TLS_KEY=/etc/hive/server.key
[Install]
WantedBy=hive.target
-16
View File
@@ -1,16 +0,0 @@
[Unit]
Description=Hive Web UI
PartOf=hive.target
After=hive-awsgw.service
[Service]
Type=simple
User=hive
Group=hive
ExecStart=/usr/local/bin/hive service hive-ui start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
[Install]
WantedBy=hive.target
-21
View File
@@ -1,21 +0,0 @@
[Unit]
Description=Hive Viperblock (Block Storage)
PartOf=hive.target
After=hive-nats.service hive-predastore.service
Requires=hive-nats.service
[Service]
Type=simple
User=hive
Group=hive
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
ExecStart=/usr/local/bin/hive service viperblock start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
EnvironmentFile=/etc/hive/systemd.env
Environment=HIVE_CONFIG_PATH=/etc/hive/hive.toml
Environment=HIVE_BASE_DIR=/var/lib/hive/viperblock/
[Install]
WantedBy=hive.target
-19
View File
@@ -1,19 +0,0 @@
[Unit]
Description=Hive VPC Daemon (OVN Networking)
PartOf=hive.target
After=hive-nats.service hive-daemon.service
Requires=hive-nats.service
[Service]
Type=simple
User=hive
Group=hive
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
ExecStart=/usr/local/bin/hive service vpcd start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=HIVE_CONFIG_PATH=/etc/hive/hive.toml
[Install]
WantedBy=hive.target
-8
View File
@@ -1,8 +0,0 @@
[Unit]
Description=Hive Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=hive-nats.service hive-predastore.service hive-viperblock.service hive-daemon.service hive-awsgw.service hive-vpcd.service hive-ui.service
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,12 @@
[Unit]
Description=Regenerate SSH host keys
ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key
Before=ssh.service
[Service]
Type=oneshot
ExecStart=/usr/bin/ssh-keygen -A
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
+54
View File
@@ -0,0 +1,54 @@
[Unit]
Description=Spinifex AWS Gateway
PartOf=spinifex.target
After=spinifex-nats.service spinifex-daemon.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-gw
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service awsgw start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/awsgw/
Environment=SPINIFEX_AWSGW_TLS_CERT=/etc/spinifex/server.pem
Environment=SPINIFEX_AWSGW_TLS_KEY=/etc/spinifex/server.key
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/awsgw /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+98
View File
@@ -0,0 +1,98 @@
[Unit]
Description=Spinifex Daemon (VM Orchestration)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service
# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
# cold boot.
[Service]
Type=simple
User=spinifex-daemon
Group=spinifex
# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
# standalone mode and reconnects to NATS in the background — process-exiting
# on a missing NATS would kill the local VM management plane.
#
# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
# where silent NATS misconfig is more confusing than helpful. Do NOT set this
# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
ExecStart=/usr/local/bin/spx service spinifex start
Restart=on-failure
RestartSec=5
# DDIL Tier 1 (mulga-siv-58): KillMode=process so systemd only signals the
# daemon's main PID, not the cgroup. QEMU children stay alive across
# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
# via the persisted local state file + pidfile + QMP socket. With the
# default control-group mode, restart would terminate every guest VM —
# defeating Scenario B's "daemon restart preserves running instances".
KillMode=process
# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
# skips VM teardown (it now just persists state and exits), so 15s is
# plenty of slack for the cluster-manager + NATS unsubscribe path.
TimeoutStopSec=15
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
# DDIL Tier 1 (mulga-siv-58): QEMU pidfiles, QMP sockets, and the
# shared qemu.log all resolve via utils.pidPath() — which falls back
# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
# pidfiles written by QEMU (running in the OLD daemon's namespace)
# are invisible to the new daemon. The /local/instances handler then
# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
# stable, namespace-independent path that both daemon incarnations and
# QEMU children resolve identically.
Environment=XDG_RUNTIME_DIR=/run/spinifex
# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
ProtectSystem=full
PrivateDevices=no
DevicePolicy=closed
DeviceAllow=/dev/kvm rw
DeviceAllow=/dev/net/tun rw
DeviceAllow=char-vfio rw
DeviceAllow=/dev/vfio/vfio rw
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
# the daemon no longer needs AF_PACKET or the softened kernel protections.
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
# VFIO GPU passthrough requires locking all guest RAM into physical memory.
# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
LimitMEMLOCK=infinity
[Install]
WantedBy=spinifex.target
+48
View File
@@ -0,0 +1,48 @@
[Unit]
Description=Spinifex NATS Message Bus
PartOf=spinifex.target
Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service
[Service]
Type=simple
User=spinifex-nats
Group=spinifex
ExecStart=/usr/local/bin/spx service nats start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/nats/nats.conf
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/nats /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/nats /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+54
View File
@@ -0,0 +1,54 @@
[Unit]
Description=Spinifex Predastore (S3-compatible Object Storage)
PartOf=spinifex.target
After=spinifex-nats.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-storage
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/var/lib/spinifex/predastore-start.sh
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_PREDASTORE_CONFIG_PATH=/etc/spinifex/predastore/predastore.toml
Environment=SPINIFEX_PREDASTORE_BASE_PATH=/var/lib/spinifex/predastore/
Environment=SPINIFEX_PREDASTORE_TLS_CERT=/etc/spinifex/server.pem
Environment=SPINIFEX_PREDASTORE_TLS_KEY=/etc/spinifex/server.key
Environment=SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE=/etc/spinifex/predastore/encryption.key
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/predastore /etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/predastore /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+49
View File
@@ -0,0 +1,49 @@
[Unit]
Description=Spinifex Web UI
PartOf=spinifex.target
After=spinifex-awsgw.service
[Service]
Type=simple
User=spinifex-ui
Group=spinifex
ExecStart=/usr/local/bin/spx service spinifex-ui start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_UI_TLS_CERT=/etc/spinifex/server.pem
Environment=SPINIFEX_UI_TLS_KEY=/etc/spinifex/server.key
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/server.pem /etc/spinifex/server.key
ReadWritePaths=/var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+52
View File
@@ -0,0 +1,52 @@
[Unit]
Description=Spinifex Viperblock (Block Storage)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-predastore.service
Wants=spinifex-nats.service
[Service]
Type=simple
User=spinifex-viperblock
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service viperblock start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
EnvironmentFile=/etc/spinifex/systemd.env
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/viperblock/
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/systemd.env /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/viperblock /var/log/spinifex /run/spinifex/nbd
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+59
View File
@@ -0,0 +1,59 @@
[Unit]
Description=Spinifex VPC Daemon (OVN Networking)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service openvswitch-switch.service ovn-controller.service
Wants=spinifex-nats.service ovn-controller.service
[Service]
Type=simple
User=spinifex-vpcd
Group=spinifex
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service vpcd start
Restart=on-failure
RestartSec=5
OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
ProtectSystem=full
PrivateDevices=no
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=no
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=no
# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
# interface. See docs/development/bugs/dhcp-allocate-error.md.
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
# CAP_NET_RAW: open AF_PACKET sockets.
# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
# Note, disabled for now, sudo issue with ovs
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
NoNewPrivileges=no
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+8
View File
@@ -0,0 +1,8 @@
[Unit]
Description=Spinifex Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service openvswitch-switch.service ovn-controller.service
[Install]
WantedBy=multi-user.target
+5
View File
@@ -0,0 +1,5 @@
# Allow spinifex group to access VFIO devices for GPU passthrough.
# QEMU (launched by spinifex-daemon, group spinifex) must open /dev/vfio/N
# to use vfio-pci passthrough devices. The kernel creates these nodes as
# root:root 0600 by default; this rule grants group-level read-write access.
SUBSYSTEM=="vfio", GROUP="spinifex", MODE="0660"
+172
View File
@@ -0,0 +1,172 @@
// dhcptest is a diagnostic tool that runs a DHCP DORA handshake using the
// same nclient4 library and options that spinifex-vpcd uses for gateway pool
// leases. Run it on a node to verify DHCP works at the library level before
// debugging the full spinifex stack.
//
// Usage (must run as root — needs AF_PACKET):
//
// sudo dhcptest --iface=br-wan
// sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
// sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
package main
import (
"context"
"flag"
"fmt"
"log"
"net"
"os"
"os/exec"
"strings"
"time"
"github.com/insomniacslk/dhcp/dhcpv4"
"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
)
func main() {
iface := flag.String("iface", "br-wan", "Interface/bridge to bind the DHCP client to")
macStr := flag.String("mac", "", "Hardware address for DORA (chaddr + AF_PACKET filter). If empty, uses the real interface MAC.")
spinifexID := flag.String("spinifex-id", "", "Generate a spinifex virtual MAC from this resource ID (e.g. 'gateway-wan1'). Overrides --mac.")
clientID := flag.String("client-id", "", "DHCP option 61 client identifier. Defaults to --spinifex-id or --mac value.")
hostname := flag.String("hostname", "", "DHCP option 12 hostname")
vendorClass := flag.String("vendor-class", "mulga-spinifex-gw", "DHCP option 60 vendor class identifier")
promisc := flag.Bool("promisc", false, "Enable promiscuous mode on the interface before the test (mirrors vpcd's setBridgePromisc)")
timeout := flag.Duration("timeout", 10*time.Second, "Per-attempt DHCP timeout")
retries := flag.Int("retries", 3, "Number of DORA attempts before giving up")
verbose := flag.Bool("verbose", false, "Print full DHCP packet contents")
flag.Parse()
// Resolve the hardware address to use.
var hwAddr net.HardwareAddr
var err error
switch {
case *spinifexID != "":
hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
if err != nil {
log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
}
fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
case *macStr != "":
hwAddr, err = net.ParseMAC(*macStr)
if err != nil {
log.Fatalf("parse --mac %q: %v", *macStr, err)
}
default:
hwAddr, err = ifaceMAC(*iface)
if err != nil {
log.Fatalf("get MAC for %s: %v", *iface, err)
}
fmt.Printf("using real interface MAC: %s\n", hwAddr)
}
// Resolve client-id: default to spinifex-id, then MAC string.
cid := *clientID
if cid == "" {
if *spinifexID != "" {
cid = *spinifexID
} else {
cid = hwAddr.String()
}
}
fmt.Printf("iface: %s\n", *iface)
fmt.Printf("hw-addr: %s\n", hwAddr)
fmt.Printf("client-id: %s\n", cid)
fmt.Printf("hostname: %s\n", *hostname)
fmt.Printf("vendor-class: %s\n", *vendorClass)
fmt.Printf("promisc: %v\n", *promisc)
fmt.Printf("timeout: %s retries: %d\n", *timeout, *retries)
fmt.Println()
if *promisc {
if err := setPromisc(*iface, true); err != nil {
log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
} else {
fmt.Printf("set %s promisc ON\n\n", *iface)
}
}
// Build nclient4 options — identical to spinifex/services/vpcd/dhcp/nclient4.go.
opts := []nclient4.ClientOpt{
nclient4.WithHWAddr(hwAddr),
nclient4.WithTimeout(*timeout),
nclient4.WithRetry(*retries),
}
if *verbose {
opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
}
client, err := nclient4.New(*iface, opts...)
if err != nil {
log.Fatalf("nclient4.New(%s): %v", *iface, err)
}
defer client.Close()
// Identity modifiers — identical to identityModifiers() in vpcd dhcp package.
var mods []dhcpv4.Modifier
if cid != "" {
mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
}
if *hostname != "" {
mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
}
if *vendorClass != "" {
mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
}
fmt.Println("Sending DHCP Discover...")
start := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), *timeout*time.Duration(*retries+1))
defer cancel()
lease, err := client.Request(ctx, mods...)
if err != nil {
fmt.Printf("\nFAILED after %s: %v\n", time.Since(start).Round(time.Millisecond), err)
os.Exit(1)
}
fmt.Printf("SUCCESS in %s\n\n", time.Since(start).Round(time.Millisecond))
ack := lease.ACK
fmt.Printf(" Your IP: %s\n", ack.YourIPAddr)
fmt.Printf(" Subnet: %s\n", ack.SubnetMask())
fmt.Printf(" Gateway: %v\n", ack.Router())
fmt.Printf(" DNS: %v\n", ack.DNS())
fmt.Printf(" Server ID: %s\n", ack.ServerIdentifier())
fmt.Printf(" Lease time: %s\n", ack.IPAddressLeaseTime(24*time.Hour))
if *verbose {
fmt.Printf("\n--- Full ACK ---\n%s\n", ack.Summary())
}
}
// generateMAC is the same deterministic hash used by vpcd's topology.go.
// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
func generateMAC(resourceID string) string {
h := uint32(0)
for _, c := range resourceID {
h = h*31 + uint32(c)
}
return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
}
func ifaceMAC(name string) (net.HardwareAddr, error) {
ifc, err := net.InterfaceByName(name)
if err != nil {
return nil, err
}
return ifc.HardwareAddr, nil
}
func setPromisc(iface string, on bool) error {
state := "on"
if !on {
state = "off"
}
out, err := exec.Command("ip", "link", "set", iface, "promisc", state).CombinedOutput()
if err != nil {
return fmt.Errorf("%w: %s", err, strings.TrimSpace(string(out)))
}
return nil
}
File diff suppressed because it is too large Load Diff
-44
View File
@@ -1,44 +0,0 @@
package cmd
import (
"os"
"path/filepath"
)
// DefaultConfigDir returns the default configuration directory.
// Production: /etc/hive (when running as root or /etc/hive exists)
// Development: ~/hive/config
func DefaultConfigDir() string {
if isProductionLayout() {
return "/etc/hive"
}
homeDir, _ := os.UserHomeDir()
return filepath.Join(homeDir, "hive", "config")
}
// DefaultDataDir returns the default data directory.
// Production: /var/lib/hive
// Development: ~/hive
func DefaultDataDir() string {
if isProductionLayout() {
return "/var/lib/hive"
}
homeDir, _ := os.UserHomeDir()
return filepath.Join(homeDir, "hive")
}
// DefaultConfigFile returns the default path to hive.toml.
func DefaultConfigFile() string {
return filepath.Join(DefaultConfigDir(), "hive.toml")
}
// isProductionLayout returns true when running in a production install.
// Detected by: running as root, or /etc/hive directory exists.
func isProductionLayout() bool {
if os.Getuid() == 0 {
if info, err := os.Stat("/etc/hive"); err == nil && info.IsDir() {
return true
}
}
return false
}
-57
View File
@@ -1,57 +0,0 @@
package cmd
import (
"os"
"path/filepath"
"testing"
)
func TestDefaultConfigDir_Dev(t *testing.T) {
// Non-root user without /etc/hive should get dev paths
if os.Getuid() == 0 {
t.Skip("test requires non-root user")
}
homeDir, _ := os.UserHomeDir()
expected := filepath.Join(homeDir, "hive", "config")
got := DefaultConfigDir()
if got != expected {
t.Errorf("DefaultConfigDir() = %q, want %q", got, expected)
}
}
func TestDefaultDataDir_Dev(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("test requires non-root user")
}
homeDir, _ := os.UserHomeDir()
expected := filepath.Join(homeDir, "hive")
got := DefaultDataDir()
if got != expected {
t.Errorf("DefaultDataDir() = %q, want %q", got, expected)
}
}
func TestDefaultConfigFile(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("test requires non-root user")
}
got := DefaultConfigFile()
homeDir, _ := os.UserHomeDir()
expected := filepath.Join(homeDir, "hive", "config", "hive.toml")
if got != expected {
t.Errorf("DefaultConfigFile() = %q, want %q", got, expected)
}
}
func TestIsProductionLayout_NonRoot(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("test requires non-root user")
}
if isProductionLayout() {
t.Error("isProductionLayout() = true for non-root user, want false")
}
}
-6
View File
@@ -1,6 +0,0 @@
# AWS Gateway configuration file in TOML format.
version = "1.0"
region = "{{.Region}}"
# Enable verbose debugging
debug = false
-64
View File
@@ -1,64 +0,0 @@
# Domain configuration file in TOML format.
version = "1.0"
epoch = 1
node = "{{.Node}}"
[nodes.{{.Node}}]
node = "{{.Node}}"
host = "{{.BindIP}}"
region = "{{.Region}}"
az = "{{.Az}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
account_id = "{{.AccountID}}"
[nodes.{{.Node}}.daemon]
host = "{{.BindIP}}:{{.Port}}"
tlskey = "config/server.key"
tlscert = "config/server.pem"
dev_networking = true
[nodes.{{.Node}}.awsgw]
host = "{{.BindIP}}:9999"
tlskey = "config/server.key"
tlscert = "config/server.pem"
debug = true
config = "awsgw/awsgw.toml"
[nodes.{{.Node}}.nats]
host = "{{.BindIP}}:4222"
[nodes.{{.Node}}.nats.acl]
token = "{{.NatsToken}}"
[nodes.{{.Node}}.nats.sub]
[nodes.{{.Node}}.viperblock]
shardwal = false
[nodes.{{.Node}}.vpcd]
ovn_nb_addr = "tcp:127.0.0.1:6641"
ovn_sb_addr = "tcp:127.0.0.1:6642"
[nodes.{{.Node}}.predastore]
host = "{{.BindIP}}:8443"
region = "{{.Region}}"
bucket = "predastore"
accesskey = "{{.AccessKey}}"
secretkey = "{{.SecretKey}}"
{{- if gt .PredastoreNodeID 0}}
node_id = {{.PredastoreNodeID}}
{{- end}}
{{- range .RemoteNodes}}
[nodes.{{.Name}}]
node = "{{.Name}}"
host = "{{.Host}}"
region = "{{.Region}}"
az = "{{.AZ}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
{{- end}}
-29
View File
@@ -1,29 +0,0 @@
package cmd
import (
"testing"
)
func TestVersionVarsExist(t *testing.T) {
// Verify the version variables are set (defaults when not injected via ldflags)
if Version == "" {
t.Error("Version should not be empty")
}
if Commit == "" {
t.Error("Commit should not be empty")
}
}
func TestVersionCommand(t *testing.T) {
// Verify the version command is registered
found := false
for _, cmd := range rootCmd.Commands() {
if cmd.Use == "version" {
found = true
break
}
}
if !found {
t.Error("version command not registered on rootCmd")
}
}
-22
View File
@@ -1,22 +0,0 @@
/*
Copyright © 2025 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import "github.com/mulgadc/hive/cmd/hive/cmd"
func main() {
cmd.Execute()
}
+350
View File
@@ -0,0 +1,350 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Package autoinstall reads headless install parameters from environment
// variables exported by spinifex-init.sh from the kernel cmdline.
// Parameters are set in the GRUB "Headless" menu entry in grub.cfg.
package autoinstall
import (
"fmt"
"log/slog"
"net"
"os"
"os/exec"
"sort"
"strconv"
"strings"
"syscall"
"time"
"github.com/mulgadc/spinifex/cmd/installer/install"
)
// Load returns an install.Config built from SPINIFEX_* environment variables,
// or nil if SPINIFEX_AUTO is not set to "1" (interactive mode).
func Load() (*install.Config, error) {
if os.Getenv("SPINIFEX_AUTO") != "1" {
return nil, nil
}
slog.Info("autoinstall: headless mode enabled via kernel cmdline")
cfg, err := buildConfig()
if err != nil {
return nil, fmt.Errorf("autoinstall config: %w", err)
}
return cfg, nil
}
// EjectAndReboot ejects the boot device (best-effort) then reboots.
func EjectAndReboot() {
srcDev, _ := os.ReadFile("/run/iso-dev")
if dev := strings.TrimSpace(string(srcDev)); dev != "" {
slog.Info("autoinstall: ejecting boot device", "device", dev)
_ = exec.Command("eject", dev).Run()
}
fmt.Println()
fmt.Println("Installation complete.")
fmt.Println("Remove the USB drive now if it was not ejected automatically.")
fmt.Println("Rebooting in 10 seconds...")
time.Sleep(10 * time.Second)
// Use the kernel syscall directly — the live environment runs spinifex-init
// as PID 1 (not systemd), so exec("reboot") fails trying to reach D-Bus.
if err := syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART); err != nil {
slog.Error("autoinstall: reboot syscall failed", "err", err)
}
}
func buildConfig() (*install.Config, error) {
password := os.Getenv("SPINIFEX_PASSWORD")
if password == "" {
return nil, fmt.Errorf("SPINIFEX_PASSWORD is required")
}
hostname := os.Getenv("SPINIFEX_HOSTNAME")
if hostname == "" {
hostname = "spinifex-node"
}
disk, err := resolveDisk(os.Getenv("SPINIFEX_DISK"))
if err != nil {
return nil, fmt.Errorf("disk: %w", err)
}
wanIface, err := resolveNIC(os.Getenv("SPINIFEX_WAN_IFACE"), "")
if err != nil {
return nil, fmt.Errorf("WAN NIC: %w", err)
}
cfg := &install.Config{
Disk: disk,
Hostname: hostname,
RootPassword: password,
// Optional on the headless path — empty means no --email passed to
// spx admin init, which omits the operator identity from telemetry.
Email: strings.TrimSpace(os.Getenv("SPINIFEX_EMAIL")),
WANInterface: wanIface,
WANDHCPMode: strings.ToLower(os.Getenv("SPINIFEX_WAN_MODE")) != "static",
}
if !cfg.WANDHCPMode {
ip := os.Getenv("SPINIFEX_WAN_IP")
mask := os.Getenv("SPINIFEX_WAN_MASK")
gw := os.Getenv("SPINIFEX_WAN_GW")
if ip == "" || mask == "" || gw == "" {
return nil, fmt.Errorf("SPINIFEX_WAN_IP, SPINIFEX_WAN_MASK, SPINIFEX_WAN_GW required for static mode")
}
cfg.WANAddress = ip
cfg.WANMask = mask
cfg.WANGateway = gw
if dns := os.Getenv("SPINIFEX_WAN_DNS"); dns != "" {
cfg.WANDNS = strings.Split(dns, ",")
}
}
if lanIface := os.Getenv("SPINIFEX_LAN_IFACE"); lanIface != "" {
lan, err := resolveNIC(lanIface, wanIface)
if err != nil {
return nil, fmt.Errorf("LAN NIC: %w", err)
}
cfg.LANInterface = lan
cfg.LANDHCPMode = strings.ToLower(os.Getenv("SPINIFEX_LAN_MODE")) != "static"
if !cfg.LANDHCPMode {
cfg.LANAddress = os.Getenv("SPINIFEX_LAN_IP")
cfg.LANMask = os.Getenv("SPINIFEX_LAN_MASK")
if dns := os.Getenv("SPINIFEX_LAN_DNS"); dns != "" {
cfg.LANDNS = strings.Split(dns, ",")
}
}
}
if os.Getenv("SPINIFEX_GPU_PASSTHROUGH") == "1" {
cfg.GPUPassthrough = true
}
role := strings.ToLower(os.Getenv("SPINIFEX_ROLE"))
if role == "" {
role = "init"
}
cfg.ClusterRole = role
if role == "join" {
joinAddr := os.Getenv("SPINIFEX_JOIN_ADDR")
if joinAddr == "" {
return nil, fmt.Errorf("SPINIFEX_JOIN_ADDR required when SPINIFEX_ROLE=join")
}
cfg.JoinAddr = joinAddr
}
return cfg, nil
}
// resolveDisk maps the spx_disk value from grub.cfg to a block device path.
//
// Supported values:
//
// auto — use the only non-removable disk; fail if multiple found
// largest — largest non-removable disk (explicit opt-in)
// smallest — smallest non-removable disk (typical OS-on-SSD pattern)
// nvme — the only NVMe disk; fail if multiple found
// /dev/sda (etc.) — exact device path; fail if not present
func resolveDisk(target string) (string, error) {
switch strings.ToLower(target) {
case "", "auto":
return singleDisk()
case "largest":
return diskBySize(true)
case "smallest":
return diskBySize(false)
case "nvme":
return nvmeDisk()
default:
if _, err := os.Stat(target); err != nil {
return "", fmt.Errorf("%q not found", target)
}
return target, nil
}
}
type diskCandidate struct {
dev string
bytes int64
}
// nonRemovableDisks returns all non-removable, non-virtual block devices.
func nonRemovableDisks() ([]diskCandidate, error) {
entries, err := os.ReadDir("/sys/block")
if err != nil {
return nil, err
}
var out []diskCandidate
for _, e := range entries {
dev := e.Name()
switch {
case strings.HasPrefix(dev, "loop"),
strings.HasPrefix(dev, "ram"),
strings.HasPrefix(dev, "dm-"),
strings.HasPrefix(dev, "sr"):
continue
}
removable, _ := os.ReadFile("/sys/block/" + dev + "/removable")
if strings.TrimSpace(string(removable)) == "1" {
continue
}
sizeRaw, _ := os.ReadFile("/sys/block/" + dev + "/size")
sectors, parseErr := strconv.ParseInt(strings.TrimSpace(string(sizeRaw)), 10, 64)
if parseErr != nil || sectors == 0 {
continue
}
out = append(out, diskCandidate{dev: dev, bytes: sectors * 512})
}
return out, nil
}
// diskList returns a human-readable list of candidates for error messages.
func diskList(disks []diskCandidate) string {
lines := make([]string, len(disks))
for i, d := range disks {
lines[i] = fmt.Sprintf(" /dev/%s (%dG)", d.dev, d.bytes>>30)
}
return strings.Join(lines, "\n")
}
// singleDisk selects the only non-removable disk, or fails listing all found.
func singleDisk() (string, error) {
disks, err := nonRemovableDisks()
if err != nil {
return "", err
}
switch len(disks) {
case 0:
return "", fmt.Errorf("no non-removable disks found")
case 1:
slog.Info("autoinstall: single disk selected", "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
return "/dev/" + disks[0].dev, nil
default:
return "", fmt.Errorf(
"multiple disks found — set SPINIFEX_DISK to one of:\n%s\n"+
" largest (largest disk)\n"+
" smallest (smallest disk)\n"+
" nvme (NVMe only)\n"+
" /dev/sdX (exact path)",
diskList(disks),
)
}
}
// diskBySize selects the largest (largest=true) or smallest non-removable disk.
func diskBySize(largest bool) (string, error) {
disks, err := nonRemovableDisks()
if err != nil {
return "", err
}
if len(disks) == 0 {
return "", fmt.Errorf("no non-removable disks found")
}
sort.Slice(disks, func(i, j int) bool {
if largest {
return disks[i].bytes > disks[j].bytes
}
return disks[i].bytes < disks[j].bytes
})
label := "smallest"
if largest {
label = "largest"
}
slog.Info("autoinstall: disk selected by size", "mode", label, "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
return "/dev/" + disks[0].dev, nil
}
// nvmeDisk selects the only NVMe disk, or fails listing all NVMe disks found.
func nvmeDisk() (string, error) {
disks, err := nonRemovableDisks()
if err != nil {
return "", err
}
var nvmes []diskCandidate
for _, d := range disks {
if strings.HasPrefix(d.dev, "nvme") {
nvmes = append(nvmes, d)
}
}
switch len(nvmes) {
case 0:
return "", fmt.Errorf("no NVMe disks found")
case 1:
slog.Info("autoinstall: NVMe disk selected", "disk", nvmes[0].dev, "size_gb", nvmes[0].bytes>>30)
return "/dev/" + nvmes[0].dev, nil
default:
return "", fmt.Errorf(
"multiple NVMe disks found — set SPINIFEX_DISK to one of:\n%s",
diskList(nvmes),
)
}
}
// virtualNICPrefixes lists interface name prefixes that identify non-physical
// interfaces (bridges, tunnels, container/OVS veth pairs, etc.). These are
// skipped when auto-selecting a NIC so we don't configure docker0 or
// ovs-system as the management interface on a machine that previously ran
// Docker or OVN.
var virtualNICPrefixes = []string{
"docker", "veth", "virbr", "br-", "ovs-", "vxlan", "genev", "tun", "tap",
}
// resolveNIC returns the interface name to use. "auto" or empty picks the
// first physical (non-loopback, non-virtual) interface that is not exclude.
func resolveNIC(name, exclude string) (string, error) {
if name != "" && name != "auto" {
return name, nil
}
ifaces, err := net.Interfaces()
if err != nil {
return "", err
}
for _, iface := range ifaces {
if iface.Flags&net.FlagLoopback != 0 {
continue
}
// Require broadcast capability — filters out point-to-point tunnels.
if iface.Flags&net.FlagBroadcast == 0 {
continue
}
// Require a MAC address — virtual/tunnel interfaces have none.
if len(iface.HardwareAddr) == 0 {
continue
}
if iface.Name == exclude {
continue
}
// Skip known virtual interface prefixes.
virtual := false
for _, pfx := range virtualNICPrefixes {
if strings.HasPrefix(iface.Name, pfx) {
virtual = true
break
}
}
if virtual {
continue
}
slog.Info("autoinstall: NIC resolved", "mode", "auto", "selected", iface.Name)
return iface.Name, nil
}
return "", fmt.Errorf("no suitable physical NIC found (non-loopback, broadcast-capable, with MAC)")
}
@@ -0,0 +1,17 @@
package autoinstall
import (
"os"
"testing"
)
func TestLoadReturnsNilWhenNotAuto(t *testing.T) {
os.Unsetenv("SPINIFEX_AUTO")
cfg, err := Load()
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg != nil {
t.Fatal("expected nil config when SPINIFEX_AUTO is not set")
}
}
+56
View File
@@ -0,0 +1,56 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Package branding centralises Mulga/Spinifex visual identity constants used
// by the installer TUI.
//
// TODO: Replace placeholder values with official Mulga/Spinifex assets once
// the brand guide and logo files are located.
package branding
import "github.com/charmbracelet/lipgloss"
// Logo is the ASCII art rendered on the installer welcome screen.
// TODO: Replace with official Spinifex/Mulga ASCII logo.
const Logo = `
███████╗██████╗ ██╗███╗ ██╗██╗███████╗███████╗██╗ ██╗
██╔════╝██╔══██╗██║████╗ ██║██║██╔════╝██╔════╝╚██╗██╔╝
███████╗██████╔╝██║██╔██╗ ██║██║█████╗ █████╗ ╚███╔╝
╚════██║██╔═══╝ ██║██║╚██╗██║██║██╔══╝ ██╔══╝ ██╔██╗
███████║██║ ██║██║ ╚████║██║██║ ███████╗██╔╝ ██╗
╚══════╝╚═╝ ╚═╝╚═╝ ╚═══╝╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝`
// Subtitle is displayed beneath the logo on the welcome screen.
const Subtitle = "Bare-Metal Node Provisioner"
// Publisher is shown in the welcome screen footer.
const Publisher = "Mulga Defense Corporation"
// Mulga color palette.
// TODO: Replace with official hex values from the Mulga brand guide.
const (
ColorPrimary = lipgloss.Color("#00A3E0") // Mulga blue (placeholder)
ColorAccent = lipgloss.Color("#FF6B00") // Mulga orange (placeholder)
ColorBackground = lipgloss.Color("#0D1117") // Dark background
ColorSurface = lipgloss.Color("#161B22") // Slightly lighter surface
ColorText = lipgloss.Color("#E6EDF3") // Light text
ColorMuted = lipgloss.Color("#8B949E") // Muted / secondary text
ColorSuccess = lipgloss.Color("#3FB950") // Green
ColorWarning = lipgloss.Color("#D29922") // Amber
ColorError = lipgloss.Color("#F85149") // Red
ColorBorder = lipgloss.Color("#30363D") // Border
)
+259
View File
@@ -0,0 +1,259 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Package firstboot writes the oneshot systemd service and configuration that
// completes Spinifex provisioning on the first real boot after installation.
package firstboot
import (
"fmt"
"os"
"path/filepath"
"strings"
"github.com/mulgadc/spinifex/cmd/installer/systemd"
)
// Config holds the values the firstboot service needs to configure the node.
type Config struct {
Hostname string
// EncapIP is the Geneve tunnel IP for OVN. Set to the LAN bridge IP when a
// dedicated LAN NIC is present, otherwise the WAN bridge IP. Empty when DHCP
// is used — setup-ovn.sh auto-detects the IP from the default route in that case.
EncapIP string
// ClusterRole is "init" or "join".
ClusterRole string
// JoinAddr is host:port of the primary node, only used when ClusterRole is "join".
JoinAddr string
// Email is the operator email collected by the TUI or SPINIFEX_EMAIL on
// the headless path. Passed to `spx admin init --email=<value>` when set;
// omitted entirely when empty.
Email string
// GPUPassthrough enables VFIO GPU passthrough by passing --gpu-passthrough
// to `spx admin init`, which writes gpu_passthrough = true in the daemon config.
GPUPassthrough bool
// InstallCallback, when non-empty, is curled once at the end of firstboot
// after the success marker is written. Generic phone-home hook for
// provisioning controllers (PXE/MAAS-style flows). Best-effort: non-2xx
// and network failures are swallowed so the install is not gated on
// controller reachability.
InstallCallback string
}
// Write drops the firstboot script and systemd unit into root, which should be
// the path of the installed system's root filesystem (e.g. /mnt/spinifex-install).
func Write(root string, cfg Config) error {
if err := writeScript(root, cfg); err != nil {
return fmt.Errorf("firstboot script: %w", err)
}
if err := systemd.WriteFirstbootUnit(root); err != nil {
return fmt.Errorf("firstboot unit: %w", err)
}
if err := systemd.EnableUnit(root, "spinifex-firstboot.service"); err != nil {
return err
}
// Banner unit is written but NOT enabled here. firstboot enables it
// (along with spinifex.target) at the end of its run, so on the first
// boot the only spinifex unit running is firstboot itself — no implicit
// pull of spinifex.target via the banner's Wants= relationship, which
// was the root cause of the boot-time race against firstboot.
if err := systemd.WriteBannerUnit(root); err != nil {
return fmt.Errorf("banner unit: %w", err)
}
return systemd.WriteGettyDropIn(root)
}
func writeScript(root string, cfg Config) error {
clusterCmd := buildClusterCmd(cfg)
// --encap-ip is optional: when DHCP is used the IP is unknown at install time
// and setup-ovn.sh auto-detects it from the default route at boot.
setupOVN := "/usr/local/bin/setup-ovn.sh --management"
if cfg.EncapIP != "" {
setupOVN += fmt.Sprintf(" --encap-ip=%s", cfg.EncapIP)
}
callbackBlock := ""
if cfg.InstallCallback != "" {
callbackBlock = fmt.Sprintf(
"\ncurl -fsS --max-time 10 --retry 2 --retry-delay 2 -o /dev/null %s || true\n",
shellEscapeSingle(cfg.InstallCallback),
)
}
script := fmt.Sprintf(`#!/bin/bash
# Spinifex firstboot — runs once after ISO installation. On success, writes
# /var/lib/spinifex/.firstboot-done and the systemd unit's
# ConditionPathExists=! prevents re-execution on subsequent boots. A partial
# run leaves no marker, so the next reboot retries from the top — safe
# because every step below is idempotent (hostnamectl, setup-ovn.sh, spx
# admin init are all "set if not set"; systemctl enable on an already-enabled
# unit is a no-op).
set -euo pipefail
DONE_MARKER=/var/lib/spinifex/.firstboot-done
# Idempotency: bail early if a previous run completed successfully. The unit
# also has ConditionPathExists=!$DONE_MARKER so we shouldn't get here on
# subsequent boots, but defend in depth in case the unit is re-triggered
# manually before the operator notices the marker exists.
if [ -f "$DONE_MARKER" ]; then
echo "[firstboot] already complete — skipping"
exit 0
fi
# Set hostname
hostnamectl set-hostname %s
# Pre-start OVS and OVN central so their databases are initialised before
# setup-ovn.sh runs. On physical hardware, first-boot DB initialisation takes
# longer than setup-ovn.sh's internal 15-second timeout allows. Starting them
# here and waiting until the NB DB is ready means setup-ovn.sh sees a live DB
# the moment it starts — no races, no timeout failures.
systemctl start openvswitch-switch
systemctl start ovn-central
echo "Waiting for OVN NB DB to initialise..."
for _i in $(seq 1 120); do
if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
echo "OVN NB DB ready (${_i}s)"
break
fi
sleep 1
done
# Create default nameservers if DHCP server returns blank
printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolvconf/resolv.conf.d/base
resolvconf -u
# Configure OVN networking.
# br-wan (and br-lan if present) are Linux bridges managed by systemd-networkd
# (declared in /etc/systemd/network/ by the installer). setup-ovn.sh
# auto-detects br-wan as the default route device and wires it to OVS via a
# veth pair — non-destructive, SSH-safe.
%s
# Write the banner (first time, spinifex-banner service will do this on reboot)
/usr/local/bin/spx admin banner --boot-check
# Cluster formation — capture credentials to file for display on console.
%s 2>&1
# Fix ownership of files spx admin init wrote. spx runs as root under
# systemd, so /etc/spinifex/{spinifex.toml,master.key,ca.key,*.pem} and any
# per-service files written under /var/lib/spinifex/* land as root:root.
# Delegate to setup.sh's fix_file_ownership (single source of truth) — it
# knows the per-file modes (ca.key 0600, *.pem 0644, toml/key 0640) that a
# blunt recursive chmod would clobber.
SETUP_STAGES=fixown /usr/local/share/spinifex/setup.sh
# The /var/lib/spinifex/awsgw/config symlink was created at build time by
# setup.sh's create_directories — it resolves to /etc/spinifex, so
# {BaseDir}/config/master.key automatically points at /etc/spinifex/master.key.
# Bootstrap the spinifex user's SSH directory with a generated key pair so
# ~/.ssh exists and the operator can immediately use ssh-keygen / ssh-copy-id
# without hitting "No such file or directory" errors documented in the setup guide.
if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
mkdir -p /home/spinifex/.ssh
chmod 700 /home/spinifex/.ssh
ssh-keygen -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$(hostname)"
chown -R spinifex:spinifex /home/spinifex/.ssh
chmod 600 /home/spinifex/.ssh/id_ed25519
chmod 644 /home/spinifex/.ssh/id_ed25519.pub
fi
# Copy AWS credentials to the spinifex user's home directory.
# spx admin init runs with HOME=/root (set by the systemd unit), so credentials
# land in /root/.aws/. Copy them to the spinifex user's home so the operator
# can use the AWS CLI without sudo.
if [ -f /root/.aws/credentials ]; then
mkdir -p /home/spinifex/.aws
cp /root/.aws/credentials /home/spinifex/.aws/credentials
cp /root/.aws/config /home/spinifex/.aws/config 2>/dev/null || true
chown -R spinifex:spinifex /home/spinifex/.aws
chmod 700 /home/spinifex/.aws
chmod 600 /home/spinifex/.aws/credentials
[ -f /home/spinifex/.aws/config ] && chmod 600 /home/spinifex/.aws/config
fi
# Enable + activate spinifex.target and the banner now that all configs are
# in place. "enable --now" creates the multi-user.target.wants/ symlinks (so
# they start directly on every subsequent boot — no longer dependent on
# firstboot running each time, since firstboot is condition-skipped after
# this run via the marker) and activates them in this boot.
# NOTE: Moved to firstinstall to enable
#systemctl enable --now spinifex.target spinifex-banner.service
# Enable services to start, on reboot
systemctl enable spinifex.target spinifex-banner.service
# Mark complete only after every step above has succeeded. Until this point,
# any failure (set -e) leaves the marker absent and the next reboot retries
# firstboot from the top.
mkdir -p "$(dirname "$DONE_MARKER")"
touch "$DONE_MARKER"
%s
systemctl start spinifex.target
# Wait for the daemon to bring up external networking. When external_mode=pool
# is configured the daemon creates br-ext (the OVN external bridge) during
# startup; launching instances before it is ready results in no public IP being
# assigned. ip link requires no root and avoids parsing daemon logs.
if grep -q 'external_mode.*pool' /etc/spinifex/spinifex.toml 2>/dev/null; then
echo "[firstboot] waiting for external networking (br-ext)..."
for _i in $(seq 1 30); do
if ip link show br-ext >/dev/null 2>&1; then
echo "[firstboot] br-ext ready (${_i}s)"
break
fi
sleep 1
done
if ! ip link show br-ext >/dev/null 2>&1; then
echo "[firstboot] warning: br-ext not up after 30s — external networking may be delayed"
fi
fi
`, cfg.Hostname, setupOVN, clusterCmd, callbackBlock)
path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
return os.WriteFile(path, []byte(script), 0o755)
}
func buildClusterCmd(cfg Config) string {
emailFlag := ""
if cfg.Email != "" {
// shellEscapeSingle keeps the email safe if it ever contains a
// character shell treats specially — belt-and-braces; the regex
// validator already rejects whitespace and @-chains.
emailFlag = " --email=" + shellEscapeSingle(cfg.Email)
}
gpuFlag := ""
if cfg.GPUPassthrough {
gpuFlag = " --gpu-passthrough"
}
switch cfg.ClusterRole {
case "join":
return fmt.Sprintf("spx admin join --node %s --host %s%s", cfg.Hostname, cfg.JoinAddr, emailFlag)
default:
return fmt.Sprintf("spx admin init --node %s --nodes 1%s%s", cfg.Hostname, emailFlag, gpuFlag)
}
}
// shellEscapeSingle wraps s in single quotes with any embedded single
// quotes escaped. Minimal — we only need this because the email value is
// interpolated into a shell script written by Write().
func shellEscapeSingle(s string) string {
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
}
+100
View File
@@ -0,0 +1,100 @@
package firstboot
import (
"os"
"path/filepath"
"strings"
"testing"
)
func makeRootDirs(t *testing.T, root string) {
t.Helper()
for _, d := range []string{
"usr/local/bin",
"etc/systemd/system",
"etc/systemd/system/multi-user.target.wants",
} {
if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil {
t.Fatalf("mkdir %s: %v", d, err)
}
}
}
func TestWriteScriptNoCallbackWhenEmpty(t *testing.T) {
root := t.TempDir()
makeRootDirs(t, root)
cfg := Config{Hostname: "test-node", ClusterRole: "init"}
if err := Write(root, cfg); err != nil {
t.Fatalf("Write: %v", err)
}
script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
if err != nil {
t.Fatalf("read script: %v", err)
}
if strings.Contains(string(script), "curl") {
t.Error("script should not contain curl when InstallCallback is empty")
}
}
func TestWriteScriptEmbedsCurlWhenCallbackSet(t *testing.T) {
root := t.TempDir()
makeRootDirs(t, root)
const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
cfg := Config{
Hostname: "test-node",
ClusterRole: "init",
InstallCallback: callbackURL,
}
if err := Write(root, cfg); err != nil {
t.Fatalf("Write: %v", err)
}
script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
if err != nil {
t.Fatalf("read script: %v", err)
}
content := string(script)
if !strings.Contains(content, "curl") {
t.Error("script missing curl command")
}
if !strings.Contains(content, callbackURL) {
t.Errorf("script missing callback URL %q", callbackURL)
}
}
func TestWriteScriptCallbackAfterDoneMarker(t *testing.T) {
root := t.TempDir()
makeRootDirs(t, root)
const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
cfg := Config{
Hostname: "node1",
ClusterRole: "init",
InstallCallback: callbackURL,
}
if err := Write(root, cfg); err != nil {
t.Fatalf("Write: %v", err)
}
script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
if err != nil {
t.Fatalf("read script: %v", err)
}
content := string(script)
doneIdx := strings.Index(content, "touch \"$DONE_MARKER\"")
curlIdx := strings.Index(content, "curl")
if doneIdx < 0 {
t.Fatal("done marker not found in script")
}
if curlIdx < 0 {
t.Fatal("curl not found in script")
}
if curlIdx < doneIdx {
t.Error("curl must appear after done marker write")
}
}
+75
View File
@@ -0,0 +1,75 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Package install performs the disk installation steps after the UI has
// collected configuration.
package install
// Config holds all values collected by the installer UI.
type Config struct {
// Disk is the block device path to install onto (e.g. /dev/sda).
Disk string
// WAN interface — always present. The management IP lives on br-wan (a Linux
// bridge over the NIC), not on the physical NIC itself. This allows OVN/OVS
// to safely attach to the bridge without disrupting host connectivity.
WANInterface string
WANDHCPMode bool
WANAddress string // empty if WANDHCPMode
WANMask string // empty if WANDHCPMode
WANGateway string // empty if WANDHCPMode
WANDNS []string // empty if WANDHCPMode; e.g. ["1.1.1.1", "8.8.8.8"]
WANWiFiSSID string // non-empty if a WiFi NIC was selected
WANWiFiPass string // non-empty if a WiFi NIC was selected
// LAN interface — only set when 2+ NICs are present. The Geneve encap IP
// for OVN tunnels lives on br-lan. When empty, the WAN address is used.
LANInterface string
LANDHCPMode bool
LANAddress string
LANMask string
LANDNS []string
LANWiFiSSID string
LANWiFiPass string
// Node identity
Hostname string
// ClusterRole is "init" or "join".
ClusterRole string
// JoinAddr is the primary node address (host:port) when ClusterRole is "join".
JoinAddr string
// CA certificate (PEM), optional.
HasCACert bool
CACert string
// RootPassword is the password to set for the root account on the installed system.
RootPassword string
// Email is the operator's email address, used by the call-home telemetry
// endpoint to notify of important system updates and security advisories.
// Required on interactive installs; may be empty on headless/CI installs
// when SPINIFEX_EMAIL is not supplied on the kernel cmdline.
Email string
// GPUPassthrough enables VFIO GPU passthrough on this node.
// Set via SPINIFEX_GPU_PASSTHROUGH=1 on headless installs.
// Passes --gpu-passthrough to `spx admin init` during firstboot.
GPUPassthrough bool
}
+825
View File
@@ -0,0 +1,825 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package install
import (
"bufio"
"fmt"
"io"
"log/slog"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"syscall"
"time"
"github.com/mulgadc/spinifex/cmd/installer/firstboot"
"github.com/mulgadc/spinifex/cmd/installer/systemd"
)
const (
mountRoot = "/mnt/spinifex-install"
efiPart = mountRoot + "/boot/efi"
)
// Run executes all installation steps in order. It is intentionally sequential
// and explicit — each step is visible in logs so failures are easy to diagnose.
func Run(cfg *Config) error {
// The live environment may not have /sbin or /usr/sbin in PATH. Set it
// explicitly so exec.Command's LookPath finds system binaries like grub-install.
_ = os.Setenv("PATH", "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin")
// Unmount unconditionally on exit so a failed step never leaves partitions
// mounted in the live environment, which would cause a retry to double-mount.
defer func() {
_ = run("umount", efiPart)
_ = run("umount", mountRoot)
}()
steps := []struct {
name string
fn func() error
}{
{"partition disk", func() error { return partitionDisk(cfg.Disk) }},
{"format partitions", func() error { return formatPartitions(cfg.Disk) }},
{"mount partitions", func() error { return mountPartitions(cfg.Disk) }},
{"copy rootfs", copyRootfs},
{"write fstab", func() error { return writeFstab(cfg.Disk) }},
{"install spinifex", func() error { return installSpinifex(cfg) }},
{"write network config", func() error { return writeNetworkConfig(cfg) }},
{"write firstboot service", func() error { return firstboot.Write(mountRoot, cfg.toFirstbootConfig()) }},
{"install bootloader", func() error { return installBootloader(cfg.Disk) }},
{"install CA cert", func() error { return installCACert(cfg) }},
}
for _, s := range steps {
slog.Info("installer", "step", s.name)
if err := s.fn(); err != nil {
return fmt.Errorf("step %q: %w", s.name, err)
}
}
slog.Info("installation complete")
promptRemoveUSB()
return reboot()
}
func partitionDisk(disk string) error {
// GPT table with three partitions:
// p1: 1MiB BIOS Boot Partition — required for grub-install i386-pc on GPT
// p2: 512MiB EFI System Partition
// p3: remainder as root (ext4)
if err := run("parted", "--script", disk,
"mklabel", "gpt",
"mkpart", "bios_boot", "1MiB", "2MiB",
"set", "1", "bios_grub", "on",
"mkpart", "ESP", "fat32", "2MiB", "514MiB",
"set", "2", "esp", "on",
"mkpart", "root", "ext4", "514MiB", "100%",
); err != nil {
return err
}
// Force the kernel to re-read the partition table and wait for udev to
// create the partition device nodes. Without this, mkfs.fat in the next
// step may race and fail with "No such file or directory" on /dev/sda2 —
// the kernel has accepted the new layout but /dev hasn't been populated
// yet. Trixie's udev seems slower at this than Bookworm's was.
return waitForPartitions(disk)
}
// waitForPartitions ensures the EFI and root partition device nodes exist
// after parted creates them. It runs partprobe (kernel re-read) and
// udevadm settle (wait for queued events), then polls /dev for the files.
func waitForPartitions(disk string) error {
// Best-effort: partprobe failure isn't fatal — udev may still pick up
// the change from the BLKRRPART ioctl that parted itself issued.
if err := run("partprobe", disk); err != nil {
slog.Warn("partprobe failed, continuing", "disk", disk, "err", err)
}
if err := run("udevadm", "settle", "--timeout=10"); err != nil {
slog.Warn("udevadm settle failed, continuing", "err", err)
}
efi, root := partitionPaths(disk)
deadline := time.Now().Add(15 * time.Second)
for _, part := range []string{efi, root} {
for {
if _, err := os.Stat(part); err == nil {
break
}
if time.Now().After(deadline) {
return fmt.Errorf("partition device %s did not appear within timeout — kernel/udev did not pick up new partition table", part)
}
time.Sleep(200 * time.Millisecond)
}
}
return nil
}
func formatPartitions(disk string) error {
efi, root := partitionPaths(disk)
if err := run("mkfs.fat", "-F32", efi); err != nil {
return err
}
return run("mkfs.ext4", "-F", root)
}
func mountPartitions(disk string) error {
_, root := partitionPaths(disk)
if err := os.MkdirAll(mountRoot, 0o755); err != nil {
return err
}
if err := run("mount", root, mountRoot); err != nil {
return err
}
if err := os.MkdirAll(efiPart, 0o755); err != nil {
return err
}
efi, _ := partitionPaths(disk)
return run("mount", efi, efiPart)
}
// copyRootfs copies the live squashfs environment onto the target disk using
// rsync. This is the air-gapped alternative to debootstrap — all packages are
// already embedded in the ISO so no network access is required.
func copyRootfs() error {
if err := run("rsync", "-aHAX", "--delete", "--info=progress2",
"--exclude=/proc",
"--exclude=/sys",
"--exclude=/dev",
"--exclude=/run",
"--exclude=/tmp",
"--exclude=/cdrom",
"--exclude=/mnt",
"--exclude=/etc/machine-id",
"--exclude=/var/lib/dbus/",
"--exclude=/etc/openvswitch/",
"--exclude=/var/lib/openvswitch/",
"--exclude=/var/lib/dhcpcd/",
"--exclude=/lost+found",
"--exclude=/boot/efi",
"/", mountRoot+"/",
); err != nil {
return err
}
// Verify critical paths exist before proceeding. rsync exits 0 on ENOSPC
// for individual file writes on some filesystems, which would produce a
// partial rootfs that boots into a panic.
critical := []string{
filepath.Join(mountRoot, "bin/bash"),
filepath.Join(mountRoot, "lib/systemd/systemd"),
filepath.Join(mountRoot, "usr/local/bin/spx"),
}
for _, p := range critical {
if _, err := os.Stat(p); err != nil {
return fmt.Errorf("copyRootfs: critical path missing after rsync (%s): %w", p, err)
}
}
// rsync skips excluded paths entirely — recreate the virtual filesystem
// mount points that systemd expects to exist on the installed system.
mountPoints := []struct {
path string
mode os.FileMode
}{
{"proc", 0o555},
{"sys", 0o555},
{"dev", 0o755},
{"run", 0o755},
{"run/lock", 0o1777},
{"tmp", 0o1777},
{"mnt", 0o755},
}
for _, mp := range mountPoints {
if err := os.MkdirAll(filepath.Join(mountRoot, mp.path), mp.mode); err != nil {
return fmt.Errorf("create mountpoint /%s: %w", mp.path, err)
}
}
return nil
}
func installSpinifex(cfg *Config) error {
// The rootfs copy already contains spx and spinifex-installer at
// /usr/local/bin/ — no binary copy needed. Regenerate machine-specific
// identity files so each installed node is unique.
// Bind-mount /dev, /proc, /sys into the chroot so PAM (chpasswd) and
// other chroot commands work correctly. Trixie's PAM requires /proc and
// /dev/urandom for password hashing.
if err := bindChrootMounts(); err != nil {
return err
}
defer unbindChrootMounts()
// Generate a unique machine-id from the kernel CSPRNG. We do not use
// systemd-machine-id-setup in the chroot: it reads the SMBIOS UUID via
// the bind-mounted /sys, which is identical on cloned VMs or hosts with
// absent/zeroed DMI data, and falls back to non-unique sources (MAC,
// hostname) when that also fails — producing the same ID on every node.
// /proc/sys/kernel/random/uuid is always unique and requires no chroot.
machineIDPath := filepath.Join(mountRoot, "etc/machine-id")
rawUUID, err := os.ReadFile("/proc/sys/kernel/random/uuid")
if err != nil {
return fmt.Errorf("installSpinifex: read kernel uuid: %w", err)
}
machineID := strings.ReplaceAll(strings.TrimSpace(string(rawUUID)), "-", "") + "\n"
if err := os.WriteFile(machineIDPath, []byte(machineID), 0o444); err != nil {
return fmt.Errorf("installSpinifex: write machine-id: %w", err)
}
// dbus mirrors /etc/machine-id; remove it so it is re-created from the
// new ID on first boot rather than carrying a stale value.
_ = os.Remove(filepath.Join(mountRoot, "var/lib/dbus/machine-id"))
// Hostname.
hostnamePath := filepath.Join(mountRoot, "etc/hostname")
if err := os.WriteFile(hostnamePath, []byte(cfg.Hostname+"\n"), 0o644); err != nil {
return err
}
// /etc/hosts entry for the hostname.
hosts := fmt.Sprintf("127.0.0.1\tlocalhost\n127.0.1.1\t%s\n", cfg.Hostname)
if err := os.WriteFile(filepath.Join(mountRoot, "etc/hosts"), []byte(hosts), 0o644); err != nil {
return err
}
// Set root + spinifex passwords. We invoke chpasswd from the LIVE
// installer (not via `chroot`), passing the target root with -R and
// forcing -c YESCRYPT. This deliberately bypasses PAM:
// * `chroot ... chpasswd` uses /etc/pam.d/chpasswd → common-password →
// pam_unix.so with the "obscure" option, which can return
// "Authentication token manipulation error" inside a chroot for
// reasons that are awkward to diagnose (audit subsystem, locked
// shadow entries from useradd, etc.).
// * -c YESCRYPT tells chpasswd to hash locally with libcrypt and write
// directly to <root>/etc/shadow — no PAM stack involved. YESCRYPT
// matches Trixie's ENCRYPT_METHOD so subsequent logins use the same
// algorithm.
// * -R <root> opens the target's passwd/shadow directly; no bind
// mounts of /dev/urandom or /proc are needed for the password step.
if cfg.RootPassword != "" {
if err := setShadowPassword("root", cfg.RootPassword); err != nil {
return fmt.Errorf("set root password: %w", err)
}
// The spinifex account is the default interactive login on the
// node (console + SSH). Root SSH is disabled, so this is the sole
// remote entry point. The user itself is created at rootfs build
// time (build-rootfs.sh) — here we just set its password.
if err := setShadowPassword("spinifex", cfg.RootPassword); err != nil {
return fmt.Errorf("set spinifex password: %w", err)
}
}
// Write /etc/spinifex/node.conf — read at runtime by spx admin banner
// to look up the current IP dynamically (handles IP changes after install).
// MANAGEMENT_IFACE is the bridge (br-wan), not the physical NIC.
// MANAGEMENT_IP is empty for DHCP — banner's --boot-check fills it in at boot.
nodeConf := fmt.Sprintf("MANAGEMENT_IP=%s\nMANAGEMENT_IFACE=br-wan\nNODE_HOSTNAME=%s\n",
cfg.WANAddress, cfg.Hostname)
confDir := filepath.Join(mountRoot, "etc/spinifex")
if err := os.MkdirAll(confDir, 0o755); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(confDir, "node.conf"), []byte(nodeConf), 0o644); err != nil {
return err
}
// dhcpcd-base is present on the installed system (used by setup-ovn.sh for
// macvlan mode). Mask the standalone dhcpcd.service so it never auto-starts
// and races with systemd-networkd's built-in DHCP client on br-wan.
if err := maskSystemdUnit(mountRoot, "dhcpcd.service"); err != nil {
slog.Warn("installSpinifex: failed to mask dhcpcd.service", "err", err)
}
return nil
}
func writeNetworkConfig(cfg *Config) error {
// IPs live on Linux bridges (br-wan, br-lan), not on the physical NICs.
// systemd-networkd owns the full lifecycle: it creates the bridge NetDevs,
// enslaves the physical NICs, and runs the DHCP client. This means the
// veth pair that setup-ovn.sh adds later (veth-wan-br) is a port of a
// networkd-known bridge, so a networkctl reload or reboot never orphans it.
netdDir := filepath.Join(mountRoot, "etc/systemd/network")
if err := os.MkdirAll(netdDir, 0o755); err != nil {
return err
}
if err := writeNetworkdBridge(netdDir, cfg.WANInterface, "br-wan", false,
cfg.WANDHCPMode, cfg.WANAddress, cfg.WANMask, cfg.WANGateway, cfg.WANDNS,
cfg.WANWiFiSSID, cfg.WANWiFiPass); err != nil {
return err
}
if cfg.LANInterface != "" {
if err := writeNetworkdBridge(netdDir, cfg.LANInterface, "br-lan", true,
cfg.LANDHCPMode, cfg.LANAddress, cfg.LANMask, "", cfg.LANDNS,
cfg.LANWiFiSSID, cfg.LANWiFiPass); err != nil {
return err
}
// br-lan timing is controlled by spinifex-lan-bridge.service, not
// networkd auto-activation — ActivationPolicy=manual in the .network
// file ensures networkd creates the bridge but does not bring it up.
if err := systemd.WriteLANBridgeUnit(mountRoot); err != nil {
return fmt.Errorf("lan bridge unit: %w", err)
}
}
// Unmask systemd-networkd-wait-online on the installed system and scope
// it to br-wan only. The live ISO masks this service (build-rootfs.sh) to
// avoid blocking the installer environment (which has no br-wan). The mask
// is copied to the installed system by copyRootfs and must be removed here
// so that network-online.target — and therefore spinifex-firstboot.service
// (After=network-online.target) — does not fire before br-wan has its
// DHCP lease and default route. Without this, setup-ovn.sh finds no default
// route and exits early, creating no veth pair.
//
// The --interface=br-wan scope means br-lan (ActivationPolicy=manual) never
// blocks the wait; --timeout=60 caps the delay on a cold switch.
waitOnlineMask := filepath.Join(mountRoot, "etc/systemd/system/systemd-networkd-wait-online.service")
_ = os.Remove(waitOnlineMask)
waitOnlineDir := filepath.Join(mountRoot, "etc/systemd/system/systemd-networkd-wait-online.service.d")
if err := os.MkdirAll(waitOnlineDir, 0o755); err != nil {
return err
}
waitOnlineConf := "[Service]\nExecStart=\nExecStart=/lib/systemd/systemd-networkd-wait-online --interface=br-wan --timeout=60\n"
if err := os.WriteFile(filepath.Join(waitOnlineDir, "spinifex-wan-only.conf"), []byte(waitOnlineConf), 0o644); err != nil {
return err
}
// Disable IPv6 via sysctl — belt-and-suspenders alongside IPv6AcceptRA=no
// in the networkd .network files.
bridges := []string{"br-wan"}
if cfg.LANInterface != "" {
bridges = append(bridges, "br-lan")
}
var sysctl strings.Builder
sysctl.WriteString("# Generated by Spinifex installer — IPv6 disabled on management bridges\n")
for _, br := range bridges {
fmt.Fprintf(&sysctl, "net.ipv6.conf.%s.disable_ipv6=1\n", br)
}
sysctlDir := filepath.Join(mountRoot, "etc/sysctl.d")
if err := os.MkdirAll(sysctlDir, 0o755); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(sysctlDir, "99-spinifex-network.conf"), []byte(sysctl.String()), 0o644); err != nil {
return err
}
// Pin each NIC name to its MAC address via udev so the installed system
// always uses the same interface name regardless of probe order.
udevDir := filepath.Join(mountRoot, "etc/udev/rules.d")
if err := os.MkdirAll(udevDir, 0o755); err != nil {
return err
}
var udevRules strings.Builder
for _, iface := range []string{cfg.WANInterface, cfg.LANInterface} {
if iface == "" {
continue
}
mac, err := os.ReadFile("/sys/class/net/" + iface + "/address")
if err != nil {
slog.Warn("writeNetworkConfig: could not read NIC MAC, skipping udev pin", "iface", iface, "err", err)
continue
}
fmt.Fprintf(&udevRules, "SUBSYSTEM==\"net\", ACTION==\"add\", ATTR{address}==\"%s\", NAME=\"%s\"\n",
strings.TrimSpace(string(mac)), iface)
}
if udevRules.Len() > 0 {
return os.WriteFile(filepath.Join(udevDir, "70-spinifex-net.rules"), []byte(udevRules.String()), 0o644)
}
return nil
}
// writeNetworkdBridge writes the three systemd-networkd files that configure a
// physical NIC enslaved to a Linux bridge:
//
// - {prio}-spinifex-{suffix}-nic.network enslaves the NIC to the bridge
// - {prio+1}-spinifex-{suffix}.netdev declares the bridge device
// - {prio+1}-spinifex-{suffix}.network configures IP on the bridge
//
// manual=true sets ActivationPolicy=manual so networkd creates the bridge but
// does not auto-activate it — used for br-lan, which spinifex-lan-bridge.service
// activates after network-online.target.
func writeNetworkdBridge(dir, nicIface, bridgeName string, manual, dhcp bool,
addr, mask, gw string, dns []string, wifiSSID, wifiPass string) error {
suffix := strings.TrimPrefix(bridgeName, "br-")
nicPrio, brPrio := 10, 11
if manual {
nicPrio, brPrio = 12, 13
}
// NIC .network — enslaves the physical NIC to the bridge.
nicNet := fmt.Sprintf("[Match]\nName=%s\n\n[Network]\nBridge=%s\n", nicIface, bridgeName)
nicFile := fmt.Sprintf("%02d-spinifex-%s-nic.network", nicPrio, suffix)
if err := os.WriteFile(filepath.Join(dir, nicFile), []byte(nicNet), 0o644); err != nil {
return err
}
// Bridge .netdev — declares the bridge device.
brNetdev := fmt.Sprintf("[NetDev]\nName=%s\nKind=bridge\n\n[Bridge]\nSTP=no\n", bridgeName)
brNetdevFile := fmt.Sprintf("%02d-spinifex-%s.netdev", brPrio, suffix)
if err := os.WriteFile(filepath.Join(dir, brNetdevFile), []byte(brNetdev), 0o644); err != nil {
return err
}
// Bridge .network — IP configuration.
var b strings.Builder
fmt.Fprintf(&b, "[Match]\nName=%s\n\n", bridgeName)
if manual {
b.WriteString("[Link]\nActivationPolicy=manual\nRequiredForOnline=no\n\n")
}
b.WriteString("[Network]\n")
if dhcp {
b.WriteString("DHCP=ipv4\n")
} else {
cidr, err := addrCIDR(addr, mask)
if err != nil {
return fmt.Errorf("bridge %s: %w", bridgeName, err)
}
fmt.Fprintf(&b, "Address=%s\n", cidr)
if gw != "" {
fmt.Fprintf(&b, "Gateway=%s\n", gw)
}
for _, ns := range dns {
if ns = strings.TrimSpace(ns); ns != "" {
fmt.Fprintf(&b, "DNS=%s\n", ns)
}
}
}
b.WriteString("IPv6AcceptRA=no\nConfigureWithoutCarrier=yes\n")
if manual && dhcp {
// br-lan is non-critical; fail fast on DHCP so a missing LAN cable
// does not stall spinifex-lan-bridge.service indefinitely.
b.WriteString("\n[DHCP]\nTimeout=10\n")
}
brNetFile := fmt.Sprintf("%02d-spinifex-%s.network", brPrio, suffix)
if err := os.WriteFile(filepath.Join(dir, brNetFile), []byte(b.String()), 0o644); err != nil {
return err
}
if wifiSSID != "" {
if err := writeWPASupplicant(nicIface, wifiSSID, wifiPass); err != nil {
return err
}
}
return nil
}
// addrCIDR converts a dotted-decimal address + subnet mask to CIDR notation.
func addrCIDR(addr, dotMask string) (string, error) {
if net.ParseIP(addr) == nil {
return "", fmt.Errorf("invalid IP: %s", addr)
}
maskIP := net.ParseIP(dotMask)
if maskIP == nil {
return "", fmt.Errorf("invalid mask: %s", dotMask)
}
m := net.IPMask(maskIP.To4())
ones, bits := m.Size()
if bits == 0 {
return "", fmt.Errorf("non-contiguous or zero subnet mask: %s", dotMask)
}
return fmt.Sprintf("%s/%d", addr, ones), nil
}
// writeWPASupplicant writes a wpa_supplicant config for nicIface and enables
// the per-interface wpa_supplicant@{iface}.service so authentication completes
// before networkd enslaves the NIC to the bridge.
func writeWPASupplicant(nicIface, ssid, psk string) error {
conf := fmt.Sprintf(
"ctrl_interface=DIR=/run/wpa_supplicant GROUP=netdev\nupdate_config=1\n\nnetwork={\n\tssid=%q\n\tpsk=%q\n}\n",
ssid, psk,
)
wpaDir := filepath.Join(mountRoot, "etc/wpa_supplicant")
if err := os.MkdirAll(wpaDir, 0o755); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(wpaDir, "wpa_supplicant-"+nicIface+".conf"), []byte(conf), 0o600); err != nil {
return err
}
// Enable via symlink into multi-user.target.wants pointing at the
// package-provided template unit.
wantsDir := filepath.Join(mountRoot, "etc/systemd/system/multi-user.target.wants")
if err := os.MkdirAll(wantsDir, 0o755); err != nil {
return err
}
link := filepath.Join(wantsDir, "wpa_supplicant@"+nicIface+".service")
_ = os.Remove(link)
return os.Symlink("/lib/systemd/system/wpa_supplicant@.service", link)
}
func installBootloader(disk string) error {
// grub-install runs in the live environment (not chroot) using the
// grub-pc-bin and grub-efi-amd64-bin packages already present on the ISO.
// --boot-directory points at the installed system's /boot.
bootDir := filepath.Join(mountRoot, "boot")
efiDir := filepath.Join(mountRoot, "boot", "efi")
efiErr := run("grub-install",
"--target=x86_64-efi",
"--efi-directory="+efiDir,
"--boot-directory="+bootDir,
"--bootloader-id=spinifex",
"--removable",
"--recheck",
)
if efiErr != nil {
slog.Warn("installBootloader: EFI install failed", "err", efiErr)
}
if biosErr := run("grub-install",
"--target=i386-pc",
"--boot-directory="+bootDir,
"--recheck",
disk,
); biosErr != nil {
if efiErr != nil {
// Both targets failed — the system will not boot.
return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
}
return biosErr
}
copySplashImage(mountRoot)
copyGrubFont(mountRoot)
// Both consoles listed: tty0 stays primary for local install, ttyS0 mirrors
// kernel output to serial so headless installs (CI, racks with serial-only
// IPMI, remote-dev SSH-to-QEMU) see boot messages. The last `console=` on
// the cmdline becomes the system console, so ttyS0 wins on serial-only
// hardware while tty0 still receives output for local display.
grubDefault := `GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=Spinifex
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 systemd.show_status=1"
`
if err := os.WriteFile(filepath.Join(mountRoot, "etc/default/grub"), []byte(grubDefault), 0o644); err != nil {
return fmt.Errorf("write /etc/default/grub: %w", err)
}
// Mirror the ISO grub.cfg graphical block exactly so the installed GRUB menu
// looks identical to the installer menu. gfxterm MUST be activated before
// serial is appended — background_image silently does nothing in text mode.
// Using exec tail so update-grub includes everything from line 3 as raw GRUB config.
grubTheme := `#!/bin/sh
exec tail -n +3 $0
insmod all_video
insmod font
if loadfont /boot/grub/fonts/unicode.pf2; then
set gfxmode=auto
insmod gfxterm
terminal_output gfxterm
fi
insmod serial
if serial --unit=0 --speed=115200 --timeout=1; then
terminal_input --append serial
terminal_output --append serial
fi
# --- Branding ---
insmod png
set theme=/boot/grub/theme.txt
export theme
`
if err := os.WriteFile(filepath.Join(mountRoot, "etc/grub.d/06_spinifex"), []byte(grubTheme), 0o755); err != nil {
return fmt.Errorf("write /etc/grub.d/06_spinifex: %w", err)
}
if err := bindChrootMounts(); err != nil {
return err
}
defer unbindChrootMounts()
return run("chroot", mountRoot, "update-grub")
}
func installCACert(cfg *Config) error {
if !cfg.HasCACert || cfg.CACert == "" {
return nil
}
certPath := filepath.Join(mountRoot, "usr/local/share/ca-certificates/spinifex-ca.crt")
if err := os.WriteFile(certPath, []byte(cfg.CACert), 0o644); err != nil {
return err
}
if err := bindChrootMounts(); err != nil {
return err
}
defer unbindChrootMounts()
return run("chroot", mountRoot, "update-ca-certificates")
}
// promptRemoveUSB prints a removal reminder and waits up to 10 seconds for
// the user to press Enter before rebooting. Reading from os.Stdin works because
// spinifex-init redirects the installer's stdin from $CONSOLE_DEV.
func promptRemoveUSB() {
fmt.Println("\n\033[1mInstallation complete.\033[0m")
fmt.Println("Remove the USB drive, then press Enter to reboot (auto-rebooting in 10 seconds)...")
done := make(chan struct{})
go func() {
scanner := bufio.NewScanner(os.Stdin)
scanner.Scan()
close(done)
}()
select {
case <-done:
case <-time.After(10 * time.Second):
}
}
func reboot() error {
// sync filesystems before reboot so nothing is lost.
_ = run("sync")
// Use the kernel syscall directly — the live environment runs spinifex-init
// as PID 1 (not systemd), so the reboot(8) utility fails trying to reach D-Bus.
return syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART)
}
// toFirstbootConfig maps installer Config to the firstboot package's Config.
func (c *Config) toFirstbootConfig() firstboot.Config {
// Geneve encap IP: use LAN bridge IP when a dedicated LAN NIC is present,
// otherwise fall back to WAN bridge IP. Empty for DHCP — setup-ovn.sh
// auto-detects the IP from the default route at boot in that case.
encapIP := c.WANAddress
if c.LANInterface != "" && c.LANAddress != "" {
encapIP = c.LANAddress
}
return firstboot.Config{
Hostname: c.Hostname,
EncapIP: encapIP,
ClusterRole: c.ClusterRole,
JoinAddr: c.JoinAddr,
Email: c.Email,
InstallCallback: strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK")),
}
}
// writeFstab writes /etc/fstab on the installed system using partition UUIDs so
// the root filesystem is mounted read-write at boot and the EFI partition is
// mounted at /boot/efi.
func writeFstab(disk string) error {
efi, root := partitionPaths(disk)
rootUUID, err := partUUID(root)
if err != nil {
return fmt.Errorf("get root UUID: %w", err)
}
efiUUID, err := partUUID(efi)
if err != nil {
return fmt.Errorf("get EFI UUID: %w", err)
}
fstab := fmt.Sprintf("# /etc/fstab — generated by Spinifex installer\nUUID=%s / ext4 errors=remount-ro 0 1\nUUID=%s /boot/efi vfat umask=0077 0 1\n",
rootUUID, efiUUID)
return os.WriteFile(filepath.Join(mountRoot, "etc/fstab"), []byte(fstab), 0o644)
}
func partUUID(dev string) (string, error) {
out, err := exec.Command("blkid", "-s", "UUID", "-o", "value", dev).Output()
if err != nil {
return "", fmt.Errorf("blkid %s: %w", dev, err)
}
uuid := strings.TrimSpace(string(out))
if uuid == "" {
return "", fmt.Errorf("blkid returned no UUID for %s — partition may not have a filesystem yet", dev)
}
return uuid, nil
}
// partitionPaths returns the EFI and root partition device paths for a given
// disk. p1 is the BIOS Boot Partition (no filesystem), p2 is EFI, p3 is root.
// Handles both /dev/sdX (→ /dev/sdX2, /dev/sdX3) and /dev/nvmeXnY
// (→ /dev/nvmeXnYp2, /dev/nvmeXnYp3).
func partitionPaths(disk string) (efi, root string) {
// NVMe devices use a 'p' separator before the partition number.
if len(disk) > 0 && disk[len(disk)-1] >= '0' && disk[len(disk)-1] <= '9' {
return disk + "p2", disk + "p3"
}
return disk + "2", disk + "3"
}
// copySplashImage copies the GRUB splash (embedded in the squashfs at build time by
// inject-bins.sh) into the installed system so the post-install GRUB shows the same
// branded background as the installer GRUB. Non-fatal — missing source is logged and skipped.
func copySplashImage(root string) {
const src = "/usr/share/spinifex/grub-splash.png"
in, err := os.Open(src)
if err != nil {
slog.Warn("copySplashImage: splash not found, skipping", "path", src)
return
}
defer in.Close()
dstDir := filepath.Join(root, "boot/grub")
if err := os.MkdirAll(dstDir, 0o755); err != nil {
slog.Warn("copySplashImage: cannot create boot/grub dir", "err", err)
return
}
out, err := os.OpenFile(filepath.Join(dstDir, "splash.png"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
if err != nil {
slog.Warn("copySplashImage: cannot open destination", "err", err)
return
}
defer out.Close()
if _, err := io.Copy(out, in); err != nil {
slog.Warn("copySplashImage: copy failed", "err", err)
}
}
// copyGrubFont copies the unicode font into the installed system's
// /boot/grub/fonts/ so the loadfont path in 05_spinifex resolves at boot.
// grub-install does not copy fonts; we mirror what build-iso.sh does.
func copyGrubFont(root string) {
const src = "/usr/share/grub/unicode.pf2"
in, err := os.Open(src)
if err != nil {
slog.Warn("copyGrubFont: font not found, graphical GRUB may not work", "path", src)
return
}
defer in.Close()
dstDir := filepath.Join(root, "boot/grub/fonts")
if err := os.MkdirAll(dstDir, 0o755); err != nil {
slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
return
}
out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
if err != nil {
slog.Warn("copyGrubFont: cannot open destination", "err", err)
return
}
defer out.Close()
if _, err := io.Copy(out, in); err != nil {
slog.Warn("copyGrubFont: copy failed", "err", err)
}
}
// maskSystemdUnit creates a symlink to /dev/null for the given unit, which is
// the standard way to permanently disable a unit so systemd will never start it.
func maskSystemdUnit(root, unit string) error {
unitPath := filepath.Join(root, "etc/systemd/system", unit)
_ = os.Remove(unitPath)
return os.Symlink("/dev/null", unitPath)
}
// setShadowPassword sets a Unix password on the installed system without
// going through PAM. See the long comment in installSpinifex for the
// rationale.
func setShadowPassword(user, password string) error {
cmd := exec.Command("chpasswd", "-c", "YESCRYPT", "-R", mountRoot)
cmd.Stdin = strings.NewReader(user + ":" + password)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Run()
}
// chrootMountPaths lists virtual filesystems to bind-mount into the chroot.
// Order matters: unbind in reverse.
var chrootMountPaths = []string{"dev", "proc", "sys"}
// bindChrootMounts bind-mounts /dev, /proc, and /sys into the installed rootfs
// so chroot commands (chpasswd, systemd-machine-id-setup, update-grub) can
// access hardware, process info, and entropy sources. Idempotent — already-
// mounted paths are skipped.
func bindChrootMounts() error {
for _, m := range chrootMountPaths {
dst := filepath.Join(mountRoot, m)
if err := os.MkdirAll(dst, 0o755); err != nil {
return fmt.Errorf("create chroot mountpoint /%s: %w", m, err)
}
if err := run("mount", "--bind", "/"+m, dst); err != nil {
return fmt.Errorf("bind-mount /%s into chroot: %w", m, err)
}
}
return nil
}
// unbindChrootMounts unmounts the virtual filesystems in reverse order.
// Errors are logged but not returned — this is best-effort cleanup.
func unbindChrootMounts() {
for i := len(chrootMountPaths) - 1; i >= 0; i-- {
_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
}
}
func run(name string, args ...string) error {
cmd := exec.Command(name, args...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Run()
}
+27
View File
@@ -0,0 +1,27 @@
package install
import (
"os"
"testing"
)
func TestToFirstbootConfigIncludesInstallCallback(t *testing.T) {
const url = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
t.Setenv("SPINIFEX_INSTALL_CALLBACK", url)
cfg := &Config{Hostname: "node1", ClusterRole: "init"}
fb := cfg.toFirstbootConfig()
if fb.InstallCallback != url {
t.Errorf("InstallCallback = %q, want %q", fb.InstallCallback, url)
}
}
func TestToFirstbootConfigEmptyCallbackWhenEnvUnset(t *testing.T) {
os.Unsetenv("SPINIFEX_INSTALL_CALLBACK")
cfg := &Config{Hostname: "node1", ClusterRole: "init"}
fb := cfg.toFirstbootConfig()
if fb.InstallCallback != "" {
t.Errorf("InstallCallback = %q, want empty", fb.InstallCallback)
}
}
+86
View File
@@ -0,0 +1,86 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package main
import (
"fmt"
"os"
"strings"
"github.com/mulgadc/spinifex/cmd/installer/autoinstall"
"github.com/mulgadc/spinifex/cmd/installer/install"
"github.com/mulgadc/spinifex/cmd/installer/ui"
)
func main() {
// Check for a headless autoinstall config on the boot media before
// launching the interactive TUI. If found and enabled, run without
// any user input then eject the USB and reboot.
if cfg, err := autoinstall.Load(); err != nil {
fmt.Fprintf(os.Stderr, "autoinstall: %v\n", err)
os.Exit(1)
} else if cfg != nil {
if err := install.Run(cfg); err != nil {
fmt.Fprintf(os.Stderr, "install failed: %v\n", err)
os.Exit(1)
}
autoinstall.EjectAndReboot()
return
}
// Normal interactive path.
ttyPath := detectTTY()
cfg, err := ui.Run(ttyPath)
if err != nil {
fmt.Fprintf(os.Stderr, "installer: %v\n", err)
os.Exit(1)
}
if err := install.Run(cfg); err != nil {
fmt.Fprintf(os.Stderr, "install failed: %v\n", err)
os.Exit(1)
}
}
// detectTTY returns the TTY device path the installer should attach to.
// It first checks the SPINIFEX_CONSOLE environment variable (set by
// spinifex-init.sh after console fallback logic), then falls back to parsing
// /proc/cmdline. spinifex-init.sh may override the GRUB-set value (e.g. when
// GRUB says tty1 but no framebuffer exists, it falls back to ttyS0 and
// exports the corrected value).
func detectTTY() string {
// Prefer the env var — spinifex-init.sh sets this after resolving the
// actual console (including headless/framebuffer fallback).
if val := os.Getenv("SPINIFEX_CONSOLE"); val != "" && val != "auto" {
return "/dev/" + val
}
// Fall back to kernel cmdline for direct invocation without spinifex-init.
data, err := os.ReadFile("/proc/cmdline")
if err == nil {
for param := range strings.FieldsSeq(string(data)) {
if after, ok := strings.CutPrefix(param, "SPINIFEX_CONSOLE="); ok {
val := after
if val != "" && val != "auto" {
return "/dev/" + val
}
}
}
}
return "/dev/tty1"
}
+182
View File
@@ -0,0 +1,182 @@
/*
Copyright © 2026 Mulga Defense Corporation
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
// Package systemd writes systemd unit files and drop-ins into an installed
// system root during the Spinifex ISO installation process.
package systemd
import (
"fmt"
"os"
"path/filepath"
)
// WriteLANBridgeUnit installs a non-critical oneshot service that activates
// br-lan via networkctl *after* network-online.target. br-lan has
// ActivationPolicy=manual in its networkd .network file, so networkd creates
// the bridge at boot but does not bring it up automatically. This service
// triggers the activation once the WAN path is confirmed online, ensuring a
// missing LAN cable or DHCP timeout on the secondary bridge never stalls
// the management interface or firstboot.
func WriteLANBridgeUnit(root string) error {
unit := `[Unit]
Description=Spinifex LAN bridge (non-critical)
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/networkctl up br-lan
RemainAfterExit=yes
# Failure is non-critical — cable unplugged or switch not ready at boot.
SuccessExitStatus=0 1
[Install]
WantedBy=multi-user.target
`
unitPath := filepath.Join(root, "etc/systemd/system/spinifex-lan-bridge.service")
if err := os.WriteFile(unitPath, []byte(unit), 0o644); err != nil {
return err
}
wantsDir := filepath.Join(root, "etc/systemd/system/multi-user.target.wants")
if err := os.MkdirAll(wantsDir, 0o755); err != nil {
return err
}
link := filepath.Join(wantsDir, "spinifex-lan-bridge.service")
if err := os.Remove(link); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove stale symlink %s: %w", link, err)
}
return os.Symlink("/etc/systemd/system/spinifex-lan-bridge.service", link)
}
// WriteFirstbootUnit writes the spinifex-firstboot.service oneshot unit that
// runs the firstboot provisioning script on the first real boot after
// installation. The script writes /var/lib/spinifex/.firstboot-done on
// success; ConditionPathExists=! on that marker condition-skips the unit on
// every subsequent boot so it never re-runs (replacing the previous
// "disable on EXIT" trap which couldn't distinguish success from failure).
func WriteFirstbootUnit(root string) error {
unit := `[Unit]
Description=Spinifex first-boot provisioning
After=network-online.target
Wants=network-online.target
ConditionPathExists=/usr/local/bin/spinifex-firstboot.sh
ConditionPathExists=!/var/lib/spinifex/.firstboot-done
[Service]
Type=oneshot
Environment=HOME=/root
ExecStart=/usr/local/bin/spinifex-firstboot.sh
RemainAfterExit=yes
# On non-zero exit, overwrite /etc/issue with a hint so the operator sees
# the failure on the console — the banner unit is never enabled in that
# case (firstboot didn't reach its enable --now line), so /etc/issue would
# otherwise stay at the Debian default with no signal that anything is wrong.
ExecStopPost=/bin/sh -c 'if [ "$EXIT_STATUS" != "0" ]; then printf "\n*** Spinifex firstboot FAILED ***\nLogin as spinifex, then run:\n sudo journalctl -u spinifex-firstboot.service\n\n" > /etc/issue; fi'
# Cap total firstboot runtime so a hang in setup-ovn.sh / spx admin init /
# ovn-central startup cannot wedge multi-user.target and keep getty from
# ever reaching the login prompt.
TimeoutStartSec=300s
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
`
path := filepath.Join(root, "etc/systemd/system/spinifex-firstboot.service")
return os.WriteFile(path, []byte(unit), 0o644)
}
// WriteBannerUnit writes the spinifex-banner.service unit that runs
// `spx admin banner --boot-check` on every boot after spinifex.target is up.
// Running After=spinifex.target ensures the banner reflects a settled system
// state and that the IP check/restart happens once services are already running.
//
// The unit file is written by the installer but NOT enabled at install time.
// firstboot enables it (along with spinifex.target) at the end of its first
// run via `systemctl enable --now`. On every subsequent boot both units are
// driven by the multi-user.target.wants/ symlinks firstboot installed.
//
// Note: only After=spinifex.target — no Wants=. A Wants= would implicitly
// pull spinifex.target up at boot, which is exactly the race this whole
// change is fixing. After= alone is a no-op when the target isn't started,
// which is correct on first boot before firstboot has provisioned configs.
func WriteBannerUnit(root string) error {
unit := `[Unit]
Description=Spinifex console banner and boot health check
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/spx admin banner --boot-check
RemainAfterExit=yes
# Banner is oneshot; cap it so a stuck boot-check (IP detection, try-restart)
# cannot block getty via the spinifex-wait.conf drop-in.
TimeoutStartSec=30s
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
`
path := filepath.Join(root, "etc/systemd/system/spinifex-banner.service")
return os.WriteFile(path, []byte(unit), 0o644)
}
// WriteGettyDropIn holds the primary consoles (tty1 and ttyS0) until
// spinifex-banner.service completes (when banner is enabled), so the MOTD
// banner is visible before the login prompt appears. Drop-ins are scoped to
// named instances (getty@tty1, serial-getty@ttyS0) so tty2, tty3, etc.
// remain available as rescue terminals.
//
// Only After=, not Wants=. A Wants= would implicitly start the (un-enabled)
// banner on the very first boot, partially undoing the "only firstboot is
// enabled on first boot" guarantee. On first boot, banner is not enabled,
// so After= is a no-op and login appears immediately with the default
// /etc/issue. Once firstboot completes and enables the banner, subsequent
// boots have banner enabled at multi-user.target and After= correctly
// orders getty after the banner has rendered /etc/issue.
func WriteGettyDropIn(root string) error {
dropIn := `[Unit]
After=spinifex-firstboot.service
`
for _, svc := range []string{"getty@tty1", "serial-getty@ttyS0"} {
dropInDir := filepath.Join(root, "etc/systemd/system/"+svc+".service.d")
if err := os.MkdirAll(dropInDir, 0o755); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(dropInDir, "spinifex-wait.conf"), []byte(dropIn), 0o644); err != nil {
return err
}
}
return nil
}
// EnableUnit creates the multi-user.target.wants symlink for serviceName,
// equivalent to `systemctl enable` for a unit that targets multi-user.target.
func EnableUnit(root, serviceName string) error {
wantsDir := filepath.Join(root, "etc/systemd/system/multi-user.target.wants")
if err := os.MkdirAll(wantsDir, 0o755); err != nil {
return err
}
link := filepath.Join(wantsDir, serviceName)
target := "/etc/systemd/system/" + serviceName
if err := os.Remove(link); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove stale symlink %s: %w", link, err)
}
return os.Symlink(target, link)
}
File diff suppressed because it is too large Load Diff
+90
View File
@@ -0,0 +1,90 @@
// lb-agent runs inside load balancer VMs and manages HAProxy configuration.
//
// It polls the AWS gateway for config updates and reports health via heartbeats.
// All communication uses SigV4-signed HTTP requests.
//
// Usage:
//
// lb-agent --lb-id=lb-xxxxx --gateway=https://192.168.1.33:9999 --access-key=AKIA... --secret-key=...
//
// Flags default to environment variables set by cloud-init:
// LB_LB_ID, LB_GATEWAY_URL, LB_ACCESS_KEY, LB_SECRET_KEY, LB_REGION.
package main
import (
"flag"
"log/slog"
"os"
"os/signal"
"syscall"
"github.com/mulgadc/spinifex/spinifex/lbagent"
)
func main() {
var (
lbID string
gatewayURL string
accessKey string
secretKey string
region string
)
hostname, _ := os.Hostname()
flag.StringVar(&lbID, "lb-id", envOrDefault("LB_LB_ID", hostname), "Load balancer ID")
flag.StringVar(&gatewayURL, "gateway", os.Getenv("LB_GATEWAY_URL"), "Gateway URL (e.g. https://192.168.1.33:9999)")
flag.StringVar(&accessKey, "access-key", os.Getenv("LB_ACCESS_KEY"), "AWS access key ID")
flag.StringVar(&secretKey, "secret-key", os.Getenv("LB_SECRET_KEY"), "AWS secret access key")
flag.StringVar(&region, "region", envOrDefault("LB_REGION", "us-east-1"), "AWS region for SigV4 signing")
flag.Parse()
if lbID == "" {
slog.Error("--lb-id is required (or set LB_LB_ID)")
os.Exit(1)
}
if gatewayURL == "" {
slog.Error("--gateway is required (or set LB_GATEWAY_URL)")
os.Exit(1)
}
if accessKey == "" || secretKey == "" {
slog.Error("--access-key and --secret-key are required (or set LB_ACCESS_KEY / LB_SECRET_KEY)")
os.Exit(1)
}
slog.Info("Starting LB agent", "lbId", lbID, "gateway", gatewayURL)
agent, err := lbagent.New(lbID, gatewayURL, accessKey, secretKey, region)
if err != nil {
slog.Error("Failed to create agent", "err", err)
os.Exit(1)
}
// Start poll loop in a goroutine — it blocks until Stop is called.
errCh := make(chan error, 1)
go func() {
if err := agent.Start(); err != nil {
errCh <- err
}
}()
// Wait for shutdown signal or fatal error.
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
select {
case sig := <-sigCh:
slog.Info("Received signal, shutting down", "signal", sig)
case err := <-errCh:
slog.Error("Agent error", "err", err)
}
agent.Stop()
}
func envOrDefault(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
File diff suppressed because it is too large Load Diff
+447
View File
@@ -0,0 +1,447 @@
package cmd
import (
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/mulgadc/spinifex/spinifex/admin"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/gpu"
"github.com/mulgadc/spinifex/spinifex/types"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var adminGpuCmd = &cobra.Command{
Use: "gpu",
Short: "Manage GPU passthrough for a node",
}
var adminGpuStatusCmd = &cobra.Command{
Use: "status",
Short: "Show GPU hardware and passthrough state",
Run: runAdminGpuStatus,
}
var adminGpuEnableCmd = &cobra.Command{
Use: "enable",
Short: "Enable GPU passthrough on a node",
Run: runAdminGpuEnable,
}
var adminGpuDisableCmd = &cobra.Command{
Use: "disable",
Short: "Disable GPU passthrough on a node (blocked while GPU instances are running)",
Run: runAdminGpuDisable,
}
var adminGpuSetupCmd = &cobra.Command{
Use: "setup",
Short: "Configure this host for GPU passthrough (IOMMU, vfio-pci, nouveau blacklist)",
Long: `Idempotent host setup for GPU passthrough. Run once before a reboot to configure
GRUB, vfio-pci early binding, and the nouveau blacklist; then run again after the
reboot to verify bindings and enable GPU passthrough in the daemon.
Must be run as root.`,
Run: runAdminGpuSetup,
}
func init() {
adminCmd.AddCommand(adminGpuCmd)
adminGpuCmd.AddCommand(adminGpuStatusCmd)
adminGpuCmd.AddCommand(adminGpuEnableCmd)
adminGpuCmd.AddCommand(adminGpuDisableCmd)
adminGpuCmd.AddCommand(adminGpuSetupCmd)
adminGpuStatusCmd.Flags().String("node", "", "Target node name (default: local node)")
// enable, disable, and setup write to the local spinifex.toml and signal the
// local daemon — they must be run directly on the target host.
}
// gpuNodeStatus queries NATS and returns the NodeStatusResponse for the target node.
func gpuNodeStatus(targetNode string) (*types.NodeStatusResponse, error) {
cfg, nc, err := loadConfigAndConnect()
if err != nil {
return nil, err
}
defer nc.Close()
if targetNode == "" {
targetNode = cfg.Node
}
responses, err := collectResponses(nc, "spinifex.node.status", 3*time.Second)
if err != nil {
return nil, fmt.Errorf("collect node status: %w", err)
}
for _, raw := range responses {
var resp types.NodeStatusResponse
if err := json.Unmarshal(raw, &resp); err != nil {
continue
}
if resp.Node == targetNode {
return &resp, nil
}
}
return nil, fmt.Errorf("node %q not found or not responding", targetNode)
}
func runAdminGpuStatus(cmd *cobra.Command, _ []string) {
targetNode, _ := cmd.Flags().GetString("node")
resp, err := gpuNodeStatus(targetNode)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
iommu := "unknown"
vfio := "unknown"
if resp.GPUCapable || len(resp.GPUModels) > 0 {
iommu = "active"
vfio = "loaded"
}
fmt.Printf("Node: %s\n", resp.Node)
if len(resp.GPUModels) > 0 {
fmt.Printf("GPU hardware: %s\n", strings.Join(resp.GPUModels, ", "))
} else {
fmt.Printf("GPU hardware: none detected\n")
}
fmt.Printf("IOMMU: %s\n", iommu)
fmt.Printf("vfio-pci: %s\n", vfio)
if resp.GPUPassthrough {
fmt.Printf("Passthrough: enabled\n")
fmt.Printf("GPU pool: %d/%d allocated\n", resp.AllocGPUs, resp.TotalGPUs)
var g5Types []string
for _, cap := range resp.InstanceTypes {
if strings.HasPrefix(cap.Name, "g5.") {
g5Types = append(g5Types, cap.Name)
}
}
if len(g5Types) > 0 {
fmt.Printf("Instance types: %s\n", strings.Join(g5Types, " "))
}
} else if resp.GPUCapable {
fmt.Printf("Passthrough: disabled\n")
fmt.Printf("Instance types: (none — run 'spx admin gpu enable' to activate g5.*)\n")
} else {
fmt.Printf("Passthrough: disabled\n")
fmt.Printf("Instance types: (prerequisites not met)\n")
}
}
// gpuToggle writes the TOML setting, sends SIGHUP, and polls until the daemon confirms the new state.
// enable/disable always operate on the local node; run the command directly on the target host.
func gpuToggle(_ *cobra.Command, enable bool) {
cfgPath := viper.GetString("config")
if cfgPath == "" {
cfgPath = DefaultConfigFile()
}
cfg, err := config.LoadConfig(cfgPath)
if err != nil {
fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
os.Exit(1)
}
localNode := cfg.Node
// Check current state first.
resp, err := gpuNodeStatus(localNode)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
if enable {
if resp.GPUPassthrough {
fmt.Println("GPU passthrough is already enabled.")
return
}
if !resp.GPUCapable {
fmt.Fprintln(os.Stderr, "Error: prerequisites not met on this node.")
fmt.Fprintln(os.Stderr, " Run 'sudo spx admin gpu setup' to configure the host.")
os.Exit(1)
}
} else {
if !resp.GPUPassthrough {
fmt.Println("GPU passthrough is already disabled.")
return
}
if resp.AllocGPUs > 0 {
fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
os.Exit(1)
}
}
// Write the TOML setting.
if err := admin.SetGPUPassthrough(cfgPath, localNode, enable); err != nil {
fmt.Fprintf(os.Stderr, "Error writing config: %v\n", err)
os.Exit(1)
}
// Signal the daemon to reload.
out, err := exec.Command("systemctl", "kill", "-s", "HUP", "spinifex-daemon").CombinedOutput()
if err != nil {
fmt.Fprintf(os.Stderr, "Error sending SIGHUP: %v\n%s\n", err, out)
os.Exit(1)
}
// Poll until the daemon confirms the new state.
action := "enable"
if !enable {
action = "disable"
}
fmt.Printf("Waiting for daemon to %s GPU passthrough", action)
deadline := time.Now().Add(30 * time.Second)
for time.Now().Before(deadline) {
time.Sleep(1 * time.Second)
fmt.Print(".")
r, err := gpuNodeStatus(localNode)
if err != nil {
continue
}
if r.GPUPassthrough == enable {
fmt.Println(" done.")
runAdminGpuStatus(adminGpuStatusCmd, nil)
return
}
}
fmt.Println()
fmt.Fprintln(os.Stderr, "Timed out waiting for daemon. Check: journalctl -u spinifex-daemon -n 50")
os.Exit(1)
}
func runAdminGpuEnable(cmd *cobra.Command, _ []string) {
gpuToggle(cmd, true)
}
func runAdminGpuDisable(cmd *cobra.Command, _ []string) {
gpuToggle(cmd, false)
}
func runAdminGpuSetup(_ *cobra.Command, _ []string) {
if os.Getuid() != 0 {
fmt.Fprintln(os.Stderr, "Error: spx admin gpu setup must be run as root.")
os.Exit(1)
}
fmt.Println("==> Detecting GPU")
devices, err := gpu.Discover()
if err != nil {
fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
os.Exit(1)
}
if len(devices) == 0 {
fmt.Fprintln(os.Stderr, "No NVIDIA or AMD GPU found — is the card seated?")
os.Exit(1)
}
for _, d := range devices {
fmt.Printf(" %s - %s\n", d.PCIAddress, d.Model)
}
fmt.Println("==> Collecting PCI IDs for vfio-pci")
ids := gpuSetupCollectVFIOIDs(devices)
idsCSV := strings.Join(ids, ",")
fmt.Printf(" IDs: %s\n", idsCSV)
rebootNeeded := false
fmt.Println("==> Checking IOMMU")
iommuEntries, _ := os.ReadDir("/sys/kernel/iommu_groups/")
if len(iommuEntries) > 0 {
fmt.Println(" Active")
} else {
params := gpuSetupIOMMUParams()
changed, err := gpuSetupAddGRUBParams(params)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to update GRUB: %v\n", err)
os.Exit(1)
}
if changed {
fmt.Printf(" GRUB updated with: %s\n", params)
} else {
fmt.Println(" GRUB already has IOMMU params — refreshing")
}
ug := exec.Command("update-grub")
ug.Stdout, ug.Stderr = os.Stdout, os.Stderr
if err := ug.Run(); err != nil {
fmt.Fprintf(os.Stderr, "update-grub failed: %v\n", err)
os.Exit(1)
}
rebootNeeded = true
}
fmt.Println("==> Configuring vfio udev rule")
const vfioUdevRule = "/etc/udev/rules.d/99-spinifex-vfio.rules"
if _, err := os.Stat(vfioUdevRule); os.IsNotExist(err) {
rule := "SUBSYSTEM==\"vfio\", GROUP=\"spinifex\", MODE=\"0660\"\n"
if err := os.WriteFile(vfioUdevRule, []byte(rule), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "Failed to write vfio udev rule: %v\n", err)
os.Exit(1)
}
fmt.Println(" Rule installed")
_ = exec.Command("udevadm", "control", "--reload-rules").Run()
_ = exec.Command("udevadm", "trigger", "--subsystem-match=vfio").Run()
} else {
fmt.Println(" Rule already present")
}
fmt.Println("==> Configuring nouveau blacklist")
const nouveauConf = "/etc/modprobe.d/blacklist-nouveau.conf"
if _, err := os.Stat(nouveauConf); os.IsNotExist(err) {
if err := os.WriteFile(nouveauConf, []byte("blacklist nouveau\noptions nouveau modeset=0\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "Failed to write nouveau blacklist: %v\n", err)
os.Exit(1)
}
fmt.Println(" nouveau blacklisted")
rebootNeeded = true
} else {
fmt.Println(" Already blacklisted")
}
fmt.Println("==> Configuring vfio-pci early binding")
const vfioPCIConf = "/etc/modprobe.d/vfio-pci.conf"
existing, _ := os.ReadFile(vfioPCIConf)
if !strings.Contains(string(existing), "ids="+idsCSV) {
if err := os.WriteFile(vfioPCIConf, []byte("options vfio-pci ids="+idsCSV+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "Failed to write vfio-pci.conf: %v\n", err)
os.Exit(1)
}
fmt.Printf(" vfio-pci.conf written (ids=%s)\n", idsCSV)
rebootNeeded = true
} else {
fmt.Printf(" Already configured for %s\n", idsCSV)
}
const initramfsModules = "/etc/initramfs-tools/modules"
modData, _ := os.ReadFile(initramfsModules)
if !strings.Contains(string(modData), "vfio_pci") {
f, err := os.OpenFile(initramfsModules, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0o644)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to open initramfs modules: %v\n", err)
os.Exit(1)
}
_, werr := fmt.Fprint(f, "\n# vfio early binding for GPU passthrough\nvfio\nvfio_iommu_type1\nvfio_pci\n")
f.Close()
if werr != nil {
fmt.Fprintf(os.Stderr, "Failed to write initramfs modules: %v\n", werr)
os.Exit(1)
}
fmt.Println(" vfio modules added to initramfs")
rebootNeeded = true
} else {
fmt.Println(" vfio modules already in initramfs")
}
if rebootNeeded {
fmt.Println("==> Updating initramfs")
ui := exec.Command("update-initramfs", "-u")
ui.Stdout, ui.Stderr = os.Stdout, os.Stderr
if err := ui.Run(); err != nil {
fmt.Fprintf(os.Stderr, "update-initramfs failed: %v\n", err)
os.Exit(1)
}
fmt.Println("\nSetup complete — reboot required.")
fmt.Println(" sudo reboot")
fmt.Println(" Then re-run: sudo spx admin gpu setup")
return
}
fmt.Println("==> Verifying vfio-pci module")
if _, err := os.Stat("/sys/module/vfio_pci"); os.IsNotExist(err) {
fmt.Fprintln(os.Stderr, "vfio_pci not loaded — check: journalctl -b | grep vfio")
os.Exit(1)
}
fmt.Println(" Loaded")
fmt.Println("==> Verifying GPU driver binding")
for _, d := range devices {
driverLink := "/sys/bus/pci/devices/" + d.PCIAddress + "/driver"
target, err := os.Readlink(driverLink)
driver := filepath.Base(target)
if err != nil || driver != "vfio-pci" {
fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci\n", d.PCIAddress, driver)
fmt.Fprintf(os.Stderr, " Check: ls -la %s\n", driverLink)
os.Exit(1)
}
fmt.Printf(" %s - vfio-pci\n", d.PCIAddress)
}
fmt.Println("==> Enabling GPU passthrough")
gpuToggle(nil, true)
}
// gpuSetupCollectVFIOIDs collects vendor:device PCI ID pairs for each GPU and
// all sibling PCI functions on the same device slot (e.g. GPU + HDMI audio).
// All siblings must bind to vfio-pci together for IOMMU group isolation.
func gpuSetupCollectVFIOIDs(devices []gpu.GPUDevice) []string {
seen := make(map[string]bool)
var ids []string
for _, d := range devices {
dot := strings.LastIndex(d.PCIAddress, ".")
if dot < 0 {
continue
}
prefix := d.PCIAddress[:dot]
matches, _ := filepath.Glob("/sys/bus/pci/devices/" + prefix + ".*")
for _, m := range matches {
vb, err1 := os.ReadFile(m + "/vendor")
db, err2 := os.ReadFile(m + "/device")
if err1 != nil || err2 != nil {
continue
}
v := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(string(vb))), "0x")
dv := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(string(db))), "0x")
id := v + ":" + dv
if !seen[id] {
seen[id] = true
ids = append(ids, id)
}
}
}
return ids
}
func gpuSetupIOMMUParams() string {
data, _ := os.ReadFile("/proc/cpuinfo")
if strings.Contains(string(data), "GenuineIntel") {
return "intel_iommu=on iommu=pt"
}
return "amd_iommu=on iommu=pt"
}
// gpuSetupAddGRUBParams appends iommu kernel params to GRUB_CMDLINE_LINUX_DEFAULT.
// Returns (true, nil) if the file was modified, (false, nil) if params were already present.
func gpuSetupAddGRUBParams(params string) (bool, error) {
const grubFile = "/etc/default/grub"
data, err := os.ReadFile(grubFile)
if err != nil {
return false, fmt.Errorf("read %s: %w", grubFile, err)
}
content := string(data)
if strings.Contains(content, "intel_iommu=on") || strings.Contains(content, "amd_iommu=on") {
return false, nil
}
re := regexp.MustCompile(`(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*)"`)
updated := re.ReplaceAllStringFunc(content, func(match string) string {
groups := re.FindStringSubmatch(match)
if len(groups) < 2 {
return match
}
return groups[1] + " " + params + `"`
})
if updated == content {
return false, fmt.Errorf("GRUB_CMDLINE_LINUX_DEFAULT not found in %s", grubFile)
}
if err := os.WriteFile(grubFile, []byte(updated), 0o644); err != nil {
return false, fmt.Errorf("write %s: %w", grubFile, err)
}
return true, nil
}
+312
View File
@@ -0,0 +1,312 @@
package cmd
import (
"bytes"
"errors"
"fmt"
"os"
"path/filepath"
"testing"
"github.com/mulgadc/spinifex/spinifex/admin"
"github.com/mulgadc/spinifex/spinifex/formation"
"github.com/mulgadc/spinifex/spinifex/utils"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Regression guard: source URL must print for every error kind, not just
// mismatch, so a 404/non-HTTPS/size-cap failure still tells the operator
// which URL to investigate.
func TestPrintChecksumError(t *testing.T) {
image := utils.Images{Checksum: "https://example.com/SUMS", ChecksumType: "sha512"}
const imageFile = "/var/lib/img.tar.xz"
const imageName = "debian-13-x86_64"
errs := []error{
fmt.Errorf("%w: expected abc got def", utils.ErrChecksumMismatch),
fmt.Errorf("%w: 404", utils.ErrChecksumFetchFailed),
errors.New("open /x: no such file"),
}
for _, e := range errs {
var buf bytes.Buffer
printChecksumError(&buf, imageFile, imageName, image, e)
out := buf.String()
assert.Contains(t, out, image.Checksum, "source URL must print for: %v", e)
assert.Contains(t, out, imageFile)
assert.Contains(t, out, "spx admin images import --name "+imageName+" --force")
}
}
// buildRemoteNodes must prefer AdvertiseIP (off-host dial target) and fall
// back to BindIP when the peer pre-dates siv-8 and didn't send AdvertiseIP.
func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
nodes := map[string]formation.NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "203.0.113.1"},
"node2": {Name: "node2", BindIP: "10.0.0.2"}, // legacy joiner
"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "203.0.113.3"},
}
got := buildRemoteNodes(nodes, "node3")
if assert.Len(t, got, 2) {
assert.Equal(t, "node1", got[0].Name)
assert.Equal(t, "203.0.113.1", got[0].Host, "advertise wins when set")
assert.Equal(t, "node2", got[1].Name)
assert.Equal(t, "10.0.0.2", got[1].Host, "bind fallback when advertise empty")
}
}
func TestResolveAdvertiseIP(t *testing.T) {
wan := &admin.DetectedNetwork{WAN: &admin.DetectedInterface{IP: "192.168.1.21"}}
noWAN := &admin.DetectedNetwork{}
tests := []struct {
name string
bindIP string
advertise string
detected *admin.DetectedNetwork
want string
wantErr bool
}{
{"single-node default, WAN detected", "0.0.0.0", "", wan, "192.168.1.21", false},
{"single-node default, no WAN", "0.0.0.0", "", noWAN, "127.0.0.1", false},
{"single-node explicit advertise", "0.0.0.0", "203.0.113.5", wan, "203.0.113.5", false},
{"multi-node init, specific bind", "10.11.12.1", "", nil, "10.11.12.1", false},
{"multi-node dual-homed override", "10.11.12.1", "203.0.113.5", nil, "203.0.113.5", false},
{"loopback-only test", "127.0.0.1", "", noWAN, "127.0.0.1", false},
{"invalid advertise flag", "0.0.0.0", "not-an-ip", wan, "", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := resolveAdvertiseIP(tt.bindIP, tt.advertise, tt.detected)
if tt.wantErr {
assert.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
// Generated spinifex.toml must include both host (listen, BindIP) and
// advertise (off-host dial target, AdvertiseIP) for the local node.
func TestSpinifexTomlTemplate_AdvertiseField(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "spinifex.toml")
settings := admin.ConfigSettings{
Node: "node1",
Az: "ap-southeast-2a",
Port: "4432",
Region: "ap-southeast-2",
BindIP: "0.0.0.0",
AdvertiseIP: "192.168.1.21",
AccessKey: "AKIATEST",
SecretKey: "SECRET",
AccountID: "123456789012",
NatsToken: "token",
ConfigDir: dir,
OVNNBAddr: "tcp:127.0.0.1:6641",
OVNSBAddr: "tcp:127.0.0.1:6642",
}
require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
data, err := os.ReadFile(path)
require.NoError(t, err)
content := string(data)
assert.Contains(t, content, `host = "0.0.0.0"`, "listen address")
assert.Contains(t, content, `advertise = "192.168.1.21"`, "off-host dial target")
}
// Empty AdvertiseIP (e.g. loading an existing cluster pre-siv-8) must NOT
// render an empty advertise = "" line — downstream fallback to Host kicks in.
func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "spinifex.toml")
settings := admin.ConfigSettings{
Node: "node1",
Az: "ap-southeast-2a",
Port: "4432",
Region: "ap-southeast-2",
BindIP: "10.11.12.1",
AccessKey: "AKIATEST",
SecretKey: "SECRET",
AccountID: "123456789012",
NatsToken: "token",
ConfigDir: dir,
OVNNBAddr: "tcp:127.0.0.1:6641",
OVNSBAddr: "tcp:127.0.0.1:6642",
}
require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
data, _ := os.ReadFile(path)
assert.NotContains(t, string(data), "advertise =")
}
// detectedDhcpBindBridge must return the default-route interface name verbatim
// when it's a bridge (Linux or OVS, br-* prefix). The old detectedWanBridge()
// returned hardcoded "br-ext" for Linux bridges — that value broke DHCP on
// consumer-router LANs because br-ext never sees LAN DHCP traffic (mulga-998).
func TestDetectedDhcpBindBridge(t *testing.T) {
tests := []struct {
name string
detected *admin.DetectedNetwork
want string
}{
{
name: "Linux bridge default route returns bridge name (not 'br-ext')",
detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-wan"}},
want: "br-wan",
},
{
name: "OVS bridge default route returned verbatim",
detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-ext"}},
want: "br-ext",
},
{
name: "physical NIC defaults to 'br-wan' (bridge the installer creates)",
detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "enp0s3"}},
want: "br-wan",
},
{
name: "nil DetectedNetwork",
detected: nil,
want: "",
},
{
name: "DetectedNetwork with nil WAN",
detected: &admin.DetectedNetwork{},
want: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, detectedDhcpBindBridge(tt.detected))
})
}
}
// Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
// alias (per mulga-998 D3). Prevents the footgun where operators inherited the
// old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
func TestCheckLegacyWanBridgeKey_TOMLRejected(t *testing.T) {
dir := t.TempDir()
cfgPath := filepath.Join(dir, "spinifex.toml")
require.NoError(t, os.WriteFile(cfgPath, []byte(`
node = "node1"
[nodes.node1.vpcd]
ovn_nb_addr = "tcp:127.0.0.1:6641"
wan_bridge = "br-ext"
`), 0o644))
viper.Reset()
t.Cleanup(viper.Reset)
viper.SetConfigFile(cfgPath)
viper.SetConfigType("toml")
require.NoError(t, viper.ReadInConfig())
err := checkLegacyWanBridgeKey("node1", cfgPath)
require.Error(t, err)
assert.Contains(t, err.Error(), "dhcp_bind_bridge")
assert.Contains(t, err.Error(), "wan_bridge")
}
func TestCheckLegacyWanBridgeKey_EnvVarRejected(t *testing.T) {
viper.Reset()
t.Cleanup(viper.Reset)
t.Setenv("SPINIFEX_VPCD_WAN_BRIDGE", "br-ext")
err := checkLegacyWanBridgeKey("node1", "/tmp/unused.toml")
require.Error(t, err)
assert.Contains(t, err.Error(), "SPINIFEX_VPCD_WAN_BRIDGE")
assert.Contains(t, err.Error(), "dhcp_bind_bridge")
}
func TestCheckLegacyWanBridgeKey_CleanConfigPasses(t *testing.T) {
dir := t.TempDir()
cfgPath := filepath.Join(dir, "spinifex.toml")
require.NoError(t, os.WriteFile(cfgPath, []byte(`
node = "node1"
[nodes.node1.vpcd]
ovn_nb_addr = "tcp:127.0.0.1:6641"
dhcp_bind_bridge = "br-wan"
`), 0o644))
viper.Reset()
t.Cleanup(viper.Reset)
viper.SetConfigFile(cfgPath)
viper.SetConfigType("toml")
require.NoError(t, viper.ReadInConfig())
assert.NoError(t, checkLegacyWanBridgeKey("node1", cfgPath))
}
// iam.nats_url must not embed a bare 0.0.0.0 — the predastore process dials
// its own NATS via loopback when listening on wildcard.
func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
nodes := []admin.PredastoreNodeConfig{
{ID: 1, Host: "10.0.0.1"},
{ID: 2, Host: "10.0.0.2"},
{ID: 3, Host: "10.0.0.3"},
}
content, err := admin.GenerateMultiNodePredastoreConfig(
predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0",
)
require.NoError(t, err)
assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
assert.NotContains(t, content, `nats_url = "nats://0.0.0.0:4222"`)
// Specific bind IP → stays as-is.
content2, err := admin.GenerateMultiNodePredastoreConfig(
predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1",
)
require.NoError(t, err)
assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
}
// writePredastoreEncryptionKey is called from both `spx admin init` and `spx
// admin join`; both depend on it producing a 32-byte file at 0600 in the
// expected layout. Predastore's keyfile loader rejects anything that isn't
// exactly that, so a regression here would break service startup on every
// node.
func TestWritePredastoreEncryptionKey(t *testing.T) {
configDir := t.TempDir()
keyPath, err := writePredastoreEncryptionKey(configDir)
require.NoError(t, err)
assert.Equal(t, filepath.Join(configDir, "predastore", "encryption.key"), keyPath)
info, err := os.Stat(keyPath)
require.NoError(t, err)
assert.Equal(t, int64(32), info.Size(), "predastore master key must be exactly 32 bytes")
assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "predastore master key must be mode 0600")
// Per-node invariant: two successive calls (on what would be two
// different nodes) must produce different key material. If they ever
// returned the same bytes we'd have silently lost the per-node
// blast-radius property.
configDir2 := t.TempDir()
keyPath2, err := writePredastoreEncryptionKey(configDir2)
require.NoError(t, err)
key1, err := os.ReadFile(keyPath)
require.NoError(t, err)
key2, err := os.ReadFile(keyPath2)
require.NoError(t, err)
assert.NotEqual(t, key1, key2, "per-node keys must differ")
}
// Joiners run `spx admin join` against a configDir that doesn't yet contain
// a predastore/ subdir. The helper must create it (predastore subdir is
// owned by spinifex-storage in prod, but the helper just needs the dir to
// exist).
func TestWritePredastoreEncryptionKey_CreatesMissingDir(t *testing.T) {
configDir := t.TempDir()
// Don't pre-create the predastore subdir — simulate a fresh joiner.
keyPath, err := writePredastoreEncryptionKey(configDir)
require.NoError(t, err)
dirInfo, err := os.Stat(filepath.Dir(keyPath))
require.NoError(t, err)
assert.True(t, dirInfo.IsDir())
}
+92
View File
@@ -0,0 +1,92 @@
package cmd
import (
"bufio"
"fmt"
"os"
"strings"
"github.com/mulgadc/spinifex/spinifex/migrate"
"github.com/spf13/cobra"
)
var upgradeCmd = &cobra.Command{
Use: "upgrade",
Short: "Apply pending config migrations",
Long: `Apply pending configuration file migrations for upgrades between Spinifex versions.
When run interactively (without --yes), shows pending migrations and prompts
for confirmation. When called from setup.sh with --yes, applies immediately.
Operators can skip migrations during install by setting INSTALL_SPINIFEX_SKIP_MIGRATE=1,
then run 'spx admin upgrade' manually to review and apply.`,
Run: runAdminUpgrade,
}
func runAdminUpgrade(cmd *cobra.Command, _ []string) {
configDir, _ := cmd.Root().Flags().GetString("config-dir")
dataDir, _ := cmd.Root().Flags().GetString("spinifex-dir")
yes, _ := cmd.Flags().GetBool("yes")
// Check that the installation exists.
spinifexToml := configDir + "/spinifex.toml"
if _, err := os.Stat(spinifexToml); os.IsNotExist(err) {
fmt.Fprintf(os.Stderr, "No Spinifex installation found at %s\nRun 'spx admin init' first.\n", configDir)
os.Exit(1)
}
// Show current versions.
versions, err := migrate.DefaultRegistry.ConfigVersions(configDir)
if err != nil {
fmt.Fprintf(os.Stderr, "Error reading config versions: %v\n", err)
os.Exit(1)
}
fmt.Println("Reading current config versions...")
for name, v := range versions {
if v == 0 {
fmt.Printf(" %-20s %d (no version marker)\n", name+":", v)
} else {
fmt.Printf(" %-20s %d\n", name+":", v)
}
}
fmt.Println()
// Check for pending migrations.
pending, err := migrate.DefaultRegistry.PendingConfig(configDir)
if err != nil {
fmt.Fprintf(os.Stderr, "Error checking pending migrations: %v\n", err)
os.Exit(1)
}
if len(pending) == 0 {
fmt.Println("No pending config migrations.")
return
}
fmt.Println("Pending config migrations:")
for _, p := range pending {
fmt.Printf(" [config] %s %d → %d: %s\n", p.Target, p.FromVersion, p.ToVersion, p.Description)
}
fmt.Println()
// Prompt for confirmation unless --yes.
if !yes {
fmt.Print("Apply? [y/N] ")
reader := bufio.NewReader(os.Stdin)
answer, _ := reader.ReadString('\n')
answer = strings.TrimSpace(strings.ToLower(answer))
if answer != "y" && answer != "yes" {
fmt.Println("Aborted.")
return
}
}
// Apply migrations.
if err := migrate.DefaultRegistry.RunAllConfig(configDir, dataDir); err != nil {
fmt.Fprintf(os.Stderr, "Migration failed: %v\n", err)
os.Exit(1)
}
fmt.Println("\nDone. Restart services to apply: sudo systemctl restart spinifex.target")
}
@@ -7,7 +7,7 @@ import (
"strings"
"time"
"github.com/mulgadc/hive/hive/daemon"
"github.com/mulgadc/spinifex/spinifex/daemon"
"github.com/nats-io/nats.go"
"github.com/spf13/cobra"
)
@@ -77,7 +77,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
// Execute phases sequentially (except INFRA which is fire-and-forget)
for _, phase := range phases {
topic := "hive.cluster.shutdown." + phase
topic := "spinifex.cluster.shutdown." + phase
req := daemon.ShutdownRequest{
Phase: phase,
Force: force,
@@ -106,7 +106,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
// For DRAIN phase, subscribe to progress updates
var progressSub *nats.Subscription
if phase == "drain" {
progressSub, err = nc.Subscribe("hive.cluster.shutdown.progress", func(msg *nats.Msg) {
progressSub, err = nc.Subscribe("spinifex.cluster.shutdown.progress", func(msg *nats.Msg) {
var progress daemon.ShutdownProgress
if err := json.Unmarshal(msg.Data, &progress); err != nil {
return
+47 -21
View File
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -21,12 +22,14 @@ import (
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/types"
"github.com/mulgadc/hive/hive/utils"
"github.com/mulgadc/spinifex/spinifex/admin"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/types"
"github.com/mulgadc/spinifex/spinifex/utils"
"github.com/nats-io/nats.go"
"github.com/pterm/pterm"
"github.com/spf13/cobra"
@@ -75,14 +78,20 @@ func loadConfigAndConnect() (*config.ClusterConfig, *nats.Conn, error) {
}
// Backfill BaseDir from config file path when not set in the config.
// Config lives at <baseDir>/config/hive.toml, so baseDir is two levels up.
if nodeConfig, ok := cfg.Nodes[cfg.Node]; ok && nodeConfig.BaseDir == "" {
nodeConfig.BaseDir = filepath.Dir(filepath.Dir(cfgPath))
if isProductionLayout() {
// Production: BaseDir is /var/lib/spinifex (data dir).
// /var/lib/spinifex/config is a symlink to /etc/spinifex.
nodeConfig.BaseDir = DefaultDataDir()
} else {
// Dev: config lives at <baseDir>/config/spinifex.toml, so baseDir is two levels up.
nodeConfig.BaseDir = filepath.Dir(filepath.Dir(cfgPath))
}
cfg.Nodes[cfg.Node] = nodeConfig
}
nodeConfig := cfg.Nodes[cfg.Node]
nc, err := utils.ConnectNATS(nodeConfig.NATS.Host, nodeConfig.NATS.ACL.Token)
nc, err := utils.ConnectNATS(admin.DialTarget(nodeConfig.NATS.Host), nodeConfig.NATS.ACL.Token, nodeConfig.NATS.CACert)
if err != nil {
return nil, nil, fmt.Errorf("failed to connect to NATS: %w", err)
}
@@ -138,6 +147,20 @@ func formatUptime(seconds int64) string {
return fmt.Sprintf("%dm", mins)
}
func formatRoles(resp types.NodeStatusResponse) string {
var roles []string
if resp.NATSRole != "" {
roles = append(roles, "nats:"+resp.NATSRole)
}
if resp.PredastoreRole != "" {
roles = append(roles, "predastore:"+resp.PredastoreRole)
}
if len(roles) == 0 {
return "-"
}
return strings.Join(roles, ",")
}
func formatMemGB(gb float64) string {
if gb >= 1.0 {
return fmt.Sprintf("%.1fGi", gb)
@@ -154,7 +177,7 @@ func runGetNodes(cmd *cobra.Command, args []string) {
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.status", timeout)
responses, err := collectResponses(nc, "spinifex.node.status", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
@@ -172,7 +195,7 @@ func runGetNodes(cmd *cobra.Command, args []string) {
// Build table: union of config-known nodes + NATS responders
tableData := pterm.TableData{
{"NAME", "STATUS", "IP", "REGION", "AZ", "UPTIME", "VMs", "SERVICES"},
{"NAME", "STATUS", "ROLES", "IP", "REGION", "AZ", "UPTIME", "VMs", "SERVICES"},
}
// Collect all node names: config + responded (union)
@@ -195,17 +218,19 @@ func runGetNodes(cmd *cobra.Command, args []string) {
tableData = append(tableData, []string{
resp.Node,
resp.Status,
formatRoles(resp),
resp.Host,
resp.Region,
resp.AZ,
formatUptime(resp.Uptime),
fmt.Sprintf("%d", resp.VMCount),
strconv.Itoa(resp.VMCount),
strings.Join(resp.Services, ","),
})
} else {
tableData = append(tableData, []string{
name,
"NotReady",
"-",
nodeCfg.Host,
nodeCfg.Region,
nodeCfg.AZ,
@@ -228,7 +253,7 @@ func runGetVMs(cmd *cobra.Command, args []string) {
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.vms", timeout)
responses, err := collectResponses(nc, "spinifex.node.vms", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
@@ -236,6 +261,7 @@ func runGetVMs(cmd *cobra.Command, args []string) {
type vmRow struct {
types.VMInfo
Node string
Host string
}
@@ -277,7 +303,7 @@ func runGetVMs(cmd *cobra.Command, args []string) {
v.InstanceID,
v.Status,
v.InstanceType,
fmt.Sprintf("%d", v.VCPU),
strconv.Itoa(v.VCPU),
formatMemGB(v.MemoryGB),
v.Node,
v.Host,
+49
View File
@@ -0,0 +1,49 @@
package cmd
import (
"testing"
"github.com/mulgadc/spinifex/spinifex/types"
"github.com/stretchr/testify/assert"
)
func TestFormatRoles(t *testing.T) {
tests := []struct {
name string
resp types.NodeStatusResponse
want string
}{
{
name: "no roles",
resp: types.NodeStatusResponse{},
want: "-",
},
{
name: "nats leader only",
resp: types.NodeStatusResponse{NATSRole: "leader"},
want: "nats:leader",
},
{
name: "predastore follower only",
resp: types.NodeStatusResponse{PredastoreRole: "follower"},
want: "predastore:follower",
},
{
name: "both roles",
resp: types.NodeStatusResponse{NATSRole: "leader", PredastoreRole: "follower"},
want: "nats:leader,predastore:follower",
},
{
name: "both leaders",
resp: types.NodeStatusResponse{NATSRole: "leader", PredastoreRole: "leader"},
want: "nats:leader,predastore:leader",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := formatRoles(tt.resp)
assert.Equal(t, tt.want, got)
})
}
}
+69
View File
@@ -0,0 +1,69 @@
package cmd
import (
"os"
"os/user"
"path/filepath"
)
// DefaultConfigDir returns the default configuration directory.
// Production: /etc/spinifex (when /etc/spinifex exists)
// Development: ~/spinifex/config
func DefaultConfigDir() string {
if isProductionLayout() {
return "/etc/spinifex"
}
return filepath.Join(realUserHomeDir(), "spinifex", "config")
}
// DefaultDataDir returns the default data directory.
// Production: /var/lib/spinifex
// Development: ~/spinifex
func DefaultDataDir() string {
if isProductionLayout() {
return "/var/lib/spinifex"
}
return filepath.Join(realUserHomeDir(), "spinifex")
}
// LogDirFor returns the log directory for a given data directory.
// Production: /var/log/spinifex (matches systemd ReadWritePaths)
// Development: <dataDir>/logs (supports custom per-node data dirs)
func LogDirFor(dataDir string) string {
if isProductionLayout() {
return "/var/log/spinifex"
}
return filepath.Join(dataDir, "logs")
}
// realUserHomeDir returns the home directory of the real (non-sudo) user.
// When running under sudo, SUDO_USER is set to the invoking user — resolve
// their home directory so config/data land in the right place.
func realUserHomeDir() string {
if sudoUser := os.Getenv("SUDO_USER"); sudoUser != "" {
if u, err := user.Lookup(sudoUser); err == nil {
return u.HomeDir
}
}
homeDir, _ := os.UserHomeDir()
return homeDir
}
// DefaultConfigFile returns the default path to spinifex.toml.
func DefaultConfigFile() string {
return filepath.Join(DefaultConfigDir(), "spinifex.toml")
}
// productionMarkerPath identifies a production install by directory presence.
// Created by setup.sh during binary install. Overridable in tests.
var productionMarkerPath = "/etc/spinifex"
// isProductionLayout returns true when running in a production install.
// No root check — allows non-root users (e.g. tf-user) to run CLI commands
// like `spx get nodes` without sudo or --config flags.
func isProductionLayout() bool {
if info, err := os.Stat(productionMarkerPath); err == nil && info.IsDir() {
return true
}
return false
}
+77
View File
@@ -0,0 +1,77 @@
package cmd
import (
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
)
// withProductionMarker temporarily overrides the marker path used to detect a
// production install. Tests that pass an empty string force "dev" mode by
// pointing at a non-existent path; tests that pass t.TempDir() force
// "production" mode by pointing at a real directory.
func withProductionMarker(t *testing.T, path string) {
t.Helper()
orig := productionMarkerPath
productionMarkerPath = path
t.Cleanup(func() { productionMarkerPath = orig })
}
func TestIsProductionLayout(t *testing.T) {
t.Run("absent marker returns false", func(t *testing.T) {
withProductionMarker(t, filepath.Join(t.TempDir(), "does-not-exist"))
assert.False(t, isProductionLayout())
})
t.Run("present marker returns true", func(t *testing.T) {
withProductionMarker(t, t.TempDir())
assert.True(t, isProductionLayout())
})
}
func TestDefaultPaths(t *testing.T) {
homeDir := t.TempDir()
t.Setenv("HOME", homeDir)
t.Setenv("SUDO_USER", "")
tests := []struct {
name string
productionMode bool
wantConfigDir string
wantDataDir string
wantConfigFile string
wantLogDir string // for LogDirFor("/data/dir")
}{
{
name: "dev layout",
productionMode: false,
wantConfigDir: filepath.Join(homeDir, "spinifex", "config"),
wantDataDir: filepath.Join(homeDir, "spinifex"),
wantConfigFile: filepath.Join(homeDir, "spinifex", "config", "spinifex.toml"),
wantLogDir: "/data/dir/logs",
},
{
name: "production layout",
productionMode: true,
wantConfigDir: "/etc/spinifex",
wantDataDir: "/var/lib/spinifex",
wantConfigFile: "/etc/spinifex/spinifex.toml",
wantLogDir: "/var/log/spinifex",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.productionMode {
withProductionMarker(t, t.TempDir())
} else {
withProductionMarker(t, filepath.Join(t.TempDir(), "absent"))
}
assert.Equal(t, tt.wantConfigDir, DefaultConfigDir())
assert.Equal(t, tt.wantDataDir, DefaultDataDir())
assert.Equal(t, tt.wantConfigFile, DefaultConfigFile())
assert.Equal(t, tt.wantLogDir, LogDirFor("/data/dir"))
})
}
}
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -19,7 +20,7 @@ import (
"fmt"
"os"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@@ -32,20 +33,17 @@ var (
// rootCmd represents the base command when called without any subcommands
var rootCmd = &cobra.Command{
Use: "hive",
Short: "Hive - Open source AWS-compatible platform for secure edge deployments",
Use: "spx",
Short: "Spinifex - Open source AWS-compatible platform for secure edge deployments",
Long: `
__/\\\________/\\\__/\\\\\\\\\\\__/\\\________/\\\__/\\\\\\\\\\\\\\\_
_\/\\\_______\/\\\_\/////\\\///__\/\\\_______\/\\\_\/\\\///////////__
_\/\\\_______\/\\\_____\/\\\_____\//\\\______/\\\__\/\\\_____________
_\/\\\\\\\\\\\\\\\_____\/\\\______\//\\\____/\\\___\/\\\\\\\\\\\_____
_\/\\\/////////\\\_____\/\\\_______\//\\\__/\\\____\/\\\///////______
_\/\\\_______\/\\\_____\/\\\________\//\\\/\\\_____\/\\\_____________
_\/\\\_______\/\\\_____\/\\\_________\//\\\\\______\/\\\_____________
_\/\\\_______\/\\\__/\\\\\\\\\\\______\//\\\_______\/\\\\\\\\\\\\\\\_
_\///________\///__\///////////________\///________\///////////////__
_____ _____ _____ _ _ _____ ______ ________ __
/ ____| __ \_ _| \ | |_ _| ____| ____\ \ / /
| (___ | |__) || | | \| | | | | |__ | |__ \ V /
\___ \| ___/ | | | . ‘ | | | | __| | __| > <
____) | | _| |_| |\ |_| |_| | | |____ / . \
|_____/|_| |_____|_| \_|_____|_| |______/_/ \_\
Hive – Open source AWS-compatible platform for secure edge deployments.
Spinifex – Open source AWS-compatible platform for secure edge deployments.
Run EC2, VPC, S3, and EBS-like services on bare metal with full control.
Built for environments where running in the cloud isn’t an option.
Whether you’re deploying to edge sites, private data-centers, or operating
@@ -67,45 +65,44 @@ func init() {
// Global flags
rootCmd.PersistentFlags().String("config", "", "config file (required)")
viper.BindEnv("config", "HIVE_CONFIG_PATH")
viper.BindEnv("config", "SPINIFEX_CONFIG_PATH")
viper.BindPFlag("config", rootCmd.PersistentFlags().Lookup("config"))
// Authentication (access_key, secret)
rootCmd.PersistentFlags().String("access-key", "", "AWS access key (overrides config file and env)")
viper.BindEnv("access-key", "HIVE_ACCESS_KEY")
viper.BindEnv("access-key", "SPINIFEX_ACCESS_KEY")
viper.BindPFlag("access-key", rootCmd.PersistentFlags().Lookup("access-key"))
rootCmd.PersistentFlags().String("secret-key", "", "AWS secret key (overrides config file and env)")
viper.BindEnv("secret-key", "HIVE_SECRET_KEY")
viper.BindEnv("secret-key", "SPINIFEX_SECRET_KEY")
viper.BindPFlag("secret-key", rootCmd.PersistentFlags().Lookup("secret-key"))
rootCmd.PersistentFlags().String("host", "", "AWS Endpoint (overrides config file and env)")
viper.BindEnv("host", "HIVE_HOST")
viper.BindEnv("host", "SPINIFEX_HOST")
viper.BindPFlag("host", rootCmd.PersistentFlags().Lookup("host"))
// Viperblock config
rootCmd.PersistentFlags().String("base-dir", "", "Viperblock base directory (overrides config file and env)")
viper.BindEnv("base-dir", "HIVE_BASE_DIR")
viper.BindEnv("base-dir", "SPINIFEX_BASE_DIR")
viper.BindPFlag("base-dir", rootCmd.PersistentFlags().Lookup("base-dir"))
// NATS specific flags
rootCmd.PersistentFlags().String("nats-host", "", "NATS server host (overrides config file and env)")
viper.BindEnv("nats-host", "HIVE_NATS_HOST")
viper.BindEnv("nats-host", "SPINIFEX_NATS_HOST")
viper.BindPFlag("nats-host", rootCmd.PersistentFlags().Lookup("nats-host"))
rootCmd.PersistentFlags().String("nats-token", "", "NATS authentication token (overrides config file and env)")
viper.BindEnv("nats-token", "HIVE_NATS_TOKEN")
viper.BindEnv("nats-token", "SPINIFEX_NATS_TOKEN")
viper.BindPFlag("nats-token", rootCmd.PersistentFlags().Lookup("nats-token"))
rootCmd.PersistentFlags().String("nats-subject", "", "NATS subscription subject (overrides config file and env)")
viper.BindEnv("nats-subject", "HIVE_NATS_SUBJECT")
viper.BindEnv("nats-subject", "SPINIFEX_NATS_SUBJECT")
viper.BindPFlag("nats-subject", rootCmd.PersistentFlags().Lookup("nats-subject"))
// Bind flags to viper
//viper.BindPFlag("nats.host", rootCmd.PersistentFlags().Lookup("nats-host"))
//viper.BindPFlag("nats.acl.token", rootCmd.PersistentFlags().Lookup("nats-token"))
//viper.BindPFlag("nats.sub.subject", rootCmd.PersistentFlags().Lookup("nats-subject"))
}
// initConfig reads in config file and ENV variables if set.
@@ -121,6 +118,6 @@ func initConfig() {
fmt.Fprintf(os.Stderr, "Error: failed to load config %s: %v\n", cfgFile, err)
os.Exit(1)
}
// No config specified — continue with env/defaults (e.g., hive --help)
// No config specified — continue with env/defaults (e.g., spx --help)
}
}
@@ -1,17 +1,18 @@
/*
Copyright © 2025 Mulga Defense Corporation
Copyright © 2026 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
http://www.apache.org/licenses/LICENSE-2.0
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package cmd
@@ -21,21 +22,21 @@ import (
"os"
"strings"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/service"
"github.com/mulgadc/hive/hive/services/hiveui"
"github.com/mulgadc/hive/hive/services/nats"
"github.com/mulgadc/hive/hive/services/predastore"
"github.com/mulgadc/hive/hive/services/viperblockd"
"github.com/mulgadc/hive/hive/services/vpcd"
"github.com/mulgadc/predastore/s3"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/service"
"github.com/mulgadc/spinifex/spinifex/services/awsgw"
"github.com/mulgadc/spinifex/spinifex/services/nats"
"github.com/mulgadc/spinifex/spinifex/services/predastore"
"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
"github.com/mulgadc/spinifex/spinifex/services/viperblockd"
"github.com/mulgadc/spinifex/spinifex/services/vpcd"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var serviceCmd = &cobra.Command{
Use: "service",
Short: "Manage Hive services",
Short: "Manage Spinifex services",
}
var predastoreCmd = &cobra.Command{
@@ -53,9 +54,9 @@ var natsCmd = &cobra.Command{
Short: "Manage the nats service",
}
var hiveCmd = &cobra.Command{
Use: "hive",
Short: "Manage the hive service",
var spinifexCmd = &cobra.Command{
Use: "spinifex",
Short: "Manage the spinifex service",
}
var awsgwCmd = &cobra.Command{
@@ -68,10 +69,10 @@ var vpcdCmd = &cobra.Command{
Short: "Manage the vpcd (VPC daemon) service",
}
var hiveUICmd = &cobra.Command{
Use: "hive-ui",
Aliases: []string{"ui", "hiveui"},
Short: "Manage the hive-ui service",
var spinifexUICmd = &cobra.Command{
Use: "spinifex-ui",
Aliases: []string{"ui", "spinifexui"},
Short: "Manage the spinifex-ui service",
}
var predastoreStartCmd = &cobra.Command{
@@ -114,13 +115,11 @@ var predastoreStartCmd = &cobra.Command{
return
}
backendType := viper.GetString("backend")
var backend s3.BackendType
encryptionKeyFile := viper.GetString("encryption-key-file")
if backendType == "distributed" {
backend = s3.BackendDistributed
} else {
backend = s3.BackendFilesystem
if encryptionKeyFile == "" {
fmt.Println("Encryption key file is not set")
return
}
nodeID := viper.GetInt("node-id")
@@ -136,8 +135,9 @@ var predastoreStartCmd = &cobra.Command{
TlsCert: tlsCert,
TlsKey: tlsKey,
Backend: backend,
NodeID: nodeID,
EncryptionKeyFile: encryptionKeyFile,
NodeID: nodeID,
PprofEnabled: pprofEnabled,
PprofOutputPath: pprofOutput,
@@ -243,13 +243,13 @@ var viperblockStartCmd = &cobra.Command{
accessKey := viper.GetString("access-key")
if accessKey != "" {
fmt.Println("Overwriting access-key:", accessKey)
fmt.Println("Overwriting access-key: ****")
nodeConfig.Predastore.AccessKey = accessKey
}
secretKey := viper.GetString("secret-key")
if secretKey != "" {
fmt.Println("Overwriting secret-key:", secretKey)
fmt.Println("Overwriting secret-key: ****")
nodeConfig.Predastore.SecretKey = secretKey
}
@@ -285,6 +285,8 @@ var viperblockStartCmd = &cobra.Command{
service, err := service.New("viperblock", &viperblockd.Config{
NatsHost: nodeConfig.NATS.Host,
NatsToken: nodeConfig.NATS.ACL.Token,
NatsCACert: nodeConfig.NATS.CACert,
PluginPath: pluginPath,
S3Host: nodeConfig.Predastore.Host,
Bucket: nodeConfig.Predastore.Bucket,
@@ -308,7 +310,7 @@ var viperblockStartCmd = &cobra.Command{
return
}
fmt.Println("Viperblock service started", service)
fmt.Println("Viperblock service started")
},
}
@@ -410,12 +412,12 @@ var natsStatusCmd = &cobra.Command{
},
}
// Repeat for hive
var hiveStartCmd = &cobra.Command{
// Repeat for spinifex
var spinifexStartCmd = &cobra.Command{
Use: "start",
Short: "Start the hive service",
Short: "Start the spinifex service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting hive service...")
fmt.Println("Starting spinifex service...")
cfgFile := viper.GetString("config")
@@ -451,53 +453,53 @@ var hiveStartCmd = &cobra.Command{
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
svc, err := service.New("hive", clusterConfig)
svc, err := service.New("spinifex", clusterConfig)
if err != nil {
fmt.Println("Error starting hive service:", err)
fmt.Println("Error starting spinifex service:", err)
return
}
// Set config path for cluster manager
if hiveSvc, ok := svc.(interface{ SetConfigPath(string) }); ok {
hiveSvc.SetConfigPath(cfgFile)
if spxSvc, ok := svc.(interface{ SetConfigPath(string) }); ok {
spxSvc.SetConfigPath(cfgFile)
}
if _, err = svc.Start(); err != nil {
fmt.Println("Error starting hive service:", err)
fmt.Println("Error starting spinifex service:", err)
os.Exit(1)
}
fmt.Println("HIVE service started")
fmt.Println("Spinifex service started")
},
}
var hiveStopCmd = &cobra.Command{
var spinifexStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the hive service",
Short: "Stop the spinifex service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping hive service...")
fmt.Println("Stopping spinifex service...")
service, err := service.New("hive", &config.ClusterConfig{})
service, err := service.New("spinifex", &config.ClusterConfig{})
if err != nil {
fmt.Println("Error stopping hive service:", err)
fmt.Println("Error stopping spinifex service:", err)
return
}
if err = service.Stop(); err != nil {
fmt.Println("Error stopping hive service:", err)
fmt.Println("Error stopping spinifex service:", err)
os.Exit(1)
}
fmt.Println("Hive service stopped")
fmt.Println("Spinifex service stopped")
},
}
var hiveStatusCmd = &cobra.Command{
var spinifexStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the hive service",
Short: "Get status of the spinifex service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Hive service status: ...")
fmt.Println("Spinifex service status: ...")
},
}
@@ -556,6 +558,7 @@ var awsgwStartCmd = &cobra.Command{
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
awsgw.SetBuildInfo(Version, Commit)
service, err := service.New("awsgw", clusterConfig)
if err != nil {
@@ -601,18 +604,18 @@ var awsgwStatusCmd = &cobra.Command{
},
}
var hiveUIStartCmd = &cobra.Command{
var spinifexUIStartCmd = &cobra.Command{
Use: "start",
Short: "Start the hive-ui service",
Short: "Start the spinifex-ui service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting hive-ui service...")
fmt.Println("Starting spinifex-ui service...")
port := viper.GetInt("hive-ui-port")
host := viper.GetString("hive-ui-host")
tlsCert := viper.GetString("hive-ui-tls-cert")
tlsKey := viper.GetString("hive-ui-tls-key")
port := viper.GetInt("spinifex-ui-port")
host := viper.GetString("spinifex-ui-host")
tlsCert := viper.GetString("spinifex-ui-tls-cert")
tlsKey := viper.GetString("spinifex-ui-tls-key")
svc, err := service.New("hive-ui", &hiveui.Config{
svc, err := service.New("spinifex-ui", &spinifexui.Config{
Port: port,
Host: host,
TLSCert: tlsCert,
@@ -620,60 +623,86 @@ var hiveUIStartCmd = &cobra.Command{
})
if err != nil {
fmt.Println("Error starting hive-ui service:", err)
fmt.Println("Error starting spinifex-ui service:", err)
return
}
if _, err = svc.Start(); err != nil {
fmt.Println("Error starting hive-ui service:", err)
fmt.Println("Error starting spinifex-ui service:", err)
os.Exit(1)
}
fmt.Println("hive-ui service started")
fmt.Println("spinifex-ui service started")
},
}
var hiveUIStopCmd = &cobra.Command{
var spinifexUIStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the hive-ui service",
Short: "Stop the spinifex-ui service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping hive-ui service...")
fmt.Println("Stopping spinifex-ui service...")
svc, err := service.New("hive-ui", &hiveui.Config{})
svc, err := service.New("spinifex-ui", &spinifexui.Config{})
if err != nil {
fmt.Println("Error stopping hive-ui service:", err)
fmt.Println("Error stopping spinifex-ui service:", err)
return
}
if err = svc.Stop(); err != nil {
fmt.Println("Error stopping hive-ui service:", err)
fmt.Println("Error stopping spinifex-ui service:", err)
os.Exit(1)
}
fmt.Println("hive-ui service stopped")
fmt.Println("spinifex-ui service stopped")
},
}
var hiveUIStatusCmd = &cobra.Command{
var spinifexUIStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the hive-ui service",
Short: "Get status of the spinifex-ui service",
Run: func(cmd *cobra.Command, args []string) {
svc, err := service.New("hive-ui", &hiveui.Config{})
svc, err := service.New("spinifex-ui", &spinifexui.Config{})
if err != nil {
fmt.Println("Error getting hive-ui service status:", err)
fmt.Println("Error getting spinifex-ui service status:", err)
return
}
status, err := svc.Status()
if err != nil {
fmt.Println("Error getting hive-ui service status:", err)
fmt.Println("Error getting spinifex-ui service status:", err)
return
}
fmt.Println("hive-ui service status:", status)
fmt.Println("spinifex-ui service status:", status)
},
}
// checkLegacyWanBridgeKey fails vpcd startup if the deprecated `wan_bridge`
// TOML key or `SPINIFEX_VPCD_WAN_BRIDGE` env-var is present. The key was
// renamed to `dhcp_bind_bridge` (see mulga-998.a) because the old name misled
// operators into pointing at the OVN-side bridge ("br-ext"), which never sees
// LAN DHCP traffic. Per D3: no silent alias, no auto-rewrite — operator must
// rename the key before vpcd will start.
func checkLegacyWanBridgeKey(node, cfgFile string) error {
legacyInTOML := viper.IsSet("nodes." + node + ".vpcd.wan_bridge")
legacyInEnv := os.Getenv("SPINIFEX_VPCD_WAN_BRIDGE") != ""
if !legacyInTOML && !legacyInEnv {
return nil
}
source := cfgFile
if legacyInEnv {
source = "env SPINIFEX_VPCD_WAN_BRIDGE"
}
return fmt.Errorf(
"vpcd: deprecated 'wan_bridge' key found in %s. "+
"Rename 'wan_bridge' to 'dhcp_bind_bridge'. The value may also be wrong — verify it is "+
"the Linux bridge holding your WAN NIC (veth mode) or the OVS bridge holding your WAN NIC (direct mode); "+
"never 'br-ext'. Typical value on a consumer-router LAN: 'br-wan'. "+
"Then: sudo systemctl restart spinifex-vpcd",
source,
)
}
var vpcdStartCmd = &cobra.Command{
Use: "start",
Short: "Start the vpcd service",
@@ -692,15 +721,65 @@ var vpcdStartCmd = &cobra.Command{
return
}
if err := checkLegacyWanBridgeKey(clusterConfig.Node, cfgFile); err != nil {
slog.Error(err.Error())
os.Exit(1)
}
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
// Map cluster-wide external pools to vpcd config
var extPools []vpcd.ExternalPoolConfig
for _, p := range clusterConfig.Network.ExternalPools {
extPools = append(extPools, vpcd.ExternalPoolConfig{
Name: p.Name,
Source: p.Source,
RangeStart: p.RangeStart,
RangeEnd: p.RangeEnd,
Gateway: p.Gateway,
GatewayIP: p.GatewayIP,
PrefixLen: p.PrefixLen,
DNSServers: p.DNSServers,
Region: p.Region,
AZ: p.AZ,
DhcpBindBridge: nodeConfig.VPCD.DhcpBindBridge,
GwLrpRangeStart: p.GwLrpRangeStart,
GwLrpRangeEnd: p.GwLrpRangeEnd,
})
}
var bootstrap *vpcd.BootstrapVPC
if clusterConfig.Bootstrap.VpcId != "" {
bootstrap = &vpcd.BootstrapVPC{
AccountID: clusterConfig.Bootstrap.AccountID,
VpcId: clusterConfig.Bootstrap.VpcId,
SubnetId: clusterConfig.Bootstrap.SubnetId,
IgwId: clusterConfig.Bootstrap.IgwId,
Cidr: clusterConfig.Bootstrap.Cidr,
SubnetCidr: clusterConfig.Bootstrap.SubnetCidr,
}
}
baseDir := viper.GetString("base-dir")
if baseDir != "" {
fmt.Println("Overwriting vpcd base-dir to:", baseDir)
nodeConfig.BaseDir = baseDir
}
svc, err := service.New("vpcd", &vpcd.Config{
NatsHost: nodeConfig.NATS.Host,
NatsToken: nodeConfig.NATS.ACL.Token,
OVNNBAddr: nodeConfig.VPCD.OVNNBAddr,
OVNSBAddr: nodeConfig.VPCD.OVNSBAddr,
BaseDir: nodeConfig.BaseDir,
Debug: false,
NatsHost: nodeConfig.NATS.Host,
NatsToken: nodeConfig.NATS.ACL.Token,
NatsCACert: nodeConfig.NATS.CACert,
OVNNBAddr: nodeConfig.VPCD.OVNNBAddr,
OVNSBAddr: nodeConfig.VPCD.OVNSBAddr,
BaseDir: nodeConfig.BaseDir,
Debug: false,
ExternalMode: clusterConfig.Network.ExternalMode,
ExternalPools: extPools,
Bootstrap: bootstrap,
ExternalInterface: nodeConfig.VPCD.ExternalInterface,
DhcpBindBridge: nodeConfig.VPCD.DhcpBindBridge,
BridgeMode: nodeConfig.VPCD.BridgeMode,
})
if err != nil {
fmt.Println("Error starting vpcd service:", err)
@@ -744,8 +823,7 @@ var vpcdStatusCmd = &cobra.Command{
}
func init() {
viper.SetEnvPrefix("HIVE") // Prefix for environment variables
viper.SetEnvPrefix("SPINIFEX") // Prefix for environment variables
viper.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))
viper.AutomaticEnv() // Read environment variables automatically
@@ -756,47 +834,54 @@ func init() {
// Predastore Port
predastoreCmd.PersistentFlags().Int("port", 8443, "Predastore (S3) port")
viper.BindEnv("port", "HIVE_PREDASTORE_PORT")
viper.BindEnv("port", "SPINIFEX_PREDASTORE_PORT")
viper.BindPFlag("port", predastoreCmd.PersistentFlags().Lookup("port"))
// Predastore Host
predastoreCmd.PersistentFlags().String("host", "0.0.0.0", "Predastore (S3) host")
viper.BindEnv("host", "HIVE_PREDASTORE_HOST")
viper.BindEnv("host", "SPINIFEX_PREDASTORE_HOST")
viper.BindPFlag("host", predastoreCmd.PersistentFlags().Lookup("host"))
// Base path
predastoreCmd.PersistentFlags().String("base-path", "", "Predastore (S3) base path")
viper.BindEnv("base-path", "HIVE_PREDASTORE_BASE_PATH")
viper.BindEnv("base-path", "SPINIFEX_PREDASTORE_BASE_PATH")
viper.BindPFlag("base-path", predastoreCmd.PersistentFlags().Lookup("base-path"))
// Predastore Config Path
predastoreCmd.PersistentFlags().String("config-path", "", "Predastore (S3) config path")
viper.BindEnv("config-path", "HIVE_PREDASTORE_CONFIG_PATH")
viper.BindEnv("config-path", "SPINIFEX_PREDASTORE_CONFIG_PATH")
viper.BindPFlag("config-path", predastoreCmd.PersistentFlags().Lookup("config-path"))
// Predastore Debug
predastoreCmd.PersistentFlags().Bool("debug", false, "Predastore (S3) debug")
viper.BindEnv("debug", "HIVE_PREDASTORE_DEBUG")
viper.BindEnv("debug", "SPINIFEX_PREDASTORE_DEBUG")
viper.BindPFlag("debug", predastoreCmd.PersistentFlags().Lookup("debug"))
// Predastore TLS Cert
predastoreCmd.PersistentFlags().String("tls-cert", "", "Predastore (S3) TLS certificate")
viper.BindEnv("tls-cert", "HIVE_PREDASTORE_TLS_CERT")
viper.BindEnv("tls-cert", "SPINIFEX_PREDASTORE_TLS_CERT")
viper.BindPFlag("tls-cert", predastoreCmd.PersistentFlags().Lookup("tls-cert"))
// Predastore TLS Key
predastoreCmd.PersistentFlags().String("tls-key", "", "Predastore (S3) TLS key")
viper.BindEnv("tls-key", "HIVE_PREDASTORE_TLS_KEY")
viper.BindEnv("tls-key", "SPINIFEX_PREDASTORE_TLS_KEY")
viper.BindPFlag("tls-key", predastoreCmd.PersistentFlags().Lookup("tls-key"))
// Predastore at-rest encryption master key (per node; mode 0600)
predastoreCmd.PersistentFlags().String("encryption-key-file", "", "Path to this node's 32-byte AES-256 master key file (required)")
viper.BindEnv("encryption-key-file", "SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE")
viper.BindPFlag("encryption-key-file", predastoreCmd.PersistentFlags().Lookup("encryption-key-file"))
// Predastore Backend
predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
viper.BindEnv("backend", "HIVE_PREDASTORE_BACKEND")
viper.BindEnv("backend", "SPINIFEX_PREDASTORE_BACKEND")
viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
// Predastore Node ID
predastoreCmd.PersistentFlags().Int("node-id", 0, "Predastore (S3) node ID")
viper.BindEnv("node-id", "HIVE_PREDASTORE_NODE_ID")
// Predastore Node ID. Default -1 is dev mode (launch every configured
// QUIC node in-process). Production deployments set this to the node's
// real ID (>= 1) via SPINIFEX_PREDASTORE_NODE_ID or --node-id.
predastoreCmd.PersistentFlags().Int("node-id", -1, "Predastore (S3) node ID (-1 = dev mode, >= 1 = production)")
viper.BindEnv("node-id", "SPINIFEX_PREDASTORE_NODE_ID")
viper.BindPFlag("node-id", predastoreCmd.PersistentFlags().Lookup("node-id"))
// Predastore CPU Profiling
@@ -815,20 +900,20 @@ func init() {
serviceCmd.AddCommand(viperblockCmd)
viperblockCmd.PersistentFlags().String("s3-host", "0.0.0.0:8443", "Predastore (S3) host URI")
viper.BindEnv("s3-host", "HIVE_VIPERBLOCK_S3_HOST")
viperblockCmd.PersistentFlags().String("s3-host", "", "Predastore (S3) host URI")
viper.BindEnv("s3-host", "SPINIFEX_VIPERBLOCK_S3_HOST")
viper.BindPFlag("s3-host", predastoreCmd.PersistentFlags().Lookup("s3-host"))
viperblockCmd.PersistentFlags().String("s3-bucket", "predastore", "Predastore (S3) bucket")
viper.BindEnv("s3-bucket", "HIVE_VIPERBLOCK_S3_BUCKET")
viper.BindEnv("s3-bucket", "SPINIFEX_VIPERBLOCK_S3_BUCKET")
viper.BindPFlag("s3-bucket", predastoreCmd.PersistentFlags().Lookup("s3-bucket"))
viperblockCmd.PersistentFlags().String("s3-region", "ap-southeast-2", "Predastore (S3) region")
viper.BindEnv("s3-region", "HIVE_VIPERBLOCK_S3_REGION")
viper.BindEnv("s3-region", "SPINIFEX_VIPERBLOCK_S3_REGION")
viper.BindPFlag("s3-region", predastoreCmd.PersistentFlags().Lookup("s3-region"))
viperblockCmd.PersistentFlags().String("plugin-path", "/opt/hive/lib/nbdkit-viperblock-plugin.so", "Pathname to the nbdkit viperblockplugin")
viper.BindEnv("plugin-path", "HIVE_VIPERBLOCK_PLUGIN_PATH")
viperblockCmd.PersistentFlags().String("plugin-path", "/opt/spinifex/lib/nbdkit-viperblock-plugin.so", "Pathname to the nbdkit viperblockplugin")
viper.BindEnv("plugin-path", "SPINIFEX_VIPERBLOCK_PLUGIN_PATH")
viper.BindPFlag("plugin-path", predastoreCmd.PersistentFlags().Lookup("plugin-path"))
viperblockCmd.AddCommand(viperblockStartCmd)
@@ -844,83 +929,83 @@ func init() {
// Add NATS flags
natsCmd.PersistentFlags().Int("port", 4222, "NATS server port")
viper.BindEnv("port", "HIVE_NATS_PORT")
viper.BindEnv("port", "SPINIFEX_NATS_PORT")
viper.BindPFlag("port", natsCmd.PersistentFlags().Lookup("port"))
natsCmd.PersistentFlags().String("host", "0.0.0.0", "NATS server host")
viper.BindEnv("host", "HIVE_NATS_HOST")
viper.BindEnv("host", "SPINIFEX_NATS_HOST")
viper.BindPFlag("host", natsCmd.PersistentFlags().Lookup("host"))
natsCmd.PersistentFlags().Bool("debug", false, "Enable debug logging")
viper.BindEnv("debug", "HIVE_NATS_DEBUG")
viper.BindEnv("debug", "SPINIFEX_NATS_DEBUG")
viper.BindPFlag("debug", natsCmd.PersistentFlags().Lookup("debug"))
natsCmd.PersistentFlags().String("data-dir", "", "NATS data directory")
viper.BindEnv("data-dir", "HIVE_NATS_DATA_DIR")
viper.BindEnv("data-dir", "SPINIFEX_NATS_DATA_DIR")
viper.BindPFlag("data-dir", natsCmd.PersistentFlags().Lookup("data-dir"))
natsCmd.PersistentFlags().Bool("jetstream", false, "Enable JetStream")
viper.BindEnv("jetstream", "HIVE_NATS_JETSTREAM")
viper.BindEnv("jetstream", "SPINIFEX_NATS_JETSTREAM")
viper.BindPFlag("jetstream", natsCmd.PersistentFlags().Lookup("jetstream"))
// Hive
serviceCmd.AddCommand(hiveCmd)
// Spinifex
serviceCmd.AddCommand(spinifexCmd)
hiveCmd.AddCommand(hiveStartCmd)
hiveCmd.AddCommand(hiveStopCmd)
hiveCmd.AddCommand(hiveStatusCmd)
spinifexCmd.AddCommand(spinifexStartCmd)
spinifexCmd.AddCommand(spinifexStopCmd)
spinifexCmd.AddCommand(spinifexStatusCmd)
hiveCmd.PersistentFlags().String("wal-dir", "", "Write-ahead log (WAL) directory. Place on high-speed NVMe disk, or tmpfs for development.")
viper.BindEnv("wal-dir", "HIVE_WAL_DIR")
viper.BindPFlag("wal-dir", hiveCmd.PersistentFlags().Lookup("wal-dir"))
spinifexCmd.PersistentFlags().String("wal-dir", "", "Write-ahead log (WAL) directory. Place on high-speed NVMe disk, or tmpfs for development.")
viper.BindEnv("wal-dir", "SPINIFEX_WAL_DIR")
viper.BindPFlag("wal-dir", spinifexCmd.PersistentFlags().Lookup("wal-dir"))
// AWS GW
serviceCmd.AddCommand(awsgwCmd)
awsgwCmd.PersistentFlags().String("host", "0.0.0.0:9999", "AWS Gateway server host")
viper.BindEnv("host", "HIVE_AWSGW_HOST")
viper.BindEnv("host", "SPINIFEX_AWSGW_HOST")
viper.BindPFlag("host", awsgwCmd.PersistentFlags().Lookup("host"))
// AWS GW TLS Cert
awsgwCmd.PersistentFlags().String("tls-cert", "", "AWS Gateway TLS certificate")
viper.BindEnv("tls-cert", "HIVE_AWSGW_TLS_CERT")
viper.BindEnv("tls-cert", "SPINIFEX_AWSGW_TLS_CERT")
viper.BindPFlag("tls-cert", awsgwCmd.PersistentFlags().Lookup("tls-cert"))
// AWS GW TLS Key
awsgwCmd.PersistentFlags().String("tls-key", "", "AWS Gateway TLS key")
viper.BindEnv("tls-key", "HIVE_AWSGW_TLS_KEY")
viper.BindEnv("tls-key", "SPINIFEX_AWSGW_TLS_KEY")
viper.BindPFlag("tls-key", awsgwCmd.PersistentFlags().Lookup("tls-key"))
awsgwCmd.PersistentFlags().Bool("debug", false, "AWS Gateway Debug")
viper.BindEnv("debug", "HIVE_AWSGW_DEBUG")
viper.BindEnv("debug", "SPINIFEX_AWSGW_DEBUG")
viper.BindPFlag("debug", awsgwCmd.PersistentFlags().Lookup("debug"))
awsgwCmd.AddCommand(awsgwStartCmd)
awsgwCmd.AddCommand(awsgwStopCmd)
awsgwCmd.AddCommand(awsgwStatusCmd)
// hive-ui
serviceCmd.AddCommand(hiveUICmd)
// spinifex-ui
serviceCmd.AddCommand(spinifexUICmd)
hiveUICmd.PersistentFlags().Int("port", 3000, "hive-ui server port")
viper.BindEnv("hive-ui-port", "HIVE_UI_PORT")
viper.BindPFlag("hive-ui-port", hiveUICmd.PersistentFlags().Lookup("port"))
spinifexUICmd.PersistentFlags().Int("port", 3000, "spinifex-ui server port")
viper.BindEnv("spinifex-ui-port", "SPINIFEX_UI_PORT")
viper.BindPFlag("spinifex-ui-port", spinifexUICmd.PersistentFlags().Lookup("port"))
hiveUICmd.PersistentFlags().String("host", "0.0.0.0", "hive-ui server host")
viper.BindEnv("hive-ui-host", "HIVE_UI_HOST")
viper.BindPFlag("hive-ui-host", hiveUICmd.PersistentFlags().Lookup("host"))
spinifexUICmd.PersistentFlags().String("host", "0.0.0.0", "spinifex-ui server host")
viper.BindEnv("spinifex-ui-host", "SPINIFEX_UI_HOST")
viper.BindPFlag("spinifex-ui-host", spinifexUICmd.PersistentFlags().Lookup("host"))
hiveUICmd.PersistentFlags().String("tls-cert", "", "TLS certificate path")
viper.BindEnv("hive-ui-tls-cert", "HIVE_UI_TLS_CERT")
viper.BindPFlag("hive-ui-tls-cert", hiveUICmd.PersistentFlags().Lookup("tls-cert"))
spinifexUICmd.PersistentFlags().String("tls-cert", "", "TLS certificate path")
viper.BindEnv("spinifex-ui-tls-cert", "SPINIFEX_UI_TLS_CERT")
viper.BindPFlag("spinifex-ui-tls-cert", spinifexUICmd.PersistentFlags().Lookup("tls-cert"))
hiveUICmd.PersistentFlags().String("tls-key", "", "TLS key path")
viper.BindEnv("hive-ui-tls-key", "HIVE_UI_TLS_KEY")
viper.BindPFlag("hive-ui-tls-key", hiveUICmd.PersistentFlags().Lookup("tls-key"))
spinifexUICmd.PersistentFlags().String("tls-key", "", "TLS key path")
viper.BindEnv("spinifex-ui-tls-key", "SPINIFEX_UI_TLS_KEY")
viper.BindPFlag("spinifex-ui-tls-key", spinifexUICmd.PersistentFlags().Lookup("tls-key"))
hiveUICmd.AddCommand(hiveUIStartCmd)
hiveUICmd.AddCommand(hiveUIStopCmd)
hiveUICmd.AddCommand(hiveUIStatusCmd)
spinifexUICmd.AddCommand(spinifexUIStartCmd)
spinifexUICmd.AddCommand(spinifexUIStopCmd)
spinifexUICmd.AddCommand(spinifexUIStatusCmd)
// vpcd
serviceCmd.AddCommand(vpcdCmd)
@@ -928,5 +1013,4 @@ func init() {
vpcdCmd.AddCommand(vpcdStartCmd)
vpcdCmd.AddCommand(vpcdStopCmd)
vpcdCmd.AddCommand(vpcdStatusCmd)
}
+28
View File
@@ -0,0 +1,28 @@
# AWS Gateway configuration file in TOML format.
version = "2"
region = "{{.Region}}"
# Enable verbose debugging
debug = false
# API request throttling (per-account, per-action token bucket)
[ratelimit]
enabled = false
rate = 20 # Default sustained requests per second per account per action
burst = 100 # Default maximum burst capacity
[ratelimit.action.RunInstances]
rate = 2
burst = 40
[ratelimit.action.TerminateInstances]
rate = 2
burst = 40
[ratelimit.action.StartInstances]
rate = 2
burst = 40
[ratelimit.action.StopInstances]
rate = 2
burst = 40

Some files were not shown because too many files have changed in this diff Show More