mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-16 01:22:00 +00:00
f8d025d07e8a8cd712d7b7e152a9774919471dcd
107 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 joshsiv-mulga
|
38373b2fc7 | fix(ci): generate epdu.env from secrets instead of pre-placed file | ||
|
joshsiv-mulga
|
e615d6f9e6 | fix(ci): remove BM_SKIP_EPDU now that ePDU SNMPv3 is provisioned | ||
|
Julian Sommer
|
97701737cb | feat: describe instance status (#221) | ||
|
joshsiv-mulga
|
a31fd8098c |
fix(ci): use sudo to clean workspace so root-owned rootfs cache is removed
The build-rootfs step creates root-owned files in the workspace; a plain rm -rf fails with permission denied on the next run's Clean Workspace step. |
||
|
Julian Sommer
|
aaa208b594 | fix: use dev for build and test | ||
|
Josh Siviour
|
45555dde49 |
test(ddil): flip Scenarios A/B/C to real assertions (#205)
* test(ddil): flip Scenarios A/B/C to real assertions Replaces the t.Skip stubs landed in #204 with the assertions described in docs/development/improvements/ddil-e2e-test-harness.md §3: - A NATSKill — KillNATS on node3, assert ModeStandalone within 30s, /local/instances still serves on the isolated node, majority /health on node1+node2 stays up, witness counters on all three nodes keep advancing, snapshot survives reconnect. - B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert ModeStandalone within 30s of restart, recovered instances present with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post PID equality dropped — see file comment; recovery + a non-zero PID give the same regression coverage without coupling to the daemon unit's KillMode. - C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep serving /health and /local/instances, isolated node3 reports ModeStandalone with /local/* responsive, witness counter on the partitioned side keeps advancing, heal converges and snapshot is preserved. Supporting changes: - harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore, the deployed daemon ClusterManager binds [nodes.<id>.daemon].host. The Phase-1 stub used a placeholder that would have blocked any /local/* assertion against the real cluster. - harness/witness.go — export WitnessVM.Terminate so scenarios can clean up their counter VMs from t.Cleanup. - scenarios/main_test.go — requireLiveCluster + launchWitnesses helpers to keep the three flipped scenarios tight. - TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED; TestCoverageDrift continues to enforce alignment with the package. Validation: built with -tags=e2e, dry-run scenarios pass, coverage drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against tofu cluster is the next step. * fix(ddil): countDropRules — single iptables -S call + filter Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs across both chains, but iptables -S takes a single optional chain — the second -S errors to stderr, the combined SSH buffer poisons the grep input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState. Switch to one `iptables -S` (lists every chain in iptables-save format), filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin. Error path now also surfaces the raw SSH output for the next time this breaks somewhere unexpected. * ci(e2e-ddil): export spinifex AWS creds for witness SDK The witness EC2 client (harness/witness.go NewWitness) calls session.NewSession with no static credentials, so the AWS SDK falls through its default chain — env vars → shared profile → IAM role — and on the CI runner none of those exist. Result on the live cluster: "NoCredentialProviders: no valid providers in chain. Deprecated." launches every Scenario A/B/C attempt straight to t.Fatal at launchWitnesses. spx admin init writes [spinifex] aws_access_key_id / aws_secret_access_key to the operator's ~/.aws/credentials on node1 during bootstrap. Pull them over SSH and export as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step resolves before it gets to the empty profile/IAM checks. Hard-fail the step if either field is missing — better than silently watching every scenario t.Fatal on the same line. * fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually green on the tofu cluster. mulga-siv-56. 1. Witness auto-discovers the smallest registered instance type (DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded t2.micro default returned InvalidInstanceType on every Scenario A/B/C launch because the cluster doesn't register that type. DDIL_WITNESS_- INSTANCE_TYPE stays as an operator override. 2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/ SKIP lines instead of a JSON wall. results.json artifact preserved for future flake-rate dashboarding. 3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope <Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>. aws-sdk-go v1's ec2query handler rejects the latter with SerializationError and drops the code, leaving Go callers without an awserr.Code() to inspect. New SDK round-trip test in gateway_test.go guards the regression. * ci(e2e-ddil): install gotestsum from prebuilt tarball `go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose tokeninternal static-assert pegs a narrow Go window (~1.22). Both older and newer runner Go versions trip `invalid array length -delta * delta`. A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely without changing the gotestsum version pin. * ci(e2e-ddil): collect cluster logs on failure Witness `findHost` is currently failing with "could not locate QEMU host for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is visible on any node. Cluster gets destroyed during teardown, so logs have to be pulled before that runs. Adds a failure-only step that ssh's each node and bundles journalctl spinifex-*, process listings, and state dirs into the ddil-results artifact for offline triage. * fix(ddil): support tap-mode QEMU networking in witness harness findHost no longer extracts hostfwd port — production launches go through auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never hostfwd. Witness was timing out across all 3 nodes. Discover guest IP via DescribeInstances.PrivateIpAddress (populated by PrepareRunInstances on ENI auto-create) and tunnel SSH through the host node to guestIP:22. Host routes to the tap bridge subnet, so the relay works without runner-side bridge routing. WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed. Closes mulga-siv-57. * fix(ddil): SSH witness via public IP, drop unreachable tunnel path tofu-cluster bootstraps with --external-mode=pool which forces dev_networking=false → no QEMU hostfwd dev NIC and no host route to the ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to host kernel). Previous tunnel-via-host fix could find the host but couldn't dial the guest. Switch to the path shell e2e already uses: default subnet has MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external pool, and the runner reaches that IP via the same WAN subnet as the cluster nodes. No host relay required, survives Scenario C (partition only DROPs peer cluster IPs, not the runner's source IP). Workflow now scp's ~/.ssh/spinifex-key from the primary node (the counterpart to the public material bootstrap-install.sh imports) and exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against cloud-init's authorized_keys. WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed. Continues mulga-siv-57. * fix(ddil): tunnel guest SSH through host node EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet, which is OVN-routed — reachable from any cluster node, not from the off-cluster runner sitting on the DDIL_NODES WAN subnet. Restore the host-jump pattern: SSH the hosting node (already known from findHost), dial publicIP:22 over that connection, run a guest SSH handshake over the tunnel. Symmetric with how shell e2e reaches guests from inside a node. Continues mulga-siv-57. * fix(ddil): wait for witness sshd before baseline read Previous run reached the guest EIP via the host tunnel but hit 'Connection refused' — sshd hadn't bound yet. waitForRunning only checks EC2 state, which flips on QEMU start, not cloud-init completion. The OVN dnat_and_snat rule for the auto-allocated EIP is also written asynchronously, so the tunnel can land before either layer is ready. awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for up to 90s, retrying on transient transport errors (refused/reset/EOF/ timeout/no-route) and failing fast on auth errors so a bad key surfaces immediately. Called once after resolvePublicIP, before the baseline counter read. Continues mulga-siv-57. * fix(ddil): case-insensitive transient SSH error match awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh emits "ssh: rejected: connect failed (Connection refused)" with capital C — strings.Contains returned false and the helper bailed on first attempt instead of polling. Last run failed in 49s, not the 90s budget. Lowercase the error before matching, and add "connect failed" as the canonical signal for "jump host dial through tunnel failed". Continues mulga-siv-57. * fix(ddil): default guest SSH user to ec2-user Shell E2E SSHes the same ami-ubuntu-* images as ec2-user (tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu and hit 'unable to authenticate, attempted methods [none publickey]' — authorized_keys is provisioned under the AMI default account, which is ec2-user, not ubuntu. Continues mulga-siv-57. * fix(ddil): wait for witness-counter.service to seed /var/lib/counter awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables witness-counter.service runs in the final cloud-init stage — after sshd binds. Baseline read landed in the window where sshd accepts but ExecStartPre hasn't created /var/lib/counter: cat: /var/lib/counter: No such file or directory Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the full path (tunnel + SSH handshake + cat) and retries on the transport errors plus "no such file or directory". Single 2-minute budget covers cloud-init final stage on tofu-cluster hardware. Continues mulga-siv-57. * fix(systemd): drop Wants=spinifex-nats from daemon unit Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately killed spinifex-nats was getting pulled back up the moment the daemon restarted — masking the DDIL Tier 1 standalone-startup path and breaking the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact: spinifex.target's Wants= activates every spinifex unit on boot. After= remains for ordering and is non-blocking when NATS is absent. scripts/test-ddil-phase1.sh hit the same trap and worked around it with a runtime drop-in plus sentinel file; the fix here makes that workaround unnecessary for any caller that restarts the daemon while NATS is stopped. * feat(daemon): peer-health probe drives standalone mode on partition Mode() is now derived from two atomic signals — natsConnected (NATS client TCP link, flipped by the existing disconnect/reconnect callbacks) and peersReachable (any peer daemon's /health responded in the most recent probe cycle). Reports cluster iff both are up; standalone otherwise. The peer-health probe is the missing signal for DDIL Scenario C: an iptables DROP between peer IPs blocks NATS cluster-routing traffic but leaves each node's local NATS client TCP connection healthy, so nats.Conn.IsConnected() keeps returning true and onNATSDisconnect never fires. The probe dials each peer's /health over the cluster network with a 2s timeout on a 2s ticker; worst-case detection of total partition is ~6s on a 3-node cluster, well inside the 30s Scenario C budget. Single-node clusters keep peersReachable permanently true (nothing to probe) so the derived mode flips to cluster as soon as the local NATS connect succeeds. The probe runs from startLocal so the partition signal exists even before (or in the absence of) a successful NATS connect. It does not touch NATS or JetStream, so the §1e Tier 1 invariant asserted by assertNoClusterServicesInitialised is unaffected. Tests: - TestMode_RequiresBothSignals — every nats×peers combination. - TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test. - TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses. - TestMonitorPeerReachability_EndToEnd — ticker-driven false→true. Closes mulga-siv-58. * fix(daemon): peer probe falls back to local daemon port `spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml; remote-node blocks carry just `host = "<ip>"` with no daemon sub-table. So `peer.Daemon.Host` was always empty in a deployed cluster and the probe failed closed, leaving peersReachable permanently false. Tofu-cluster run 25665158964 surfaced this as a Scenario A regression: after StartNATS, natsConnected flipped true on reconnect but Mode() stayed standalone because the derived signal required peersReachable. `a_nats_kill_test.go:75: node3 did not return to cluster mode … (still reporting another mode)`. Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port when present and otherwise reuses the local daemon's port (clusters are symmetric — every node binds the same port the template renders). New TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port- available case; TestProbePeerHealth_RejectsBadAddr is updated to clear self config so it still asserts the fail-closed path. * fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit DDIL Scenario B asserts that `systemctl restart spinifex-daemon` preserves every guest VM so the new daemon can reattach via the persisted local state file. The previous setupShutdown SIGTERM handler called vmMgr.StopAll(), gracefully QMP-powering-down every guest before the daemon exited — ~90s wall-clock and zero surviving QEMU on the other side. CI run 25666369604 surfaced this as `recovered instance i-... has no live QEMU PID`. Two changes restore the Tier 1 contract: - setupShutdown no longer calls StopAll. It still flips d.shuttingDown so crash/restart handlers bail out, persists state, unsubscribes from NATS, and exits. VMs are only stopped when coordinated cluster shutdown drains them — handleShutdownDrain still calls StopAll and flips the flag before the SIGTERM path runs. Operator-visible behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs running. To stop VMs, use the cluster-shutdown topics or terminate via the AWS API. - The systemd unit gains `KillMode=process` so systemd only signals the daemon's main PID. With the default control-group mode, QEMU children were killed when systemd tore down the daemon's cgroup — defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the new fast-exit grace window. Also nil-guards natsConn.Close in the shutdown path: when a daemon restarts while NATS is unreachable (Scenario B), natsConn is nil and the close would panic. Mulga-siv-58 (reopened). * fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart) QEMU stdout/stderr were piped to daemon-side scanner goroutines. When the daemon exits (e.g. `systemctl restart spinifex-daemon`), the pipe read ends close; QEMU's next write triggers SIGPIPE and the guest dies — defeating DDIL Scenario B's "daemon restart preserves running instances" guarantee. Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr instead. The FD is dup'd into QEMU at fork, so closing it on the daemon side after Start() leaves QEMU writing into the file indefinitely. POSIX O_APPEND makes concurrent QEMU writes interleave atomically per line, so a single shared log is safe and avoids per-instance clutter in RuntimeDir. Trade-off: QEMU lines no longer land in journalctl -u spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly (path logged at instance start under the qemu_log field). mulga-siv-58 (DDIL Scenario B iteration 4) * fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's `PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()` falling through to `os.TempDir()` (=/tmp): - spinifex-daemon HOME=/var/lib/spinifex/spinifex, so `$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not exist and pidPath() returns /tmp. - Every daemon start gets a fresh private /tmp namespace. QEMU stays in the OLD namespace after the daemon dies (mount namespaces aren't changed by reparenting). The new daemon's /tmp is empty. - `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace the file doesn't exist, so PID=0. Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d, already in ReadWritePaths) and survives daemon restart. pidPath() short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and RuntimeDir() all resolve to the stable path. Both daemon incarnations and QEMU children see the same files. Revert iteration 4's pipe→file QEMU log change (commit |
||
|
Julian Sommer
|
dc3b48f17e | feat: security groups enforcement (#201) | ||
|
Julian Sommer
|
bb08335831 | fix: explicit workflow permission | ||
|
Julian Sommer
|
f9dddd3c9c | fix: min package age 3 days | ||
|
Julian Sommer
|
51f7afab4c | fix(ui): github workflow | ||
|
joshsiv-mulga
|
9ca8b84bea | ci(baremetal): skip ePDU power-cycle until credentials are wired in | ||
|
joshsiv-mulga
|
e9aec9e784 | ci(baremetal): restore bm-ci SSH key from secret at run start | ||
|
joshsiv-mulga
|
8c6d0747e0 | fix(ci): pass SPINIFEX_SRC/VIPERBLOCK_SRC to iso-builder, build viperblock .so | ||
|
joshsiv-mulga
|
374f2ed7a3 |
feat(ci): add e2e-baremetal workflow — workflow_dispatch only
Separate from normal e2e. Checks out spinifex, checks out mulga scripts (sparse: iso-builder + baremetal), builds spx + test tar via bm-build-artifacts.sh, builds PXE artifacts (squashfs/initrd/vmlinuz) via iso-builder with persistent cache on banksia, then runs run-baremetal-e2e.sh against the single bare-metal node. |
||
|
tomnewton-mulga
|
28ca689130 |
feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Julian Sommer
|
56cc83f723 | chore: merge main into dev | ||
|
Julian Sommer
|
a216d6f596 |
chore(ci): point nightly e2e at dev branch
Scheduled runs were using github.ref_name, which equals the default branch (main) — so the nightly was testing main. Hardcode GIT_BRANCH=dev so the build-install-artifacts.sh + tofu provisioning target dev's HEAD instead. GitHub always loads scheduled-workflow files from the default branch, so this change must stay in sync on main for the schedule to use it. |
||
|
Julian Sommer
|
cbe335e903 | chore: bump single node e2e timeout to 35 mins | ||
|
Julian Sommer
|
84278b794f |
chore: reintroduce dev branch in CI + scripts
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml) - workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch so checkout@v6 uses the default branch implicitly - dependabot: route gomod/github-actions/npm PRs to dev - diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev - iac/proxmox-template: default git_branch to dev for cluster builds Refs docs/development/improvements/reintroduce-dev-branch.md |
||
|
Josh Siviour
|
04f63ad8c6 |
feat(iso-installer): DHCP race fix + vpcd reconcile gate + ELB mgmt route + HashMAC + diagnostics (#172)
* preflight tweak
* fix: pin gateway LRP to link-local, end intra-cluster ARP collision
Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on
attach. Multi-VPC clusters all GARP'd the same .160 with three different
deterministic MACs on the shared br-ext localnet, flapping the upstream
ARP cache and dropping 100% of inbound traffic to dnat_and_snat external
IPs (mulga-siv-26).
handleIGWAttach and reconcileIGW now unconditionally set the gateway
router port to 169.254.0.1/30. The default route's OutputPort already
pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay
allocated only for per-VM dnat_and_snat external addresses, which are
unique by construction.
Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites
stale Networks in place via the new UpdateLogicalRouterPort path. Needed
because Reconcile gates IGW work on "ext switch missing", so a cluster
shipped with the buggy CIDR never re-enters reconcileIGW.
Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision
and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests
updated to assert link-local and OutputPort=&gwPortName. Log field
gateway_ip renamed to lrp_network to signal the semantic shift.
* fix: rewrite utils.HashMAC to SHA-256 / 46-bit output
The old 24-bit rolling hash with literal class prefix had birthday-paradox
collision probability of 1% at ~580 ids and 50% at 4096 — well within the
range of a single deployment's resource population (LRPs, ENIs, dev/mgmt
NICs). On a shared L2 segment, two clusters could trivially produce
duplicate MACs, breaking neighbour discovery and DHCP.
Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes
and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the
first octet. Output entropy goes from 24 bits to 46 bits — 1% collision
boundary moves from ~580 to ~1.2M ids.
Drop the redundant prefix arg: it was no longer the literal output prefix
under the new scheme, just folded into the hash input. Class disambiguation
moves to the caller, where it's explicit. Three of four callers had nothing
to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and
mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags.
Also consolidate vpcd.generateMAC, which was a local duplicate of the
broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation.
Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md
Bead: mulga-siv-27
* fix: pin HashMAC first octet to 0x02
Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast,
bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived.
Technically valid LAA unicast, but produced MACs like 4a:.. and fe:..
that visually encroach on vendor OUI space and tripped CI/ops checks
expecting all generated MACs to start with 02:.
Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output
entropy drops from 46 to 40 bits — birthday-paradox 1% collision at
~150k ids, still 256x the old 24-bit bound (~580).
* chore(e2e): dump cross-chassis OVN state on guest SSH failure
Adds chassis registrations, port_binding claims, gateway_chassis rows,
ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to
dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis
dataplane stalls (cr-gw chassis vs remote chassis with the VM bound).
* fix: gate vpcd startup reconcile behind NATS KV leader election
N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router
with no atomicity, producing duplicate rows that ovn-nbctl rejects with
"Multiple logical routers named '...'. Use a UUID."
Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key
in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile,
ReconcileFromKV, and the two retrofit passes — others Subscribe and rely
on the vpcd-workers queue group for runtime events. MaxAge=60s on the
bucket caps recovery time when a leader crashes mid-reconcile.
Falls through (elected=true) when KV is unavailable so first-boot does
not deadlock on a missing JetStream.
mulga-siv-29
* fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30)
Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s
because the lb-agent inside the ALB VM has no return path to AWSGW.
The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP
for single-node and deliberately skips adding the /32 mgmt host route — the
WAN-return-path concern in the comment is real but applies only to
internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that
gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when
the agent dials advertiseIP the reply targets the VM's private subnet IP from
the host's WAN with no route back.
Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the
existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and
falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only
internal LBs need the /32 route, so internet-facing single-node still gets
no bootcmd and the host-initiated WAN path stays clean.
Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the
fallback has the inputs it needs.
* fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31)
The test was calling real SetupComputeNode("chassis-test", ...) which shelled
out via sudoCommand to set external_ids:system-id=chassis-test on the host's
live OVS. On a dev box with OVS installed this overwrote the persistent UUID
(owned by openvswitch-switch and re-applied only on reboot), so vpcd's
discoverChassis filtered out the legitimate SBDB chassis row as 'stale local'
and crashlooped — taking out all VPC ops including ALB ENI provisioning.
Bind sudoCommand to a var so tests can swap it. Stub it in the test to return
exec.Command("/bin/false") and assert SetupComputeNode returns an error. The
test's intent (signature + error-path coverage) is preserved; nothing on the
host is touched.
* chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32)
Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8
DORA timeouts correlate with concurrent Acquire calls on the same
bridge. Tracks per-bridge in-flight count and wallclock duration of each
Acquire; logs WARN on failure with both fields, DEBUG on success.
No behavioural change.
* ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32)
run-multinode-e2e.sh dumps node journals via its own EXIT trap, but
run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts
fail the cluster is destroyed in Teardown without surfacing any daemon
logs, blocking root-cause analysis.
Add a workflow step that runs after the test scripts but before
Teardown when a previous step failed, ssh'ing to each node and pulling
the last 200 lines of every spinifex-* unit. Best-effort: skips
gracefully when inventory or nodes are unreachable.
Diagnostic-only; no behavioural change to the test path.
* fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32)
Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when
dhcpcd-base is installed. It binds br-wan and races vpcd's in-process
nclient4 client for upstream OFFERs, draining the lab DHCP pool with
duplicate leases and causing intermittent DORA timeouts in the e2e
public-IP path (worst on cell-8 real-multi/dhcp/debian-13).
The ISO installer already masks the unit at install time
(cmd/installer/install/install.go:316). The binary install path used by
e2e nightly missed it, so debian hosts installed via setup.sh ran with
both DHCP clients live. Mirror the ISO mask here.
Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable
dhcpcd.service on Noble).
* ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32)
Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the
dhcpcd.service journal alongside the spinifex-* units, plus
systemctl is-enabled / is-active and the DHCP client port (UDP 68)
listener so we can see directly whether dhcpcd is running and bound
to br-wan when an Acquire times out.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
|
||
|
Julian Sommer
|
a3994510e0 | feat: e2e nightly reboot tests (#168) | ||
|
Julian Sommer
|
138083f6c8 | chore: only run e2e nightly mon to friday | ||
|
Julian Sommer
|
f2dcb2f33f | fix: tofu nightly e2e | ||
|
Julian Sommer
|
7128dc785c | chore: dhcp e2e nightly | ||
|
dependabot[bot]
|
f3cfee3189 | chore: bump actions/upload-artifact in the github-actions group (#163) | ||
|
joshsiv-mulga
|
8735275d84 | Merge commit 'eeb7ebc5db77e06db11ed7a58b9a70cdcc4128fb' into feat/iso-installer | ||
|
Julian Sommer
|
eeb7ebc5db | chore: enable nightly nginx-alb workbook (#161) | ||
|
joshsiv-mulga
|
c01207b342 |
merge: fix/dhcp-vpcd-issues (mulga-998.b Fixes 1+1b+2+3) into feat/iso-installer
Fixes DHCP/vpcd external-path issues discovered during reboot drill on josh-banksia: - Fix 1: persist veth-wan pair via networkd .netdev/.network - Fix 1b: admin-up veth-wan-ovs via second .network unit (peer stayed LOWERLAYERDOWN) - Fix 2: loud detectBridgeMode + verifyBridgeMode sanity check - Fix 3: retrofit nat-addresses=router + network_name=external on every vpcd startup via RetrofitAllExternalLocalnetOptions Bead: mulga-siv-13 (closed). Plan: docs/development/bugs/dhcp-vpcd-issues.md. Reboot-drill evidence: smoketest 9/9, Drill 2 retrofit verified post-reboot. # Conflicts: # scripts/reset-dev-env.sh # spinifex/handlers/ec2/vpc/external_ipam.go # spinifex/services/vpcd/vpcd.go |
||
|
Julian Sommer
|
c7d4040dd3 | feat: nightly e2e (#155) | ||
|
Josh Siviour
|
068aef5589 |
feat: DDIL E2E test harness (mulga-siv-5) (#160)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc. |
||
|
tomnewton-mulga
|
265556dc4e | updated ISO doc | ||
|
tomnewton-mulga
|
f0d48c4274 | Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer | ||
|
Ben Duncan
|
93a6704944 | Merge remote-tracking branch 'origin/main' into feat/iso-installer | ||
|
Julian Sommer
|
1ca8621e16 | fix: allow mulga feat branch | ||
|
tomnewton-mulga
|
56a1ead706 | extra pics for ISO docs | ||
|
tomnewton-mulga
|
3c6077ef58 | ISO install doc tweaks | ||
|
tomnewton-mulga
|
7b9fdeb60c | new docs | ||
|
Julian Sommer
|
8d6f4df526 | fix: setting up cluster image | ||
|
Julian Sommer
|
ff9f9485d2 | feat: nats TLS (#144) | ||
|
Julian Sommer
|
68ae17dcf1 | chore: rename app-id to client-id | ||
|
Julian Sommer
|
5b39efb078 | fix: make e2e use env vars | ||
|
Julian Sommer
|
7cf0783122 | fix: set dev tags as prerelease | ||
|
Julian Sommer
|
b32111102d | fix: README.md | ||
|
Ben Duncan
|
f99eb42ca7 | Doc tweaks and README for release | ||
|
dependabot[bot]
|
36da673ecd | chore: bump the github-actions group with 2 updates (#133) | ||
|
Josh Siviour
|
ab95566795 |
feat: initial ELBv2 implementation (#122)
* feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commit |
||
|
Julian Sommer
|
a56a2baf1f | feat: privilege separation (#114) | ||
|
Julian Sommer
|
dda4575888 | fix: use binary install for single node e2e (#111) | ||
|
Julian Sommer
|
f5ec8c3bae | feat: formation server HTTPS (#108) | ||
|
Julian Sommer
|
8795a7b926 | fix: remove dev branch |