mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-23 04:36:40 +00:00
6d38fcad5c2c8b207737a921eed5d2ebb2c3d22b
117 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 joshsiv-mulga
|
b348003922 |
test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition
After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute
instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on
the new image. Make them detect at runtime so v3 and v6 envs both pass.
- tests/e2e/run-e2e.sh: DescribeImages filter accepts both
ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match.
- tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img,
fall back to ubuntu-24.04.img; pass detected version to spx import.
- tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first,
then 24.04), stop on the first one DescribeImages can resolve.
- .github/workflows/e2e-baremetal.yml: download resolute (26.04) into
BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path.
- docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}:
AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values=
array is ORed).
Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline)
land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/*
under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the
v6 gold image and 24.04 entries are removed from spinifex/utils/images.go.
|
||
|
Josh Siviour
|
0a07aff199 |
ci(e2e-go): cluster JUnit failures + name likely root cause (#230)
* ci(e2e-go): cluster JUnit failures + name likely root cause
New composite action `.github/actions/e2e-analyze` runs after
`go-junit-report` converts each suite log. For every `<failure>` it
strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps,
file:line) to a stable signature, picks the earliest non-cascade
failure per suite as the likely root cause, and buckets the rest by
signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to
`<log-dir>/analysis.md` so it travels with the artifact bundle.
Cascade markers (`must populate fix.`, `Should NOT be empty`,
`Expected value not to be nil`) collapse the typical "one upstream
phase fails, ten downstream `require.NotEmpty` assertions trip" pattern
into a single grouped entry under the real first failure.
The analyser is advisory — it always exits 0. The existing
"Fail job if any suite failed" gate keeps owning exit-code semantics.
Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md`
(mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3
(rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder
seeded here) are independent follow-ups.
CI wiring:
- e2e-go.yml: new "Analyze e2e failures" step between "Publish test
summary" and the artifact upload, always-runs.
- spinifex.yml: new `make test-actions` step so the golden-file tests
under `.github/actions/e2e-analyze/testdata/` run on every PR.
- Makefile: `test-actions` target; kept out of `test-cover` so CI-only
tooling doesn't dilute coverage %.
Golden fixtures: a synthetic cascade-vpcs case (one Phase 5
InvalidParameterValue root + 15 downstream cascades), one real
single-failure case captured from run 26037892507, and a clean
no-failures case.
* ci(e2e-go): per-failure log bundles with time-window slicing
Stage 2 of the e2e failure analyzer. For each failure surfaced by
Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing:
failure.txt — summary (test, start, duration, file-site, error)
journal.log — spinifex-journal.log sliced to
[StartAt - 5s, StartAt + Duration + 5s]
test.log — test-<suite>.log block between `=== RUN <name>`
and the matching `--- FAIL` marker
<copies> — any files the harness already dropped at
<log-dir>/<TestName_with_/_replaced>/
Report links each entry to its bundle so a reader on the Actions
page goes straight to the relevant slice.
Workflow now invokes journalctl with --output=short-iso so the
analyzer can parse wall-clock timestamps for the journal window.
Lines without a parseable timestamp inherit the previous line's
classification (keeps multi-line records and stack traces
contiguous).
Render output is unchanged when BundlePath is empty, so the Stage 1
golden snapshots still match.
* ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump
setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local",
which makes Go refuse to auto-download the toolchain version named
in go.mod. The self-hosted ci-single runner ships /usr/local/go
older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh
prepends that path so the system Go wins the resolution race. Under
v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6
("local"), the Build Install Artifacts step errors out:
go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local)
Re-set the env to "auto" so behaviour matches pre-v6 runs.
* ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go
Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local
to $GITHUB_ENV, which overrides workflow env in every later step.
Add a dedicated step right after setup-go that appends
GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step
(which uses /usr/local/go from a stale runner path) can self-heal by
fetching the toolchain named in go.mod, matching pre-v6 behaviour.
* fix(vpcd): add flows-ready barrier after dnat_and_snat add
handleAddNAT wrote the NAT rule to OVN NB and returned immediately,
mirroring the gap siv-105 closed for handleIGWAttach but on the NAT
add path. ec2d/RunInstances then returned success while ovn-northd
was still compiling NB->SB and ovn-controller hadn't installed the
DNAT/SNAT flows on the gateway chassis, so the freshly launched VM
was unreachable on its public IP for whatever flow-install latency
the chassis happened to incur.
Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at
192.168.0.212 timed out at 3min while diagnostics showed
Port_Binding up=true for both the VM eni and the chassis-redirect
gateway. SB-side convergence was complete; the missing piece was the
same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach,
applied at the end of handleAddNAT as well.
Reuses the existing package-var `waitForFlowsHV` so tests can stub
it; no new infra. Parallel unit test asserts the call site fires
exactly once per add-nat.
* ci(e2e-analyze): real-artifact fixes for journal slicing
Three bugs surfaced after Stage 2 ran against real CI artifacts in
run 26072432957:
1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`)
that systemd's `journalctl --output=short-iso` actually emits on
most versions. Added a parser that tries RFC3339 first and falls
back to the no-colon `-0700` layout. Result: 0-line journal slices
become populated.
2. go-junit-report's `timestamp` attr on <testsuite> records the
moment the XML was produced (post-suite), not when the tests
started. Computing StartAt as `that + cumulative-priors` placed
every failure AFTER the journal collection ended, so windows
always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start`
right before running each suite; analyzer prefers that file over
the junit timestamp.
3. Persisted `OffsetFromSuiteStart` on each Failure so the
suite-start override can recompute StartAt without re-parsing the
XML.
Unit tests added for the colon-offset journal format. End-to-end
test (TestWriteBundles_EndToEnd) still passes with the new layout.
* ci(e2e-analyze): skip parent testcases when summing suite offsets
go-junit-report emits one <testcase> for the parent Go test AND each
subtest, with the parent's time = sum of its children. ParseFile was
summing every testcase into the suite-cumulative-offset counter, so
every leaf's offset was inflated by ~the parent's total duration.
Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes
after the suite finished collecting logs — beyond the journal window,
so the per-failure journal slice was empty.
Pre-compute the leaf set per suite (a name is a leaf iff no other
testcase name starts with `<name>/`) and only accumulate cumul on
leaves. Golden snapshots updated; the offset for the only failure in
each fixture now matches the actual leaf-sum start time.
* test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail
Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout
reproduced on the e2e-failure-analysis branch with siv-105's IGW
barrier AND the new add-nat barrier BOTH firing cleanly, so the
residual gap is downstream of `--wait=hv sync` — most likely stale
OVS conntrack, stale upstream-host ARP, or an OF flow-install miss.
DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the
external IP, logical IP, and per-test artifact dir. When set, the
helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and
`ip -4 neigh show` for the external IP / logical IP and snapshots
`ovn-nbctl find NAT external_ip=…`. Each capture is both printed
inline (existing affordance) and written to a separate file under
fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside
the journal + test-log slices, no analyzer changes needed.
Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the
captures fire on the next failure; updates the t.Fatalf message to
point at siv-111 instead of siv-105 (which is the now-closed parent).
* fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b
Phase 8b stands up a non-default VPC and runs an instance there. Its
auto-created default SG denies all ingress, so the SSH probe times out
even when every datapath barrier converges — OF flow captures show the
SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between
tables 13 and 17. The bash driver relies on admin init having pre-baked
this rule on the *default* VPC's default SG; a fresh VPC has not.
Add a DescribeSecurityGroups + AuthorizeSSHIngress step between
route-table setup and run-instances, using the existing harness helper.
Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL
drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the
bead plan doc.
|
||
|
joshsiv-mulga
|
09636987ab |
ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7
GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the rest of the repo; bump to match versions already in use: - actions/setup-go@v5 -> @v6 (matches spinifex.yml) - actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml) checkout@v6 and create-github-app-token@v3 are already current across all workflows. The local junit-report action is `using: composite` (shell-only), so the Node runtime change does not apply. |
||
|
joshsiv-mulga
|
073acc9243 |
ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners
Reap Orphan Procs (job start) and the in-Teardown pkill block were using `pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name .terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and deleting their state-lock files — exactly the kind of cross-runner collateral the reap was meant to recover from. Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's workspace dir is unique on the host, so only this run's own tofu procs match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE. |
||
|
joshsiv-mulga
|
21b1d5f02b |
ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy
Two related leaks made cancelled runs strand QEMU work on the libvirt host: 1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran without -tt, so when the runner cancelled the step the local ssh died but sshd never sent SIGHUP through (no PTY). The remote `.test` binary orphaned and kept running until its own 30m `-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file handles. 2. Teardown went straight to `tofu destroy` without first killing any stragglers on the VM, so destroy raced the .test's open file handles on the root disk — sometimes successfully, sometimes leaving the libvirt host with a half-released qcow2. Fix: - Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children when the local ssh dies. - Add a trap (EXIT INT TERM) in the run-suites step that ssh's back to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces. - Mirror the same pkill at the top of Teardown so destroy runs against a quiesced VM. Both cleanups are scoped to the per-run wan_ip — safe to land while multiple ci-single runner agents run concurrent jobs against different tofu envs on the same host (we deliberately keep that parallelism). Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected; no journal access yet to confirm). |
||
|
joshsiv-mulga
|
ab68effaa9 |
ci(e2e-go): unbuffer sed in test pipeline for live log streaming
GNU sed defaults to block-buffered stdout when stdout is a pipe (GitHub Actions runner). Accumulates 4-64KB before flushing, so test output appeared in sporadic batches unaligned to test boundaries. -u flag forces line-buffered output so each go test -v line surfaces in the Actions UI as it is produced. Side effect: none. No process count change, no SSH session change, no test execution change. Pure output flushing on the runner host. |
||
|
joshsiv-mulga
|
651377fe9c |
ci(e2e-go): drop push trigger, manual dispatch only
Pull the on.push branch trigger so the workflow only fires from workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push trigger lifts again once the single-node Go port (mulga-siv-89) is green end-to-end and run-e2e.sh is retired. |
||
|
joshsiv-mulga
|
5c269a3970 |
ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge
A cancelled run on the single-runner ci-single host has been leaving
orphan tofu/terraform procs (and stale tfstate locks) behind, which
caused the next push's tofu destroy to hang for 45m+ and saturate the
runner host until sshd refused new connections.
- Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new
push to feat/e2e-go-harness cancels the in-flight run instead of
stacking on the same self-hosted runner.
- New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any
lingering `tofu`/`terraform` from the runner user, and unlink any
stale `.terraform.tfstate.lock.info`.
- Harden `Teardown`: drop `set -e` (scrub-env must always run as the
libvirt-level fallback), reap orphans before destroy, and wrap
`tofu destroy` in `timeout 600` so a wedged provider can't pin the
job to its 45m timeout.
|
||
|
joshsiv-mulga
|
2965ff97e1 |
ci(e2e-go): strip indent + file:line on display, propagate color env over SSH
Two display bugs spotted on the previous run:
1. Phase banners came out white instead of bold-cyan because the test
binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS
nor a TTY is present, so harness.log.go init() disables ANSI escapes.
Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env
so the helper keeps emitting color codes.
2. Every t.Logf line still rendered as
lb_test.go:61: instance type: t3.nano
(4-space indent + `<file>:<line>:` prefix) because t.Helper()
bubbles file:line to the call site. Fix: pipe the test binary's
stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'`
so the display strips both. Tee to the artifact log happens BEFORE
the sed so raw output is preserved for triage; only the live UI
sees the cleaned form.
|
||
|
joshsiv-mulga
|
be966b20c4 |
ci: colored phase banners + quiet-on-success tofu output
Pretty up the live Actions log without flooding it. Three changes: 1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad, warn, note, quiet_run) using ANSI colors that the Actions UI renders. 2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect into $TOFU_LOG (the previous tee'd version regressed live noise vs the original silent redirect). Each phase now prints one colored ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single collapsed group, so the live log stays scannable but the failing tail is one click away. 3. junit-report render.py — compact summary: short status icons (✅/❌) in a single column, no separate err column, single-line failure list (truncated to 120 chars) instead of <details> blocks that wrap badly on narrow screens. Per-suite test runs still wrap in ::group:: with an inline cyan banner so pass output collapses and the suite name is obvious when expanded. |
||
|
joshsiv-mulga
|
1a494fed58 |
ci: render JUnit into job summary + collapsible groups in noisy steps
Add a reusable composite action (.github/actions/junit-report) that parses a glob of JUnit XML files, writes a totals table + collapsible per-failure details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per failed/errored test case so failures surface in the run header instead of being buried in step logs. Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum now also emits --junitfile so the action has input). Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env, tar-over-ssh, per-suite test runs, cluster-log collection) in ::group::/::endgroup:: markers so the Actions UI collapses the chatter by default and the human can expand only the failing phase. Tofu output still tees to $TOFU_LOG so the artifact bundle is unchanged. Future workflows (multinode, baremetal) can adopt the same pattern via the README in .github/actions/junit-report. |
||
|
joshsiv-mulga
|
7e97ef20a2 | ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time | ||
|
joshsiv-mulga
|
3b272295a0 |
ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM
Replaces the source-tree + go + gotestsum payload that the e2e-go.yml workflow previously shipped to the tofu CI VM with self-contained test binaries built on the runner via `go test -c`. testdata for the LB suite is already //go:embed'd into the binaries, so the bundle on the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed). Workflow changes - actions/setup-go on runner. - New step builds binaries via `make e2e-build SUITES_BUILD=...`. - Install go-junit-report on runner; convert each VM-side test log to junit-<suite>.xml after artifact pull so "Publish test summary" still parses JUnit XML. - Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network). - Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth). - VM step runs `./<suite>.test -test.v -test.timeout=30m` directly. Makefile changes - New `e2e-build` target compiles each `SUITES_BUILD` entry into `BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also removes BIN_DIR. Closes mulga-siv-80. |
||
|
joshsiv-mulga
|
1a8729b2d6 |
ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync
bootstrap-install.sh ships a minimal binary VM with no rsync; the ship-source and collect-artifacts steps failed with 'rsync: command not found'. Stream a tarball over ssh instead — same exclude list, no extra dependency on the VM. |
||
|
joshsiv-mulga
|
32486be5a6 |
ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there
Single-node tofu VM only has the spx binary + bash test bundle at ~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert' from $HOME/Development/mulga/spinifex which doesn't exist, producing: make: *** No rule to make target 'e2e-cert'. Stop. Add a checkout of the spinifex repo on the runner and rsync it (minus .git, build/, bin/, node_modules) to the VM as ~/Development/mulga/spinifex before installing Go + gotestsum. Interim solution; mulga-siv-80 tracks replacing the rsync with shipping pre-built test binaries from the runner. |
||
|
joshsiv-mulga
|
7b9e0c859f |
ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility
gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile under Go 1.26 with: invalid array length -delta * delta (constant -256 of type int64) Switching to @latest pulls a newer x/tools that builds clean. |
||
|
joshsiv-mulga
|
fe329384e7 |
ci(e2e): install Go on tofu VM before running Go E2E suites
The tofu-provisioned VM ships with the spx binary and bash test scripts but no go toolchain, so the prior workflow's "Install gotestsum on VM" step failed with `bash: line 2: go: command not found`. Add a curl-based Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and update the test step's PATH so make e2e-* runs the freshly-installed go binary. |
||
|
joshsiv-mulga
|
5591915163 |
ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in
Standalone workflow runs the new Go-based E2E harness on the existing ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex, installs gotestsum, runs each requested suite via the e2e Makefile, and uploads per-test artifacts + JUnit XML + JSON streams as a build artifact. Test summary parses the JUnit and emits ::error file=...:: annotations so failures surface in the PR diff. Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't interfere with the existing e2e.yml while the Go conversion lands. tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR under a suite-tagged filename derived from the first PKG component. |
||
|
Josh Siviour
|
4ae2a4ac53 |
feat(elbv2): QEMU microvm direct-boot for LB system instances (#228)
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline
vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go
config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput
image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts
* feat(daemon): direct-boot launch path for microvm system instances
When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration
PC-machine path (DirectBoot=false) unchanged.
Closes mulga-siv-65
* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold
* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed
* fix(microvm): end-to-end fixes for direct-boot LB path
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
* fix(build): auto-detect docker/podman when apk unavailable for microvm image build
* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
* fix(microvm): fix mgmt NIC routing for all LB schemes
Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
* feat(microvm): Phase 2 kernel stripping + boot optimisation
- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches
* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip
The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.
Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.
* fix(build): keep failover modules in initramfs strip
virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.
Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.
* fix(daemon): restore slog default after viperblock.New mutates it
viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.
Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.
Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.
Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).
* perf(microvm): hit Phase 2 boot/memory gates
t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.
Root causes:
1. build-microvm-image.sh staged the entire chroot into the cpio
without dropping /boot, embedding a duplicate vmlinuz, Alpine's
stock initramfs-virt, System.map, and a "boot → ." symlink loop
(~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
the kernel is copied out, before the cpio. Initramfs drops from
31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
42 MiB → 22 MiB.
2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
no firmware, ROM, or block layer). Both extracted to
directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
qemuStartSettleWait to 50 ms.
3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
Added boot-time perf flags: quiet loglevel=3 (silence printk over
115200-baud isa-serial), mitigations=off (KVM guest on trusted
host, single-tenant LB workload), tsc=reliable, no_timer_check,
reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).
4. init.sh dropped modprobe virtio_mmio (built-in:
CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).
Phase 2 gates:
- t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms
- artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script)
- ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure,
unrelated to microvm path)
* build(microvm): wire build into deploy + distro + release pipeline
Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).
Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.
Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.
* build(ansible): include microvm artifacts in build role tarball
Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).
* feat(config): enable microvm direct-boot LB path by default
Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.
Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).
* fix(microvm): bootstrap apk.static when no host rootfs tool exists
Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.
When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.
No-op when apk, docker, or podman is already present.
Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244
* revert: drop apk.static bootstrap in build-microvm-image
Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.
* docs(workbook): drop lb-alpine import for microvm direct-boot LB
LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.
* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN
When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").
Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.
* fix(e2e): redirect lb probe chatter to stderr so stalls are visible
test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.
Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.
* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node
buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.
resolveMgmtRoute already encodes the correct policy:
- multi-node any scheme → explicit MgmtRoute{Gateway,Target}
- single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP)
- single-node internet-facing → empty (reach AWSGW via VPC SNAT)
Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.
Test covers all three branches.
* feat(ansible): dev-deploy playbook for in-place artifact swap
Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.
Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.
Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.
* chore(microvm): remove legacy LB Alpine image build pipeline
Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.
build-system-image stays — other manifests (e.g. GPU images) still
use it.
* refactor(microvm): remove legacy PC-machine LB boot path
Phase 3 — direct-boot becomes the only ELBv2 LB launch path.
- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
[daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
paths) and refresh stale comments that still mentioned lbVMUserData.
ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.
* docs: drop lb-alpine catalog entry from cluster setup guide
LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.
* feat(ui): remove LB image import banner + LB-AMI gating
Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:
- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
system-managed.ts (isSystemManagedImage stays — still used by the
EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
wizard pages
- The corresponding tests that asserted the banner appears
* revert: restore lb.conf + lb-setup.sh for mulga install pipeline
Phase 3 microvm removal (commit
|
||
|
Julian Sommer
|
b7a932d6b0 |
ci(e2e-nightly): target dispatched branch on workflow_dispatch
GIT_BRANCH was hardcoded to dev, so workflow_dispatch runs ignored the ref chosen in the UI. Use github.ref_name for dispatch (and any non-cron trigger), keep dev as the explicit target for scheduled cron runs. |
||
|
joshsiv-mulga
|
7042f25861 |
feat(e2e): add run-bm-multinode-e2e.sh for bare-metal N≥2 node clusters
Bare-metal variant of run-multinode-e2e.sh: - Accepts 2+ node IPs (not hardcoded 3) - SSH key/user via PEER_SSH_KEY / PEER_SSH_USER env vars (default: bm-ci / spinifex) - NODE3_IP optional; node3 survival check skipped when absent Workflow updated: job renamed to Multi-Node E2E, timeout 60→120 min. |
||
|
joshsiv-mulga
|
9fd89ec1ad | fix(ci): cache lb-alpine image on runner alongside ubuntu, pass to bm-bootstrap | ||
|
joshsiv-mulga
|
58c2e18ad1 |
fix(ci): cache ubuntu-24.04 cloud image on runner, pass to bm-bootstrap
Add a step before the E2E run that ensures the Ubuntu 24.04 noble cloud image is cached at BM_ISO_CACHE. Downloads from ubuntu.com if missing, prints cached size if present. Passes path via BM_UBUNTU_IMAGE to bm-bootstrap.sh which SCPs it to the node for --file import. |
||
|
joshsiv-mulga
|
7859d9610b | fix(ci): write bm-ci key to RUNNER_TEMP and derive .pub for bm-bootstrap | ||
|
joshsiv-mulga
|
38373b2fc7 | fix(ci): generate epdu.env from secrets instead of pre-placed file | ||
|
joshsiv-mulga
|
e615d6f9e6 | fix(ci): remove BM_SKIP_EPDU now that ePDU SNMPv3 is provisioned | ||
|
Julian Sommer
|
97701737cb | feat: describe instance status (#221) | ||
|
joshsiv-mulga
|
a31fd8098c |
fix(ci): use sudo to clean workspace so root-owned rootfs cache is removed
The build-rootfs step creates root-owned files in the workspace; a plain rm -rf fails with permission denied on the next run's Clean Workspace step. |
||
|
Julian Sommer
|
aaa208b594 | fix: use dev for build and test | ||
|
Josh Siviour
|
45555dde49 |
test(ddil): flip Scenarios A/B/C to real assertions (#205)
* test(ddil): flip Scenarios A/B/C to real assertions Replaces the t.Skip stubs landed in #204 with the assertions described in docs/development/improvements/ddil-e2e-test-harness.md §3: - A NATSKill — KillNATS on node3, assert ModeStandalone within 30s, /local/instances still serves on the isolated node, majority /health on node1+node2 stays up, witness counters on all three nodes keep advancing, snapshot survives reconnect. - B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert ModeStandalone within 30s of restart, recovered instances present with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post PID equality dropped — see file comment; recovery + a non-zero PID give the same regression coverage without coupling to the daemon unit's KillMode. - C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep serving /health and /local/instances, isolated node3 reports ModeStandalone with /local/* responsive, witness counter on the partitioned side keeps advancing, heal converges and snapshot is preserved. Supporting changes: - harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore, the deployed daemon ClusterManager binds [nodes.<id>.daemon].host. The Phase-1 stub used a placeholder that would have blocked any /local/* assertion against the real cluster. - harness/witness.go — export WitnessVM.Terminate so scenarios can clean up their counter VMs from t.Cleanup. - scenarios/main_test.go — requireLiveCluster + launchWitnesses helpers to keep the three flipped scenarios tight. - TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED; TestCoverageDrift continues to enforce alignment with the package. Validation: built with -tags=e2e, dry-run scenarios pass, coverage drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against tofu cluster is the next step. * fix(ddil): countDropRules — single iptables -S call + filter Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs across both chains, but iptables -S takes a single optional chain — the second -S errors to stderr, the combined SSH buffer poisons the grep input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState. Switch to one `iptables -S` (lists every chain in iptables-save format), filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin. Error path now also surfaces the raw SSH output for the next time this breaks somewhere unexpected. * ci(e2e-ddil): export spinifex AWS creds for witness SDK The witness EC2 client (harness/witness.go NewWitness) calls session.NewSession with no static credentials, so the AWS SDK falls through its default chain — env vars → shared profile → IAM role — and on the CI runner none of those exist. Result on the live cluster: "NoCredentialProviders: no valid providers in chain. Deprecated." launches every Scenario A/B/C attempt straight to t.Fatal at launchWitnesses. spx admin init writes [spinifex] aws_access_key_id / aws_secret_access_key to the operator's ~/.aws/credentials on node1 during bootstrap. Pull them over SSH and export as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step resolves before it gets to the empty profile/IAM checks. Hard-fail the step if either field is missing — better than silently watching every scenario t.Fatal on the same line. * fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually green on the tofu cluster. mulga-siv-56. 1. Witness auto-discovers the smallest registered instance type (DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded t2.micro default returned InvalidInstanceType on every Scenario A/B/C launch because the cluster doesn't register that type. DDIL_WITNESS_- INSTANCE_TYPE stays as an operator override. 2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/ SKIP lines instead of a JSON wall. results.json artifact preserved for future flake-rate dashboarding. 3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope <Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>. aws-sdk-go v1's ec2query handler rejects the latter with SerializationError and drops the code, leaving Go callers without an awserr.Code() to inspect. New SDK round-trip test in gateway_test.go guards the regression. * ci(e2e-ddil): install gotestsum from prebuilt tarball `go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose tokeninternal static-assert pegs a narrow Go window (~1.22). Both older and newer runner Go versions trip `invalid array length -delta * delta`. A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely without changing the gotestsum version pin. * ci(e2e-ddil): collect cluster logs on failure Witness `findHost` is currently failing with "could not locate QEMU host for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is visible on any node. Cluster gets destroyed during teardown, so logs have to be pulled before that runs. Adds a failure-only step that ssh's each node and bundles journalctl spinifex-*, process listings, and state dirs into the ddil-results artifact for offline triage. * fix(ddil): support tap-mode QEMU networking in witness harness findHost no longer extracts hostfwd port — production launches go through auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never hostfwd. Witness was timing out across all 3 nodes. Discover guest IP via DescribeInstances.PrivateIpAddress (populated by PrepareRunInstances on ENI auto-create) and tunnel SSH through the host node to guestIP:22. Host routes to the tap bridge subnet, so the relay works without runner-side bridge routing. WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed. Closes mulga-siv-57. * fix(ddil): SSH witness via public IP, drop unreachable tunnel path tofu-cluster bootstraps with --external-mode=pool which forces dev_networking=false → no QEMU hostfwd dev NIC and no host route to the ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to host kernel). Previous tunnel-via-host fix could find the host but couldn't dial the guest. Switch to the path shell e2e already uses: default subnet has MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external pool, and the runner reaches that IP via the same WAN subnet as the cluster nodes. No host relay required, survives Scenario C (partition only DROPs peer cluster IPs, not the runner's source IP). Workflow now scp's ~/.ssh/spinifex-key from the primary node (the counterpart to the public material bootstrap-install.sh imports) and exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against cloud-init's authorized_keys. WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed. Continues mulga-siv-57. * fix(ddil): tunnel guest SSH through host node EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet, which is OVN-routed — reachable from any cluster node, not from the off-cluster runner sitting on the DDIL_NODES WAN subnet. Restore the host-jump pattern: SSH the hosting node (already known from findHost), dial publicIP:22 over that connection, run a guest SSH handshake over the tunnel. Symmetric with how shell e2e reaches guests from inside a node. Continues mulga-siv-57. * fix(ddil): wait for witness sshd before baseline read Previous run reached the guest EIP via the host tunnel but hit 'Connection refused' — sshd hadn't bound yet. waitForRunning only checks EC2 state, which flips on QEMU start, not cloud-init completion. The OVN dnat_and_snat rule for the auto-allocated EIP is also written asynchronously, so the tunnel can land before either layer is ready. awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for up to 90s, retrying on transient transport errors (refused/reset/EOF/ timeout/no-route) and failing fast on auth errors so a bad key surfaces immediately. Called once after resolvePublicIP, before the baseline counter read. Continues mulga-siv-57. * fix(ddil): case-insensitive transient SSH error match awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh emits "ssh: rejected: connect failed (Connection refused)" with capital C — strings.Contains returned false and the helper bailed on first attempt instead of polling. Last run failed in 49s, not the 90s budget. Lowercase the error before matching, and add "connect failed" as the canonical signal for "jump host dial through tunnel failed". Continues mulga-siv-57. * fix(ddil): default guest SSH user to ec2-user Shell E2E SSHes the same ami-ubuntu-* images as ec2-user (tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu and hit 'unable to authenticate, attempted methods [none publickey]' — authorized_keys is provisioned under the AMI default account, which is ec2-user, not ubuntu. Continues mulga-siv-57. * fix(ddil): wait for witness-counter.service to seed /var/lib/counter awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables witness-counter.service runs in the final cloud-init stage — after sshd binds. Baseline read landed in the window where sshd accepts but ExecStartPre hasn't created /var/lib/counter: cat: /var/lib/counter: No such file or directory Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the full path (tunnel + SSH handshake + cat) and retries on the transport errors plus "no such file or directory". Single 2-minute budget covers cloud-init final stage on tofu-cluster hardware. Continues mulga-siv-57. * fix(systemd): drop Wants=spinifex-nats from daemon unit Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately killed spinifex-nats was getting pulled back up the moment the daemon restarted — masking the DDIL Tier 1 standalone-startup path and breaking the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact: spinifex.target's Wants= activates every spinifex unit on boot. After= remains for ordering and is non-blocking when NATS is absent. scripts/test-ddil-phase1.sh hit the same trap and worked around it with a runtime drop-in plus sentinel file; the fix here makes that workaround unnecessary for any caller that restarts the daemon while NATS is stopped. * feat(daemon): peer-health probe drives standalone mode on partition Mode() is now derived from two atomic signals — natsConnected (NATS client TCP link, flipped by the existing disconnect/reconnect callbacks) and peersReachable (any peer daemon's /health responded in the most recent probe cycle). Reports cluster iff both are up; standalone otherwise. The peer-health probe is the missing signal for DDIL Scenario C: an iptables DROP between peer IPs blocks NATS cluster-routing traffic but leaves each node's local NATS client TCP connection healthy, so nats.Conn.IsConnected() keeps returning true and onNATSDisconnect never fires. The probe dials each peer's /health over the cluster network with a 2s timeout on a 2s ticker; worst-case detection of total partition is ~6s on a 3-node cluster, well inside the 30s Scenario C budget. Single-node clusters keep peersReachable permanently true (nothing to probe) so the derived mode flips to cluster as soon as the local NATS connect succeeds. The probe runs from startLocal so the partition signal exists even before (or in the absence of) a successful NATS connect. It does not touch NATS or JetStream, so the §1e Tier 1 invariant asserted by assertNoClusterServicesInitialised is unaffected. Tests: - TestMode_RequiresBothSignals — every nats×peers combination. - TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test. - TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses. - TestMonitorPeerReachability_EndToEnd — ticker-driven false→true. Closes mulga-siv-58. * fix(daemon): peer probe falls back to local daemon port `spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml; remote-node blocks carry just `host = "<ip>"` with no daemon sub-table. So `peer.Daemon.Host` was always empty in a deployed cluster and the probe failed closed, leaving peersReachable permanently false. Tofu-cluster run 25665158964 surfaced this as a Scenario A regression: after StartNATS, natsConnected flipped true on reconnect but Mode() stayed standalone because the derived signal required peersReachable. `a_nats_kill_test.go:75: node3 did not return to cluster mode … (still reporting another mode)`. Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port when present and otherwise reuses the local daemon's port (clusters are symmetric — every node binds the same port the template renders). New TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port- available case; TestProbePeerHealth_RejectsBadAddr is updated to clear self config so it still asserts the fail-closed path. * fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit DDIL Scenario B asserts that `systemctl restart spinifex-daemon` preserves every guest VM so the new daemon can reattach via the persisted local state file. The previous setupShutdown SIGTERM handler called vmMgr.StopAll(), gracefully QMP-powering-down every guest before the daemon exited — ~90s wall-clock and zero surviving QEMU on the other side. CI run 25666369604 surfaced this as `recovered instance i-... has no live QEMU PID`. Two changes restore the Tier 1 contract: - setupShutdown no longer calls StopAll. It still flips d.shuttingDown so crash/restart handlers bail out, persists state, unsubscribes from NATS, and exits. VMs are only stopped when coordinated cluster shutdown drains them — handleShutdownDrain still calls StopAll and flips the flag before the SIGTERM path runs. Operator-visible behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs running. To stop VMs, use the cluster-shutdown topics or terminate via the AWS API. - The systemd unit gains `KillMode=process` so systemd only signals the daemon's main PID. With the default control-group mode, QEMU children were killed when systemd tore down the daemon's cgroup — defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the new fast-exit grace window. Also nil-guards natsConn.Close in the shutdown path: when a daemon restarts while NATS is unreachable (Scenario B), natsConn is nil and the close would panic. Mulga-siv-58 (reopened). * fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart) QEMU stdout/stderr were piped to daemon-side scanner goroutines. When the daemon exits (e.g. `systemctl restart spinifex-daemon`), the pipe read ends close; QEMU's next write triggers SIGPIPE and the guest dies — defeating DDIL Scenario B's "daemon restart preserves running instances" guarantee. Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr instead. The FD is dup'd into QEMU at fork, so closing it on the daemon side after Start() leaves QEMU writing into the file indefinitely. POSIX O_APPEND makes concurrent QEMU writes interleave atomically per line, so a single shared log is safe and avoids per-instance clutter in RuntimeDir. Trade-off: QEMU lines no longer land in journalctl -u spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly (path logged at instance start under the qemu_log field). mulga-siv-58 (DDIL Scenario B iteration 4) * fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's `PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()` falling through to `os.TempDir()` (=/tmp): - spinifex-daemon HOME=/var/lib/spinifex/spinifex, so `$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not exist and pidPath() returns /tmp. - Every daemon start gets a fresh private /tmp namespace. QEMU stays in the OLD namespace after the daemon dies (mount namespaces aren't changed by reparenting). The new daemon's /tmp is empty. - `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace the file doesn't exist, so PID=0. Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d, already in ReadWritePaths) and survives daemon restart. pidPath() short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and RuntimeDir() all resolve to the stable path. Both daemon incarnations and QEMU children see the same files. Revert iteration 4's pipe→file QEMU log change (commit |
||
|
Julian Sommer
|
dc3b48f17e | feat: security groups enforcement (#201) | ||
|
Julian Sommer
|
bb08335831 | fix: explicit workflow permission | ||
|
Julian Sommer
|
51f7afab4c | fix(ui): github workflow | ||
|
joshsiv-mulga
|
9ca8b84bea | ci(baremetal): skip ePDU power-cycle until credentials are wired in | ||
|
joshsiv-mulga
|
e9aec9e784 | ci(baremetal): restore bm-ci SSH key from secret at run start | ||
|
joshsiv-mulga
|
8c6d0747e0 | fix(ci): pass SPINIFEX_SRC/VIPERBLOCK_SRC to iso-builder, build viperblock .so | ||
|
joshsiv-mulga
|
374f2ed7a3 |
feat(ci): add e2e-baremetal workflow — workflow_dispatch only
Separate from normal e2e. Checks out spinifex, checks out mulga scripts (sparse: iso-builder + baremetal), builds spx + test tar via bm-build-artifacts.sh, builds PXE artifacts (squashfs/initrd/vmlinuz) via iso-builder with persistent cache on banksia, then runs run-baremetal-e2e.sh against the single bare-metal node. |
||
|
Julian Sommer
|
56cc83f723 | chore: merge main into dev | ||
|
Julian Sommer
|
a216d6f596 |
chore(ci): point nightly e2e at dev branch
Scheduled runs were using github.ref_name, which equals the default branch (main) — so the nightly was testing main. Hardcode GIT_BRANCH=dev so the build-install-artifacts.sh + tofu provisioning target dev's HEAD instead. GitHub always loads scheduled-workflow files from the default branch, so this change must stay in sync on main for the schedule to use it. |
||
|
Julian Sommer
|
cbe335e903 | chore: bump single node e2e timeout to 35 mins | ||
|
Julian Sommer
|
84278b794f |
chore: reintroduce dev branch in CI + scripts
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml) - workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch so checkout@v6 uses the default branch implicitly - dependabot: route gomod/github-actions/npm PRs to dev - diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev - iac/proxmox-template: default git_branch to dev for cluster builds Refs docs/development/improvements/reintroduce-dev-branch.md |
||
|
Josh Siviour
|
04f63ad8c6 |
feat(iso-installer): DHCP race fix + vpcd reconcile gate + ELB mgmt route + HashMAC + diagnostics (#172)
* preflight tweak
* fix: pin gateway LRP to link-local, end intra-cluster ARP collision
Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on
attach. Multi-VPC clusters all GARP'd the same .160 with three different
deterministic MACs on the shared br-ext localnet, flapping the upstream
ARP cache and dropping 100% of inbound traffic to dnat_and_snat external
IPs (mulga-siv-26).
handleIGWAttach and reconcileIGW now unconditionally set the gateway
router port to 169.254.0.1/30. The default route's OutputPort already
pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay
allocated only for per-VM dnat_and_snat external addresses, which are
unique by construction.
Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites
stale Networks in place via the new UpdateLogicalRouterPort path. Needed
because Reconcile gates IGW work on "ext switch missing", so a cluster
shipped with the buggy CIDR never re-enters reconcileIGW.
Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision
and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests
updated to assert link-local and OutputPort=&gwPortName. Log field
gateway_ip renamed to lrp_network to signal the semantic shift.
* fix: rewrite utils.HashMAC to SHA-256 / 46-bit output
The old 24-bit rolling hash with literal class prefix had birthday-paradox
collision probability of 1% at ~580 ids and 50% at 4096 — well within the
range of a single deployment's resource population (LRPs, ENIs, dev/mgmt
NICs). On a shared L2 segment, two clusters could trivially produce
duplicate MACs, breaking neighbour discovery and DHCP.
Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes
and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the
first octet. Output entropy goes from 24 bits to 46 bits — 1% collision
boundary moves from ~580 to ~1.2M ids.
Drop the redundant prefix arg: it was no longer the literal output prefix
under the new scheme, just folded into the hash input. Class disambiguation
moves to the caller, where it's explicit. Three of four callers had nothing
to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and
mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags.
Also consolidate vpcd.generateMAC, which was a local duplicate of the
broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation.
Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md
Bead: mulga-siv-27
* fix: pin HashMAC first octet to 0x02
Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast,
bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived.
Technically valid LAA unicast, but produced MACs like 4a:.. and fe:..
that visually encroach on vendor OUI space and tripped CI/ops checks
expecting all generated MACs to start with 02:.
Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output
entropy drops from 46 to 40 bits — birthday-paradox 1% collision at
~150k ids, still 256x the old 24-bit bound (~580).
* chore(e2e): dump cross-chassis OVN state on guest SSH failure
Adds chassis registrations, port_binding claims, gateway_chassis rows,
ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to
dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis
dataplane stalls (cr-gw chassis vs remote chassis with the VM bound).
* fix: gate vpcd startup reconcile behind NATS KV leader election
N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router
with no atomicity, producing duplicate rows that ovn-nbctl rejects with
"Multiple logical routers named '...'. Use a UUID."
Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key
in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile,
ReconcileFromKV, and the two retrofit passes — others Subscribe and rely
on the vpcd-workers queue group for runtime events. MaxAge=60s on the
bucket caps recovery time when a leader crashes mid-reconcile.
Falls through (elected=true) when KV is unavailable so first-boot does
not deadlock on a missing JetStream.
mulga-siv-29
* fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30)
Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s
because the lb-agent inside the ALB VM has no return path to AWSGW.
The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP
for single-node and deliberately skips adding the /32 mgmt host route — the
WAN-return-path concern in the comment is real but applies only to
internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that
gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when
the agent dials advertiseIP the reply targets the VM's private subnet IP from
the host's WAN with no route back.
Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the
existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and
falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only
internal LBs need the /32 route, so internet-facing single-node still gets
no bootcmd and the host-initiated WAN path stays clean.
Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the
fallback has the inputs it needs.
* fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31)
The test was calling real SetupComputeNode("chassis-test", ...) which shelled
out via sudoCommand to set external_ids:system-id=chassis-test on the host's
live OVS. On a dev box with OVS installed this overwrote the persistent UUID
(owned by openvswitch-switch and re-applied only on reboot), so vpcd's
discoverChassis filtered out the legitimate SBDB chassis row as 'stale local'
and crashlooped — taking out all VPC ops including ALB ENI provisioning.
Bind sudoCommand to a var so tests can swap it. Stub it in the test to return
exec.Command("/bin/false") and assert SetupComputeNode returns an error. The
test's intent (signature + error-path coverage) is preserved; nothing on the
host is touched.
* chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32)
Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8
DORA timeouts correlate with concurrent Acquire calls on the same
bridge. Tracks per-bridge in-flight count and wallclock duration of each
Acquire; logs WARN on failure with both fields, DEBUG on success.
No behavioural change.
* ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32)
run-multinode-e2e.sh dumps node journals via its own EXIT trap, but
run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts
fail the cluster is destroyed in Teardown without surfacing any daemon
logs, blocking root-cause analysis.
Add a workflow step that runs after the test scripts but before
Teardown when a previous step failed, ssh'ing to each node and pulling
the last 200 lines of every spinifex-* unit. Best-effort: skips
gracefully when inventory or nodes are unreachable.
Diagnostic-only; no behavioural change to the test path.
* fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32)
Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when
dhcpcd-base is installed. It binds br-wan and races vpcd's in-process
nclient4 client for upstream OFFERs, draining the lab DHCP pool with
duplicate leases and causing intermittent DORA timeouts in the e2e
public-IP path (worst on cell-8 real-multi/dhcp/debian-13).
The ISO installer already masks the unit at install time
(cmd/installer/install/install.go:316). The binary install path used by
e2e nightly missed it, so debian hosts installed via setup.sh ran with
both DHCP clients live. Mirror the ISO mask here.
Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable
dhcpcd.service on Noble).
* ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32)
Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the
dhcpcd.service journal alongside the spinifex-* units, plus
systemctl is-enabled / is-active and the DHCP client port (UDP 68)
listener so we can see directly whether dhcpcd is running and bound
to br-wan when an Acquire times out.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
|
||
|
Julian Sommer
|
a3994510e0 | feat: e2e nightly reboot tests (#168) | ||
|
Julian Sommer
|
138083f6c8 | chore: only run e2e nightly mon to friday | ||
|
Julian Sommer
|
f2dcb2f33f | fix: tofu nightly e2e | ||
|
Julian Sommer
|
7128dc785c | chore: dhcp e2e nightly | ||
|
dependabot[bot]
|
f3cfee3189 | chore: bump actions/upload-artifact in the github-actions group (#163) | ||
|
Julian Sommer
|
eeb7ebc5db | chore: enable nightly nginx-alb workbook (#161) | ||
|
Julian Sommer
|
c7d4040dd3 | feat: nightly e2e (#155) | ||
|
Josh Siviour
|
068aef5589 |
feat: DDIL E2E test harness (mulga-siv-5) (#160)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc. |