mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 19:25:52 +00:00
908aa89dbb96a3eadb37690ce4facfaba112aa69
144 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 joshsiv-mulga
|
908aa89dbb |
test(e2e-single): extend SSH wait budget to 3m on slow boot paths
CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while sshd was still finishing pam/cloud-init. Bump all four wait sites to 3m as a stop-gap: - single/instance_test.go phase5aii_SSHProbe - single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre) - single/vpc_test.go phase8b_VPCSubnetE2E - single/natgw_test.go phase8d bastion SSH wait Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH that probes the actual handshake, not just tcp/22 reachability). |
||
|
joshsiv-mulga
|
760acd1eb4 |
test(e2e-lb): resolve peer availability before shared fixture setup
setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes on capacity-tight dev hosts) and previously ran BEFORE the peer pickup + ssh.Ping. On single-node, the InternetFacing_* subtests would correctly t.Skip on peer == "" — but only after the fixture work and once subtests started, so the wait appeared to be attached to the first subtest name gotestsum printed. Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip" immediately, then build the fixture for the internal subtests. No behavior change for multi-node + peer-reachable; on single-node the skip rationale surfaces before the fixture wait. |
||
|
joshsiv-mulga
|
afaf3fe8da |
test(e2e): parallelise hot phases + tune fast-path polls
Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four
quick wins from decomposing run-e2e.sh into Go phase subtests:
- Phase 8 negatives (single/negative_test.go): t.Parallel() inside
each of the 18 read-only error-path sub-tests. ~5-10x speedup.
- IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays
seeded sequentially (populates fix.IAMAdminAccount); the other
four policies fan out as siblings under a wrapping
create_policies_parallel t.Run. ~5x on the Phase 4 sub-step.
- Volume + snapshot fast paths (single/{volume,snapshot}_test.go):
pass harness.WithPoll(500ms) on the five state transitions that
finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock.
- Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4
inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes).
No harness changes, no Fixture changes, no test-execution-order
changes. mulga-siv-96.
|
||
|
joshsiv-mulga
|
19f11ffea9 |
fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops
Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs filter fix: 1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then saw "Permission denied (publickey)" / "Identity file not accessible" and the cascade left the instance in a state that broke 7c-pre's `start-instances`. Move the PEM under a TestSingleNode-scoped `fix.TmpDir` so it survives until the whole test exits. 2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was intended as a no-op (Phase 4 populates the field), but require.NotEmpty fails on empty regardless of message. Removed the bogus check. 3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the policy-version family (IAM Phase 4): `CreatePolicyVersion`, `ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`. They return `InvalidAction` and abort the phase. Wrap each block in a t.Run sub-test with t.Skip so the surrounding phase still runs and the skips show up explicitly in the test report. |
||
|
joshsiv-mulga
|
4280972018 |
test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go
Stage G of the single-node Go port (mulga-siv-89) — final teardown + post-teardown verification. Adds tests/e2e/single/teardown_test.go (226 lines): - phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage (CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated -> DeleteKeyPair(KeyName). Each call swallows *NotFound via harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b always run. - phase9a_VerifyTeardown: DescribeInstances must show terminated or NotFound; DescribeImages must NOT include the deregistered AMI; DescribeKeyPairs must NOT include the deleted key. Clears the consumed Fixture fields. - phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the terminated InstanceID; orphan i- rows surface as soft Logf warnings to avoid flakes when parallel suites share the node. Wired into TestSingleNode. With this, mulga-siv-89's single-node Go port (Stages A-G) is feature-complete; run-e2e.sh remains in tree until a separate cleanup bead deletes it after CI burn-in. |
||
|
joshsiv-mulga
|
1d4e851532 |
test(e2e-single): port run-e2e.sh phases 8b-8e to Go
Stage F of the single-node Go port (mulga-siv-89). Adds three files under tests/e2e/single/: - vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch assertions, IGW attach, custom route table + 0.0.0.0/0 route, instance launch + SSH + external connectivity probe) and Phase 8c (route table validation — main RT association, local + IGW route presence, custom RT CRUD + association/disassociation, negative delete-local + delete-main assertions). - natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E — CreateNatGateway with EIP, private subnet route via NAT, bastion- hopped SSH into private VM, baseline egress drop / post-NAT egress succeed / post-teardown egress lost). Inline waitForNATGatewayState helper bounds the pending->available->deleted state machine via harness.EventuallyErr. - datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath — two SGs with UserIdGroupPair ingress rule, OVN port-group membership + SB address-set wait loops, allowed traffic curl, denied traffic nc, revoke->drop, re-add->restore). Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures teardown unwinds even on partial failure. Wired into TestSingleNode. |
||
|
joshsiv-mulga
|
5427cbe37b |
fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs
AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the daemon's filter allowlist enforces that. The harness sent the camelCase SDK field name (isDefault), so the daemon returned InvalidParameterValue 400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty and cascading 16 downstream Single-Node E2E subtests to fail. |
||
|
joshsiv-mulga
|
562d7d66fd |
test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go
Stage E of the single-node Go port (mulga-siv-89). Adds two files
under tests/e2e/single/:
- iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key
Lifecycle, User Authentication (via harness.NewAWSClientWithCreds),
Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion /
DeletePolicyVersion which bash lacks), Policy Attachment &
Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle
(DeleteConflict while attached), and idempotent Cleanup.
- account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps
(Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet /
IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases,
Cleanup). Uses harness.AccountCarousel for the multi-profile
credential map and SpxAdminAccountCreate for admin account
bootstrap.
Fixture extended with IAM cross-phase state (IAMAdminAccount,
IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive
even when an intermediate IAM phase short-circuits. All 11 Phase 8
Steps share local state in the outer function; t.Cleanup mirrors
Step 11 LIFO so a mid-Step failure still tears down accounts.
Wired into TestSingleNode. run-e2e.sh stays in tree until step-7
cleanup per parent plan.
|
||
|
joshsiv-mulga
|
3da760c692 |
ci(e2e-go): wire e2e-single target + add single to build set
Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D landed) be exercised in CI via explicit `suites: e2e-single` dispatch input ahead of Stage G flipping the workflow default. Workflow default suites unchanged until Phase 9 teardown lands. |
||
|
joshsiv-mulga
|
d697686003 |
test(e2e-single): port run-e2e.sh phases 5f-8 to Go
Stage D of the single-node Go port (mulga-siv-89). Adds four files under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement), Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start, attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error codes via harness.ExpectError / AssertAWSError / PostAWSAction). Wired into TestSingleNode in main_test.go; phase ordering preserved because Phase 7 chain mutates the shared instance state (stopped at end of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings) and Phase 8 negatives need a running instance with the AMI/key created earlier. Removes the placeholder phase7a_AttachToStoppedError stub from volume_test.go (real impl now in lifecycle_test.go). run-e2e.sh stays in tree until step-7 cleanup per the parent plan. |
||
|
joshsiv-mulga
|
162d840cf1 |
test(e2e-single): port run-e2e.sh phases 5-5e to Go
Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle that everything else hangs off: - Phase 5: run-instances (default VPC + default SG with tcp/22 opened via idempotent harness.AuthorizeSSHIngress), wait running, resolve root volume from BlockDeviceMappings. - Phase 5a-pre: spx get vms now contains the running instance. - Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs). - Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume size (±1 GiB tolerance — lsblk rounds down), soft hostname check. - Phase 5a-iii: get-console-output round-trip on InstanceId. - Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) → attach-volume → detach-volume → delete-volume, polling each state transition through harness.WaitForVolumeState. - Phase 5b-ii: describe-volume-status on the root volume. - Phase 5c: create-snapshot → wait completed → copy-snapshot → both visible → delete original → copy survives → delete copy. - Phase 5d: AMI snapshot-backed launch verification via DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side config.json reads are TODO (no S3 client in harness yet). - Phase 5e: create-image --no-reboot, wait available, capture backing snapshot ID for Phase 9 cleanup. Phase 7a (attach to stopped → error) stubbed in volume_test.go for Stage D to fill once the lifecycle stop/start lands. |
||
|
joshsiv-mulga
|
242d47074b |
test(e2e-single): port run-e2e.sh phases 1-4 to Go
Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture
that threads state between sequential phase subtests, plus phases:
- Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS
readiness polling, region/AZ smoke.
- Phase 1b: spx get nodes / top nodes / get vms (single-node only).
- Phase 2: nano instance-type + arch discovery.
- Phase 2b: serial-console-access default/enable/disable round-trips.
- Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM
kept for Phase 5 instance launch (cleanup deferred to teardown phase
in stage G).
- Phase 4: architecture-aware AMI discovery via name filter
ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output.
Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ).
detectPoolMode reads [network] external_mode from spinifex.toml so
stages F/G can skip pool-only phases on dev_networking fixtures.
No production code changed. cert + lb scenarios continue to compile.
|
||
|
joshsiv-mulga
|
23143bd0de |
feat(e2e-harness): extend harness for single-node Go suite port
Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh port consumes: - AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds for per-tenant account-scoping in Phase 8. - awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent of the bash expect_error helper used throughout the negative phases. - poll.go: WaitForInstance/Volume/Snapshot/Image state to any target, with per-resource default timeouts tuned from the bash sleep loops. - spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes, spx admin account create with output parsing for tenant-id/access-key. - rawhttp.go: signed raw POST against the AWS gateway for the Phase 8 InvalidAction probe (gateway rejects on auth before action parsing). - ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e datapath checks; WaitForPortGroupMember handles propagation polls. - ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd), LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC, idempotent AuthorizeSSHIngress. - account.go: AccountCarousel maps tenant name -> scoped AWSClient, built atop NewAWSClientWithCreds for the Phase 8 isolation tests. Pure groundwork — no new tests, no production behaviour change. The cert + lb scenarios still compile against the extended harness. |
||
|
joshsiv-mulga
|
ae789dd22d |
test(e2e): add harness.Phase / Step / Detail for human-readable test output
Test fixtures were emitting flat walls of t.Logf with every line prefixed
by the same `lb_test.go:61:` (file:line of the call site that the
t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim,
hard to find where one phase ends and the next begins.
Add three helpers in tests/e2e/harness/log.go:
Phase(t, fmt, ...) — flush-left colored banner via fmt.Println.
Bypasses t.Logf so the line lands without the
testing framework's indent/file:line prefix.
Pure visual structure; the diagnostic detail
stays on t.Logf.
Step(t, fmt, ...) — single-line sub-phase marker via t.Logf.
Keeps file:line attribution so a triager can
find the source of a progress message.
Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same
attribution rationale as Step — IDs are
diagnostic and want to land in JUnit XML
bucketed with the owning test.
Why the split: Phase is purely a banner; losing JUnit attribution costs
nothing because banners aren't diagnostic. Step/Detail carry triage
context (instance IDs, AMI, etc.) and a failing test's JUnit testcase
node should still contain them, so t.Logf attribution is kept.
ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS
isn't set, so piped local `go test -v` stays clean.
Apply to lb_test setupSharedFixture — five Phase banners replace the
implicit run of helper logs ("Discovering cluster capacity",
"Ensuring SSH key pair", "Creating shared VPC topology",
"Launching app instances", "Launching probe client") so the live CI log
reads as a story rather than a flat resource dump.
|
||
|
joshsiv-mulga
|
964f4038e0 |
test(ddil): quarantine Scenario C until predastore §2 lands
CI run 26016840819 surfaced that ScenarioC_CleanPartition silently depends on predastore-ddil-hardening §2 (write-quorum with deferred parity repair). Root cause from node3 daemon log: predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted predastore: putObjectViaQUIC: put failed spx QMP: BLOCK_IO_ERROR instanceId=<witness> spx: StopOrTerminateInstance: Terminating spx: Instance state transition running → shutting-down The 'sendmsg: operation not permitted' is the EPERM Linux returns when iptables drops an outbound packet — exactly the OUTPUT DROP rules PartitionNode installs. The witness VM's block I/O is backed by distributed predastore, so the same partition that severs node3↔peer NATS traffic also severs the witness disk's cross-shard QUIC writes. QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's reconciler terminates the instance, and the next AssertProgressed dials a sshd that's already gone (hence the 'handshake failed: EOF' the test reports). Quarantine by collapsing C to the same scenarioSkip stub D/E/F already use, citing predastore-ddil-hardening §2 as the dependency. Flip C back to ENABLED in the same PR that lands §2 — that's when the witness disk can survive a peer partition without QEMU halting. TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine note explaining the predastore dependency. |
||
|
joshsiv-mulga
|
e1c164d663 |
fix(e2e): open default SG :22 ingress via structured form before witness launch
The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress form silently (mulga-siv-79), so the default SG starts with no inbound :22 and the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the workaround already present in lb_test by issuing the structured IpPermissions form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns in the same cluster are safe. |
||
|
joshsiv-mulga
|
06ca05d611 |
refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness
Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes
Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness
into the shared package so the LB, multinode, and BM scenarios in
steps 4-5 can build on the same primitives instead of duplicating SSH
+ cluster plumbing.
Moves
- ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go
-> harness/ (plus the witness testdata embed dir).
- ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a
new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/
RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot
now take harness.Cluster / harness.Node / harness.SSH explicitly.
- ddil/harness/retry.go::Run retry-with-reset extracted as
harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go
keeps the DDIL-only quarantine wrapper that calls into it.
- WaitForMode moved from fault.go to harness/daemon_client.go since
it polls daemon state and has no fault-injection concern.
- The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved
as harness/peer_ssh.go so the lb_test suite keeps working unchanged;
the typed x/crypto/ssh client from DDIL now owns harness/ssh.go.
- New harness/util.go exports ShellQuote (was unexported shellQuote in
netem.go) so both ddil/fault and reset.go/witness.go can use it.
Env var rename
- SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace
DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured
with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps
passing until .github/workflows/e2e-ddil.yml drops the old names.
DDIL_QUARANTINED stays DDIL-specific.
DDIL parity preserved — scenarios still call ddilh.Run for quarantine
semantics; the underlying retry-with-reset is the same code path,
just shared.
Closes mulga-siv-74.
|
||
|
joshsiv-mulga
|
3b272295a0 |
ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM
Replaces the source-tree + go + gotestsum payload that the e2e-go.yml workflow previously shipped to the tofu CI VM with self-contained test binaries built on the runner via `go test -c`. testdata for the LB suite is already //go:embed'd into the binaries, so the bundle on the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed). Workflow changes - actions/setup-go on runner. - New step builds binaries via `make e2e-build SUITES_BUILD=...`. - Install go-junit-report on runner; convert each VM-side test log to junit-<suite>.xml after artifact pull so "Publish test summary" still parses JUnit XML. - Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network). - Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth). - VM step runs `./<suite>.test -test.v -test.timeout=30m` directly. Makefile changes - New `e2e-build` target compiles each `SUITES_BUILD` entry into `BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also removes BIN_DIR. Closes mulga-siv-80. |
||
|
joshsiv-mulga
|
9580cfd67a |
fix(e2e): discover single-node IPs + awsgw bind in LoadEnv
run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and adds every non-loopback global IPv4 to the node list. The Go harness defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the LAN IP, not loopback) saw 'connection refused' on every TLS check. Port the same discovery: ModeSingle now walks net.InterfaceAddrs for NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS. |
||
|
joshsiv-mulga
|
afa933ac05 |
feat(e2e): share probe client across both internal LB suites + SG fixes
- Replaces per-suite probe client with a single shared VM launched in the
fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB
internal suites.
- Client cloud-init exposes a JSON trigger endpoint on :9090 in addition
to the http.server on :80 (results dir). Each suite POSTs
{proto, ip, count, outfile, http_port, tcp_port} to run a probe loop
against the LB private IP and writes the result file under /tmp/httpd
for the driver to fetch back.
- configureDefaultSG now uses the structured IpPermissions form (top-level
shortcut is silently ignored by vpcd — tracked separately) and opens
port 9090 in addition to 80 + 9000.
- deleteLB captures the underlying sys.micro VM id via the LB ENI before
delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so
the next suite doesn't race the capacity reclaim.
- Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate
fully before its own createLB fires. DescribeLoadBalancers can report
the ALB gone before the vCPU/memory allocation is actually released on
capacity-tight dev hosts; without this settle, NLB's createLB trips the
sys.micro reserve and lands in terminal state=failed. Still cheaper
than cold-booting a per-suite client.
|
||
|
joshsiv-mulga
|
5591915163 |
ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in
Standalone workflow runs the new Go-based E2E harness on the existing ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex, installs gotestsum, runs each requested suite via the e2e Makefile, and uploads per-test artifacts + JUnit XML + JSON streams as a build artifact. Test summary parses the JUnit and emits ::error file=...:: annotations so failures surface in the PR diff. Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't interfere with the existing e2e.yml while the Go conversion lands. tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR under a suite-tagged filename derived from the first PKG component. |
||
|
joshsiv-mulga
|
51078d2613 |
fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state
Reverts the shared probe-server VM that wouldn't come up reliably during the InternalFacing suites — back to the proven bash pattern of launching a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on the suite kind). Adds renderClientUserData + a conditional template so each client probes only its target LB. WaitForLBActive now bails immediately on terminal LB states (failed, provisioning_failed) and emits a "still waiting" log line every 30s so hangs are visible instead of silently burning the full 5-minute timeout. |
||
|
joshsiv-mulga
|
f902131841 |
refactor(e2e): serialise LB suites + share probe client across internal tests
Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB × internet-facing/internal). Each suite creates its own TG + LB + listener, runs its traffic test, and tears down before the next starts. Replaces the per-suite throwaway client VM with a single long-lived probe server launched at fixture setup; per-suite internal traffic tests GET /probe?ip=...&proto=... against it instead of cold-booting a fresh client. Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the internal phase passes on capacity-constrained dev nodes that previously hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying spinifex bugs). |
||
|
joshsiv-mulga
|
48f9b86984 |
fix(e2e): log client launch params + dump daemon log on InvalidInstanceType
Adds diagnostics for the LB internal client launch path. Logs the exact InstanceType, AMI, subnet, key name, and user-data size right before RunInstances, and registers a daemon-log dump on failure so we can correlate the 400 InvalidInstanceType with what spinifex saw. |
||
|
joshsiv-mulga
|
0cb4b7795d |
feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario
Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246 lines of bash with a typed Go scenario exercising all 4 LB variants (ALB/NLB × internet-facing/internal) against a shared VPC + app instance pool. Internet-facing phase auto-skips when no peer node is configured; internal phase always runs. Adds shared harness primitives: - harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway - harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, WaitForInstanceRunning/Terminated, InstanceENI - harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts - harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh (replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74) User-data scripts embedded via //go:embed under lb/testdata/. Bash script stays in place for one CI cycle before deletion. |
||
|
joshsiv-mulga
|
3298d89c2a |
feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh
Lands step 1 of docs/development/improvements/e2e-go-harness.md: shared harness primitives (env, wait, artifacts, cert, tls) plus the TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files gated by //go:build e2e so default builds and `make preflight` are unaffected. Bash script stays in place for one CI cycle before deletion. |
||
|
Josh Siviour
|
4ae2a4ac53 |
feat(elbv2): QEMU microvm direct-boot for LB system instances (#228)
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline
vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go
config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput
image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts
* feat(daemon): direct-boot launch path for microvm system instances
When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration
PC-machine path (DirectBoot=false) unchanged.
Closes mulga-siv-65
* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold
* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed
* fix(microvm): end-to-end fixes for direct-boot LB path
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
* fix(build): auto-detect docker/podman when apk unavailable for microvm image build
* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
* fix(microvm): fix mgmt NIC routing for all LB schemes
Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
* feat(microvm): Phase 2 kernel stripping + boot optimisation
- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches
* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip
The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.
Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.
* fix(build): keep failover modules in initramfs strip
virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.
Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.
* fix(daemon): restore slog default after viperblock.New mutates it
viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.
Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.
Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.
Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).
* perf(microvm): hit Phase 2 boot/memory gates
t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.
Root causes:
1. build-microvm-image.sh staged the entire chroot into the cpio
without dropping /boot, embedding a duplicate vmlinuz, Alpine's
stock initramfs-virt, System.map, and a "boot → ." symlink loop
(~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
the kernel is copied out, before the cpio. Initramfs drops from
31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
42 MiB → 22 MiB.
2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
no firmware, ROM, or block layer). Both extracted to
directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
qemuStartSettleWait to 50 ms.
3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
Added boot-time perf flags: quiet loglevel=3 (silence printk over
115200-baud isa-serial), mitigations=off (KVM guest on trusted
host, single-tenant LB workload), tsc=reliable, no_timer_check,
reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).
4. init.sh dropped modprobe virtio_mmio (built-in:
CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).
Phase 2 gates:
- t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms
- artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script)
- ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure,
unrelated to microvm path)
* build(microvm): wire build into deploy + distro + release pipeline
Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).
Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.
Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.
* build(ansible): include microvm artifacts in build role tarball
Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).
* feat(config): enable microvm direct-boot LB path by default
Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.
Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).
* fix(microvm): bootstrap apk.static when no host rootfs tool exists
Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.
When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.
No-op when apk, docker, or podman is already present.
Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244
* revert: drop apk.static bootstrap in build-microvm-image
Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.
* docs(workbook): drop lb-alpine import for microvm direct-boot LB
LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.
* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN
When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").
Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.
* fix(e2e): redirect lb probe chatter to stderr so stalls are visible
test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.
Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.
* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node
buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.
resolveMgmtRoute already encodes the correct policy:
- multi-node any scheme → explicit MgmtRoute{Gateway,Target}
- single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP)
- single-node internet-facing → empty (reach AWSGW via VPC SNAT)
Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.
Test covers all three branches.
* feat(ansible): dev-deploy playbook for in-place artifact swap
Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.
Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.
Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.
* chore(microvm): remove legacy LB Alpine image build pipeline
Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.
build-system-image stays — other manifests (e.g. GPU images) still
use it.
* refactor(microvm): remove legacy PC-machine LB boot path
Phase 3 — direct-boot becomes the only ELBv2 LB launch path.
- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
[daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
paths) and refresh stale comments that still mentioned lbVMUserData.
ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.
* docs: drop lb-alpine catalog entry from cluster setup guide
LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.
* feat(ui): remove LB image import banner + LB-AMI gating
Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:
- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
system-managed.ts (isSystemManagedImage stays — still used by the
EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
wizard pages
- The corresponding tests that asserted the banner appears
* revert: restore lb.conf + lb-setup.sh for mulga install pipeline
Phase 3 microvm removal (commit
|
||
|
Josh Siviour
|
7eff142c34 |
feat(e2e): bare-metal multi-node cluster support + density-aware guards (#227)
* feat(predastore): support 2-node distributed mode with RS(1,1)
Lower minimum node threshold from 3 to 2. Emit data=1,parity=1 when
exactly 2 nodes configured — effectively mirroring. 3+ nodes continue
to use RS(2,1).
* fix(ec2): route StartInstances back to original node with node-down fallback
ec2.start used a plain queue group, so NATS could route a restart request to
any daemon — including a capacity-constrained NUC that can't fit a second
instance. This caused InsufficientInstanceCapacity in the Phase 10 stop/start
test on 2-node bare-metal clusters where one node has significantly fewer
resources.
Each daemon now also subscribes to ec2.start.{nodeId} (no queue group). The
queue-group handler peeks at the instance ID, looks up LastNode from the
stopped KV, and forwards to ec2.start.{lastNode}. If the original node does
not respond within 5 seconds (down or partitioned), the handler falls back to
starting locally on whichever node picked up the request — preserving the
existing behaviour for node-failure scenarios.
The direct handler (ec2.start.{nodeId}) always starts locally and never
forwards, preventing routing loops.
* fix(ec2): fall back to local start when original node returns InsufficientInstanceCapacity
The forwarding handler was relaying all responses from the targeted node
unconditionally, including InsufficientInstanceCapacity. This meant that if
the NUC (node2) was the LastNode but had no remaining capacity, the error
propagated to the caller rather than falling back to the local node which may
have room.
Now only success and non-capacity errors are relayed; an InsufficientInstance-
Capacity response from the targeted node triggers a local start attempt, giving
any other node in the cluster a chance to absorb the instance.
* test(e2e): make VPC stop/start restart non-fatal on InsufficientInstanceCapacity
Asymmetric hardware (NUC node2) may not have capacity to restart all
VPC instances. IP-persists-while-stopped is still verified and fails
hard; restart verification is skipped with a warning on capacity errors.
* test(e2e): density-aware capacity guard across bare-metal multi-node suite
Sub-3-node bare-metal clusters with asymmetric/undersized nodes (e.g. a
4-vCPU NUC alongside a 16-vCPU box) can fit only one t3.nano on the
smallest node after the daemon's host reserve. Phases that direct or
indirectly land >1 instance on that node hit InsufficientInstanceCapacity
and previously hard-failed the entire suite.
Add is_density_skip helper and skip_test counter, and wrap every
density-sensitive run-instances / start-instances / wait-for-running call
site (phases 3, 5, 7, 10, 11) so capacity exhaustion on sub-3-node
clusters records a SKIP rather than a FAIL. On 3+ node clusters the
guard short-circuits to false, preserving existing failure semantics.
Summary block now prints Skipped count + names so degraded coverage is
visible in CI logs.
Refs: mulga-siv-72
* test(e2e): parameterise lb SSH key + user via PEER_SSH_KEY / PEER_SSH_USER
Allow run-lb-e2e.sh to run against clusters that don't use tf-user with the
tofu-cluster SSH key (e.g. bare-metal nodes which use spinifex@ + bm-ci key).
Defaults preserve existing behaviour on the tofu-cluster CI path.
Refs: mulga-siv-73
|
||
|
joshsiv-mulga
|
5b3ced92f9 |
fix(e2e): increase SSH wait timeout and fix console log path for bm
- SSH wait: 60 → 120 attempts; NBD-backed VMs on bare metal take longer to boot than tofu-cluster VMs with local storage - Diagnostic console log: add /run/spinifex/console-<id>.log which is where Spinifex QEMU actually writes serial output |
||
|
joshsiv-mulga
|
ee986fefe6 |
fix(e2e): remove hardcoded 3-node assumptions from bm multinode test
Replace all literal `2`/`3` node counts with `NODE_COUNT`-derived expressions so the test runs correctly against any cluster size. - NATS recovery peer threshold: `>= 2` → `>= NODE_COUNT - 1` - spx get nodes after recovery: `>= 3` → `>= NODE_COUNT` - Spread placement: `--count 3` / loop `0 1 2` → `NODE_COUNT`; thresholds updated - NAT GW step label: static "3 nodes" → `$NODE_COUNT` |
||
|
joshsiv-mulga
|
7b755227d5 |
fix(e2e): NATS peer count expects NODE_COUNT-1 routes not hardcoded 2
/routez reports outbound routes; with N nodes each node has N-1 routes. A 2-node cluster has 1 route, not 2. Test was written for 3-node tofu cluster and failed on 2-node baremetal. |
||
|
joshsiv-mulga
|
f667e2e0a6 |
fix(e2e): fix unbound variable on 2-node bare-metal cluster
Hardcoded 'for i in 1 2' loops assumed 3+ nodes, accessing NODE_IPS[2] which doesn't exist on a 2-node cluster. With set -u this is a fatal unbound variable error. Use seq 1 $((NODE_COUNT-1)) instead. |
||
|
Julian Sommer
|
de3ef1bb96 | fix: extra logging | ||
|
Julian Sommer
|
970b3bb58d | fix: allow port 80 | ||
|
Julian Sommer
|
d3290be6e7 | fix(e2e): use custom security group for alb | ||
|
joshsiv-mulga
|
08935ff7e7 |
fix(e2e): replace guest SSH uptime check with LaunchTime assertion
Phase 8.3 polled VPC instance IPs via two-hop SSH. In single-node macvlan mode the host has no route to VPC subnets, so SSH always timed out (10 min of dead retries) even when the VMs were healthy (ALB targets passing health checks). Replace with a LaunchTime comparison: the daemon resets LaunchTime to time.Now() when it relaunches QEMU on recovery, so LaunchTime >= REBOOT_START is a fast, network-topology-independent reboot confirmation. |
||
|
joshsiv-mulga
|
7042f25861 |
feat(e2e): add run-bm-multinode-e2e.sh for bare-metal N≥2 node clusters
Bare-metal variant of run-multinode-e2e.sh: - Accepts 2+ node IPs (not hardcoded 3) - SSH key/user via PEER_SSH_KEY / PEER_SSH_USER env vars (default: bm-ci / spinifex) - NODE3_IP optional; node3 survival check skipped when absent Workflow updated: job renamed to Multi-Node E2E, timeout 60→120 min. |
||
|
Julian Sommer
|
7c08c62e0d | chore: increase tofu e2e logging | ||
|
Julian Sommer
|
8d1258e4b7 | fix: tofu e2e | ||
|
Josh Siviour
|
45555dde49 |
test(ddil): flip Scenarios A/B/C to real assertions (#205)
* test(ddil): flip Scenarios A/B/C to real assertions Replaces the t.Skip stubs landed in #204 with the assertions described in docs/development/improvements/ddil-e2e-test-harness.md §3: - A NATSKill — KillNATS on node3, assert ModeStandalone within 30s, /local/instances still serves on the isolated node, majority /health on node1+node2 stays up, witness counters on all three nodes keep advancing, snapshot survives reconnect. - B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert ModeStandalone within 30s of restart, recovered instances present with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post PID equality dropped — see file comment; recovery + a non-zero PID give the same regression coverage without coupling to the daemon unit's KillMode. - C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep serving /health and /local/instances, isolated node3 reports ModeStandalone with /local/* responsive, witness counter on the partitioned side keeps advancing, heal converges and snapshot is preserved. Supporting changes: - harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore, the deployed daemon ClusterManager binds [nodes.<id>.daemon].host. The Phase-1 stub used a placeholder that would have blocked any /local/* assertion against the real cluster. - harness/witness.go — export WitnessVM.Terminate so scenarios can clean up their counter VMs from t.Cleanup. - scenarios/main_test.go — requireLiveCluster + launchWitnesses helpers to keep the three flipped scenarios tight. - TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED; TestCoverageDrift continues to enforce alignment with the package. Validation: built with -tags=e2e, dry-run scenarios pass, coverage drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against tofu cluster is the next step. * fix(ddil): countDropRules — single iptables -S call + filter Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs across both chains, but iptables -S takes a single optional chain — the second -S errors to stderr, the combined SSH buffer poisons the grep input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState. Switch to one `iptables -S` (lists every chain in iptables-save format), filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin. Error path now also surfaces the raw SSH output for the next time this breaks somewhere unexpected. * ci(e2e-ddil): export spinifex AWS creds for witness SDK The witness EC2 client (harness/witness.go NewWitness) calls session.NewSession with no static credentials, so the AWS SDK falls through its default chain — env vars → shared profile → IAM role — and on the CI runner none of those exist. Result on the live cluster: "NoCredentialProviders: no valid providers in chain. Deprecated." launches every Scenario A/B/C attempt straight to t.Fatal at launchWitnesses. spx admin init writes [spinifex] aws_access_key_id / aws_secret_access_key to the operator's ~/.aws/credentials on node1 during bootstrap. Pull them over SSH and export as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step resolves before it gets to the empty profile/IAM checks. Hard-fail the step if either field is missing — better than silently watching every scenario t.Fatal on the same line. * fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually green on the tofu cluster. mulga-siv-56. 1. Witness auto-discovers the smallest registered instance type (DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded t2.micro default returned InvalidInstanceType on every Scenario A/B/C launch because the cluster doesn't register that type. DDIL_WITNESS_- INSTANCE_TYPE stays as an operator override. 2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/ SKIP lines instead of a JSON wall. results.json artifact preserved for future flake-rate dashboarding. 3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope <Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>. aws-sdk-go v1's ec2query handler rejects the latter with SerializationError and drops the code, leaving Go callers without an awserr.Code() to inspect. New SDK round-trip test in gateway_test.go guards the regression. * ci(e2e-ddil): install gotestsum from prebuilt tarball `go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose tokeninternal static-assert pegs a narrow Go window (~1.22). Both older and newer runner Go versions trip `invalid array length -delta * delta`. A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely without changing the gotestsum version pin. * ci(e2e-ddil): collect cluster logs on failure Witness `findHost` is currently failing with "could not locate QEMU host for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is visible on any node. Cluster gets destroyed during teardown, so logs have to be pulled before that runs. Adds a failure-only step that ssh's each node and bundles journalctl spinifex-*, process listings, and state dirs into the ddil-results artifact for offline triage. * fix(ddil): support tap-mode QEMU networking in witness harness findHost no longer extracts hostfwd port — production launches go through auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never hostfwd. Witness was timing out across all 3 nodes. Discover guest IP via DescribeInstances.PrivateIpAddress (populated by PrepareRunInstances on ENI auto-create) and tunnel SSH through the host node to guestIP:22. Host routes to the tap bridge subnet, so the relay works without runner-side bridge routing. WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed. Closes mulga-siv-57. * fix(ddil): SSH witness via public IP, drop unreachable tunnel path tofu-cluster bootstraps with --external-mode=pool which forces dev_networking=false → no QEMU hostfwd dev NIC and no host route to the ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to host kernel). Previous tunnel-via-host fix could find the host but couldn't dial the guest. Switch to the path shell e2e already uses: default subnet has MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external pool, and the runner reaches that IP via the same WAN subnet as the cluster nodes. No host relay required, survives Scenario C (partition only DROPs peer cluster IPs, not the runner's source IP). Workflow now scp's ~/.ssh/spinifex-key from the primary node (the counterpart to the public material bootstrap-install.sh imports) and exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against cloud-init's authorized_keys. WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed. Continues mulga-siv-57. * fix(ddil): tunnel guest SSH through host node EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet, which is OVN-routed — reachable from any cluster node, not from the off-cluster runner sitting on the DDIL_NODES WAN subnet. Restore the host-jump pattern: SSH the hosting node (already known from findHost), dial publicIP:22 over that connection, run a guest SSH handshake over the tunnel. Symmetric with how shell e2e reaches guests from inside a node. Continues mulga-siv-57. * fix(ddil): wait for witness sshd before baseline read Previous run reached the guest EIP via the host tunnel but hit 'Connection refused' — sshd hadn't bound yet. waitForRunning only checks EC2 state, which flips on QEMU start, not cloud-init completion. The OVN dnat_and_snat rule for the auto-allocated EIP is also written asynchronously, so the tunnel can land before either layer is ready. awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for up to 90s, retrying on transient transport errors (refused/reset/EOF/ timeout/no-route) and failing fast on auth errors so a bad key surfaces immediately. Called once after resolvePublicIP, before the baseline counter read. Continues mulga-siv-57. * fix(ddil): case-insensitive transient SSH error match awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh emits "ssh: rejected: connect failed (Connection refused)" with capital C — strings.Contains returned false and the helper bailed on first attempt instead of polling. Last run failed in 49s, not the 90s budget. Lowercase the error before matching, and add "connect failed" as the canonical signal for "jump host dial through tunnel failed". Continues mulga-siv-57. * fix(ddil): default guest SSH user to ec2-user Shell E2E SSHes the same ami-ubuntu-* images as ec2-user (tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu and hit 'unable to authenticate, attempted methods [none publickey]' — authorized_keys is provisioned under the AMI default account, which is ec2-user, not ubuntu. Continues mulga-siv-57. * fix(ddil): wait for witness-counter.service to seed /var/lib/counter awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables witness-counter.service runs in the final cloud-init stage — after sshd binds. Baseline read landed in the window where sshd accepts but ExecStartPre hasn't created /var/lib/counter: cat: /var/lib/counter: No such file or directory Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the full path (tunnel + SSH handshake + cat) and retries on the transport errors plus "no such file or directory". Single 2-minute budget covers cloud-init final stage on tofu-cluster hardware. Continues mulga-siv-57. * fix(systemd): drop Wants=spinifex-nats from daemon unit Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately killed spinifex-nats was getting pulled back up the moment the daemon restarted — masking the DDIL Tier 1 standalone-startup path and breaking the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact: spinifex.target's Wants= activates every spinifex unit on boot. After= remains for ordering and is non-blocking when NATS is absent. scripts/test-ddil-phase1.sh hit the same trap and worked around it with a runtime drop-in plus sentinel file; the fix here makes that workaround unnecessary for any caller that restarts the daemon while NATS is stopped. * feat(daemon): peer-health probe drives standalone mode on partition Mode() is now derived from two atomic signals — natsConnected (NATS client TCP link, flipped by the existing disconnect/reconnect callbacks) and peersReachable (any peer daemon's /health responded in the most recent probe cycle). Reports cluster iff both are up; standalone otherwise. The peer-health probe is the missing signal for DDIL Scenario C: an iptables DROP between peer IPs blocks NATS cluster-routing traffic but leaves each node's local NATS client TCP connection healthy, so nats.Conn.IsConnected() keeps returning true and onNATSDisconnect never fires. The probe dials each peer's /health over the cluster network with a 2s timeout on a 2s ticker; worst-case detection of total partition is ~6s on a 3-node cluster, well inside the 30s Scenario C budget. Single-node clusters keep peersReachable permanently true (nothing to probe) so the derived mode flips to cluster as soon as the local NATS connect succeeds. The probe runs from startLocal so the partition signal exists even before (or in the absence of) a successful NATS connect. It does not touch NATS or JetStream, so the §1e Tier 1 invariant asserted by assertNoClusterServicesInitialised is unaffected. Tests: - TestMode_RequiresBothSignals — every nats×peers combination. - TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test. - TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses. - TestMonitorPeerReachability_EndToEnd — ticker-driven false→true. Closes mulga-siv-58. * fix(daemon): peer probe falls back to local daemon port `spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml; remote-node blocks carry just `host = "<ip>"` with no daemon sub-table. So `peer.Daemon.Host` was always empty in a deployed cluster and the probe failed closed, leaving peersReachable permanently false. Tofu-cluster run 25665158964 surfaced this as a Scenario A regression: after StartNATS, natsConnected flipped true on reconnect but Mode() stayed standalone because the derived signal required peersReachable. `a_nats_kill_test.go:75: node3 did not return to cluster mode … (still reporting another mode)`. Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port when present and otherwise reuses the local daemon's port (clusters are symmetric — every node binds the same port the template renders). New TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port- available case; TestProbePeerHealth_RejectsBadAddr is updated to clear self config so it still asserts the fail-closed path. * fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit DDIL Scenario B asserts that `systemctl restart spinifex-daemon` preserves every guest VM so the new daemon can reattach via the persisted local state file. The previous setupShutdown SIGTERM handler called vmMgr.StopAll(), gracefully QMP-powering-down every guest before the daemon exited — ~90s wall-clock and zero surviving QEMU on the other side. CI run 25666369604 surfaced this as `recovered instance i-... has no live QEMU PID`. Two changes restore the Tier 1 contract: - setupShutdown no longer calls StopAll. It still flips d.shuttingDown so crash/restart handlers bail out, persists state, unsubscribes from NATS, and exits. VMs are only stopped when coordinated cluster shutdown drains them — handleShutdownDrain still calls StopAll and flips the flag before the SIGTERM path runs. Operator-visible behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs running. To stop VMs, use the cluster-shutdown topics or terminate via the AWS API. - The systemd unit gains `KillMode=process` so systemd only signals the daemon's main PID. With the default control-group mode, QEMU children were killed when systemd tore down the daemon's cgroup — defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the new fast-exit grace window. Also nil-guards natsConn.Close in the shutdown path: when a daemon restarts while NATS is unreachable (Scenario B), natsConn is nil and the close would panic. Mulga-siv-58 (reopened). * fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart) QEMU stdout/stderr were piped to daemon-side scanner goroutines. When the daemon exits (e.g. `systemctl restart spinifex-daemon`), the pipe read ends close; QEMU's next write triggers SIGPIPE and the guest dies — defeating DDIL Scenario B's "daemon restart preserves running instances" guarantee. Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr instead. The FD is dup'd into QEMU at fork, so closing it on the daemon side after Start() leaves QEMU writing into the file indefinitely. POSIX O_APPEND makes concurrent QEMU writes interleave atomically per line, so a single shared log is safe and avoids per-instance clutter in RuntimeDir. Trade-off: QEMU lines no longer land in journalctl -u spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly (path logged at instance start under the qemu_log field). mulga-siv-58 (DDIL Scenario B iteration 4) * fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's `PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()` falling through to `os.TempDir()` (=/tmp): - spinifex-daemon HOME=/var/lib/spinifex/spinifex, so `$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not exist and pidPath() returns /tmp. - Every daemon start gets a fresh private /tmp namespace. QEMU stays in the OLD namespace after the daemon dies (mount namespaces aren't changed by reparenting). The new daemon's /tmp is empty. - `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace the file doesn't exist, so PID=0. Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d, already in ReadWritePaths) and survives daemon restart. pidPath() short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and RuntimeDir() all resolve to the stable path. Both daemon incarnations and QEMU children see the same files. Revert iteration 4's pipe→file QEMU log change (commit |
||
|
Julian Sommer
|
dc3b48f17e | feat: security groups enforcement (#201) | ||
|
joshsiv-mulga
|
5ea7df6e7c |
fix(e2e): revert to internet-facing ALB; keep LaunchTime reboot check
The internal-ALB approach was wrong on two counts: - ALB_ENI_MAC is the OVN VPC ENI MAC (lives on br-int), never appears in br-mgmt neighbour table, so the mgmt IP lookup always fails. - Switching to internal masked the real bug: OVN EIP DNAT not restoring properly after a host reboot, which is exactly what this test should catch. Restore internet-facing ALB and EIP-based traffic bursts. The post-reboot traffic failure (Phase 8.6) is the correct signal for the outstanding OVN EIP DNAT persistence bug. Keep the Phase 8.3 LaunchTime fix: two-hop SSH to VPC private IPs doesn't work in single-node macvlan (host has no route to VPC subnets). LaunchTime from describe-instances is fast and topology-independent. |
||
|
joshsiv-mulga
|
94405effec |
fix(e2e): cell-18 internal ALB + LaunchTime reboot check
Two changes to fix cell-18 (single/static/debian-13/reboot-single): 1. Switch ALB to --scheme internal. Internet-facing ALBs work pre-reboot on single-node but post-reboot the OVN EIP DNAT (nat-addresses=router GARP) is not re-established in time, causing 100% curl timeouts. Internal ALB routes traffic via br-mgmt using the mgmt IP discovered from the ALB ENI MAC in the br-mgmt neighbour table, bypassing OVN external NAT entirely. 2. Replace Phase 8.3 guest SSH uptime check with LaunchTime assertion. Two-hop SSH to VPC private IPs fails in single-node macvlan: the host has no route to VPC subnets. LaunchTime from describe-instances is reset by the daemon on QEMU relaunch, so LaunchTime >= REBOOT_START reliably confirms recovery without network reachability to the guest. |
||
|
Julian Sommer
|
46d7a875ab | fix: single node e2e direct s3 access | ||
|
joshsiv-mulga
|
14b8fc234a | fix: counts: unbound variable in bash 5.2 + set -u + empty declare -A | ||
|
Julian Sommer
|
56cc83f723 | chore: merge main into dev | ||
|
Julian Sommer
|
ce0bbde8dc |
fix(e2e): poll guest SSH after reboot in reboot-e2e cell
The post-reboot guest uptime check was a single-shot ssh with a 10s connect timeout. After the host comes back, guest QEMU + cloud-init + sshd inside each guest still need a few seconds to settle, so the check raced them. Poll up to GUEST_SSH_SECS (default 180s) per guest, mirroring the host TCP/22 poll. Run 25329248329 cell-18 reproduced this — ALB targets went healthy ~2s after the second SSH attempt failed. |
||
|
joshsiv-mulga
|
9ed59aac79 |
fix(e2e): prevent silent set -e abort on var=$(func) patterns
When a function returning non-zero is captured via command substitution under `set -euo pipefail`, the assignment exits the script before the following `if [ -z "$var" ]` diagnostic can run. CI failures showed up as a sudden jump to the EXIT trap with no error message. - run-multinode-e2e.sh: wrap get_remote_ssh_port in `if !`, bump retry 10 -> 30, add dump_guest_ssh_diagnostics on miss - run-pseudo-multinode-e2e.sh, run-e2e.sh: wrap get_ssh_port in `if !` - run-lb-e2e.sh: add `|| VAR=""` fallback on pick_peer |
||
|
Josh Siviour
|
04f63ad8c6 |
feat(iso-installer): DHCP race fix + vpcd reconcile gate + ELB mgmt route + HashMAC + diagnostics (#172)
* preflight tweak
* fix: pin gateway LRP to link-local, end intra-cluster ARP collision
Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on
attach. Multi-VPC clusters all GARP'd the same .160 with three different
deterministic MACs on the shared br-ext localnet, flapping the upstream
ARP cache and dropping 100% of inbound traffic to dnat_and_snat external
IPs (mulga-siv-26).
handleIGWAttach and reconcileIGW now unconditionally set the gateway
router port to 169.254.0.1/30. The default route's OutputPort already
pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay
allocated only for per-VM dnat_and_snat external addresses, which are
unique by construction.
Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites
stale Networks in place via the new UpdateLogicalRouterPort path. Needed
because Reconcile gates IGW work on "ext switch missing", so a cluster
shipped with the buggy CIDR never re-enters reconcileIGW.
Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision
and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests
updated to assert link-local and OutputPort=&gwPortName. Log field
gateway_ip renamed to lrp_network to signal the semantic shift.
* fix: rewrite utils.HashMAC to SHA-256 / 46-bit output
The old 24-bit rolling hash with literal class prefix had birthday-paradox
collision probability of 1% at ~580 ids and 50% at 4096 — well within the
range of a single deployment's resource population (LRPs, ENIs, dev/mgmt
NICs). On a shared L2 segment, two clusters could trivially produce
duplicate MACs, breaking neighbour discovery and DHCP.
Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes
and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the
first octet. Output entropy goes from 24 bits to 46 bits — 1% collision
boundary moves from ~580 to ~1.2M ids.
Drop the redundant prefix arg: it was no longer the literal output prefix
under the new scheme, just folded into the hash input. Class disambiguation
moves to the caller, where it's explicit. Three of four callers had nothing
to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and
mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags.
Also consolidate vpcd.generateMAC, which was a local duplicate of the
broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation.
Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md
Bead: mulga-siv-27
* fix: pin HashMAC first octet to 0x02
Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast,
bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived.
Technically valid LAA unicast, but produced MACs like 4a:.. and fe:..
that visually encroach on vendor OUI space and tripped CI/ops checks
expecting all generated MACs to start with 02:.
Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output
entropy drops from 46 to 40 bits — birthday-paradox 1% collision at
~150k ids, still 256x the old 24-bit bound (~580).
* chore(e2e): dump cross-chassis OVN state on guest SSH failure
Adds chassis registrations, port_binding claims, gateway_chassis rows,
ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to
dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis
dataplane stalls (cr-gw chassis vs remote chassis with the VM bound).
* fix: gate vpcd startup reconcile behind NATS KV leader election
N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router
with no atomicity, producing duplicate rows that ovn-nbctl rejects with
"Multiple logical routers named '...'. Use a UUID."
Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key
in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile,
ReconcileFromKV, and the two retrofit passes — others Subscribe and rely
on the vpcd-workers queue group for runtime events. MaxAge=60s on the
bucket caps recovery time when a leader crashes mid-reconcile.
Falls through (elected=true) when KV is unavailable so first-boot does
not deadlock on a missing JetStream.
mulga-siv-29
* fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30)
Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s
because the lb-agent inside the ALB VM has no return path to AWSGW.
The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP
for single-node and deliberately skips adding the /32 mgmt host route — the
WAN-return-path concern in the comment is real but applies only to
internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that
gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when
the agent dials advertiseIP the reply targets the VM's private subnet IP from
the host's WAN with no route back.
Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the
existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and
falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only
internal LBs need the /32 route, so internet-facing single-node still gets
no bootcmd and the host-initiated WAN path stays clean.
Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the
fallback has the inputs it needs.
* fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31)
The test was calling real SetupComputeNode("chassis-test", ...) which shelled
out via sudoCommand to set external_ids:system-id=chassis-test on the host's
live OVS. On a dev box with OVS installed this overwrote the persistent UUID
(owned by openvswitch-switch and re-applied only on reboot), so vpcd's
discoverChassis filtered out the legitimate SBDB chassis row as 'stale local'
and crashlooped — taking out all VPC ops including ALB ENI provisioning.
Bind sudoCommand to a var so tests can swap it. Stub it in the test to return
exec.Command("/bin/false") and assert SetupComputeNode returns an error. The
test's intent (signature + error-path coverage) is preserved; nothing on the
host is touched.
* chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32)
Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8
DORA timeouts correlate with concurrent Acquire calls on the same
bridge. Tracks per-bridge in-flight count and wallclock duration of each
Acquire; logs WARN on failure with both fields, DEBUG on success.
No behavioural change.
* ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32)
run-multinode-e2e.sh dumps node journals via its own EXIT trap, but
run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts
fail the cluster is destroyed in Teardown without surfacing any daemon
logs, blocking root-cause analysis.
Add a workflow step that runs after the test scripts but before
Teardown when a previous step failed, ssh'ing to each node and pulling
the last 200 lines of every spinifex-* unit. Best-effort: skips
gracefully when inventory or nodes are unreachable.
Diagnostic-only; no behavioural change to the test path.
* fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32)
Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when
dhcpcd-base is installed. It binds br-wan and races vpcd's in-process
nclient4 client for upstream OFFERs, draining the lab DHCP pool with
duplicate leases and causing intermittent DORA timeouts in the e2e
public-IP path (worst on cell-8 real-multi/dhcp/debian-13).
The ISO installer already masks the unit at install time
(cmd/installer/install/install.go:316). The binary install path used by
e2e nightly missed it, so debian hosts installed via setup.sh ran with
both DHCP clients live. Mirror the ISO mask here.
Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable
dhcpcd.service on Noble).
* ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32)
Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the
dhcpcd.service journal alongside the spinifex-* units, plus
systemctl is-enabled / is-active and the DHCP client port (UDP 68)
listener so we can see directly whether dhcpcd is running and bound
to br-wan when an Acquire times out.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
|
||
|
Julian Sommer
|
94735e7d19 | chore: interface compliance comp time check |