mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
v1.12.1
155 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 joshsiv-mulga
|
b147e2b869 |
ci: make Setup Go Workspace idempotent on ci-single runner
The publish-eks-ami matrix fans out four legs on the single persistent ci-single runner, all reusing one workspace root. go work init is create-only and checkout never scrubs the root go.work, so legs after the first fail with 'go.work already exists'. Remove any stale go.work before init so each leg re-inits cleanly. |
||
|
joshsiv-mulga
|
2b44de27bc |
ci: publish eks-node-gpu + ecs-agent-gpu AMIs
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish scripts are already manifest-driven and Ubuntu-aware; only the workflow lacked the GPU legs. Refactor the single-job body into a matrix over the four image manifests, resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs can be skipped when only the CPU AMIs need republishing. GPU images do not fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's on-disk _temp; legs run serially (max-parallel 1) on the single ci runner. |
||
|
Josh Siviour
|
1e1d2f3c6d |
ci: GPU e2e workflow on ci-gpu self-hosted runner (#518)
* ci: GPU e2e workflow on ci-gpu self-hosted runner Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent, hand-bootstrapped wattle shared host, driven over SSH by a banksia runner labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner installed on the shared box. - runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress false (single A1000 = single-flight, queue not stomp) - workflow_dispatch + GPU-path filter (not every push) - manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU testing blocks CI - if: always() cleanup terminates launched GPU instances; never resets the box (no setup-ovn.sh/dev-reset/cluster-reset) - drives the control plane with spx + the already-configured aws spinifex profile directly (no wrapper scripts, nothing to install on wattle) * ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs The suite is self-guarding — each test skips when its AMI is absent. The hard-fail on the base ubuntu GPU AMI blocked the whole run even when the ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and fail only when none are present. * ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported the self-guarding suite skipped every test, passing as a false green. Make the raw-EC2 GPU passthrough test the always-on baseline. - Add an import-if-absent setup step that pulls the prebuilt base GPU image from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's persistent loop-fs. Runs spx under sg spinifex (local storage write). - Preflight now hard-requires the base AMI (fail, not skip) so a silent import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn. - Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and launch-gpu.sh's nvidia default now use the same canonical name, so a locally built+imported image also satisfies the suite. * ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches spx admin images import --name selects the catalog entry but names the resulting AMI generically (ami-<distro>-<version>-<arch>), so the import landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same flag build-system-image.sh uses) and poll briefly for availability. * fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout Auto-assigned public IPs were never carried through the network reconciler. applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no gatewayLRP->guest convergence. Their dnat_and_snat was created once at RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the guest-port hop never converged: inbound DNATs at the gateway but never reaches the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK). floatingIPSpecs now folds every auto-assigned/ELB public IP into the same AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user EIP wins), making the datapath self-healing across reconcile ticks and deploys. qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB, floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers. * fix(vm): scale QMP handshake command deadline for VFIO launches The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU guest sends its QMP greeting before the RAM pin completes, then cannot service qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and the launch cleanup SIGKILLed a QEMU that would have come up. Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s) for qmp_capabilities. Other callers keep the 30s default. * test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle |
||
|
Josh Siviour
|
e78d0dbac9 |
fix(ci): upload full tofu apply/destroy log as an artifact after teardown (#489)
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/ destroy output to that file, but the Teardown step (which runs the destroy) always ran AFTER the existing upload-artifact step, so the full log — including destroy-time failures — was silently discarded on every run. Add an if: always() upload-artifact step after Teardown in each tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml) that uploads $TOFU_LOG directly. Since quiet_run appends both apply and destroy output to the same file, one upload after Teardown now captures everything, recoverable even when destroy itself fails. Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe, which will consume this artifact. |
||
|
Julian Sommer
|
47604f43dc | chore: go 1.26.5 | ||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
joshsiv-mulga
|
53a87fad8c |
ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. |
||
|
joshsiv-mulga
|
ee5c467a40 |
ci: publish ecs-agent AMI alongside eks-node
Add a second build+publish step to the EKS AMI workflow so the same manual run also builds the ecs-agent system AMI and uploads its qcow2 + .sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup; renames the workflow to cover both node images. |
||
|
Julian Sommer
|
485cca941f | feat: IAM inline role policies (Put/Get/DeleteRolePolicy + real ListRolePolicies) (#429) | ||
|
Julian Sommer
|
446f93ae8c | chore: rename ubuntu-latest to ubuntu-26.04 | ||
|
Julian Sommer
|
8a94886b13 | feat: cloud-init EC2 datasource (#411) | ||
|
dependabot[bot]
|
79e881f1c8 | chore: bump actions/checkout from 6 to 7 in the github-actions group (#405) | ||
|
Julian Sommer
|
51706938e8 | chore: use ubuntu-26.04 runners | ||
|
Josh Siviour
|
41dc53b7f8 |
fix(ci): prevent stale suite-status from false-failing e2e re-runs (#387)
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a re-run on the same self-hosted runner reused a prior attempt's append-only suite-status.txt. The 'Fail job if any suite failed' gate's -s check then failed the job even when the surviving attempt passed with rc=0. Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt steps.e2e.outputs.rc (fail-closed: an empty rc still fails). |
||
|
Julian Sommer
|
28c6e8ccff | chore: bump CI time | ||
|
Julian Sommer
|
eced3f733a | feat: split IAM e2e (#372) | ||
|
Julian Sommer
|
058da3eff4 | fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) | ||
|
Josh Siviour
|
17c83cb3ad |
fix: build microVM rootfs via buildx to stop overlay2 export leak (#358)
docker export of a container leaks its overlay mount on Docker's overlay2 driver — the mount survives docker rm, so the layer dirs are never reaped except by a daemon restart. The lb-agent microVM build ran docker export on every e2e job across ~9 runners, accumulating ~69G of orphaned overlay2 layers on the CI host (disk hit 94%). Build the rootfs with `docker buildx --output type=local` from a small generated Dockerfile instead, which writes the image filesystem to a directory and releases all mounts — nothing leaks. Keep the podman path (its store is unaffected). Tighten the e2e cleanup image prune to dangling-only so the alpine base stays cached instead of re-pulling. |
||
|
Julian Sommer
|
c6c26c04e2 | chore: trim workflow comments | ||
|
Julian Sommer
|
2df35d3822 | chore: remove reboot e2e | ||
|
Josh Siviour
|
d076a434d4 |
Feat/e2e targeted suites (#352)
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
|
||
|
Josh Siviour
|
90369371ee |
Feat/nightly e2e fix (#330)
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for the composite actions and pulls /run/spinifex/console-*.log so LB microVM (lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal) stay on bash — no Go suite exists. * ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the cell-17 step's tee'd test-tofu.log flows through the same go-junit-report -> e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR, analyze title, and artifact name with the matrix cells so CI-RCA discovers its bundle uniformly. * ci(e2e-nightly): wire host journal + VM consoles into RCA bundles Collect now emits the canonical spinifex-journal.log (concat of all nodes) that the e2e-analyze bundler slices into each failure's journal.log, so elbv2-service/vpcd/daemon lines in the failure window travel with the bundle (previously the journal uploaded as spinifex-journal-node*.log, a name the bundler never reads). Rename the console capture to vm-console-node* and note it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB, cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots. * ci(e2e): self-heal runner /dev + --one-file-system on workspace clean Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates /dev/null if a prior build clobbered it, and uses rm --one-file-system so the clean can't descend into a still-mounted devtmpfs and delete host device nodes. Guards both the matrix cells and the bare-metal job against the host-wide /dev/null wipe that fails go build with 'open /dev/null: no such file or directory'. |
||
|
Josh Siviour
|
7bcd365549 |
ci(eks-ami): build on self-hosted ci-single runner (#326)
* ci(eks-ami): build on self-hosted ci-single-1 runner GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network: passt fails (PID file denied under its AppArmor confine / privilege drop) and the SLIRP fallback has no outbound, so virt-customize --network dies. The self-hosted ci-single-1 runner has /dev/kvm and real networking, so the existing libguestfs build works unmodified. * fix(build-system-image): add --no-cache, force fresh build on publish The <10min cached-raw skip avoids duplicate work for concurrent e2e jobs, but on the persistent publish runner it could republish a stale image from a prior or concurrent run. Add --no-cache to bypass the skip and have publish-system-image.sh pass it on every build. * ci(eks-ami): target ci-single label (no ci-single-1 label exists) The self-hosted runners are named ci-single/-2/-3 but all share the single label 'ci-single'; runs-on matches labels, not names, so ci-single-1 never schedules. * ci(eks-ami): disable Go cache on self-hosted runner setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the Actions backend on every run, taking ~2.5min vs the 36s build. The self-hosted runner already keeps the cache on local disk, so cache:false. |
||
|
Josh Siviour
|
4ef19fc488 | feat(ui): EKS console (#310) | ||
|
Josh Siviour
|
5ac8dc3fb9 |
fix(network): round-robin external pool cursor — stop instant EIP reuse (recycled EIP unreachable, EKS CI) (#320)
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit
|
||
|
Josh Siviour
|
467e4de715 |
feat(eks): publish eks-node AMI to R2 + spx catalogue entry (#313)
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry Add a spinifex-eks-node catalogue entry so operators install the EKS node image with a single command: spx admin images import --name spinifex-eks-node The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its sha256, and stamps spinifex:managed-by=eks — the tag the control-plane and node-group launchers resolve the boot AMI by, so no extra config is needed. Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to convert the built raw to a compressed qcow2, emit a checksum sidecar, and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via the S3 API. * ci(eks): add manual workflow to build + publish eks-node AMI to R2 Build the eks-node AMI on the runner (libguestfs) and publish via publish-system-image.sh using the existing CF_R2_* secrets. Manual workflow_dispatch with a dry-run input. Fix the publish script's default bucket to spinifex-iso (the bucket iso.mulgadc.com is served from). |
||
|
Josh Siviour
|
a1f3fcc531 |
ci(e2e): exclude eks suite from multinode job (#312)
EKS create-cluster fails deterministically on the multinode job with ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream stream and create-cluster needs an elected leader, but multinode JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg). EKS is validated single-node only at this stage. Drop eks from the multinode build+run suite filters (alongside the existing reboot/single exclusions) so the multinode job reflects what's actually supported. Single-node continues to run the full eks suite. |
||
|
Josh Siviour
|
a4817b1afc |
ci: run baremetal E2E nightly + hard-gate releases on it (#306)
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input) so the nightly and release pipelines can invoke the real PXE-reimaged hardware suite, which previously only ran on manual dispatch. Nightly: add a baremetal job (conceptual cell 20) that runs every weeknight on the existing cron, or on dispatch when 20 is selected. It tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix (different runner, provisioning, 120m timeout) and shares the e2e-baremetal-cluster concurrency group, so all baremetal runs serialise on the single physical node. Release: a release builds and uploads an ISO to R2 but never booted that ISO/squashfs path on hardware. Add a baremetal job (needs: release) and hard-gate on it: promotion moves to a dedicated promote job needs:[verify, baremetal] (with a matching mark-failed job on failure), and the iso job's upload gate now waits for both verify and baremetal to conclude success before pushing to R2. Bump the iso job timeout to 180m since it now waits on the parallel ~2h baremetal run. |
||
|
Josh Siviour
|
44d0548fd4 |
ci: make microVM depmod best-effort, drop apt kmod install (#308)
build-microvm-image required depmod (kmod) as a mandatory tool to generate modules.dep. Self-hosted CI runners don't ship kmod and can't reliably apt-install it, so the inline 'apt-get install kmod' steps added to e2e/e2e-nightly/e2e-ddil were unreliable. modules.dep is an optimisation, not a boot requirement: the guest init's load_mod() (build/microvm/init.sh) falls back to insmod-by-path when modprobe can't resolve a module. So: - Drop depmod from the mandatory-tool preflight. - Run depmod only when present; still assert a non-empty modules.dep in that case to keep catching the trixie-kmod empty-dep bug. - Warn and skip when depmod is absent (guest uses insmod fallback). - Remove the apt-get kmod install steps from all three workflows. Gold-image bake still installs kmod (provision.sh), so production images get a proper modules.dep; only kmod-less CI runners take the fallback. |
||
|
Josh Siviour
|
8e0edef7c2 |
ci: install kmod (depmod) before microVM image build (e2e + nightly + ddil) (#307)
* ci(e2e): install kmod (depmod) before microVM image build #305 made depmod a hard requirement in build-microvm-image.sh, but the self-hosted ci-single/ci-multi runners don't ship the kmod package, so the Build Install Artifacts step aborts with 'required tool not found: depmod' before any E2E suite runs. Add an idempotent kmod install ahead of each Build Install Artifacts step (single/multi/eks jobs); the gold-image builder bakes kmod in too, so this no-ops once runners pick up the new image. * ci: install kmod (depmod) in nightly + ddil too e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make build-microvm-image) on the same self-hosted ci-single/ci-multi runners as e2e.yml, so they hit the identical 'depmod not found' abort. Add the same idempotent kmod guard before Build Install Artifacts in both. release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal runs on ci-baremetal which already has it (green today) — neither needs the guard. |
||
|
Josh Siviour
|
496424c41c |
test(eks): EKS control-plane E2E suite (Sprint 6g) (#304)
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
|
||
|
joshsiv-mulga
|
d8aa31e699 |
ci(baremetal): capture combined log + junit + analyze bundle on failure
Bare-metal e2e produced no artifact, so RCA fell back to truncated log-failed output. Tee bootstrap + e2e output to test-baremetal.log, generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and upload the bundle as e2e-baremetal-artifacts-<run_id>. |
||
|
joshsiv-mulga
|
f4ba115316 | ci(baremetal): revert e2e to single-node | ||
|
joshsiv-mulga
|
3bb90c9fd3 |
fix(ci): guard release Clean Workspace against leaked iso-builder devtmpfs
The iso job mounts a kernel-global devtmpfs under the workspace (.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks the bind; the verify job (same ci-single runner) then ran a bare 'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev. Detach any mount under the workspace before deleting and add --one-file-system so rm never crosses a mount boundary. Same backstop added to the iso job's own cleanup. |
||
|
joshsiv-mulga
|
da26b1db69 |
fix(ci): stop image-build /dev bind from wiping host /dev
A killed ubuntu system-image build can leave `mount --bind /dev` under its build dir. The e2e Clean Workspace step then ran `sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind, and deleted the host's /dev (incl. /dev/null), wedging the runner. - e2e.yml: umount anything under the workspace (deepest-first) before deleting, and pass `rm --one-file-system` so rm can never cross a mount boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace steps. - build-system-image.sh: recursive lazy unmount in cleanup() detaches the chroot binds even when busy, and mark the build subtree rprivate so the /dev bind can't propagate into the host namespace. |
||
|
tomnewton-mulga
|
6be3ccbb23 | fix: unify .iso and .iso.sha filenames | ||
|
tomnewton-mulga
|
158b3a25c5 | fix: updated release.yml for clean iso uploads | ||
|
joshsiv-mulga
|
af690b3299 |
test(elbv2): collect LB microVM guest console in CI + surface NLB probe failures
NLB target health stalls at 0/N in CI while the data plane works locally. The lb-agent's nginx engine activation, any nginx -t/reload error, and its active-probe results are logged to the LB microVM serial console — which the host journal never captures and the GetConsoleOutput API path reports as NotFound, so the failure is undiagnosable from CI artifacts. - e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and multi-node collect steps (files persist past VM teardown) into lb-console/. - lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable targets surface at the agent's default level on the captured console. |
||
|
Julian Sommer
|
c666c8146d | fix: e2e.yml | ||
|
Julian Sommer
|
044d35737e | feat: remove bash e2e for e2e-go | ||
|
Julian Sommer
|
9508aaacce | chore: bump go 1.26.4 | ||
|
joshsiv-mulga
|
7cd054cbf1 | chore(e2e): dump Logical_Router_Policy + Logical_Router_Static_Route in W5 diag | ||
|
joshsiv-mulga
|
ba7253e2c6 |
ci(e2e): fix Start Packet Capture EACCES + PID-tracking on tcpdump (siv-167)
Two bugs in
|
||
|
joshsiv-mulga
|
afd63ed967 |
ci(e2e): Phase 1 diag — tcpdump br-wan + bridge fdb + hairpin modes (siv-167)
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB single-host return-path drop class: e2e_single only: - New step 'Start Packet Capture' between Bootstrap and Run E2E Tests. SSHes to test VM and starts background tcpdump on br-wan capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap. PID stored at /tmp/spinifex-diag/tcpdump.pid. All three jobs (single, pseudo-multi, real-multi): - Collect Diagnostics: kill tcpdump (single only — PID file guarded) -INT for graceful flush, falls through to -KILL after 5s. - Adds journal-spinifex-full.log (no -n truncation) alongside the existing -n 10000 capped one. - Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood for every br-wan brif — needed to confirm/disconfirm the hairpin-loss hypothesis in the plan doc. - Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap (created by sudo tcpdump as root) is readable by tf-user. bash -n verified on all four rendered scripts. YAML parses. Zero new gold-image deps (tcpdump, bridge present in base image). |
||
|
joshsiv-mulga
|
3f4613c28b |
ci(e2e): capture OVN/OVS/journal diagnostics on every run
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
|
||
|
Josh Siviour
|
3415d9f3de |
fix(network): IGW route-table AWS parity + per-subnet egress gating (#270)
* fix(network): IGW default route AWS-parity via per-subnet LR policy
ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.
Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.
Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet
Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
route; EnsureSubnetEgress installs the per-subnet policy with the
resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics
* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry
handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
(mirror of the IGW DeleteRoute path) so the subscriber tears down both
the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
the subscriber can install the matching Logical_Router_Policy.
network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
Same nexthop and output port as the IGW priority sibling; lower priority
so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
validation, lookup, and slog tagging.
network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
has no route to egress NAT'd packets and private-subnet traffic is
dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.
Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.
* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate
In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.
DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).
* fix(routetable): publish per-subnet events for implicit main-RT membership
E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.
Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).
Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
for main RT's routes after deletes for departing RT
* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list
ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.
* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests
EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.
start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.
Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.
Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.
* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV
Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.
LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.
* fix(routes): serialize OVN egress wiring via request-reply
vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).
Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.
handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.
* Revert "fix(routes): serialize OVN egress wiring via request-reply"
This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.
* fix(igw): restore default static route in AttachIGW/DetachIGW
OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.
Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.
Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).
* feat(network): per-subnet egress drop gating for AWS RT parity
VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.
Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.
Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
attached VPCs whose effective RT lacks 0.0.0.0/0.
Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.
* fix(igw): fan out subnet egress gate decisions on IGW attach/detach
The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.
Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.
Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).
* fix(igw): skip gate fan-out on attach to avoid bootstrap race
AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).
Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.
Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.
* fix(vpc): prevent duplicate main route table from bootstrap race
Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.
CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).
Two-part fix:
1. createMainRouteTable scans rtbKV for an existing IsMain=true record
on the same VPC before Put-ing a new one. Race window narrows from
"always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
between scan and Put.
2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
tiebreak when multiple IsMain=true records exist: more routes >
oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
that slips through (cross-AZ restore, manual KV edit, residual race)
still produces consistent egress gate decisions instead of flapping
on KV scan order.
* fix(policy): atomic ACL replacement for SG mutations
Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).
Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
|
||
|
Josh Siviour
|
fd3dc6b680 |
feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255)
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
|
||
|
dependabot[bot]
|
ef097bf4ab | chore: bump actions/setup-go from 5 to 6 in the github-actions group (#251) | ||
|
joshsiv-mulga
|
80ab3fb37a |
ci: prune rootless docker + trap container leaks in microvm build
Self-hosted runner accumulated 18GB in overlay2 from leaked container layers. Two causes: build-microvm-image.sh only removed the alpine container on the success path, and no workflow step pruned docker between runs. - scripts/build-microvm-image.sh: container id now removed via the existing EXIT trap, so failures, signals, and runner kills no longer leak the writable layer. - .github/workflows/e2e-go.yml: prune containers / dangling images / buildx cache before each Build Install Artifacts step, in all three e2e jobs (single, multinode, reboot). |
||
|
Josh Siviour
|
bedc35344f |
feat(e2e/multinode) (#242)
* feat(e2e/harness): multinode cluster wait helpers + AWSClientForGateway (Stage A)
Stage A of mulga-siv-90. Compile-only; no tests wired yet.
multinode_cluster.go:
- Cluster.WaitNATSPeers polls /routez per node, dedupes by remote_id.
want=2 on healthy 3-node, =1 after a node-down event.
- Cluster.WaitPredastoreHealthy / WaitGatewayHealthy probe HTTPS per node
via insecureHTTPClient (multinode certs are self-signed across peers).
- Cluster.WaitDaemonReady runs DescribeInstanceTypes through every
node's gateway, not just env.ServiceIPs[0] — catches a daemon that's
only registered on one node.
multinode_aws_via.go:
- AWSClientForGateway clones *Env, swaps ServiceIPs to a single node's
Addr, returns a fresh AWSClient. Lets phases 6/7 issue describe/
stop/start across every node in turn (bash aws_via equivalent).
Init/Join helpers deferred to the pseudo-multinode bead (mulga-siv-91)
where they belong — real-multinode bootstrap is owned by tofu +
bootstrap-install.sh before the Go suite starts.
* feat(e2e/multinode): TestMultinodePreflight + TestMultinodeClusterHealth (Stage B)
Stage B of mulga-siv-90. First runnable Test* against a 3-node cluster.
multinode/main_test.go:
- Package-scoped singleton fixture, mirrors single/main_test.go.
- requireMultiNodeFixture skips when SPINIFEX_E2E unset or mode != multinode.
- Fixture = {Env, AWS, Harness, Cluster, Artifacts, TmpDir} — adds Cluster
to the single-node shape via harness.ClusterFromEnv (reads SPINIFEX_NODES
+ SPINIFEX_SSH_USER/KEY).
- TestMain drains harness.Fixture.Close + removes TmpDir at process exit.
multinode/helpers_test.go:
- needAZ / needInstanceTypeArch / needAMI shims (memoized via harness).
- readyNodeCount substring counter — matches bash `grep -c "Ready"`.
multinode/tests_test.go:
- Top-level Test* wrappers (TestMultinodePreflight, TestMultinodeClusterHealth).
multinode/preflight_test.go (phase 1):
- /dev/kvm writable check (no /dev/kvm = test environment is wrong, fail fast).
- peer_ssh hostname on every remote node. Skips node[0] to match bash —
the runner's loopback SSHd is not part of the contract.
multinode/cluster_health_test.go (phase 2):
- Cluster.WaitNATSPeers >= 2 per node.
- Cluster.WaitPredastoreHealthy / WaitGatewayHealthy across nodes.
- Cluster.WaitDaemonReady (DescribeInstanceTypes per gateway).
- spx get nodes shows N Ready, where N = cluster size.
- spx get vms returns (length only — content may be empty).
* feat(e2e/multinode): TestMultinodeInstanceDistribution + GuestSSH (Stage C)
Stage C of mulga-siv-90. Phase 3 (distribution) + phase 4 (guest SSH).
harness/multinode_distribution.go:
- InstanceHostingNode: SSH each peer, ps grep qemu-system | <id>.
Reuses NewPeerSSH; returns nil on no match (caller decides fatality).
- CountInstancesPerNode + AssertSpreadAcrossNodes: fan-out + minNodes
gate. Bash phase 3 warns-on-stacking; Go matches by accepting
minNodes=1 + caller-side log.
harness/multinode_guest_ssh.go:
- GuestSSHPort polls hosting node's qemu cmdline for hostfwd=tcp:...
reusing the existing hostfwdRE from ec2helpers.go (single regex, two
capture groups: host + port).
- GuestSSHEndpoint prefers PublicIpAddress, falls back to hosting-node
+ hostfwd. Mirrors the phase 4 conditional.
- GuestSSHReady polls `echo ready` via SSH with bash-equivalent -o flags.
multinode/helpers_test.go:
- needKeyPair (EnsureKeyPair shim).
- needInstanceTrio: pkg-level sync.Once launches 3 nano instances on
default VPC, registers terminate-on-process-exit. Phase 4+ reuse the
same trio without re-launch.
multinode/distribution_test.go (phase 3):
- launch trio + assert >= 1 node hosting (log warn on stacking).
- spx get vms contains every trio ID.
multinode/guest_ssh_test.go (phase 4):
- For each trio instance: resolve endpoint, poll SSH ready, assert id
reports ec2-user, log lsblk line count.
multinode/tests_test.go: TestMultinodeInstanceDistribution + TestMultinodeGuestSSH.
* feat(e2e/multinode): TestMultinode{Volume,CrossNodeGateway,CrossNodeOps} (Stage D)
Stage D of mulga-siv-90. Phases 5/6/7 — pure AWS API exercise (no
node-state churn).
volume_test.go (phase 5):
- Create 10GiB volume in discovered AZ, attach to trio[0] at /dev/sdf,
detach, delete. Inline lifecycle — DeleteVolume in-test would race a
registered Ensure* cleanup, surface as InvalidVolume.NotFound.
gateway_test.go (phase 6):
- Baseline DescribeInstances via the local gateway, then fan out
AWSClientForGateway per non-local node. Require.Equal across all —
cross-node fan-out regression surfaces immediately.
crossnode_ops_test.go (phase 7):
- Find trio[0]'s hosting node, stop via a different gateway, start via
a third (or same other on 2-node fallback). Returns to running so
later Test*s (phase 4 SSH, etc.) don't cascade-fail. otherTwoGateways
mirrors the bash fallback chain.
* feat(e2e/multinode): TestMultinodeNode{Failure,Recovery} (Stage E)
Stage E of mulga-siv-90. Phase 8 (failure) + phase 9 (recovery) +
diagnostic primitives.
harness/multinode_node_lifecycle.go:
- StopNode SSHes `sudo systemctl stop spinifex.target`. Non-fatal on
exit error (bash matches — cluster shutdown races the SSH conn).
- StartNode mirrors with sudo systemctl start.
- WaitNodeServiceReady polls a single node's HTTPS gateway. Distinct
from cluster-wide WaitGatewayHealthy so a recovering node can be
tracked without re-checking peers.
harness/multinode_diag.go:
- DumpAllNodeLogs fans out journalctl -u 'spinifex-*' per node into
artifactsDir/journal-<node>.log. Tolerates per-node SSH failure.
- SpxGetNodesAcrossCluster strips trailing whitespace off SpxGetNodes.
multinode/node_failure_test.go (phase 8):
- Stop node2, register t.Cleanup restart safety net.
- WaitNATSPeers want=1 (degraded), DescribeInstanceTypes via node1+node3,
DescribeInstances still returns the trio.
multinode/node_recovery_test.go (phase 9):
- StartNode node2 (idempotent — phase 9 in isolation still works).
- WaitNATSPeers want=2, WaitNodeServiceReady, spx get nodes 3 Ready,
DescribeInstanceTypes via the recovered gateway.
* feat(e2e/multinode): TestMultinodeVPCNetworking (Stage F)
Ports phase 10 from run-multinode-e2e.sh. Stands up a fresh non-default
VPC + subnet (10.200.0.0/16 + 10.200.1.0/24), launches 3 instances into
the subnet, then asserts:
- PrivateIpAddress is non-empty and matches 10.200.1.0/24
- All 3 IPs are unique
- SubnetId / VpcId on the instance match the freshly-created ids
- IPs persist across stop -> start cycles
Independent of the package-level trio (which uses the default VPC) —
this phase needs full control over subnet allocation so resources are
created and cleaned up inline via t.Cleanup (LIFO: instances ->
subnet -> VPC).
* feat(e2e/multinode): TestMultinodePlacementAndNAT (Stage G)
Ports phase 11 from run-multinode-e2e.sh — the largest single phase
(~340 bash lines covering spread placement + NAT Gateway egress). Eight
sub-steps preserved with matching harness.Step markers so the Go run
cross-references cleanly with the bash log:
Step 1: VPC + pub/priv subnets + IGW + main RT route + 2 SGs
(bastion: tcp/22 anywhere; private: tcp/22 from bastion-sg,
icmp from VPC)
Step 2: bastion launch, wait SSH, scp key
Step 3: spread placement group + 3 private instances via the group
Step 4: validate spread (>=2 unique hosting nodes; >=3 = full PASS)
Step 5: pre-NAT baseline — confirm NO internet from private VMs
Step 6: allocate EIP + create NAT GW + private RT + 0.0.0.0/0 -> NGW
Step 7: post-NAT — every private VM reaches 8.8.8.8 via NAT GW
Step 8: inline teardown of NAT GW + private RT + EIP
Each resource is owned by the test (no Ensure* memoization) so a failure
mid-flight tears the full VPC graph down via t.Cleanup in LIFO order,
matching the bash teardown sequence. NAT GW state polling is duplicated
locally (waitForNATGatewayStateFatal/Best); the equivalents in
single/natgw_test.go are unexported. Promote to harness/poll.go once a
third consumer lands.
* ci(e2e-go): add multi-node Go E2E job + cert/lb on 3-node cluster (Stage H)
Adds e2e_go_multinode as a parallel job alongside e2e_go_single on the
e2e-go workflow. The single-node job is untouched apart from extending
its SUITES filter to drop the multinode suite (which only runs on the
ci-multi cluster).
What the new job does:
- Provisions a 3-node tofu cluster via the same build-install-artifacts
+ bootstrap-install path as e2e_real_multi (bash).
- Builds multinode.test / cert.test / lb.test on the runner, ships
them to node1.
- Runs each suite with SPINIFEX_MODE=multinode and the cluster csv
in SPINIFEX_NODES / SPINIFEX_NODE_IPS. cert / lb branch internally
on env.Mode so the same binaries cover both single + multi.
- Pulls per-node journals (spinifex-journal-node1..N.log) for the
e2e-analyze action so a node2/3 failure isn't blind.
- Same JUnit + analyze + artifact-upload + status pipeline as the
single-node job.
Other moves:
- Default SUITES now includes e2e-multinode so push triggers fan
out across single + multi automatically. e2e_go_reboot stays
explicit-opt-in (unchanged).
- tests/e2e/Makefile: e2e-multinode now sets MODE=multinode so
`make e2e-multinode` doesn't fall back to single (which would
skip every TestMultinode* on the SPINIFEX_MODE check).
- tests/e2e/TEST_COVERAGE.md: cross-reference the Go port at the
head of the Real Multi-Node section. The bash driver stays in
place as executable spec until the Go run is soak-tested.
- e2e_go_status now needs e2e_go_multinode; multi=skipped counts
as success so a single-only dispatch still posts green.
* fix(e2e/multinode): NATS routez via peer_ssh + StartNode timeout + spx best-effort
CI run 26161792146 surfaced three regressions in the freshly-landed
multinode suite:
1. WaitNATSPeers dialled node.Addr:8222 directly. NATS monitor binds
127.0.0.1:8222 only (cmd/spinifex/cmd/templates/nats.conf:65), so
every routez poll returned "connection refused". Switched to
peer_ssh + curl http://127.0.0.1:8222/routez per node — matches
the bash run-multinode-e2e.sh phase 2 pattern.
2. StartNode / StopNode capped ssh at a 30s context. spinifex.target
has 6+ dependent units (predastore, NATS, awsgw, vpcd, daemon, ui)
and a clean systemctl start runs 60-90s; the 30s budget was
killing ssh mid-command ("signal: killed"). Bump StartNode to 5min
and StopNode to 3min.
3. SpxGetVMs is best-effort in bash (`spx get vms --timeout 5s
2>/dev/null` with no exit-code check) because the CLI's NATS dial
can race the cluster join shortly after bootstrap. The Go port
fatal'd on non-zero exit and broke TestMultinodeClusterHealth +
TestMultinodeInstanceDistribution downstream. Added
harness.SpxRunBestEffort and rewired both call sites.
* fix(e2e/multinode): spx CLI lenience + trio capacity retry + diag
Run 26163994815 cascaded from two issues that bash silently tolerated.
The Go port hard-failed where bash counts the failure and continues, so
3+ tests cascaded off transient first-failures.
- spx get nodes: cluster_health + node_recovery now downgrade <Ready
count to WARN with SpxRunBestEffort, matching bash phase 2
fail_test counter and phase 9 `|| echo ""`. Gateway+daemon health
checks above already prove end-to-end NATS routing.
- needInstanceTrio: retry RunInstances up to 6× on
InsufficientInstanceCapacity (10s backoff). Daemon resourceMgr can
briefly report 0 capacity right after node bootstrap before its
first inventory scan completes — AWS itself surfaces this
transiently. Stagger 2s between launches to match bash phase 3.
- CI workflow: new "Pre-flight diagnostics on node1" step dumps
groups/perms/spx output/systemctl status/journal tail to
preflight-diag.log so next run's artifacts surface what tf-user
sees before the test binary starts.
* fix(e2e): strip SPINIFEX_* from spx child env
Root cause of `spx get nodes` returning "nats: no servers available for
connection" in the multinode Go suite (mulga-siv-90 runs 26163994815,
26195522455): spx CLI calls viper.AutomaticEnv() with prefix "SPINIFEX"
inside config.LoadConfig. The CI workflow exports SPINIFEX_NODES (csv
of cluster WAN IPs) so harness.ClusterFromEnv can build the *Cluster.
viper picks SPINIFEX_NODES up as the `nodes` config key and silently
clobbers the loaded `[nodes.*]` map → nodeConfig.NATS.Host = "" →
nats.Connect("") → ErrNoServers.
Bash never set SPINIFEX_NODES so the bash port was unaffected. Direct
ssh `spx get nodes` from the runner also worked because the env was
clean. Only the test-binary-spawned spx hit the collision.
Fix: SpxRun / SpxRunBestEffort build a sanitized child env that
preserves the small SPINIFEX_* set spx's root command explicitly
BindEnvs (config path, AWS keys, host, base-dir, NATS dial overrides)
and drops everything else.
* fix(vpcd): make handleAddNAT idempotent (mulga-siv-124)
PR #232 (commit
|