* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
* ci(eks-ami): build on self-hosted ci-single-1 runner
GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network:
passt fails (PID file denied under its AppArmor confine / privilege drop)
and the SLIRP fallback has no outbound, so virt-customize --network dies.
The self-hosted ci-single-1 runner has /dev/kvm and real networking, so
the existing libguestfs build works unmodified.
* fix(build-system-image): add --no-cache, force fresh build on publish
The <10min cached-raw skip avoids duplicate work for concurrent e2e
jobs, but on the persistent publish runner it could republish a stale
image from a prior or concurrent run. Add --no-cache to bypass the skip
and have publish-system-image.sh pass it on every build.
* ci(eks-ami): target ci-single label (no ci-single-1 label exists)
The self-hosted runners are named ci-single/-2/-3 but all share the
single label 'ci-single'; runs-on matches labels, not names, so
ci-single-1 never schedules.
* ci(eks-ami): disable Go cache on self-hosted runner
setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the
Actions backend on every run, taking ~2.5min vs the 36s build. The
self-hosted runner already keeps the cache on local disk, so cache:false.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
Two bugs in afd63ed9:
1. `sudo mkdir` makes /tmp/spinifex-diag root-owned 0755. The
subsequent `> /tmp/spinifex-diag/tcpdump.log` is performed by
the calling tf-user shell BEFORE sudo gets the command, failing
with Permission denied. action shell runs with `set -e` so the
step aborts.
2. `sudo nohup ... &; echo $!` captures the wrong PID — `$!` is the
backgrounded sudo, which may exit when its target dies, leaving
stale tcpdump.pid.
Fixes:
- chmod 0777 on /tmp/spinifex-diag after creating it (next steps'
redirects from tf-user shell now work).
- Wrap the tcpdump launch in `sudo bash -c '...'` so backgrounding
and `$!` happen inside the sudo'd shell — `$!` is the actual
tcpdump PID (nohup execs, sudo bash holds it). Captures stdin
to /dev/null, stdout/stderr to /tmp/spinifex-diag/tcpdump.log.
- sleep 2 (was 1) — gives tcpdump time to fully init pcap header.
- Hard fail (exit 1) if PID is missing or process dead, so the
step's failure is visible instead of a silent green.
bash -n verified.
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB
single-host return-path drop class:
e2e_single only:
- New step 'Start Packet Capture' between Bootstrap and Run E2E
Tests. SSHes to test VM and starts background tcpdump on br-wan
capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap.
PID stored at /tmp/spinifex-diag/tcpdump.pid.
All three jobs (single, pseudo-multi, real-multi):
- Collect Diagnostics: kill tcpdump (single only — PID file
guarded) -INT for graceful flush, falls through to -KILL after 5s.
- Adds journal-spinifex-full.log (no -n truncation) alongside
the existing -n 10000 capped one.
- Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood
for every br-wan brif — needed to confirm/disconfirm the
hairpin-loss hypothesis in the plan doc.
- Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap
(created by sudo tcpdump as root) is readable by tf-user.
bash -n verified on all four rendered scripts. YAML parses.
Zero new gold-image deps (tcpdump, bridge present in base image).
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
* fix(network): IGW default route AWS-parity via per-subnet LR policy
ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.
Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.
Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet
Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
route; EnsureSubnetEgress installs the per-subnet policy with the
resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics
* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry
handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
(mirror of the IGW DeleteRoute path) so the subscriber tears down both
the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
the subscriber can install the matching Logical_Router_Policy.
network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
Same nexthop and output port as the IGW priority sibling; lower priority
so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
validation, lookup, and slog tagging.
network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
has no route to egress NAT'd packets and private-subnet traffic is
dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.
Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.
* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate
In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.
DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).
* fix(routetable): publish per-subnet events for implicit main-RT membership
E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.
Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).
Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
for main RT's routes after deletes for departing RT
* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list
ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.
* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests
EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.
start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.
Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.
Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.
* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV
Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.
LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.
* fix(routes): serialize OVN egress wiring via request-reply
vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).
Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.
handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.
* Revert "fix(routes): serialize OVN egress wiring via request-reply"
This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.
* fix(igw): restore default static route in AttachIGW/DetachIGW
OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.
Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.
Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).
* feat(network): per-subnet egress drop gating for AWS RT parity
VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.
Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.
Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
attached VPCs whose effective RT lacks 0.0.0.0/0.
Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.
* fix(igw): fan out subnet egress gate decisions on IGW attach/detach
The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.
Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.
Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).
* fix(igw): skip gate fan-out on attach to avoid bootstrap race
AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).
Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.
Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.
* fix(vpc): prevent duplicate main route table from bootstrap race
Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.
CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).
Two-part fix:
1. createMainRouteTable scans rtbKV for an existing IsMain=true record
on the same VPC before Put-ing a new one. Race window narrows from
"always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
between scan and Put.
2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
tiebreak when multiple IsMain=true records exist: more routes >
oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
that slips through (cross-AZ restore, manual KV edit, residual race)
still produces consistent egress gate decisions instead of flapping
on KV scan order.
* fix(policy): atomic ACL replacement for SG mutations
Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).
Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
Julian Sommer