mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
Josh Siviour
7bbd99c792
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
e2e-analyze
Post-test failure-clustering action for the Go E2E suites.
What it does
Runs after go-junit-report has converted each suite's stdout into
junit-<suite>.xml. For every JUnit file matched by junit-glob:
- Parses each
<testcase>with a<failure>/<error>child. - Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs, IPs, timestamps, request-ids, source line numbers) to produce a stable signature per failure.
- Picks the earliest non-cascade failure per suite as the likely
root cause. "Cascade" = failure message matches a fixture-dependency
marker (
must populate fix.,Should NOT be empty,Expected value not to be nil). When every failure is a cascade (whole suite tripped on one upstream gap), the earliest cascade is the fallback root. - Buckets the remaining failures by signature and renders a markdown
report. The report is appended to
$GITHUB_STEP_SUMMARYand also written to<log-dir>/analysis.mdso it travels with the uploaded artifact bundle.
The action never fails the job — exit-code semantics stay with the existing "Fail job if any suite failed" gate.
Inputs
| Input | Required | Description |
|---|---|---|
junit-glob |
yes | Shell glob for the JUnit XML files (e.g. ${ARTIFACT_DIR}/junit-*.xml). |
log-dir |
yes | Directory the rendered report is written into (analysis.md). |
title |
no | Heading at the top of the summary section. Default E2E failure analysis. |
Outputs
None. The summary is appended to $GITHUB_STEP_SUMMARY; the same
content lands at <log-dir>/analysis.md.
Local invocation
Outside Actions, the analyser prints to stdout instead of appending to a summary file:
cd spinifex/.github/actions/e2e-analyze
go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
Tests
Golden-file tests live in testdata/. Each subdir holds a scenario:
| Scenario | Source |
|---|---|
cascade-vpcs |
Synthetic: one Phase 5 root + 15 cascades |
single-failure |
Real artifact from run 26037892507 |
no-failures |
Synthetic: clean run |
Run normally:
go test ./.github/actions/e2e-analyze/
When the report format changes intentionally, regenerate the snapshots:
go test ./.github/actions/e2e-analyze/ -update
Per-failure bundles (Stage 2)
After parsing, the action materialises a bundle directory per failure
under <log-dir>/analysis/NN-<TestName>/. The report links to it from
each entry so a reader on the Actions page can click straight through
to the relevant log slice.
Each bundle contains:
| File | Source |
|---|---|
failure.txt |
Synthesised summary: test, start, duration, file-site, error line. |
journal.log |
spinifex-journal.log sliced to [StartAt − 5s, StartAt + Duration + 5s]. |
test.log |
test-<suite>.log block between === RUN <name> and the matching --- FAIL. |
<other files> |
Anything the harness already dropped into <log-dir>/<TestName>/ at run-time. |
The journal slice relies on the workflow running
journalctl --output=short-iso so timestamps are parseable. Lines
without a parseable timestamp inherit the previous line's
in-window/out-of-window classification (so multi-line records and stack
traces stay contiguous).
Roadmap
- Stage 1 (shipped): first-failure surfacing + cascade clustering.
- Stage 2 (this update): time-window correlation — per-failure bundle with sliced journal + test-log + any per-test artifacts.
- Stage 3: rule-based root-cause classifier driven by
spinifex/tests/e2e/.failure-rules.yaml.
See docs/development/improvements/e2e-go-failure-analysis.md in the
parent mulga repo.