mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 03:11:46 +00:00
7bbd99c792
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
176 lines
4.5 KiB
Go
176 lines
4.5 KiB
Go
package main
|
|
|
|
// Stage 2: per-failure log slicers used by bundle.go to materialise time-window /
|
|
// testcase-window log excerpts next to each named failure.
|
|
|
|
import (
|
|
"bufio"
|
|
"io"
|
|
"os"
|
|
"regexp"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
// shortIsoTSRe matches the timestamp prefix produced by
|
|
// `journalctl --output=short-iso`. The offset format varies across
|
|
// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
|
|
// (no colon). Match either — the parser below picks the right layout.
|
|
var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
|
|
|
|
// shortIsoLayouts are tried in order; the first parse that succeeds wins.
|
|
var shortIsoLayouts = []string{
|
|
time.RFC3339, // 2026-05-19T12:32:30+10:00
|
|
"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
|
|
}
|
|
|
|
func parseShortIso(s string) (time.Time, error) {
|
|
var lastErr error
|
|
for _, layout := range shortIsoLayouts {
|
|
t, err := time.Parse(layout, s)
|
|
if err == nil {
|
|
return t, nil
|
|
}
|
|
lastErr = err
|
|
}
|
|
return time.Time{}, lastErr
|
|
}
|
|
|
|
// SliceJournal copies lines from src whose `short-iso` timestamp falls in
|
|
// [start, end] to dst. Lines without a parseable timestamp inherit the
|
|
// previous line's classification — that keeps multi-line journal records
|
|
// (stack traces, panics) intact rather than orphaning their continuation
|
|
// lines. Lines that precede the first parseable timestamp are dropped.
|
|
//
|
|
// Returns the number of lines written; missing src is not an error (the
|
|
// journal may be empty if collection failed).
|
|
func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
|
|
f, err := os.Open(src)
|
|
if err != nil {
|
|
if os.IsNotExist(err) {
|
|
return 0, nil
|
|
}
|
|
return 0, err
|
|
}
|
|
defer f.Close()
|
|
|
|
out, err := os.Create(dst)
|
|
if err != nil {
|
|
return 0, err
|
|
}
|
|
defer out.Close()
|
|
|
|
bw := bufio.NewWriter(out)
|
|
defer bw.Flush()
|
|
|
|
sc := bufio.NewScanner(f)
|
|
// Journal lines can be long (panic dumps); bump scanner buffer.
|
|
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
|
|
|
keep := false
|
|
written := 0
|
|
for sc.Scan() {
|
|
line := sc.Text()
|
|
if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
|
|
ts, err := parseShortIso(m[1])
|
|
if err != nil {
|
|
continue
|
|
}
|
|
keep = !ts.Before(start) && !ts.After(end)
|
|
}
|
|
if keep {
|
|
if _, err := bw.WriteString(line); err != nil {
|
|
return written, err
|
|
}
|
|
if err := bw.WriteByte('\n'); err != nil {
|
|
return written, err
|
|
}
|
|
written++
|
|
}
|
|
}
|
|
return written, sc.Err()
|
|
}
|
|
|
|
// runRePrefix and finishRePrefix bracket a single testcase's output in
|
|
// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
|
|
// markers, so slicing on the exact testcase name yields just that test's
|
|
// lines (including any nested phase logs).
|
|
var (
|
|
runRe = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
|
|
finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
|
|
)
|
|
|
|
// SliceTestLog copies lines belonging to the named testcase from src to
|
|
// dst. The block starts at `=== RUN <name>` and ends at the matching
|
|
// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
|
|
// never seen the destination file is created empty so the bundle has a
|
|
// stable shape.
|
|
//
|
|
// Returns the number of lines written; missing src is not an error.
|
|
func SliceTestLog(src, testName, dst string) (int, error) {
|
|
f, err := os.Open(src)
|
|
if err != nil {
|
|
if os.IsNotExist(err) {
|
|
return 0, nil
|
|
}
|
|
return 0, err
|
|
}
|
|
defer f.Close()
|
|
|
|
out, err := os.Create(dst)
|
|
if err != nil {
|
|
return 0, err
|
|
}
|
|
defer out.Close()
|
|
|
|
bw := bufio.NewWriter(out)
|
|
defer bw.Flush()
|
|
|
|
sc := bufio.NewScanner(f)
|
|
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
|
|
|
in := false
|
|
written := 0
|
|
for sc.Scan() {
|
|
line := sc.Text()
|
|
if !in {
|
|
if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
|
|
in = true
|
|
} else {
|
|
continue
|
|
}
|
|
}
|
|
if _, err := bw.WriteString(line); err != nil {
|
|
return written, err
|
|
}
|
|
if err := bw.WriteByte('\n'); err != nil {
|
|
return written, err
|
|
}
|
|
written++
|
|
if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
|
|
return written, nil
|
|
}
|
|
}
|
|
return written, sc.Err()
|
|
}
|
|
|
|
// copyFile is a small helper used when bundling pre-collected per-test
|
|
// artifacts. Best-effort: missing src is silently skipped.
|
|
func copyFile(src, dst string) error {
|
|
in, err := os.Open(src)
|
|
if err != nil {
|
|
if os.IsNotExist(err) {
|
|
return nil
|
|
}
|
|
return err
|
|
}
|
|
defer in.Close()
|
|
out, err := os.Create(dst)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer out.Close()
|
|
_, err = io.Copy(out, in)
|
|
return err
|
|
}
|