mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
809ec5f1feca54f6534877b301eb080ee08ea8ba
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
7474fa2073 | chore: use native loadx509KeyPair | ||
|
Julian Sommer
|
c81abd1836 | fix: add fipsboot guard to missing services | ||
|
Julian Sommer
|
97acfbbd80 | refactor: use bluebottle/pkg/tlsconfig for the TLS curve allowlist (#910) | ||
|
Julian Sommer
|
4c1f7864f8 | chore: more cleanup | ||
|
Josh Siviour
|
7bbd99c792 |
feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459)
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
|
||
|
Julian Sommer
|
7371358537 | chore: more cleanups | ||
|
Josh Siviour
|
496424c41c |
test(eks): EKS control-plane E2E suite (Sprint 6g) (#304)
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
|
||
|
Josh Siviour
|
5aaf8c23d2 |
fix(eks): make aws eks get-token → kubectl auth work end-to-end (Sprint 6f) (#303)
* feat(eks): webhook observability — console log sink + per-request TokenReview decision * fix(eks): wire token-webhook authenticator into generated k3s apiserver config The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked config.yaml.skel, but its kube-apiserver-arg list omitted the authentication-token-webhook-config-file arg. The running apiserver therefore never loaded the token webhook, so bearer tokens minted by 'aws eks get-token' 401'd without the webhook ever being invoked. * fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal With --api-audiences set, kube-apiserver sends spec.audiences in the TokenReview and discards an authenticated=true response that does not confirm a matching audience. The webhook returned an empty status.audiences, so a valid IAM principal surfaced as a 401. Echo the request's audiences on success. * fix(eks): pin token-webhook version v1 so apiserver accepts the response kube-apiserver --authentication-token-webhook-version defaults to v1beta1, but the eks-token-webhook responds with an authentication.k8s.io/v1 TokenReview. The apiserver decodes the response against the expected GVK and rejects the mismatch, surfacing a valid principal as a 401. Pin v1 to match the webhook. |
||
|
Josh Siviour
|
bfacca7789 |
feat(eks): 6c token webhook + AccessEntry (IAM auth → kubectl) (#284)
* feat(eks): 6c AccessEntry CRUD + access-policy association
Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real
KV-backed handlers (Q9, API auth mode):
- AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash};
the principal ARN is hashed into the key (ARNs carry ':', illegal in
NATS KV keys) while the record keeps the plaintext ARN.
- AssociateAccessPolicy/Disassociate/ListAssociated against the four
supported AWS-managed policies; unsupported policy ARN or bad access
scope -> InvalidParameterValue.
- ListAccessPolicies returns the static supported catalogue.
- Cluster-existence gate -> ResourceNotFoundException; duplicate create
-> ResourceInUseException; node-type entries rejected until 6d.
Updates the daemon routing test for the now-implemented handlers.
* feat(eks): plumb caller principal ARN + bootstrap-creator-admin
Thread the caller's resolved IAM principal ARN from the gateway to the
EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster
can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9):
- utils.NATSRequest takes optional extra headers; PrincipalARNHeader +
PrincipalARNFromMsg added.
- Gateway resolves the caller ARN best-effort (buildCallerARN) and passes
it down the EKS route table; only CreateCluster consumes it.
- CreateCluster signature gains callerPrincipalARN; default-true bootstrap
flag mints a system:masters entry keyed by the caller's exact ARN (the
token webhook resolves the same ARN from STS). '' degrades gracefully.
- Dedicated daemon handler reads both headers.
* feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring
The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl
requests that present an `aws eks get-token` bearer token: decode the presigned
STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster
AccessEntry, and answer the apiserver TokenReview with the mapped username +
groups.
STS is hosted in-process in the awsgw service and is not otherwise on the bus,
so awsgw now serves an `eks.VerifyToken` NATS subject that wraps
VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces
the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly
over its own NATS connection.
Apiserver wiring: the k3s config skeleton sets
authentication-token-webhook-config-file; the webhook writes its own self-signed
serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a
bounded wait in the k3s start_pre so the apiserver never loads a missing file).
* fix(eks): omit empty spec from token webhook TokenReview response
* fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD
VerifyPresignedGetCallerIdentity built the SigV4 canonical request with
"UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore
and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws
eks get-token` produces) with the SHA256 of the empty string. Every real
get-token therefore failed signature verification with InvalidIdentityToken,
breaking the entire EKS IAM token-auth flow end to end.
Use the empty-string SHA256. The existing hand-rolled presign tests masked the
bug by signing with the same wrong constant; they now sign correctly, and a new
test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the
CLI uses) so this can't regress silently.
Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to
the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid
token presented for a different cluster is still rejected (anti-replay intact).
* fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve
IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI
percent-encodes in the request path. The dispatcher matched r.URL.Path
(decoded), so %2F became a literal '/' and split the ARN across the
([^/]+) capture, dropping every describe/update/delete-access-entry and
associate/disassociate/list-access-policy call to InvalidActionException.
Match r.URL.EscapedPath() so the encoded ARN stays one segment, then
url.PathUnescape each captured param before it reaches the handler.
* test(eks): unit-cover access-entry scope validation, AWS converters, store guards
|
||
|
Josh Siviour
|
8883984f3f |
feat(eks): eks-server + eks-agent AMI manifests (#273)
* feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES
Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12).
Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in
isolation before CreateCluster orchestration lands.
Builder:
- build-system-image.sh gains optional INSTALL_FILES (src:dst pairs,
mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES.
- ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts
the manifest references are on disk before rc-update enables them.
eks-server image (Alpine 3.21.7, q35, 4G):
- K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the
upstream sha256sum-amd64.txt trust anchor).
- nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream
release zip; unzip installed + purged in same setup step).
- OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot.
- Cron periodics under /etc/periodic/{15min,daily}/ for state-report +
etcd-snapshot.
- k3s-first-boot publishes node-token + admin kubeconfig over NATS on
first successful K3s healthz, then self-disables (sentinel + rc-update
del).
- config.yaml.skel without webhook-token-auth — wiring it before the real
webhook lands (cs-eks-6c) would block anonymous probes against /healthz.
Orchestrator drops the live config + restarts k3s.
eks-agent image (Alpine 3.21.7, q35, 2G):
- Same pinned K3s release for server/agent lock-step.
- k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN —
fail-closed pre-orchestration.
eks-token-webhook stub (cmd/eks-token-webhook/main.go):
- 503 + Retry-After:30 on every path. Statically linked. Lets the AMI
build succeed and kube-apiserver boot cleanly. Replaced wholesale by
cs-eks-6c.
Init system divergence from spec wording: Alpine ships OpenRC, not
systemd. Unit files use .initd extensions and cron periodics instead of
.service / .timer. Bead-level intent (auto-enable on boot, periodic
execution) preserved.
Smoke-tested:
- eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on
6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401
is K3s v1.32 default RBAC behaviour against anonymous, not a regression).
- eks-agent: builds 280M qcow2; qemu boot mounts root cleanly.
* feat(eks): make targets for eks-server + eks-agent AMI builds
- build-system-image gains subdir-manifest fallback: tries
scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf
so the new EKS layout works alongside the existing flat lb manifest.
- build-eks-server-image / build-eks-agent-image — explicit convenience
targets that delegate to the generic one.
|