mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
dev
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
8e11908dfb |
fix: make the EKS control plane's disk latency observable, and give it room to run (#1009)
* fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> |
||
|
Josh Siviour
|
f7bf921ce7 |
fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511)
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group names exceed the ELBv2 32-char limit, failing cluster creation asynchronously with no visible cause. safeELBv2Name now deterministically truncates the name and appends a stable hash of the full cluster name when needed, so the same name is derived on every call (create, lookup, delete). DescribeCluster also now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster is FAILED, instead of leaving callers to guess why. * fix(eks): plumb per-nodegroup Ready-node counts through state reports The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready count keyed by the eks.amazonaws.com/nodegroup label value, alongside the existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's observe step, and SetClusterHealthState now carry this map through to ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next commit) can gate on a nodegroup's own workers instead of the cluster total. * fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches waitWorkersReady compared the cluster-wide Ready-node count against a nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's Ready workers could mask another whose own workers never registered. It now reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the eks.amazonaws.com/nodegroup=<name> label. Separately, launchNodegroupInfra never retried a failed worker launch (a transient QMP timeout or host capacity pressure), leaving the nodegroup stuck CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches only the shortfall on failure, logging each retry, until the desired count is reached or workerLaunchRetryTimeout elapses. Both fixes are covered by regression tests: a two-nodegroup scenario where only one has Ready workers, a transient-failure-then-success relaunch, and an exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which this change does not add. * fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and kubectl logs/exec against pods on that node failed "No agent available". Adds an Exists toleration for nvidia.com/gpu:NoSchedule. Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster available to test against right now. * fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS |
||
|
Josh Siviour
|
e47baf321c |
feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480)
* feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight. |
||
|
Josh Siviour
|
2d4f0fa94c |
feat: IMDS instance-role creds for lb-agent + EKS server VMs (siv-231.6) (#450)
* feat: IMDS-capable gateway signer for lb-agent + EKS clients Add internal/gwsign, a SigV4 signer that draws credentials from either fixed static keys or the AWS SDK default credential chain. In a guest VM the chain falls back to the in-VM IMDS endpoint, yielding scoped, auto-rotating instance-role credentials; a session token is signed as X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds. Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client (internal/eksgw). When static keys are absent the clients now sign with IMDS credentials instead of failing; when present, behaviour is unchanged (static path). This is the client-side foundation for dropping long-lived secrets from VM user-data. Credentials are retrieved per request so rotated IMDS creds take effect without an agent restart. * feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C) Attach a Spinifex-managed system instance profile to the k3s control-plane VM and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating IMDS instance-role credentials instead of a long-lived static secret baked into user-data. - Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a role (EC2 trust) plus its instance profile and asserts an inline permission policy. The gateway enforces per-action authz on assumed-role principals, so the roles carry exactly the actions each VM calls: - EKS server: eks:PublishInternal, eks:ListInternalAddons. - LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig. - Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves the role. - Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/ EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from the lb-agent fw_cfg env; relax k3s server validation to accept profile-only; drop the EKS key requires from the first-boot and state-report scripts. - Both verticals fall back to baked static creds when IAM is unwired, so an absent master key never blocks a launch. * fix: resolve IMDS instance-role creds under the instance account for system VMs Three boot-path defects kept the LB VM's lb-agent from authenticating to the gateway with IMDS-served instance-role creds; each produced a silent empty IMDS role list and a heartbeat failing with "unexpected empty EC2 IMDS role list". Found and fixed against env19. - Build the system IAM ensurer lazily on the first system-VM launch instead of eagerly at daemon startup, where it raced the NATS KV backend and fell back to baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider resolved at launch time. - Create the LB instance profile in the system account the LB VM runs under (utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile under the instance's account, so a customer-account role was invisible to it. - Record the attached instance's owning account on the ENI (InstanceOwnerId, mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every IAM/STS lookup under it. The LB ENI lives in the customer account while the system LB VM and its role live in the system account; keying the whole IMDS chain off the ENI account left the instance invisible and the role lookup missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account. A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role principal with no static keys in user-data. * test(e2e): assert LB VM authenticates via IMDS instance-role creds The LB suite's targets-healthy assertion already depends on the lb-agent authenticating to the gateway, and with static keys dropped from user-data that auth is IMDS-only. Add an explicit check that a recent gateway authentication for the LB used an assumed-role (ASIA) principal, so a regression to the instance-account IMDS resolution fails with a clear message instead of a mysterious datapath timeout. The LB VM is a system-account instance invisible to the customer AWS API, so the assertion observes the gateway journal on the cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and skips when no node is SSH-reachable. |
||
|
Josh Siviour
|
a7241e01f9 |
feat: run EKS apiserver with anonymous-auth=false (CIS) (#447)
Flip the k3s server apiserver to anonymous-auth=false. The only consumer of the anonymous path was mulga-eks-state-report's /healthz probe; switch it to an authenticated kubectl get --raw=/healthz through the node admin kubeconfig (an unauthenticated curl now returns 401). The NLB target group uses a TCP health check, and the host-side reconciler rides the authenticated NATS state-report, so neither depends on the anonymous apiserver path. |
||
|
Josh Siviour
|
5025a37006 |
feat: Containers — internal-ECR pulls, DNS-free registry, EKS hardening + docs (#409)
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
|
||
|
Josh Siviour
|
4ef19fc488 | feat(ui): EKS console (#310) | ||
|
Josh Siviour
|
496424c41c |
test(eks): EKS control-plane E2E suite (Sprint 6g) (#304)
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
|
||
|
Josh Siviour
|
a22bc2f716 |
feat(eks): 6d nodegroup provisioning + NATS-gated cluster health (#300)
* feat(eks): nodegroup provisioning + K3s agent join (6d)
CreateNodegroup launches N customer-owned worker EC2 from the unified
eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN
baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env).
Workers are customer-owned and DescribeInstances-visible (no ManagedBy,
no mgmt NIC) via the normal RunInstances path, matching real EKS.
K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the
cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB
endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the
decrypted cluster node-token persisted at server bootstrap.
Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup
SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250,
intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred.
* feat: gate EKS cluster health via NATS state reports
The K3s apiserver binds the VPC node-ip, unreachable from the host-side
spinifex daemon, so the reconciler cannot probe /healthz directly. Add a
control-plane state reporter that publishes {healthz, node_count} over the
management NATS bus the daemon already shares.
- eks-node server role runs mulga-eks-state-report as an OpenRC service,
publishing eks.state.{account}.{cluster}.server every 30s.
- ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on
a healthy report, and records Ready node_count into ClusterMeta.
- node_count counts Ready nodes only; k3s retains terminated workers as
NotReady until pruned, so a raw count never drops on scale-down.
|
||
|
Josh Siviour
|
f14fb64562 |
feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288)
* feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first. |