mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 03:11:46 +00:00
b23fa3bf60
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
140 lines
4.6 KiB
Go
140 lines
4.6 KiB
Go
// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
|
|
// control-plane resource from the host through the AWS gateway, instead of
|
|
// dialing core NATS directly. It is the read-side companion to
|
|
// eks-gateway-publish.
|
|
//
|
|
// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
|
|
// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
|
|
// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
|
|
// as tab-separated lines the on-VM addon-sync shell agent consumes without a
|
|
// JSON parser:
|
|
//
|
|
// <addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
|
|
//
|
|
// The configuration values are base64-encoded so embedded tabs/newlines cannot
|
|
// break the line framing.
|
|
//
|
|
// Usage:
|
|
//
|
|
// eks-gateway-fetch -resource addons # GET /clusters/{cluster}/internal-addons/{accountId}
|
|
//
|
|
// Flags default to environment variables seeded by cloud-init:
|
|
// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
|
|
// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
|
|
package main
|
|
|
|
import (
|
|
"bufio"
|
|
"encoding/base64"
|
|
"encoding/json"
|
|
"flag"
|
|
"fmt"
|
|
"io"
|
|
"log/slog"
|
|
"os"
|
|
"time"
|
|
|
|
"github.com/mulgadc/spinifex/internal/eksgw"
|
|
|
|
_ "github.com/mulgadc/spinifex/internal/fipsboot"
|
|
)
|
|
|
|
// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
|
|
// here to keep this tiny VM binary free of the handler package's dependency
|
|
// graph.
|
|
type stagedAddon struct {
|
|
AddonName string `json:"addonName"`
|
|
AddonVersion string `json:"addonVersion"`
|
|
ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
|
|
ConfigurationValues string `json:"configurationValues"`
|
|
}
|
|
|
|
type internalAddonsResponse struct {
|
|
Addons []stagedAddon `json:"addons"`
|
|
}
|
|
|
|
const (
|
|
maxAttempts = 30
|
|
retryDelay = 5 * time.Second
|
|
)
|
|
|
|
func main() {
|
|
var (
|
|
gatewayURL string
|
|
gatewayCA string
|
|
accessKey string
|
|
secretKey string
|
|
region string
|
|
accountID string
|
|
clusterName string
|
|
resource string
|
|
)
|
|
|
|
flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
|
|
flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
|
|
flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
|
|
flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
|
|
flag.StringVar(®ion, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
|
|
flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
|
|
flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
|
|
flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
|
|
flag.Parse()
|
|
|
|
switch {
|
|
case accountID == "":
|
|
fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
|
|
case clusterName == "":
|
|
fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
|
|
case resource != "addons":
|
|
fatal("--resource must be addons")
|
|
}
|
|
|
|
client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
|
|
if err != nil {
|
|
fatal(fmt.Sprintf("build gateway client: %v", err))
|
|
}
|
|
path := "/clusters/" + clusterName + "/internal-addons/" + accountID
|
|
|
|
var lastErr error
|
|
for attempt := 1; attempt <= maxAttempts; attempt++ {
|
|
body, err := client.Get(path)
|
|
if err != nil {
|
|
lastErr = err
|
|
slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
|
|
if attempt < maxAttempts {
|
|
time.Sleep(retryDelay)
|
|
}
|
|
continue
|
|
}
|
|
if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
|
|
fatal(fmt.Sprintf("render response: %v", werr))
|
|
}
|
|
return
|
|
}
|
|
fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
|
|
}
|
|
|
|
// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
|
|
// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
|
|
// every locally-rendered manifest), which is the correct steady state for a
|
|
// cluster with no managed add-ons.
|
|
func emitAddonsTSV(out io.Writer, body []byte) error {
|
|
var resp internalAddonsResponse
|
|
if err := json.Unmarshal(body, &resp); err != nil {
|
|
return fmt.Errorf("unmarshal internal-addons response: %w", err)
|
|
}
|
|
w := bufio.NewWriter(out)
|
|
for _, a := range resp.Addons {
|
|
cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
|
|
if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
return w.Flush()
|
|
}
|
|
|
|
func fatal(msg string) {
|
|
slog.Error("eks-gateway-fetch: " + msg)
|
|
os.Exit(1)
|
|
}
|