Files
spinifex/internal/eksgw/client.go
T
Josh Siviour b23fa3bf60 feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356)
* feat(eks): VM-side managed-addon delivery transport

Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).

- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
  re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
  observability

* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP

The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.

Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.

- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
  drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
  when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP

* feat: auto-wire IGW for internet-facing EKS cluster endpoint

An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.

EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.

* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak

A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.

- failClusterLaunch now runs a best-effort purgeClusterInfra on the
  in-memory meta (freshest refs even when the failing step was the meta
  write itself) before MarkClusterFailed. purgeClusterInfra is
  idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
  the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
  TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
  that fails after AssociateAddress releases the eipService allocation
  instead of leaking it (MarkFailed's ReleasePublicIP only knows the
  externalIPAM path).

* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs

CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.

This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.

Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.

* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID

The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.

Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.

* fix(eks): explicitly delete addon objects on unstage

K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
2026-06-12 22:29:24 +10:00

113 lines
3.3 KiB
Go

// Package eksgw is the on-VM SigV4 client used by EKS helpers (eks-gateway-publish,
// eks-token-webhook) to POST over HTTPS to the AWS gateway instead of NATS.
// Signs with Predastore credentials for service "eks".
package eksgw
import (
"bytes"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
"github.com/mulgadc/predastore/auth"
"github.com/mulgadc/spinifex/internal/tlsconfig"
)
// Client posts SigV4-signed requests to the AWS gateway for the "eks" service.
type Client struct {
baseURL string
accessKey string
secretKey string
region string
httpClient *http.Client
}
// New builds a client. caPath optionally pins the gateway TLS CA; when empty the
// client relies on the system trust store. region defaults to us-east-1 when
// empty (SigV4 requires a non-empty region).
func New(baseURL, caPath, accessKey, secretKey, region string) (*Client, error) {
if baseURL == "" {
return nil, fmt.Errorf("eksgw: baseURL is required")
}
if accessKey == "" || secretKey == "" {
return nil, fmt.Errorf("eksgw: access key and secret key are required")
}
if region == "" {
region = "us-east-1"
}
tlsCfg := &tls.Config{
MinVersion: tls.VersionTLS13,
CurvePreferences: tlsconfig.Curves,
}
if caPath != "" {
pem, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("eksgw: read gateway CA %q: %w", caPath, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("eksgw: gateway CA %q has no usable certificates", caPath)
}
tlsCfg.RootCAs = pool
}
return &Client{
baseURL: strings.TrimRight(baseURL, "/"),
accessKey: accessKey,
secretKey: secretKey,
region: region,
httpClient: &http.Client{
Timeout: 10 * time.Second,
Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
},
}, nil
}
// Post SigV4-signs and sends body to path, returning the response body on 2xx.
// Errors include the gateway status and body. No retry; callers wrap as needed.
func (c *Client) Post(path string, body []byte) ([]byte, error) {
return c.do(http.MethodPost, path, body)
}
// Get SigV4-signs and sends a single GET to path (e.g.
// "/clusters/alpha/internal-addons?accountId=..."). Same contract as Post: 2xx
// body or an error carrying the gateway status. No retry.
func (c *Client) Get(path string) ([]byte, error) {
return c.do(http.MethodGet, path, nil)
}
func (c *Client) do(method, path string, body []byte) ([]byte, error) {
req, err := http.NewRequest(method, c.baseURL+path, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
if body != nil {
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
}
sum := sha256.Sum256(body)
if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "eks", c.region); err != nil {
return nil, fmt.Errorf("sign request: %w", err)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, fmt.Errorf("send request: %w", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("gateway returned %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
}
return respBody, nil
}