mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
2acb37342d
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution
DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.
* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle
Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:
- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
count from two to one so an ALB provisions in a single-AZ deployment.
* feat(eks): render IRSA into managed addons + node-minted webhook cert
Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.
* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller
Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.
* fix(eks): point the LB controller at the customer VPC + customer-facing gateway
The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:
- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
which the gateway's DescribeVpcs does not return for customer calls, so
backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
and their subnets live and where the ALB is provisioned.
- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
bootstrap, state-report and token-webhook paths.
* fix(elbv2): resolve forward target group from ForwardConfig
listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.
* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records
The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).
Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.
* test(daemon): pin synthetic capacity in host-dependent allocate tests
TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.
Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.
* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)
In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.
Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.
* feat: bump aws-load-balancer-controller addon bundle to v2.11.0
v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.
Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.
* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)
LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.
Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.
* fix: synthesize taggable ARN for default listener rule
Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.
Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.
* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB
LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.
Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.
* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams
Live verification showed two gaps in the LBC addon bundle:
LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.
The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.
* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets
k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.
* fix(elbv2): honor SubnetMappings in CreateLoadBalancer
LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.
* fix(vm): capture GCInterval before spawning GC goroutine
TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
148 lines
4.6 KiB
Go
148 lines
4.6 KiB
Go
// Command eks-webhook-cert mints, then reuses, a self-signed ECDSA-P256 serving
|
|
// certificate for an in-cluster admission webhook and prints its base64 PEM so
|
|
// the addon renderer can embed it as the webhook Secret + each caBundle.
|
|
//
|
|
// The eks-node AMI carries no openssl, so this is the Go equivalent of the
|
|
// upstream Helm chart's install-time genSelfSignedCert: one cert per cluster,
|
|
// cached on disk under -dir, stable across addon-sync ticks. The single cert is
|
|
// its own CA (IsCA + ServerAuth), so the same PEM serves as both the TLS leaf
|
|
// and the caBundle the apiserver trusts.
|
|
//
|
|
// Usage:
|
|
//
|
|
// eks-webhook-cert -dir /var/lib/spinifex-eks/webhook-certs/<addon> \
|
|
// -cn <svc>.<ns>.svc -dns <svc>.<ns>.svc,<svc>.<ns>.svc.cluster.local
|
|
//
|
|
// Output (one line): <ca_b64>\t<tls_crt_b64>\t<tls_key_b64>
|
|
package main
|
|
|
|
import (
|
|
"crypto/ecdsa"
|
|
"crypto/elliptic"
|
|
"crypto/rand"
|
|
"crypto/x509"
|
|
"crypto/x509/pkix"
|
|
"encoding/base64"
|
|
"encoding/pem"
|
|
"flag"
|
|
"fmt"
|
|
"log/slog"
|
|
"math/big"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
func main() {
|
|
dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
|
|
cn := flag.String("cn", "", "certificate common name")
|
|
dnsCSV := flag.String("dns", "", "comma-separated DNS SANs")
|
|
flag.Parse()
|
|
|
|
if *dir == "" || *cn == "" || *dnsCSV == "" {
|
|
fatal("eks-webhook-cert: -dir, -cn and -dns are required")
|
|
}
|
|
dnsNames := splitCSV(*dnsCSV)
|
|
if len(dnsNames) == 0 {
|
|
fatal("eks-webhook-cert: -dns lists no names")
|
|
}
|
|
|
|
certPEM, keyPEM, err := ensureCert(*dir, *cn, dnsNames)
|
|
if err != nil {
|
|
fatal("eks-webhook-cert: " + err.Error())
|
|
}
|
|
b := base64.StdEncoding.EncodeToString
|
|
fmt.Printf("%s\t%s\t%s\n", b(certPEM), b(certPEM), b(keyPEM))
|
|
}
|
|
|
|
// ensureCert returns the cached cert/key under dir, minting and persisting a
|
|
// fresh self-signed pair (for cn + dnsNames) when none is present or readable.
|
|
func ensureCert(dir, cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
|
|
certPath := filepath.Join(dir, "tls.crt")
|
|
keyPath := filepath.Join(dir, "tls.key")
|
|
|
|
if c, cerr := os.ReadFile(certPath); cerr == nil {
|
|
if k, kerr := os.ReadFile(keyPath); kerr == nil {
|
|
if _, perr := x509.ParseCertificate(decodeFirst(c)); perr == nil && len(k) > 0 {
|
|
return c, k, nil
|
|
}
|
|
}
|
|
}
|
|
|
|
certPEM, keyPEM, err = generateSelfSigned(cn, dnsNames)
|
|
if err != nil {
|
|
return nil, nil, err
|
|
}
|
|
if err = os.MkdirAll(dir, 0o700); err != nil {
|
|
return nil, nil, fmt.Errorf("mkdir %s: %w", dir, err)
|
|
}
|
|
if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
|
|
return nil, nil, fmt.Errorf("write %s: %w", certPath, err)
|
|
}
|
|
if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
|
|
return nil, nil, fmt.Errorf("write %s: %w", keyPath, err)
|
|
}
|
|
return certPEM, keyPEM, nil
|
|
}
|
|
|
|
// generateSelfSigned mints an ECDSA-P256 self-signed cert valid as both a CA and
|
|
// a server cert, with the given common name and DNS SANs.
|
|
func generateSelfSigned(cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
|
|
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
|
if err != nil {
|
|
return nil, nil, fmt.Errorf("generate key: %w", err)
|
|
}
|
|
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
|
|
if err != nil {
|
|
return nil, nil, fmt.Errorf("serial: %w", err)
|
|
}
|
|
now := time.Now().UTC()
|
|
tmpl := x509.Certificate{
|
|
SerialNumber: serial,
|
|
Subject: pkix.Name{CommonName: cn},
|
|
NotBefore: now.Add(-time.Hour),
|
|
NotAfter: now.AddDate(10, 0, 0),
|
|
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
|
|
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
|
BasicConstraintsValid: true,
|
|
IsCA: true,
|
|
DNSNames: dnsNames,
|
|
}
|
|
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
|
|
if err != nil {
|
|
return nil, nil, fmt.Errorf("create cert: %w", err)
|
|
}
|
|
keyDER, err := x509.MarshalECPrivateKey(priv)
|
|
if err != nil {
|
|
return nil, nil, fmt.Errorf("marshal key: %w", err)
|
|
}
|
|
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
|
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
|
|
return certPEM, keyPEM, nil
|
|
}
|
|
|
|
// decodeFirst returns the DER bytes of the first PEM block, or nil.
|
|
func decodeFirst(pemBytes []byte) []byte {
|
|
blk, _ := pem.Decode(pemBytes)
|
|
if blk == nil {
|
|
return nil
|
|
}
|
|
return blk.Bytes
|
|
}
|
|
|
|
func splitCSV(s string) []string {
|
|
var out []string
|
|
for p := range strings.SplitSeq(s, ",") {
|
|
if p = strings.TrimSpace(p); p != "" {
|
|
out = append(out, p)
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
func fatal(msg string) {
|
|
slog.Error(msg)
|
|
os.Exit(1)
|
|
}
|