mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-16 01:22:00 +00:00
feat(eks): join nodegroup workers via cluster NLB endpoint + private endpoint access (#362)
* feat: join nodegroup workers via cluster NLB endpoint Under the Set-B managed-CP-VPC topology the control plane runs in the system-account CP VPC while nodegroup workers launch in the customer VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP (:6443), which sits in a different, unpeered VPC and is unreachable, so the join path was dead. Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint, https://<frontend-ip>:443, the :443->:6443 listener kubectl already uses). Guard CreateNodegroup on a non-empty endpoint. mulga-siv-298 * feat: gate nodegroup ACTIVE on workers registering Ready launchNodegroupInfra flipped the nodegroup to ACTIVE the moment RunInstances returned, so a worker that booted but never joined left the nodegroup falsely ACTIVE with no usable capacity. Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed by the cluster reconciler from the CP NATS state report — Ready nodes only) rising by the desired worker count over the create-time baseline. On timeout the nodegroup is marked CREATE_FAILED with its instance IDs retained for the reclaim path. Mirrors the CP reconciler's healthy-observe gating. Bounded by injectable timeout/poll knobs. mulga-siv-299 * feat(eks): private endpoint access via cross-account Set A ENI Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the control plane through a customer-VPC (Set A) resident ENI on the cluster NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path. Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not its primary, so the instance-terminate cascade never reclaims it. Detach (store-clear) the stale attachment before deleting, so private clusters delete cleanly instead of looping on InvalidNetworkInterface.InUse. Refs: mulga-siv-301 * fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule resources by SecurityGroupRuleId with empty IpPermissions. The handlers only read IpPermissions, so they returned success while removing nothing; the surviving SG-reference rule then pinned both SGs undeletable (DependencyViolation), deadlocking destroy of mutually-referencing SGs. Resolve SecurityGroupRuleIds to their stored rules and remove them (malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound). Refs: mulga-siv-321 * fix(vpc): reap the main route table on DeleteVpc CreateVpc auto-creates a main route table and DeleteRouteTable refuses to delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps the VPC's main route table before deleting the VPC record. Refs: mulga-siv-322 * fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS Concurrent writes to a single role / route table record used a blind read-modify-Put with no optimistic-concurrency guard, so parallel writers racing from the same revision clobbered each other and lost updates. Verified on env19: three parallel policy attachments to one node role kept only one; two parallel subnet associations to one route table kept only one (the provider then timed out waiting for state 'associated'). Route the mutating paths (AttachRolePolicy/DetachRolePolicy, AssociateRouteTable/DisassociateRouteTable) through a revision-guarded Update + bounded retry, matching the existing accountCounterBucket CAS idiom. Add concurrent regression tests covering both lost updates. * docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress) Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a browsable load-balanced demo app: - eks-quickstart: minimal single-node cluster + NodePort demo - eks-addons-access: CoreDNS managed addon + access-entry RBAC - eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS with an ACM-imported self-signed certificate Each covers the Spinifex-specific endpoints, the auto-managed worker SG NodePort pattern, and the omitted addon_version constraint.
This commit is contained in:
Josh Siviour
@@ -0,0 +1,181 @@
|
||||
---
|
||||
title: "EKS with Addons + Access-Entry RBAC"
|
||||
description: "Extend a single-node EKS cluster with a managed CoreDNS addon, access-entry RBAC, and a browsable load-balanced demo app, using Terraform on Spinifex."
|
||||
category: "Terraform Workbooks"
|
||||
tags:
|
||||
- terraform
|
||||
- eks
|
||||
- kubernetes
|
||||
- addons
|
||||
- rbac
|
||||
- iam
|
||||
- workbook
|
||||
resources:
|
||||
- title: "Terraform AWS Provider"
|
||||
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
|
||||
- title: "Terraform Kubernetes Provider"
|
||||
url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
|
||||
- title: "Spinifex Repository"
|
||||
url: "https://github.com/mulgadc/spinifex"
|
||||
- title: "OpenTofu"
|
||||
url: "https://opentofu.org/"
|
||||
---
|
||||
|
||||
# Terraform: EKS with Addons + Access-Entry RBAC
|
||||
|
||||
> Builds on the quickstart by installing a managed addon (CoreDNS) and granting a second IAM principal read-only cluster access — then deploys the same browsable, load-balanced demo app so you can see the result.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Instructions](#instructions)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This workbook takes the same single-node cluster (and the same browsable demo app) from [EKS Quickstart](../eks-quickstart) and adds the two things you reach for on almost any real cluster:
|
||||
|
||||
1. **A managed addon.** CoreDNS is installed through the EKS API (`aws_eks_addon`) rather than `kubectl apply`, so its lifecycle is managed by the cluster. The demo app is deployed *after* CoreDNS, so a healthy page also confirms in-cluster DNS is working.
|
||||
2. **Access-entry RBAC.** A second IAM role is granted read-only access via an `aws_eks_access_entry` plus an `aws_eks_access_policy_association` bound to the AWS-managed `AmazonEKSViewPolicy`. This is the modern access-entry path (`authentication_mode = "API"`) — no `aws-auth` ConfigMap.
|
||||
|
||||
When `apply` finishes, open the **`demo_url`** output and refresh — the `nginxdemos/hello` page reports which pod served each request.
|
||||
|
||||
**What you'll learn:**
|
||||
|
||||
- Installing a managed EKS addon with `aws_eks_addon`
|
||||
- Registering an IAM principal with `aws_eks_access_entry` and scoping it with `aws_eks_access_policy_association`
|
||||
- The difference between the cluster creator's admin access and a scoped viewer
|
||||
- Deploying a workload with the Kubernetes provider once the addon is healthy
|
||||
|
||||
**What gets created**
|
||||
|
||||
| Resource | Name | Purpose |
|
||||
|---|---|---|
|
||||
| VPC | `eks-addons-vpc` | Isolated network (10.31.0.0/16) |
|
||||
| Subnets | `eks-addons-subnet-a/-b` | Public subnets for the cluster and worker |
|
||||
| Internet Gateway | `eks-addons-igw` | Egress so the worker can pull the demo image |
|
||||
| IAM Roles | `eks-addons-cluster-role`, `eks-addons-node-role` | Control-plane and worker roles |
|
||||
| IAM Role | `eks-addons-viewer` | Stand-in principal granted read-only access |
|
||||
| EKS Cluster | `eks-addons` | Public endpoint, API auth mode |
|
||||
| Node Group | `default` | One `t3.medium` worker |
|
||||
| Addon | `coredns` | Cluster DNS, managed via the EKS API |
|
||||
| Access Entry + Assoc. | `eks-addons-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
|
||||
| SG Ingress Rule | `eks-addons-demo-nodeport` | Opens the demo NodePort on the worker SG |
|
||||
| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 2 replicas, NodePort |
|
||||
|
||||
**Spinifex specifics**
|
||||
|
||||
- Addon catalog is limited. Spinifex currently serves `coredns`, `aws-load-balancer-controller` and `spinifex-noop`. There is no `vpc-cni`, `kube-proxy`, `ebs-csi` or `metrics-server`.
|
||||
- **Leave `addon_version` unset.** Spinifex defaults an unset version to its catalog default (`coredns` 1.11.1). Pinning is a dead end: the AWS provider rejects a bare `1.11.1` (it requires the `v`-prefixed `v1.11.1-eksbuild.N` form), while Spinifex's catalog only accepts the bare `1.11.1` — so no single string satisfies both.
|
||||
- Cluster access policies are the four AWS-managed `cluster-access-policy/Amazon EKS{ClusterAdmin,Admin,Edit,View}Policy` ARNs.
|
||||
- The worker SG is auto-managed and the worker AMI is always the `eks-node` image; the NodePort is exposed by adding one rule to the looked-up worker SG.
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
- Spinifex installed and running
|
||||
- The Spinifex `eks-node` image available on the cluster
|
||||
- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
|
||||
|
||||
## Instructions
|
||||
|
||||
### Step 1. Get the Template
|
||||
|
||||
```bash
|
||||
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
|
||||
cd spinifex-tf
|
||||
git sparse-checkout set docs/terraform-workbooks
|
||||
cd docs/terraform-workbooks/eks-addons-access
|
||||
```
|
||||
|
||||
Or create a `main.tf` file and paste the full configuration below.
|
||||
|
||||
<!-- INCLUDE: main.tf lang:hcl -->
|
||||
|
||||
### Step 2. Deploy
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
tofu init
|
||||
tofu apply
|
||||
```
|
||||
|
||||
Allow a few minutes for the cluster to reach `ACTIVE` and the worker to register. CoreDNS installs onto the worker, then Terraform deploys the demo app.
|
||||
|
||||
> Keep `AWS_PROFILE=spinifex` exported for the whole `apply` — the Kubernetes provider runs `aws eks get-token` against the Spinifex STS endpoint.
|
||||
|
||||
### Step 3. Open the Demo
|
||||
|
||||
```bash
|
||||
tofu output demo_url
|
||||
```
|
||||
|
||||
Open it in a browser and refresh — the `nginxdemos/hello` page shows the pod that answered, alternating between the two replicas.
|
||||
|
||||
### Step 4. Verify the Addon and Access Entry
|
||||
|
||||
```bash
|
||||
aws eks list-addons --cluster-name eks-addons
|
||||
aws eks describe-addon --cluster-name eks-addons --addon-name coredns --query 'addon.status'
|
||||
|
||||
aws eks list-access-entries --cluster-name eks-addons
|
||||
aws eks list-associated-access-policies \
|
||||
--cluster-name eks-addons \
|
||||
--principal-arn "$(tofu output -raw viewer_principal_arn)"
|
||||
```
|
||||
|
||||
The viewer principal is bound to `AmazonEKSViewPolicy` at cluster scope — it can read across the cluster but cannot mutate anything. The principal that ran `apply` keeps full cluster-admin from `bootstrap_cluster_creator_admin_permissions`.
|
||||
|
||||
### Cleanup
|
||||
|
||||
```bash
|
||||
tofu destroy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Demo URL Doesn't Load
|
||||
|
||||
Same chain as the quickstart — check the worker's public IP, that the demo pods are `Running`, and that the NodePort rule landed on the worker SG:
|
||||
|
||||
```bash
|
||||
kubectl get pods -o wide
|
||||
aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-addons-nodegroup-sg" \
|
||||
--query 'SecurityGroups[0].IpPermissions'
|
||||
```
|
||||
|
||||
If the demo pods are stuck `ContainerCreating` with DNS errors, confirm CoreDNS is healthy first:
|
||||
|
||||
```bash
|
||||
kubectl -n kube-system get deploy coredns
|
||||
```
|
||||
|
||||
### Addon Stuck in CREATING / DEGRADED
|
||||
|
||||
CoreDNS needs a schedulable worker. Confirm the node group is `ACTIVE` and the node is `Ready`:
|
||||
|
||||
```bash
|
||||
aws eks describe-nodegroup --cluster-name eks-addons --nodegroup-name default --query 'nodegroup.status'
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
### Addon Version Rejected
|
||||
|
||||
This workbook leaves `addon_version` unset so Spinifex picks its catalog default. If you add an explicit `addon_version`, expect a wall: the AWS provider requires the `v`-prefixed `v1.11.1-eksbuild.N` form, but Spinifex's catalog only knows the bare `1.11.1` — a value that passes one is rejected by the other. Prefer omitting the field.
|
||||
|
||||
### Access Entry Not Taking Effect
|
||||
|
||||
Assume the viewer role and confirm it can read but not write:
|
||||
|
||||
```bash
|
||||
kubectl auth can-i list pods --all-namespaces # yes
|
||||
kubectl auth can-i create deployments # no
|
||||
```
|
||||
|
||||
### Provider Connection Refused
|
||||
|
||||
```bash
|
||||
sudo systemctl status spinifex.target
|
||||
curl -k https://localhost:9999/
|
||||
```
|
||||
@@ -0,0 +1,468 @@
|
||||
# Example: EKS with an Addon + Access-Entry RBAC (with a visible demo app)
|
||||
#
|
||||
# Builds on eks-quickstart: same single-worker cluster and browsable demo app,
|
||||
# but adds two things you reach for on a real cluster:
|
||||
#
|
||||
# * a managed addon (CoreDNS) installed through the EKS API, and
|
||||
# * a second IAM principal granted scoped, read-only access via an EKS access
|
||||
# entry + access-policy association (authentication_mode = "API").
|
||||
#
|
||||
# After `apply`, open the demo_url output — the page reports the pod that
|
||||
# served it, so a refresh shows requests landing on different replicas. CoreDNS
|
||||
# being healthy is what lets in-cluster name resolution work.
|
||||
#
|
||||
# Usage:
|
||||
# cd spinifex/docs/terraform-workbooks/eks-addons-access
|
||||
# export AWS_PROFILE=spinifex
|
||||
# tofu init && tofu apply
|
||||
# # then open the demo_url output
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.6.0"
|
||||
|
||||
required_providers {
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = ">= 5.40, < 6.0"
|
||||
}
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = ">= 2.20"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Variables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
default = "ap-southeast-2"
|
||||
}
|
||||
|
||||
variable "cluster_name" {
|
||||
type = string
|
||||
default = "eks-addons"
|
||||
}
|
||||
|
||||
variable "k8s_version" {
|
||||
type = string
|
||||
default = "1.32"
|
||||
}
|
||||
|
||||
variable "instance_type" {
|
||||
type = string
|
||||
default = "t3.medium"
|
||||
}
|
||||
|
||||
variable "node_port" {
|
||||
type = number
|
||||
default = 30080
|
||||
}
|
||||
|
||||
variable "replicas" {
|
||||
type = number
|
||||
default = 2
|
||||
}
|
||||
|
||||
variable "browse_cidr" {
|
||||
type = string
|
||||
default = "0.0.0.0/0"
|
||||
description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
|
||||
}
|
||||
|
||||
variable "spinifex_endpoint" {
|
||||
type = string
|
||||
default = "https://127.0.0.1:9999"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Providers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
provider "aws" {
|
||||
region = var.region
|
||||
|
||||
endpoints {
|
||||
ec2 = var.spinifex_endpoint
|
||||
iam = var.spinifex_endpoint
|
||||
sts = var.spinifex_endpoint
|
||||
eks = var.spinifex_endpoint
|
||||
}
|
||||
|
||||
skip_credentials_validation = true
|
||||
skip_metadata_api_check = true
|
||||
skip_requesting_account_id = true
|
||||
skip_region_validation = true
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
host = aws_eks_cluster.this.endpoint
|
||||
cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
|
||||
|
||||
exec {
|
||||
api_version = "client.authentication.k8s.io/v1beta1"
|
||||
command = "aws"
|
||||
args = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
|
||||
}
|
||||
}
|
||||
|
||||
data "aws_availability_zones" "available" {
|
||||
state = "available"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# VPC + two public subnets
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_vpc" "main" {
|
||||
cidr_block = "10.31.0.0/16"
|
||||
enable_dns_hostnames = true
|
||||
enable_dns_support = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-vpc"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_internet_gateway" "igw" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-igw"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "a" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.31.1.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
map_public_ip_on_launch = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-subnet-a"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "b" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.31.2.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
map_public_ip_on_launch = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-subnet-b"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table" "public" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
route {
|
||||
cidr_block = "0.0.0.0/0"
|
||||
gateway_id = aws_internet_gateway.igw.id
|
||||
}
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-public-rt"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "a" {
|
||||
subnet_id = aws_subnet.a.id
|
||||
route_table_id = aws_route_table.public.id
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "b" {
|
||||
subnet_id = aws_subnet.b.id
|
||||
route_table_id = aws_route_table.public.id
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — cluster role
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "cluster" {
|
||||
name = "${var.cluster_name}-cluster-role"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "eks.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "cluster" {
|
||||
role = aws_iam_role.cluster.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — node role
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "node" {
|
||||
name = "${var.cluster_name}-node-role"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "ec2.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_worker" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_cni" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_ecr" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — a second principal to grant read-only cluster access to
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "viewer" {
|
||||
name = "${var.cluster_name}-viewer"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "ec2.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# EKS cluster
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_cluster" "this" {
|
||||
name = var.cluster_name
|
||||
role_arn = aws_iam_role.cluster.arn
|
||||
version = var.k8s_version
|
||||
|
||||
vpc_config {
|
||||
subnet_ids = [aws_subnet.a.id, aws_subnet.b.id]
|
||||
endpoint_public_access = true
|
||||
endpoint_private_access = false
|
||||
}
|
||||
|
||||
access_config {
|
||||
authentication_mode = "API"
|
||||
bootstrap_cluster_creator_admin_permissions = true
|
||||
}
|
||||
|
||||
depends_on = [aws_iam_role_policy_attachment.cluster]
|
||||
|
||||
tags = {
|
||||
Name = var.cluster_name
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node group
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_node_group" "default" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
node_group_name = "default"
|
||||
node_role_arn = aws_iam_role.node.arn
|
||||
subnet_ids = [aws_subnet.a.id, aws_subnet.b.id]
|
||||
|
||||
scaling_config {
|
||||
desired_size = 1
|
||||
min_size = 1
|
||||
max_size = 2
|
||||
}
|
||||
|
||||
instance_types = [var.instance_type]
|
||||
ami_type = "AL2_x86_64"
|
||||
|
||||
depends_on = [
|
||||
aws_iam_role_policy_attachment.node_worker,
|
||||
aws_iam_role_policy_attachment.node_cni,
|
||||
aws_iam_role_policy_attachment.node_ecr,
|
||||
]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-default"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Addon — CoreDNS
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_addon" "coredns" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
addon_name = "coredns"
|
||||
resolve_conflicts_on_create = "OVERWRITE"
|
||||
|
||||
# addon_version omitted on purpose. Spinifex defaults an unset version to its
|
||||
# catalog default (coredns 1.11.1). The AWS provider rejects a bare "1.11.1"
|
||||
# (it demands a v-prefixed form) which Spinifex's catalog would in turn
|
||||
# reject — so pinning is a dead end here; let the server choose.
|
||||
depends_on = [aws_eks_node_group.default]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-coredns"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Access entry — grant the viewer principal read-only access
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_access_entry" "viewer" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
principal_arn = aws_iam_role.viewer.arn
|
||||
type = "STANDARD"
|
||||
}
|
||||
|
||||
resource "aws_eks_access_policy_association" "viewer" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
principal_arn = aws_iam_role.viewer.arn
|
||||
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
|
||||
|
||||
access_scope {
|
||||
type = "cluster"
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_access_entry.viewer]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Open the NodePort on the auto-managed nodegroup SG (see eks-quickstart)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
data "aws_security_group" "nodegroup" {
|
||||
filter {
|
||||
name = "group-name"
|
||||
values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
|
||||
}
|
||||
|
||||
filter {
|
||||
name = "vpc-id"
|
||||
values = [aws_vpc.main.id]
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.default]
|
||||
}
|
||||
|
||||
resource "aws_vpc_security_group_ingress_rule" "nodeport" {
|
||||
security_group_id = data.aws_security_group.nodegroup.id
|
||||
cidr_ipv4 = var.browse_cidr
|
||||
from_port = var.node_port
|
||||
to_port = var.node_port
|
||||
ip_protocol = "tcp"
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-demo-nodeport"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Demo workload — deployed after CoreDNS so name resolution is up
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "kubernetes_deployment_v1" "hello" {
|
||||
metadata {
|
||||
name = "hello"
|
||||
namespace = "default"
|
||||
labels = { app = "hello" }
|
||||
}
|
||||
|
||||
spec {
|
||||
replicas = var.replicas
|
||||
|
||||
selector {
|
||||
match_labels = { app = "hello" }
|
||||
}
|
||||
|
||||
template {
|
||||
metadata {
|
||||
labels = { app = "hello" }
|
||||
}
|
||||
|
||||
spec {
|
||||
container {
|
||||
name = "hello"
|
||||
image = "nginxdemos/hello"
|
||||
|
||||
port {
|
||||
container_port = 80
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_addon.coredns]
|
||||
}
|
||||
|
||||
resource "kubernetes_service_v1" "hello" {
|
||||
metadata {
|
||||
name = "hello"
|
||||
namespace = "default"
|
||||
}
|
||||
|
||||
spec {
|
||||
selector = { app = "hello" }
|
||||
type = "NodePort"
|
||||
|
||||
port {
|
||||
port = 80
|
||||
target_port = 80
|
||||
node_port = var.node_port
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [kubernetes_deployment_v1.hello]
|
||||
}
|
||||
|
||||
data "aws_instances" "workers" {
|
||||
instance_tags = {
|
||||
"spinifex:eks-cluster" = aws_eks_cluster.this.name
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.default]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Outputs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
output "cluster_name" {
|
||||
value = aws_eks_cluster.this.name
|
||||
}
|
||||
|
||||
output "viewer_principal_arn" {
|
||||
value = aws_iam_role.viewer.arn
|
||||
}
|
||||
|
||||
output "demo_url" {
|
||||
value = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
|
||||
description = "Open in a browser; refresh to see different pods answer"
|
||||
}
|
||||
|
||||
output "update_kubeconfig" {
|
||||
value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
title: "3-Node EKS behind an HTTPS ALB (ACM TLS)"
|
||||
description: "Provision an HA-shaped EKS cluster with public + private API endpoints and three workers, deploy a demo app, and publish it through an internet-facing ALB that terminates TLS with an ACM-imported certificate, using Terraform on Spinifex."
|
||||
category: "Terraform Workbooks"
|
||||
tags:
|
||||
- terraform
|
||||
- eks
|
||||
- kubernetes
|
||||
- acm
|
||||
- tls
|
||||
- alb
|
||||
- elbv2
|
||||
- vpc
|
||||
- workbook
|
||||
resources:
|
||||
- title: "Terraform AWS Provider"
|
||||
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
|
||||
- title: "Terraform Kubernetes Provider"
|
||||
url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
|
||||
- title: "Spinifex Repository"
|
||||
url: "https://github.com/mulgadc/spinifex"
|
||||
- title: "OpenTofu"
|
||||
url: "https://opentofu.org/"
|
||||
---
|
||||
|
||||
# Terraform: 3-Node EKS behind an HTTPS ALB
|
||||
|
||||
> The largest EKS workbook, and the clearest demo. An HA-shaped cluster with public + private API endpoints and three workers, running a demo app published over HTTPS through an internet-facing ALB that terminates TLS with an ACM-imported certificate.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Instructions](#instructions)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This workbook is sized for a **3-node Spinifex deployment** and is built to show off. It brings together the pieces from the smaller EKS examples, deploys a demo app across three workers, and fronts it with a TLS-terminating load balancer. When `apply` finishes, fetch the ALB's public IP and open `https://<ip>` — the `nginxdemos/hello` page reports which pod answered, and refreshing moves between the three replicas spread across the cluster. That single page demonstrates a multi-node Kubernetes cluster load-balancing behind HTTPS.
|
||||
|
||||
- **Public + private API endpoints.** `endpoint_public_access` serves `kubectl` and the Kubernetes provider from your host (CIDR-restricted); `endpoint_private_access` exposes an in-VPC private endpoint the workers use to join.
|
||||
- **Private workers + NAT.** The three workers sit in private subnets. They *join* the cluster over the in-VPC private endpoint, and a NAT gateway gives them outbound access to pull the demo container image.
|
||||
- **CoreDNS** installed as a managed addon.
|
||||
- **TLS via ACM.** A self-signed certificate is generated with the `tls` provider and **imported into ACM** (Spinifex ACM supports `ImportCertificate`, not `RequestCertificate`). The ALB's HTTPS listener references the ACM ARN and forwards to the workers' NodePort.
|
||||
- **The app, deployed by Terraform.** The Kubernetes provider deploys the Deployment + NodePort Service, and one security-group rule lets the ALB reach the NodePort.
|
||||
|
||||
<p align="center">
|
||||
<img src="../../../.github/assets/diagrams/tf-eks-https-ingress.svg" alt="3-node EKS — public+private API endpoints, ALB in public subnets terminating TLS, three workers in private subnets behind a NAT gateway" width="900">
|
||||
</p>
|
||||
|
||||
**What you'll learn:**
|
||||
|
||||
- Running an EKS cluster with both public and private API endpoints
|
||||
- Placing workers in private subnets that join over the private endpoint and egress via NAT
|
||||
- Importing a certificate into ACM and terminating TLS on an ALB HTTPS listener
|
||||
- Wiring an ALB target group to managed node-group workers via a NodePort, including the SG rule that allows it
|
||||
- Deploying a load-balanced workload with the Kubernetes provider
|
||||
|
||||
**What gets created**
|
||||
|
||||
| Resource | Name | Purpose |
|
||||
|---|---|---|
|
||||
| VPC | `eks-https-vpc` | Isolated network (10.32.0.0/16) |
|
||||
| Public Subnets | `eks-https-public-a/-b` | Host the ALB and NAT gateway |
|
||||
| Private Subnets | `eks-https-private-a/-b` | Host the workers |
|
||||
| Internet Gateway | `eks-https-igw` | Internet for the public subnets |
|
||||
| NAT Gateway (+ EIP) | `eks-https-nat` | Outbound for the private workers (image pulls) |
|
||||
| Route Tables | `eks-https-public-rt`, `eks-https-private-rt` | Public via IGW; private via NAT |
|
||||
| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
|
||||
| EKS Cluster | `eks-https` | Public (CIDR-restricted) + private endpoints |
|
||||
| Node Group | `workers` | Three `t3.large` workers in the private subnets |
|
||||
| Addon | `coredns` | Cluster DNS |
|
||||
| ACM Cert | `eks-https-ingress` | Self-signed cert imported into ACM |
|
||||
| ALB + HTTPS Listener | `eks-https-alb` | Internet-facing, terminates TLS on :443 |
|
||||
| Target Group | `eks-https-tg` | Instance targets on the NodePort |
|
||||
| SG Ingress Rule | `eks-https-alb-nodeport` | Lets the ALB SG reach the worker NodePort |
|
||||
| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 3 replicas, NodePort |
|
||||
|
||||
**Spinifex specifics**
|
||||
|
||||
- ACM supports `ImportCertificate`, `DescribeCertificate`, `ListCertificates`, `DeleteCertificate` — there is no `RequestCertificate`/DNS-validation flow, hence the self-signed import.
|
||||
- The ALB HTTPS listener resolves the cert from ACM. Valid SSL policies are `ELBSecurityPolicy-2016-08` (default) and `ELBSecurityPolicy-TLS13-1-2-2021-06`.
|
||||
- The worker SG is auto-managed (`vpc_config.security_group_ids` is ignored) and admits only intra-cluster traffic. The ALB→NodePort rule is added by looking the SG up by its deterministic name and admitting the ALB SG.
|
||||
- Worker discovery uses the `spinifex:eks-cluster` tag Spinifex stamps on node-group instances; the target-group attachment count is the known `node_desired_size`.
|
||||
- Leave `addon_version` unset (see the addons workbook for why pinning it can't satisfy both the provider and the catalog).
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
- A 3-node Spinifex deployment, installed and running
|
||||
- The Spinifex `eks-node` image available on the cluster
|
||||
- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
|
||||
|
||||
## Instructions
|
||||
|
||||
### Step 1. Get the Template
|
||||
|
||||
```bash
|
||||
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
|
||||
cd spinifex-tf
|
||||
git sparse-checkout set docs/terraform-workbooks
|
||||
cd docs/terraform-workbooks/eks-https-ingress
|
||||
```
|
||||
|
||||
Or create a `main.tf` file and paste the full configuration below.
|
||||
|
||||
<!-- INCLUDE: main.tf lang:hcl -->
|
||||
|
||||
### Step 2. Deploy
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
tofu init
|
||||
tofu apply
|
||||
```
|
||||
|
||||
Expect several minutes: the control plane bootstraps, three workers launch and join over the private endpoint, the NAT gateway comes up, CoreDNS installs, the ALB is provisioned, and the Kubernetes provider rolls out the demo app.
|
||||
|
||||
> **Keep `AWS_PROFILE=spinifex` exported** for the whole `apply` — the Kubernetes provider authenticates with `aws eks get-token` against the Spinifex STS endpoint.
|
||||
|
||||
> **Tighten access in production.** `api_public_access_cidr` and `alb_ingress_cidr` both default to `0.0.0.0/0`. Set them to your own CIDR:
|
||||
>
|
||||
> ```bash
|
||||
> export TF_VAR_api_public_access_cidr="203.0.113.10/32"
|
||||
> export TF_VAR_alb_ingress_cidr="203.0.113.10/32"
|
||||
> ```
|
||||
|
||||
### Step 3. Open the Demo over HTTPS
|
||||
|
||||
The ALB DNS name (`*.elb.spinifex.local`) won't resolve from your host, so fetch its public IP:
|
||||
|
||||
```bash
|
||||
ALB_IP=$(aws elbv2 describe-load-balancers --names eks-https-alb \
|
||||
--query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
|
||||
--output text)
|
||||
```
|
||||
|
||||
The certificate is self-signed, so a browser will warn (proceed anyway) and `curl` needs `-k`:
|
||||
|
||||
```bash
|
||||
curl -k https://$ALB_IP
|
||||
curl -k https://$ALB_IP # refresh: the "Server name" changes between pods
|
||||
```
|
||||
|
||||
Open `https://$ALB_IP` in a browser and refresh to watch requests land on different pods across the three nodes.
|
||||
|
||||
### Step 4. Inspect (optional)
|
||||
|
||||
```bash
|
||||
aws eks update-kubeconfig --name eks-https --region ap-southeast-2
|
||||
kubectl get nodes
|
||||
kubectl get pods -o wide # three replicas, one per node
|
||||
|
||||
TG_ARN=$(aws elbv2 describe-target-groups --names eks-https-tg \
|
||||
--query 'TargetGroups[0].TargetGroupArn' --output text)
|
||||
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
|
||||
```
|
||||
|
||||
### Cleanup
|
||||
|
||||
```bash
|
||||
tofu destroy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Targets Unhealthy / ALB Returns 5xx
|
||||
|
||||
The ALB health-checks `/` on the NodePort. Targets turn healthy once the demo pods are running *and* the SG rule admits the ALB. Check, in order:
|
||||
|
||||
```bash
|
||||
kubectl get pods -o wide # pods Running?
|
||||
aws ec2 describe-security-groups \
|
||||
--filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
|
||||
--query 'SecurityGroups[0].IpPermissions' # NodePort rule present?
|
||||
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
|
||||
```
|
||||
|
||||
If pods are `ImagePullBackOff`, the workers can't reach Docker Hub — confirm the NAT gateway is `available`:
|
||||
|
||||
```bash
|
||||
aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
|
||||
```
|
||||
|
||||
### Workers Never Join
|
||||
|
||||
The workers join over the private endpoint. Confirm it's enabled and the node group is progressing:
|
||||
|
||||
```bash
|
||||
aws eks describe-cluster --name eks-https --query 'cluster.resourcesVpcConfig.endpointPrivateAccess'
|
||||
aws eks describe-nodegroup --cluster-name eks-https --nodegroup-name workers --query 'nodegroup.status'
|
||||
```
|
||||
|
||||
### Target-Group Attachment Errors at Apply
|
||||
|
||||
The attachment indexes the discovered worker IDs by `node_desired_size`. If `apply` errors with an index out of range, the workers weren't all tagged/running when the `aws_instances` lookup ran. Re-run `tofu apply` — the node group will be `ACTIVE` by then:
|
||||
|
||||
```bash
|
||||
aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-https" \
|
||||
--query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text
|
||||
```
|
||||
|
||||
### kubernetes provider: connection refused / Unauthorized
|
||||
|
||||
The provider runs `aws eks get-token` against the Spinifex STS endpoint. If the cluster was still `CREATING` when the provider first connected, re-run `tofu apply`. Otherwise confirm:
|
||||
|
||||
```bash
|
||||
aws sts get-caller-identity
|
||||
aws eks describe-cluster --name eks-https --query 'cluster.status'
|
||||
```
|
||||
|
||||
### ACM Import / HTTPS Listener Errors
|
||||
|
||||
```bash
|
||||
aws acm list-certificates
|
||||
aws acm describe-certificate --certificate-arn "$(tofu output -raw certificate_arn)" \
|
||||
--query 'Certificate.Status'
|
||||
```
|
||||
|
||||
A `CertificateNotFound` on the listener means the ACM ARN didn't resolve — check the cert exists and belongs to the same account/profile.
|
||||
|
||||
### Provider Connection Refused
|
||||
|
||||
```bash
|
||||
sudo systemctl status spinifex.target
|
||||
curl -k https://localhost:9999/
|
||||
```
|
||||
@@ -0,0 +1,671 @@
|
||||
# Example: 3-Node EKS behind an HTTPS ALB (ACM TLS) with a load-balanced app
|
||||
#
|
||||
# The largest EKS workbook, and the one with the clearest payoff. It stands up
|
||||
# an HA-shaped cluster sized for a 3-node Spinifex deployment, deploys a demo
|
||||
# web app across the workers, and publishes it through an internet-facing ALB
|
||||
# that terminates TLS with a certificate held in ACM.
|
||||
#
|
||||
# After `apply`, fetch the ALB's public IP (see alb_public_ip_hint) and open
|
||||
# https://<ip> in a browser. The page reports which pod answered; refresh and
|
||||
# it moves between the three replicas spread across the cluster — a live picture
|
||||
# of a multi-node Kubernetes cluster load-balancing behind HTTPS.
|
||||
#
|
||||
# What it builds:
|
||||
# * VPC with two public subnets (ALB + NAT) and two private subnets (workers).
|
||||
# * A NAT gateway so the private workers can pull the demo container image.
|
||||
# * EKS cluster with BOTH a public endpoint (kubectl from your host) and a
|
||||
# private endpoint (the in-VPC path the workers use to join).
|
||||
# * A managed node group of three workers (t3.large) in the private subnets.
|
||||
# * CoreDNS addon.
|
||||
# * A self-signed cert imported into ACM (Spinifex ACM supports
|
||||
# ImportCertificate, not RequestCertificate), wired to the ALB's HTTPS
|
||||
# listener, which forwards to the workers' NodePort.
|
||||
# * The demo Deployment + NodePort Service, deployed by the Kubernetes
|
||||
# provider, plus the one SG rule that lets the ALB reach the NodePort.
|
||||
#
|
||||
# Usage:
|
||||
# cd spinifex/docs/terraform-workbooks/eks-https-ingress
|
||||
# export AWS_PROFILE=spinifex
|
||||
# tofu init && tofu apply
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.6.0"
|
||||
|
||||
required_providers {
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = ">= 5.40, < 6.0"
|
||||
}
|
||||
tls = {
|
||||
source = "hashicorp/tls"
|
||||
version = ">= 4.0"
|
||||
}
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = ">= 2.20"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Variables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
default = "ap-southeast-2"
|
||||
}
|
||||
|
||||
variable "cluster_name" {
|
||||
type = string
|
||||
default = "eks-https"
|
||||
}
|
||||
|
||||
variable "k8s_version" {
|
||||
type = string
|
||||
default = "1.32"
|
||||
}
|
||||
|
||||
variable "node_instance_type" {
|
||||
type = string
|
||||
default = "t3.large"
|
||||
}
|
||||
|
||||
variable "node_desired_size" {
|
||||
type = number
|
||||
default = 3
|
||||
description = "Worker count; spread across a 3-node Spinifex deployment for HA"
|
||||
}
|
||||
|
||||
variable "node_port" {
|
||||
type = number
|
||||
default = 30080
|
||||
description = "NodePort the ALB forwards to and the demo Service is published on"
|
||||
}
|
||||
|
||||
variable "cert_common_name" {
|
||||
type = string
|
||||
default = "eks-https.spinifex.local"
|
||||
}
|
||||
|
||||
variable "api_public_access_cidr" {
|
||||
type = string
|
||||
default = "0.0.0.0/0"
|
||||
description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
|
||||
}
|
||||
|
||||
variable "alb_ingress_cidr" {
|
||||
type = string
|
||||
default = "0.0.0.0/0"
|
||||
description = "CIDR allowed to reach the ALB HTTPS listener"
|
||||
}
|
||||
|
||||
variable "spinifex_endpoint" {
|
||||
type = string
|
||||
default = "https://127.0.0.1:9999"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Providers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
provider "aws" {
|
||||
region = var.region
|
||||
|
||||
endpoints {
|
||||
ec2 = var.spinifex_endpoint
|
||||
iam = var.spinifex_endpoint
|
||||
sts = var.spinifex_endpoint
|
||||
eks = var.spinifex_endpoint
|
||||
acm = var.spinifex_endpoint
|
||||
elasticloadbalancingv2 = var.spinifex_endpoint
|
||||
}
|
||||
|
||||
skip_credentials_validation = true
|
||||
skip_metadata_api_check = true
|
||||
skip_requesting_account_id = true
|
||||
skip_region_validation = true
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
host = aws_eks_cluster.this.endpoint
|
||||
cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
|
||||
|
||||
exec {
|
||||
api_version = "client.authentication.k8s.io/v1beta1"
|
||||
command = "aws"
|
||||
args = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
|
||||
}
|
||||
}
|
||||
|
||||
data "aws_availability_zones" "available" {
|
||||
state = "available"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# VPC — public subnets for the ALB + NAT, private subnets for the workers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_vpc" "main" {
|
||||
cidr_block = "10.32.0.0/16"
|
||||
enable_dns_hostnames = true
|
||||
enable_dns_support = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-vpc"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_internet_gateway" "igw" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-igw"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "public_a" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.32.1.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
map_public_ip_on_launch = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-public-a"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "public_b" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.32.2.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
map_public_ip_on_launch = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-public-b"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "private_a" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.32.11.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-private-a"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "private_b" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.32.12.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-private-b"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table" "public" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
route {
|
||||
cidr_block = "0.0.0.0/0"
|
||||
gateway_id = aws_internet_gateway.igw.id
|
||||
}
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-public-rt"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "public_a" {
|
||||
subnet_id = aws_subnet.public_a.id
|
||||
route_table_id = aws_route_table.public.id
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "public_b" {
|
||||
subnet_id = aws_subnet.public_b.id
|
||||
route_table_id = aws_route_table.public.id
|
||||
}
|
||||
|
||||
# NAT gateway gives the private workers outbound internet to pull the demo
|
||||
# container image. The workers still *join* the cluster over the in-VPC private
|
||||
# endpoint, not through the NAT.
|
||||
resource "aws_eip" "nat" {
|
||||
domain = "vpc"
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-nat-eip"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_nat_gateway" "nat" {
|
||||
allocation_id = aws_eip.nat.id
|
||||
subnet_id = aws_subnet.public_a.id
|
||||
|
||||
depends_on = [aws_internet_gateway.igw]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-nat"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table" "private" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
route {
|
||||
cidr_block = "0.0.0.0/0"
|
||||
nat_gateway_id = aws_nat_gateway.nat.id
|
||||
}
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-private-rt"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "private_a" {
|
||||
subnet_id = aws_subnet.private_a.id
|
||||
route_table_id = aws_route_table.private.id
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "private_b" {
|
||||
subnet_id = aws_subnet.private_b.id
|
||||
route_table_id = aws_route_table.private.id
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — cluster role
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "cluster" {
|
||||
name = "${var.cluster_name}-cluster-role"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "eks.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "cluster" {
|
||||
role = aws_iam_role.cluster.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — node role
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "node" {
|
||||
name = "${var.cluster_name}-node-role"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "ec2.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_worker" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_cni" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_ecr" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# EKS cluster — public + private endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_cluster" "this" {
|
||||
name = var.cluster_name
|
||||
role_arn = aws_iam_role.cluster.arn
|
||||
version = var.k8s_version
|
||||
|
||||
vpc_config {
|
||||
subnet_ids = [aws_subnet.private_a.id, aws_subnet.private_b.id]
|
||||
endpoint_public_access = true
|
||||
endpoint_private_access = true
|
||||
public_access_cidrs = [var.api_public_access_cidr]
|
||||
}
|
||||
|
||||
access_config {
|
||||
authentication_mode = "API"
|
||||
bootstrap_cluster_creator_admin_permissions = true
|
||||
}
|
||||
|
||||
depends_on = [aws_iam_role_policy_attachment.cluster]
|
||||
|
||||
tags = {
|
||||
Name = var.cluster_name
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Managed node group — three workers in the private subnets
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_node_group" "workers" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
node_group_name = "workers"
|
||||
node_role_arn = aws_iam_role.node.arn
|
||||
subnet_ids = [aws_subnet.private_a.id, aws_subnet.private_b.id]
|
||||
|
||||
scaling_config {
|
||||
desired_size = var.node_desired_size
|
||||
min_size = var.node_desired_size
|
||||
max_size = var.node_desired_size * 2
|
||||
}
|
||||
|
||||
instance_types = [var.node_instance_type]
|
||||
ami_type = "AL2_x86_64"
|
||||
|
||||
depends_on = [
|
||||
aws_iam_role_policy_attachment.node_worker,
|
||||
aws_iam_role_policy_attachment.node_cni,
|
||||
aws_iam_role_policy_attachment.node_ecr,
|
||||
]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-workers"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Addon — CoreDNS
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_addon" "coredns" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
addon_name = "coredns"
|
||||
resolve_conflicts_on_create = "OVERWRITE"
|
||||
|
||||
# addon_version omitted on purpose. Spinifex defaults an unset version to its
|
||||
# catalog default (coredns 1.11.1). The AWS provider also rejects a bare
|
||||
# "1.11.1" (it demands a v-prefixed form) which Spinifex's catalog would in
|
||||
# turn reject — so pinning is a dead end here; let the server choose.
|
||||
depends_on = [aws_eks_node_group.workers]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-coredns"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TLS — self-signed certificate imported into ACM
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "tls_private_key" "ingress" {
|
||||
algorithm = "RSA"
|
||||
rsa_bits = 2048
|
||||
}
|
||||
|
||||
resource "tls_self_signed_cert" "ingress" {
|
||||
private_key_pem = tls_private_key.ingress.private_key_pem
|
||||
|
||||
subject {
|
||||
common_name = var.cert_common_name
|
||||
organization = "Spinifex EKS Demo"
|
||||
}
|
||||
|
||||
dns_names = [var.cert_common_name]
|
||||
validity_period_hours = 8760
|
||||
early_renewal_hours = 720
|
||||
|
||||
allowed_uses = [
|
||||
"key_encipherment",
|
||||
"digital_signature",
|
||||
"server_auth",
|
||||
]
|
||||
}
|
||||
|
||||
resource "aws_acm_certificate" "ingress" {
|
||||
private_key = tls_private_key.ingress.private_key_pem
|
||||
certificate_body = tls_self_signed_cert.ingress.cert_pem
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-ingress"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ALB — internet-facing, HTTPS terminated with the ACM cert
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_security_group" "alb" {
|
||||
name = "${var.cluster_name}-alb-sg"
|
||||
description = "ALB HTTPS inbound, NodePort to workers"
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
ingress {
|
||||
description = "HTTPS"
|
||||
from_port = 443
|
||||
to_port = 443
|
||||
protocol = "tcp"
|
||||
cidr_blocks = [var.alb_ingress_cidr]
|
||||
}
|
||||
|
||||
egress {
|
||||
description = "All outbound"
|
||||
from_port = 0
|
||||
to_port = 0
|
||||
protocol = "-1"
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-alb-sg"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_lb" "ingress" {
|
||||
name = "${var.cluster_name}-alb"
|
||||
internal = false
|
||||
load_balancer_type = "application"
|
||||
security_groups = [aws_security_group.alb.id]
|
||||
subnets = [aws_subnet.public_a.id, aws_subnet.public_b.id]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-alb"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_lb_target_group" "workers" {
|
||||
name = "${var.cluster_name}-tg"
|
||||
port = var.node_port
|
||||
protocol = "HTTP"
|
||||
target_type = "instance"
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
health_check {
|
||||
path = "/"
|
||||
protocol = "HTTP"
|
||||
matcher = "200-399"
|
||||
healthy_threshold = 2
|
||||
unhealthy_threshold = 3
|
||||
timeout = 5
|
||||
interval = 10
|
||||
}
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-tg"
|
||||
}
|
||||
}
|
||||
|
||||
# Discover the launched workers by their Spinifex EKS tag so the ALB can target
|
||||
# them. depends_on the node group: the lookup must run after the workers exist.
|
||||
data "aws_instances" "workers" {
|
||||
instance_tags = {
|
||||
"spinifex:eks-cluster" = aws_eks_cluster.this.name
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.workers]
|
||||
}
|
||||
|
||||
# count is the known desired size, so it's resolvable at plan time even though
|
||||
# the worker IDs themselves are only known after the node group launches.
|
||||
resource "aws_lb_target_group_attachment" "workers" {
|
||||
count = var.node_desired_size
|
||||
target_group_arn = aws_lb_target_group.workers.arn
|
||||
target_id = data.aws_instances.workers.ids[count.index]
|
||||
port = var.node_port
|
||||
}
|
||||
|
||||
resource "aws_lb_listener" "https" {
|
||||
load_balancer_arn = aws_lb.ingress.arn
|
||||
port = 443
|
||||
protocol = "HTTPS"
|
||||
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"
|
||||
certificate_arn = aws_acm_certificate.ingress.arn
|
||||
|
||||
default_action {
|
||||
type = "forward"
|
||||
target_group_arn = aws_lb_target_group.workers.arn
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Let the ALB reach the workers' NodePort
|
||||
#
|
||||
# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
|
||||
# Look it up by its deterministic name and admit the ALB SG on the NodePort.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
data "aws_security_group" "nodegroup" {
|
||||
filter {
|
||||
name = "group-name"
|
||||
values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
|
||||
}
|
||||
|
||||
filter {
|
||||
name = "vpc-id"
|
||||
values = [aws_vpc.main.id]
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.workers]
|
||||
}
|
||||
|
||||
resource "aws_vpc_security_group_ingress_rule" "nodeport_from_alb" {
|
||||
security_group_id = data.aws_security_group.nodegroup.id
|
||||
referenced_security_group_id = aws_security_group.alb.id
|
||||
from_port = var.node_port
|
||||
to_port = var.node_port
|
||||
ip_protocol = "tcp"
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-alb-nodeport"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Demo workload — three replicas spread across the workers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "kubernetes_deployment_v1" "hello" {
|
||||
metadata {
|
||||
name = "hello"
|
||||
namespace = "default"
|
||||
labels = { app = "hello" }
|
||||
}
|
||||
|
||||
spec {
|
||||
replicas = var.node_desired_size
|
||||
|
||||
selector {
|
||||
match_labels = { app = "hello" }
|
||||
}
|
||||
|
||||
template {
|
||||
metadata {
|
||||
labels = { app = "hello" }
|
||||
}
|
||||
|
||||
spec {
|
||||
container {
|
||||
name = "hello"
|
||||
image = "nginxdemos/hello"
|
||||
|
||||
port {
|
||||
container_port = 80
|
||||
}
|
||||
}
|
||||
|
||||
# Spread one replica per node so a refresh visibly hits different nodes.
|
||||
topology_spread_constraint {
|
||||
max_skew = 1
|
||||
topology_key = "kubernetes.io/hostname"
|
||||
when_unsatisfiable = "ScheduleAnyway"
|
||||
|
||||
label_selector {
|
||||
match_labels = { app = "hello" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_addon.coredns]
|
||||
}
|
||||
|
||||
resource "kubernetes_service_v1" "hello" {
|
||||
metadata {
|
||||
name = "hello"
|
||||
namespace = "default"
|
||||
}
|
||||
|
||||
spec {
|
||||
selector = { app = "hello" }
|
||||
type = "NodePort"
|
||||
|
||||
port {
|
||||
port = 80
|
||||
target_port = 80
|
||||
node_port = var.node_port
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [kubernetes_deployment_v1.hello]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Outputs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
output "cluster_name" {
|
||||
value = aws_eks_cluster.this.name
|
||||
}
|
||||
|
||||
output "certificate_arn" {
|
||||
value = aws_acm_certificate.ingress.arn
|
||||
}
|
||||
|
||||
output "alb_dns_name" {
|
||||
value = aws_lb.ingress.dns_name
|
||||
}
|
||||
|
||||
output "alb_public_ip_hint" {
|
||||
value = "aws elbv2 describe-load-balancers --names ${var.cluster_name}-alb --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' --output text"
|
||||
}
|
||||
|
||||
output "demo_url_hint" {
|
||||
value = "Fetch the ALB IP via alb_public_ip_hint, then open https://<that-ip> (self-signed cert: curl -k). Refresh to see different pods answer."
|
||||
}
|
||||
|
||||
output "update_kubeconfig" {
|
||||
value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
title: "EKS Quickstart"
|
||||
description: "Stand up a minimal managed-Kubernetes cluster and a browsable demo app — VPC, IAM roles, an EKS cluster, a single-worker node group, and a load-balanced web page — using Terraform on Spinifex."
|
||||
category: "Terraform Workbooks"
|
||||
tags:
|
||||
- terraform
|
||||
- eks
|
||||
- kubernetes
|
||||
- iam
|
||||
- vpc
|
||||
- workbook
|
||||
resources:
|
||||
- title: "Terraform AWS Provider"
|
||||
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
|
||||
- title: "Terraform Kubernetes Provider"
|
||||
url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
|
||||
- title: "Spinifex Repository"
|
||||
url: "https://github.com/mulgadc/spinifex"
|
||||
- title: "OpenTofu"
|
||||
url: "https://opentofu.org/"
|
||||
---
|
||||
|
||||
# Terraform: EKS Quickstart
|
||||
|
||||
> The smallest end-to-end EKS example on Spinifex: a VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app you can open in a browser.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Instructions](#instructions)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook deploys a small demo app onto it and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page is served by `nginxdemos/hello` and prints the name of the pod that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.
|
||||
|
||||
Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a single-node `aws_eks_node_group`. Terraform then uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.
|
||||
|
||||
**What you'll learn:**
|
||||
|
||||
- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
|
||||
- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
|
||||
- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
|
||||
- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
|
||||
- Opening one rule on the auto-managed worker security group to expose a NodePort
|
||||
|
||||
**What gets created**
|
||||
|
||||
| Resource | Name | Purpose |
|
||||
|---|---|---|
|
||||
| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
|
||||
| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
|
||||
| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
|
||||
| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
|
||||
| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
|
||||
| Node Group | `default` | One `t3.medium` worker |
|
||||
| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
|
||||
| K8s Deployment | `hello` | `nginxdemos/hello`, 2 replicas |
|
||||
| K8s Service | `hello` | NodePort publishing the demo on the worker |
|
||||
|
||||
**Spinifex specifics**
|
||||
|
||||
- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
|
||||
- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
|
||||
- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
|
||||
- The workers need outbound internet (here via the IGW) to pull the demo container image.
|
||||
|
||||
**Prerequisites:**
|
||||
|
||||
- Spinifex installed and running (see [Installing Spinifex](/docs/install))
|
||||
- The Spinifex `eks-node` image available on the cluster
|
||||
- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
|
||||
|
||||
## Instructions
|
||||
|
||||
### Step 1. Get the Template
|
||||
|
||||
```bash
|
||||
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
|
||||
cd spinifex-tf
|
||||
git sparse-checkout set docs/terraform-workbooks
|
||||
cd docs/terraform-workbooks/eks-quickstart
|
||||
```
|
||||
|
||||
Or create a `main.tf` file and paste the full configuration below.
|
||||
|
||||
<!-- INCLUDE: main.tf lang:hcl -->
|
||||
|
||||
### Step 2. Deploy
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
tofu init
|
||||
tofu apply
|
||||
```
|
||||
|
||||
This single `apply` does the whole job: it creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker, opens the NodePort, then deploys the demo app and waits for the pods to roll out.
|
||||
|
||||
> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for the whole `apply`.
|
||||
|
||||
### Step 3. Open the Demo
|
||||
|
||||
```bash
|
||||
tofu output demo_url
|
||||
```
|
||||
|
||||
Open that URL in a browser. You'll see the `nginxdemos/hello` page showing the **Server name** (the pod that handled the request). Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.
|
||||
|
||||
### Step 4. Inspect with kubectl (optional)
|
||||
|
||||
```bash
|
||||
aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
|
||||
kubectl get nodes
|
||||
kubectl get pods -o wide
|
||||
```
|
||||
|
||||
`kubectl get pods -o wide` shows the demo pods and which node each landed on.
|
||||
|
||||
### Cleanup
|
||||
|
||||
```bash
|
||||
tofu destroy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Demo URL Doesn't Load
|
||||
|
||||
The page is served from a NodePort on the worker's public IP. Work through the chain:
|
||||
|
||||
```bash
|
||||
# Is the worker running with a public IP?
|
||||
aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
|
||||
--query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text
|
||||
|
||||
# Did the demo pods roll out?
|
||||
kubectl get pods -o wide
|
||||
|
||||
# Is the NodePort rule on the worker SG?
|
||||
aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
|
||||
--query 'SecurityGroups[0].IpPermissions'
|
||||
```
|
||||
|
||||
If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't reach Docker Hub; confirm the IGW route and the worker's public IP.
|
||||
|
||||
### kubernetes provider: connection refused / Unauthorized
|
||||
|
||||
The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:
|
||||
|
||||
```bash
|
||||
aws sts get-caller-identity
|
||||
aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
|
||||
```
|
||||
|
||||
If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.
|
||||
|
||||
### Cluster Stuck in CREATING
|
||||
|
||||
Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:
|
||||
|
||||
```bash
|
||||
aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
|
||||
aws ec2 describe-instances --profile spinifex
|
||||
```
|
||||
|
||||
### Provider Connection Refused
|
||||
|
||||
```bash
|
||||
sudo systemctl status spinifex.target
|
||||
curl -k https://localhost:9999/
|
||||
```
|
||||
@@ -0,0 +1,428 @@
|
||||
# Example: EKS Quickstart on Spinifex
|
||||
#
|
||||
# A minimal managed-Kubernetes cluster that ends in something you can see: a
|
||||
# VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app
|
||||
# deployed onto it by Terraform and exposed on the worker's public IP.
|
||||
#
|
||||
# Once `apply` finishes, open the demo_url output in a browser. The page is
|
||||
# served by the nginxdemos/hello image and prints the pod name that answered —
|
||||
# refresh and it bounces between the app's replicas, showing the cluster is
|
||||
# really scheduling and load-balancing pods.
|
||||
#
|
||||
# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
|
||||
# and managed aws_eks_node_group, the Kubernetes provider authenticating to the
|
||||
# cluster, and a Deployment + NodePort Service reachable from your browser.
|
||||
#
|
||||
# Usage:
|
||||
# cd spinifex/docs/terraform-workbooks/eks-quickstart
|
||||
# export AWS_PROFILE=spinifex
|
||||
# tofu init && tofu apply
|
||||
# # then open the demo_url output
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.6.0"
|
||||
|
||||
required_providers {
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = ">= 5.40, < 6.0"
|
||||
}
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = ">= 2.20"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Variables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
default = "ap-southeast-2"
|
||||
}
|
||||
|
||||
variable "cluster_name" {
|
||||
type = string
|
||||
default = "eks-quickstart"
|
||||
}
|
||||
|
||||
variable "k8s_version" {
|
||||
type = string
|
||||
default = "1.32"
|
||||
description = "Kubernetes minor version for the control plane and workers"
|
||||
}
|
||||
|
||||
variable "instance_type" {
|
||||
type = string
|
||||
default = "t3.medium"
|
||||
}
|
||||
|
||||
variable "node_port" {
|
||||
type = number
|
||||
default = 30080
|
||||
description = "NodePort the demo Service is published on"
|
||||
}
|
||||
|
||||
variable "replicas" {
|
||||
type = number
|
||||
default = 2
|
||||
description = "Demo app replicas; refresh the page to see requests land on different pods"
|
||||
}
|
||||
|
||||
variable "browse_cidr" {
|
||||
type = string
|
||||
default = "0.0.0.0/0"
|
||||
description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
|
||||
}
|
||||
|
||||
variable "spinifex_endpoint" {
|
||||
type = string
|
||||
default = "https://127.0.0.1:9999"
|
||||
description = "Spinifex AWS gateway endpoint"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Providers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
provider "aws" {
|
||||
region = var.region
|
||||
|
||||
endpoints {
|
||||
ec2 = var.spinifex_endpoint
|
||||
iam = var.spinifex_endpoint
|
||||
sts = var.spinifex_endpoint
|
||||
eks = var.spinifex_endpoint
|
||||
}
|
||||
|
||||
skip_credentials_validation = true
|
||||
skip_metadata_api_check = true
|
||||
skip_requesting_account_id = true
|
||||
skip_region_validation = true
|
||||
}
|
||||
|
||||
# Authenticates to the cluster with the same `aws eks get-token` exec flow the
|
||||
# generated kubeconfig uses, so the Kubernetes provider can deploy the demo app.
|
||||
provider "kubernetes" {
|
||||
host = aws_eks_cluster.this.endpoint
|
||||
cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
|
||||
|
||||
exec {
|
||||
api_version = "client.authentication.k8s.io/v1beta1"
|
||||
command = "aws"
|
||||
args = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data sources
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
data "aws_availability_zones" "available" {
|
||||
state = "available"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# VPC + two public subnets (workers get public IPs to pull the demo image)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_vpc" "main" {
|
||||
cidr_block = "10.30.0.0/16"
|
||||
enable_dns_hostnames = true
|
||||
enable_dns_support = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-vpc"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_internet_gateway" "igw" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-igw"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "a" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.30.1.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
map_public_ip_on_launch = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-subnet-a"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_subnet" "b" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
cidr_block = "10.30.2.0/24"
|
||||
availability_zone = data.aws_availability_zones.available.names[0]
|
||||
map_public_ip_on_launch = true
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-subnet-b"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table" "public" {
|
||||
vpc_id = aws_vpc.main.id
|
||||
|
||||
route {
|
||||
cidr_block = "0.0.0.0/0"
|
||||
gateway_id = aws_internet_gateway.igw.id
|
||||
}
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-public-rt"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "a" {
|
||||
subnet_id = aws_subnet.a.id
|
||||
route_table_id = aws_route_table.public.id
|
||||
}
|
||||
|
||||
resource "aws_route_table_association" "b" {
|
||||
subnet_id = aws_subnet.b.id
|
||||
route_table_id = aws_route_table.public.id
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — EKS cluster role
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "cluster" {
|
||||
name = "${var.cluster_name}-cluster-role"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "eks.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "cluster" {
|
||||
role = aws_iam_role.cluster.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IAM — worker node role
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_iam_role" "node" {
|
||||
name = "${var.cluster_name}-node-role"
|
||||
|
||||
assume_role_policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Effect = "Allow"
|
||||
Action = "sts:AssumeRole"
|
||||
Principal = { Service = "ec2.amazonaws.com" }
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_worker" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_cni" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy_attachment" "node_ecr" {
|
||||
role = aws_iam_role.node.name
|
||||
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# EKS cluster — public API endpoint, API (access-entry) auth mode
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_cluster" "this" {
|
||||
name = var.cluster_name
|
||||
role_arn = aws_iam_role.cluster.arn
|
||||
version = var.k8s_version
|
||||
|
||||
vpc_config {
|
||||
subnet_ids = [aws_subnet.a.id, aws_subnet.b.id]
|
||||
endpoint_public_access = true
|
||||
endpoint_private_access = false
|
||||
}
|
||||
|
||||
access_config {
|
||||
authentication_mode = "API"
|
||||
bootstrap_cluster_creator_admin_permissions = true
|
||||
}
|
||||
|
||||
depends_on = [aws_iam_role_policy_attachment.cluster]
|
||||
|
||||
tags = {
|
||||
Name = var.cluster_name
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Managed node group — one worker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "aws_eks_node_group" "default" {
|
||||
cluster_name = aws_eks_cluster.this.name
|
||||
node_group_name = "default"
|
||||
node_role_arn = aws_iam_role.node.arn
|
||||
subnet_ids = [aws_subnet.a.id, aws_subnet.b.id]
|
||||
|
||||
scaling_config {
|
||||
desired_size = 1
|
||||
min_size = 1
|
||||
max_size = 2
|
||||
}
|
||||
|
||||
instance_types = [var.instance_type]
|
||||
ami_type = "AL2_x86_64"
|
||||
|
||||
depends_on = [
|
||||
aws_iam_role_policy_attachment.node_worker,
|
||||
aws_iam_role_policy_attachment.node_cni,
|
||||
aws_iam_role_policy_attachment.node_ecr,
|
||||
]
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-default"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Open the NodePort on the auto-managed nodegroup SG
|
||||
#
|
||||
# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
|
||||
# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
|
||||
# from a browser, look the SG up by its deterministic name and add one rule.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
data "aws_security_group" "nodegroup" {
|
||||
filter {
|
||||
name = "group-name"
|
||||
values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
|
||||
}
|
||||
|
||||
filter {
|
||||
name = "vpc-id"
|
||||
values = [aws_vpc.main.id]
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.default]
|
||||
}
|
||||
|
||||
resource "aws_vpc_security_group_ingress_rule" "nodeport" {
|
||||
security_group_id = data.aws_security_group.nodegroup.id
|
||||
cidr_ipv4 = var.browse_cidr
|
||||
from_port = var.node_port
|
||||
to_port = var.node_port
|
||||
ip_protocol = "tcp"
|
||||
|
||||
tags = {
|
||||
Name = "${var.cluster_name}-demo-nodeport"
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Demo workload — deployed onto the cluster by Terraform
|
||||
#
|
||||
# nginxdemos/hello renders a page showing which pod served the request; with
|
||||
# multiple replicas, refreshing the demo_url alternates between them.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
resource "kubernetes_deployment_v1" "hello" {
|
||||
metadata {
|
||||
name = "hello"
|
||||
namespace = "default"
|
||||
labels = { app = "hello" }
|
||||
}
|
||||
|
||||
spec {
|
||||
replicas = var.replicas
|
||||
|
||||
selector {
|
||||
match_labels = { app = "hello" }
|
||||
}
|
||||
|
||||
template {
|
||||
metadata {
|
||||
labels = { app = "hello" }
|
||||
}
|
||||
|
||||
spec {
|
||||
container {
|
||||
name = "hello"
|
||||
image = "nginxdemos/hello"
|
||||
|
||||
port {
|
||||
container_port = 80
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.default]
|
||||
}
|
||||
|
||||
resource "kubernetes_service_v1" "hello" {
|
||||
metadata {
|
||||
name = "hello"
|
||||
namespace = "default"
|
||||
}
|
||||
|
||||
spec {
|
||||
selector = { app = "hello" }
|
||||
type = "NodePort"
|
||||
|
||||
port {
|
||||
port = 80
|
||||
target_port = 80
|
||||
node_port = var.node_port
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [kubernetes_deployment_v1.hello]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Discover the worker's public IP for the demo URL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
data "aws_instances" "workers" {
|
||||
instance_tags = {
|
||||
"spinifex:eks-cluster" = aws_eks_cluster.this.name
|
||||
}
|
||||
|
||||
depends_on = [aws_eks_node_group.default]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Outputs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
output "cluster_name" {
|
||||
value = aws_eks_cluster.this.name
|
||||
}
|
||||
|
||||
output "demo_url" {
|
||||
value = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
|
||||
description = "Open in a browser; refresh to see different pods answer"
|
||||
}
|
||||
|
||||
output "update_kubeconfig" {
|
||||
value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
|
||||
}
|
||||
@@ -24,6 +24,16 @@ import (
|
||||
// Compile-time check that Daemon implements SystemInstanceLauncher.
|
||||
var _ handlers_elbv2.SystemInstanceLauncher = (*Daemon)(nil)
|
||||
|
||||
// resolveENIAccount returns owner when set, else fallback. A cross-account ENI
|
||||
// (e.g. an EKS cluster NLB fronting a customer-VPC ENI) carries its own account;
|
||||
// same-account ENIs leave it empty and inherit the primary/system account.
|
||||
func resolveENIAccount(owner, fallback string) string {
|
||||
if owner != "" {
|
||||
return owner
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
// LaunchSystemInstance creates and starts a system-managed VM (ELBv2 LB).
|
||||
// The VM is owned by the system account (GlobalAccountID), is not visible to
|
||||
// customer DescribeInstances calls, and always boots via direct kernel boot
|
||||
@@ -36,10 +46,7 @@ func (d *Daemon) LaunchSystemInstance(input *handlers_elbv2.SystemInstanceInput)
|
||||
accountID := utils.GlobalAccountID
|
||||
// ENI account may differ from system account — the ENI is created under
|
||||
// the caller's account, so lookups/updates must use that account ID.
|
||||
eniAccountID := input.AccountID
|
||||
if eniAccountID == "" {
|
||||
eniAccountID = accountID
|
||||
}
|
||||
eniAccountID := resolveENIAccount(input.AccountID, accountID)
|
||||
|
||||
// Validate instance type
|
||||
instanceType, exists := d.resourceMgr.instanceTypes[input.InstanceType]
|
||||
@@ -127,7 +134,11 @@ func (d *Daemon) LaunchSystemInstance(input *handlers_elbv2.SystemInstanceInput)
|
||||
// for one of them (we clean up on failure below).
|
||||
for idx, extra := range input.ExtraENIs {
|
||||
if d.vpcService != nil {
|
||||
if _, attachErr := d.vpcService.AttachENI(eniAccountID, extra.ENIID, instance.ID, int64(idx+1)); attachErr != nil {
|
||||
// A cross-account extra ENI (e.g. an EKS cluster NLB fronting a
|
||||
// customer-VPC ENI) lives in its own account; the ENI record is
|
||||
// account-keyed, so attach under extra.AccountID when set.
|
||||
extraAccount := resolveENIAccount(extra.AccountID, eniAccountID)
|
||||
if _, attachErr := d.vpcService.AttachENI(extraAccount, extra.ENIID, instance.ID, int64(idx+1)); attachErr != nil {
|
||||
slog.Error("LaunchSystemInstance: failed to attach extra ENI", "eniId", extra.ENIID, "instanceId", instance.ID, "err", attachErr)
|
||||
d.cleanupFailedSystemInstance(instance, instanceType)
|
||||
return nil, fmt.Errorf("attach extra ENI %s: %w", extra.ENIID, attachErr)
|
||||
|
||||
@@ -156,6 +156,16 @@ func TestBuildNetcfgBlob_MultiNIC_DefaultFlags(t *testing.T) {
|
||||
// tapNameForNIC
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func TestResolveENIAccount_OwnerWhenSet(t *testing.T) {
|
||||
// A cross-account extra ENI attaches under its own account.
|
||||
assert.Equal(t, "999988887777", resolveENIAccount("999988887777", "000000000000"))
|
||||
}
|
||||
|
||||
func TestResolveENIAccount_FallsBackWhenEmpty(t *testing.T) {
|
||||
// A same-account extra ENI (empty AccountID) inherits the primary account.
|
||||
assert.Equal(t, "000000000000", resolveENIAccount("", "000000000000"))
|
||||
}
|
||||
|
||||
func TestTapNameForNIC_PrimaryENI(t *testing.T) {
|
||||
input := &handlers_elbv2.SystemInstanceInput{
|
||||
ENIID: "eni-0123456789abcdef0",
|
||||
|
||||
@@ -115,6 +115,60 @@ func (s *RouteTableServiceImpl) putRouteTable(accountID string, record *RouteTab
|
||||
return nil
|
||||
}
|
||||
|
||||
// rtbCASMaxRetries bounds optimistic-concurrency retries when a concurrent
|
||||
// writer wins the CAS race on a route table record. High enough to absorb the
|
||||
// parallel AssociateRouteTable calls Terraform fans out per route table.
|
||||
const rtbCASMaxRetries = 16
|
||||
|
||||
// mutateRouteTableCAS applies mutate to a route table under optimistic
|
||||
// concurrency: read with revision, mutate, then Update guarded by that
|
||||
// revision, retrying when a concurrent writer wins. A blind read-modify-Put
|
||||
// loses updates when callers associate several subnets with one route table at
|
||||
// once. mutate reports whether it changed the record; a false return commits
|
||||
// nothing.
|
||||
func (s *RouteTableServiceImpl) mutateRouteTableCAS(accountID, rtbID string, mutate func(*RouteTableRecord) (bool, error)) error {
|
||||
key := utils.AccountKey(accountID, rtbID)
|
||||
for range rtbCASMaxRetries {
|
||||
entry, err := s.rtbKV.Get(key)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrKeyNotFound) {
|
||||
return errors.New(awserrors.ErrorInvalidRouteTableIDNotFound)
|
||||
}
|
||||
slog.Error("Failed to read route table from KV", "routeTableId", rtbID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
var record RouteTableRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil {
|
||||
slog.Error("Corrupt route table record in KV", "routeTableId", rtbID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
changed, err := mutate(&record)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !changed {
|
||||
return nil
|
||||
}
|
||||
|
||||
data, err := json.Marshal(&record)
|
||||
if err != nil {
|
||||
slog.Error("Failed to marshal route table record", "routeTableId", rtbID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if _, err := s.rtbKV.Update(key, data, entry.Revision()); err != nil {
|
||||
if errors.Is(err, nats.ErrKeyExists) {
|
||||
continue // CAS conflict — another writer won, re-read and retry.
|
||||
}
|
||||
slog.Error("Failed to write route table to KV", "routeTableId", rtbID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// getVPCCidr looks up a VPC's CIDR block from the VPC KV bucket
|
||||
func (s *RouteTableServiceImpl) getVPCCidr(accountID, vpcID string) (string, error) {
|
||||
entry, err := s.vpcKV.Get(utils.AccountKey(accountID, vpcID))
|
||||
@@ -892,13 +946,21 @@ func (s *RouteTableServiceImpl) AssociateRouteTable(input *ec2.AssociateRouteTab
|
||||
}
|
||||
|
||||
assocID := utils.GenerateResourceID("rtbassoc")
|
||||
record.Associations = append(record.Associations, AssociationRecord{
|
||||
AssociationId: assocID,
|
||||
SubnetId: subnetID,
|
||||
Main: false,
|
||||
})
|
||||
|
||||
if err := s.putRouteTable(accountID, record); err != nil {
|
||||
if err := s.mutateRouteTableCAS(accountID, rtbID, func(rec *RouteTableRecord) (bool, error) {
|
||||
// Re-check same-table association under the fresh read: a concurrent
|
||||
// associate for this subnet may have landed since the precheck.
|
||||
for _, assoc := range rec.Associations {
|
||||
if assoc.SubnetId == subnetID && !assoc.Main {
|
||||
return false, errors.New(awserrors.ErrorResourceAlreadyAssociated)
|
||||
}
|
||||
}
|
||||
rec.Associations = append(rec.Associations, AssociationRecord{
|
||||
AssociationId: assocID,
|
||||
SubnetId: subnetID,
|
||||
Main: false,
|
||||
})
|
||||
return true, nil
|
||||
}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -971,14 +1033,21 @@ func (s *RouteTableServiceImpl) DisassociateRouteTable(input *ec2.DisassociateRo
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
for i, assoc := range record.Associations {
|
||||
for _, assoc := range record.Associations {
|
||||
if assoc.AssociationId == assocID {
|
||||
if assoc.Main {
|
||||
return nil, errors.New(awserrors.ErrorInvalidParameterValue)
|
||||
}
|
||||
departingSubnetID := assoc.SubnetId
|
||||
record.Associations = append(record.Associations[:i], record.Associations[i+1:]...)
|
||||
if err := s.putRouteTable(accountID, &record); err != nil {
|
||||
if err := s.mutateRouteTableCAS(accountID, record.RouteTableId, func(rec *RouteTableRecord) (bool, error) {
|
||||
for j, a := range rec.Associations {
|
||||
if a.AssociationId == assocID {
|
||||
rec.Associations = append(rec.Associations[:j], rec.Associations[j+1:]...)
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
return false, nil // already removed by a concurrent writer — idempotent
|
||||
}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package handlers_ec2_routetable
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -430,6 +431,44 @@ func TestAssociateRouteTable_DuplicateSubnet(t *testing.T) {
|
||||
assert.EqualError(t, err, awserrors.ErrorResourceAlreadyAssociated)
|
||||
}
|
||||
|
||||
// TestAssociateRouteTable_ConcurrentDistinctSubnets reproduces the env19
|
||||
// lost-update: Terraform associates two subnets with one route table in
|
||||
// parallel. A blind read-modify-Put kept only one association (the other raced
|
||||
// from the same revision and was clobbered, so the provider timed out waiting
|
||||
// for 'associated'); the CAS path must persist both.
|
||||
func TestAssociateRouteTable_ConcurrentDistinctSubnets(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
rtbID := createTestRtb(t, svc)
|
||||
|
||||
subnets := []string{"subnet-test1", "subnet-priv1"}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
errs := make([]error, len(subnets))
|
||||
for i, sn := range subnets {
|
||||
wg.Add(1)
|
||||
go func(i int, sn string) {
|
||||
defer wg.Done()
|
||||
_, errs[i] = svc.AssociateRouteTable(&ec2.AssociateRouteTableInput{
|
||||
RouteTableId: aws.String(rtbID),
|
||||
SubnetId: aws.String(sn),
|
||||
}, testAccountID)
|
||||
}(i, sn)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
for i, err := range errs {
|
||||
require.NoErrorf(t, err, "associate %s", subnets[i])
|
||||
}
|
||||
|
||||
record, err := svc.getRouteTable(testAccountID, rtbID)
|
||||
require.NoError(t, err)
|
||||
got := make([]string, 0, len(record.Associations))
|
||||
for _, a := range record.Associations {
|
||||
got = append(got, a.SubnetId)
|
||||
}
|
||||
assert.ElementsMatch(t, subnets, got, "all concurrently-associated subnets must persist")
|
||||
}
|
||||
|
||||
func TestDisassociateRouteTable(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
rtbID := createTestRtb(t, svc)
|
||||
|
||||
@@ -795,6 +795,11 @@ func (s *VPCServiceImpl) RevokeSecurityGroupIngress(input *ec2.RevokeSecurityGro
|
||||
slog.Warn("RevokeSecurityGroupIngress: invalid rule", "groupId", groupId, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorInvalidParameterValue)
|
||||
}
|
||||
idRules, err := resolveRuleIDsToRemove(record.IngressRules, input.SecurityGroupRuleIds)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
revokeRules = append(revokeRules, idRules...)
|
||||
if len(revokeRules) == 0 {
|
||||
return &ec2.RevokeSecurityGroupIngressOutput{Return: aws.Bool(true)}, nil
|
||||
}
|
||||
@@ -852,6 +857,11 @@ func (s *VPCServiceImpl) RevokeSecurityGroupEgress(input *ec2.RevokeSecurityGrou
|
||||
slog.Warn("RevokeSecurityGroupEgress: invalid rule", "groupId", groupId, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorInvalidParameterValue)
|
||||
}
|
||||
idRules, err := resolveRuleIDsToRemove(record.EgressRules, input.SecurityGroupRuleIds)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
revokeRules = append(revokeRules, idRules...)
|
||||
if len(revokeRules) == 0 {
|
||||
return &ec2.RevokeSecurityGroupEgressOutput{Return: aws.Bool(true)}, nil
|
||||
}
|
||||
@@ -1070,6 +1080,38 @@ func sgRuleKey(r SGRule) string {
|
||||
return fmt.Sprintf("%s:%d:%d:%s:%s", r.IpProtocol, r.FromPort, r.ToPort, r.CidrIp, r.SourceSG)
|
||||
}
|
||||
|
||||
// resolveRuleIDsToRemove maps SecurityGroupRuleIds to the matching rules in the
|
||||
// SG's existing set. Modern Terraform (aws_vpc_security_group_ingress_rule)
|
||||
// revokes per-rule resources by SecurityGroupRuleId with empty IpPermissions;
|
||||
// without this the revoke silently no-ops and the leftover rule pins the SG
|
||||
// undeletable (DependencyViolation) when it references a peer SG.
|
||||
func resolveRuleIDsToRemove(existing []SGRule, ruleIDs []*string) ([]SGRule, error) {
|
||||
if len(ruleIDs) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
byID := make(map[string]SGRule, len(existing))
|
||||
for _, r := range existing {
|
||||
if r.RuleId != "" {
|
||||
byID[r.RuleId] = r
|
||||
}
|
||||
}
|
||||
out := make([]SGRule, 0, len(ruleIDs))
|
||||
for _, idp := range ruleIDs {
|
||||
if idp == nil || *idp == "" {
|
||||
continue
|
||||
}
|
||||
if !SGRuleIDRegex.MatchString(*idp) {
|
||||
return nil, errors.New(awserrors.ErrorInvalidSecurityGroupRuleIdMalformed)
|
||||
}
|
||||
r, ok := byID[*idp]
|
||||
if !ok {
|
||||
return nil, errors.New(awserrors.ErrorInvalidSecurityGroupRuleIdNotFound)
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// vpcdSGEventTimeout bounds the synchronous vpcd round-trip for SG events.
|
||||
const vpcdSGEventTimeout = 5 * time.Second
|
||||
|
||||
|
||||
@@ -417,6 +417,128 @@ func TestRevokeSecurityGroupEgress_RuleNotFound(t *testing.T) {
|
||||
assert.Contains(t, err.Error(), "InvalidPermission.NotFound")
|
||||
}
|
||||
|
||||
// --- Revoke by SecurityGroupRuleId (Terraform per-rule destroy) ---
|
||||
|
||||
func authorizeIngressFromSG(t *testing.T, svc *VPCServiceImpl, sgID, srcSG string) {
|
||||
t.Helper()
|
||||
_, err := svc.AuthorizeSecurityGroupIngress(&ec2.AuthorizeSecurityGroupIngressInput{
|
||||
GroupId: aws.String(sgID),
|
||||
IpPermissions: []*ec2.IpPermission{{
|
||||
IpProtocol: aws.String("tcp"),
|
||||
FromPort: aws.Int64(443),
|
||||
ToPort: aws.Int64(443),
|
||||
UserIdGroupPairs: []*ec2.UserIdGroupPair{{GroupId: aws.String(srcSG)}},
|
||||
}},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func ingressRuleIDReferencing(t *testing.T, svc *VPCServiceImpl, sgID, srcSG string) string {
|
||||
t.Helper()
|
||||
out, err := svc.DescribeSecurityGroupRules(&ec2.DescribeSecurityGroupRulesInput{
|
||||
Filters: []*ec2.Filter{{Name: aws.String("group-id"), Values: []*string{aws.String(sgID)}}},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
for _, r := range out.SecurityGroupRules {
|
||||
if r.ReferencedGroupInfo != nil && aws.StringValue(r.ReferencedGroupInfo.GroupId) == srcSG {
|
||||
return aws.StringValue(r.SecurityGroupRuleId)
|
||||
}
|
||||
}
|
||||
t.Fatalf("no ingress rule on %s referencing %s", sgID, srcSG)
|
||||
return ""
|
||||
}
|
||||
|
||||
// TestRevokeSecurityGroupIngress_ByRuleID_BreaksMutualReferenceTeardown
|
||||
// reproduces a Terraform destroy of two mutually-referencing SGs. Modern TF
|
||||
// (aws_vpc_security_group_ingress_rule) revokes each rule by SecurityGroupRuleId
|
||||
// with empty IpPermissions; if the handler ignores SecurityGroupRuleIds the
|
||||
// cross-ref rule survives and DeleteSecurityGroup deadlocks on DependencyViolation.
|
||||
func TestRevokeSecurityGroupIngress_ByRuleID_BreaksMutualReferenceTeardown(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.0.0.0/16")
|
||||
sgA := createTestSG(t, svc, vpcID, "tf-sg-a")
|
||||
sgB := createTestSG(t, svc, vpcID, "tf-sg-b")
|
||||
|
||||
authorizeIngressFromSG(t, svc, sgA, sgB)
|
||||
authorizeIngressFromSG(t, svc, sgB, sgA)
|
||||
|
||||
// Precondition: each SG is pinned undeletable while the cross-ref stands.
|
||||
_, err := svc.DeleteSecurityGroup(&ec2.DeleteSecurityGroupInput{GroupId: aws.String(sgA)}, testAccountID)
|
||||
require.ErrorContains(t, err, awserrors.ErrorDependencyViolation)
|
||||
|
||||
ruleA := ingressRuleIDReferencing(t, svc, sgA, sgB)
|
||||
ruleB := ingressRuleIDReferencing(t, svc, sgB, sgA)
|
||||
|
||||
for _, rv := range []struct{ sg, rule string }{{sgA, ruleA}, {sgB, ruleB}} {
|
||||
_, err = svc.RevokeSecurityGroupIngress(&ec2.RevokeSecurityGroupIngressInput{
|
||||
GroupId: aws.String(rv.sg),
|
||||
SecurityGroupRuleIds: []*string{aws.String(rv.rule)},
|
||||
}, testAccountID)
|
||||
require.NoErrorf(t, err, "revoke %s by rule id must remove the cross-ref", rv.sg)
|
||||
}
|
||||
|
||||
_, err = svc.DeleteSecurityGroup(&ec2.DeleteSecurityGroupInput{GroupId: aws.String(sgA)}, testAccountID)
|
||||
require.NoError(t, err, "sgA must tear down once its cross-ref is revoked")
|
||||
_, err = svc.DeleteSecurityGroup(&ec2.DeleteSecurityGroupInput{GroupId: aws.String(sgB)}, testAccountID)
|
||||
require.NoError(t, err, "sgB must tear down once its cross-ref is revoked")
|
||||
}
|
||||
|
||||
func TestRevokeSecurityGroupEgress_ByRuleID_RemovesRule(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.0.0.0/16")
|
||||
sgID := createTestSG(t, svc, vpcID, "tf-egress-byid")
|
||||
|
||||
rules, err := svc.DescribeSecurityGroupRules(&ec2.DescribeSecurityGroupRulesInput{
|
||||
Filters: []*ec2.Filter{{Name: aws.String("group-id"), Values: []*string{aws.String(sgID)}}},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
var egressID string
|
||||
for _, r := range rules.SecurityGroupRules {
|
||||
if aws.BoolValue(r.IsEgress) && aws.StringValue(r.GroupId) == sgID {
|
||||
egressID = aws.StringValue(r.SecurityGroupRuleId)
|
||||
}
|
||||
}
|
||||
require.NotEmpty(t, egressID, "new SG has a default all-egress rule")
|
||||
|
||||
_, err = svc.RevokeSecurityGroupEgress(&ec2.RevokeSecurityGroupEgressInput{
|
||||
GroupId: aws.String(sgID),
|
||||
SecurityGroupRuleIds: []*string{aws.String(egressID)},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
after, err := svc.DescribeSecurityGroupRules(&ec2.DescribeSecurityGroupRulesInput{
|
||||
Filters: []*ec2.Filter{{Name: aws.String("group-id"), Values: []*string{aws.String(sgID)}}},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
for _, r := range after.SecurityGroupRules {
|
||||
assert.NotEqual(t, egressID, aws.StringValue(r.SecurityGroupRuleId), "revoked egress rule must be gone")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRevokeSecurityGroupIngress_ByRuleID_NotFound(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.0.0.0/16")
|
||||
sgID := createTestSG(t, svc, vpcID, "tf-byid-notfound")
|
||||
|
||||
_, err := svc.RevokeSecurityGroupIngress(&ec2.RevokeSecurityGroupIngressInput{
|
||||
GroupId: aws.String(sgID),
|
||||
SecurityGroupRuleIds: []*string{aws.String("sgr-00000000000000000")},
|
||||
}, testAccountID)
|
||||
require.ErrorContains(t, err, awserrors.ErrorInvalidSecurityGroupRuleIdNotFound)
|
||||
}
|
||||
|
||||
func TestRevokeSecurityGroupIngress_ByRuleID_Malformed(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.0.0.0/16")
|
||||
sgID := createTestSG(t, svc, vpcID, "tf-byid-malformed")
|
||||
|
||||
_, err := svc.RevokeSecurityGroupIngress(&ec2.RevokeSecurityGroupIngressInput{
|
||||
GroupId: aws.String(sgID),
|
||||
SecurityGroupRuleIds: []*string{aws.String("not-a-rule-id")},
|
||||
}, testAccountID)
|
||||
require.ErrorContains(t, err, awserrors.ErrorInvalidSecurityGroupRuleIdMalformed)
|
||||
}
|
||||
|
||||
// --- Helper function tests ---
|
||||
|
||||
func TestIpPermissionsToSGRules_TCP(t *testing.T) {
|
||||
|
||||
@@ -403,6 +403,22 @@ func (s *VPCServiceImpl) DeleteVpc(input *ec2.DeleteVpcInput, accountID string)
|
||||
}
|
||||
}
|
||||
|
||||
// Reap the VPC's main route table: CreateVpc auto-creates it and
|
||||
// DeleteRouteTable refuses to delete a main RT, so nothing else reclaims it
|
||||
// — without this rtbKV leaks one orphaned main RT per deleted VPC.
|
||||
rtbID, err := s.findMainRouteTableID(accountID, vpcID)
|
||||
if err != nil {
|
||||
slog.Error("DeleteVpc: main route table lookup failed", "vpcId", vpcID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if rtbID != "" {
|
||||
if err := s.rtbKV.Delete(utils.AccountKey(accountID, rtbID)); err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
|
||||
slog.Error("DeleteVpc: main route table reap failed", "vpcId", vpcID, "routeTableId", rtbID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
slog.Info("DeleteVpc: reaped main route table", "vpcId", vpcID, "routeTableId", rtbID)
|
||||
}
|
||||
|
||||
if err := s.vpcKV.Delete(key); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
@@ -599,6 +599,28 @@ func TestCreateMainRouteTable_Idempotent(t *testing.T) {
|
||||
assert.Equal(t, 1, mains, "exactly one IsMain=true record after duplicate call")
|
||||
}
|
||||
|
||||
// TestDeleteVpc_ReapsMainRouteTable asserts DeleteVpc reclaims the VPC's
|
||||
// auto-created main route table. DeleteRouteTable refuses to delete a main RT
|
||||
// (AWS-faithful), so without DeleteVpc reaping it the rtbKV bucket leaks one
|
||||
// orphaned main RT per deleted VPC.
|
||||
func TestDeleteVpc_ReapsMainRouteTable(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.77.0.0/16") // auto-creates the main RT
|
||||
|
||||
rtbID, err := svc.findMainRouteTableID(testAccountID, vpcID)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, rtbID, "CreateVpc should have created a main route table")
|
||||
require.Equal(t, 1, countMainRouteTablesForVPC(t, svc, vpcID))
|
||||
|
||||
_, err = svc.DeleteVpc(&ec2.DeleteVpcInput{VpcId: aws.String(vpcID)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
gone, err := svc.findMainRouteTableID(testAccountID, vpcID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, gone, "DeleteVpc must reap the main route table, not leak it")
|
||||
assert.Equal(t, 0, countMainRouteTablesForVPC(t, svc, vpcID))
|
||||
}
|
||||
|
||||
// countMainRouteTablesForVPC scans rtbKV directly to surface duplicate-main
|
||||
// state in tests. Returns the number of IsMain=true records for vpcID.
|
||||
func countMainRouteTablesForVPC(t *testing.T, svc *VPCServiceImpl, vpcID string) int {
|
||||
|
||||
@@ -107,6 +107,8 @@ func TestCreateNodegroup_ConcurrentSameNameSingleOwner(t *testing.T) {
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
// The single winning owner launches one worker; let its Ready-gate resolve.
|
||||
markWorkersReady(t, f, "c1", 1)
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
ok, inUse := classifyCreateErrs(t, errs)
|
||||
|
||||
@@ -43,7 +43,13 @@ type ClusterMeta struct {
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
// EndpointIP is the NLB front-end IP (external-pool for public, VPC IP for private).
|
||||
// The apiserver serving cert SANs this IP for TLS verification.
|
||||
EndpointIP string `json:"endpointIp,omitempty"`
|
||||
EndpointIP string `json:"endpointIp,omitempty"`
|
||||
// PrivateEndpointIP is the customer-VPC (Set A) IP of the ENI threaded onto the
|
||||
// cluster NLB's LB VM when EndpointPrivateAccess is on. In-VPC workers + kubectl
|
||||
// reach the control plane here on :443 (no public hairpin / NAT GW). SANed on
|
||||
// the apiserver cert. PrivateEndpointENIID is its ENI, kept for teardown.
|
||||
PrivateEndpointIP string `json:"privateEndpointIp,omitempty"`
|
||||
PrivateEndpointENIID string `json:"privateEndpointEniId,omitempty"`
|
||||
OIDCIssuer string `json:"oidcIssuer,omitempty"`
|
||||
CertificateAuthorityB64 string `json:"certificateAuthorityB64,omitempty"`
|
||||
ResourcesVpcConfig *ClusterVpcConfig `json:"resourcesVpcConfig,omitempty"`
|
||||
|
||||
@@ -3,6 +3,7 @@ package handlers_eks
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
@@ -89,6 +90,10 @@ func newEKSServiceFixture(t *testing.T) *eksServiceFixture {
|
||||
Scheduler: &fakeHostScheduler{},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
// Tight Ready-gating knobs so nodegroup launches resolve in tests without the
|
||||
// production 10m timeout; tests drive meta.NodeCount to simulate the reconciler.
|
||||
svc.nodegroupReadyTimeout = 1 * time.Second
|
||||
svc.nodegroupReadyPoll = 2 * time.Millisecond
|
||||
t.Cleanup(svc.Shutdown)
|
||||
|
||||
return &eksServiceFixture{svc: svc, kv: kv, nlb: nlb, inst: inst, vpc: vpc, ami: ami, eip: eip, sg: sg, worker: worker, vpcMgr: vpcMgr, igw: igw, ngw: ngw, rt: rt}
|
||||
@@ -203,6 +208,31 @@ func TestDeleteCluster_AllTeardownSucceedsSweepsKV(t *testing.T) {
|
||||
assert.Len(t, f.vpc.deleteCalls, 1)
|
||||
}
|
||||
|
||||
// TestDeleteCluster_DetachesPrivateEndpointENIBeforeDelete locks the mulga-siv-301
|
||||
// teardown fix: the Set A private-endpoint ENI is an extra NIC on the cluster NLB's
|
||||
// LB VM, not that VM's primary ENI, so the instance-terminate cascade never reclaims
|
||||
// it. purgeClusterInfra must detach (store-clear) the stale attachment before
|
||||
// deleting, or DeleteNetworkInterface loops on InvalidNetworkInterface.InUse and the
|
||||
// cluster wedges in DELETING. The fake models that in-use-until-detached semantics,
|
||||
// so the teardown only completes if detach precedes delete.
|
||||
func TestDeleteCluster_DetachesPrivateEndpointENIBeforeDelete(t *testing.T) {
|
||||
f := newDeleteClusterFixture(t, "alpha")
|
||||
|
||||
meta, err := GetClusterMeta(f.kv, "alpha")
|
||||
require.NoError(t, err)
|
||||
meta.PrivateEndpointENIID = "eni-pe-001"
|
||||
require.NoError(t, PutClusterMeta(f.kv, meta))
|
||||
f.vpc.inUseUntilDetached = map[string]bool{"eni-pe-001": true}
|
||||
|
||||
out, err := f.svc.DeleteCluster(deleteInput("alpha"), testAccountID)
|
||||
require.NoError(t, err, "teardown must complete: the private-endpoint ENI is detached before delete")
|
||||
require.NotNil(t, out)
|
||||
|
||||
assert.Contains(t, f.vpc.detachCalls, "eni-pe-001", "private-endpoint ENI must be detached before delete")
|
||||
_, getErr := GetClusterMeta(f.kv, "alpha")
|
||||
assert.ErrorIs(t, getErr, ErrClusterNotFound, "successful teardown must sweep the meta")
|
||||
}
|
||||
|
||||
func TestDeleteCluster_NLBFailureLeavesMetaAndDELETING(t *testing.T) {
|
||||
f := newDeleteClusterFixture(t, "alpha")
|
||||
|
||||
|
||||
@@ -80,6 +80,8 @@ func (v *seqK3sVPC) DescribeNetworkInterfaces(*ec2.DescribeNetworkInterfacesInpu
|
||||
return &ec2.DescribeNetworkInterfacesOutput{}, nil
|
||||
}
|
||||
|
||||
func (v *seqK3sVPC) DetachENI(_, _ string) error { return nil }
|
||||
|
||||
// seqK3sInst returns instance ID "i-<nodeID>" so launches map deterministically
|
||||
// back to their target host. failNodes makes a node's launch return an error.
|
||||
type seqK3sInst struct {
|
||||
|
||||
@@ -30,6 +30,7 @@ type k3sVPCProvisioner interface {
|
||||
CreateNetworkInterface(input *ec2.CreateNetworkInterfaceInput, accountID string) (*ec2.CreateNetworkInterfaceOutput, error)
|
||||
DeleteNetworkInterface(input *ec2.DeleteNetworkInterfaceInput, accountID string) (*ec2.DeleteNetworkInterfaceOutput, error)
|
||||
DescribeNetworkInterfaces(input *ec2.DescribeNetworkInterfacesInput, accountID string) (*ec2.DescribeNetworkInterfacesOutput, error)
|
||||
DetachENI(accountID, eniID string) error
|
||||
}
|
||||
|
||||
// k3sInstanceLauncher is the system-instance launch surface for the K3s CP VM.
|
||||
@@ -102,7 +103,11 @@ type K3sServerInput struct {
|
||||
NLBDNS string
|
||||
// EndpointIP is the NLB front-end IP added to the apiserver cert SANs for TLS.
|
||||
// Empty for an internal endpoint with no front-end IP.
|
||||
EndpointIP string
|
||||
EndpointIP string
|
||||
// PrivateEndpointIP is the customer-VPC (Set A) private-endpoint IP added to the
|
||||
// apiserver cert SANs so in-VPC clients validate TLS via https://<ip>:443.
|
||||
// Empty when private access is off.
|
||||
PrivateEndpointIP string
|
||||
OIDCIssuer string
|
||||
OIDCPrivateKeyPEM string
|
||||
OIDCPublicKeyPEM string
|
||||
@@ -466,6 +471,10 @@ func buildK3sUserData(in K3sServerInput) string {
|
||||
if in.EndpointIP != "" {
|
||||
configLines = append(configLines, " - "+in.EndpointIP)
|
||||
}
|
||||
// The Set A private-endpoint IP is what in-VPC clients connect to; SAN it too.
|
||||
if in.PrivateEndpointIP != "" && in.PrivateEndpointIP != in.EndpointIP {
|
||||
configLines = append(configLines, " - "+in.PrivateEndpointIP)
|
||||
}
|
||||
configLines = append(configLines,
|
||||
"kube-apiserver-arg:",
|
||||
" - service-account-key-file="+k3sOIDCPublicKeyPath,
|
||||
|
||||
@@ -19,10 +19,17 @@ import (
|
||||
type fakeK3sVPC struct {
|
||||
createCalls []*ec2.CreateNetworkInterfaceInput
|
||||
deleteCalls []*ec2.DeleteNetworkInterfaceInput
|
||||
detachCalls []string
|
||||
|
||||
createOut *ec2.CreateNetworkInterfaceOutput
|
||||
createErr error
|
||||
deleteErr error
|
||||
detachErr error
|
||||
// inUseUntilDetached models real VPC semantics: DeleteNetworkInterface returns
|
||||
// InvalidNetworkInterface.InUse for these ENIs until DetachENI clears the
|
||||
// attachment. detached tracks which ENIs DetachENI has cleared.
|
||||
inUseUntilDetached map[string]bool
|
||||
detached map[string]bool
|
||||
// deleteInUseUntil models the async instance-terminate cascade: the first N
|
||||
// DeleteNetworkInterface calls return InvalidNetworkInterface.InUse, then the
|
||||
// cascade has force-deleted the ENI so the next call returns NotFound.
|
||||
@@ -56,6 +63,9 @@ func (f *fakeK3sVPC) CreateNetworkInterface(input *ec2.CreateNetworkInterfaceInp
|
||||
|
||||
func (f *fakeK3sVPC) DeleteNetworkInterface(input *ec2.DeleteNetworkInterfaceInput, _ string) (*ec2.DeleteNetworkInterfaceOutput, error) {
|
||||
f.deleteCalls = append(f.deleteCalls, input)
|
||||
if id := aws.StringValue(input.NetworkInterfaceId); f.inUseUntilDetached[id] && !f.detached[id] {
|
||||
return nil, errors.New(awserrors.ErrorInvalidNetworkInterfaceInUse)
|
||||
}
|
||||
if f.deleteInUseUntil > 0 {
|
||||
if len(f.deleteCalls) <= f.deleteInUseUntil {
|
||||
return nil, errors.New(awserrors.ErrorInvalidNetworkInterfaceInUse)
|
||||
@@ -68,6 +78,18 @@ func (f *fakeK3sVPC) DeleteNetworkInterface(input *ec2.DeleteNetworkInterfaceInp
|
||||
return &ec2.DeleteNetworkInterfaceOutput{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeK3sVPC) DetachENI(_, eniID string) error {
|
||||
f.detachCalls = append(f.detachCalls, eniID)
|
||||
if f.detachErr != nil {
|
||||
return f.detachErr
|
||||
}
|
||||
if f.detached == nil {
|
||||
f.detached = map[string]bool{}
|
||||
}
|
||||
f.detached[eniID] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeK3sVPC) DescribeNetworkInterfaces(input *ec2.DescribeNetworkInterfacesInput, _ string) (*ec2.DescribeNetworkInterfacesOutput, error) {
|
||||
if f.describeErr != nil {
|
||||
return nil, f.describeErr
|
||||
@@ -317,6 +339,26 @@ func TestLaunchK3sServerVM_TargetNodeIDRoutedToLauncher(t *testing.T) {
|
||||
assert.Equal(t, "node-c", inst.launchNodes[0], "TargetNodeID pins placement to a specific host")
|
||||
}
|
||||
|
||||
func TestBuildK3sUserData_TLSSANIncludesPrivateEndpointIP(t *testing.T) {
|
||||
in := validK3sInput()
|
||||
in.EndpointIP = "203.0.113.9"
|
||||
in.PrivateEndpointIP = "10.20.0.5"
|
||||
|
||||
ud := buildK3sUserData(in)
|
||||
assert.Contains(t, ud, " - "+in.NLBDNS)
|
||||
assert.Contains(t, ud, " - 203.0.113.9")
|
||||
assert.Contains(t, ud, " - 10.20.0.5", "Set A private-endpoint IP must be a cert SAN")
|
||||
}
|
||||
|
||||
func TestBuildK3sUserData_TLSSANDedupsWhenPrivateEqualsEndpoint(t *testing.T) {
|
||||
in := validK3sInput()
|
||||
in.EndpointIP = "10.20.0.5"
|
||||
in.PrivateEndpointIP = "10.20.0.5"
|
||||
|
||||
ud := buildK3sUserData(in)
|
||||
assert.Equal(t, 1, strings.Count(ud, " - 10.20.0.5"), "must not emit a duplicate SAN")
|
||||
}
|
||||
|
||||
func TestLaunchK3sServerVM_UserDataContainsAllArtifacts(t *testing.T) {
|
||||
vpc, inst, ami := &fakeK3sVPC{}, &fakeK3sInst{}, &fakeK3sAMI{}
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/elbv2"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/sysinstance"
|
||||
"github.com/mulgadc/spinifex/spinifex/tags"
|
||||
)
|
||||
|
||||
@@ -14,6 +15,9 @@ import (
|
||||
type nlbProvisioner interface {
|
||||
CreateLoadBalancer(input *elbv2.CreateLoadBalancerInput, accountID string) (*elbv2.CreateLoadBalancerOutput, error)
|
||||
CreateLoadBalancerSync(input *elbv2.CreateLoadBalancerInput, accountID string) (*elbv2.CreateLoadBalancerOutput, error)
|
||||
// CreateClusterNLBSync is CreateLoadBalancerSync plus cross-account ENIs threaded
|
||||
// onto the LB VM at launch (the customer-VPC Set A private-endpoint NIC).
|
||||
CreateClusterNLBSync(input *elbv2.CreateLoadBalancerInput, accountID string, crossAccountENIs []sysinstance.ExtraENIInput) (*elbv2.CreateLoadBalancerOutput, error)
|
||||
DescribeLoadBalancers(input *elbv2.DescribeLoadBalancersInput, accountID string) (*elbv2.DescribeLoadBalancersOutput, error)
|
||||
DeleteLoadBalancer(input *elbv2.DeleteLoadBalancerInput, accountID string) (*elbv2.DeleteLoadBalancerOutput, error)
|
||||
|
||||
@@ -83,7 +87,7 @@ func ClusterTargetGroupName(clusterName string) string {
|
||||
// NLB managed-SG ingress; ignored for an internal NLB (its ingress already
|
||||
// tracks the VPC CIDR) and for the wide-open default, which the LB carries out
|
||||
// of the box.
|
||||
func EnsureClusterNLB(nlbp nlbProvisioner, accountID, clusterName string, subnetIDs []string, internetFacing bool, publicAccessCidrs []string) (*ClusterNLB, error) {
|
||||
func EnsureClusterNLB(nlbp nlbProvisioner, accountID, clusterName string, subnetIDs []string, internetFacing bool, publicAccessCidrs []string, crossAccountENIs []sysinstance.ExtraENIInput) (*ClusterNLB, error) {
|
||||
if clusterName == "" {
|
||||
return nil, errors.New("eks: EnsureClusterNLB empty cluster name")
|
||||
}
|
||||
@@ -101,7 +105,7 @@ func EnsureClusterNLB(nlbp nlbProvisioner, accountID, clusterName string, subnet
|
||||
|
||||
out := &ClusterNLB{}
|
||||
|
||||
if err := ensureClusterLB(nlbp, accountID, clusterName, lbName, subnetIDs, internetFacing, out); err != nil {
|
||||
if err := ensureClusterLB(nlbp, accountID, clusterName, lbName, subnetIDs, internetFacing, crossAccountENIs, out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ensureClusterTG(nlbp, accountID, clusterName, tgName, out); err != nil {
|
||||
@@ -268,7 +272,7 @@ func deleteClusterTG(nlbp nlbProvisioner, accountID, tgName string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func ensureClusterLB(nlbp nlbProvisioner, accountID, clusterName, lbName string, subnetIDs []string, internetFacing bool, out *ClusterNLB) error {
|
||||
func ensureClusterLB(nlbp nlbProvisioner, accountID, clusterName, lbName string, subnetIDs []string, internetFacing bool, crossAccountENIs []sysinstance.ExtraENIInput, out *ClusterNLB) error {
|
||||
if lb, err := lookupLBByName(nlbp, accountID, lbName); err != nil {
|
||||
return err
|
||||
} else if lb != nil {
|
||||
@@ -291,7 +295,9 @@ func ensureClusterLB(nlbp nlbProvisioner, accountID, clusterName, lbName string,
|
||||
// awaits the data-plane launch and gets back an LB carrying its front-end IP —
|
||||
// which must be baked into the apiserver cert SAN before the control-plane VM
|
||||
// boots (the async CreateLoadBalancer would return before the IP is allocated).
|
||||
created, err := nlbp.CreateLoadBalancerSync(&elbv2.CreateLoadBalancerInput{
|
||||
// Cross-account ENIs (the Set A private-endpoint NIC) are threaded onto the LB
|
||||
// VM at launch, so the create must carry them when present.
|
||||
in := &elbv2.CreateLoadBalancerInput{
|
||||
Name: aws.String(lbName),
|
||||
Type: aws.String(elbv2.LoadBalancerTypeEnumNetwork),
|
||||
Scheme: aws.String(scheme),
|
||||
@@ -300,7 +306,14 @@ func ensureClusterLB(nlbp nlbProvisioner, accountID, clusterName, lbName string,
|
||||
{Key: aws.String(tags.ManagedByKey), Value: aws.String(tags.ManagedByEKS)},
|
||||
{Key: aws.String(clusterEKSClusterTagKey), Value: aws.String(clusterName)},
|
||||
},
|
||||
}, accountID)
|
||||
}
|
||||
var created *elbv2.CreateLoadBalancerOutput
|
||||
var err error
|
||||
if len(crossAccountENIs) > 0 {
|
||||
created, err = nlbp.CreateClusterNLBSync(in, accountID, crossAccountENIs)
|
||||
} else {
|
||||
created, err = nlbp.CreateLoadBalancerSync(in, accountID)
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("create NLB %s: %w", lbName, err)
|
||||
}
|
||||
|
||||
@@ -7,24 +7,26 @@ import (
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/elbv2"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/sysinstance"
|
||||
"github.com/mulgadc/spinifex/spinifex/tags"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type fakeNLBProvisioner struct {
|
||||
createLBCalls []*elbv2.CreateLoadBalancerInput
|
||||
createLBSyncCalls []*elbv2.CreateLoadBalancerInput
|
||||
describeLBCalls []*elbv2.DescribeLoadBalancersInput
|
||||
deleteLBCalls []*elbv2.DeleteLoadBalancerInput
|
||||
createTGCalls []*elbv2.CreateTargetGroupInput
|
||||
describeTGCalls []*elbv2.DescribeTargetGroupsInput
|
||||
deleteTGCalls []*elbv2.DeleteTargetGroupInput
|
||||
createListenerCalls []*elbv2.CreateListenerInput
|
||||
describeListeners []*elbv2.DescribeListenersInput
|
||||
registerCalls []*elbv2.RegisterTargetsInput
|
||||
deregisterCalls []*elbv2.DeregisterTargetsInput
|
||||
setIngressCalls []setIngressCIDRsCall
|
||||
createLBCalls []*elbv2.CreateLoadBalancerInput
|
||||
createLBSyncCalls []*elbv2.CreateLoadBalancerInput
|
||||
createClusterNLBExtras [][]sysinstance.ExtraENIInput
|
||||
describeLBCalls []*elbv2.DescribeLoadBalancersInput
|
||||
deleteLBCalls []*elbv2.DeleteLoadBalancerInput
|
||||
createTGCalls []*elbv2.CreateTargetGroupInput
|
||||
describeTGCalls []*elbv2.DescribeTargetGroupsInput
|
||||
deleteTGCalls []*elbv2.DeleteTargetGroupInput
|
||||
createListenerCalls []*elbv2.CreateListenerInput
|
||||
describeListeners []*elbv2.DescribeListenersInput
|
||||
registerCalls []*elbv2.RegisterTargetsInput
|
||||
deregisterCalls []*elbv2.DeregisterTargetsInput
|
||||
setIngressCalls []setIngressCIDRsCall
|
||||
|
||||
// frontendIP is the address CreateLoadBalancerSync stamps onto the launched
|
||||
// LB's AZ addresses (both public + private slots, so frontendIPFromLB resolves
|
||||
@@ -85,6 +87,13 @@ func (f *fakeNLBProvisioner) CreateLoadBalancerSync(input *elbv2.CreateLoadBalan
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// CreateClusterNLBSync records the cross-account ENIs and delegates to the sync
|
||||
// path so existing front-end-IP assertions still hold.
|
||||
func (f *fakeNLBProvisioner) CreateClusterNLBSync(input *elbv2.CreateLoadBalancerInput, accountID string, crossAccountENIs []sysinstance.ExtraENIInput) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
f.createClusterNLBExtras = append(f.createClusterNLBExtras, crossAccountENIs)
|
||||
return f.CreateLoadBalancerSync(input, accountID)
|
||||
}
|
||||
|
||||
func (f *fakeNLBProvisioner) CreateLoadBalancer(input *elbv2.CreateLoadBalancerInput, _ string) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
f.createLBCalls = append(f.createLBCalls, input)
|
||||
if f.createLBErr != nil {
|
||||
@@ -257,10 +266,10 @@ func (f *fakeNLBProvisioner) SetLoadBalancerIngressCIDRs(lbArn string, cidrs []s
|
||||
func TestEnsureClusterNLB_EmptyInputsRejected(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "", []string{"subnet-aaa"}, false, nil)
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.Error(t, err)
|
||||
|
||||
_, err = EnsureClusterNLB(nlbp, "111122223333", "alpha", nil, false, nil)
|
||||
_, err = EnsureClusterNLB(nlbp, "111122223333", "alpha", nil, false, nil, nil)
|
||||
require.Error(t, err)
|
||||
|
||||
assert.Empty(t, nlbp.createLBCalls)
|
||||
@@ -270,7 +279,7 @@ func TestEnsureClusterNLB_NameTooLongRejected(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
|
||||
longName := strings.Repeat("x", maxELBv2NameLen) // "eks-" + 32x = 36 chars
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", longName, []string{"subnet-aaa"}, false, nil)
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", longName, []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "exceeds")
|
||||
assert.Empty(t, nlbp.createLBCalls)
|
||||
@@ -279,7 +288,7 @@ func TestEnsureClusterNLB_NameTooLongRejected(t *testing.T) {
|
||||
func TestEnsureClusterNLB_FreshCreatesAllThree(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa", "subnet-bbb"}, false, nil)
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa", "subnet-bbb"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, out)
|
||||
assert.NotEmpty(t, out.LoadBalancerArn)
|
||||
@@ -320,7 +329,7 @@ func TestEnsureClusterNLB_NoFrontendIPFailsLoud(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
nlbp.frontendIP = "" // no external IP pool → LB comes up without a reachable address
|
||||
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, nil)
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, nil, nil)
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, ErrClusterNLBFrontendIPUnavailable)
|
||||
// The LB was still created (sync attempt) before the loud fail.
|
||||
@@ -331,7 +340,7 @@ func TestEnsureClusterNLB_InternetFacingUsesPublicAddress(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
nlbp.frontendIP = "203.0.113.7"
|
||||
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, nil)
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, nil, nil)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "203.0.113.7", out.FrontendIP)
|
||||
require.Len(t, nlbp.createLBSyncCalls, 1)
|
||||
@@ -341,14 +350,14 @@ func TestEnsureClusterNLB_InternetFacingUsesPublicAddress(t *testing.T) {
|
||||
func TestEnsureClusterNLB_IdempotentReusesExisting(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
|
||||
first, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil)
|
||||
first, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
createLBCount := len(nlbp.createLBCalls)
|
||||
createTGCount := len(nlbp.createTGCalls)
|
||||
createListenerCount := len(nlbp.createListenerCalls)
|
||||
|
||||
second, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil)
|
||||
second, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, first.LoadBalancerArn, second.LoadBalancerArn)
|
||||
@@ -365,7 +374,7 @@ func TestEnsureClusterNLB_LBCreateErrorSurfaced(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
nlbp.createLBErr = errors.New("InsufficientCapacity")
|
||||
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil)
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "create NLB eks-alpha")
|
||||
assert.Empty(t, nlbp.createTGCalls, "TG create should not run when LB create fails")
|
||||
@@ -391,7 +400,7 @@ func TestEnsureClusterNLB_InternetFacingSchemeAndPublicFrontendIP(t *testing.T)
|
||||
}},
|
||||
}
|
||||
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, nil)
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, nil, nil)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, nlbp.createLBCalls, 1)
|
||||
assert.Equal(t, elbv2.LoadBalancerSchemeEnumInternetFacing, aws.StringValue(nlbp.createLBCalls[0].Scheme))
|
||||
@@ -416,7 +425,7 @@ func TestEnsureClusterNLB_InternalSchemeUsesPrivateFrontendIP(t *testing.T) {
|
||||
}},
|
||||
}
|
||||
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil)
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, nlbp.createLBCalls, 1)
|
||||
assert.Equal(t, elbv2.LoadBalancerSchemeEnumInternal, aws.StringValue(nlbp.createLBCalls[0].Scheme))
|
||||
@@ -427,7 +436,7 @@ func TestEnsureClusterNLB_NarrowedPublicAccessSetsIngressCIDRs(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
|
||||
cidrs := []string{"203.0.113.0/24", "198.51.100.7/32"}
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, cidrs)
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, cidrs, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, nlbp.setIngressCalls, 1, "narrowed public access should drive SetLoadBalancerIngressCIDRs")
|
||||
@@ -446,7 +455,7 @@ func TestEnsureClusterNLB_DefaultPublicAccessSkipsIngressCIDRs(t *testing.T) {
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, tc.cidrs)
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, tc.cidrs, nil)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, nlbp.setIngressCalls, "wide-open front-end needs no ingress override")
|
||||
})
|
||||
@@ -458,7 +467,7 @@ func TestEnsureClusterNLB_InternalSkipsIngressCIDRs(t *testing.T) {
|
||||
|
||||
// Even with narrowed CIDRs, an internal NLB ignores them — its ingress
|
||||
// already tracks the VPC CIDR, not a public front-end.
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, []string{"203.0.113.0/24"})
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, []string{"203.0.113.0/24"}, nil)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, nlbp.setIngressCalls)
|
||||
}
|
||||
@@ -467,7 +476,7 @@ func TestEnsureClusterNLB_SetIngressErrorSurfaced(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
nlbp.setIngressErr = errors.New("InvalidLoadBalancer")
|
||||
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, []string{"203.0.113.0/24"})
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, true, []string{"203.0.113.0/24"}, nil)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "set NLB ingress CIDRs for eks-alpha")
|
||||
}
|
||||
@@ -555,7 +564,7 @@ func TestDeregisterClusterTarget_PostsENIIPAndAPIPort(t *testing.T) {
|
||||
|
||||
func TestDeleteClusterNLB_DeletesBoth(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil)
|
||||
out, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, DeleteClusterNLB(nlbp, "111122223333", "alpha"))
|
||||
@@ -575,7 +584,7 @@ func TestDeleteClusterNLB_MissingResourcesNoOp(t *testing.T) {
|
||||
|
||||
func TestDeleteClusterNLB_FirstErrorSurfacedSweepContinues(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil)
|
||||
_, err := EnsureClusterNLB(nlbp, "111122223333", "alpha", []string{"subnet-aaa"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
nlbp.deleteLBErr = errors.New("LoadBalancerInUse")
|
||||
|
||||
|
||||
@@ -23,6 +23,15 @@ import (
|
||||
// when the caller omits instanceTypes.
|
||||
const defaultNodegroupInstanceType = "t3.medium"
|
||||
|
||||
// defaultNodegroupReadyTimeout / defaultNodegroupReadyPoll bound how long
|
||||
// launchNodegroupInfra waits for its workers to register Ready (observed via the
|
||||
// CP state report's Ready-node count, refreshed at the reconcile cadence) before
|
||||
// marking the nodegroup CREATE_FAILED.
|
||||
const (
|
||||
defaultNodegroupReadyTimeout = 10 * time.Minute
|
||||
defaultNodegroupReadyPoll = 15 * time.Second
|
||||
)
|
||||
|
||||
// ErrNodegroupNotFound is returned by GetNodegroupRecord when the record key is
|
||||
// absent. Callers translate to the AWS shape (ResourceNotFoundException) at the
|
||||
// service boundary.
|
||||
@@ -180,6 +189,10 @@ func (s *EKSServiceImpl) createNodegroup(acctKV nats.KeyValue, input *eks.Create
|
||||
slog.Error("createNodegroup: cluster has no control-plane ENI IP", "cluster", cluster)
|
||||
return nil, errors.New(awserrors.ErrorInvalidRequest)
|
||||
}
|
||||
if meta.Endpoint == "" {
|
||||
slog.Error("createNodegroup: cluster has no published endpoint", "cluster", cluster)
|
||||
return nil, errors.New(awserrors.ErrorInvalidRequest)
|
||||
}
|
||||
if meta.ResourcesVpcConfig == nil || meta.ResourcesVpcConfig.VpcId == "" {
|
||||
slog.Error("createNodegroup: cluster has no VPC", "cluster", cluster)
|
||||
return nil, errors.New(awserrors.ErrorInvalidRequest)
|
||||
@@ -314,7 +327,20 @@ func (s *EKSServiceImpl) launchNodegroupInfra(lc nodegroupLaunchCtx) {
|
||||
return
|
||||
}
|
||||
|
||||
// v1: ACTIVE on RunInstances success; node-Ready gating is a future improvement.
|
||||
// Gate ACTIVE on the workers registering Ready (observed via the CP state
|
||||
// report's Ready-node count), not merely on RunInstances success — a worker
|
||||
// that boots but never joins must surface CREATE_FAILED, not falsely ACTIVE.
|
||||
// Baseline is the create-time node count; the workers add lc.desired Ready nodes.
|
||||
if err := s.waitWorkersReady(acctKV, cluster, meta.NodeCount, lc.desired); err != nil {
|
||||
rec.Status = eks.NodegroupStatusCreateFailed
|
||||
rec.StatusReason = "workers did not become Ready: " + err.Error()
|
||||
rec.ModifiedAt = time.Now().UTC()
|
||||
if perr := PutNodegroupRecord(acctKV, rec); perr != nil {
|
||||
slog.Error("createNodegroup: persist CREATE_FAILED record", "cluster", cluster, "nodegroup", ng, "err", perr)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
rec.Status = eks.NodegroupStatusActive
|
||||
rec.ModifiedAt = time.Now().UTC()
|
||||
if err := PutNodegroupRecord(acctKV, rec); err != nil {
|
||||
@@ -322,6 +348,35 @@ func (s *EKSServiceImpl) launchNodegroupInfra(lc nodegroupLaunchCtx) {
|
||||
}
|
||||
}
|
||||
|
||||
// waitWorkersReady blocks until the cluster's Ready-node count rises by want over
|
||||
// baseline — every nodegroup worker registered Ready — or the timeout / bgCtx
|
||||
// fires. Ready count is meta.NodeCount, which the ClusterReconciler refreshes
|
||||
// from the CP's NATS state report (Ready nodes only). Mirrors the CP reconciler's
|
||||
// healthy-observe gating: a nodegroup is ACTIVE only once its workers are observed
|
||||
// Ready, not merely launched.
|
||||
func (s *EKSServiceImpl) waitWorkersReady(acctKV nats.KeyValue, cluster string, baseline, want int) error {
|
||||
target := baseline + want
|
||||
deadline := time.Now().Add(s.nodegroupReadyTimeout)
|
||||
for {
|
||||
meta, err := GetClusterMeta(acctKV, cluster)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if meta.NodeCount >= target {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("timed out after %s: cluster reports %d Ready nodes, want >= %d (baseline %d + %d workers)",
|
||||
s.nodegroupReadyTimeout, meta.NodeCount, target, baseline, want)
|
||||
}
|
||||
select {
|
||||
case <-s.bgCtx.Done():
|
||||
return s.bgCtx.Err()
|
||||
case <-time.After(s.nodegroupReadyPoll):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// launchWorkers issues count customer-owned RunInstances calls (one per worker
|
||||
// so each gets a distinct node name + node label in its user-data) and returns
|
||||
// the launched instance IDs. On a partial failure it returns the IDs that did
|
||||
@@ -341,11 +396,11 @@ func (s *EKSServiceImpl) launchWorkers(acctKV nats.KeyValue, accountID string, r
|
||||
for i := range count {
|
||||
shortID := uuid.NewString()[:8]
|
||||
userData := buildAgentUserData(agentUserDataInput{
|
||||
ClusterName: rec.ClusterName,
|
||||
NodegroupName: rec.Name,
|
||||
ControlPlaneENIIP: meta.ControlPlaneENIIP,
|
||||
JoinToken: token,
|
||||
NodeName: fmt.Sprintf("%s-%s-%s", rec.ClusterName, rec.Name, shortID),
|
||||
ClusterName: rec.ClusterName,
|
||||
NodegroupName: rec.Name,
|
||||
ServerURL: clusterJoinEndpoint(meta),
|
||||
JoinToken: token,
|
||||
NodeName: fmt.Sprintf("%s-%s-%s", rec.ClusterName, rec.Name, shortID),
|
||||
})
|
||||
runInput := &ec2.RunInstancesInput{
|
||||
ImageId: aws.String(amiID),
|
||||
|
||||
@@ -59,6 +59,7 @@ func seedActiveClusterWithToken(t *testing.T, f *eksServiceFixture, cluster stri
|
||||
Status: ClusterStatusActive,
|
||||
Version: "1.32",
|
||||
ControlPlaneENIIP: "10.0.1.42",
|
||||
Endpoint: "https://10.254.0.9:443",
|
||||
ResourcesVpcConfig: &ClusterVpcConfig{
|
||||
SubnetIds: []string{"subnet-aaa"},
|
||||
VpcId: "vpc-aaa",
|
||||
@@ -72,6 +73,15 @@ func seedActiveClusterWithToken(t *testing.T, f *eksServiceFixture, cluster stri
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// markWorkersReady simulates the cluster reconciler observing n Ready nodes from
|
||||
// the CP state report, so launchNodegroupInfra's Ready-gate (waitWorkersReady)
|
||||
// can resolve. Seed clusters start at NodeCount 0, so n is the create baseline
|
||||
// (0) plus the worker count the test expects Ready.
|
||||
func markWorkersReady(t *testing.T, f *eksServiceFixture, cluster string, n int) {
|
||||
t.Helper()
|
||||
require.NoError(t, SetClusterHealthState(f.kv, cluster, "", n))
|
||||
}
|
||||
|
||||
func createNGInput(cluster, ng string, desired int64) *eks.CreateNodegroupInput {
|
||||
return &eks.CreateNodegroupInput{
|
||||
ClusterName: aws.String(cluster),
|
||||
@@ -170,16 +180,16 @@ func TestNodegroupRecord_CRUDRoundTrip(t *testing.T) {
|
||||
|
||||
func TestBuildAgentUserData_Shape(t *testing.T) {
|
||||
ud := buildAgentUserData(agentUserDataInput{
|
||||
ClusterName: "c1",
|
||||
NodegroupName: "ng1",
|
||||
ControlPlaneENIIP: "10.0.1.42",
|
||||
JoinToken: "K10secret::server:xyz",
|
||||
NodeName: "c1-ng1-abc123de",
|
||||
ClusterName: "c1",
|
||||
NodegroupName: "ng1",
|
||||
ServerURL: "https://10.254.0.9:443",
|
||||
JoinToken: "K10secret::server:xyz",
|
||||
NodeName: "c1-ng1-abc123de",
|
||||
})
|
||||
|
||||
assert.True(t, strings.HasPrefix(ud, "#cloud-config\n"))
|
||||
assert.Contains(t, ud, "SPINIFEX_K3S_ROLE=agent")
|
||||
assert.Contains(t, ud, "K3S_URL=https://10.0.1.42:6443")
|
||||
assert.Contains(t, ud, "K3S_URL=https://10.254.0.9:443")
|
||||
assert.Contains(t, ud, "K3S_TOKEN=K10secret::server:xyz")
|
||||
assert.Contains(t, ud, "K3S_NODE_NAME=c1-ng1-abc123de")
|
||||
assert.Contains(t, ud, "eks.amazonaws.com/nodegroup=ng1")
|
||||
@@ -234,6 +244,8 @@ func TestCreateNodegroup_HappyPath(t *testing.T) {
|
||||
assert.Equal(t, int64(2), aws.Int64Value(out.Nodegroup.ScalingConfig.DesiredSize))
|
||||
assert.Contains(t, aws.StringValue(out.Nodegroup.NodegroupArn), ":nodegroup/c1/ng1/")
|
||||
|
||||
// Both workers register Ready → the Ready-gate lets the nodegroup go ACTIVE.
|
||||
markWorkersReady(t, f, "c1", 2)
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
// The async launch transitions the record to ACTIVE once workers run.
|
||||
@@ -266,6 +278,26 @@ func TestCreateNodegroup_HappyPath(t *testing.T) {
|
||||
require.EqualError(t, err, awserrors.ErrorEKSResourceInUse)
|
||||
}
|
||||
|
||||
// Workers that launch but never register Ready (no rise in the cluster's Ready-
|
||||
// node count) must drive the nodegroup to CREATE_FAILED, not a falsely-ACTIVE
|
||||
// record. Instance IDs are retained so the reclaim path tears the workers down.
|
||||
func TestCreateNodegroup_WorkersNeverReady_CreateFailed(t *testing.T) {
|
||||
f := newEKSServiceFixture(t)
|
||||
seedActiveClusterWithToken(t, f, "c1")
|
||||
|
||||
_, err := f.svc.CreateNodegroup(createNGInput("c1", "ng1", 2), testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// No markWorkersReady → NodeCount stays at the baseline; the Ready-gate times out.
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
rec, err := GetNodegroupRecord(f.kv, "c1", "ng1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, eks.NodegroupStatusCreateFailed, rec.Status)
|
||||
assert.Contains(t, rec.StatusReason, "did not become Ready")
|
||||
assert.Len(t, rec.InstanceIDs, 2, "launched workers retained for the reclaim path")
|
||||
}
|
||||
|
||||
func TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping(t *testing.T) {
|
||||
f := newEKSServiceFixture(t)
|
||||
seedActiveClusterWithToken(t, f, "c1")
|
||||
@@ -275,6 +307,7 @@ func TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping(t *testing.T) {
|
||||
_, err := f.svc.CreateNodegroup(in, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
markWorkersReady(t, f, "c1", 1)
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
require.Len(t, f.worker.runCalls, 1)
|
||||
@@ -291,6 +324,7 @@ func TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping(t *testing.T) {
|
||||
_, err := f.svc.CreateNodegroup(createNGInput("c1", "ng1", 1), testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
markWorkersReady(t, f, "c1", 1)
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
require.Len(t, f.worker.runCalls, 1)
|
||||
@@ -322,6 +356,7 @@ func TestDescribeAndListNodegroups(t *testing.T) {
|
||||
seedActiveClusterWithToken(t, f, "c1")
|
||||
_, err := f.svc.CreateNodegroup(createNGInput("c1", "ng1", 1), testAccountID)
|
||||
require.NoError(t, err)
|
||||
markWorkersReady(t, f, "c1", 1)
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
desc, err := f.svc.DescribeNodegroup(&eks.DescribeNodegroupInput{
|
||||
@@ -346,6 +381,7 @@ func TestUpdateNodegroupConfig_ScaleUpAndDown(t *testing.T) {
|
||||
seedActiveClusterWithToken(t, f, "c1")
|
||||
_, err := f.svc.CreateNodegroup(createNGInput("c1", "ng1", 1), testAccountID)
|
||||
require.NoError(t, err)
|
||||
markWorkersReady(t, f, "c1", 1)
|
||||
f.svc.WaitLaunches()
|
||||
require.Len(t, f.worker.runCalls, 1)
|
||||
|
||||
@@ -381,6 +417,7 @@ func TestDeleteNodegroup_TerminatesAndIsIdempotent(t *testing.T) {
|
||||
seedActiveClusterWithToken(t, f, "c1")
|
||||
_, err := f.svc.CreateNodegroup(createNGInput("c1", "ng1", 2), testAccountID)
|
||||
require.NoError(t, err)
|
||||
markWorkersReady(t, f, "c1", 2)
|
||||
f.svc.WaitLaunches()
|
||||
|
||||
_, err = f.svc.DeleteNodegroup(&eks.DeleteNodegroupInput{
|
||||
|
||||
@@ -2,26 +2,27 @@ package handlers_eks
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// agentUserDataInput is the input shape for buildAgentUserData. ControlPlaneENIIP
|
||||
// is the cluster's in-VPC control-plane ENI private IP (the K3s join target);
|
||||
// agentUserDataInput is the input shape for buildAgentUserData. ServerURL is the
|
||||
// cluster's published apiserver endpoint (the NLB front-end, https://<ip>:443),
|
||||
// the K3s join target reachable from the customer VPC under the Set-B topology;
|
||||
// JoinToken is the decrypted K3s node token.
|
||||
type agentUserDataInput struct {
|
||||
ClusterName string
|
||||
NodegroupName string
|
||||
ControlPlaneENIIP string
|
||||
JoinToken string
|
||||
NodeName string
|
||||
ClusterName string
|
||||
NodegroupName string
|
||||
ServerURL string
|
||||
JoinToken string
|
||||
NodeName string
|
||||
}
|
||||
|
||||
// buildAgentUserData renders the cloud-config YAML for a nodegroup worker VM.
|
||||
// Mirrors buildK3sUserData structure; SPINIFEX_K3S_ROLE=agent enables k3s-agent.
|
||||
func buildAgentUserData(in agentUserDataInput) string {
|
||||
// K3S_URL uses the CP ENI IP directly (NLB DNS not yet resolvable in-VPC).
|
||||
k3sURL := "https://" + net.JoinHostPort(in.ControlPlaneENIIP, "6443")
|
||||
// K3S_URL is the published NLB endpoint (:443→:6443); the in-VPC CP ENI IP
|
||||
// lives in the unpeered managed CP VPC and is unreachable from the worker VPC.
|
||||
k3sURL := in.ServerURL
|
||||
nodeLabel := "eks.amazonaws.com/nodegroup=" + in.NodegroupName
|
||||
|
||||
envBody := strings.Join([]string{
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package handlers_eks
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/tags"
|
||||
)
|
||||
|
||||
// PrivateEndpointENI is the customer-VPC (Set A) ENI threaded onto the cluster
|
||||
// NLB's LB VM to give in-VPC clients a private apiserver endpoint.
|
||||
type PrivateEndpointENI struct {
|
||||
ENIID string
|
||||
ENIIP string
|
||||
ENIMac string
|
||||
SGID string
|
||||
}
|
||||
|
||||
// EnsurePrivateEndpointENI provisions the customer-VPC SG (admitting the customer
|
||||
// VPC CIDR on :443) and a customer-account ENI in the first customer subnet. The
|
||||
// ENI's private IP is the private apiserver endpoint: it is threaded onto the
|
||||
// cluster NLB's LB VM as a cross-account extra NIC, so in-VPC workers + kubectl
|
||||
// reach the control plane without the public hairpin / NAT GW egress. Created
|
||||
// before the NLB + CP VMs so its IP is a known cert SAN.
|
||||
func EnsurePrivateEndpointENI(
|
||||
vpcSvc k3sVPCProvisioner,
|
||||
sgp sgProvisioner,
|
||||
resolver SubnetVPCResolver,
|
||||
accountID, clusterName, subnetID, vpcID string,
|
||||
) (*PrivateEndpointENI, error) {
|
||||
if accountID == "" {
|
||||
return nil, errors.New("eks: EnsurePrivateEndpointENI empty account id")
|
||||
}
|
||||
if subnetID == "" {
|
||||
return nil, errors.New("eks: EnsurePrivateEndpointENI empty subnet id")
|
||||
}
|
||||
if vpcID == "" {
|
||||
return nil, errors.New("eks: EnsurePrivateEndpointENI empty vpc id")
|
||||
}
|
||||
|
||||
vpcCIDR, err := resolver.GetVPCCIDR(accountID, vpcID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve customer VPC %s CIDR: %w", vpcID, err)
|
||||
}
|
||||
|
||||
sgID, err := EnsurePrivateEndpointSG(sgp, accountID, clusterName, vpcID, vpcCIDR)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
eniOut, err := vpcSvc.CreateNetworkInterface(&ec2.CreateNetworkInterfaceInput{
|
||||
SubnetId: aws.String(subnetID),
|
||||
Description: aws.String("EKS private-endpoint ENI for " + clusterName),
|
||||
Groups: aws.StringSlice([]string{sgID}),
|
||||
TagSpecifications: []*ec2.TagSpecification{{
|
||||
ResourceType: aws.String("network-interface"),
|
||||
Tags: []*ec2.Tag{
|
||||
{Key: aws.String(tags.ManagedByKey), Value: aws.String(tags.ManagedByEKS)},
|
||||
{Key: aws.String(clusterEKSClusterTagKey), Value: aws.String(clusterName)},
|
||||
{Key: aws.String(clusterEKSAccountTagKey), Value: aws.String(accountID)},
|
||||
{Key: aws.String(clusterEKSRoleTagKey), Value: aws.String(clusterEKSRolePrivateEndpoint)},
|
||||
},
|
||||
}},
|
||||
}, accountID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create private-endpoint ENI in subnet %s: %w", subnetID, err)
|
||||
}
|
||||
if eniOut == nil || eniOut.NetworkInterface == nil ||
|
||||
aws.StringValue(eniOut.NetworkInterface.NetworkInterfaceId) == "" ||
|
||||
aws.StringValue(eniOut.NetworkInterface.PrivateIpAddress) == "" {
|
||||
return nil, errors.New("eks: CreateNetworkInterface returned incomplete private-endpoint ENI")
|
||||
}
|
||||
|
||||
pe := &PrivateEndpointENI{
|
||||
ENIID: aws.StringValue(eniOut.NetworkInterface.NetworkInterfaceId),
|
||||
ENIIP: aws.StringValue(eniOut.NetworkInterface.PrivateIpAddress),
|
||||
ENIMac: aws.StringValue(eniOut.NetworkInterface.MacAddress),
|
||||
SGID: sgID,
|
||||
}
|
||||
slog.Info("EnsurePrivateEndpointENI provisioned",
|
||||
"clusterName", clusterName, "accountID", accountID,
|
||||
"eniId", pe.ENIID, "eniIp", pe.ENIIP, "sgId", sgID)
|
||||
return pe, nil
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
package handlers_eks
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/sysinstance"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestClusterJoinEndpoint_PrivateAccessPrefersSetA(t *testing.T) {
|
||||
// public+private: published endpoint is public, but workers join via Set A.
|
||||
meta := &ClusterMeta{
|
||||
Endpoint: "https://203.0.113.9:443",
|
||||
PrivateEndpointIP: "10.20.0.5",
|
||||
ResourcesVpcConfig: &ClusterVpcConfig{
|
||||
EndpointPublicAccess: true,
|
||||
EndpointPrivateAccess: true,
|
||||
},
|
||||
}
|
||||
assert.Equal(t, "https://10.20.0.5:443", clusterJoinEndpoint(meta))
|
||||
}
|
||||
|
||||
func TestClusterJoinEndpoint_PrivateOnlyMatchesPublished(t *testing.T) {
|
||||
// private-only: meta.Endpoint already equals the Set A endpoint.
|
||||
meta := &ClusterMeta{
|
||||
Endpoint: "https://10.20.0.5:443",
|
||||
PrivateEndpointIP: "10.20.0.5",
|
||||
ResourcesVpcConfig: &ClusterVpcConfig{
|
||||
EndpointPublicAccess: false,
|
||||
EndpointPrivateAccess: true,
|
||||
},
|
||||
}
|
||||
assert.Equal(t, "https://10.20.0.5:443", clusterJoinEndpoint(meta))
|
||||
}
|
||||
|
||||
func TestClusterJoinEndpoint_PublicOnlyUsesPublished(t *testing.T) {
|
||||
meta := &ClusterMeta{
|
||||
Endpoint: "https://203.0.113.9:443",
|
||||
ResourcesVpcConfig: &ClusterVpcConfig{
|
||||
EndpointPublicAccess: true,
|
||||
EndpointPrivateAccess: false,
|
||||
},
|
||||
}
|
||||
assert.Equal(t, "https://203.0.113.9:443", clusterJoinEndpoint(meta))
|
||||
}
|
||||
|
||||
func TestEnsurePrivateEndpointSG_AuthorizesVPCCIDROn443(t *testing.T) {
|
||||
sgp := newFakeSGProvisioner()
|
||||
sgp.createIDs = []string{"sg-pe-001"}
|
||||
|
||||
sgID, err := EnsurePrivateEndpointSG(sgp, "111122223333", "alpha", "vpc-aaa", "10.0.0.0/16")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "sg-pe-001", sgID)
|
||||
|
||||
require.Len(t, sgp.authorizeCalls, 1)
|
||||
in := sgp.authorizeCalls[0]
|
||||
assert.Equal(t, "sg-pe-001", aws.StringValue(in.GroupId))
|
||||
require.Len(t, in.IpPermissions, 1)
|
||||
perm := in.IpPermissions[0]
|
||||
assert.Equal(t, "tcp", aws.StringValue(perm.IpProtocol))
|
||||
assert.Equal(t, clusterNLBListenPort, aws.Int64Value(perm.FromPort))
|
||||
assert.Equal(t, clusterNLBListenPort, aws.Int64Value(perm.ToPort))
|
||||
require.Len(t, perm.IpRanges, 1)
|
||||
assert.Equal(t, "10.0.0.0/16", aws.StringValue(perm.IpRanges[0].CidrIp))
|
||||
}
|
||||
|
||||
func TestEnsurePrivateEndpointSG_EmptyInputsRejected(t *testing.T) {
|
||||
sgp := newFakeSGProvisioner()
|
||||
_, err := EnsurePrivateEndpointSG(sgp, "111122223333", "", "vpc-aaa", "10.0.0.0/16")
|
||||
require.Error(t, err)
|
||||
_, err = EnsurePrivateEndpointSG(sgp, "111122223333", "alpha", "", "10.0.0.0/16")
|
||||
require.Error(t, err)
|
||||
_, err = EnsurePrivateEndpointSG(sgp, "111122223333", "alpha", "vpc-aaa", "")
|
||||
require.Error(t, err)
|
||||
assert.Empty(t, sgp.createCalls)
|
||||
}
|
||||
|
||||
func TestEnsurePrivateEndpointSG_DuplicateIngressTolerated(t *testing.T) {
|
||||
sgp := newFakeSGProvisioner()
|
||||
sgp.authorizeErr = errors.New(awserrors.ErrorInvalidPermissionDuplicate)
|
||||
|
||||
_, err := EnsurePrivateEndpointSG(sgp, "111122223333", "alpha", "vpc-aaa", "10.0.0.0/16")
|
||||
require.NoError(t, err, "a duplicate :443 rule on re-run must be treated as success")
|
||||
}
|
||||
|
||||
func TestEnsurePrivateEndpointENI_HappyPath(t *testing.T) {
|
||||
vpcSvc := &fakeK3sVPC{
|
||||
createOut: &ec2.CreateNetworkInterfaceOutput{
|
||||
NetworkInterface: &ec2.NetworkInterface{
|
||||
NetworkInterfaceId: aws.String("eni-pe-001"),
|
||||
PrivateIpAddress: aws.String("10.0.1.50"),
|
||||
MacAddress: aws.String("02:0a:01:23:45:67"),
|
||||
SubnetId: aws.String("subnet-aaa"),
|
||||
},
|
||||
},
|
||||
}
|
||||
sgp := newFakeSGProvisioner()
|
||||
sgp.createIDs = []string{"sg-pe-001"}
|
||||
|
||||
pe, err := EnsurePrivateEndpointENI(vpcSvc, sgp, fakeSubnetResolver{}, "111122223333", "alpha", "subnet-aaa", "vpc-aaa")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "eni-pe-001", pe.ENIID)
|
||||
assert.Equal(t, "10.0.1.50", pe.ENIIP)
|
||||
assert.Equal(t, "02:0a:01:23:45:67", pe.ENIMac)
|
||||
assert.Equal(t, "sg-pe-001", pe.SGID)
|
||||
|
||||
// ENI must be created in the customer subnet, bound to the private-endpoint SG.
|
||||
require.Len(t, vpcSvc.createCalls, 1)
|
||||
createIn := vpcSvc.createCalls[0]
|
||||
assert.Equal(t, "subnet-aaa", aws.StringValue(createIn.SubnetId))
|
||||
require.Len(t, createIn.Groups, 1)
|
||||
assert.Equal(t, "sg-pe-001", aws.StringValue(createIn.Groups[0]))
|
||||
}
|
||||
|
||||
func TestEnsurePrivateEndpointENI_EmptyInputsRejected(t *testing.T) {
|
||||
vpcSvc := &fakeK3sVPC{}
|
||||
sgp := newFakeSGProvisioner()
|
||||
_, err := EnsurePrivateEndpointENI(vpcSvc, sgp, fakeSubnetResolver{}, "", "alpha", "subnet-aaa", "vpc-aaa")
|
||||
require.Error(t, err)
|
||||
_, err = EnsurePrivateEndpointENI(vpcSvc, sgp, fakeSubnetResolver{}, "111122223333", "alpha", "", "vpc-aaa")
|
||||
require.Error(t, err)
|
||||
_, err = EnsurePrivateEndpointENI(vpcSvc, sgp, fakeSubnetResolver{}, "111122223333", "alpha", "subnet-aaa", "")
|
||||
require.Error(t, err)
|
||||
assert.Empty(t, vpcSvc.createCalls)
|
||||
}
|
||||
|
||||
func TestEnsureClusterNLB_ThreadsCrossAccountENIToSyncCreate(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
extras := []sysinstance.ExtraENIInput{{
|
||||
ENIID: "eni-pe-001",
|
||||
ENIMac: "02:0a:01:23:45:67",
|
||||
ENIIP: "10.0.1.50",
|
||||
SubnetID: "subnet-aaa",
|
||||
AccountID: "111122223333",
|
||||
}}
|
||||
|
||||
_, err := EnsureClusterNLB(nlbp, "000000000000", "alpha", []string{"subnet-cp"}, false, nil, extras)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, nlbp.createClusterNLBExtras, 1, "extras present must route through CreateClusterNLBSync")
|
||||
require.Len(t, nlbp.createClusterNLBExtras[0], 1)
|
||||
assert.Equal(t, "eni-pe-001", nlbp.createClusterNLBExtras[0][0].ENIID)
|
||||
assert.Equal(t, "111122223333", nlbp.createClusterNLBExtras[0][0].AccountID)
|
||||
}
|
||||
|
||||
func TestEnsureClusterNLB_NoExtrasUsesPlainSync(t *testing.T) {
|
||||
nlbp := newFakeNLBProvisioner()
|
||||
|
||||
_, err := EnsureClusterNLB(nlbp, "000000000000", "alpha", []string{"subnet-cp"}, false, nil, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, nlbp.createClusterNLBExtras, "no extras must use the plain sync create path")
|
||||
assert.Len(t, nlbp.createLBSyncCalls, 1)
|
||||
}
|
||||
@@ -35,10 +35,17 @@ const clusterEKSAccountTagKey = "spinifex:eks-cluster-account"
|
||||
const clusterEKSNodegroupTagKey = "spinifex:eks-nodegroup"
|
||||
|
||||
const (
|
||||
clusterEKSRoleControlPlane = "control-plane"
|
||||
clusterEKSRoleNodegroup = "nodegroup"
|
||||
clusterEKSRoleControlPlane = "control-plane"
|
||||
clusterEKSRoleNodegroup = "nodegroup"
|
||||
clusterEKSRolePrivateEndpoint = "private-endpoint"
|
||||
)
|
||||
|
||||
// ClusterPrivateEndpointSGName returns the deterministic customer-VPC SG name for
|
||||
// the cluster's private-endpoint ENI (the Set A NIC on the cluster NLB's LB VM).
|
||||
func ClusterPrivateEndpointSGName(clusterName string) string {
|
||||
return fmt.Sprintf("eks-cluster-%s-private-endpoint-sg", clusterName)
|
||||
}
|
||||
|
||||
// ClusterControlPlaneSGName returns the deterministic control-plane SG name for a cluster.
|
||||
func ClusterControlPlaneSGName(clusterName string) string {
|
||||
return fmt.Sprintf("eks-cluster-%s-control-plane-sg", clusterName)
|
||||
@@ -111,6 +118,70 @@ func DeleteClusterSGs(sgp sgProvisioner, accountID, clusterName, vpcID string) e
|
||||
return firstErr
|
||||
}
|
||||
|
||||
// EnsurePrivateEndpointSG creates (or reuses) the customer-VPC SG for the
|
||||
// private-endpoint ENI and admits the customer VPC CIDR on the NLB listen port
|
||||
// (:443), so in-VPC workers + kubectl reach the apiserver via the Set A NIC.
|
||||
// Idempotent: SG lookup-or-create + duplicate-tolerant ingress authorize.
|
||||
func EnsurePrivateEndpointSG(sgp sgProvisioner, accountID, clusterName, vpcID, vpcCIDR string) (string, error) {
|
||||
if clusterName == "" {
|
||||
return "", errors.New("eks: EnsurePrivateEndpointSG empty cluster name")
|
||||
}
|
||||
if vpcID == "" {
|
||||
return "", errors.New("eks: EnsurePrivateEndpointSG empty vpc id")
|
||||
}
|
||||
if vpcCIDR == "" {
|
||||
return "", errors.New("eks: EnsurePrivateEndpointSG empty vpc cidr")
|
||||
}
|
||||
|
||||
sgID, err := ensureClusterSG(sgp, accountID, vpcID, clusterName,
|
||||
ClusterPrivateEndpointSGName(clusterName),
|
||||
fmt.Sprintf("EKS private-endpoint SG for cluster %s", clusterName),
|
||||
clusterEKSRolePrivateEndpoint)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
perm := &ec2.IpPermission{
|
||||
IpProtocol: aws.String("tcp"),
|
||||
FromPort: aws.Int64(clusterNLBListenPort),
|
||||
ToPort: aws.Int64(clusterNLBListenPort),
|
||||
IpRanges: []*ec2.IpRange{{
|
||||
CidrIp: aws.String(vpcCIDR),
|
||||
Description: aws.String("EKS in-VPC clients to private apiserver endpoint"),
|
||||
}},
|
||||
}
|
||||
_, err = sgp.AuthorizeSecurityGroupIngress(&ec2.AuthorizeSecurityGroupIngressInput{
|
||||
GroupId: aws.String(sgID),
|
||||
IpPermissions: []*ec2.IpPermission{perm},
|
||||
}, accountID)
|
||||
if err != nil && !awserrors.IsErrorCode(err, awserrors.ErrorInvalidPermissionDuplicate) {
|
||||
return "", fmt.Errorf("authorize private-endpoint ingress on %s: %w", sgID, err)
|
||||
}
|
||||
return sgID, nil
|
||||
}
|
||||
|
||||
// DeletePrivateEndpointSG removes the cluster's private-endpoint SG. Missing SG
|
||||
// is a no-op. The private-endpoint ENI must be deleted first (it references this SG).
|
||||
func DeletePrivateEndpointSG(sgp sgProvisioner, accountID, clusterName, vpcID string) error {
|
||||
if clusterName == "" {
|
||||
return errors.New("eks: DeletePrivateEndpointSG empty cluster name")
|
||||
}
|
||||
if vpcID == "" {
|
||||
return errors.New("eks: DeletePrivateEndpointSG empty vpc id")
|
||||
}
|
||||
id, err := lookupSGByName(sgp, accountID, vpcID, ClusterPrivateEndpointSGName(clusterName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if id == "" {
|
||||
return nil
|
||||
}
|
||||
if _, err := sgp.DeleteSecurityGroup(&ec2.DeleteSecurityGroupInput{GroupId: aws.String(id)}, accountID); err != nil && !awserrors.IsNotFound(err) {
|
||||
return fmt.Errorf("delete private-endpoint SG %s: %w", id, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ensureClusterSG(sgp sgProvisioner, accountID, vpcID, clusterName, name, description, role string) (string, error) {
|
||||
if id, err := lookupSGByName(sgp, accountID, vpcID, name); err != nil {
|
||||
return "", err
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/sysinstance"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
@@ -98,6 +99,12 @@ type EKSServiceImpl struct {
|
||||
|
||||
// launchWG tracks in-flight async create launches for test determinism (WaitLaunches).
|
||||
launchWG sync.WaitGroup
|
||||
|
||||
// nodegroupReadyTimeout / nodegroupReadyPoll bound how long launchNodegroupInfra
|
||||
// waits for its workers to register Ready before marking the nodegroup
|
||||
// CREATE_FAILED. Tests inject small values.
|
||||
nodegroupReadyTimeout time.Duration
|
||||
nodegroupReadyPoll time.Duration
|
||||
}
|
||||
|
||||
var _ EKSService = (*EKSServiceImpl)(nil)
|
||||
@@ -122,11 +129,13 @@ func NewEKSServiceImpl(deps EKSServiceDeps) (*EKSServiceImpl, error) {
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
return &EKSServiceImpl{
|
||||
deps: deps,
|
||||
leaderKV: leaderKV,
|
||||
registry: NewReconcilerRegistry(),
|
||||
bgCtx: ctx,
|
||||
bgCancel: cancel,
|
||||
deps: deps,
|
||||
leaderKV: leaderKV,
|
||||
registry: NewReconcilerRegistry(),
|
||||
bgCtx: ctx,
|
||||
bgCancel: cancel,
|
||||
nodegroupReadyTimeout: defaultNodegroupReadyTimeout,
|
||||
nodegroupReadyPoll: defaultNodegroupReadyPoll,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -351,6 +360,7 @@ func (s *EKSServiceImpl) CreateCluster(input *eks.CreateClusterInput, accountID,
|
||||
subnetIDs: subnetIDs,
|
||||
vpcID: vpcID,
|
||||
publicAccess: publicAccess,
|
||||
privateAccess: privateAccess,
|
||||
publicCidrs: publicCidrs,
|
||||
input: input,
|
||||
meta: meta,
|
||||
@@ -370,6 +380,7 @@ type clusterLaunchCtx struct {
|
||||
subnetIDs []string
|
||||
vpcID string
|
||||
publicAccess bool
|
||||
privateAccess bool
|
||||
publicCidrs []string
|
||||
input *eks.CreateClusterInput
|
||||
meta *ClusterMeta
|
||||
@@ -409,6 +420,7 @@ func (s *EKSServiceImpl) launchClusterInfra(lc clusterLaunchCtx) {
|
||||
name := lc.name
|
||||
region := lc.region
|
||||
publicAccess := lc.publicAccess
|
||||
privateAccess := lc.privateAccess
|
||||
publicCidrs := lc.publicCidrs
|
||||
input := lc.input
|
||||
meta := lc.meta
|
||||
@@ -457,22 +469,57 @@ func (s *EKSServiceImpl) launchClusterInfra(lc clusterLaunchCtx) {
|
||||
return
|
||||
}
|
||||
|
||||
// Private endpoint (301): when private access is on, give the cluster NLB a
|
||||
// customer-VPC (Set A) front-end so in-VPC workers + kubectl reach the control
|
||||
// plane without the public hairpin / NAT GW egress. Provision the customer-
|
||||
// account ENI (admitted by a customer-VPC SG on :443) before the NLB + CP VMs so
|
||||
// its IP is a known cert SAN, then thread it onto the LB VM as a cross-account
|
||||
// extra NIC. Persist its refs immediately so teardown reclaims them on any later
|
||||
// failure. nginx(L4) on the LB VM binds all addresses, so it answers the Set A
|
||||
// NIC and proxies to the CP target group with no data-plane change.
|
||||
var crossAccountENIs []sysinstance.ExtraENIInput
|
||||
if privateAccess {
|
||||
pe, perr := EnsurePrivateEndpointENI(s.deps.VPCK3s, s.deps.VPCSG, s.deps.VPCSubnet, accountID, name, lc.subnetIDs[0], lc.vpcID)
|
||||
if perr != nil {
|
||||
s.failClusterLaunch(acctKV, name, accountID, meta, "ensure private endpoint ENI", perr)
|
||||
return
|
||||
}
|
||||
meta.PrivateEndpointENIID = pe.ENIID
|
||||
meta.PrivateEndpointIP = pe.ENIIP
|
||||
if err := PutClusterMeta(acctKV, meta); err != nil {
|
||||
s.failClusterLaunch(acctKV, name, accountID, meta, "persist private endpoint refs", err)
|
||||
return
|
||||
}
|
||||
crossAccountENIs = []sysinstance.ExtraENIInput{{
|
||||
ENIID: pe.ENIID,
|
||||
ENIMac: pe.ENIMac,
|
||||
ENIIP: pe.ENIIP,
|
||||
SubnetID: lc.subnetIDs[0],
|
||||
AccountID: accountID,
|
||||
}}
|
||||
}
|
||||
|
||||
// The cluster NLB lives in the CP VPC public subnet. publicAccess selects the
|
||||
// scheme (internet-facing external-pool IP vs internal VPC IP); the CP VPC IGW
|
||||
// (attached by EnsureClusterCPVPC) makes an internet-facing front-end IP
|
||||
// answerable on the wire.
|
||||
nlb, err := EnsureClusterNLB(s.deps.NLB, sysAcct, name, []string{cpRefs.PublicSubnetID}, publicAccess, publicCidrs)
|
||||
nlb, err := EnsureClusterNLB(s.deps.NLB, sysAcct, name, []string{cpRefs.PublicSubnetID}, publicAccess, publicCidrs, crossAccountENIs)
|
||||
if err != nil {
|
||||
s.failClusterLaunch(acctKV, name, accountID, meta, "ensure cluster NLB", err)
|
||||
return
|
||||
}
|
||||
// The reachable front-end IP is the kubeconfig endpoint host so the apiserver
|
||||
// serving cert (which SANs this IP) validates with TLS verification on.
|
||||
// EnsureClusterNLB guarantees a non-empty FrontendIP (it fails the launch
|
||||
// otherwise), so there is no DNS-name fallback to bake an unresolvable,
|
||||
// non-SANed endpoint.
|
||||
meta.Endpoint = "https://" + net.JoinHostPort(nlb.FrontendIP, strconv.FormatInt(clusterNLBListenPort, 10))
|
||||
meta.EndpointIP = nlb.FrontendIP
|
||||
// Endpoint resolution (301): publish the reachable front-end. With public access
|
||||
// on (public-only or public+private) that is the NLB's public front-end IP. A
|
||||
// private-only cluster publishes its Set A private endpoint instead — the
|
||||
// internal NLB's Set B IP is unreachable from the customer VPC. Either way the
|
||||
// host is a cert SAN, so TLS validates with verification on.
|
||||
if publicAccess {
|
||||
meta.Endpoint = "https://" + net.JoinHostPort(nlb.FrontendIP, strconv.FormatInt(clusterNLBListenPort, 10))
|
||||
meta.EndpointIP = nlb.FrontendIP
|
||||
} else {
|
||||
meta.Endpoint = "https://" + net.JoinHostPort(meta.PrivateEndpointIP, strconv.FormatInt(clusterNLBListenPort, 10))
|
||||
meta.EndpointIP = meta.PrivateEndpointIP
|
||||
}
|
||||
meta.NLBArn = nlb.LoadBalancerArn
|
||||
meta.NLBTargetGroupArn = nlb.TargetGroupArn
|
||||
|
||||
@@ -515,6 +562,7 @@ func (s *EKSServiceImpl) launchClusterInfra(lc clusterLaunchCtx) {
|
||||
ControlPlaneSGID: cpSG,
|
||||
NLBDNS: nlb.DNSName,
|
||||
EndpointIP: nlb.FrontendIP,
|
||||
PrivateEndpointIP: meta.PrivateEndpointIP,
|
||||
OIDCIssuer: oidcIssuer,
|
||||
OIDCPrivateKeyPEM: privPEM,
|
||||
OIDCPublicKeyPEM: pubPEM,
|
||||
@@ -854,6 +902,21 @@ func (s *EKSServiceImpl) purgeClusterInfra(accountID, name string, meta *Cluster
|
||||
}
|
||||
}
|
||||
|
||||
// Private endpoint (301): the Set A ENI lives in the customer VPC under the
|
||||
// customer account and was attached to the cluster NLB's LB VM (terminated by
|
||||
// DeleteClusterNLB above). It is an extra NIC, not the LB VM's primary ENI, so
|
||||
// the instance-terminate cascade never reclaims it — detach (store-clear) the
|
||||
// stale attachment first, then delete before its SG. Detach is best-effort: a
|
||||
// missing/already-detached ENI is fine; the delete is the authoritative gate.
|
||||
if meta.PrivateEndpointENIID != "" {
|
||||
if err := s.deps.VPCK3s.DetachENI(accountID, meta.PrivateEndpointENIID); err != nil {
|
||||
slog.Warn("purgeClusterInfra: detach private-endpoint ENI", "cluster", name, "eni", meta.PrivateEndpointENIID, "err", err)
|
||||
}
|
||||
if err := deleteENIAwaitingTerminate(s.deps.VPCK3s, accountID, meta.PrivateEndpointENIID); err != nil {
|
||||
teardownErrs = append(teardownErrs, fmt.Errorf("delete private-endpoint ENI: %w", err))
|
||||
}
|
||||
}
|
||||
|
||||
for _, cp := range controlPlaneTeardownNodes(meta) {
|
||||
if err := TerminateK3sServerVM(s.deps.VPCK3s, s.deps.Instance,
|
||||
infraAcct, cp.InstanceID, cp.ENIID); err != nil {
|
||||
@@ -902,6 +965,14 @@ func (s *EKSServiceImpl) purgeClusterInfra(accountID, name string, meta *Cluster
|
||||
return errors.Join(teardownErrs...)
|
||||
}
|
||||
|
||||
// Private endpoint (301): reclaim the customer-VPC SG now its ENI is gone.
|
||||
// Best-effort — its only billable dependant (the ENI) is already deleted.
|
||||
if meta.ResourcesVpcConfig != nil && meta.ResourcesVpcConfig.VpcId != "" {
|
||||
if err := DeletePrivateEndpointSG(s.deps.VPCSG, accountID, name, meta.ResourcesVpcConfig.VpcId); err != nil {
|
||||
slog.Warn("purgeClusterInfra: delete private-endpoint SG failed", "cluster", name, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
if meta.ManagedCPVPC != nil && meta.ManagedCPVPC.VpcId != "" {
|
||||
// New topology: SGs + IGW + subnets + route tables + NAT GW + VPC all live
|
||||
// in the managed CP VPC under the system account. DeleteClusterCPVPC removes
|
||||
@@ -1506,6 +1577,19 @@ func publicAccessCidrs(vpc *eks.VpcConfigRequest, public bool) []string {
|
||||
return []string{defaultPublicAccessCidr}
|
||||
}
|
||||
|
||||
// clusterJoinEndpoint is the apiserver URL nodegroup workers join through. With
|
||||
// private access on, workers prefer the customer-VPC (Set A) private endpoint on
|
||||
// :443 so a private-only cluster needs no NAT GW egress — the core 301 win.
|
||||
// Otherwise (public-only, or no provisioned private endpoint) the published
|
||||
// endpoint is used. For a private-only cluster meta.Endpoint already equals the
|
||||
// private endpoint, so the two agree.
|
||||
func clusterJoinEndpoint(meta *ClusterMeta) string {
|
||||
if meta.ResourcesVpcConfig != nil && meta.ResourcesVpcConfig.EndpointPrivateAccess && meta.PrivateEndpointIP != "" {
|
||||
return "https://" + net.JoinHostPort(meta.PrivateEndpointIP, strconv.FormatInt(clusterNLBListenPort, 10))
|
||||
}
|
||||
return meta.Endpoint
|
||||
}
|
||||
|
||||
func (s *EKSServiceImpl) markFailed(kv nats.KeyValue, name string) {
|
||||
if err := SetClusterStatus(kv, name, ClusterStatusFailed); err != nil {
|
||||
slog.Warn("CreateCluster: SetClusterStatus(FAILED) failed", "cluster", name, "err", err)
|
||||
|
||||
@@ -338,8 +338,14 @@ func (s *ELBv2ServiceImpl) buildMicrovmNICs(primaryIP, primaryMAC, primarySubnet
|
||||
for _, extra := range extraENIs {
|
||||
extraCIDR := ""
|
||||
extraGateway := ""
|
||||
// Cross-account extras carry their own AccountID; the subnet record is
|
||||
// account-keyed, so resolve under it. Empty falls back to accountID.
|
||||
extraAccount := extra.AccountID
|
||||
if extraAccount == "" {
|
||||
extraAccount = accountID
|
||||
}
|
||||
if s.VPCService != nil && extra.SubnetID != "" {
|
||||
if subnet, err := s.VPCService.GetSubnet(accountID, extra.SubnetID); err == nil && subnet != nil {
|
||||
if subnet, err := s.VPCService.GetSubnet(extraAccount, extra.SubnetID); err == nil && subnet != nil {
|
||||
extraCIDR = subnetCIDRForIP(extra.ENIIP, subnet.CidrBlock)
|
||||
extraGateway = subnetGatewayIP(subnet.CidrBlock)
|
||||
} else {
|
||||
@@ -418,7 +424,7 @@ type lbVMLaunch struct {
|
||||
// launchLBVM boots the system VM for a load balancer. The first ENI is the
|
||||
// primary NIC; extras give multi-subnet data-plane presence. No-op when no
|
||||
// launcher is configured. Shared by CreateLoadBalancer and SetSubnets.
|
||||
func (s *ELBv2ServiceImpl) launchLBVM(lbID, scheme string, eniIDs, subnets []string, accountID string) lbVMLaunch {
|
||||
func (s *ELBv2ServiceImpl) launchLBVM(lbID, scheme string, eniIDs, subnets []string, accountID string, crossAccountENIs []ExtraENIInput) lbVMLaunch {
|
||||
var res lbVMLaunch
|
||||
if s.InstanceLauncher == nil || len(eniIDs) == 0 || len(subnets) == 0 {
|
||||
return res
|
||||
@@ -433,7 +439,11 @@ func (s *ELBv2ServiceImpl) launchLBVM(lbID, scheme string, eniIDs, subnets []str
|
||||
primaryMAC = aws.StringValue(primary.MacAddress)
|
||||
}
|
||||
|
||||
// Same-account extras come from the (already-described) primary ENI set;
|
||||
// cross-account extras arrive fully populated (the caller created them in the
|
||||
// other account) and are not in eniDetails, so append them directly.
|
||||
extraENIInputs := buildExtraENIInputs(eniIDs, eniDetails)
|
||||
extraENIInputs = append(extraENIInputs, crossAccountENIs...)
|
||||
|
||||
if s.GatewayURL == "" || s.SystemAccessKey == "" || s.SystemSecretKey == "" {
|
||||
slog.Error("launchLBVM: system credentials not configured — cannot launch LB VM", "lbId", lbID)
|
||||
@@ -528,6 +538,9 @@ func (s *ELBv2ServiceImpl) RebuildSystemInstanceInput(ctx RecoveryContext) (*Sys
|
||||
return nil, fmt.Errorf("no MAC for primary ENI %s of lb %s", lb.ENIs[0], lb.LoadBalancerID)
|
||||
}
|
||||
extraENIs := buildExtraENIInputs(lb.ENIs, eniDetails)
|
||||
// Cross-account extras live in another account and aren't in lb.ENIs/eniDetails;
|
||||
// they were persisted fully populated, so re-attach them as-is.
|
||||
extraENIs = append(extraENIs, lb.CrossAccountENIs...)
|
||||
|
||||
nics := s.buildMicrovmNICs(lb.VPCIP, primaryMAC, lb.Subnets[0], lb.ENIs[0], lb.Scheme, extraENIs, lb.AccountID)
|
||||
// Re-inject mgmt NIC MAC/CIDR — buildMicrovmNICs leaves them blank.
|
||||
@@ -1037,7 +1050,7 @@ func isCompatibleProtocol(listenerProto, tgProto string) bool {
|
||||
// need the LB's front-end address before proceeding — e.g. EKS, which bakes the
|
||||
// IP into the control-plane apiserver cert SAN — must use CreateLoadBalancerSync.
|
||||
func (s *ELBv2ServiceImpl) CreateLoadBalancer(input *elbv2.CreateLoadBalancerInput, accountID string) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
return s.createLoadBalancer(input, accountID, false)
|
||||
return s.createLoadBalancer(input, accountID, false, nil)
|
||||
}
|
||||
|
||||
// CreateLoadBalancerSync creates the LB and drives its data-plane launch
|
||||
@@ -1048,10 +1061,20 @@ func (s *ELBv2ServiceImpl) CreateLoadBalancer(input *elbv2.CreateLoadBalancerInp
|
||||
// running it on the shared gateway responder would reintroduce 267.4. The LB
|
||||
// still flips provisioning → active on the lb-agent's first heartbeat.
|
||||
func (s *ELBv2ServiceImpl) CreateLoadBalancerSync(input *elbv2.CreateLoadBalancerInput, accountID string) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
return s.createLoadBalancer(input, accountID, true)
|
||||
return s.createLoadBalancer(input, accountID, true, nil)
|
||||
}
|
||||
|
||||
func (s *ELBv2ServiceImpl) createLoadBalancer(input *elbv2.CreateLoadBalancerInput, accountID string, syncLaunch bool) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
// CreateClusterNLBSync creates an NLB synchronously (like CreateLoadBalancerSync)
|
||||
// and also threads one or more cross-account ENIs onto the LB VM at launch. EKS
|
||||
// uses it to give an otherwise system-VPC cluster NLB a customer-VPC (Set A)
|
||||
// front-end, so in-VPC clients and workers reach the control plane privately
|
||||
// while inheriting CP-level HA from the NLB target group. Each ExtraENIInput
|
||||
// carries its own AccountID (the customer account that owns the Set A ENI).
|
||||
func (s *ELBv2ServiceImpl) CreateClusterNLBSync(input *elbv2.CreateLoadBalancerInput, accountID string, crossAccountENIs []ExtraENIInput) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
return s.createLoadBalancer(input, accountID, true, crossAccountENIs)
|
||||
}
|
||||
|
||||
func (s *ELBv2ServiceImpl) createLoadBalancer(input *elbv2.CreateLoadBalancerInput, accountID string, syncLaunch bool, crossAccountENIs []ExtraENIInput) (*elbv2.CreateLoadBalancerOutput, error) {
|
||||
if input.Name == nil || *input.Name == "" {
|
||||
return nil, errors.New(awserrors.ErrorMissingParameter)
|
||||
}
|
||||
@@ -1216,24 +1239,25 @@ func (s *ELBv2ServiceImpl) createLoadBalancer(input *elbv2.CreateLoadBalancerInp
|
||||
// Attributes left nil — defaults derived from lb.Type on read.
|
||||
// InstanceID/VPCIP/HostPorts are filled by the async launch.
|
||||
record := &LoadBalancerRecord{
|
||||
LoadBalancerArn: lbArn,
|
||||
LoadBalancerID: lbID,
|
||||
DNSName: dnsName,
|
||||
Name: name,
|
||||
Scheme: scheme,
|
||||
Type: lbType,
|
||||
State: state,
|
||||
VpcId: vpcID,
|
||||
SecurityGroups: securityGroups,
|
||||
NLBManagedSGID: nlbManagedSGID,
|
||||
Subnets: subnets,
|
||||
AvailZones: availZones,
|
||||
ENIs: eniIDs,
|
||||
IPAddressType: IPAddressTypeIPv4,
|
||||
NodeID: s.nodeID,
|
||||
Tags: tags,
|
||||
AccountID: accountID,
|
||||
CreatedAt: time.Now().UTC(),
|
||||
LoadBalancerArn: lbArn,
|
||||
LoadBalancerID: lbID,
|
||||
DNSName: dnsName,
|
||||
Name: name,
|
||||
Scheme: scheme,
|
||||
Type: lbType,
|
||||
State: state,
|
||||
VpcId: vpcID,
|
||||
SecurityGroups: securityGroups,
|
||||
NLBManagedSGID: nlbManagedSGID,
|
||||
Subnets: subnets,
|
||||
AvailZones: availZones,
|
||||
ENIs: eniIDs,
|
||||
CrossAccountENIs: crossAccountENIs,
|
||||
IPAddressType: IPAddressTypeIPv4,
|
||||
NodeID: s.nodeID,
|
||||
Tags: tags,
|
||||
AccountID: accountID,
|
||||
CreatedAt: time.Now().UTC(),
|
||||
}
|
||||
|
||||
if err := s.store.PutLoadBalancer(record); err != nil {
|
||||
@@ -1246,7 +1270,7 @@ func (s *ELBv2ServiceImpl) createLoadBalancer(input *elbv2.CreateLoadBalancerInp
|
||||
if willLaunch {
|
||||
// Snapshot the launch inputs; the async goroutine must not read the record
|
||||
// the caller is about to return.
|
||||
lc := lbLaunchCtx{lbID: lbID, lbArn: lbArn, scheme: scheme, eniIDs: eniIDs, subnets: subnets, accountID: accountID}
|
||||
lc := lbLaunchCtx{lbID: lbID, lbArn: lbArn, scheme: scheme, eniIDs: eniIDs, subnets: subnets, accountID: accountID, crossAccountENIs: crossAccountENIs}
|
||||
if syncLaunch {
|
||||
// Drive the data-plane launch inline so the returned record carries the
|
||||
// allocated front-end IP. The caller is off the gateway responder, so
|
||||
@@ -1292,6 +1316,9 @@ type lbLaunchCtx struct {
|
||||
eniIDs []string
|
||||
subnets []string
|
||||
accountID string
|
||||
// crossAccountENIs are extra ENIs owned by a different account than eniIDs
|
||||
// (each carries its own AccountID), threaded onto the LB VM at launch.
|
||||
crossAccountENIs []ExtraENIInput
|
||||
}
|
||||
|
||||
// provisionLBDataPlane boots the LB-VM and folds the result into the persisted
|
||||
@@ -1303,7 +1330,7 @@ type lbLaunchCtx struct {
|
||||
// whether to run it inline (CreateLoadBalancerSync) or on a background goroutine
|
||||
// (launchLBVMAsync).
|
||||
func (s *ELBv2ServiceImpl) provisionLBDataPlane(lc lbLaunchCtx) error {
|
||||
launch := s.launchLBVM(lc.lbID, lc.scheme, lc.eniIDs, lc.subnets, lc.accountID)
|
||||
launch := s.launchLBVM(lc.lbID, lc.scheme, lc.eniIDs, lc.subnets, lc.accountID, lc.crossAccountENIs)
|
||||
|
||||
record, err := s.store.GetLoadBalancerByArn(lc.lbArn)
|
||||
if err != nil || record == nil {
|
||||
|
||||
@@ -214,7 +214,7 @@ func (s *ELBv2ServiceImpl) SetSubnets(input *elbv2.SetSubnetsInput, accountID st
|
||||
}
|
||||
}
|
||||
|
||||
launch := s.launchLBVM(lb.LoadBalancerID, lb.Scheme, newENIs, desired, accountID)
|
||||
launch := s.launchLBVM(lb.LoadBalancerID, lb.Scheme, newENIs, desired, accountID, lb.CrossAccountENIs)
|
||||
availZones := rebuildAvailZones(desired, lb.AvailZones, newAZBySubnet)
|
||||
if launch.publicIP != "" && len(availZones) > 0 {
|
||||
availZones[0].PublicIP = launch.publicIP
|
||||
|
||||
@@ -926,3 +926,56 @@ func TestRebuildSystemInstanceInput_MultiENI(t *testing.T) {
|
||||
assert.Equal(t, orig.ENIIP, e.ENIIP)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateClusterNLBSync_CarriesCrossAccountENI(t *testing.T) {
|
||||
svc, vpcSvc := setupTestServiceWithVPC(t)
|
||||
|
||||
subnets, err := vpcSvc.DescribeSubnets(&ec2.DescribeSubnetsInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
subnetID := *subnets.Subnets[0].SubnetId
|
||||
|
||||
mock := &mockSystemInstanceLauncher{
|
||||
launchResult: &SystemInstanceOutput{InstanceID: "i-cluster-nlb", PrivateIP: "10.0.1.9"},
|
||||
}
|
||||
svc.InstanceLauncher = mock
|
||||
svc.GatewayURL = "https://10.0.0.1:9999"
|
||||
svc.SystemAccessKey = "AKID"
|
||||
svc.SystemSecretKey = "SECRET"
|
||||
|
||||
// The Set A ENI lives in a different (customer) account than the LB's own
|
||||
// system-account ENIs; it arrives fully populated as the caller created it.
|
||||
extra := ExtraENIInput{
|
||||
ENIID: "eni-seta-001",
|
||||
ENIMac: "02:0a:01:23:45:67",
|
||||
ENIIP: "10.20.0.5",
|
||||
SubnetID: "subnet-seta",
|
||||
AccountID: "999988887777",
|
||||
}
|
||||
|
||||
_, err = svc.CreateClusterNLBSync(&elbv2.CreateLoadBalancerInput{
|
||||
Name: aws.String("eks-alpha"),
|
||||
Type: aws.String("network"),
|
||||
Scheme: aws.String("internal"),
|
||||
Subnets: []*string{aws.String(subnetID)},
|
||||
}, testAccountID, []ExtraENIInput{extra})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, mock.launchCalls, 1)
|
||||
launch := mock.launchCalls[0]
|
||||
|
||||
// The cross-account ENI must be threaded onto the LB VM with its own account.
|
||||
require.Len(t, launch.ExtraENIs, 1)
|
||||
assert.Equal(t, "eni-seta-001", launch.ExtraENIs[0].ENIID)
|
||||
assert.Equal(t, "999988887777", launch.ExtraENIs[0].AccountID)
|
||||
|
||||
// NIC[0] primary, NIC[1] mgmt, NIC[2] the Set A extra NIC.
|
||||
require.Len(t, launch.NICs, 3)
|
||||
assert.Equal(t, "02:0a:01:23:45:67", launch.NICs[2].MAC)
|
||||
|
||||
// And it is persisted for host-reboot recovery.
|
||||
record, err := svc.store.GetLoadBalancerByName("eks-alpha", testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, record)
|
||||
require.Len(t, record.CrossAccountENIs, 1)
|
||||
assert.Equal(t, "eni-seta-001", record.CrossAccountENIs[0].ENIID)
|
||||
}
|
||||
|
||||
@@ -116,24 +116,31 @@ type LoadBalancerRecord struct {
|
||||
// NLBIngressCIDRs overrides the scheme-based default client CIDRs that
|
||||
// listener ports are opened to on NLBManagedSGID. Empty ⇒ default
|
||||
// (internet-facing: 0.0.0.0/0; internal: the VPC CIDR).
|
||||
NLBIngressCIDRs []string `json:"nlb_ingress_cidrs,omitempty"`
|
||||
Subnets []string `json:"subnets"`
|
||||
AvailZones []AvailZoneInfo `json:"availability_zones"`
|
||||
ENIs []string `json:"enis,omitempty"` // ENI IDs created for this ALB (internal)
|
||||
InstanceID string `json:"instance_id,omitempty"` // ALB VM instance ID (system-managed)
|
||||
VPCIP string `json:"vpc_ip,omitempty"` // VPC private IP of the ALB VM
|
||||
ConfigText string `json:"config_text,omitempty"` // Pre-computed HAProxy config
|
||||
ConfigHash string `json:"config_hash,omitempty"` // SHA256 of ConfigText + cert material
|
||||
CertFiles map[string]string `json:"cert_files,omitempty"` // Absolute path → combined PEM (cert+chain+key), delivered with config
|
||||
HealthTargets []HealthTargetSpec `json:"health_targets,omitempty"` // NLB only: backends the nginx agent actively probes (HAProxy reports its own)
|
||||
LastHeartbeat time.Time `json:"last_heartbeat"` // Last agent heartbeat timestamp
|
||||
HostPorts map[int]int `json:"host_ports,omitempty"` // Dev mode: guest port → host port forwarding
|
||||
NodeID string `json:"node_id"` // Daemon node running this ALB
|
||||
IPAddressType string `json:"ip_address_type"` // "ipv4"
|
||||
Attributes map[string]string `json:"attributes,omitempty"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
AccountID string `json:"account_id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
NLBIngressCIDRs []string `json:"nlb_ingress_cidrs,omitempty"`
|
||||
Subnets []string `json:"subnets"`
|
||||
AvailZones []AvailZoneInfo `json:"availability_zones"`
|
||||
ENIs []string `json:"enis,omitempty"` // ENI IDs created for this ALB (internal)
|
||||
// CrossAccountENIs are extra ENIs owned by a different account than the
|
||||
// primary LB ENIs (e.g. an EKS cluster NLB whose LB VM lives in the system
|
||||
// CP VPC but also fronts a customer-VPC Set A ENI so in-VPC clients reach the
|
||||
// control plane privately, inheriting CP-level HA from the NLB target group).
|
||||
// Threaded onto the LB VM at launch and re-attached on host-reboot recovery;
|
||||
// kept out of ENIs so same-account describe/recovery stays single-account.
|
||||
CrossAccountENIs []ExtraENIInput `json:"cross_account_enis,omitempty"`
|
||||
InstanceID string `json:"instance_id,omitempty"` // ALB VM instance ID (system-managed)
|
||||
VPCIP string `json:"vpc_ip,omitempty"` // VPC private IP of the ALB VM
|
||||
ConfigText string `json:"config_text,omitempty"` // Pre-computed HAProxy config
|
||||
ConfigHash string `json:"config_hash,omitempty"` // SHA256 of ConfigText + cert material
|
||||
CertFiles map[string]string `json:"cert_files,omitempty"` // Absolute path → combined PEM (cert+chain+key), delivered with config
|
||||
HealthTargets []HealthTargetSpec `json:"health_targets,omitempty"` // NLB only: backends the nginx agent actively probes (HAProxy reports its own)
|
||||
LastHeartbeat time.Time `json:"last_heartbeat"` // Last agent heartbeat timestamp
|
||||
HostPorts map[int]int `json:"host_ports,omitempty"` // Dev mode: guest port → host port forwarding
|
||||
NodeID string `json:"node_id"` // Daemon node running this ALB
|
||||
IPAddressType string `json:"ip_address_type"` // "ipv4"
|
||||
Attributes map[string]string `json:"attributes,omitempty"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
AccountID string `json:"account_id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
// HealthTargetSpec is a backend the nginx agent actively health-checks, computed
|
||||
|
||||
@@ -24,6 +24,11 @@ const (
|
||||
defaultMaxSessionDuration = int64(3600)
|
||||
minMaxSessionDuration = int64(900)
|
||||
maxMaxSessionDuration = int64(43200)
|
||||
|
||||
// Bound on optimistic-concurrency retries when a concurrent writer wins the
|
||||
// CAS race on a role record. High enough to absorb the handful of parallel
|
||||
// AttachRolePolicy calls Terraform fans out per role.
|
||||
roleCASMaxRetries = 16
|
||||
)
|
||||
|
||||
func (s *IAMServiceImpl) CreateRole(accountID string, input *iam.CreateRoleInput) (*iam.CreateRoleOutput, error) {
|
||||
@@ -256,24 +261,17 @@ func (s *IAMServiceImpl) AttachRolePolicy(accountID string, input *iam.AttachRol
|
||||
}
|
||||
}
|
||||
|
||||
role, err := s.getRole(accountID, roleName)
|
||||
err := s.updateRoleCAS(accountID, roleName, func(role *Role) (bool, error) {
|
||||
if slices.Contains(role.AttachedPolicies, policyARN) {
|
||||
return false, nil
|
||||
}
|
||||
role.AttachedPolicies = append(role.AttachedPolicies, policyARN)
|
||||
return true, nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if slices.Contains(role.AttachedPolicies, policyARN) {
|
||||
return &iam.AttachRolePolicyOutput{}, nil
|
||||
}
|
||||
|
||||
role.AttachedPolicies = append(role.AttachedPolicies, policyARN)
|
||||
data, err := json.Marshal(role)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal role: %w", err)
|
||||
}
|
||||
if _, err := s.rolesBucket.Put(accountID+"."+roleName, data); err != nil {
|
||||
return nil, fmt.Errorf("update role: %w", err)
|
||||
}
|
||||
|
||||
slog.Info("IAM policy attached to role", "accountID", accountID, "roleName", roleName, "policyArn", policyARN)
|
||||
return &iam.AttachRolePolicyOutput{}, nil
|
||||
}
|
||||
@@ -282,33 +280,18 @@ func (s *IAMServiceImpl) DetachRolePolicy(accountID string, input *iam.DetachRol
|
||||
roleName := *input.RoleName
|
||||
policyARN := *input.PolicyArn
|
||||
|
||||
role, err := s.getRole(accountID, roleName)
|
||||
err := s.updateRoleCAS(accountID, roleName, func(role *Role) (bool, error) {
|
||||
idx := slices.Index(role.AttachedPolicies, policyARN)
|
||||
if idx < 0 {
|
||||
return false, errors.New(awserrors.ErrorIAMNoSuchEntity)
|
||||
}
|
||||
role.AttachedPolicies = slices.Delete(role.AttachedPolicies, idx, idx+1)
|
||||
return true, nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
found := false
|
||||
remaining := make([]string, 0, len(role.AttachedPolicies))
|
||||
for _, arn := range role.AttachedPolicies {
|
||||
if arn == policyARN {
|
||||
found = true
|
||||
} else {
|
||||
remaining = append(remaining, arn)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return nil, errors.New(awserrors.ErrorIAMNoSuchEntity)
|
||||
}
|
||||
|
||||
role.AttachedPolicies = remaining
|
||||
data, err := json.Marshal(role)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal role: %w", err)
|
||||
}
|
||||
if _, err := s.rolesBucket.Put(accountID+"."+roleName, data); err != nil {
|
||||
return nil, fmt.Errorf("update role: %w", err)
|
||||
}
|
||||
|
||||
slog.Info("IAM policy detached from role", "accountID", accountID, "roleName", roleName, "policyArn", policyARN)
|
||||
return &iam.DetachRolePolicyOutput{}, nil
|
||||
}
|
||||
@@ -402,6 +385,51 @@ func (s *IAMServiceImpl) getRole(accountID, roleName string) (*Role, error) {
|
||||
return &role, nil
|
||||
}
|
||||
|
||||
// updateRoleCAS applies mutate to a role under optimistic concurrency: read the
|
||||
// record with its revision, mutate, then Update guarded by that revision,
|
||||
// retrying when a concurrent writer wins the race. A blind read-modify-Put
|
||||
// loses updates when callers (e.g. Terraform attaching several managed policies
|
||||
// to one role at once) write the same record concurrently. mutate reports
|
||||
// whether it changed the record; a false return commits nothing.
|
||||
func (s *IAMServiceImpl) updateRoleCAS(accountID, roleName string, mutate func(*Role) (bool, error)) error {
|
||||
key := accountID + "." + roleName
|
||||
for range roleCASMaxRetries {
|
||||
entry, err := s.rolesBucket.Get(key)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrKeyNotFound) {
|
||||
return errors.New(awserrors.ErrorIAMNoSuchEntity)
|
||||
}
|
||||
return fmt.Errorf("get role: %w", err)
|
||||
}
|
||||
|
||||
var role Role
|
||||
if err := json.Unmarshal(entry.Value(), &role); err != nil {
|
||||
return fmt.Errorf("unmarshal role: %w", err)
|
||||
}
|
||||
|
||||
changed, err := mutate(&role)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !changed {
|
||||
return nil
|
||||
}
|
||||
|
||||
data, err := json.Marshal(&role)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal role: %w", err)
|
||||
}
|
||||
if _, err := s.rolesBucket.Update(key, data, entry.Revision()); err != nil {
|
||||
if errors.Is(err, nats.ErrKeyExists) {
|
||||
continue // CAS conflict — another writer won, re-read and retry.
|
||||
}
|
||||
return fmt.Errorf("update role: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// findInstanceProfilesForRole scans the instance-profiles bucket for any
|
||||
// profile in the account that references the given role. Fails closed on
|
||||
// per-key Get or unmarshal errors so DeleteRole cannot succeed while a
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
@@ -590,6 +591,50 @@ func TestDetachRolePolicy_NotAttached(t *testing.T) {
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorIAMNoSuchEntity)
|
||||
}
|
||||
|
||||
// TestAttachRolePolicy_ConcurrentDistinctPolicies reproduces the env19
|
||||
// lost-update: Terraform attaches the three EKS managed policies to one node
|
||||
// role in parallel. A blind read-modify-Put kept only one (the others raced
|
||||
// from the same revision and clobbered each other); the CAS path must persist
|
||||
// all three.
|
||||
func TestAttachRolePolicy_ConcurrentDistinctPolicies(t *testing.T) {
|
||||
svc := setupTestIAMService(t)
|
||||
role := createTestRole(t, svc, "eks-node-role")
|
||||
|
||||
arns := []string{
|
||||
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
|
||||
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
|
||||
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
errs := make([]error, len(arns))
|
||||
for i, arn := range arns {
|
||||
wg.Add(1)
|
||||
go func(i int, arn string) {
|
||||
defer wg.Done()
|
||||
_, errs[i] = svc.AttachRolePolicy(testAccountID, &iam.AttachRolePolicyInput{
|
||||
RoleName: role.RoleName,
|
||||
PolicyArn: aws.String(arn),
|
||||
})
|
||||
}(i, arn)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
for i, err := range errs {
|
||||
require.NoErrorf(t, err, "attach %s", arns[i])
|
||||
}
|
||||
|
||||
out, err := svc.ListAttachedRolePolicies(testAccountID, &iam.ListAttachedRolePoliciesInput{
|
||||
RoleName: role.RoleName,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
got := make([]string, 0, len(out.AttachedPolicies))
|
||||
for _, p := range out.AttachedPolicies {
|
||||
got = append(got, *p.PolicyArn)
|
||||
}
|
||||
assert.ElementsMatch(t, arns, got, "all concurrently-attached policies must persist")
|
||||
}
|
||||
|
||||
func TestListAttachedRolePolicies_Empty(t *testing.T) {
|
||||
svc := setupTestIAMService(t)
|
||||
role := createTestRole(t, svc, "empty-attach")
|
||||
|
||||
@@ -88,6 +88,11 @@ type ExtraENIInput struct {
|
||||
ENIMac string `json:"eni_mac"`
|
||||
ENIIP string `json:"eni_ip"`
|
||||
SubnetID string `json:"subnet_id"`
|
||||
// AccountID is the owner account of this ENI when it differs from the VM's
|
||||
// primary-ENI account (cross-account attach, e.g. an EKS cluster NLB whose
|
||||
// LB VM lives in the system CP VPC but fronts a customer-VPC ENI). Empty
|
||||
// falls back to the primary AccountID — back-compat for same-account extras.
|
||||
AccountID string `json:"account_id,omitempty"`
|
||||
}
|
||||
|
||||
// NICConfig describes a single network interface for a BootDirect microVM.
|
||||
|
||||
Reference in New Issue
Block a user