Files
spinifex/docs/terraform-workbooks/eks-https-ingress
Josh Siviour 129bfc0fb5 feat(eks): join nodegroup workers via cluster NLB endpoint + private endpoint access (#362)
* feat: join nodegroup workers via cluster NLB endpoint

Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.

Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.

mulga-siv-298

* feat: gate nodegroup ACTIVE on workers registering Ready

launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.

Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.

mulga-siv-299

* feat(eks): private endpoint access via cross-account Set A ENI

Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.

Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.

Refs: mulga-siv-301

* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress

Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.

Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).

Refs: mulga-siv-321

* fix(vpc): reap the main route table on DeleteVpc

CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.

Refs: mulga-siv-322

* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS

Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').

Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.

* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)

Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
  with an ACM-imported self-signed certificate

Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.
2026-06-16 00:59:36 +10:00
..

title, description, category, tags, resources
title description category tags resources
3-Node EKS behind an HTTPS ALB (ACM TLS) Provision an HA-shaped EKS cluster with public + private API endpoints and three workers, deploy a demo app, and publish it through an internet-facing ALB that terminates TLS with an ACM-imported certificate, using Terraform on Spinifex. Terraform Workbooks
terraform
eks
kubernetes
acm
tls
alb
elbv2
vpc
workbook
title url
Terraform AWS Provider https://registry.terraform.io/providers/hashicorp/aws/latest
title url
Terraform Kubernetes Provider https://registry.terraform.io/providers/hashicorp/kubernetes/latest
title url
Spinifex Repository https://github.com/mulgadc/spinifex
title url
OpenTofu https://opentofu.org/

Terraform: 3-Node EKS behind an HTTPS ALB

The largest EKS workbook, and the clearest demo. An HA-shaped cluster with public + private API endpoints and three workers, running a demo app published over HTTPS through an internet-facing ALB that terminates TLS with an ACM-imported certificate.

Table of Contents


Overview

This workbook is sized for a 3-node Spinifex deployment and is built to show off. It brings together the pieces from the smaller EKS examples, deploys a demo app across three workers, and fronts it with a TLS-terminating load balancer. When apply finishes, fetch the ALB's public IP and open https://<ip> — the nginxdemos/hello page reports which pod answered, and refreshing moves between the three replicas spread across the cluster. That single page demonstrates a multi-node Kubernetes cluster load-balancing behind HTTPS.

  • Public + private API endpoints. endpoint_public_access serves kubectl and the Kubernetes provider from your host (CIDR-restricted); endpoint_private_access exposes an in-VPC private endpoint the workers use to join.
  • Private workers + NAT. The three workers sit in private subnets. They join the cluster over the in-VPC private endpoint, and a NAT gateway gives them outbound access to pull the demo container image.
  • CoreDNS installed as a managed addon.
  • TLS via ACM. A self-signed certificate is generated with the tls provider and imported into ACM (Spinifex ACM supports ImportCertificate, not RequestCertificate). The ALB's HTTPS listener references the ACM ARN and forwards to the workers' NodePort.
  • The app, deployed by Terraform. The Kubernetes provider deploys the Deployment + NodePort Service, and one security-group rule lets the ALB reach the NodePort.

3-node EKS — public+private API endpoints, ALB in public subnets terminating TLS, three workers in private subnets behind a NAT gateway

What you'll learn:

  • Running an EKS cluster with both public and private API endpoints
  • Placing workers in private subnets that join over the private endpoint and egress via NAT
  • Importing a certificate into ACM and terminating TLS on an ALB HTTPS listener
  • Wiring an ALB target group to managed node-group workers via a NodePort, including the SG rule that allows it
  • Deploying a load-balanced workload with the Kubernetes provider

What gets created

Resource Name Purpose
VPC eks-https-vpc Isolated network (10.32.0.0/16)
Public Subnets eks-https-public-a/-b Host the ALB and NAT gateway
Private Subnets eks-https-private-a/-b Host the workers
Internet Gateway eks-https-igw Internet for the public subnets
NAT Gateway (+ EIP) eks-https-nat Outbound for the private workers (image pulls)
Route Tables eks-https-public-rt, eks-https-private-rt Public via IGW; private via NAT
IAM Roles eks-https-cluster-role, eks-https-node-role Control-plane and worker roles
EKS Cluster eks-https Public (CIDR-restricted) + private endpoints
Node Group workers Three t3.large workers in the private subnets
Addon coredns Cluster DNS
ACM Cert eks-https-ingress Self-signed cert imported into ACM
ALB + HTTPS Listener eks-https-alb Internet-facing, terminates TLS on :443
Target Group eks-https-tg Instance targets on the NodePort
SG Ingress Rule eks-https-alb-nodeport Lets the ALB SG reach the worker NodePort
K8s Deployment + Service hello nginxdemos/hello, 3 replicas, NodePort

Spinifex specifics

  • ACM supports ImportCertificate, DescribeCertificate, ListCertificates, DeleteCertificate — there is no RequestCertificate/DNS-validation flow, hence the self-signed import.
  • The ALB HTTPS listener resolves the cert from ACM. Valid SSL policies are ELBSecurityPolicy-2016-08 (default) and ELBSecurityPolicy-TLS13-1-2-2021-06.
  • The worker SG is auto-managed (vpc_config.security_group_ids is ignored) and admits only intra-cluster traffic. The ALB→NodePort rule is added by looking the SG up by its deterministic name and admitting the ALB SG.
  • Worker discovery uses the spinifex:eks-cluster tag Spinifex stamps on node-group instances; the target-group attachment count is the known node_desired_size.
  • Leave addon_version unset (see the addons workbook for why pinning it can't satisfy both the provider and the catalog).

Prerequisites:

  • A 3-node Spinifex deployment, installed and running
  • The Spinifex eks-node image available on the cluster
  • OpenTofu or Terraform, plus kubectl and the AWS CLI

Instructions

Step 1. Get the Template

git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/eks-https-ingress

Or create a main.tf file and paste the full configuration below.

Step 2. Deploy

export AWS_PROFILE=spinifex
tofu init
tofu apply

Expect several minutes: the control plane bootstraps, three workers launch and join over the private endpoint, the NAT gateway comes up, CoreDNS installs, the ALB is provisioned, and the Kubernetes provider rolls out the demo app.

Keep AWS_PROFILE=spinifex exported for the whole apply — the Kubernetes provider authenticates with aws eks get-token against the Spinifex STS endpoint.

Tighten access in production. api_public_access_cidr and alb_ingress_cidr both default to 0.0.0.0/0. Set them to your own CIDR:

export TF_VAR_api_public_access_cidr="203.0.113.10/32"
export TF_VAR_alb_ingress_cidr="203.0.113.10/32"

Step 3. Open the Demo over HTTPS

The ALB DNS name (*.elb.spinifex.local) won't resolve from your host, so fetch its public IP:

ALB_IP=$(aws elbv2 describe-load-balancers --names eks-https-alb \
  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
  --output text)

The certificate is self-signed, so a browser will warn (proceed anyway) and curl needs -k:

curl -k https://$ALB_IP
curl -k https://$ALB_IP   # refresh: the "Server name" changes between pods

Open https://$ALB_IP in a browser and refresh to watch requests land on different pods across the three nodes.

Step 4. Inspect (optional)

aws eks update-kubeconfig --name eks-https --region ap-southeast-2
kubectl get nodes
kubectl get pods -o wide          # three replicas, one per node

TG_ARN=$(aws elbv2 describe-target-groups --names eks-https-tg \
  --query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"

Cleanup

tofu destroy

Troubleshooting

Targets Unhealthy / ALB Returns 5xx

The ALB health-checks / on the NodePort. Targets turn healthy once the demo pods are running and the SG rule admits the ALB. Check, in order:

kubectl get pods -o wide                                  # pods Running?
aws ec2 describe-security-groups \
  --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
  --query 'SecurityGroups[0].IpPermissions'               # NodePort rule present?
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"

If pods are ImagePullBackOff, the workers can't reach Docker Hub — confirm the NAT gateway is available:

aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'

Workers Never Join

The workers join over the private endpoint. Confirm it's enabled and the node group is progressing:

aws eks describe-cluster --name eks-https --query 'cluster.resourcesVpcConfig.endpointPrivateAccess'
aws eks describe-nodegroup --cluster-name eks-https --nodegroup-name workers --query 'nodegroup.status'

Target-Group Attachment Errors at Apply

The attachment indexes the discovered worker IDs by node_desired_size. If apply errors with an index out of range, the workers weren't all tagged/running when the aws_instances lookup ran. Re-run tofu apply — the node group will be ACTIVE by then:

aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-https" \
  --query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text

kubernetes provider: connection refused / Unauthorized

The provider runs aws eks get-token against the Spinifex STS endpoint. If the cluster was still CREATING when the provider first connected, re-run tofu apply. Otherwise confirm:

aws sts get-caller-identity
aws eks describe-cluster --name eks-https --query 'cluster.status'

ACM Import / HTTPS Listener Errors

aws acm list-certificates
aws acm describe-certificate --certificate-arn "$(tofu output -raw certificate_arn)" \
  --query 'Certificate.Status'

A CertificateNotFound on the listener means the ACM ARN didn't resolve — check the cert exists and belongs to the same account/profile.

Provider Connection Refused

sudo systemctl status spinifex.target
curl -k https://localhost:9999/