2 Commits

Author	SHA1	Message	Date
Josh Siviour	07722e563f	docs: ECS service guide + register ecs-quickstart workbook (#434) ... * docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts (cluster, task definition, task, service, container instances via the ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart, task IAM roles via the credential endpoint, awsvpc networking, and the honest v1 limitations (no service discovery, awslogs discarded, no rolling deployments, no capacity providers, no distinct execution role). Convert the ecs-quickstart workbook README to the doc-page format (frontmatter + overview/prerequisites/instructions/troubleshooting) so it renders as a Terraform Workbooks entry alongside the EKS workbooks. * docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites flagging the required node-image import (eks-node / spinifex-ecs-node), mirroring the install page's WAN-bridge banner, each with a describe-images verification command. Add a basic Spinifex topology diagram (SVG) to each Overview, in the tf-nginx-alb visual style: the EKS managed control plane + node-group workers, and the ECS ALB + container instances with the gateway/agent and task-credential path.	2026-06-29 10:44:19 +10:00
Josh Siviour	5025a37006	feat: Containers — internal-ECR pulls, DNS-free registry, EKS hardening + docs (#409) ... * docs: add Containers section with EKS and ECR service pages New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks. * docs(eks): rework Terraform workbooks into a capability ladder Rework the three EKS workbooks into a coherent ladder and publish them to the Containers section. All tiers are single- or three-node capable via node_desired_size and serve a shared Spinifex-themed demo app from ECR, replacing nginxdemos/hello. - demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/ region + hit counter; optional /data persistence), distroless image. - eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort. - eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer- controller addon + ACM, public/private VPC with NAT, managed-ingress=false. - eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access entry RBAC, HTTPS Ingress carried over. Pending live e2e verification (image build/push, apply, ALB + PVC bind). * fix(ecr): implement ListTagsForResource (empty tags) for client parity The AWS provider calls ecr:ListTagsForResource on every repository read, which returned 501 NotImplementedException and broke 'tofu apply' of any workbook that creates an aws_ecr_repository. Implement it to return an empty tag set (resource tagging stays a deferred feature) so reads succeed. Drop the tags block from the three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed TagResource on create. * feat(eks): pull from internal ECR via kubelet credential provider Worker nodes could not pull custom images from the internal Spinifex ECR registry: the eks-node had no DNS for the registry host, no containerd CA trust, no port rewrite to :9999, and no pull credentials. Only public images worked. Add cmd/ecr-credential-provider, a kubelet exec credential provider that fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic auth for the registry host. Thread the gateway URL, account, region, and internal suffix into the agent userdata, which now writes the gateway CA, a registries.yaml (CA trust + :9999 endpoint), the kubelet credential-provider config and config.yaml.d drop-in, and an /etc/hosts entry mapping the registry host to the gateway IP. Ship the provider binary in the eks-node AMI. Also remove a stray paragraph from the admin-cli README. * fix: dial EKS ECR registry mirror by gateway IP cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on every boot, wiping the registry-host mapping appended via bootcmd, so worker nodes could not resolve the internal ECR host and pulls failed with no such host. Point the k3s registries.yaml mirror endpoint and tls config at the gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image ref port-less and dropping the DNS leg until northstar lands. Fall back to the hostname endpoint when the gateway is not an IP. * fix: attach node-role instance profile to EKS workers for ECR EKS workers launched with no IAM instance profile, so IMDS served no role and the ECR credential provider could not mint a registry token, leaving internal ECR pulls failing with 401. Real EKS implicitly creates an instance profile for the node role; do the same here. Wire a KV-backed IAM service into the EKS deps and, at worker launch, find-or-create an instance profile named for the node role, attach the role, and pass it to RunInstances so IMDS exposes the role to the credential provider. * feat: use mulga logo in EKS demo app * feat: add ACM certificate tag support (List/Add/RemoveTags) Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate. * test: cover ACM tag gateway and NATS wrapper paths * feat: support EKS nodegroup tags (Tag/Untag/ListTags) Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException. * fix: provision EKS ALB ingress via node credentials The aws-load-balancer-controller addon could not provision an ALB. Three fixes: - TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE, falling back to the Go system trust pool and rejecting the gateway cert as 'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool. - Credentials: an empty AWS_ROLE_ARN beside the web-identity token file half-arms the SDK web-identity provider, which then signs anonymously (MissingAuthenticationToken). Inject the IRSA env only when a service-account role ARN is staged; otherwise omit it so the default chain falls back to the node instance-profile credentials served over IMDS. - Authorization: the node role lacked the ec2/elbv2 actions the controller needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController managed policy to the node role (CreatePolicy + AttachRolePolicy). * fix: add force_delete to EKS workbook demo ECR repos tofu destroy of an EKS workbook left the demo ECR repository behind: it still holds the pushed demo image, so DeleteRepository (force=false) returns RepositoryNotEmptyException. Add force_delete = true to each workbook's aws_ecr_repository.demo so teardown removes the repo with its images. * fix: force-delete EKS control-plane ENIs on teardown TerminateK3sServerVM deleted the CP ENI with a plain force=false DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the record's Status/InstanceId/AttachmentId are set, so when the K3s VM is gone but the ENI record's attachment fields were never cleared the delete returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper re-drove the same non-idempotent call every 90s, wedging the cluster in DELETING. Teardown owns its CP ENIs and has already terminated the VM, so the attachment is authoritatively dead. Replace the InUse-retry budget loop with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the stale attachment fields, then DeleteNetworkInterface passes the in-use guard. Both calls tolerate an already-gone ENI (NotFound) so a race with the async instance-terminate cascade resolves to idempotent success. purgeClusterInfra's private-endpoint ENI path reuses the same helper. * fix: detach+delete worker ENIs on EKS nodegroup teardown A terminated nodegroup worker could leave its primary ENI in spinifex-vpc-enis as in-use/attached, and a dangling ENI throws DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two gaps: TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped any worker owned by another node, stranding both the VM and its ENI on multi-node clusters. Route the terminate per-instance via ec2.cmd.<id> (mirroring the gateway TerminateInstances path) so the owning daemon runs the full teardown — detach + force-delete the primary ENI and clear its record. No responder / NotFound payload is treated as already-gone so a retried DeleteNodegroup stays idempotent. deleteInstanceENI (stopped-instance terminate) issued a force=false DeleteNetworkInterface with no prior detach: a stale in-use attachment made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind after the instance was already written to the terminated bucket. Detach first, then delete, tolerating NotFound. * fix: heal volume State drift on detach/terminate teardown A detach/terminate could leave a volume's State empty instead of 'available'. DeleteVolume gated on State=='available' exactly, so it returned VolumeInUse for an unattached volume nothing was using, leaving it undeletable and blocking stack teardown. DescribeVolumes cosmetically rendered an empty State as 'available', masking the drift. - DeleteVolume now treats an empty State with no attachment as deletable (an attached volume still returns VolumeInUse). - DescribeVolumes derives an empty State from the attachment (attached -> in-use, unattached -> available) instead of blindly rendering available, so an empty-but-attached volume is no longer hidden. - UpdateVolumeState normalizes an unattached volume with an empty state to available, so a detach writeback that omits the state cannot strand the volume in drift. * fix: break all SG cross-references on EKS teardown DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked only ingress (IpPermissions), never egress (IpPermissionsEgress), so a cross-reference via an egress rule kept the SG pinned on DependencyViolation. It also resolved only the two named cluster SGs, so any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG, the private-endpoint SG, user/TF SGs — was never revoked and pinned the referenced SG on delete, leaving the VPC stuck. Enumerate every non-default SG in the VPC and revoke BOTH ingress and egress on each before deleting, breaking any cycle order-independently, then delete only the cluster-owned SGs (referrer SGs are deleted by their own teardown). The default SG is left untouched. Since DeleteClusterSGs runs against both the customer VPC and the managed control-plane VPC, the broadened sweep covers both teardown paths. * feat: support TargetGroup ProtocolVersion in elbv2 CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target groups to HTTP1) and DescribeTargetGroups/Create output return it. Without this the aws-load-balancer-controller sees a perpetual immutable-field mismatch (desired HTTP1 vs empty) and recreates the target group every reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it omitted to match AWS. * fix: omit IRSA env from ebs-csi addon when no role staged The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}} plus the web-identity token volume. With no service-account role staged the placeholder rendered empty, half-arming the SDK web-identity provider, which panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}} markers like the LBC bundle so the IRSA env and token volume are omitted when no role is staged, falling back to node instance-profile credentials over IMDS. The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes. Requires an eks-node AMI rebake to take effect. * docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup with host routing (app./argocd. on a wildcard cert) instead of one ALB per ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers need (Spinifex wires controller creds at the node level, not IRSA) and an ignore_changes block for the cluster's access_config and vpc_config security_group_ids, which Spinifex DescribeCluster does not echo back, so the provider stops trying to replace the cluster on every plan. * fix: stop addon-sync substituting IRSA markers named in comments The ebs-csi controller manifest's header comment spelled out the literal {{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync matched each marker by substring, so the comment lines were replaced with the env/mount blocks, injecting a bare YAML sequence between two documents. K3s' auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal array into unstructured), so the ebs-csi Addon CR never applied and the EKS addon hung in CREATING. Reword the comment to not contain the literal tokens, and anchor the marker substitution to lines that are exactly the marker (whitespace aside) so a future comment mentioning a marker can't retrigger it. * fix: disable K3s local-storage on EKS clusters in parity mode In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server disabled traefik and servicelb but left local-storage enabled. Its local-path provisioner ships a StorageClass marked default, which races the EBS CSI ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind nondeterministically. EKS has no local-path, so disable local-storage too and let ebs-gp3 be the sole default. The built-in-ingress path is unchanged. * fix: re-encrypt HTTPS target groups in haproxy ALB backends An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's own check then spoke HTTP to a TLS port, failed, and marked the server DOWN, returning 503 — even though the lb-agent's separate HTTPS health probe reported the target healthy, so describe-target-health masked the fault. Emit check-ssl ssl verify none on the server line for HTTPS target groups so both the check and proxied traffic re-encrypt to the backend. * fix: reconnect wedged QMP client on next command A timed-out QMP decode under host contention left the cached client's shared json.Decoder mid-message, wedging every later command and breaking EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead on encode/decode error and redial the retained socket path (re-running the qmp_capabilities handshake in place) on the next send. * fix: round-trip security-group rule descriptions SG ingress/egress rules dropped the per-rule Description on store and describe. The AWS Load Balancer Controller tags the worker-node SG rules it manages with a UserIdGroupPair description and identifies them by that value to revoke them on teardown. Without the round-trip the controller cannot match its rules, never revokes them, and the backend security group stays referenced — DeleteSecurityGroup returns DependencyViolation forever, hanging the ingress finalizer and tofu destroy with orphaned security groups. Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not identity) and render it back through describe. * fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown In the managed control-plane VPC topology, launchNodegroupInfra creates the cluster control-plane and nodegroup security groups in the customer VPC for worker<->control-plane networking, but purgeClusterInfra only reclaimed the security groups in the managed CP VPC. The customer-VPC SGs orphaned, cross-referenced each other, and pinned the customer VPC with DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the managed-CP-VPC branch too, matching the legacy customer-VPC branch. * fix: reclaim ALB EIP when system VM already gone on terminate TerminateSystemInstance only released an internet-facing ALB's Elastic IP on the local path, where the VM is still present in the manager. When the VM was already gone (health-reaped, crashed, or raced by ENI teardown), it took the remote/no-owner path and returned idempotent success without reconciling the EIP, leaking the wan-pool address still associated to a deleted ENI. Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic scan that disassociates then releases any EIP recorded against an instance, and call it as a backstop in TerminateSystemInstance for both branches. * fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook The demo Application carried no resources finalizer, so tofu destroy removed the Application CR without pruning the synced PVC. The PVC survived, the cluster was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the resources-finalizer.argocd.argoproj.io finalizer so deleting the Application cascades to the PVC and ebs-csi reaps the volume before cluster teardown. * feat: raw-IP access + documented Argo CD creds in eks-gitops workbook The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password command as a tofu output and document the credential + IP URLs in the README. * fix: reclaim ALB EIP on the owning-node terminate path The EIP backstop lived only in TerminateSystemInstance, but the ALB delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's cached allocation which is often empty. Extract reclaimSystemInstanceEIP (authoritative KV reconcile via ReleaseAddressByInstanceID) and call it from terminateSystemInstanceLocal as well as the no-owner backstop, so an internet-facing ALB's EIP is released on every terminate route. * chore(eks): drop coredns from the managed addon catalog Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as a managed addon — there is no bundled coredns manifest, so CreateAddon coredns would stage but fail VM-side render. Remove the phantom catalog entry so DescribeAddonVersions and CreateAddon no longer advertise it; repoint the addon tests that used coredns as their sample to argocd. * feat(ui): group EKS and ECR under a Containers nav section Fold the separate EKS and ECR sidebar groups into one Containers group with EKS (clusters) and ECR (repositories) entries. * fix(ui): satisfy strict null checks in ECR component tests Index mock.calls via optional chaining and narrow the getAllByRole result before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix. * refactor(eks): hide internal fixture addons from the public catalog Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays creatable (lookupAddon finds it, so the delivery-transport e2e still drives it) but no longer surfaces in the unfiltered DescribeAddonVersions listing. * refactor(eks): remove K3s built-in ingress; LBC-only Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik, servicelb, and local-storage are now always disabled on EKS nodes (AWS parity — Service type=LoadBalancer and Ingress are the Load Balancer Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in the state reporter. * fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation Parallelize the independent subnet-create and route-table-association loops in the create-VPC mutation with Promise.all (each task records its resource in the rollback list before any further call), and switch the CIDR-validation regex to named capture groups. Clears the no-await-in-loop and prefer-named-capture-group lint errors. * fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal The ECR registry is served on the gateway port (:9999), but proxyEndpoint and repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and failed. Derive the port from the awsgw listen host and append it to the registry host (omitted for 443). Also rename the default internal DNS suffix. * docs: drop verbose ECR :9999 host note * docs: fix literal-asterisk emphasis (single-* not rendered) * docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust * feat: default ECR registry URIs to the gateway host (no-DNS interim) The registry authenticates by token, not hostname, so the per-account <acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port (already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need no DNS; the parity host still works server-side where DNS exists. * fix: serve ECR registry on a DNS-free gateway host end to end Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable host:9999 carried in the server cert SANs rather than the unresolvable per-account parity name. Docker push then dials it directly with no DNS. Mirror the same host on EKS workers: the registries.yaml mirror keys and the credential provider matchImages now cover the port-less parity name, the port-ful parity name, and the gateway IP:9999, all redirecting to the dialed gateway endpoint. A port-ful pull from the API-returned repository_url matches the mirror instead of bypassing it into a failed DNS lookup. Net: docker push host, pod image host, and the API's repository_url are the same value, with no DNS or /etc/hosts. * docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS) * docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS) * docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks	2026-06-23 15:29:14 +10:00