test: cover nodegroup delete failure paths, profile ownership refusals and worker absence checks

docs/containers/eks/README.md

@@ -59,6 +59,9 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
 
 - **Access entries are the only auth path.** `authentication_mode` accepts `API` (the default when unset) and `API_AND_CONFIG_MAP`; bare `CONFIG_MAP` is rejected. Spinifex has no `aws-auth` ConfigMap path, so `API_AND_CONFIG_MAP` behaves as `API`: grant access with access entries.
 - **The cluster security group is auto-managed.** As on EKS, Spinifex creates one `eks-cluster-sg-<name>-<id>` in your VPC, tagged `kubernetes.io/cluster/<name>=owned` and `aws:eks:cluster-name`, allowing all traffic from itself. It is attached to the workers and the private-endpoint network interface, and `DescribeCluster` reports it as `clusterSecurityGroupId`. A launch template that names security groups replaces it on the workers, as on AWS. Clusters created before this change keep their `-control-plane-sg` and `-nodegroup-sg` until you recreate them. Deleting a cluster deletes the cluster security group and removes only the rules on your other security groups that reference it. `vpc_config.security_group_ids` is ignored.
+- **Node group deletes are asynchronous.** `DeleteNodegroup` returns at once with the group `DELETING`, and `DescribeNodegroup` answers `ResourceNotFoundException` once the workers and the group's instance profile are gone. Deleting a cluster or creating a group of the same name while a group is `DELETING` returns `ResourceInUseException`, as does a second delete of a `DELETING` group.
+- **If a node group delete cannot finish,** the group moves to `DELETE_FAILED` after 30 minutes, with a `NODE_TERMINATION_FAILURE` health issue naming the workers that are still running. Run `DeleteNodegroup` again to resume it.
+- **Each node group owns an EKS-managed instance profile.** Spinifex creates it, tags it `spinifex:eks-cluster`, `spinifex:eks-nodegroup` and `spinifex:eks-profile-id`, and puts the node role in it. It is removed with the group, so you can delete the node role afterwards with no extra steps. Spinifex never deletes an instance profile it did not create, or one whose tags, name or role no longer match what it recorded; it leaves the profile and logs a warning with its name.
 - **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
 - **Workers launch with IMDSv2 required and an IMDS hop limit of 2.**
   This matches AWS managed node groups and lets a pod without IRSA reach the node role through the SDK default credential chain.
@@ -377,6 +380,15 @@ Control plane readiness and kubectl authentication above lists which clusters re
 
 **`create-cluster` rejected with an authentication-mode error.** `CONFIG_MAP` is the one mode Spinifex rejects, because it has no `aws-auth` ConfigMap path. Set `authentication_mode` to `API` or `API_AND_CONFIG_MAP`, or leave it unset, and grant access with access entries.
 
+**`delete-role` fails with `DeleteConflict` after deleting a node group.** Node groups created before per-group instance profiles left an instance profile named after the role, which Spinifex no longer removes because it cannot prove it created it. Remove it yourself:
+
+```bash
+aws iam remove-role-from-instance-profile --instance-profile-name <role-name> --role-name <role-name>
+aws iam delete-instance-profile --instance-profile-name <role-name>
+```
+
+**A node group is `DELETE_FAILED`.** `aws eks describe-nodegroup` lists the workers that did not stop under `health.issues`. Once they are gone, or after you fix the cause, run `aws eks delete-nodegroup` again.
+
 **Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, use the AWS form from `DescribeAddonVersions` (see Add-on versions above), or leave `addon_version` unset so Spinifex selects the catalog default.
 
 ## Reference Architectures

docs/coverage/coverage.json

@@ -2413,6 +2413,7 @@
           "ListAssociatedAccessPolicies",
           "ListClusters",
           "ListInternalAddons",
+          "ListInternalWorkers",
           "ListNodegroups",
           "ListTagsForResource",
           "PublishInternal",
@@ -2502,6 +2503,7 @@
         "extra": [
           "GetRecoveryDirective",
           "ListInternalAddons",
+          "ListInternalWorkers",
           "PublishInternal",
           "WebhookTokenReview"
         ]

docs/coverage/internal-report.md

@@ -903,7 +903,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 
 ## eks
 
-`2017-11-01` — 27 of 70 modelled operations implemented (38.6%), 0 stubbed, 0 not applicable, 4 registered outside the pinned model.
+`2017-11-01` — 27 of 70 modelled operations implemented (38.6%), 0 stubbed, 0 not applicable, 5 registered outside the pinned model.
 
 ### Not implemented
 
@@ -955,6 +955,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
 
 - `GetRecoveryDirective`
 - `ListInternalAddons`
+- `ListInternalWorkers`
 - `PublishInternal`
 - `WebhookTokenReview`
 

docs/package-boundary/eks-cluster-lifecycle.md

@@ -34,7 +34,7 @@ Node group workers and CSI-provisioned EBS volumes are never removed by a cluste
 
 ## Gaps
 
-1. Four `eks.*` handlers still hold their request for work that grows with the resource, and the subject table names them as not conformant: `CreateCluster` (reclaiming a `FAILED` cluster of the same name tears its infra down inline), `DeleteNodegroup` (one awaited terminate per worker), `UpdateNodegroupConfig` (one awaited launch or terminate per worker added or removed) and `RestoreSnapshot` (launches a control-plane VM and fences the old one inline).
+1. Three `eks.*` handlers still hold their request for work that grows with the resource, and the subject table names them as not conformant: `CreateCluster` (reclaiming a `FAILED` cluster of the same name tears its infra down inline), `UpdateNodegroupConfig` (one awaited launch or terminate per worker added or removed) and `RestoreSnapshot` (launches a control-plane VM and fences the old one inline).
 2. AWS's answer to a repeat delete on a `DELETING` cluster has not been observed; `DELETING` from the record is the chosen behaviour.
 3. Control-plane, load balancer and VPC resources in the system account are keyed by the bare cluster name, so a teardown by name can reach another tenant's same-named cluster.
 4. The teardown deletes load balancers the in-cluster controller created in the customer account, which AWS leaves behind.

spinifex/domains/eks/cluster/delete_cluster_review_test.go

@@ -205,6 +205,8 @@ func TestS2_EKSLastNodegroupDeletedResumesTheClusterDelete(t *testing.T) {
 		ClusterName: aws.String("alpha"), NodegroupName: aws.String("ng1"),
 	}, testAccountID)
 	require.NoError(t, err)
+	_, err = f.svc.NewNodegroupDeletingReaper().Sweep(context.Background())
+	require.NoError(t, err)
 	f.svc.WaitTeardowns()
 
 	_, err = GetClusterMeta(t.Context(), f.kv, "alpha")

spinifex/domains/eks/cluster/delete_cluster_test.go

@@ -105,6 +105,9 @@ func newEKSServiceFixture(t *testing.T) *eksServiceFixture {
 	require.NoError(t, err)
 	// Tight Ready-gating knobs so nodegroup launches resolve in tests without the
 	// production 10m timeout; tests drive meta.NodeCount to simulate the reconciler.
+	svc.noDeleteKick = true
+	svc.workersEndWait = 20 * time.Millisecond
+	svc.workersEndPoll = time.Millisecond
 	svc.nodegroupReadyTimeout = 50 * time.Millisecond
 	svc.nodegroupReadyPoll = 2 * time.Millisecond
 	// Tight worker-launch-retry knobs so a launch-failure retry test resolves

spinifex/domains/eks/cluster/eks_nodegroup_deleting_reaper.go

@@ -0,0 +1,284 @@
+package cluster
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"log/slog"
+	"strings"
+	"time"
+
+	"github.com/aws/aws-sdk-go/service/eks"
+	"github.com/mulgadc/spinifex/spinifex/runtime/compute/vm"
+	"github.com/nats-io/nats.go/jetstream"
+)
+
+// A DELETING nodegroup becomes DELETE_FAILED once it has been deleting this
+// long and this many passes have failed. Vars for tests.
+var (
+	nodegroupDeleteFailAfter    = 30 * time.Minute
+	nodegroupDeleteFailAttempts = 5
+)
+
+// A pass waits this long, polling, for ended workers to be proven gone.
+const (
+	defaultWorkersEndWait = 90 * time.Second
+	defaultWorkersEndPoll = 3 * time.Second
+)
+
+// errWorkersEnding means workers were told to end but are not yet proven gone.
+var errWorkersEnding = errors.New("workers still ending")
+
+// EKSNodegroupDeletingReaper finishes recorded nodegroup deletes: it ends the
+// workers, removes the nodegroup's own instance profile and only then the
+// record, so a nodegroup is never gone while something it made still is.
+type EKSNodegroupDeletingReaper struct {
+	svc *EKSServiceImpl
+}
+
+var _ vm.Reaper = (*EKSNodegroupDeletingReaper)(nil)
+
+// NewNodegroupDeletingReaper builds the DELETING nodegroup converger.
+func (s *EKSServiceImpl) NewNodegroupDeletingReaper() *EKSNodegroupDeletingReaper {
+	return &EKSNodegroupDeletingReaper{svc: s}
+}
+
+func (r *EKSNodegroupDeletingReaper) Class() string         { return "eks-nodegroup-deleting" }
+func (r *EKSNodegroupDeletingReaper) Scope() vm.ReaperScope { return vm.ScopeNodeLocal }
+
+// Sweep hands every due DELETING nodegroup to a background converger and
+// returns how many it started. The lease decides which node runs each one.
+func (r *EKSNodegroupDeletingReaper) Sweep(ctx context.Context) (int, error) {
+	if !r.svc.depsReadyForOrchestration() {
+		return 0, nil
+	}
+	js, err := jetstream.New(r.svc.deps.NATSConn)
+	if err != nil {
+		return 0, fmt.Errorf("jetstream: %w", err)
+	}
+	names, err := accountBucketNames(ctx, r.svc.deps.NATSConn)
+	if err != nil {
+		return 0, fmt.Errorf("enumerate account buckets: %w", err)
+	}
+
+	started := 0
+	for _, name := range names {
+		select {
+		case <-ctx.Done():
+			return started, ctx.Err()
+		default:
+		}
+		accountID := strings.TrimPrefix(name, KVBucketEKSAccountPrefix)
+		acctKV, err := js.KeyValue(ctx, name)
+		if err != nil {
+			continue
+		}
+		clusters, err := listClusterNames(ctx, acctKV)
+		if err != nil {
+			continue
+		}
+		for _, cluster := range clusters {
+			recs, err := ListNodegroupRecords(ctx, acctKV, cluster)
+			if err != nil {
+				continue
+			}
+			for _, rec := range recs {
+				if r.due(rec) && r.svc.startNodegroupDelete(accountID, acctKV, cluster, rec.Name) {
+					started++
+				}
+			}
+		}
+	}
+	return started, nil
+}
+
+// due reports whether rec is DELETING and past its retry backoff.
+func (r *EKSNodegroupDeletingReaper) due(rec *NodegroupRecord) bool {
+	if rec.Status != eks.NodegroupStatusDeleting {
+		return false
+	}
+	if rec.DeleteReapAttempts == 0 || rec.LastDeleteReapAttempt.IsZero() {
+		return true
+	}
+	return time.Since(rec.LastDeleteReapAttempt) >= deleteReapBackoff(deletingReapMinAge, rec.DeleteReapAttempts)
+}
+
+// startNodegroupDelete runs one converger pass in the background and reports
+// whether it started; a pass already running here or on another node refuses.
+func (s *EKSServiceImpl) startNodegroupDelete(accountID string, kv jetstream.KeyValue, cluster, ng string) bool {
+	release, _, ok := s.holdTeardown(accountID, cluster+"/nodegroup/"+ng, false)
+	if !ok {
+		return false
+	}
+	s.teardownWG.Go(func() {
+		defer release()
+		ctx, cancel := context.WithTimeout(s.bgCtx, teardownPassTimeout)
+		defer cancel()
+		if err := s.convergeNodegroupDeletion(ctx, accountID, kv, cluster, ng); err != nil {
+			slog.Warn("eks: nodegroup delete incomplete, the record stays DELETING for retry",
+				"cluster", cluster, "nodegroup", ng, "account", accountID, "err", err)
+		}
+	})
+	return true
+}
+
+// WaitNodegroupDeletes blocks until every converger this process started has
+// returned. Intended for tests.
+func (s *EKSServiceImpl) WaitNodegroupDeletes() { s.teardownWG.Wait() }
+
+// mutateNodegroup applies fn to the stored record under compare-and-swap. fn
+// returns false to leave the record unchanged.
+func (s *EKSServiceImpl) mutateNodegroup(ctx context.Context, kv jetstream.KeyValue, cluster, ng string, fn func(*NodegroupRecord) bool) (*NodegroupRecord, error) {
+	for range ngCASMaxRetries {
+		rec, rev, err := getNodegroupEntry(ctx, kv, cluster, ng)
+		if err != nil {
+			return nil, err
+		}
+		if !fn(rec) {
+			return rec, nil
+		}
+		ok, err := s.casPutNodegroup(ctx, kv, rec, rev)
+		if err != nil {
+			return nil, err
+		}
+		if ok {
+			return rec, nil
+		}
+	}
+	return nil, fmt.Errorf("nodegroup %s/%s: concurrent writers kept winning", cluster, ng)
+}
+
+// convergeNodegroupDeletion finishes one recorded delete. It acts only on a
+// record that is DELETING when re-read, and keeps the record on any failure.
+func (s *EKSServiceImpl) convergeNodegroupDeletion(ctx context.Context, accountID string, kv jetstream.KeyValue, cluster, ng string) error {
+	rec, err := s.mutateNodegroup(ctx, kv, cluster, ng, func(r *NodegroupRecord) bool {
+		if r.Status != eks.NodegroupStatusDeleting {
+			return false
+		}
+		r.DeleteReapAttempts++
+		r.LastDeleteReapAttempt = time.Now().UTC()
+		return true
+	})
+	if err != nil {
+		if errors.Is(err, ErrNodegroupNotFound) {
+			return nil
+		}
+		return err
+	}
+	if rec.Status != eks.NodegroupStatusDeleting {
+		return nil
+	}
+
+	if err := s.releaseNodegroupResources(ctx, accountID, kv, rec); err != nil {
+		if errors.Is(err, errWorkersEnding) {
+			s.noteWorkersEnding(ctx, kv, cluster, ng, err)
+			return err
+		}
+		s.noteNodegroupDeleteFailure(ctx, kv, rec, err)
+		return err
+	}
+	if err := DeleteNodegroupRecord(ctx, kv, cluster, ng); err != nil {
+		return err
+	}
+	// A cluster delete waits on its node groups; the last one going lets it run.
+	if meta, err := GetClusterMeta(ctx, kv, cluster); err == nil && meta.Status == ClusterStatusDeleting {
+		if rest, err := ListNodegroupRecords(ctx, kv, cluster); err == nil && len(rest) == 0 {
+			s.startTeardown(accountID, cluster)
+		}
+	}
+	return nil
+}
+
+// releaseNodegroupResources ends the workers, then removes the profile. The
+// profile and the recorded worker ids go only once every worker is proven
+// terminated or absent, since an accepted terminate is only a request.
+func (s *EKSServiceImpl) releaseNodegroupResources(ctx context.Context, accountID string, kv jetstream.KeyValue, rec *NodegroupRecord) error {
+	if len(rec.InstanceIDs) > 0 {
+		if err := s.deps.Worker.TerminateWorkerInstances(ctx, rec.InstanceIDs, accountID); err != nil {
+			return fmt.Errorf("terminate workers: %w", err)
+		}
+		if err := s.awaitWorkersGone(ctx, rec.InstanceIDs, accountID); err != nil {
+			return err
+		}
+		if _, err := s.mutateNodegroup(ctx, kv, rec.ClusterName, rec.Name, func(r *NodegroupRecord) bool {
+			r.InstanceIDs = nil
+			return true
+		}); err != nil {
+			return fmt.Errorf("record workers ended: %w", err)
+		}
+	}
+	return s.deleteNodegroupProfile(ctx, accountID, kv, rec)
+}
+
+// awaitWorkersGone polls until no worker is live, up to the end wait,
+// and returns errWorkersEnding if some still are.
+func (s *EKSServiceImpl) awaitWorkersGone(ctx context.Context, ids []string, accountID string) error {
+	wait, poll := s.workersEndWait, s.workersEndPoll
+	if wait <= 0 {
+		wait = defaultWorkersEndWait
+	}
+	if poll <= 0 {
+		poll = defaultWorkersEndPoll
+	}
+	deadline := time.Now().Add(wait)
+	for {
+		live, err := s.deps.Worker.LiveWorkerInstances(ctx, ids, accountID)
+		if err != nil {
+			return fmt.Errorf("confirm workers ended: %w", err)
+		}
+		if len(live) == 0 {
+			return nil
+		}
+		if !time.Now().Before(deadline) {
+			return fmt.Errorf("%w: %s", errWorkersEnding, strings.Join(live, ","))
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(poll):
+		}
+	}
+}
+
+// noteWorkersEnding handles workers that are not yet gone. Early on that is no
+// failed attempt: the next sweep checks again without backoff. Once the group
+// has been deleting past nodegroupDeleteFailAfter it settles DELETE_FAILED
+// naming the instances, keeping the profile and record for a customer retry.
+func (s *EKSServiceImpl) noteWorkersEnding(ctx context.Context, kv jetstream.KeyValue, cluster, ng string, cause error) {
+	_, err := s.mutateNodegroup(ctx, kv, cluster, ng, func(r *NodegroupRecord) bool {
+		if r.Status != eks.NodegroupStatusDeleting {
+			return false
+		}
+		if time.Since(r.DeletingSince) >= nodegroupDeleteFailAfter {
+			r.Status = eks.NodegroupStatusDeleteFailed
+			r.StatusReason = "delete: " + cause.Error()
+			r.ModifiedAt = time.Now().UTC()
+			return true
+		}
+		r.DeleteReapAttempts--
+		r.LastDeleteReapAttempt = time.Time{}
+		return true
+	})
+	if err != nil {
+		slog.WarnContext(ctx, "eks: record workers still ending", "cluster", cluster, "nodegroup", ng, "err", err)
+	}
+}
+
+// noteNodegroupDeleteFailure records why a pass failed and, once the group has
+// been deleting long enough and failed often enough, settles it DELETE_FAILED.
+func (s *EKSServiceImpl) noteNodegroupDeleteFailure(ctx context.Context, kv jetstream.KeyValue, rec *NodegroupRecord, cause error) {
+	_, err := s.mutateNodegroup(ctx, kv, rec.ClusterName, rec.Name, func(r *NodegroupRecord) bool {
+		if r.Status != eks.NodegroupStatusDeleting {
+			return false
+		}
+		r.StatusReason = "delete: " + cause.Error()
+		r.ModifiedAt = time.Now().UTC()
+		if r.DeleteReapAttempts >= nodegroupDeleteFailAttempts && time.Since(r.DeletingSince) >= nodegroupDeleteFailAfter {
+			r.Status = eks.NodegroupStatusDeleteFailed
+		}
+		return true
+	})
+	if err != nil {
+		slog.WarnContext(ctx, "eks: record nodegroup delete failure", "cluster", rec.ClusterName, "nodegroup", rec.Name, "err", err)
+	}
+}

spinifex/domains/eks/cluster/nodegroup.go

@@ -379,6 +379,12 @@ func (s *EKSServiceImpl) createNodegroup(ctx context.Context, acctKV jetstream.K
 		ModifiedAt:     now,
 	}
 
+	// Recorded before the claim, so a name is durable before any worker launches.
+	if rec.NodeRole != "" {
+		rec.InstanceProfileID = uuid.NewV4().String()
+		rec.InstanceProfileName = nodegroupProfilePrefix + rec.InstanceProfileID
+	}
+
 	// Atomically claim the record; concurrent/duplicate CreateNodegroup requests lose here.
 	owned, err := ClaimNodegroupRecord(ctx, acctKV, rec)
 	if err != nil {
@@ -716,7 +722,7 @@ func (s *EKSServiceImpl) launchOneWorker(ctx context.Context, rec *NodegroupReco
 	// Attach the node role's instance profile so IMDS serves the role to the ECR
 	// credential provider; without it worker pulls from the internal ECR get 401.
 	if s.iamEnsurer() != nil && rec.NodeRole != "" {
-		profileARN, perr := s.ensureNodeInstanceProfile(accountID, rec.NodeRole)
+		profileARN, perr := s.nodeProfileForLaunch(accountID, rec)
 		if perr != nil {
 			return "", fmt.Errorf("ensure node instance profile: %w", perr)
 		}
@@ -855,14 +861,10 @@ func gatewayHostIP(gatewayURL string) string {
 	return host
 }
 
-// ensureNodeInstanceProfile guarantees an IAM instance profile named for the
-// nodegroup's node role exists and carries that role, returning its ARN. Workers
-// launch with this profile so IMDS serves the node role to the ECR credential
-// provider, mirroring the implicit instance profile real EKS creates for a node
-// role. Idempotent: concurrent worker launches converge on the same profile.
-func (s *EKSServiceImpl) ensureNodeInstanceProfile(accountID, nodeRoleARN string) (string, error) {
-	// Resolution compares the supplied ARN against the stored one, so an invented
-	// path cannot bind a profile to the role its trailing name happens to match.
+// resolveNodeRoleName returns the name of the stored role nodeRoleARN names.
+// Resolution compares the supplied ARN against the stored one, so an invented
+// path cannot bind a profile to the role its trailing name happens to match.
+func (s *EKSServiceImpl) resolveNodeRoleName(accountID, nodeRoleARN string) (string, error) {
 	_, roleName, err := auth.ResolveRoleARN(nodeRoleARN, func(account, name string) (string, error) {
 		// The profile is created under the caller's account, so a role from
 		// another account would silently bind a name that account does not own.
@@ -881,6 +883,19 @@ func (s *EKSServiceImpl) ensureNodeInstanceProfile(accountID, nodeRoleARN string
 	if err != nil {
 		return "", fmt.Errorf("node role ARN %q: %w", nodeRoleARN, err)
 	}
+	return roleName, nil
+}
+
+// ensureNodeInstanceProfile guarantees an IAM instance profile named for the
+// nodegroup's node role exists and carries that role, returning its ARN. Workers
+// launch with this profile so IMDS serves the node role to the ECR credential
+// provider, mirroring the implicit instance profile real EKS creates for a node
+// role. Idempotent: concurrent worker launches converge on the same profile.
+func (s *EKSServiceImpl) ensureNodeInstanceProfile(accountID, nodeRoleARN string) (string, error) {
+	roleName, err := s.resolveNodeRoleName(accountID, nodeRoleARN)
+	if err != nil {
+		return "", err
+	}
 	profileName := roleName
 
 	out, err := s.iamEnsurer().GetInstanceProfile(accountID, &iam.GetInstanceProfileInput{
@@ -1018,6 +1033,9 @@ func (s *EKSServiceImpl) reconcileNodegroup(ctx context.Context, acctKV jetstrea
 			}
 			return nil, err
 		}
+		if rec.Status == eks.NodegroupStatusDeleting || rec.Status == eks.NodegroupStatusDeleteFailed {
+			return nil, errors.New(awserrors.ErrorEKSResourceInUse)
+		}
 		applyScalingUpdate(rec, scaling)
 		if labels != nil {
 			rec.Labels = applyLabelUpdate(rec.Labels, labels)
@@ -1227,33 +1245,49 @@ func (s *EKSServiceImpl) deleteNodegroup(ctx context.Context, acctKV jetstream.K
 	if err := requireCluster(ctx, acctKV, cluster); err != nil {
 		return nil, err
 	}
-	rec, err := GetNodegroupRecord(ctx, acctKV, cluster, ng)
+	rec, err := s.markNodegroupDeleting(ctx, acctKV, cluster, ng)
 	if err != nil {
-		if errors.Is(err, ErrNodegroupNotFound) {
-			return nil, nodegroupNotFound(ng)
-		}
 		return nil, err
 	}
-
-	if len(rec.InstanceIDs) > 0 {
-		if err := s.deps.Worker.TerminateWorkerInstances(ctx, rec.InstanceIDs, accountID); err != nil {
-			return nil, fmt.Errorf("terminate workers: %w", err)
-		}
+	if !s.noDeleteKick {
+		s.startNodegroupDelete(accountID, acctKV, cluster, ng)
 	}
-	if err := DeleteNodegroupRecord(ctx, acctKV, cluster, ng); err != nil {
-		return nil, err
-	}
-	// A cluster delete waits on its node groups; the last one going lets it run.
-	if meta, err := GetClusterMeta(ctx, acctKV, cluster); err == nil && meta.Status == ClusterStatusDeleting {
-		if rest, err := ListNodegroupRecords(ctx, acctKV, cluster); err == nil && len(rest) == 0 {
-			s.startTeardown(accountID, cluster)
-		}
-	}
-
-	rec.Status = eks.NodegroupStatusDeleting
 	return &eks.DeleteNodegroupOutput{Nodegroup: nodegroupRecordToAWS(rec)}, nil
 }
 
+// markNodegroupDeleting records DELETING and returns. The workers, the profile
+// and the record itself are the nodegroup reaper's to remove, so the request
+// never waits on work that scales with the group (a launching group is refused).
+func (s *EKSServiceImpl) markNodegroupDeleting(ctx context.Context, kv jetstream.KeyValue, cluster, ng string) (*NodegroupRecord, error) {
+	for range ngCASMaxRetries {
+		rec, rev, err := getNodegroupEntry(ctx, kv, cluster, ng)
+		if err != nil {
+			if errors.Is(err, ErrNodegroupNotFound) {
+				return nil, nodegroupNotFound(ng)
+			}
+			return nil, err
+		}
+		if rec.Status == eks.NodegroupStatusDeleting || rec.Status == eks.NodegroupStatusCreating {
+			return nil, errors.New(awserrors.ErrorEKSResourceInUse)
+		}
+		now := time.Now().UTC()
+		rec.Status = eks.NodegroupStatusDeleting
+		rec.StatusReason = ""
+		rec.DeletingSince = now
+		rec.DeleteReapAttempts = 0
+		rec.LastDeleteReapAttempt = time.Time{}
+		rec.ModifiedAt = now
+		ok, err := s.casPutNodegroup(ctx, kv, rec, rev)
+		if err != nil {
+			return nil, err
+		}
+		if ok {
+			return rec, nil
+		}
+	}
+	return nil, errors.New(awserrors.ErrorServerInternal)
+}
+
 // decryptNodeToken reads + decrypts the cluster's K3s join token from KV.
 func (s *EKSServiceImpl) decryptNodeToken(ctx context.Context, kv jetstream.KeyValue, cluster string) (string, error) {
 	if kv == nil {
@@ -1496,8 +1530,12 @@ func nodegroupRecordToAWS(rec *NodegroupRecord) *eks.Nodegroup {
 	// wrong; a nil slice here would marshal the key away entirely.
 	issues := make([]*eks.Issue, 0, 1)
 	if rec.StatusReason != "" {
+		code := eks.NodegroupIssueCodeNodeCreationFailure
+		if rec.Status == eks.NodegroupStatusDeleting || rec.Status == eks.NodegroupStatusDeleteFailed {
+			code = eks.NodegroupIssueCodeNodeTerminationFailure
+		}
 		issues = append(issues, &eks.Issue{
-			Code:        aws.String(eks.NodegroupIssueCodeNodeCreationFailure),
+			Code:        aws.String(code),
 			Message:     aws.String(rec.StatusReason),
 			ResourceIds: aws.StringSlice(rec.InstanceIDs),
 		})

spinifex/domains/eks/cluster/nodegroup_delete_failure_test.go

@@ -0,0 +1,255 @@
+package cluster
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/service/eks"
+	"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func (f *fakeWorkerLauncher) terminateCallCount() int {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return len(f.terminateCalls)
+}
+
+func requireNodegroupStatus(t *testing.T, f *eksServiceFixture, cluster, ng, want string) *NodegroupRecord {
+	t.Helper()
+	rec, err := GetNodegroupRecord(t.Context(), f.kv, cluster, ng)
+	require.NoError(t, err)
+	require.Equal(t, want, rec.Status)
+	return rec
+}
+
+func TestNodegroupDeletingReaper_SweepWithoutOrchestrationDepsStartsNothing(t *testing.T) {
+	t.Parallel()
+	n, err := (&EKSServiceImpl{}).NewNodegroupDeletingReaper().Sweep(t.Context())
+	require.NoError(t, err)
+	assert.Zero(t, n)
+}
+
+func TestNodegroupDeletingReaper_CancelledSweepLeavesTheDeleteRecorded(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	ctx, cancel := context.WithCancel(t.Context())
+	cancel()
+	n, err := f.svc.NewNodegroupDeletingReaper().Sweep(ctx)
+	f.svc.WaitNodegroupDeletes()
+
+	require.ErrorIs(t, err, context.Canceled)
+	assert.Zero(t, n)
+	requireNodegroupStatus(t, f, "c1", "ng1", eks.NodegroupStatusDeleting)
+}
+
+func TestNodegroupDeletingReaper_OnlyDeletingGroupsPastBackoffAreStarted(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "active", 1)
+	createReadyNodegroup(t, f, "c1", "backoff", 1)
+	_, err := deleteNodegroupCall(f, "c1", "backoff")
+	require.NoError(t, err)
+	rec := requireNodegroupStatus(t, f, "c1", "backoff", eks.NodegroupStatusDeleting)
+	rec.DeleteReapAttempts = 3
+	rec.LastDeleteReapAttempt = time.Now().UTC()
+	require.NoError(t, PutNodegroupRecord(t.Context(), f.kv, rec))
+
+	assert.Zero(t, sweepNodegroups(t, f), "an ACTIVE group and a DELETING one inside its backoff are not due")
+	requireNodegroupStatus(t, f, "c1", "active", eks.NodegroupStatusActive)
+	assert.Zero(t, f.worker.terminateCallCount())
+}
+
+func TestConvergeNodegroupDeletion_MissingOrNotDeletingRecordIsLeftAlone(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+
+	require.NoError(t, f.svc.convergeNodegroupDeletion(t.Context(), testAccountID, f.kv, "c1", "never-existed"))
+	require.NoError(t, f.svc.convergeNodegroupDeletion(t.Context(), testAccountID, f.kv, "c1", "ng1"))
+
+	requireNodegroupStatus(t, f, "c1", "ng1", eks.NodegroupStatusActive)
+	assert.Zero(t, f.worker.terminateCallCount(), "a group nobody asked to delete keeps its workers")
+	assert.Contains(t, fixtureIAM(t, f).Profiles, mustProfileName(t, f, "c1", "ng1"))
+}
+
+func mustProfileName(t *testing.T, f *eksServiceFixture, cluster, ng string) string {
+	t.Helper()
+	rec, _ := ownedProfile(t, f, cluster, ng)
+	return rec.InstanceProfileName
+}
+
+func TestAwaitWorkersGone_CancelledWhileWorkersEndingReturnsTheContextError(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	f.worker.stillLive = []string{"i-slow"}
+	f.svc.workersEndWait = time.Hour
+	f.svc.workersEndPoll = time.Hour
+	ctx, cancel := context.WithCancel(t.Context())
+	time.AfterFunc(50*time.Millisecond, cancel)
+
+	err := f.svc.awaitWorkersGone(ctx, []string{"i-slow"}, testAccountID)
+
+	require.ErrorIs(t, err, context.Canceled)
+	assert.NotErrorIs(t, err, errWorkersEnding, "a cancelled wait is not a verdict on the workers")
+}
+
+func TestDeleteNodegroup_IAMReadFailureKeepsGroupDeletingAndProfileIntact(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+	m.GetInstanceProfileErr = errors.New("iam read timed out")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+
+	cur := requireNodegroupStatus(t, f, "c1", "ng1", eks.NodegroupStatusDeleting)
+	assert.Contains(t, cur.StatusReason, "get instance profile", "the failed read is recorded for the customer")
+	assert.Contains(t, m.Profiles, rec.InstanceProfileName, "an unreadable profile is not a missing one")
+	assert.Zero(t, m.DeleteInstanceProfileCalls)
+
+	m.GetInstanceProfileErr = nil
+	backdateNodegroupReap(t, f, "c1", "ng1")
+	sweepNodegroups(t, f)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.NotContains(t, m.Profiles, rec.InstanceProfileName)
+}
+
+func TestDeleteNodegroup_ProfileAlreadyGoneStillFinishesTheGroup(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+	delete(m.Profiles, rec.InstanceProfileName)
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.Zero(t, m.RemoveRoleCalls)
+	assert.Zero(t, m.DeleteInstanceProfileCalls)
+}
+
+func TestDeleteNodegroup_ProfileVanishingMidDeleteStillFinishesTheGroup(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	m := fixtureIAM(t, f)
+	m.DeleteInstanceProfileErr = awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile gone.")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.Equal(t, 1, m.RemoveRoleCalls)
+}
+
+func TestDeleteNodegroup_DeleteProfileFailureKeepsGroupDeletingUntilItClears(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+	m.DeleteInstanceProfileErr = errors.New("iam unavailable")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+	cur := requireNodegroupStatus(t, f, "c1", "ng1", eks.NodegroupStatusDeleting)
+	assert.Contains(t, cur.StatusReason, "delete instance profile")
+	assert.Empty(t, cur.InstanceIDs, "workers proven gone are not terminated again")
+
+	m.DeleteInstanceProfileErr = nil
+	backdateNodegroupReap(t, f, "c1", "ng1")
+	sweepNodegroups(t, f)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.NotContains(t, m.Profiles, rec.InstanceProfileName)
+}
+
+func TestEnsureNodegroupProfile_RecreatesAMissingRecordedProfileAsItsOwn(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+	delete(m.Profiles, rec.InstanceProfileName)
+
+	arn, err := f.svc.ensureNodegroupProfile(testAccountID, rec)
+
+	require.NoError(t, err)
+	assert.Contains(t, arn, rec.InstanceProfileName)
+	got := m.Profiles[rec.InstanceProfileName]
+	require.NotNil(t, got)
+	assert.True(t, profileOwnedBy(got, rec), "a re-created profile carries this incarnation's tags")
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(got))
+}
+
+func TestEnsureNodegroupProfile_RefusesAProfileThatIsNotThisGroups(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, p := ownedProfile(t, f, "c1", "ng1")
+	p.Tags = nil
+	callsBefore := fixtureIAM(t, f).AddRoleToInstanceProfileCalls
+
+	_, err := f.svc.ensureNodegroupProfile(testAccountID, rec)
+
+	require.ErrorContains(t, err, "is not this nodegroup's")
+	assert.Equal(t, callsBefore, fixtureIAM(t, f).AddRoleToInstanceProfileCalls, "no role is attached to a profile EKS does not own")
+}
+
+func TestEnsureNodegroupProfile_IAMFailuresSurfaceWithoutHalfWork(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+
+	m.GetInstanceProfileErr = errors.New("iam read failed")
+	_, err := f.svc.ensureNodegroupProfile(testAccountID, rec)
+	require.ErrorContains(t, err, "get instance profile")
+	m.GetInstanceProfileErr = nil
+
+	delete(m.Profiles, rec.InstanceProfileName)
+	m.CreateInstanceProfileErr = errors.New("iam write failed")
+	_, err = f.svc.ensureNodegroupProfile(testAccountID, rec)
+	require.ErrorContains(t, err, "create instance profile")
+	assert.NotContains(t, m.Profiles, rec.InstanceProfileName)
+}
+
+func TestMarkNodegroupDeleting_UnknownGroupIsResourceNotFound(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+
+	_, err := f.svc.markNodegroupDeleting(t.Context(), f.kv, "c1", "ghost")
+
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+}

spinifex/domains/eks/cluster/nodegroup_delete_test.go

@@ -0,0 +1,662 @@
+package cluster
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/aws/aws-sdk-go/service/eks"
+	"github.com/aws/aws-sdk-go/service/iam"
+	iammock "github.com/mulgadc/spinifex/spinifex/domains/iam/mock"
+	"github.com/mulgadc/spinifex/spinifex/foundation/aws/arn"
+	"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	profileClusterTag   = "spinifex:eks-cluster"
+	profileNodegroupTag = "spinifex:eks-nodegroup"
+	fixtureNodeRoleName = "eks-node"
+)
+
+func fixtureIAM(t *testing.T, f *eksServiceFixture) *iammock.SystemInstanceRoleEnsurer {
+	t.Helper()
+	m, ok := f.svc.deps.IAM.(*iammock.SystemInstanceRoleEnsurer)
+	require.True(t, ok, "the fixture IAM must be the in-memory mock")
+	return m
+}
+
+// createReadyNodegroup creates a nodegroup whose n workers register Ready.
+func createReadyNodegroup(t *testing.T, f *eksServiceFixture, cluster, ng string, n int64) {
+	t.Helper()
+	markWorkersReady(t, f, cluster, ng, int(n))
+	_, err := f.svc.CreateNodegroup(context.Background(), createNGInput(cluster, ng, n), testAccountID)
+	require.NoError(t, err)
+	f.svc.WaitLaunches()
+}
+
+// createUnreadyNodegroup creates a nodegroup that never registers, so it ends
+// CREATE_FAILED holding the workers it launched.
+func createUnreadyNodegroup(t *testing.T, f *eksServiceFixture, cluster, ng string, n int64) {
+	t.Helper()
+	_, err := f.svc.CreateNodegroup(context.Background(), createNGInput(cluster, ng, n), testAccountID)
+	require.NoError(t, err)
+	f.svc.WaitLaunches()
+}
+
+func deleteNodegroupCall(f *eksServiceFixture, cluster, ng string) (*eks.DeleteNodegroupOutput, error) {
+	return f.svc.DeleteNodegroup(context.Background(), &eks.DeleteNodegroupInput{
+		ClusterName: aws.String(cluster), NodegroupName: aws.String(ng),
+	}, testAccountID)
+}
+
+func describeNodegroupCall(f *eksServiceFixture, cluster, ng string) (*eks.DescribeNodegroupOutput, error) {
+	return f.svc.DescribeNodegroup(context.Background(), &eks.DescribeNodegroupInput{
+		ClusterName: aws.String(cluster), NodegroupName: aws.String(ng),
+	}, testAccountID)
+}
+
+func sweepNodegroups(t *testing.T, f *eksServiceFixture) int {
+	t.Helper()
+	n, err := f.svc.NewNodegroupDeletingReaper().Sweep(context.Background())
+	require.NoError(t, err)
+	f.svc.WaitNodegroupDeletes()
+	return n
+}
+
+// backdateNodegroupReap moves the last reap pass far enough back that any
+// retry backoff has elapsed.
+func backdateNodegroupReap(t *testing.T, f *eksServiceFixture, cluster, ng string) {
+	t.Helper()
+	rec, err := GetNodegroupRecord(t.Context(), f.kv, cluster, ng)
+	require.NoError(t, err)
+	rec.LastDeleteReapAttempt = time.Now().UTC().Add(-24 * time.Hour)
+	rec.DeletingSince = time.Now().UTC().Add(-24 * time.Hour)
+	require.NoError(t, PutNodegroupRecord(t.Context(), f.kv, rec))
+}
+
+func requireErrorCode(t *testing.T, err error, want string) {
+	t.Helper()
+	require.Error(t, err)
+	code, ok := awserrors.ResolveErrorCode(err)
+	require.True(t, ok, "error %v carries no AWS code", err)
+	assert.Equal(t, want, code)
+}
+
+func profileTag(p *iam.InstanceProfile, key string) string {
+	for _, tg := range p.Tags {
+		if aws.StringValue(tg.Key) == key {
+			return aws.StringValue(tg.Value)
+		}
+	}
+	return ""
+}
+
+func profileHasTagValue(p *iam.InstanceProfile, value string) bool {
+	for _, tg := range p.Tags {
+		if aws.StringValue(tg.Value) == value {
+			return true
+		}
+	}
+	return false
+}
+
+func profileRoleNames(p *iam.InstanceProfile) []string {
+	var names []string
+	for _, r := range p.Roles {
+		names = append(names, aws.StringValue(r.RoleName))
+	}
+	return names
+}
+
+// ownedProfile returns the nodegroup's record and the instance profile it
+// names, failing the test if either is missing.
+func ownedProfile(t *testing.T, f *eksServiceFixture, cluster, ng string) (*NodegroupRecord, *iam.InstanceProfile) {
+	t.Helper()
+	rec, err := GetNodegroupRecord(t.Context(), f.kv, cluster, ng)
+	require.NoError(t, err)
+	require.NotEmpty(t, rec.InstanceProfileName, "the record must name the profile it owns")
+	require.NotEmpty(t, rec.InstanceProfileID, "the record must carry the per-incarnation id")
+	p, ok := fixtureIAM(t, f).Profiles[rec.InstanceProfileName]
+	require.True(t, ok, "profile %q must exist", rec.InstanceProfileName)
+	return rec, p
+}
+
+func seedCustomerProfile(f *eksServiceFixture, t *testing.T, name, role string, tags map[string]string) {
+	t.Helper()
+	p := &iam.InstanceProfile{
+		InstanceProfileName: aws.String(name),
+		Arn:                 aws.String("arn:aws:iam::" + testAccountID + ":instance-profile/" + name),
+	}
+	if role != "" {
+		p.Roles = []*iam.Role{{RoleName: aws.String(role)}}
+	}
+	for k, v := range tags {
+		p.Tags = append(p.Tags, &iam.Tag{Key: aws.String(k), Value: aws.String(v)})
+	}
+	fixtureIAM(t, f).Profiles[name] = p
+}
+
+func TestDeleteNodegroup_ReturnsDeletingWithoutTerminatingInRequest(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 2)
+
+	out, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, eks.NodegroupStatusDeleting, aws.StringValue(out.Nodegroup.Status))
+
+	assert.Zero(t, f.worker.terminatedCount(), "the request must not wait for workers to end")
+	rec, err := GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
+	require.NoError(t, err, "the record stays until the reaper has finished")
+	assert.Equal(t, eks.NodegroupStatusDeleting, rec.Status)
+
+	desc, err := describeNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, eks.NodegroupStatusDeleting, aws.StringValue(desc.Nodegroup.Status))
+}
+
+func TestDeleteNodegroup_ReaperEndsWorkersThenRemovesGroup(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 2)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, 1, sweepNodegroups(t, f))
+
+	require.Len(t, f.worker.terminateCalls, 1)
+	assert.ElementsMatch(t, rec.InstanceIDs, f.worker.terminateCalls[0])
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	_, ok := fixtureIAM(t, f).Profiles[rec.InstanceProfileName]
+	assert.False(t, ok, "the group is not gone until its profile is")
+}
+
+func TestDeleteNodegroup_RepeatedDeleteOnDeletingGroupIsResourceInUse(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	_, err = deleteNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceInUse)
+}
+
+func TestDeleteNodegroup_DeletingGroupRefusesUpdates(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	_, err = f.svc.UpdateNodegroupConfig(context.Background(), &eks.UpdateNodegroupConfigInput{
+		ClusterName:   aws.String("c1"),
+		NodegroupName: aws.String("ng1"),
+		ScalingConfig: &eks.NodegroupScalingConfig{MinSize: aws.Int64(1), MaxSize: aws.Int64(3), DesiredSize: aws.Int64(3)},
+	}, testAccountID)
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceInUse)
+	assert.Equal(t, 1, f.worker.runCount(), "a DELETING group must not launch workers")
+}
+
+func TestDeleteNodegroup_SameNameCreateWhileDeletingIsResourceInUse(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	_, err = f.svc.CreateNodegroup(context.Background(), createNGInput("c1", "ng1", 1), testAccountID)
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceInUse)
+}
+
+func TestDeleteNodegroup_DeleteClusterRefusedWhileGroupDeleting(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	_, err = f.svc.DeleteCluster(context.Background(), &eks.DeleteClusterInput{Name: aws.String("c1")}, testAccountID)
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceInUse)
+}
+
+func TestDeleteNodegroup_BootReclaimLeavesDeletingGroupToReaper(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	require.NoError(t, PutNodegroupRecord(t.Context(), f.kv, &NodegroupRecord{
+		ClusterName: "c1", Name: "ng1", Status: eks.NodegroupStatusDeleting,
+		InstanceIDs: []string{"i-w1", "i-w2"}, NodeRole: fixtureNodeRoleARN,
+	}))
+
+	f.svc.reclaimOrphanedNodegroups(t.Context(), testAccountID, f.kv, "c1")
+
+	assert.Zero(t, f.worker.terminatedCount())
+	rec, err := GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, eks.NodegroupStatusDeleting, rec.Status)
+	assert.Equal(t, []string{"i-w1", "i-w2"}, rec.InstanceIDs)
+}
+
+func TestDeleteNodegroup_RepeatedReaperFailureEndsDeleteFailedWithHealthIssue(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	f.worker.termErr = errors.New("terminate refused")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	var rec *NodegroupRecord
+	for range 30 {
+		sweepNodegroups(t, f)
+		rec, err = GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
+		require.NoError(t, err, "a failing delete must leave the record, never drop it")
+		if rec.Status == eks.NodegroupStatusDeleteFailed {
+			break
+		}
+		backdateNodegroupReap(t, f, "c1", "ng1")
+	}
+	require.Equal(t, eks.NodegroupStatusDeleteFailed, rec.Status)
+
+	desc, err := describeNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	require.NotEmpty(t, desc.Nodegroup.Health.Issues, "DELETE_FAILED must say why")
+	assert.Equal(t, eks.NodegroupIssueCodeNodeTerminationFailure, aws.StringValue(desc.Nodegroup.Health.Issues[0].Code))
+
+	// The customer retries the delete once the cause is gone.
+	f.worker.termErr = nil
+	_, err = deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+}
+
+func TestCreateNodegroup_OwnsATaggedProfileRecordedBeforeFirstLaunch(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	var seenAtLaunch string
+	f.worker.beforeRun = func(in *ec2.RunInstancesInput) {
+		if rec, err := GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1"); err == nil && seenAtLaunch == "" {
+			seenAtLaunch = rec.InstanceProfileName
+		}
+	}
+	createReadyNodegroup(t, f, "c1", "ng1", 2)
+
+	rec, p := ownedProfile(t, f, "c1", "ng1")
+	assert.NotEqual(t, fixtureNodeRoleName, rec.InstanceProfileName, "the profile is the group's own, not the role-named one")
+	assert.Equal(t, rec.InstanceProfileName, seenAtLaunch, "the name must be durable before any worker launches")
+	assert.Equal(t, "c1", profileTag(p, profileClusterTag))
+	assert.Equal(t, "ng1", profileTag(p, profileNodegroupTag))
+	assert.True(t, profileHasTagValue(p, rec.InstanceProfileID), "a tag must carry the incarnation id")
+	for _, tg := range p.Tags {
+		assert.True(t, strings.HasPrefix(aws.StringValue(tg.Key), "spinifex:"), "tag %q: aws: keys are reserved, ownership tags must be spinifex:", aws.StringValue(tg.Key))
+	}
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(p))
+
+	require.Equal(t, 2, f.worker.runCount())
+	for _, in := range f.worker.runCalls {
+		require.NotNil(t, in.IamInstanceProfile)
+		assert.True(t, strings.HasSuffix(aws.StringValue(in.IamInstanceProfile.Arn), "/"+rec.InstanceProfileName),
+			"every worker launches with the group's own profile, got %s", aws.StringValue(in.IamInstanceProfile.Arn))
+	}
+}
+
+func TestCreateNodegroup_TwoGroupsOnOneRoleGetTwoProfiles(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createUnreadyNodegroup(t, f, "c1", "ng-a", 1)
+	createUnreadyNodegroup(t, f, "c1", "ng-b", 1)
+
+	recA, pA := ownedProfile(t, f, "c1", "ng-a")
+	recB, pB := ownedProfile(t, f, "c1", "ng-b")
+	assert.NotEqual(t, recA.InstanceProfileName, recB.InstanceProfileName)
+	assert.NotEqual(t, recA.InstanceProfileID, recB.InstanceProfileID)
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(pA))
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(pB))
+}
+
+func TestCreateNodegroup_RecreatedGroupOfSameNameGetsANewProfile(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createUnreadyNodegroup(t, f, "c1", "ng1", 1)
+	first, _ := ownedProfile(t, f, "c1", "ng1")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+
+	createUnreadyNodegroup(t, f, "c1", "ng1", 1)
+	second, _ := ownedProfile(t, f, "c1", "ng1")
+	assert.NotEqual(t, first.InstanceProfileName, second.InstanceProfileName)
+	assert.NotEqual(t, first.InstanceProfileID, second.InstanceProfileID)
+	_, stillThere := fixtureIAM(t, f).Profiles[first.InstanceProfileName]
+	assert.False(t, stillThere, "the first incarnation's profile must have gone with it")
+}
+
+func TestDeleteNodegroup_AfterCreateFailedRemovesProfileAndGroup(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createUnreadyNodegroup(t, f, "c1", "ng1", 2)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	require.Equal(t, eks.NodegroupStatusCreateFailed, rec.Status)
+
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+
+	_, ok := fixtureIAM(t, f).Profiles[rec.InstanceProfileName]
+	assert.False(t, ok)
+	assert.Equal(t, 2, f.worker.terminatedCount())
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+}
+
+func TestDeleteNodegroup_ProfileSurvivesWhileWorkersAreLive(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	f.worker.termErr = errors.New("terminate refused")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+
+	m := fixtureIAM(t, f)
+	assert.Contains(t, m.Profiles, rec.InstanceProfileName, "a profile in use by live workers must not be touched")
+	assert.Zero(t, m.RemoveRoleCalls)
+	assert.Zero(t, m.DeleteInstanceProfileCalls)
+}
+
+func TestDeleteNodegroup_PartialProfileDeleteConvergesOnRetry(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+	m.RemoveRoleFromInstanceProfileErr = errors.New("iam unavailable")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+	still, err := GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
+	require.NoError(t, err, "the group is not gone while its profile is")
+	assert.Equal(t, eks.NodegroupStatusDeleting, still.Status)
+
+	m.RemoveRoleFromInstanceProfileErr = nil
+	backdateNodegroupReap(t, f, "c1", "ng1")
+	sweepNodegroups(t, f)
+	assert.NotContains(t, m.Profiles, rec.InstanceProfileName)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+}
+
+func TestDeleteNodegroup_CustomerRoleNamedProfileSurvives(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	seedCustomerProfile(f, t, fixtureNodeRoleName, fixtureNodeRoleName, nil)
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+
+	m := fixtureIAM(t, f)
+	assert.NotContains(t, m.Profiles, rec.InstanceProfileName, "the group's own profile goes")
+	require.Contains(t, m.Profiles, fixtureNodeRoleName, "a customer's profile is never EKS's to delete")
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(m.Profiles[fixtureNodeRoleName]))
+	assert.Equal(t, 1, m.DeleteInstanceProfileCalls)
+}
+
+func TestDeleteNodegroup_RecordedProfileMissingOwnershipTagSurvives(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	m := fixtureIAM(t, f)
+	_, err := m.UntagInstanceProfile(testAccountID, &iam.UntagInstanceProfileInput{
+		InstanceProfileName: aws.String(rec.InstanceProfileName),
+		TagKeys:             aws.StringSlice([]string{profileClusterTag}),
+	})
+	require.NoError(t, err)
+
+	_, err = deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+
+	require.Contains(t, m.Profiles, rec.InstanceProfileName)
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(m.Profiles[rec.InstanceProfileName]))
+	assert.Zero(t, m.RemoveRoleCalls)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+}
+
+func TestDeleteNodegroup_OldLayoutGroupDeletesAndLeavesRoleNamedProfile(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	seedCustomerProfile(f, t, fixtureNodeRoleName, fixtureNodeRoleName, nil)
+	require.NoError(t, PutNodegroupRecord(t.Context(), f.kv, &NodegroupRecord{
+		ClusterName: "c1", Name: "old", Status: eks.NodegroupStatusActive,
+		InstanceIDs: []string{"i-old1"}, NodeRole: fixtureNodeRoleARN,
+	}))
+
+	_, err := deleteNodegroupCall(f, "c1", "old")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+
+	require.Len(t, f.worker.terminateCalls, 1)
+	assert.Equal(t, []string{"i-old1"}, f.worker.terminateCalls[0])
+	_, err = describeNodegroupCall(f, "c1", "old")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	m := fixtureIAM(t, f)
+	require.Contains(t, m.Profiles, fixtureNodeRoleName)
+	assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(m.Profiles[fixtureNodeRoleName]))
+	assert.Zero(t, m.RemoveRoleCalls)
+	assert.Zero(t, m.DeleteInstanceProfileCalls)
+}
+
+// No path deletes a profile unless it is recorded, tagged for this
+// incarnation, holds the record's node role and is named by no other record.
+func TestInvariant_NodegroupDeleteNeverRemovesAProfileItDoesNotOwn(t *testing.T) {
+	t.Parallel()
+	ownership := map[string]string{profileClusterTag: "c1", profileNodegroupTag: "ng1"}
+
+	t.Run("customer pre-creates a profile at a guessed name", func(t *testing.T) {
+		t.Parallel()
+		f := newEKSServiceFixture(t)
+		seedActiveClusterWithToken(t, f, "c1")
+		guesses := []string{
+			"eks-" + arn.EKSNodegroupDiscriminator(testAccountID, "c1", "ng1"),
+			"eks-c1-ng1",
+			"c1-ng1",
+			fixtureNodeRoleName,
+		}
+		for _, name := range guesses {
+			seedCustomerProfile(f, t, name, fixtureNodeRoleName, ownership)
+		}
+		createReadyNodegroup(t, f, "c1", "ng1", 1)
+		rec, _ := ownedProfile(t, f, "c1", "ng1")
+		require.NotContains(t, guesses, rec.InstanceProfileName, "the name must not be guessable")
+
+		_, err := deleteNodegroupCall(f, "c1", "ng1")
+		require.NoError(t, err)
+		sweepNodegroups(t, f)
+
+		m := fixtureIAM(t, f)
+		for _, name := range guesses {
+			require.Contains(t, m.Profiles, name, "customer profile %q was deleted", name)
+			assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(m.Profiles[name]), "customer profile %q lost its role", name)
+		}
+	})
+
+	t.Run("recorded profile was swapped for one holding another role", func(t *testing.T) {
+		t.Parallel()
+		f := newEKSServiceFixture(t)
+		seedActiveClusterWithToken(t, f, "c1")
+		createReadyNodegroup(t, f, "c1", "ng1", 1)
+		rec, p := ownedProfile(t, f, "c1", "ng1")
+		p.Roles = []*iam.Role{{RoleName: aws.String("customer-role")}}
+
+		_, err := deleteNodegroupCall(f, "c1", "ng1")
+		require.NoError(t, err)
+		sweepNodegroups(t, f)
+
+		m := fixtureIAM(t, f)
+		require.Contains(t, m.Profiles, rec.InstanceProfileName)
+		assert.Equal(t, []string{"customer-role"}, profileRoleNames(m.Profiles[rec.InstanceProfileName]))
+		assert.Zero(t, m.RemoveRoleCalls)
+		assert.Zero(t, m.DeleteInstanceProfileCalls)
+	})
+
+	t.Run("another nodegroup record names the same profile", func(t *testing.T) {
+		t.Parallel()
+		f := newEKSServiceFixture(t)
+		seedActiveClusterWithToken(t, f, "c1")
+		createReadyNodegroup(t, f, "c1", "ng1", 1)
+		rec, _ := ownedProfile(t, f, "c1", "ng1")
+		require.NoError(t, PutNodegroupRecord(t.Context(), f.kv, &NodegroupRecord{
+			ClusterName: "c1", Name: "ng2", Status: eks.NodegroupStatusActive, NodeRole: fixtureNodeRoleARN,
+			InstanceProfileName: rec.InstanceProfileName, InstanceProfileID: rec.InstanceProfileID,
+		}))
+
+		_, err := deleteNodegroupCall(f, "c1", "ng1")
+		require.NoError(t, err)
+		sweepNodegroups(t, f)
+
+		m := fixtureIAM(t, f)
+		require.Contains(t, m.Profiles, rec.InstanceProfileName)
+		assert.Equal(t, []string{fixtureNodeRoleName}, profileRoleNames(m.Profiles[rec.InstanceProfileName]))
+		assert.Zero(t, m.DeleteInstanceProfileCalls)
+	})
+}
+
+func TestDeleteNodegroup_ProfileAndRecordKeptUntilWorkersProvenGone(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 2)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	f.worker.stillLive = []string{rec.InstanceIDs[0]}
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+
+	m := fixtureIAM(t, f)
+	assert.Contains(t, m.Profiles, rec.InstanceProfileName)
+	assert.Zero(t, m.RemoveRoleCalls)
+	assert.Zero(t, m.DeleteInstanceProfileCalls)
+	got, err := describeNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, eks.NodegroupStatusDeleting, aws.StringValue(got.Nodegroup.Status))
+	cur, err := GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, rec.InstanceIDs, cur.InstanceIDs, "worker ids stay recorded until they are proven gone")
+	assert.Zero(t, cur.DeleteReapAttempts, "workers still ending is not a failed pass")
+
+	f.worker.mu.Lock()
+	f.worker.stillLive = nil
+	f.worker.mu.Unlock()
+	sweepNodegroups(t, f)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.NotContains(t, m.Profiles, rec.InstanceProfileName)
+}
+
+func TestDeleteNodegroup_UndecidableWorkerStateKeepsProfile(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	f.worker.liveErr = errors.New("describe fan-out did not complete")
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+
+	m := fixtureIAM(t, f)
+	assert.Contains(t, m.Profiles, rec.InstanceProfileName)
+	assert.Zero(t, m.DeleteInstanceProfileCalls)
+}
+
+func TestDeleteNodegroup_KickFinishesTheDeleteWithoutAReaperSweep(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 1)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	f.svc.noDeleteKick = false
+
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	f.svc.WaitNodegroupDeletes()
+
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.NotContains(t, fixtureIAM(t, f).Profiles, rec.InstanceProfileName)
+}
+
+func TestDeleteNodegroup_WorkerStuckPastDeadlineEndsDeleteFailedKeepingProfile(t *testing.T) {
+	t.Parallel()
+	f := newEKSServiceFixture(t)
+	seedActiveClusterWithToken(t, f, "c1")
+	createReadyNodegroup(t, f, "c1", "ng1", 2)
+	rec, _ := ownedProfile(t, f, "c1", "ng1")
+	stuck := rec.InstanceIDs[0]
+	f.worker.stillLive = []string{stuck}
+	_, err := deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+
+	sweepNodegroups(t, f)
+	cur, err := GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
+	require.NoError(t, err)
+	require.Equal(t, eks.NodegroupStatusDeleting, cur.Status, "within the deadline the group keeps waiting")
+
+	backdateNodegroupReap(t, f, "c1", "ng1")
+	sweepNodegroups(t, f)
+
+	desc, err := describeNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	assert.Equal(t, eks.NodegroupStatusDeleteFailed, aws.StringValue(desc.Nodegroup.Status))
+	require.NotEmpty(t, desc.Nodegroup.Health.Issues)
+	issue := desc.Nodegroup.Health.Issues[0]
+	assert.Equal(t, eks.NodegroupIssueCodeNodeTerminationFailure, aws.StringValue(issue.Code))
+	assert.Contains(t, aws.StringValue(issue.Message), stuck)
+	assert.Contains(t, fixtureIAM(t, f).Profiles, rec.InstanceProfileName)
+
+	f.worker.mu.Lock()
+	f.worker.stillLive = nil
+	f.worker.mu.Unlock()
+	_, err = deleteNodegroupCall(f, "c1", "ng1")
+	require.NoError(t, err)
+	sweepNodegroups(t, f)
+	_, err = describeNodegroupCall(f, "c1", "ng1")
+	requireErrorCode(t, err, awserrors.ErrorEKSResourceNotFound)
+	assert.NotContains(t, fixtureIAM(t, f).Profiles, rec.InstanceProfileName)
+}

spinifex/domains/eks/cluster/nodegroup_profile.go

@@ -0,0 +1,174 @@
+package cluster
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"strings"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/iam"
+	"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
+	"github.com/nats-io/nats.go/jetstream"
+)
+
+const (
+	nodegroupProfilePrefix = "eks-"
+
+	profileClusterTagKey   = "spinifex:eks-cluster"
+	profileNodegroupTagKey = "spinifex:eks-nodegroup"
+	profileIDTagKey        = "spinifex:eks-profile-id"
+)
+
+// nodeProfileForLaunch returns the ARN of the instance profile a worker of rec
+// launches with. A record that names no profile of its own predates per-group
+// profiles and keeps the role-named one it already uses.
+func (s *EKSServiceImpl) nodeProfileForLaunch(accountID string, rec *NodegroupRecord) (string, error) {
+	if rec.InstanceProfileName == "" {
+		return s.ensureNodeInstanceProfile(accountID, rec.NodeRole)
+	}
+	return s.ensureNodegroupProfile(accountID, rec)
+}
+
+func nodegroupProfileTags(rec *NodegroupRecord) []*iam.Tag {
+	return []*iam.Tag{
+		{Key: aws.String(profileClusterTagKey), Value: aws.String(rec.ClusterName)},
+		{Key: aws.String(profileNodegroupTagKey), Value: aws.String(rec.Name)},
+		{Key: aws.String(profileIDTagKey), Value: aws.String(rec.InstanceProfileID)},
+	}
+}
+
+// profileOwnedBy reports whether p carries the ownership tags of this exact
+// incarnation of rec.
+func profileOwnedBy(p *iam.InstanceProfile, rec *NodegroupRecord) bool {
+	have := map[string]string{}
+	for _, t := range p.Tags {
+		have[aws.StringValue(t.Key)] = aws.StringValue(t.Value)
+	}
+	for _, want := range nodegroupProfileTags(rec) {
+		if have[aws.StringValue(want.Key)] != aws.StringValue(want.Value) {
+			return false
+		}
+	}
+	return true
+}
+
+func roleNameFromARN(roleARN string) string {
+	return roleARN[strings.LastIndex(roleARN, "/")+1:]
+}
+
+// ensureNodegroupProfile finds or creates the instance profile recorded for
+// rec, tagged as its own, and makes sure it holds the node role. Concurrent
+// worker launches converge on the one profile.
+func (s *EKSServiceImpl) ensureNodegroupProfile(accountID string, rec *NodegroupRecord) (string, error) {
+	roleName, err := s.resolveNodeRoleName(accountID, rec.NodeRole)
+	if err != nil {
+		return "", err
+	}
+	name := rec.InstanceProfileName
+	ens := s.iamEnsurer()
+
+	got, err := ens.GetInstanceProfile(accountID, &iam.GetInstanceProfileInput{InstanceProfileName: aws.String(name)})
+	if err != nil {
+		if code, _ := awserrors.ResolveErrorCode(err); code != awserrors.ErrorIAMNoSuchEntity {
+			return "", fmt.Errorf("get instance profile %q: %w", name, err)
+		}
+		_, cerr := ens.CreateInstanceProfile(accountID, &iam.CreateInstanceProfileInput{
+			InstanceProfileName: aws.String(name),
+			Tags:                nodegroupProfileTags(rec),
+		})
+		if cerr != nil {
+			// A racing launch may have created it first; re-read and converge.
+			if code, _ := awserrors.ResolveErrorCode(cerr); code != awserrors.ErrorIAMEntityAlreadyExists {
+				return "", fmt.Errorf("create instance profile %q: %w", name, cerr)
+			}
+		}
+		got, err = ens.GetInstanceProfile(accountID, &iam.GetInstanceProfileInput{InstanceProfileName: aws.String(name)})
+		if err != nil {
+			return "", fmt.Errorf("re-get instance profile %q: %w", name, err)
+		}
+	}
+	if !profileOwnedBy(got.InstanceProfile, rec) {
+		return "", fmt.Errorf("instance profile %q exists but is not this nodegroup's", name)
+	}
+	return s.attachRoleToProfile(accountID, name, roleName,
+		aws.StringValue(got.InstanceProfile.Arn), len(got.InstanceProfile.Roles) > 0)
+}
+
+// deleteNodegroupProfile removes the profile recorded for rec, and only that
+// one: it must be recorded, tagged for this incarnation, hold exactly the node
+// role and be named by no other record. Anything else is left alone.
+func (s *EKSServiceImpl) deleteNodegroupProfile(ctx context.Context, accountID string, kv jetstream.KeyValue, rec *NodegroupRecord) error {
+	name := rec.InstanceProfileName
+	ens := s.iamEnsurer()
+	if name == "" || ens == nil {
+		return nil
+	}
+	got, err := ens.GetInstanceProfile(accountID, &iam.GetInstanceProfileInput{InstanceProfileName: aws.String(name)})
+	if err != nil {
+		if code, _ := awserrors.ResolveErrorCode(err); code == awserrors.ErrorIAMNoSuchEntity {
+			return nil
+		}
+		return fmt.Errorf("get instance profile %q: %w", name, err)
+	}
+	p := got.InstanceProfile
+
+	if !profileOwnedBy(p, rec) {
+		slog.WarnContext(ctx, "eks: leaving instance profile whose ownership tags do not match its nodegroup",
+			"cluster", rec.ClusterName, "nodegroup", rec.Name, "profile", name)
+		return nil
+	}
+	nodeRole := roleNameFromARN(rec.NodeRole)
+	if len(p.Roles) > 1 || (len(p.Roles) == 1 && aws.StringValue(p.Roles[0].RoleName) != nodeRole) {
+		slog.WarnContext(ctx, "eks: leaving instance profile that holds a role other than the node role",
+			"cluster", rec.ClusterName, "nodegroup", rec.Name, "profile", name)
+		return nil
+	}
+	shared, err := s.profileNamedByOtherRecord(ctx, kv, rec)
+	if err != nil {
+		return err
+	}
+	if shared {
+		slog.WarnContext(ctx, "eks: leaving instance profile another nodegroup record names",
+			"cluster", rec.ClusterName, "nodegroup", rec.Name, "profile", name)
+		return nil
+	}
+
+	if len(p.Roles) == 1 {
+		_, err := ens.RemoveRoleFromInstanceProfile(accountID, &iam.RemoveRoleFromInstanceProfileInput{
+			InstanceProfileName: aws.String(name), RoleName: aws.String(nodeRole),
+		})
+		if err != nil {
+			if code, _ := awserrors.ResolveErrorCode(err); code != awserrors.ErrorIAMNoSuchEntity {
+				return fmt.Errorf("remove role %q from instance profile %q: %w", nodeRole, name, err)
+			}
+		}
+	}
+	if _, err := ens.DeleteInstanceProfile(accountID, &iam.DeleteInstanceProfileInput{InstanceProfileName: aws.String(name)}); err != nil {
+		if code, _ := awserrors.ResolveErrorCode(err); code != awserrors.ErrorIAMNoSuchEntity {
+			return fmt.Errorf("delete instance profile %q: %w", name, err)
+		}
+	}
+	return nil
+}
+
+// profileNamedByOtherRecord reports whether any nodegroup record in the
+// account other than rec names rec's profile.
+func (s *EKSServiceImpl) profileNamedByOtherRecord(ctx context.Context, kv jetstream.KeyValue, rec *NodegroupRecord) (bool, error) {
+	clusters, err := listClusterNames(ctx, kv)
+	if err != nil {
+		return false, fmt.Errorf("list clusters: %w", err)
+	}
+	for _, cluster := range clusters {
+		recs, err := ListNodegroupRecords(ctx, kv, cluster)
+		if err != nil {
+			return false, fmt.Errorf("list nodegroups of %s: %w", cluster, err)
+		}
+		for _, other := range recs {
+			if other.InstanceProfileName == rec.InstanceProfileName && (other.ClusterName != rec.ClusterName || other.Name != rec.Name) {
+				return true, nil
+			}
+		}
+	}
+	return false, nil
+}

spinifex/domains/eks/cluster/nodegroup_test.go

@@ -86,6 +86,12 @@ type fakeWorkerLauncher struct {
 	// before subsequent calls succeed — simulates a transient launch failure
 	// (QMP timeout, host pressure) that clears on retry.
 	failNextN int
+	// beforeRun, when set, runs at the start of every launch, before it is
+	// recorded, so a test can inspect durable state at launch time.
+	beforeRun func(*ec2.RunInstancesInput)
+	// stillLive lists instances LiveWorkerInstances keeps reporting as live.
+	stillLive []string
+	liveErr   error
 }
 
 var _ WorkerLauncher = (*fakeWorkerLauncher)(nil)
@@ -101,6 +107,9 @@ func (f *fakeWorkerLauncher) RunWorkerInstance(ctx context.Context, input *ec2.R
 func (f *fakeWorkerLauncher) RunWorkerInstanceOnNode(_ context.Context, nodeID string, input *ec2.RunInstancesInput, _ string) (*ec2.Reservation, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
+	if f.beforeRun != nil {
+		f.beforeRun(input)
+	}
 	f.runCalls = append(f.runCalls, input)
 	f.runNodes = append(f.runNodes, nodeID)
 	if f.failNextN > 0 {
@@ -129,6 +138,21 @@ func (f *fakeWorkerLauncher) TerminateWorkerInstances(_ context.Context, ids []s
 	return f.termErr
 }
 
+func (f *fakeWorkerLauncher) LiveWorkerInstances(_ context.Context, ids []string, _ string) ([]string, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if f.liveErr != nil {
+		return nil, f.liveErr
+	}
+	var live []string
+	for _, id := range ids {
+		if slices.Contains(f.stillLive, id) {
+			live = append(live, id)
+		}
+	}
+	return live, nil
+}
+
 // runCount / terminatedCount return the recorded call totals under the lock so
 // concurrent tests can read them safely after joining.
 func (f *fakeWorkerLauncher) runCount() int {
@@ -927,7 +951,7 @@ func TestUpdateNodegroupConfig_CapacityErrorSurfacesCode(t *testing.T) {
 	})
 }
 
-func TestDeleteNodegroup_TerminatesAndIsIdempotent(t *testing.T) {
+func TestDeleteNodegroup_ReapedGroupIsGoneAndRepeatDeleteIsNotFound(t *testing.T) {
 	t.Parallel()
 	f := newEKSServiceFixture(t)
 	seedActiveClusterWithToken(t, f, "c1")
@@ -940,13 +964,16 @@ func TestDeleteNodegroup_TerminatesAndIsIdempotent(t *testing.T) {
 		ClusterName: aws.String("c1"), NodegroupName: aws.String("ng1"),
 	}, testAccountID)
 	require.NoError(t, err)
+	_, err = f.svc.NewNodegroupDeletingReaper().Sweep(context.Background())
+	require.NoError(t, err)
+	f.svc.WaitNodegroupDeletes()
 	require.Len(t, f.worker.terminateCalls, 1)
 	assert.Len(t, f.worker.terminateCalls[0], 2)
 
 	_, err = GetNodegroupRecord(t.Context(), f.kv, "c1", "ng1")
 	assert.ErrorIs(t, err, ErrNodegroupNotFound)
 
-	// Second delete → ResourceNotFoundException (record already gone).
+	// Once the reaper has finished the group is gone: ResourceNotFoundException.
 	_, err = f.svc.DeleteNodegroup(context.Background(), &eks.DeleteNodegroupInput{
 		ClusterName: aws.String("c1"), NodegroupName: aws.String("ng1"),
 	}, testAccountID)

spinifex/domains/eks/cluster/service_impl.go

@@ -201,20 +201,26 @@ type WorkerLauncher interface {
 	// host spread. An empty nodeID launches on the local node like RunWorkerInstance.
 	RunWorkerInstanceOnNode(ctx context.Context, nodeID string, input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error)
 	TerminateWorkerInstances(ctx context.Context, instanceIDs []string, accountID string) error
+	// LiveWorkerInstances returns the given workers not yet proven terminated or
+	// absent. An undecidable answer is an error, never an empty list.
+	LiveWorkerInstances(ctx context.Context, instanceIDs []string, accountID string) ([]string, error)
 }
 
 // iamEnsurer resolves the IAM ensurer, preferring the test-injected deps.IAM,
 // then the lazy IAMProvider (built at launch time to dodge the daemon-startup
 // NATS-KV race). Returns nil when neither yields a service, so callers fall back
-// to static creds / skip the profile. The provider's concrete service satisfies
-// instanceProfileEnsurer (identical method set to SystemInstanceRoleEnsurer).
+// to static creds / skip the profile.
 func (s *EKSServiceImpl) iamEnsurer() instanceProfileEnsurer {
 	if s.deps.IAM != nil {
 		return s.deps.IAM
 	}
 	if s.deps.IAMProvider != nil {
 		if e := s.deps.IAMProvider(); e != nil {
-			return e
+			// The concrete service has the wider surface; the shared interface
+			// stays narrow so other callers are not forced to grow it.
+			if ep, ok := e.(instanceProfileEnsurer); ok {
+				return ep
+			}
 		}
 	}
 	return nil
@@ -227,6 +233,8 @@ type instanceProfileEnsurer interface {
 	GetInstanceProfile(accountID string, input *iam.GetInstanceProfileInput) (*iam.GetInstanceProfileOutput, error)
 	CreateInstanceProfile(accountID string, input *iam.CreateInstanceProfileInput) (*iam.CreateInstanceProfileOutput, error)
 	AddRoleToInstanceProfile(accountID string, input *iam.AddRoleToInstanceProfileInput) (*iam.AddRoleToInstanceProfileOutput, error)
+	RemoveRoleFromInstanceProfile(accountID string, input *iam.RemoveRoleFromInstanceProfileInput) (*iam.RemoveRoleFromInstanceProfileOutput, error)
+	DeleteInstanceProfile(accountID string, input *iam.DeleteInstanceProfileInput) (*iam.DeleteInstanceProfileOutput, error)
 
 	// Role find-or-create surface for the system-managed control-plane role.
 	// Unlike worker node roles (customer-supplied), the k3s server role is
@@ -276,6 +284,12 @@ type EKSServiceImpl struct {
 	teardownMu      sync.Mutex
 	teardownRunning map[string]struct{}
 	teardownWG      sync.WaitGroup
+	// noDeleteKick, set by tests, leaves a nodegroup delete to the reaper sweep.
+	noDeleteKick bool
+	// workersEndWait and workersEndPoll bound how long one delete pass waits for
+	// ended workers to be proven gone; zero means the defaults.
+	workersEndWait time.Duration
+	workersEndPoll time.Duration
 	// teardownKV, when set by a test, replaces the converger's account bucket.
 	teardownKV func(ctx context.Context, accountID string) (jetstream.KeyValue, error)
 

spinifex/domains/eks/cluster/types.go

@@ -49,6 +49,16 @@ type NodegroupRecord struct {
 	Tags           map[string]string `json:"tags,omitempty"`
 	InstanceIDs    []string          `json:"instanceIds,omitempty"`
 	Health         string            `json:"health,omitempty"`
+	// InstanceProfileID and InstanceProfileName identify the instance profile
+	// EKS made for this incarnation of the nodegroup; both are empty on a record
+	// that predates it.
+	InstanceProfileID   string `json:"instanceProfileId,omitempty"`
+	InstanceProfileName string `json:"instanceProfileName,omitempty"`
+	// DeletingSince, DeleteReapAttempts and LastDeleteReapAttempt pace the reaper that finishes
+	// a DELETING nodegroup.
+	DeletingSince         time.Time `json:"deletingSince,omitzero"`
+	DeleteReapAttempts    int       `json:"deleteReapAttempts,omitempty"`
+	LastDeleteReapAttempt time.Time `json:"lastDeleteReapAttempt,omitzero"`
 	// GPUEnabled/GPUVendor cache whether InstanceTypes includes a GPU family and
 	// which vendor, so worker launches resolve the matching GPU node AMI without
 	// re-scanning InstanceTypes on every launch.

spinifex/domains/eks/cluster/worker_membership_test.go

@@ -56,6 +56,9 @@ func TestListManagedWorkers_DropsWorkersTheNodegroupTerminated(t *testing.T) {
 		ClusterName: aws.String("c1"), NodegroupName: aws.String("ng2"),
 	}, testAccountID)
 	require.NoError(t, err)
+	_, err = f.svc.NewNodegroupDeletingReaper().Sweep(context.Background())
+	require.NoError(t, err)
+	f.svc.WaitNodegroupDeletes()
 	got = managedWorkers(t, f, "c1")
 	assert.Len(t, got, 1)
 	for _, id := range ng2.InstanceIDs {

spinifex/domains/iam/mock/mock.go

@@ -36,6 +36,9 @@ type SystemInstanceRoleEnsurer struct {
 	CreateRoleCalls               int
 	CreateInstanceProfileCalls    int
 	AddRoleToInstanceProfileCalls int
+	TagInstanceProfileCalls       int
+	RemoveRoleCalls               int
+	DeleteInstanceProfileCalls    int
 
 	// LastTrustDoc captures the AssumeRolePolicyDocument of the most recent
 	// CreateRole call.
@@ -54,6 +57,14 @@ type SystemInstanceRoleEnsurer struct {
 	// AddRoleToInstanceProfileErr, when set, is returned by
 	// AddRoleToInstanceProfile instead of attaching.
 	AddRoleToInstanceProfileErr error
+	// RemoveRoleFromInstanceProfileErr and DeleteInstanceProfileErr, when set,
+	// are returned by the matching call instead of acting.
+	RemoveRoleFromInstanceProfileErr error
+	DeleteInstanceProfileErr         error
+	// GetInstanceProfileErr, when set, is returned by GetInstanceProfile
+	// instead of reading, for exercising callers that must tell a failed read
+	// from a missing profile.
+	GetInstanceProfileErr error
 	// EmptyInstanceProfileARN makes CreateInstanceProfile return an empty
 	// ARN, for exercising callers that must reject it.
 	EmptyInstanceProfileARN bool
@@ -81,6 +92,7 @@ func copyRole(r *iam.Role) *iam.Role {
 func copyProfile(p *iam.InstanceProfile) *iam.InstanceProfile {
 	c := *p
 	c.Roles = append([]*iam.Role(nil), p.Roles...)
+	c.Tags = append([]*iam.Tag(nil), p.Tags...)
 	return &c
 }
 
@@ -130,6 +142,9 @@ func (e *SystemInstanceRoleEnsurer) GetInstanceProfile(_ string, in *iam.GetInst
 	e.mu.Lock()
 	defer e.mu.Unlock()
 
+	if e.GetInstanceProfileErr != nil {
+		return nil, e.GetInstanceProfileErr
+	}
 	p, ok := e.Profiles[aws.StringValue(in.InstanceProfileName)]
 	if !ok {
 		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile %s cannot be found.", aws.StringValue(in.InstanceProfileName))
@@ -157,6 +172,7 @@ func (e *SystemInstanceRoleEnsurer) CreateInstanceProfile(accountID string, in *
 	p := &iam.InstanceProfile{
 		InstanceProfileName: aws.String(name),
 		Arn:                 aws.String(arn),
+		Tags:                append([]*iam.Tag(nil), in.Tags...),
 	}
 	e.Profiles[name] = p
 	return &iam.CreateInstanceProfileOutput{InstanceProfile: copyProfile(p)}, nil
@@ -181,3 +197,101 @@ func (e *SystemInstanceRoleEnsurer) AddRoleToInstanceProfile(_ string, in *iam.A
 	p.Roles = []*iam.Role{{RoleName: in.RoleName}}
 	return &iam.AddRoleToInstanceProfileOutput{}, nil
 }
+
+func (e *SystemInstanceRoleEnsurer) TagInstanceProfile(_ string, in *iam.TagInstanceProfileInput) (*iam.TagInstanceProfileOutput, error) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+
+	e.TagInstanceProfileCalls++
+	p, ok := e.Profiles[aws.StringValue(in.InstanceProfileName)]
+	if !ok {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile %s cannot be found.", aws.StringValue(in.InstanceProfileName))
+	}
+	for _, t := range in.Tags {
+		replaced := false
+		for i, have := range p.Tags {
+			if aws.StringValue(have.Key) == aws.StringValue(t.Key) {
+				p.Tags[i] = t
+				replaced = true
+				break
+			}
+		}
+		if !replaced {
+			p.Tags = append(p.Tags, t)
+		}
+	}
+	return &iam.TagInstanceProfileOutput{}, nil
+}
+
+func (e *SystemInstanceRoleEnsurer) UntagInstanceProfile(_ string, in *iam.UntagInstanceProfileInput) (*iam.UntagInstanceProfileOutput, error) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+
+	p, ok := e.Profiles[aws.StringValue(in.InstanceProfileName)]
+	if !ok {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile %s cannot be found.", aws.StringValue(in.InstanceProfileName))
+	}
+	drop := map[string]bool{}
+	for _, k := range in.TagKeys {
+		drop[aws.StringValue(k)] = true
+	}
+	kept := p.Tags[:0:0]
+	for _, t := range p.Tags {
+		if !drop[aws.StringValue(t.Key)] {
+			kept = append(kept, t)
+		}
+	}
+	p.Tags = kept
+	return &iam.UntagInstanceProfileOutput{}, nil
+}
+
+func (e *SystemInstanceRoleEnsurer) ListInstanceProfileTags(_ string, in *iam.ListInstanceProfileTagsInput) (*iam.ListInstanceProfileTagsOutput, error) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+
+	p, ok := e.Profiles[aws.StringValue(in.InstanceProfileName)]
+	if !ok {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile %s cannot be found.", aws.StringValue(in.InstanceProfileName))
+	}
+	return &iam.ListInstanceProfileTagsOutput{Tags: append([]*iam.Tag(nil), p.Tags...)}, nil
+}
+
+func (e *SystemInstanceRoleEnsurer) RemoveRoleFromInstanceProfile(_ string, in *iam.RemoveRoleFromInstanceProfileInput) (*iam.RemoveRoleFromInstanceProfileOutput, error) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+
+	e.RemoveRoleCalls++
+	if e.RemoveRoleFromInstanceProfileErr != nil {
+		return nil, e.RemoveRoleFromInstanceProfileErr
+	}
+	p, ok := e.Profiles[aws.StringValue(in.InstanceProfileName)]
+	if !ok {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile %s cannot be found.", aws.StringValue(in.InstanceProfileName))
+	}
+	if len(p.Roles) == 0 || aws.StringValue(p.Roles[0].RoleName) != aws.StringValue(in.RoleName) {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "The role with name %s cannot be found.", aws.StringValue(in.RoleName))
+	}
+	p.Roles = nil
+	return &iam.RemoveRoleFromInstanceProfileOutput{}, nil
+}
+
+// DeleteInstanceProfile refuses while a role is attached, as IAM does.
+func (e *SystemInstanceRoleEnsurer) DeleteInstanceProfile(_ string, in *iam.DeleteInstanceProfileInput) (*iam.DeleteInstanceProfileOutput, error) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+
+	e.DeleteInstanceProfileCalls++
+	if e.DeleteInstanceProfileErr != nil {
+		return nil, e.DeleteInstanceProfileErr
+	}
+	name := aws.StringValue(in.InstanceProfileName)
+	p, ok := e.Profiles[name]
+	if !ok {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMNoSuchEntity, "Instance Profile %s cannot be found.", name)
+	}
+	if len(p.Roles) > 0 {
+		return nil, awserrors.Errorf(awserrors.ErrorIAMDeleteConflict, "Cannot delete entity, must remove roles from instance profile first.")
+	}
+	delete(e.Profiles, name)
+	return &iam.DeleteInstanceProfileOutput{}, nil
+}

spinifex/domains/iam/mock/mock_test.go

@@ -83,3 +83,112 @@ func TestPutRolePolicy_InjectedErrorStillRecordsCall(t *testing.T) {
 		t.Fatalf("RolePolicies[r1] = %q, want %q", e.RolePolicies["r1"], "doc")
 	}
 }
+
+func seedProfile(t *testing.T, e *mock.SystemInstanceRoleEnsurer, name string) {
+	t.Helper()
+	if _, err := e.CreateInstanceProfile("acct", &iam.CreateInstanceProfileInput{InstanceProfileName: aws.String(name)}); err != nil {
+		t.Fatalf("CreateInstanceProfile: %v", err)
+	}
+}
+
+func requireCode(t *testing.T, err error, want string) {
+	t.Helper()
+	if err == nil || awserrors.ValidErrorCodeFromError(err) != want {
+		t.Fatalf("err = %v, want %s", err, want)
+	}
+}
+
+func TestInstanceProfileTags_TagReplacesSameKeyUntagDropsListReturnsCopy(t *testing.T) {
+	e := mock.New()
+	seedProfile(t, e, "p1")
+	tag := func(k, v string) *iam.Tag { return &iam.Tag{Key: aws.String(k), Value: aws.String(v)} }
+	if _, err := e.TagInstanceProfile("acct", &iam.TagInstanceProfileInput{
+		InstanceProfileName: aws.String("p1"), Tags: []*iam.Tag{tag("a", "1"), tag("b", "2")},
+	}); err != nil {
+		t.Fatalf("TagInstanceProfile: %v", err)
+	}
+	if _, err := e.TagInstanceProfile("acct", &iam.TagInstanceProfileInput{
+		InstanceProfileName: aws.String("p1"), Tags: []*iam.Tag{tag("a", "9")},
+	}); err != nil {
+		t.Fatalf("re-tag: %v", err)
+	}
+	if _, err := e.UntagInstanceProfile("acct", &iam.UntagInstanceProfileInput{
+		InstanceProfileName: aws.String("p1"), TagKeys: aws.StringSlice([]string{"b"}),
+	}); err != nil {
+		t.Fatalf("UntagInstanceProfile: %v", err)
+	}
+	out, err := e.ListInstanceProfileTags("acct", &iam.ListInstanceProfileTagsInput{InstanceProfileName: aws.String("p1")})
+	if err != nil {
+		t.Fatalf("ListInstanceProfileTags: %v", err)
+	}
+	if len(out.Tags) != 1 || aws.StringValue(out.Tags[0].Key) != "a" || aws.StringValue(out.Tags[0].Value) != "9" {
+		t.Fatalf("tags = %v, want only a=9", out.Tags)
+	}
+}
+
+func TestInstanceProfileTags_MissingProfileIsNoSuchEntity(t *testing.T) {
+	e := mock.New()
+	name := aws.String("absent")
+	_, err := e.TagInstanceProfile("acct", &iam.TagInstanceProfileInput{InstanceProfileName: name})
+	requireCode(t, err, awserrors.ErrorIAMNoSuchEntity)
+	_, err = e.UntagInstanceProfile("acct", &iam.UntagInstanceProfileInput{InstanceProfileName: name})
+	requireCode(t, err, awserrors.ErrorIAMNoSuchEntity)
+	_, err = e.ListInstanceProfileTags("acct", &iam.ListInstanceProfileTagsInput{InstanceProfileName: name})
+	requireCode(t, err, awserrors.ErrorIAMNoSuchEntity)
+}
+
+func TestDeleteInstanceProfile_RefusesWhileRoleAttachedThenDeletes(t *testing.T) {
+	e := mock.New()
+	seedProfile(t, e, "p1")
+	if _, err := e.AddRoleToInstanceProfile("acct", &iam.AddRoleToInstanceProfileInput{
+		InstanceProfileName: aws.String("p1"), RoleName: aws.String("r1"),
+	}); err != nil {
+		t.Fatalf("AddRoleToInstanceProfile: %v", err)
+	}
+	del := &iam.DeleteInstanceProfileInput{InstanceProfileName: aws.String("p1")}
+	_, err := e.DeleteInstanceProfile("acct", del)
+	requireCode(t, err, awserrors.ErrorIAMDeleteConflict)
+	if _, ok := e.Profiles["p1"]; !ok {
+		t.Fatal("a refused delete must leave the profile in place")
+	}
+
+	_, err = e.RemoveRoleFromInstanceProfile("acct", &iam.RemoveRoleFromInstanceProfileInput{
+		InstanceProfileName: aws.String("p1"), RoleName: aws.String("other"),
+	})
+	requireCode(t, err, awserrors.ErrorIAMNoSuchEntity)
+	if _, err := e.RemoveRoleFromInstanceProfile("acct", &iam.RemoveRoleFromInstanceProfileInput{
+		InstanceProfileName: aws.String("p1"), RoleName: aws.String("r1"),
+	}); err != nil {
+		t.Fatalf("RemoveRoleFromInstanceProfile: %v", err)
+	}
+	if _, err := e.DeleteInstanceProfile("acct", del); err != nil {
+		t.Fatalf("DeleteInstanceProfile: %v", err)
+	}
+	if _, ok := e.Profiles["p1"]; ok {
+		t.Fatal("profile still present after delete")
+	}
+	_, err = e.DeleteInstanceProfile("acct", del)
+	requireCode(t, err, awserrors.ErrorIAMNoSuchEntity)
+}
+
+func TestInstanceProfileMutations_MissingProfileAndInjectedErrors(t *testing.T) {
+	e := mock.New()
+	name := aws.String("absent")
+	_, err := e.RemoveRoleFromInstanceProfile("acct", &iam.RemoveRoleFromInstanceProfileInput{InstanceProfileName: name, RoleName: aws.String("r")})
+	requireCode(t, err, awserrors.ErrorIAMNoSuchEntity)
+
+	boom := errors.New("boom")
+	e.RemoveRoleFromInstanceProfileErr, e.DeleteInstanceProfileErr, e.GetInstanceProfileErr = boom, boom, boom
+	if _, err := e.RemoveRoleFromInstanceProfile("acct", &iam.RemoveRoleFromInstanceProfileInput{InstanceProfileName: name, RoleName: aws.String("r")}); !errors.Is(err, boom) {
+		t.Fatalf("RemoveRole err = %v, want injected", err)
+	}
+	if _, err := e.DeleteInstanceProfile("acct", &iam.DeleteInstanceProfileInput{InstanceProfileName: name}); !errors.Is(err, boom) {
+		t.Fatalf("Delete err = %v, want injected", err)
+	}
+	if _, err := e.GetInstanceProfile("acct", &iam.GetInstanceProfileInput{InstanceProfileName: name}); !errors.Is(err, boom) {
+		t.Fatalf("Get err = %v, want injected", err)
+	}
+	if e.RemoveRoleCalls != 2 || e.DeleteInstanceProfileCalls != 1 {
+		t.Fatalf("calls = remove %d delete %d, want 2 and 1", e.RemoveRoleCalls, e.DeleteInstanceProfileCalls)
+	}
+}

spinifex/runtime/daemon/daemon.go

@@ -2562,6 +2562,7 @@ func (d *Daemon) startCluster() error {
 		if d.eksService != nil {
 			reapers = append(reapers, d.eksService.NewBillableReaper(d.nodeRunningVMs))
 			reapers = append(reapers, d.eksService.NewDeletingReaper())
+			reapers = append(reapers, d.eksService.NewNodegroupDeletingReaper())
 		}
 		// The RDS automated-backup retention sweep is the one reaper here that
 		// deletes customer data, which is why it rides this backstop rather than the

spinifex/runtime/daemon/eks_worker_launch.go

@@ -8,11 +8,15 @@ import (
 	"fmt"
 	"github.com/mulgadc/spinifex/spinifex/foundation/messaging/nats"
 	"log/slog"
+	"maps"
+	"slices"
+	"strings"
 	"time"
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/contracts/ec2/v1"
+	ec2instanceapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/instance"
 	ec2instance "github.com/mulgadc/spinifex/spinifex/domains/ec2/instance"
 	ekscluster "github.com/mulgadc/spinifex/spinifex/domains/eks/cluster"
 	"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
@@ -84,6 +88,55 @@ func (d *Daemon) TerminateWorkerInstances(ctx context.Context, instanceIDs []str
 	return nil
 }
 
+// LiveWorkerInstances returns the workers that are not yet proven gone: any
+// instance that is not terminated and not absent. Absence is only accepted from
+// a describe that heard every configured node; an unresolved describe is an
+// error, never an empty answer.
+func (d *Daemon) LiveWorkerInstances(ctx context.Context, instanceIDs []string, accountID string) ([]string, error) {
+	if d.natsConn == nil {
+		return nil, errors.New("eks worker: NATS connection not initialized")
+	}
+	var nodeIDs []string
+	if d.clusterConfig != nil {
+		nodeIDs = slices.Sorted(maps.Keys(d.clusterConfig.Nodes))
+	}
+	var live []string
+	for _, id := range instanceIDs {
+		if id == "" {
+			continue
+		}
+		out, err := ec2instanceapi.DescribeInstancesChecked(ctx, &ec2.DescribeInstancesInput{
+			InstanceIds: []*string{aws.String(id)},
+		}, d.natsConn, 0, nodeIDs, accountID)
+		if err != nil {
+			if strings.Contains(err.Error(), awserrors.ErrorInvalidInstanceIDNotFound) {
+				continue
+			}
+			return nil, fmt.Errorf("describe worker %s: %w", id, err)
+		}
+		if workerStillPresent(out) {
+			live = append(live, id)
+		}
+	}
+	return live, nil
+}
+
+// workerStillPresent reports whether the describe output holds an instance that
+// has not reached terminated.
+func workerStillPresent(out *ec2.DescribeInstancesOutput) bool {
+	for _, res := range out.Reservations {
+		for _, inst := range res.Instances {
+			if inst == nil {
+				continue
+			}
+			if inst.State == nil || aws.StringValue(inst.State.Name) != ec2.InstanceStateNameTerminated {
+				return true
+			}
+		}
+	}
+	return false
+}
+
 // terminateWorkerInstance routes a single worker terminate to its owning node.
 // StopInstance is set alongside TerminateInstance so the owner does not restart
 // it. A NoResponders reply means no daemon owns the instance: it is already

spinifex/runtime/daemon/eks_worker_live_test.go

@@ -0,0 +1,92 @@
+package daemon
+
+import (
+	"encoding/json"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestWorkerStillPresent(t *testing.T) {
+	t.Parallel()
+	inst := func(state string) *ec2.DescribeInstancesOutput {
+		return &ec2.DescribeInstancesOutput{Reservations: []*ec2.Reservation{{Instances: []*ec2.Instance{
+			{State: &ec2.InstanceState{Name: aws.String(state)}},
+		}}}}
+	}
+	assert.False(t, workerStillPresent(&ec2.DescribeInstancesOutput{}))
+	assert.False(t, workerStillPresent(inst(ec2.InstanceStateNameTerminated)))
+	assert.True(t, workerStillPresent(inst(ec2.InstanceStateNameShuttingDown)))
+	assert.True(t, workerStillPresent(inst(ec2.InstanceStateNameRunning)))
+	assert.True(t, workerStillPresent(&ec2.DescribeInstancesOutput{Reservations: []*ec2.Reservation{{Instances: []*ec2.Instance{{}}}}}))
+}
+
+// answerBucket serves one instance-bucket describe subject, answering only
+// requests that name id so tests sharing the NATS server do not see each other.
+func answerBucket(t *testing.T, nc *nats.Conn, subject, id, state string) {
+	t.Helper()
+	sub, err := nc.Subscribe(subject, func(msg *nats.Msg) {
+		var in ec2.DescribeInstancesInput
+		if json.Unmarshal(msg.Data, &in) != nil || len(in.InstanceIds) != 1 || aws.StringValue(in.InstanceIds[0]) != id {
+			return
+		}
+		out := ec2.DescribeInstancesOutput{Reservations: []*ec2.Reservation{{
+			ReservationId: aws.String("r-" + id),
+			Instances:     []*ec2.Instance{{InstanceId: aws.String(id), State: &ec2.InstanceState{Name: aws.String(state)}}},
+		}}}
+		data, _ := json.Marshal(out)
+		_ = msg.Respond(data)
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+func TestLiveWorkerInstances_NeedsNATS(t *testing.T) {
+	t.Parallel()
+	live, err := (&Daemon{}).LiveWorkerInstances(t.Context(), []string{"i-x"}, "111122223333")
+	require.Error(t, err)
+	assert.Nil(t, live)
+}
+
+func TestLiveWorkerInstances_BlankIDsAreSkipped(t *testing.T) {
+	t.Parallel()
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+
+	live, err := (&Daemon{natsConn: nc}).LiveWorkerInstances(t.Context(), []string{"", ""}, "111122223333")
+
+	require.NoError(t, err)
+	assert.Empty(t, live)
+}
+
+func TestLiveWorkerInstances_ReportsOnlyWorkersNotYetTerminated(t *testing.T) {
+	t.Parallel()
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+	answerBucket(t, nc, "ec2.DescribeStoppedInstances", "i-live-stopped", ec2.InstanceStateNameStopped)
+	answerBucket(t, nc, "ec2.DescribeTerminatedInstances", "i-live-done", ec2.InstanceStateNameTerminated)
+
+	live, err := (&Daemon{natsConn: nc}).LiveWorkerInstances(t.Context(),
+		[]string{"i-live-stopped", "i-live-done"}, "111122223333")
+
+	require.NoError(t, err)
+	assert.Equal(t, []string{"i-live-stopped"}, live, "a stopped worker is not gone; a terminated one is")
+}
+
+func TestLiveWorkerInstances_UnprovableAbsenceIsAnErrorNotAnEmptyAnswer(t *testing.T) {
+	t.Parallel()
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+
+	live, err := (&Daemon{natsConn: nc}).LiveWorkerInstances(t.Context(), []string{"i-nobody-answers"}, "111122223333")
+
+	require.ErrorContains(t, err, "describe worker i-nobody-answers")
+	assert.Nil(t, live, "no node answered, so the worker cannot be called gone")
+}

tests/e2e/eks/eks_test.go

@@ -242,6 +242,12 @@ func TestEKS(t *testing.T) {
 		runEBSCSIVolume(t, c, env, artifacts, fx)
 	})
 
+	parent := t
+	t.Run("NodegroupDelete", func(t *testing.T) {
+		requireClusterReady(t, fx)
+		runNodegroupDelete(t, parent, c, fx)
+	})
+
 	t.Run("DeleteCluster", func(t *testing.T) {
 		requireClusterReady(t, fx)
 		clusterSG := clusterSecurityGroupID(t, fx)
@@ -256,6 +262,10 @@ func TestEKS(t *testing.T) {
 		waitClusterGoneWhileDeleting(t, c, fx.ClusterName)
 		fx.Deleted = true
 		assertBystanderRulesAfterDelete(t, c, bystander, clusterSG)
+		if fx.BystanderProfile != "" {
+			_, perr := c.IAM.GetInstanceProfile(&iam.GetInstanceProfileInput{InstanceProfileName: aws.String(fx.BystanderProfile)})
+			require.NoError(t, perr, "a customer-made instance profile must survive DeleteCluster")
+		}
 
 		_, err = c.EKS.DeleteCluster(&eks.DeleteClusterInput{Name: aws.String(fx.ClusterName)})
 		var aerr awserr.Error
@@ -452,12 +462,9 @@ func createWebIdentityRole(t *testing.T, c *harness.AWSClient, fx *clusterFixtur
 // EC2-service trust policy — real EKS has the same prerequisite: the customer
 // creates the node instance role before CreateNodegroup). Both runIRSAPod and
 // runEBSCSIVolume call this independently rather than one leaking the role for
-// the other to pick up by naming convention: CreateNodegroup's worker launch
-// attaches the role to a same-named instance profile (ensureNodeInstanceProfile
-// in nodegroup.go), so DeleteRole fails while that attachment stands. The
-// cleanup here detaches it first and asserts the delete instead of discarding
-// its error, so a real leak fails the test instead of leaving the role behind
-// for the next subtest to silently depend on.
+// the other to pick up by naming convention. The cleanup asserts the delete
+// with no hand step, so a nodegroup that leaves its instance profile attached
+// fails the test instead of leaving the role behind.
 func ensureNodeRole(t *testing.T, c *harness.AWSClient, fx *clusterFixture) (roleArn, roleName string) {
 	t.Helper()
 	roleName = fx.ClusterName + "-node"
@@ -470,13 +477,6 @@ func ensureNodeRole(t *testing.T, c *harness.AWSClient, fx *clusterFixture) (rol
 	require.NoError(t, err, "create-node-role")
 	roleArn = aws.StringValue(out.Role.Arn)
 	t.Cleanup(func() {
-		_, err := c.IAM.RemoveRoleFromInstanceProfile(&iam.RemoveRoleFromInstanceProfileInput{
-			InstanceProfileName: aws.String(roleName),
-			RoleName:            aws.String(roleName),
-		})
-		if err != nil && !harness.ErrorCodeIs(err, iam.ErrCodeNoSuchEntityException) {
-			t.Errorf("remove-role-from-instance-profile %s: %v", roleName, err)
-		}
 		if _, err := c.IAM.DeleteRole(&iam.DeleteRoleInput{RoleName: aws.String(roleName)}); err != nil {
 			t.Errorf("delete-node-role %s: %v", roleName, err)
 		}
@@ -1055,6 +1055,9 @@ type clusterFixture struct {
 	Egress      *harness.WorkerEgress
 	Cluster     *eks.Cluster
 	Deleted     bool
+	// BystanderProfile is a customer-made instance profile that must outlive
+	// the nodegroup and cluster deletes.
+	BystanderProfile string
 }
 
 func setupClusterFixture(t *testing.T, c *harness.AWSClient, env *harness.Env, artifacts string) *clusterFixture {

tests/e2e/eks/nodegroup_profile_test.go

@@ -0,0 +1,104 @@
+//go:build e2e
+
+package eks
+
+import (
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/eks"
+	"github.com/aws/aws-sdk-go/service/iam"
+	"github.com/mulgadc/spinifex/tests/e2e/harness"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// runNodegroupDelete proves DeleteNodegroup is asynchronous and leaves nothing
+// behind that blocks the customer: the group answers DELETING first, then
+// ResourceNotFound, after which the node role deletes with no hand step and an
+// instance profile the customer made is untouched. The bystander profile is
+// cleaned up on the parent test so it outlives this subtest and DeleteCluster
+// can check it too.
+func runNodegroupDelete(t, parent *testing.T, c *harness.AWSClient, fx *clusterFixture) {
+	const trustPolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}`
+	roleName := fx.ClusterName + "-ngdel-node"
+	roleOut, err := c.IAM.CreateRole(&iam.CreateRoleInput{
+		RoleName:                 aws.String(roleName),
+		AssumeRolePolicyDocument: aws.String(trustPolicy),
+	})
+	require.NoError(t, err, "create-node-role")
+	roleDeleted := false
+	t.Cleanup(func() {
+		if !roleDeleted {
+			_, _ = c.IAM.DeleteRole(&iam.DeleteRoleInput{RoleName: aws.String(roleName)})
+		}
+	})
+
+	fx.BystanderProfile = fx.ClusterName + "-bystander"
+	_, err = c.IAM.CreateInstanceProfile(&iam.CreateInstanceProfileInput{InstanceProfileName: aws.String(fx.BystanderProfile)})
+	require.NoError(t, err, "create-bystander-instance-profile")
+	parent.Cleanup(func() {
+		_, _ = c.IAM.DeleteInstanceProfile(&iam.DeleteInstanceProfileInput{InstanceProfileName: aws.String(fx.BystanderProfile)})
+	})
+
+	const nodegroup = "ngdel-e2e-ng"
+	harness.Phase(t, "Creating worker nodegroup %s", nodegroup)
+	_, err = c.EKS.CreateNodegroup(&eks.CreateNodegroupInput{
+		ClusterName:   aws.String(fx.ClusterName),
+		NodegroupName: aws.String(nodegroup),
+		Subnets:       aws.StringSlice([]string{fx.SubnetID}),
+		NodeRole:      roleOut.Role.Arn,
+		ScalingConfig: &eks.NodegroupScalingConfig{MinSize: aws.Int64(1), MaxSize: aws.Int64(1), DesiredSize: aws.Int64(1)},
+	})
+	require.NoError(t, err, "create-nodegroup")
+	t.Cleanup(func() { harness.DeleteEKSNodegroupAndWait(t, c, fx.ClusterName, nodegroup) })
+	describe := func() (*eks.Nodegroup, error) {
+		out, derr := c.EKS.DescribeNodegroup(&eks.DescribeNodegroupInput{
+			ClusterName: aws.String(fx.ClusterName), NodegroupName: aws.String(nodegroup),
+		})
+		if derr != nil {
+			return nil, derr
+		}
+		return out.Nodegroup, nil
+	}
+	harness.EventuallyErr(t, func() error {
+		ng, derr := describe()
+		if derr != nil {
+			return fmt.Errorf("describe-nodegroup: %w", derr)
+		}
+		if s := aws.StringValue(ng.Status); s != eks.NodegroupStatusActive {
+			return fmt.Errorf("nodegroup status %q, want ACTIVE", s)
+		}
+		return nil
+	}, 8*time.Minute, 10*time.Second)
+
+	started := time.Now()
+	out, err := c.EKS.DeleteNodegroup(&eks.DeleteNodegroupInput{
+		ClusterName: aws.String(fx.ClusterName), NodegroupName: aws.String(nodegroup),
+	})
+	require.NoError(t, err, "delete-nodegroup")
+	assert.Less(t, time.Since(started), 5*time.Second, "DeleteNodegroup must return without waiting for the workers")
+	assert.Equal(t, eks.NodegroupStatusDeleting, aws.StringValue(out.Nodegroup.Status))
+	ng, err := describe()
+	require.NoError(t, err, "the group must still be described right after DeleteNodegroup returns")
+	assert.Equal(t, eks.NodegroupStatusDeleting, aws.StringValue(ng.Status))
+
+	harness.EventuallyErr(t, func() error {
+		_, derr := describe()
+		if derr == nil {
+			return fmt.Errorf("nodegroup %s still exists", nodegroup)
+		}
+		if harness.ErrorCodeIs(derr, eks.ErrCodeResourceNotFoundException) {
+			return nil
+		}
+		return fmt.Errorf("describe-nodegroup: %w", derr)
+	}, 5*time.Minute, 3*time.Second)
+
+	_, err = c.IAM.DeleteRole(&iam.DeleteRoleInput{RoleName: aws.String(roleName)})
+	require.NoError(t, err, "delete-role must succeed once the nodegroup is gone, with no instance profile to detach by hand")
+	roleDeleted = true
+	_, err = c.IAM.GetInstanceProfile(&iam.GetInstanceProfileInput{InstanceProfileName: aws.String(fx.BystanderProfile)})
+	require.NoError(t, err, "an instance profile the customer made must survive the nodegroup delete")
+}