# EKS managed add-on: lifecycle contract

Present contract under ADR-0003 S1, recorded before the resource moves.
It describes the code as it stands, so it claims no conformance; the gaps below are observed behaviour, not targets.
The resource-specific target is proposed requirement Q-146 and the recorded conflict is C-82 (`docs/specs/requirements/managed-services.md` in the mulga repo); both are open proposals pending review, not accepted decisions, and this record does not close either.
Paths are relative to `spinifex/` unless they start with `tests/` or `scripts/`, which are relative to the spinifex repo root.

## Scope

An add-on has an asynchronous realization: the control-plane VM renders a baked bundle into the K3s auto-deploy directory and reports delivery back.
The host side owns a desired descriptor (the record plus a staged manifest sub-key); the guest side (`scripts/images/eks-node/mulga-eks-addon-sync.sh`) pulls staged manifests through the gateway and publishes per-add-on delivery reports, which the cluster reconciler folds back into the record.
There is no generation, so the record mixes desired fields with the latest observation.

## Contract

| Item | Present behaviour | Evidence |
|---|---|---|
| Identity | Cluster name plus add-on name; one add-on per name per cluster. ARN `arn:aws:eks:{region}:{account}:addon/{cluster}/{addon}` (`foundation/aws/arn/eks.go` `FormatEKSAddon`), deterministic by name, with no per-incarnation suffix. The gateway authorizer derives the same ARN (`gateway/eks/authz.go` `addonARN`). | `handlers/eks/addon_lifecycle_test.go` `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `gateway/eks/authz_test.go` `TestResourceARNsFidelity` |
| Scope | Account and Region, within one cluster: keys live under `clusters/{cluster}/addons/` in the `eks-account-{account}` bucket. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster` |
| Durable record | `clusters/{cluster}/addons/{addon}` holds `Record` (`domains/eks/addon/record.go`): `addonName`, `addonVersion`, `status`, `serviceAccountRoleArn`, `configurationValues`, `health`, `arn`, `tags`, `createdAt`, `modifiedAt` (the optional ones `omitempty`). `clusters/{cluster}/addons/{addon}/manifest` holds `Manifest` (`domains/eks/addon/record.go`): `addonName`, `addonVersion`, `serviceAccountRoleArn`, `configurationValues`. Listing skips the manifest sub-key by segment count. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestAddonRecord_DecodesHandWrittenJSON`, `domains/eks/addon/store_test.go` `TestRecord_RoundTrip`, `TestList_SkipsManifestSubKeys` |
| Writers | Record: the owner in `domains/eks/addon/owner.go`, reached from the `CreateAddon`, `UpdateAddon` and `DeleteAddon` handlers, the cluster reconciler's report relay and the GPU node group launch: `Owner.Create` (unconditional put), `Owner.Update` and its failure marking (revision CAS), `Owner.Delete` (get then delete), `Owner.ApplyReport` (revision CAS), `Owner.EnsureGPUDevicePlugin` through `Owner.Create`; and the cluster prefix purge (`DeleteClusterPrefix`). Manifest: `StagingInstaller.Install` (unconditional put on create and update), `StagingInstaller.Uninstall` (`domains/eks/addon/delivery.go`), `Owner.Delete` and the prefix purge. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster`, `domains/eks/addon/store_test.go` `TestCasUpdate`, `TestDeleteRecord_RemovesRecordAndManifest` |
| Readers | `ListAddons`, `DescribeAddon`, `CreateAddon` (existence check) and the reconciler read the record. `ListStagedAddonManifests` reads every `/manifest` key for the cluster (`addon.ListManifests`); it is served on `eks.ListStagedAddonManifests` (`daemon/daemon.go`) and reached only through the gateway's internal-addons route. | `handlers/eks/addons_test.go` `TestAddon_DescribeListDelete`, `TestListStagedAddonManifests` |
| Catalogue | Static in-binary catalogue (`domains/eks/addon/catalog.go`): `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`, and hidden-but-creatable `nvidia-device-plugin` and `spinifex-noop`; one version each, the first being the default. Create and update reject unknown names and versions with `InvalidParameterValue`. | `domains/eks/addon/catalog_test.go`, `handlers/eks/addons_test.go` `TestDescribeAddonVersions_ReturnsCatalog`, `TestCreateAddon_UnknownAddonRejected`, `TestCreateAddon_UnknownVersionRejected`, `TestUpdateAddon_UnknownVersionRejected` |
| Desired state | Version, `serviceAccountRoleArn`, `configurationValues` and create-time tags. `resolveConflicts`, `podIdentityAssociations` and `clientRequestToken` are accepted and dropped; `preserve` on delete is ignored. `configurationValues` reaches the guest but the agent discards it (`mulga-eks-addon-sync.sh` `sync_once`), so it has no effect. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest` |
| Observed state | `status` and `health` are overwritten by guest reports on the same record as the desired fields. `health` is the latest failure message, projected as one `AddonIssue` with only `message`. Statuses held: `CREATING`, `ACTIVE`, `UPDATING`, `DEGRADED`, `CREATE_FAILED`; `DELETING` is defined but never stored. | `TestAddonRecord_DecodesHandWrittenJSON`, `handlers/eks/addon_status_reconciler_test.go` `TestApplyAddonStatusReport` |
| Generation | None. Reports carry `version` and `ts`, and neither is read: any report for the add-on name applies to whatever is currently desired. | `TestAddonStatusReport_MismatchedVersionIsAccepted` |
| Create | Validate against the catalogue, require the cluster meta (`acctKVForCluster`), get then put a `CREATING` record, then stage the manifest. Returns `CREATING`. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `handlers/eks/addons_test.go` `TestCreateAddon_CreatesStagingAndDefaultsVersion`, `TestStagingInstaller_StagesManifest`, `TestCreateAddon_UnknownClusterIsNotFound` |
| Update | Validate the version, CAS the new version, config and role onto the record with status `UPDATING`, re-stage the manifest, and return an `Update` with status `Successful`, type `AddonUpdate` and `id` equal to the add-on ARN (the same for every update). Allowed from any stored status. | `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `handlers/eks/addons_test.go` `TestUpdateAddon_ChangesVersionAndReinstalls`, `TestUpdateAddon_MissingIsNotFound` |
| Readiness | `ready` maps any non-`ACTIVE` status to `ACTIVE` and clears `health`; `applied` and unknown phases change nothing. The agent reports `ready` when K3s records the add-on's `gvks` annotation, or after a 90 s grace once the bundle's workloads are rolled out; the check is per rendered file, not per version. | `domains/eks/addon/owner_test.go` `TestNextStatus`, `TestOwnerApplyReport`, `handlers/eks/addon_status_reconciler_test.go` `TestApplyAddonStatusReport`, `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` |
| Failure | Staging failure on create or update marks the record `CREATE_FAILED` with the error as `health` and returns the raw error. A `failed` report maps `ACTIVE` to `DEGRADED`, keeps `DEGRADED` and `CREATE_FAILED`, and maps anything else (including `UPDATING`) to `CREATE_FAILED`. Neither failure state is terminal: a later `ready` report returns the add-on to `ACTIVE`. | `TestCreateAddon_InstallerFailureLeavesDurableCreateFailedRecord`, `TestUpdateAddon_InstallerFailureMarksCreateFailed`, `TestAddonStatusReport_MismatchedVersionIsAccepted`, `handlers/eks/addons_test.go` `TestCreateAddon_InstallerFailureMarksCreateFailed` |
| Delete | Read the record, build a `DELETING` response from it, delete the manifest, then delete the record and manifest again (`Owner.Delete`). Both keys are gone when the call returns. The guest later notices the missing manifest, removes the rendered file and runs `kubectl delete --wait=false`; it sends no report, so nothing observes removal. | `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `TestListStagedAddonManifests`, `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` |
| Guest fetch | `GET /clusters/{cluster}/internal-addons/{accountId}` (`gateway/eks.go`, `ListInternalAddons`) is admitted only for a control-plane agent serving that cluster (`AuthorizeInternal`), relays to `eks.ListStagedAddonManifests`, and wraps the result as `contracts/eks/v1` `InternalAddonsResponse`. The body is written by the SDK REST-JSON marshaller, which ignores the `json` tags: it is `{"Addons":[{"AddonName","AddonVersion","ServiceAccountRoleArn","ConfigurationValues"}]}` with empty strings present, sorted by add-on name. | `gateway/eks_addon_wire_test.go` `TestEKSRequest_InternalAddonsWireBody`, `contracts/eks/v1` `TestInternalAddonsResponse_RESTJSONBody`, `gateway/eks_authz_test.go` `TestEKSRequest_InternalRoutesAdmitTheCPAgentPrincipal`, `gateway/eks/internal_authz_test.go` `TestAuthorizeInternal_RejectsClusterTheCallerDoesNotServe` |
| TSV conversion | `eks-gateway-fetch -resource addons` (`agents/eks/gatewayfetch`) decodes the body into its own copy of the shape (matching the Go field names only because `encoding/json` is case-insensitive) and writes `name\tversion\troleArn\tbase64(config)` per add-on; an empty set writes nothing. It retries the GET up to 30 times at 5 s. | `agents/eks/gatewayfetch/addons_wire_test.go` `TestEmitAddonsTSV_GatewayWireBody`, `agents/eks/gatewayfetch/emit_test.go` `TestEmitAddonsTSV`, `TestEmitAddonsTSV_Empty`, `agents/eks/gatewayfetch/fetch_test.go` `TestRun_RendersAddons`, `TestRun_FailsAfterAttemptBudget` |
| Guest render and report | Every 30 s (`ADDON_SYNC_INTERVAL`) the agent renders each staged add-on, substituting the role ARN into web-identity env when set, then publishes `{"addon","version","phase","message","ts"}` through `eks-gateway-publish -channel addon`. The gateway relays the raw payload to core NATS `eks.addon.{account}.{cluster}.status` (`gateway/eks/internal_publish.go`). A fetch failure keeps the current render. | `gateway/eks/internal_publish_test.go` `TestPublishInternal_RelaysToSubjects`, `agents/eks/gatewaypublish/publish_test.go` `TestRun_PublishesWrappedPayload`, `TestRun_FailsAfterAttemptBudget` |
| Retry meaning | Reports are level-triggered: each tick re-sends the current phase for every staged add-on, so a lost report is repeated on the next tick. `eks-gateway-publish` retries the POST up to 30 times at 5 s and the agent ignores its failure. Core NATS keeps nothing: a report published while no reconciler holds the cluster's lease, or after the reconciler stops on a terminal cluster status, is dropped. | none |
| Report ordering | The reconciler subscribes plainly (no queue group) on the lease holder and applies reports in arrival order through a revision CAS. Nothing orders a report against a later update: a report produced for the previous version settles the new `UPDATING` record, and a stale `failed` turns it `CREATE_FAILED`. A report for a deleted add-on is a no-op. | `TestAddonStatusReport_MismatchedVersionIsAccepted`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest` |
| Cross-resource dependency | A GPU node group's launch calls `stageGPUDeviceAddon`, which checks the cluster exists and calls `Owner.EnsureGPUDevicePlugin` for `nvidia-device-plugin`; an existing record is a no-op and every other error is logged only. Deleting the node group never unstages it, and an existing record in any status, including `CREATE_FAILED`, blocks re-staging. | `handlers/eks/nodegroup_test.go` `TestCreateNodegroup_GPUNodegroupStagesDevicePluginAddon`, `domains/eks/addon/owner_test.go` `TestOwnerEnsureGPUDevicePlugin_StagesDefaultVersionOnceAndKeepsAnyExistingRecord`, `TestCreateNodegroup_NonGPUNodegroupDoesNotStageDevicePluginAddon`, `tests/e2e/gpu/eks_gpu_test.go` `TestEKSGPUPodExposure` |
| Idempotency | `clientRequestToken` is ignored. A duplicate create, including an SDK retry with the same token, returns `ResourceInUseException`. A repeated update re-stages and returns another `Successful` with the same id. A repeated delete returns `ResourceNotFoundException`. | `TestCreateAddon_RepeatedClientRequestTokenIsResourceInUse`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `handlers/eks/addons_test.go` `TestCreateAddon_CreatesStagingAndDefaultsVersion` |
| Absent resource | An unknown cluster returns `ResourceNotFoundException` for every action, including the internal manifest listing. A missing add-on returns `ResourceNotFoundException` from describe, update and delete. | `handlers/eks/addons_test.go` `TestCreateAddon_UnknownClusterIsNotFound`, `TestDescribeDeleteAddon_MissingIsNotFound`, `TestUpdateAddon_MissingIsNotFound`, `TestListStagedAddonManifests` |
| Dependencies | Only the cluster meta's existence is checked; cluster status is not read, so add-ons can be created on a `DELETING` cluster. | `TestAddonActionsIgnoreClusterLifecycleStatus` |
| Authorization | Gateway policy uses the add-on ARN above. `serviceAccountRoleArn` requires `iam:PassRole` on `CreateAddon` and `UpdateAddon` (`gateway/eks/passrole.go`), the same check `CreateCluster`/`CreateNodegroup` apply, including that the role exists. Tag actions on an add-on ARN return `NotImplemented`. | `gateway/eks_addon_wire_test.go` `TestPassedRoleARNs_AddonServiceAccountRoleIsChecked`, `gateway/eks_passrole_test.go` `TestEKSCreateAddon_PassRole`, `TestEKSUpdateAddon_PassRole`, `gateway/eks/authz_test.go` `TestResourceARNsFidelity` |
| Cluster deletion | Add-on records and manifests are erased only by the cluster prefix purge, after the cluster's infrastructure teardown succeeds; there is no per-add-on delete. Reclaiming a failed cluster keeps the meta and does not purge, so the recreated cluster inherits the add-ons. | `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster`, `handlers/eks/cluster_state_test.go` `TestDeleteClusterPrefix_SweepsEveryKey` |
| Restart and interruption | Create interrupted between the record put and staging leaves a `CREATING` record with no manifest that nothing re-stages and that blocks a retry. Delete interrupted after unstaging leaves a record in its previous status while the guest removes the software. An update interrupted after the CAS leaves `UPDATING` with the previous manifest. A daemon restart loses nothing durable, but reports sent while no reconciler runs are lost until the next tick. | none |
| End-to-end | `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` (create to `ACTIVE`, objects present, delete removes the namespace) and `tests/e2e/gpu/eks_gpu_test.go` `TestEKSGPUPodExposure`; `tests/integration/cross_tenant_test.go` `TestCrossTenant_EveryScopedActionIsAccountedFor`. No E2E for update, failure, configuration or the role. No Terraform apply/destroy evidence for `aws_eks_addon`. | |

## Gaps

1. Create is get-then-put (`domains/eks/addon/owner.go` `Owner.Create`), so concurrent creates of one add-on both succeed and the later write wins; internal NATS redelivery of a create returns `ResourceInUseException`.
2. No client-token support: `clientRequestToken` is dropped on create and update, so an SDK retry of a successful create fails instead of returning the add-on as AWS does.
3. Delivery reports ignore their supplied `version` (and `ts`), so a stale or mismatched report settles the current desired state; there is no generation or incarnation fence.
4. `UpdateAddon` returns `Successful` while the add-on remains `UPDATING`, and the update `id` is the add-on ARN. AWS returns an in-progress update with its own id, tracked through `DescribeUpdate`; not verified against live AWS.
5. `DELETING` exists only in the delete response; the record and manifest are erased before any guest removal, so `DescribeAddon` returns not found immediately.
6. No observed deletion acknowledgement: the agent removes objects with `--wait=false` and reports nothing.
7. `configurationValues` is stored, staged and transported but discarded by the agent; `resolveConflicts`, `preserve` and `podIdentityAssociations` are accepted and never used.
8. Fixed: `serviceAccountRoleArn` now requires `iam:PassRole` on `CreateAddon` and `UpdateAddon`, matching `CreateCluster`/`CreateNodegroup` (`gateway/eks/passrole.go`).
9. AWS `UPDATE_FAILED` and `DELETE_FAILED` are not represented: a failed update becomes `CREATE_FAILED` and a failed delete leaves the prior status.
10. The add-on ARN lacks the per-incarnation suffix AWS appends, so a recreated add-on reuses its predecessor's ARN. This is a compatibility gap; extraction must keep the current ARN.
11. The guest/controller wire shape is versioned as `contracts/eks/v1` (subject, delivery phases, status report, staged manifest, internal-addons body), used directly by the handler, gateway and `agents/eks/gatewayfetch`. The add-on owner keeps its own `Manifest` and `Report`, and the shell agent still builds the report with `printf`. The HTTP body uses Go field names because the REST-JSON marshaller ignores the `json` tags, and the agent decodes it only through case-insensitive matching; the contract pins that form rather than changing it.
12. Reports travel over core NATS to a lease-held subscription, so they are lost whenever no reconciler runs for the cluster; delivery relies on the agent re-sending every tick.
13. The record mixes desired fields with the latest observation (`status`, `health`), and `ready` lifts `CREATE_FAILED` back to `ACTIVE`.
14. GPU auto-staging crosses from the node group into the add-on through `Owner.EnsureGPUDevicePlugin`, swallows every error, and is never undone.
15. `ListAddons` ignores `maxResults` and `nextToken`; tag actions on add-on ARNs return `NotImplemented`.
16. No ADR-0003 S5 failure or recovery evidence.