spinifex/docs/package-boundary/eks-addon-lifecycle.md

EKS managed add-on: lifecycle contract

Present contract under ADR-0003 S1, recorded before the resource moves. It describes the code as it stands, so it claims no conformance; the gaps below are observed behaviour, not targets. The resource-specific target is proposed requirement Q-146 and the recorded conflict is C-82 (docs/specs/requirements/managed-services.md in the mulga repo); both are open proposals pending review, not accepted decisions, and this record does not close either. Paths are relative to spinifex/ unless they start with tests/ or scripts/, which are relative to the spinifex repo root.

Scope

An add-on has an asynchronous realization: the control-plane VM renders a baked bundle into the K3s auto-deploy directory and reports delivery back. The host side owns a desired descriptor (the record plus a staged manifest sub-key); the guest side (scripts/images/eks-node/mulga-eks-addon-sync.sh) pulls staged manifests through the gateway and publishes per-add-on delivery reports, which the cluster reconciler folds back into the record. There is no generation, so the record mixes desired fields with the latest observation.

Contract

Item	Present behaviour	Evidence
Identity	Cluster name plus add-on name; one add-on per name per cluster. ARN arn:aws:eks:{region}:{account}:addon/{cluster}/{addon} (foundation/aws/arn/eks.go FormatEKSAddon), deterministic by name, with no per-incarnation suffix. The gateway authorizer derives the same ARN (gateway/eks/authz.go addonARN).	handlers/eks/addon_lifecycle_test.go TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames, gateway/eks/authz_test.go TestResourceARNsFidelity
Scope	Account and Region, within one cluster: keys live under clusters/{cluster}/addons/ in the eks-account-{account} bucket.	TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames, TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster
Durable record	clusters/{cluster}/addons/{addon} holds Record (domains/eks/addon/record.go): addonName, addonVersion, status, serviceAccountRoleArn, configurationValues, health, arn, tags, createdAt, modifiedAt (the optional ones omitempty). clusters/{cluster}/addons/{addon}/manifest holds Manifest (domains/eks/addon/record.go): addonName, addonVersion, serviceAccountRoleArn, configurationValues. Listing skips the manifest sub-key by segment count.	TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames, TestAddonRecord_DecodesHandWrittenJSON, domains/eks/addon/store_test.go TestRecord_RoundTrip, TestList_SkipsManifestSubKeys
Writers	Record: the owner in domains/eks/addon/owner.go, reached from the CreateAddon, UpdateAddon and DeleteAddon handlers, the cluster reconciler's report relay and the GPU node group launch: Owner.Create (unconditional put), Owner.Update and its failure marking (revision CAS), Owner.Delete (get then delete), Owner.ApplyReport (revision CAS), Owner.EnsureGPUDevicePlugin through Owner.Create; and the cluster prefix purge (DeleteClusterPrefix). Manifest: StagingInstaller.Install (unconditional put on create and update), StagingInstaller.Uninstall (domains/eks/addon/delivery.go), Owner.Delete and the prefix purge.	TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames, TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating, TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest, TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster, domains/eks/addon/store_test.go TestCasUpdate, TestDeleteRecord_RemovesRecordAndManifest
Readers	ListAddons, DescribeAddon, CreateAddon (existence check) and the reconciler read the record. ListStagedAddonManifests reads every /manifest key for the cluster (addon.ListManifests); it is served on eks.ListStagedAddonManifests (daemon/daemon.go) and reached only through the gateway's internal-addons route.	handlers/eks/addons_test.go TestAddon_DescribeListDelete, TestListStagedAddonManifests
Catalogue	Static in-binary catalogue (domains/eks/addon/catalog.go): aws-load-balancer-controller, argocd, aws-ebs-csi-driver, and hidden-but-creatable nvidia-device-plugin and spinifex-noop; one version each, the first being the default. Create and update reject unknown names and versions with InvalidParameterValue.	domains/eks/addon/catalog_test.go, handlers/eks/addons_test.go TestDescribeAddonVersions_ReturnsCatalog, TestCreateAddon_UnknownAddonRejected, TestCreateAddon_UnknownVersionRejected, TestUpdateAddon_UnknownVersionRejected
Desired state	Version, serviceAccountRoleArn, configurationValues and create-time tags. resolveConflicts, podIdentityAssociations and clientRequestToken are accepted and dropped; preserve on delete is ignored. configurationValues reaches the guest but the agent discards it (mulga-eks-addon-sync.sh sync_once), so it has no effect.	TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames, TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest
Observed state	status and health are overwritten by guest reports on the same record as the desired fields. health is the latest failure message, projected as one AddonIssue with only message. Statuses held: CREATING, ACTIVE, UPDATING, DEGRADED, CREATE_FAILED; DELETING is defined but never stored.	TestAddonRecord_DecodesHandWrittenJSON, handlers/eks/addon_status_reconciler_test.go TestApplyAddonStatusReport
Generation	None. Reports carry version and ts, and neither is read: any report for the add-on name applies to whatever is currently desired.	TestAddonStatusReport_MismatchedVersionIsAccepted
Create	Validate against the catalogue, require the cluster meta (acctKVForCluster), get then put a CREATING record, then stage the manifest. Returns CREATING.	TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames, handlers/eks/addons_test.go TestCreateAddon_CreatesStagingAndDefaultsVersion, TestStagingInstaller_StagesManifest, TestCreateAddon_UnknownClusterIsNotFound
Update	Validate the version, CAS the new version, config and role onto the record with status UPDATING, re-stage the manifest, and return an Update with status Successful, type AddonUpdate and id equal to the add-on ARN (the same for every update). Allowed from any stored status.	TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating, handlers/eks/addons_test.go TestUpdateAddon_ChangesVersionAndReinstalls, TestUpdateAddon_MissingIsNotFound
Readiness	ready maps any non-ACTIVE status to ACTIVE and clears health; applied and unknown phases change nothing. The agent reports ready when K3s records the add-on's gvks annotation, or after a 90 s grace once the bundle's workloads are rolled out; the check is per rendered file, not per version.	domains/eks/addon/owner_test.go TestNextStatus, TestOwnerApplyReport, handlers/eks/addon_status_reconciler_test.go TestApplyAddonStatusReport, tests/e2e/eks/eks_test.go TestEKS/AddonDelivery
Failure	Staging failure on create or update marks the record CREATE_FAILED with the error as health and returns the raw error. A failed report maps ACTIVE to DEGRADED, keeps DEGRADED and CREATE_FAILED, and maps anything else (including UPDATING) to CREATE_FAILED. Neither failure state is terminal: a later ready report returns the add-on to ACTIVE.	TestCreateAddon_InstallerFailureLeavesDurableCreateFailedRecord, TestUpdateAddon_InstallerFailureMarksCreateFailed, TestAddonStatusReport_MismatchedVersionIsAccepted, handlers/eks/addons_test.go TestCreateAddon_InstallerFailureMarksCreateFailed
Delete	Read the record, build a DELETING response from it, delete the manifest, then delete the record and manifest again (Owner.Delete). Both keys are gone when the call returns. The guest later notices the missing manifest, removes the rendered file and runs kubectl delete --wait=false; it sends no report, so nothing observes removal.	TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest, TestListStagedAddonManifests, tests/e2e/eks/eks_test.go TestEKS/AddonDelivery
Guest fetch	GET /clusters/{cluster}/internal-addons/{accountId} (gateway/eks.go, ListInternalAddons) is admitted only for a control-plane agent serving that cluster (AuthorizeInternal), relays to eks.ListStagedAddonManifests, and wraps the result as contracts/eks/v1 InternalAddonsResponse. The body is written by the SDK REST-JSON marshaller, which ignores the json tags: it is {"Addons":[{"AddonName","AddonVersion","ServiceAccountRoleArn","ConfigurationValues"}]} with empty strings present, sorted by add-on name.	gateway/eks_addon_wire_test.go TestEKSRequest_InternalAddonsWireBody, contracts/eks/v1 TestInternalAddonsResponse_RESTJSONBody, gateway/eks_authz_test.go TestEKSRequest_InternalRoutesAdmitTheCPAgentPrincipal, gateway/eks/internal_authz_test.go TestAuthorizeInternal_RejectsClusterTheCallerDoesNotServe
TSV conversion	eks-gateway-fetch -resource addons (agents/eks/gatewayfetch) decodes the body into its own copy of the shape (matching the Go field names only because encoding/json is case-insensitive) and writes name\tversion\troleArn\tbase64(config) per add-on; an empty set writes nothing. It retries the GET up to 30 times at 5 s.	agents/eks/gatewayfetch/addons_wire_test.go TestEmitAddonsTSV_GatewayWireBody, agents/eks/gatewayfetch/emit_test.go TestEmitAddonsTSV, TestEmitAddonsTSV_Empty, agents/eks/gatewayfetch/fetch_test.go TestRun_RendersAddons, TestRun_FailsAfterAttemptBudget
Guest render and report	Every 30 s (ADDON_SYNC_INTERVAL) the agent renders each staged add-on, substituting the role ARN into web-identity env when set, then publishes {"addon","version","phase","message","ts"} through eks-gateway-publish -channel addon. The gateway relays the raw payload to core NATS eks.addon.{account}.{cluster}.status (gateway/eks/internal_publish.go). A fetch failure keeps the current render.	gateway/eks/internal_publish_test.go TestPublishInternal_RelaysToSubjects, agents/eks/gatewaypublish/publish_test.go TestRun_PublishesWrappedPayload, TestRun_FailsAfterAttemptBudget
Retry meaning	Reports are level-triggered: each tick re-sends the current phase for every staged add-on, so a lost report is repeated on the next tick. eks-gateway-publish retries the POST up to 30 times at 5 s and the agent ignores its failure. Core NATS keeps nothing: a report published while no reconciler holds the cluster's lease, or after the reconciler stops on a terminal cluster status, is dropped.	none
Report ordering	The reconciler subscribes plainly (no queue group) on the lease holder and applies reports in arrival order through a revision CAS. Nothing orders a report against a later update: a report produced for the previous version settles the new UPDATING record, and a stale failed turns it CREATE_FAILED. A report for a deleted add-on is a no-op.	TestAddonStatusReport_MismatchedVersionIsAccepted, TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest
Cross-resource dependency	A GPU node group's launch calls stageGPUDeviceAddon, which checks the cluster exists and calls Owner.EnsureGPUDevicePlugin for nvidia-device-plugin; an existing record is a no-op and every other error is logged only. Deleting the node group never unstages it, and an existing record in any status, including CREATE_FAILED, blocks re-staging.	handlers/eks/nodegroup_test.go TestCreateNodegroup_GPUNodegroupStagesDevicePluginAddon, domains/eks/addon/owner_test.go TestOwnerEnsureGPUDevicePlugin_StagesDefaultVersionOnceAndKeepsAnyExistingRecord, TestCreateNodegroup_NonGPUNodegroupDoesNotStageDevicePluginAddon, tests/e2e/gpu/eks_gpu_test.go TestEKSGPUPodExposure
Idempotency	clientRequestToken is ignored. A duplicate create, including an SDK retry with the same token, returns ResourceInUseException. A repeated update re-stages and returns another Successful with the same id. A repeated delete returns ResourceNotFoundException.	TestCreateAddon_RepeatedClientRequestTokenIsResourceInUse, TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating, TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest, handlers/eks/addons_test.go TestCreateAddon_CreatesStagingAndDefaultsVersion
Absent resource	An unknown cluster returns ResourceNotFoundException for every action, including the internal manifest listing. A missing add-on returns ResourceNotFoundException from describe, update and delete.	handlers/eks/addons_test.go TestCreateAddon_UnknownClusterIsNotFound, TestDescribeDeleteAddon_MissingIsNotFound, TestUpdateAddon_MissingIsNotFound, TestListStagedAddonManifests
Dependencies	Only the cluster meta's existence is checked; cluster status is not read, so add-ons can be created on a DELETING cluster.	TestAddonActionsIgnoreClusterLifecycleStatus
Authorization	Gateway policy uses the add-on ARN above. serviceAccountRoleArn requires iam:PassRole on CreateAddon and UpdateAddon (gateway/eks/passrole.go), the same check CreateCluster/CreateNodegroup apply, including that the role exists. Tag actions on an add-on ARN return NotImplemented.	gateway/eks_addon_wire_test.go TestPassedRoleARNs_AddonServiceAccountRoleIsChecked, gateway/eks_passrole_test.go TestEKSCreateAddon_PassRole, TestEKSUpdateAddon_PassRole, gateway/eks/authz_test.go TestResourceARNsFidelity
Cluster deletion	Add-on records and manifests are erased only by the cluster prefix purge, after the cluster's infrastructure teardown succeeds; there is no per-add-on delete. Reclaiming a failed cluster keeps the meta and does not purge, so the recreated cluster inherits the add-ons.	TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster, handlers/eks/cluster_state_test.go TestDeleteClusterPrefix_SweepsEveryKey
Restart and interruption	Create interrupted between the record put and staging leaves a CREATING record with no manifest that nothing re-stages and that blocks a retry. Delete interrupted after unstaging leaves a record in its previous status while the guest removes the software. An update interrupted after the CAS leaves UPDATING with the previous manifest. A daemon restart loses nothing durable, but reports sent while no reconciler runs are lost until the next tick.	none
End-to-end	tests/e2e/eks/eks_test.go TestEKS/AddonDelivery (create to ACTIVE, objects present, delete removes the namespace) and tests/e2e/gpu/eks_gpu_test.go TestEKSGPUPodExposure; tests/integration/cross_tenant_test.go TestCrossTenant_EveryScopedActionIsAccountedFor. No E2E for update, failure, configuration or the role. No Terraform apply/destroy evidence for aws_eks_addon.

Gaps

1. Create is get-then-put (domains/eks/addon/owner.go Owner.Create), so concurrent creates of one add-on both succeed and the later write wins; internal NATS redelivery of a create returns ResourceInUseException.
2. No client-token support: clientRequestToken is dropped on create and update, so an SDK retry of a successful create fails instead of returning the add-on as AWS does.
3. Delivery reports ignore their supplied version (and ts), so a stale or mismatched report settles the current desired state; there is no generation or incarnation fence.
4. UpdateAddon returns Successful while the add-on remains UPDATING, and the update id is the add-on ARN. AWS returns an in-progress update with its own id, tracked through DescribeUpdate; not verified against live AWS.
5. DELETING exists only in the delete response; the record and manifest are erased before any guest removal, so DescribeAddon returns not found immediately.
6. No observed deletion acknowledgement: the agent removes objects with --wait=false and reports nothing.
7. configurationValues is stored, staged and transported but discarded by the agent; resolveConflicts, preserve and podIdentityAssociations are accepted and never used.
8. Fixed: serviceAccountRoleArn now requires iam:PassRole on CreateAddon and UpdateAddon, matching CreateCluster/CreateNodegroup (gateway/eks/passrole.go).
9. AWS UPDATE_FAILED and DELETE_FAILED are not represented: a failed update becomes CREATE_FAILED and a failed delete leaves the prior status.
10. The add-on ARN lacks the per-incarnation suffix AWS appends, so a recreated add-on reuses its predecessor's ARN. This is a compatibility gap; extraction must keep the current ARN.
11. The guest/controller wire shape is versioned as contracts/eks/v1 (subject, delivery phases, status report, staged manifest, internal-addons body), used directly by the handler, gateway and agents/eks/gatewayfetch. The add-on owner keeps its own Manifest and Report, and the shell agent still builds the report with printf. The HTTP body uses Go field names because the REST-JSON marshaller ignores the json tags, and the agent decodes it only through case-insensitive matching; the contract pins that form rather than changing it.
12. Reports travel over core NATS to a lease-held subscription, so they are lost whenever no reconciler runs for the cluster; delivery relies on the agent re-sending every tick.
13. The record mixes desired fields with the latest observation (status, health), and ready lifts CREATE_FAILED back to ACTIVE.
14. GPU auto-staging crosses from the node group into the add-on through Owner.EnsureGPUDevicePlugin, swallows every error, and is never undone.
15. ListAddons ignores maxResults and nextToken; tag actions on add-on ARNs return NotImplemented.
16. No ADR-0003 S5 failure or recovery evidence.