gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-20 19:25:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: record EKS add-on lifecycle contract

Browse Source
This commit is contained in:
joshsiv-mulga
2026-10-08 22:33:28 +10:00
parent 52de5e517f
commit 675341b561
2 changed files with 64 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PACKAGE_BOUNDARY_MIGRATION.md
+1
View File
@@ -299,6 +299,7 @@ while the branch is in flight; they do not describe the umbrella repository's
| `4b5524781` | `spinifex/handlers/rds` (`Engine`, engine registry and lookup, family mapping, version/username/DB-name validation, `postgres`/`mariadb` catalogues, `ParameterSpec`, `ApplyType*`, `ParamType*`, `ResolveEffectiveParameters`, `db.*` classes, engine-version and orderable selection) | `spinifex/domains/rds/engine` (`Engine`, `LookupEngine`, `EngineForFamily`, `ParameterSpec`, `Setting`, `InstanceTypes`, `Sizing`, `NewSizing`, `EngineVersions`, `OrderableOptions`) | ADR-0004 owner move of static metadata; boundary note in `docs/package-boundary/rds-engine-catalogue-boundary.md`. Two deliberate boundaries: EC2 instance-type memory is consumed only through `Sizing`, never by a direct RDS to EC2 package dependency; agent engine metadata is consumed only through the agent-declared `EngineCatalog` contract composed in `cmd/rds-agent`, never by importing the catalogue. Effective parameters cross as engine-owned `Setting`, converted to `Parameter` in `handlers/rds`. AWS output, paging, apply-method mapping, storage limits and the runnable check stay in the adapter path. |
| `ca1dd9aa9` | — | — | RDS DB parameter group characterization before extraction (ADR-0003 S1, ADR-0004): persisted meta and override field names, pre-existing JSON decodes, default groups synthesised and never stored, repeated delete is not found, a pending attachment blocks delete, a dynamic `pending-reboot` override is still applied live, propagation reaches a stopped instance. Present contract and gaps in `docs/package-boundary/rds-db-parameter-group-lifecycle.md`; propagation to instances is recorded as instance-side realization outside the group, and no conformance is claimed. |
| `d9c3f3ca6` | `spinifex/handlers/rds` (`DBParameterGroupRecord`, `DBParameterRecord`, the parameter-group key helpers, `ListDBParameterGroupNames`, `ListDBParameterOverrides`, `getDBParameterGroup` and the create/modify/delete writes) | `spinifex/domains/rds/parametergroup` (`Owner`, `Record`, `Override`, `MetaKey`, `ParamsPrefix`, `ParamKey`, `Prefix`) | ADR-0004 owner extraction (contract in `docs/package-boundary/rds-db-parameter-group-lifecycle.md`). Stored keys and JSON field names unchanged. Delete reaches instances through the owner-declared `Dependants`; instance-side resolution reads through the instance-declared `parameterGroupValues`. `SetOverrides` refuses an unknown group. The handler keeps AWS validation (it refuses every `default.*` name; the owner refuses only a real engine default), projection, paging and propagation. Proves ownership and dependency direction only; no conformance claimed. |
| `52de5e517` | — | — | EKS add-on characterization before extraction (ADR-0003 S1, ADR-0004): persisted record and manifest keys and field names, durable CREATING and staged descriptor, installer failure as CREATE_FAILED (update too), update returning Successful while UPDATING, mismatched-version reports accepted, delete erasing both keys while answering DELETING, the ignored client token, no PassRole check on serviceAccountRoleArn, and the exact internal add-ons body (Go field names, not the declared json tags) and agent TSV. Present contract and gaps in `docs/package-boundary/eks-addon-lifecycle.md`; no conformance claimed. |
The EC2 contract row is intentionally different from the directory moves: it creates
a compatibility boundary. `ec2.cmd.*` retains its deployed one-token NATS
docs/package-boundary/eks-addon-lifecycle.md
+63
View File
@@ -0,0 +1,63 @@
# EKS managed add-on: lifecycle contract
Present contract under ADR-0003 S1, recorded before the resource moves.
It describes the code as it stands, so it claims no conformance; the gaps below are observed behaviour, not targets.
The resource-specific target is proposed requirement Q-146 and the recorded conflict is C-82 (`docs/specs/requirements/managed-services.md` in the mulga repo); both are open proposals pending review, not accepted decisions, and this record does not close either.
Paths are relative to `spinifex/` unless they start with `tests/` or `scripts/`, which are relative to the spinifex repo root.
## Scope
An add-on has an asynchronous realization: the control-plane VM renders a baked bundle into the K3s auto-deploy directory and reports delivery back.
The host side owns a desired descriptor (the record plus a staged manifest sub-key); the guest side (`scripts/images/eks-node/mulga-eks-addon-sync.sh`) pulls staged manifests through the gateway and publishes per-add-on delivery reports, which the cluster reconciler folds back into the record.
There is no generation, so the record mixes desired fields with the latest observation.
## Contract
| Item | Present behaviour | Evidence |
|---|---|---|
| Identity | Cluster name plus add-on name; one add-on per name per cluster. ARN `arn:aws:eks:{region}:{account}:addon/{cluster}/{addon}` (`foundation/aws/arn/eks.go` `FormatEKSAddon`), deterministic by name, with no per-incarnation suffix. The gateway authorizer derives the same ARN (`gateway/eks/authz.go` `addonARN`). | `handlers/eks/addon_lifecycle_test.go` `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `gateway/eks/authz_test.go` `TestResourceARNsFidelity` |
| Scope | Account and Region, within one cluster: keys live under `clusters/{cluster}/addons/` in the `eks-account-{account}` bucket. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster` |
| Durable record | `clusters/{cluster}/addons/{addon}` holds `AddonRecord` (`handlers/eks/addon_state.go`): `addonName`, `addonVersion`, `status`, `serviceAccountRoleArn`, `configurationValues`, `health`, `arn`, `tags`, `createdAt`, `modifiedAt` (the optional ones `omitempty`). `clusters/{cluster}/addons/{addon}/manifest` holds `StagedAddonManifest` (`handlers/eks/addon_installer.go`): `addonName`, `addonVersion`, `serviceAccountRoleArn`, `configurationValues`. Listing skips the manifest sub-key by segment count. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestAddonRecord_DecodesHandWrittenJSON`, `handlers/eks/addon_state_test.go` `TestAddonRecord_RoundTrip`, `TestListAddonRecords_SkipsManifestSubKeys` |
| Writers | Record: `CreateAddon` (unconditional put), `UpdateAddon` and `markAddonFailed` (revision CAS), `DeleteAddon` via `DeleteAddonRecord` (get then delete), `ClusterReconciler.applyAddonStatusReport` (revision CAS), `stageGPUDeviceAddon` through `CreateAddon`, and the cluster prefix purge (`DeleteClusterPrefix`). Manifest: `stagingInstaller.Install` (unconditional put on create and update), `stagingInstaller.Uninstall`, `DeleteAddonRecord` and the prefix purge. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster`, `handlers/eks/addon_state_test.go` `TestCasUpdateAddon`, `TestDeleteAddonRecord_RemovesRecordAndManifest` |
| Readers | `ListAddons`, `DescribeAddon`, `CreateAddon` (existence check) and the reconciler read the record. `ListStagedAddonManifests` reads every `/manifest` key for the cluster; it is served on `eks.ListStagedAddonManifests` (`daemon/daemon.go`) and reached only through the gateway's internal-addons route. | `handlers/eks/addons_test.go` `TestAddon_DescribeListDelete`, `TestListStagedAddonManifests` |
| Catalogue | Static in-binary catalogue (`handlers/eks/addon_catalog.go`): `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`, and hidden-but-creatable `nvidia-device-plugin` and `spinifex-noop`; one version each, the first being the default. Create and update reject unknown names and versions with `InvalidParameterValue`. | `handlers/eks/addon_catalog_test.go`, `handlers/eks/addons_test.go` `TestDescribeAddonVersions_ReturnsCatalog`, `TestCreateAddon_UnknownAddonRejected`, `TestCreateAddon_UnknownVersionRejected`, `TestUpdateAddon_UnknownVersionRejected` |
| Desired state | Version, `serviceAccountRoleArn`, `configurationValues` and create-time tags. `resolveConflicts`, `podIdentityAssociations` and `clientRequestToken` are accepted and dropped; `preserve` on delete is ignored. `configurationValues` reaches the guest but the agent discards it (`mulga-eks-addon-sync.sh` `sync_once`), so it has no effect. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest` |
| Observed state | `status` and `health` are overwritten by guest reports on the same record as the desired fields. `health` is the latest failure message, projected as one `AddonIssue` with only `message`. Statuses held: `CREATING`, `ACTIVE`, `UPDATING`, `DEGRADED`, `CREATE_FAILED`; `DELETING` is defined but never stored. | `TestAddonRecord_DecodesHandWrittenJSON`, `handlers/eks/addon_status_reconciler_test.go` `TestApplyAddonStatusReport` |
| Generation | None. Reports carry `version` and `ts`, and neither is read: any report for the add-on name applies to whatever is currently desired. | `TestAddonStatusReport_MismatchedVersionIsAccepted` |
| Create | Validate against the catalogue, require the cluster meta (`acctKVForCluster`), get then put a `CREATING` record, then stage the manifest. Returns `CREATING`. | `TestAddonPersistedLayout_RecordAndManifestKeysAndFieldNames`, `handlers/eks/addons_test.go` `TestCreateAddon_CreatesStagingAndDefaultsVersion`, `TestStagingInstaller_StagesManifest`, `TestCreateAddon_UnknownClusterIsNotFound` |
| Update | Validate the version, CAS the new version, config and role onto the record with status `UPDATING`, re-stage the manifest, and return an `Update` with status `Successful`, type `AddonUpdate` and `id` equal to the add-on ARN (the same for every update). Allowed from any stored status. | `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `handlers/eks/addons_test.go` `TestUpdateAddon_ChangesVersionAndReinstalls`, `TestUpdateAddon_MissingIsNotFound` |
| Readiness | `ready` maps any non-`ACTIVE` status to `ACTIVE` and clears `health`; `applied` and unknown phases change nothing. The agent reports `ready` when K3s records the add-on's `gvks` annotation, or after a 90 s grace once the bundle's workloads are rolled out; the check is per rendered file, not per version. | `handlers/eks/addon_status_reconciler_test.go` `TestNextAddonStatus`, `TestApplyAddonStatusReport`, `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` |
| Failure | Staging failure on create or update marks the record `CREATE_FAILED` with the error as `health` and returns the raw error. A `failed` report maps `ACTIVE` to `DEGRADED`, keeps `DEGRADED` and `CREATE_FAILED`, and maps anything else (including `UPDATING`) to `CREATE_FAILED`. Neither failure state is terminal: a later `ready` report returns the add-on to `ACTIVE`. | `TestCreateAddon_InstallerFailureLeavesDurableCreateFailedRecord`, `TestUpdateAddon_InstallerFailureMarksCreateFailed`, `TestAddonStatusReport_MismatchedVersionIsAccepted`, `handlers/eks/addons_test.go` `TestCreateAddon_InstallerFailureMarksCreateFailed` |
| Delete | Read the record, build a `DELETING` response from it, delete the manifest, then delete the record and manifest again in `DeleteAddonRecord`. Both keys are gone when the call returns. The guest later notices the missing manifest, removes the rendered file and runs `kubectl delete --wait=false`; it sends no report, so nothing observes removal. | `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `TestListStagedAddonManifests`, `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` |
| Guest fetch | `GET /clusters/{cluster}/internal-addons/{accountId}` (`gateway/eks.go`, `ListInternalAddons`) is admitted only for a control-plane agent serving that cluster (`AuthorizeInternal`), relays to `eks.ListStagedAddonManifests`, and wraps the result as `internalAddonsOutput`. The body is written by the SDK REST-JSON marshaller, which ignores the `json` tags: it is `{"Addons":[{"AddonName","AddonVersion","ServiceAccountRoleArn","ConfigurationValues"}]}` with empty strings present, sorted by add-on name. | `gateway/eks_addon_wire_test.go` `TestEKSRequest_InternalAddonsWireBody`, `gateway/eks_authz_test.go` `TestEKSRequest_InternalRoutesAdmitTheCPAgentPrincipal`, `gateway/eks/internal_authz_test.go` `TestAuthorizeInternal_RejectsClusterTheCallerDoesNotServe` |
| TSV conversion | `eks-gateway-fetch -resource addons` (`agents/eks/gatewayfetch`) decodes the body into its own copy of the shape (matching the Go field names only because `encoding/json` is case-insensitive) and writes `name\tversion\troleArn\tbase64(config)` per add-on; an empty set writes nothing. It retries the GET up to 30 times at 5 s. | `agents/eks/gatewayfetch/addons_wire_test.go` `TestEmitAddonsTSV_GatewayWireBody`, `agents/eks/gatewayfetch/emit_test.go` `TestEmitAddonsTSV`, `TestEmitAddonsTSV_Empty`, `agents/eks/gatewayfetch/fetch_test.go` `TestRun_RendersAddons`, `TestRun_FailsAfterAttemptBudget` |
| Guest render and report | Every 30 s (`ADDON_SYNC_INTERVAL`) the agent renders each staged add-on, substituting the role ARN into web-identity env when set, then publishes `{"addon","version","phase","message","ts"}` through `eks-gateway-publish -channel addon`. The gateway relays the raw payload to core NATS `eks.addon.{account}.{cluster}.status` (`gateway/eks/internal_publish.go`). A fetch failure keeps the current render. | `gateway/eks/internal_publish_test.go` `TestPublishInternal_RelaysToSubjects`, `agents/eks/gatewaypublish/publish_test.go` `TestRun_PublishesWrappedPayload`, `TestRun_FailsAfterAttemptBudget` |
| Retry meaning | Reports are level-triggered: each tick re-sends the current phase for every staged add-on, so a lost report is repeated on the next tick. `eks-gateway-publish` retries the POST up to 30 times at 5 s and the agent ignores its failure. Core NATS keeps nothing: a report published while no reconciler holds the cluster's lease, or after the reconciler stops on a terminal cluster status, is dropped. | none |
| Report ordering | The reconciler subscribes plainly (no queue group) on the lease holder and applies reports in arrival order through a revision CAS. Nothing orders a report against a later update: a report produced for the previous version settles the new `UPDATING` record, and a stale `failed` turns it `CREATE_FAILED`. A report for a deleted add-on is a no-op. | `TestAddonStatusReport_MismatchedVersionIsAccepted`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest` |
| Cross-resource dependency | A GPU node group's launch calls `stageGPUDeviceAddon`, which calls `CreateAddon` for `nvidia-device-plugin` and ignores `ResourceInUse` and every other error (logged only). Deleting the node group never unstages it, and an existing record in any status, including `CREATE_FAILED`, blocks re-staging. | `handlers/eks/nodegroup_test.go` `TestCreateNodegroup_GPUNodegroupStagesDevicePluginAddon`, `TestCreateNodegroup_NonGPUNodegroupDoesNotStageDevicePluginAddon`, `tests/e2e/gpu/eks_gpu_test.go` `TestEKSGPUPodExposure` |
| Idempotency | `clientRequestToken` is ignored. A duplicate create, including an SDK retry with the same token, returns `ResourceInUseException`. A repeated update re-stages and returns another `Successful` with the same id. A repeated delete returns `ResourceNotFoundException`. | `TestCreateAddon_RepeatedClientRequestTokenIsResourceInUse`, `TestUpdateAddon_ReportsSuccessfulWhileRecordIsUpdating`, `TestDeleteAddon_ReturnsDeletingAfterErasingRecordAndManifest`, `handlers/eks/addons_test.go` `TestCreateAddon_CreatesStagingAndDefaultsVersion` |
| Absent resource | An unknown cluster returns `ResourceNotFoundException` for every action, including the internal manifest listing. A missing add-on returns `ResourceNotFoundException` from describe, update and delete. | `handlers/eks/addons_test.go` `TestCreateAddon_UnknownClusterIsNotFound`, `TestDescribeDeleteAddon_MissingIsNotFound`, `TestUpdateAddon_MissingIsNotFound`, `TestListStagedAddonManifests` |
| Dependencies | Only the cluster meta's existence is checked; cluster status is not read, so add-ons can be created on a `DELETING` cluster. | `TestAddonActionsIgnoreClusterLifecycleStatus` |
| Authorization | Gateway policy uses the add-on ARN above. `serviceAccountRoleArn` is stored and rendered into the add-on's web-identity env without `iam:PassRole` (`gateway/eks/passrole.go` covers `CreateCluster` and `CreateNodegroup` only) and without checking the role exists. Tag actions on an add-on ARN return `NotImplemented`. | `gateway/eks_addon_wire_test.go` `TestPassedRoleARNs_AddonServiceAccountRoleIsNotChecked`, `gateway/eks/authz_test.go` `TestResourceARNsFidelity` |
| Cluster deletion | Add-on records and manifests are erased only by the cluster prefix purge, after the cluster's infrastructure teardown succeeds; there is no per-add-on delete. Reclaiming a failed cluster keeps the meta and does not purge, so the recreated cluster inherits the add-ons. | `TestDeleteClusterPrefix_SweepsAddonRecordsAndManifestsScopedToCluster`, `handlers/eks/cluster_state_test.go` `TestDeleteClusterPrefix_SweepsEveryKey` |
| Restart and interruption | Create interrupted between the record put and staging leaves a `CREATING` record with no manifest that nothing re-stages and that blocks a retry. Delete interrupted after unstaging leaves a record in its previous status while the guest removes the software. An update interrupted after the CAS leaves `UPDATING` with the previous manifest. A daemon restart loses nothing durable, but reports sent while no reconciler runs are lost until the next tick. | none |
| End-to-end | `tests/e2e/eks/eks_test.go` `TestEKS/AddonDelivery` (create to `ACTIVE`, objects present, delete removes the namespace) and `tests/e2e/gpu/eks_gpu_test.go` `TestEKSGPUPodExposure`; `tests/integration/cross_tenant_test.go` `TestCrossTenant_EveryScopedActionIsAccountedFor`. No E2E for update, failure, configuration or the role. No Terraform apply/destroy evidence for `aws_eks_addon`. | |
## Gaps
1. Create is get-then-put (`handlers/eks/addons.go` `CreateAddon`), so concurrent creates of one add-on both succeed and the later write wins; internal NATS redelivery of a create returns `ResourceInUseException`.
2. No client-token support: `clientRequestToken` is dropped on create and update, so an SDK retry of a successful create fails instead of returning the add-on as AWS does.
3. Delivery reports ignore their supplied `version` (and `ts`), so a stale or mismatched report settles the current desired state; there is no generation or incarnation fence.
4. `UpdateAddon` returns `Successful` while the add-on remains `UPDATING`, and the update `id` is the add-on ARN. AWS returns an in-progress update with its own id, tracked through `DescribeUpdate`; not verified against live AWS.
5. `DELETING` exists only in the delete response; the record and manifest are erased before any guest removal, so `DescribeAddon` returns not found immediately.
6. No observed deletion acknowledgement: the agent removes objects with `--wait=false` and reports nothing.
7. `configurationValues` is stored, staged and transported but discarded by the agent; `resolveConflicts`, `preserve` and `podIdentityAssociations` are accepted and never used.
8. `serviceAccountRoleArn` is not checked for `iam:PassRole` on `CreateAddon` or `UpdateAddon`, which AWS requires; not verified against live AWS. Recorded, not fixed: it touches the IAM boundary.
9. AWS `UPDATE_FAILED` and `DELETE_FAILED` are not represented: a failed update becomes `CREATE_FAILED` and a failed delete leaves the prior status.
10. The add-on ARN lacks the per-incarnation suffix AWS appends, so a recreated add-on reuses its predecessor's ARN. This is a compatibility gap; extraction must keep the current ARN.
11. The guest/controller wire shape is duplicated and unversioned: `StagedAddonManifest` and `internalAddonsOutput` in the handler and gateway, `stagedAddon` in `agents/eks/gatewayfetch`, `AddonStatusReport` in `handlers/eks/state_report.go` and a `printf` in the shell agent. The HTTP body uses Go field names because the REST-JSON marshaller ignores the `json` tags, and the agent decodes it only through case-insensitive matching.
12. Reports travel over core NATS to a lease-held subscription, so they are lost whenever no reconciler runs for the cluster; delivery relies on the agent re-sending every tick.
13. The record mixes desired fields with the latest observation (`status`, `health`), and `ready` lifts `CREATE_FAILED` back to `ACTIVE`.
14. GPU auto-staging crosses from the node group into the add-on through the public `CreateAddon`, swallows every error, and is never undone.
15. `ListAddons` ignores `maxResults` and `nextToken`; tag actions on add-on ARNs return `NotImplemented`.
16. No ADR-0003 S5 failure or recovery evidence.