mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
docs: post-ADR-0001 source-tree doc fixes and move planning material to Mulga (#1210)
* docs: update source-tree docs and comments after the ADR-0001 package moves * docs: move package-boundary planning material to Mulga
This commit is contained in:
Josh Siviour
+26
-13
@@ -305,22 +305,32 @@ Configs are generated by `spx admin init` (leader) or `spx admin join` (follower
|
||||
|
||||
## File Structure
|
||||
|
||||
The layout follows Mulga ADR-0001 (Spinifex package boundary and layout).
|
||||
`architecture/layering` enforces its dependency rules in `make preflight`.
|
||||
|
||||
```
|
||||
spinifex/
|
||||
├── cmd/
|
||||
│ ├── spinifex/ # spx CLI: service entry points and `spx admin`
|
||||
│ ├── installer/ # Platform installer
|
||||
│ └── ... # In-guest agents (lb, ecs, rds, eks helpers) and dev tools
|
||||
├── cmd/ # Thin executable mains: spx, installer, in-guest agents, dev tools
|
||||
├── contracts/ # Versioned cross-process wire contracts (contracts/<domain>/vN)
|
||||
├── spinifex/
|
||||
│ ├── services/ # Long-running services (awsgw, spinifex, nats, predastore, ...)
|
||||
│ ├── gateway/ # AWS API surface: routing, SigV4 auth, per-service dispatch
|
||||
│ ├── handlers/ # NATS-side service implementations (ec2, iam, eks, rds, ...)
|
||||
│ ├── daemon/ # Per-node daemon and its NATS subscription table
|
||||
│ ├── vm/, qmp/ # VM lifecycle and the QEMU Machine Protocol client
|
||||
│ ├── vpcd/, network/ # VPC daemon and the layered OVN/OVS network stack
|
||||
│ ├── admin/ # Cluster bootstrap (init/join, master.key, CA, tokens)
|
||||
│ └── config/ # Configuration structures
|
||||
├── internal/ # In-guest SigV4 clients and the AWS Smithy models
|
||||
│ ├── foundation/ # Shared primitives: AWS errors/ARNs/filters/tags, KV state, messaging, telemetry, PKI
|
||||
│ ├── providers/ # Backend adapters: EBS (Viperblock), object store, cloud (OCI, Exoscale)
|
||||
│ ├── domains/ # Service domains (ec2, acm, ecr, dns, network, ochre, rds, tagging, ...)
|
||||
│ │ └── <domain>/awsapi/ # A domain's AWS action adapters
|
||||
│ ├── ingress/ # AWS protocol ingress: query/REST parsing, dispatch registry, rate limiting
|
||||
│ ├── runtime/
|
||||
│ │ ├── roles/ # Long-running service roles (awsgw, spinifex, vpcd, nats, predastore, ...)
|
||||
│ │ └── compute/ # VM lifecycle, QMP, NBD, GPU
|
||||
│ ├── bootstrap/ # Cluster and node configuration structures, preflight checks
|
||||
│ ├── operator/ # spx CLI commands, admin use cases, installer, upgrade
|
||||
│ ├── agents/ # In-guest agent libraries (ecs, eks, rds)
|
||||
│ ├── gateway/ # Legacy root: HTTP route assembly, SigV4 auth, legacy per-service dispatch
|
||||
│ ├── handlers/ # Legacy root: ecs, eks, elbv2, iam, rds, sts service implementations
|
||||
│ ├── daemon/ # Legacy root: per-node daemon and its NATS subscription table
|
||||
│ ├── accountteardown/ # Legacy root: account deletion cleanup
|
||||
│ └── lbagent/ # Legacy root: load-balancer agent
|
||||
├── internal/ # In-guest SigV4 clients, AWS Smithy models and test kits
|
||||
├── architecture/ # Executable architecture rules (package layering)
|
||||
├── build/ # systemd units, node helper scripts, microVM build
|
||||
├── scripts/ # Install, dev-environment, image-build scripts
|
||||
└── tests/
|
||||
@@ -328,6 +338,9 @@ spinifex/
|
||||
└── e2e/ # Live-cluster suites (see docs/testing/e2e-coverage.md)
|
||||
```
|
||||
|
||||
The five legacy roots predate ADR-0001 and are frozen: they may shrink but not gain packages.
|
||||
[`PACKAGE_BOUNDARY_MIGRATION.md`](PACKAGE_BOUNDARY_MIGRATION.md) records what remains in each and what blocks its move.
|
||||
|
||||
## Development
|
||||
|
||||
### Running the Stack
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# Package-boundary contracts
|
||||
|
||||
This directory holds code-adjacent evidence that versions with the code: present lifecycle contracts and boundary notes describing a package as it stands.
|
||||
Plans, discovery reports, inventories and draft designs belong in the Mulga repository; the earlier ones are in `docs/development/reference/spinifex-package-boundary/` there.
|
||||
@@ -1,388 +0,0 @@
|
||||
# ADR-0004 source inventory — ECS
|
||||
|
||||
Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`.
|
||||
It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it.
|
||||
Items marked "verify" or "uncertain" are unconfirmed.
|
||||
Gaps record observed behaviour, not target behaviour.
|
||||
|
||||
|
||||
Repo: `spinifex` submodule, branch `refactor/spinifex-package-boundaries`; paths below are relative to `spinifex/spinifex/` unless stated otherwise.
|
||||
Binding inputs: ADR-0001..0004 and Q-78 at `origin/main`.
|
||||
This inventory reports what the code does; it accepts or closes no gap.
|
||||
"Verify" marks claims about AWS behaviour, or runtime behaviour, that a test or live system has not confirmed here.
|
||||
|
||||
## 1. File table
|
||||
|
||||
Totals: `handlers/ecs` is 7,760 production lines over 28 files, `gateway/ecs` + `gateway/ecs.go` is 1,236 lines, and `daemon/ecs_deps.go` + `accountteardown/reapers_ecs.go` is 132 lines.
|
||||
|
||||
| File | Lines | Class | Reason / split |
|
||||
|---|---|---|---|
|
||||
| `handlers/ecs/bus/messages.go` | 173 | guest/controller wire contract (part temporary residue) | `Assign`, `AssignContainer`, `StopDirective`, `TaskState`, `ContainerStatus`, `InstanceCapacity`, `PortMapping` and `SystemControl` are live payloads of the agent and gateway path. `RegisterInstance` (:19) and `Heartbeat` (:33) have no production publisher, so they are residue. Split: live types go to a versioned `contracts/ecs/v1`; the residue types are deleted once the scheduler bus subscriptions are removed. |
|
||||
| `handlers/ecs/bus/subjects.go` | 45 | temporary residue (prerequisite: drop scheduler bus subs; removal: delete unused builders) | `RegisterSubject`, `HeartbeatSubject`, `AssignSubject` and `TaskStateSubject` (:19-38) have no production publisher. `ServiceReconcileSubject` (:43) is live and is an internal cross-node wake owned by the service reconcile. |
|
||||
| `handlers/ecs/capacity.go` | 223 | cross-resource orchestration (container-instance capacity provisioning) | `ProvisionCapacity` (:70) is a non-AWS action. It composes an IAM system role, EC2 AMI lookup and EC2 `RunInstances`, and records nothing durable in ECS. |
|
||||
| `handlers/ecs/capacity_providers.go` | 233 | resource owner (`capacityprovider`), mixed | Create, Describe and Delete belong to `capacityprovider`. `PutClusterCapacityProviders` (:16-40) mutates the cluster record and belongs to `cluster`. |
|
||||
| `handlers/ecs/cluster_teardown.go` | 171 | mixed: cross-resource orchestration + resource owner (`containerinstance`) | `DeleteCluster` (:24-53) is a cascade over every child, which is orchestration. `DeregisterContainerInstance` (:60), `UpdateContainerInstancesState` (:103), `instanceActiveTasks` (:143) and `drainInstanceServiceTasks` (:160) are container-instance commands that also stop tasks. Split three ways: cluster delete orchestration, container-instance owner, and task-stop calls through a task capability. |
|
||||
| `handlers/ecs/conv.go` | 291 | mixed | `ClusterShortName`, `ContainerInstanceShortID`, `TaskShortID` and `CapacityProviderShortName` (:27-60) are per-resource identity parsers that the gateway authz also uses. The `containerDefsFromAWS`, `runtimePlatformFromAWS` and `ContainerDef.toAWS` group (:87-265) is the `taskdefinition` projection. `toAssignContainer` (:268) is wire-contract projection. |
|
||||
| `handlers/ecs/deployments.go` | 267 | resource owner (`service`) | This is pure deployment, rollout and circuit-breaker logic on `ServiceRecord`, with no KV I/O. |
|
||||
| `handlers/ecs/eni.go` | 155 | cross-resource orchestration (task realization, network capability) | The consumer-owned `eniController` interface (:37) belongs with `task`. `natsENIController` (:46-128) binds it to EC2 subjects and belongs at the composition root. |
|
||||
| `handlers/ecs/gpu_report.go` | 63 | resource owner (`task`), mixed with wire contract | `ReportTaskGPU` (:31) writes the task record. `ReportTaskGPUInput`, `ContainerGPUReport` and `ReportTaskGPUOutput` (:10-25) are agent wire shapes. |
|
||||
| `handlers/ecs/instancerole.go` | 48 | cross-resource orchestration (identity for container instances) | It ensures and converges the `ecsInstanceRole` through `handlers_iam` (:37-48). |
|
||||
| `handlers/ecs/instances.go` | 532 | mixed: resource owner (`containerinstance`) + `task` + orchestration | Register, Describe, List, `upsertInstance`, `recordRegister` and `recordHeartbeat` (:19-357) belong to `containerinstance`. `recordTaskState` (:361) and `SubmitTaskStateChange` (:445) belong to `task`. `releaseReservation` (:505) is placement against the instance record. `recordDeploymentFailure` (:481) belongs to `service`. |
|
||||
| `handlers/ecs/netcfg.go` | 78 | resource owner (`task`) | It resolves the network mode and parses awsvpc input for task launch. |
|
||||
| `handlers/ecs/placement.go` | 144 | cross-resource orchestration (placement) | It contains pure fit and strategy selection over container-instance capacity, with no I/O. |
|
||||
| `handlers/ecs/poll.go` | 107 | guest/controller wire contract | It holds the agent inbox drain and ack (`PollAssignments` :37), and the task-owned inbox reclaim (:93-107). |
|
||||
| `handlers/ecs/records.go` | 525 | mixed (all resources' records) | It holds the status constants, ARN builders and every record type for every resource. Split each record, its ARN builder and its statuses into the owning resource. |
|
||||
| `handlers/ecs/scheduler.go` | 449 | cross-resource orchestration + temporary residue | Lease, service reconcile loop, instance reaper and role converge are orchestration. `onRegister`, `onHeartbeat` and `onTaskState` (:236-320) are residue because nothing publishes those subjects. |
|
||||
| `handlers/ecs/service.go` | 706 | mixed: facade + `cluster` + `taskdefinition` + shared KV helpers | `ECSService` (:27-81) is an AWS-shaped facade that also serves as an internal API. `Deps` (:98) is composition. Cluster CRUD (:231-411) belongs to `cluster`. Task-definition CRUD and `ParseTaskDefRef` (:416-706) belong to `taskdefinition`. `getJSON`, `putJSON` and `keysWithPrefix` (:164-225) are shared persistence helpers. |
|
||||
| `handlers/ecs/service_nats.go` | 177 | AWS protocol adapter (gateway→daemon NATS hop) | `NATSECSService` forwards 34 methods onto `ecs.<Action>`. It is also used as an internal API by `accountteardown` and by gateway PassRole resolution. |
|
||||
| `handlers/ecs/services.go` | 786 | mixed: resource owner (`service`) + orchestration bindings | Service CRUD and reconcile (:128-665) belong to `service`. `targetRegistrar` / `natsTargetRegistrar` (:28-59, ELBv2) and `eipManager` / `natsEIPManager` (:64-121, EC2 EIP) are consumer-owned capabilities, but their NATS bindings belong at the composition root. `listTaskRecords` (:605) reads task records directly. |
|
||||
| `handlers/ecs/store.go` | 227 | temporary residue (shared account bucket; prerequisite: decide per-owner key namespaces; removal: key helpers move to owners) | One bucket holds the keys of all six resources (:36-150). It also holds the leader bucket factory (:221) and the account enumeration used by every sweep (:186). |
|
||||
| `handlers/ecs/tags.go` | 277 | cross-resource orchestration (tag dispatch) | It parses the ARN and then reads and writes each resource's record directly (:76-203). It should become one tag command per owner behind ARN dispatch. |
|
||||
| `handlers/ecs/task_stop.go` | 301 | mixed: resource owner (`task`) + orchestration | `StopTask`, `StartTask`, `reserveOnInstance`, `requestStopTask` and `forceStopTask` (:28-209) belong to `task`, with placement mixed into `reserveOnInstance`. `applyServiceTargets` (:235) and `assignTaskPublicIP` / `releaseTaskPublicIP` (:264-301) are ELBv2 and EIP orchestration. |
|
||||
| `handlers/ecs/tasks.go` | 438 | mixed: resource owner (`task`) + placement orchestration + wire | `RunTask`, Describe, List and the record/projection code belong to `task`. `reservePlacement` (:180) does a CAS on the container-instance record. `publishAssign` (:249) writes the agent inbox. |
|
||||
| `handlers/ecs/tasks_gc.go` | 130 | resource owner (`task`) reconciliation | It retains STOPPED tasks, retries owed ENI releases and prunes old tasks. It is hosted on `Scheduler` (:21). |
|
||||
| `handlers/ecs/userdata.go` | 58 | guest/controller wire contract | It writes the cloud-config that seeds the agent env (`ECS_GATEWAY_URL`, `ECS_GATEWAY_CA`, `ECS_REGION`, `ECS_CLUSTER`) (:31-58). |
|
||||
| `gateway/ecs.go` | 80 | AWS protocol adapter | `ECS_Request` (:26) does target parsing, the scope gate (:54-60), the PassRole closure (:65-67) and response encoding. |
|
||||
| `gateway/ecs/handler.go` | 165 | AWS protocol adapter | It holds the authoritative action table (:50-112), `StubbedActionNames` (:115), the raw-JSON action list (:131) and the JSON 1.1 writers. |
|
||||
| `gateway/ecs/actions.go` | 452 | AWS protocol adapter (mixed: PassRole resolution) | Each action decodes and forwards over NATS. `checkTaskLaunchRoles` (:44) queries `DescribeTaskDefinition` over the ECS NATS facade to find role ARNs. |
|
||||
| `gateway/ecs/authz.go` | 350 | AWS protocol adapter | The per-action resource-ARN scope table (:57-124) and resolver (:139-350) are built on the `handlers_ecs` parsers and ARN builders. |
|
||||
| `gateway/ecs/azrebalancing.go` | 103 | AWS protocol adapter | It rejects `availabilityZoneRebalancing=ENABLED` (:28) and adds the SDK-gap field to service responses (:54). |
|
||||
| `gateway/ecs/clienttoken.go` | 86 | AWS protocol adapter holding durable retry state (flag) | It holds the RunTask ClientToken store `spinifex-ecs-clienttokens` (:22). ADR-0003 S3 says an adapter does not own retry bookkeeping, so this belongs with `task`. |
|
||||
| `daemon/ecs_deps.go` | 63 | composition root (nearest class: cross-resource orchestration) | It binds IAM, images, `RunWorkerInstance` and the gateway URL/CA into `handlers_ecs.Deps` (:17-63). |
|
||||
| `daemon/daemon.go` (ECS sections only) | n/a | AWS protocol adapter transport + composition | Import at :74, fields at :168-169, 34 `ecs.<Action>` subscriptions at :1131-1167, construction at :2023-2043. |
|
||||
| `accountteardown/reapers_ecs.go` | 69 | cross-resource orchestration (account teardown) | It has one cluster reaper that calls the ECS AWS facade over NATS (:20-58). |
|
||||
| `gateway/operations.go` (:7, :30-32) | n/a | AWS protocol adapter | It registers the ECS operation inventory from `gateway_ecs.Actions` and `StubbedActionNames`. |
|
||||
| `gateway/gateway.go` (:301, :403, :523-524, :639) | n/a | AWS protocol adapter | Service routing to `ECS_Request` and the `ecs-tasks.amazonaws.com` PassRole principal constant live here. |
|
||||
|
||||
Other ECS-aware code outside scope, with no import of ECS:
|
||||
`handlers/sts/assume_role.go:47-55,350-355` duplicates the `ecsInstanceRole` name and the `ecs-tasks.amazonaws.com` principal, and documents that it cannot import `handlers_ecs`.
|
||||
`foundation/aws/tags/tags.go:24` defines `ManagedByECS` for AMI resolution.
|
||||
`operator/imagecatalog/images.go:611-637` defines the `spinifex-ecs-node` and `spinifex-ecs-node-gpu` system AMIs.
|
||||
`domains/acm/renewal.go:169` is a comment-only reference.
|
||||
`../internal/ecsgw/client.go` is the agent's SigV4 gateway client, with target prefix at :29 and :101.
|
||||
|
||||
Guest agent counterparty (`agents/ecs/agent/`), not inventoried file by file:
|
||||
It reaches the control plane only through signed gateway HTTP (`controlplane.go:18-35`), not NATS.
|
||||
It imports both `handlers/ecs/bus` and the `handlers_ecs` implementation package (`registration.go:37-38`, `controlplane.go:128,142,150`, `taskrun.go:416-419`).
|
||||
That is the ADR-0004 S5 debt: the agent imports a domain implementation package to get wire types.
|
||||
|
||||
## 2. Served AWS actions and protocol
|
||||
|
||||
Protocol: AWS JSON 1.1, with the action taken from the `X-Amz-Target` suffix (`gateway/ecs.go:15-20`, any prefix accepted).
|
||||
Requests use the canonical prefix `AmazonEC2ContainerServiceV20141113` (`gateway/ecs/handler.go:26`).
|
||||
Responses are encoded with `jsonutil` (epoch-second times), except `PollAssignments`, which uses `encoding/json` (RFC3339 times) (`handler.go:131-157`).
|
||||
Errors are `awserrors` codes rendered by the shared ErrorHandler.
|
||||
An action missing from the table returns `InvalidAction` (`gateway/ecs.go:32-36`), and a stub returns `NotImplemented` (501) (`handler.go:43-45`).
|
||||
Request order is: account check (:40-46), bounded body (:48), `ResourceARNs` scope derivation (:54), `checkPolicyResources(r, "ecs", action, …)` (:58), handler with PassRole closure (:69), then encode.
|
||||
Gateway to daemon is a NATS request on `ecs.<Action>` with a 30 s timeout (`handlers/ecs/service_nats.go:12,29-177`), served by queue group `spinifex-workers` (`daemon/daemon.go:1131-1167`).
|
||||
`gateway/operations.go:30-32` publishes Registered = all 39 table keys and Stubbed = 5.
|
||||
`ecsScopes` (`authz.go:57-124`) has an entry for every table action, and `ResourceARNs` fails closed if an entry is missing (:140-144).
|
||||
|
||||
| Action | Status | Resource | Handler (daemon) |
|
||||
|---|---|---|---|
|
||||
| CreateCluster | registered | cluster | `service.go:231` |
|
||||
| DescribeClusters | registered | cluster (+counts from instance/task/service) | `service.go:268` |
|
||||
| ListClusters | registered | cluster | `service.go:338` |
|
||||
| DeleteCluster | registered | cluster (cascade orchestration) | `cluster_teardown.go:24` |
|
||||
| UpdateCluster | stubbed | cluster | — |
|
||||
| PutClusterCapacityProviders | registered | cluster (references capacityprovider) | `capacity_providers.go:16` |
|
||||
| CreateCapacityProvider | registered | capacityprovider | `capacity_providers.go:44` |
|
||||
| DescribeCapacityProviders | registered | capacityprovider | `capacity_providers.go:80` |
|
||||
| DeleteCapacityProvider | registered | capacityprovider | `capacity_providers.go:125` |
|
||||
| RegisterTaskDefinition | registered (PassRole on task/execution role, `actions.go:99`) | taskdefinition | `service.go:416` |
|
||||
| DeregisterTaskDefinition | registered | taskdefinition | `service.go:546` |
|
||||
| DescribeTaskDefinition | registered | taskdefinition | `service.go:571` |
|
||||
| ListTaskDefinitions | registered | taskdefinition | `service.go:586` |
|
||||
| ListTaskDefinitionFamilies | stubbed | taskdefinition | — |
|
||||
| RunTask | registered (PassRole via taskdef lookup, ClientToken) | task (+placement, containerinstance) | `tasks.go:28` |
|
||||
| StartTask | registered (PassRole via taskdef lookup) | task (+containerinstance) | `task_stop.go:53` |
|
||||
| StopTask | registered | task | `task_stop.go:28` |
|
||||
| DescribeTasks | registered | task | `tasks.go:273` |
|
||||
| ListTasks | registered | task | `tasks.go:299` |
|
||||
| SubmitTaskStateChange | registered (AWS action; used by agent) | task | `instances.go:445` |
|
||||
| PollAssignments | registered (non-AWS, agent-only, raw JSON) | task inbox (wire) | `poll.go:37` |
|
||||
| ReportTaskGPU | registered (non-AWS, agent-only) | task | `gpu_report.go:31` |
|
||||
| CreateService | registered (PassRole via taskdef lookup; AZ-rebalancing gate) | service | `services.go:128` |
|
||||
| UpdateService | registered (same) | service | `services.go:209` |
|
||||
| DeleteService | registered | service | `services.go:267` |
|
||||
| DescribeServices | registered | service | `services.go:302` |
|
||||
| ListServices | registered | service | `services.go:326` |
|
||||
| ListServicesByNamespace | stubbed | service | — |
|
||||
| RegisterContainerInstance | registered (agent boot + 30 s liveness) | containerinstance | `instances.go:65` |
|
||||
| DeregisterContainerInstance | registered | containerinstance | `cluster_teardown.go:60` |
|
||||
| UpdateContainerInstancesState | registered | containerinstance | `cluster_teardown.go:103` |
|
||||
| DescribeContainerInstances | registered | containerinstance | `instances.go:127` |
|
||||
| ListContainerInstances | registered | containerinstance | `instances.go:167` |
|
||||
| ProvisionCapacity | registered (non-AWS) | containerinstance capacity (EC2 launch orchestration) | `capacity.go:70` |
|
||||
| PutAccountSetting | stubbed | account setting | — |
|
||||
| ListAccountSettings | stubbed | account setting | — |
|
||||
| TagResource | registered | cross-resource (by ARN) | `tags.go:227` |
|
||||
| UntagResource | registered | cross-resource (by ARN) | `tags.go:254` |
|
||||
| ListTagsForResource | registered | cross-resource (by ARN) | `tags.go:207` |
|
||||
|
||||
Unsupported actions are any ECS action absent from the table, and they return `InvalidAction`.
|
||||
Examples include task sets, `ExecuteCommand`, `UpdateCapacityProvider`, `DeleteTaskDefinitions`, attribute APIs and service deployments/revisions.
|
||||
Three agent-facing actions (`PollAssignments`, `ReportTaskGPU`, `ProvisionCapacity`) are non-AWS shapes on the AWS ECS signing surface.
|
||||
`SubmitTaskStateChange` is an AWS action that our agent uses as its state-report channel.
|
||||
|
||||
## 3. Durable records / KV buckets
|
||||
|
||||
| Bucket | Key shape | Record type | Owning resource | Writers | Readers |
|
||||
|---|---|---|---|---|---|
|
||||
| `ecs-account-<accountID>` (history 1, schema v1, created lazily, `store.go:27-29,207-217`) | `clusters/<name>/meta` | `ClusterRecord` (`records.go:155`) | cluster | CreateCluster `service.go:259`; PutClusterCapacityProviders `capacity_providers.go:36`; tags `tags.go:152`; DeleteCluster removes the subtree `cluster_teardown.go:47` | cluster CRUD; RunTask/StartTask/CreateService existence checks (`tasks.go:36`, `task_stop.go:60`, `services.go:148`) |
|
||||
| same | `clusters/<c>/instances/<id>` | `InstanceRecord` (`records.go:333`) | containerinstance | `upsertInstance` plain Put `instances.go:215`; `recordHeartbeat` `:356`; UpdateContainerInstancesState `cluster_teardown.go:131`; reaper `scheduler.go:390`; tags `tags.go:200`; CAS reserve/release `tasks.go:212`, `task_stop.go:148`, `instances.go:527`; Deregister deletes `cluster_teardown.go:90` | placement (`tasks.go:182`), cluster counts (`service.go:301`), reaper, describe/list |
|
||||
| same | `clusters/<c>/tasks/<taskID>` | `TaskRecord` (`records.go:389`) | task | RunTask `tasks.go:84`; StartTask `task_stop.go:104`; stop/force-stop `task_stop.go:171,199`; recordTaskState `instances.go:409,433`; EIP assign `task_stop.go:284`; ENI rollback `tasks.go:168`; GPU report `gpu_report.go:58`; sweep `tasks_gc.go:98,119`; tags `tags.go:189` | service reconcile (`services.go:589-622`), cluster/instance counts, reaper, sweep |
|
||||
| same | `clusters/<c>/assignments/<ciID>/<taskID>` | `bus.Assign` | task (agent inbox) | `publishAssign` `tasks.go:269` | agent via `PollAssignments` (`poll.go:58`); acked and reclaimed via `poll.go:49,97` |
|
||||
| same | `clusters/<c>/stops/<ciID>/<taskID>` | `bus.StopDirective` | task (agent inbox) | `requestStopTask` `task_stop.go:176` | agent via `PollAssignments` (`poll.go:74`); acked and reclaimed via `poll.go:55,106` |
|
||||
| same | `clusters/<c>/services/<name>` | `ServiceRecord` (`records.go:454`) incl. `Deployments`, `Events` | service | Create/Update/Delete `services.go:198,256,295`; reconcile `services.go:433`; deployment-failure count `instances.go:496`; tags `tags.go:178` | task paths read it for LB and EIP config (`task_stop.go:241,273`) |
|
||||
| same | `taskdef-families/<family>/latest-rev`, `taskdef-families/<family>/revs/<rev>` | `int`, `TaskDefRecord` (`records.go:280`) | taskdefinition | Register `service.go:452,455`; Deregister `:564`; tags `tags.go:167` | `resolveTaskDef` (`service.go:622`) for RunTask/StartTask/Create/UpdateService; gateway PassRole via NATS `DescribeTaskDefinition` |
|
||||
| same | `capacity-providers/<name>` | `CapacityProviderRecord` (`records.go:200`) | capacityprovider | Create `capacity_providers.go:72`; Delete `:142` | Describe only |
|
||||
| `spinifex-ecs-leader` (TTL 60 s, `store.go:31-33,221`) | `scheduler` (`scheduler.go:24`) | kvlease lease | scheduler (orchestration) | `kvlease` in `Scheduler` (`scheduler.go:96-109`) | the same |
|
||||
| `spinifex-ecs-clienttokens` (TTL 15 min, `gateway/ecs/clienttoken.go:22-26`) | `<accountID>.<token>` (`foundation/lifecycle/idempotency/store.go:91-93`, no namespace) | `idempotency` entry with `ecs.RunTaskOutput` | task (currently held by the adapter) | gateway `runTaskWithClientToken` (`clienttoken.go:71-86`) | the same |
|
||||
|
||||
`store.go:23-25`, `:134-138` and `LeaderLeaseKey` describe per-cluster `<accountID>/<clusterName>` lease keys, but the code uses one global key `scheduler`, and `LeaderLeaseKey` has no production caller.
|
||||
No reader outside ECS opens any of these buckets.
|
||||
A Go grep for bucket names and helpers outside `handlers/ecs` and `gateway/ecs` found only RDS's own `AccountBucketName`.
|
||||
`accountteardown` and the agent reach ECS state only through the ECS API (NATS facade or gateway), not through KV.
|
||||
No KV migration is registered for `ecs-account-*`; it runs `migrate.DefaultRegistry.RunKV` at version 1 (`store.go:213`).
|
||||
No production code was found that deletes an `ecs-account-*` bucket.
|
||||
|
||||
## 4. Identity and scope
|
||||
|
||||
The partition is hard-coded to `aws`.
|
||||
The Region is the daemon `config.Region` captured at construction (`daemon/daemon.go:2029`).
|
||||
The account is the SigV4 caller's account, carried on every NATS request.
|
||||
All records live in the caller's per-account bucket, so the account scope is structural.
|
||||
|
||||
| Resource | Identifier | ARN (`records.go:107-135`) | Scope |
|
||||
|---|---|---|---|
|
||||
| cluster | name; omitted means `default` (`conv.go:27-36`, `service.go:233-235`) | `arn:aws:ecs:<region>:<acct>:cluster/<name>` | account + Region; no AZ |
|
||||
| taskdefinition | `family:revision`; revision is `latest-rev + 1` (`service.go:531-541`) | `…:task-definition/<family>:<rev>` | account + Region |
|
||||
| service | name unique per cluster | `…:service/<cluster>/<name>` | account + Region; subnets/SGs stored, not validated |
|
||||
| task | UUIDv4 (`tasks.go:65`, `task_stop.go:85`) | `…:task/<cluster>/<taskID>` | account + Region; AZ is implicit through its container instance and is not stored on the task |
|
||||
| containerinstance | the `InstanceIdentityDocument` string supplied by the agent (in practice the EC2 instance ID), with fallback `ci-<UTC timestamp>` (`instances.go:67-70`) | `…:container-instance/<cluster>/<id>` | account + Region; AZ from the registration attribute (`instances.go:100-101,331`) |
|
||||
| capacityprovider | name | `…:capacity-provider/<name>` | account + Region |
|
||||
|
||||
AWS parity note (verify): AWS mints a UUID container-instance ID that is distinct from `ec2InstanceId`, and AWS expects `instanceIdentityDocument` to be the signed IMDS JSON document.
|
||||
Ours uses the supplied string as both values (`instances.go:276`).
|
||||
Identifier reuse: DeleteCluster removes every record (`cluster_teardown.go:47`), so a cluster name and its child names are reusable immediately.
|
||||
There is no tombstone or identifier-reuse rule (Q-43/Q-44).
|
||||
Task-definition revisions are not reserved against reuse after a concurrent collision (see gap G9).
|
||||
Container instances authenticate as the account's `ecsInstanceRole` (`instancerole.go:12-30`).
|
||||
`PollAssignments`, `SubmitTaskStateChange` and `ReportTaskGPU` check only account scope plus IAM policy on the named instance or task ARN (`authz.go:83-95`).
|
||||
The handlers do not bind the caller to the specific container instance (`poll.go:37-56`, `instances.go:445-475`, `gpu_report.go:31`).
|
||||
|
||||
## 5. Lifecycle paths per resource
|
||||
|
||||
There are no desired or observed generation fields on any ECS record (`records.go:155-493`).
|
||||
The only fencing mechanisms are:
|
||||
- one global scheduler lease (`scheduler.go:96-109`), checked only at the start of each pass (`:170`, `:189`);
|
||||
- CAS (`kv.Update` with revision) on container-instance reservations (`tasks.go:212`, `task_stop.go:148`, `instances.go:527`).
|
||||
|
||||
Every other write is a read-modify-`Put` without revision (`service.go:179-188`).
|
||||
ClientToken idempotency exists only for RunTask (`gateway/ecs/actions.go:203-238`).
|
||||
|
||||
### cluster
|
||||
Create is `CreateCluster` (`service.go:231-264`): get, then a plain Put if absent.
|
||||
A repeat create by name returns the existing record without comparing parameters; AWS also treats it as idempotent (verify).
|
||||
Readiness: the cluster is `ACTIVE` on create (`:250`), synchronously.
|
||||
There is no reconcile loop.
|
||||
Delete is `DeleteCluster` (`cluster_teardown.go:24-53`): it force-stops every task (:43-45), deletes the whole `clusters/<n>/` subtree (:47), and returns a synthetic `INACTIVE` (:51).
|
||||
No durable deletion intent is written.
|
||||
On teardown it is driven by `accountteardown/reapers_ecs.go:53-58`.
|
||||
|
||||
### taskdefinition
|
||||
Create is `RegisterTaskDefinition` (`service.go:416-459`): it validates (`:472-511`), computes `rev = latest + 1` without CAS (`:530-541`), Puts the revision (`:452`), then Puts `latest-rev` (`:455`).
|
||||
There is no client token, and AWS has none for this action.
|
||||
Readiness: the definition is `ACTIVE` synchronously.
|
||||
Delete is `DeregisterTaskDefinition` (`:546-568`), which sets `INACTIVE` and retains the record.
|
||||
There is no hard delete (`DeleteTaskDefinitions` is not served).
|
||||
`resolveTaskDef` (`:622-647`) does not check status, so RunTask, StartTask and CreateService accept an `INACTIVE` revision.
|
||||
AWS refuses new runs and services on an INACTIVE revision after a short window (verify).
|
||||
|
||||
### service
|
||||
Create is `CreateService` (`services.go:128-206`): an `ACTIVE` service with the same name is returned as-is (:159-160), and `ClientToken` is ignored.
|
||||
It Puts the record with a PRIMARY deployment (:196-198), then runs `reconcileService` synchronously on the serving worker (:202).
|
||||
Readiness: the service record is `ACTIVE` immediately.
|
||||
Rollout state moves `IN_PROGRESS` → `COMPLETED` when the primary running count reaches the desired count and no old-deployment tasks remain (`deployments.go:170-196`).
|
||||
The circuit breaker sets `FAILED` after 3 failed starts (`deployments.go:222-243`, `records.go:54`), with optional rollback (`:247-267`).
|
||||
Launch backoff is 15 s to 5 min (`deployments.go:206-216`).
|
||||
Reconcile has two triggers:
|
||||
- the leader loop `ecs/services` (`scheduler.go:143-148`) runs `reconcileAllServices` (`services.go:445-476`) on a wake from `ecs.bus.<acct>.<cluster>.service-reconcile` (`instances.go:472`), a 10 s revisit while unsettled, and a 5 min resync;
|
||||
- `CreateService` and `UpdateService` also reconcile inline on the API worker (`services.go:202`, `:259`).
|
||||
The reconcile launches through `s.RunTask` without a client token (`services.go:495-525`) and stops surplus tasks (`:531-581`).
|
||||
Delete is `DeleteService` (`:267-299`): it requests a stop for each task (:288-289), then sets `INACTIVE` immediately (:291).
|
||||
`ServiceStatusDraining` (`records.go:30`) is never written.
|
||||
There is no `force` or desired-count precondition check, and the INACTIVE record is never pruned.
|
||||
|
||||
### task
|
||||
Create is `RunTask` (`tasks.go:28-93`) or `StartTask` (`task_stop.go:53-113`), in this order:
|
||||
1. Reserve capacity by CAS on the container instance (`tasks.go:180-218` / `task_stop.go:118-153`).
|
||||
2. Build the record in memory (`tasks.go:221-243`; desired `RUNNING`, last `PENDING`).
|
||||
3. For awsvpc, create and attach the ENI through EC2 (`tasks.go:131-176`).
|
||||
4. Make the first durable task write (`tasks.go:84` / `task_stop.go:104`).
|
||||
5. Write the assignment inbox (`tasks.go:249-270`); a failure there is only logged (`:87-89`).
|
||||
Readiness comes from the agent's `SubmitTaskStateChange`, which goes through `recordTaskState` (`instances.go:361-439`).
|
||||
`RUNNING` sets `StartedAt`, registers ELBv2 targets and assigns an EIP when the service asks for it (:415-418).
|
||||
`STOPPED` releases the target, EIP, inboxes, ENI and reservation (:422-436).
|
||||
Stop has two paths:
|
||||
- `StopTask` calls `requestStopTask` (`task_stop.go:162-179`), which sets desired `STOPPED` and posts a stop directive for the agent.
|
||||
- `forceStopTask` (`:186-209`) is used by the cluster delete, deregister, drain and reaper paths; it marks the task `STOPPED` without telling the agent.
|
||||
Repair: the leader `ecs/timers` loop runs `sweepStoppedTasks` (`tasks_gc.go:21-105`).
|
||||
That sweep retries owed ENI releases (at most 20 per pass, 30 s to 15 min backoff) and deletes STOPPED tasks after 1 h.
|
||||
Idempotency: RunTask honours ClientToken through `idempotency.Do` (`gateway/ecs/clienttoken.go:71-86`); a partial placement is finalized and replayed.
|
||||
StartTask has no token, and AWS has none for it (verify).
|
||||
Internal launch from the service reconcile is not idempotent per logical launch.
|
||||
|
||||
### containerinstance
|
||||
Create is `RegisterContainerInstance` (`instances.go:65-124`), which is an upsert with a plain Put (`:195-219`).
|
||||
The agent calls it at boot and on every 30 s heartbeat (`agents/ecs/agent/controlplane.go:22-25`).
|
||||
Readiness: the instance is `ACTIVE` on first registration (`:207`).
|
||||
A zero reported capacity drains it (`Reaped=true`), and returning capacity restores it (`:44-60`).
|
||||
`AgentConnected` is derived from `Status == ACTIVE` (`:278`), not from liveness.
|
||||
Repair: the leader reaper (`scheduler.go:340-396`) marks an instance `DRAINING` with `Reaped=true` after 90 s without `LastSeen`, and force-stops its tasks (`:400-416`).
|
||||
`recordRegister` and `recordHeartbeat` (`instances.go:325-357`) are fed only by bus subjects that nothing publishes.
|
||||
Operator state: `UpdateContainerInstancesState` (`cluster_teardown.go:103-140`) sets the status.
|
||||
`DRAINING` force-stops service tasks immediately (`:160-171`).
|
||||
Delete is `DeregisterContainerInstance` (`:60-97`): it refuses if tasks are active unless `force`, deletes the assignments inbox and the record (the stops inbox is left), and does not touch the EC2 instance.
|
||||
Capacity is provisioned by `ProvisionCapacity` (`capacity.go:70-170`), which calls `RunInstances` directly with no ECS-side intent record and no idempotency.
|
||||
|
||||
### capacityprovider
|
||||
Create is `CreateCapacityProvider` (`capacity_providers.go:44-76`): an existing name returns the existing record without comparing parameters, and the ASG ARN is stored without validation.
|
||||
Readiness: it is `ACTIVE` immediately and inert, with no scheduler coupling (`service.go:74-76`).
|
||||
Delete is a hard key delete (`:125-146`), with no check that a cluster references the provider.
|
||||
|
||||
## 6. NATS subjects and guest/controller messages
|
||||
|
||||
| Subject | Direction / type | Payload | Retry / redelivery |
|
||||
|---|---|---|---|
|
||||
| `ecs.<Action>` ×34 (`service_nats.go:29-177`; subs `daemon.go:1134-1167`) | gateway (and `accountteardown`, gateway PassRole lookup) → daemon, request/reply, queue `spinifex-workers` | aws-sdk-go ECS input/output or `PollAssignments*`, `ReportTaskGPU*`, `ProvisionCapacity*` structs | 30 s timeout. There is no retry in `NATSECSService`, and `natsmsg.NATSRequest` (`foundation/messaging/nats/nats.go:269`) retry behaviour was not verified here. |
|
||||
| `ecs.bus.<acct>.<cluster>.service-reconcile` (`bus/subjects.go:43`) | any worker → leader, core publish | none | Fire-and-forget (`instances.go:472-474`); loss is covered by the 10 s revisit and 5 min resync (`scheduler.go:30-58`). |
|
||||
| `ecs.bus.*.*.instance-register.*`, `…instance-heartbeat.*`, `…task-state.*` (`scheduler.go:236-245`) | subscribed by leader | `bus.RegisterInstance`, `bus.Heartbeat`, `bus.TaskState` | No production publisher (residue). Core NATS with no ack or redelivery, and messages are dropped while there is no leader. |
|
||||
| `ecs.bus.<acct>.<cluster>.assign.<id>` (`bus/subjects.go:30`) | unused | — | Residue; assignment moved to the KV inbox. |
|
||||
| `ec2.CreateNetworkInterface`, `ec2.DeleteNetworkInterface` (`eni.go:64,119`) | ECS → EC2 AWS-action subjects | aws-sdk-go EC2 types | 30 s timeout (`eni.go:19`). NotFound counts as converged (`:148-155`). A failed release is retried by the sweep. |
|
||||
| `ec2v1.InstanceCommandSubject(<instanceID>)` (`eni.go:126-128`) | ECS → owning node, `contracts/ec2/v1` | `EC2InstanceCommand{AttachENI or DetachENI}` | Versioned contract; detach is forced (`:109`). |
|
||||
| `ec2.AllocateAddress`, `ec2.AssociateAddress`, `ec2.DescribeAddresses`, `ec2.DisassociateAddress`, `ec2.ReleaseAddress` via `ec2eip.NewNATSEIPService` (`services.go:81-121`; subjects in `domains/ec2/eip/service_nats.go:26-34`) | ECS → EC2 EIP AWS adapter | aws-sdk-go EC2 types | A failed associate releases immediately (`services.go:91-96`). A failed release is logged, and the fields are cleared anyway (`task_stop.go:296-300`). |
|
||||
| ELBv2 `RegisterTargets` / `DeregisterTargets` via `handlers_elbv2.NewNATSELBv2Service` (`services.go:45-58`) | ECS → ELBv2 AWS adapter | aws-sdk-go ELBv2 types | Best effort and log-only (`task_stop.go:252-255`), with no retry. The exact subject string was not verified. |
|
||||
| `ec2.RunInstances.<type>.<node>` | not used by ECS | — | ECS passes no node, so `RunWorkerInstance` runs the local in-process `instanceService.RunInstances` (`daemon/eks_worker_launch.go:29-51`). |
|
||||
|
||||
Guest protocol: agent ↔ gateway HTTP, SigV4-signed with the instance role through `internal/ecsgw`; it is not NATS.
|
||||
- `RegisterContainerInstance`: boot, then every 30 s as liveness.
|
||||
- `PollAssignments`: delivery is at-least-once. The inboxes are KV keys, and an entry is redelivered on every poll until the agent acks it in `AckTaskIDs` or `AckStopIDs` on the next poll (`poll.go:45-56`), or until the task stops (`instances.go:430-431`, `task_stop.go:204-205`).
|
||||
- `SubmitTaskStateChange`: duplicate reports are tolerated only through edge detection on `prev != status` (`instances.go:415,422`). There is no ordering or monotonicity guard and no check of the reporting instance (`:372-373`).
|
||||
- `ReportTaskGPU`.
|
||||
Bootstrap contract: cloud-config writes `/etc/spinifex-ecs/agent.env` and the gateway CA (`userdata.go:31-58`).
|
||||
|
||||
## 7. Boundary-crossing imports
|
||||
|
||||
ECS → other packages (non-stdlib, non-foundation first):
|
||||
|
||||
| From | Imports | Symbol(s) | Flag |
|
||||
|---|---|---|---|
|
||||
| `handlers/ecs/service.go:19`, `instancerole.go:6` | `handlers/iam` | `SystemInstanceRoleEnsurer` (`service.go:92`), `EnsureSystemInstanceProfile` (`instancerole.go:38`), `ConvergeSystemRolePolicy` (`:46`) | IAM |
|
||||
| `handlers/ecs/services.go:20` | `handlers/elbv2` | `NewNATSELBv2Service().RegisterTargets/DeregisterTargets` | other domain's AWS adapter used as an internal API |
|
||||
| `handlers/ecs/services.go:18` | `domains/ec2/eip` | `NewNATSEIPService` (Allocate/Associate/Describe/Disassociate/Release) | EC2 / network; AWS adapter as internal API |
|
||||
| `handlers/ecs/eni.go:13` | `contracts/ec2/v1` | `EC2InstanceCommand`, `EC2CommandAttributes`, `AttachENIData`, `DetachENIData`, `InstanceCommandSubject` | EC2 instance and network (versioned contract) |
|
||||
| `handlers/ecs/eni.go:64,119` | string subjects `ec2.CreateNetworkInterface` / `ec2.DeleteNetworkInterface` | — | network; EC2 AWS-action subject used as an internal API |
|
||||
| `handlers/ecs/capacity.go:11-12` | `domains/ec2/ebs/metadata`, `domains/ec2/instancetypes` | `RootDeviceName`; `IsGPUTypeName`, `GPUVendorForType` | EC2 |
|
||||
| `handlers/ecs/placement.go:9` | `domains/ec2/instancetypes` | `HotPlugENISlotsForType` | EC2 |
|
||||
| `handlers/ecs/service.go:16`, `capacity.go:10`, `eni.go:12`, `services.go:15-17` | aws-sdk-go `service/ec2`, `service/elbv2` | `RunInstancesInput`, `DescribeImages*`, ENI/EIP/target types | EC2 / ELBv2 types |
|
||||
| `handlers/ecs/*` | foundation | `messaging/nats.NATSRequest`, `state/kvlease`, `state/kvutil`, `state/migrate`, `lifecycle/reconciler`, `aws/errors`, `aws/arn`, `aws/ami`, `aws/tags` | — |
|
||||
| `gateway/ecs/authz.go:11-12` | `ingress/aws/bodyscope`, `ingress/aws/query` | `bodyscope.Parse`, `query.MaxSliceLen` | ingress |
|
||||
| `gateway/ecs/clienttoken.go:13` | `foundation/lifecycle/idempotency` | `OpenStore`, `Do`, `ParamHash`, `ErrParamMismatch`, `ErrUnavailable` | — |
|
||||
| `daemon/ecs_deps.go:8` | `github.com/mulgadc/bluebottle/pkg/masterkey` | `ReadShared` (:39) | Bluebottle (daemon wiring only; `handlers/ecs` itself does not import Bluebottle) |
|
||||
| `daemon/ecs_deps.go:10,50-52` | `handlers/iam` | `NewIAMServiceImpl`, `*IAMServiceImpl` | IAM |
|
||||
| `daemon/ecs_deps.go:29,32` | EKS-named daemon helpers | `resolveGatewayBaseURL` (`daemon/eks_deps.go:164`), `RunWorkerInstance` (`daemon/eks_worker_launch.go:29`) | EC2 instance launch shared with EKS |
|
||||
|
||||
Packages that import ECS:
|
||||
|
||||
| Importer | Imports | Symbol(s) |
|
||||
|---|---|---|
|
||||
| `gateway/ecs/actions.go:12` | `handlers/ecs` | `NewNATSECSService`, `ECSService`, `ProvisionCapacityInput`, `PollAssignmentsInput`, `ReportTaskGPUInput` |
|
||||
| `gateway/ecs/authz.go:10` | `handlers/ecs` | `ClusterShortName`, `ServiceShortName`, `TaskShortID`, `ContainerInstanceShortID`, `CapacityProviderShortName`, `ParseTaskDefRef`, `ResourceARNSegment`, `ClusterARN`, `ServiceARN`, `TaskARN`, `ContainerInstanceARN`, `CapacityProviderARN`, `TaskDefARN`, `TaskDefRefARN` |
|
||||
| `gateway/ecs.go:10`, `gateway/operations.go:7` | `gateway/ecs` | `Actions`, `ResourceARNs`, `RawJSONActions`, `WriteJSONResponse`, `WriteRawJSONResponse`, `StubbedActionNames` |
|
||||
| `daemon/daemon.go:74`, `daemon/ecs_deps.go:9` | `handlers/ecs` | `Service`, `Scheduler`, `NewService`, `WithDeps`, `InitLeaderBucket`, `NewScheduler`, `Deps` |
|
||||
| `accountteardown/reapers_ecs.go:9` | `handlers/ecs` | `NewNATSECSService`, `ECSService` (`ListClusters`, `DeleteCluster`) |
|
||||
| `agents/ecs/agent/{registration,controlplane,taskrun}.go` | `handlers/ecs` (implementation package) | `AttrInstanceType`, `AttrAvailabilityZone`, `PollAssignmentsInput/Output`, `ReportTaskGPUInput`, `ContainerGPUReport` |
|
||||
| `agents/ecs/agent/{agent,controlplane,reconcile,registration,taskrun}.go` | `handlers/ecs/bus` | `Assign`, `StopDirective`, `TaskState`, `ContainerStatus`, `InstanceCapacity`, status constants |
|
||||
|
||||
Explicit flags:
|
||||
- **IAM / PassRole.** PassRole is enforced only in the gateway (`gateway/ecs.go:65-67`, through `gw.checkPassRole` at `gateway/gateway.go:645-648` with `iam:PassedToService=ecs-tasks.amazonaws.com`).
|
||||
RegisterTaskDefinition checks the role ARNs in the body (`actions.go:99`).
|
||||
RunTask, StartTask, CreateService and UpdateService first fetch the task definition through the ECS NATS facade (`actions.go:44-56`).
|
||||
`ProvisionCapacity` attaches the system `ecsInstanceRole` instance profile (`capacity.go:103,141`) and launches through `RunWorkerInstance`, which goes straight to `d.instanceService.RunInstances` (`daemon/eks_worker_launch.go:51`).
|
||||
That bypasses the EC2 gateway's PassRole closure (`gateway/ec2.go:199`).
|
||||
`handlers/sts/assume_role.go:52-55` duplicates `ecsInstanceRole`, which is implicit coupling with no import.
|
||||
- **Quota.** `handlers/ecs` has no quota import.
|
||||
`ProvisionCapacity` appears to bypass `gw.Quota.EnforceLaunch` and `ChargeLaunch` (`gateway/ec2.go:196-205`), because it calls the in-process instance service (verify whether `instanceService.RunInstances` enforces quota itself).
|
||||
ECS resources themselves have no quota.
|
||||
- **EC2 instance.** `RunInstances` (via `Deps.RunInstances`), `DescribeImages` (via `Deps.Images` = `d.imageService`), `instancetypes`, EBS root-device metadata, and the `contracts/ec2/v1` instance command for ENI hot-plug.
|
||||
- **Network.** ENI create/delete through EC2 AWS subjects, EIP through the `domains/ec2/eip` NATS adapter, subnets and SGs passed through unvalidated.
|
||||
- **Bluebottle.** Only `masterkey.ReadShared` in `daemon/ecs_deps.go:39`, used to build the IAM service.
|
||||
Without it, `ProvisionCapacity` is disabled and the scheduler refuses leadership (`scheduler.go:226-231`).
|
||||
|
||||
## 8. Known gaps against Q-78 / ADR-0003 (observed)
|
||||
|
||||
- **G1. No generations.** No ECS record has a desired or observed generation (`records.go:155-493`). Reconcilers act on whole records with plain Put (`service.go:179-188`).
|
||||
- **G2. Realization before intent.** RunTask and StartTask reserve capacity and create/attach an ENI before the task record exists (`tasks.go:66-84`, `task_stop.go:86-104`). A crash in that window leaves a reservation (`PlacedTasks` naming an unknown task) and possibly an ENI with no owning record. No reconciler targets orphan reservations; `tasks_gc.go` handles only owed ENI release on STOPPED tasks.
|
||||
- **G3. Lost-update window on instance reservations.** Reservations use CAS (`tasks.go:212`, `task_stop.go:148`, `instances.go:527`), but the same key is rewritten with a plain read-modify-Put every 30 s by agent registration (`instances.go:195-215`). The same happens in UpdateContainerInstancesState (`cluster_teardown.go:131`), the reaper (`scheduler.go:390`) and tags (`tags.go:200`), so a CAS commit between their Get and Put can be overwritten. `handlers/ecs/task_accounting_test.go` (`TestDescribeContainerInstances_StalePlacedTasksDoNotInflateCount`) acknowledges stale `PlacedTasks`.
|
||||
- **G4. Unfenced concurrent writers.** The leader lease is checked only at pass start (`scheduler.go:170,189`). KV writes are not fenced by the lease. API workers also run `reconcileService` inline (`services.go:202,259`) while the leader runs `reconcileAllServices` (`scheduler.go:177`), so two writers can launch through `RunTask` for the same service. This is the observed code shape; no test was found covering it.
|
||||
- **G5. Cascade delete removes cleanup ownership.** `DeleteCluster` (`cluster_teardown.go:43-47`) deletes the whole subtree, including STOPPED tasks whose `ENIReleased=false` or whose EIP or target cleanup failed. That removes the only record of owed cleanup, which Q-78 forbids. There is no deletion intent or incomplete-cleanup state. AWS refuses DeleteCluster while services, tasks or registered instances exist (verify against AWS docs and a test).
|
||||
- **G6. Force-stop does effects before persisting.** `forceStopTask` releases the EIP and ENI before persisting STOPPED (`task_stop.go:197-199`). A failed EIP release is logged and its allocation ID cleared (`:296-300`), so it cannot be retried. Target deregistration is best effort with no retry (`:252-255`). `forceStopTask` also posts no stop directive, so whether a live agent stops its containers is unverified.
|
||||
- **G7. EIP before record.** `assignTaskPublicIP` allocates and associates the EIP before persisting its allocation ID (`task_stop.go:277-286`).
|
||||
- **G8. Silent reservation leak.** `releaseReservation` gives up silently after 5 CAS conflicts and returns nil (`instances.go:506-531`), which leaks capacity with no reconciler.
|
||||
- **G9. Non-atomic revision allocation.** `RegisterTaskDefinition` computes the revision as read + 1 with no CAS (`service.go:430,531-541`). It writes the revision before `latest-rev` (`:452-455`), so concurrent registers can produce the same revision and overwrite it.
|
||||
- **G10. No guard on task-state reports.** `recordTaskState` sets `LastStatus` unconditionally (`instances.go:372-373`), so a late RUNNING after STOPPED regresses the state. The reporting `InstanceID` is ignored.
|
||||
- **G11. DeleteService is immediate.** It sets INACTIVE directly, with no DRAINING (`services.go:291`; `records.go:30` is unused). It has no `force` or desired-count check, AWS requires one (verify), and the INACTIVE record is never pruned.
|
||||
- **G12. Create idempotency without comparison.** CreateService ignores `ClientToken` and returns an existing ACTIVE service without comparing parameters (`services.go:159-160`). CreateCapacityProvider (`capacity_providers.go:60-61`) and CreateCluster (`service.go:246-262`) behave the same way.
|
||||
- **G13. Retry bookkeeping in the adapter.** The RunTask idempotency store lives in the AWS adapter (`gateway/ecs/clienttoken.go:22-56`). ADR-0003 S3 says retry bookkeeping is not the adapter's.
|
||||
- **G14. ProvisionCapacity.** It has no durable intent and no idempotency, and it silently caps count at 10 (`capacity.go:89-91,158`). Launched instances are linked only by the tag `spinifex:ecs-cluster` (`:150`).
|
||||
- **G15. Liveness derived from status.** Container-instance `AgentConnected` is derived from Status, not liveness (`instances.go:278`). Registration makes an instance ACTIVE immediately (`:207`).
|
||||
- **G16. Residue subscriptions.** The scheduler subscribes to three bus subjects that nothing publishes (`scheduler.go:236-245`).
|
||||
- **G17. AWS adapters as internal APIs.** Account teardown uses the ECS facade (`accountteardown/reapers_ecs.go:21`), and gateway PassRole resolution calls `DescribeTaskDefinition` over NATS (`actions.go:48`). ECS calls the ELBv2 and EIP adapters and the `ec2.*NetworkInterface` subjects (`services.go:46,82`, `eni.go:64,119`). ADR-0003 S4 and ADR-0004 S4 forbid an AWS adapter used as an internal service API.
|
||||
- **G18. Incomplete account teardown.** It reaps only clusters (`reapers_ecs.go:20-22`). Task definitions, capacity providers and the `ecs-account-*` bucket remain; no deleting code was found.
|
||||
- **G19. No pagination.** List APIs ignore `nextToken` and `maxResults` (`service.go:338,586`, `tasks.go:299`, `services.go:326`, `instances.go:167`). Q-89 requires listing tests that exceed one page.
|
||||
- **G20. Status not checked on launch.** RunTask, StartTask and CreateService accept an INACTIVE task-definition revision (`service.go:622-647`), but AWS refuses (verify).
|
||||
- **G21. Capacity providers not taggable.** Capacity-provider ARNs are rejected by the tag parser (`tags.go:52-71`), although create accepts tags.
|
||||
- **G22. Delete leaves stop inboxes.** `DeregisterContainerInstance` deletes the assignments inbox but not the stops inbox (`cluster_teardown.go:87-90`).
|
||||
|
||||
## 9. First-resource candidates (ranked)
|
||||
|
||||
1. **taskdefinition.**
|
||||
Size: about 300 lines in `service.go:416-706` plus about 180 in `conv.go:70-265`, plus `TaskDefRecord` and `ContainerDef` (`records.go:211-328`).
|
||||
Private state: `taskdef-families/*` only.
|
||||
Dependencies: none at the handler. PassRole is checked in the gateway on role ARN strings, so extraction does not require deciding IAM authority. It has no EC2, network or Bluebottle dependency.
|
||||
Consumers needing a narrow query: task launch, service create/update (`resolveTaskDef`), gateway PassRole lookup, and tags.
|
||||
Tests: `handlers/ecs/taskdef_validation_test.go`, `service_test.go` (`RegisterTaskDefinition_*`, `ListTaskDefinitions_*`, `DescribeTaskDefinition_Unknown`), `cluster_teardown_test.go` (`DeregisterTaskDefinition_*`), `conv_test.go`, `records_test.go`, `tags_test.go` (`TaskDefinitionRoundTrip`).
|
||||
Known gaps: G9 and G20.
|
||||
Assessment: the best first slice. Its `resolveTaskDef` edge also gives the ADR-0004 rollout item 3 capability seam.
|
||||
2. **capacityprovider.**
|
||||
Size: about 220 lines in `capacity_providers.go` plus records.
|
||||
Private state: `capacity-providers/*`.
|
||||
Dependencies: none; the provider is inert, and the ASG ARN is opaque.
|
||||
Prerequisite split: `PutClusterCapacityProviders` must move to `cluster`.
|
||||
Tests: `handlers/ecs/capacity_providers_test.go`.
|
||||
Gaps: G12, G21, and an unchecked delete.
|
||||
Assessment: the smallest and safest slice, with no IAM, EC2 or Bluebottle dependency. It has almost no consumers, so it demonstrates little about cross-resource capabilities.
|
||||
3. **cluster.**
|
||||
Size: CRUD of about 180 lines (`service.go:231-411`) plus `PutClusterCapacityProviders`.
|
||||
Private state: `clusters/<n>/meta`.
|
||||
Dependencies: counts read instance, task and service records (`service.go:300-335`), and its existence is checked by task and service launch. The delete cascade is orchestration over every child (G5).
|
||||
Tests: `service_test.go` (`CreateCluster_Idempotent`, `DescribeClusters_*`, `ListClusters_*`), `cluster_teardown_test.go` (`DeleteCluster_*`), `capacity_providers_test.go`, `tags_test.go`, `accountteardown/reapers_ecs_test.go`.
|
||||
Assessment: a CRUD-only slice is feasible without IAM or Bluebottle. DeleteCluster has to stay as domain-root orchestration until the children have owners.
|
||||
4. **containerinstance.**
|
||||
Size: about 450 lines across `instances.go`, part of `cluster_teardown.go` and `placement.go`.
|
||||
Private state: `instances/*` plus the reservation fields, which placement and task CAS-mutate.
|
||||
Dependencies: the agent wire contract, EC2 (`instancetypes`; `RunInstances` via `ProvisionCapacity`) and IAM (`ecsInstanceRole`).
|
||||
Tests: `instance_recovery_test.go`, `instance_list_status_test.go`, `placement_test.go`, `eni_placement_test.go`, `capacity_test.go`, `scheduler_test.go`, `task_accounting_test.go`, `service_test.go` (GPU/register/heartbeat/reaper).
|
||||
Assessment: G3 must be characterized first, and reservation ownership needs a decision. It is not a first slice.
|
||||
5. **service.**
|
||||
Size: about 1,050 lines (`services.go`, `deployments.go`).
|
||||
Dependencies: task launch and stop, taskdefinition, ELBv2 targets and EC2 EIP. Its reconcile runs on both the leader and the API workers (G4).
|
||||
Tests: `services_test.go`, `deployments_test.go`, `revisit_test.go`, `eip_test.go`, `services_eip_nats_test.go`, gateway `actions_test.go` and `azrebalancing_test.go`.
|
||||
Assessment: it depends on the task and taskdefinition owners existing.
|
||||
6. **task.**
|
||||
Size: about 1,300 lines (`tasks.go`, `task_stop.go`, `tasks_gc.go`, `poll.go`, `eni.go`, `netcfg.go`, `gpu_report.go`, part of `instances.go`).
|
||||
Dependencies: EC2 ENI and EIP, ELBv2, container-instance reservations, the agent contract, the ClientToken store and gateway PassRole.
|
||||
Tests: `eni_test.go`, `eni_placement_test.go`, `tasks_gc_test.go`, `task_accounting_test.go`, `poll_test.go`, `gpu_report_test.go`, `service_test.go` (`RunTask_*`, `RecordTaskState_*`), gateway `clienttoken_test.go`, `actions_test.go`, and e2e `tests/e2e/ecs/ecs_test.go` and `tests/e2e/gpu/ecs_gpu_test.go` in the mulga root.
|
||||
Assessment: it carries most of G2, G3, G6, G7, G8 and G10, so it should be extracted last.
|
||||
@@ -1,348 +0,0 @@
|
||||
# ADR-0004 source inventory — EKS
|
||||
|
||||
Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`.
|
||||
It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it.
|
||||
Items marked "verify" or "uncertain" are unconfirmed.
|
||||
Gaps record observed behaviour, not target behaviour.
|
||||
|
||||
Add-on update: every add-on row, the add-on boundary edges (Section 7.1) and the add-on gaps were re-read from the code at `18794e017`, after the add-on owner and guest wire contract were extracted.
|
||||
That update is a code read only; it is not verified against a live system.
|
||||
Everything else still describes `6c50349bd`, and line numbers outside the add-on rows may have drifted.
|
||||
The add-on lifecycle contract and its gap list are in `docs/package-boundary/eks-addon-lifecycle.md`; the proposed lifecycle correction is in `docs/package-boundary/eks-addon-lifecycle-design.md`.
|
||||
|
||||
|
||||
Repo: `spinifex` submodule, branch `refactor/spinifex-package-boundaries`; all paths are relative to `spinifex/spinifex/` unless stated.
|
||||
Binding inputs: ADR-0001, ADR-0002, ADR-0003, ADR-0004 and Q-78 as on `origin/main` of mulga.
|
||||
Proposed resource vocabulary (ADR-0004 S1): `cluster`, `nodegroup`, `addon`, `access`.
|
||||
Line numbers are from the working tree at time of writing; "uncertain" marks inferences not verified against a test or live system.
|
||||
|
||||
## 1. File table
|
||||
|
||||
Production `.go` files only; `_test.go` excluded.
|
||||
Line counts are `wc -l`.
|
||||
|
||||
### 1.1 `handlers/eks/` (single package `handlers_eks`, 37 files, 12,310 lines at `6c50349bd`)
|
||||
|
||||
The add-on files `addon_catalog.go`, `addon_installer.go` and `addon_state.go` have left this package; Section 1.1a lists where they went.
|
||||
The add-on rows below are at `18794e017`.
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `access_entry.go` | 255 | resource owner (`access`) | AccessEntry record CRUD/CAS, supported access-policy catalogue, scope validation, ARN/record projection. |
|
||||
| `addon_status_reconciler.go` | 32 | temporary residue (report relay hosted on `cluster`) | `applyAddonStatusReport` (`:22`) converts a `contracts/eks/v1` `AddonStatusReport` into `addon.Report` and calls the reconciler-declared `addonReports` capability (`:13-15`, bound to `*addon.Owner` at `:17`); `report.Version` and `report.TS` are dropped. Removal: add-on delivery observation gets its own host (Section 7.1, edge E3). |
|
||||
| `addons.go` | 287 | AWS protocol adapter (`addon`) + internal guest route | Six AWS add-on actions validate against `addon.Lookup`, check cluster existence (`acctKVForCluster`), call `*addon.Owner` and project `addon.Record` to SDK shapes (`addonRecordToAWS` `:222`); `UpdateAddon` builds the `Successful` update with `Id = rec.Arn` (`:192-197`). Internal `ListStagedAddonManifests` (`:117`) copies `addon.Manifest` into `eksv1.StagedAddonManifest`. `addons()` (`:265`) builds a new `Owner` per call; `addonBucket` (`:278`) opens the account bucket for the staging installer. |
|
||||
| `awsgw_endpoint.go` | 42 | resource owner (`cluster`) | Builds the per-cluster OIDC issuer URL. |
|
||||
| `claim.go` | 31 | resource owner (`nodegroup`) | CAS-create helper; only caller is `nodegroup.go:123`. |
|
||||
| `clienttoken.go` | 57 | resource owner (`cluster`) | CreateCluster `clientRequestToken` store (`spinifex-eks-clustertokens`). |
|
||||
| `cluster_reconciler.go` | 1209 | resource owner (`cluster`), mixed | Cluster readiness/health, CP restart, member replacement, etcd reset; also hosts the add-on status subscription for `addon` (`addonReports` field `:171-176`, `WithAddonStatusSource` `:264-269`, plain subscribe `:447-458`), wired in `service_impl.go:2160-2163` (add-on rows at `18794e017`). |
|
||||
| `cluster_state.go` | 515 | resource owner (`cluster`) | `ClusterMeta` record, status enum, CAS helpers, delete-reap bookkeeping, `DeleteClusterPrefix` (`:493`, which also erases nodegroup, access and add-on record and manifest keys by prefix without calling the add-on owner). |
|
||||
| `cp_vpc.go` | 128 | resource owner (`cluster`) | EKS-specific managed CP VPC realization via `domains/network/systemvpc`. |
|
||||
| `eks_billable_reaper.go` | 133 | resource owner (`cluster`) | Node-local GC backstop for CP VMs whose meta is gone; cites `ADR-0003:S2` as the general lifecycle envelope, while the resource's own contract remains incomplete. |
|
||||
| `eks_deleting_reaper.go` | 159 | resource owner (`cluster`) | Re-drives DELETING clusters with backoff and exhaustion. |
|
||||
| `igw.go` | 32 | resource owner (`cluster`) | Legacy-topology IGW ensure/delete via `systemvpc`. |
|
||||
| `k3s_ha_control_plane.go` | 628 | resource owner (`cluster`), mixed | CP placement, spread, replacement, fresh CP; also contains `natsHostScheduler` (`:461-628`), a NATS fan-out over `contracts/cluster/v1` that is a capability implementation and should move to the composition root or the scheduler owner. |
|
||||
| `k3s_server_role.go` | 51 | resource owner (`cluster`) | Ensures the system CP instance profile through `handlers_iam.EnsureSystemInstanceProfile` (identity dependency). |
|
||||
| `k3s_server_vm.go` | 700 | resource owner (`cluster`) + guest wire contract | CP VM/ENI launch/terminate and AMI lookup; `buildK3sUserData` (`:524+`) is the CP guest boot contract; `lookupEKSGPUNodeAMI` is used by `nodegroup`. |
|
||||
| `nats_bootstrap.go` | 347 | resource owner (`cluster`) + guest wire contract | Bootstrap subject vocabulary and envelope (`:47-57`) are a guest contract; persistence of token/kubeconfig/JWKS/CA is cluster state. |
|
||||
| `nlb.go` | 591 | resource owner (`cluster`) | EKS-specific NLB realization via an ELBv2-shaped capability; also `ReapLBCLoadBalancers` (`:517`) for LBC-created ALBs in the customer VPC. |
|
||||
| `nodegroup.go` | 1409 | resource owner (`nodegroup`), mixed | Nodegroup CRUD, scaling CAS, worker launch/terminate; also creates customer IAM roles/instance profiles (`:768-838`), stages a GPU add-on (`stageGPUDeviceAddon` `:405-416`, called from `:473`, through `addons().EnsureGPUDevicePlugin`; at `18794e017`), and decrypts the cluster join token (`:1160`, reads `cluster` secrets). |
|
||||
| `nodegroup_userdata.go` | 171 | resource owner (`nodegroup`) + guest wire contract | Worker cloud-init (K3S_TOKEN, registry mirrors) is the worker guest boot contract. |
|
||||
| `oidc_keypair.go` | 202 | resource owner (`cluster`) | Per-cluster OIDC signing key (encrypted) and JWKS. |
|
||||
| `private_endpoint.go` | 88 | resource owner (`cluster`) | Customer-VPC private-endpoint ENI. |
|
||||
| `reconciler_registry.go` | 183 | resource owner (`cluster`) | In-process registry of per-cluster reconciler goroutines. |
|
||||
| `recovery_directive.go` | 144 | resource owner (`cluster`) + guest wire contract | Per-member etcd recovery directive; `RecoveryDirective` is read by the on-VM recovery agent. |
|
||||
| `restore_snapshot.go` | 355 | resource owner (`cluster`) | Operator-only total-loss DR path; reads etcd snapshots through `providers/objectstore`. |
|
||||
| `security_groups.go` | 505 | resource owner (`cluster`), mixed | Cluster SG realization in system and customer VPCs; `EnsureNodegroupSGRules` (`:351`) is nodegroup realization. |
|
||||
| `service.go` | 73 | temporary residue | `EKSService`, one method per AWS action plus internal methods; it is the NATS RPC surface between gateway and daemon and is also consumed by `accountteardown` and the operator CLI; removal condition: per-resource commands behind `domains/eks/awsapi` registration (ADR-0004 S3) with any remaining process boundary versioned under `contracts/eks/v1`. |
|
||||
| `service_impl.go` | 2518 | mixed: cross-resource orchestration + resource owners | Split: CreateCluster launch sequence and `purgeClusterInfra` (`:504-1013`, `:1268-1452`) are cluster realization orchestration; cluster read/delete (`:1041-1162`) is `cluster`; access-entry actions (`:1527-1816`) are `access`; nodegroup wrappers (`:1465-1525`) are `nodegroup`; Tag actions (`:1842-1992`) span cluster and nodegroup; IdP stubs (`:1818-1832`); `DesiredDNSChanges` (`:2143`) is a DNS projection; `EKSServiceDeps` and capability interfaces (`:39-245`). |
|
||||
| `service_nats.go` | 190 | temporary residue | Gateway-side NATS client implementing `EKSService` over `eks.<Action>` subjects; removal condition as for `service.go`. |
|
||||
| `state_report.go` | 75 | guest/controller wire contract (`cluster`) | `eks.state.*` subject and `ServerStateReport`; the add-on subject and report moved to `contracts/eks/v1`, leaving only `unmarshalAddonStatusReport` (`:69`), which decodes into `eksv1.AddonStatusReport` (add-on rows at `18794e017`). |
|
||||
| `store.go` | 255 | resource owner (`cluster`), mixed | Bucket names, leader bucket, migrations, key helpers for every resource (the add-on key helpers have moved to `domains/eks/addon/store.go`; 217 lines at `18794e017`); split per-resource key helpers into each owner; `AccountWatchBuckets` (`:215`) is effectively a projection consumed by DNS. |
|
||||
| `token_review.go` | 128 | resource owner (`access`) | Authenticates a `get-token` bearer against STS and maps to AccessEntry groups; `ResolveTokenReview` is called from the gateway process and opens EKS KV there. |
|
||||
| `token_webhook.go` | 57 | guest/controller wire contract | `eks.VerifyToken` subject and request/response consumed by `runtime/roles/awsgw`. |
|
||||
| `types.go` | 131 | mixed: `nodegroup` + `access` + residue | `NodegroupRecord` (nodegroup), `AccessEntryRecord`/`AccessScope` (access); `ClusterRecord` (`:19`) and `OIDCProviderConfigRecord` (`:125`) have no production readers or writers (temporary residue; removal: delete). |
|
||||
| `volume_reclaim.go` | 81 | resource owner (`cluster`) | Surfaces (never deletes) CSI-created EBS volumes at cluster delete. |
|
||||
|
||||
### 1.1a Add-on owner and wire contract (at `18794e017`)
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `domains/eks/addon/record.go` | 52 | resource owner (`addon`) | `Record` and `Manifest` (persisted JSON fields unchanged from the former `AddonRecord`), `Status` (six values; `UPDATE_FAILED` and `DELETE_FAILED` absent), `ErrNotFound`, `ErrExists`. Moved from `addon_state.go`. |
|
||||
| `domains/eks/addon/store.go` | 237 | resource owner (`addon`) | `Prefix`, `Key`, `ManifestKey`, `Get`, `List`, `ListManifests` (exported) and `put`, `putManifest`, `deleteManifest`, `deleteRecord`, `casUpdate` (private). `put` is unconditional, so create is get-then-put. Moved from `addon_state.go` and the add-on key helpers in `store.go`. |
|
||||
| `domains/eks/addon/owner.go` | 205 | resource owner (`addon`) | `Owner` is the sole writer: `Create` (`:48`), `EnsureGPUDevicePlugin` (`:79`), `Update` (`:94`), `Delete` (`:122`), `ApplyReport` (`:140`), `markFailed` (`:171`, always `CREATE_FAILED`), `nextStatus` (`:186`). `Desired` and `Change` are the validated inputs. Moved from `addons.go`, `addon_status_reconciler.go` and the former `markAddonFailed`/`nextAddonStatus`. |
|
||||
| `domains/eks/addon/delivery.go` | 77 | resource owner (`addon`) | `Installer` interface, `StagingInstaller` (writes or deletes the manifest sub-key), `Bucket`, and the owner's own `Phase`/`Report` (no version or timestamp). Moved from `addon_installer.go`. |
|
||||
| `domains/eks/addon/catalog.go` | 122 | resource owner (`addon`), read-only catalogue | Static bundles `aws-load-balancer-controller` 2.11.0, `argocd` 3.0.23, `aws-ebs-csi-driver` 1.40.1, hidden `nvidia-device-plugin` 0.17.4 and hidden `spinifex-noop` 0.1.0; `Lookup`, `Specs`, `ValidateCatalog`. Moved from `addon_catalog.go`. |
|
||||
| `contracts/eks/v1/addon_status.go` (repo root) | 37 | guest/controller wire contract | `AddonStatusSubject`, `AddonDeliveryPhase` (`applied`, `ready`, `failed`), `AddonStatusReport` (`addon`, `version`, `phase`, `message`, `ts`). Moved from `state_report.go`. |
|
||||
| `contracts/eks/v1/addon_manifest.go` (repo root) | 18 | guest/controller wire contract | `StagedAddonManifest` and `InternalAddonsResponse`; the deployed HTTP body uses Go field names because the REST-JSON marshaller ignores the `json` tags, which `addon_manifest_test.go` pins. Moved from `addon_installer.go`, `gateway/eks` (`internalAddonsOutput`) and `agents/eks/gatewayfetch` (`stagedAddon`, `internalAddonsResponse`). |
|
||||
| `contracts/eks/v1/README.md` | n/a | contract record | Compatibility boundary: a generation-aware protocol needs a successor version, not a v1 edit. |
|
||||
|
||||
Guest counterparty: `scripts/images/eks-node/mulga-eks-addon-sync.sh` (417 lines) runs on the primary server only, renders per add-on name, GCs rendered files whose name is no longer staged with `kubectl delete --wait=false`, ignores `configurationValues`, and builds the report with `printf`.
|
||||
|
||||
Additional dead code observed in `store.go`: `OIDCProviderKey` (`:95`) and `EventKey` (`:132`) have no production callers.
|
||||
|
||||
### 1.2 `gateway/eks.go` and `gateway/eks/` (15 files, 1,473 lines)
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `gateway/eks.go` | 379 | AWS protocol adapter | REST-JSON route table (`:27-219`), dispatch, authorization ordering, `iam:PassRole` check (`:319-347`), caller identity extraction. |
|
||||
| `gateway/eks/access.go` | 89 | AWS protocol adapter | Unmarshal and NATS call for access actions. |
|
||||
| `gateway/eks/addons.go` | 60 | AWS protocol adapter | Same for add-on actions. |
|
||||
| `gateway/eks/authz.go` | 310 | AWS protocol adapter | Per-action resource ARN derivation (`eksScopes` `:45-109`); imports `handlers_eks.PrincipalARNHash`. |
|
||||
| `gateway/eks/cluster.go` | 66 | AWS protocol adapter | Cluster action wrappers. |
|
||||
| `gateway/eks/handler.go` | 47 | AWS protocol adapter | JSON response/error writers. |
|
||||
| `gateway/eks/internal_addons.go` | 33 | guest/controller wire contract | `GET /clusters/{c}/internal-addons/{acct}` for the addon-sync agent; calls `handlers_eks.NewNATSEKSService(...).ListStagedAddonManifests` and wraps the result as `eksv1.InternalAddonsResponse` (at `18794e017`). |
|
||||
| `gateway/eks/internal_authz.go` | 129 | AWS protocol adapter + temporary residue | Principal gate for internal routes; `lookupClusterMeta` (`:102-129`) reads `eks-account-*` and `ClusterMeta` directly from the gateway process; removal: owner-published membership query. |
|
||||
| `gateway/eks/internal_publish.go` | 91 | guest/controller wire contract | HTTP-to-NATS relay of bootstrap/state/addon channels; the `addon` channel relays the raw payload to `eksv1.AddonStatusSubject` (`:75-76`, at `18794e017`). |
|
||||
| `gateway/eks/internal_recovery.go` | 37 | guest/controller wire contract | `GET .../internal-recovery/{acct}/{instance}`. |
|
||||
| `gateway/eks/nodegroup.go` | 65 | AWS protocol adapter | Nodegroup wrappers. |
|
||||
| `gateway/eks/oidc.go` | 45 | AWS protocol adapter | IdP-config wrappers (handlers are stubs). |
|
||||
| `gateway/eks/passrole.go` | 43 | AWS protocol adapter | Extracts passed role ARNs for CreateCluster, CreateNodegroup, CreateAddon and UpdateAddon. |
|
||||
| `gateway/eks/tags.go` | 35 | AWS protocol adapter | Tag wrappers. |
|
||||
| `gateway/eks/token_review.go` | 51 | guest/controller wire contract | `POST /clusters/{c}/token-review` for the in-VM token webhook. |
|
||||
|
||||
### 1.3 Other production files wiring or reading EKS
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `daemon/eks_deps.go` | 220 | cross-resource orchestration (composition-root binding) | Builds `EKSServiceDeps`, binding VPC, ELBv2, image, EIP, IGW, NAT-GW, route-table, placement-group, IAM, EBS and object-store owners. |
|
||||
| `daemon/eks_adapter.go` | 58 | cross-resource orchestration (composition-root binding) | `SubnetVPCResolver` over the VPC service. |
|
||||
| `daemon/eks_cp_control.go` | 67 | temporary residue | CP describe/start/stop calls `domains/ec2/awsapi/instance.{DescribeInstances,StartInstances,StopInstances}`, i.e. an EC2 AWS adapter used as an internal API (ADR-0004 S3/S4); removal: EC2-owned instance-control capability. |
|
||||
| `daemon/eks_worker_launch.go` | 175 | cross-resource orchestration (composition-root binding) | `WorkerLauncher` via `instanceService.RunInstances`, `ec2.RunInstances.{type}.{node}` and `contracts/ec2/v1` instance commands. |
|
||||
| `daemon/daemon.go` (EKS parts: `:75`, `:167`, `:1085-1128`, `:2018-2019`, `:2175`, `:2302-2304`, `:3017-3018`) | n/a | temporary residue | NATS subscription table for 38 `eks.*` subjects, service init, reconciler spawn, reaper registration, shutdown; comment at `:1085-1087` says every handler returns NotImplemented, which is stale. |
|
||||
| `daemon/dns_reconcile.go` (`:27`, `:64-68`, `:97-98`) | n/a | temporary residue | Watches every `eks-account-*` bucket via `AccountWatchBuckets` and pulls `DesiredDNSChanges`; removal: EKS-published DNS projection. |
|
||||
| `gateway/oidc_discovery.go` | 117 | temporary residue | Public OIDC discovery/JWKS endpoints read `eks-account-*`, `GetClusterMeta` and `OIDCJWKSKey` directly; removal: cluster-owned OIDC projection. |
|
||||
| `handlers/sts/oidc_jwks.go` | 126 | temporary residue (STS-owned file) | `FetchClusterJWKS` reads `eks-account-*`/`OIDCJWKSKey` (`:101-110`); ADR-0001 step 7 names this STS-to-EKS import for removal via an identity-owned contract or EKS projection. |
|
||||
| `runtime/roles/awsgw/eks_token_verify.go` | 75 | guest/controller wire contract (server) | Queue-subscribes `eks.VerifyToken` (`:30`) and verifies presigned STS URLs. |
|
||||
| `accountteardown/reapers_eks.go` | 138 | cross-resource orchestration | Account teardown deletes nodegroups then clusters through the `EKSService` NATS client. |
|
||||
| `operator/cli/admin_eks.go` | 77 | guest/controller wire contract (operator) | `spx admin eks restore-snapshot` over `eks.RestoreSnapshot`. |
|
||||
| `gateway/gateway.go` (`:300`, `:345-346`, `:403`, `:455`, `:513`), `gateway/auth.go` (`:278-279`), `gateway/operations.go` (`:34-36`) | n/a | AWS protocol adapter (legacy registration residue) | Legacy dispatch, OIDC public routes and inventory; removal per ADR-0002 when `domains/eks/awsapi` registers. |
|
||||
| `bootstrap/config/config.go:44` | n/a | AWS protocol adapter (config) | `eks` listed in `AWSGWServiceNames`. |
|
||||
|
||||
Guest counterparties (not inventoried individually): `agents/eks/{gatewaypublish,gatewayfetch,tokenwebhook,credentialprovider,konnectivitycert,webhookcert}` and `scripts/images/eks-node/*`.
|
||||
`agents/eks/gatewayfetch/fetch.go` decodes the internal-addons body through `contracts/eks/v1` `InternalAddonsResponse` (at `18794e017`).
|
||||
`agents/eks/tokenwebhook/webhook.go` imports `handlers_eks.WebhookTokenReviewResult`, so a guest binary imports the domain monolith (ADR-0004 S5 debt).
|
||||
|
||||
## 2. Served AWS actions and protocol
|
||||
|
||||
Protocol: REST-JSON, method+path routed by `rest.NewRouter("eks", eksRoutes)` (`gateway/eks.go:234`); unknown routes return `InvalidAction` (`:241`).
|
||||
Authorization order (`gateway/eks.go:237-312`): route lookup, account from auth context, internal-route principal gate, body read, query folding, `ResourceARNs`, `checkPolicyResources`, `checkEKSPassRole`, handler over NATS.
|
||||
Gateway-to-daemon transport: NATS request/reply `eks.<Action>`, queue group `spinifex-workers` (`daemon/daemon.go:1090-1127`).
|
||||
`gateway/operations.go:34-36` declares EKS as `Registered: eksActionNames()` with no `Stubbed` or `Unsupported` list, so stubs and internal routes are indistinguishable in the inventory.
|
||||
|
||||
| Action | Route | Status | Owner |
|
||||
|---|---|---|---|
|
||||
| CreateCluster | POST /clusters | implemented (async) | cluster |
|
||||
| ListClusters | GET /clusters | implemented | cluster |
|
||||
| DescribeCluster | GET /clusters/{c} | implemented | cluster |
|
||||
| DeleteCluster | DELETE /clusters/{c} | implemented (synchronous purge) | cluster |
|
||||
| UpdateClusterConfig | POST /clusters/{c}/update-config | stub NotImplemented (`service_impl.go:1455`) | cluster |
|
||||
| UpdateClusterVersion | POST /clusters/{c}/updates | stub NotImplemented (`service_impl.go:1459`) | cluster |
|
||||
| ListUpdates | GET /clusters/{c}/updates | stub at gateway (`gateway/eks.go:49-52`) | cluster (update records do not exist) |
|
||||
| DescribeUpdate | GET /clusters/{c}/updates/{id} | stub at gateway (`gateway/eks.go:53-56`) | cluster/nodegroup/addon |
|
||||
| CreateNodegroup | POST /clusters/{c}/node-groups | implemented (async) | nodegroup |
|
||||
| ListNodegroups | GET /clusters/{c}/node-groups | implemented | nodegroup |
|
||||
| DescribeNodegroup | GET .../node-groups/{n} | implemented | nodegroup |
|
||||
| UpdateNodegroupConfig | POST .../node-groups/{n}/update-config | implemented (synchronous, returns `Successful`) | nodegroup |
|
||||
| UpdateNodegroupVersion | POST .../node-groups/{n}/update-version | stub NotImplemented (`nodegroup.go:1120-1127`) | nodegroup |
|
||||
| DeleteNodegroup | DELETE .../node-groups/{n} | implemented (synchronous) | nodegroup |
|
||||
| CreateAccessEntry | POST /clusters/{c}/access-entries | implemented (STANDARD type only, `service_impl.go:1561-1564`) | access |
|
||||
| ListAccessEntries | GET /clusters/{c}/access-entries | implemented | access |
|
||||
| DescribeAccessEntry | GET .../access-entries/{p} | implemented | access |
|
||||
| UpdateAccessEntry | POST .../access-entries/{p} | implemented | access |
|
||||
| DeleteAccessEntry | DELETE .../access-entries/{p} | implemented | access |
|
||||
| AssociateAccessPolicy | POST .../access-entries/{p}/access-policies | implemented (cluster scope only, `service_impl.go:1692`) | access |
|
||||
| DisassociateAccessPolicy | DELETE .../access-policies/{policy} | implemented | access |
|
||||
| ListAssociatedAccessPolicies | GET .../access-entries/{p}/access-policies | implemented | access |
|
||||
| ListAccessPolicies | GET /access-policies | implemented (static catalogue) | access |
|
||||
| DescribeAddonVersions | GET /addons/supported-versions | implemented (static catalogue) | addon |
|
||||
| ListAddons | GET /clusters/{c}/addons | implemented | addon |
|
||||
| CreateAddon | POST /clusters/{c}/addons | implemented (async via guest) | addon |
|
||||
| DescribeAddon | GET .../addons/{a} | implemented | addon |
|
||||
| UpdateAddon | POST .../addons/{a}/update | implemented | addon |
|
||||
| DeleteAddon | DELETE .../addons/{a} | implemented | addon |
|
||||
| AssociateIdentityProviderConfig | POST .../identity-provider-configs/associate | stub NotImplemented (`service_impl.go:1818`) | cluster (no IdP resource) |
|
||||
| DescribeIdentityProviderConfig | POST .../identity-provider-configs/describe | stub | cluster |
|
||||
| DisassociateIdentityProviderConfig | POST .../identity-provider-configs/disassociate | stub | cluster |
|
||||
| ListIdentityProviderConfigs | GET .../identity-provider-configs | stub | cluster |
|
||||
| TagResource / UntagResource / ListTagsForResource | POST/DELETE/GET /tags/* | implemented for cluster and nodegroup ARNs; NotImplemented for addon/access-entry ARNs (`service_impl.go:1882,1925,1958`) | cluster, nodegroup |
|
||||
| PublishInternal | POST /clusters/{c}/internal-publish | internal (not AWS) | cluster guest contract |
|
||||
| WebhookTokenReview | POST /clusters/{c}/token-review | internal (not AWS) | access guest contract |
|
||||
| ListInternalAddons | GET /clusters/{c}/internal-addons/{acct} | internal (not AWS) | addon guest contract |
|
||||
| GetRecoveryDirective | GET /clusters/{c}/internal-recovery/{acct}/{id} | internal (not AWS) | cluster guest contract |
|
||||
|
||||
Internal NATS-only methods (no HTTP route): `SetRecoveryDirective`, `RestoreSnapshot`, `ListStagedAddonManifests` (backs ListInternalAddons).
|
||||
Unregistered AWS EKS operations (InvalidAction today) include Fargate profiles, pod identity associations, encryption config, insights, `RegisterCluster`/`DeregisterCluster`, `DescribeAddonConfiguration`, `DescribeClusterVersions` and EKS Anywhere subscriptions (list from memory of the EKS API, not verified against the pinned model).
|
||||
`iam:PassRole` is enforced for CreateCluster `roleArn`, CreateNodegroup `nodeRole`, and CreateAddon/UpdateAddon `serviceAccountRoleArn` (`gateway/eks/passrole.go`).
|
||||
|
||||
## 3. Durable records and KV buckets
|
||||
|
||||
| Bucket | Key | Record | Owner | Writers | Readers outside EKS |
|
||||
|---|---|---|---|---|---|
|
||||
| `eks-account-{acct}` (history 1, migration v1, `store.go:30-38,235-245`) | `clusters/{c}/meta` | `ClusterMeta` (`cluster_state.go:65`) | cluster | `service_impl.go` create/launch/delete/tags, `cluster_state.go` CAS helpers, `cluster_reconciler.go`, `restore_snapshot.go:150`, `nats_bootstrap.go:297` (CA) | **`gateway/eks/internal_authz.go:110-124`**, **`gateway/oidc_discovery.go:38,60`** |
|
||||
| same | `clusters/{c}/nodegroups/{ng}` | `NodegroupRecord` (`types.go:33`) | nodegroup | `nodegroup.go` (claim, CAS, put, delete), `service_impl.go:1964` (tags) | none observed |
|
||||
| same | `clusters/{c}/access-entries/{sha256(principalARN)}` | `AccessEntryRecord` (`types.go:93`) | access | `service_impl.go:1548-1785`, creator-admin seed `service_impl.go:994-1001` (cluster launch) | **gateway process via `ResolveTokenReview` (`token_review.go:113`)** |
|
||||
| same | `clusters/{c}/addons/{a}` | `addon.Record` (`domains/eks/addon/record.go`) | addon | `domains/eks/addon/owner.go` only, reached from `addons.go`, `addon_status_reconciler.go` and `nodegroup.go:405-416` (GPU add-on); plus the `DeleteClusterPrefix` sweep (at `18794e017`) | none observed |
|
||||
| same | `clusters/{c}/addons/{a}/manifest` | `addon.Manifest` (`domains/eks/addon/record.go`) | addon | `domains/eks/addon/delivery.go` `StagingInstaller`, `Owner.Delete`, the `DeleteClusterPrefix` sweep | guest via internal-addons route, as `contracts/eks/v1` `StagedAddonManifest` |
|
||||
| same | `clusters/{c}/oidc-signing-key.pem.enc` | encrypted ECDSA key | cluster | `oidc_keypair.go:86`, zeroized `service_impl.go:1281` | none |
|
||||
| same | `clusters/{c}/oidc-jwks.json` | JWKS | cluster | `oidc_keypair.go:94` | **`handlers/sts/oidc_jwks.go:110`**, **`gateway/oidc_discovery.go:93`** |
|
||||
| same | `clusters/{c}/oidc-jwks-verified` | marker | cluster | `nats_bootstrap.go:291` | none |
|
||||
| same | `clusters/{c}/admin-kubeconfig.enc` | encrypted kubeconfig | cluster | `nats_bootstrap.go:269` | none observed in Go (uncertain whether a CLI reads it) |
|
||||
| same | `clusters/{c}/k3s-node-token.enc` | encrypted join token | cluster | `nats_bootstrap.go:251` | read by `nodegroup.go:1164` (cross-resource) |
|
||||
| same | `clusters/{c}/recovery/{instanceId}` | `RecoveryDirective` | cluster | `recovery_directive.go:102`, reconciler | guest via internal-recovery route |
|
||||
| same (whole bucket) | all keys | watch | cluster | n/a | **`daemon/dns_reconcile.go:68`** via `AccountWatchBuckets` |
|
||||
| `spinifex-eks-leader` (TTL 60s, `store.go:40-42,249`) | `{acct}/{c}` reconciler lease; `.../teardown` lease | kvlease | cluster | `cluster_reconciler.go:391-403`, `service_impl.go:1174-1197` | none |
|
||||
| `spinifex-eks-clustertokens` (TTL 15m, `clienttoken.go:16-21`) | idempotency store keyed by account+token | `idempotency.Store[string]` | cluster | `service_impl.go:529-558,612-619` | none |
|
||||
|
||||
Dead key helpers: `OIDCProviderKey` (`store.go:95`), `EventKey` (`store.go:132`).
|
||||
Secret material in `ClusterMeta`: `ControlPlaneTemplate` is a copy of `K3sServerInput` that clears creds but not `OIDCPrivateKeyPEM` or `JoinToken` (`service_impl.go:945-959`; fields at `k3s_server_vm.go:146,171`), so the OIDC private key and k3s join token appear to be persisted unencrypted in the meta record that the gateway also reads; no scrubbing or custom marshaller was found (verify with a KV dump before acting).
|
||||
`ClusterMeta` combines desired intent (`Version`, `RoleArn`, `ResourcesVpcConfig`, `Logging`, `Tags`), realization refs (CP nodes, NLB, CP VPC, EIP, ENIs), observations (`HealthIssue`, `NodeCount`, `NodegroupNodeCounts`, `CertificateAuthorityB64`) and delete bookkeeping (`DeletingSince`, `DeleteReapAttempts`).
|
||||
Hidden realizations outside EKS stores: CP VMs (system account, `ManagedBy=eks`), CP VPC/subnets/NAT-GW/route tables/IGW, NLB and target groups (ELBv2), SGs, private-endpoint ENI, egress EIP, HA placement group, worker instances (customer account), IAM roles/instance profiles, CSI EBS volumes.
|
||||
|
||||
## 4. Identity and scope
|
||||
|
||||
| Resource | Identifier / ARN | Account | Region | AZ |
|
||||
|---|---|---|---|---|
|
||||
| cluster | name; `arn:aws:eks:{region}:{acct}:cluster/{name}` (`foundation/aws/arn/eks.go:21`) | tenant account owns the record; CP VMs, CP VPC and NLB are realized in the system account `awsidentifiers.GlobalAccountID` (`service_impl.go:738`, `:2240-2248`); customer-VPC SGs and the private-endpoint ENI in the tenant account | single Region from `deps.Region`; not stored as a field | Regional; HA CP spread over hosts by AZ (`k3s_ha_control_plane.go:528`) |
|
||||
| nodegroup | `arn:...:nodegroup/{c}/{ng}/{discriminator}` where the discriminator is derived from (account, cluster, name) (`nodegroup.go:333`), so a recreated nodegroup gets the same ARN, unlike AWS's per-incarnation UUID | tenant | deps.Region | AZs implied by subnets |
|
||||
| addon | `arn:...:addon/{c}/{a}` (`arn/eks.go` `FormatEKSAddon`, minted in `domains/eks/addon/owner.go` `Create`; the gateway authorizer derives the same string, `gateway/eks/authz.go` `addonARN`); no per-incarnation suffix, whereas the AWS API reference examples show `addon/{c}/{a}/{uuid}` | tenant | owner `region` from deps.Region | n/a |
|
||||
| access | `arn:...:access-entry/{c}/{sha256(principal)}` (`access_entry.go:244`); AWS's shape differs (verify) | tenant | deps.Region | n/a |
|
||||
|
||||
Uniqueness is per account bucket and cluster-name prefix; there is no immutable internal UID distinct from the name for any EKS resource.
|
||||
|
||||
## 5. Lifecycle paths per resource
|
||||
|
||||
### cluster
|
||||
Create: `CreateCluster` `service_impl.go:504`; validates, optional token claim (`:529-558`), resolves VPC, builds `ClusterMeta{Status: CREATING}`, `claimClusterName` CAS-create with FAILED-to-CREATING reclaim (`:1200-1265`), finalizes token, then launches `launchClusterInfra` on an in-process goroutine with `context.Background()` (`:625-645`).
|
||||
Launch stages (`:718-1012`): CP VPC, SGs, CP ingress, customer SGs, private ENI, NLB, OIDC keypair, join token, CP placement/launch, persist CP refs, NLB target registration, persist final meta, creator-admin AccessEntry, `spawnBootstrap`, `spawnReconciler`; any failure calls `failClusterLaunch` (`:696`) which marks FAILED.
|
||||
Readiness: `ClusterReconciler.reconcileOnce` (`cluster_reconciler.go:594`) flips CREATING to ACTIVE when bootstrap artifacts exist (`:1173-1189`) and the CP state report is healthy; CREATING becomes FAILED after `createTimeout` (`:1151-1171`).
|
||||
Reconciliation/repair (ACTIVE): health recording, CP restart (`:708`), etcd reset via recovery directives (`:780`), member replacement (`:1010`), all under the per-cluster lease `spinifex-eks-leader/{acct}/{c}` (`:391-418`).
|
||||
Restart: `SpawnRegisteredReconcilers` (`service_impl.go:342-425`) rescans buckets, reclaims orphaned nodegroups, resubscribes missing bootstrap kinds and respawns reconcilers; it does not resume an interrupted `launchClusterInfra`.
|
||||
Delete: `DeleteCluster` (`service_impl.go:1104-1155`) returns success for an absent cluster, takes the teardown lease (losers return DELETING), sets DELETING, runs `purgeClusterInfra` synchronously in the request (`:1268-1452`), and only erases `clusters/{c}/` after all blocking steps succeed; `EKSDeletingReaper` re-drives with backoff and exhaustion (`eks_deleting_reaper.go:46-159`); `EKSBillableReaper` terminates CP VMs whose meta is gone (`eks_billable_reaper.go:46-100`).
|
||||
Fencing: leases only; no desired/observed generation (no `generation` field anywhere in `handlers/eks`).
|
||||
Idempotency: `clientRequestToken` honoured for CreateCluster only, 15-minute TTL, replays current meta by name.
|
||||
|
||||
### nodegroup
|
||||
Create: `createNodegroup` (`nodegroup.go:205`) requires cluster ACTIVE (`:226-229`), `ClaimNodegroupRecord` CAS-create (`:112-126`, ResourceInUse at `:363`), then background `launchNodegroupInfra` (`:373`, `:449-535`).
|
||||
Readiness: `waitWorkersReady` (`:537-563`) polls the cluster's `NodegroupNodeCounts` observation until ready count reaches baseline plus desired, then sets ACTIVE (`:521-523`), else CREATE_FAILED.
|
||||
Update: `UpdateNodegroupConfig` runs `reconcileNodegroup` inline under revision CAS (`:914-979`, `launchWorkersCAS` `:985`, `recordLaunchedWorker` `:1038`) and returns `Update{Status: Successful}` (`:900-906`).
|
||||
Restart: `reclaimOrphanedNodegroups` (`:1189-1228`) terminates workers of CREATING/partial CREATE_FAILED records and marks them CREATE_FAILED; it does not resume.
|
||||
Delete: `deleteNodegroup` (`:1129-1158`) terminates workers then deletes the record synchronously and returns a DELETING projection of a record that no longer exists.
|
||||
Fencing: KV revision CAS; no generation, no lease.
|
||||
Idempotency: no `clientRequestToken`; duplicate create loses the claim and gets ResourceInUse.
|
||||
|
||||
### addon
|
||||
Paths in this subsection are at `18794e017`.
|
||||
Create: `CreateAddon` (`addons.go:61`) validates against the catalogue and calls `Owner.Create` (`domains/eks/addon/owner.go:48`), which is get-then-put (not CAS-create) and then stages the manifest (`delivery.go` `StagingInstaller.Install`); a staging failure marks `CREATE_FAILED`; returns CREATING.
|
||||
Readiness: guest addon-sync agent publishes `eks.addon.{acct}.{c}.status` via internal-publish; `applyAddonStatusReport` (`addon_status_reconciler.go:22`) relays to `Owner.ApplyReport` (`owner.go:140`), whose `nextStatus` (`:186`) maps `ready` to ACTIVE from any status and `failed` to CREATE_FAILED/DEGRADED; `report.Version` and `report.TS` are dropped before the owner sees them.
|
||||
Update: `UpdateAddon` (`addons.go:163`) calls `Owner.Update` (`owner.go:94`), which CAS-sets UPDATING from any status and restages; the handler returns `Successful` with `Id = rec.Arn` (`addons.go:193-194`); a staging failure marks `CREATE_FAILED`.
|
||||
Delete: `DeleteAddon` (`addons.go:201`) calls `Owner.Delete` (`owner.go:122`), which unstages and deletes the record and manifest immediately; the DELETING status is only in the response.
|
||||
Fencing: revision CAS on the record; no generation or incarnation.
|
||||
Idempotency: none (`clientRequestToken` ignored).
|
||||
|
||||
### access
|
||||
Create: `CreateAccessEntry` (`service_impl.go:1548-1581`) get-then-put (`:1570-1578`), synchronous; creator-admin entry seeded during cluster launch (`:994-1001`).
|
||||
Mutations: `casUpdateAccessEntry` (`access_entry.go:141`) for update/associate/disassociate; delete via `DeleteAccessEntryRecord`.
|
||||
Effect: read at authentication time by `Authenticate`/`ResolveTokenReview` (`token_review.go:31-128`); no in-cluster realization or readiness.
|
||||
Delete with cluster: swept by `DeleteClusterPrefix` (`cluster_state.go:493`).
|
||||
Idempotency: none (`clientRequestToken` ignored).
|
||||
|
||||
## 6. NATS subjects and guest/controller messages
|
||||
|
||||
| Subject / route | Direction | Delivery | Defined | Consumer |
|
||||
|---|---|---|---|---|
|
||||
| `eks.<Action>` (38 subjects) | gateway (or teardown/CLI) to daemon | core NATS request/reply, queue `spinifex-workers` | `service_nats.go:30-189`, `daemon/daemon.go:1090-1127` | `EKSServiceImpl` |
|
||||
| `eks.bus.{acct}.{c}.{k3s-bootstrap-token,k3s-admin-kubeconfig,k3s-oidc-jwks,k3s-ca}` | CP VM via gateway relay to daemon | core NATS publish/subscribe, at-most-once | `nats_bootstrap.go:47-57`, subscribe `:197` | `NATSBootstrap` persists to KV with `persistWithRetry` (`:32`) |
|
||||
| `eks.state.{acct}.{c}.server` | CP VM via relay | core NATS, periodic | `state_report.go:13` | `ClusterReconciler` (`cluster_reconciler.go:432`) |
|
||||
| `eks.addon.{acct}.{c}.status` | CP VM via relay | core NATS, level-triggered every 30 s per staged add-on | `contracts/eks/v1/addon_status.go:10` (at `18794e017`) | `ClusterReconciler` (`:448`), relayed to `addon.Owner.ApplyReport` |
|
||||
| `eks.VerifyToken` | EKS (gateway process) to awsgw STS | request/reply, queue `awsgw-eks-token-verify` | `token_webhook.go:13` | `runtime/roles/awsgw/eks_token_verify.go:30` |
|
||||
| `contracts/cluster/v1` `NodeStatusSubject`, `NodeVMsSubject` | EKS to every node | fan-out request | `k3s_ha_control_plane.go:502,579` | daemons |
|
||||
| `ec2.RunInstances.{type}.{node}` and `contracts/ec2/v1` instance commands | EKS (daemon binding) to EC2 | request/reply | `daemon/eks_worker_launch.go` | EC2 instance owner |
|
||||
| HTTP `internal-publish`, `token-review`, `internal-addons`, `internal-recovery` | guest to gateway (SigV4, system CP role) | HTTP | `gateway/eks.go:59-87` | gateway relays/queries |
|
||||
| HTTP `/oidc/eks/{region}/{acct}/{c}/.well-known/openid-configuration`, `/keys` | public | HTTP, unauthenticated | `gateway/gateway.go:345-346` | `gateway/oidc_discovery.go` |
|
||||
|
||||
Retry and recovery paths: guest `eks-gateway-publish` retries until HTTP 2xx (`agents/eks/gatewaypublish/publish.go:26-31,97-114`), but the gateway returns 2xx after a core-NATS publish even with no subscriber (`gateway/eks/internal_publish.go:84-89`), so a bootstrap message published while the daemon subscriber is absent is lost.
|
||||
On restart, `BootstrapPendingKinds` resubscribes only missing kinds (`nats_bootstrap.go:115-139`, `service_impl.go:410-416`); a lost one-shot message is not replayed, so the cluster reaches FAILED at the create timeout.
|
||||
A bootstrap subscriber that exits with error marks the cluster FAILED (`service_impl.go:2192-2212`).
|
||||
SDK retries of `eks.<Action>` land on any node of the queue group; dedup relies on token store, claim CAS, teardown lease or nothing (addon/access create).
|
||||
|
||||
## 7. Boundary-crossing imports
|
||||
|
||||
### Outbound from `handlers/eks`
|
||||
- IAM/identity: `handlers/iam` `EncryptSecret`/`DecryptSecret` (`nats_bootstrap.go`, `oidc_keypair.go`, `nodegroup.go`), `EnsureSystemInstanceProfile` (`k3s_server_role.go`), `SystemInstanceRoleEnsurer` (`service_impl.go`); `instanceProfileEnsurer` capability with `GetRole/CreateRole/PutRolePolicy/CreateInstanceProfile/AddRoleToInstanceProfile` (`service_impl.go:214`) means EKS creates IAM roles and instance profiles.
|
||||
- Bluebottle: `bluebottle/pkg/auth` `ResolveRoleARN` (`nodegroup.go`) and `ParseRoleARN` (`service_impl.go`); daemon binding uses `bluebottle/pkg/masterkey`; gateway uses `bluebottle/pkg/auth` (`gateway/eks.go`).
|
||||
- STS: token verification over `eks.VerifyToken` (process contract, no import).
|
||||
- EC2 instance: `domains/ec2/systeminstance` (`SystemInstanceInput/Output`, `ExtraENIInput`, `BootAMI`), `domains/ec2/instancetypes`, `domains/ec2/placementgroup` (alias `ec2placementgroup`), `domains/ec2/ebs/metadata`, `runtime/compute/vm` (reapers), plus `WorkerLauncher`, `k3sInstanceLauncher`, `k3sAMIResolver`, `eipProvisioner`, `csiVolumeReclaimer` capabilities.
|
||||
- Network: `domains/network/systemvpc` (`Ensure`, `Delete`, `EnsureIGW`, `DeleteIGW`, provisioner interfaces); `sgProvisioner` and `k3sVPCProvisioner` capabilities (EC2 SDK shapes).
|
||||
- DNS: `domains/dns` `EKSName`, `EKSChanges`, `ResolveBaseDomain`, `Change`, `ActionUpsert` (`service_impl.go`).
|
||||
- ACM/CA: no import; the platform/NATS CA PEM is passed as `GatewayCACert` from `d.config.NATS.CACert` (`daemon/eks_deps.go:64-71`) and baked into CP user data; the k3s cluster CA is self-generated in the guest and returned on `k3s-ca`.
|
||||
- ELBv2: `nlbProvisioner` capability with ELBv2 SDK shapes (`nlb.go:19`), bound to `d.elbv2Service`.
|
||||
- Storage/object: `providers/objectstore` (etcd snapshots), Predastore system creds via deps.
|
||||
- Contracts: `contracts/cluster/v1`; `contracts/eks/v1` from `addons.go`, `addon_status_reconciler.go`, `state_report.go` and `service_impl.go` (at `18794e017`).
|
||||
- EKS add-on owner: `domains/eks/addon` from `addons.go`, `addon_status_reconciler.go`, `service_impl.go` and, through `addons()`, `nodegroup.go` (at `18794e017`; edges in Section 7.1).
|
||||
- Foundation: `kvutil`, `kvstore`, `kvlease`, `migrate`, `idempotency`, `reconciler`, `telemetry`, `aws/{arn,errors,tags,identifiers,ami}`, `messaging/nats`.
|
||||
- Bootstrap: `bootstrap/config.Config` in `EKSServiceDeps`.
|
||||
|
||||
### Inbound to `handlers/eks`
|
||||
- `gateway/eks/*`: `NewNATSEKSService`, input/output types (including `ListStagedAddonManifestsInput` in `internal_addons.go`), `PrincipalARNHash`, `ClusterMeta`, `ClusterMetaKey`, `AccountBucketName`, `CPInstanceRoleName`, `ResolveTokenReview`, bootstrap/state subject builders; the add-on subject now comes from `contracts/eks/v1` (at `18794e017`).
|
||||
- `gateway/oidc_discovery.go`: `AccountBucketName`, `GetClusterMeta`, `OIDCJWKSKey`.
|
||||
- `handlers/sts/oidc_jwks.go`: `AccountBucketName`, `OIDCJWKSKey` (STS reads EKS private KV).
|
||||
- `runtime/roles/awsgw/eks_token_verify.go`: `TokenVerifySubject`, `TokenVerifyRequest`, `TokenVerifyResponse`.
|
||||
- `agents/eks/tokenwebhook/webhook.go`: `WebhookTokenReviewResult` (guest imports domain).
|
||||
- `daemon/*`: `EKSServiceImpl`, `NewEKSServiceImpl`, `EKSServiceDeps`, `NewNATSHostScheduler`, `SubnetVPCResolver`, `WorkerLauncher`, `AccountWatchBuckets`.
|
||||
- `accountteardown/reapers_eks.go`: `EKSService`, `NewNATSEKSService`.
|
||||
- `operator/cli/admin_eks.go`: `NewNATSEKSService`, `RestoreSnapshotInput`.
|
||||
|
||||
### Reverse knowledge of EKS in other domains (comments or tag conventions, no import)
|
||||
`domains/ec2/instance/service_impl.go:394-396` (worker tag comment), `domains/ec2/systeminstance/launcher.go:4,25`, `handlers/elbv2/{launcher.go:5,service_impl.go:1232-1250,service_impl_nlb_sg.go:196}`, `domains/ec2/vpc/eni.go:54`, `domains/ec2/guestmetadata/resolver.go:37`, `domains/dns/{naming.go:118-124,reconcile.go:58,303}`, `handlers/iam/managed_policies.go` (EKS managed policies), `accountteardown/reapers_iam.go:26-33` (OIDC providers).
|
||||
|
||||
### 7.1 Add-on boundary edges remaining at `18794e017`
|
||||
|
||||
Each edge is temporary unless marked otherwise, and each clears only on its stated condition; moving a file does not clear any of them.
|
||||
E1 to E5 match the `handlers/eks` → `domains/eks/addon` debt entry in `docs/PACKAGE_BOUNDARY_MIGRATION.md`; E6 to E9 match the further-edges entry beside it.
|
||||
|
||||
| ID | Edge | Kind | Where | Removal condition |
|
||||
|---|---|---|---|---|
|
||||
| E1 | Cluster delete erases add-on records and manifests | handler → owner state (legacy) | `handlers/eks/cluster_state.go:493` `DeleteClusterPrefix` sweeps `clusters/{c}/` without calling `addon.Owner` | Cluster delete invokes an add-on-owned teardown capability. |
|
||||
| E2 | GPU node-group launch stages `nvidia-device-plugin` | node group → add-on orchestration | `handlers/eks/nodegroup.go:405-416` `stageGPUDeviceAddon`, called at `:473`, through `Owner.EnsureGPUDevicePlugin`; best-effort, errors logged and swallowed, never undone on node-group delete, and any existing record (including `CREATE_FAILED`) blocks re-staging | The node-group owner is extracted and the dependency is designed. |
|
||||
| E3 | Add-on delivery reports are received by the cluster reconciler | cluster hosts add-on observation | `handlers/eks/cluster_reconciler.go:171-176,264-269,447-458` subscribes; `addon_status_reconciler.go:22` relays through the reconciler-declared `addonReports`; wired in `service_impl.go:2160-2163` | Add-on delivery observation has its own host under the lifecycle correction. |
|
||||
| E4 | Cluster existence is checked by the handler | handler admission | `addons.go` actions call `acctKVForCluster`, which reads `ClusterMeta` existence only; cluster status is ignored | Moves into the owner when cluster admission is defined. |
|
||||
| E5 | Guest wire types | versioned contract (not debt) | `contracts/eks/v1` imported directly by `handlers/eks` (`addons.go`, `addon_status_reconciler.go`, `state_report.go`, `service_impl.go`), `gateway/eks` (`internal_addons.go`, `internal_publish.go`) and `agents/eks/gatewayfetch` | Permanent while v1 guests exist; v1 is retired only through a successor version and an explicit retirement, never by editing v1. |
|
||||
| E6 | The persisted `Record` crosses to the AWS adapter | owner record exposed | `addons.go` `addonRecordToAWS` (`:222`) and `ListAddons`/`DescribeAddon` read `addon.Get`/`addon.List` directly | Replace with an owner-returned view when the add-on AWS adapter moves to `domains/eks/awsapi`. |
|
||||
| E7 | The guest route reaches the owner through the monolith RPC | gateway → `handlers/eks` | `gateway/eks/internal_addons.go` uses `handlers_eks.NewNATSEKSService` and `ListStagedAddonManifestsInput`; `handlers/eks/addons.go:117` maps `addon.Manifest` to `eksv1.StagedAddonManifest` | As for `service.go`/`service_nats.go`: per-resource commands behind `domains/eks/awsapi` registration, with the internal request versioned with the add-on contract. |
|
||||
| E8 | Owner and contract keep separate shapes | deliberate duplication (not debt) | `addon.Manifest`/`addon.Report` beside `eksv1.StagedAddonManifest`/`eksv1.AddonStatusReport`; the handler maps between them | None: the owner does not import the wire contract. Revisit only if a successor contract makes the mapping lossy. |
|
||||
| E9 | The add-on ARN is derived in two places | duplicated identity rule | `domains/eks/addon/owner.go` `Create` and `gateway/eks/authz.go` `addonARN` both call `arn.FormatEKSAddon` from names | Clears when authorization obtains the ARN from the owner, which becomes necessary if an incarnation suffix is adopted. |
|
||||
|
||||
## 8. Known gaps against Q-78 / ADR-0003 (observed)
|
||||
|
||||
1. No desired or observed generation for any EKS resource; fencing is leases (cluster) or record-revision CAS (nodegroup, addon, access).
|
||||
2. Intent and observation share `ClusterMeta` (`cluster_state.go:65-190`), and nodegroup readiness reads a cluster observation (`nodegroup.go:537-549`).
|
||||
3. Cluster realization runs in request-spawned goroutines (`service_impl.go:625-645`); restart does not resume the launch, so an interrupted create ends FAILED after timeout (`cluster_reconciler.go:1151-1171`) rather than converging the same intent.
|
||||
4. CP VMs are live before their IDs are persisted (`service_impl.go:960-970` comment); a crash in that window leaves VMs reachable only by the billable reaper after meta removal.
|
||||
5. Bootstrap artifacts travel on at-most-once core NATS (`nats_bootstrap.go:197`, `gateway/eks/internal_publish.go:84`); a lost message is not replayed.
|
||||
6. DeleteCluster purges synchronously inside the request (`service_impl.go:1141-1154`) rather than accepting asynchronously as AWS does.
|
||||
7. DeleteCluster does not reject a cluster with nodegroups (AWS returns ResourceInUseException) and `DeleteClusterPrefix` erases nodegroup records while their worker instances keep running; `accountteardown/reapers_eks.go:15-18` documents this; workers carry no `ManagedBy` tag (`nodegroup.go:669-670`) and no reaper reclaims them (uncertain whether the e2e suite exercises this order).
|
||||
8. DeleteNodegroup is synchronous and removes the record immediately (`nodegroup.go:1145-1157`), so the resource is absent before EC2 termination completes; no durable deletion intent.
|
||||
9. UpdateNodegroupConfig and UpdateAddon return `Successful` synchronously (`nodegroup.go:900-906`; `addons.go:192-197` at `18794e017`) and `DescribeUpdate`/`ListUpdates` are stubs, so there is no truthful pending update.
|
||||
10. CreateAddon and CreateAccessEntry use get-then-put (`domains/eks/addon/owner.go` `Owner.Create` at `18794e017`; `service_impl.go:1570-1578`), so concurrent creates can both succeed.
|
||||
11. Addon status reports are not fenced by version or incarnation (`addon_status_reconciler.go:22` drops `AddonStatusReport.Version` and `TS` at `18794e017`), so a stale report can flip UPDATING to ACTIVE, or a stale `failed` to CREATE_FAILED.
|
||||
12. DeleteAddon deletes the record and manifest before in-cluster removal is observed (`domains/eks/addon/owner.go` `Owner.Delete` at `18794e017`), and the guest sends no removal report.
|
||||
13. `clientRequestToken` is honoured only by CreateCluster; CreateNodegroup, CreateAddon, CreateAccessEntry, UpdateNodegroupConfig, UpdateAddon ignore it.
|
||||
14. Nodegroup and addon ARNs are deterministic by name, so identifiers are reused across incarnations (Q-78 references Q-43 identifier-reuse rules).
|
||||
15. ADR-0004 S3 inventory: EKS has no `Stubbed`/`Unsupported` classification (`gateway/operations.go:34-36`) although nine actions return NotImplemented, and four internal routes are mixed into the AWS action inventory.
|
||||
16. ADR-0003 S4 / ADR-0004 S4: three non-owner readers of EKS private KV (`gateway/eks/internal_authz.go`, `gateway/oidc_discovery.go`, `handlers/sts/oidc_jwks.go`) plus `daemon/dns_reconcile.go` watching all EKS buckets.
|
||||
17. ADR-0004 S3: `daemon/eks_cp_control.go` uses the EC2 AWS adapter package as an internal API.
|
||||
18. ADR-0004 S5: guest `agents/eks/tokenwebhook` imports `handlers/eks`.
|
||||
19. Observed security risk outside Q-78: plaintext OIDC private key and k3s join token in `ClusterMeta.ControlPlaneTemplate` (Section 3).
|
||||
20. Comment hygiene: EKS lifecycle evidence now cites `ADR-0003:S2`; `daemon/daemon.go:1085-1087` is stale.
|
||||
|
||||
## 9. First-resource candidates
|
||||
|
||||
| Rank | Resource | Approx. size | Private state | Dependencies | Tests | Extractable without EKS identity/IAM/Bluebottle? |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | access | about 670 lines (`access_entry.go` 255, `service_impl.go:1527-1816` about 290, `token_review.go` 128, parts of `types.go`) | `clusters/{c}/access-entries/*` only | cluster existence query (`acctKVForCluster` reads `ClusterMeta`), cluster launch seeds creator admin, gateway authz needs `PrincipalARNHash`, gateway token-review calls `ResolveTokenReview`, STS verify contract | `access_entry_test.go`, `service_impl_test.go` (access cases), `token_review_test.go`, `token_webhook_test.go`, `gateway/eks/authz_test.go`, `gateway/eks/token_review_test.go`, e2e `TestEKS/AccessEntry`, `GetToken` | The record CRUD and policy catalogue are extractable now with a narrow cluster-exists query; the token-review part touches STS identity (`eks.VerifyToken` contract) but not IAM or Bluebottle, so it can stay behind its existing NATS contract. |
|
||||
| 2 | addon (owner extracted) | at `18794e017`: owner 693 lines in `domains/eks/addon`, wire contract 55 lines in `contracts/eks/v1`, adapter and relay 319 lines left in `handlers/eks` (`addons.go`, `addon_status_reconciler.go`) | `clusters/{c}/addons/*` incl. manifests | the edges in Section 7.1 | `domains/eks/addon/*_test.go`, `contracts/eks/v1/*_test.go`, `handlers/eks/addon_lifecycle_test.go`, `addons_test.go`, `addon_status_reconciler_test.go`, `gateway/eks_addon_wire_test.go`, `agents/eks/gatewayfetch/addons_wire_test.go`, e2e `AddonDelivery`, `TestEKSGPUPodExposure` | Extracted without IAM or Bluebottle; the report-relay seam (E3) and the guest contract (E5) exist, and every lifecycle gap remains. |
|
||||
| 3 | nodegroup | about 1,650 lines (`nodegroup.go`, `nodegroup_userdata.go`, `claim.go`, part of `types.go`, nodegroup SG rules) | `clusters/{c}/nodegroups/*` | cluster meta (ACTIVE, CP ENI IP, VPC, readiness counts), cluster join token decrypt (`handlers_iam.DecryptSecret`), IAM role/profile creation, Bluebottle `ResolveRoleARN`, EC2 worker launch, AMI lookup, cluster SGs, GPU addon | `nodegroup_test.go` (39 tests), `nodegroup_recreate_test.go`, `nodegroup_instance_profile_test.go`, `nodegroup_userdata_test.go`, `daemon/eks_worker_launch_test.go`, e2e IRSA pod path | No; it needs IAM, Bluebottle and cluster secrets capabilities first. |
|
||||
| 4 | cluster | about 8,500 lines (remainder of `handlers/eks`) | meta, OIDC material, kubeconfig, join token, recovery directives, leader/token buckets | every external owner (EC2, VPC, ELBv2, DNS, IAM, objectstore, placement, scheduler) and all guest contracts | `create_cluster_test.go`, `delete_cluster_test.go`, `cluster_reconciler*_test.go` (5 files), `lifecycle_invariants_test.go` (RLC1-3), reapers, `nats_bootstrap_test.go`, `oidc_keypair_test.go`, `k3s_*_test.go`, `nlb_test.go`, `security_groups_test.go`, `cp_vpc_test.go`, `restore_snapshot_test.go`, e2e `TestEKS` | No; it is the orchestration hub and depends on identity, Bluebottle and the CA. |
|
||||
|
||||
Recommendation: extract `access` first, defining a cluster-owned "cluster exists and is not deleting" query and moving `PrincipalARNHash` and the key helper into the access package.
|
||||
Its shortcomings (gap 10, no client token, no generation) are small and can be recorded in its lifecycle contract without first resolving cluster realization.
|
||||
@@ -1,310 +0,0 @@
|
||||
# ADR-0004 source inventory — RDS
|
||||
|
||||
Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`.
|
||||
It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it.
|
||||
Items marked "verify" or "uncertain" are unconfirmed.
|
||||
Gaps record observed behaviour, not target behaviour.
|
||||
|
||||
|
||||
Repo: `/home/tf-user/Development/mulga/spinifex`, branch `refactor/spinifex-package-boundaries`, paths relative to `spinifex/`.
|
||||
Inputs read: ADR-0001, ADR-0003, ADR-0004 and Q-78 from `origin/main`.
|
||||
Method: static reading of production code only; nothing here was verified against a live cluster.
|
||||
Line references are `file:line` at the time of reading.
|
||||
"Uncertain" marks a claim inferred from code shape rather than traced end to end.
|
||||
|
||||
## 0. Scope findings
|
||||
|
||||
`daemon/instance_records.go` (217) and `daemon/instance_records_migrate.go` (361) are **not RDS**.
|
||||
They own the EC2 per-instance record space `i.<id>` in the EC2 instance-state bucket (`daemon/instance_records.go:25`) and its migrations (`daemon/instance_records_migrate.go:29-70`), and import only `runtime/compute/vm`, `kvstore`, `kvutil`, `migrate`.
|
||||
They are excluded from the RDS inventory and belong to the EC2 instance owner.
|
||||
|
||||
Other production files that wire or consume RDS, found by import search:
|
||||
|
||||
- `daemon/daemon.go` (RDS sections only: `:170-171`, `:1171-1206`, `:2046-2078`, `:2306-2312`, `:2374-2382`).
|
||||
- `daemon/rds_deps.go`.
|
||||
- `daemon/dns_reconcile.go:71-76` and `:99-104` (RDS bucket watch and RDS DNS desired set).
|
||||
- `daemon/bedrock_deps.go:43` (Ochre/Bedrock borrows `handlers_rds.NewNATSVolumeAttacher`).
|
||||
- `handlers/quota/live.go:82-93` (`EnforceRDSInstances`).
|
||||
- `accountteardown/reapers_rds.go` (instance, subnet-group, parameter-group reapers).
|
||||
- `gateway/rds.go`, `gateway/operations.go:43-45`, `gateway/gateway.go:302,525-526` (service map and dispatch).
|
||||
- `bootstrap/config/config.go:338` (`RDSConfig`).
|
||||
- Test-support (non-`_test.go` but not production): `tests/integration/service_daemon_lite.go:51` reflects every `Service` method onto `rds.<Method>`; `tests/e2e/harness/rds*.go`.
|
||||
- Counterparty only: `agents/rds/agent/*.go` and `cmd/rds-agent` import `handlers/rds` for wire types.
|
||||
|
||||
## 1. File table
|
||||
|
||||
Total in-scope production lines: 16,468 (handlers/rds 14,873; gateway/rds 853; gateway/rds.go 95; daemon/rds_deps.go 98; daemon/instance_records* 578, not RDS).
|
||||
|
||||
### handlers/rds
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `store.go` | 451 | temporary residue (prerequisite: per-resource packages exist; removal: each resource owns its own key helpers and bucket view) | One file holds every resource's key space, bucket configs, migrations hook, leader bucket and cross-tenant enumeration (`:20-47`, `:39-50` key list). **Mixed**: split into instance (`db-instances/`, `bootstrap-payloads/`, `instance-index/`, leader bucket), snapshot (`db-snapshots/`, `backups/`, `retained-volumes/`), subnetgroup (`db-subnet-groups/`), parametergroup (`db-parameter-groups/`), events (`events/`), plus a domain-root bucket factory. |
|
||||
| `records.go` | 501 | temporary residue (same prerequisite) | All durable record structs in one file. **Mixed**: `DBInstanceRecord`/`PendingModifiedValues`/`ModifyLease`/`BootstrapState`/`AgentState`/`SnapshotOperation` → instance; `DBSnapshotRecord`/`AutomatedBackupRecord`/`RetainedVolumeRecord` → snapshot; `DBSubnetGroupRecord` → subnetgroup; `DBParameterGroupRecord`/`DBParameterRecord` → parametergroup; `Parameter`, `EngineHealth`, `Heartbeat*` → guest/controller wire contract. |
|
||||
| `service.go` | 314 | temporary residue (prerequisite: resource owners; removal: each owner holds its own deps, domain root holds composition) | Single `Service` with one `Deps` bag for every resource (`:19-45` in stripped view), per-account bucket cache and generic JSON CAS helpers. **Mixed**: `Deps` splits by resource; CAS helpers could move to `foundation/state/kvstore` (already exists) rather than a new common package. |
|
||||
| `service_nats.go` | 189 | temporary residue (prerequisite: awsapi calls owners in-process or via a domain contract; removal: no AWS-SDK-typed NATS client used as internal API) | AWS-SDK-shaped NATS client used by the gateway, quota (`handlers/quota/live.go:89`) and account teardown (`accountteardown/reapers_rds.go:19`). It is currently the de facto internal API, which ADR-0004 S3/S4 forbid for quota. |
|
||||
| `subjects.go` | 90 | **Mixed**: guest/controller wire contract (`rds.bus.*`, `CommandQueueGroup`, `:6-19`, `:65-90`) + temporary residue (Layer-1 `rds.<Action>` gateway→daemon subjects, `:21-63`) | Bus subjects are the real agent-path process boundary; the per-action subjects exist only because the gateway and daemon are separate dispatch hops. |
|
||||
| `arn.go` | 92 | AWS protocol adapter (with per-resource kinds) | `FormatARN`/`ParseARN`/`ResourceKind` (`:15-40`) are used for projection, tags and gateway authz scopes. Kinds could live with each resource; parsing belongs in awsapi. |
|
||||
| `filters.go` | 68 | AWS protocol adapter | Generic AWS `Filter` parsing (`ReadFilters`, `:38`), used by gateway catalogs and describes. |
|
||||
| `paging.go` | 74 | AWS protocol adapter | AWS `MaxRecords`/`Marker` paging (`:23`) and per-type page keys (`:52-74`). |
|
||||
| `validate.go` | 271 | resource owner (instance) — partly AWS protocol adapter | `CreateDBInstance` request validation (`:69`), identifier rules (`:196`), unimplemented-parameter rejection (`:225`). **Mixed**: SDK input parsing → awsapi; identifier/engine/class invariants → instance. |
|
||||
| `status.go` | 69 | resource owner (instance) | DB instance status enum and transition table (`:7-60`). |
|
||||
| `create.go` | 305 | resource owner (instance) | `CreateDBInstance` orchestration, record reservation, rollback (`:29-147`, `:195-305`). |
|
||||
| `describe.go` | 298 | **Mixed**: resource owner (instance) query + AWS protocol adapter (projection to `rds.DBInstance`, `:145-298`) | Projection is awsapi; record read/filter is the owner's query. |
|
||||
| `lifecycle.go` | 454 | resource owner (instance) — **contains an EC2 adapter** | Reboot/Stop/Start (`:57-162`), transition helpers (`:276-405`). `natsInstanceCommander` (`:407-454`) sends `ec2v1` instance commands and `ec2.start`; that is an EC2-owned capability implemented inside RDS (residue). |
|
||||
| `launch.go` | 629 | cross-resource orchestration (instance realization over EC2/VPC/EBS/IAM) — **contains an EC2 adapter** | `LaunchDBInstanceVM` (`:178-366`) composes system instance, two ENIs, data volume, AMI lookup. `natsVolumeAttacher` (`:560-629`) is an EC2 volume-attach NATS client that Bedrock also imports (`daemon/bedrock_deps.go:43`); it is EC2/EBS residue living in RDS. |
|
||||
| `network.go` | 200 | resource owner (instance) with cross-resource read of subnetgroup | Endpoint placement (`:36-75`) reads the subnet-group record directly (`:43`) and EC2 VPC/subnet/SG via `networkResolver` (`:16-20`). |
|
||||
| `vpc.go` | 163 | cross-resource orchestration (system VPC/SG realization for instances) | `EnsureSystemVPC`/`EnsureSystemSecurityGroup` (`:78`, `:104`) over `domains/network/systemvpc`. |
|
||||
| `storage.go` | 155 | resource owner (instance) | Storage grow via `volumeResizer` (EC2 `DescribeVolumes`/`ModifyVolume`, `:23-26`). |
|
||||
| `modify.go` | 528 | resource owner (instance) | `ModifyDBInstance` plan/immediate/pending paths (`:76-198`, `:199-528`). |
|
||||
| `modify_lease.go` | 173 | resource owner (instance) | Per-instance modify lease stored on the instance record (`:24-173`). |
|
||||
| `pending.go` | 256 | resource owner (instance) — with parametergroup orchestration | Applies `PendingModifiedValues` (`:28`), parameter-group apply to an instance (`:149`), filesystem grow finish (`:240`). `applyParameterGroup` is the instance side of parameter-group propagation. |
|
||||
| `replace.go` | 252 | resource owner (instance) — replacement | VM replace for class/storage change (`:43-166`), index rewrite (`:200`). |
|
||||
| `recovery.go` | 220 | resource owner (instance) — failure detection | Health classification (`:54`, `:76`); reports failure, does not repair (`:19-21`). |
|
||||
| `reconciler.go` | 896 | **Mixed**: resource owner (instance reconcile, `:238-660`, `:797-866`) + snapshot reconcile (`:674-796`) + foundation wiring (leader lease, `:83-188`) | Split: instance reconciler, snapshot creating-resolver, domain-root leader/loop. `describeInstanceState` (`:867-896`) is an EC2 projection adapter. |
|
||||
| `window.go` | 356 | resource owner (instance) — maintenance/backup windows | Pure window arithmetic for an instance's backup and maintenance windows. |
|
||||
| `backup.go` | 597 | **Mixed**: resource owner (instance: retention, windows, `runBackupWindow` `:206`, `runMaintenanceWindow` `:340`) + snapshot (automated snapshot identity `:385-394`) + AWS adapter (`DescribeDBInstanceAutomatedBackups` `:396-536`) | Automated backup is an instance obligation that produces snapshot records. |
|
||||
| `backup_sweep.go` | 340 | resource owner (snapshot) — retention reaper | `BackupRetentionReaper` (`:24-112`) deletes automated snapshots and reclaims retained volumes (`:310`); implements `vm.Reaper` from `runtime/compute/vm` (`:31`). |
|
||||
| `snapshot.go` | 768 | resource owner (snapshot) — tightly coupled to instance | Create/Describe/Delete (`:49`, `:224`, `:299`); create CASes the instance into `backing-up` (`:672-723`) and quiesces the engine. |
|
||||
| `restore.go` | 411 | cross-resource orchestration (snapshot → new instance) | `RestoreDBInstanceFromDBSnapshot` (`:24-162`) reads a snapshot record, creates a volume from it and runs the instance create path. |
|
||||
| `delete.go` | 548 | resource owner (instance) — with snapshot orchestration | `DeleteDBInstance` (`:34`), teardown (`:178-227`), final snapshot (`:247-447`), retained data volume (`:449-535`). |
|
||||
| `agent.go` | 667 | **Mixed**: guest/controller wire contract (IO types, `Command`, `CommandReply`, `InstanceIndexEntry`, `:30-160`, `:628-636`) + resource owner (instance agent-path handlers `:161-627`) | Split: `contracts/rds/v1` vocabulary vs instance owner handlers. |
|
||||
| `commands.go` | 239 | **Mixed**: guest/controller wire contract (command types and params, `:12-40`) + resource owner (instance: `issueCommand` issuer, `:86-230`) | Issuer is instance realization; constants are wire vocabulary. |
|
||||
| `bootstrap.go` | 280 | resource owner (instance) — bootstrap secret | Encrypted master-password payload (`:31-280`), encrypted via `handlers_iam.EncryptSecret` (`:105`, `:199`). |
|
||||
| `servingcert.go` | 109 | resource owner (instance) — realization helper | Per-instance serving cert minted from cluster CA (`:46`). Uncertain whether this should sit with a PKI owner instead. |
|
||||
| `userdata.go` | 68 | guest/controller wire contract | Agent user-data rendering (`:36`), the boot-time contract with the DB VM. |
|
||||
| `iam.go` | 83 | **Mixed**: resource owner (instance — instance profile ensure, `:56-83`) + guest/controller wire contract (`InternalAgentActions`, `:24-30`) | The internal-action list is shared by the gateway gate (`gateway/rds/authz.go:52-63`) and the role grant. |
|
||||
| `dns.go` | 72 | resource owner (instance) — publishes a projection | `DesiredDNSChanges` (`:25-51`) is an RDS-owned projection consumed by the daemon DNS reconcile. |
|
||||
| `events.go` | 312 | cross-resource orchestration (event journal) — uncertain final home | Per-source event rings for instance and snapshot (`:88-141`) plus `DescribeEvents` (`:159`). Not in the ADR-0004 resource table; candidate for the domain root, not a `common` package. |
|
||||
| `tags.go` | 280 | cross-resource orchestration + AWS adapter | One tag implementation dispatching on ARN kind over all four record types (`:51-73`, `:202-230`). Split: each owner exposes a tag mutation on its own record; awsapi routes by ARN. |
|
||||
| `engine.go` | 287 | resource owner (engine) — read-only catalogue | Engine registry, version/username/dbname/password rules (`:15-287`); consumed by the agent (`LookupEngine`). |
|
||||
| `engine_postgres.go` | 505 | resource owner (engine) | Postgres engine catalogue and parameter specs/validators. |
|
||||
| `engine_mariadb.go` | 503 | resource owner (engine) | MariaDB engine catalogue and parameter specs/validators. |
|
||||
| `paramcatalog.go` | 449 | resource owner (engine) | Parameter spec model, size-derived defaults, `ResolveEffectiveParameters` (`:354`); imports `domains/ec2/instancetypes` for class memory (`:219`). |
|
||||
| `catalog.go` | 214 | **Mixed**: resource owner (engine — `EngineVersions`, `OrderableOptions` `:102-141`) + AWS protocol adapter (filter value sets, SDK projection `:142-214`) | |
|
||||
| `sizing.go` | 57 | resource owner (engine/instance-class facade) | `db.*` → EC2 instance type map (`:14-21`), validated against `domains/ec2/instancetypes` (`:30`). |
|
||||
| `subnetgroup.go` | 340 | resource owner (subnetgroup) | Create/Describe/Delete (`:39`, `:94`, `:138`), subnet resolution via EC2 (`:176`). Reads instance records for in-use check (`:258-277`). |
|
||||
| `parametergroup.go` | 599 | resource owner (parametergroup) — with instance orchestration | CRUD + DescribeDBParameters (`:29-380`); `propagateParameterGroup` (`:210-236`) pushes changes into instances; `resolveGroupParameters` (`:549`) is called by instance create/modify/restore. |
|
||||
|
||||
### gateway/rds and gateway/rds.go
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `gateway/rds.go` | 95 | AWS protocol adapter (ingress glue) | Query parse, action check, caller derivation, authz order, quota injection and `ExpectedNodes` injection (`:16-75`). Imports `bluebottle/pkg/auth.ParseRoleARN` (`:87`). |
|
||||
| `gateway/rds/handler.go` | 220 | AWS protocol adapter | Action inventory (registered/internal/unsupported, `:96-169`), typed Query→XML adapter (`:47-79`), `Env{ExpectedNodes, QuotaCheck}` (`:33-43`). |
|
||||
| `gateway/rds/authz.go` | 112 | AWS protocol adapter | Principal-class gate and per-action resource ARN scopes (`:20-112`). |
|
||||
| `gateway/rds/instance.go` | 50 | AWS protocol adapter | Instance actions + `DescribeEvents`; runs `QuotaCheck` before forwarding create (`:13-19`). |
|
||||
| `gateway/rds/snapshot.go` | 35 | AWS protocol adapter | Forwards snapshot/restore/automated-backup actions. |
|
||||
| `gateway/rds/subnetgroup.go` | 23 | AWS protocol adapter | Forwards subnet-group actions. |
|
||||
| `gateway/rds/parametergroup.go` | 31 | AWS protocol adapter | Forwards parameter-group actions. |
|
||||
| `gateway/rds/tags.go` | 23 | AWS protocol adapter | Forwards tag actions. |
|
||||
| `gateway/rds/enginecatalog.go` | 184 | **Mixed**: AWS protocol adapter (engine catalogue actions) + temporary residue (prerequisite: EC2-owned instance-type-availability projection; removal: `clusterRunnableTypes` replaced by that projection) | `clusterRunnableTypes` (`:146-174`) calls the EC2 AWS adapter `ec2instanceapi.DescribeInstanceTypes` with `env.ExpectedNodes`. |
|
||||
| `gateway/rds/agent.go` | 287 | **Mixed**: guest/controller wire contract (agent-facing internal actions, long poll, reply relay `:146-287`) + AWS protocol adapter (agent identity gate `:36-111`) | Reads the `rds-system` bucket directly via JetStream (`:72-100`), i.e. the gateway reads RDS private state. |
|
||||
|
||||
### daemon and other wiring
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `daemon/rds_deps.go` | 98 | cross-resource orchestration (composition root) — contains residue | Binds EC2/VPC/IGW/RT/NGW/EIP/image/volume/snapshot services, IAM ensurer, master key and config into `handlers_rds.Deps` (`:20-85`). `describeInstancesFanOut` (`:90-97`) calls the EC2 AWS adapter with `len(clusterConfig.Nodes)` as expected-node count (residue; same blocker family as `ExpectedNodes`). Loads `bluebottle/pkg/masterkey` (`:52`). |
|
||||
| `daemon/daemon.go` (RDS sections) | n/a | cross-resource orchestration (composition root) | Subscriptions (`:1171-1206`), service init (`:2046-2066`), reconciler start (`:2070-2078`), backup reaper registration (`:2310-2312`), and the RDS reconciler lease reused as the cluster-wide GC election (`:2374-2382`). The last is residue: an RDS lease elects unrelated reapers. |
|
||||
| `daemon/dns_reconcile.go` (`:71-76`, `:99-104`) | n/a | cross-resource orchestration | Watches RDS per-account buckets via `handlers_rds.AccountWatchBuckets` and consumes `DesiredDNSChanges`. Bucket watch is a read of RDS private storage by the daemon. |
|
||||
| `daemon/bedrock_deps.go:43` | n/a | temporary residue (prerequisite: EC2/EBS-owned attach capability; removal: Bedrock and RDS both bind the EC2 one) | Another domain imports RDS for an EC2 volume-attach client. |
|
||||
| `handlers/quota/live.go:82-93` | n/a | temporary residue (prerequisite: RDS-owned count projection per ADR-0004 S4; removal: quota no longer calls `NATSService.DescribeDBInstances`) | Quota counts instances through the AWS-shaped RDS API. |
|
||||
| `accountteardown/reapers_rds.go` | n/a | temporary residue / cross-domain consumer | Drives instance/subnet-group/parameter-group deletion through `NATSService` AWS-shaped calls (`:41`, `:76`, `:87`, `:109`, `:125`, `:139`, `:159`). Account teardown is explicitly out of ADR-0004 scope. |
|
||||
| `daemon/instance_records.go`, `daemon/instance_records_migrate.go` | 217 / 361 | not RDS | EC2 instance record space; excluded. |
|
||||
|
||||
## 2. Served AWS actions and protocol
|
||||
|
||||
Protocol: AWS Query in (form `Action=`), XML out using the IAM-style `<ActionResponse><ActionResult>` envelope (`gateway/rds.go:14-26`, `gateway/rds/handler.go:59-75`).
|
||||
Order: unknown action → `InvalidAction` before policy (`gateway/rds.go:29-32`); principal-class gate before policy (`:41-43`); resource ARN derivation and policy (`:45-51`); dispatch (`:57-64`).
|
||||
Unsupported actions return `OperationNotSupported` from `Dispatch` (`gateway/rds/handler.go:182-185`).
|
||||
Path: gateway adapter → NATS request `rds.<Action>` (queue group `spinifex-workers`) → daemon `Service` method (`daemon/daemon.go:1173-1206`), except where noted.
|
||||
|
||||
| Action | Status | Owning resource | Notes |
|
||||
|---|---|---|---|
|
||||
| CreateDBInstance | registered | instance | `QuotaCheck` in gateway before forward (`gateway/rds/instance.go:13-19`); 5 min budget (`service_nats.go:16`). |
|
||||
| DescribeDBInstances | registered | instance | |
|
||||
| ModifyDBInstance | registered, scoped | instance | Also binds parametergroup. |
|
||||
| DeleteDBInstance | registered, scoped | instance | Orchestrates final snapshot. |
|
||||
| RebootDBInstance / StartDBInstance / StopDBInstance | registered, scoped | instance | |
|
||||
| CreateDBSnapshot | registered, scopes instance + snapshot | snapshot (orchestrates instance) | |
|
||||
| DescribeDBSnapshots | registered | snapshot | |
|
||||
| DeleteDBSnapshot | registered, scoped | snapshot | |
|
||||
| RestoreDBInstanceFromDBSnapshot | registered, scopes snapshot + instance | cross-resource (snapshot → instance) | |
|
||||
| DescribeDBInstanceAutomatedBackups | registered | instance (automated backup) / snapshot | |
|
||||
| CreateDBSubnetGroup / DescribeDBSubnetGroups | registered | subnetgroup | |
|
||||
| DeleteDBSubnetGroup | registered, scoped | subnetgroup | |
|
||||
| CreateDBParameterGroup / DescribeDBParameterGroups | registered | parametergroup | |
|
||||
| ModifyDBParameterGroup / DescribeDBParameters / DeleteDBParameterGroup | registered, scoped | parametergroup | Modify propagates to instances. |
|
||||
| AddTagsToResource / RemoveTagsFromResource / ListTagsForResource | registered, ARN scope | per ARN kind (all four) | `tags.go:51-73`. |
|
||||
| DescribeEvents | registered | cross-resource (events) | |
|
||||
| DescribeDBEngineVersions | registered | engine | Served **in the gateway process** with no NATS hop (`gateway/rds/enginecatalog.go:41-73`). |
|
||||
| DescribeOrderableDBInstanceOptions | registered (`typedEnv`) | engine (+ EC2 availability) | Served in the gateway; NATS fan-out to EC2 `DescribeInstanceTypes` using `ExpectedNodes` (`gateway/rds/enginecatalog.go:146-174`). |
|
||||
| RegisterDBInstance, SubmitDBStateChange | registered, internal | instance (agent path) | Gateway → bus `rds.bus.{acct}.{db}.register/health` (`service_nats.go:49-57`). |
|
||||
| GetDBBootstrapConfig, AcknowledgeDBBootstrap | registered, internal | instance (bootstrap) | Gateway → `rds.GetDBBootstrapConfig` / `rds.AcknowledgeDBBootstrap`. |
|
||||
| PollDBCommands | registered, internal | instance (command channel) | Served **in the gateway**: queue-subscribes the bus command subject; no daemon handler (`gateway/rds/agent.go:182-247`). |
|
||||
| CreateDBInstanceReadReplica, PromoteReadReplica, CreateDBCluster, ModifyDBCluster, DeleteDBCluster, DescribeDBClusters, FailoverDBCluster, CreateOptionGroup, ModifyOptionGroup, DeleteOptionGroup, DescribeOptionGroups, RestoreDBInstanceToPointInTime | unsupported (12) | n/a | `gateway/rds/handler.go:157-168`. |
|
||||
|
||||
Totals: 26 customer actions registered, 5 internal agent actions, 12 unsupported, 43 in the inventory (`ActionNames`/`UnsupportedActionNames` feed `gateway/operations.go:43-45`).
|
||||
Daemon subscribes 29 `rds.*` subjects (`daemon/daemon.go:1175-1205`); the two catalogue actions and `PollDBCommands` have no daemon subject.
|
||||
Any action outside the table and not in the map is `InvalidAction`.
|
||||
|
||||
## 3. Durable records and KV buckets
|
||||
|
||||
| Bucket | Key | Record | Owning resource (proposed) | Writers | Readers (flag = outside RDS owner) |
|
||||
|---|---|---|---|---|---|
|
||||
| `rds-account-{accountID}` (history 1, version 1, lazily created; `store.go:20-38`, `:201-213`) | `db-instances/{id}` | `DBInstanceRecord` | instance | create, restore, lifecycle, modify, pending, replace, agent, delete, reconciler, recovery, backup, snapshot (`backing-up` CAS), tags | subnetgroup (`subnetgroup.go:258-277`), parametergroup (`parametergroup.go:211`, `:342`), dns.go, **daemon DNS reconcile via bucket watch** (`daemon/dns_reconcile.go:75`), **quota** indirectly via `DescribeDBInstances` (`handlers/quota/live.go:89`), **account teardown** via `NATSService` |
|
||||
| same | `bootstrap-payloads/{id}` | `BootstrapPayloadEnvelope` (encrypted) | instance | create, restore, replace (via bootstrap.go) | agent.go (`GetDBBootstrapConfig`, `AcknowledgeDBBootstrap`), reconciler (stale pending report) |
|
||||
| same | `db-snapshots/{id}` (colon → `.`, `store.go:84-97`) | `DBSnapshotRecord` (manual and automated) | snapshot | snapshot.go, delete.go (final snapshot), backup.go (automated) | restore.go, backup_sweep.go, reconciler snapshot pass, describe, tags |
|
||||
| same | `backups/{dbId}/automated/{ts}` | `AutomatedBackupRecord` | snapshot (automated-backup index) — instance obligation | backup.go | backup_sweep.go, DescribeDBInstanceAutomatedBackups |
|
||||
| same | `retained-volumes/{volumeID}` | `RetainedVolumeRecord` | snapshot (cleanup obligation for a former instance's volume) | delete.go `retainDataVolume` (`:484`) | snapshot.go `releaseRetainedVolume`/`reclaimRetainedVolume` (`:414`, `:430`), backup_sweep `reclaimOrphanedVolumes` (`:310`) |
|
||||
| same | `db-subnet-groups/{name}` | `DBSubnetGroupRecord` | subnetgroup | subnetgroup.go, tags.go | network.go `resolvePlacement` (`:43`, instance create/restore), describe, **account teardown** via `NATSService` |
|
||||
| same | `db-parameter-groups/{name}/meta` | `DBParameterGroupRecord` | parametergroup | parametergroup.go, tags.go | instance create/modify/restore via `resolveGroupParameters` (`parametergroup.go:549`), pending.go |
|
||||
| same | `db-parameter-groups/{name}/params/{key}` | `DBParameterRecord` | parametergroup | parametergroup.go (`:188`, delete `:361`) | `ListDBParameterOverrides` (`store.go`), instance apply paths |
|
||||
| same | `events/{sourceType}/{sourceId}` | event ring (bounded, 14-day trim) | events (cross-resource) | `RecordEvent` from every resource (`events.go:88`) | `DescribeEvents` |
|
||||
| `rds-system` (history 1, version 1; `store.go:24-26`, `:184-195`) | `instance-index/{ec2InstanceID}` | `InstanceIndexEntry{AccountID, DBInstanceIdentifier, VMGeneration}` (`agent.go:628-636`) | instance (agent identity binding) | create (`create.go:124`), restore (`restore.go:142`), replace (`replace.go:200-215`), delete teardown | **gateway reads JetStream directly** (`gateway/rds/agent.go:72-100`); `Service.LookupInstanceIndex` (`agent.go:637`) |
|
||||
| `spinifex-rds-leader` (TTL 60s; `store.go:28-33`, `:216-223`) | `reconciler` | kvlease record | domain-root reconciler leadership | `kvlease` via `Reconciler` (`reconciler.go:104-118`) | **daemon GC uses it as the cluster-wide reaper election** (`daemon/daemon.go:2374-2382`) |
|
||||
|
||||
Non-durable state that affects decisions: in-memory heartbeat liveness map per daemon (`service.go` `liveness`, `agent.go:284-321`), persisted to KV only on change or every `HeartbeatPersistFloor` (5 beats, `records.go` constants).
|
||||
The reconciler leader reads this node-local map (`recovery.go:191`), so a beat answered by another node's queue-group member is seen only via the KV floor (uncertain impact; observation, not intent).
|
||||
|
||||
## 4. Identity and scope per resource
|
||||
|
||||
| Resource | Identifier | Immutable internal ID | ARN | Account | Region | AZ |
|
||||
|---|---|---|---|---|---|---|
|
||||
| instance | `DBInstanceIdentifier`, account-unique via `createJSONRevision` (`create.go:62`), reusable after delete | `DbiResourceID` `db-…` (`create.go:153`) | `arn:aws:rds:{region}:{acct}:db:{id}` (`arn.go:15-27`) | customer account in bucket name and record; VM, system ENI and IAM profile live in the system account `GlobalAccountID` (`create.go:51-55`, `delete.go:544-546`) | not stored; taken from `Service.region` at projection (`service.go`) | not recorded on the instance; derived only via subnet (uncertain whether projected at all; no `AvailabilityZone` reference in `describe.go`) |
|
||||
| snapshot | `DBSnapshotIdentifier`; automated ones `rds:{db}-{ts}` (`backup.go:385`) | none beyond backing EC2 `SnapshotID` | `…:snapshot:{id}` | customer | from service | none |
|
||||
| subnetgroup | `DBSubnetGroupName` (case-sensitive key; `default` rejected, `subnetgroup.go:318`) | none | `…:subgrp:{name}` | customer | from service | each member subnet carries its AZ (`records.go` `DBSubnetGroupSubnet`) |
|
||||
| parametergroup | `DBParameterGroupName`; `default.*` synthesized per engine, not stored (`parametergroup.go:24`, `:456-482`) | none | `…:pg:{name}` | customer | from service | n/a |
|
||||
| engine | engine name + version | n/a (static) | none | global | global | n/a |
|
||||
| agent identity | EC2 instance ID as `RoleSessionName` of the system instance role (`gateway/rds/agent.go:36-69`) | `VMGeneration` in index | n/a | system account principal mapped to customer account via index | n/a | n/a |
|
||||
|
||||
## 5. Lifecycle paths per resource
|
||||
|
||||
### instance
|
||||
|
||||
- Create: validate (`validate.go:69`) → placement reads subnet group / default VPC (`network.go:36`) → resolve parameter group (`create.go:47`) → ensure IAM instance profile in system account (`create.go:51`) → reserve record at `creating` with `createJSONRevision` (`create.go:62`) → stage encrypted bootstrap payload (`create.go:83`) → `LaunchDBInstanceVM` inline in the request (`create.go:87`) → `recordLaunch` CAS (`create.go:115`, `:195-244`) → write instance index (`create.go:124`) → event, return `creating`.
|
||||
- Create failure: deferred rollback deletes payload and CAS-deletes the record if `DbiResourceID` unchanged (`create.go:246-295`); post-launch failures run `Unwind` (`create.go:297-305`).
|
||||
- Readiness: reconciler `reconcileCreating` moves `creating → available` on first healthy agent beat (`reconciler.go:448-464`); `failed` after `BootstrapTimeout` (default 20 min, `reconciler.go:30`).
|
||||
- Steady state: `reconcileHealth` (`recovery.go:76-101`) records/clears failure; `StatusRecovering` exists in the table (`status.go:22`, `:34`, `:43`, `:47`) but **no code transitions into it** (only `status.go` references it), so there is no automatic repair.
|
||||
- Reboot/Start: `beginTransition` CAS (`lifecycle.go:65`, `:143`) then reconciler `reconcileRestarting` (`reconciler.go:479`).
|
||||
- Stop: CAS to `stopping`, inline engine stop + VM stop, inline `completeTransition` to `stopped` before responding (`lifecycle.go:103-133`); reconciler resumes a dead stop (`reconciler.go:584-605`).
|
||||
- Modify: immediate non-disruptive in-request; disruptive writes `PendingModifiedValues` (`modify.go:153`), then either waits for the maintenance window (`backup.go:340`) or runs under the modify lease (`modify.go:176`); reconciler resumes under the same lease (`reconciler.go:510-583`); class/storage change replaces the VM (`pending.go:80`, `replace.go:43-166`).
|
||||
- Replace: launch new VM (`replace.go:113`), rewrite index (`:125`), CAS record with `VMGeneration++` (`:137`), rollback on error (`:218`).
|
||||
- Delete: validate, deletion protection, reserve final snapshot (`delete.go:68`), CAS to `deleting` (`delete.go:73-91`), inline teardown (`delete.go:178-227`): stop engine, terminate VM, final snapshot, purge automated backups, release or retain data volume, delete ENIs (best-effort), delete index, delete payload, delete record last; reconciler replays teardown (`reconciler.go:611-628`) and marks `failed` after `transitionTimeout` 10 min.
|
||||
- Fencing: KV revision CAS on the record everywhere; per-instance `ModifyLease` on the record (`modify_lease.go:24-173`); cluster `kvlease` for the reconciler (`reconciler.go:104-118`); `VMGeneration` binds bootstrap payloads and acknowledgements (`agent.go:345`, `:470`, `:552-553`) and the index.
|
||||
- No desired generation exists on `DBInstanceRecord` (`records.go`); `VMGeneration` is a realization generation, not desired-intent generation.
|
||||
- Idempotency: AWS `CreateDBInstance` has no client token, and a repeat returns `DBInstanceAlreadyExists` (`create.go:64-66`); delete retry re-runs teardown and must repeat the snapshot choice (`delete.go:57-63`); register is idempotent (`agent.go:159-160`).
|
||||
|
||||
### snapshot
|
||||
|
||||
- Create: check name free (`snapshot.go:92`) → CAS instance to `backing-up` with `SnapshotOperation` (`:672-705`) → create record `creating` (`:109-115`) → quiesce + EC2 snapshot inline (`:144-188`) → `putJSON` record `available` (`:126-130`) → restore instance status (`:706-723`).
|
||||
- Readiness: synchronous; the API returns `available`.
|
||||
- Repair: reconciler resolves records stuck `creating` past 15 min against EC2 (`reconciler.go:674-751`, `findEC2SnapshotFor` `:752`); returns instances stuck in `backing-up` (`reconciler.go:630-652`).
|
||||
- Delete: `removeDBSnapshot` deletes EC2 snapshot then record and releases a retained volume when last holder (`snapshot.go:330-470`).
|
||||
- Automated: window-driven `takeAutomatedBackup` (`backup.go:270`), stamped on instance record so a leader change cannot double-fire (`reconciler.go:654-660` comment); retention sweep `BackupRetentionReaper` via the daemon GC backstop (`backup_sweep.go:45-112`, `daemon/daemon.go:2310`).
|
||||
- Fencing: instance-record CAS into `backing-up`; `createJSON` on snapshot key; no snapshot generation.
|
||||
|
||||
### subnetgroup
|
||||
|
||||
- Create: validate, resolve subnets through EC2 `DescribeSubnets` (`subnetgroup.go:176-239`), `createJSON` (`:78`); synchronous; status always `Complete` (`:31`, `:287`).
|
||||
- Describe: KV read and list (`:94-133`).
|
||||
- Delete: read, scan all instance records for `DBSubnetGroupName == name` (`:154-163`), plain `kv.Delete` (`:165`).
|
||||
- No modify action is served (`ModifyDBSubnetGroup` is neither registered nor unsupported, so it returns `InvalidAction`).
|
||||
- No reconciliation; no generation; tags via `tags.go` CAS.
|
||||
|
||||
### parametergroup
|
||||
|
||||
- Create: validate family against engine registry, `createJSON` meta (`parametergroup.go:29-84`); synchronous.
|
||||
- Modify: validate whole batch, then `putJSON` one key per parameter (`:186-196`), then synchronously push to every attached instance via agent `apply-params` (`:198`, `:210-236`, `pending.go:149`).
|
||||
- Delete: refuse `default.*`, scan instances for current or pending use (`:342-352`), delete params then meta (`:354-367`).
|
||||
- `default.*` groups are synthesized from the engine catalogue (`:456-482`).
|
||||
- No generation; no reconciliation of a failed propagation beyond `ParameterApplyFailed` on the instance (`pending.go:214`).
|
||||
|
||||
### engine
|
||||
|
||||
- Static, process-local catalogue validated at daemon boot (`engine.go:69`, `daemon/daemon.go:2055`); no lifecycle; ADR-0003 S1 excludes pure read-only catalogues from the lifecycle contract.
|
||||
|
||||
## 6. NATS subjects and controller-to-agent messages
|
||||
|
||||
Gateway → daemon (request/reply, queue group `spinifex-workers`): `rds.<Action>` for the 24 customer actions listed in `subjects.go:21-63`, plus `rds.GetDBBootstrapConfig` and `rds.AcknowledgeDBBootstrap` (`subjects.go:17-19`).
|
||||
Budgets: default 30 s, create 5 min, lifecycle 4 min, delete 6 min, modify 6 min, snapshot 6 min, snapshot delete 4 min, restore 8 min (`service_nats.go:12-36`).
|
||||
A gateway timeout does not cancel the daemon's inline orchestration (uncertain; inferred from request/reply shape), so the client may see an error while the create continues.
|
||||
|
||||
Agent path (agent speaks SigV4 Query to awsgw over the mgmt bridge, `daemon/rds_deps.go:71-73`):
|
||||
|
||||
- `RegisterDBInstance` → gateway → `rds.bus.{acct}.{db}.register` → daemon `SubjectRegisterWildcard` (`subjects.go:12-13`, `:65-67`).
|
||||
- `SubmitDBStateChange` (heartbeat folded into state, 30 s interval, stale after 90 s; `records.go` constants) → `rds.bus.{acct}.{db}.health`.
|
||||
- `GetDBBootstrapConfig` / `AcknowledgeDBBootstrap` → Layer-1 subjects; replay is allowed until acknowledged for the bound `VMGeneration` (`gateway/rds/agent.go:150-180`).
|
||||
- `PollDBCommands` long poll (1-20 s) queue-subscribes `rds.bus.{acct}.{db}.command` with group `spinifex-rds-agents` and publishes carried replies to `rds.bus.{acct}.{db}.command-reply` before subscribing (`gateway/rds/agent.go:198-287`).
|
||||
- Command types: `set-password`, `apply-params`, `stop-engine`, `grow-filesystem`, `quiesce`, `unquiesce` (`commands.go:12-23`); params `MasterUsername`, `MasterUserPassword`, `QuiesceLabel`, `QuiesceDeadlineSeconds` (`commands.go:27-35`).
|
||||
- Issuer: `issueCommand` subscribes the reply subject, then republishes the same `CommandID` every 2 s until reply or budget (`commands.go:144-199`, `:72`); core NATS, not durable, by design for the password (`subjects.go:77-80`).
|
||||
- Agent retry: results are retained and re-carried until a poll succeeds (`agents/rds/agent/command.go:100-133`).
|
||||
- Agent dedupe: the agent executes every delivered command and has **no CommandID dedupe** (`agents/rds/agent/command.go:128-131`, `:140-157`); a republish delivered after execution but before the issuer sees the reply can execute twice (uncertain; depends on timing between reply relay and the 2 s republish).
|
||||
- Commands carry no `VMGeneration` (`agent.go:134-139`), so the fence against a superseded VM is the bus subject plus the gateway's index lookup rather than a generation in the message.
|
||||
- `LookupEngine`, `Engine`, `ParamType*`, `ApplyType*`, `Parameter`, `EngineHealth*`, `HeartbeatInterval`, `BootstrapMode*`, `ParameterRollbackMessage`, `SupportedInstanceClasses`, `SmallestInstanceClass` are imported by the agent from `handlers/rds` (full list from import scan); this is the S5 "accidental import of a domain monolith".
|
||||
|
||||
RDS → EC2 subjects used as capabilities: `ec2v1.InstanceCommandSubject(id)` for stop/reboot/attach (`lifecycle.go:434`, `launch.go:583`), `ec2.start` (`lifecycle.go:451`), `ec2.DescribeStoppedInstances` (`launch.go:616`), plus EC2 `DescribeInstances` fan-out (`daemon/rds_deps.go:90-97`) and `DescribeInstanceTypes` fan-out (`gateway/rds/enginecatalog.go:147`).
|
||||
|
||||
Recovery/redelivery: there is no durable work queue; recovery relies on the reconciler re-reading record status (`reconciler.go:238-282`) under `kvlease`, a 5 min resync and KV watch wakeups (`reconciler.go:120-145`).
|
||||
|
||||
## 7. Boundary-crossing imports
|
||||
|
||||
### Outbound from RDS (handlers/rds, gateway/rds, gateway/rds.go, daemon/rds_deps.go)
|
||||
|
||||
- **Quota**: none from `handlers/rds`; the gateway injects `gw.Quota.EnforceRDSInstances` as `Env.QuotaCheck` (`gateway/rds.go:59-62`).
|
||||
- **ExpectedNodes / EC2 instance availability**: `gateway/rds/enginecatalog.go` → `domains/ec2/awsapi/instance.DescribeInstanceTypes` with `env.ExpectedNodes` (`:147`); `daemon/rds_deps.go` → `domains/ec2/awsapi/instance.DescribeInstances` with node count (`:96`).
|
||||
- **EC2 instance**: `domains/ec2/systeminstance` (`launch.go:99-100`, `delete.go:236`), `domains/ec2/instance` (`lifecycle.go`, `StartStoppedInstanceOutput`), `contracts/ec2/v1` (`launch.go`, `lifecycle.go`), `runtime/compute/vm` (`agent.go`, `create.go` `vm.VolumeSerial`, `backup_sweep.go` `vm.Reaper`), `foundation/aws/ami`, `domains/ec2/instancetypes` (`sizing.go:30`, `paramcatalog.go:219`).
|
||||
- **VPC/network**: `domains/network/systemvpc` (`vpc.go`, `launch.go`, `daemon/rds_deps.go`), EC2 SDK VPC/subnet/SG/ENI calls through narrow interfaces `networkResolver` (`network.go:16`) and `launchVPCProvisioner` (`launch.go:76-96`); `foundation/aws/tags` (`ManagedByRDS`).
|
||||
- **Storage/EBS**: EC2 SDK `CreateVolume`/`DeleteVolume` (`launch.go:107-111`), `DescribeVolumes`/`ModifyVolume` (`storage.go:23-26`), EC2 snapshot provider (`snapshot.go:25`), volume attach over EC2 NATS (`launch.go:560-629`).
|
||||
- **IAM**: `handlers/iam` for `SystemInstanceRoleEnsurer`/`EnsureSystemInstanceProfile` (`iam.go:56-77`) and `EncryptSecret`/`DecryptSecret` for bootstrap payloads (`bootstrap.go:105`, `:199`).
|
||||
- **DNS**: `domains/dns` (`dns.go:17`, `dns.RDSName`, `dns.Change`).
|
||||
- **Bluebottle**: `bluebottle/pkg/auth.ParseRoleARN` (`gateway/rds.go:87`), `bluebottle/pkg/masterkey.ReadShared` (`daemon/rds_deps.go:52`); `handlers/rds` imports no Bluebottle package.
|
||||
- **Ingress/foundation**: `ingress/aws/query`, `foundation/aws/xml`, `foundation/aws/errors`, `foundation/aws/identifiers`, `foundation/state/{kvstore,kvlease,kvutil,migrate}`, `foundation/lifecycle/reconciler`, `foundation/messaging/nats`, `foundation/pki`, `bootstrap/config`.
|
||||
|
||||
### Inbound into RDS (who imports `handlers/rds` or `gateway/rds`)
|
||||
|
||||
- `handlers/quota/live.go:14,89` → `NewNATSService(...).DescribeDBInstances` (quota reads RDS through the AWS-shaped API).
|
||||
- `accountteardown/reapers_rds.go:9` → `NATSService` describe/modify/delete for instances, subnet groups, parameter groups.
|
||||
- `daemon/daemon.go:78` → `Service`, `Reconciler`, all `Subject*`, `ValidateEngineRegistry`, `NewService`, `NewReconciler`, `NewBackupRetentionReaper`.
|
||||
- `daemon/rds_deps.go` → `Deps`, `LaunchDeps`, `BackupPolicy`, `NewNATSVolumeAttacher`, `NewDescribeInstanceState`, `NewNATSInstanceCommander`.
|
||||
- `daemon/dns_reconcile.go:16,75` → `AccountWatchBuckets` (raw bucket handles).
|
||||
- `daemon/bedrock_deps.go:10,43` → `NewNATSVolumeAttacher` (Bedrock/Ochre depends on RDS).
|
||||
- `gateway/operations.go:8,44-45` → `gateway_rds.ActionNames`, `UnsupportedActionNames`.
|
||||
- `gateway/rds/*` → `KVBucketRDSSystem`, `InstanceIndexKey`, `InstanceIndexEntry`, `InstanceRoleName`, `InternalAgentActions`, `ResourceKind*`, `FormatARN`, `ParseARN`, catalogue functions, `NATSService`, wire types.
|
||||
- `agents/rds/agent/*` → wire types and engine catalogue (section 6).
|
||||
- `tests/integration/service_daemon_lite.go:51`, `tests/e2e/harness/rds.go` (test support).
|
||||
|
||||
## 8. Known gaps against Q-78 / ADR-0003 (observed in code, not live-verified)
|
||||
|
||||
1. No desired generation on any RDS record (`records.go`); mutations advance only the KV revision, and `PendingModifiedValues` is overwritten by a later modify (`modify.go:153`), so Q-78 "atomically commits a monotonically increasing desired generation" is not met.
|
||||
2. Desired and observed state share one record: `Status`, `Agent`, `UnhealthySince`, `FailureReason` sit beside requested class/storage on `DBInstanceRecord` (`records.go`).
|
||||
3. Create performs all external realization inline in the request (`create.go:87`); if the daemon dies after `LaunchDBInstanceVM` returns and before `recordLaunch` (`create.go:115`), the record stays `creating` with no `InstanceID`, times out to `failed` (`reconciler.go:458-462`), and a later delete has no IDs to clean (`delete.go:229-233`, `:537-547`); the VM, ENIs and volume are tagged `ManagedBy=rds` (`launch.go:260`, `:314`, `:400`) but no RDS orphan reaper for them was found.
|
||||
4. The same interruption window exists for restore (`restore.go:111` → `:136`) and replace (`replace.go:113` → `:137`); replace's rollback runs only on error return, not on process death.
|
||||
5. Delete removes the only ownership record (`delete.go:217`) after ENI deletion that is best-effort and only logs failures (`delete.go:203`, `launch.go:510-520`), contrary to Q-78 "a handler never removes the only reconstructible ownership of required cleanup".
|
||||
6. `deleting` that exceeds 10 min becomes `failed` (`reconciler.go:621-624`); a `failed` instance is then handled by `reconcileHealth` rather than continuing teardown, so incomplete cleanup is not shown as a distinct state (uncertain; inferred from the `switch` at `reconciler.go:256-272`).
|
||||
7. `StatusRecovering` is declared and reachable in the transition table but never entered, so there is no repair path (`status.go:22`; `recovery.go:19-21` says v1.0 reports rather than repairs).
|
||||
8. Synchronous completion where AWS is asynchronous: AWS `CreateDBSnapshot` returns `Status=creating`, here it returns `available` after the snapshot is taken inline (`snapshot.go:126-130`); AWS `StopDBInstance` returns `stopping`, here the handler completes to `stopped` inline (`lifecycle.go:126`); AWS `DeleteDBInstance` leaves the instance describable as `deleting`, here the record is deleted before the response when teardown succeeds inline (`delete.go:97-104`, `:217`).
|
||||
9. DB subnet group and parameter group deletes are check-then-delete across keys with no fence against a concurrent `CreateDBInstance` that has already read the group (`subnetgroup.go:151-165`, `parametergroup.go:339-367`; `network.go:43`, `create.go:47`), so an instance can reference a deleted group.
|
||||
10. `ModifyDBParameterGroup` writes one key per parameter with no batch atomicity (`parametergroup.go:186-196`); a KV failure mid-loop leaves a half-applied batch despite the comment at `:140-143`, and propagation to instances is synchronous best-effort (`:210-236`).
|
||||
11. Quota admission is a non-atomic count before create (`gateway/rds/instance.go:13-19`, `handlers/quota/live.go:85-93`), so concurrent creates can exceed the cap; it counts `len(DescribeDBInstances)` which pages at 100 by default (`paging.go:14`, `describe.go:71`), so an account above 100 instances is under-counted (uncertain whether quota limits ever exceed 100).
|
||||
12. Agent commands are republished every 2 s with no agent-side `CommandID` dedupe and no generation (section 6), so internal redelivery is not proven idempotent for `grow-filesystem`, `apply-params` or `set-password`.
|
||||
13. Heartbeat liveness lives partly in per-node process memory (`agent.go:284-321`) and the leader judges health from its own map plus a KV floor (`recovery.go:191-209`), which Q-78 discourages for recovery decisions (observation-side, lower severity).
|
||||
14. The RDS reconciler lease doubles as the cluster-wide GC election for unrelated reapers (`daemon/daemon.go:2374-2382`); without RDS service init every cluster-wide reaper is skipped.
|
||||
15. No lifecycle contract record exists for any RDS resource (ADR-0003 S1); nothing in `handlers/rds` states readiness predicates or deletion obligations as a contract.
|
||||
|
||||
## 9. First-resource candidates
|
||||
|
||||
| Rank | Resource | Size (prod lines) | Private state | Dependencies | Tests | Extractable without quota / ExpectedNodes / IAM / Bluebottle? |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | **subnetgroup** | ~340 (`subnetgroup.go`) + its slice of `store.go`, `records.go`, `tags.go`, `arn.go`; gateway 23 | `db-subnet-groups/{name}` only | EC2 `DescribeSubnets` through a narrow interface already (`network.go:16-20`, `subnetgroup.go:189`); reverse dependency: needs an instance-owned query "instances referencing subnet group X" to replace `instancesUsingGroup` (`subnetgroup.go:154`); forward dependency: instance placement reads the group record directly (`network.go:43`), which must become a subnetgroup query | `handlers/rds/subnetgroup_test.go` (338), `tags_test.go`, `gateway/rds/handler_test.go`/`authz_test.go`, e2e `tests/e2e/rds/groups_test.go` (`TestSubnetAndParameterGroups`), `isolation_test.go`, `api_matrix_test.go` (uncertain coverage depth for the last two) | **Yes.** No quota, ExpectedNodes, IAM, Bluebottle, agent or EC2-launch dependency. Synchronous AWS semantics (status `Complete`) match the code. Gap 9 (delete race) should be recorded, not fixed, during the move. |
|
||||
| 2 | **parametergroup** | ~600 (`parametergroup.go`) + slice of store/records/tags; gateway 31 | `db-parameter-groups/{name}/meta`, `…/params/{key}` | Engine catalogue (`engineForFamily`, `LookupParameter`, `ResolveEffectiveParameters`) — owned by **engine**; `SmallestInstanceClass` (EC2 `instancetypes`); cluster CA availability (`checkTLSEnforceable`, `parametergroup.go:581`); instance records for in-use check; **instance orchestration** in Modify (`propagateParameterGroup` → `applyParameterGroup` → agent `apply-params`) | `parametergroup_test.go` (806), `paramcatalog_test.go`, `enginefamily_test.go`, `tlsenforcement_test.go` (uncertain scope), `pending_test.go` (propagation), e2e `groups_test.go` | **Mostly.** No quota/ExpectedNodes/IAM/Bluebottle, but Create/Describe/Delete are synchronous while Modify crosses into instance realization and the agent wire contract; extraction requires engine first (or a narrow engine capability) and moving propagation to explicit cross-resource orchestration. |
|
||||
| 3 | **engine** | ~1,960 (`engine.go`, `engine_postgres.go`, `engine_mariadb.go`, `paramcatalog.go`, `catalog.go`, `sizing.go`) | none (static catalogue) | `domains/ec2/instancetypes`; consumed by the agent, parametergroup, instance | `engine_test.go`, `engine_mariadb_test.go`, `enginefamily_test.go`, `paramcatalog_test.go`, `catalog_test.go`, `sizing_test.go`, `gateway/rds/enginecatalog_test.go` | **Package: yes**; it is the natural S5 agent capability. **awsapi action `DescribeOrderableDBInstanceOptions`: no**, blocked on the EC2 instance-type-availability projection (`gateway/rds/enginecatalog.go:146-174`). Not a durable resource under ADR-0003 S1, so it is weak evidence for ADR-0004 rollout item 2. |
|
||||
| 4 | **snapshot** | ~1,700 (`snapshot.go`, `backup_sweep.go`, share of `backup.go`, `restore.go`, `reconciler.go:674-796`) | `db-snapshots/`, `backups/…/automated/`, `retained-volumes/` | Instance record CAS into `backing-up`, engine quiesce over agent commands, EC2 snapshot/volume services, instance delete (final snapshot, retained volumes), restore runs the instance create path | `snapshot_test.go` (879), `restore_test.go`, `backup_test.go` (1013), `backup_sweep_test.go`, `delete_test.go`; e2e `snapshot_restore_test.go`, `automated_backup_test.go` | No quota/ExpectedNodes/IAM/Bluebottle on snapshot itself, but restore inherits instance create (IAM, launch); snapshot creation needs instance-owned quiesce/hold capabilities first. Not a first candidate. |
|
||||
| 5 | **instance** | ~8,000+ (create, describe, lifecycle, launch, network, vpc, storage, modify, modify_lease, pending, replace, recovery, reconciler, window, backup, delete, agent, commands, bootstrap, servingcert, userdata, iam, dns, validate, status) | `db-instances/`, `bootstrap-payloads/`, `rds-system` `instance-index/`, `spinifex-rds-leader` | Quota admission, ExpectedNodes-style EC2 fan-out (`daemon/rds_deps.go:90-97`), IAM ensurer and secret crypto, Bluebottle master key (via composition), EC2 system instance / ENI / volume / snapshot / attach, system VPC, DNS, agent wire contract | `create_test.go`, `launch_test.go`, `lifecycle_test.go`, `delete_test.go`, `modify_test.go`, `modify_lease_test.go`, `pending_test.go`, `replace_test.go`, `reconciler_test.go`, `recovery_test.go`, `revisit_test.go`, `agent_test.go`, `bootstrap_test.go`, `commands_test.go`, `storage_test.go`, `network_test.go`, `vpc_test.go`, `describe_test.go`, `status_test.go`, `iam_test.go`, `userdata_test.go`, `watch_source_test.go`, `gateway/rds/agent_test.go`, `authz_test.go`; e2e `create_describe`, `lifecycle`, `modify`, `delete`, `failure`, `graceful_reboot`, `connect`, `cross_subnet`, `isolation`, `authz`, `mariadb*` | **No.** Every listed blocker applies. Extract last. |
|
||||
|
||||
Cross-resource files to settle alongside the first extraction: `tags.go` (per-ARN dispatch), `events.go` (journal), `arn.go`/`filters.go`/`paging.go` (awsapi), `store.go`/`records.go`/`service.go` (split per owner).
|
||||
|
||||
Recommendation: extract **subnetgroup** first, with two narrow capabilities defined by the consumer: an instance-owned "references to subnet group" query for delete, and a subnetgroup-owned "resolve placement subnets" query for instance create.
|
||||
Then do **engine** (package and agent capability, leaving the orderable action on the legacy path until the EC2 projection exists), then **parametergroup** with propagation made explicit orchestration.
|
||||
Test coverage for subnetgroup is unit plus one e2e test; ADR-0003 S5 failure/recovery evidence for its delete race does not exist yet.
|
||||
@@ -1,416 +0,0 @@
|
||||
# EKS managed add-on: per-bundle readiness and removal inventory
|
||||
|
||||
Code read at 3a55e2ee1; not verified live.
|
||||
|
||||
This is a discovery record for the add-on lifecycle work: for every bundle in the catalogue it lists what is rendered, how identity is wired, what "ready" currently proves, what removal deletes and leaves, and where the bundle collides with other objects.
|
||||
It proposes no design; gaps are stated per bundle and collected at the end.
|
||||
It complements `docs/package-boundary/eks-addon-lifecycle.md` (present contract) and `docs/package-boundary/eks-addon-lifecycle-design.md` (draft correction, which refines open proposal Q-146: delete holds until owned objects and identity associations are observably removed, and `ACTIVE` means workloads applied and ready).
|
||||
Paths starting `spinifex/` are Go sources; `scripts/` and `tests/` are relative to the spinifex repo root; bundle files are under `scripts/images/eks-node/addons/<addon>/<version>/` and are abbreviated to `<addon>/<version>/<file>`.
|
||||
The sync agent `scripts/images/eks-node/mulga-eks-addon-sync.sh` is abbreviated to `sync.sh`.
|
||||
|
||||
## 1. Catalogue
|
||||
|
||||
| Name | Version | `RequiresIRSA` | Hidden | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| `aws-load-balancer-controller` | 2.11.0 | true | no | `spinifex/domains/eks/addon/catalog.go:31-33` |
|
||||
| `argocd` | 3.0.23 | false | no | `catalog.go:34-36` |
|
||||
| `aws-ebs-csi-driver` | 1.40.1 | true | no | `catalog.go:37-39` |
|
||||
| `nvidia-device-plugin` | 0.17.4 | false | yes (auto-staged by GPU node groups) | `catalog.go:42-44`, `spinifex/domains/eks/addon/owner.go:79-89`, `spinifex/handlers/eks/nodegroup.go:408-416` |
|
||||
| `spinifex-noop` | 0.1.0 | false | yes (e2e fixture) | `catalog.go:48-50` |
|
||||
|
||||
`RequiresIRSA` is read only by `DescribeAddonVersions` as `requiresIamPermissions` (`spinifex/handlers/eks/addons.go:255`); `CreateAddon` neither requires a role for the two `true` bundles nor rejects one for the three `false` bundles (`addons.go:61-99`).
|
||||
The bundle manifests are baked into the AMI (`scripts/images/eks-node/manifest.conf:58-67`), but container images are not: every image below is pulled from a public registry at pod start.
|
||||
|
||||
## 2. AWS reference semantics
|
||||
|
||||
Checked against the API reference on 2026-10-09.
|
||||
|
||||
| Item | AWS documents | Source |
|
||||
|---|---|---|
|
||||
| `status` | Enum `CREATING`, `ACTIVE`, `CREATE_FAILED`, `UPDATING`, `DELETING`, `DELETE_FAILED`, `DEGRADED`, `UPDATE_FAILED`; the API reference gives no per-value definition of `ACTIVE` or `DEGRADED`. | API_Addon |
|
||||
| `health` | `AddonHealth.issues`, a list of `AddonIssue`. | API_AddonHealth |
|
||||
| `AddonIssue` | `code` one of `AccessDenied`, `InternalFailure`, `ClusterUnreachable`, `InsufficientNumberOfReplicas`, `ConfigurationConflict`, `AdmissionRequestDenied`, `UnsupportedAddonModification`, `K8sResourceNotFound`, `AddonSubscriptionNeeded`, `AddonPermissionFailure`; plus `message` and `resourceIds`. | API_AddonIssue |
|
||||
| `serviceAccountRoleArn` | "The ARN of the IAM role that's bound to the Kubernetes `ServiceAccount` object that the add-on uses"; on create, "If you don't specify an existing IAM role, then the add-on uses the permissions assigned to the node IAM role", and an IAM OIDC provider must exist for the cluster. | API_Addon, API_CreateAddon |
|
||||
| `configurationValues` | "validated against the schema returned by `DescribeAddonConfiguration`". | API_CreateAddon |
|
||||
| `resolveConflicts` | `NONE`/`PRESERVE` leave a self-managed install's values and may fail creation; `OVERWRITE` replaces them; with no prior install all three behave alike. | API_CreateAddon |
|
||||
| `DeleteAddon` | "When you remove an add-on, it's deleted from the cluster"; `preserve` "preserves the add-on software on your cluster but Amazon EKS stops managing any settings for the add-on. If an IAM account is associated with the add-on, it isn't removed." The example response is `DELETING`. | API_DeleteAddon |
|
||||
|
||||
The issue codes imply AWS distinguishes replica shortfall (`InsufficientNumberOfReplicas`), missing objects (`K8sResourceNotFound`), field conflicts (`ConfigurationConflict`) and admission denials; how EKS computes them, and whether `DEGRADED` is driven by them, is not stated in the API reference and was not verified.
|
||||
Whether AWS offers each of these five names as an EKS add-on was not checked against a live `DescribeAddonVersions`.
|
||||
|
||||
## 3. Shared mechanics (apply to every bundle)
|
||||
|
||||
### 3.1 Render
|
||||
|
||||
| Step | Behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Input | One TSV line per staged add-on: `name`, `version`, `roleArn`, `base64(configurationValues)`. | `spinifex/agents/eks/gatewayfetch/fetch.go` `emitAddonsTSV`, `sync.sh:357` |
|
||||
| Source | Every `*.yaml` in `/usr/share/spinifex-eks/addons/<name>/<version>/`, in glob order, concatenated with `---`. A missing directory reports `failed`. | `sync.sh:41`, `sync.sh:82-89`, `sync.sh:219-242` |
|
||||
| Output | One file per add-on name, `/var/lib/rancher/k3s/server/manifests/spinifex-addon-<name>.yaml`; the name carries no version. Replaced only when content changes. | `sync.sh:42-43`, `sync.sh:83`, `sync.sh:248-253` |
|
||||
| Block markers | `{{IRSA_ENV}}`, `{{IRSA_VOLUME}}`, `{{IRSA_VOLUME_MOUNT}}`, `{{ELB_INGRESS_PARAMS_SPEC}}` replaced by awk when alone on a line. | `sync.sh:224-230` |
|
||||
| Scalar placeholders | `{{SERVICE_ACCOUNT_ROLE_ARN}}`, `{{CLUSTER_NAME}}`, `{{AWS_REGION}}`, `{{AWS_VPC_ID}}`, `{{GATEWAY_ENDPOINT}}`, `{{GATEWAY_CA_PEM_B64}}`, `{{WEBHOOK_CA_B64}}`, `{{WEBHOOK_TLS_CRT_B64}}`, `{{WEBHOOK_TLS_KEY_B64}}` via `sed s|..|..|g`. The role ARN is inserted raw: it is not escaped for sed (`|`, `&`, `\`) or for YAML. | `sync.sh:231-240` |
|
||||
| Webhook cert | Opt-in by `webhook.conf`; minted once per add-on under `/var/lib/spinifex-eks/webhook-certs/<name>`, 10-year validity, reused thereafter. | `sync.sh:98-120`, `spinifex/agents/eks/webhookcert/cert.go:109-110` |
|
||||
| `configurationValues` | Read from the TSV and discarded; no bundle has a hook. | `sync.sh:359` |
|
||||
| Executor | Primary server only; server-join nodes do not run the agent. | `scripts/images/eks-node/eks-node-role.sh:138-149` |
|
||||
|
||||
### 3.2 Identity (IRSA) wiring
|
||||
|
||||
| Element | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Role carrier | The role ARN lives in the add-on record and the staged manifest (`serviceAccountRoleArn`), and after render in the rendered file only. No pod identity association, no EKS-side binding object. | `spinifex/domains/eks/addon/record.go:24-45`, `sync.sh:144-160` |
|
||||
| Credential env | With a role: `AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE`, `AWS_STS_REGIONAL_ENDPOINTS`, region, `AWS_ENDPOINT_URL_STS` (gateway), `AWS_CA_BUNDLE` and `SSL_CERT_FILE`. Without one: region and CA only, so the SDK falls through to node instance-profile credentials over IMDS. | `sync.sh:144-195` |
|
||||
| Token | A projected ServiceAccount token, audience `sts.amazonaws.com`, 86400 s, mounted at `/var/run/secrets/eks.amazonaws.com/serviceaccount`, injected statically by the render. There is no pod-identity mutating webhook in the image, so the `eks.amazonaws.com/role-arn` ServiceAccount annotation is descriptive only; the env and volume are what take effect. | `sync.sh:161-178`; no `pod-identity-webhook` in the tree |
|
||||
| Issuer | The apiserver's `/openid/v1/jwks` is published at first boot; STS `AssumeRoleWithWebIdentity` resolves `iss` to a registered IAM OIDC provider in the role's account, fetches the cluster JWKS, and evaluates the role trust policy (`{iss}:sub`, `{iss}:aud`). | `scripts/images/eks-node/k3s-first-boot.sh:141-162`, `spinifex/handlers/sts/assume_role_with_web_identity.go:33-213`, `spinifex/handlers/sts/web_identity_trust_policy.go:137-170` |
|
||||
| Objects outside the bundle | The IAM role, its trust policy (naming `system:serviceaccount:<ns>:<sa>`) and the IAM OIDC provider are user-owned IAM objects; nothing in the add-on path creates, modifies or deletes them. | `tests/e2e/eks/eks_test.go:658-700` creates them in the test |
|
||||
| Authorisation | `iam:PassRole` on the role at `CreateAddon` and `UpdateAddon`. | `spinifex/gateway/eks/passrole.go:31-37` |
|
||||
| `eks-token-webhook` | Not part of IRSA: it is the apiserver TokenReview authenticator for `aws eks get-token` bearer tokens. | `scripts/images/eks-node/eks-token-webhook.initd:1-8` |
|
||||
|
||||
Because identity association lives only in IAM (user-owned) and in the rendered pod spec, "identity associations observably removed" has no Spinifex-side object to observe for any bundle today: the role is not detached from anything when the add-on is deleted, matching AWS only in that AWS also does not delete the IAM role.
|
||||
|
||||
### 3.3 Readiness
|
||||
|
||||
| Path | Condition | Evidence |
|
||||
|---|---|---|
|
||||
| Primary | The K3s `Addon` CR in `kube-system` whose `.spec.source` is the rendered file carries a non-empty `addon.k3s.cattle.io/gvks` annotation. Reported `ready` on every tick this holds. No pod, replica or CRD condition is consulted. | `sync.sh:266-273`, `sync.sh:361-365` |
|
||||
| Fallback | Only after `ADDON_READY_GRACE` (90 s) with the annotation still empty: `kubectl get -f <rendered> --ignore-not-found`, then require at least one Deployment, StatefulSet or DaemonSet and that every one returned is rolled out (Deployment `availableReplicas >= spec.replicas` and generation observed; StatefulSet `readyReplicas >= spec.replicas`; DaemonSet `desiredNumberScheduled > 0` and `numberReady >= desiredNumberScheduled`). | `sync.sh:56`, `sync.sh:301-320`, `sync.sh:367-372` |
|
||||
| Otherwise | `applied`, which changes no status. | `sync.sh:374-376`, `spinifex/domains/eks/addon/owner.go:186-205` |
|
||||
| `failed` | Emitted only for a missing bundle directory or webhook cert failure; never for apply errors, crash-looping pods or lost replicas. | `sync.sh:87`, `sync.sh:112` |
|
||||
| Status mapping | `ready` lifts any non-`ACTIVE` status (including `CREATE_FAILED`, `DEGRADED`, `UPDATING`) to `ACTIVE` and clears `health`; `failed` maps `ACTIVE` to `DEGRADED`. Reports carry no generation. | `owner.go:140-168`, `owner.go:186-205` |
|
||||
|
||||
Consequences common to all bundles:
|
||||
|
||||
1. On the primary path `ready` means "K3s recorded the GVKs it applied", not "workloads running"; a bundle whose pods never start (image pull failure, unschedulable, crash loop) is `ACTIVE` and stays so.
|
||||
2. `ACTIVE` never degrades on workload loss: the annotation persists, so `ready` is re-sent every tick and no `failed` is ever produced for runtime health.
|
||||
3. `--ignore-not-found` drops absent objects from the fallback set, so a workload that failed to apply is simply not checked; one rolled-out workload is enough.
|
||||
4. After `UpdateAddon` the same tick that rewrites the file reads the annotation from the previous apply; whether K3s clears it before that read is not established, so `UPDATING` may settle `ACTIVE` on the old apply (the agent comment at `sync.sh:244-247` says a re-apply clears it, which suggests a window exists).
|
||||
5. The agent never deletes the K3s `Addon` CR; if K3s keeps that CR after the file is removed, a later re-create would match it by `.spec.source` and could report `ready` from a stale annotation before the new apply (K3s behaviour not verified).
|
||||
|
||||
### 3.4 Removal
|
||||
|
||||
| Step | Behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Trigger | A rendered `spinifex-addon-<name>.yaml` whose name is absent from a successful fetch. A failed fetch skips GC. | `sync.sh:344-348`, `sync.sh:386-391` |
|
||||
| Selection | The object list is the rendered file itself (`kubectl delete -f`), not labels or a GVK inventory. Objects created at runtime by the add-on, or by users through it, are never selected. | `sync.sh:397-401` |
|
||||
| Order | File order, which is glob order of the bundle files then document order. | `sync.sh:219` |
|
||||
| Waiting | `--wait=false`; errors go to syslog; the copy is deleted; nothing is reported. | `sync.sh:400-402` |
|
||||
| Leftovers on the node | `/var/lib/spinifex-eks/webhook-certs/<name>` and the K3s `Addon` CR. The `/tmp` sentinels are cleared. | `sync.sh:47`, `sync.sh:393` |
|
||||
| Control plane | Record and manifest keys are erased before any guest action. | `owner.go:122-137` |
|
||||
| Namespace completion | Namespace deletion needs every aggregated API available; the e2e asserts `v1beta1.metrics.k8s.io` is `Available` first because otherwise the namespace stays `Terminating`. | `tests/e2e/eks/eks_test.go:197-231` |
|
||||
|
||||
`preserve` is ignored (record `eks-addon-lifecycle.md` row Desired state); for every bundle a preserve would need to keep the whole rendered object set and stop only the render and GC, so the per-bundle `preserve` rows below list what in addition would break or dangle.
|
||||
|
||||
## 4. `spinifex-noop` 0.1.0
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| Namespace | `spinifex-noop` | | `spinifex-noop/0.1.0/noop.yaml:5-10` |
|
||||
| ConfigMap | `spinifex-noop` | `spinifex-noop` | `noop.yaml:12-20` |
|
||||
|
||||
Images: none.
|
||||
Both objects carry `app.kubernetes.io/managed-by: spinifex-addon` (`noop.yaml:9-10`, `noop.yaml:17-18`); nothing reads the label.
|
||||
|
||||
### Identity
|
||||
|
||||
No IRSA markers, no ServiceAccount; a supplied `serviceAccountRoleArn` is stored and inert.
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | Ready once K3s records the two GVKs. This is the path the e2e exercises (`tests/e2e/eks/eks_test.go:158-195`). |
|
||||
| Fallback | Never succeeds: no workload kinds, so `length > 0` is false (`sync.sh:308`). |
|
||||
| Meaning of ready | Both objects applied; there is nothing to serve. |
|
||||
| False ready | None material. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | Namespace (cascading the ConfigMap) and ConfigMap. |
|
||||
| Left behind | The K3s `Addon` CR. |
|
||||
| Preserve | Keep both; nothing dangles. |
|
||||
| Completion signal | `kubectl get ns spinifex-noop` NotFound; this is exactly what the e2e asserts (`eks_test.go:221-231`). Blocked indefinitely by an unavailable aggregated API. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
Namespace name could collide with a user namespace of the same name, which GC would then delete.
|
||||
No configuration hook.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. The only bundle with removal proven end-to-end; its completion signal (namespace NotFound) does not generalise to bundles in `kube-system`.
|
||||
|
||||
## 5. `argocd` 3.0.23
|
||||
|
||||
### Objects
|
||||
|
||||
Single file `argocd/3.0.23/argocd.yaml` (26,819 lines), upstream `install.yaml` passed through kustomize with `namespace: argocd` (`argocd.yaml:1-11`).
|
||||
|
||||
| Kind | Count | Names | Namespace | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| Namespace | 1 | `argocd` (first document) | | `argocd.yaml:12-17` |
|
||||
| CustomResourceDefinition | 3 | `applications.argoproj.io`, `applicationsets.argoproj.io`, `appprojects.argoproj.io` | | `argocd.yaml:25`, `:5904`, `:23563` |
|
||||
| ServiceAccount | 7 | `argocd-application-controller`, `-applicationset-controller`, `-dex-server`, `-notifications-controller`, `-redis`, `-repo-server`, `-server` | `argocd` | parsed |
|
||||
| Role / RoleBinding | 6 / 6 | per component | `argocd` | parsed |
|
||||
| ClusterRole / ClusterRoleBinding | 3 / 3 | `argocd-application-controller`, `argocd-applicationset-controller`, `argocd-server` | | `argocd.yaml:24243-24554` |
|
||||
| ConfigMap | 7 | `argocd-cm`, `-cmd-params-cm`, `-gpg-keys-cm`, `-notifications-cm`, `-rbac-cm`, `-ssh-known-hosts-cm`, `-tls-certs-cm` | `argocd` | parsed |
|
||||
| Secret | 2 | `argocd-notifications-secret`, `argocd-secret` | `argocd` | `argocd.yaml:24740`, `:24751` |
|
||||
| Service | 8 | applicationset-controller, dex-server, metrics, notifications-controller-metrics, redis, repo-server, server, server-metrics | `argocd` | parsed |
|
||||
| Deployment | 6 | `argocd-applicationset-controller`, `-dex-server`, `-notifications-controller`, `-redis`, `-repo-server`, `-server` (default 1 replica) | `argocd` | `argocd.yaml:24925-25887` |
|
||||
| StatefulSet | 1 | `argocd-application-controller`, 1 replica | `argocd` | `argocd.yaml:26301-26310` |
|
||||
| NetworkPolicy | 7 | one per component | `argocd` | parsed |
|
||||
|
||||
Images: `quay.io/argoproj/argocd:v3.0.23` (8 containers), `ghcr.io/dexidp/dex:v2.41.1` (`argocd.yaml:25243`), `redis:7.2.11-alpine` (`argocd.yaml:25478`).
|
||||
Only the Namespace carries `app.kubernetes.io/managed-by: spinifex-addon`.
|
||||
|
||||
### Identity
|
||||
|
||||
No IRSA markers (`argocd.yaml:11`); a supplied role is stored and inert.
|
||||
All seven ServiceAccounts are created by the bundle.
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | Per the agent comment, K3s never records the `gvks` annotation for this bundle, so the primary path does not fire (`sync.sh:49-55`); why is not established. |
|
||||
| Fallback | After 90 s: all 6 Deployments available at 1 replica and the StatefulSet at 1 ready replica, among those `kubectl get -f` finds. |
|
||||
| Meaning of ready | Every returned workload rolled out once. CRDs are not checked for `Established`; an absent CRD or workload is dropped by `--ignore-not-found` rather than failing the check. |
|
||||
| False ready | A partial apply that left any document unapplied still reads ready if the remaining workloads roll out; if the missing `gvks` is caused by an apply error on one object, this is the case in practice (hypothesis, not verified). Dex and notifications are counted though unused by default. |
|
||||
| Runtime | Images pulled from three public registries; on an air-gapped site the bundle never rolls out and stays `CREATING`. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | Everything in the file, Namespace first. CRD deletion removes every `Application`, `ApplicationSet` and `AppProject` in the cluster, including the runtime `default` AppProject. |
|
||||
| Left behind | Workloads Argo CD deployed into other namespaces (user applications) unless their `Application` carries the cascade finalizer and the controller survives long enough to process it; runtime Secrets in `argocd` go with the namespace. |
|
||||
| Stuck risk | `Application` objects carrying `resources-finalizer.argocd.argoproj.io` (user-set, upstream behaviour) need the application controller to clear them; it is deleted in the same pass, so the CRs, the `applications` CRD and the `argocd` namespace can stay `Terminating` indefinitely (not verified live). |
|
||||
| Preserve | Keep all objects; the cluster-scoped ClusterRoles and CRDs then remain unowned. |
|
||||
| Completion signal | `argocd` namespace NotFound and the three CRDs NotFound and the three ClusterRoles and ClusterRoleBindings NotFound; namespace and CRD finalizers must be watched explicitly, since NotFound on the namespace alone does not cover cluster-scoped objects. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
A user-installed Argo CD (same CRDs, same ClusterRole names, namespace `argocd`) would be overwritten on create and wholly deleted on DeleteAddon; this is the `resolveConflicts` case.
|
||||
No configuration hook, although `argocd-cm`/`argocd-rbac-cm` are the natural targets.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. Readiness depends entirely on the fallback, which neither checks CRDs nor detects missing objects.
|
||||
2. Removal deletes user-owned `Application` data cluster-wide and can wedge on finalizers; nothing reports it.
|
||||
3. No e2e coverage at all.
|
||||
|
||||
## 6. `aws-ebs-csi-driver` 1.40.1
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| ServiceAccount | `ebs-csi-controller-sa` (role annotation) | `kube-system` | `aws-ebs-csi-driver/1.40.1/00-rbac.yaml:5-14` |
|
||||
| ServiceAccount | `ebs-csi-node-sa` | `kube-system` | `00-rbac.yaml:16-23` |
|
||||
| ClusterRole | `ebs-external-attacher-role`, `ebs-csi-node-role`, `ebs-external-provisioner-role`, `ebs-external-resizer-role`, `ebs-external-snapshotter-role` | | `00-rbac.yaml:25`, `:65`, `:94`, `:184`, `:241` |
|
||||
| ClusterRoleBinding | `ebs-csi-attacher-binding`, `ebs-csi-node-getter-binding`, `ebs-csi-provisioner-binding`, `ebs-csi-resizer-binding`, `ebs-csi-snapshotter-binding` | | `00-rbac.yaml:309-381` |
|
||||
| Role / RoleBinding | `ebs-csi-leases-role` / `ebs-csi-leases-rolebinding` | `kube-system` | `00-rbac.yaml:384-417` |
|
||||
| CSIDriver | `ebs.csi.aws.com` | | `00-rbac.yaml:421-430` |
|
||||
| PodDisruptionBudget | `ebs-csi-controller` (maxUnavailable 1) | `kube-system` | `00-rbac.yaml:432-443` |
|
||||
| Secret | `spinifex-gateway-ca` (shared name, see Conflicts) | `kube-system` | `10-controller.yaml:8-17` |
|
||||
| Deployment | `ebs-csi-controller`, 2 replicas, preferred anti-affinity, tolerates `CriticalAddonsOnly` and `nvidia.com/gpu` | `kube-system` | `10-controller.yaml:19-82` |
|
||||
| DaemonSet | `ebs-csi-node`, tolerates `nvidia.com/gpu` only | `kube-system` | `20-node.yaml:5-58` |
|
||||
| StorageClass | `ebs-gp3`, default class, `WaitForFirstConsumer` | | `30-storageclass.yaml:5-18` |
|
||||
|
||||
Images: `public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.40.1` (`10-controller.yaml:89`, `20-node.yaml:67`), `public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v5.2.0-eks-1-32-7` (`:147`), `external-attacher:v4.8.1-eks-1-32-7` (`:180`), `external-snapshotter/csi-snapshotter:v8.2.1-eks-1-32-7` (`:209`), `external-resizer:v1.13.2-eks-1-32-7` (`:238`), `livenessprobe:v2.14.0-eks-1-32-7` (`:270`, `20-node.yaml:155`), `node-driver-registrar:v2.13.0-eks-1-32-7` (`20-node.yaml:118`).
|
||||
The bundle header says it is vendored from upstream v1.41.0 while the catalogue version is 1.40.1 (`00-rbac.yaml:1-3`); the driver image is v1.40.1.
|
||||
|
||||
### Identity
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Takes `serviceAccountRoleArn` | Yes (`RequiresIRSA`), not required at create. |
|
||||
| Annotated SA | `ebs-csi-controller-sa` (`00-rbac.yaml:12-13`, quoted, so an empty role renders `''`). |
|
||||
| Injected pod | Controller Deployment only: `{{IRSA_ENV}}` (`10-controller.yaml:110`), `{{IRSA_VOLUME_MOUNT}}` (`:114`), `{{IRSA_VOLUME}}` (`:289`). The node DaemonSet gets no credentials and makes no EC2 calls (`20-node.yaml:1-3`). |
|
||||
| SA owner | The bundle creates both ServiceAccounts. |
|
||||
| Without a role | Controller uses node instance-profile credentials over IMDS. |
|
||||
| Evidence | `tests/e2e/eks/eks_test.go:658-800` installs it with a role, creates the OIDC provider and role, and drives a PVC through CreateVolume/AttachVolume. |
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | `ready` as soon as K3s records GVKs; neither replica need be running. The e2e waits for `ACTIVE` and then separately proves the data path (`eks_test.go:767-800`). |
|
||||
| Fallback | Controller Deployment 2 of 2 available and node DaemonSet `desired > 0` and all ready. |
|
||||
| Fallback false negative | A cluster with no worker node that tolerates scheduling (only the control plane, which carries `CriticalAddonsOnly` the DaemonSet does not tolerate) has `desired = 0`, so the fallback never passes. |
|
||||
| False ready | Controller pending or crash-looping (including STS web-identity failures from a bad trust policy), or sidecars failing, still `ACTIVE` via the primary path. The bundle ships no `VolumeSnapshot` CRDs while running `csi-snapshotter`; the sidecar's behaviour without them was not verified and nothing in readiness would show it. |
|
||||
| AWS comparison | `InsufficientNumberOfReplicas` and `AccessDenied` are the codes that would correspond to these states; neither is ever produced. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | All objects above, including the default StorageClass and the CSIDriver, in file order (RBAC and CSIDriver first, then Secret and controller, then node DaemonSet, then StorageClass). |
|
||||
| Left behind (in cluster) | PersistentVolumes and PVCs it provisioned; `VolumeAttachment` objects; runtime leader-election Leases in `kube-system`; `CSINode` driver entries until node plugins unregister. |
|
||||
| Left behind (outside cluster) | Spinifex EBS volumes backing those PVs, still allocated and possibly attached. |
|
||||
| Stuck risk | PVs with `reclaimPolicy: Delete` and VolumeAttachments carry CSI sidecar finalizers that only the deleted controller clears, so later PVC deletion or node drain hangs and the backing volume is never deleted (upstream sidecar behaviour, not verified on this build). |
|
||||
| Preserve | Keep the whole set; existing mounts and new provisioning keep working with the last-rendered role. |
|
||||
| Completion signal | All 20 rendered objects NotFound (all in `kube-system` or cluster-scoped, so there is no namespace to watch); a meaningful "removed" additionally needs zero `VolumeAttachment`s and PVs with `spec.csi.driver == ebs.csi.aws.com`, or an explicit statement that those are user data that outlive the add-on. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
| Object | Collides with |
|
||||
|---|---|
|
||||
| `kube-system/spinifex-gateway-ca` | The identical name in `aws-load-balancer-controller` (`10-aws-load-balancer-controller.yaml:33-41`). Deleting either add-on deletes the Secret the other mounts; K3s re-applies the survivor's file only when it changes, so the survivor's next pod start fails until then (re-apply behaviour not verified). A removal check by NotFound on this Secret also never completes while the other add-on re-creates it. |
|
||||
| `StorageClass/ebs-gp3` with `is-default-class` | Any user default StorageClass, giving two defaults; K3s `local-storage` is disabled (`scripts/images/eks-node/setup.sh:98-101`). |
|
||||
| `CSIDriver/ebs.csi.aws.com`, the five ClusterRoles, `ebs-csi-*` names | A self-managed (Helm) EBS CSI install, the `resolveConflicts` case; DeleteAddon would remove the user's driver too. |
|
||||
|
||||
No configuration hook; upstream exposes replica count, tolerations and StorageClass parameters through Helm values.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. `ACTIVE` does not imply a running controller or working credentials.
|
||||
2. Removal orphans PVs, VolumeAttachments and Spinifex volumes, and can wedge them behind finalizers.
|
||||
3. Shares `spinifex-gateway-ca` with the load balancer controller, so removal of either is not isolated.
|
||||
4. Bundle provenance comment (v1.41.0) disagrees with the catalogue version (1.40.1).
|
||||
|
||||
## 7. `aws-load-balancer-controller` 2.11.0
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| CustomResourceDefinition | `ingressclassparams.elbv2.k8s.aws` | | `aws-load-balancer-controller/2.11.0/00-crds.yaml:1-8` |
|
||||
| CustomResourceDefinition | `targetgroupbindings.elbv2.k8s.aws` | | `00-crds.yaml:256-263` |
|
||||
| ServiceAccount | `aws-load-balancer-controller` (role annotation, unquoted) | `kube-system` | `10-aws-load-balancer-controller.yaml:2-13` |
|
||||
| Secret | `aws-load-balancer-webhook-tls` (node-minted cert) | `kube-system` | `10-…yaml:17-28` |
|
||||
| Secret | `spinifex-gateway-ca` (shared name) | `kube-system` | `10-…yaml:32-41` |
|
||||
| Role / RoleBinding | `aws-load-balancer-controller-leader-election-role` / `-rolebinding` | `kube-system` | `10-…yaml:44-48`, `:242-254` |
|
||||
| ClusterRole / ClusterRoleBinding | `aws-load-balancer-controller-role` / `-rolebinding` | | `10-…yaml:85-89`, `:258-269` |
|
||||
| Service | `aws-load-balancer-webhook-service` | `kube-system` | `10-…yaml:273-277` |
|
||||
| Deployment | `aws-load-balancer-controller`, 1 replica, tolerates `CriticalAddonsOnly` and `nvidia.com/gpu` | `kube-system` | `10-…yaml:288-372` |
|
||||
| List → IngressClassParams | `alb` (subnets injected from `EKS_ELB_SUBNET_IDS`) | | `10-…yaml:375-386`, `sync.sh:197-215` |
|
||||
| List → IngressClass | `alb` | | `10-…yaml:388-400` |
|
||||
| MutatingWebhookConfiguration | `aws-load-balancer-webhook`: `mservice` (Services, all namespaces), `mpod` (Pods in namespaces labelled `elbv2.k8s.aws/pod-readiness-gate-inject`), `mtargetgroupbinding`; all `failurePolicy: Fail` | | `10-…yaml:402-488` |
|
||||
| ValidatingWebhookConfiguration | `aws-load-balancer-webhook`: `vingressclassparams`, `vtargetgroupbinding`, `vingress`; all `failurePolicy: Fail` | | `10-…yaml:489-564` |
|
||||
|
||||
Image: `public.ecr.aws/eks/aws-load-balancer-controller:v2.11.0` (`10-…yaml:320`).
|
||||
Webhook SANs come from `webhook.conf` (`aws-load-balancer-controller/2.11.0/webhook.conf`).
|
||||
|
||||
### Identity
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Takes `serviceAccountRoleArn` | Yes (`RequiresIRSA`), not required at create. |
|
||||
| Annotated SA | `aws-load-balancer-controller` (`10-…yaml:11-12`); the placeholder is unquoted, so the raw ARN becomes a YAML scalar and an empty role renders a null annotation value. |
|
||||
| Injected pod | Controller Deployment: `{{IRSA_ENV}}` (`:319`), `{{IRSA_VOLUME_MOUNT}}` (`:349`), `{{IRSA_VOLUME}}` (`:372`); STS, EC2, ELBv2 and ACM endpoints pinned to the gateway by args (`:309-314`). |
|
||||
| SA owner | The bundle. |
|
||||
| Without a role | Node instance-profile credentials over IMDS. |
|
||||
| Evidence | No e2e installs this add-on; IRSA is proven only for EBS CSI and a hand-written pod (`tests/e2e/eks/eks_test.go:234-247`). |
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | `ready` when GVKs are recorded; the controller need not be running. |
|
||||
| Fallback | The one Deployment at 1 available replica. |
|
||||
| False ready | Controller pending, crash-looping, or failing STS/ELBv2 calls is still `ACTIVE`. Worse than a passive false ready: with the controller absent, the six `failurePolicy: Fail` webhooks reject every Service create or update cluster-wide (`mservice` matches all namespaces except LBC's own objects) and every Ingress, so an `ACTIVE` add-on can break unrelated workloads. CRD `Established` is not checked. |
|
||||
| AWS comparison | `AdmissionRequestDenied` and `InsufficientNumberOfReplicas` are the matching issue codes; neither is produced. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | CRDs first (00 file sorts first), then ServiceAccount, Secrets, RBAC, Service, Deployment, IngressClassParams, IngressClass, and the webhook configurations last. |
|
||||
| Left behind (in cluster) | User Ingresses and `LoadBalancer` Services it reconciled, with the controller's finalizers still set; the runtime leader Lease. |
|
||||
| Left behind (outside cluster) | Spinifex ELBv2 load balancers, target groups, listeners and any security groups the controller created for those Ingresses and Services. |
|
||||
| Stuck risk | Deleting the `targetgroupbindings` CRD waits on TargetGroupBindings whose finalizer only the controller clears; user Ingresses and Services keep finalizers no one removes, so they cannot be deleted (upstream controller behaviour, not verified on this build). Between the Deployment deletion and the webhook deletion, Service and Ingress writes fail; if the webhook delete errors, that state persists. |
|
||||
| Preserve | Keep the whole set; the webhook cert and gateway CA must stay valid (cert is 10-year). |
|
||||
| Completion signal | All rendered objects NotFound plus both CRDs gone (no `Terminating`); a meaningful "removed" also needs no remaining object carrying the controller's finalizers and no remaining Spinifex ELBv2 resources tagged for this cluster, which only the host side can observe. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
| Object | Collides with |
|
||||
|---|---|
|
||||
| `kube-system/spinifex-gateway-ca` | `aws-ebs-csi-driver` (Section 6). |
|
||||
| `IngressClass/alb`, `IngressClassParams/alb` | Any user IngressClass named `alb`, including a self-managed LBC. |
|
||||
| CRDs, ClusterRole, webhook configuration names | A self-managed LBC (Helm), the `resolveConflicts` case. |
|
||||
|
||||
No configuration hook; `EKS_ELB_SUBNET_IDS` is the only per-cluster input and it comes from the node environment, not from `configurationValues`.
|
||||
The design draft names this bundle as the first configuration target.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. `ACTIVE` can coexist with fail-closed webhooks that have no backend, which breaks Service and Ingress writes cluster-wide.
|
||||
2. Removal orphans user Ingresses/Services behind finalizers and Spinifex ELBv2 resources outside the cluster; nothing in the guest can observe the latter.
|
||||
3. Shares `spinifex-gateway-ca` with EBS CSI.
|
||||
4. Unquoted role placeholder takes the raw ARN as YAML.
|
||||
5. No e2e coverage.
|
||||
|
||||
## 8. `nvidia-device-plugin` 0.17.4
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| DaemonSet | `nvidia-device-plugin-daemonset`; nodeSelector `nvidia.com/gpu.present=true`; tolerates `nvidia.com/gpu`; `runtimeClassName: nvidia`; hostPath `/` read-only, `/var/lib/kubelet/device-plugins`, `/var/run/cdi` | `kube-system` | `nvidia-device-plugin/0.17.4/10-daemonset.yaml:14-75` |
|
||||
|
||||
Image: `nvcr.io/nvidia/k8s-device-plugin:v0.17.4` (`10-daemonset.yaml:41`).
|
||||
Depends on a `RuntimeClass` named `nvidia` and on the node label written by `mulga-eks-provider-id` (`10-daemonset.yaml:1-12`); neither is in the bundle.
|
||||
|
||||
### Identity
|
||||
|
||||
No IRSA markers and no ServiceAccount object: pods run as the `kube-system/default` ServiceAccount that Kubernetes creates.
|
||||
`EnsureGPUDevicePlugin` stages it with no role (`owner.go:79-89`); a user-created one may carry an inert role.
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | `ready` when K3s records the DaemonSet GVK, regardless of GPU nodes. |
|
||||
| Fallback | Requires `desiredNumberScheduled > 0`: never passes on a cluster with no GPU node. |
|
||||
| False ready | `ACTIVE` with zero GPU nodes (nothing serving); `ACTIVE` after every GPU node group is deleted; `FAIL_ON_INIT_ERROR=false` (`10-daemonset.yaml:46-47`) and no readiness probe mean a pod whose NVML init failed is Ready while advertising no `nvidia.com/gpu`. The e2e proves the real signal separately: DaemonSet `1=1` and node allocatable `nvidia.com/gpu=1` (`tests/e2e/gpu/eks_gpu_test.go:260-275`). |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | The DaemonSet. |
|
||||
| Left behind | CDI spec files the plugin wrote to host `/var/run/cdi`; node `nvidia.com/gpu` capacity until kubelet reconciles the unregistered plugin (kubelet behaviour not verified). |
|
||||
| Re-staging | Deleting it while GPU node groups exist leaves GPUs unschedulable until the next GPU node group create re-stages it; deleting the last GPU node group never unstages it (`eks-addon-lifecycle.md` row Cross-resource dependency). |
|
||||
| Preserve | Keep the DaemonSet; nothing dangles. |
|
||||
| Completion signal | DaemonSet NotFound; pods gone from GPU nodes. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
A user-installed NVIDIA device plugin or GPU Operator would run a second plugin on the same nodes and contend for the kubelet device-plugin socket; the DaemonSet name matches the upstream static manifest, so a self-managed upstream install is overwritten.
|
||||
No configuration hook (device-list strategy, time-slicing and MIG are upstream options).
|
||||
|
||||
### Gaps
|
||||
|
||||
1. Ready is indifferent to whether any GPU is exposed; the workload fallback is unreachable on non-GPU clusters.
|
||||
2. Ownership is split between the auto-stager and the user, with no record of which.
|
||||
|
||||
## 9. Cross-bundle findings
|
||||
|
||||
| # | Finding | Bundles | Evidence |
|
||||
|---|---|---|---|
|
||||
| X1 | Readiness is "applied" on the primary path; workload state is never consulted while `gvks` is present, and runtime failure never yields `failed`. | all | `sync.sh:361-365`, `sync.sh:87`, `sync.sh:112` |
|
||||
| X2 | The fallback silently drops absent objects (`--ignore-not-found`) and never checks CRD `Established`. | argocd, LBC, EBS | `sync.sh:304-319` |
|
||||
| X3 | DaemonSet `desired > 0` makes the fallback unreachable for node-dependent bundles on clusters without matching nodes, while the primary path reports ready for the same state. | EBS, nvidia | `sync.sh:317` |
|
||||
| X4 | Removal selects only objects in the rendered file; runtime-created and user-created dependants (PVs, VolumeAttachments, Applications, Ingresses with finalizers, Leases) are not inventoried, and out-of-cluster resources (EBS volumes, ELBv2) are invisible to the guest. | EBS, LBC, argocd | `sync.sh:397-401` |
|
||||
| X5 | `kube-system/spinifex-gateway-ca` is rendered by two bundles under one name; either removal deletes the other's dependency, and a NotFound completion check on it is ill-defined while both exist. | LBC, EBS | `10-aws-load-balancer-controller.yaml:32-41`, `aws-ebs-csi-driver/1.40.1/10-controller.yaml:8-17` |
|
||||
| X6 | No bundle has a stable ownership label set: only noop and the argocd Namespace carry `managed-by: spinifex-addon`; LBC, EBS and nvidia carry only upstream `app.kubernetes.io/name` labels, which a self-managed install of the same software carries too. A label-based inventory or conflict check has nothing reliable to select on. | all | per-bundle sources |
|
||||
| X7 | Identity association is only the rendered pod env plus user-owned IAM objects; there is nothing Spinifex-owned to remove or observe, and the SA annotation is not acted on by any webhook. | LBC, EBS | Section 3.2 |
|
||||
| X8 | `RequiresIRSA` is advisory: no create-time check either way, so a role on argocd, nvidia or noop is stored and inert. | all | `addons.go:61-99`, `addons.go:255` |
|
||||
| X9 | The role ARN is substituted raw into sed and YAML; only `iam:PassRole` (which requires the role to exist) constrains it. | LBC, EBS | `sync.sh:231`, `passrole.go:31-37` |
|
||||
| X10 | `configurationValues` has no schema, no `DescribeAddonConfiguration`, and no render hook for any bundle. | all | `sync.sh:359` |
|
||||
| X11 | Removal runs only on the primary server and only from the local rendered file; a replaced primary has no file and removes nothing. | all | `eks-node-role.sh:138-142`, `sync.sh:386` |
|
||||
| X12 | E2E covers create, ready and delete only for noop, create and data path for EBS (delete only in cleanup, not asserted), and DaemonSet rollout for nvidia; argocd and LBC have none. | all | `tests/e2e/eks/eks_test.go:158-247`, `tests/e2e/gpu/eks_gpu_test.go:260-275` |
|
||||
|
||||
## 10. Per-bundle observable removal signal (summary)
|
||||
|
||||
| Bundle | Objects that must be NotFound | Finalizer or dependant to watch | Out-of-cluster residue |
|
||||
|---|---|---|---|
|
||||
| `spinifex-noop` | Namespace `spinifex-noop` | Namespace `kubernetes` finalizer (aggregated API availability) | none |
|
||||
| `argocd` | Namespace `argocd`, 3 CRDs, 3 ClusterRoles, 3 ClusterRoleBindings | `Application` resources finalizer; CRD and namespace `Terminating` | none from the bundle; user workloads it deployed |
|
||||
| `aws-ebs-csi-driver` | 20 objects, all `kube-system` or cluster-scoped | PV and VolumeAttachment CSI finalizers | Spinifex EBS volumes |
|
||||
| `aws-load-balancer-controller` | 2 CRDs plus 13 objects, webhooks included | Ingress, Service and TargetGroupBinding controller finalizers; CRD `Terminating` | Spinifex ELBv2 load balancers, target groups, security groups |
|
||||
| `nvidia-device-plugin` | 1 DaemonSet | none | host CDI spec files |
|
||||
@@ -1,141 +0,0 @@
|
||||
# EKS managed add-on: delivery executor and fencing inventory
|
||||
|
||||
Code read at 3a55e2ee1; not verified live.
|
||||
|
||||
This inventory supports the "Delivery executor" section (Section 10) and review decision R6 of `docs/package-boundary/eks-addon-lifecycle-design.md`, and the draft proposal PROP-SPINIFEXARCH-007 (`docs/development/proposals/spinifex-architecture/0007-contract-guest-and-persisted-state-evolution.md` in the mulga repo), which is not accepted.
|
||||
It records what the code does today and which evidence exists for choosing and fencing a guest-side delivery executor; it proposes no design.
|
||||
Paths are relative to `spinifex/` unless they start with `scripts/`, `images/`, `contracts/` or `tests/`, which are relative to the spinifex repo root.
|
||||
|
||||
## 1. Which guest process delivers add-ons
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Delivery process | `mulga-eks-addon-sync` (shell loop): fetch staged set, render each bundle into `/var/lib/rancher/k3s/server/manifests/spinifex-addon-<name>.yaml`, report a phase per add-on, GC rendered files whose add-on is no longer staged with `kubectl delete --wait=false`. | `scripts/images/eks-node/mulga-eks-addon-sync.sh:341-405` (sync), `:80-254` (render), `:383-404` (GC) |
|
||||
| Loop and cadence | Every `ADDON_SYNC_INTERVAL` (30 s) forever; one-shot when unset. | `mulga-eks-addon-sync.sh:409-417`, `mulga-eks-addon-sync.initd:18`, `images/mkosi.profiles/eks-server/mkosi.extra/etc/systemd/system/mulga-eks-addon-sync.service:13` |
|
||||
| Fetch failure | Logs and keeps the current render; no GC runs that tick, so already-rendered files stay in the auto-deploy dir. | `mulga-eks-addon-sync.sh:344-348` |
|
||||
| Report | `printf` JSON `{addon,version,phase,message,ts}` piped to `eks-gateway-publish -channel addon`; failure ignored. No executor identity, instance ID or epoch in the payload. | `mulga-eks-addon-sync.sh:68-73`, `contracts/eks/v1/addon_status.go:31-37` |
|
||||
| Local state | Per-add-on sentinels in `/tmp` (`.addon-firstapply-*`, `.addon-diag-*`) and webhook certs in `/var/lib/spinifex-eks/webhook-certs`; no record of which generation or executor rendered a file. | `mulga-eks-addon-sync.sh:47`, `:279-293`, `:326-328` |
|
||||
| Credentials | Sources `/etc/spinifex-eks/first-boot.env`; signs with `EKS_ACCESS_KEY`/`EKS_SECRET_KEY` when present, else the SDK chain reads IMDS instance-role credentials. | `mulga-eks-addon-sync.sh:19-33`, `handlers/eks/k3s_server_vm.go:549-557` |
|
||||
| OpenRC unit | `/etc/init.d/mulga-eks-addon-sync`, `after k3s`, refuses to start without `first-boot.env`. | `scripts/images/eks-node/mulga-eks-addon-sync.initd:1-30`, installed by `scripts/images/eks-node/manifest.conf:43-44` |
|
||||
| systemd unit | `mulga-eks-addon-sync.service`, `After=k3s.service`, `WantedBy=multi-user.target`; baked disabled by preset. | `images/mkosi.profiles/eks-server/mkosi.extra/etc/systemd/system/mulga-eks-addon-sync.service:1-17`, `images/mkosi.profiles/eks-server/mkosi.extra/usr/lib/systemd/system-preset/40-mulga-eks-role.preset:73` |
|
||||
| Bake-time enablement | OpenRC image enables only `crond eks-node-role mulga-ebs-byid mulga-eks-provider-id mulga-vpc-mtu`; systemd image enables only `eks-node-role` and the snapshot/otel units. | `scripts/images/eks-node/manifest.conf:83`, `40-mulga-eks-role.preset:59-75` |
|
||||
|
||||
## 2. How a VM decides it is the primary
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Role source | Host renders `SPINIFEX_K3S_ROLE=server` when `ServerURL` is empty and `server-join` otherwise; workers get `agent`. | `handlers/eks/k3s_server_vm.go:527-545`, `handlers/eks/nodegroup_userdata.go:64` |
|
||||
| Who gets `server` | Create: the first VM of the spread (`nodes[0]`) or the single CP; every other spread member gets `ServerURL` and so `server-join`. | `handlers/eks/k3s_ha_control_plane.go:290-323`, `:272-282`; test `k3s_ha_control_plane_test.go` `TestPlaceControlPlane_SpreadFirstInitsRestJoin` |
|
||||
| Selector | `eks-node-role.sh` runs once at first boot, writes `/etc/spinifex-eks/role`, enables and starts role services, then disables itself; a later boot exits early if the role file exists. | `scripts/images/eks-node/eks-node-role.sh:90-96`, `:118-122`, `:177` |
|
||||
| Primary-only service | Only the `server` branch enables and starts `mulga-eks-addon-sync` (and `k3s-first-boot`); `server-join` does not. | `eks-node-role.sh:124-150` vs `:151-166`; `scripts/images/eks-node/eks-node-role_test.sh:129`, `:162` |
|
||||
| Primacy is static | "Primary" is a first-boot, per-VM, irreversible choice; nothing in the guest re-evaluates it, and no kube Lease, k3s leader or etcd leader is consulted. | `eks-node-role.sh:90-96`; no lease/leader reference in `scripts/images/eks-node/*.sh` |
|
||||
| Can a non-primary run it | Not by any code path read: `server-join` never enables it, the preset and OpenRC runlevel bake it disabled. Only manual enablement inside the VM would start a second instance, and the gateway would admit it (Section 4). | `eks-node-role.sh:151-166`, `40-mulga-eks-role.preset:73`, `manifest.conf:83` |
|
||||
| K3s auto-deploy in HA | Each K3s server watches its own local `server/manifests` dir; whether the auto-deploy controller applies on every server or only on the elected K3s leader is unverified. If it runs only on the leader, a primary that is not the K3s leader would not apply. | not verified; `mulga-eks-addon-sync.sh:261-273` relies on the `Addon` CR the controller creates |
|
||||
|
||||
## 3. Control-plane membership on the host
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Record | `ClusterMeta` at `clusters/{name}/meta` in `eks-account-{account}` (history 1). | `handlers/eks/store.go:28-40`, `handlers/eks/cluster_state.go:64-201` |
|
||||
| Member list | `ControlPlaneNodes []ControlPlaneNode{NodeID, InstanceID, ENIID, ENIIP, MgmtIP}`; scalar `ControlPlaneInstanceID` etc. mirror `[0]`. No role, boot incarnation, executor or epoch field. | `cluster_state.go:109-125`, `:203-212` |
|
||||
| "Primary" in meta | Index `[0]` by comment and by the create path; the scalars mirror it. | `cluster_state.go:119-124`, `handlers/eks/service_impl.go:941-949` |
|
||||
| Membership predicate | `IsControlPlaneMember(instanceID)`: any entry in `ControlPlaneNodes` or the scalar. | `cluster_state.go:238-251` |
|
||||
| Template | `ControlPlaneTemplate` persisted with `ServerURL`, creds and per-node fields cleared, replayed for replacements. | `cluster_state.go:128-133`, `service_impl.go:951-963` |
|
||||
| Meta revision | Written by every `casUpdateMeta` caller, including health and node-count changes, so its KV revision is not a membership counter. | `cluster_state.go:276`, `:416-436`, `cluster_reconciler.go:620`, `:635` |
|
||||
|
||||
## 4. Gateway authentication of the internal routes
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| CP principal | Assumed-role session of role `spinifex-eks-server` in the system account; IMDS mints it with `RoleSessionName` = instance ID. | `handlers/eks/k3s_server_role.go:14`, `handlers/sts/assume_role.go:136-161`, `domains/ec2/guestmetadata/metadata.go:476-511` |
|
||||
| Role policy | `eks:PublishInternal`, `eks:ListInternalAddons`, `eks:WebhookTokenReview`, `eks:GetRecoveryDirective` on `Resource: "*"`; one role shared by every cluster's CP VMs. | `k3s_server_role.go:27`, `:42-43` |
|
||||
| Caller mapping | `eksCaller`: `SessionName` from `ctxIdentity`, role name from the underlying role ARN, not the session name. | `gateway/eks.go:307-320` |
|
||||
| `AuthorizeInternal` | Requires the CP principal class, then `meta.IsControlPlaneMember(caller.SessionName)` for the named cluster; `GetRecoveryDirective` additionally requires the path instance ID to equal the caller's. | `gateway/eks/internal_authz.go:44-96`; tests `gateway/eks_authz_test.go` `TestEKSRequest_InternalRoutesAdmitTheCPAgentPrincipal`, `gateway/eks/internal_authz_test.go` `TestAuthorizeInternal_RejectsClusterTheCallerDoesNotServe` |
|
||||
| Which routes it gates | All four internal routes. `ListInternalAddons` and `GetRecoveryDirective` are scoped `sourceInternalCluster`; `PublishInternal` and `WebhookTokenReview` are scoped `sourceInternalBodyCluster`, binding the URL cluster and the body account (read by the same decoder the handler uses) to the caller's membership before policy evaluation. Before that correction both were policy-only. | `gateway/eks/authz.go`, `gateway/eks/internal_authz.go`; tests `gateway/eks_authz_test.go` `TestEKSRequest_PublishInternal*`, `TestEKSRequest_WebhookTokenReview*` |
|
||||
| Member vs primary | The gateway can tell members apart (instance ID) and bind a fetch to a cluster, but cannot identify the delivery executor: meta has no role field, and `[0]` stops being the `server`-role VM after `[0]` is replaced (Section 6). | `internal_authz.go:74`, `cluster_state.go:463-490` |
|
||||
| Static-cred fallback | When the instance profile cannot be ensured, the VM gets the shared system access key; it has no instance identity, so every internal route, including `PublishInternal` and `WebhookTokenReview`, is denied for the VM's life; this fallback is unsupported. | `k3s_server_role.go:30-50`, `k3s_ha_control_plane.go:161-166`, `k3s_server_vm.go:549-557` |
|
||||
| Handler-side identity | `ListStagedAddonManifests` receives only the cluster name; no caller identity reaches the daemon. | `gateway/eks/internal_addons.go:20-33`, `handlers/eks/addons.go:117-128`, `daemon/daemon.go:1116` |
|
||||
|
||||
## 5. Status relay path
|
||||
|
||||
| Hop | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Agent | `eks-gateway-publish` wraps `{accountId, channel, kind, payload}` and POSTs `/clusters/{cluster}/internal-publish`, 30 attempts. | `agents/eks/gatewaypublish/publish.go:29`, `:57-114` |
|
||||
| Gateway authz | IAM policy against `arn:aws:eks:{region}:{bodyAccount}:cluster/{name}`; with the role's `Resource: "*"`, any CP-role session of any cluster can publish to any cluster's add-on subject. | `gateway/eks/authz.go:155-160`, `k3s_server_role.go:27` |
|
||||
| Relay | `PublishInternal` publishes the raw payload to core NATS `eks.addon.{account}.{cluster}.status`; only a trace header is added. No caller instance ID, session or epoch is attached. | `gateway/eks/internal_publish.go:48-91`, `contracts/eks/v1/addon_status.go:10-12` |
|
||||
| Subscriber | Lease holder's `ClusterReconciler.Run` subscribes plainly; decodes the report and calls `Owner.ApplyReport` with addon, phase and message only (version and ts dropped). | `handlers/eks/cluster_reconciler.go:447-461`, `handlers/eks/addon_status_reconciler.go:22-31` |
|
||||
| Apply | Revision CAS on the add-on record; no check of sender, membership, version or epoch. | `domains/eks/addon/owner.go:140-168`, `domains/eks/addon/store.go:198` |
|
||||
| State reports | `mulga-eks-state-report` runs on both server roles and uses the same unbound relay; the report carries no member identity and the reconciler keeps the newest. | `eks-node-role.sh:137`, `:160`, `handlers/eks/state_report.go:24-42`, `cluster_reconciler.go:535` |
|
||||
|
||||
## 6. Primary failure, replacement and rebuild today
|
||||
|
||||
| Event | What happens to add-on delivery | Evidence |
|
||||
|---|---|---|
|
||||
| Primary stopped or errored | Reconciler restarts it in place after `restartGrace` (same instance ID, same root volume); role file and enabled units persist, so `mulga-eks-addon-sync` resumes with its rendered files. | `cluster_reconciler.go:708-771`, `:927-937` |
|
||||
| Etcd reform (HA, all VMs running) | `members[0]` gets `cluster-reset`, others `wipe-rejoin`; members are stopped and restarted. Wipe removes only `server/db/etcd`, so rendered manifests survive. | `cluster_reconciler.go:780-880`, `scripts/images/eks-node/mulga-eks-k3s-recovery.sh:38`, `:183-186` |
|
||||
| HA member lost (terminated, gone, or describe error) | `ProvisionReplacementCP` launches with `ServerURL` set, so the replacement is `server-join` and never runs `mulga-eks-addon-sync`; `SwapControlPlaneMember` puts it at the dead member's index. | `cluster_reconciler.go:1010-1096`, `k3s_ha_control_plane.go:142-187`, `cluster_state.go:463-490` |
|
||||
| HA primary (`[0]`) lost | Same path: after the swap, `[0]` is a `server-join` VM and no member runs delivery. Staged manifests stay in KV, but nothing fetches, renders, GCs or reports until an operator intervenes; add-ons keep their last status. | as above; no other enable path for the service |
|
||||
| Describe error treated as lost | A member whose state query fails counts as lost and can be replaced after `replaceGrace` while still running; if it was the primary, it keeps rendering and reporting (Section 7). | `cluster_reconciler.go:982-1007`; test `cluster_reconciler_replace_test.go` `TestMaybeReplaceCP_DescribeErrorMemberTreatedAsLost` |
|
||||
| Single-CP total loss | Operator `RestoreSnapshot` launches a fresh `server` VM (`ServerURL` empty), so delivery resumes on the new VM from the KV-staged set; meta is committed before the old CP is terminated, and termination is retried 3 times. | `handlers/eks/restore_snapshot.go:217-353`, `k3s_ha_control_plane.go:206-246`, `restore_snapshot.go:28`, `:185-212` |
|
||||
| Cluster create timeout or bootstrap failure | Marked `FAILED` without purging; CP VMs keep running `mulga-eks-addon-sync`, `ListInternalAddons` still admits them (membership only), and the reconciler has exited, so reports are dropped. | `cluster_reconciler.go:1151-1167`, `service_impl.go:2138`, `cluster_reconciler.go:645-648` |
|
||||
| Async launch failure | `failClusterLaunch` purges infra (terminates CP VMs) and marks `FAILED`, keeping meta. | `service_impl.go:701-717` |
|
||||
| Reclaim of a `FAILED` name | CAS `FAILED`→`CREATING`, purge old infra, overwrite meta; add-on keys are not purged, so the new primary renders the inherited staged set. | `service_impl.go:1203-1261` |
|
||||
| Delete | `purgeClusterInfra` stops the reconciler and terminates the CP VMs. | `service_impl.go:1271-1272` |
|
||||
| Resume from durable state | A fresh `server` VM resumes because the staged set is durable KV and the listing has no per-executor cursor; a `server-join` replacement cannot resume because it never runs the agent. | `handlers/eks/addons.go:117-128`, `eks-node-role.sh:151-166` |
|
||||
|
||||
## 7. Existing leases, epochs and generations
|
||||
|
||||
| Mechanism | Scope | What it proves | Evidence |
|
||||
|---|---|---|---|
|
||||
| ClusterReconciler lease | Host daemon per cluster; key `{account}/{cluster}` in `spinifex-eks-leader`, value = daemon `HolderID`, CAS create, renew by revision CAS every 20 s, bucket TTL 60 s, `Lost()` ends `Run`. | One daemon reconciles at a time. The revision is private (no accessor) and is not passed to report apply or guest routes. Unrelated to the guest executor. | `cluster_reconciler.go:26`, `:391-418`, `:463-470`; `foundation/state/kvlease/lease.go:96-178`, `:201-225`; `store.go:38-40` |
|
||||
| Lease takeover | `SpawnRegisteredReconcilers` runs only at daemon start; a lease loser is not retried elsewhere in the code read. | Reports for a cluster whose lease holder died are dropped until some daemon restarts. | `daemon/daemon.go:2209`, `service_impl.go:385-428`, `handlers/eks/reconciler_registry.go:12-14`, `:166-171` |
|
||||
| Recovery directive epoch | Per member instance, `epoch = prev+1` on each set; guest applies once and records the applied epoch locally. | At-most-once boot-time directive. Not a fence: it is pulled only at boot and has no `disable delivery` action. | `handlers/eks/recovery_directive.go:30-43`, `:90-107`; `mulga-eks-k3s-recovery.sh:131-197` |
|
||||
| Per-cluster epoch or generation | None in `ClusterMeta`; add-on records have no generation. | Nothing. | `cluster_state.go:64-201`, `docs/package-boundary/eks-addon-lifecycle.md` row "Generation" |
|
||||
| K3s server roles | `server` (cluster-init) and `server-join`; fixed at first boot. | Identifies the delivery VM only until `[0]` is replaced. | `k3s_server_vm.go:527-531`, `:585-589` |
|
||||
| Etcd / kube leader election | K3s embedded etcd and kube controllers elect internally; nothing in Spinifex reads them. | Not used. | no reference in `scripts/images/eks-node/*.sh` or `handlers/eks` |
|
||||
|
||||
## 8. Candidate fencing evidence available today
|
||||
|
||||
"Fences" means the old executor provably cannot render or apply, and cannot have its reports accepted.
|
||||
|
||||
| Evidence | Available how | Proves old executor cannot render/apply? | Gap and what would be needed |
|
||||
|---|---|---|---|
|
||||
| EC2 state `terminated` or instance not found | `cpControlAdapter.InstanceState` (system account `DescribeInstances`); `TerminateK3sServerVM` treats not-found as success. | Yes for render and apply once the VM is gone, if the instance record is authoritative; host-partition behaviour of the state record is unverified. | Reports already queued or retried by `eks-gateway-publish` are still accepted. Needs: state proven authoritative under host loss, and report acceptance bound to the current executor. `service_impl.go:153-170`, `k3s_server_vm.go:292-330` |
|
||||
| EC2 state `stopped` / `error` | Same query. | Only momentarily: the reconciler's restart ladder starts stopped/error members automatically, and the VM resumes delivery with the same identity. | Needs a stop that nothing restarts, or a guest-side disable that survives restart. `cluster_reconciler.go:708-771`, `:927-937` |
|
||||
| Describe error | Treated as lost. | No: the VM may be running. | Must not count as fencing evidence. `cluster_reconciler.go:982-1007` |
|
||||
| Removal from `ControlPlaneNodes` | `SwapControlPlaneMember`, `replaceControlPlaneForRestore`. | Partly: `ListInternalAddons` is denied, but the agent keeps its current render on fetch failure, K3s keeps applying those files; its reports are now denied by the membership check, but reports already in flight are still applied. | Needs: publish gated by executor and identity attached on relay; a guest rule that a denied fetch stops delivery. `internal_authz.go:74`, `mulga-eks-addon-sync.sh:344-348` |
|
||||
| Instance ID / incarnation | Session name = instance ID; stable across stop/start. | No: there is no per-boot incarnation, so a restarted VM is indistinguishable from before. | Needs an executor epoch recorded host-side and checked per request; instance ID alone identifies, it does not fence. `assume_role.go:136-161` |
|
||||
| KV revisions | Meta revision, add-on record revision, lease revision. | No: meta revision churns with health writes, add-on CAS ignores sender, lease revision is host-only and private. | Needs a dedicated, monotonic executor epoch written only on reassignment. `cluster_state.go:416-436`, `kvlease/lease.go:44-52` |
|
||||
| IMDS credential expiry | Instance-role creds last 3600 s, re-minted 5 min early. | No: up to an hour of validity after the last mint, and a running VM keeps re-minting. | Not a fence on its own. `domains/ec2/guestmetadata/credentials.go:14-20` |
|
||||
| Credential revocation | `RevokeRoleSessions` deletes sessions by role (called on role delete); no per-instance revocation. | No per-executor effect: the role is shared by every CP VM of every cluster. | Needs a per-session-name revocation, and proof the gateway rejects a revoked session; a running VM would still re-mint via IMDS unless IMDS also refuses it. `handlers/sts/session_credentials.go:212-219`, `handlers/iam/roles.go:221` |
|
||||
| Static system keys | Shared system access key on fallback. | No: shared, unrevocable per VM, no identity. | Fallback VMs cannot be executors under any identity-based rule. `k3s_server_role.go:30-50` |
|
||||
| K3s node removal / etcd member prune | `EKS_ETCD_PRUNE_PEER_IP` handled in `k3s-first-boot.sh`. | Would stop apply through the shared datastore if the member is removed, but not render to the VM's own disk. The prune path appears unreachable: the variable is set only on replacements, which are `server-join`, and `k3s-first-boot` runs only for `server`. | Needs a reachable prune, verification that a removed member's apiserver cannot write, and still the report gate. `k3s-first-boot.sh:165-187`, `k3s_ha_control_plane.go:153`, `eks-node-role.sh:136`, `:151-166` |
|
||||
| Guest service disable | None remotely; the only host-to-guest instruction channel is the boot-time recovery directive. | No. | Needs an online directive the agent polls, plus an acknowledgement the host can verify; an acknowledgement from the old executor proves only that it was alive to send it. `recovery_directive.go:12-28` |
|
||||
| Restore fence | `confirmOldCPTerminated` after meta commit. | Yes for the old VM once terminate succeeds, but it runs after the new VM was launched and committed. | Ordering is successor-first; R6 requires fence-first. `restore_snapshot.go:262-336` |
|
||||
|
||||
## 9. Evidence each design option would need
|
||||
|
||||
No option is chosen here; each row lists only what it would rely on.
|
||||
|
||||
| Option | Evidence it needs that does not exist today |
|
||||
|---|---|
|
||||
| Host-assigned executor `{instanceId, epoch}` on the cluster record | An executor field and epoch advanced by CAS; the gateway attaching the caller instance ID (and epoch) to the relayed message; report apply checking it; a reassignment trigger that does not fire on describe errors; fence evidence from Section 8 rows that say "Yes". |
|
||||
| Executor follows meta `[0]` | A guest that can become executor after first boot (today role is fixed and replacements are `server-join`); the same gateway and relay identity checks; `[0]` reassignment tied to fencing of the old `[0]`. |
|
||||
| Run the agent on every server with guest-side leader election (kube Lease) | Host-visible proof of the guest leader (the host cannot read the kube Lease today); verification of K3s auto-deploy behaviour across servers; reports still need gateway-attached identity, since a partitioned old leader can still publish. |
|
||||
| Disable-on-handover directive | An online directive channel and agent polling; a rule that a fetch denial or missing directive stops rendering; an acknowledgement or instance-state proof before the successor acts. |
|
||||
| Credential-based fence | Per-instance session revocation, IMDS refusing to re-mint for a fenced instance, and gateway rejection of revoked sessions; still insufficient for files already rendered to the old VM's disk. |
|
||||
|
||||
## 10. Gaps
|
||||
|
||||
1. Delivery runs only on the first-boot `server` VM; replacing HA `[0]` leaves no member running `mulga-eks-addon-sync`, with no detection or recovery.
|
||||
2. Corrected: `PublishInternal` and `WebhookTokenReview` were policy-only, so any CP-role session, any tenant holding `eks:*`, or static system credentials could publish reports or resolve tokens for any cluster and account. Both now require cluster membership (see Section 3); the binding is to a member, not to the delivery executor.
|
||||
3. The relay attaches no caller identity, and the payload carries none, so the owner cannot tell which member, executor or epoch produced a report; this conflicts with the direction in draft PROP-SPINIFEXARCH-007 S3, which is not accepted.
|
||||
4. `ClusterMeta` has no role, executor or epoch field; `[0]` as "primary" is a convention that diverges from the guest role after a swap.
|
||||
5. A member removed from meta is denied fetches but keeps applying its last render and keeps having its reports accepted.
|
||||
6. Describe errors count as member loss, so a running primary can be replaced without fencing.
|
||||
7. The restore path launches and commits the successor before fencing the old CP.
|
||||
8. No per-instance credential revocation; the CP role and instance profile are shared across all clusters.
|
||||
9. `FAILED` clusters (create timeout, bootstrap failure) keep running CP VMs whose agent is still admitted while reports are dropped.
|
||||
10. The etcd peer prune for replacements appears unreachable (`server-join` does not run `k3s-first-boot`); not verified live.
|
||||
11. Whether K3s auto-deploy applies from a non-leader server's manifests dir is unverified, which bears on any multi-server option.
|
||||
12. No lease takeover for a cluster whose reconciler holder died until a daemon restarts, so reports are dropped meanwhile.
|
||||
@@ -1,450 +0,0 @@
|
||||
# EKS managed add-on: lifecycle correction design
|
||||
|
||||
**Status:** Draft for requirements review; no behaviour changes until approved.
|
||||
Reviewer decisions recorded (see "Review decisions"); the body is consistent with them.
|
||||
|
||||
This is the Phase 2 design for correcting the EKS managed add-on lifecycle, written against the code at `18794e017`.
|
||||
The present contract and its gaps are in `docs/package-boundary/eks-addon-lifecycle.md`, and that record stays authoritative for what the code does today.
|
||||
Paths are relative to `spinifex/` unless they start with `tests/`, `scripts/` or `contracts/`, which are relative to the spinifex repo root.
|
||||
|
||||
## Standing
|
||||
|
||||
The design is resource-specific.
|
||||
It refines proposed requirement Q-146 and the recorded conflict C-82 (`docs/specs/requirements/managed-services.md` in the mulga repo); both are open and pending review, and this document closes neither.
|
||||
It does not create an architecture proposal, a universal lifecycle interface, a generic finalizer package or a shared repository; ADR-0003 S2 (accepted) rules those out without two real consumers.
|
||||
It applies ADR-0003 S1 to S5 and ADR-0004 (both accepted) to one resource.
|
||||
It is not a NATS or JetStream redesign: every transport stays as it is.
|
||||
Edge IDs E1 to E9 refer to Section 7.1 of `docs/package-boundary/adr-0004-inventory-eks.md`.
|
||||
|
||||
Q-75 (contract evolution) and Q-76 (command, event and notification semantics) in `docs/specs/requirements/resource-lifecycle.md` are not accepted ADRs.
|
||||
They are Decided requirements: a named person stated them, and they have normative effect on this design.
|
||||
They are not open questions here; only the mechanics of implementing them are.
|
||||
Q-75 requires a safe compatibility path for older consumers, so a weaker substitute for a guarantee (such as a timing heuristic standing in for a fence) does not satisfy it.
|
||||
|
||||
## Review decisions
|
||||
|
||||
The reviewer decided the following; the sections below apply them.
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| R1 | Q-75 and Q-76 are Decided requirements with human provenance and normative effect, not accepted ADRs; only their implementation mechanics remain open. |
|
||||
| R2 | No v1 timing heuristics: neither "version match after a settle window" nor "delete complete after two listings" is used, because neither is a fence or proof of removal and both produce false success. |
|
||||
| R3 | `CREATE_FAILED` and `UPDATE_FAILED` are terminal for their generation. A late `ready` is recorded diagnostically and does not restore `ACTIVE`. A new `UpdateAddon` starts a new generation, and `DeleteAddon` stays available. `DEGRADED` recovers to `ACTIVE` on a valid report for the current generation. |
|
||||
| R4 | The external ARN stays name-only for this correction; the internal immutable `incarnationId` (UUID) is added now. The AWS-style per-incarnation ARN is a separate, explicitly migrated future change, because it affects IAM resource policies and must not break them silently. The divergence stays recorded. |
|
||||
| R5 | No v1 fallback. v1 stays byte-compatible and readable, but lifecycle mutations that need fencing or observed deletion require an authenticated, v2-capable active control-plane executor; otherwise they return `InvalidRequestException` telling the operator to upgrade the control-plane image. This is a deliberate compatibility gate, not a hard cutover. |
|
||||
| R6 | Delivery belongs to the EKS cluster, not to whichever server was primary. The cluster control plane assigns the active delivery executor, identified by executor identity and epoch; only that executor may fetch and report, a replaced executor is fenced before its successor acts, and the successor resumes from durable desired state without advancing the generation. The add-on side declares a narrow capability that the cluster owner provides. |
|
||||
| R7 | The v2 guest contract does not emit `InvalidParameterValue`; it uses structured internal reasons. The AWS adapter maps invalid customer input to `InvalidParameterException`, and a bad directive detected by the guest becomes a reported realisation failure. v1 is unchanged. |
|
||||
| R8 | `namespaceConfig` is rejected at the gateway while the pinned SDK cannot represent or apply it, and is never silently dropped; this is its own early slice. |
|
||||
| R9 | Non-empty `configurationValues` stay rejected until a bundle publishes a schema and actually renders the value. |
|
||||
| R10 | The `iam:PassRole` check on `serviceAccountRoleArn` stays; it is already implemented (`gateway/eks/passrole.go`). |
|
||||
| R11 | `InvalidRequestException` answers a mutation that conflicts with an in-progress lifecycle operation; `ResourceInUseException` is reserved for a conflicting resource on create. |
|
||||
| R12 | Deadlines are profile-configured and the chosen deadline is persisted when the operation is accepted; no per-bundle timing values yet. |
|
||||
| R13 | A delete that supersedes an in-flight update marks that update `Cancelled`. |
|
||||
| R14 | `serviceAccountRoleArn` is rejected for bundles with no IRSA marker. |
|
||||
| R15 | The GPU add-on is not re-created automatically after a user deliberately deleted it. |
|
||||
| R16 | Finalisation is an explicit, operator-visible add-on migration action with preflight evidence that legacy consumers are gone, as Q-75 requires. |
|
||||
|
||||
## AWS behaviour relied on
|
||||
|
||||
Verified against the AWS EKS API reference on 2026-10-08:
|
||||
|
||||
- `Addon.status` is one of `CREATING`, `ACTIVE`, `CREATE_FAILED`, `UPDATING`, `DELETING`, `DELETE_FAILED`, `DEGRADED`, `UPDATE_FAILED` (API_Addon).
|
||||
- `CreateAddon` and `DeleteAddon` examples show `addonArn` as `arn:aws:eks:{region}:{account}:addon/{cluster}/{addon}/{uuid}`, so each incarnation has its own suffix (API_CreateAddon, API_DeleteAddon examples; the type reference does not state the format).
|
||||
- `UpdateAddon` returns an `Update` with its own `id`; the example returns `status: InProgress` and `type: AddonUpdate` with `params` such as `ServiceAccountRoleArn` and `ResolveConflicts` (API_UpdateAddon).
|
||||
- `Update.id` is "a UUID that is used to track the update"; `Update.status` is `InProgress`, `Failed`, `Cancelled` or `Successful`; `errors` carries details of a `Failed` update (API_Update).
|
||||
- `DescribeUpdate` takes `addonName` as a query parameter, "required if the update is an add-on update"; `Successful` means complete and `Failed` carries an error detail (API_DescribeUpdate).
|
||||
- `DeleteAddon` returns the add-on with `status: DELETING` (example), and `preserve` "preserves the add-on software on your cluster but Amazon EKS stops managing any settings for the add-on. If an IAM account is associated with the add-on, it isn't removed" (API_DeleteAddon).
|
||||
- `resolveConflicts` is `OVERWRITE`, `NONE` or `PRESERVE`; on create, with no self-managed version installed, "Amazon EKS sets all values to default values, regardless of the option that you specify"; on update, `NONE` leaves a changed value and "the update might fail", `OVERWRITE` resets it, `PRESERVE` keeps it (API_CreateAddon, API_UpdateAddon).
|
||||
- `configurationValues` "are validated against the schema returned by `DescribeAddonConfiguration`" (API_CreateAddon, API_UpdateAddon).
|
||||
- `podIdentityAssociations` on update: "If this value is left blank, no change. If an empty array is provided, existing associations owned by the add-on are deleted" (API_UpdateAddon).
|
||||
- The documented errors for `CreateAddon` and `UpdateAddon` are `ClientException`, `InvalidParameterException`, `InvalidRequestException` ("invalid given the state of the cluster"), `ResourceInUseException` (409), `ResourceNotFoundException` (404) and `ServerException`; `DeleteAddon` lists the same set without `ResourceInUseException`.
|
||||
`IdempotentParameterMismatch` is not among them.
|
||||
- `CreateAddon` accepts `namespaceConfig`, which the pinned `aws-sdk-go` v1.55.8 model does not carry.
|
||||
|
||||
Not verified, and marked where used: which error AWS returns for an update or delete issued while another operation is in progress (R11 decides ours); whether `ready` after `CREATE_FAILED` or `UPDATE_FAILED` returns the add-on to `ACTIVE` (R3 decides ours); how long AWS honours a `clientRequestToken`; what a token reused with different parameters returns; whether `DescribeUpdate` finds an update of a deleted predecessor; the `AddonIssue` code set; whether a role passed to an add-on that needs no IAM permissions is rejected (R14 decides ours); and the Terraform provider's waiters.
|
||||
|
||||
## 1. Identity and generation
|
||||
|
||||
| Item | Design |
|
||||
|---|---|
|
||||
| Incarnation | `incarnationId`: a random UUID minted when `CreateAddon` is accepted and never changed. Delete followed by create of the same name yields a new incarnation. |
|
||||
| Desired generation | `desired.generation`: an unsigned integer, 1 on create, incremented by exactly one in the same CAS that accepts each `UpdateAddon` and each `DeleteAddon`. Tag changes do not advance it, because tags are not delivered. |
|
||||
| Where stored | In the add-on record `clusters/{cluster}/addons/{addon}` in `eks-account-{account}`, next to the existing fields, so one revision CAS covers intent, generation and status. No new bucket or key family. |
|
||||
| Lookup key | Still cluster plus add-on name; the incarnation is a fence, not an address. |
|
||||
|
||||
The external ARN stays `addon/{cluster}/{addon}`, minted by the owner and independently derived by the gateway authorizer from names (inventory edge E9), and `incarnationId` is internal (R4).
|
||||
This keeps authorization and existing IAM policies unchanged.
|
||||
The divergence from AWS remains recorded as contract gap 10: AWS's `CreateAddon` reference examples carry a per-incarnation suffix, and a recreated Spinifex add-on reuses its predecessor's ARN.
|
||||
Adopting the suffix is a separate change with its own explicit migration, because policies naming today's ARN would stop matching new add-ons and the authorizer would have to obtain the ARN from the owner before its policy check.
|
||||
Shaping `incarnationId` as a UUID keeps that change possible.
|
||||
|
||||
## 2. Directives, reports and the stale-report rule
|
||||
|
||||
A directive is what the controller wants the guest to realise for one add-on; a report is what the guest observed.
|
||||
In the successor contract (Section 9) both carry `incarnationId` and `generation`, and both carry the delivery executor's epoch (Section 10).
|
||||
|
||||
Directive listing (v2): the response carries `executorEpoch` once, then one entry per add-on:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `addonName`, `incarnationId`, `generation` | Identity and the desired generation this directive represents. |
|
||||
| `mode` | `install`, `remove` or `relinquish`. |
|
||||
| `addonVersion`, `serviceAccountRoleArn`, `configurationValues` | The desired spec; for `remove` and `relinquish`, the last installed spec, so a guest without local state can still render the object list it must remove. |
|
||||
|
||||
Report (v2): `addon`, `incarnationId`, `generation`, `executorEpoch`, `phase` (`applied`, `ready`, `failed`, `removing`, `removed`, `relinquished`), `reason` (a structured code, required when `phase` is `failed`), `message`, `ts`.
|
||||
The gateway attaches the executor identity it derived from the authenticated caller; no executor identity is read from the payload.
|
||||
|
||||
Exact comparison rule, applied by the owner to record `R` and report `P`, where `X` is the cluster's current active executor (identity and epoch):
|
||||
|
||||
1. If `P`'s gateway-attached executor identity is not `X`'s identity, or `P.executorEpoch` is not `X`'s epoch, discard `P` as a stale executor.
|
||||
2. If `R` is absent, discard `P`.
|
||||
3. If `P.incarnationId != R.incarnationId`, discard `P` as a stale incarnation.
|
||||
This rejects every report from before a delete and recreate.
|
||||
4. If `P.generation < R.desired.generation`, discard `P` as stale.
|
||||
This rejects reports produced for a superseded update and reports produced before a delete was accepted.
|
||||
5. If `P.generation > R.desired.generation`, discard `P` and log a protocol error; a guest cannot learn a generation the controller has not issued.
|
||||
6. If `P.phase` does not fit `R.desired.mode` (`ready` while removing, `removed` while installing), discard `P` and log a protocol error.
|
||||
7. Otherwise apply the transition in Section 6 through a revision CAS on `R`; on a CAS conflict, re-read `R` and start again from step 1.
|
||||
|
||||
Within one generation, reports apply in arrival order.
|
||||
A reordered pair from the same generation describes the same desired state, and the level-triggered agent corrects it on its next tick.
|
||||
Discards never write the record; they are counted in a metric, labelled by rule, so a stuck or fenced guest is visible.
|
||||
|
||||
## 3. Desired intent separated from observed state
|
||||
|
||||
Record shape sketch (JSON field names; existing fields unchanged):
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 2,
|
||||
"addonName": "aws-load-balancer-controller",
|
||||
"arn": "arn:aws:eks:...:addon/prod/aws-load-balancer-controller",
|
||||
"incarnationId": "6f1c...",
|
||||
"createdAt": "...", "modifiedAt": "...", "tags": {},
|
||||
"origin": "user",
|
||||
"createToken": {"token": "...", "paramHash": "..."},
|
||||
"desired": {
|
||||
"generation": 3, "acceptedAt": "...", "mode": "install",
|
||||
"addonVersion": "2.11.0", "serviceAccountRoleArn": "...", "configurationValues": "",
|
||||
"resolveConflicts": "OVERWRITE",
|
||||
"deletion": null
|
||||
},
|
||||
"observed": {"generation": 2, "phase": "ready", "reason": "", "message": "", "receivedAt": "...",
|
||||
"contract": "v2", "executorEpoch": 4},
|
||||
"status": "UPDATING",
|
||||
"health": [{"code": "...", "message": "..."}],
|
||||
"deadline": {"at": "...", "duration": "...", "profileKey": "..."},
|
||||
"updates": [{"id": "...", "type": "AddonUpdate", "status": "InProgress", "generation": 3,
|
||||
"params": [{"type": "AddonVersion", "value": "2.11.0"}], "createdAt": "...",
|
||||
"token": "...", "paramHash": "...", "errors": []}],
|
||||
"addonVersion": "2.11.0", "serviceAccountRoleArn": "...", "configurationValues": ""
|
||||
}
|
||||
```
|
||||
|
||||
`desired` is written only by API commands; `observed` only by accepted reports (and by diagnostic late reports, Section 6); `status`, `health` and `deadline` are owner-derived from both and stored so reads stay a single get.
|
||||
`origin` is `user` or `auto-staged` (Section 13).
|
||||
The trailing top-level `addonVersion`, `serviceAccountRoleArn` and `configurationValues` are the legacy fields, kept in step with `desired` during the transition so an older binary can still read the record.
|
||||
Updates are kept inline, bounded to the most recent N per incarnation, so accepting an update and advancing the generation is one CAS rather than a two-key write.
|
||||
|
||||
The staged manifest sub-key becomes a projection of `desired`.
|
||||
The internal listing is built from the records, and the manifest key is still written for rollback until finalisation.
|
||||
That removes today's crash window between the record put and staging (contract row "Restart and interruption"), because there is no longer a second write that can be missed.
|
||||
|
||||
Migration of existing records, as an idempotent and resumable expand step:
|
||||
|
||||
| Today's status | `incarnationId` | `desired.generation` | `observed` | Status after |
|
||||
|---|---|---|---|---|
|
||||
| `ACTIVE`, `DEGRADED` | newly minted | 1 | generation 1, phase from status, contract `v1` | unchanged |
|
||||
| `CREATING`, `UPDATING` | newly minted | 1 | none | unchanged; a deadline from the profile starts at migration |
|
||||
| `CREATE_FAILED` | newly minted | 1 | none | unchanged, with the stored `health` as an issue |
|
||||
|
||||
Minting at migration is safe because no deployed report carries an incarnation.
|
||||
Existing non-empty `configurationValues` are preserved, never silently dropped, and gain a health issue stating they are not applied (Section 7).
|
||||
An older binary that rewrites a migrated record decodes it into today's `Record` and drops the new fields, so the new lifecycle semantics activate only once every daemon in the Region is known to run the new release (Section 9).
|
||||
Activation does not end rollback; the separate finalisation action does (Section 9, R16).
|
||||
|
||||
## 4. Deletion
|
||||
|
||||
On a cluster whose active executor has proven v2, `DeleteAddon` creates a durable obligation and keeps ownership until guest-side removal is observed:
|
||||
|
||||
1. One CAS sets `status: DELETING`, advances `desired.generation` to `D`, sets `desired.mode` to `remove` (or `relinquish` when `preserve=true`), records `desired.deletion.requestedAt`, persists the delete deadline and, if an update is in flight, marks it `Cancelled` (R13).
|
||||
The response is the record in that state.
|
||||
2. The listing now carries a `remove` or `relinquish` directive for generation `D` instead of omitting the add-on.
|
||||
3. The record stays until a report for (`incarnationId`, `D`) with phase `removed` or `relinquished` arrives; the owner then deletes the record and the manifest key.
|
||||
4. While the record exists, `CreateAddon` of the same name returns `ResourceInUseException`, and `DescribeAddon` and `ListAddons` show it as `DELETING` or `DELETE_FAILED`.
|
||||
Recreate is therefore serialised behind removal, so a new incarnation never overlaps its predecessor's objects in the guest.
|
||||
|
||||
On a cluster without a v2 executor, `DeleteAddon` returns `InvalidRequestException` (Section 9).
|
||||
|
||||
What "observed" means for a v2 guest:
|
||||
today's GC loop deletes the rendered file, runs `kubectl delete --wait=false` on a temporary copy and forgets it.
|
||||
The v2 agent keeps the removed manifest in a durable per-add-on location instead of `mktemp`, and on every tick runs `kubectl get -f <copy> --ignore-not-found`.
|
||||
It reports `removing` while any listed object remains and `removed` only when every object in the rendered manifest returns NotFound, including namespaces held by finalizers.
|
||||
If the apiserver is unreachable, it sends nothing for that add-on rather than `removed`.
|
||||
If it has no local copy (state lost, never rendered, or a newly assigned executor), it renders the bundle from the directive's last spec into a scratch file and uses that object list.
|
||||
It keeps reporting `removed` on every tick while the directive is listed, so a lost report is repeated.
|
||||
|
||||
For `relinquish`, the v2 agent deletes only the rendered file, leaving the objects in place (K3s does not delete applied objects when the auto-deploy file disappears, per the agent's own comment), and reports `relinquished` once the file is gone.
|
||||
What K3s does with its `Addon` object in that case is unverified.
|
||||
|
||||
Unavailable guest: the record stays `DELETING` and the directive stays listed.
|
||||
When the persisted delete deadline passes, the owner sets `DELETE_FAILED` with a health issue, keeps the record and keeps listing the directive.
|
||||
`DELETE_FAILED` is recoverable: a later `removed` report for (`incarnationId`, `D`) completes the deletion, and a repeated `DeleteAddon` re-arms the deadline without advancing the generation, since the intent is unchanged.
|
||||
A guest that recovers after hours therefore converges the retained directive with no operator action.
|
||||
|
||||
Cluster deletion destroys the control-plane VMs and with them every in-cluster object, so add-on removal is implied there.
|
||||
That path should call an add-on-owned teardown capability that records removal-by-cluster-teardown instead of the blanket `DeleteClusterPrefix` sweep (inventory edge E1); it is not subject to the v2 gate.
|
||||
|
||||
## 5. Update as a durable asynchronous operation
|
||||
|
||||
`UpdateAddon` is accepted in one CAS that advances `desired.generation` to `G`, sets `status: UPDATING`, appends an update `{id: new UUID, type: AddonUpdate, status: InProgress, generation: G, params}` and persists the update deadline.
|
||||
The response is that update, not the add-on ARN marked `Successful`.
|
||||
|
||||
| Event for generation `G` | Update status | Add-on status |
|
||||
|---|---|---|
|
||||
| `ready` report | `Successful` | `ACTIVE` |
|
||||
| `failed` report | `Failed`, with `reason` and `message` in `errors` | `UPDATE_FAILED` |
|
||||
| Deadline passes with no `ready` | `Failed` ("timed out") | `UPDATE_FAILED` |
|
||||
| `DeleteAddon` accepted first | `Cancelled` | `DELETING` |
|
||||
|
||||
`DescribeUpdate` with `addonName` routes to the add-on owner and finds the update by `id` among the record's retained updates for the current incarnation; an unknown `id` is `ResourceNotFoundException`.
|
||||
Whether AWS still resolves an update of a deleted predecessor is unverified.
|
||||
`ListUpdates` with `addonName` lists the retained ids.
|
||||
Both are gateway stubs today, so this work also implements them for add-ons only; cluster and node-group updates stay stubs.
|
||||
|
||||
`UpdateAddon` is accepted from `ACTIVE`, `DEGRADED`, `CREATE_FAILED` and `UPDATE_FAILED`.
|
||||
From `CREATING`, `UPDATING`, `DELETING` and `DELETE_FAILED` it conflicts with an in-progress lifecycle operation and returns `InvalidRequestException` (R11).
|
||||
On a cluster without a v2 executor it returns `InvalidRequestException` (Section 9).
|
||||
An update whose parameters equal the current desired spec is still accepted as a new generation, so a client gets a trackable update.
|
||||
|
||||
## 6. Status set and transitions
|
||||
|
||||
The owner adds `UPDATE_FAILED` and `DELETE_FAILED` to its `Status`; `DELETING` becomes a stored status.
|
||||
Every report row below assumes the report passed the rule in Section 2.
|
||||
|
||||
| From | Event | To | Notes |
|
||||
|---|---|---|---|
|
||||
| absent | `CreateAddon` accepted | `CREATING` | generation 1; create deadline persisted |
|
||||
| `CREATING` | `ready` | `ACTIVE` | |
|
||||
| `CREATING` | `failed`, or create deadline | `CREATE_FAILED` | terminal for generation 1 |
|
||||
| `ACTIVE` | `failed` | `DEGRADED` | |
|
||||
| `DEGRADED` | `ready` | `ACTIVE` | health clears |
|
||||
| `ACTIVE`, `DEGRADED`, `CREATE_FAILED`, `UPDATE_FAILED` | `UpdateAddon` accepted | `UPDATING` | generation +1 |
|
||||
| `UPDATING` | `ready` | `ACTIVE` | update `Successful` |
|
||||
| `UPDATING` | `failed`, or update deadline | `UPDATE_FAILED` | update `Failed`; terminal for that generation |
|
||||
| `CREATING`, `ACTIVE`, `DEGRADED`, `CREATE_FAILED`, `UPDATE_FAILED` | `DeleteAddon` accepted | `DELETING` | generation +1 |
|
||||
| `UPDATING` | `DeleteAddon` accepted | `DELETING` | generation +1; in-flight update `Cancelled` |
|
||||
| `DELETING` | `DeleteAddon` | `DELETING` | idempotent; same generation; record returned |
|
||||
| `DELETE_FAILED` | `DeleteAddon` | `DELETING` | same generation; deadline re-armed |
|
||||
| `DELETING` | delete deadline | `DELETE_FAILED` | record and directive retained |
|
||||
| `DELETING`, `DELETE_FAILED` | `removed` or `relinquished` | absent | record and manifest deleted |
|
||||
| `CREATE_FAILED`, `UPDATE_FAILED` | `ready` for the same generation | unchanged | recorded diagnostically in `observed`; does not restore `ACTIVE` |
|
||||
|
||||
Failure is terminal for its generation and recoverable only by a new generation (R3).
|
||||
Today `ready` lifts `CREATE_FAILED` to `ACTIVE`; that changes, because a failed update must keep a stable `Failed` outcome in `DescribeUpdate` and Terraform should not see a failed resource silently turn healthy.
|
||||
The cost is low: the agent reports `failed` only for deterministic causes (missing baked bundle, webhook certificate generation, a directive the guest cannot realise), and a slow rollout reports `applied`, not `failed`.
|
||||
`DEGRADED` is different: it describes a realised generation whose health dipped, so a valid `ready` for the current generation returns it to `ACTIVE`.
|
||||
Staging is no longer a failure source, because the listing is a projection of the record.
|
||||
|
||||
## 7. Per-field decisions
|
||||
|
||||
The catalogue holds five bundles, one version each: `aws-load-balancer-controller` 2.11.0 and `aws-ebs-csi-driver` 1.40.1 (both `RequiresIRSA`), `argocd` 3.0.23, and the hidden `nvidia-device-plugin` 0.17.4 and `spinifex-noop` 0.1.0.
|
||||
No accepted field may be stored and ignored.
|
||||
Every rejection of invalid customer input uses `InvalidParameterException` (`awserrors.ErrorEKSInvalidParameter`); today's add-on validation returns `InvalidParameterValue`, which is not an EKS error code, and correcting it is part of slice 2.
|
||||
State conflicts use `InvalidRequestException` and an existing resource on create uses `ResourceInUseException` (R11).
|
||||
|
||||
| Field | Today | Decision |
|
||||
|---|---|---|
|
||||
| `addonVersion` | Validated against the catalogue; applied. | Keep. Each bundle supports its one version; an unknown version is rejected. A change advances the generation. |
|
||||
| `serviceAccountRoleArn` | Stored, `iam:PassRole` checked (already implemented, `gateway/eks/passrole.go`), rendered as web-identity env when the bundle has IRSA markers. | Implement for `aws-load-balancer-controller` and `aws-ebs-csi-driver`; keep the `iam:PassRole` check (R10). Reject for `argocd`, `nvidia-device-plugin` and `spinifex-noop`, whose bundles have no IRSA markers, so the role would be inert (R14). An update that changes it advances the generation. |
|
||||
| `configurationValues` | Stored, staged, transported, discarded by the agent. | Reject a non-empty value for every bundle until that bundle publishes a schema through `DescribeAddonConfiguration` and the agent renders it (R9). Implement per bundle afterwards, `aws-load-balancer-controller` first. Existing stored values are kept and flagged (Section 3). |
|
||||
| `resolveConflicts` | Accepted and dropped. | Create: accept all three values; with no pre-existing install AWS treats them alike. The v2 agent with `NONE` or `PRESERVE` checks the bundle's objects before the first render and reports `failed` on a conflict. Update: accept `OVERWRITE` (what K3s auto-deploy does); reject `NONE` and `PRESERVE` until field-level drift detection exists, because K3s will overwrite regardless. Persist the value and return it as an update param. |
|
||||
| `preserve` (delete) | Ignored. | Implement as `relinquish` (Section 4). Clusters without a v2 executor cannot delete at all (Section 9). |
|
||||
| `podIdentityAssociations` | Accepted and dropped. | Reject a non-empty list; EKS Pod Identity is not implemented. Accept an absent field and, on update, an empty list, which is a correct no-op because no associations exist. Return the field empty. |
|
||||
| `clientRequestToken` | Dropped. | Implement for create and update (Section 8). |
|
||||
| `tags` | Stored on create and returned; tag actions on the add-on ARN return `NotImplemented`. | Keep create-time tags. Implement `TagResource`, `UntagResource` and `ListTagsForResource` for add-on ARNs as an owner-provided tag mutation that does not advance the generation and is not gated, because a Terraform tag change on `aws_eks_addon` otherwise fails. |
|
||||
| `namespaceConfig` | Not in the pinned SDK model, so silently dropped when the body is decoded. | Reject at the gateway when present in the raw body, while the pinned SDK cannot represent or apply it; never drop it silently (R8, slice 1). |
|
||||
|
||||
## 8. Client token idempotency
|
||||
|
||||
The token is stored in the record it created, not in a TTL bucket, so create stays one atomic write.
|
||||
|
||||
- `CreateAddon` becomes a CAS-create (`kv.Create`), fixing contract gap 1.
|
||||
The record stores the token and a hash of the request without the token.
|
||||
- A create that loses to an existing record compares tokens: same token and same hash returns the existing add-on as it now stands; same token and a different hash is rejected; a different or absent token is `ResourceInUseException`.
|
||||
- `UpdateAddon` with a token already held by a retained update of the current incarnation returns that update if the hash matches and is rejected if it does not; the replayed update is not re-applied and does not advance the generation.
|
||||
- `DeleteAddon` has no token; a repeat while `DELETING` returns the record again, and a repeat once the record is gone is `ResourceNotFoundException`, as today.
|
||||
- The token lives as long as its record or retained update. A retry after the add-on was deleted creates a new incarnation.
|
||||
- A token reused with different parameters: recommend `InvalidParameterException`, because `IdempotentParameterMismatch` is not in the EKS error lists, although `CreateCluster` returns it today (open question 1).
|
||||
|
||||
## 9. Guest contract evolution and the v1 compatibility gate
|
||||
|
||||
`contracts/eks/v1` is frozen; its README already says a generation-aware protocol needs a successor.
|
||||
|
||||
- `contracts/eks/v2` adds the directive and report fields of Section 2, the new phases and structured `reason` codes.
|
||||
It is served on a new internal route (for example `GET /clusters/{c}/internal-addons/v2/{account}`) and reports go on a new subject (for example `eks.addon.v2.{account}.{cluster}.status`), so neither side has to sniff payloads.
|
||||
- Following Q-75, the producer never emits v2 to a guest that has not asked for it, and v1 routes, subject and bodies stay byte-identical, pinned by the existing contract tests.
|
||||
- The v2 internal routes do not return AWS error codes such as `InvalidParameterValue` (R7).
|
||||
A malformed guest request gets a structured internal reason (for example `NotActiveExecutor`, `StaleExecutorEpoch`, `UnknownContractMajor`, `MalformedReport`).
|
||||
Invalid customer input never reaches the guest: the AWS adapter rejects it with `InvalidParameterException`.
|
||||
A directive the guest finds it cannot realise (for example a bundle missing from the image) is reported as `failed` with a `reason`, and becomes `CREATE_FAILED`, `UPDATE_FAILED` or `DEGRADED` through Section 6.
|
||||
- The v2 agent ships in a new EKS node image, so a cluster stays on v1 until its control-plane VM is replaced; the mixed window can be long.
|
||||
|
||||
How v2 capability is established: the cluster owner records it per executor epoch, from the authenticated caller only.
|
||||
The gateway already authenticates the internal routes as a control-plane VM session and binds the session name (the VM's instance ID) to the named cluster (`AuthorizeInternal`).
|
||||
For the v2 route it additionally requires that instance ID to be the cluster's active executor for the current epoch (Section 10), and only then does the cluster owner mark that epoch v2-capable.
|
||||
No field in a guest payload can assert capability or executor identity.
|
||||
A new epoch starts without capability, so the gate re-closes on every executor change until the new executor fetches v2.
|
||||
|
||||
Gated actions: on a cluster whose current epoch is not v2-capable, `CreateAddon`, `UpdateAddon` and `DeleteAddon` (with or without `preserve`) return `InvalidRequestException` with a message instructing the operator to upgrade the cluster's control-plane image.
|
||||
Each of these needs either a fenced report (create and update reach `ACTIVE` only on a report for the current incarnation and generation) or observed deletion, which v1 cannot provide.
|
||||
Not gated: `DescribeAddon`, `ListAddons`, `DescribeUpdate`, `ListUpdates`, `DescribeAddonVersions`, `DescribeAddonConfiguration`, tag actions, and cluster-teardown removal (Section 4).
|
||||
|
||||
v1 clusters during the transition:
|
||||
|
||||
| Concern | v1 handling |
|
||||
|---|---|
|
||||
| Reads | Records are readable and listed; the v1 route serves a v1 projection of the records. |
|
||||
| Reports | v1 reports carry no incarnation, generation or epoch, so they never complete a create, update or delete. They are recorded diagnostically in `observed` with contract `v1`. Whether they may still move an add-on with no operation in flight between `ACTIVE` and `DEGRADED` is open question 6. |
|
||||
| Operations in flight at activation | A migrated `CREATING` or `UPDATING` record cannot complete on v1 and reaches `CREATE_FAILED` or `UPDATE_FAILED` at its deadline; it converges after the image upgrade through a new `UpdateAddon` or a `DeleteAddon`. |
|
||||
| Conformance | Q-146 conformance is claimed only for clusters whose executor has proven v2. |
|
||||
|
||||
This is a compatibility gate, not a hard cutover: v1 guests keep working, nothing is migrated destructively, and the gate opens per cluster once its image is upgraded.
|
||||
|
||||
Activation and finalisation are separate steps:
|
||||
|
||||
- **Activation** turns on the new lifecycle semantics and the gate once every daemon in the Region is known to run the new release, so no older binary can complete a delete by erasing the record; the transitional record stays readable by the previous release, so rollback is still possible.
|
||||
- **Finalisation** is an explicit, operator-visible add-on migration action (R16).
|
||||
Its preflight shows evidence that legacy consumers are gone: no daemon older than the activating release in the Region, and no remaining reader of the manifest key or legacy record fields.
|
||||
It then stops writing the manifest key and legacy fields, and ends rollback.
|
||||
It never runs because a new binary started.
|
||||
|
||||
Removing v1 requires a published retirement, detection that no cluster in the Region still fetches the v1 route, and evidence that every cluster has moved to a v2 image; until then both are served.
|
||||
|
||||
## 10. Delivery executor
|
||||
|
||||
Delivery belongs to the EKS cluster, not to whichever server VM was first booted as primary (R6).
|
||||
Today `eks-node-role.sh` enables `mulga-eks-addon-sync` only on the primary server, and any control-plane member that passes `AuthorizeInternal` may fetch and report.
|
||||
|
||||
Ownership split:
|
||||
|
||||
| Owner | Owns |
|
||||
|---|---|
|
||||
| Add-on owner (`domains/eks/addon`) | Desired state, generation, observed lifecycle and the comparison rule. |
|
||||
| Cluster owner | Assignment of the active delivery executor: `executor {instanceId, epoch}`, plus whether that epoch has proven v2. |
|
||||
|
||||
The add-on package declares a narrow capability, for example `ActiveDeliveryExecutor(account, cluster) -> {instanceId, epoch, v2Capable}`, and the cluster owner provides it.
|
||||
The add-on package does not read `ClusterMeta` or other cluster internals.
|
||||
The gateway's internal-route check uses the same capability.
|
||||
|
||||
Rules:
|
||||
|
||||
- Only the active executor for the current epoch may fetch the v2 listing or post a v2 report; the gateway derives the executor identity from the authenticated caller and rejects any other control-plane member with `NotActiveExecutor`.
|
||||
- Every report carries the `executorEpoch` it was issued in its listing; the gateway rejects a mismatched epoch, and the owner re-checks it (Section 2 rule 1) because the epoch may advance between the gateway check and the CAS.
|
||||
- A replaced executor is fenced before its successor is assigned: the cluster owner advances the epoch only after the old executor is confirmed unable to render (VM stopped or terminated, or its delivery service disabled), not merely because it is unreachable.
|
||||
There is never an autonomous second primary.
|
||||
- The new executor reads durable desired state from the listing and resumes delivery and reconciliation, including retained `remove` directives; nothing advances `desired.generation`.
|
||||
- The handover interval is guest unavailability: operations keep their persisted deadlines, and a deadline that passes during handover produces the same `CREATE_FAILED`, `UPDATE_FAILED` or `DELETE_FAILED` as any outage.
|
||||
A cluster with no assigned executor is an unavailable guest.
|
||||
|
||||
The cluster lease `spinifex-eks-leader/{account}/{cluster}` elects the host-side daemon that reconciles the cluster; it is unrelated to the executor epoch, which fences the guest side.
|
||||
|
||||
## 11. Restart reconciliation
|
||||
|
||||
Every obligation is in the record, so restart needs no request memory.
|
||||
An add-on reconcile pass runs under the cluster's existing lease (`spinifex-eks-leader/{account}/{cluster}`), at start and on a periodic tick, owned by `domains/eks/addon` and given its own host instead of the cluster reconciler's report callback (inventory edge E3).
|
||||
For each record it:
|
||||
|
||||
- re-projects the listing and, until finalisation, rewrites a missing or stale manifest key, which re-drives a create or update interrupted after its CAS;
|
||||
- keeps the `remove` or `relinquish` directive listed for `DELETING` and `DELETE_FAILED` records, which re-drives an interrupted delete;
|
||||
- applies the persisted deadlines (create, update, delete); and
|
||||
- retries nothing that already completed, because each transition is a CAS keyed on generation.
|
||||
|
||||
The only window left is a crash before the accepting CAS commits, in which case nothing was accepted and the client's retry with the same token is a new attempt.
|
||||
Which daemon re-drives: whichever holds the cluster lease after restart, through the existing `SpawnRegisteredReconcilers` respawn; no new process role is introduced.
|
||||
|
||||
## 12. Status publications are notifications
|
||||
|
||||
Under Q-76 a delivery report is a notification, a hint that observed state may have changed, and never proof of completion by itself: completion is the owner's CAS after the Section 2 rule accepts a report for the current executor epoch, incarnation and generation.
|
||||
Core NATS is therefore acceptable as the transport:
|
||||
|
||||
- **Lost:** the agent re-sends the current phase for every listed directive on every tick, including `removed` for a retained deletion, and the directive stays listed until the owner records completion; a report lost during a restart or with no lease holder is replaced within one tick.
|
||||
- **Duplicated:** applying the same (`incarnationId`, `generation`, `phase`) twice changes nothing.
|
||||
- **Delayed or reordered:** a late report from an earlier generation, incarnation or executor epoch fails the comparison rule; within one generation the next tick corrects it.
|
||||
- **Never arriving:** deadlines turn silence into `CREATE_FAILED`, `UPDATE_FAILED` or a retained `DELETE_FAILED`, so liveness does not depend on delivery.
|
||||
|
||||
Correctness rests on durable desired state, periodic guest reporting and restart reconciliation, not on the transport, so no JetStream stream, queue group or delivery change is proposed.
|
||||
v1 reports never drive a lifecycle completion (Section 9), so no weaker rule exists for v1.
|
||||
|
||||
## 13. GPU node-group auto-staging
|
||||
|
||||
`stageGPUDeviceAddon` (inventory edge E2) keeps calling the owner and gains the new create path: incarnation, generation 1, no token, no IRSA, `origin: auto-staged`.
|
||||
Interactions under this design:
|
||||
|
||||
- An existing record in any status, including `DELETING`, remains a no-op for the node group, so the plugin is not re-staged while it is being removed.
|
||||
- A user's deliberate deletion is remembered (R15): when a user-initiated `DeleteAddon` of `nvidia-device-plugin` completes, the add-on owner keeps a cluster-scoped declined marker in its own key space, and auto-staging skips the add-on while the marker exists.
|
||||
An explicit user `CreateAddon` of the add-on clears it, and cluster teardown removes it with the cluster.
|
||||
The marker's key, and whether a deletion of an `origin: user` record also sets it, are open question 7.
|
||||
- On a cluster whose executor has not proven v2, the create path returns `InvalidRequestException` (Section 9), so auto-staging is logged and skipped, as its errors are today.
|
||||
- `CREATE_FAILED` now persists (Section 6), so a device plugin that failed because the baked bundle was missing stays failed until updated or deleted; node-group readiness does not wait on it, and the failure is visible in `DescribeAddon`.
|
||||
- Errors stay logged and swallowed, and node-group delete never unstages, until the node-group owner is extracted and the dependency is designed; this design does not change that edge.
|
||||
|
||||
## 14. Required evidence
|
||||
|
||||
Unit and owner tests (fault injection at the owner's store and installer seams), E2E and Terraform:
|
||||
|
||||
1. Interruption before the accepting CAS (nothing accepted; retry with the same token creates once) and after it (reconcile re-projects the directive; no second incarnation).
|
||||
2. Repeated create (token replay returns the same incarnation), repeated update (same update id, generation advanced once) and repeated delete (same `DELETING` record, no second generation).
|
||||
3. Stale report after a newer update: a `ready` and a `failed` for generation `G-1` arriving after `G` is accepted leave the add-on `UPDATING`.
|
||||
4. Stale report after delete and recreate: a report carrying the old `incarnationId` never touches the new record.
|
||||
5. Terminal failure: a `ready` for the failed generation after `CREATE_FAILED` or `UPDATE_FAILED` is recorded in `observed` and leaves the status unchanged; `DEGRADED` returns to `ACTIVE` on a current-generation `ready`.
|
||||
6. Delete during `UPDATING` marks the update `Cancelled`.
|
||||
7. Controller restart while a create, an update and a delete are each outstanding: every one converges after respawn with no request memory.
|
||||
8. Unavailable guest during delete: the record stays `DELETING`, becomes `DELETE_FAILED` at the persisted deadline, and the directive stays listed.
|
||||
9. Eventual guest recovery: a `removed` report after `DELETE_FAILED` completes deletion; an `applied` then `ready` after a long outage completes a pending create or update if its deadline has not passed.
|
||||
10. Executor fencing: a fetch or report from a control-plane member that is not the active executor, or from a superseded epoch, is rejected at the gateway and at the owner; a handover resumes a pending create, update and delete without advancing the generation; the epoch does not advance while the old executor is merely unreachable.
|
||||
11. v1 compatibility gate: `CreateAddon`, `UpdateAddon` and `DeleteAddon` on a cluster without a v2-capable epoch return `InvalidRequestException`; reads and tag actions still work; v1 reports never complete an operation; v1 body and subject bytes are unchanged; capability cannot be asserted through a payload field.
|
||||
12. Field decisions: invalid customer input, including `namespaceConfig`, non-empty `configurationValues` and a role on a bundle without IRSA markers, returns `InvalidParameterException`; the role reaches the rendered bundle for both IRSA bundles.
|
||||
13. `DescribeUpdate` and `ListUpdates` for add-ons: `InProgress`, `Successful`, `Failed` with errors, and `Cancelled`.
|
||||
14. GPU auto-staging does not re-create `nvidia-device-plugin` after a user deleted it, and does after an explicit user create.
|
||||
15. Finalisation preflight refuses while a legacy consumer remains, and finalisation is visible to the operator.
|
||||
16. E2E in `tests/e2e/eks`: create, update (version and role), failed update, delete with observed removal, delete with `preserve`, guest outage across a delete, and executor replacement during an update.
|
||||
17. Terraform `aws_eks_addon`: repeated apply and destroy, an in-place update (version or role) and a tag change, alongside the unit and E2E evidence, as ADR-0003 S5 requires.
|
||||
The Terraform provider's create, update and delete waiters are unverified and must be read before the fixture is written.
|
||||
|
||||
## 15. Implementation slices
|
||||
|
||||
Each slice is small and ordered; behaviour-changing slices are named as such.
|
||||
|
||||
| # | Slice | Behaviour change |
|
||||
|---|---|---|
|
||||
| 0 | Expand the record: write `schemaVersion`, `incarnationId`, `origin`, `desired`, `observed` beside the legacy fields; migrate existing records; nothing reads the new fields yet. | None visible; persisted bytes grow. |
|
||||
| 1 | Reject `namespaceConfig` at the gateway when present in the raw body. | Yes: a previously dropped input is rejected. |
|
||||
| 2 | CAS-create, create token in the record, `InvalidParameterException` for add-on validation errors. | Yes: concurrent and retried creates; error code. |
|
||||
| 3 | Add-on reconcile pass with its own host under the cluster lease; listing built from records; manifest key kept for rollback. | None visible; clears the create-staging crash window. |
|
||||
| 4 | Remaining field decisions of Section 7, except `preserve` and `resolveConflicts` conflict detection. | Yes: previously accepted inputs are rejected. |
|
||||
| 5 | Cluster-owned delivery executor: executor identity and epoch, the declared capability, and the gateway's active-executor check on v2 routes. | None for v1 routes. |
|
||||
| 6 | `contracts/eks/v2`, v2 route and subject, structured reasons, v2 agent in a new node image, executor-epoch, incarnation and generation fencing, v2 capability recorded per epoch. | Yes, for v2 clusters only. |
|
||||
| 7 | Activation and the v1 compatibility gate for `CreateAddon`, `UpdateAddon` and `DeleteAddon`. | Yes: lifecycle mutations on v1 clusters return `InvalidRequestException`. |
|
||||
| 8 | Durable update operation, `UPDATE_FAILED`, persisted profile deadlines, `DescribeUpdate`/`ListUpdates` for add-ons, update token, terminal failure. | Yes: update response and status. |
|
||||
| 9 | Durable `DELETING`, `DELETE_FAILED`, delete deadline, recreate blocked while deleting, update `Cancelled` on delete. | Yes: delete response persists and describe no longer returns not found at once. |
|
||||
| 10 | `preserve` as `relinquish` and create-time conflict detection. | Yes. |
|
||||
| 11 | Tag actions on add-on ARNs. | Yes: `NotImplemented` becomes supported. |
|
||||
| 12 | GPU declined marker for deliberately deleted auto-staged add-ons. | Yes: a deleted device plugin stays deleted. |
|
||||
| 13 | Finalisation action with preflight; stop writing the manifest key and legacy fields. | None for current binaries; ends rollback. |
|
||||
|
||||
The per-incarnation ARN suffix and v1 retirement are separate later changes with their own migrations, not slices of this correction.
|
||||
|
||||
## 16. Open questions for the reviewer
|
||||
|
||||
These are implementation mechanics; the decisions above are not reopened.
|
||||
|
||||
1. Token reused with different parameters: `InvalidParameterException` (recommended) or `IdempotentParameterMismatch` to match `CreateCluster`, and should `CreateCluster` be revisited?
|
||||
2. Existing stored `configurationValues`: keep and flag (recommended), or reject further updates to such add-ons until cleared?
|
||||
3. How many updates to retain inline per incarnation?
|
||||
4. Is there an existing Region-wide release gate that can establish "every daemon runs the new release" for activation and supply finalisation's preflight evidence, or does slice 7 need one?
|
||||
5. Executor fencing mechanics: what evidence the cluster owner accepts that a replaced executor cannot render (instance state from the EC2 owner, a disabled service, an operator assertion), and who triggers reassignment.
|
||||
6. May a v1 report still move an add-on with no operation in flight between `ACTIVE` and `DEGRADED`, given it carries no generation or epoch?
|
||||
7. GPU declined marker: its key in the add-on owner's space, and whether deleting a user-created `nvidia-device-plugin` also sets it.
|
||||
8. Profile keys and default values for the create, update and delete deadlines.
|
||||
9. Rollback after activation: how the previous release presents records in `DELETING`, `DELETE_FAILED` or `UPDATE_FAILED`, which it does not model today.
|
||||
@@ -1,86 +0,0 @@
|
||||
# EKS add-on v1 compatibility: who would drop, erase or ignore added fields
|
||||
|
||||
Code read at 3a55e2ee1; not verified live.
|
||||
|
||||
Question: if a newer binary adds JSON fields (for example `incarnationId`, `generation`, `observedGeneration`, `origin`, `executorEpoch`) to the add-on record `clusters/{c}/addons/{a}`, the staged manifest `clusters/{c}/addons/{a}/manifest`, or a `contracts/eks/v1` payload, what does an older binary still running the code below do with them?
|
||||
|
||||
Verdicts:
|
||||
|
||||
- **DROP**: decodes into a Go struct and writes the re-encoded struct back, erasing undeclared fields (`encoding/json` keeps no unknown-field bag).
|
||||
- **OVERWRITE**: writes a freshly built value with no read of the stored bytes; whatever was stored is replaced.
|
||||
- **ERASE**: deletes the key; content is irrelevant.
|
||||
- **IGNORE**: read-only decode; unknown fields are skipped and nothing is written back.
|
||||
- **FAIL**: decode error when a known field's JSON type changes (not an added field).
|
||||
|
||||
## Version fences that exist today
|
||||
|
||||
| Fence | Where | Effect |
|
||||
|---|---|---|
|
||||
| Record schema/version field | none: `spinifex/domains/eks/addon/record.go:24` | No per-record version, so an old binary cannot tell a newer record from a v1 one. |
|
||||
| Manifest schema/version field | none: `spinifex/domains/eks/addon/record.go:40` | Same for the manifest. |
|
||||
| Per-account bucket schema stamp | `spinifex/handlers/eks/store.go:29` (`KVBucketEKSAccountVersion = 1`), checked at `store.go:203` | Bumping it makes every older binary refuse the whole `eks-account-{id}` bucket with `SchemaAheadError` (`spinifex/foundation/state/migrate/kv.go:87`), i.e. all EKS calls for that account, not just add-ons. |
|
||||
| Registered migrations for the bucket | none | `RegisterKV` is keyed by exact bucket name (`migrate/kv.go:48`, lookup at `kv.go:94`) while EKS buckets are named per account (`store.go:146`). |
|
||||
| Wire version | `contracts/eks/v1/addon_status.go:1-3` package doc | Says generation-aware delivery needs a successor version; nothing on the wire carries a version marker. |
|
||||
|
||||
## Record and manifest writers
|
||||
|
||||
| Writer | file:line | Record | Manifest |
|
||||
|---|---|---|---|
|
||||
| `Owner.Create` (via `CreateAddon` `handlers/eks/addons.go:84`) | `domains/eks/addon/owner.go:48`; `Get` then unconditional `put` `store.go:39,51` | OVERWRITE (the Get/Put pair is not `kv.Create`, so a record created between them is replaced wholesale) | OVERWRITE via `StagingInstaller.Install` `delivery.go:39` → `putManifest` `store.go:153` (built from the four v1 fields) |
|
||||
| `Owner.EnsureGPUDevicePlugin` (via `nodegroup.go:411`) | `owner.go:79` | IGNORE when a record exists (`ErrExists` swallowed); else as Create | as Create |
|
||||
| `Owner.Update` (via `UpdateAddon` `addons.go:181`) | `owner.go:94` → `casUpdate` `store.go:198` (Unmarshal `:209`, Marshal `:215`, `kv.Update` `:219`) | DROP | OVERWRITE via Install `owner.go:113` → `putManifest` |
|
||||
| `Owner.ApplyReport` (via reconciler `handlers/eks/addon_status_reconciler.go:23`) | `owner.go:140` → `casUpdate` | DROP when status or health changes; IGNORE when the report changes nothing (mutate returns false, no write) | untouched |
|
||||
| `Owner.markFailed` (staging failure in Create/Update) | `owner.go:171` → `casUpdate` | DROP | untouched |
|
||||
| `Owner.Delete` (via `DeleteAddon` `addons.go:211`) | `owner.go:122` → Uninstall `delivery.go:51`, `deleteRecord` `store.go:181,189` | ERASE | ERASE (`deleteManifest` `store.go:172`) |
|
||||
| `DeleteClusterPrefix` (DeleteCluster `service_impl.go:1153`, DELETING reaper `eks_deleting_reaper.go:141`, both via `purgeClusterInfra` `service_impl.go:1451`) | `handlers/eks/cluster_state.go:493` | ERASE (every key under `clusters/{c}/`) | ERASE |
|
||||
| `purgeClusterInfra(deleteMeta=false)` (`service_impl.go:712,1251`) | `service_impl.go:1450` | not touched | not touched |
|
||||
| Account teardown | `spinifex/accountteardown/reapers_eks.go:103` | ERASE, only indirectly through `DeleteCluster`; no add-on reaper and no EKS bucket delete | ERASE (same path) |
|
||||
| `TagResource` / `UntagResource` on an add-on ARN | `service_impl.go:1772,1812,1855` | not a writer: add-on ARNs return `NotImplemented`; create-time tags live only in `Record.Tags` | n/a |
|
||||
|
||||
## Record and manifest readers
|
||||
|
||||
| Reader | file:line | Verdict |
|
||||
|---|---|---|
|
||||
| `addon.Get` (DescribeAddon `addons.go:152`, Create existence check, Delete) | `store.go:58,70` | IGNORE; FAIL on a type change |
|
||||
| `addon.List` (ListAddons `addons.go:26`) | `store.go:78,108` | IGNORE; one undecodable record fails the whole listing |
|
||||
| `addon.ListManifests` (`ListStagedAddonManifests` `addons.go:125`) | `store.go:119,141` | IGNORE; one undecodable manifest fails delivery for every add-on in the cluster |
|
||||
| `addonRecordToAWS` | `addons.go:222` | IGNORE (projects v1 fields onto the SDK shape) |
|
||||
| `ListStagedAddonManifests` reply | `addons.go:129-137` | Projects the four v1 fields field-by-field, so unknown manifest fields never leave the daemon even if `addon.Manifest` gained them |
|
||||
|
||||
## contracts/eks/v1 consumers
|
||||
|
||||
| Consumer | file:line | Mechanism | Added field in a newer payload |
|
||||
|---|---|---|---|
|
||||
| Gateway NATS client for `eks.ListStagedAddonManifests` | `handlers/eks/service_nats.go:128`; `foundation/messaging/nats/nats.go:316` | `encoding/json` into `ListStagedAddonManifestsOutput` | DROP on relay: an older gateway re-renders only the fields its `StagedAddonManifest` declares |
|
||||
| Gateway `ListInternalAddons` response | `gateway/eks/internal_addons.go:20,32`; route `gateway/eks.go:55`; `gateway/eks/handler.go:21-22` | `jsonutil.BuildJSON` (Go field names, json tags ignored) | Emits whatever fields its build declares; nothing to ignore |
|
||||
| `eks-gateway-fetch -resource addons` | `agents/eks/gatewayfetch/fetch.go:115-123` | `encoding/json` (case-insensitive) then fixed four-column printf | IGNORE: extra top-level or per-add-on fields never reach the TSV |
|
||||
| `mulga-eks-addon-sync.sh` sync loop | `scripts/images/eks-node/mulga-eks-addon-sync.sh:357` | `IFS=tab read -r _addon _version _role _cfg_b64` | A fifth TSV column would land inside `_cfg_b64` (last variable takes the rest), which is unused today (`:359`); inserting a column before column 4 would shift positions. Fetch binary and script ship in the same image, so they move together. |
|
||||
| `mulga-eks-addon-sync.sh` report | `mulga-eks-addon-sync.sh:67-72` | printf of five fixed keys | N/A for added fields: an older guest never emits them |
|
||||
| Gateway `PublishInternal` relay | `gateway/eks/internal_publish.go:30,82-85` | `Payload json.RawMessage` published verbatim | Passes added report fields through untouched |
|
||||
| Reconciler report subscriber | `handlers/eks/cluster_reconciler.go:449-455`; `state_report.go:69-72`; `addon_status_reconciler.go:22-27` | `encoding/json` into `AddonStatusReport`, then only `Addon`, `Phase`, `Message` reach the owner | IGNORE: a report carrying `incarnationId`/`generation` is applied exactly as today, so an older reconciler cannot reject a stale incarnation's report |
|
||||
| Phase / status enums | `owner.go:186-204` | `nextStatus` default branch | An unknown phase changes nothing; an unknown stored status decodes, and a `failed` report moves it to `CREATE_FAILED` |
|
||||
|
||||
## Consequences for an expand phase
|
||||
|
||||
- Every path that writes the record back (Update, ApplyReport with a change, markFailed) DROPs added fields, so while any older binary can serve an add-on write, added record fields are best-effort and can silently revert to absent.
|
||||
- Every Install rebuilds the manifest from the record, so an older binary's Create/Update erases added manifest fields too.
|
||||
- Delete and cluster deletion ERASE regardless of content, so a newer `origin`/`declined` marker would not stop an older binary deleting the record.
|
||||
- Reads and all wire consumers tolerate added fields; changing the JSON type of an existing field instead fails whole listings on older binaries.
|
||||
- The only hard fence is the per-account bucket stamp, which is all-or-nothing for EKS in that account.
|
||||
|
||||
## Characterisation tests pinning the above
|
||||
|
||||
| Test | file |
|
||||
|---|---|
|
||||
| `TestOwnerWrites_DropUnknownRecordAndManifestFields` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestOwnerApplyReport_NoOpKeepsUnknownFields` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestOwnerDelete_ErasesRecordAndManifestWithUnknownFields` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestReads_IgnoreUnknownFieldsWithoutRewriting` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestList_OneUndecodableEntryFailsTheListing` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestAddonStatusReport_IgnoresUnknownFields` | `contracts/eks/v1/v1_compat_test.go` |
|
||||
| `TestInternalAddonsResponse_IgnoresUnknownFields` | `contracts/eks/v1/v1_compat_test.go` |
|
||||
| `TestEmitAddonsTSV_IgnoresUnknownFields` | `spinifex/agents/eks/gatewayfetch/v1_compat_test.go` |
|
||||
| `TestAddonStatusReport_UnknownFieldsAreAppliedAsToday` | `spinifex/handlers/eks/addon_v1_compat_test.go` |
|
||||
| `TestListStagedAddonManifests_ProjectsOnlyV1Fields` | `spinifex/handlers/eks/addon_v1_compat_test.go` |
|
||||
|
||||
Already covered elsewhere: cluster-prefix sweep of add-on keys (`handlers/eks/addon_lifecycle_test.go:306`), hand-written v1 record decode (`addon_lifecycle_test.go:85`), unknown phase is a no-op (`domains/eks/addon/owner_test.go:49`).
|
||||
@@ -1,541 +0,0 @@
|
||||
# EKS managed add-on: v2 guest contract design
|
||||
|
||||
**Status:** Draft for review; contract design only, nothing implemented.
|
||||
|
||||
This is the guest/controller wire contract that `docs/package-boundary/eks-addon-lifecycle-design.md` (Section 9, slice 6) calls `contracts/eks/v2`.
|
||||
It applies that design's review decisions R1 to R16 and does not reopen them.
|
||||
It defines payloads, routes, subjects, authorization binding, capability, executor identity and compatibility; the owner transitions, deadlines, record migration and agent implementation stay in the lifecycle design.
|
||||
Present-state claims cite code read on this branch; `docs/package-boundary/eks-addon-lifecycle.md` stays authoritative for what the code does today.
|
||||
Paths are relative to `spinifex/` unless they start with `scripts/`, `contracts/` or `tests/`, which are relative to the spinifex repo root.
|
||||
|
||||
## Standing
|
||||
|
||||
Q-75 (contract evolution) and Q-76 (asynchronous message semantics) in `docs/specs/requirements/resource-lifecycle.md` (mulga repo) are Decided requirements with human provenance and normative effect (R1); they are not ADRs.
|
||||
Q-146 and C-82 (`docs/specs/requirements/managed-services.md`) are open, and this document closes neither.
|
||||
ADR-0006 (`docs/adr/0006-contract-guest-and-persisted-state-evolution.md`, mulga repo) is accepted and gives Q-75 and Q-76 their normative shape: S2 for the N/N-1 window on a coordinated Mulga Regional release and the explicit compatibility matrix on an independently released component, S3 for closed majors and the ban on timing heuristics as capability proof, S4 for expand/migrate/contract on persisted formats, S5 for prepare/activate/migrate/finalise on release activation, and S6 for the guest legacy-mode lifecycle.
|
||||
This design applies ADR-0006 and cites its clauses as decisions wherever it relies on them.
|
||||
Where ADR-0006 leaves a mechanism to implementation (the release-membership record, the activation gate), it states the dependency in Section 3.
|
||||
Capability is demonstrated end-to-end only by an accepted, fenced v2 report; a `contractVersion` field, an image tag or a successful fetch alone never proves it (Section 2.3; V8).
|
||||
|
||||
## 1. Purpose and non-goals
|
||||
|
||||
Purpose: give the add-on owner reports it can fence by executor, incarnation and generation, and observed proof of removal, without changing a byte of v1.
|
||||
|
||||
| Non-goal | Consequence |
|
||||
|---|---|
|
||||
| NATS or JetStream redesign | Reports stay on core NATS as notifications (Q-76); correctness rests on durable desired state, level-triggered re-sending and owner deadlines (lifecycle design Section 12). |
|
||||
| Changing v1 | `contracts/eks/v1` route, subject, both JSON forms and Go field names stay byte-identical and readable, pinned by `contracts/eks/v1/addon_status_test.go`, `addon_manifest_test.go` and `v1_compat_test.go`. |
|
||||
| Timing heuristics | No settle window, poll count, retry budget, credential expiry or deadline is used as a fence, capability proof or completion condition (R2, Q-75, ADR-0006 S3: negotiation is never by timing). |
|
||||
| Universal framework | No generic versioning, capability-negotiation or guest-protocol library; every type here is EKS add-on specific (ADR-0003 S2 rules out a shared one without two consumers). |
|
||||
| Lifecycle implementation | Owner state machine, deadlines, record migration, token idempotency and agent code are the lifecycle design's slices, not this contract. |
|
||||
| Per-incarnation ARN | The external ARN stays name-only (R4); `incarnationId` is internal. |
|
||||
|
||||
## 2. Version and capability negotiation
|
||||
|
||||
### 2.1 Versions
|
||||
|
||||
| Item | Rule |
|
||||
|---|---|
|
||||
| Major | Every v2 body carries `"contractVersion": 2`. A receiver rejects any other value with reason `UnknownContractMajor` and changes no state (Q-75, ADR-0006 S3). |
|
||||
| Separation | v2 has its own routes and subject (Section 7); no receiver sniffs payloads to tell v1 from v2. |
|
||||
| Extensibility | Declared per body below: added optional fields may be ignored; an unknown value of a closed enum (`mode`, `phase`, `reason`, object `state`) is rejected for that entry (Q-75 "reject unknown required semantics or enum values"; ADR-0006 S3). |
|
||||
| Producer rule | The host never emits a v2 directive to a caller that has not called the v2 route as the authenticated active executor; v1 callers get v1 only (Q-75 "oldest active compatible representation"; ADR-0006 S3 "the producer emits the oldest active compatible representation unless an authenticated capability exchange proves support for the newer one"). |
|
||||
|
||||
### 2.2 What N and N-1 mean here
|
||||
|
||||
| Party | N-1 | N |
|
||||
|---|---|---|
|
||||
| Host release (daemon, awsgw, add-on owner, cluster owner) | Today's release: v1 route and subject only. | Serves v1 unchanged plus the v2 routes and subject, the executor record and the v1 compatibility gate. |
|
||||
| Guest image (control-plane VM agent) | v1 agent: `mulga-eks-addon-sync.sh` with `eks-gateway-fetch`/`eks-gateway-publish`. | v2 agent shipped in a new EKS node image. |
|
||||
|
||||
The host release groups daemon, awsgw, the add-on owner and the cluster owner as one coordinated Mulga Regional release, not as independently released components (ADR-0006 S2): they ship from the same Spinifex build, and `cluster-deploy.yml`'s rolling restart runs the fleet mid-roll on mixed N and N-1 builds (release-and-version-inventory.md Section 2), which is exactly the N/N-1 coexistence S2 describes for a coordinated release.
|
||||
ADR-0006 S2's own worked example, that an AWSGW deployment or restart must not force a compatible Predastore deployment to restart, is about awsgw's cross-submodule relationship to Predastore, an independently released product with its own compatibility declaration; this contract does not touch Predastore, so that example does not reclassify awsgw here.
|
||||
The host window is N and N-1 in both rollout orders.
|
||||
The guest image is a long-lived consumer outside the Regional upgrade (a cluster keeps its image until its control-plane VM is replaced), so the v1 guest mode publishes its own support and retirement lifecycle as Q-75 and ADR-0006 S6 require; N+1 does not drop v1 merely because one release passed.
|
||||
New EKS images are picked newest-AMI-wins (`handlers/eks/k3s_server_vm.go:353-363`, release inventory Section 5), so an N-1 host can launch a v2 image; Section 8 covers that row.
|
||||
|
||||
### 2.3 How capability is established
|
||||
|
||||
Capability is never read from a guest payload, image tag, user-data value or AMI name.
|
||||
|
||||
1. The caller authenticates with the control-plane instance role session that IMDS mints with `RoleSessionName` = instance ID (`handlers/sts/assume_role.go:136-148`); the role trusts only `ec2.amazonaws.com` (`handlers/iam/system_role.go:14`), and the gateway takes the role name from the underlying role ARN, never the session name (`gateway/eks.go:307-320`).
|
||||
2. The gateway runs the existing class and membership checks (`gateway/eks/internal_authz.go:44-96`), then the executor check (Section 4.3).
|
||||
3. The gateway relays the fetch to the daemon with the derived executor identity attached and lists v2 directives for it; a fetch alone marks no capability (V8).
|
||||
4. The gateway relays the first v2 report that passes checks 1 to 6 of Section 4.3; the cluster owner re-checks the epoch against the executor record and, in the same CAS that accepts that report, marks the epoch `addonContract: 2` if it is not already marked.
|
||||
5. The add-on owner reads capability only through the narrow interface in Section 4.2.
|
||||
|
||||
A new epoch starts unmarked, so the compatibility gate re-closes on every executor change until the new executor's first valid, fenced v2 report is accepted (lifecycle design Section 9; V8).
|
||||
A fetch, or a directive the guest never acts on, may be retained as diagnostic evidence only; it is never sufficient on its own to prove capability (V8).
|
||||
|
||||
## 3. Prerequisites: what must exist before v2 activates
|
||||
|
||||
The release inventory found no release-membership record and no activation gate (`docs/package-boundary/release-and-version-inventory.md` Section 7).
|
||||
v2 can ship in the Prepare stage without them, but the v2 routes answer `NotActivated` and the gate stays open (old semantics for everyone) until each item below exists.
|
||||
|
||||
| # | Prerequisite | Gap it closes | Why v2 needs it |
|
||||
|---|---|---|---|
|
||||
| P1 | An authoritative, cluster-visible record of the release each daemon and awsgw process runs | G1, G10 | Activation must prove no N-1 writer remains: N-1 `casUpdate` writers DROP added record fields and N-1 `Owner.Delete` ERASEs without observed removal (`eks-addon-v1-compatibility.md` "Record and manifest writers"). awsgw processes count, because an N-1 gateway serves v1 membership-only authorization and has no v2 route. |
|
||||
| P2 | An operator-visible activation record for the add-on lifecycle change, written only from P1 evidence | G2 | The v2 routes and the v1 compatibility gate key on it. It is EKS-add-on-specific for now (V9); the release-membership mechanism (P1) should be reusable by another resource, but this design does not build a platform-wide activation framework, unless a platform gate supersedes it first (lifecycle design open question 4). |
|
||||
| P3 | Record expand complete: `schemaVersion`, `incarnationId`, `desired`, `observed` written beside legacy fields by an idempotent, resumable pass | G7, G8 | Directives carry `incarnationId` and `generation`; a record without them cannot be listed in v2 (Section 5.4). |
|
||||
| P4 | The cluster-owned executor record and the bound v2 report route (Sections 4 and 7) | executor inventory gaps 2 to 4 | Without it no report can be bound to a caller, cluster or executor. |
|
||||
| P5 | A documented controlled-rollback procedure for the activated state | G4 | An N-1 binary that returns after activation must not complete or reinterpret v2 obligations (lifecycle design open question 9). |
|
||||
|
||||
The per-account bucket stamp is not the fence: bumping `KVBucketEKSAccountVersion` (`handlers/eks/store.go:29`) makes every N-1 binary refuse the whole `eks-account-{id}` bucket with `SchemaAheadError` (`foundation/state/migrate/kv.go:55-67`), taking down every EKS action in the account, not just add-ons.
|
||||
The per-record `schemaVersion` plus P2 replaces it.
|
||||
|
||||
## 4. Executor identity and epoch
|
||||
|
||||
### 4.1 Shape and owner
|
||||
|
||||
The cluster owner owns a new key `clusters/{cluster}/delivery-executor` in `eks-account-{account}`, beside `meta` rather than inside it.
|
||||
A separate key keeps the epoch off the meta revision, which churns on every health write (`handlers/eks/cluster_state.go:416-436`), and away from N-1 `casUpdateMeta` writers, which would DROP an added meta field.
|
||||
N-1 binaries ignore the key: cluster listing selects only keys ending `/meta` (`handlers/eks/service_impl.go:2422`), and `DeleteClusterPrefix` erases it with the cluster, which is correct.
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"instanceId": "i-0a1b2c3d4e5f60718",
|
||||
"epoch": 5,
|
||||
"assignedAt": "2026-10-09T10:00:00Z",
|
||||
"addonContract": 2,
|
||||
"addonContractProvenAt": "2026-10-09T10:00:31Z",
|
||||
"fence": {
|
||||
"previousInstanceId": "i-09f8e7d6c5b4a3921",
|
||||
"previousEpoch": 4,
|
||||
"evidence": "instance-terminated",
|
||||
"observedAt": "2026-10-09T09:59:58Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Rule |
|
||||
|---|---|
|
||||
| `instanceId` | The internal EC2 instance ID of a current control-plane member; empty means no executor is assigned (an unavailable guest). |
|
||||
| `epoch` | Unsigned, starts at 1, advanced by exactly one in the CAS that changes `instanceId`; never decreases while the key exists. |
|
||||
| `addonContract` | `0` (unproven) or `2`; reset to `0` in the CAS that advances `epoch`; set to `2` only in the CAS that accepts the first valid, fenced v2 report for the epoch, never on a fetch (Section 2.3; V8). |
|
||||
| `addonContractProvenAt` | RFC 3339 UTC; set in the same CAS as `addonContract: 2`; diagnostic only. |
|
||||
| `fence` | The evidence that let the epoch advance (Section 4.4); absent for epoch 1. |
|
||||
|
||||
### 4.2 Interface the add-on side declares
|
||||
|
||||
The add-on package declares, and the cluster owner implements, one read:
|
||||
|
||||
```go
|
||||
// In domains/eks/addon.
|
||||
type DeliveryExecutor struct {
|
||||
InstanceID string
|
||||
Epoch uint64
|
||||
AddonContract int // 0 or 2
|
||||
}
|
||||
|
||||
type DeliveryExecutors interface {
|
||||
ActiveDeliveryExecutor(ctx context.Context, account, cluster string) (DeliveryExecutor, error)
|
||||
}
|
||||
```
|
||||
|
||||
The implementation carries the compile-time check (`var _ addon.DeliveryExecutors = (*Impl)(nil)`).
|
||||
The add-on package does not read `ClusterMeta`, the executor key or the gateway caller (R6).
|
||||
Assignment, reassignment and capability marking are cluster-owner methods the add-on side never calls.
|
||||
|
||||
### 4.3 Gateway derivation and rejection rule
|
||||
|
||||
The executor identity is `Caller.SessionName` after `requireCPAgent` passes (`gateway/eks/internal_authz.go:85-96`); nothing in the path, query or body names it.
|
||||
Static-credential fallback VMs have no session name and can never be executors (`handlers/eks/k3s_server_role.go:30-50`).
|
||||
|
||||
| Order | Check | v2 reason | HTTP |
|
||||
|---|---|---|---|
|
||||
| 1 | SigV4, IAM policy, CP-agent class | existing gateway envelope (`AccessDenied`); these run before the v2 handler | 403 |
|
||||
| 2 | Activation record present (P2) | `NotActivated` if the record is absent; `ActivationStateUnavailable` if it is unreadable, invalid or unavailable | 503 |
|
||||
| 3 | Caller is a member of the named cluster in the named account (`IsControlPlaneMember`, `internal_authz.go:74`) | `NotClusterMember` | 403 |
|
||||
| 4 | Executor record names an instance | `NoActiveExecutor` | 503 |
|
||||
| 5 | `Caller.SessionName == executor.instanceId` | `NotActiveExecutor` | 403 |
|
||||
| 6 | Report only: body `executorEpoch == executor.epoch` | `StaleExecutorEpoch` | 409 |
|
||||
| 7 | Body decodes against v2 with `contractVersion == 2` | `UnknownContractMajor` or `MalformedReport` | 400 |
|
||||
|
||||
The daemon re-runs checks 4 to 6 against the record it reads, and the owner re-checks the epoch inside its CAS loop (lifecycle design Section 2 rule 1), because the epoch can advance between the gateway check and the write.
|
||||
That same CAS marks `addonContract: 2` if unmarked, only for the first report it accepts under the current epoch (Section 4.1; V8).
|
||||
|
||||
### 4.4 Fencing prerequisites and evidence options
|
||||
|
||||
The epoch may advance only after the previous executor provably cannot render, apply, or have reports accepted (R6).
|
||||
Report acceptance is covered by checks 5 and 6 once the epoch advances; render and apply need evidence about the old VM.
|
||||
The options below come from the executor inventory (`docs/package-boundary/eks-addon-executor-inventory.md` Sections 8 and 9); the reviewer chooses.
|
||||
|
||||
| Option | Evidence it requires | Exists today |
|
||||
|---|---|---|
|
||||
| A. Instance terminated or not found | EC2 instance record `terminated` or absent, proven authoritative under host partition; reassignment never triggered by a describe error (`cluster_reconciler.go:982-1007`) | Query exists; authority under host loss unverified; describe-error trigger must be excluded |
|
||||
| B. Instance stopped and pinned stopped | `stopped` plus a durable "do not restart" marker the reconciler's restart ladder honours (`cluster_reconciler.go:708-771`) | No |
|
||||
| C. Disable-on-handover directive | An online host-to-guest directive the agent polls, an acknowledgement, and a guest rule that a denied fetch stops rendering; an acknowledgement proves only that the old VM was alive to send it | No (only the boot-time recovery directive) |
|
||||
| D. Credential-based | Per-session revocation, IMDS refusing to re-mint for the fenced instance, gateway rejecting revoked sessions; still leaves files already rendered on the old disk | No |
|
||||
| E. Guest leader election (kube Lease) | Host-visible proof of the guest leader, verified K3s auto-deploy behaviour across servers; reports still need checks 5 and 6 | No |
|
||||
| F. Operator assertion | An explicit, audited operator action naming the old instance, recorded as `fence.evidence: "operator-asserted"` | No |
|
||||
|
||||
**Approved (V10):** A is the only automatic evidence: an authoritative record that the former instance terminated or no longer exists.
|
||||
A lookup error, a network issue or a partition is never evidence of termination, and reassignment is never triggered by a describe error.
|
||||
F, an explicit, audited operator assertion, is the alternative when A cannot be obtained.
|
||||
The guest stand-by rule of Section 6.4 is defence in depth and is never counted as fence evidence.
|
||||
B to E each need new machinery and either prove less than A or prove only liveness.
|
||||
Under A, the restore path's successor-first ordering (`restore_snapshot.go:262-336`) and the unreachable etcd prune (executor inventory gap 10) must be resolved before the restore path may advance an epoch.
|
||||
|
||||
## 5. Directive (fetch) payload v2
|
||||
|
||||
### 5.1 Body
|
||||
|
||||
Response to `GET /clusters/{clusterName}/internal-addons-v2/{accountId}` (Section 7), `Content-Type: application/json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"contractVersion": 2,
|
||||
"cluster": "prod",
|
||||
"executor": {"instanceId": "i-0a1b2c3d4e5f60718", "epoch": 5},
|
||||
"issuedAt": "2026-10-09T10:01:00Z",
|
||||
"directives": [
|
||||
{
|
||||
"addonName": "argocd",
|
||||
"incarnationId": "0b7e4f7a-2d55-4f0e-9a51-6f1fd1c0e2a4",
|
||||
"generation": 4,
|
||||
"mode": "remove",
|
||||
"spec": {"addonVersion": "3.0.23"},
|
||||
"deletion": {"requestedAt": "2026-10-09T09:58:00Z", "preserve": false},
|
||||
"deadline": "2026-10-09T10:28:00Z"
|
||||
},
|
||||
{
|
||||
"addonName": "aws-load-balancer-controller",
|
||||
"incarnationId": "6f1c2a4e-8b0d-4c3e-a1f2-93b7d4e5c6a8",
|
||||
"generation": 3,
|
||||
"mode": "install",
|
||||
"spec": {
|
||||
"addonVersion": "2.11.0",
|
||||
"serviceAccountRoleArn": "arn:aws:iam::123456789012:role/lbc-irsa"
|
||||
},
|
||||
"deadline": "2026-10-09T10:20:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Rule |
|
||||
|---|---|---|
|
||||
| `contractVersion` | int | Always 2. |
|
||||
| `cluster` | string | The path cluster, echoed for logs. |
|
||||
| `executor.instanceId`, `executor.epoch` | string, uint64 | From the executor record, never from the request; the guest echoes `epoch` in every report. |
|
||||
| `issuedAt` | RFC 3339 UTC | Diagnostic only. |
|
||||
| `directives` | array | One entry per add-on record of the cluster, sorted by `addonName`; `[]`, never `null`, when empty. |
|
||||
| `addonName`, `incarnationId`, `generation` | string, UUID string, uint64 | Identity and the desired generation this entry represents. |
|
||||
| `mode` | enum `install`, `remove`, `relinquish` | Closed set. |
|
||||
| `spec.addonVersion` | string | Catalogue version; for `remove` and `relinquish`, the last installed spec, so a guest without local state can still render the object list. |
|
||||
| `spec.serviceAccountRoleArn` | string, omitted when unset | An IAM role ARN; see 5.3. |
|
||||
| `spec.configurationValues` | string, omitted | Never emitted until a bundle publishes a schema and a renderer (R9); a guest that receives it for a bundle without a renderer reports `failed`/`UnsupportedConfiguration`. |
|
||||
| `deletion` | object, omitted unless `mode` is `remove` or `relinquish` | `requestedAt` and `preserve`, the obligation being discharged. |
|
||||
| `deadline` | RFC 3339 UTC | The persisted operation deadline (R12), informational; the guest must not change what it reports, stop reporting or infer completion from it. The owner alone enforces it. |
|
||||
|
||||
Extensibility: the guest ignores unknown optional fields at any level; it rejects an unknown `mode` for that entry by reporting `failed`/`InvalidDirective`.
|
||||
|
||||
### 5.2 Rendering and the v1 quirk
|
||||
|
||||
v1's body is written by `WriteJSONResponse`, which calls the SDK REST-JSON marshaller `jsonutil.BuildJSON` (`gateway/eks/handler.go:19-22`); it ignores `json` tags, so the deployed v1 body uses Go field names (`{"Addons":[{"AddonName",...}]}`) and the guest matches it only through `encoding/json`'s case-insensitive decoding (`contracts/eks/v1/README.md`).
|
||||
v2 avoids this by construction:
|
||||
|
||||
- The v2 route has its own writer that calls `encoding/json` (`json.Marshal`, default HTML escaping) and sets `application/json`; it never goes through `WriteJSONResponse`.
|
||||
- v2 types carry only `json` tags, no `locationName`, and the literal-byte tests (Section 9) pin camelCase keys, so a regression to `BuildJSON` fails a test.
|
||||
- Optional fields use `omitempty`; arrays that must be present (`directives`, `reports`, `objects`) are initialised non-nil.
|
||||
- The guest decodes v2 strictly by key case (a Go decoder that rejects a case-mismatched key, for example by comparing a re-marshal), so the two forms can never be confused again.
|
||||
|
||||
### 5.3 Validation the host applies before listing
|
||||
|
||||
| Value | Rule | On violation |
|
||||
|---|---|---|
|
||||
| `serviceAccountRoleArn` | `arn:{partition}:iam::{12 digits}:role/{path}{name}` with IAM's role-name character set; already `iam:PassRole`-checked at the adapter (R10) and rejected for bundles with no IRSA marker (R14). | The AWS adapter returns `InvalidParameterException`, so it never reaches a record; a stored legacy value that fails is listed without the field and the owner records a health issue. |
|
||||
| `addonVersion` | In the catalogue. | Same. |
|
||||
| `configurationValues` | Never listed (R9). | Stored legacy values stay flagged on the record (lifecycle design Section 3). |
|
||||
|
||||
The role-ARN format check at the adapter boundary is a standalone security correction, not v2 lifecycle work: it is being fixed directly on the existing v1 path and applies to v1 and v2 alike, independent of this design.
|
||||
v2 adds only the guest-side re-validation below; it is not what closes the adapter-side gap.
|
||||
The guest re-validates the role ARN with the same pattern before rendering, because the v1 renderer substitutes it raw into `sed` and YAML (`eks-addon-bundle-inventory.md` X9); a failure is `failed`/`InvalidDirective`.
|
||||
|
||||
### 5.4 Records that cannot be listed
|
||||
|
||||
A record without `incarnationId` or `desired.generation` (expand not yet run, or rewritten by an N-1 writer before activation) is omitted from the v2 listing and counted in a metric; the add-on reconcile pass re-expands it.
|
||||
After activation no N-1 writer exists (P1), so the omission is transient.
|
||||
|
||||
## 6. Report payload v2
|
||||
|
||||
### 6.1 Guest body
|
||||
|
||||
`POST /clusters/{clusterName}/internal-addons-v2/{accountId}/reports`, one batch per tick:
|
||||
|
||||
```json
|
||||
{
|
||||
"contractVersion": 2,
|
||||
"executorEpoch": 5,
|
||||
"reports": [
|
||||
{
|
||||
"addonName": "aws-load-balancer-controller",
|
||||
"incarnationId": "6f1c2a4e-8b0d-4c3e-a1f2-93b7d4e5c6a8",
|
||||
"generation": 3,
|
||||
"phase": "degraded",
|
||||
"reason": "InsufficientReplicas",
|
||||
"message": "deployment kube-system/aws-load-balancer-controller available 0/1",
|
||||
"observedAt": "2026-10-09T10:05:00Z",
|
||||
"objects": [
|
||||
{"group": "apps", "kind": "Deployment", "namespace": "kube-system", "name": "aws-load-balancer-controller", "state": "present", "detail": "available=0/1"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"addonName": "argocd",
|
||||
"incarnationId": "0b7e4f7a-2d55-4f0e-9a51-6f1fd1c0e2a4",
|
||||
"generation": 4,
|
||||
"phase": "removing",
|
||||
"reason": "FinalizerBlocked",
|
||||
"message": "namespace argocd Terminating",
|
||||
"observedAt": "2026-10-09T10:05:00Z",
|
||||
"objects": [
|
||||
{"group": "", "kind": "Namespace", "namespace": "", "name": "argocd", "state": "terminating", "detail": "finalizers=kubernetes"},
|
||||
{"group": "apiextensions.k8s.io", "kind": "CustomResourceDefinition", "namespace": "", "name": "applications.argoproj.io", "state": "absent", "detail": ""}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Rule |
|
||||
|---|---|
|
||||
| `executorEpoch` | The epoch from the last successful directive fetch; one per batch. |
|
||||
| `reports[]` | At most one entry per `addonName`; a duplicate rejects the batch as `MalformedReport`. |
|
||||
| `phase` | Closed enum, Section 6.2. |
|
||||
| `reason` | Closed enum, Section 6.3; required for `failed`, `degraded` and `removing`, optional otherwise. |
|
||||
| `message` | Free text, at most 1024 bytes after truncation by the guest. |
|
||||
| `objects[]` | Per-object observations (Sections 6.6 and 6.7); required for `removing`, `removed` and `ready`, optional otherwise; bounded to the bundle's declared set. |
|
||||
| `ts` | Not present; `observedAt` replaces v1's unused `ts`. |
|
||||
|
||||
The guest builds the body with a JSON encoder (a Go helper in the image, as `eks-gateway-publish` already is), never `printf`.
|
||||
v1's `printf` (`scripts/images/eks-node/mulga-eks-addon-sync.sh:67-72`) does not escape `message`, so a quote or backslash in a message produces invalid JSON; v2 removes that class of defect rather than escaping in shell.
|
||||
|
||||
### 6.2 Phases
|
||||
|
||||
| Phase | Valid in mode | Meaning |
|
||||
|---|---|---|
|
||||
| `applied` | `install` | Rendered for this generation; readiness (Section 6.6) not yet met. Never a failure. |
|
||||
| `ready` | `install` | Readiness definition for this bundle met for this generation. |
|
||||
| `degraded` | `install` | Was ready for this generation and no longer meets the definition. |
|
||||
| `failed` | `install` | A deterministic cause prevents realising this generation (`reason` required). |
|
||||
| `removing` | `remove` | Some object of the removal set still exists. |
|
||||
| `removed` | `remove` | Every object of the removal set observed absent. |
|
||||
| `relinquished` | `relinquish` | The rendered file is gone and the agent no longer manages the add-on. |
|
||||
|
||||
`degraded` is an addition to the phase list in the lifecycle design Section 2: without it the guest must choose between `applied` (which cannot move `ACTIVE` to `DEGRADED`) and `failed` (which is terminal while `CREATING` or `UPDATING`, R3), and it cannot see the host status to choose.
|
||||
`degraded` is approved (V2) as an observed condition for an otherwise `ACTIVE` add-on: the owner maps `ACTIVE` plus `degraded` to `DEGRADED` and records it as a health issue.
|
||||
A `degraded` report during `CREATING` or `UPDATING` records the same health issue but does not by itself turn the operation into a terminal failure; the operation's deadline and an explicit terminal report (`failed`, `ready` or `removed`) decide that (V2).
|
||||
|
||||
### 6.3 Reason codes
|
||||
|
||||
Reasons are internal contract values, not AWS error codes (R7); the owner maps some onto `AddonIssue.code` (pinned SDK `AddonIssueCode*`, `aws-sdk-go` v1.55.8 `service/eks/api.go`) when it builds `health`.
|
||||
|
||||
| Reason | Phases | `AddonIssue.code` | Meaning |
|
||||
|---|---|---|---|
|
||||
| `BundleMissing` | `failed` | `InternalFailure` | No baked bundle for name and version in this image. |
|
||||
| `InvalidDirective` | `failed` | `InternalFailure` | Unknown mode, invalid role ARN, or a field the guest cannot realise. |
|
||||
| `UnsupportedConfiguration` | `failed` | `InternalFailure` | `configurationValues` present for a bundle without a renderer. |
|
||||
| `RenderFailed` | `failed` | `InternalFailure` | Template or webhook certificate generation failed. |
|
||||
| `ApplyRejected` | `failed`, `degraded` | `AdmissionRequestDenied` | The apiserver or an admission webhook rejected an object of the bundle. |
|
||||
| `ObjectConflict` | `failed` | `ConfigurationConflict` | `resolveConflicts` `NONE`/`PRESERVE` and an object of the bundle exists without the add-on's ownership labels. |
|
||||
| `ObjectMissing` | `degraded` | `K8sResourceNotFound` | An object of the readiness set is absent after it was applied. |
|
||||
| `InsufficientReplicas` | `degraded` | `InsufficientNumberOfReplicas` | A workload of the readiness set is below its replica target. |
|
||||
| `WebhookUnavailable` | `degraded` | `AdmissionRequestDenied` | A fail-closed webhook of the bundle has no ready endpoint. |
|
||||
| `CredentialFailure` | `degraded` | `AccessDenied` | Only where a bundle's readiness definition observes it; none does yet. |
|
||||
| `FinalizerBlocked` | `removing` | none | An object of the removal set is `Terminating` behind a finalizer. |
|
||||
| `DeletePending` | `removing` | none | Delete issued; objects still present without a finalizer cause. |
|
||||
|
||||
No reason exists for an unreachable apiserver: the guest sends nothing for that add-on, as the lifecycle design specifies, and silence is handled by deadlines.
|
||||
The gateway-side rejection reasons of Section 4.3 (`NotActivated`, `ActivationStateUnavailable`, `NotClusterMember`, `NoActiveExecutor`, `NotActiveExecutor`, `StaleExecutorEpoch`, `UnknownContractMajor`, `MalformedReport`) are response reasons, never report reasons.
|
||||
|
||||
The reason table above is the closed set for `contractVersion: 2` (V4; Q-75, ADR-0006 S3).
|
||||
A genuinely new wire-level reason has new semantics and therefore requires a new major, never a silent addition to this enum; free-form detail that does not need enum-level routing belongs in `message`, not in an unbounded `reason` value.
|
||||
|
||||
### 6.4 Gateway response and guest stand-by rule
|
||||
|
||||
The v2 routes answer with `application/json` `{"contractVersion":2,"error":{"reason":"...","message":"..."}}`, rendered with `encoding/json`, instead of the AWS error envelope (R7); only check 1 of Section 4.3 keeps the gateway's existing envelope, because it runs before the route handler.
|
||||
A successful report POST returns `{"contractVersion":2,"accepted":N}` where `accepted` counts entries relayed, not entries applied: acceptance by the owner is a separate CAS that the guest never learns of (Q-76: a transport acknowledgement never implies domain completion).
|
||||
|
||||
On `NotActiveExecutor`, `NotClusterMember` or `StaleExecutorEpoch`, the v2 agent stops delivery: it removes its rendered `spinifex-addon-*.yaml` files without `kubectl delete`, sends no reports and re-fetches on its next tick.
|
||||
This complements, but never substitutes for, the fence evidence of Section 4.4.
|
||||
|
||||
### 6.5 Host-internal envelope and subject
|
||||
|
||||
The gateway does not relay the guest bytes verbatim, unlike v1's `PublishInternal` (`gateway/eks/internal_publish.go:47-91`).
|
||||
It decodes the batch strictly, then publishes one host-internal envelope rendered with `encoding/json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"contractVersion": 2,
|
||||
"account": "123456789012",
|
||||
"cluster": "prod",
|
||||
"executor": {"instanceId": "i-0a1b2c3d4e5f60718", "epoch": 5},
|
||||
"receivedAt": "2026-10-09T10:05:01Z",
|
||||
"reports": [ ... the decoded entries, re-encoded ... ]
|
||||
}
|
||||
```
|
||||
|
||||
`executor.instanceId` comes from the authenticated caller and `executor.epoch` is the value check 6 verified; no guest field reaches `executor`.
|
||||
Subject: `eks.addon.v2.{account}.{cluster}.status`, core NATS, no queue group, subscribed by the add-on reconcile pass under the cluster lease (lifecycle design Section 11).
|
||||
The account and cluster tokens come from the route captures after check 3, so they are bound to a cluster the caller serves; v1's subject is built from the body `accountId` (`internal_publish.go:63-76`).
|
||||
|
||||
### 6.6 Readiness evidence per bundle
|
||||
|
||||
v1 reports `ready` when K3s records the `addon.k3s.cattle.io/gvks` annotation, which proves only that K3s applied some GVKs, says nothing about workloads, and may reflect the previous generation (`eks-addon-bundle-inventory.md` Section 3.3, X1 to X3).
|
||||
v2 drops the annotation as a readiness source entirely and defines ready per bundle.
|
||||
|
||||
Generation binding: the v2 renderer stamps every rendered object with annotations `eks.spinifex.io/addon-incarnation` and `eks.spinifex.io/addon-generation` and labels `app.kubernetes.io/managed-by: spinifex-addon` and `eks.spinifex.io/addon-name`.
|
||||
An object counts toward readiness only if it is present and carries the current incarnation and generation, so a report can never describe a previous apply; this also gives a stable ownership selector (X6).
|
||||
|
||||
Common rule: every object of the bundle's readiness set is read individually (no `--ignore-not-found` aggregation); a missing one is `ObjectMissing`.
|
||||
|
||||
| Bundle | Ready when, for the current generation |
|
||||
|---|---|
|
||||
| `spinifex-noop` | Namespace `spinifex-noop` and ConfigMap `spinifex-noop/spinifex-noop` present. |
|
||||
| `argocd` | Namespace `argocd` present; CRDs `applications`, `applicationsets`, `appprojects.argoproj.io` `Established`; all 6 Deployments `observedGeneration >= generation`, `updatedReplicas == availableReplicas == spec.replicas`; StatefulSet `argocd-application-controller` `updatedReplicas == readyReplicas == spec.replicas`. |
|
||||
| `aws-ebs-csi-driver` | All rendered objects present; Deployment `ebs-csi-controller` available equals `spec.replicas` (2); DaemonSet `ebs-csi-node` `numberReady == updatedNumberScheduled == desiredNumberScheduled`, where `desiredNumberScheduled == 0` is ready with a `message` saying no eligible node. |
|
||||
| `aws-load-balancer-controller` | CRDs `ingressclassparams` and `targetgroupbindings.elbv2.k8s.aws` `Established`; Deployment available equals `spec.replicas`; Service `aws-load-balancer-webhook-service` has at least one ready endpoint (EndpointSlice), because the bundle's six webhooks are `failurePolicy: Fail`; both webhook configurations present. |
|
||||
| `nvidia-device-plugin` | DaemonSet `nvidia-device-plugin-daemonset` `numberReady == updatedNumberScheduled == desiredNumberScheduled`; `desiredNumberScheduled == 0` is ready with a no-GPU-node message. |
|
||||
|
||||
Not part of any ready definition: IRSA credential success, GPU allocatable exposure and data-path checks; open question 2.
|
||||
|
||||
### 6.7 Removal evidence per bundle
|
||||
|
||||
`removed` means every object of the bundle's removal set is observed NotFound by an individual `get` for each object (namespaced and cluster-scoped), not by listing a namespace.
|
||||
The removal set is scoped to add-on-owned Kubernetes objects only (V6): it is the rendered object list of the last installed spec, minus objects shared with another add-on that is still listed (below), and it never includes downstream workloads, PVs or other unrelated resources outside that set.
|
||||
`removing` reports carry `objects[]` with `present`, `terminating` or `absent` for every member of the set, so `DescribeAddon` health can show what blocks.
|
||||
|
||||
| Bundle | Removal set (all NotFound) | Blockers reported as `FinalizerBlocked` | Residue outside the set |
|
||||
|---|---|---|---|
|
||||
| `spinifex-noop` | Namespace, ConfigMap | Namespace `kubernetes` finalizer (aggregated API availability) | none |
|
||||
| `argocd` | Namespace `argocd`, 3 CRDs, 3 ClusterRoles, 3 ClusterRoleBindings, and the namespaced objects | `Application` `resources-finalizer.argocd.argoproj.io`; CRD and namespace `Terminating` | user workloads Argo CD deployed |
|
||||
| `aws-ebs-csi-driver` | 20 rendered objects, `kube-system` or cluster-scoped | PV and VolumeAttachment CSI finalizers | PVs, VolumeAttachments, Spinifex EBS volumes |
|
||||
| `aws-load-balancer-controller` | 2 CRDs plus 13 rendered objects, webhook configurations included | Ingress, Service and TargetGroupBinding controller finalizers; CRD `Terminating` | Spinifex ELBv2 load balancers, target groups, security groups |
|
||||
| `nvidia-device-plugin` | 1 DaemonSet | none | host CDI files |
|
||||
|
||||
Residue is reported in the final `removed` report's `objects[]` with `state: "residue"` where the guest can see it (PVs with `spec.csi.driver == ebs.csi.aws.com`); out-of-cluster residue is invisible to the guest.
|
||||
`removed` does not wait for in-cluster residue outside the removal set (V6): downstream workloads, unreleased PVs and other dependents are reported as residue, not waited on, and the record proceeds once the removal set itself is gone.
|
||||
Preserve-mode deletion (`deletion.preserve`) explicitly relinquishes the add-on's ownership of its objects rather than claiming their removal (V6); that obligation is discharged by the `relinquished` phase (Section 6.2), not by `removed`.
|
||||
|
||||
Shared `kube-system/spinifex-gateway-ca` Secret (bundle inventory X5): v1 renders it under one shared name for both IRSA bundles, so either removal deletes the other's dependency and a NotFound check on it never completes while the other re-creates it.
|
||||
v2 renders a per-bundle name (`spinifex-gateway-ca-<addon>`) for every new install, so the overlap does not recur (V7).
|
||||
Compatibility path for a cluster migrating from v1 (V7): while a bundle's rendered objects still carry the old shared name, the shared-object rule applies unchanged, so an object rendered by more than one listed add-on is excluded from every removal set while more than one `install` directive renders it, and the guest deletes it only with the last one.
|
||||
A bundle moves to its per-bundle name only on its next v2 install or update, after which the shared-object rule no longer applies to it.
|
||||
|
||||
## 7. Routes, subjects and authorization binding
|
||||
|
||||
| Direction | v1 (unchanged) | v2 |
|
||||
|---|---|---|
|
||||
| Fetch route | `GET /clusters/{clusterName}/internal-addons/{accountId}`, action `ListInternalAddons` (`gateway/eks.go:55-57`) | `GET /clusters/{clusterName}/internal-addons-v2/{accountId}`, action `ListInternalAddonDirectives` |
|
||||
| Report route | `POST /clusters/{clusterName}/internal-publish`, channel `addon`, action `PublishInternal` (`gateway/eks.go:39-41`) | `POST /clusters/{clusterName}/internal-addons-v2/{accountId}/reports`, action `ReportInternalAddons` |
|
||||
| Gateway to daemon | NATS `eks.ListStagedAddonManifests` | NATS `eks.ListAddonDirectives`, request carries the gateway-derived executor identity |
|
||||
| Report subject | `eks.addon.{account}.{cluster}.status` | `eks.addon.v2.{account}.{cluster}.status` |
|
||||
| Response renderer | `jsonutil.BuildJSON` | `encoding/json` |
|
||||
| Scope | `ListInternalAddons` is `sourceInternalCluster`; `PublishInternal` is `sourceBodyAccountCluster` (`gateway/eks/authz.go:55,59`) | Both `sourceInternalCluster`, so `IsInternalAction` is true and `AuthorizeInternal` runs (`internal_authz.go:37-39`) |
|
||||
| IAM | Role policy `Resource: "*"` (`handlers/eks/k3s_server_role.go:27`) | The two new actions added to the same inline policy; the binding comes from `AuthorizeInternal` plus Section 4.3, not from the policy resource |
|
||||
|
||||
Binding finding carried forward: today `PublishInternal` is not gated by `AuthorizeInternal`, so with the role's `Resource: "*"` any control-plane session of any cluster can publish add-on reports for any account and cluster, and the relay attaches no caller identity (executor inventory gaps 2 and 3).
|
||||
v2 does not reuse `PublishInternal`; its report route takes the cluster and account from the path, requires membership and the active executor, and attaches identity.
|
||||
Binding v1's `PublishInternal` to the caller's cluster is being fixed directly on the v1 path, as a standalone security correction independent of this design, not as v2 lifecycle work.
|
||||
v2 keeps its own binding regardless (Section 4.3, Section 6.5) and neither depends on, nor is credited for, that correction; post-activation v1 reports stay diagnostic-only either way (lifecycle design Section 9).
|
||||
|
||||
After an executor is assigned for a cluster (P4), the N gateway binds the v1 fetch route to that assigned executor by applying check 5 of Section 4.3 to it, so a v2 agent on a non-executor member cannot fall back to v1 and become a second renderer (V5).
|
||||
v1 payload bytes are unchanged by this; before an executor is assigned, the v1 fetch route keeps its existing behaviour unchanged (V5).
|
||||
|
||||
## 8. Compatibility matrix
|
||||
|
||||
"Pre-activation" means P2 has not been recorded; "post-activation" means it has, which by P1 implies no N-1 host process remains.
|
||||
|
||||
| Guest | Host | Fetch | Reports | Lifecycle mutations | Conformance |
|
||||
|---|---|---|---|---|---|
|
||||
| v1 | N-1 | v1 | v1, applied as today | Today's semantics | None claimed |
|
||||
| v1 | N, pre-activation | v1, byte-identical | v1, applied as today | Today's semantics | None claimed |
|
||||
| v1 | N, post-activation | v1 projection: records in `install` mode only | v1, recorded diagnostically in `observed`, never complete an operation | Gated (below) | None claimed |
|
||||
| v2 | N-1 | v2 answered `InvalidAction` (`gateway/eks.go:196-199`); agent uses legacy v1 mode that tick | v1 | Today's semantics | None claimed |
|
||||
| v2 | N, pre-activation | v2 answered `NotActivated`; legacy v1 mode that tick | v1 | Today's semantics | None claimed |
|
||||
| v2, active executor | N, post-activation | v2 | v2, fenced; the first report accepted for the epoch marks `addonContract: 2` if unmarked (V8) | Ungated once marked | Q-146 claimed for this cluster only |
|
||||
| v2, not executor | N, post-activation | `NotActiveExecutor`; stand-by (Section 6.4); v1 fetch also denied | none | n/a | n/a |
|
||||
|
||||
Legacy-mode fallback is approved only for `InvalidAction` from the fetch route and `NotActivated` (V1).
|
||||
The guest never falls back on an authentication, membership, executor, epoch or other fencing failure, including `NotClusterMember`, `NoActiveExecutor`, `NotActiveExecutor` and `StaleExecutorEpoch`, because doing so would bypass the fence.
|
||||
This is a guest protocol fallback for an N-1 or not-yet-activated host, not a lifecycle fallback, and it never opens the gate; it does not conflict with R5.
|
||||
|
||||
### 8.1 Gated mutations
|
||||
|
||||
Per R5, the lifecycle design Section 9 and ADR-0006 S5 (migration follows activation, never the reverse), on a cluster whose current epoch is not marked `addonContract: 2`:
|
||||
|
||||
| Action | Gated | Reason |
|
||||
|---|---|---|
|
||||
| `CreateAddon` (including GPU auto-staging through the owner) | Yes | `ACTIVE` needs a fenced report for the current incarnation and generation. |
|
||||
| `UpdateAddon` | Yes | Same. |
|
||||
| `DeleteAddon`, with or without `preserve` | Yes | Needs observed removal or relinquish. |
|
||||
| `DescribeAddon`, `ListAddons`, `DescribeUpdate`, `ListUpdates`, `DescribeAddonVersions`, `DescribeAddonConfiguration`, tag actions, cluster-teardown removal | No | No guest realisation required. |
|
||||
|
||||
Response: `InvalidRequestException`, HTTP 400, message (proposed exact text):
|
||||
|
||||
- epoch unmarked: `Add-on lifecycle operations on cluster {cluster} require an upgraded control-plane image. The cluster's add-on delivery agent has not established add-on contract v2. Upgrade the cluster's control-plane image and retry.`
|
||||
- no executor assigned: `Add-on lifecycle operations on cluster {cluster} are unavailable: the cluster has no active add-on delivery agent. Retry after the control plane recovers.`
|
||||
|
||||
Pre-activation, nothing is gated; the gate exists only once P2 is recorded (lifecycle design slice 7).
|
||||
|
||||
### 8.2 What old binaries do with new records
|
||||
|
||||
From `docs/package-boundary/eks-addon-v1-compatibility.md`:
|
||||
|
||||
| N-1 path | Effect on a v2 record | Why it forces ordering |
|
||||
|---|---|---|
|
||||
| `Owner.Update`, `Owner.ApplyReport` with a change, `markFailed` via `casUpdate` (`domains/eks/addon/store.go:198-219`) | DROP: re-encodes v1 `Record` and erases `incarnationId`, `desired`, `observed` | Expand must precede activation, and v2 semantics stay off while any N-1 writer can serve the cluster (P1, P2; ADR-0006 S4 expand/migrate/contract, S5 prepare/activate/migrate/finalise). |
|
||||
| `StagingInstaller.Install` | OVERWRITE the manifest from four v1 fields | The manifest key is a projection only until finalisation (R16). |
|
||||
| `Owner.Delete`, `DeleteClusterPrefix` | ERASE regardless of content | An N-1 delete would complete a v2 delete without observed removal; only activation prevents it. |
|
||||
| Readers (`Get`, `List`, `ListManifests`) | IGNORE added fields; FAIL the whole listing on a type change (`store.go:78,108`) | No existing field may change JSON type. |
|
||||
| Reconciler on the v1 subject | IGNORE added report fields; never subscribes to the v2 subject | v2 reports reach only an N lease holder; they are notifications, so a drop is repaired next tick. |
|
||||
| Bucket stamp | Bumping it makes N-1 refuse the whole account bucket | Use per-record `schemaVersion`, not the stamp. |
|
||||
|
||||
Type-change hazard in the lifecycle design's record sketch: today `health` is a string (`domains/eks/addon/record.go:30`), and the sketch shows `health` as an array of issues.
|
||||
Writing that would make every N-1 `ListAddons` and `DescribeAddon` on the cluster fail to decode; `health` stays a string, and the issue list is carried in a new, optional `healthIssues` field for v2, never by changing `health`'s existing type (V3).
|
||||
|
||||
## 9. Compatibility tests the v2 contract needs
|
||||
|
||||
| # | Test | Pins |
|
||||
|---|---|---|
|
||||
| T1 | Literal-byte encode of a directive response and of a report batch, including empty `directives`, empty `objects`, omitted optional fields and HTML-escaped characters | camelCase keys, field order, `[]` not `null`, no `BuildJSON` |
|
||||
| T2 | Literal-byte decode of the same bytes into the v2 types, both directions (host encode to guest decode, guest encode to host decode) | Round trip without loss |
|
||||
| T3 | Unknown optional fields at top level, per directive, per report and per object are ignored | Declared extensibility |
|
||||
| T4 | Unknown `mode`, `phase`, `reason`, object `state`, and `contractVersion` 1 or 3 are rejected with the named reason and change no state | Q-75 closed enums and majors |
|
||||
| T5 | Case-mismatched keys (`AddonName`, `Directives`) are rejected by the guest decoder | The v1 quirk cannot recur |
|
||||
| T6 | Directives sorted by `addonName`; duplicate `addonName` in a report batch rejects the batch | Ordering and uniqueness |
|
||||
| T7 | A `message` containing quote, backslash, newline and non-ASCII round-trips through the guest encoder and the gateway envelope | No `printf` JSON |
|
||||
| T8 | The gateway envelope's `executor.instanceId` equals the authenticated session name even when the guest body carries `executor`, `instanceId` or `account` fields | No payload-asserted identity |
|
||||
| T9 | Each rejection row of Section 4.3, one test per row, including a member of another cluster and a non-executor member | Authorization binding |
|
||||
| T10 | A v1 body, v1 subject and v1 report remain byte-identical after v2 is added; the existing `contracts/eks/v1` tests run unchanged | v1 frozen |
|
||||
| T11 | A v2 report on the v1 subject and a v1 report on the v2 subject are both rejected or ignored without state change | No cross-major interpretation |
|
||||
| T12 | Guest fallback: `InvalidAction` and `NotActivated` select v1 mode for one tick; every other reason does not | Fallback cannot bypass the fence |
|
||||
| T13 | An N-1 decoder (`contracts/eks/v1` types) reading a v2 directive or report body fails or ignores safely, never yields a populated v1 value that would be applied | Mis-routing safety |
|
||||
| T14 | Readiness and removal fixtures per bundle: objects with a previous generation annotation never produce `ready`; a shared Secret is excluded while another add-on renders it | Section 6.6 and 6.7 |
|
||||
|
||||
## 10. Review decisions
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| V1 | Legacy-mode fallback is approved only for `InvalidAction` and `NotActivated` (Section 8). The guest never falls back on an authentication, membership, executor, epoch or other fencing failure. |
|
||||
| V2 | `DEGRADED` (Section 6.2) is an observed condition for an otherwise `ACTIVE` add-on. A health loss reported during `CREATING` or `UPDATING` records a health issue but does not by itself turn the operation into a terminal failure; the operation's deadline and an explicit terminal report decide that. |
|
||||
| V3 | Legacy `health` stays a string (Section 8.2); v2 adds an optional `healthIssues` field. `health`'s type is never changed. |
|
||||
| V4 | v2 reason codes (Section 6.3) are a closed enum. A genuinely new wire-level reason has new semantics and requires v3; free-form detail goes in `message`, never in an unbounded `reason` value. |
|
||||
| V5 | The v1 fetch route (Section 7) is bound to the assigned executor once one exists, without changing v1 payload bytes; before assignment, existing v1 behaviour is preserved. |
|
||||
| V6 | Removal (Section 6.7) means removal of add-on-owned Kubernetes objects only; `removed` does not wait indefinitely for downstream workloads, PVs or unrelated resources. Preserve-mode deletion explicitly relinquishes ownership rather than claiming removal. |
|
||||
| V7 | Per-bundle CA-secret names for v2 (Section 6.7), with a deliberate compatibility path for existing v1 shared-secret bundles. |
|
||||
| V8 | An executor is not v2-capable merely because it fetched a directive. Capability is marked only after the first valid, fenced v2 report is accepted (Section 2.3, Section 4.1). A fetch may be retained as diagnostic evidence only. |
|
||||
| V9 | Activation (Section 3, P2) is EKS-add-on-specific for now. The release-membership mechanism should be reusable, but this design does not build a platform-wide activation framework. |
|
||||
| V10 | Automatic executor fencing (Section 4.4) uses only authoritative evidence that the former instance terminated or no longer exists. A lookup error, network issue or partition is never evidence of termination. The alternative is an audited operator assertion. |
|
||||
| V11 | `ActivationStateUnavailable` (HTTP 503, retryable) is defined in v2 from the outset for an unreadable, invalid or unavailable activation record. Only an absent record is `NotActivated`, and only `NotActivated` is eligible for the bounded fallback (V1); `ActivationStateUnavailable` never triggers fallback, because it is not evidence that v2 is inactive. |
|
||||
|
||||
The v1 `PublishInternal` authorization-binding gap and the `serviceAccountRoleArn` format-validation gap (Sections 5.3 and 7) are being fixed directly on the v1 path as standalone security corrections, not as v2 lifecycle work; v2 keeps its own binding and its own guest-side validation regardless of either fix's timing.
|
||||
|
||||
## 11. Open questions for the reviewer
|
||||
|
||||
1. Route and subject names: `internal-addons-v2` and `eks.addon.v2.{account}.{cluster}.status` as proposed, or another spelling?
|
||||
2. Should readiness for the IRSA bundles include a credential check, and for `nvidia-device-plugin` a GPU allocatable check, or stay as defined in Section 6.6?
|
||||
3. Report batching: one batch per tick (proposed) or one POST per add-on as in v1?
|
||||
4. Should the directive carry `deadline` at all, given the guest must ignore it for behaviour, or only in diagnostics?
|
||||
5. v1 guest support lifecycle: what retirement notice and detection (no cluster fetching the v1 route in the Region) is published, and from which release is it counted?
|
||||
@@ -1,205 +0,0 @@
|
||||
# EKS add-on v2: release membership and activation (case study)
|
||||
|
||||
Code read at `25b3efdec`; not verified live.
|
||||
|
||||
**Status:** Draft for review; design dependency only, nothing implemented.
|
||||
|
||||
This is the design dependency for prerequisites P1 (release membership), P2 (activation record) and P5 (controlled rollback) of `docs/package-boundary/eks-addon-v2-contract-design.md` Section 3, plus the finalisation action of R16 (`docs/package-boundary/eks-addon-lifecycle-design.md`).
|
||||
It is written as an EKS case study: every finding is classed as **(a)** a generic compatibility or activation concept that may be reusable later, **(b)** an EKS add-on implementation need, or **(c)** out-of-scope platform-wide release orchestration.
|
||||
Paths are relative to `spinifex/` unless they start with `docs/`; `mulga/` paths are relative to the umbrella repo.
|
||||
|
||||
## Standing
|
||||
|
||||
ADR-0006 (`mulga/docs/adr/0006-contract-guest-and-persisted-state-evolution.md`) is accepted and cited by clause: S1 (boundary declaration), S2 (N/N-1 for a coordinated Regional release), S3 (no timing heuristic as capability proof; capability invalidated on replacement or rollback), S4 (an older writer that discards fields is not compatible), S5 (prepare, activate, migrate, finalise; scope limited to components that read, write or enforce the contract; controlled rollback), S6 (guest legacy mode) and S8 (evidence).
|
||||
ADR-0006's non-goals leave the membership data source and the `spx admin` command shape to implementation, and its Consequences say the add-on work must not decide the platform-wide release mechanism by itself.
|
||||
ADR-0003:S2 rules out a shared abstraction without two real consumers.
|
||||
Q-75 and Q-76 (`mulga/docs/specs/requirements/resource-lifecycle.md:261,275`) are Decided requirements; Q-183 and Q-184 are Open and used as context only.
|
||||
R1 to R16 and V1 to V10 are applied, not reopened; V9 keeps activation EKS-add-on-specific.
|
||||
|
||||
## 1. Facts this design depends on
|
||||
|
||||
| Fact | Evidence | Class |
|
||||
|---|---|---|
|
||||
| No cluster-visible record of the release each process runs; no activation gate | `docs/package-boundary/release-and-version-inventory.md` Section 7 (G1, G2) | a |
|
||||
| Only awsgw receives the ldflags build version; the daemon never does | `operator/cli/version.go:10-15`, `operator/cli/service.go:613`, `runtime/roles/awsgw/awsgw.go:59-71` | b |
|
||||
| Daemons rewrite `heartbeat.<node>` every 10 s in `spinifex-cluster-state` (1 h TTL), with no build or capability field | `daemon/heartbeat.go:9,41-72`, `daemon/jetstream.go:180-191,224-248` | b |
|
||||
| The only prefix lister of that bucket reads `heartbeat.`; other keys are read by exact name | `runtime/compute/cache/liveness.go:25,44-49,103-123`, `daemon/jetstream.go:327,362,408` | b |
|
||||
| Liveness already treats a missing or unreadable heartbeat as Unknown, never Stale | `runtime/compute/cache/liveness.go:32-41,74-95` | a |
|
||||
| Every daemon, N-1 included, answers a per-node health subject without any build field | `daemon/daemon.go:1030`, `daemon/daemon_handlers.go:337-360`, `contracts/cluster/v1/health.go:15-24` | b |
|
||||
| awsgw answers admin-only `GetVersion` for itself only; it writes nothing to cluster state | `gateway/spinifex.go:16-17,53-56,61-62`, `runtime/roles/awsgw/awsgw.go:584-586` | b |
|
||||
| Expected membership is the node config map with per-node services (`daemon`, `awsgw`, ...) | `bootstrap/config/config.go:20-28,186-197,544-562` | a |
|
||||
| A Region is one formed Spinifex cluster | Q-72 (`mulga/docs/specs/requirements/resource-lifecycle.md:215-225`) | a |
|
||||
| Every daemon serves every `eks.*` add-on subject in queue group `spinifex-workers` | `daemon/daemon.go:1089-1122` | b |
|
||||
| Reconciler lease holder and GC sweep holder are daemons identified by node name | `daemon/eks_deps.go:80`, `handlers/eks/cluster_reconciler.go:395,448`, `daemon/daemon.go:2336-2348,2411-2416` | b |
|
||||
| awsgw reads cluster meta for internal-route authz and relays fetch and report | `gateway/eks/internal_authz.go:48-80,143-170`, `gateway/eks.go:40-42,57` | b |
|
||||
| The v1 listing reads manifest keys, and a v1 guest removes any add-on absent from it | `handlers/eks/addons.go:132-156` | b |
|
||||
| Bumping a bucket stamp makes N-1 refuse the whole bucket | `foundation/state/migrate/kv.go:55-67,81-88`, `handlers/eks/store.go:28-29,197-208` | a |
|
||||
| No durable operator audit log; the account-deletion job is the one durable operator record (client token, `kv.Create` claim, replay) | `gateway/requestaudit.go:17-38`, `gateway/admin_deleteaccount.go:80-97,359-386` | a |
|
||||
| `spx admin eks` exists with one subcommand reaching the daemon over NATS | `operator/cli/admin_eks.go:14-77` | b |
|
||||
|
||||
## 2. (a) Generic concepts this case study relies on
|
||||
|
||||
These are concepts and invariants, not a protocol specification or a package.
|
||||
|
||||
| Concept | Meaning |
|
||||
|---|---|
|
||||
| Contract scope | The roles that read, write or enforce one contract (ADR-0006:S5). Other roles are never consulted, so no global "everything upgraded" dependency arises. |
|
||||
| Slot | (Region, node, role), derived from configured membership, never from who reports. |
|
||||
| Incarnation | Random per process start; a restart is a new incarnation. |
|
||||
| Capability token | Compiled into the binary; never configurable; a build version string is diagnostic only. |
|
||||
| Self-report | Written only by the process it describes; additive to existing storage. |
|
||||
| Probe | One per-slot request at evaluation time proving the current occupant; a pre-protocol binary answers without the new fields, which is positive legacy evidence. |
|
||||
| Legacy signal | A record the pre-protocol binary already rewrites wholesale; a fresh rewrite without the protocol marker means a legacy occupant. |
|
||||
| Activation record | One per contract and scope, no TTL, single-CAS transitions, audit inside the record. |
|
||||
| Activation generation | Advanced on each activation; long-lived consumers' capability proofs are tagged with it, so rollback invalidates them (ADR-0006:S3). |
|
||||
|
||||
| Invariant | Why |
|
||||
|---|---|
|
||||
| Unknown is never compatible: `stale`, `absent`, `unreachable` and `unexpected` slots block exactly as `legacy` does | Q-75; ADR-0006:S3 |
|
||||
| Time can only make the answer more conservative: staleness moves a slot from capable to stale, never the reverse | Ageing out removes evidence, never manufactures it; the staleness threshold is the only time constant and is never a fence |
|
||||
| Ageing out never removes an expected slot; a dead member blocks until a capable occupant reports or the node leaves configured membership | Slots come from configuration |
|
||||
| Prepare snapshots every slot's incarnation; activate requires the same incarnations | Replaces a settle window with identity (no timing heuristic) |
|
||||
| No automatic transition; only an operator action moves the state | Q-75: finalisation never follows a binary start |
|
||||
| Unreadable activation state selects neither old nor new semantics | Fail closed |
|
||||
| No bucket stamp bump as the fence | Stamp refusal is all-or-nothing (Section 1) |
|
||||
| Rollback is freeze, drain, deactivate, then downgrade; deactivation needs no binary evidence because it moves toward shared semantics | ADR-0006:S5 |
|
||||
| Long-lived consumers never gate activation; they gate per resource, and may gate finalisation | ADR-0006:S5 step 2, S6 |
|
||||
|
||||
State machine (generic):
|
||||
|
||||
| From | To | Evidence |
|
||||
|---|---|---|
|
||||
| absent or `PREPARED` | `PREPARED` | All in-scope slots capable; one config epoch; no unexpected slot; no unresolved legacy observation; contract readiness; snapshot stored |
|
||||
| `PREPARED` | `ACTIVE` | Same, plus incarnations equal the snapshot |
|
||||
| `ACTIVE` | `DEACTIVATING` | Authorisation only; freezes N-only mutations |
|
||||
| `DEACTIVATING` | `PREPARED` | Zero outstanding N-only obligations; every record in the N-1-readable form |
|
||||
| `DEACTIVATING` | `ACTIVE` | Activation evidence again (abort rollback) |
|
||||
| `ACTIVE` | `FINALISED` | Slots capable; no legacy observation since activation; migration complete; affected long-lived consumers proven; explicit confirmation |
|
||||
|
||||
Concurrent operators: every transition is one CAS on the record revision with a client token; the loser gets the current state and changes nothing, and a repeat with the same token returns the recorded outcome.
|
||||
A crash before the CAS changes nothing; after it, the new state already carries its audit entry.
|
||||
|
||||
## 3. (b) EKS add-on implementation needs
|
||||
|
||||
### 3.1 Relevant participants
|
||||
|
||||
| Participant | Why relevant | Must report |
|
||||
|---|---|---|
|
||||
| Every configured `daemon` slot | Any daemon may serve any add-on mutation; N-1 DROPs, OVERWRITEs and ERASEs (`docs/package-boundary/eks-addon-v1-compatibility.md`) | Preserves v2 record fields on every write; delete is an obligation, not an erase; honours the executor record; lease holder subscribes the v2 subject |
|
||||
| Lease holder and GC sweep holder | Daemons, covered by the daemon slot | as daemon |
|
||||
| Every configured `awsgw` slot | Enforcer and relay; an N-1 gateway answers v2 `InvalidAction`, letting a v2 guest fall back (V1) to an unbound v1 fetch (V5 lost) | v2 routes; v1 fetch bound to executor; v1 listing projected from records |
|
||||
| Control-plane VM agent | Long-lived consumer; capability per executor epoch (V8) | Not a slot; `addonContract: 2` tagged with the activation generation |
|
||||
| Not participants | predastore, viperblock, vpcd, nats, northstar, ui, non-EKS services; account teardown and OIDC readers write no add-on keys (`accountteardown/reapers_eks.go:102-107`, `handlers/sts/oidc_jwks.go:101`) | nothing |
|
||||
|
||||
### 3.2 What the v2 design may depend on
|
||||
|
||||
| Need | Choice | Reason |
|
||||
|---|---|---|
|
||||
| Daemon report | `release.daemon.<node>` in `spinifex-cluster-state` | TTL ages it out; invisible to N-1 readers (Section 1) |
|
||||
| Daemon legacy signal | Additive marker in `heartbeat.<node>` | N-1 rewrites the heartbeat without it every 10 s |
|
||||
| Daemon probe | Additive fields in the per-node health reply | N-1 answers without them |
|
||||
| awsgw report and probe | `release.awsgw.<node>`; additive fields in `GetVersion` to the node's configured gateway | N-1 answers `GetVersion` without them |
|
||||
| Activation record | EKS-owned, no-TTL storage keyed `eks-addon-lifecycle/2` | Not cluster-state (TTL), not per-account (scope is the Region), not the stamp |
|
||||
| Operator surface | `spx admin eks addon-lifecycle {status,prepare,activate,deactivate,finalise}`, with client token and reason; evaluated by a daemon on a subject only N daemons subscribe | Extends the existing group; an N-1 evaluator cannot exist |
|
||||
| Audit | Operator principal (or `local:<node>`), token, from, to, reason and evidence digest, in the same CAS, plus a log line | No durable audit facility exists |
|
||||
|
||||
### 3.3 Activation preflight (EKS)
|
||||
|
||||
| Check | Refusal (summary) |
|
||||
|---|---|
|
||||
| Every configured daemon and awsgw slot capable | `{role} on node {node} is {class}` |
|
||||
| One config epoch; no unexpected slot | `node {node} reports epoch {e}` or `is not in the Region configuration` |
|
||||
| No unresolved legacy observation | `an older {role} was observed on node {node}` |
|
||||
| Record expand complete (P3) over every `eks-account-*` bucket; enumeration fails closed (`handlers/eks/store.go:149-171`) | `{n} add-on records not yet expanded` or `buckets could not be enumerated` |
|
||||
| Executor record present per non-deleting cluster (P4); empty executor allowed and gated per cluster | `cluster {c} has no delivery-executor record` |
|
||||
| Incarnations unchanged since prepare | `{role} on node {node} restarted after prepare; run prepare again` |
|
||||
|
||||
Clusters whose executor is not yet proven are reported, not refused: the per-cluster gate (v2 design Section 8.1) handles them.
|
||||
|
||||
### 3.4 Rollback before finalisation (P5)
|
||||
|
||||
| N-1 behaviour on activated state (compat doc) | Deactivation rule |
|
||||
|---|---|
|
||||
| DROPs `incarnationId`, `desired`, `observed` on update, report, failure | Allowed only with no obligation in flight; re-activation re-expands |
|
||||
| ERASEs on delete and cluster-prefix delete | No record in `DELETING` or `DELETE_FAILED`; each completes on an observed `removed` or `relinquished` report first |
|
||||
| Does not model `UPDATING`, `UPDATE_FAILED`, `DELETING`, `DELETE_FAILED` | None in those statuses; `UPDATE_FAILED` is resolved by `UpdateAddon` or `DeleteAddon` first |
|
||||
| OVERWRITEs the manifest from v1 fields | Harmless: manifest and legacy fields are still written until finalisation (R16) |
|
||||
| Readers IGNORE added fields | Safe |
|
||||
|
||||
While `DEACTIVATING`, create, update and delete are refused with a rollback-in-progress message; reports, deadlines and reconcile continue so obligations drain under v2 semantics.
|
||||
After deactivation, v2 routes answer `NotActivated` and v2 guests fall back to v1 (V1).
|
||||
On re-activation, no earlier `addonContract: 2` marking counts until a fresh fenced v2 report under the new activation generation, because an N-1 reconciler may have swapped control-plane members without advancing any epoch.
|
||||
|
||||
### 3.5 Finalisation (R16)
|
||||
|
||||
| Preflight | Why |
|
||||
|---|---|
|
||||
| Every daemon and awsgw slot capable; no legacy observation since activation | No N-1 host remains |
|
||||
| Every record migrated; every awsgw and daemon projects the v1 listing from records | Manifest keys stop being written |
|
||||
| Every cluster explicitly activated onto v2 has current proof: its executor epoch proven v2 under the current activation generation; such a cluster with no executor, or `FAILED` with running VMs, refuses | A v1 guest served from manifest keys would remove every add-on (`handlers/eks/addons.go:132-134`) |
|
||||
| Every cluster still on v1 is enumerated as legacy in the finalisation output and audit entry | v1 clusters are visible, never silently treated as compatible |
|
||||
| Explicit `--confirm eks-addon-lifecycle/2` | Q-75, ADR-0006:S5 step 4 |
|
||||
|
||||
Finalisation permits stopping, then removing, the manifest key and legacy top-level fields; it makes deactivation and any N-1 rejoin impossible; it ends rollback for the activated v2 track only, and does not retire the v1 routes or v1 guest mode or permit removing their support, which stay an ADR-0006:S6 retirement.
|
||||
|
||||
### 3.6 Coverage
|
||||
|
||||
| Prerequisite | Covered by | Not covered |
|
||||
|---|---|---|
|
||||
| P1 | 3.1, 3.2 | Post-activation awsgw legacy signal (gap 3) |
|
||||
| P2 | 3.2, 3.3 | none |
|
||||
| P5 | 3.4 | Repair after an N-1 writer runs post-activation (gap 2) |
|
||||
| P3, P4 | Consumed as evidence only | Expand pass; executor record and V10 fencing |
|
||||
|
||||
### 3.7 Evidence required later
|
||||
|
||||
| Test | Pins |
|
||||
|---|---|
|
||||
| Rolling upgrade in both orders (daemons first, gateways first); activation refused until the last slot is capable; v1 unchanged | ADR-0006:S2, S8 item 1 |
|
||||
| Activation refused with an N-1 daemon, an N-1 awsgw, a stopped node, a never-reported slot, an unexpected node, an unexpanded record, an unenumerable bucket | Unknown is never compatible |
|
||||
| Activation refused after a restart between prepare and activate | No settle window |
|
||||
| Rollback: freeze; pending delete completes on `removed`; deactivation refused while an obligation or unmodelled status exists; N-1 then serves every record with today's semantics | ADR-0006:S5, S8 items 5 and 6 |
|
||||
| Re-activation re-closes every per-cluster gate until a fresh v2 report | ADR-0006:S3 invalidation |
|
||||
| Finalisation refused while one v1 guest, one `FAILED` cluster with VMs, or one legacy observation remains, and without confirmation | R16, Q-75 |
|
||||
| Crash before and after each CAS; same-token retry returns the recorded outcome | Crash safety |
|
||||
| Two concurrent operators: one transition, one audit entry | CAS |
|
||||
| N-1 daemon started after activation is observed and freezes mutations; resolves only on a capable new incarnation | Detection |
|
||||
| Unreadable activation state: no semantics chosen | Fail closed |
|
||||
| N-1 binaries unaffected by the added keys and fields | ADR-0006:S2 |
|
||||
|
||||
## 4. (c) Out of scope
|
||||
|
||||
Global release orchestration, release managers and rollout drivers (ADR-0006:S5; Q-183 and Q-184 open); any "all Spinifex components upgraded" gate; Predastore, Viperblock, Bluebottle and non-EKS services; shared packages (ADR-0003:S2); NATS or JetStream redesign and stamp bumps; node enrolment, credential revocation and installer downgrade refusal (Q-71, Q-184).
|
||||
|
||||
## 5. Other likely consumers (not enrolled)
|
||||
|
||||
- EBS provider protocol: exact-match wire version today; would need provider slots and a newer-major activation.
|
||||
- RDS and ECS guest agents: versions recorded, never compared; would need per-instance capability under long-lived-consumer rules.
|
||||
- Predastore: independently released; would need a declared version matrix evaluated by capability, not N/N-1.
|
||||
- Viperblock formats: exact-match headers; would need expand and finalise for retained artefacts.
|
||||
- Bluebottle: a linked library, so the linking binary is the participant.
|
||||
- Spinifex KV crossings such as the instance-record migration: first-opener activation could become an operator action.
|
||||
|
||||
Future service work can reuse or challenge the generic concepts in Section 2; this design does not decide that.
|
||||
|
||||
## 6. Gaps and conflicts (flagged, not resolved)
|
||||
|
||||
1. V9 limits activation to EKS; Section 2 states generic concepts, which V9 did not anticipate, though no framework or package is built.
|
||||
2. No fence stops an N-1 daemon started after activation; it can DROP or ERASE before detection, and the heartbeat starts after subscriptions (`daemon/daemon.go:2257,2309`). ADR-0006:S5's rule on automatic downgrade is met only on the controlled path.
|
||||
3. awsgw has no post-activation legacy signal; an N-1 gateway re-opens the V1 fallback and an unbound v1 fetch until its report goes stale.
|
||||
4. The daemon has no runtime build version (`operator/cli/service.go:613`).
|
||||
5. Finalisation requiring every cluster to prove v2 keeps the rollback window open as long as the slowest image upgrade; lifecycle design Section 9 frames finalisation around host consumers and v1 retirement separately.
|
||||
6. Section 3.3 and a fail-closed unreadable state may need a new v2 response reason, while v2 design Section 6.3 lists response reasons as fixed.
|
||||
7. Slots key on the configured node name, while Q-71 requires an enrolled identity (authority open under O-1).
|
||||
8. A node removed from configuration is not revoked (`operator/cli/get.go:80` uses a config-held token), so its record ageing out is paired with configuration evidence only.
|
||||
9. Not every reader of `spinifex-cluster-state` was confirmed (`operator/cli/admin_jsprobe.go:82` opens it).
|
||||
10. The restore path's successor-first ordering still blocks automatic epoch advance after re-activation (v2 design Section 4.4).
|
||||
|
||||
## 7. Review decisions
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| M1 | Finalisation requires current v2 proof only for clusters explicitly activated onto v2. Clusters remaining on v1 are enumerated as legacy, never treated as compatible. Finalisation ends rollback for the activated v2 track; it does not retire v1 or permit removing its support (Section 3.5). |
|
||||
| M2 | An older relevant binary detected after activation freezes EKS add-on mutations Region-wide. Reads continue; create, update, delete, activation and executor reassignment fail closed until an operator upgrades or removes the incompatible writer or performs an explicit controlled rollback. The freeze covers the add-on contract only, not the platform. |
|
||||
| M3 | An unreadable, invalid or unavailable activation record returns the retryable v2 reason `ActivationStateUnavailable` (HTTP 503), defined in v2 from the outset. Only an absent record is `NotActivated`, the sole case eligible for the bounded v2-to-v1 fallback (v2 design V1, V11). |
|
||||
| M4 | The old-writer gap (Section 6 items 2 and 3), the unenrolled-node gap (items 7 and 8) and restore-before-fencing (item 10) are explicit prerequisites for eventual activation, recorded as design debt. They are not package-boundary work and do not extend it. |
|
||||
@@ -1,148 +0,0 @@
|
||||
# Release, Version and Upgrade Mechanism Inventory
|
||||
|
||||
Code read at `3a55e2ee1`; not verified live.
|
||||
|
||||
This is a read-only inventory of the release-membership, rolling-upgrade, rollback, schema-version and guest-image mechanisms that exist today, measured against what the draft Prepare → Activate → Migrate → Finalise sequence and N/N-1 negotiation would need.
|
||||
The draft proposal (`docs/development/proposals/spinifex-architecture/0007-…`, unaccepted) and Q-75/Q-76 in `docs/specs/requirements/resource-lifecycle.md` are the yardstick only; nothing here treats them as decided.
|
||||
The open question it answers is the EKS add-on design's question 4 (`eks-addon-lifecycle-design.md:445`): is there an existing Region-wide release gate that can establish "every daemon runs the new release"?
|
||||
**Short answer: no.**
|
||||
|
||||
Path convention: paths without a prefix are relative to the `spinifex` repo root; `mulga/`, `predastore/`, `viperblock/` and `ansible/` paths are relative to the umbrella repo.
|
||||
Shell scripts, Ansible and docs were read, not run.
|
||||
|
||||
## 1. How a node advertises its build version
|
||||
|
||||
| Mechanism | Evidence | What it carries | Cluster-visible? |
|
||||
|---|---|---|---|
|
||||
| Build-time ldflags | `Makefile:60-62` | `git describe --tags --always --dirty` and short commit into `operator/cli.Version`/`Commit` | No; baked into the binary |
|
||||
| `spx version` | `spinifex/operator/cli/version.go:12-22` | Prints `Version (Commit) os/arch` of the local binary | No; local process only |
|
||||
| Gateway `GetVersion` | `spinifex/operator/cli/service.go:613`, `spinifex/runtime/roles/awsgw/awsgw.go:58-71,487-488`, `spinifex/gateway/spinifex.go:61-62`, `spinifex/gateway/spx/spx.go:27-39` | Version of whichever awsgw process answered the request | Only for the answering gateway; no per-node fan-out |
|
||||
| Daemon heartbeat (KV `spinifex-cluster-state`, key `heartbeat.<node>`) | `spinifex/daemon/jetstream.go:224-236,247`, `spinifex/daemon/heartbeat.go:56-72` | Node, config `Epoch`, services, capacity | Yes, but **no build version field** |
|
||||
| Node health (`spinifex.admin.<node>.health`, HTTPS `/health`) | `contracts/cluster/v1/health.go:15-24` | `ConfigHash`, `Epoch`, uptime, services | Yes, **no build version** |
|
||||
| Node status fan-out (`spinifex.node.status`, `spx get nodes`) | `contracts/cluster/v1/status.go:34-68`, `spinifex/gateway/spx/spx.go:47-60` | Capacity, roles, GPUs | Yes, **no build version** |
|
||||
| Node discovery (`spinifex.nodes.discover`) | `contracts/cluster/v1/discovery.go:14-16` | Node name only | Yes, no version |
|
||||
| Formation join | `spinifex/runtime/formation/formation.go:25-37,70-80` | Name, IPs, region/AZ, services | Join exchange has **no version or protocol field** |
|
||||
| `spinifex.toml` `version` / `epoch` | `spinifex/operator/cli/templates/spinifex.toml:2-3`, `spinifex/bootstrap/config/config.go:19-21`, `spinifex/daemon/daemon.go:2743-2757` | Config-format version ("4") and leader-bumped config epoch, hashed into `ConfigHash` | Indirectly via `ConfigHash`; this is config agreement, not binary version |
|
||||
| Deploy manifest file | `mulga/scripts/update-nodes.sh:165-189,320-321` | Five sub-repo describe strings, written to `/etc/spinifex/deploy-manifest.txt` and the operator's `~/.spinifex/deploy-manifests/` | No; a file on disk, only written by this dev script |
|
||||
|
||||
**Finding:** there is no cluster-wide record of which release each member runs.
|
||||
Every version observation is either local (`spx version`, deploy manifest) or names one process (`GetVersion`).
|
||||
The only cross-node comparison is done by shell scripts over SSH (Section 2).
|
||||
|
||||
## 2. How upgrades are performed today
|
||||
|
||||
| Path | Evidence | Ordering | Drain | Health gate | Version gate |
|
||||
|---|---|---|---|---|---|
|
||||
| Online installer re-run (`curl … \| bash`) | `docs/admin/update/README.md:31-53`, `scripts/setup.sh:828-847,1187-1194` | Per node, operator-driven; no cluster coordination | Whatever `spinifex.target` stop does | None in the script beyond service restart | Optional `--version` pin, SHA-256 verified; no peer comparison |
|
||||
| Manual: install binary, `spx admin upgrade`, restart | `docs/admin/update/README.md:64-100`, `spinifex/operator/cli/admin_upgrade.go:55-120` | Per node | None | None | None; applies local config migrations and systemd unit reconcile only |
|
||||
| `mulga/scripts/update-nodes.sh` (dev/test) | `mulga/scripts/update-nodes.sh:29-38,218-243,329-348,359-377` | **Stop-the-world**: `spx admin cluster shutdown`, update all, start all | Coordinated GATE→DRAIN→STORAGE→PERSIST→INFRA with all-node ACKs | Retries `spx get nodes` until no `NotReady` | Prints `spx version` per host after the fact; header states mixed NATS/predastore/OVN versions are deliberately avoided |
|
||||
| Ansible `cluster-deploy.yml` (dev clusters) | `ansible/playbooks/cluster-deploy.yml:1-37`, `ansible/roles/deploy/tasks/main.yml:97-109,372,390,443-459,516-534` | **Rolling** `serial: 1`, `max_fail_percentage: 0` | `systemctl stop spinifex.target` (drain via `spinifex-shutdown.service`) | awsgw port, target active, predastore Raft quorum rejoin, stale-inode restart | None; no `spx admin upgrade` step, so the cluster runs mixed builds mid-roll with no compatibility check |
|
||||
| `spinifex/scripts/install-node.sh` (formation) | `scripts/install-node.sh:9-11,286-304,347-354` | n/a; forms a cluster | n/a | Smoke test | Refuses to form if `spx version` strings differ (exact match, SSH) |
|
||||
| Coordinated shutdown | `spinifex/operator/cli/admin.go:105-112`, `spinifex/daemon/shutdown.go:29-45` | Phase order with ACKs | Yes | ACK per phase | None |
|
||||
| Local drain | `spinifex/operator/cli/admin.go:120-133` | Local node | GATE+DRAIN only | n/a | None |
|
||||
| `spx admin preflight` | `spinifex/operator/cli/admin_preflight.go:15-28`, `docs/admin/update/README.md:156-183` | Local | n/a | Detects a unit still executing a replaced `spx` inode | Local only; not a peer comparison |
|
||||
|
||||
The two multi-node paths disagree: `update-nodes.sh` avoids mixed versions by stopping the cluster, while `cluster-deploy.yml` rolls one node at a time and so runs N and N-1 together with nothing proving they interoperate.
|
||||
A known mixed-version hazard is already documented: an older daemon ignores the `Target` field of `ShutdownRequest` and treats a single-node drain as cluster-wide (`spinifex/daemon/shutdown.go:34-37`, `docs/admin/host-lifecycle/README.md:123`).
|
||||
RFC2-18 (`mulga/docs/specs/spinifex/rfc2-18_operator-lifecycle.md:34-53,88-104,165-177`, DRAFTING) records the same observations and states that after-the-fact version printing is not a compatibility gate.
|
||||
|
||||
## 3. Feature flags, capability negotiation, minimum-version gates
|
||||
|
||||
| Mechanism | Evidence | Scope | Shape |
|
||||
|---|---|---|---|
|
||||
| EBS provider capabilities | `spinifex/providers/ebs/provider.go:55-70`, `spinifex/providers/ebs/viperblock/provider_handlers.go:236-262`, `spinifex/daemon/daemon.go:2397` | Spinifex ↔ block provider | Boolean feature set fetched at runtime; the only real capability exchange found |
|
||||
| EBS provider wire version | `spinifex/providers/ebs/provider.go:13-25`, `spinifex/providers/ebs/errors.go:90-95`, `spinifex/providers/ebs/nats.go:212-316` | Every provider request/response | **Exact match** (`!= SchemaVersion` → `ErrUnsupportedVersion`); no N-1 window |
|
||||
| RDS data-volume contract image tag | `scripts/images/rds-postgres/manifest.conf:49-51`, `scripts/images/rds-mariadb/manifest.conf:59`, `spinifex/handlers/rds/launch.go:39-45,532`, `spinifex/operator/imagecatalog/images.go:666,686` | Control plane → guest image | AMI filter `rds-data-volume-contract=format-auth-v1` excludes older images; a static image-tag capability, not a runtime proof |
|
||||
| RDS bootstrap envelope | `spinifex/handlers/rds/bootstrap.go:23-36,190-192,212` | Host ↔ RDS guest | `envelopeVersion` exact match |
|
||||
| KV schema-ahead refusal | `spinifex/foundation/state/migrate/kv.go:55-67,87-89` | Any process opening a versioned bucket | Older build refuses to open a bucket stamped newer; fail-closed, not negotiation |
|
||||
| systemd unit marker | `spinifex/operator/host/systemd/reconcile.go:28-33,131-140`, `spinifex/operator/host/systemd/units_gen.go` | Host units | `# spinifex-unit-version: N`; replace only when older |
|
||||
|
||||
Searches for `featuregate`, `feature_flag`, `minVersion` (non-TLS), `minimumVersion`, `semver` and version handshakes in Spinifex found nothing else.
|
||||
There is no "all members at version ≥ X" check, no release gate record and no cluster capability set.
|
||||
|
||||
## 4. Persisted-state versions, migrations and rollback
|
||||
|
||||
### 4.1 KV bucket schema versions
|
||||
|
||||
| Item | Evidence | Notes |
|
||||
|---|---|---|
|
||||
| Version stamp | `spinifex/foundation/state/kvutil/state.go:3-4`, `spinifex/foundation/state/kvutil/kvutil.go:211-244` | `_version` key per bucket; `WriteVersion` is read-then-unconditional-put, not CAS |
|
||||
| Migration runner | `spinifex/foundation/state/migrate/kv.go:17-137` | Chain validation, run, stamp after each step; runs at bucket-open time in whichever process opens it |
|
||||
| Bucket-level versions | e.g. `spinifex/daemon/jetstream.go:53-55,139,154,187`, `spinifex/domains/ec2/vpc/service_impl.go:42-44,171-211`, `spinifex/handlers/rds/store.go:27-38,184,209` | About 40 constants, almost all `1`; IPAM, security groups and static pool are at `2` with **no registered migration** (`spinifex/domains/ec2/vpc/ipam.go:17`, `spinifex/domains/ec2/vpc/security_group.go:128`, `spinifex/domains/network/external/static_pool.go:22`) |
|
||||
| Registered migrations | `spinifex/daemon/instance_records_migrate.go:31-72`, `spinifex/migrate/VERSIONS.md` | Only `spinifex-instance-state` (1→5) and `spinifex-terminated-instances` (1→3) |
|
||||
| Catalogue | `spinifex/migrate/VERSIONS.md` | Lists three KV buckets; the other constants are not catalogued |
|
||||
|
||||
A bucket stamped `1` whose target became `2` with no registered step fails `RunKV` with "no migrations registered" (`spinifex/foundation/state/migrate/kv.go:110-111`); `VERSIONS.md` says those installs required `spx admin init --force`, which is a hard cutover with no documented Q-75 exception record.
|
||||
|
||||
### 4.2 Expand/contract precedent
|
||||
|
||||
The instance-record crossing is the only expand–migrate–contract precedent found.
|
||||
`VERSIONS.md` states version 5 of `spinifex-instance-state` froze the legacy `instance.<id>` and `node.<id>` keys and left them in place so the crossing can be rolled back, and the tests cover idempotent re-run and not overwriting fresher records (`spinifex/daemon/instance_records_transition_test.go:221-315`).
|
||||
It is not a full Prepare→Activate→Finalise sequence: activation happens when the first upgraded process opens the bucket, the bump itself makes the older build refuse to start (`SchemaAheadError`), and there is no operator finalisation step that deletes the frozen keys.
|
||||
So "rollback" means a binary downgrade is refused rather than supported; it would need the version stamp to be reset by hand (not documented).
|
||||
|
||||
### 4.3 Record- and file-level schema versions
|
||||
|
||||
| Format | Evidence | Rule |
|
||||
|---|---|---|
|
||||
| EBS volume/snapshot/AMI metadata documents | `spinifex/domains/ec2/ebs/metadata/metadata.go:14-17,166-221` | `schema_version` = 2; decode rejects anything `!= 2` as corrupt, so N-1 cannot read N's documents and vice versa |
|
||||
| Node local state (disk + KV `node.<id>`) | `spinifex/daemon/local_state.go:14-34,131-133` | Exact match `1` |
|
||||
| Config files | `spinifex/migrate/VERSIONS.md`, `spinifex/migrate/migrate.go:191-246,327-332` | Per-file version; `RunConfig` backs up before each step; no config migrations registered |
|
||||
| Object-store migrations | `spinifex/migrate/migrate.go:126-161`, `spinifex/migrate/VERSIONS.md` | Stamp in a KV bucket, read-then-write; none registered |
|
||||
| Predastore cluster config | `predastore/internal/config/config.go:25-28,358` | `Version = 1`, exact match, "no migration between versions" |
|
||||
| Predastore placement record | `predastore/internal/gate/handlers/placement_record.go:17-48,85,125-139` | Readers accept v1–v3 and reject unknown; writer always emits v3 (not "oldest compatible") |
|
||||
| Viperblock WAL/chunk/checkpoint headers | `viperblock/viperblock/snapshot.go:346-349`, `viperblock/viperblock/viperblock.go:3622,4788,6532` | Magic + `Version` uint16, exact match |
|
||||
| Viperblock flat-section version byte | `viperblock/viperblock/snapshot.go:634,726-729` | Reserved, not checked |
|
||||
|
||||
No resource record carries a per-record `schemaVersion` today; the EKS add-on design proposes adding one (`eks-addon-lifecycle-design.md:119,421`).
|
||||
|
||||
### 4.4 Rollback
|
||||
|
||||
No rollback command, procedure or test exists for the platform release.
|
||||
`spx admin upgrade` has no reverse mode (`spinifex/operator/cli/admin_upgrade.go:55-71`); a downgraded unit marker is reported as a conflict rather than restored (`spinifex/operator/host/systemd/reconcile.go:133-140`); config backups are files, not managed restore points (RFC2-18 C-127 at `mulga/docs/specs/spinifex/rfc2-18_operator-lifecycle.md:97-104`).
|
||||
The update runbook covers forward upgrade and stale-binary troubleshooting only (`docs/admin/update/README.md:31-213`).
|
||||
|
||||
## 5. Guest images and agents
|
||||
|
||||
| Guest | Image version source | Agent version | Control plane records | Upgrade path |
|
||||
|---|---|---|---|---|
|
||||
| EKS node (CP and worker) | `scripts/images/eks-node/manifest.conf:2,4,25,87-88`, `scripts/images/eks-node/setup.sh:16` (K3s `v1.32.5+k3s1`), catalogue `spinifex/operator/imagecatalog/images.go:576-590` (Version = Alpine `3.21.7`, mutable URL, sidecar SHA-256) | Guest binaries built without version ldflags; no guest reports one | `ClusterMeta.Version` = requested Kubernetes version, default `1.32`, not validated against the image (`spinifex/handlers/eks/service_impl.go:291,587`); `ControlPlaneInstanceID` (`spinifex/handlers/eks/cluster_state.go:109`); nodegroup `ReleaseVersion` is a fixed placeholder `1.0.0-spinifex-eks-node` (`spinifex/handlers/eks/nodegroup.go:43-46,296-299`); `PlatformVersion` is constant `eks.1` (`spinifex/handlers/eks/service_impl.go:2188-2191`) | None: `UpdateClusterVersion`, `UpdateNodegroupVersion`, `ListUpdates` and `DescribeUpdate` have no gateway route at this revision (`spinifex/gateway/eks.go`, 31 routes, none match); plan only in `mulga/docs/development/feature/eks-cluster-version-upgrade.md` |
|
||||
| EKS add-on bundles | Baked per version into the image under `addons/<name>/<version>` (`scripts/images/eks-node/mulga-eks-addon-sync.sh:38,81-87`) | Add-on status report carries add-on version only (`contracts/eks/v1/addon_status.go:31-37`) | Add-on record `addonVersion` | A new add-on version needs a new image; v1 guest protocol has no protocol version or capability field |
|
||||
| ECS container instance | `scripts/images/ecs-agent/manifest.conf:4,23,40-41` | `version = "dev"` unless ldflags set (`cmd/ecs-agent/main.go:18-24`); `Makefile:78` sets it but the image manifest build does not, so imaged agents likely report `dev` (not verified live) | `AgentVersion` stored and echoed in `VersionInfo` (`spinifex/handlers/ecs/records.go:237`, `spinifex/handlers/ecs/instances.go:283,333`); never compared | Relaunch capacity; no agent-update path |
|
||||
| RDS (postgres, mariadb) | `scripts/images/rds-postgres/manifest.conf:4,19,47-51`, `scripts/images/rds-mariadb/manifest.conf:4,21,55-59`, tags `engine`, `engine-version`, `rds-data-volume-contract` | `version = "dev"` default (`cmd/rds-agent/main.go:21-23,72`); image build sets no ldflags | `Agent.AgentVersion`, `EngineVersion` recorded (`spinifex/handlers/rds/records.go:405-415`, `spinifex/handlers/rds/agent.go:189`); never compared; AMI ID used at launch is logged, not stored on the DB record (`spinifex/handlers/rds/launch.go:189,267,353`) | `ModifyDBInstance EngineVersion` returns unimplemented (`spinifex/handlers/rds/modify.go:477-478`) |
|
||||
|
||||
AMI selection for EKS, RDS and ECS takes the newest image matching tags (`spinifex/handlers/eks/k3s_server_vm.go:353-363`, `spinifex/handlers/rds/launch.go:552-558`, `spinifex/handlers/ecs/capacity.go:108-119`), so importing a new image silently changes what new resources receive while existing ones keep the old one.
|
||||
The control plane can find a guest's AMI only indirectly through the backing EC2 instance; it cannot learn the guest's protocol support from the guest.
|
||||
|
||||
## 6. Offline artefacts and backups
|
||||
|
||||
| Artefact | Evidence | Version field |
|
||||
|---|---|---|
|
||||
| EBS snapshot / AMI metadata | `spinifex/domains/ec2/ebs/metadata/metadata.go:57-58,79-80` | `schema_version` 2, exact match |
|
||||
| Viperblock snapshot checkpoint | `viperblock/viperblock/snapshot.go:346-349` | Header magic + version, exact match |
|
||||
| RDS DB snapshot record | `spinifex/handlers/rds/records.go:291-320` | `EngineVersion` only; no record schema version; data lives in an EC2 snapshot |
|
||||
| EKS etcd backups (`eks-backups-system` bucket) | `spinifex/handlers/eks/restore_snapshot.go:19-22,53-75` | Name-encoded tier and timestamp only; no K3s/etcd version recorded |
|
||||
| Account export | Not found | n/a |
|
||||
|
||||
No artefact declares a support or retirement window, and none records the Spinifex release that wrote it.
|
||||
|
||||
## 7. Gaps against Prepare → Activate → Migrate → Finalise and N/N-1
|
||||
|
||||
Stated as gaps only; no design is implied.
|
||||
|
||||
| # | Need | Current state | Gap |
|
||||
|---|---|---|---|
|
||||
| G1 | Authoritative record of each member's release | Heartbeat, health, status and formation carry no build version (Section 1) | No release-membership data source exists; activation evidence ("every writer can preserve the transitional form") cannot be produced |
|
||||
| G2 | Region-wide activation gate | None; only `install-node.sh` exact string match over SSH at formation | Nothing operator-visible records "activated"; the EKS add-on slice 7 would have to build it |
|
||||
| G3 | Finalisation step and preflight | Frozen instance keys are never retired; no finalise command | No operator-visible end of rollback for any change |
|
||||
| G4 | Controlled rollback | Schema-ahead refusal blocks downgrade; no reverse migration, no unit downgrade, no runbook | Rollback is unsupported once any bucket is bumped |
|
||||
| G5 | N/N-1 wire compatibility | Most contracts unversioned (C-9); versioned ones use exact match (EBS provider, RDS envelope, local state, EBS metadata, viperblock headers) | No contract implements a two-version window or "emit oldest compatible" |
|
||||
| G6 | Capability exchange | Only EBS provider booleans and an RDS image tag | No authenticated per-peer or per-guest capability record; no invalidation on replacement |
|
||||
| G7 | Fenced, owned migration | Migrations run on bucket open in any process; version stamp is read-then-put | No migration owner, fence or per-record progress; concurrent runners rely on idempotence |
|
||||
| G8 | Per-record schema version | Bucket-level only | Lazy per-record migration under a resource owner has no field to key on |
|
||||
| G9 | Coherent rolling procedure | Stop-the-world script vs `serial: 1` Ansible roll with no version or compatibility check; known mixed-version drain hazard | No supported rolling-upgrade contract with a stop-on-skew rule |
|
||||
| G10 | Guest version visibility | Agent versions recorded but default `dev` in images and never compared; EKS guests report none | Control plane cannot tell a v1 from a v2-capable guest except by a protocol it does not yet have |
|
||||
| G11 | Guest upgrade operations | EKS version/nodegroup updates unrouted; RDS engine upgrade unimplemented; newest-AMI-wins selection | No owned path to move a guest to a new image, and image choice is implicit |
|
||||
| G12 | Release manifest | `update-nodes.sh` writes a dev-only text manifest; installer pins one tarball | No immutable release manifest binding binary, contract versions and supported source versions (RFC2-18 Q-183 target) |
|
||||
| G13 | Artefact lifecycle | Snapshot/backup formats lack writer release and support window; etcd backups record no K3s version | Retained artefacts cannot be checked for usability before a format is retired |
|
||||
| G14 | Version catalogue accuracy | `VERSIONS.md` lists 3 of about 40 KV buckets | No single inventory of persisted formats to drive a finalisation preflight |
|
||||
@@ -1,75 +0,0 @@
|
||||
# STS OIDC discovery projection plan
|
||||
|
||||
**Status: Proposed** — scope is ADR-0005 S4 only; the Q-22 cut uses an EKS-owned in-process projection, not a wire contract.
|
||||
|
||||
## Current state
|
||||
|
||||
- `handlers/sts/oidc_jwks.go:14` imports `handlers/eks` for exactly two symbols: `AccountBucketName` (line 101) and `OIDCJWKSKey` (line 110).
|
||||
- No EKS store type, record or `ClusterMeta` is used by STS production code; the JetStream access is `s.js.KeyValue("eks-account-{issuerAccount}")` then `kv.Get("clusters/{cluster}/oidc-jwks.json")` (`oidc_jwks.go:93-126`).
|
||||
- The only caller is the `AssumeRoleWithWebIdentity` key function (`handlers/sts/assume_role_with_web_identity.go:159-193`): parse the `iss` claim (`ParseEKSIssuerURL`, line 173), check the IAM OIDC-provider registration in the role account (line 177), read the JWKS from the issuer account (line 180), select by `kid` (line 187).
|
||||
- Every failure in the key function becomes `InvalidIdentityToken` (HTTP 400) at line 71; there is no cache, so each request reads KV afresh.
|
||||
- The public `/oidc/eks/{region}/{accountID}/{clusterName}/.well-known/openid-configuration` and `/keys` routes are served by the root `gateway` package (`gateway/gateway.go:351-352`, `gateway/oidc_discovery.go`), not by STS, and are outside the Q-22 rule because the root gateway package has no domain.
|
||||
- The composition root is `runtime/roles/awsgw/awsgw.go:222` (`NewSTSServiceImpl`), which already wires one cross-service capability at line 228 (`SetSessionRevoker`).
|
||||
- The JWKS key is written once per cluster launch (`handlers/eks/service_impl.go:874` → `oidc_keypair.go:94`) as a single-key document, read by the EKS bootstrap cross-check (`nats_bootstrap.go:283`), and removed only by `DeleteClusterPrefix` when teardown completes (`service_impl.go:1442`); there is no rotation path.
|
||||
|
||||
## Resolver (consumer-owned, in `handlers/sts`)
|
||||
|
||||
```go
|
||||
// ClusterJWKSResolver returns the published JWKS document for an EKS cluster issuer.
|
||||
type ClusterJWKSResolver interface {
|
||||
LookupClusterJWKS(ctx context.Context, accountID, clusterName string) (jwks []byte, found bool, err error)
|
||||
}
|
||||
```
|
||||
|
||||
| Today's outcome | Resolver result | STS result (unchanged) |
|
||||
| --- | --- | --- |
|
||||
| Account bucket missing (`ErrBucketNotFound`) | `nil, false, nil` | `InvalidIdentityToken` |
|
||||
| Cluster missing, key absent, key deleted by teardown | `nil, false, nil` | `InvalidIdentityToken` |
|
||||
| Bucket open or key read fails (timeout, unavailable) | `nil, false, err` | `InvalidIdentityToken` |
|
||||
| Key present, any bytes | `raw, true, nil` | STS decodes; malformed or empty set is `InvalidIdentityToken` |
|
||||
|
||||
- The resolver returns raw bytes so JSON decoding, the empty-set rule and `JWK`/`JWKS` types stay in STS, and the implementation never imports `handlers/sts`.
|
||||
- `FetchClusterJWKS` keeps its `(nil, nil)` / error contract, taking the resolver instead of `jetstream.JetStream`; `ParseEKSIssuerURL` and the registration-before-lookup ordering stay in STS.
|
||||
- `NewSTSServiceImpl` takes the resolver as a required argument and rejects nil, mirroring the nil-IAM check, so a missing binding fails at start-up rather than as a permissive path.
|
||||
|
||||
## Placement: `handlers/eks`, not a new domain package
|
||||
|
||||
- Rejected: a new `domains/eks/projection` package. `Domains_ImportNoAgentsIngressOrLegacy` (`architecture/layering/rules_test.go:130`) forbids any new-path `domains/*` package from importing a `handlers/*` legacy root, and the bucket/key layout lives only in `handlers/eks` today, so a projection package could not read it without that forbidden edge.
|
||||
- Instead: `handlers/eks` implements the resolver itself, beside the layout it already owns. `OIDCJWKSReader` (`handlers/eks/oidc_jwks_reader.go`), built with `NewOIDCJWKSReader(js jetstream.JetStream)`, does the same bucket-open-then-`Get`, the same timeouts and the same not-found classification `handlers/sts/oidc_jwks.go` used, next to `AccountBucketName` and `OIDCJWKSKey` in `store.go`.
|
||||
- This is transitional: `handlers/eks` is the current owner of the layout, not a permanent home for a cross-domain adapter. When `handlers/eks` is extracted to a `domains/eks` package, the resolver implementation moves with it.
|
||||
- `handlers/eks` must not import `handlers/sts` (that edge is still forbidden), so the compile-time interface check `var _ handlers_sts.ClusterJWKSResolver = (*handlers_eks.OIDCJWKSReader)(nil)` lives in the composition root, `runtime/roles/awsgw`, not beside the implementation.
|
||||
|
||||
## Wiring
|
||||
|
||||
- `runtime/roles/awsgw/awsgw.go:222` builds `handlers_eks.NewOIDCJWKSReader(js)` and passes it to `NewSTSServiceImpl` as a required, nil-checked argument.
|
||||
- `tests/integration/harness.go:150` is the only other production-shaped STS construction site and is wired identically.
|
||||
|
||||
## Imports and allowlist
|
||||
|
||||
- Disappears: `handlers/sts/oidc_jwks.go` `handlers_eks` import, the last non-test STS import of any EKS package.
|
||||
- Goes stale and is deleted: `{From: "handlers/sts", To: "handlers/eks"}` (`architecture/layering/allowlist_test.go:161`); the layering test fails on a stale entry, so it cannot be left in place.
|
||||
- Stays: `handlers/sts → handlers/iam` and `gateway/sts → handlers/iam`, because the IAM registration read, trust policy and session crypto are out of scope.
|
||||
- Test files that exercise the real KV layout move to `handlers/eks`, next to the layout they pin.
|
||||
- Every remaining `handlers/sts` test, including the web-identity fixtures and the `newTestSetup` constructor wiring, uses a test-owned `fakeJWKSResolver` and imports no EKS package.
|
||||
- The `Q22_STSImportsNoEKSImplementation` rule stays and now passes with no exception: STS imports no EKS package at all, not even through an allowlist entry, and that holds for test files too (`grep -rn 'handlers/eks' spinifex/handlers/sts/` is empty).
|
||||
|
||||
## Test placement (post-cut)
|
||||
|
||||
- `handlers/eks/oidc_jwks_reader_test.go`: the raw KV layout and literal bucket/key pin (`TestOIDCJWKSReader_UsesLiteralBucketAndKey`), missing bucket (`TestOIDCJWKSReader_AccountBucketMissingIsNotFound`), dependency failures and a tombstoned key (`TestOIDCJWKSReader_DependencyFailureSemantics`), input validation (`TestOIDCJWKSReader_NilJetStream`, `TestOIDCJWKSReader_EmptyArgs`), cluster status never consulted (`TestCurrentBehaviour_ClusterRecordStateIsNotConsulted`, moved from `handlers/sts`), and a deleted cluster prefix reading as not found (`TestCurrentBehaviour_ClusterPrefixDeletedIsNotFound`, moved from `handlers/sts`).
|
||||
- `handlers/sts/fake_jwks_resolver_test.go`: the `fakeJWKSResolver` test double — a map keyed by `(accountID, clusterName)` with seed/seedErr and call recording — used by every STS test in this package.
|
||||
- `handlers/sts/oidc_jwks_test.go`: `FetchClusterJWKS` unit tests against the fake, including `TestFetchClusterJWKS_PropagatesResolverErrorUnchanged` (renamed from `..._ResolverErrorIsWrapped`), which pins that a resolver error passes through unchanged via `assert.Same` and `errors.Is`.
|
||||
- `handlers/sts/oidc_jwks_current_behaviour_test.go`: the STS-owned outcomes — registration runs before any resolver call (`TestCurrentBehaviour_UnregisteredIssuerNeverCallsResolver`), the issuer's account is queried, not the role's (`TestCurrentBehaviour_CrossAccountIssuerQueriesIssuerAccount`), every resolver outcome (not-found, malformed JWKS, dependency error) maps to `InvalidIdentityToken` (`TestCurrentBehaviour_ClusterMissingIsInvalidIdentityToken`, `TestCurrentBehaviour_JWKSMalformedIsInvalidIdentityToken`, `TestCurrentBehaviour_ResolverErrorIsInvalidIdentityToken`), and rotation is read fresh each request (`TestCurrentBehaviour_JWKSRotationIsReadFreshEachRequest`).
|
||||
- `handlers/sts/service_impl_test.go` and `assume_role_with_web_identity_test.go`: `newTestSetup` wires a `*fakeJWKSResolver`, and the web-identity fixture seeds it directly instead of writing EKS KV records.
|
||||
- `tests/integration/sts_eks_oidc_issuer_test.go`: the one test that deliberately imports both `handlers/eks` and `handlers/sts` — `TestEKSClusterOIDCIssuer_RoundTripsThroughSTSParseEKSIssuerURL` proves a `ClusterOIDCIssuer` URL parses through `ParseEKSIssuerURL` to the same `(accountID, clusterName)`.
|
||||
- `gateway/oidc_discovery_current_behaviour_test.go` is unchanged: `TestCurrentBehaviour_WebIdentityJWKSMissingWireResponse` still builds a real `OIDCJWKSReader` plus a real STS service and pins the wire HTTP 400 with `<Code>InvalidIdentityToken</Code>`. The root `gateway` package has no domain classification and composes both packages intentionally, so this is not a package-boundary violation.
|
||||
- `gateway/oidc_discovery_current_behaviour_test.go` also pins the public discovery/JWKS endpoints: 404/500 bodies, verbatim JWKS bytes and `Cache-Control: public, max-age=300`, unaffected by this cut.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Moving IAM state, the OIDC-provider registry read or `handlers_iam` imports.
|
||||
- Redesigning web-identity verification, issuer parsing, audience or trust-policy handling, or fetching JWKS over HTTPS as AWS STS does.
|
||||
- Moving `handlers/sts` to `domains/sts`, or changing the cluster-status blindness, DELETING/FAILED key retention, or adding rotation.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should the root-gateway discovery handlers (`gateway/oidc_discovery.go:38,93`) consume an EKS-owned resolver in a later change, the same pattern as STS? The root gateway's `handlers/eks` import has no allowlist entry today because the root gateway has no domain classification, so it is tracked as debt rather than fixed here; S4 is about STS only.
|
||||
@@ -1,516 +0,0 @@
|
||||
# Tagging capabilities discovery
|
||||
|
||||
**Status:** Discovery only.
|
||||
Nothing here is a decision.
|
||||
It records current behaviour at the head of the ADR-0001 checkpoint branch and proposes a capability shape for review.
|
||||
The ADR-0001 structural cut in [section 4](#4-proposal-the-adr-0001-structural-cut) has since been implemented; the [future direction](#future-direction-outside-the-adr-0001-structural-cut) has not.
|
||||
|
||||
## Scope and method
|
||||
|
||||
This covers every Spinifex path that reads or writes resource tags, grouped by the owner of the record the tags live on.
|
||||
File references are relative to the repository root at that revision.
|
||||
References to `gateway/tagging` describe the code before the structural cut moved `GetResources` to `domains/tagging` and the listers to `runtime/roles/awsgw`.
|
||||
|
||||
What was measured:
|
||||
|
||||
- Code reading of every tag entry point found by searching for `Tag`/`Untag`/`AddTags`/`RemoveTags`/`ListTags`/`DescribeTags` handlers and the EC2 record-mirror fan-out.
|
||||
- Sixteen characterisation tests named `TestCurrentBehaviour_Tagging_*` (listed under [Tests](#tests)), run through real handlers on embedded JetStream.
|
||||
- AWS documentation fetched on 2026-10-13 for the pages cited in [AWS parity](#aws-parity).
|
||||
|
||||
What was not measured:
|
||||
|
||||
- No live cluster run, so multi-node interleavings are shown by injecting a write between a handler's read and its write, not by real concurrent nodes.
|
||||
- No test drives the full daemon CreateTags path including record mirrors on a multi-node cluster.
|
||||
- Predastore's own S3 bucket tagging was not exercised; Spinifex does not read it.
|
||||
|
||||
Related work this does not duplicate:
|
||||
|
||||
- `mulga-6maav`: CreateTags accepts unvalidated or nonexistent resource IDs.
|
||||
- `mulga-5eh43`: IAM CreatePolicy rejects `aws:RequestTag` condition keys.
|
||||
- `mulga-lmq4l`: tag-based ABAC condition keys.
|
||||
- `docs/development/bugs/concurrent-createtags-lost-updates.md` (mulga repo): moved the EC2 central index to JetStream KV under CAS (`mulga-tt77o`, closed).
|
||||
That plan's "Not done" list leaves the two-store split in place, which is where the record-mirror finding below sits.
|
||||
- `docs/development/bugs/creation-time-tags-describetags.md` (mulga repo): creation-time tags reach the central index only for some EC2 owners.
|
||||
|
||||
## 1. Map of tag reads and writes by owner
|
||||
|
||||
### EC2 and the VPC family
|
||||
|
||||
EC2 has two tag stores.
|
||||
|
||||
- **Central index:** JetStream KV bucket `ec2-tags`, key `{accountID}.{resourceID}`, value the whole tag map, owned by `TagsServiceImpl` (`spinifex/spinifex/domains/ec2/tags/service_impl.go:50`).
|
||||
A legacy Predastore object `tags/{accountID}/{resourceID}.json` is read through and deleted on teardown (`service_impl.go:129`, `:156`).
|
||||
`DescribeTags` and `GetResources` read only this store.
|
||||
- **Resource record:** each owning service's own KV or metadata record carries a `Tags` map.
|
||||
Each resource's own describe call and its tag filters read only this.
|
||||
|
||||
Write paths:
|
||||
|
||||
| Path | Package | Writes | Concurrency |
|
||||
| --- | --- | --- | --- |
|
||||
| AWSGW `CreateTags` validation | `domains/ec2/awsapi/tags/CreateTags.go:15` | nothing; refuses empty key and `aws:` prefix | n/a |
|
||||
| Daemon `createTags` / `deleteTags` | `daemon/daemon_handlers_tag.go:89`, `:125` | central index for non-instance IDs, then every record mirror, then instance owners | sequential, first error returns |
|
||||
| Central index merge | `domains/ec2/tags/service_impl.go:195`, `:377`; `store.go:77` | `ec2-tags` | CAS via `kvutil.Update` |
|
||||
| Central index overwrite | `domains/ec2/tags/store.go:102` (`PutResourceTags`) | `ec2-tags` | CAS put |
|
||||
| Record mirrors (`ApplyRecordTags`/`RemoveRecordTags`) | vpc `domains/ec2/vpc/service_impl.go:1091`, `:1110`, `:1166`; image `domains/ec2/image/service_impl.go:681`; volume `domains/ec2/volume/service_impl.go:1175`, `:1208`; snapshot `domains/ec2/snapshot/service_impl.go:997`; route table `domains/ec2/routetable/service_impl.go:1588`; igw `domains/ec2/igw/service_impl.go:598`; eigw `domains/ec2/eigw/service_impl.go:277`; eip `domains/ec2/eip/service_impl.go:938`; natgw `domains/ec2/natgw/service_impl.go:538`; key pair `domains/ec2/key/service_impl.go:1032` | each owner's record | plain read then `Put` (`domains/ec2/tagmirror/tagmirror.go:83`), no revision |
|
||||
| Instance tags | `daemon/daemon_handlers_tag.go:194` routes to the owning daemon over `ec2.cmd.<id>`, or `TagStoppedInstance` when no responder | instance record and central index together | ownership-gated |
|
||||
| Creation-time `TagSpecifications` | each create path | record; central index only for instance, vpc/subnet/sg/eni (`domains/ec2/vpc/service_impl.go`, `projectRecordTags`) and key pair (`domains/ec2/key/service_impl.go:55`) | central write failure is logged and swallowed |
|
||||
|
||||
The daemon fan-out (`daemon/daemon_handlers_tag.go:34`) calls every mirror with every non-instance ID, and each mirror skips IDs whose prefix it does not own.
|
||||
`launchtemplate` and `placementgroup` are not in the mirror list, so a `CreateTags` on them reaches only the central index.
|
||||
That last point is from code and is not tested.
|
||||
|
||||
Read paths:
|
||||
|
||||
- `DescribeTags` lists every key in the bucket and filters by account prefix (`domains/ec2/tags/service_impl.go:243`, `:290`).
|
||||
An ID with no recognised prefix is reported with resource type `unknown` (`:84`).
|
||||
- Resource describes filter on their own record's tags (for example `domains/ec2/instance/service_impl.go:2040`).
|
||||
|
||||
Cross-owner reach: the daemon is the composition point that writes the central index and then reaches every record owner; the record mirrors each write only their own record.
|
||||
|
||||
### ELBv2
|
||||
|
||||
- **Store:** load balancer, target group, listener and rule records, each with a `Tags` map (`spinifex/spinifex/handlers/elbv2/service_impl_tags.go:25`).
|
||||
- **Writes:** `AddTags` and `RemoveTags` (`service_impl_tags.go:101`, `:149`).
|
||||
Load balancer, listener and rule tag writes re-put the whole record read earlier (`:41`, `:74`, `:88`).
|
||||
Target group writes go through `UpdateTargetGroupByArn` CAS, but the merged map is computed from the earlier read and assigned inside the CAS closure (`:57`).
|
||||
- **Reads:** `DescribeTags` returns one entry per ARN, tagged or not (`handlers/elbv2/service_impl.go:3041`).
|
||||
- **Cross-owner reach:** `gateway/tagging` calls `DescribeLoadBalancers`, `DescribeTargetGroups` and `DescribeTags` over NATS (`spinifex/spinifex/gateway/tagging/GetResources.go:106`).
|
||||
|
||||
### RDS
|
||||
|
||||
- **Store:** DB instance, DB snapshot, subnet group and parameter group records in the account bucket (`spinifex/spinifex/handlers/rds/tags.go:56`).
|
||||
- **Writes:** `AddTagsToResource`, `RemoveTagsFromResource` (`tags.go:104`, `:141`) through a registry of `TaggedRecord` kinds, with a revision-checked update retried up to 16 times (`tags.go:207`).
|
||||
- **Validation:** 50 tags after merge, 128-character key, 256-character value, `aws:` refused (`tags.go:21`, `:236`).
|
||||
- **ARN check:** region and account must match the caller (`handlers/rds/arn.go:66`, `:69`).
|
||||
- **Cross-owner reach:** `handlers/rds/tags.go` writes `domains/rds/subnetgroup` and `domains/rds/parametergroup` records directly (`tags.go:53`, `:67`, `:72`).
|
||||
These are the `handlers/rds` → `domains/rds/subnetgroup` and `handlers/rds` → `domains/rds/parametergroup` legacy edges in the `docs/PACKAGE_BOUNDARY_MIGRATION.md` "Dependency-rule debt to clear after the moves" list.
|
||||
|
||||
### ECS
|
||||
|
||||
- **Store:** cluster, task definition revision, service, task and container instance records in the account bucket (`spinifex/spinifex/handlers/ecs/tags.go:77`).
|
||||
- **Writes:** `TagResource`, `UntagResource` (`tags.go:228`, `:255`) via `mutateResourceTags` (`tags.go:141`), which re-puts the whole record with no revision.
|
||||
- **ARN check:** `parseResourceARN` keeps only the resource segment (`tags.go:41`); the ARN's account and region are not compared with the caller's.
|
||||
- **Validation:** none on count, length or prefix.
|
||||
- **Cross-owner reach:** reads and writes `domains/ecs/taskdefinition` records directly (`tags.go:89`, `:154`), the `handlers/ecs` → `domains/ecs/taskdefinition` legacy edge in the `docs/PACKAGE_BOUNDARY_MIGRATION.md` "Dependency-rule debt to clear after the moves" list.
|
||||
|
||||
### EKS
|
||||
|
||||
- **Store:** cluster meta and nodegroup records (`spinifex/spinifex/handlers/eks/service_impl.go:1763`).
|
||||
- **Writes:** `TagResource`, `UntagResource` (`:1763`, `:1806`) under CAS (`casUpdateMeta`, `casUpdateNodegroupTags`).
|
||||
Other EKS ARNs return NotImplemented.
|
||||
- **ARN check:** `clusterNameFromARN` and `nodegroupRefFromARN` discard account and region (`:1925`, `:1940`).
|
||||
- **Validation:** none on count, length or prefix.
|
||||
|
||||
### ECR
|
||||
|
||||
- **Store:** repository meta record (`RepoMeta.Tags`) in the account's ECR KV bucket (`spinifex/spinifex/domains/ecr/meta.go:163`).
|
||||
- **Writes:** `TagResource`, `UntagResource` in the AWS adapter (`spinifex/spinifex/domains/ecr/awsapi/tags.go:59`, `:90`), which reads the whole meta over NATS, edits it in the gateway process, and puts it back with a plain `kv.Put`.
|
||||
- **ARN check:** `RepositoryNameFromResourceARN` keeps only the repository name (`domains/ecr/awsapi/authz.go:217`).
|
||||
- **Validation:** missing or empty key only (`domains/ecr/awsapi/validation.go:64`).
|
||||
|
||||
### ACM
|
||||
|
||||
- **Store:** certificate record (`CertRecord.Tags`) (`spinifex/spinifex/domains/acm/store.go:223`).
|
||||
- **Writes:** `AddTagsToCertificate`, `RemoveTagsFromCertificate` (`spinifex/spinifex/domains/acm/service_impl.go:724`, `:751`), through `Store.UpdateTags` (`store.go:502`).
|
||||
Both check ownership through `lookupOwned`, then change only `Tags` through `Store.UpdateTags`, a revision-checked update of the raw record that leaves the encrypted private key untouched.
|
||||
Before `mulga-ncn9f` was fixed they wrote back the `GetCertMetadata` copy through `PutCert`, which replaced the stored key with an empty one.
|
||||
- **Validation:** none on count, length or prefix.
|
||||
|
||||
### IAM
|
||||
|
||||
- **Store:** role, user, customer-managed policy, instance profile and OIDC provider records in IAM KV buckets.
|
||||
- **Writes:** `TagRole`/`UntagRole` (`spinifex/spinifex/handlers/iam/roles.go:521`, `:545`), `TagPolicy`/`UntagPolicy` (`handlers/iam/service_impl.go:1942`, `:1965`) and `TagInstanceProfile`/`UntagInstanceProfile` (`handlers/iam/instance_profiles.go:300`, `:325`) use CAS.
|
||||
`TagUser`/`UntagUser` (`handlers/iam/service_impl.go:1871`, `:1893`, `:1919`) and `TagOpenIDConnectProvider`/`UntagOpenIDConnectProvider` (`handlers/iam/oidc_providers.go:293`, `:311`, `:334`) read and then plain `Put`.
|
||||
- **Validation:** count, key and value length, pattern, `aws:` prefix and duplicate keys (`handlers/iam/service_impl.go:279`).
|
||||
|
||||
### Resources with no Spinifex tag path
|
||||
|
||||
- Lambda and CloudWatch Logs have no package under `spinifex/spinifex/`.
|
||||
- Bedrock (`gateway/bedrock*`, `domains/ochre`) has no tag action.
|
||||
- S3 bucket tagging is served by Predastore (`predastore/internal/gate/handlers/bucket_tagging.go`); Spinifex's `GetResources` does not list buckets.
|
||||
|
||||
### Cross-service tagging API
|
||||
|
||||
- `gateway/tagging.go:19` serves only `GetResources`; any other action, including `TagResources`, `UntagResources`, `GetTagKeys` and `GetTagValues`, returns `InvalidAction` (`gateway/tagging.go:41`).
|
||||
- Authorization evaluates `tagging:GetResources` against `*` (`spinifex/spinifex/gateway/tagging/authz.go:28`).
|
||||
- `GetResources` aggregates ELBv2 load balancers and target groups plus the EC2 central index (`gateway/tagging/GetResources.go:45`), then filters by tag, sorts by ARN and slices by an integer offset token (`:264`).
|
||||
- The layering ratchet records the two cross-owner imports as debt (two `gateway/tagging` rows in `architecture/layering/allowlist_test.go`), and `gateway/tagging` is a frozen legacy package (`architecture/layering/legacy_test.go`).
|
||||
|
||||
### Authorization against tags
|
||||
|
||||
- The policy evaluator allowlists five condition keys and none read a tag (`bluebottle/pkg/iampolicy/conditions.go`).
|
||||
- IAM `CreatePolicy` refuses any unsupported key at write time (`handlers/iam/service_impl.go:2496`), with a specific reason for `aws:RequestTag/` (`:2514`).
|
||||
- No code path reads a resource's tags to evaluate a request.
|
||||
|
||||
### Summary table
|
||||
|
||||
| Owner | Tag store | Service write API | Write concurrency | Limits enforced | ARN account checked | Listed by `GetResources` | Cross-owner reach |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| EC2 central index | `ec2-tags` KV | `CreateTags`/`DeleteTags` | CAS | `aws:` prefix only, at AWSGW | IDs are not ARNs; stored under caller account | yes | daemon fans out to record owners |
|
||||
| EC2 record owners (11 services) | own record | via daemon mirror | plain read then put | none | n/a | no (index only) | none |
|
||||
| EC2 instance | instance record and index | via owning daemon | ownership-gated | `aws:` at AWSGW | n/a | yes, via index | none |
|
||||
| ELBv2 | LB/TG/listener/rule records | `AddTags`/`RemoveTags` | LB, listener, rule plain put; TG CAS over a stale merge | empty key only | owner field compared | LB and TG only | `gateway/tagging` reads over NATS |
|
||||
| RDS | four record kinds | `AddTagsToResource`/`RemoveTagsFromResource` | CAS, 16 attempts | 50/128/256, `aws:` | yes, and region | no | writes `domains/rds` subnet and parameter group records |
|
||||
| ECS | five record kinds | `TagResource`/`UntagResource` | plain read then put | none | no | no | writes `domains/ecs/taskdefinition` records |
|
||||
| EKS | cluster meta, nodegroup | `TagResource`/`UntagResource` | CAS | none | no | no | none |
|
||||
| ECR | repo meta | `TagResource`/`UntagResource` | plain read then put, from the gateway process | empty key only | no | no | none |
|
||||
| ACM | certificate record | `AddTagsToCertificate`/`RemoveTagsFromCertificate` | revision-checked update of the tag field only | none | owner checked via `lookupOwned` | no | none |
|
||||
| IAM | five record kinds | `Tag*`/`Untag*` | CAS for role, policy, instance profile; plain put for user, OIDC provider | full | names, not ARNs, except policy and OIDC | no | none |
|
||||
|
||||
## 2. Error and atomicity behaviour
|
||||
|
||||
Each line states current behaviour and how it was established.
|
||||
"Test" names a `TestCurrentBehaviour_Tagging_*` test listed under [Tests](#tests); "code" means read from source and not tested here; "INFERRED" means a consequence of code that no test demonstrates.
|
||||
|
||||
### Read-modify-write and revision checks
|
||||
|
||||
- EC2 record mirror: a tag write and a non-tag record change that both land between the mirror's read and its put are reverted, and every call reports success.
|
||||
Test: `RecordMirrorRevertsInterleavedWrites`.
|
||||
Every record owner that routes through `tagmirror.UpdateKVRecordTags` shares this; volume and image use their own read-then-put with the same shape (INFERRED for those two).
|
||||
- EC2 central index: CAS, so concurrent merges on one resource keep each other's keys.
|
||||
Code, and the closed plan's tests.
|
||||
- ECS: the tag write reverts a record change committed between its read and its put.
|
||||
Test: `ECSTagWriteRevertsInterleavedRecordWrite` (an interleaved `DesiredCount` change is lost).
|
||||
- ACM: tag writes are revision-checked and change only the tag field; the stored private key survives (resolved by `mulga-ncn9f`).
|
||||
Test: `ACMTagWritesPreserveStoredPrivateKey`.
|
||||
A certificate that lost its key before the fix can no longer be served by any listener that loads it (INFERRED); recovery is `mulga-o436p`.
|
||||
- ELBv2: load balancer, listener and rule tag writes can revert a concurrent record change, and the target group CAS assigns a map merged from an earlier read, so two concurrent `AddTags` on one target group can lose a key.
|
||||
INFERRED from `service_impl_tags.go:41`, `:57`, `:74`, `:88`.
|
||||
- ECR, IAM user, IAM OIDC provider: read then plain put; concurrent tag writes can lose a key.
|
||||
INFERRED.
|
||||
- RDS, EKS, IAM role, policy and instance profile: revision-checked; code.
|
||||
|
||||
### Partial failure across resources in one call
|
||||
|
||||
- ELBv2 `AddTags` writes each ARN before resolving the next, so an unknown ARN after a valid one fails the call with the first resource already tagged.
|
||||
Test: `ELBv2AddTagsPartiallyAppliesBeforeUnknownARN`.
|
||||
- EC2 `CreateTags`/`DeleteTags`: the central loop stops at the first failing resource after committing earlier ones; the daemon then writes the record mirrors only if the central write succeeded, and a mirror error is returned after the central index is already committed.
|
||||
Code (`daemon/daemon_handlers_tag.go:98`); the resulting index/record divergence is INFERRED.
|
||||
- Instance IDs are tagged concurrently and the first error in input order is returned; other instances may already be tagged.
|
||||
Code (`daemon/daemon_handlers_tag.go:172`).
|
||||
|
||||
### Validation limits
|
||||
|
||||
- EC2: `aws:` refused at AWSGW with `InvalidParameterValue`; no count, key-length or value-length limit.
|
||||
Test: `EC2CreateTagsAppliesNoLengthOrCountLimits` (51 tags, a 129-character key and a 257-character value all stored).
|
||||
- ELBv2, ECS, ACM: no count, length or `aws:` rule.
|
||||
Tests: `ELBv2AddTagsAppliesNoTagLimits`, `ECSTagResourceAppliesNoTagLimits`, `ACMAddTagsAppliesNoTagLimits`.
|
||||
- ECR: empty key only; EKS: none.
|
||||
Code.
|
||||
- RDS and IAM: AWS limits enforced.
|
||||
Code.
|
||||
|
||||
### Unknown, nonexistent and cross-account resources
|
||||
|
||||
- EC2: a well-formed ID that names nothing, an ID with no recognised prefix, and another account's ID are all accepted and stored under the caller's account.
|
||||
Test: `EC2CreateTagsStoresUnresolvedIDsInCallerNamespace` (extends `mulga-6maav`).
|
||||
- `GetResources` then lists those IDs, an unrecognised one as `arn:aws:ec2:<region>:<account>:unknown/<id>`.
|
||||
Test: `GetResourcesListsUnresolvedEC2IDs`.
|
||||
- ECS: an ARN naming another account and region resolves to the caller's same-named resource and tags it.
|
||||
Test: `ECSForeignAccountARNTagsCallerResource`.
|
||||
- EKS and ECR discard the ARN account the same way.
|
||||
Code; the effect is INFERRED.
|
||||
- ELBv2: unknown or foreign ARNs return the per-type not-found code; RDS returns `InvalidParameterValue` for a foreign account or region.
|
||||
Code.
|
||||
|
||||
### Error codes
|
||||
|
||||
| Owner | Unknown resource | Bad tag | Store failure |
|
||||
| --- | --- | --- | --- |
|
||||
| EC2 | none (accepted) | `ErrorInvalidParameterValue` (`aws:` and empty key only) | `ErrorServerInternal` |
|
||||
| ELBv2 | `ErrorELBv2LoadBalancerNotFound`, `ErrorELBv2TargetGroupNotFound`, `ErrorELBv2ListenerNotFound`, `ErrorELBv2RuleNotFound` | `ErrorInvalidParameterValue` (empty key only) | `ErrorServerInternal` |
|
||||
| RDS | the kind's not-found fault (`tags.go:60`) | `ErrorInvalidParameterValue` | error passed through |
|
||||
| ECS | `ErrorECSClusterNotFound`, `ErrorECSServiceNotFound`, otherwise `ErrorECSInvalidParameter` | none | error passed through |
|
||||
| EKS | `ErrorEKSResourceNotFound` for clusters (`service_impl.go:1958`); other ARNs `ErrorNotImplemented` | none | error passed through |
|
||||
| ECR | `RepositoryNotFoundError` | `ErrorInvalidTagParameter` (empty key only) | error passed through |
|
||||
| ACM | `ErrorACMInvalidArn` or `ErrorResourceNotFound` (`service_impl.go:614`; also when the certificate is deleted between the ownership check and the write) | none | `ErrorInternalError` |
|
||||
| IAM | `ErrorIAMNoSuchEntity` | `validationError`, `ErrorIAMInvalidInput`, `ErrorIAMLimitExceeded` | error passed through |
|
||||
|
||||
Names are `awserrors` constants or package helpers.
|
||||
This table is from code.
|
||||
|
||||
### GetResources during and after writes
|
||||
|
||||
- A resource whose tags were all removed drops out of the result.
|
||||
Test: `GetResourcesOmitsResourceWhoseTagsWereAllRemoved`.
|
||||
- The pagination token is an integer offset into a set recomputed per call, so a resource tagged between page reads that sorts ahead of the offset makes the next page repeat an entry.
|
||||
Test: `GetResourcesOffsetTokenRepeatsAfterInterleavedWrite`.
|
||||
The same mechanism can skip an entry when a resource ahead of the offset is untagged (INFERRED).
|
||||
- A non-integer token returns `InvalidParameterValue`, and `ResourcesPerPage` of 500 is accepted.
|
||||
Test: `GetResourcesPaginationValidation`.
|
||||
- Only EC2 index entries and ELBv2 load balancers and target groups are aggregated; any other type filter returns an empty list without error.
|
||||
Test: `GetResourcesAggregatesOnlyEC2AndELBv2`.
|
||||
- ELBv2 load balancers and target groups with no tags are included, because ELBv2 `DescribeTags` returns one entry per ARN.
|
||||
Code (`handlers/elbv2/service_impl.go:3041`).
|
||||
|
||||
### Authorization
|
||||
|
||||
- No path supplies resource tags, request tags or tag keys to the policy evaluator.
|
||||
Code.
|
||||
- IAM `CreatePolicy` refuses policies conditioned on `aws:ResourceTag/`, `aws:RequestTag/`, `aws:TagKeys` and `ec2:ResourceTag/` with `MalformedPolicyDocument`.
|
||||
Test: `CreatePolicyRefusesTagConditionKeys` (links `mulga-5eh43`, `mulga-lmq4l`).
|
||||
|
||||
## 3. AWS parity
|
||||
|
||||
Each item states AWS behaviour first, then ours.
|
||||
Sources were fetched on 2026-10-13.
|
||||
|
||||
- **Tag limits.**
|
||||
AWS: 50 tags per resource, 128-character keys, 256-character values, keys and values case-sensitive, `aws:` reserved ([EC2 tag restrictions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions); [ECS TagResource](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TagResource.html); [EKS TagResource](https://docs.aws.amazon.com/eks/latest/APIReference/API_TagResource.html) gives 50 entries, key 1..128, value up to 256; [ECR TagResource](https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_TagResource.html) gives `TooManyTagsException` at 50 and `InvalidTagParameterException`; [ACM AddTagsToCertificate](https://docs.aws.amazon.com/acm/latest/APIReference/API_AddTagsToCertificate.html) gives 1..50 items, `InvalidTagException` for `aws:`, `TooManyTagsException`; [ELBv2 AddTags](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_AddTags.html) lists `TooManyTags` and `DuplicateTagKeys`).
|
||||
Ours: EC2, ELBv2, ECS, EKS, ECR and ACM do not enforce the count or lengths; ELBv2, ECS, EKS, ECR and ACM do not refuse `aws:`.
|
||||
Divergence, verified by the limit tests.
|
||||
- **EC2 nonexistent or malformed IDs.**
|
||||
AWS: the error table defines `InvalidID` as "The specified ID for the resource you are trying to tag is not valid" ([EC2 error codes](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html)).
|
||||
What AWS returns for a well-formed ID naming no resource on `CreateTags` is unverified.
|
||||
Ours: accepted and stored.
|
||||
The malformed-ID case is a divergence, verified by test.
|
||||
- **EC2 cross-account IDs.**
|
||||
AWS: "When you tag public or shared resources, the tags you assign are available only to your AWS account" ([EC2 tag restrictions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions)).
|
||||
AWS behaviour for an ID owned by another account and not shared with the caller is unverified.
|
||||
Ours stores under the caller in both cases, which matches the documented shared case and is unverified for the unshared case.
|
||||
- **EC2 consistency.**
|
||||
AWS: the EC2 API "follows an eventual consistency model" ([EC2 error codes, Eventual consistency](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html#api-eventual-consistency)).
|
||||
A lost tag write is not eventual consistency; the record-mirror revert is a defect regardless of this model.
|
||||
- **Multi-resource atomicity.**
|
||||
AWS: Resource Groups Tagging `TagResources` reports per-ARN failures in `FailedResourcesMap` ([TagResources](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_TagResources.html)), so partial success is part of that contract.
|
||||
Whether ELBv2 `AddTags` or EC2 `CreateTags` is all-or-nothing across resources is unverified.
|
||||
Ours: both apply partially before failing.
|
||||
- **GetResources content.**
|
||||
AWS: "`GetResources` does not return untagged resources", and without a tag filter "Resources that were previously tagged, but do not currently have tags, are shown with an empty tag set" ([GetResources](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html)).
|
||||
Ours drops previously tagged resources with no tags (test) and includes never-tagged ELBv2 load balancers and target groups (code).
|
||||
Both are divergences.
|
||||
- **GetResources coverage.**
|
||||
AWS: "All tagged resources, whether the tagging used the Resource Groups Tagging API or not, are returned" (same page).
|
||||
Ours returns EC2 and two ELBv2 types only.
|
||||
Divergence, verified by test.
|
||||
- **GetResources pagination.**
|
||||
AWS: `ResourcesPerPage` "minimum of 1 and a maximum value of 100"; invalid input is `InvalidParameterException`; a token is valid for at most 15 minutes, after which `PaginationTokenExpiredException` (same page).
|
||||
Ours accepts 500 and returns `InvalidParameterValue` for a bad token.
|
||||
Divergence, verified by test.
|
||||
AWS's page consistency under concurrent writes is unverified, so the repeat on interleaved writes is reported as current behaviour, not as a parity gap.
|
||||
- **TagResources and UntagResources.**
|
||||
AWS: requires `tag:TagResources` and the owning service's tagging permission, for example `ec2:CreateTags`; at most 20 ARNs and 50 tags per call (TagResources page).
|
||||
Ours: not served (code).
|
||||
- **Tag condition keys.**
|
||||
AWS: `aws:ResourceTag/<key>` compares the tag on the resource the request is made against; `aws:RequestTag/<key>` the tag passed in the request; `aws:TagKeys` the list of keys passed ([IAM global condition keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html)).
|
||||
Ours: refused at policy write and never evaluated.
|
||||
Divergence, verified by test; tracked by `mulga-5eh43` and `mulga-lmq4l`.
|
||||
- **ECS, EKS and ECR ARN with another account.**
|
||||
AWS error for that case is unverified.
|
||||
Ours acts on the caller's same-named resource (ECS by test, EKS and ECR by code).
|
||||
- **ACM private key.**
|
||||
No AWS documentation addresses this; tagging is metadata only on every AWS page fetched.
|
||||
The key-loss defect is resolved (`mulga-ncn9f`).
|
||||
|
||||
## 4. Proposal: the ADR-0001 structural cut
|
||||
|
||||
**Status:** Done.
|
||||
Steps 1 to 4 of the [migration order](#migration-order-and-exit-criteria) are implemented and recorded in `docs/PACKAGE_BOUNDARY_MIGRATION.md`.
|
||||
|
||||
ADR-0001 (accepted 2026-09-28) gives the rule this proposal works within: a consumer declares its own narrow in-process capability interface, an owner owns any versioned process contract under `contracts/<domain>/v1`, and a composition root wires them.
|
||||
The ADR's package map row for `tags` sends generic helpers to `foundation/aws/tags` and says each resource owner retains canonical tag state.
|
||||
|
||||
### Scope
|
||||
|
||||
Decided: `domains/tagging` is a stateless aggregator composed in the AWSGW process, not a new daemon.
|
||||
Decided: any owner wire contract belongs under that owner's `contracts/<owner>/v1`, and that there is no shared tagging wire format.
|
||||
|
||||
Inside the cut:
|
||||
|
||||
- `domains/tagging` owns the current `GetResources` aggregation path and nothing else.
|
||||
Today `gateway/tagging` serves only `GetResources` (`spinifex/spinifex/gateway/tagging.go:19`), so this is a structural move with no new API.
|
||||
- `domains/tagging` declares one read/list capability, enough to reach the EC2 tag index and ELBv2 without importing either owner.
|
||||
- The AWSGW composition root supplies one adapter per owner that implements that capability.
|
||||
- `GetResources` behaviour is unchanged, including every behaviour recorded in [Findings](#findings).
|
||||
|
||||
Outside the cut, listed under [Future direction](#future-direction-outside-the-adr-0001-structural-cut): pagination semantics, supported-resource scope, tag limits, `aws:` handling, ever-tagged tracking, tag-policy evaluation (`ResourceTagReader`), and every mutation API (`TagResources`, `UntagResources` and the per-service tag writes).
|
||||
|
||||
### Capability
|
||||
|
||||
`domains/tagging` is the consumer, so it declares the interface:
|
||||
|
||||
```go
|
||||
package tagging
|
||||
|
||||
// TaggedLister returns one owner's resources for GetResources aggregation.
|
||||
type TaggedLister interface {
|
||||
ListTagged(ctx context.Context, accountID string, typeFilters map[string]bool) ([]TaggedResource, error)
|
||||
}
|
||||
|
||||
type TaggedResource struct {
|
||||
ARN string
|
||||
Tags map[string]string
|
||||
}
|
||||
```
|
||||
|
||||
`typeFilters` keeps today's meaning: the lower-cased `ResourceTypeFilters` set, where an entry may be a full `service:type` or a bare service (`gateway/tagging/GetResources.go:206`).
|
||||
Each adapter applies it to the types it serves, as `listELBv2` and `listEC2` do now.
|
||||
Tag filtering, ARN sorting and the offset token stay in the aggregator unchanged.
|
||||
The interface carries no mutation, no limits and no ever-tagged flag; those belong to the future direction.
|
||||
|
||||
### Adapters and compile-time checks
|
||||
|
||||
The owners run in daemons and the aggregator runs in AWSGW.
|
||||
A daemon-side owner cannot satisfy a Go interface declared in the AWSGW process, so the capability is implemented by an AWSGW-side NATS client adapter.
|
||||
|
||||
- **EC2 index adapter:** wraps today's `listEC2` path, `ec2tags.NewNATSTagsService(...).DescribeTags` plus `buildEC2Mappings` (`gateway/tagging/GetResources.go:95`).
|
||||
It is constructed with the region, since EC2 IDs become ARNs there.
|
||||
- **ELBv2 adapter:** wraps today's `elbv2Resources` path, `DescribeLoadBalancers`, `DescribeTargetGroups` and batched `DescribeTags` through `handlers_elbv2.NewNATSELBv2Service` (`gateway/tagging/GetResources.go:106`).
|
||||
- **Placement:** the adapters live under the AWSGW composition root, `runtime/roles/awsgw`, which the ratchet already names as the role allowed to wire domain registrations (`architecture/layering/rules_test.go:17`) and which already imports `handlers/*` and `domains/*` packages (`runtime/roles/awsgw/awsgw.go:28` to `:46`).
|
||||
- **Compile-time checks:** `var _ tagging.TaggedLister = (*ec2IndexLister)(nil)` and `var _ tagging.TaggedLister = (*elbv2Lister)(nil)` sit beside the adapters in AWSGW.
|
||||
No owner package carries a check against a `domains/tagging` interface.
|
||||
- **Wire contracts:** both adapters reuse subjects and clients that exist today, so the cut defines no new wire contract.
|
||||
The remote owner of each subject owns its request/reply contract and handler; the EC2 `ec2.DescribeTags` and ELBv2 `elbv2.Describe*` subjects are where that contract would be formalised, under `contracts/ec2/v1` and `contracts/elbv2/v1`, if a later step needs it.
|
||||
- **Wiring:** `runtime/roles/awsgw` builds the aggregator with both adapters and hands it to the gateway, which today calls `gateway_tagging.GetResources` with its NATS connection (`gateway/tagging.go:21`).
|
||||
|
||||
### Ratchet rows removed
|
||||
|
||||
- `architecture/layering/allowlist_test.go` row `gateway/tagging` → `domains/ec2/tags`.
|
||||
- `architecture/layering/allowlist_test.go` row `gateway/tagging` → `handlers/elbv2`.
|
||||
- `architecture/layering/legacy_test.go` entry `gateway/tagging` as a frozen legacy package.
|
||||
- The `gateway/tagging` row of the `docs/PACKAGE_BOUNDARY_MIGRATION.md` "Close-out status" table and the "One narrow missing boundary" row of its "Remaining legacy roots by blocker class" matrix record `gateway/tagging` as open debt; both rows are updated when it clears.
|
||||
|
||||
`domains/tagging` starts with no allowlist rows and imports only `foundation` packages and the AWS SDK model types it already uses.
|
||||
|
||||
### Migration order and exit criteria
|
||||
|
||||
1. **Interfaces in `domains/tagging`, read/list only.**
|
||||
Exit: `TaggedLister` and `TaggedResource` exist, the package imports no domain, handler or contract package, and the ratchet classifies it with zero allowlist rows.
|
||||
Done in `0d830e901`.
|
||||
2. **EC2 index and ELBv2 adapters in AWSGW.**
|
||||
Exit: both adapters compile with their `var _ tagging.TaggedLister` checks, each has a test against embedded JetStream equivalent to today's `listEC2` and `elbv2Resources`, and nothing calls them yet.
|
||||
Done in `f3545e3ea`; type matching is exported once as `tagging.MatchesType`.
|
||||
3. **Move `GetResources` aggregation from `gateway/tagging` to `domains/tagging`.**
|
||||
The request parsing, type and tag filtering, sorting, offset token and `authz.go` move; the gateway dispatch in `gateway/tagging.go` calls the aggregator wired by AWSGW.
|
||||
Exit: the `TestCurrentBehaviour_Tagging_GetResources*` tests and the existing `gateway/tagging` unit tests pass with unchanged assertions after moving with the code.
|
||||
Done in `aedd0b2c4`: AWSGW sets `GatewayConfig.TaggingResources`, the `GetResources` characterisation tests run in `runtime/roles/awsgw`, and the aggregator unit tests run in `domains/tagging`.
|
||||
4. **Remove the ratchet rows.**
|
||||
Exit: the two `gateway/tagging` allowlist rows and its `legacy_test.go` frozen-legacy entry are deleted, the `gateway/tagging` package is gone, `make preflight`'s layering tests pass, and the "Close-out status" `gateway/tagging` row and the "One narrow missing boundary" matrix row record the debt as cleared.
|
||||
Done: the package and ratchet rows went with `aedd0b2c4`, and the migration record now marks `gateway/tagging` cleared.
|
||||
|
||||
The RDS and ECS record-writing edges in the `docs/PACKAGE_BOUNDARY_MIGRATION.md` "Dependency-rule debt to clear after the moves" list are not cleared by this cut.
|
||||
They need owner tag mutation capabilities, which are future direction.
|
||||
|
||||
## Future direction (outside the ADR-0001 structural cut)
|
||||
|
||||
Nothing in this section is part of the ADR-0001 track.
|
||||
It records a possible shape for later work so the cut above does not block it.
|
||||
|
||||
### Owner tag mutation
|
||||
|
||||
A later `domains/tagging` could declare a mutation capability:
|
||||
|
||||
```go
|
||||
type Tagger interface {
|
||||
ResourceTypes() []string // "service:type", e.g. "ec2:security-group"
|
||||
TagResource(ctx context.Context, accountID, arn string, add map[string]string) error
|
||||
UntagResource(ctx context.Context, accountID, arn string, keys []string) error
|
||||
ListTags(ctx context.Context, accountID, arn string) (map[string]string, error)
|
||||
}
|
||||
```
|
||||
|
||||
For owners in another process, the same model as the cut applies: the owner defines a request/reply contract under `contracts/<owner>/v1` and serves it, and an AWSGW-side adapter implements `Tagger` and carries the `var _ tagging.Tagger` check.
|
||||
An owner in the same process as its consumer can implement the interface directly.
|
||||
|
||||
Errors would be sentinels declared in `domains/tagging` (`ErrNotFound`, `ErrInvalidARN`, `ErrInvalidTag`, `ErrTooManyTags`, `ErrConflict`), wrapping the owner's own `awserrors` code so per-service APIs keep today's codes.
|
||||
|
||||
Guarantees each owner would give:
|
||||
|
||||
1. Per-resource atomic: validate first, then one revision-checked write that changes only the tag field, retried on conflict with the mutation re-applied to the fresh record.
|
||||
2. Existence and ownership: the ARN's partition, region and account must match the caller, and the resource must exist, before any write.
|
||||
3. Limits from `foundation/aws/tags` with the owner's documented variations.
|
||||
4. A tag write never rewrites a non-tag field.
|
||||
|
||||
No owner would be asked for cross-resource atomicity.
|
||||
|
||||
### Work that would use it
|
||||
|
||||
- `TagResources` and `UntagResources` in `domains/tagging`, which are an API expansion, reporting per-ARN failures in `FailedResourcesMap`.
|
||||
- Clearing the RDS subnet-group and parameter-group edges and the ECS task-definition edge (the `handlers/rds` and `handlers/ecs` legacy edges in the `docs/PACKAGE_BOUNDARY_MIGRATION.md` "Dependency-rule debt to clear after the moves" list) by calling owner mutation capabilities.
|
||||
- ACM, EKS, ECR and IAM owners; ACM's key-loss defect is already fixed (`mulga-ncn9f`).
|
||||
- EC2 record owners, once the canonical EC2 tag store is decided (`mulga-36wn6`, `mulga-6maav`).
|
||||
- `foundation/aws/tags` validation helpers shared by every owner.
|
||||
- `GetResources` pagination, supported-resource scope and ever-tagged tracking (`mulga-t6cuv`).
|
||||
|
||||
### Tag-policy evaluation
|
||||
|
||||
The policy door would declare its own read-only capability:
|
||||
|
||||
```go
|
||||
type ResourceTagReader interface {
|
||||
ResourceTags(ctx context.Context, accountID, arn string) (map[string]string, bool, error)
|
||||
}
|
||||
```
|
||||
|
||||
`aws:RequestTag` and `aws:TagKeys` come from the parsed request and need no owner call.
|
||||
This depends on `mulga-lmq4l` and `mulga-5eh43`.
|
||||
|
||||
## Open decisions
|
||||
|
||||
These need an owner's decision and are not settled by this document.
|
||||
None of them blocks the ADR-0001 cut.
|
||||
|
||||
1. **EC2 canonical store.** Whether the `ec2-tags` index becomes the canonical store for every EC2 type, or records stay canonical and the index becomes a projection each owner maintains.
|
||||
2. **Pagination token format and lifetime.** Keyset token versus snapshot, and whether to mirror AWS's 15-minute expiry.
|
||||
3. **Cross-account ARN behaviour** for ECS, EKS and ECR, given that AWS behaviour for these is unverified here.
|
||||
4. **Ever-tagged tracking.** Whether owners must remember that a resource was once tagged, to match `GetResources` returning `"Tags": []`.
|
||||
5. **Which `TagResources` permission check runs where.** The gateway door, the aggregator, or each owner.
|
||||
6. **Internal callers and `aws:` keys.** EC2 refuses `aws:` only at AWSGW; whether internal writers (system VPC, EKS controllers) may keep writing reserved keys through a mutation capability.
|
||||
|
||||
Decided by the user and removed from this list: where `domains/tagging` runs (AWSGW, stateless) and the process contract shape (owner contracts only, no shared format).
|
||||
|
||||
## Findings
|
||||
|
||||
Current behaviour only; none is fixed except finding 1, marked resolved.
|
||||
|
||||
| # | Finding | Evidence | Bead |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Resolved: ACM tag writes erased the stored private key; they now preserve it. Certificates tagged before the fix may have lost their key and need re-import or re-issuance. | Test: `ACMTagWritesPreserveStoredPrivateKey` | `mulga-ncn9f` (P1, closed); remediation `mulga-o436p` (P1, open) |
|
||||
| 2 | EC2 record mirrors revert concurrent tag and non-tag writes. | Test: `RecordMirrorRevertsInterleavedWrites` | `mulga-36wn6` (P1) |
|
||||
| 3 | ECS tag writes revert concurrent record writes, including a `DesiredCount` change. | Test: `ECSTagWriteRevertsInterleavedRecordWrite` | `mulga-c0d5x` (P1) |
|
||||
| 4 | ECS acts on the caller's resource for an ARN naming another account. | Test: `ECSForeignAccountARNTagsCallerResource`; EKS and ECR INFERRED | `mulga-k8d9b` (P2) |
|
||||
| 5 | EC2 stores tags for unresolved and other-account IDs. | Test: `EC2CreateTagsStoresUnresolvedIDsInCallerNamespace` | `mulga-6maav` |
|
||||
| 6 | EC2, ELBv2, ECS and ACM enforce no tag count or length limits. | Tests: `EC2CreateTagsAppliesNoLengthOrCountLimits`, `ELBv2AddTagsAppliesNoTagLimits`, `ECSTagResourceAppliesNoTagLimits`, `ACMAddTagsAppliesNoTagLimits`; EKS and ECR code | none |
|
||||
| 7 | ELBv2 `AddTags` partially applies before an unknown ARN. | Test: `ELBv2AddTagsPartiallyAppliesBeforeUnknownARN` | `mulga-liyz5` (P2) |
|
||||
| 8 | `GetResources` lists unresolved EC2 IDs, including type `unknown`. | Test: `GetResourcesListsUnresolvedEC2IDs` | `mulga-t6cuv` (P2), `mulga-6maav` |
|
||||
| 9 | `GetResources` drops resources whose tags were all removed. | Test: `GetResourcesOmitsResourceWhoseTagsWereAllRemoved` | `mulga-t6cuv` (P2) |
|
||||
| 10 | `GetResources` offset token repeats an entry after an interleaved write. | Test: `GetResourcesOffsetTokenRepeatsAfterInterleavedWrite` | `mulga-t6cuv` (P2) |
|
||||
| 11 | `GetResources` accepts page sizes over 100 and returns `InvalidParameterValue` for a bad token. | Test: `GetResourcesPaginationValidation` | `mulga-t6cuv` (P2) |
|
||||
| 12 | `GetResources` covers only EC2 and ELBv2 load balancers and target groups. | Test: `GetResourcesAggregatesOnlyEC2AndELBv2` | `mulga-t6cuv` (P2) |
|
||||
| 13 | `GetResources` includes untagged ELBv2 load balancers and target groups. | Code | `mulga-t6cuv` (P2) |
|
||||
| 14 | `TagResources`, `UntagResources`, `GetTagKeys`, `GetTagValues` are not served. | Code | none |
|
||||
| 15 | Tag condition keys are refused at policy write and never evaluated. | Test: `CreatePolicyRefusesTagConditionKeys` | `mulga-5eh43`, `mulga-lmq4l` |
|
||||
| 16 | ELBv2 target group tag CAS merges from a stale read; LB, listener and rule tag writes are whole-record puts. | INFERRED | none |
|
||||
| 17 | ECR, IAM user and IAM OIDC provider tag writes are read then plain put. | INFERRED | none |
|
||||
| 18 | The EC2 daemon writes the central index before record mirrors, and a mirror failure leaves them divergent. | Code; divergence INFERRED | none |
|
||||
| 19 | Creation-time tags reach the central index only for instance, vpc/subnet/sg/eni and key pair. | Code; matches `creation-time-tags-describetags.md` | none |
|
||||
| 20 | `launchtemplate` and `placementgroup` tag writes reach only the central index. | Code | none |
|
||||
|
||||
`mulga-t6cuv` groups the `GetResources` parity gaps; whether it covers findings 8 and 13 is for its owner to confirm.
|
||||
|
||||
## Tests
|
||||
|
||||
Each asserts current behaviour:
|
||||
|
||||
| Test | File |
|
||||
| --- | --- |
|
||||
| `TestCurrentBehaviour_Tagging_RecordMirrorRevertsInterleavedWrites` | `spinifex/spinifex/domains/ec2/tagmirror/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_EC2CreateTagsStoresUnresolvedIDsInCallerNamespace` | `spinifex/spinifex/daemon/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_EC2CreateTagsAppliesNoLengthOrCountLimits` | `spinifex/spinifex/gateway/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_GetResourcesListsUnresolvedEC2IDs` | `spinifex/spinifex/runtime/roles/awsgw/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_GetResourcesOmitsResourceWhoseTagsWereAllRemoved` | same |
|
||||
| `TestCurrentBehaviour_Tagging_GetResourcesOffsetTokenRepeatsAfterInterleavedWrite` | same |
|
||||
| `TestCurrentBehaviour_Tagging_GetResourcesPaginationValidation` | same |
|
||||
| `TestCurrentBehaviour_Tagging_GetResourcesAggregatesOnlyEC2AndELBv2` | same |
|
||||
| `TestCurrentBehaviour_Tagging_ELBv2AddTagsPartiallyAppliesBeforeUnknownARN` | `spinifex/spinifex/handlers/elbv2/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_ELBv2AddTagsAppliesNoTagLimits` | same |
|
||||
| `TestCurrentBehaviour_Tagging_ECSTagWriteRevertsInterleavedRecordWrite` | `spinifex/spinifex/handlers/ecs/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_ECSForeignAccountARNTagsCallerResource` | same |
|
||||
| `TestCurrentBehaviour_Tagging_ECSTagResourceAppliesNoTagLimits` | same |
|
||||
| `TestCurrentBehaviour_Tagging_ACMTagWritesPreserveStoredPrivateKey` | `spinifex/spinifex/domains/acm/tagging_current_behaviour_test.go` |
|
||||
| `TestCurrentBehaviour_Tagging_ACMAddTagsAppliesNoTagLimits` | same |
|
||||
| `TestCurrentBehaviour_Tagging_CreatePolicyRefusesTagConditionKeys` | `spinifex/spinifex/handlers/iam/tagging_current_behaviour_test.go` |
|
||||
|
||||
The gateway and AWSGW tests serve the EC2 tag subjects from a real `TagsServiceImpl` without the daemon's record-mirror fan-out, so they cover the central index only.
|
||||
The interleaving tests wrap the KV handle to run one write between the handler's read and its write; they show the window exists, not how often real nodes hit it.
|
||||
@@ -51,9 +51,10 @@ services:
|
||||
# forwards to vpcd / viperblockd / predastore.
|
||||
#
|
||||
# Source layout: spinifex/runtime/roles/spinifex (entry), spinifex/daemon
|
||||
# (subscribe registration), spinifex/domains/ec2/<resource> and spinifex/handlers/{ecs,elbv2,iam,eks} (per-
|
||||
# subject handler bodies). Selector treats these paths as belonging
|
||||
# to this service via the `additional_paths` field.
|
||||
# (subscribe registration), spinifex/domains/<domain> and the legacy
|
||||
# spinifex/handlers/{ecs,eks,elbv2,iam,rds} packages (per-subject handler
|
||||
# bodies). Selector treats these paths as belonging to this service via
|
||||
# the `additional_paths` field.
|
||||
additional_paths:
|
||||
- spinifex/daemon
|
||||
- spinifex/domains/ec2/account
|
||||
@@ -148,7 +149,7 @@ services:
|
||||
- "elbv2.CreateListener"
|
||||
- "elbv2.DeleteListener"
|
||||
- "elbv2.DescribeListeners"
|
||||
# ACM-alike certificate store (handlers/acm/service_nats.go topic map).
|
||||
# ACM-alike certificate store (domains/acm; subscribed in daemon.go).
|
||||
- "acm.ImportCertificate"
|
||||
- "acm.DescribeCertificate"
|
||||
- "acm.ListCertificates"
|
||||
@@ -426,7 +427,7 @@ suites:
|
||||
e2e-ecr:
|
||||
path: tests/e2e/ecr
|
||||
covers: [awsgw, spinifex_daemon, predastore]
|
||||
# ECR control plane (handlers/ecr, gateway/ecr*) + OCI data plane: blobs and
|
||||
# ECR control plane (domains/ecr, gateway/ecr*) + OCI data plane: blobs and
|
||||
# manifests in predastore, control + auth bridge in awsgw, KV via
|
||||
# spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
|
||||
# client binary or registry-host resolution is absent.
|
||||
|
||||
@@ -184,7 +184,7 @@ fstype_of() {
|
||||
# weights_device_ordinal maps this member's BEDROCK_WEIGHTS_DEVICE (/dev/sdf,
|
||||
# /dev/sdg, ...) to its 0-based position among the bundle's members: sdf -> 0,
|
||||
# sdg -> 1, and so on, mirroring nthWeightsDevice's own 'f'+n scheme
|
||||
# (handlers/bedrock/launch.go) exactly. Named-letter arithmetic, not ASCII
|
||||
# (spinifex/domains/ochre/launch.go) exactly. Named-letter arithmetic, not ASCII
|
||||
# math: POSIX sh has no portable char-to-int builtin, and a bundle never
|
||||
# carries more than a handful of co-served members, so the alphabet is not a
|
||||
# real ceiling to enumerate by hand. Prints the ordinal and returns 0, or
|
||||
|
||||
@@ -61,8 +61,8 @@ git show <commit>^:spinifex/migrate/003_ipam_purpose.go
|
||||
|
||||
## Framework
|
||||
|
||||
The framework (`migrate.go`, `version_readers.go`) covers three kinds of target:
|
||||
The framework covers three kinds of target, split between two packages:
|
||||
|
||||
- `RunKV` is called from service-startup paths to stamp NATS KV bucket versions.
|
||||
- `RunConfig` / `RunAllConfig` / `PendingConfig` back `spx admin upgrade`, which `scripts/setup.sh` runs with `--yes` while the services are stopped.
|
||||
- `RunKV` (`spinifex/foundation/state/migrate/kv.go`) is called from service-startup paths to stamp NATS KV bucket versions.
|
||||
- `RunConfig` / `RunAllConfig` / `PendingConfig` (this package's `migrate.go` and `version_readers.go`) back `spx admin upgrade`, which `scripts/setup.sh` runs with `--yes` while the services are stopped.
|
||||
- `RunObject` migrates object-store data, stamping its version in a caller-supplied JetStream KV bucket. The stamp is a read-then-write, not a compare-and-swap, so two nodes can race to run the same step — every `ObjectMigration.Run` must be safe to re-execute.
|
||||
|
||||
Reference in New Issue
Block a user