diff --git a/docs/PACKAGE_BOUNDARY_MIGRATION.md b/docs/PACKAGE_BOUNDARY_MIGRATION.md index 44bf2b79b..3e8e11f72 100644 --- a/docs/PACKAGE_BOUNDARY_MIGRATION.md +++ b/docs/PACKAGE_BOUNDARY_MIGRATION.md @@ -385,6 +385,34 @@ These residuals were recorded by individual slices and must be resolved once the - Instance adapter consumers outside EC2: `gateway/rds/enginecatalog.go`, `daemon/eks_cp_control.go`, `daemon/rds_deps.go` and `handlers/quota/vcpu.go` call `domains/ec2/awsapi/instance` directly; each needs an EC2 capability rather than the AWS action adapter. - Merge-time follow-up in the umbrella repository: the `Makefile` `invariants-check` target, the `docs-pr.yml` clause cross-check, `.claude/hooks/architecture-traps.sh` and a `check-invariants.sh` comment hard-code `spinifex/spinifex/network/invariants`. They must change with the pointer bump that brings in this branch, or the docs-pr cross-check silently skips. Live specs and proposals citing `spinifex/network/` will report unresolved citations until updated. +## ADR-0004 source inventory and first-resource ranking + +The per-file inventories for ECS, EKS and RDS are in `docs/package-boundary/adr-0004-inventory-{ecs,eks,rds}.md`. +Each classifies every production file as resource owner, cross-resource orchestration, AWS protocol adapter, guest/controller wire contract, or temporary residue, and maps actions, durable records, identity and scope, lifecycle paths, NATS subjects and boundary-crossing imports. +They were produced by reading the code at `6c50349bd` and are not verified against a live system. + +Observed gaps common to all three: no resource carries a desired generation, all resources of a service share one per-account KV bucket, and deletes remove records before dependent cleanup is confirmed. + +First-resource ranking: + +1. RDS DB subnet group: about 340 lines with its own `db-subnet-groups/{name}` keys, synchronous AWS semantics, and no quota, expected-node, IAM, Bluebottle or agent dependency. + It has a real deletion graph, because delete refuses while an instance references the group, so it exercises a consumer-owned capability in both directions: an instance-owned "instances referencing this group" query for delete, and a subnet-group-owned placement query for instance create. +2. ECS task definition: own keys and good tests, but `RegisterTaskDefinition` allocates revisions non-atomically and the gateway adapter carries the `iam:PassRole` checks, so it sits closer to a blocked boundary. +3. EKS access entry: own keys, but entries are read during token review, which places it next to EKS identity. + +Blocked dependencies recorded, not designed around: + +- RDS instance: quota admission, the duplicated expected-node fan-out (`gateway/rds/enginecatalog.go`, `daemon/rds_deps.go`), IAM ensurer and Bluebottle master key. +- RDS `DescribeOrderableDBInstanceOptions`: the EC2 instance-type-availability projection. +- ECS task, service and capacity: `iam:PassRole` in the gateway adapter, IAM `ecsInstanceRole`, and EC2/ELBv2/EIP adapters called as internal APIs. +- EKS nodegroup and cluster: IAM role and instance-profile creation, Bluebottle `auth`, the NATS CA, and EKS identity (token review, OIDC). +- Guest agents: `agents/ecs`, `agents/rds` and `agents/eks/tokenwebhook` import the service implementation packages, which ADR-0004 S5 forbids; they need wire contracts. + +Findings to resolve before extraction: + +- The branch is 59 commits behind `dev`, including `ModifyDBSubnetGroup` (#1152) and the EKS stub unregistration (#1172), so the subnet-group inventory must be redone after `dev` is merged in. +- `ClusterMeta.ControlPlaneTemplate` (`handlers/eks/service_impl.go`) clears credentials but keeps `OIDCPrivateKeyPEM` and the k3s `JoinToken`, so both persist in the cluster KV record, which the gateway also reads; whether that record is encrypted at rest is unconfirmed. + ## Recording rule For every later slice, add the old and new path, source commit, focused diff --git a/docs/package-boundary/adr-0004-inventory-ecs.md b/docs/package-boundary/adr-0004-inventory-ecs.md new file mode 100644 index 000000000..c7150dc69 --- /dev/null +++ b/docs/package-boundary/adr-0004-inventory-ecs.md @@ -0,0 +1,388 @@ +# ADR-0004 source inventory — ECS + +Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`. +It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it. +Items marked "verify" or "uncertain" are unconfirmed. +Gaps record observed behaviour, not target behaviour. + + +Repo: `spinifex` submodule, branch `refactor/spinifex-package-boundaries`; paths below are relative to `spinifex/spinifex/` unless stated otherwise. +Binding inputs: ADR-0001..0004 and Q-78 at `origin/main`. +This inventory reports what the code does; it accepts or closes no gap. +"Verify" marks claims about AWS behaviour, or runtime behaviour, that a test or live system has not confirmed here. + +## 1. File table + +Totals: `handlers/ecs` is 7,760 production lines over 28 files, `gateway/ecs` + `gateway/ecs.go` is 1,236 lines, and `daemon/ecs_deps.go` + `accountteardown/reapers_ecs.go` is 132 lines. + +| File | Lines | Class | Reason / split | +|---|---|---|---| +| `handlers/ecs/bus/messages.go` | 173 | guest/controller wire contract (part temporary residue) | `Assign`, `AssignContainer`, `StopDirective`, `TaskState`, `ContainerStatus`, `InstanceCapacity`, `PortMapping` and `SystemControl` are live payloads of the agent and gateway path. `RegisterInstance` (:19) and `Heartbeat` (:33) have no production publisher, so they are residue. Split: live types go to a versioned `contracts/ecs/v1`; the residue types are deleted once the scheduler bus subscriptions are removed. | +| `handlers/ecs/bus/subjects.go` | 45 | temporary residue (prerequisite: drop scheduler bus subs; removal: delete unused builders) | `RegisterSubject`, `HeartbeatSubject`, `AssignSubject` and `TaskStateSubject` (:19-38) have no production publisher. `ServiceReconcileSubject` (:43) is live and is an internal cross-node wake owned by the service reconcile. | +| `handlers/ecs/capacity.go` | 223 | cross-resource orchestration (container-instance capacity provisioning) | `ProvisionCapacity` (:70) is a non-AWS action. It composes an IAM system role, EC2 AMI lookup and EC2 `RunInstances`, and records nothing durable in ECS. | +| `handlers/ecs/capacity_providers.go` | 233 | resource owner (`capacityprovider`), mixed | Create, Describe and Delete belong to `capacityprovider`. `PutClusterCapacityProviders` (:16-40) mutates the cluster record and belongs to `cluster`. | +| `handlers/ecs/cluster_teardown.go` | 171 | mixed: cross-resource orchestration + resource owner (`containerinstance`) | `DeleteCluster` (:24-53) is a cascade over every child, which is orchestration. `DeregisterContainerInstance` (:60), `UpdateContainerInstancesState` (:103), `instanceActiveTasks` (:143) and `drainInstanceServiceTasks` (:160) are container-instance commands that also stop tasks. Split three ways: cluster delete orchestration, container-instance owner, and task-stop calls through a task capability. | +| `handlers/ecs/conv.go` | 291 | mixed | `ClusterShortName`, `ContainerInstanceShortID`, `TaskShortID` and `CapacityProviderShortName` (:27-60) are per-resource identity parsers that the gateway authz also uses. The `containerDefsFromAWS`, `runtimePlatformFromAWS` and `ContainerDef.toAWS` group (:87-265) is the `taskdefinition` projection. `toAssignContainer` (:268) is wire-contract projection. | +| `handlers/ecs/deployments.go` | 267 | resource owner (`service`) | This is pure deployment, rollout and circuit-breaker logic on `ServiceRecord`, with no KV I/O. | +| `handlers/ecs/eni.go` | 155 | cross-resource orchestration (task realization, network capability) | The consumer-owned `eniController` interface (:37) belongs with `task`. `natsENIController` (:46-128) binds it to EC2 subjects and belongs at the composition root. | +| `handlers/ecs/gpu_report.go` | 63 | resource owner (`task`), mixed with wire contract | `ReportTaskGPU` (:31) writes the task record. `ReportTaskGPUInput`, `ContainerGPUReport` and `ReportTaskGPUOutput` (:10-25) are agent wire shapes. | +| `handlers/ecs/instancerole.go` | 48 | cross-resource orchestration (identity for container instances) | It ensures and converges the `ecsInstanceRole` through `handlers_iam` (:37-48). | +| `handlers/ecs/instances.go` | 532 | mixed: resource owner (`containerinstance`) + `task` + orchestration | Register, Describe, List, `upsertInstance`, `recordRegister` and `recordHeartbeat` (:19-357) belong to `containerinstance`. `recordTaskState` (:361) and `SubmitTaskStateChange` (:445) belong to `task`. `releaseReservation` (:505) is placement against the instance record. `recordDeploymentFailure` (:481) belongs to `service`. | +| `handlers/ecs/netcfg.go` | 78 | resource owner (`task`) | It resolves the network mode and parses awsvpc input for task launch. | +| `handlers/ecs/placement.go` | 144 | cross-resource orchestration (placement) | It contains pure fit and strategy selection over container-instance capacity, with no I/O. | +| `handlers/ecs/poll.go` | 107 | guest/controller wire contract | It holds the agent inbox drain and ack (`PollAssignments` :37), and the task-owned inbox reclaim (:93-107). | +| `handlers/ecs/records.go` | 525 | mixed (all resources' records) | It holds the status constants, ARN builders and every record type for every resource. Split each record, its ARN builder and its statuses into the owning resource. | +| `handlers/ecs/scheduler.go` | 449 | cross-resource orchestration + temporary residue | Lease, service reconcile loop, instance reaper and role converge are orchestration. `onRegister`, `onHeartbeat` and `onTaskState` (:236-320) are residue because nothing publishes those subjects. | +| `handlers/ecs/service.go` | 706 | mixed: facade + `cluster` + `taskdefinition` + shared KV helpers | `ECSService` (:27-81) is an AWS-shaped facade that also serves as an internal API. `Deps` (:98) is composition. Cluster CRUD (:231-411) belongs to `cluster`. Task-definition CRUD and `ParseTaskDefRef` (:416-706) belong to `taskdefinition`. `getJSON`, `putJSON` and `keysWithPrefix` (:164-225) are shared persistence helpers. | +| `handlers/ecs/service_nats.go` | 177 | AWS protocol adapter (gateway→daemon NATS hop) | `NATSECSService` forwards 34 methods onto `ecs.`. It is also used as an internal API by `accountteardown` and by gateway PassRole resolution. | +| `handlers/ecs/services.go` | 786 | mixed: resource owner (`service`) + orchestration bindings | Service CRUD and reconcile (:128-665) belong to `service`. `targetRegistrar` / `natsTargetRegistrar` (:28-59, ELBv2) and `eipManager` / `natsEIPManager` (:64-121, EC2 EIP) are consumer-owned capabilities, but their NATS bindings belong at the composition root. `listTaskRecords` (:605) reads task records directly. | +| `handlers/ecs/store.go` | 227 | temporary residue (shared account bucket; prerequisite: decide per-owner key namespaces; removal: key helpers move to owners) | One bucket holds the keys of all six resources (:36-150). It also holds the leader bucket factory (:221) and the account enumeration used by every sweep (:186). | +| `handlers/ecs/tags.go` | 277 | cross-resource orchestration (tag dispatch) | It parses the ARN and then reads and writes each resource's record directly (:76-203). It should become one tag command per owner behind ARN dispatch. | +| `handlers/ecs/task_stop.go` | 301 | mixed: resource owner (`task`) + orchestration | `StopTask`, `StartTask`, `reserveOnInstance`, `requestStopTask` and `forceStopTask` (:28-209) belong to `task`, with placement mixed into `reserveOnInstance`. `applyServiceTargets` (:235) and `assignTaskPublicIP` / `releaseTaskPublicIP` (:264-301) are ELBv2 and EIP orchestration. | +| `handlers/ecs/tasks.go` | 438 | mixed: resource owner (`task`) + placement orchestration + wire | `RunTask`, Describe, List and the record/projection code belong to `task`. `reservePlacement` (:180) does a CAS on the container-instance record. `publishAssign` (:249) writes the agent inbox. | +| `handlers/ecs/tasks_gc.go` | 130 | resource owner (`task`) reconciliation | It retains STOPPED tasks, retries owed ENI releases and prunes old tasks. It is hosted on `Scheduler` (:21). | +| `handlers/ecs/userdata.go` | 58 | guest/controller wire contract | It writes the cloud-config that seeds the agent env (`ECS_GATEWAY_URL`, `ECS_GATEWAY_CA`, `ECS_REGION`, `ECS_CLUSTER`) (:31-58). | +| `gateway/ecs.go` | 80 | AWS protocol adapter | `ECS_Request` (:26) does target parsing, the scope gate (:54-60), the PassRole closure (:65-67) and response encoding. | +| `gateway/ecs/handler.go` | 165 | AWS protocol adapter | It holds the authoritative action table (:50-112), `StubbedActionNames` (:115), the raw-JSON action list (:131) and the JSON 1.1 writers. | +| `gateway/ecs/actions.go` | 452 | AWS protocol adapter (mixed: PassRole resolution) | Each action decodes and forwards over NATS. `checkTaskLaunchRoles` (:44) queries `DescribeTaskDefinition` over the ECS NATS facade to find role ARNs. | +| `gateway/ecs/authz.go` | 350 | AWS protocol adapter | The per-action resource-ARN scope table (:57-124) and resolver (:139-350) are built on the `handlers_ecs` parsers and ARN builders. | +| `gateway/ecs/azrebalancing.go` | 103 | AWS protocol adapter | It rejects `availabilityZoneRebalancing=ENABLED` (:28) and adds the SDK-gap field to service responses (:54). | +| `gateway/ecs/clienttoken.go` | 86 | AWS protocol adapter holding durable retry state (flag) | It holds the RunTask ClientToken store `spinifex-ecs-clienttokens` (:22). ADR-0003 S3 says an adapter does not own retry bookkeeping, so this belongs with `task`. | +| `daemon/ecs_deps.go` | 63 | composition root (nearest class: cross-resource orchestration) | It binds IAM, images, `RunWorkerInstance` and the gateway URL/CA into `handlers_ecs.Deps` (:17-63). | +| `daemon/daemon.go` (ECS sections only) | n/a | AWS protocol adapter transport + composition | Import at :74, fields at :168-169, 34 `ecs.` subscriptions at :1131-1167, construction at :2023-2043. | +| `accountteardown/reapers_ecs.go` | 69 | cross-resource orchestration (account teardown) | It has one cluster reaper that calls the ECS AWS facade over NATS (:20-58). | +| `gateway/operations.go` (:7, :30-32) | n/a | AWS protocol adapter | It registers the ECS operation inventory from `gateway_ecs.Actions` and `StubbedActionNames`. | +| `gateway/gateway.go` (:301, :403, :523-524, :639) | n/a | AWS protocol adapter | Service routing to `ECS_Request` and the `ecs-tasks.amazonaws.com` PassRole principal constant live here. | + +Other ECS-aware code outside scope, with no import of ECS: +`handlers/sts/assume_role.go:47-55,350-355` duplicates the `ecsInstanceRole` name and the `ecs-tasks.amazonaws.com` principal, and documents that it cannot import `handlers_ecs`. +`foundation/aws/tags/tags.go:24` defines `ManagedByECS` for AMI resolution. +`operator/imagecatalog/images.go:611-637` defines the `spinifex-ecs-node` and `spinifex-ecs-node-gpu` system AMIs. +`domains/acm/renewal.go:169` is a comment-only reference. +`../internal/ecsgw/client.go` is the agent's SigV4 gateway client, with target prefix at :29 and :101. + +Guest agent counterparty (`agents/ecs/agent/`), not inventoried file by file: +It reaches the control plane only through signed gateway HTTP (`controlplane.go:18-35`), not NATS. +It imports both `handlers/ecs/bus` and the `handlers_ecs` implementation package (`registration.go:37-38`, `controlplane.go:128,142,150`, `taskrun.go:416-419`). +That is the ADR-0004 S5 debt: the agent imports a domain implementation package to get wire types. + +## 2. Served AWS actions and protocol + +Protocol: AWS JSON 1.1, with the action taken from the `X-Amz-Target` suffix (`gateway/ecs.go:15-20`, any prefix accepted). +Requests use the canonical prefix `AmazonEC2ContainerServiceV20141113` (`gateway/ecs/handler.go:26`). +Responses are encoded with `jsonutil` (epoch-second times), except `PollAssignments`, which uses `encoding/json` (RFC3339 times) (`handler.go:131-157`). +Errors are `awserrors` codes rendered by the shared ErrorHandler. +An action missing from the table returns `InvalidAction` (`gateway/ecs.go:32-36`), and a stub returns `NotImplemented` (501) (`handler.go:43-45`). +Request order is: account check (:40-46), bounded body (:48), `ResourceARNs` scope derivation (:54), `checkPolicyResources(r, "ecs", action, …)` (:58), handler with PassRole closure (:69), then encode. +Gateway to daemon is a NATS request on `ecs.` with a 30 s timeout (`handlers/ecs/service_nats.go:12,29-177`), served by queue group `spinifex-workers` (`daemon/daemon.go:1131-1167`). +`gateway/operations.go:30-32` publishes Registered = all 39 table keys and Stubbed = 5. +`ecsScopes` (`authz.go:57-124`) has an entry for every table action, and `ResourceARNs` fails closed if an entry is missing (:140-144). + +| Action | Status | Resource | Handler (daemon) | +|---|---|---|---| +| CreateCluster | registered | cluster | `service.go:231` | +| DescribeClusters | registered | cluster (+counts from instance/task/service) | `service.go:268` | +| ListClusters | registered | cluster | `service.go:338` | +| DeleteCluster | registered | cluster (cascade orchestration) | `cluster_teardown.go:24` | +| UpdateCluster | stubbed | cluster | — | +| PutClusterCapacityProviders | registered | cluster (references capacityprovider) | `capacity_providers.go:16` | +| CreateCapacityProvider | registered | capacityprovider | `capacity_providers.go:44` | +| DescribeCapacityProviders | registered | capacityprovider | `capacity_providers.go:80` | +| DeleteCapacityProvider | registered | capacityprovider | `capacity_providers.go:125` | +| RegisterTaskDefinition | registered (PassRole on task/execution role, `actions.go:99`) | taskdefinition | `service.go:416` | +| DeregisterTaskDefinition | registered | taskdefinition | `service.go:546` | +| DescribeTaskDefinition | registered | taskdefinition | `service.go:571` | +| ListTaskDefinitions | registered | taskdefinition | `service.go:586` | +| ListTaskDefinitionFamilies | stubbed | taskdefinition | — | +| RunTask | registered (PassRole via taskdef lookup, ClientToken) | task (+placement, containerinstance) | `tasks.go:28` | +| StartTask | registered (PassRole via taskdef lookup) | task (+containerinstance) | `task_stop.go:53` | +| StopTask | registered | task | `task_stop.go:28` | +| DescribeTasks | registered | task | `tasks.go:273` | +| ListTasks | registered | task | `tasks.go:299` | +| SubmitTaskStateChange | registered (AWS action; used by agent) | task | `instances.go:445` | +| PollAssignments | registered (non-AWS, agent-only, raw JSON) | task inbox (wire) | `poll.go:37` | +| ReportTaskGPU | registered (non-AWS, agent-only) | task | `gpu_report.go:31` | +| CreateService | registered (PassRole via taskdef lookup; AZ-rebalancing gate) | service | `services.go:128` | +| UpdateService | registered (same) | service | `services.go:209` | +| DeleteService | registered | service | `services.go:267` | +| DescribeServices | registered | service | `services.go:302` | +| ListServices | registered | service | `services.go:326` | +| ListServicesByNamespace | stubbed | service | — | +| RegisterContainerInstance | registered (agent boot + 30 s liveness) | containerinstance | `instances.go:65` | +| DeregisterContainerInstance | registered | containerinstance | `cluster_teardown.go:60` | +| UpdateContainerInstancesState | registered | containerinstance | `cluster_teardown.go:103` | +| DescribeContainerInstances | registered | containerinstance | `instances.go:127` | +| ListContainerInstances | registered | containerinstance | `instances.go:167` | +| ProvisionCapacity | registered (non-AWS) | containerinstance capacity (EC2 launch orchestration) | `capacity.go:70` | +| PutAccountSetting | stubbed | account setting | — | +| ListAccountSettings | stubbed | account setting | — | +| TagResource | registered | cross-resource (by ARN) | `tags.go:227` | +| UntagResource | registered | cross-resource (by ARN) | `tags.go:254` | +| ListTagsForResource | registered | cross-resource (by ARN) | `tags.go:207` | + +Unsupported actions are any ECS action absent from the table, and they return `InvalidAction`. +Examples include task sets, `ExecuteCommand`, `UpdateCapacityProvider`, `DeleteTaskDefinitions`, attribute APIs and service deployments/revisions. +Three agent-facing actions (`PollAssignments`, `ReportTaskGPU`, `ProvisionCapacity`) are non-AWS shapes on the AWS ECS signing surface. +`SubmitTaskStateChange` is an AWS action that our agent uses as its state-report channel. + +## 3. Durable records / KV buckets + +| Bucket | Key shape | Record type | Owning resource | Writers | Readers | +|---|---|---|---|---|---| +| `ecs-account-` (history 1, schema v1, created lazily, `store.go:27-29,207-217`) | `clusters//meta` | `ClusterRecord` (`records.go:155`) | cluster | CreateCluster `service.go:259`; PutClusterCapacityProviders `capacity_providers.go:36`; tags `tags.go:152`; DeleteCluster removes the subtree `cluster_teardown.go:47` | cluster CRUD; RunTask/StartTask/CreateService existence checks (`tasks.go:36`, `task_stop.go:60`, `services.go:148`) | +| same | `clusters//instances/` | `InstanceRecord` (`records.go:333`) | containerinstance | `upsertInstance` plain Put `instances.go:215`; `recordHeartbeat` `:356`; UpdateContainerInstancesState `cluster_teardown.go:131`; reaper `scheduler.go:390`; tags `tags.go:200`; CAS reserve/release `tasks.go:212`, `task_stop.go:148`, `instances.go:527`; Deregister deletes `cluster_teardown.go:90` | placement (`tasks.go:182`), cluster counts (`service.go:301`), reaper, describe/list | +| same | `clusters//tasks/` | `TaskRecord` (`records.go:389`) | task | RunTask `tasks.go:84`; StartTask `task_stop.go:104`; stop/force-stop `task_stop.go:171,199`; recordTaskState `instances.go:409,433`; EIP assign `task_stop.go:284`; ENI rollback `tasks.go:168`; GPU report `gpu_report.go:58`; sweep `tasks_gc.go:98,119`; tags `tags.go:189` | service reconcile (`services.go:589-622`), cluster/instance counts, reaper, sweep | +| same | `clusters//assignments//` | `bus.Assign` | task (agent inbox) | `publishAssign` `tasks.go:269` | agent via `PollAssignments` (`poll.go:58`); acked and reclaimed via `poll.go:49,97` | +| same | `clusters//stops//` | `bus.StopDirective` | task (agent inbox) | `requestStopTask` `task_stop.go:176` | agent via `PollAssignments` (`poll.go:74`); acked and reclaimed via `poll.go:55,106` | +| same | `clusters//services/` | `ServiceRecord` (`records.go:454`) incl. `Deployments`, `Events` | service | Create/Update/Delete `services.go:198,256,295`; reconcile `services.go:433`; deployment-failure count `instances.go:496`; tags `tags.go:178` | task paths read it for LB and EIP config (`task_stop.go:241,273`) | +| same | `taskdef-families//latest-rev`, `taskdef-families//revs/` | `int`, `TaskDefRecord` (`records.go:280`) | taskdefinition | Register `service.go:452,455`; Deregister `:564`; tags `tags.go:167` | `resolveTaskDef` (`service.go:622`) for RunTask/StartTask/Create/UpdateService; gateway PassRole via NATS `DescribeTaskDefinition` | +| same | `capacity-providers/` | `CapacityProviderRecord` (`records.go:200`) | capacityprovider | Create `capacity_providers.go:72`; Delete `:142` | Describe only | +| `spinifex-ecs-leader` (TTL 60 s, `store.go:31-33,221`) | `scheduler` (`scheduler.go:24`) | kvlease lease | scheduler (orchestration) | `kvlease` in `Scheduler` (`scheduler.go:96-109`) | the same | +| `spinifex-ecs-clienttokens` (TTL 15 min, `gateway/ecs/clienttoken.go:22-26`) | `.` (`foundation/lifecycle/idempotency/store.go:91-93`, no namespace) | `idempotency` entry with `ecs.RunTaskOutput` | task (currently held by the adapter) | gateway `runTaskWithClientToken` (`clienttoken.go:71-86`) | the same | + +`store.go:23-25`, `:134-138` and `LeaderLeaseKey` describe per-cluster `/` lease keys, but the code uses one global key `scheduler`, and `LeaderLeaseKey` has no production caller. +No reader outside ECS opens any of these buckets. +A Go grep for bucket names and helpers outside `handlers/ecs` and `gateway/ecs` found only RDS's own `AccountBucketName`. +`accountteardown` and the agent reach ECS state only through the ECS API (NATS facade or gateway), not through KV. +No KV migration is registered for `ecs-account-*`; it runs `migrate.DefaultRegistry.RunKV` at version 1 (`store.go:213`). +No production code was found that deletes an `ecs-account-*` bucket. + +## 4. Identity and scope + +The partition is hard-coded to `aws`. +The Region is the daemon `config.Region` captured at construction (`daemon/daemon.go:2029`). +The account is the SigV4 caller's account, carried on every NATS request. +All records live in the caller's per-account bucket, so the account scope is structural. + +| Resource | Identifier | ARN (`records.go:107-135`) | Scope | +|---|---|---|---| +| cluster | name; omitted means `default` (`conv.go:27-36`, `service.go:233-235`) | `arn:aws:ecs:::cluster/` | account + Region; no AZ | +| taskdefinition | `family:revision`; revision is `latest-rev + 1` (`service.go:531-541`) | `…:task-definition/:` | account + Region | +| service | name unique per cluster | `…:service//` | account + Region; subnets/SGs stored, not validated | +| task | UUIDv4 (`tasks.go:65`, `task_stop.go:85`) | `…:task//` | account + Region; AZ is implicit through its container instance and is not stored on the task | +| containerinstance | the `InstanceIdentityDocument` string supplied by the agent (in practice the EC2 instance ID), with fallback `ci-` (`instances.go:67-70`) | `…:container-instance//` | account + Region; AZ from the registration attribute (`instances.go:100-101,331`) | +| capacityprovider | name | `…:capacity-provider/` | account + Region | + +AWS parity note (verify): AWS mints a UUID container-instance ID that is distinct from `ec2InstanceId`, and AWS expects `instanceIdentityDocument` to be the signed IMDS JSON document. +Ours uses the supplied string as both values (`instances.go:276`). +Identifier reuse: DeleteCluster removes every record (`cluster_teardown.go:47`), so a cluster name and its child names are reusable immediately. +There is no tombstone or identifier-reuse rule (Q-43/Q-44). +Task-definition revisions are not reserved against reuse after a concurrent collision (see gap G9). +Container instances authenticate as the account's `ecsInstanceRole` (`instancerole.go:12-30`). +`PollAssignments`, `SubmitTaskStateChange` and `ReportTaskGPU` check only account scope plus IAM policy on the named instance or task ARN (`authz.go:83-95`). +The handlers do not bind the caller to the specific container instance (`poll.go:37-56`, `instances.go:445-475`, `gpu_report.go:31`). + +## 5. Lifecycle paths per resource + +There are no desired or observed generation fields on any ECS record (`records.go:155-493`). +The only fencing mechanisms are: +- one global scheduler lease (`scheduler.go:96-109`), checked only at the start of each pass (`:170`, `:189`); +- CAS (`kv.Update` with revision) on container-instance reservations (`tasks.go:212`, `task_stop.go:148`, `instances.go:527`). + +Every other write is a read-modify-`Put` without revision (`service.go:179-188`). +ClientToken idempotency exists only for RunTask (`gateway/ecs/actions.go:203-238`). + +### cluster +Create is `CreateCluster` (`service.go:231-264`): get, then a plain Put if absent. +A repeat create by name returns the existing record without comparing parameters; AWS also treats it as idempotent (verify). +Readiness: the cluster is `ACTIVE` on create (`:250`), synchronously. +There is no reconcile loop. +Delete is `DeleteCluster` (`cluster_teardown.go:24-53`): it force-stops every task (:43-45), deletes the whole `clusters//` subtree (:47), and returns a synthetic `INACTIVE` (:51). +No durable deletion intent is written. +On teardown it is driven by `accountteardown/reapers_ecs.go:53-58`. + +### taskdefinition +Create is `RegisterTaskDefinition` (`service.go:416-459`): it validates (`:472-511`), computes `rev = latest + 1` without CAS (`:530-541`), Puts the revision (`:452`), then Puts `latest-rev` (`:455`). +There is no client token, and AWS has none for this action. +Readiness: the definition is `ACTIVE` synchronously. +Delete is `DeregisterTaskDefinition` (`:546-568`), which sets `INACTIVE` and retains the record. +There is no hard delete (`DeleteTaskDefinitions` is not served). +`resolveTaskDef` (`:622-647`) does not check status, so RunTask, StartTask and CreateService accept an `INACTIVE` revision. +AWS refuses new runs and services on an INACTIVE revision after a short window (verify). + +### service +Create is `CreateService` (`services.go:128-206`): an `ACTIVE` service with the same name is returned as-is (:159-160), and `ClientToken` is ignored. +It Puts the record with a PRIMARY deployment (:196-198), then runs `reconcileService` synchronously on the serving worker (:202). +Readiness: the service record is `ACTIVE` immediately. +Rollout state moves `IN_PROGRESS` → `COMPLETED` when the primary running count reaches the desired count and no old-deployment tasks remain (`deployments.go:170-196`). +The circuit breaker sets `FAILED` after 3 failed starts (`deployments.go:222-243`, `records.go:54`), with optional rollback (`:247-267`). +Launch backoff is 15 s to 5 min (`deployments.go:206-216`). +Reconcile has two triggers: +- the leader loop `ecs/services` (`scheduler.go:143-148`) runs `reconcileAllServices` (`services.go:445-476`) on a wake from `ecs.bus...service-reconcile` (`instances.go:472`), a 10 s revisit while unsettled, and a 5 min resync; +- `CreateService` and `UpdateService` also reconcile inline on the API worker (`services.go:202`, `:259`). +The reconcile launches through `s.RunTask` without a client token (`services.go:495-525`) and stops surplus tasks (`:531-581`). +Delete is `DeleteService` (`:267-299`): it requests a stop for each task (:288-289), then sets `INACTIVE` immediately (:291). +`ServiceStatusDraining` (`records.go:30`) is never written. +There is no `force` or desired-count precondition check, and the INACTIVE record is never pruned. + +### task +Create is `RunTask` (`tasks.go:28-93`) or `StartTask` (`task_stop.go:53-113`), in this order: +1. Reserve capacity by CAS on the container instance (`tasks.go:180-218` / `task_stop.go:118-153`). +2. Build the record in memory (`tasks.go:221-243`; desired `RUNNING`, last `PENDING`). +3. For awsvpc, create and attach the ENI through EC2 (`tasks.go:131-176`). +4. Make the first durable task write (`tasks.go:84` / `task_stop.go:104`). +5. Write the assignment inbox (`tasks.go:249-270`); a failure there is only logged (`:87-89`). +Readiness comes from the agent's `SubmitTaskStateChange`, which goes through `recordTaskState` (`instances.go:361-439`). +`RUNNING` sets `StartedAt`, registers ELBv2 targets and assigns an EIP when the service asks for it (:415-418). +`STOPPED` releases the target, EIP, inboxes, ENI and reservation (:422-436). +Stop has two paths: +- `StopTask` calls `requestStopTask` (`task_stop.go:162-179`), which sets desired `STOPPED` and posts a stop directive for the agent. +- `forceStopTask` (`:186-209`) is used by the cluster delete, deregister, drain and reaper paths; it marks the task `STOPPED` without telling the agent. +Repair: the leader `ecs/timers` loop runs `sweepStoppedTasks` (`tasks_gc.go:21-105`). +That sweep retries owed ENI releases (at most 20 per pass, 30 s to 15 min backoff) and deletes STOPPED tasks after 1 h. +Idempotency: RunTask honours ClientToken through `idempotency.Do` (`gateway/ecs/clienttoken.go:71-86`); a partial placement is finalized and replayed. +StartTask has no token, and AWS has none for it (verify). +Internal launch from the service reconcile is not idempotent per logical launch. + +### containerinstance +Create is `RegisterContainerInstance` (`instances.go:65-124`), which is an upsert with a plain Put (`:195-219`). +The agent calls it at boot and on every 30 s heartbeat (`agents/ecs/agent/controlplane.go:22-25`). +Readiness: the instance is `ACTIVE` on first registration (`:207`). +A zero reported capacity drains it (`Reaped=true`), and returning capacity restores it (`:44-60`). +`AgentConnected` is derived from `Status == ACTIVE` (`:278`), not from liveness. +Repair: the leader reaper (`scheduler.go:340-396`) marks an instance `DRAINING` with `Reaped=true` after 90 s without `LastSeen`, and force-stops its tasks (`:400-416`). +`recordRegister` and `recordHeartbeat` (`instances.go:325-357`) are fed only by bus subjects that nothing publishes. +Operator state: `UpdateContainerInstancesState` (`cluster_teardown.go:103-140`) sets the status. +`DRAINING` force-stops service tasks immediately (`:160-171`). +Delete is `DeregisterContainerInstance` (`:60-97`): it refuses if tasks are active unless `force`, deletes the assignments inbox and the record (the stops inbox is left), and does not touch the EC2 instance. +Capacity is provisioned by `ProvisionCapacity` (`capacity.go:70-170`), which calls `RunInstances` directly with no ECS-side intent record and no idempotency. + +### capacityprovider +Create is `CreateCapacityProvider` (`capacity_providers.go:44-76`): an existing name returns the existing record without comparing parameters, and the ASG ARN is stored without validation. +Readiness: it is `ACTIVE` immediately and inert, with no scheduler coupling (`service.go:74-76`). +Delete is a hard key delete (`:125-146`), with no check that a cluster references the provider. + +## 6. NATS subjects and guest/controller messages + +| Subject | Direction / type | Payload | Retry / redelivery | +|---|---|---|---| +| `ecs.` ×34 (`service_nats.go:29-177`; subs `daemon.go:1134-1167`) | gateway (and `accountteardown`, gateway PassRole lookup) → daemon, request/reply, queue `spinifex-workers` | aws-sdk-go ECS input/output or `PollAssignments*`, `ReportTaskGPU*`, `ProvisionCapacity*` structs | 30 s timeout. There is no retry in `NATSECSService`, and `natsmsg.NATSRequest` (`foundation/messaging/nats/nats.go:269`) retry behaviour was not verified here. | +| `ecs.bus...service-reconcile` (`bus/subjects.go:43`) | any worker → leader, core publish | none | Fire-and-forget (`instances.go:472-474`); loss is covered by the 10 s revisit and 5 min resync (`scheduler.go:30-58`). | +| `ecs.bus.*.*.instance-register.*`, `…instance-heartbeat.*`, `…task-state.*` (`scheduler.go:236-245`) | subscribed by leader | `bus.RegisterInstance`, `bus.Heartbeat`, `bus.TaskState` | No production publisher (residue). Core NATS with no ack or redelivery, and messages are dropped while there is no leader. | +| `ecs.bus...assign.` (`bus/subjects.go:30`) | unused | — | Residue; assignment moved to the KV inbox. | +| `ec2.CreateNetworkInterface`, `ec2.DeleteNetworkInterface` (`eni.go:64,119`) | ECS → EC2 AWS-action subjects | aws-sdk-go EC2 types | 30 s timeout (`eni.go:19`). NotFound counts as converged (`:148-155`). A failed release is retried by the sweep. | +| `ec2v1.InstanceCommandSubject()` (`eni.go:126-128`) | ECS → owning node, `contracts/ec2/v1` | `EC2InstanceCommand{AttachENI or DetachENI}` | Versioned contract; detach is forced (`:109`). | +| `ec2.AllocateAddress`, `ec2.AssociateAddress`, `ec2.DescribeAddresses`, `ec2.DisassociateAddress`, `ec2.ReleaseAddress` via `ec2eip.NewNATSEIPService` (`services.go:81-121`; subjects in `domains/ec2/eip/service_nats.go:26-34`) | ECS → EC2 EIP AWS adapter | aws-sdk-go EC2 types | A failed associate releases immediately (`services.go:91-96`). A failed release is logged, and the fields are cleared anyway (`task_stop.go:296-300`). | +| ELBv2 `RegisterTargets` / `DeregisterTargets` via `handlers_elbv2.NewNATSELBv2Service` (`services.go:45-58`) | ECS → ELBv2 AWS adapter | aws-sdk-go ELBv2 types | Best effort and log-only (`task_stop.go:252-255`), with no retry. The exact subject string was not verified. | +| `ec2.RunInstances..` | not used by ECS | — | ECS passes no node, so `RunWorkerInstance` runs the local in-process `instanceService.RunInstances` (`daemon/eks_worker_launch.go:29-51`). | + +Guest protocol: agent ↔ gateway HTTP, SigV4-signed with the instance role through `internal/ecsgw`; it is not NATS. +- `RegisterContainerInstance`: boot, then every 30 s as liveness. +- `PollAssignments`: delivery is at-least-once. The inboxes are KV keys, and an entry is redelivered on every poll until the agent acks it in `AckTaskIDs` or `AckStopIDs` on the next poll (`poll.go:45-56`), or until the task stops (`instances.go:430-431`, `task_stop.go:204-205`). +- `SubmitTaskStateChange`: duplicate reports are tolerated only through edge detection on `prev != status` (`instances.go:415,422`). There is no ordering or monotonicity guard and no check of the reporting instance (`:372-373`). +- `ReportTaskGPU`. +Bootstrap contract: cloud-config writes `/etc/spinifex-ecs/agent.env` and the gateway CA (`userdata.go:31-58`). + +## 7. Boundary-crossing imports + +ECS → other packages (non-stdlib, non-foundation first): + +| From | Imports | Symbol(s) | Flag | +|---|---|---|---| +| `handlers/ecs/service.go:19`, `instancerole.go:6` | `handlers/iam` | `SystemInstanceRoleEnsurer` (`service.go:92`), `EnsureSystemInstanceProfile` (`instancerole.go:38`), `ConvergeSystemRolePolicy` (`:46`) | IAM | +| `handlers/ecs/services.go:20` | `handlers/elbv2` | `NewNATSELBv2Service().RegisterTargets/DeregisterTargets` | other domain's AWS adapter used as an internal API | +| `handlers/ecs/services.go:18` | `domains/ec2/eip` | `NewNATSEIPService` (Allocate/Associate/Describe/Disassociate/Release) | EC2 / network; AWS adapter as internal API | +| `handlers/ecs/eni.go:13` | `contracts/ec2/v1` | `EC2InstanceCommand`, `EC2CommandAttributes`, `AttachENIData`, `DetachENIData`, `InstanceCommandSubject` | EC2 instance and network (versioned contract) | +| `handlers/ecs/eni.go:64,119` | string subjects `ec2.CreateNetworkInterface` / `ec2.DeleteNetworkInterface` | — | network; EC2 AWS-action subject used as an internal API | +| `handlers/ecs/capacity.go:11-12` | `domains/ec2/ebs/metadata`, `domains/ec2/instancetypes` | `RootDeviceName`; `IsGPUTypeName`, `GPUVendorForType` | EC2 | +| `handlers/ecs/placement.go:9` | `domains/ec2/instancetypes` | `HotPlugENISlotsForType` | EC2 | +| `handlers/ecs/service.go:16`, `capacity.go:10`, `eni.go:12`, `services.go:15-17` | aws-sdk-go `service/ec2`, `service/elbv2` | `RunInstancesInput`, `DescribeImages*`, ENI/EIP/target types | EC2 / ELBv2 types | +| `handlers/ecs/*` | foundation | `messaging/nats.NATSRequest`, `state/kvlease`, `state/kvutil`, `state/migrate`, `lifecycle/reconciler`, `aws/errors`, `aws/arn`, `aws/ami`, `aws/tags` | — | +| `gateway/ecs/authz.go:11-12` | `ingress/aws/bodyscope`, `ingress/aws/query` | `bodyscope.Parse`, `query.MaxSliceLen` | ingress | +| `gateway/ecs/clienttoken.go:13` | `foundation/lifecycle/idempotency` | `OpenStore`, `Do`, `ParamHash`, `ErrParamMismatch`, `ErrUnavailable` | — | +| `daemon/ecs_deps.go:8` | `github.com/mulgadc/bluebottle/pkg/masterkey` | `ReadShared` (:39) | Bluebottle (daemon wiring only; `handlers/ecs` itself does not import Bluebottle) | +| `daemon/ecs_deps.go:10,50-52` | `handlers/iam` | `NewIAMServiceImpl`, `*IAMServiceImpl` | IAM | +| `daemon/ecs_deps.go:29,32` | EKS-named daemon helpers | `resolveGatewayBaseURL` (`daemon/eks_deps.go:164`), `RunWorkerInstance` (`daemon/eks_worker_launch.go:29`) | EC2 instance launch shared with EKS | + +Packages that import ECS: + +| Importer | Imports | Symbol(s) | +|---|---|---| +| `gateway/ecs/actions.go:12` | `handlers/ecs` | `NewNATSECSService`, `ECSService`, `ProvisionCapacityInput`, `PollAssignmentsInput`, `ReportTaskGPUInput` | +| `gateway/ecs/authz.go:10` | `handlers/ecs` | `ClusterShortName`, `ServiceShortName`, `TaskShortID`, `ContainerInstanceShortID`, `CapacityProviderShortName`, `ParseTaskDefRef`, `ResourceARNSegment`, `ClusterARN`, `ServiceARN`, `TaskARN`, `ContainerInstanceARN`, `CapacityProviderARN`, `TaskDefARN`, `TaskDefRefARN` | +| `gateway/ecs.go:10`, `gateway/operations.go:7` | `gateway/ecs` | `Actions`, `ResourceARNs`, `RawJSONActions`, `WriteJSONResponse`, `WriteRawJSONResponse`, `StubbedActionNames` | +| `daemon/daemon.go:74`, `daemon/ecs_deps.go:9` | `handlers/ecs` | `Service`, `Scheduler`, `NewService`, `WithDeps`, `InitLeaderBucket`, `NewScheduler`, `Deps` | +| `accountteardown/reapers_ecs.go:9` | `handlers/ecs` | `NewNATSECSService`, `ECSService` (`ListClusters`, `DeleteCluster`) | +| `agents/ecs/agent/{registration,controlplane,taskrun}.go` | `handlers/ecs` (implementation package) | `AttrInstanceType`, `AttrAvailabilityZone`, `PollAssignmentsInput/Output`, `ReportTaskGPUInput`, `ContainerGPUReport` | +| `agents/ecs/agent/{agent,controlplane,reconcile,registration,taskrun}.go` | `handlers/ecs/bus` | `Assign`, `StopDirective`, `TaskState`, `ContainerStatus`, `InstanceCapacity`, status constants | + +Explicit flags: +- **IAM / PassRole.** PassRole is enforced only in the gateway (`gateway/ecs.go:65-67`, through `gw.checkPassRole` at `gateway/gateway.go:645-648` with `iam:PassedToService=ecs-tasks.amazonaws.com`). + RegisterTaskDefinition checks the role ARNs in the body (`actions.go:99`). + RunTask, StartTask, CreateService and UpdateService first fetch the task definition through the ECS NATS facade (`actions.go:44-56`). + `ProvisionCapacity` attaches the system `ecsInstanceRole` instance profile (`capacity.go:103,141`) and launches through `RunWorkerInstance`, which goes straight to `d.instanceService.RunInstances` (`daemon/eks_worker_launch.go:51`). + That bypasses the EC2 gateway's PassRole closure (`gateway/ec2.go:199`). + `handlers/sts/assume_role.go:52-55` duplicates `ecsInstanceRole`, which is implicit coupling with no import. +- **Quota.** `handlers/ecs` has no quota import. + `ProvisionCapacity` appears to bypass `gw.Quota.EnforceLaunch` and `ChargeLaunch` (`gateway/ec2.go:196-205`), because it calls the in-process instance service (verify whether `instanceService.RunInstances` enforces quota itself). + ECS resources themselves have no quota. +- **EC2 instance.** `RunInstances` (via `Deps.RunInstances`), `DescribeImages` (via `Deps.Images` = `d.imageService`), `instancetypes`, EBS root-device metadata, and the `contracts/ec2/v1` instance command for ENI hot-plug. +- **Network.** ENI create/delete through EC2 AWS subjects, EIP through the `domains/ec2/eip` NATS adapter, subnets and SGs passed through unvalidated. +- **Bluebottle.** Only `masterkey.ReadShared` in `daemon/ecs_deps.go:39`, used to build the IAM service. + Without it, `ProvisionCapacity` is disabled and the scheduler refuses leadership (`scheduler.go:226-231`). + +## 8. Known gaps against Q-78 / ADR-0003 (observed) + +- **G1. No generations.** No ECS record has a desired or observed generation (`records.go:155-493`). Reconcilers act on whole records with plain Put (`service.go:179-188`). +- **G2. Realization before intent.** RunTask and StartTask reserve capacity and create/attach an ENI before the task record exists (`tasks.go:66-84`, `task_stop.go:86-104`). A crash in that window leaves a reservation (`PlacedTasks` naming an unknown task) and possibly an ENI with no owning record. No reconciler targets orphan reservations; `tasks_gc.go` handles only owed ENI release on STOPPED tasks. +- **G3. Lost-update window on instance reservations.** Reservations use CAS (`tasks.go:212`, `task_stop.go:148`, `instances.go:527`), but the same key is rewritten with a plain read-modify-Put every 30 s by agent registration (`instances.go:195-215`). The same happens in UpdateContainerInstancesState (`cluster_teardown.go:131`), the reaper (`scheduler.go:390`) and tags (`tags.go:200`), so a CAS commit between their Get and Put can be overwritten. `handlers/ecs/task_accounting_test.go` (`TestDescribeContainerInstances_StalePlacedTasksDoNotInflateCount`) acknowledges stale `PlacedTasks`. +- **G4. Unfenced concurrent writers.** The leader lease is checked only at pass start (`scheduler.go:170,189`). KV writes are not fenced by the lease. API workers also run `reconcileService` inline (`services.go:202,259`) while the leader runs `reconcileAllServices` (`scheduler.go:177`), so two writers can launch through `RunTask` for the same service. This is the observed code shape; no test was found covering it. +- **G5. Cascade delete removes cleanup ownership.** `DeleteCluster` (`cluster_teardown.go:43-47`) deletes the whole subtree, including STOPPED tasks whose `ENIReleased=false` or whose EIP or target cleanup failed. That removes the only record of owed cleanup, which Q-78 forbids. There is no deletion intent or incomplete-cleanup state. AWS refuses DeleteCluster while services, tasks or registered instances exist (verify against AWS docs and a test). +- **G6. Force-stop does effects before persisting.** `forceStopTask` releases the EIP and ENI before persisting STOPPED (`task_stop.go:197-199`). A failed EIP release is logged and its allocation ID cleared (`:296-300`), so it cannot be retried. Target deregistration is best effort with no retry (`:252-255`). `forceStopTask` also posts no stop directive, so whether a live agent stops its containers is unverified. +- **G7. EIP before record.** `assignTaskPublicIP` allocates and associates the EIP before persisting its allocation ID (`task_stop.go:277-286`). +- **G8. Silent reservation leak.** `releaseReservation` gives up silently after 5 CAS conflicts and returns nil (`instances.go:506-531`), which leaks capacity with no reconciler. +- **G9. Non-atomic revision allocation.** `RegisterTaskDefinition` computes the revision as read + 1 with no CAS (`service.go:430,531-541`). It writes the revision before `latest-rev` (`:452-455`), so concurrent registers can produce the same revision and overwrite it. +- **G10. No guard on task-state reports.** `recordTaskState` sets `LastStatus` unconditionally (`instances.go:372-373`), so a late RUNNING after STOPPED regresses the state. The reporting `InstanceID` is ignored. +- **G11. DeleteService is immediate.** It sets INACTIVE directly, with no DRAINING (`services.go:291`; `records.go:30` is unused). It has no `force` or desired-count check, AWS requires one (verify), and the INACTIVE record is never pruned. +- **G12. Create idempotency without comparison.** CreateService ignores `ClientToken` and returns an existing ACTIVE service without comparing parameters (`services.go:159-160`). CreateCapacityProvider (`capacity_providers.go:60-61`) and CreateCluster (`service.go:246-262`) behave the same way. +- **G13. Retry bookkeeping in the adapter.** The RunTask idempotency store lives in the AWS adapter (`gateway/ecs/clienttoken.go:22-56`). ADR-0003 S3 says retry bookkeeping is not the adapter's. +- **G14. ProvisionCapacity.** It has no durable intent and no idempotency, and it silently caps count at 10 (`capacity.go:89-91,158`). Launched instances are linked only by the tag `spinifex:ecs-cluster` (`:150`). +- **G15. Liveness derived from status.** Container-instance `AgentConnected` is derived from Status, not liveness (`instances.go:278`). Registration makes an instance ACTIVE immediately (`:207`). +- **G16. Residue subscriptions.** The scheduler subscribes to three bus subjects that nothing publishes (`scheduler.go:236-245`). +- **G17. AWS adapters as internal APIs.** Account teardown uses the ECS facade (`accountteardown/reapers_ecs.go:21`), and gateway PassRole resolution calls `DescribeTaskDefinition` over NATS (`actions.go:48`). ECS calls the ELBv2 and EIP adapters and the `ec2.*NetworkInterface` subjects (`services.go:46,82`, `eni.go:64,119`). ADR-0003 S4 and ADR-0004 S4 forbid an AWS adapter used as an internal service API. +- **G18. Incomplete account teardown.** It reaps only clusters (`reapers_ecs.go:20-22`). Task definitions, capacity providers and the `ecs-account-*` bucket remain; no deleting code was found. +- **G19. No pagination.** List APIs ignore `nextToken` and `maxResults` (`service.go:338,586`, `tasks.go:299`, `services.go:326`, `instances.go:167`). Q-89 requires listing tests that exceed one page. +- **G20. Status not checked on launch.** RunTask, StartTask and CreateService accept an INACTIVE task-definition revision (`service.go:622-647`), but AWS refuses (verify). +- **G21. Capacity providers not taggable.** Capacity-provider ARNs are rejected by the tag parser (`tags.go:52-71`), although create accepts tags. +- **G22. Delete leaves stop inboxes.** `DeregisterContainerInstance` deletes the assignments inbox but not the stops inbox (`cluster_teardown.go:87-90`). + +## 9. First-resource candidates (ranked) + +1. **taskdefinition.** + Size: about 300 lines in `service.go:416-706` plus about 180 in `conv.go:70-265`, plus `TaskDefRecord` and `ContainerDef` (`records.go:211-328`). + Private state: `taskdef-families/*` only. + Dependencies: none at the handler. PassRole is checked in the gateway on role ARN strings, so extraction does not require deciding IAM authority. It has no EC2, network or Bluebottle dependency. + Consumers needing a narrow query: task launch, service create/update (`resolveTaskDef`), gateway PassRole lookup, and tags. + Tests: `handlers/ecs/taskdef_validation_test.go`, `service_test.go` (`RegisterTaskDefinition_*`, `ListTaskDefinitions_*`, `DescribeTaskDefinition_Unknown`), `cluster_teardown_test.go` (`DeregisterTaskDefinition_*`), `conv_test.go`, `records_test.go`, `tags_test.go` (`TaskDefinitionRoundTrip`). + Known gaps: G9 and G20. + Assessment: the best first slice. Its `resolveTaskDef` edge also gives the ADR-0004 rollout item 3 capability seam. +2. **capacityprovider.** + Size: about 220 lines in `capacity_providers.go` plus records. + Private state: `capacity-providers/*`. + Dependencies: none; the provider is inert, and the ASG ARN is opaque. + Prerequisite split: `PutClusterCapacityProviders` must move to `cluster`. + Tests: `handlers/ecs/capacity_providers_test.go`. + Gaps: G12, G21, and an unchecked delete. + Assessment: the smallest and safest slice, with no IAM, EC2 or Bluebottle dependency. It has almost no consumers, so it demonstrates little about cross-resource capabilities. +3. **cluster.** + Size: CRUD of about 180 lines (`service.go:231-411`) plus `PutClusterCapacityProviders`. + Private state: `clusters//meta`. + Dependencies: counts read instance, task and service records (`service.go:300-335`), and its existence is checked by task and service launch. The delete cascade is orchestration over every child (G5). + Tests: `service_test.go` (`CreateCluster_Idempotent`, `DescribeClusters_*`, `ListClusters_*`), `cluster_teardown_test.go` (`DeleteCluster_*`), `capacity_providers_test.go`, `tags_test.go`, `accountteardown/reapers_ecs_test.go`. + Assessment: a CRUD-only slice is feasible without IAM or Bluebottle. DeleteCluster has to stay as domain-root orchestration until the children have owners. +4. **containerinstance.** + Size: about 450 lines across `instances.go`, part of `cluster_teardown.go` and `placement.go`. + Private state: `instances/*` plus the reservation fields, which placement and task CAS-mutate. + Dependencies: the agent wire contract, EC2 (`instancetypes`; `RunInstances` via `ProvisionCapacity`) and IAM (`ecsInstanceRole`). + Tests: `instance_recovery_test.go`, `instance_list_status_test.go`, `placement_test.go`, `eni_placement_test.go`, `capacity_test.go`, `scheduler_test.go`, `task_accounting_test.go`, `service_test.go` (GPU/register/heartbeat/reaper). + Assessment: G3 must be characterized first, and reservation ownership needs a decision. It is not a first slice. +5. **service.** + Size: about 1,050 lines (`services.go`, `deployments.go`). + Dependencies: task launch and stop, taskdefinition, ELBv2 targets and EC2 EIP. Its reconcile runs on both the leader and the API workers (G4). + Tests: `services_test.go`, `deployments_test.go`, `revisit_test.go`, `eip_test.go`, `services_eip_nats_test.go`, gateway `actions_test.go` and `azrebalancing_test.go`. + Assessment: it depends on the task and taskdefinition owners existing. +6. **task.** + Size: about 1,300 lines (`tasks.go`, `task_stop.go`, `tasks_gc.go`, `poll.go`, `eni.go`, `netcfg.go`, `gpu_report.go`, part of `instances.go`). + Dependencies: EC2 ENI and EIP, ELBv2, container-instance reservations, the agent contract, the ClientToken store and gateway PassRole. + Tests: `eni_test.go`, `eni_placement_test.go`, `tasks_gc_test.go`, `task_accounting_test.go`, `poll_test.go`, `gpu_report_test.go`, `service_test.go` (`RunTask_*`, `RecordTaskState_*`), gateway `clienttoken_test.go`, `actions_test.go`, and e2e `tests/e2e/ecs/ecs_test.go` and `tests/e2e/gpu/ecs_gpu_test.go` in the mulga root. + Assessment: it carries most of G2, G3, G6, G7, G8 and G10, so it should be extracted last. diff --git a/docs/package-boundary/adr-0004-inventory-eks.md b/docs/package-boundary/adr-0004-inventory-eks.md new file mode 100644 index 000000000..91e465e0f --- /dev/null +++ b/docs/package-boundary/adr-0004-inventory-eks.md @@ -0,0 +1,308 @@ +# ADR-0004 source inventory — EKS + +Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`. +It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it. +Items marked "verify" or "uncertain" are unconfirmed. +Gaps record observed behaviour, not target behaviour. + + +Repo: `spinifex` submodule, branch `refactor/spinifex-package-boundaries`; all paths are relative to `spinifex/spinifex/` unless stated. +Binding inputs: ADR-0001, ADR-0002, ADR-0003, ADR-0004 and Q-78 as on `origin/main` of mulga. +Proposed resource vocabulary (ADR-0004 S1): `cluster`, `nodegroup`, `addon`, `access`. +Line numbers are from the working tree at time of writing; "uncertain" marks inferences not verified against a test or live system. + +## 1. File table + +Production `.go` files only; `_test.go` excluded. +Line counts are `wc -l`. + +### 1.1 `handlers/eks/` (single package `handlers_eks`, 37 files, 12,310 lines) + +| File | Lines | Class | Reason | +|---|---|---|---| +| `access_entry.go` | 255 | resource owner (`access`) | AccessEntry record CRUD/CAS, supported access-policy catalogue, scope validation, ARN/record projection. | +| `addon_catalog.go` | 121 | resource owner (`addon`) | Read-only add-on version catalogue backing CreateAddon/DescribeAddonVersions. | +| `addon_installer.go` | 97 | resource owner (`addon`) + guest wire contract | Stages manifests in KV; `StagedAddonManifest` is the payload the on-VM addon-sync agent reads, so the type should split to a contract. | +| `addon_state.go` | 186 | resource owner (`addon`) | AddonRecord, status enum, CAS update, list/delete. | +| `addon_status_reconciler.go` | 79 | resource owner (`addon`), hosted on cluster reconciler | Transition function is addon semantics but is a method on `ClusterReconciler`; split: addon owns `nextAddonStatus`/apply, cluster reconciler only relays reports. | +| `addons.go` | 348 | resource owner (`addon`) + guest wire contract | Six AWS add-on actions plus internal `ListStagedAddonManifests` served to the guest. | +| `awsgw_endpoint.go` | 42 | resource owner (`cluster`) | Builds the per-cluster OIDC issuer URL. | +| `claim.go` | 31 | resource owner (`nodegroup`) | CAS-create helper; only caller is `nodegroup.go:123`. | +| `clienttoken.go` | 57 | resource owner (`cluster`) | CreateCluster `clientRequestToken` store (`spinifex-eks-clustertokens`). | +| `cluster_reconciler.go` | 1209 | resource owner (`cluster`), mixed | Cluster readiness/health, CP restart, member replacement, etcd reset; also subscribes the addon status subject (`:265-268`, `:447-458`) on behalf of `addon`. | +| `cluster_state.go` | 515 | resource owner (`cluster`) | `ClusterMeta` record, status enum, CAS helpers, delete-reap bookkeeping, `DeleteClusterPrefix` (which also erases nodegroup/addon/access keys). | +| `cp_vpc.go` | 128 | resource owner (`cluster`) | EKS-specific managed CP VPC realization via `domains/network/systemvpc`. | +| `eks_billable_reaper.go` | 133 | resource owner (`cluster`) | Node-local GC backstop for CP VMs whose meta is gone; comment at `:3` cites "ADR-0006 §5", which is not an accepted ADR. | +| `eks_deleting_reaper.go` | 159 | resource owner (`cluster`) | Re-drives DELETING clusters with backoff and exhaustion. | +| `igw.go` | 32 | resource owner (`cluster`) | Legacy-topology IGW ensure/delete via `systemvpc`. | +| `k3s_ha_control_plane.go` | 628 | resource owner (`cluster`), mixed | CP placement, spread, replacement, fresh CP; also contains `natsHostScheduler` (`:461-628`), a NATS fan-out over `contracts/cluster/v1` that is a capability implementation and should move to the composition root or the scheduler owner. | +| `k3s_server_role.go` | 51 | resource owner (`cluster`) | Ensures the system CP instance profile through `handlers_iam.EnsureSystemInstanceProfile` (identity dependency). | +| `k3s_server_vm.go` | 700 | resource owner (`cluster`) + guest wire contract | CP VM/ENI launch/terminate and AMI lookup; `buildK3sUserData` (`:524+`) is the CP guest boot contract; `lookupEKSGPUNodeAMI` is used by `nodegroup`. | +| `nats_bootstrap.go` | 347 | resource owner (`cluster`) + guest wire contract | Bootstrap subject vocabulary and envelope (`:47-57`) are a guest contract; persistence of token/kubeconfig/JWKS/CA is cluster state. | +| `nlb.go` | 591 | resource owner (`cluster`) | EKS-specific NLB realization via an ELBv2-shaped capability; also `ReapLBCLoadBalancers` (`:517`) for LBC-created ALBs in the customer VPC. | +| `nodegroup.go` | 1409 | resource owner (`nodegroup`), mixed | Nodegroup CRUD, scaling CAS, worker launch/terminate; also creates customer IAM roles/instance profiles (`:768-838`), stages a GPU add-on (`:410`, crosses into `addon`), and decrypts the cluster join token (`:1160`, reads `cluster` secrets). | +| `nodegroup_userdata.go` | 171 | resource owner (`nodegroup`) + guest wire contract | Worker cloud-init (K3S_TOKEN, registry mirrors) is the worker guest boot contract. | +| `oidc_keypair.go` | 202 | resource owner (`cluster`) | Per-cluster OIDC signing key (encrypted) and JWKS. | +| `private_endpoint.go` | 88 | resource owner (`cluster`) | Customer-VPC private-endpoint ENI. | +| `reconciler_registry.go` | 183 | resource owner (`cluster`) | In-process registry of per-cluster reconciler goroutines. | +| `recovery_directive.go` | 144 | resource owner (`cluster`) + guest wire contract | Per-member etcd recovery directive; `RecoveryDirective` is read by the on-VM recovery agent. | +| `restore_snapshot.go` | 355 | resource owner (`cluster`) | Operator-only total-loss DR path; reads etcd snapshots through `providers/objectstore`. | +| `security_groups.go` | 505 | resource owner (`cluster`), mixed | Cluster SG realization in system and customer VPCs; `EnsureNodegroupSGRules` (`:351`) is nodegroup realization. | +| `service.go` | 73 | temporary residue | `EKSService`, one method per AWS action plus internal methods; it is the NATS RPC surface between gateway and daemon and is also consumed by `accountteardown` and the operator CLI; removal condition: per-resource commands behind `domains/eks/awsapi` registration (ADR-0004 S3) with any remaining process boundary versioned under `contracts/eks/v1`. | +| `service_impl.go` | 2518 | mixed: cross-resource orchestration + resource owners | Split: CreateCluster launch sequence and `purgeClusterInfra` (`:504-1013`, `:1268-1452`) are cluster realization orchestration; cluster read/delete (`:1041-1162`) is `cluster`; access-entry actions (`:1527-1816`) are `access`; nodegroup wrappers (`:1465-1525`) are `nodegroup`; Tag actions (`:1842-1992`) span cluster and nodegroup; IdP stubs (`:1818-1832`); `DesiredDNSChanges` (`:2143`) is a DNS projection; `EKSServiceDeps` and capability interfaces (`:39-245`). | +| `service_nats.go` | 190 | temporary residue | Gateway-side NATS client implementing `EKSService` over `eks.` subjects; removal condition as for `service.go`. | +| `state_report.go` | 111 | guest/controller wire contract | `eks.state.*` and `eks.addon.*` subjects and payloads published by the CP VM. | +| `store.go` | 255 | resource owner (`cluster`), mixed | Bucket names, leader bucket, migrations, key helpers for every resource; split per-resource key helpers into each owner; `AccountWatchBuckets` (`:215`) is effectively a projection consumed by DNS. | +| `token_review.go` | 128 | resource owner (`access`) | Authenticates a `get-token` bearer against STS and maps to AccessEntry groups; `ResolveTokenReview` is called from the gateway process and opens EKS KV there. | +| `token_webhook.go` | 57 | guest/controller wire contract | `eks.VerifyToken` subject and request/response consumed by `runtime/roles/awsgw`. | +| `types.go` | 131 | mixed: `nodegroup` + `access` + residue | `NodegroupRecord` (nodegroup), `AccessEntryRecord`/`AccessScope` (access); `ClusterRecord` (`:19`) and `OIDCProviderConfigRecord` (`:125`) have no production readers or writers (temporary residue; removal: delete). | +| `volume_reclaim.go` | 81 | resource owner (`cluster`) | Surfaces (never deletes) CSI-created EBS volumes at cluster delete. | + +Additional dead code observed in `store.go`: `OIDCProviderKey` (`:95`) and `EventKey` (`:132`) have no production callers. + +### 1.2 `gateway/eks.go` and `gateway/eks/` (15 files, 1,473 lines) + +| File | Lines | Class | Reason | +|---|---|---|---| +| `gateway/eks.go` | 379 | AWS protocol adapter | REST-JSON route table (`:27-219`), dispatch, authorization ordering, `iam:PassRole` check (`:319-347`), caller identity extraction. | +| `gateway/eks/access.go` | 89 | AWS protocol adapter | Unmarshal and NATS call for access actions. | +| `gateway/eks/addons.go` | 60 | AWS protocol adapter | Same for add-on actions. | +| `gateway/eks/authz.go` | 310 | AWS protocol adapter | Per-action resource ARN derivation (`eksScopes` `:45-109`); imports `handlers_eks.PrincipalARNHash`. | +| `gateway/eks/cluster.go` | 66 | AWS protocol adapter | Cluster action wrappers. | +| `gateway/eks/handler.go` | 47 | AWS protocol adapter | JSON response/error writers. | +| `gateway/eks/internal_addons.go` | 38 | guest/controller wire contract | `GET /clusters/{c}/internal-addons/{acct}` for the addon-sync agent. | +| `gateway/eks/internal_authz.go` | 129 | AWS protocol adapter + temporary residue | Principal gate for internal routes; `lookupClusterMeta` (`:102-129`) reads `eks-account-*` and `ClusterMeta` directly from the gateway process; removal: owner-published membership query. | +| `gateway/eks/internal_publish.go` | 91 | guest/controller wire contract | HTTP-to-NATS relay of bootstrap/state/addon channels. | +| `gateway/eks/internal_recovery.go` | 37 | guest/controller wire contract | `GET .../internal-recovery/{acct}/{instance}`. | +| `gateway/eks/nodegroup.go` | 65 | AWS protocol adapter | Nodegroup wrappers. | +| `gateway/eks/oidc.go` | 45 | AWS protocol adapter | IdP-config wrappers (handlers are stubs). | +| `gateway/eks/passrole.go` | 31 | AWS protocol adapter | Extracts passed role ARNs for CreateCluster/CreateNodegroup only. | +| `gateway/eks/tags.go` | 35 | AWS protocol adapter | Tag wrappers. | +| `gateway/eks/token_review.go` | 51 | guest/controller wire contract | `POST /clusters/{c}/token-review` for the in-VM token webhook. | + +### 1.3 Other production files wiring or reading EKS + +| File | Lines | Class | Reason | +|---|---|---|---| +| `daemon/eks_deps.go` | 220 | cross-resource orchestration (composition-root binding) | Builds `EKSServiceDeps`, binding VPC, ELBv2, image, EIP, IGW, NAT-GW, route-table, placement-group, IAM, EBS and object-store owners. | +| `daemon/eks_adapter.go` | 58 | cross-resource orchestration (composition-root binding) | `SubnetVPCResolver` over the VPC service. | +| `daemon/eks_cp_control.go` | 67 | temporary residue | CP describe/start/stop calls `domains/ec2/awsapi/instance.{DescribeInstances,StartInstances,StopInstances}`, i.e. an EC2 AWS adapter used as an internal API (ADR-0004 S3/S4); removal: EC2-owned instance-control capability. | +| `daemon/eks_worker_launch.go` | 175 | cross-resource orchestration (composition-root binding) | `WorkerLauncher` via `instanceService.RunInstances`, `ec2.RunInstances.{type}.{node}` and `contracts/ec2/v1` instance commands. | +| `daemon/daemon.go` (EKS parts: `:75`, `:167`, `:1085-1128`, `:2018-2019`, `:2175`, `:2302-2304`, `:3017-3018`) | n/a | temporary residue | NATS subscription table for 38 `eks.*` subjects, service init, reconciler spawn, reaper registration, shutdown; comment at `:1085-1087` says every handler returns NotImplemented, which is stale. | +| `daemon/dns_reconcile.go` (`:27`, `:64-68`, `:97-98`) | n/a | temporary residue | Watches every `eks-account-*` bucket via `AccountWatchBuckets` and pulls `DesiredDNSChanges`; removal: EKS-published DNS projection. | +| `gateway/oidc_discovery.go` | 117 | temporary residue | Public OIDC discovery/JWKS endpoints read `eks-account-*`, `GetClusterMeta` and `OIDCJWKSKey` directly; removal: cluster-owned OIDC projection. | +| `handlers/sts/oidc_jwks.go` | 126 | temporary residue (STS-owned file) | `FetchClusterJWKS` reads `eks-account-*`/`OIDCJWKSKey` (`:101-110`); ADR-0001 step 7 names this STS-to-EKS import for removal via an identity-owned contract or EKS projection. | +| `runtime/roles/awsgw/eks_token_verify.go` | 75 | guest/controller wire contract (server) | Queue-subscribes `eks.VerifyToken` (`:30`) and verifies presigned STS URLs. | +| `accountteardown/reapers_eks.go` | 138 | cross-resource orchestration | Account teardown deletes nodegroups then clusters through the `EKSService` NATS client. | +| `operator/cli/admin_eks.go` | 77 | guest/controller wire contract (operator) | `spx admin eks restore-snapshot` over `eks.RestoreSnapshot`. | +| `gateway/gateway.go` (`:300`, `:345-346`, `:403`, `:455`, `:513`), `gateway/auth.go` (`:278-279`), `gateway/operations.go` (`:34-36`) | n/a | AWS protocol adapter (legacy registration residue) | Legacy dispatch, OIDC public routes and inventory; removal per ADR-0002 when `domains/eks/awsapi` registers. | +| `bootstrap/config/config.go:44` | n/a | AWS protocol adapter (config) | `eks` listed in `AWSGWServiceNames`. | + +Guest counterparties (not inventoried individually): `agents/eks/{gatewaypublish,gatewayfetch,tokenwebhook,credentialprovider,konnectivitycert,webhookcert}` and `scripts/images/eks-node/*`. +`agents/eks/tokenwebhook/webhook.go` imports `handlers_eks.WebhookTokenReviewResult`, so a guest binary imports the domain monolith (ADR-0004 S5 debt). + +## 2. Served AWS actions and protocol + +Protocol: REST-JSON, method+path routed by `rest.NewRouter("eks", eksRoutes)` (`gateway/eks.go:234`); unknown routes return `InvalidAction` (`:241`). +Authorization order (`gateway/eks.go:237-312`): route lookup, account from auth context, internal-route principal gate, body read, query folding, `ResourceARNs`, `checkPolicyResources`, `checkEKSPassRole`, handler over NATS. +Gateway-to-daemon transport: NATS request/reply `eks.`, queue group `spinifex-workers` (`daemon/daemon.go:1090-1127`). +`gateway/operations.go:34-36` declares EKS as `Registered: eksActionNames()` with no `Stubbed` or `Unsupported` list, so stubs and internal routes are indistinguishable in the inventory. + +| Action | Route | Status | Owner | +|---|---|---|---| +| CreateCluster | POST /clusters | implemented (async) | cluster | +| ListClusters | GET /clusters | implemented | cluster | +| DescribeCluster | GET /clusters/{c} | implemented | cluster | +| DeleteCluster | DELETE /clusters/{c} | implemented (synchronous purge) | cluster | +| UpdateClusterConfig | POST /clusters/{c}/update-config | stub NotImplemented (`service_impl.go:1455`) | cluster | +| UpdateClusterVersion | POST /clusters/{c}/updates | stub NotImplemented (`service_impl.go:1459`) | cluster | +| ListUpdates | GET /clusters/{c}/updates | stub at gateway (`gateway/eks.go:49-52`) | cluster (update records do not exist) | +| DescribeUpdate | GET /clusters/{c}/updates/{id} | stub at gateway (`gateway/eks.go:53-56`) | cluster/nodegroup/addon | +| CreateNodegroup | POST /clusters/{c}/node-groups | implemented (async) | nodegroup | +| ListNodegroups | GET /clusters/{c}/node-groups | implemented | nodegroup | +| DescribeNodegroup | GET .../node-groups/{n} | implemented | nodegroup | +| UpdateNodegroupConfig | POST .../node-groups/{n}/update-config | implemented (synchronous, returns `Successful`) | nodegroup | +| UpdateNodegroupVersion | POST .../node-groups/{n}/update-version | stub NotImplemented (`nodegroup.go:1120-1127`) | nodegroup | +| DeleteNodegroup | DELETE .../node-groups/{n} | implemented (synchronous) | nodegroup | +| CreateAccessEntry | POST /clusters/{c}/access-entries | implemented (STANDARD type only, `service_impl.go:1561-1564`) | access | +| ListAccessEntries | GET /clusters/{c}/access-entries | implemented | access | +| DescribeAccessEntry | GET .../access-entries/{p} | implemented | access | +| UpdateAccessEntry | POST .../access-entries/{p} | implemented | access | +| DeleteAccessEntry | DELETE .../access-entries/{p} | implemented | access | +| AssociateAccessPolicy | POST .../access-entries/{p}/access-policies | implemented (cluster scope only, `service_impl.go:1692`) | access | +| DisassociateAccessPolicy | DELETE .../access-policies/{policy} | implemented | access | +| ListAssociatedAccessPolicies | GET .../access-entries/{p}/access-policies | implemented | access | +| ListAccessPolicies | GET /access-policies | implemented (static catalogue) | access | +| DescribeAddonVersions | GET /addons/supported-versions | implemented (static catalogue) | addon | +| ListAddons | GET /clusters/{c}/addons | implemented | addon | +| CreateAddon | POST /clusters/{c}/addons | implemented (async via guest) | addon | +| DescribeAddon | GET .../addons/{a} | implemented | addon | +| UpdateAddon | POST .../addons/{a}/update | implemented | addon | +| DeleteAddon | DELETE .../addons/{a} | implemented | addon | +| AssociateIdentityProviderConfig | POST .../identity-provider-configs/associate | stub NotImplemented (`service_impl.go:1818`) | cluster (no IdP resource) | +| DescribeIdentityProviderConfig | POST .../identity-provider-configs/describe | stub | cluster | +| DisassociateIdentityProviderConfig | POST .../identity-provider-configs/disassociate | stub | cluster | +| ListIdentityProviderConfigs | GET .../identity-provider-configs | stub | cluster | +| TagResource / UntagResource / ListTagsForResource | POST/DELETE/GET /tags/* | implemented for cluster and nodegroup ARNs; NotImplemented for addon/access-entry ARNs (`service_impl.go:1882,1925,1958`) | cluster, nodegroup | +| PublishInternal | POST /clusters/{c}/internal-publish | internal (not AWS) | cluster guest contract | +| WebhookTokenReview | POST /clusters/{c}/token-review | internal (not AWS) | access guest contract | +| ListInternalAddons | GET /clusters/{c}/internal-addons/{acct} | internal (not AWS) | addon guest contract | +| GetRecoveryDirective | GET /clusters/{c}/internal-recovery/{acct}/{id} | internal (not AWS) | cluster guest contract | + +Internal NATS-only methods (no HTTP route): `SetRecoveryDirective`, `RestoreSnapshot`, `ListStagedAddonManifests` (backs ListInternalAddons). +Unregistered AWS EKS operations (InvalidAction today) include Fargate profiles, pod identity associations, encryption config, insights, `RegisterCluster`/`DeregisterCluster`, `DescribeAddonConfiguration`, `DescribeClusterVersions` and EKS Anywhere subscriptions (list from memory of the EKS API, not verified against the pinned model). +`iam:PassRole` is enforced for CreateCluster `roleArn` and CreateNodegroup `nodeRole` only (`gateway/eks/passrole.go`); CreateAddon/UpdateAddon `serviceAccountRoleArn` is not PassRole-checked (AWS requires `iam:PassRole` for it; verify before treating as a divergence). + +## 3. Durable records and KV buckets + +| Bucket | Key | Record | Owner | Writers | Readers outside EKS | +|---|---|---|---|---|---| +| `eks-account-{acct}` (history 1, migration v1, `store.go:30-38,235-245`) | `clusters/{c}/meta` | `ClusterMeta` (`cluster_state.go:65`) | cluster | `service_impl.go` create/launch/delete/tags, `cluster_state.go` CAS helpers, `cluster_reconciler.go`, `restore_snapshot.go:150`, `nats_bootstrap.go:297` (CA) | **`gateway/eks/internal_authz.go:110-124`**, **`gateway/oidc_discovery.go:38,60`** | +| same | `clusters/{c}/nodegroups/{ng}` | `NodegroupRecord` (`types.go:33`) | nodegroup | `nodegroup.go` (claim, CAS, put, delete), `service_impl.go:1964` (tags) | none observed | +| same | `clusters/{c}/access-entries/{sha256(principalARN)}` | `AccessEntryRecord` (`types.go:93`) | access | `service_impl.go:1548-1785`, creator-admin seed `service_impl.go:994-1001` (cluster launch) | **gateway process via `ResolveTokenReview` (`token_review.go:113`)** | +| same | `clusters/{c}/addons/{a}` | `AddonRecord` (`addon_state.go:29`) | addon | `addons.go`, `addon_status_reconciler.go`, `nodegroup.go:410` (GPU add-on) | none observed | +| same | `clusters/{c}/addons/{a}/manifest` | staged manifest descriptor | addon | `addon_installer.go:80-92` | guest via internal-addons route | +| same | `clusters/{c}/oidc-signing-key.pem.enc` | encrypted ECDSA key | cluster | `oidc_keypair.go:86`, zeroized `service_impl.go:1281` | none | +| same | `clusters/{c}/oidc-jwks.json` | JWKS | cluster | `oidc_keypair.go:94` | **`handlers/sts/oidc_jwks.go:110`**, **`gateway/oidc_discovery.go:93`** | +| same | `clusters/{c}/oidc-jwks-verified` | marker | cluster | `nats_bootstrap.go:291` | none | +| same | `clusters/{c}/admin-kubeconfig.enc` | encrypted kubeconfig | cluster | `nats_bootstrap.go:269` | none observed in Go (uncertain whether a CLI reads it) | +| same | `clusters/{c}/k3s-node-token.enc` | encrypted join token | cluster | `nats_bootstrap.go:251` | read by `nodegroup.go:1164` (cross-resource) | +| same | `clusters/{c}/recovery/{instanceId}` | `RecoveryDirective` | cluster | `recovery_directive.go:102`, reconciler | guest via internal-recovery route | +| same (whole bucket) | all keys | watch | cluster | n/a | **`daemon/dns_reconcile.go:68`** via `AccountWatchBuckets` | +| `spinifex-eks-leader` (TTL 60s, `store.go:40-42,249`) | `{acct}/{c}` reconciler lease; `.../teardown` lease | kvlease | cluster | `cluster_reconciler.go:391-403`, `service_impl.go:1174-1197` | none | +| `spinifex-eks-clustertokens` (TTL 15m, `clienttoken.go:16-21`) | idempotency store keyed by account+token | `idempotency.Store[string]` | cluster | `service_impl.go:529-558,612-619` | none | + +Dead key helpers: `OIDCProviderKey` (`store.go:95`), `EventKey` (`store.go:132`). +Secret material in `ClusterMeta`: `ControlPlaneTemplate` is a copy of `K3sServerInput` that clears creds but not `OIDCPrivateKeyPEM` or `JoinToken` (`service_impl.go:945-959`; fields at `k3s_server_vm.go:146,171`), so the OIDC private key and k3s join token appear to be persisted unencrypted in the meta record that the gateway also reads; no scrubbing or custom marshaller was found (verify with a KV dump before acting). +`ClusterMeta` combines desired intent (`Version`, `RoleArn`, `ResourcesVpcConfig`, `Logging`, `Tags`), realization refs (CP nodes, NLB, CP VPC, EIP, ENIs), observations (`HealthIssue`, `NodeCount`, `NodegroupNodeCounts`, `CertificateAuthorityB64`) and delete bookkeeping (`DeletingSince`, `DeleteReapAttempts`). +Hidden realizations outside EKS stores: CP VMs (system account, `ManagedBy=eks`), CP VPC/subnets/NAT-GW/route tables/IGW, NLB and target groups (ELBv2), SGs, private-endpoint ENI, egress EIP, HA placement group, worker instances (customer account), IAM roles/instance profiles, CSI EBS volumes. + +## 4. Identity and scope + +| Resource | Identifier / ARN | Account | Region | AZ | +|---|---|---|---|---| +| cluster | name; `arn:aws:eks:{region}:{acct}:cluster/{name}` (`foundation/aws/arn/eks.go:21`) | tenant account owns the record; CP VMs, CP VPC and NLB are realized in the system account `awsidentifiers.GlobalAccountID` (`service_impl.go:738`, `:2240-2248`); customer-VPC SGs and the private-endpoint ENI in the tenant account | single Region from `deps.Region`; not stored as a field | Regional; HA CP spread over hosts by AZ (`k3s_ha_control_plane.go:528`) | +| nodegroup | `arn:...:nodegroup/{c}/{ng}/{discriminator}` where the discriminator is derived from (account, cluster, name) (`nodegroup.go:333`), so a recreated nodegroup gets the same ARN, unlike AWS's per-incarnation UUID | tenant | deps.Region | AZs implied by subnets | +| addon | `arn:...:addon/{c}/{a}` (`arn/eks.go:51`, `addons.go:100`); no per-incarnation suffix (AWS appends one; verify) | tenant | deps.Region | n/a | +| access | `arn:...:access-entry/{c}/{sha256(principal)}` (`access_entry.go:244`); AWS's shape differs (verify) | tenant | deps.Region | n/a | + +Uniqueness is per account bucket and cluster-name prefix; there is no immutable internal UID distinct from the name for any EKS resource. + +## 5. Lifecycle paths per resource + +### cluster +Create: `CreateCluster` `service_impl.go:504`; validates, optional token claim (`:529-558`), resolves VPC, builds `ClusterMeta{Status: CREATING}`, `claimClusterName` CAS-create with FAILED-to-CREATING reclaim (`:1200-1265`), finalizes token, then launches `launchClusterInfra` on an in-process goroutine with `context.Background()` (`:625-645`). +Launch stages (`:718-1012`): CP VPC, SGs, CP ingress, customer SGs, private ENI, NLB, OIDC keypair, join token, CP placement/launch, persist CP refs, NLB target registration, persist final meta, creator-admin AccessEntry, `spawnBootstrap`, `spawnReconciler`; any failure calls `failClusterLaunch` (`:696`) which marks FAILED. +Readiness: `ClusterReconciler.reconcileOnce` (`cluster_reconciler.go:594`) flips CREATING to ACTIVE when bootstrap artifacts exist (`:1173-1189`) and the CP state report is healthy; CREATING becomes FAILED after `createTimeout` (`:1151-1171`). +Reconciliation/repair (ACTIVE): health recording, CP restart (`:708`), etcd reset via recovery directives (`:780`), member replacement (`:1010`), all under the per-cluster lease `spinifex-eks-leader/{acct}/{c}` (`:391-418`). +Restart: `SpawnRegisteredReconcilers` (`service_impl.go:342-425`) rescans buckets, reclaims orphaned nodegroups, resubscribes missing bootstrap kinds and respawns reconcilers; it does not resume an interrupted `launchClusterInfra`. +Delete: `DeleteCluster` (`service_impl.go:1104-1155`) returns success for an absent cluster, takes the teardown lease (losers return DELETING), sets DELETING, runs `purgeClusterInfra` synchronously in the request (`:1268-1452`), and only erases `clusters/{c}/` after all blocking steps succeed; `EKSDeletingReaper` re-drives with backoff and exhaustion (`eks_deleting_reaper.go:46-159`); `EKSBillableReaper` terminates CP VMs whose meta is gone (`eks_billable_reaper.go:46-100`). +Fencing: leases only; no desired/observed generation (no `generation` field anywhere in `handlers/eks`). +Idempotency: `clientRequestToken` honoured for CreateCluster only, 15-minute TTL, replays current meta by name. + +### nodegroup +Create: `createNodegroup` (`nodegroup.go:205`) requires cluster ACTIVE (`:226-229`), `ClaimNodegroupRecord` CAS-create (`:112-126`, ResourceInUse at `:363`), then background `launchNodegroupInfra` (`:373`, `:449-535`). +Readiness: `waitWorkersReady` (`:537-563`) polls the cluster's `NodegroupNodeCounts` observation until ready count reaches baseline plus desired, then sets ACTIVE (`:521-523`), else CREATE_FAILED. +Update: `UpdateNodegroupConfig` runs `reconcileNodegroup` inline under revision CAS (`:914-979`, `launchWorkersCAS` `:985`, `recordLaunchedWorker` `:1038`) and returns `Update{Status: Successful}` (`:900-906`). +Restart: `reclaimOrphanedNodegroups` (`:1189-1228`) terminates workers of CREATING/partial CREATE_FAILED records and marks them CREATE_FAILED; it does not resume. +Delete: `deleteNodegroup` (`:1129-1158`) terminates workers then deletes the record synchronously and returns a DELETING projection of a record that no longer exists. +Fencing: KV revision CAS; no generation, no lease. +Idempotency: no `clientRequestToken`; duplicate create loses the claim and gets ResourceInUse. + +### addon +Create: `CreateAddon` (`addons.go:65-111`) validates against the catalogue, get-then-put (`:88-105`, not CAS-create), stages manifest (`addon_installer.go:62-85`), returns CREATING. +Readiness: guest addon-sync agent publishes `eks.addon.{acct}.{c}.status` via internal-publish; `applyAddonStatusReport` (`addon_status_reconciler.go:15-47`) maps `ready` to ACTIVE and `failed` to CREATE_FAILED/DEGRADED; `report.Version` is ignored. +Update: `UpdateAddon` (`addons.go:203-252`) CAS sets UPDATING, restages, returns `Successful` with `Id = rec.Arn`. +Delete: `DeleteAddon` (`addons.go:254-285`) unstages and deletes the record immediately; the DELETING status is only in the response. +Fencing: revision CAS on the record; no generation. +Idempotency: none (`clientRequestToken` ignored). + +### access +Create: `CreateAccessEntry` (`service_impl.go:1548-1581`) get-then-put (`:1570-1578`), synchronous; creator-admin entry seeded during cluster launch (`:994-1001`). +Mutations: `casUpdateAccessEntry` (`access_entry.go:141`) for update/associate/disassociate; delete via `DeleteAccessEntryRecord`. +Effect: read at authentication time by `Authenticate`/`ResolveTokenReview` (`token_review.go:31-128`); no in-cluster realization or readiness. +Delete with cluster: swept by `DeleteClusterPrefix` (`cluster_state.go:493`). +Idempotency: none (`clientRequestToken` ignored). + +## 6. NATS subjects and guest/controller messages + +| Subject / route | Direction | Delivery | Defined | Consumer | +|---|---|---|---|---| +| `eks.` (38 subjects) | gateway (or teardown/CLI) to daemon | core NATS request/reply, queue `spinifex-workers` | `service_nats.go:30-189`, `daemon/daemon.go:1090-1127` | `EKSServiceImpl` | +| `eks.bus.{acct}.{c}.{k3s-bootstrap-token,k3s-admin-kubeconfig,k3s-oidc-jwks,k3s-ca}` | CP VM via gateway relay to daemon | core NATS publish/subscribe, at-most-once | `nats_bootstrap.go:47-57`, subscribe `:197` | `NATSBootstrap` persists to KV with `persistWithRetry` (`:32`) | +| `eks.state.{acct}.{c}.server` | CP VM via relay | core NATS, periodic | `state_report.go:13` | `ClusterReconciler` (`cluster_reconciler.go:432`) | +| `eks.addon.{acct}.{c}.status` | CP VM via relay | core NATS | `state_report.go:73` | `ClusterReconciler` (`:448`) | +| `eks.VerifyToken` | EKS (gateway process) to awsgw STS | request/reply, queue `awsgw-eks-token-verify` | `token_webhook.go:13` | `runtime/roles/awsgw/eks_token_verify.go:30` | +| `contracts/cluster/v1` `NodeStatusSubject`, `NodeVMsSubject` | EKS to every node | fan-out request | `k3s_ha_control_plane.go:502,579` | daemons | +| `ec2.RunInstances.{type}.{node}` and `contracts/ec2/v1` instance commands | EKS (daemon binding) to EC2 | request/reply | `daemon/eks_worker_launch.go` | EC2 instance owner | +| HTTP `internal-publish`, `token-review`, `internal-addons`, `internal-recovery` | guest to gateway (SigV4, system CP role) | HTTP | `gateway/eks.go:59-87` | gateway relays/queries | +| HTTP `/oidc/eks/{region}/{acct}/{c}/.well-known/openid-configuration`, `/keys` | public | HTTP, unauthenticated | `gateway/gateway.go:345-346` | `gateway/oidc_discovery.go` | + +Retry and recovery paths: guest `eks-gateway-publish` retries until HTTP 2xx (`agents/eks/gatewaypublish/publish.go:26-31,97-114`), but the gateway returns 2xx after a core-NATS publish even with no subscriber (`gateway/eks/internal_publish.go:84-89`), so a bootstrap message published while the daemon subscriber is absent is lost. +On restart, `BootstrapPendingKinds` resubscribes only missing kinds (`nats_bootstrap.go:115-139`, `service_impl.go:410-416`); a lost one-shot message is not replayed, so the cluster reaches FAILED at the create timeout. +A bootstrap subscriber that exits with error marks the cluster FAILED (`service_impl.go:2192-2212`). +SDK retries of `eks.` land on any node of the queue group; dedup relies on token store, claim CAS, teardown lease or nothing (addon/access create). + +## 7. Boundary-crossing imports + +### Outbound from `handlers/eks` +- IAM/identity: `handlers/iam` `EncryptSecret`/`DecryptSecret` (`nats_bootstrap.go`, `oidc_keypair.go`, `nodegroup.go`), `EnsureSystemInstanceProfile` (`k3s_server_role.go`), `SystemInstanceRoleEnsurer` (`service_impl.go`); `instanceProfileEnsurer` capability with `GetRole/CreateRole/PutRolePolicy/CreateInstanceProfile/AddRoleToInstanceProfile` (`service_impl.go:214`) means EKS creates IAM roles and instance profiles. +- Bluebottle: `bluebottle/pkg/auth` `ResolveRoleARN` (`nodegroup.go`) and `ParseRoleARN` (`service_impl.go`); daemon binding uses `bluebottle/pkg/masterkey`; gateway uses `bluebottle/pkg/auth` (`gateway/eks.go`). +- STS: token verification over `eks.VerifyToken` (process contract, no import). +- EC2 instance: `domains/ec2/systeminstance` (`SystemInstanceInput/Output`, `ExtraENIInput`, `BootAMI`), `domains/ec2/instancetypes`, `domains/ec2/placementgroup` (alias `ec2placementgroup`), `domains/ec2/ebs/metadata`, `runtime/compute/vm` (reapers), plus `WorkerLauncher`, `k3sInstanceLauncher`, `k3sAMIResolver`, `eipProvisioner`, `csiVolumeReclaimer` capabilities. +- Network: `domains/network/systemvpc` (`Ensure`, `Delete`, `EnsureIGW`, `DeleteIGW`, provisioner interfaces); `sgProvisioner` and `k3sVPCProvisioner` capabilities (EC2 SDK shapes). +- DNS: `domains/dns` `EKSName`, `EKSChanges`, `ResolveBaseDomain`, `Change`, `ActionUpsert` (`service_impl.go`). +- ACM/CA: no import; the platform/NATS CA PEM is passed as `GatewayCACert` from `d.config.NATS.CACert` (`daemon/eks_deps.go:64-71`) and baked into CP user data; the k3s cluster CA is self-generated in the guest and returned on `k3s-ca`. +- ELBv2: `nlbProvisioner` capability with ELBv2 SDK shapes (`nlb.go:19`), bound to `d.elbv2Service`. +- Storage/object: `providers/objectstore` (etcd snapshots), Predastore system creds via deps. +- Contracts: `contracts/cluster/v1`. +- Foundation: `kvutil`, `kvstore`, `kvlease`, `migrate`, `idempotency`, `reconciler`, `telemetry`, `aws/{arn,errors,tags,identifiers,ami}`, `messaging/nats`. +- Bootstrap: `bootstrap/config.Config` in `EKSServiceDeps`. + +### Inbound to `handlers/eks` +- `gateway/eks/*`: `NewNATSEKSService`, input/output types, `PrincipalARNHash`, `ClusterMeta`, `ClusterMetaKey`, `AccountBucketName`, `CPInstanceRoleName`, `ResolveTokenReview`, bootstrap/state/addon subject builders. +- `gateway/oidc_discovery.go`: `AccountBucketName`, `GetClusterMeta`, `OIDCJWKSKey`. +- `handlers/sts/oidc_jwks.go`: `AccountBucketName`, `OIDCJWKSKey` (STS reads EKS private KV). +- `runtime/roles/awsgw/eks_token_verify.go`: `TokenVerifySubject`, `TokenVerifyRequest`, `TokenVerifyResponse`. +- `agents/eks/tokenwebhook/webhook.go`: `WebhookTokenReviewResult` (guest imports domain). +- `daemon/*`: `EKSServiceImpl`, `NewEKSServiceImpl`, `EKSServiceDeps`, `NewNATSHostScheduler`, `SubnetVPCResolver`, `WorkerLauncher`, `AccountWatchBuckets`. +- `accountteardown/reapers_eks.go`: `EKSService`, `NewNATSEKSService`. +- `operator/cli/admin_eks.go`: `NewNATSEKSService`, `RestoreSnapshotInput`. + +### Reverse knowledge of EKS in other domains (comments or tag conventions, no import) +`domains/ec2/instance/service_impl.go:394-396` (worker tag comment), `domains/ec2/systeminstance/launcher.go:4,25`, `handlers/elbv2/{launcher.go:5,service_impl.go:1232-1250,service_impl_nlb_sg.go:196}`, `domains/ec2/vpc/eni.go:54`, `domains/ec2/guestmetadata/resolver.go:37`, `domains/dns/{naming.go:118-124,reconcile.go:58,303}`, `handlers/iam/managed_policies.go` (EKS managed policies), `accountteardown/reapers_iam.go:26-33` (OIDC providers). + +## 8. Known gaps against Q-78 / ADR-0003 (observed) + +1. No desired or observed generation for any EKS resource; fencing is leases (cluster) or record-revision CAS (nodegroup, addon, access). +2. Intent and observation share `ClusterMeta` (`cluster_state.go:65-190`), and nodegroup readiness reads a cluster observation (`nodegroup.go:537-549`). +3. Cluster realization runs in request-spawned goroutines (`service_impl.go:625-645`); restart does not resume the launch, so an interrupted create ends FAILED after timeout (`cluster_reconciler.go:1151-1171`) rather than converging the same intent. +4. CP VMs are live before their IDs are persisted (`service_impl.go:960-970` comment); a crash in that window leaves VMs reachable only by the billable reaper after meta removal. +5. Bootstrap artifacts travel on at-most-once core NATS (`nats_bootstrap.go:197`, `gateway/eks/internal_publish.go:84`); a lost message is not replayed. +6. DeleteCluster purges synchronously inside the request (`service_impl.go:1141-1154`) rather than accepting asynchronously as AWS does. +7. DeleteCluster does not reject a cluster with nodegroups (AWS returns ResourceInUseException) and `DeleteClusterPrefix` erases nodegroup records while their worker instances keep running; `accountteardown/reapers_eks.go:15-18` documents this; workers carry no `ManagedBy` tag (`nodegroup.go:669-670`) and no reaper reclaims them (uncertain whether the e2e suite exercises this order). +8. DeleteNodegroup is synchronous and removes the record immediately (`nodegroup.go:1145-1157`), so the resource is absent before EC2 termination completes; no durable deletion intent. +9. UpdateNodegroupConfig and UpdateAddon return `Successful` synchronously (`nodegroup.go:900-906`, `addons.go:246-251`) and `DescribeUpdate`/`ListUpdates` are stubs, so there is no truthful pending update. +10. CreateAddon and CreateAccessEntry use get-then-put (`addons.go:88-105`, `service_impl.go:1570-1578`), so concurrent creates can both succeed. +11. Addon status reports are not fenced by version (`addon_status_reconciler.go:15-47` ignores `AddonStatusReport.Version`), so a stale report can flip UPDATING to ACTIVE. +12. DeleteAddon deletes the record before in-cluster removal is observed (`addons.go:270-285`). +13. `clientRequestToken` is honoured only by CreateCluster; CreateNodegroup, CreateAddon, CreateAccessEntry, UpdateNodegroupConfig, UpdateAddon ignore it. +14. Nodegroup and addon ARNs are deterministic by name, so identifiers are reused across incarnations (Q-78 references Q-43 identifier-reuse rules). +15. ADR-0004 S3 inventory: EKS has no `Stubbed`/`Unsupported` classification (`gateway/operations.go:34-36`) although nine actions return NotImplemented, and four internal routes are mixed into the AWS action inventory. +16. ADR-0003 S4 / ADR-0004 S4: three non-owner readers of EKS private KV (`gateway/eks/internal_authz.go`, `gateway/oidc_discovery.go`, `handlers/sts/oidc_jwks.go`) plus `daemon/dns_reconcile.go` watching all EKS buckets. +17. ADR-0004 S3: `daemon/eks_cp_control.go` uses the EC2 AWS adapter package as an internal API. +18. ADR-0004 S5: guest `agents/eks/tokenwebhook` imports `handlers/eks`. +19. Observed security risk outside Q-78: plaintext OIDC private key and k3s join token in `ClusterMeta.ControlPlaneTemplate` (Section 3). +20. Comment hygiene: `eks_billable_reaper.go:3` cites an unaccepted "ADR-0006"; `daemon/daemon.go:1085-1087` is stale. + +## 9. First-resource candidates + +| Rank | Resource | Approx. size | Private state | Dependencies | Tests | Extractable without EKS identity/IAM/Bluebottle? | +|---|---|---|---|---|---|---| +| 1 | access | about 670 lines (`access_entry.go` 255, `service_impl.go:1527-1816` about 290, `token_review.go` 128, parts of `types.go`) | `clusters/{c}/access-entries/*` only | cluster existence query (`acctKVForCluster` reads `ClusterMeta`), cluster launch seeds creator admin, gateway authz needs `PrincipalARNHash`, gateway token-review calls `ResolveTokenReview`, STS verify contract | `access_entry_test.go`, `service_impl_test.go` (access cases), `token_review_test.go`, `token_webhook_test.go`, `gateway/eks/authz_test.go`, `gateway/eks/token_review_test.go`, e2e `TestEKS/AccessEntry`, `GetToken` | The record CRUD and policy catalogue are extractable now with a narrow cluster-exists query; the token-review part touches STS identity (`eks.VerifyToken` contract) but not IAM or Bluebottle, so it can stay behind its existing NATS contract. | +| 2 | addon | about 830 lines (`addons.go`, `addon_state.go`, `addon_installer.go`, `addon_status_reconciler.go`, `addon_catalog.go`) | `clusters/{c}/addons/*` incl. manifests | cluster existence query, cluster reconciler relays `eks.addon.*` reports, guest internal-addons route, nodegroup GPU staging, IRSA role ARN stored only | `addons_test.go`, `addon_state_test.go`, `addon_status_reconciler_test.go`, `addon_catalog_test.go`, `gateway/eks/wrappers_test.go`, e2e `AddonDelivery` | Yes for IAM/Bluebottle; needs a report-relay seam out of `ClusterReconciler` and a guest contract for `StagedAddonManifest`/`AddonStatusReport`. | +| 3 | nodegroup | about 1,650 lines (`nodegroup.go`, `nodegroup_userdata.go`, `claim.go`, part of `types.go`, nodegroup SG rules) | `clusters/{c}/nodegroups/*` | cluster meta (ACTIVE, CP ENI IP, VPC, readiness counts), cluster join token decrypt (`handlers_iam.DecryptSecret`), IAM role/profile creation, Bluebottle `ResolveRoleARN`, EC2 worker launch, AMI lookup, cluster SGs, GPU addon | `nodegroup_test.go` (39 tests), `nodegroup_recreate_test.go`, `nodegroup_instance_profile_test.go`, `nodegroup_userdata_test.go`, `daemon/eks_worker_launch_test.go`, e2e IRSA pod path | No; it needs IAM, Bluebottle and cluster secrets capabilities first. | +| 4 | cluster | about 8,500 lines (remainder of `handlers/eks`) | meta, OIDC material, kubeconfig, join token, recovery directives, leader/token buckets | every external owner (EC2, VPC, ELBv2, DNS, IAM, objectstore, placement, scheduler) and all guest contracts | `create_cluster_test.go`, `delete_cluster_test.go`, `cluster_reconciler*_test.go` (5 files), `lifecycle_invariants_test.go` (RLC1-3), reapers, `nats_bootstrap_test.go`, `oidc_keypair_test.go`, `k3s_*_test.go`, `nlb_test.go`, `security_groups_test.go`, `cp_vpc_test.go`, `restore_snapshot_test.go`, e2e `TestEKS` | No; it is the orchestration hub and depends on identity, Bluebottle and the CA. | + +Recommendation: extract `access` first, defining a cluster-owned "cluster exists and is not deleting" query and moving `PrincipalARNHash` and the key helper into the access package. +Its shortcomings (gap 10, no client token, no generation) are small and can be recorded in its lifecycle contract without first resolving cluster realization. diff --git a/docs/package-boundary/adr-0004-inventory-rds.md b/docs/package-boundary/adr-0004-inventory-rds.md new file mode 100644 index 000000000..ffc2dd62f --- /dev/null +++ b/docs/package-boundary/adr-0004-inventory-rds.md @@ -0,0 +1,310 @@ +# ADR-0004 source inventory — RDS + +Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`. +It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it. +Items marked "verify" or "uncertain" are unconfirmed. +Gaps record observed behaviour, not target behaviour. + + +Repo: `/home/tf-user/Development/mulga/spinifex`, branch `refactor/spinifex-package-boundaries`, paths relative to `spinifex/`. +Inputs read: ADR-0001, ADR-0003, ADR-0004 and Q-78 from `origin/main`. +Method: static reading of production code only; nothing here was verified against a live cluster. +Line references are `file:line` at the time of reading. +"Uncertain" marks a claim inferred from code shape rather than traced end to end. + +## 0. Scope findings + +`daemon/instance_records.go` (217) and `daemon/instance_records_migrate.go` (361) are **not RDS**. +They own the EC2 per-instance record space `i.` in the EC2 instance-state bucket (`daemon/instance_records.go:25`) and its migrations (`daemon/instance_records_migrate.go:29-70`), and import only `runtime/compute/vm`, `kvstore`, `kvutil`, `migrate`. +They are excluded from the RDS inventory and belong to the EC2 instance owner. + +Other production files that wire or consume RDS, found by import search: + +- `daemon/daemon.go` (RDS sections only: `:170-171`, `:1171-1206`, `:2046-2078`, `:2306-2312`, `:2374-2382`). +- `daemon/rds_deps.go`. +- `daemon/dns_reconcile.go:71-76` and `:99-104` (RDS bucket watch and RDS DNS desired set). +- `daemon/bedrock_deps.go:43` (Ochre/Bedrock borrows `handlers_rds.NewNATSVolumeAttacher`). +- `handlers/quota/live.go:82-93` (`EnforceRDSInstances`). +- `accountteardown/reapers_rds.go` (instance, subnet-group, parameter-group reapers). +- `gateway/rds.go`, `gateway/operations.go:43-45`, `gateway/gateway.go:302,525-526` (service map and dispatch). +- `bootstrap/config/config.go:338` (`RDSConfig`). +- Test-support (non-`_test.go` but not production): `tests/integration/service_daemon_lite.go:51` reflects every `Service` method onto `rds.`; `tests/e2e/harness/rds*.go`. +- Counterparty only: `agents/rds/agent/*.go` and `cmd/rds-agent` import `handlers/rds` for wire types. + +## 1. File table + +Total in-scope production lines: 16,468 (handlers/rds 14,873; gateway/rds 853; gateway/rds.go 95; daemon/rds_deps.go 98; daemon/instance_records* 578, not RDS). + +### handlers/rds + +| File | Lines | Class | Reason | +|---|---|---|---| +| `store.go` | 451 | temporary residue (prerequisite: per-resource packages exist; removal: each resource owns its own key helpers and bucket view) | One file holds every resource's key space, bucket configs, migrations hook, leader bucket and cross-tenant enumeration (`:20-47`, `:39-50` key list). **Mixed**: split into instance (`db-instances/`, `bootstrap-payloads/`, `instance-index/`, leader bucket), snapshot (`db-snapshots/`, `backups/`, `retained-volumes/`), subnetgroup (`db-subnet-groups/`), parametergroup (`db-parameter-groups/`), events (`events/`), plus a domain-root bucket factory. | +| `records.go` | 501 | temporary residue (same prerequisite) | All durable record structs in one file. **Mixed**: `DBInstanceRecord`/`PendingModifiedValues`/`ModifyLease`/`BootstrapState`/`AgentState`/`SnapshotOperation` → instance; `DBSnapshotRecord`/`AutomatedBackupRecord`/`RetainedVolumeRecord` → snapshot; `DBSubnetGroupRecord` → subnetgroup; `DBParameterGroupRecord`/`DBParameterRecord` → parametergroup; `Parameter`, `EngineHealth`, `Heartbeat*` → guest/controller wire contract. | +| `service.go` | 314 | temporary residue (prerequisite: resource owners; removal: each owner holds its own deps, domain root holds composition) | Single `Service` with one `Deps` bag for every resource (`:19-45` in stripped view), per-account bucket cache and generic JSON CAS helpers. **Mixed**: `Deps` splits by resource; CAS helpers could move to `foundation/state/kvstore` (already exists) rather than a new common package. | +| `service_nats.go` | 189 | temporary residue (prerequisite: awsapi calls owners in-process or via a domain contract; removal: no AWS-SDK-typed NATS client used as internal API) | AWS-SDK-shaped NATS client used by the gateway, quota (`handlers/quota/live.go:89`) and account teardown (`accountteardown/reapers_rds.go:19`). It is currently the de facto internal API, which ADR-0004 S3/S4 forbid for quota. | +| `subjects.go` | 90 | **Mixed**: guest/controller wire contract (`rds.bus.*`, `CommandQueueGroup`, `:6-19`, `:65-90`) + temporary residue (Layer-1 `rds.` gateway→daemon subjects, `:21-63`) | Bus subjects are the real agent-path process boundary; the per-action subjects exist only because the gateway and daemon are separate dispatch hops. | +| `arn.go` | 92 | AWS protocol adapter (with per-resource kinds) | `FormatARN`/`ParseARN`/`ResourceKind` (`:15-40`) are used for projection, tags and gateway authz scopes. Kinds could live with each resource; parsing belongs in awsapi. | +| `filters.go` | 68 | AWS protocol adapter | Generic AWS `Filter` parsing (`ReadFilters`, `:38`), used by gateway catalogs and describes. | +| `paging.go` | 74 | AWS protocol adapter | AWS `MaxRecords`/`Marker` paging (`:23`) and per-type page keys (`:52-74`). | +| `validate.go` | 271 | resource owner (instance) — partly AWS protocol adapter | `CreateDBInstance` request validation (`:69`), identifier rules (`:196`), unimplemented-parameter rejection (`:225`). **Mixed**: SDK input parsing → awsapi; identifier/engine/class invariants → instance. | +| `status.go` | 69 | resource owner (instance) | DB instance status enum and transition table (`:7-60`). | +| `create.go` | 305 | resource owner (instance) | `CreateDBInstance` orchestration, record reservation, rollback (`:29-147`, `:195-305`). | +| `describe.go` | 298 | **Mixed**: resource owner (instance) query + AWS protocol adapter (projection to `rds.DBInstance`, `:145-298`) | Projection is awsapi; record read/filter is the owner's query. | +| `lifecycle.go` | 454 | resource owner (instance) — **contains an EC2 adapter** | Reboot/Stop/Start (`:57-162`), transition helpers (`:276-405`). `natsInstanceCommander` (`:407-454`) sends `ec2v1` instance commands and `ec2.start`; that is an EC2-owned capability implemented inside RDS (residue). | +| `launch.go` | 629 | cross-resource orchestration (instance realization over EC2/VPC/EBS/IAM) — **contains an EC2 adapter** | `LaunchDBInstanceVM` (`:178-366`) composes system instance, two ENIs, data volume, AMI lookup. `natsVolumeAttacher` (`:560-629`) is an EC2 volume-attach NATS client that Bedrock also imports (`daemon/bedrock_deps.go:43`); it is EC2/EBS residue living in RDS. | +| `network.go` | 200 | resource owner (instance) with cross-resource read of subnetgroup | Endpoint placement (`:36-75`) reads the subnet-group record directly (`:43`) and EC2 VPC/subnet/SG via `networkResolver` (`:16-20`). | +| `vpc.go` | 163 | cross-resource orchestration (system VPC/SG realization for instances) | `EnsureSystemVPC`/`EnsureSystemSecurityGroup` (`:78`, `:104`) over `domains/network/systemvpc`. | +| `storage.go` | 155 | resource owner (instance) | Storage grow via `volumeResizer` (EC2 `DescribeVolumes`/`ModifyVolume`, `:23-26`). | +| `modify.go` | 528 | resource owner (instance) | `ModifyDBInstance` plan/immediate/pending paths (`:76-198`, `:199-528`). | +| `modify_lease.go` | 173 | resource owner (instance) | Per-instance modify lease stored on the instance record (`:24-173`). | +| `pending.go` | 256 | resource owner (instance) — with parametergroup orchestration | Applies `PendingModifiedValues` (`:28`), parameter-group apply to an instance (`:149`), filesystem grow finish (`:240`). `applyParameterGroup` is the instance side of parameter-group propagation. | +| `replace.go` | 252 | resource owner (instance) — replacement | VM replace for class/storage change (`:43-166`), index rewrite (`:200`). | +| `recovery.go` | 220 | resource owner (instance) — failure detection | Health classification (`:54`, `:76`); reports failure, does not repair (`:19-21`). | +| `reconciler.go` | 896 | **Mixed**: resource owner (instance reconcile, `:238-660`, `:797-866`) + snapshot reconcile (`:674-796`) + foundation wiring (leader lease, `:83-188`) | Split: instance reconciler, snapshot creating-resolver, domain-root leader/loop. `describeInstanceState` (`:867-896`) is an EC2 projection adapter. | +| `window.go` | 356 | resource owner (instance) — maintenance/backup windows | Pure window arithmetic for an instance's backup and maintenance windows. | +| `backup.go` | 597 | **Mixed**: resource owner (instance: retention, windows, `runBackupWindow` `:206`, `runMaintenanceWindow` `:340`) + snapshot (automated snapshot identity `:385-394`) + AWS adapter (`DescribeDBInstanceAutomatedBackups` `:396-536`) | Automated backup is an instance obligation that produces snapshot records. | +| `backup_sweep.go` | 340 | resource owner (snapshot) — retention reaper | `BackupRetentionReaper` (`:24-112`) deletes automated snapshots and reclaims retained volumes (`:310`); implements `vm.Reaper` from `runtime/compute/vm` (`:31`). | +| `snapshot.go` | 768 | resource owner (snapshot) — tightly coupled to instance | Create/Describe/Delete (`:49`, `:224`, `:299`); create CASes the instance into `backing-up` (`:672-723`) and quiesces the engine. | +| `restore.go` | 411 | cross-resource orchestration (snapshot → new instance) | `RestoreDBInstanceFromDBSnapshot` (`:24-162`) reads a snapshot record, creates a volume from it and runs the instance create path. | +| `delete.go` | 548 | resource owner (instance) — with snapshot orchestration | `DeleteDBInstance` (`:34`), teardown (`:178-227`), final snapshot (`:247-447`), retained data volume (`:449-535`). | +| `agent.go` | 667 | **Mixed**: guest/controller wire contract (IO types, `Command`, `CommandReply`, `InstanceIndexEntry`, `:30-160`, `:628-636`) + resource owner (instance agent-path handlers `:161-627`) | Split: `contracts/rds/v1` vocabulary vs instance owner handlers. | +| `commands.go` | 239 | **Mixed**: guest/controller wire contract (command types and params, `:12-40`) + resource owner (instance: `issueCommand` issuer, `:86-230`) | Issuer is instance realization; constants are wire vocabulary. | +| `bootstrap.go` | 280 | resource owner (instance) — bootstrap secret | Encrypted master-password payload (`:31-280`), encrypted via `handlers_iam.EncryptSecret` (`:105`, `:199`). | +| `servingcert.go` | 109 | resource owner (instance) — realization helper | Per-instance serving cert minted from cluster CA (`:46`). Uncertain whether this should sit with a PKI owner instead. | +| `userdata.go` | 68 | guest/controller wire contract | Agent user-data rendering (`:36`), the boot-time contract with the DB VM. | +| `iam.go` | 83 | **Mixed**: resource owner (instance — instance profile ensure, `:56-83`) + guest/controller wire contract (`InternalAgentActions`, `:24-30`) | The internal-action list is shared by the gateway gate (`gateway/rds/authz.go:52-63`) and the role grant. | +| `dns.go` | 72 | resource owner (instance) — publishes a projection | `DesiredDNSChanges` (`:25-51`) is an RDS-owned projection consumed by the daemon DNS reconcile. | +| `events.go` | 312 | cross-resource orchestration (event journal) — uncertain final home | Per-source event rings for instance and snapshot (`:88-141`) plus `DescribeEvents` (`:159`). Not in the ADR-0004 resource table; candidate for the domain root, not a `common` package. | +| `tags.go` | 280 | cross-resource orchestration + AWS adapter | One tag implementation dispatching on ARN kind over all four record types (`:51-73`, `:202-230`). Split: each owner exposes a tag mutation on its own record; awsapi routes by ARN. | +| `engine.go` | 287 | resource owner (engine) — read-only catalogue | Engine registry, version/username/dbname/password rules (`:15-287`); consumed by the agent (`LookupEngine`). | +| `engine_postgres.go` | 505 | resource owner (engine) | Postgres engine catalogue and parameter specs/validators. | +| `engine_mariadb.go` | 503 | resource owner (engine) | MariaDB engine catalogue and parameter specs/validators. | +| `paramcatalog.go` | 449 | resource owner (engine) | Parameter spec model, size-derived defaults, `ResolveEffectiveParameters` (`:354`); imports `domains/ec2/instancetypes` for class memory (`:219`). | +| `catalog.go` | 214 | **Mixed**: resource owner (engine — `EngineVersions`, `OrderableOptions` `:102-141`) + AWS protocol adapter (filter value sets, SDK projection `:142-214`) | | +| `sizing.go` | 57 | resource owner (engine/instance-class facade) | `db.*` → EC2 instance type map (`:14-21`), validated against `domains/ec2/instancetypes` (`:30`). | +| `subnetgroup.go` | 340 | resource owner (subnetgroup) | Create/Describe/Delete (`:39`, `:94`, `:138`), subnet resolution via EC2 (`:176`). Reads instance records for in-use check (`:258-277`). | +| `parametergroup.go` | 599 | resource owner (parametergroup) — with instance orchestration | CRUD + DescribeDBParameters (`:29-380`); `propagateParameterGroup` (`:210-236`) pushes changes into instances; `resolveGroupParameters` (`:549`) is called by instance create/modify/restore. | + +### gateway/rds and gateway/rds.go + +| File | Lines | Class | Reason | +|---|---|---|---| +| `gateway/rds.go` | 95 | AWS protocol adapter (ingress glue) | Query parse, action check, caller derivation, authz order, quota injection and `ExpectedNodes` injection (`:16-75`). Imports `bluebottle/pkg/auth.ParseRoleARN` (`:87`). | +| `gateway/rds/handler.go` | 220 | AWS protocol adapter | Action inventory (registered/internal/unsupported, `:96-169`), typed Query→XML adapter (`:47-79`), `Env{ExpectedNodes, QuotaCheck}` (`:33-43`). | +| `gateway/rds/authz.go` | 112 | AWS protocol adapter | Principal-class gate and per-action resource ARN scopes (`:20-112`). | +| `gateway/rds/instance.go` | 50 | AWS protocol adapter | Instance actions + `DescribeEvents`; runs `QuotaCheck` before forwarding create (`:13-19`). | +| `gateway/rds/snapshot.go` | 35 | AWS protocol adapter | Forwards snapshot/restore/automated-backup actions. | +| `gateway/rds/subnetgroup.go` | 23 | AWS protocol adapter | Forwards subnet-group actions. | +| `gateway/rds/parametergroup.go` | 31 | AWS protocol adapter | Forwards parameter-group actions. | +| `gateway/rds/tags.go` | 23 | AWS protocol adapter | Forwards tag actions. | +| `gateway/rds/enginecatalog.go` | 184 | **Mixed**: AWS protocol adapter (engine catalogue actions) + temporary residue (prerequisite: EC2-owned instance-type-availability projection; removal: `clusterRunnableTypes` replaced by that projection) | `clusterRunnableTypes` (`:146-174`) calls the EC2 AWS adapter `ec2instanceapi.DescribeInstanceTypes` with `env.ExpectedNodes`. | +| `gateway/rds/agent.go` | 287 | **Mixed**: guest/controller wire contract (agent-facing internal actions, long poll, reply relay `:146-287`) + AWS protocol adapter (agent identity gate `:36-111`) | Reads the `rds-system` bucket directly via JetStream (`:72-100`), i.e. the gateway reads RDS private state. | + +### daemon and other wiring + +| File | Lines | Class | Reason | +|---|---|---|---| +| `daemon/rds_deps.go` | 98 | cross-resource orchestration (composition root) — contains residue | Binds EC2/VPC/IGW/RT/NGW/EIP/image/volume/snapshot services, IAM ensurer, master key and config into `handlers_rds.Deps` (`:20-85`). `describeInstancesFanOut` (`:90-97`) calls the EC2 AWS adapter with `len(clusterConfig.Nodes)` as expected-node count (residue; same blocker family as `ExpectedNodes`). Loads `bluebottle/pkg/masterkey` (`:52`). | +| `daemon/daemon.go` (RDS sections) | n/a | cross-resource orchestration (composition root) | Subscriptions (`:1171-1206`), service init (`:2046-2066`), reconciler start (`:2070-2078`), backup reaper registration (`:2310-2312`), and the RDS reconciler lease reused as the cluster-wide GC election (`:2374-2382`). The last is residue: an RDS lease elects unrelated reapers. | +| `daemon/dns_reconcile.go` (`:71-76`, `:99-104`) | n/a | cross-resource orchestration | Watches RDS per-account buckets via `handlers_rds.AccountWatchBuckets` and consumes `DesiredDNSChanges`. Bucket watch is a read of RDS private storage by the daemon. | +| `daemon/bedrock_deps.go:43` | n/a | temporary residue (prerequisite: EC2/EBS-owned attach capability; removal: Bedrock and RDS both bind the EC2 one) | Another domain imports RDS for an EC2 volume-attach client. | +| `handlers/quota/live.go:82-93` | n/a | temporary residue (prerequisite: RDS-owned count projection per ADR-0004 S4; removal: quota no longer calls `NATSService.DescribeDBInstances`) | Quota counts instances through the AWS-shaped RDS API. | +| `accountteardown/reapers_rds.go` | n/a | temporary residue / cross-domain consumer | Drives instance/subnet-group/parameter-group deletion through `NATSService` AWS-shaped calls (`:41`, `:76`, `:87`, `:109`, `:125`, `:139`, `:159`). Account teardown is explicitly out of ADR-0004 scope. | +| `daemon/instance_records.go`, `daemon/instance_records_migrate.go` | 217 / 361 | not RDS | EC2 instance record space; excluded. | + +## 2. Served AWS actions and protocol + +Protocol: AWS Query in (form `Action=`), XML out using the IAM-style `` envelope (`gateway/rds.go:14-26`, `gateway/rds/handler.go:59-75`). +Order: unknown action → `InvalidAction` before policy (`gateway/rds.go:29-32`); principal-class gate before policy (`:41-43`); resource ARN derivation and policy (`:45-51`); dispatch (`:57-64`). +Unsupported actions return `OperationNotSupported` from `Dispatch` (`gateway/rds/handler.go:182-185`). +Path: gateway adapter → NATS request `rds.` (queue group `spinifex-workers`) → daemon `Service` method (`daemon/daemon.go:1173-1206`), except where noted. + +| Action | Status | Owning resource | Notes | +|---|---|---|---| +| CreateDBInstance | registered | instance | `QuotaCheck` in gateway before forward (`gateway/rds/instance.go:13-19`); 5 min budget (`service_nats.go:16`). | +| DescribeDBInstances | registered | instance | | +| ModifyDBInstance | registered, scoped | instance | Also binds parametergroup. | +| DeleteDBInstance | registered, scoped | instance | Orchestrates final snapshot. | +| RebootDBInstance / StartDBInstance / StopDBInstance | registered, scoped | instance | | +| CreateDBSnapshot | registered, scopes instance + snapshot | snapshot (orchestrates instance) | | +| DescribeDBSnapshots | registered | snapshot | | +| DeleteDBSnapshot | registered, scoped | snapshot | | +| RestoreDBInstanceFromDBSnapshot | registered, scopes snapshot + instance | cross-resource (snapshot → instance) | | +| DescribeDBInstanceAutomatedBackups | registered | instance (automated backup) / snapshot | | +| CreateDBSubnetGroup / DescribeDBSubnetGroups | registered | subnetgroup | | +| DeleteDBSubnetGroup | registered, scoped | subnetgroup | | +| CreateDBParameterGroup / DescribeDBParameterGroups | registered | parametergroup | | +| ModifyDBParameterGroup / DescribeDBParameters / DeleteDBParameterGroup | registered, scoped | parametergroup | Modify propagates to instances. | +| AddTagsToResource / RemoveTagsFromResource / ListTagsForResource | registered, ARN scope | per ARN kind (all four) | `tags.go:51-73`. | +| DescribeEvents | registered | cross-resource (events) | | +| DescribeDBEngineVersions | registered | engine | Served **in the gateway process** with no NATS hop (`gateway/rds/enginecatalog.go:41-73`). | +| DescribeOrderableDBInstanceOptions | registered (`typedEnv`) | engine (+ EC2 availability) | Served in the gateway; NATS fan-out to EC2 `DescribeInstanceTypes` using `ExpectedNodes` (`gateway/rds/enginecatalog.go:146-174`). | +| RegisterDBInstance, SubmitDBStateChange | registered, internal | instance (agent path) | Gateway → bus `rds.bus.{acct}.{db}.register/health` (`service_nats.go:49-57`). | +| GetDBBootstrapConfig, AcknowledgeDBBootstrap | registered, internal | instance (bootstrap) | Gateway → `rds.GetDBBootstrapConfig` / `rds.AcknowledgeDBBootstrap`. | +| PollDBCommands | registered, internal | instance (command channel) | Served **in the gateway**: queue-subscribes the bus command subject; no daemon handler (`gateway/rds/agent.go:182-247`). | +| CreateDBInstanceReadReplica, PromoteReadReplica, CreateDBCluster, ModifyDBCluster, DeleteDBCluster, DescribeDBClusters, FailoverDBCluster, CreateOptionGroup, ModifyOptionGroup, DeleteOptionGroup, DescribeOptionGroups, RestoreDBInstanceToPointInTime | unsupported (12) | n/a | `gateway/rds/handler.go:157-168`. | + +Totals: 26 customer actions registered, 5 internal agent actions, 12 unsupported, 43 in the inventory (`ActionNames`/`UnsupportedActionNames` feed `gateway/operations.go:43-45`). +Daemon subscribes 29 `rds.*` subjects (`daemon/daemon.go:1175-1205`); the two catalogue actions and `PollDBCommands` have no daemon subject. +Any action outside the table and not in the map is `InvalidAction`. + +## 3. Durable records and KV buckets + +| Bucket | Key | Record | Owning resource (proposed) | Writers | Readers (flag = outside RDS owner) | +|---|---|---|---|---|---| +| `rds-account-{accountID}` (history 1, version 1, lazily created; `store.go:20-38`, `:201-213`) | `db-instances/{id}` | `DBInstanceRecord` | instance | create, restore, lifecycle, modify, pending, replace, agent, delete, reconciler, recovery, backup, snapshot (`backing-up` CAS), tags | subnetgroup (`subnetgroup.go:258-277`), parametergroup (`parametergroup.go:211`, `:342`), dns.go, **daemon DNS reconcile via bucket watch** (`daemon/dns_reconcile.go:75`), **quota** indirectly via `DescribeDBInstances` (`handlers/quota/live.go:89`), **account teardown** via `NATSService` | +| same | `bootstrap-payloads/{id}` | `BootstrapPayloadEnvelope` (encrypted) | instance | create, restore, replace (via bootstrap.go) | agent.go (`GetDBBootstrapConfig`, `AcknowledgeDBBootstrap`), reconciler (stale pending report) | +| same | `db-snapshots/{id}` (colon → `.`, `store.go:84-97`) | `DBSnapshotRecord` (manual and automated) | snapshot | snapshot.go, delete.go (final snapshot), backup.go (automated) | restore.go, backup_sweep.go, reconciler snapshot pass, describe, tags | +| same | `backups/{dbId}/automated/{ts}` | `AutomatedBackupRecord` | snapshot (automated-backup index) — instance obligation | backup.go | backup_sweep.go, DescribeDBInstanceAutomatedBackups | +| same | `retained-volumes/{volumeID}` | `RetainedVolumeRecord` | snapshot (cleanup obligation for a former instance's volume) | delete.go `retainDataVolume` (`:484`) | snapshot.go `releaseRetainedVolume`/`reclaimRetainedVolume` (`:414`, `:430`), backup_sweep `reclaimOrphanedVolumes` (`:310`) | +| same | `db-subnet-groups/{name}` | `DBSubnetGroupRecord` | subnetgroup | subnetgroup.go, tags.go | network.go `resolvePlacement` (`:43`, instance create/restore), describe, **account teardown** via `NATSService` | +| same | `db-parameter-groups/{name}/meta` | `DBParameterGroupRecord` | parametergroup | parametergroup.go, tags.go | instance create/modify/restore via `resolveGroupParameters` (`parametergroup.go:549`), pending.go | +| same | `db-parameter-groups/{name}/params/{key}` | `DBParameterRecord` | parametergroup | parametergroup.go (`:188`, delete `:361`) | `ListDBParameterOverrides` (`store.go`), instance apply paths | +| same | `events/{sourceType}/{sourceId}` | event ring (bounded, 14-day trim) | events (cross-resource) | `RecordEvent` from every resource (`events.go:88`) | `DescribeEvents` | +| `rds-system` (history 1, version 1; `store.go:24-26`, `:184-195`) | `instance-index/{ec2InstanceID}` | `InstanceIndexEntry{AccountID, DBInstanceIdentifier, VMGeneration}` (`agent.go:628-636`) | instance (agent identity binding) | create (`create.go:124`), restore (`restore.go:142`), replace (`replace.go:200-215`), delete teardown | **gateway reads JetStream directly** (`gateway/rds/agent.go:72-100`); `Service.LookupInstanceIndex` (`agent.go:637`) | +| `spinifex-rds-leader` (TTL 60s; `store.go:28-33`, `:216-223`) | `reconciler` | kvlease record | domain-root reconciler leadership | `kvlease` via `Reconciler` (`reconciler.go:104-118`) | **daemon GC uses it as the cluster-wide reaper election** (`daemon/daemon.go:2374-2382`) | + +Non-durable state that affects decisions: in-memory heartbeat liveness map per daemon (`service.go` `liveness`, `agent.go:284-321`), persisted to KV only on change or every `HeartbeatPersistFloor` (5 beats, `records.go` constants). +The reconciler leader reads this node-local map (`recovery.go:191`), so a beat answered by another node's queue-group member is seen only via the KV floor (uncertain impact; observation, not intent). + +## 4. Identity and scope per resource + +| Resource | Identifier | Immutable internal ID | ARN | Account | Region | AZ | +|---|---|---|---|---|---|---| +| instance | `DBInstanceIdentifier`, account-unique via `createJSONRevision` (`create.go:62`), reusable after delete | `DbiResourceID` `db-…` (`create.go:153`) | `arn:aws:rds:{region}:{acct}:db:{id}` (`arn.go:15-27`) | customer account in bucket name and record; VM, system ENI and IAM profile live in the system account `GlobalAccountID` (`create.go:51-55`, `delete.go:544-546`) | not stored; taken from `Service.region` at projection (`service.go`) | not recorded on the instance; derived only via subnet (uncertain whether projected at all; no `AvailabilityZone` reference in `describe.go`) | +| snapshot | `DBSnapshotIdentifier`; automated ones `rds:{db}-{ts}` (`backup.go:385`) | none beyond backing EC2 `SnapshotID` | `…:snapshot:{id}` | customer | from service | none | +| subnetgroup | `DBSubnetGroupName` (case-sensitive key; `default` rejected, `subnetgroup.go:318`) | none | `…:subgrp:{name}` | customer | from service | each member subnet carries its AZ (`records.go` `DBSubnetGroupSubnet`) | +| parametergroup | `DBParameterGroupName`; `default.*` synthesized per engine, not stored (`parametergroup.go:24`, `:456-482`) | none | `…:pg:{name}` | customer | from service | n/a | +| engine | engine name + version | n/a (static) | none | global | global | n/a | +| agent identity | EC2 instance ID as `RoleSessionName` of the system instance role (`gateway/rds/agent.go:36-69`) | `VMGeneration` in index | n/a | system account principal mapped to customer account via index | n/a | n/a | + +## 5. Lifecycle paths per resource + +### instance + +- Create: validate (`validate.go:69`) → placement reads subnet group / default VPC (`network.go:36`) → resolve parameter group (`create.go:47`) → ensure IAM instance profile in system account (`create.go:51`) → reserve record at `creating` with `createJSONRevision` (`create.go:62`) → stage encrypted bootstrap payload (`create.go:83`) → `LaunchDBInstanceVM` inline in the request (`create.go:87`) → `recordLaunch` CAS (`create.go:115`, `:195-244`) → write instance index (`create.go:124`) → event, return `creating`. +- Create failure: deferred rollback deletes payload and CAS-deletes the record if `DbiResourceID` unchanged (`create.go:246-295`); post-launch failures run `Unwind` (`create.go:297-305`). +- Readiness: reconciler `reconcileCreating` moves `creating → available` on first healthy agent beat (`reconciler.go:448-464`); `failed` after `BootstrapTimeout` (default 20 min, `reconciler.go:30`). +- Steady state: `reconcileHealth` (`recovery.go:76-101`) records/clears failure; `StatusRecovering` exists in the table (`status.go:22`, `:34`, `:43`, `:47`) but **no code transitions into it** (only `status.go` references it), so there is no automatic repair. +- Reboot/Start: `beginTransition` CAS (`lifecycle.go:65`, `:143`) then reconciler `reconcileRestarting` (`reconciler.go:479`). +- Stop: CAS to `stopping`, inline engine stop + VM stop, inline `completeTransition` to `stopped` before responding (`lifecycle.go:103-133`); reconciler resumes a dead stop (`reconciler.go:584-605`). +- Modify: immediate non-disruptive in-request; disruptive writes `PendingModifiedValues` (`modify.go:153`), then either waits for the maintenance window (`backup.go:340`) or runs under the modify lease (`modify.go:176`); reconciler resumes under the same lease (`reconciler.go:510-583`); class/storage change replaces the VM (`pending.go:80`, `replace.go:43-166`). +- Replace: launch new VM (`replace.go:113`), rewrite index (`:125`), CAS record with `VMGeneration++` (`:137`), rollback on error (`:218`). +- Delete: validate, deletion protection, reserve final snapshot (`delete.go:68`), CAS to `deleting` (`delete.go:73-91`), inline teardown (`delete.go:178-227`): stop engine, terminate VM, final snapshot, purge automated backups, release or retain data volume, delete ENIs (best-effort), delete index, delete payload, delete record last; reconciler replays teardown (`reconciler.go:611-628`) and marks `failed` after `transitionTimeout` 10 min. +- Fencing: KV revision CAS on the record everywhere; per-instance `ModifyLease` on the record (`modify_lease.go:24-173`); cluster `kvlease` for the reconciler (`reconciler.go:104-118`); `VMGeneration` binds bootstrap payloads and acknowledgements (`agent.go:345`, `:470`, `:552-553`) and the index. +- No desired generation exists on `DBInstanceRecord` (`records.go`); `VMGeneration` is a realization generation, not desired-intent generation. +- Idempotency: AWS `CreateDBInstance` has no client token, and a repeat returns `DBInstanceAlreadyExists` (`create.go:64-66`); delete retry re-runs teardown and must repeat the snapshot choice (`delete.go:57-63`); register is idempotent (`agent.go:159-160`). + +### snapshot + +- Create: check name free (`snapshot.go:92`) → CAS instance to `backing-up` with `SnapshotOperation` (`:672-705`) → create record `creating` (`:109-115`) → quiesce + EC2 snapshot inline (`:144-188`) → `putJSON` record `available` (`:126-130`) → restore instance status (`:706-723`). +- Readiness: synchronous; the API returns `available`. +- Repair: reconciler resolves records stuck `creating` past 15 min against EC2 (`reconciler.go:674-751`, `findEC2SnapshotFor` `:752`); returns instances stuck in `backing-up` (`reconciler.go:630-652`). +- Delete: `removeDBSnapshot` deletes EC2 snapshot then record and releases a retained volume when last holder (`snapshot.go:330-470`). +- Automated: window-driven `takeAutomatedBackup` (`backup.go:270`), stamped on instance record so a leader change cannot double-fire (`reconciler.go:654-660` comment); retention sweep `BackupRetentionReaper` via the daemon GC backstop (`backup_sweep.go:45-112`, `daemon/daemon.go:2310`). +- Fencing: instance-record CAS into `backing-up`; `createJSON` on snapshot key; no snapshot generation. + +### subnetgroup + +- Create: validate, resolve subnets through EC2 `DescribeSubnets` (`subnetgroup.go:176-239`), `createJSON` (`:78`); synchronous; status always `Complete` (`:31`, `:287`). +- Describe: KV read and list (`:94-133`). +- Delete: read, scan all instance records for `DBSubnetGroupName == name` (`:154-163`), plain `kv.Delete` (`:165`). +- No modify action is served (`ModifyDBSubnetGroup` is neither registered nor unsupported, so it returns `InvalidAction`). +- No reconciliation; no generation; tags via `tags.go` CAS. + +### parametergroup + +- Create: validate family against engine registry, `createJSON` meta (`parametergroup.go:29-84`); synchronous. +- Modify: validate whole batch, then `putJSON` one key per parameter (`:186-196`), then synchronously push to every attached instance via agent `apply-params` (`:198`, `:210-236`, `pending.go:149`). +- Delete: refuse `default.*`, scan instances for current or pending use (`:342-352`), delete params then meta (`:354-367`). +- `default.*` groups are synthesized from the engine catalogue (`:456-482`). +- No generation; no reconciliation of a failed propagation beyond `ParameterApplyFailed` on the instance (`pending.go:214`). + +### engine + +- Static, process-local catalogue validated at daemon boot (`engine.go:69`, `daemon/daemon.go:2055`); no lifecycle; ADR-0003 S1 excludes pure read-only catalogues from the lifecycle contract. + +## 6. NATS subjects and controller-to-agent messages + +Gateway → daemon (request/reply, queue group `spinifex-workers`): `rds.` for the 24 customer actions listed in `subjects.go:21-63`, plus `rds.GetDBBootstrapConfig` and `rds.AcknowledgeDBBootstrap` (`subjects.go:17-19`). +Budgets: default 30 s, create 5 min, lifecycle 4 min, delete 6 min, modify 6 min, snapshot 6 min, snapshot delete 4 min, restore 8 min (`service_nats.go:12-36`). +A gateway timeout does not cancel the daemon's inline orchestration (uncertain; inferred from request/reply shape), so the client may see an error while the create continues. + +Agent path (agent speaks SigV4 Query to awsgw over the mgmt bridge, `daemon/rds_deps.go:71-73`): + +- `RegisterDBInstance` → gateway → `rds.bus.{acct}.{db}.register` → daemon `SubjectRegisterWildcard` (`subjects.go:12-13`, `:65-67`). +- `SubmitDBStateChange` (heartbeat folded into state, 30 s interval, stale after 90 s; `records.go` constants) → `rds.bus.{acct}.{db}.health`. +- `GetDBBootstrapConfig` / `AcknowledgeDBBootstrap` → Layer-1 subjects; replay is allowed until acknowledged for the bound `VMGeneration` (`gateway/rds/agent.go:150-180`). +- `PollDBCommands` long poll (1-20 s) queue-subscribes `rds.bus.{acct}.{db}.command` with group `spinifex-rds-agents` and publishes carried replies to `rds.bus.{acct}.{db}.command-reply` before subscribing (`gateway/rds/agent.go:198-287`). +- Command types: `set-password`, `apply-params`, `stop-engine`, `grow-filesystem`, `quiesce`, `unquiesce` (`commands.go:12-23`); params `MasterUsername`, `MasterUserPassword`, `QuiesceLabel`, `QuiesceDeadlineSeconds` (`commands.go:27-35`). +- Issuer: `issueCommand` subscribes the reply subject, then republishes the same `CommandID` every 2 s until reply or budget (`commands.go:144-199`, `:72`); core NATS, not durable, by design for the password (`subjects.go:77-80`). +- Agent retry: results are retained and re-carried until a poll succeeds (`agents/rds/agent/command.go:100-133`). +- Agent dedupe: the agent executes every delivered command and has **no CommandID dedupe** (`agents/rds/agent/command.go:128-131`, `:140-157`); a republish delivered after execution but before the issuer sees the reply can execute twice (uncertain; depends on timing between reply relay and the 2 s republish). +- Commands carry no `VMGeneration` (`agent.go:134-139`), so the fence against a superseded VM is the bus subject plus the gateway's index lookup rather than a generation in the message. +- `LookupEngine`, `Engine`, `ParamType*`, `ApplyType*`, `Parameter`, `EngineHealth*`, `HeartbeatInterval`, `BootstrapMode*`, `ParameterRollbackMessage`, `SupportedInstanceClasses`, `SmallestInstanceClass` are imported by the agent from `handlers/rds` (full list from import scan); this is the S5 "accidental import of a domain monolith". + +RDS → EC2 subjects used as capabilities: `ec2v1.InstanceCommandSubject(id)` for stop/reboot/attach (`lifecycle.go:434`, `launch.go:583`), `ec2.start` (`lifecycle.go:451`), `ec2.DescribeStoppedInstances` (`launch.go:616`), plus EC2 `DescribeInstances` fan-out (`daemon/rds_deps.go:90-97`) and `DescribeInstanceTypes` fan-out (`gateway/rds/enginecatalog.go:147`). + +Recovery/redelivery: there is no durable work queue; recovery relies on the reconciler re-reading record status (`reconciler.go:238-282`) under `kvlease`, a 5 min resync and KV watch wakeups (`reconciler.go:120-145`). + +## 7. Boundary-crossing imports + +### Outbound from RDS (handlers/rds, gateway/rds, gateway/rds.go, daemon/rds_deps.go) + +- **Quota**: none from `handlers/rds`; the gateway injects `gw.Quota.EnforceRDSInstances` as `Env.QuotaCheck` (`gateway/rds.go:59-62`). +- **ExpectedNodes / EC2 instance availability**: `gateway/rds/enginecatalog.go` → `domains/ec2/awsapi/instance.DescribeInstanceTypes` with `env.ExpectedNodes` (`:147`); `daemon/rds_deps.go` → `domains/ec2/awsapi/instance.DescribeInstances` with node count (`:96`). +- **EC2 instance**: `domains/ec2/systeminstance` (`launch.go:99-100`, `delete.go:236`), `domains/ec2/instance` (`lifecycle.go`, `StartStoppedInstanceOutput`), `contracts/ec2/v1` (`launch.go`, `lifecycle.go`), `runtime/compute/vm` (`agent.go`, `create.go` `vm.VolumeSerial`, `backup_sweep.go` `vm.Reaper`), `foundation/aws/ami`, `domains/ec2/instancetypes` (`sizing.go:30`, `paramcatalog.go:219`). +- **VPC/network**: `domains/network/systemvpc` (`vpc.go`, `launch.go`, `daemon/rds_deps.go`), EC2 SDK VPC/subnet/SG/ENI calls through narrow interfaces `networkResolver` (`network.go:16`) and `launchVPCProvisioner` (`launch.go:76-96`); `foundation/aws/tags` (`ManagedByRDS`). +- **Storage/EBS**: EC2 SDK `CreateVolume`/`DeleteVolume` (`launch.go:107-111`), `DescribeVolumes`/`ModifyVolume` (`storage.go:23-26`), EC2 snapshot provider (`snapshot.go:25`), volume attach over EC2 NATS (`launch.go:560-629`). +- **IAM**: `handlers/iam` for `SystemInstanceRoleEnsurer`/`EnsureSystemInstanceProfile` (`iam.go:56-77`) and `EncryptSecret`/`DecryptSecret` for bootstrap payloads (`bootstrap.go:105`, `:199`). +- **DNS**: `domains/dns` (`dns.go:17`, `dns.RDSName`, `dns.Change`). +- **Bluebottle**: `bluebottle/pkg/auth.ParseRoleARN` (`gateway/rds.go:87`), `bluebottle/pkg/masterkey.ReadShared` (`daemon/rds_deps.go:52`); `handlers/rds` imports no Bluebottle package. +- **Ingress/foundation**: `ingress/aws/query`, `foundation/aws/xml`, `foundation/aws/errors`, `foundation/aws/identifiers`, `foundation/state/{kvstore,kvlease,kvutil,migrate}`, `foundation/lifecycle/reconciler`, `foundation/messaging/nats`, `foundation/pki`, `bootstrap/config`. + +### Inbound into RDS (who imports `handlers/rds` or `gateway/rds`) + +- `handlers/quota/live.go:14,89` → `NewNATSService(...).DescribeDBInstances` (quota reads RDS through the AWS-shaped API). +- `accountteardown/reapers_rds.go:9` → `NATSService` describe/modify/delete for instances, subnet groups, parameter groups. +- `daemon/daemon.go:78` → `Service`, `Reconciler`, all `Subject*`, `ValidateEngineRegistry`, `NewService`, `NewReconciler`, `NewBackupRetentionReaper`. +- `daemon/rds_deps.go` → `Deps`, `LaunchDeps`, `BackupPolicy`, `NewNATSVolumeAttacher`, `NewDescribeInstanceState`, `NewNATSInstanceCommander`. +- `daemon/dns_reconcile.go:16,75` → `AccountWatchBuckets` (raw bucket handles). +- `daemon/bedrock_deps.go:10,43` → `NewNATSVolumeAttacher` (Bedrock/Ochre depends on RDS). +- `gateway/operations.go:8,44-45` → `gateway_rds.ActionNames`, `UnsupportedActionNames`. +- `gateway/rds/*` → `KVBucketRDSSystem`, `InstanceIndexKey`, `InstanceIndexEntry`, `InstanceRoleName`, `InternalAgentActions`, `ResourceKind*`, `FormatARN`, `ParseARN`, catalogue functions, `NATSService`, wire types. +- `agents/rds/agent/*` → wire types and engine catalogue (section 6). +- `tests/integration/service_daemon_lite.go:51`, `tests/e2e/harness/rds.go` (test support). + +## 8. Known gaps against Q-78 / ADR-0003 (observed in code, not live-verified) + +1. No desired generation on any RDS record (`records.go`); mutations advance only the KV revision, and `PendingModifiedValues` is overwritten by a later modify (`modify.go:153`), so Q-78 "atomically commits a monotonically increasing desired generation" is not met. +2. Desired and observed state share one record: `Status`, `Agent`, `UnhealthySince`, `FailureReason` sit beside requested class/storage on `DBInstanceRecord` (`records.go`). +3. Create performs all external realization inline in the request (`create.go:87`); if the daemon dies after `LaunchDBInstanceVM` returns and before `recordLaunch` (`create.go:115`), the record stays `creating` with no `InstanceID`, times out to `failed` (`reconciler.go:458-462`), and a later delete has no IDs to clean (`delete.go:229-233`, `:537-547`); the VM, ENIs and volume are tagged `ManagedBy=rds` (`launch.go:260`, `:314`, `:400`) but no RDS orphan reaper for them was found. +4. The same interruption window exists for restore (`restore.go:111` → `:136`) and replace (`replace.go:113` → `:137`); replace's rollback runs only on error return, not on process death. +5. Delete removes the only ownership record (`delete.go:217`) after ENI deletion that is best-effort and only logs failures (`delete.go:203`, `launch.go:510-520`), contrary to Q-78 "a handler never removes the only reconstructible ownership of required cleanup". +6. `deleting` that exceeds 10 min becomes `failed` (`reconciler.go:621-624`); a `failed` instance is then handled by `reconcileHealth` rather than continuing teardown, so incomplete cleanup is not shown as a distinct state (uncertain; inferred from the `switch` at `reconciler.go:256-272`). +7. `StatusRecovering` is declared and reachable in the transition table but never entered, so there is no repair path (`status.go:22`; `recovery.go:19-21` says v1.0 reports rather than repairs). +8. Synchronous completion where AWS is asynchronous: AWS `CreateDBSnapshot` returns `Status=creating`, here it returns `available` after the snapshot is taken inline (`snapshot.go:126-130`); AWS `StopDBInstance` returns `stopping`, here the handler completes to `stopped` inline (`lifecycle.go:126`); AWS `DeleteDBInstance` leaves the instance describable as `deleting`, here the record is deleted before the response when teardown succeeds inline (`delete.go:97-104`, `:217`). +9. DB subnet group and parameter group deletes are check-then-delete across keys with no fence against a concurrent `CreateDBInstance` that has already read the group (`subnetgroup.go:151-165`, `parametergroup.go:339-367`; `network.go:43`, `create.go:47`), so an instance can reference a deleted group. +10. `ModifyDBParameterGroup` writes one key per parameter with no batch atomicity (`parametergroup.go:186-196`); a KV failure mid-loop leaves a half-applied batch despite the comment at `:140-143`, and propagation to instances is synchronous best-effort (`:210-236`). +11. Quota admission is a non-atomic count before create (`gateway/rds/instance.go:13-19`, `handlers/quota/live.go:85-93`), so concurrent creates can exceed the cap; it counts `len(DescribeDBInstances)` which pages at 100 by default (`paging.go:14`, `describe.go:71`), so an account above 100 instances is under-counted (uncertain whether quota limits ever exceed 100). +12. Agent commands are republished every 2 s with no agent-side `CommandID` dedupe and no generation (section 6), so internal redelivery is not proven idempotent for `grow-filesystem`, `apply-params` or `set-password`. +13. Heartbeat liveness lives partly in per-node process memory (`agent.go:284-321`) and the leader judges health from its own map plus a KV floor (`recovery.go:191-209`), which Q-78 discourages for recovery decisions (observation-side, lower severity). +14. The RDS reconciler lease doubles as the cluster-wide GC election for unrelated reapers (`daemon/daemon.go:2374-2382`); without RDS service init every cluster-wide reaper is skipped. +15. No lifecycle contract record exists for any RDS resource (ADR-0003 S1); nothing in `handlers/rds` states readiness predicates or deletion obligations as a contract. + +## 9. First-resource candidates + +| Rank | Resource | Size (prod lines) | Private state | Dependencies | Tests | Extractable without quota / ExpectedNodes / IAM / Bluebottle? | +|---|---|---|---|---|---|---| +| 1 | **subnetgroup** | ~340 (`subnetgroup.go`) + its slice of `store.go`, `records.go`, `tags.go`, `arn.go`; gateway 23 | `db-subnet-groups/{name}` only | EC2 `DescribeSubnets` through a narrow interface already (`network.go:16-20`, `subnetgroup.go:189`); reverse dependency: needs an instance-owned query "instances referencing subnet group X" to replace `instancesUsingGroup` (`subnetgroup.go:154`); forward dependency: instance placement reads the group record directly (`network.go:43`), which must become a subnetgroup query | `handlers/rds/subnetgroup_test.go` (338), `tags_test.go`, `gateway/rds/handler_test.go`/`authz_test.go`, e2e `tests/e2e/rds/groups_test.go` (`TestSubnetAndParameterGroups`), `isolation_test.go`, `api_matrix_test.go` (uncertain coverage depth for the last two) | **Yes.** No quota, ExpectedNodes, IAM, Bluebottle, agent or EC2-launch dependency. Synchronous AWS semantics (status `Complete`) match the code. Gap 9 (delete race) should be recorded, not fixed, during the move. | +| 2 | **parametergroup** | ~600 (`parametergroup.go`) + slice of store/records/tags; gateway 31 | `db-parameter-groups/{name}/meta`, `…/params/{key}` | Engine catalogue (`engineForFamily`, `LookupParameter`, `ResolveEffectiveParameters`) — owned by **engine**; `SmallestInstanceClass` (EC2 `instancetypes`); cluster CA availability (`checkTLSEnforceable`, `parametergroup.go:581`); instance records for in-use check; **instance orchestration** in Modify (`propagateParameterGroup` → `applyParameterGroup` → agent `apply-params`) | `parametergroup_test.go` (806), `paramcatalog_test.go`, `enginefamily_test.go`, `tlsenforcement_test.go` (uncertain scope), `pending_test.go` (propagation), e2e `groups_test.go` | **Mostly.** No quota/ExpectedNodes/IAM/Bluebottle, but Create/Describe/Delete are synchronous while Modify crosses into instance realization and the agent wire contract; extraction requires engine first (or a narrow engine capability) and moving propagation to explicit cross-resource orchestration. | +| 3 | **engine** | ~1,960 (`engine.go`, `engine_postgres.go`, `engine_mariadb.go`, `paramcatalog.go`, `catalog.go`, `sizing.go`) | none (static catalogue) | `domains/ec2/instancetypes`; consumed by the agent, parametergroup, instance | `engine_test.go`, `engine_mariadb_test.go`, `enginefamily_test.go`, `paramcatalog_test.go`, `catalog_test.go`, `sizing_test.go`, `gateway/rds/enginecatalog_test.go` | **Package: yes**; it is the natural S5 agent capability. **awsapi action `DescribeOrderableDBInstanceOptions`: no**, blocked on the EC2 instance-type-availability projection (`gateway/rds/enginecatalog.go:146-174`). Not a durable resource under ADR-0003 S1, so it is weak evidence for ADR-0004 rollout item 2. | +| 4 | **snapshot** | ~1,700 (`snapshot.go`, `backup_sweep.go`, share of `backup.go`, `restore.go`, `reconciler.go:674-796`) | `db-snapshots/`, `backups/…/automated/`, `retained-volumes/` | Instance record CAS into `backing-up`, engine quiesce over agent commands, EC2 snapshot/volume services, instance delete (final snapshot, retained volumes), restore runs the instance create path | `snapshot_test.go` (879), `restore_test.go`, `backup_test.go` (1013), `backup_sweep_test.go`, `delete_test.go`; e2e `snapshot_restore_test.go`, `automated_backup_test.go` | No quota/ExpectedNodes/IAM/Bluebottle on snapshot itself, but restore inherits instance create (IAM, launch); snapshot creation needs instance-owned quiesce/hold capabilities first. Not a first candidate. | +| 5 | **instance** | ~8,000+ (create, describe, lifecycle, launch, network, vpc, storage, modify, modify_lease, pending, replace, recovery, reconciler, window, backup, delete, agent, commands, bootstrap, servingcert, userdata, iam, dns, validate, status) | `db-instances/`, `bootstrap-payloads/`, `rds-system` `instance-index/`, `spinifex-rds-leader` | Quota admission, ExpectedNodes-style EC2 fan-out (`daemon/rds_deps.go:90-97`), IAM ensurer and secret crypto, Bluebottle master key (via composition), EC2 system instance / ENI / volume / snapshot / attach, system VPC, DNS, agent wire contract | `create_test.go`, `launch_test.go`, `lifecycle_test.go`, `delete_test.go`, `modify_test.go`, `modify_lease_test.go`, `pending_test.go`, `replace_test.go`, `reconciler_test.go`, `recovery_test.go`, `revisit_test.go`, `agent_test.go`, `bootstrap_test.go`, `commands_test.go`, `storage_test.go`, `network_test.go`, `vpc_test.go`, `describe_test.go`, `status_test.go`, `iam_test.go`, `userdata_test.go`, `watch_source_test.go`, `gateway/rds/agent_test.go`, `authz_test.go`; e2e `create_describe`, `lifecycle`, `modify`, `delete`, `failure`, `graceful_reboot`, `connect`, `cross_subnet`, `isolation`, `authz`, `mariadb*` | **No.** Every listed blocker applies. Extract last. | + +Cross-resource files to settle alongside the first extraction: `tags.go` (per-ARN dispatch), `events.go` (journal), `arn.go`/`filters.go`/`paging.go` (awsapi), `store.go`/`records.go`/`service.go` (split per owner). + +Recommendation: extract **subnetgroup** first, with two narrow capabilities defined by the consumer: an instance-owned "references to subnet group" query for delete, and a subnetgroup-owned "resolve placement subnets" query for instance create. +Then do **engine** (package and agent capability, leaving the orderable action on the legacy path until the EC2 projection exists), then **parametergroup** with propagation made explicit orchestration. +Test coverage for subnetgroup is unit plus one e2e test; ADR-0003 S5 failure/recovery evidence for its delete race does not exist yet.