mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
docs: record ECS, EKS and RDS source inventory and first-resource ranking
This commit is contained in:
joshsiv-mulga
@@ -385,6 +385,34 @@ These residuals were recorded by individual slices and must be resolved once the
|
||||
- Instance adapter consumers outside EC2: `gateway/rds/enginecatalog.go`, `daemon/eks_cp_control.go`, `daemon/rds_deps.go` and `handlers/quota/vcpu.go` call `domains/ec2/awsapi/instance` directly; each needs an EC2 capability rather than the AWS action adapter.
|
||||
- Merge-time follow-up in the umbrella repository: the `Makefile` `invariants-check` target, the `docs-pr.yml` clause cross-check, `.claude/hooks/architecture-traps.sh` and a `check-invariants.sh` comment hard-code `spinifex/spinifex/network/invariants`. They must change with the pointer bump that brings in this branch, or the docs-pr cross-check silently skips. Live specs and proposals citing `spinifex/network/` will report unresolved citations until updated.
|
||||
|
||||
## ADR-0004 source inventory and first-resource ranking
|
||||
|
||||
The per-file inventories for ECS, EKS and RDS are in `docs/package-boundary/adr-0004-inventory-{ecs,eks,rds}.md`.
|
||||
Each classifies every production file as resource owner, cross-resource orchestration, AWS protocol adapter, guest/controller wire contract, or temporary residue, and maps actions, durable records, identity and scope, lifecycle paths, NATS subjects and boundary-crossing imports.
|
||||
They were produced by reading the code at `6c50349bd` and are not verified against a live system.
|
||||
|
||||
Observed gaps common to all three: no resource carries a desired generation, all resources of a service share one per-account KV bucket, and deletes remove records before dependent cleanup is confirmed.
|
||||
|
||||
First-resource ranking:
|
||||
|
||||
1. RDS DB subnet group: about 340 lines with its own `db-subnet-groups/{name}` keys, synchronous AWS semantics, and no quota, expected-node, IAM, Bluebottle or agent dependency.
|
||||
It has a real deletion graph, because delete refuses while an instance references the group, so it exercises a consumer-owned capability in both directions: an instance-owned "instances referencing this group" query for delete, and a subnet-group-owned placement query for instance create.
|
||||
2. ECS task definition: own keys and good tests, but `RegisterTaskDefinition` allocates revisions non-atomically and the gateway adapter carries the `iam:PassRole` checks, so it sits closer to a blocked boundary.
|
||||
3. EKS access entry: own keys, but entries are read during token review, which places it next to EKS identity.
|
||||
|
||||
Blocked dependencies recorded, not designed around:
|
||||
|
||||
- RDS instance: quota admission, the duplicated expected-node fan-out (`gateway/rds/enginecatalog.go`, `daemon/rds_deps.go`), IAM ensurer and Bluebottle master key.
|
||||
- RDS `DescribeOrderableDBInstanceOptions`: the EC2 instance-type-availability projection.
|
||||
- ECS task, service and capacity: `iam:PassRole` in the gateway adapter, IAM `ecsInstanceRole`, and EC2/ELBv2/EIP adapters called as internal APIs.
|
||||
- EKS nodegroup and cluster: IAM role and instance-profile creation, Bluebottle `auth`, the NATS CA, and EKS identity (token review, OIDC).
|
||||
- Guest agents: `agents/ecs`, `agents/rds` and `agents/eks/tokenwebhook` import the service implementation packages, which ADR-0004 S5 forbids; they need wire contracts.
|
||||
|
||||
Findings to resolve before extraction:
|
||||
|
||||
- The branch is 59 commits behind `dev`, including `ModifyDBSubnetGroup` (#1152) and the EKS stub unregistration (#1172), so the subnet-group inventory must be redone after `dev` is merged in.
|
||||
- `ClusterMeta.ControlPlaneTemplate` (`handlers/eks/service_impl.go`) clears credentials but keeps `OIDCPrivateKeyPEM` and the k3s `JoinToken`, so both persist in the cluster KV record, which the gateway also reads; whether that record is encrypted at rest is unconfirmed.
|
||||
|
||||
## Recording rule
|
||||
|
||||
For every later slice, add the old and new path, source commit, focused
|
||||
|
||||
@@ -0,0 +1,388 @@
|
||||
# ADR-0004 source inventory — ECS
|
||||
|
||||
Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`.
|
||||
It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it.
|
||||
Items marked "verify" or "uncertain" are unconfirmed.
|
||||
Gaps record observed behaviour, not target behaviour.
|
||||
|
||||
|
||||
Repo: `spinifex` submodule, branch `refactor/spinifex-package-boundaries`; paths below are relative to `spinifex/spinifex/` unless stated otherwise.
|
||||
Binding inputs: ADR-0001..0004 and Q-78 at `origin/main`.
|
||||
This inventory reports what the code does; it accepts or closes no gap.
|
||||
"Verify" marks claims about AWS behaviour, or runtime behaviour, that a test or live system has not confirmed here.
|
||||
|
||||
## 1. File table
|
||||
|
||||
Totals: `handlers/ecs` is 7,760 production lines over 28 files, `gateway/ecs` + `gateway/ecs.go` is 1,236 lines, and `daemon/ecs_deps.go` + `accountteardown/reapers_ecs.go` is 132 lines.
|
||||
|
||||
| File | Lines | Class | Reason / split |
|
||||
|---|---|---|---|
|
||||
| `handlers/ecs/bus/messages.go` | 173 | guest/controller wire contract (part temporary residue) | `Assign`, `AssignContainer`, `StopDirective`, `TaskState`, `ContainerStatus`, `InstanceCapacity`, `PortMapping` and `SystemControl` are live payloads of the agent and gateway path. `RegisterInstance` (:19) and `Heartbeat` (:33) have no production publisher, so they are residue. Split: live types go to a versioned `contracts/ecs/v1`; the residue types are deleted once the scheduler bus subscriptions are removed. |
|
||||
| `handlers/ecs/bus/subjects.go` | 45 | temporary residue (prerequisite: drop scheduler bus subs; removal: delete unused builders) | `RegisterSubject`, `HeartbeatSubject`, `AssignSubject` and `TaskStateSubject` (:19-38) have no production publisher. `ServiceReconcileSubject` (:43) is live and is an internal cross-node wake owned by the service reconcile. |
|
||||
| `handlers/ecs/capacity.go` | 223 | cross-resource orchestration (container-instance capacity provisioning) | `ProvisionCapacity` (:70) is a non-AWS action. It composes an IAM system role, EC2 AMI lookup and EC2 `RunInstances`, and records nothing durable in ECS. |
|
||||
| `handlers/ecs/capacity_providers.go` | 233 | resource owner (`capacityprovider`), mixed | Create, Describe and Delete belong to `capacityprovider`. `PutClusterCapacityProviders` (:16-40) mutates the cluster record and belongs to `cluster`. |
|
||||
| `handlers/ecs/cluster_teardown.go` | 171 | mixed: cross-resource orchestration + resource owner (`containerinstance`) | `DeleteCluster` (:24-53) is a cascade over every child, which is orchestration. `DeregisterContainerInstance` (:60), `UpdateContainerInstancesState` (:103), `instanceActiveTasks` (:143) and `drainInstanceServiceTasks` (:160) are container-instance commands that also stop tasks. Split three ways: cluster delete orchestration, container-instance owner, and task-stop calls through a task capability. |
|
||||
| `handlers/ecs/conv.go` | 291 | mixed | `ClusterShortName`, `ContainerInstanceShortID`, `TaskShortID` and `CapacityProviderShortName` (:27-60) are per-resource identity parsers that the gateway authz also uses. The `containerDefsFromAWS`, `runtimePlatformFromAWS` and `ContainerDef.toAWS` group (:87-265) is the `taskdefinition` projection. `toAssignContainer` (:268) is wire-contract projection. |
|
||||
| `handlers/ecs/deployments.go` | 267 | resource owner (`service`) | This is pure deployment, rollout and circuit-breaker logic on `ServiceRecord`, with no KV I/O. |
|
||||
| `handlers/ecs/eni.go` | 155 | cross-resource orchestration (task realization, network capability) | The consumer-owned `eniController` interface (:37) belongs with `task`. `natsENIController` (:46-128) binds it to EC2 subjects and belongs at the composition root. |
|
||||
| `handlers/ecs/gpu_report.go` | 63 | resource owner (`task`), mixed with wire contract | `ReportTaskGPU` (:31) writes the task record. `ReportTaskGPUInput`, `ContainerGPUReport` and `ReportTaskGPUOutput` (:10-25) are agent wire shapes. |
|
||||
| `handlers/ecs/instancerole.go` | 48 | cross-resource orchestration (identity for container instances) | It ensures and converges the `ecsInstanceRole` through `handlers_iam` (:37-48). |
|
||||
| `handlers/ecs/instances.go` | 532 | mixed: resource owner (`containerinstance`) + `task` + orchestration | Register, Describe, List, `upsertInstance`, `recordRegister` and `recordHeartbeat` (:19-357) belong to `containerinstance`. `recordTaskState` (:361) and `SubmitTaskStateChange` (:445) belong to `task`. `releaseReservation` (:505) is placement against the instance record. `recordDeploymentFailure` (:481) belongs to `service`. |
|
||||
| `handlers/ecs/netcfg.go` | 78 | resource owner (`task`) | It resolves the network mode and parses awsvpc input for task launch. |
|
||||
| `handlers/ecs/placement.go` | 144 | cross-resource orchestration (placement) | It contains pure fit and strategy selection over container-instance capacity, with no I/O. |
|
||||
| `handlers/ecs/poll.go` | 107 | guest/controller wire contract | It holds the agent inbox drain and ack (`PollAssignments` :37), and the task-owned inbox reclaim (:93-107). |
|
||||
| `handlers/ecs/records.go` | 525 | mixed (all resources' records) | It holds the status constants, ARN builders and every record type for every resource. Split each record, its ARN builder and its statuses into the owning resource. |
|
||||
| `handlers/ecs/scheduler.go` | 449 | cross-resource orchestration + temporary residue | Lease, service reconcile loop, instance reaper and role converge are orchestration. `onRegister`, `onHeartbeat` and `onTaskState` (:236-320) are residue because nothing publishes those subjects. |
|
||||
| `handlers/ecs/service.go` | 706 | mixed: facade + `cluster` + `taskdefinition` + shared KV helpers | `ECSService` (:27-81) is an AWS-shaped facade that also serves as an internal API. `Deps` (:98) is composition. Cluster CRUD (:231-411) belongs to `cluster`. Task-definition CRUD and `ParseTaskDefRef` (:416-706) belong to `taskdefinition`. `getJSON`, `putJSON` and `keysWithPrefix` (:164-225) are shared persistence helpers. |
|
||||
| `handlers/ecs/service_nats.go` | 177 | AWS protocol adapter (gateway→daemon NATS hop) | `NATSECSService` forwards 34 methods onto `ecs.<Action>`. It is also used as an internal API by `accountteardown` and by gateway PassRole resolution. |
|
||||
| `handlers/ecs/services.go` | 786 | mixed: resource owner (`service`) + orchestration bindings | Service CRUD and reconcile (:128-665) belong to `service`. `targetRegistrar` / `natsTargetRegistrar` (:28-59, ELBv2) and `eipManager` / `natsEIPManager` (:64-121, EC2 EIP) are consumer-owned capabilities, but their NATS bindings belong at the composition root. `listTaskRecords` (:605) reads task records directly. |
|
||||
| `handlers/ecs/store.go` | 227 | temporary residue (shared account bucket; prerequisite: decide per-owner key namespaces; removal: key helpers move to owners) | One bucket holds the keys of all six resources (:36-150). It also holds the leader bucket factory (:221) and the account enumeration used by every sweep (:186). |
|
||||
| `handlers/ecs/tags.go` | 277 | cross-resource orchestration (tag dispatch) | It parses the ARN and then reads and writes each resource's record directly (:76-203). It should become one tag command per owner behind ARN dispatch. |
|
||||
| `handlers/ecs/task_stop.go` | 301 | mixed: resource owner (`task`) + orchestration | `StopTask`, `StartTask`, `reserveOnInstance`, `requestStopTask` and `forceStopTask` (:28-209) belong to `task`, with placement mixed into `reserveOnInstance`. `applyServiceTargets` (:235) and `assignTaskPublicIP` / `releaseTaskPublicIP` (:264-301) are ELBv2 and EIP orchestration. |
|
||||
| `handlers/ecs/tasks.go` | 438 | mixed: resource owner (`task`) + placement orchestration + wire | `RunTask`, Describe, List and the record/projection code belong to `task`. `reservePlacement` (:180) does a CAS on the container-instance record. `publishAssign` (:249) writes the agent inbox. |
|
||||
| `handlers/ecs/tasks_gc.go` | 130 | resource owner (`task`) reconciliation | It retains STOPPED tasks, retries owed ENI releases and prunes old tasks. It is hosted on `Scheduler` (:21). |
|
||||
| `handlers/ecs/userdata.go` | 58 | guest/controller wire contract | It writes the cloud-config that seeds the agent env (`ECS_GATEWAY_URL`, `ECS_GATEWAY_CA`, `ECS_REGION`, `ECS_CLUSTER`) (:31-58). |
|
||||
| `gateway/ecs.go` | 80 | AWS protocol adapter | `ECS_Request` (:26) does target parsing, the scope gate (:54-60), the PassRole closure (:65-67) and response encoding. |
|
||||
| `gateway/ecs/handler.go` | 165 | AWS protocol adapter | It holds the authoritative action table (:50-112), `StubbedActionNames` (:115), the raw-JSON action list (:131) and the JSON 1.1 writers. |
|
||||
| `gateway/ecs/actions.go` | 452 | AWS protocol adapter (mixed: PassRole resolution) | Each action decodes and forwards over NATS. `checkTaskLaunchRoles` (:44) queries `DescribeTaskDefinition` over the ECS NATS facade to find role ARNs. |
|
||||
| `gateway/ecs/authz.go` | 350 | AWS protocol adapter | The per-action resource-ARN scope table (:57-124) and resolver (:139-350) are built on the `handlers_ecs` parsers and ARN builders. |
|
||||
| `gateway/ecs/azrebalancing.go` | 103 | AWS protocol adapter | It rejects `availabilityZoneRebalancing=ENABLED` (:28) and adds the SDK-gap field to service responses (:54). |
|
||||
| `gateway/ecs/clienttoken.go` | 86 | AWS protocol adapter holding durable retry state (flag) | It holds the RunTask ClientToken store `spinifex-ecs-clienttokens` (:22). ADR-0003 S3 says an adapter does not own retry bookkeeping, so this belongs with `task`. |
|
||||
| `daemon/ecs_deps.go` | 63 | composition root (nearest class: cross-resource orchestration) | It binds IAM, images, `RunWorkerInstance` and the gateway URL/CA into `handlers_ecs.Deps` (:17-63). |
|
||||
| `daemon/daemon.go` (ECS sections only) | n/a | AWS protocol adapter transport + composition | Import at :74, fields at :168-169, 34 `ecs.<Action>` subscriptions at :1131-1167, construction at :2023-2043. |
|
||||
| `accountteardown/reapers_ecs.go` | 69 | cross-resource orchestration (account teardown) | It has one cluster reaper that calls the ECS AWS facade over NATS (:20-58). |
|
||||
| `gateway/operations.go` (:7, :30-32) | n/a | AWS protocol adapter | It registers the ECS operation inventory from `gateway_ecs.Actions` and `StubbedActionNames`. |
|
||||
| `gateway/gateway.go` (:301, :403, :523-524, :639) | n/a | AWS protocol adapter | Service routing to `ECS_Request` and the `ecs-tasks.amazonaws.com` PassRole principal constant live here. |
|
||||
|
||||
Other ECS-aware code outside scope, with no import of ECS:
|
||||
`handlers/sts/assume_role.go:47-55,350-355` duplicates the `ecsInstanceRole` name and the `ecs-tasks.amazonaws.com` principal, and documents that it cannot import `handlers_ecs`.
|
||||
`foundation/aws/tags/tags.go:24` defines `ManagedByECS` for AMI resolution.
|
||||
`operator/imagecatalog/images.go:611-637` defines the `spinifex-ecs-node` and `spinifex-ecs-node-gpu` system AMIs.
|
||||
`domains/acm/renewal.go:169` is a comment-only reference.
|
||||
`../internal/ecsgw/client.go` is the agent's SigV4 gateway client, with target prefix at :29 and :101.
|
||||
|
||||
Guest agent counterparty (`agents/ecs/agent/`), not inventoried file by file:
|
||||
It reaches the control plane only through signed gateway HTTP (`controlplane.go:18-35`), not NATS.
|
||||
It imports both `handlers/ecs/bus` and the `handlers_ecs` implementation package (`registration.go:37-38`, `controlplane.go:128,142,150`, `taskrun.go:416-419`).
|
||||
That is the ADR-0004 S5 debt: the agent imports a domain implementation package to get wire types.
|
||||
|
||||
## 2. Served AWS actions and protocol
|
||||
|
||||
Protocol: AWS JSON 1.1, with the action taken from the `X-Amz-Target` suffix (`gateway/ecs.go:15-20`, any prefix accepted).
|
||||
Requests use the canonical prefix `AmazonEC2ContainerServiceV20141113` (`gateway/ecs/handler.go:26`).
|
||||
Responses are encoded with `jsonutil` (epoch-second times), except `PollAssignments`, which uses `encoding/json` (RFC3339 times) (`handler.go:131-157`).
|
||||
Errors are `awserrors` codes rendered by the shared ErrorHandler.
|
||||
An action missing from the table returns `InvalidAction` (`gateway/ecs.go:32-36`), and a stub returns `NotImplemented` (501) (`handler.go:43-45`).
|
||||
Request order is: account check (:40-46), bounded body (:48), `ResourceARNs` scope derivation (:54), `checkPolicyResources(r, "ecs", action, …)` (:58), handler with PassRole closure (:69), then encode.
|
||||
Gateway to daemon is a NATS request on `ecs.<Action>` with a 30 s timeout (`handlers/ecs/service_nats.go:12,29-177`), served by queue group `spinifex-workers` (`daemon/daemon.go:1131-1167`).
|
||||
`gateway/operations.go:30-32` publishes Registered = all 39 table keys and Stubbed = 5.
|
||||
`ecsScopes` (`authz.go:57-124`) has an entry for every table action, and `ResourceARNs` fails closed if an entry is missing (:140-144).
|
||||
|
||||
| Action | Status | Resource | Handler (daemon) |
|
||||
|---|---|---|---|
|
||||
| CreateCluster | registered | cluster | `service.go:231` |
|
||||
| DescribeClusters | registered | cluster (+counts from instance/task/service) | `service.go:268` |
|
||||
| ListClusters | registered | cluster | `service.go:338` |
|
||||
| DeleteCluster | registered | cluster (cascade orchestration) | `cluster_teardown.go:24` |
|
||||
| UpdateCluster | stubbed | cluster | — |
|
||||
| PutClusterCapacityProviders | registered | cluster (references capacityprovider) | `capacity_providers.go:16` |
|
||||
| CreateCapacityProvider | registered | capacityprovider | `capacity_providers.go:44` |
|
||||
| DescribeCapacityProviders | registered | capacityprovider | `capacity_providers.go:80` |
|
||||
| DeleteCapacityProvider | registered | capacityprovider | `capacity_providers.go:125` |
|
||||
| RegisterTaskDefinition | registered (PassRole on task/execution role, `actions.go:99`) | taskdefinition | `service.go:416` |
|
||||
| DeregisterTaskDefinition | registered | taskdefinition | `service.go:546` |
|
||||
| DescribeTaskDefinition | registered | taskdefinition | `service.go:571` |
|
||||
| ListTaskDefinitions | registered | taskdefinition | `service.go:586` |
|
||||
| ListTaskDefinitionFamilies | stubbed | taskdefinition | — |
|
||||
| RunTask | registered (PassRole via taskdef lookup, ClientToken) | task (+placement, containerinstance) | `tasks.go:28` |
|
||||
| StartTask | registered (PassRole via taskdef lookup) | task (+containerinstance) | `task_stop.go:53` |
|
||||
| StopTask | registered | task | `task_stop.go:28` |
|
||||
| DescribeTasks | registered | task | `tasks.go:273` |
|
||||
| ListTasks | registered | task | `tasks.go:299` |
|
||||
| SubmitTaskStateChange | registered (AWS action; used by agent) | task | `instances.go:445` |
|
||||
| PollAssignments | registered (non-AWS, agent-only, raw JSON) | task inbox (wire) | `poll.go:37` |
|
||||
| ReportTaskGPU | registered (non-AWS, agent-only) | task | `gpu_report.go:31` |
|
||||
| CreateService | registered (PassRole via taskdef lookup; AZ-rebalancing gate) | service | `services.go:128` |
|
||||
| UpdateService | registered (same) | service | `services.go:209` |
|
||||
| DeleteService | registered | service | `services.go:267` |
|
||||
| DescribeServices | registered | service | `services.go:302` |
|
||||
| ListServices | registered | service | `services.go:326` |
|
||||
| ListServicesByNamespace | stubbed | service | — |
|
||||
| RegisterContainerInstance | registered (agent boot + 30 s liveness) | containerinstance | `instances.go:65` |
|
||||
| DeregisterContainerInstance | registered | containerinstance | `cluster_teardown.go:60` |
|
||||
| UpdateContainerInstancesState | registered | containerinstance | `cluster_teardown.go:103` |
|
||||
| DescribeContainerInstances | registered | containerinstance | `instances.go:127` |
|
||||
| ListContainerInstances | registered | containerinstance | `instances.go:167` |
|
||||
| ProvisionCapacity | registered (non-AWS) | containerinstance capacity (EC2 launch orchestration) | `capacity.go:70` |
|
||||
| PutAccountSetting | stubbed | account setting | — |
|
||||
| ListAccountSettings | stubbed | account setting | — |
|
||||
| TagResource | registered | cross-resource (by ARN) | `tags.go:227` |
|
||||
| UntagResource | registered | cross-resource (by ARN) | `tags.go:254` |
|
||||
| ListTagsForResource | registered | cross-resource (by ARN) | `tags.go:207` |
|
||||
|
||||
Unsupported actions are any ECS action absent from the table, and they return `InvalidAction`.
|
||||
Examples include task sets, `ExecuteCommand`, `UpdateCapacityProvider`, `DeleteTaskDefinitions`, attribute APIs and service deployments/revisions.
|
||||
Three agent-facing actions (`PollAssignments`, `ReportTaskGPU`, `ProvisionCapacity`) are non-AWS shapes on the AWS ECS signing surface.
|
||||
`SubmitTaskStateChange` is an AWS action that our agent uses as its state-report channel.
|
||||
|
||||
## 3. Durable records / KV buckets
|
||||
|
||||
| Bucket | Key shape | Record type | Owning resource | Writers | Readers |
|
||||
|---|---|---|---|---|---|
|
||||
| `ecs-account-<accountID>` (history 1, schema v1, created lazily, `store.go:27-29,207-217`) | `clusters/<name>/meta` | `ClusterRecord` (`records.go:155`) | cluster | CreateCluster `service.go:259`; PutClusterCapacityProviders `capacity_providers.go:36`; tags `tags.go:152`; DeleteCluster removes the subtree `cluster_teardown.go:47` | cluster CRUD; RunTask/StartTask/CreateService existence checks (`tasks.go:36`, `task_stop.go:60`, `services.go:148`) |
|
||||
| same | `clusters/<c>/instances/<id>` | `InstanceRecord` (`records.go:333`) | containerinstance | `upsertInstance` plain Put `instances.go:215`; `recordHeartbeat` `:356`; UpdateContainerInstancesState `cluster_teardown.go:131`; reaper `scheduler.go:390`; tags `tags.go:200`; CAS reserve/release `tasks.go:212`, `task_stop.go:148`, `instances.go:527`; Deregister deletes `cluster_teardown.go:90` | placement (`tasks.go:182`), cluster counts (`service.go:301`), reaper, describe/list |
|
||||
| same | `clusters/<c>/tasks/<taskID>` | `TaskRecord` (`records.go:389`) | task | RunTask `tasks.go:84`; StartTask `task_stop.go:104`; stop/force-stop `task_stop.go:171,199`; recordTaskState `instances.go:409,433`; EIP assign `task_stop.go:284`; ENI rollback `tasks.go:168`; GPU report `gpu_report.go:58`; sweep `tasks_gc.go:98,119`; tags `tags.go:189` | service reconcile (`services.go:589-622`), cluster/instance counts, reaper, sweep |
|
||||
| same | `clusters/<c>/assignments/<ciID>/<taskID>` | `bus.Assign` | task (agent inbox) | `publishAssign` `tasks.go:269` | agent via `PollAssignments` (`poll.go:58`); acked and reclaimed via `poll.go:49,97` |
|
||||
| same | `clusters/<c>/stops/<ciID>/<taskID>` | `bus.StopDirective` | task (agent inbox) | `requestStopTask` `task_stop.go:176` | agent via `PollAssignments` (`poll.go:74`); acked and reclaimed via `poll.go:55,106` |
|
||||
| same | `clusters/<c>/services/<name>` | `ServiceRecord` (`records.go:454`) incl. `Deployments`, `Events` | service | Create/Update/Delete `services.go:198,256,295`; reconcile `services.go:433`; deployment-failure count `instances.go:496`; tags `tags.go:178` | task paths read it for LB and EIP config (`task_stop.go:241,273`) |
|
||||
| same | `taskdef-families/<family>/latest-rev`, `taskdef-families/<family>/revs/<rev>` | `int`, `TaskDefRecord` (`records.go:280`) | taskdefinition | Register `service.go:452,455`; Deregister `:564`; tags `tags.go:167` | `resolveTaskDef` (`service.go:622`) for RunTask/StartTask/Create/UpdateService; gateway PassRole via NATS `DescribeTaskDefinition` |
|
||||
| same | `capacity-providers/<name>` | `CapacityProviderRecord` (`records.go:200`) | capacityprovider | Create `capacity_providers.go:72`; Delete `:142` | Describe only |
|
||||
| `spinifex-ecs-leader` (TTL 60 s, `store.go:31-33,221`) | `scheduler` (`scheduler.go:24`) | kvlease lease | scheduler (orchestration) | `kvlease` in `Scheduler` (`scheduler.go:96-109`) | the same |
|
||||
| `spinifex-ecs-clienttokens` (TTL 15 min, `gateway/ecs/clienttoken.go:22-26`) | `<accountID>.<token>` (`foundation/lifecycle/idempotency/store.go:91-93`, no namespace) | `idempotency` entry with `ecs.RunTaskOutput` | task (currently held by the adapter) | gateway `runTaskWithClientToken` (`clienttoken.go:71-86`) | the same |
|
||||
|
||||
`store.go:23-25`, `:134-138` and `LeaderLeaseKey` describe per-cluster `<accountID>/<clusterName>` lease keys, but the code uses one global key `scheduler`, and `LeaderLeaseKey` has no production caller.
|
||||
No reader outside ECS opens any of these buckets.
|
||||
A Go grep for bucket names and helpers outside `handlers/ecs` and `gateway/ecs` found only RDS's own `AccountBucketName`.
|
||||
`accountteardown` and the agent reach ECS state only through the ECS API (NATS facade or gateway), not through KV.
|
||||
No KV migration is registered for `ecs-account-*`; it runs `migrate.DefaultRegistry.RunKV` at version 1 (`store.go:213`).
|
||||
No production code was found that deletes an `ecs-account-*` bucket.
|
||||
|
||||
## 4. Identity and scope
|
||||
|
||||
The partition is hard-coded to `aws`.
|
||||
The Region is the daemon `config.Region` captured at construction (`daemon/daemon.go:2029`).
|
||||
The account is the SigV4 caller's account, carried on every NATS request.
|
||||
All records live in the caller's per-account bucket, so the account scope is structural.
|
||||
|
||||
| Resource | Identifier | ARN (`records.go:107-135`) | Scope |
|
||||
|---|---|---|---|
|
||||
| cluster | name; omitted means `default` (`conv.go:27-36`, `service.go:233-235`) | `arn:aws:ecs:<region>:<acct>:cluster/<name>` | account + Region; no AZ |
|
||||
| taskdefinition | `family:revision`; revision is `latest-rev + 1` (`service.go:531-541`) | `…:task-definition/<family>:<rev>` | account + Region |
|
||||
| service | name unique per cluster | `…:service/<cluster>/<name>` | account + Region; subnets/SGs stored, not validated |
|
||||
| task | UUIDv4 (`tasks.go:65`, `task_stop.go:85`) | `…:task/<cluster>/<taskID>` | account + Region; AZ is implicit through its container instance and is not stored on the task |
|
||||
| containerinstance | the `InstanceIdentityDocument` string supplied by the agent (in practice the EC2 instance ID), with fallback `ci-<UTC timestamp>` (`instances.go:67-70`) | `…:container-instance/<cluster>/<id>` | account + Region; AZ from the registration attribute (`instances.go:100-101,331`) |
|
||||
| capacityprovider | name | `…:capacity-provider/<name>` | account + Region |
|
||||
|
||||
AWS parity note (verify): AWS mints a UUID container-instance ID that is distinct from `ec2InstanceId`, and AWS expects `instanceIdentityDocument` to be the signed IMDS JSON document.
|
||||
Ours uses the supplied string as both values (`instances.go:276`).
|
||||
Identifier reuse: DeleteCluster removes every record (`cluster_teardown.go:47`), so a cluster name and its child names are reusable immediately.
|
||||
There is no tombstone or identifier-reuse rule (Q-43/Q-44).
|
||||
Task-definition revisions are not reserved against reuse after a concurrent collision (see gap G9).
|
||||
Container instances authenticate as the account's `ecsInstanceRole` (`instancerole.go:12-30`).
|
||||
`PollAssignments`, `SubmitTaskStateChange` and `ReportTaskGPU` check only account scope plus IAM policy on the named instance or task ARN (`authz.go:83-95`).
|
||||
The handlers do not bind the caller to the specific container instance (`poll.go:37-56`, `instances.go:445-475`, `gpu_report.go:31`).
|
||||
|
||||
## 5. Lifecycle paths per resource
|
||||
|
||||
There are no desired or observed generation fields on any ECS record (`records.go:155-493`).
|
||||
The only fencing mechanisms are:
|
||||
- one global scheduler lease (`scheduler.go:96-109`), checked only at the start of each pass (`:170`, `:189`);
|
||||
- CAS (`kv.Update` with revision) on container-instance reservations (`tasks.go:212`, `task_stop.go:148`, `instances.go:527`).
|
||||
|
||||
Every other write is a read-modify-`Put` without revision (`service.go:179-188`).
|
||||
ClientToken idempotency exists only for RunTask (`gateway/ecs/actions.go:203-238`).
|
||||
|
||||
### cluster
|
||||
Create is `CreateCluster` (`service.go:231-264`): get, then a plain Put if absent.
|
||||
A repeat create by name returns the existing record without comparing parameters; AWS also treats it as idempotent (verify).
|
||||
Readiness: the cluster is `ACTIVE` on create (`:250`), synchronously.
|
||||
There is no reconcile loop.
|
||||
Delete is `DeleteCluster` (`cluster_teardown.go:24-53`): it force-stops every task (:43-45), deletes the whole `clusters/<n>/` subtree (:47), and returns a synthetic `INACTIVE` (:51).
|
||||
No durable deletion intent is written.
|
||||
On teardown it is driven by `accountteardown/reapers_ecs.go:53-58`.
|
||||
|
||||
### taskdefinition
|
||||
Create is `RegisterTaskDefinition` (`service.go:416-459`): it validates (`:472-511`), computes `rev = latest + 1` without CAS (`:530-541`), Puts the revision (`:452`), then Puts `latest-rev` (`:455`).
|
||||
There is no client token, and AWS has none for this action.
|
||||
Readiness: the definition is `ACTIVE` synchronously.
|
||||
Delete is `DeregisterTaskDefinition` (`:546-568`), which sets `INACTIVE` and retains the record.
|
||||
There is no hard delete (`DeleteTaskDefinitions` is not served).
|
||||
`resolveTaskDef` (`:622-647`) does not check status, so RunTask, StartTask and CreateService accept an `INACTIVE` revision.
|
||||
AWS refuses new runs and services on an INACTIVE revision after a short window (verify).
|
||||
|
||||
### service
|
||||
Create is `CreateService` (`services.go:128-206`): an `ACTIVE` service with the same name is returned as-is (:159-160), and `ClientToken` is ignored.
|
||||
It Puts the record with a PRIMARY deployment (:196-198), then runs `reconcileService` synchronously on the serving worker (:202).
|
||||
Readiness: the service record is `ACTIVE` immediately.
|
||||
Rollout state moves `IN_PROGRESS` → `COMPLETED` when the primary running count reaches the desired count and no old-deployment tasks remain (`deployments.go:170-196`).
|
||||
The circuit breaker sets `FAILED` after 3 failed starts (`deployments.go:222-243`, `records.go:54`), with optional rollback (`:247-267`).
|
||||
Launch backoff is 15 s to 5 min (`deployments.go:206-216`).
|
||||
Reconcile has two triggers:
|
||||
- the leader loop `ecs/services` (`scheduler.go:143-148`) runs `reconcileAllServices` (`services.go:445-476`) on a wake from `ecs.bus.<acct>.<cluster>.service-reconcile` (`instances.go:472`), a 10 s revisit while unsettled, and a 5 min resync;
|
||||
- `CreateService` and `UpdateService` also reconcile inline on the API worker (`services.go:202`, `:259`).
|
||||
The reconcile launches through `s.RunTask` without a client token (`services.go:495-525`) and stops surplus tasks (`:531-581`).
|
||||
Delete is `DeleteService` (`:267-299`): it requests a stop for each task (:288-289), then sets `INACTIVE` immediately (:291).
|
||||
`ServiceStatusDraining` (`records.go:30`) is never written.
|
||||
There is no `force` or desired-count precondition check, and the INACTIVE record is never pruned.
|
||||
|
||||
### task
|
||||
Create is `RunTask` (`tasks.go:28-93`) or `StartTask` (`task_stop.go:53-113`), in this order:
|
||||
1. Reserve capacity by CAS on the container instance (`tasks.go:180-218` / `task_stop.go:118-153`).
|
||||
2. Build the record in memory (`tasks.go:221-243`; desired `RUNNING`, last `PENDING`).
|
||||
3. For awsvpc, create and attach the ENI through EC2 (`tasks.go:131-176`).
|
||||
4. Make the first durable task write (`tasks.go:84` / `task_stop.go:104`).
|
||||
5. Write the assignment inbox (`tasks.go:249-270`); a failure there is only logged (`:87-89`).
|
||||
Readiness comes from the agent's `SubmitTaskStateChange`, which goes through `recordTaskState` (`instances.go:361-439`).
|
||||
`RUNNING` sets `StartedAt`, registers ELBv2 targets and assigns an EIP when the service asks for it (:415-418).
|
||||
`STOPPED` releases the target, EIP, inboxes, ENI and reservation (:422-436).
|
||||
Stop has two paths:
|
||||
- `StopTask` calls `requestStopTask` (`task_stop.go:162-179`), which sets desired `STOPPED` and posts a stop directive for the agent.
|
||||
- `forceStopTask` (`:186-209`) is used by the cluster delete, deregister, drain and reaper paths; it marks the task `STOPPED` without telling the agent.
|
||||
Repair: the leader `ecs/timers` loop runs `sweepStoppedTasks` (`tasks_gc.go:21-105`).
|
||||
That sweep retries owed ENI releases (at most 20 per pass, 30 s to 15 min backoff) and deletes STOPPED tasks after 1 h.
|
||||
Idempotency: RunTask honours ClientToken through `idempotency.Do` (`gateway/ecs/clienttoken.go:71-86`); a partial placement is finalized and replayed.
|
||||
StartTask has no token, and AWS has none for it (verify).
|
||||
Internal launch from the service reconcile is not idempotent per logical launch.
|
||||
|
||||
### containerinstance
|
||||
Create is `RegisterContainerInstance` (`instances.go:65-124`), which is an upsert with a plain Put (`:195-219`).
|
||||
The agent calls it at boot and on every 30 s heartbeat (`agents/ecs/agent/controlplane.go:22-25`).
|
||||
Readiness: the instance is `ACTIVE` on first registration (`:207`).
|
||||
A zero reported capacity drains it (`Reaped=true`), and returning capacity restores it (`:44-60`).
|
||||
`AgentConnected` is derived from `Status == ACTIVE` (`:278`), not from liveness.
|
||||
Repair: the leader reaper (`scheduler.go:340-396`) marks an instance `DRAINING` with `Reaped=true` after 90 s without `LastSeen`, and force-stops its tasks (`:400-416`).
|
||||
`recordRegister` and `recordHeartbeat` (`instances.go:325-357`) are fed only by bus subjects that nothing publishes.
|
||||
Operator state: `UpdateContainerInstancesState` (`cluster_teardown.go:103-140`) sets the status.
|
||||
`DRAINING` force-stops service tasks immediately (`:160-171`).
|
||||
Delete is `DeregisterContainerInstance` (`:60-97`): it refuses if tasks are active unless `force`, deletes the assignments inbox and the record (the stops inbox is left), and does not touch the EC2 instance.
|
||||
Capacity is provisioned by `ProvisionCapacity` (`capacity.go:70-170`), which calls `RunInstances` directly with no ECS-side intent record and no idempotency.
|
||||
|
||||
### capacityprovider
|
||||
Create is `CreateCapacityProvider` (`capacity_providers.go:44-76`): an existing name returns the existing record without comparing parameters, and the ASG ARN is stored without validation.
|
||||
Readiness: it is `ACTIVE` immediately and inert, with no scheduler coupling (`service.go:74-76`).
|
||||
Delete is a hard key delete (`:125-146`), with no check that a cluster references the provider.
|
||||
|
||||
## 6. NATS subjects and guest/controller messages
|
||||
|
||||
| Subject | Direction / type | Payload | Retry / redelivery |
|
||||
|---|---|---|---|
|
||||
| `ecs.<Action>` ×34 (`service_nats.go:29-177`; subs `daemon.go:1134-1167`) | gateway (and `accountteardown`, gateway PassRole lookup) → daemon, request/reply, queue `spinifex-workers` | aws-sdk-go ECS input/output or `PollAssignments*`, `ReportTaskGPU*`, `ProvisionCapacity*` structs | 30 s timeout. There is no retry in `NATSECSService`, and `natsmsg.NATSRequest` (`foundation/messaging/nats/nats.go:269`) retry behaviour was not verified here. |
|
||||
| `ecs.bus.<acct>.<cluster>.service-reconcile` (`bus/subjects.go:43`) | any worker → leader, core publish | none | Fire-and-forget (`instances.go:472-474`); loss is covered by the 10 s revisit and 5 min resync (`scheduler.go:30-58`). |
|
||||
| `ecs.bus.*.*.instance-register.*`, `…instance-heartbeat.*`, `…task-state.*` (`scheduler.go:236-245`) | subscribed by leader | `bus.RegisterInstance`, `bus.Heartbeat`, `bus.TaskState` | No production publisher (residue). Core NATS with no ack or redelivery, and messages are dropped while there is no leader. |
|
||||
| `ecs.bus.<acct>.<cluster>.assign.<id>` (`bus/subjects.go:30`) | unused | — | Residue; assignment moved to the KV inbox. |
|
||||
| `ec2.CreateNetworkInterface`, `ec2.DeleteNetworkInterface` (`eni.go:64,119`) | ECS → EC2 AWS-action subjects | aws-sdk-go EC2 types | 30 s timeout (`eni.go:19`). NotFound counts as converged (`:148-155`). A failed release is retried by the sweep. |
|
||||
| `ec2v1.InstanceCommandSubject(<instanceID>)` (`eni.go:126-128`) | ECS → owning node, `contracts/ec2/v1` | `EC2InstanceCommand{AttachENI or DetachENI}` | Versioned contract; detach is forced (`:109`). |
|
||||
| `ec2.AllocateAddress`, `ec2.AssociateAddress`, `ec2.DescribeAddresses`, `ec2.DisassociateAddress`, `ec2.ReleaseAddress` via `ec2eip.NewNATSEIPService` (`services.go:81-121`; subjects in `domains/ec2/eip/service_nats.go:26-34`) | ECS → EC2 EIP AWS adapter | aws-sdk-go EC2 types | A failed associate releases immediately (`services.go:91-96`). A failed release is logged, and the fields are cleared anyway (`task_stop.go:296-300`). |
|
||||
| ELBv2 `RegisterTargets` / `DeregisterTargets` via `handlers_elbv2.NewNATSELBv2Service` (`services.go:45-58`) | ECS → ELBv2 AWS adapter | aws-sdk-go ELBv2 types | Best effort and log-only (`task_stop.go:252-255`), with no retry. The exact subject string was not verified. |
|
||||
| `ec2.RunInstances.<type>.<node>` | not used by ECS | — | ECS passes no node, so `RunWorkerInstance` runs the local in-process `instanceService.RunInstances` (`daemon/eks_worker_launch.go:29-51`). |
|
||||
|
||||
Guest protocol: agent ↔ gateway HTTP, SigV4-signed with the instance role through `internal/ecsgw`; it is not NATS.
|
||||
- `RegisterContainerInstance`: boot, then every 30 s as liveness.
|
||||
- `PollAssignments`: delivery is at-least-once. The inboxes are KV keys, and an entry is redelivered on every poll until the agent acks it in `AckTaskIDs` or `AckStopIDs` on the next poll (`poll.go:45-56`), or until the task stops (`instances.go:430-431`, `task_stop.go:204-205`).
|
||||
- `SubmitTaskStateChange`: duplicate reports are tolerated only through edge detection on `prev != status` (`instances.go:415,422`). There is no ordering or monotonicity guard and no check of the reporting instance (`:372-373`).
|
||||
- `ReportTaskGPU`.
|
||||
Bootstrap contract: cloud-config writes `/etc/spinifex-ecs/agent.env` and the gateway CA (`userdata.go:31-58`).
|
||||
|
||||
## 7. Boundary-crossing imports
|
||||
|
||||
ECS → other packages (non-stdlib, non-foundation first):
|
||||
|
||||
| From | Imports | Symbol(s) | Flag |
|
||||
|---|---|---|---|
|
||||
| `handlers/ecs/service.go:19`, `instancerole.go:6` | `handlers/iam` | `SystemInstanceRoleEnsurer` (`service.go:92`), `EnsureSystemInstanceProfile` (`instancerole.go:38`), `ConvergeSystemRolePolicy` (`:46`) | IAM |
|
||||
| `handlers/ecs/services.go:20` | `handlers/elbv2` | `NewNATSELBv2Service().RegisterTargets/DeregisterTargets` | other domain's AWS adapter used as an internal API |
|
||||
| `handlers/ecs/services.go:18` | `domains/ec2/eip` | `NewNATSEIPService` (Allocate/Associate/Describe/Disassociate/Release) | EC2 / network; AWS adapter as internal API |
|
||||
| `handlers/ecs/eni.go:13` | `contracts/ec2/v1` | `EC2InstanceCommand`, `EC2CommandAttributes`, `AttachENIData`, `DetachENIData`, `InstanceCommandSubject` | EC2 instance and network (versioned contract) |
|
||||
| `handlers/ecs/eni.go:64,119` | string subjects `ec2.CreateNetworkInterface` / `ec2.DeleteNetworkInterface` | — | network; EC2 AWS-action subject used as an internal API |
|
||||
| `handlers/ecs/capacity.go:11-12` | `domains/ec2/ebs/metadata`, `domains/ec2/instancetypes` | `RootDeviceName`; `IsGPUTypeName`, `GPUVendorForType` | EC2 |
|
||||
| `handlers/ecs/placement.go:9` | `domains/ec2/instancetypes` | `HotPlugENISlotsForType` | EC2 |
|
||||
| `handlers/ecs/service.go:16`, `capacity.go:10`, `eni.go:12`, `services.go:15-17` | aws-sdk-go `service/ec2`, `service/elbv2` | `RunInstancesInput`, `DescribeImages*`, ENI/EIP/target types | EC2 / ELBv2 types |
|
||||
| `handlers/ecs/*` | foundation | `messaging/nats.NATSRequest`, `state/kvlease`, `state/kvutil`, `state/migrate`, `lifecycle/reconciler`, `aws/errors`, `aws/arn`, `aws/ami`, `aws/tags` | — |
|
||||
| `gateway/ecs/authz.go:11-12` | `ingress/aws/bodyscope`, `ingress/aws/query` | `bodyscope.Parse`, `query.MaxSliceLen` | ingress |
|
||||
| `gateway/ecs/clienttoken.go:13` | `foundation/lifecycle/idempotency` | `OpenStore`, `Do`, `ParamHash`, `ErrParamMismatch`, `ErrUnavailable` | — |
|
||||
| `daemon/ecs_deps.go:8` | `github.com/mulgadc/bluebottle/pkg/masterkey` | `ReadShared` (:39) | Bluebottle (daemon wiring only; `handlers/ecs` itself does not import Bluebottle) |
|
||||
| `daemon/ecs_deps.go:10,50-52` | `handlers/iam` | `NewIAMServiceImpl`, `*IAMServiceImpl` | IAM |
|
||||
| `daemon/ecs_deps.go:29,32` | EKS-named daemon helpers | `resolveGatewayBaseURL` (`daemon/eks_deps.go:164`), `RunWorkerInstance` (`daemon/eks_worker_launch.go:29`) | EC2 instance launch shared with EKS |
|
||||
|
||||
Packages that import ECS:
|
||||
|
||||
| Importer | Imports | Symbol(s) |
|
||||
|---|---|---|
|
||||
| `gateway/ecs/actions.go:12` | `handlers/ecs` | `NewNATSECSService`, `ECSService`, `ProvisionCapacityInput`, `PollAssignmentsInput`, `ReportTaskGPUInput` |
|
||||
| `gateway/ecs/authz.go:10` | `handlers/ecs` | `ClusterShortName`, `ServiceShortName`, `TaskShortID`, `ContainerInstanceShortID`, `CapacityProviderShortName`, `ParseTaskDefRef`, `ResourceARNSegment`, `ClusterARN`, `ServiceARN`, `TaskARN`, `ContainerInstanceARN`, `CapacityProviderARN`, `TaskDefARN`, `TaskDefRefARN` |
|
||||
| `gateway/ecs.go:10`, `gateway/operations.go:7` | `gateway/ecs` | `Actions`, `ResourceARNs`, `RawJSONActions`, `WriteJSONResponse`, `WriteRawJSONResponse`, `StubbedActionNames` |
|
||||
| `daemon/daemon.go:74`, `daemon/ecs_deps.go:9` | `handlers/ecs` | `Service`, `Scheduler`, `NewService`, `WithDeps`, `InitLeaderBucket`, `NewScheduler`, `Deps` |
|
||||
| `accountteardown/reapers_ecs.go:9` | `handlers/ecs` | `NewNATSECSService`, `ECSService` (`ListClusters`, `DeleteCluster`) |
|
||||
| `agents/ecs/agent/{registration,controlplane,taskrun}.go` | `handlers/ecs` (implementation package) | `AttrInstanceType`, `AttrAvailabilityZone`, `PollAssignmentsInput/Output`, `ReportTaskGPUInput`, `ContainerGPUReport` |
|
||||
| `agents/ecs/agent/{agent,controlplane,reconcile,registration,taskrun}.go` | `handlers/ecs/bus` | `Assign`, `StopDirective`, `TaskState`, `ContainerStatus`, `InstanceCapacity`, status constants |
|
||||
|
||||
Explicit flags:
|
||||
- **IAM / PassRole.** PassRole is enforced only in the gateway (`gateway/ecs.go:65-67`, through `gw.checkPassRole` at `gateway/gateway.go:645-648` with `iam:PassedToService=ecs-tasks.amazonaws.com`).
|
||||
RegisterTaskDefinition checks the role ARNs in the body (`actions.go:99`).
|
||||
RunTask, StartTask, CreateService and UpdateService first fetch the task definition through the ECS NATS facade (`actions.go:44-56`).
|
||||
`ProvisionCapacity` attaches the system `ecsInstanceRole` instance profile (`capacity.go:103,141`) and launches through `RunWorkerInstance`, which goes straight to `d.instanceService.RunInstances` (`daemon/eks_worker_launch.go:51`).
|
||||
That bypasses the EC2 gateway's PassRole closure (`gateway/ec2.go:199`).
|
||||
`handlers/sts/assume_role.go:52-55` duplicates `ecsInstanceRole`, which is implicit coupling with no import.
|
||||
- **Quota.** `handlers/ecs` has no quota import.
|
||||
`ProvisionCapacity` appears to bypass `gw.Quota.EnforceLaunch` and `ChargeLaunch` (`gateway/ec2.go:196-205`), because it calls the in-process instance service (verify whether `instanceService.RunInstances` enforces quota itself).
|
||||
ECS resources themselves have no quota.
|
||||
- **EC2 instance.** `RunInstances` (via `Deps.RunInstances`), `DescribeImages` (via `Deps.Images` = `d.imageService`), `instancetypes`, EBS root-device metadata, and the `contracts/ec2/v1` instance command for ENI hot-plug.
|
||||
- **Network.** ENI create/delete through EC2 AWS subjects, EIP through the `domains/ec2/eip` NATS adapter, subnets and SGs passed through unvalidated.
|
||||
- **Bluebottle.** Only `masterkey.ReadShared` in `daemon/ecs_deps.go:39`, used to build the IAM service.
|
||||
Without it, `ProvisionCapacity` is disabled and the scheduler refuses leadership (`scheduler.go:226-231`).
|
||||
|
||||
## 8. Known gaps against Q-78 / ADR-0003 (observed)
|
||||
|
||||
- **G1. No generations.** No ECS record has a desired or observed generation (`records.go:155-493`). Reconcilers act on whole records with plain Put (`service.go:179-188`).
|
||||
- **G2. Realization before intent.** RunTask and StartTask reserve capacity and create/attach an ENI before the task record exists (`tasks.go:66-84`, `task_stop.go:86-104`). A crash in that window leaves a reservation (`PlacedTasks` naming an unknown task) and possibly an ENI with no owning record. No reconciler targets orphan reservations; `tasks_gc.go` handles only owed ENI release on STOPPED tasks.
|
||||
- **G3. Lost-update window on instance reservations.** Reservations use CAS (`tasks.go:212`, `task_stop.go:148`, `instances.go:527`), but the same key is rewritten with a plain read-modify-Put every 30 s by agent registration (`instances.go:195-215`). The same happens in UpdateContainerInstancesState (`cluster_teardown.go:131`), the reaper (`scheduler.go:390`) and tags (`tags.go:200`), so a CAS commit between their Get and Put can be overwritten. `handlers/ecs/task_accounting_test.go` (`TestDescribeContainerInstances_StalePlacedTasksDoNotInflateCount`) acknowledges stale `PlacedTasks`.
|
||||
- **G4. Unfenced concurrent writers.** The leader lease is checked only at pass start (`scheduler.go:170,189`). KV writes are not fenced by the lease. API workers also run `reconcileService` inline (`services.go:202,259`) while the leader runs `reconcileAllServices` (`scheduler.go:177`), so two writers can launch through `RunTask` for the same service. This is the observed code shape; no test was found covering it.
|
||||
- **G5. Cascade delete removes cleanup ownership.** `DeleteCluster` (`cluster_teardown.go:43-47`) deletes the whole subtree, including STOPPED tasks whose `ENIReleased=false` or whose EIP or target cleanup failed. That removes the only record of owed cleanup, which Q-78 forbids. There is no deletion intent or incomplete-cleanup state. AWS refuses DeleteCluster while services, tasks or registered instances exist (verify against AWS docs and a test).
|
||||
- **G6. Force-stop does effects before persisting.** `forceStopTask` releases the EIP and ENI before persisting STOPPED (`task_stop.go:197-199`). A failed EIP release is logged and its allocation ID cleared (`:296-300`), so it cannot be retried. Target deregistration is best effort with no retry (`:252-255`). `forceStopTask` also posts no stop directive, so whether a live agent stops its containers is unverified.
|
||||
- **G7. EIP before record.** `assignTaskPublicIP` allocates and associates the EIP before persisting its allocation ID (`task_stop.go:277-286`).
|
||||
- **G8. Silent reservation leak.** `releaseReservation` gives up silently after 5 CAS conflicts and returns nil (`instances.go:506-531`), which leaks capacity with no reconciler.
|
||||
- **G9. Non-atomic revision allocation.** `RegisterTaskDefinition` computes the revision as read + 1 with no CAS (`service.go:430,531-541`). It writes the revision before `latest-rev` (`:452-455`), so concurrent registers can produce the same revision and overwrite it.
|
||||
- **G10. No guard on task-state reports.** `recordTaskState` sets `LastStatus` unconditionally (`instances.go:372-373`), so a late RUNNING after STOPPED regresses the state. The reporting `InstanceID` is ignored.
|
||||
- **G11. DeleteService is immediate.** It sets INACTIVE directly, with no DRAINING (`services.go:291`; `records.go:30` is unused). It has no `force` or desired-count check, AWS requires one (verify), and the INACTIVE record is never pruned.
|
||||
- **G12. Create idempotency without comparison.** CreateService ignores `ClientToken` and returns an existing ACTIVE service without comparing parameters (`services.go:159-160`). CreateCapacityProvider (`capacity_providers.go:60-61`) and CreateCluster (`service.go:246-262`) behave the same way.
|
||||
- **G13. Retry bookkeeping in the adapter.** The RunTask idempotency store lives in the AWS adapter (`gateway/ecs/clienttoken.go:22-56`). ADR-0003 S3 says retry bookkeeping is not the adapter's.
|
||||
- **G14. ProvisionCapacity.** It has no durable intent and no idempotency, and it silently caps count at 10 (`capacity.go:89-91,158`). Launched instances are linked only by the tag `spinifex:ecs-cluster` (`:150`).
|
||||
- **G15. Liveness derived from status.** Container-instance `AgentConnected` is derived from Status, not liveness (`instances.go:278`). Registration makes an instance ACTIVE immediately (`:207`).
|
||||
- **G16. Residue subscriptions.** The scheduler subscribes to three bus subjects that nothing publishes (`scheduler.go:236-245`).
|
||||
- **G17. AWS adapters as internal APIs.** Account teardown uses the ECS facade (`accountteardown/reapers_ecs.go:21`), and gateway PassRole resolution calls `DescribeTaskDefinition` over NATS (`actions.go:48`). ECS calls the ELBv2 and EIP adapters and the `ec2.*NetworkInterface` subjects (`services.go:46,82`, `eni.go:64,119`). ADR-0003 S4 and ADR-0004 S4 forbid an AWS adapter used as an internal service API.
|
||||
- **G18. Incomplete account teardown.** It reaps only clusters (`reapers_ecs.go:20-22`). Task definitions, capacity providers and the `ecs-account-*` bucket remain; no deleting code was found.
|
||||
- **G19. No pagination.** List APIs ignore `nextToken` and `maxResults` (`service.go:338,586`, `tasks.go:299`, `services.go:326`, `instances.go:167`). Q-89 requires listing tests that exceed one page.
|
||||
- **G20. Status not checked on launch.** RunTask, StartTask and CreateService accept an INACTIVE task-definition revision (`service.go:622-647`), but AWS refuses (verify).
|
||||
- **G21. Capacity providers not taggable.** Capacity-provider ARNs are rejected by the tag parser (`tags.go:52-71`), although create accepts tags.
|
||||
- **G22. Delete leaves stop inboxes.** `DeregisterContainerInstance` deletes the assignments inbox but not the stops inbox (`cluster_teardown.go:87-90`).
|
||||
|
||||
## 9. First-resource candidates (ranked)
|
||||
|
||||
1. **taskdefinition.**
|
||||
Size: about 300 lines in `service.go:416-706` plus about 180 in `conv.go:70-265`, plus `TaskDefRecord` and `ContainerDef` (`records.go:211-328`).
|
||||
Private state: `taskdef-families/*` only.
|
||||
Dependencies: none at the handler. PassRole is checked in the gateway on role ARN strings, so extraction does not require deciding IAM authority. It has no EC2, network or Bluebottle dependency.
|
||||
Consumers needing a narrow query: task launch, service create/update (`resolveTaskDef`), gateway PassRole lookup, and tags.
|
||||
Tests: `handlers/ecs/taskdef_validation_test.go`, `service_test.go` (`RegisterTaskDefinition_*`, `ListTaskDefinitions_*`, `DescribeTaskDefinition_Unknown`), `cluster_teardown_test.go` (`DeregisterTaskDefinition_*`), `conv_test.go`, `records_test.go`, `tags_test.go` (`TaskDefinitionRoundTrip`).
|
||||
Known gaps: G9 and G20.
|
||||
Assessment: the best first slice. Its `resolveTaskDef` edge also gives the ADR-0004 rollout item 3 capability seam.
|
||||
2. **capacityprovider.**
|
||||
Size: about 220 lines in `capacity_providers.go` plus records.
|
||||
Private state: `capacity-providers/*`.
|
||||
Dependencies: none; the provider is inert, and the ASG ARN is opaque.
|
||||
Prerequisite split: `PutClusterCapacityProviders` must move to `cluster`.
|
||||
Tests: `handlers/ecs/capacity_providers_test.go`.
|
||||
Gaps: G12, G21, and an unchecked delete.
|
||||
Assessment: the smallest and safest slice, with no IAM, EC2 or Bluebottle dependency. It has almost no consumers, so it demonstrates little about cross-resource capabilities.
|
||||
3. **cluster.**
|
||||
Size: CRUD of about 180 lines (`service.go:231-411`) plus `PutClusterCapacityProviders`.
|
||||
Private state: `clusters/<n>/meta`.
|
||||
Dependencies: counts read instance, task and service records (`service.go:300-335`), and its existence is checked by task and service launch. The delete cascade is orchestration over every child (G5).
|
||||
Tests: `service_test.go` (`CreateCluster_Idempotent`, `DescribeClusters_*`, `ListClusters_*`), `cluster_teardown_test.go` (`DeleteCluster_*`), `capacity_providers_test.go`, `tags_test.go`, `accountteardown/reapers_ecs_test.go`.
|
||||
Assessment: a CRUD-only slice is feasible without IAM or Bluebottle. DeleteCluster has to stay as domain-root orchestration until the children have owners.
|
||||
4. **containerinstance.**
|
||||
Size: about 450 lines across `instances.go`, part of `cluster_teardown.go` and `placement.go`.
|
||||
Private state: `instances/*` plus the reservation fields, which placement and task CAS-mutate.
|
||||
Dependencies: the agent wire contract, EC2 (`instancetypes`; `RunInstances` via `ProvisionCapacity`) and IAM (`ecsInstanceRole`).
|
||||
Tests: `instance_recovery_test.go`, `instance_list_status_test.go`, `placement_test.go`, `eni_placement_test.go`, `capacity_test.go`, `scheduler_test.go`, `task_accounting_test.go`, `service_test.go` (GPU/register/heartbeat/reaper).
|
||||
Assessment: G3 must be characterized first, and reservation ownership needs a decision. It is not a first slice.
|
||||
5. **service.**
|
||||
Size: about 1,050 lines (`services.go`, `deployments.go`).
|
||||
Dependencies: task launch and stop, taskdefinition, ELBv2 targets and EC2 EIP. Its reconcile runs on both the leader and the API workers (G4).
|
||||
Tests: `services_test.go`, `deployments_test.go`, `revisit_test.go`, `eip_test.go`, `services_eip_nats_test.go`, gateway `actions_test.go` and `azrebalancing_test.go`.
|
||||
Assessment: it depends on the task and taskdefinition owners existing.
|
||||
6. **task.**
|
||||
Size: about 1,300 lines (`tasks.go`, `task_stop.go`, `tasks_gc.go`, `poll.go`, `eni.go`, `netcfg.go`, `gpu_report.go`, part of `instances.go`).
|
||||
Dependencies: EC2 ENI and EIP, ELBv2, container-instance reservations, the agent contract, the ClientToken store and gateway PassRole.
|
||||
Tests: `eni_test.go`, `eni_placement_test.go`, `tasks_gc_test.go`, `task_accounting_test.go`, `poll_test.go`, `gpu_report_test.go`, `service_test.go` (`RunTask_*`, `RecordTaskState_*`), gateway `clienttoken_test.go`, `actions_test.go`, and e2e `tests/e2e/ecs/ecs_test.go` and `tests/e2e/gpu/ecs_gpu_test.go` in the mulga root.
|
||||
Assessment: it carries most of G2, G3, G6, G7, G8 and G10, so it should be extracted last.
|
||||
@@ -0,0 +1,308 @@
|
||||
# ADR-0004 source inventory — EKS
|
||||
|
||||
Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`.
|
||||
It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it.
|
||||
Items marked "verify" or "uncertain" are unconfirmed.
|
||||
Gaps record observed behaviour, not target behaviour.
|
||||
|
||||
|
||||
Repo: `spinifex` submodule, branch `refactor/spinifex-package-boundaries`; all paths are relative to `spinifex/spinifex/` unless stated.
|
||||
Binding inputs: ADR-0001, ADR-0002, ADR-0003, ADR-0004 and Q-78 as on `origin/main` of mulga.
|
||||
Proposed resource vocabulary (ADR-0004 S1): `cluster`, `nodegroup`, `addon`, `access`.
|
||||
Line numbers are from the working tree at time of writing; "uncertain" marks inferences not verified against a test or live system.
|
||||
|
||||
## 1. File table
|
||||
|
||||
Production `.go` files only; `_test.go` excluded.
|
||||
Line counts are `wc -l`.
|
||||
|
||||
### 1.1 `handlers/eks/` (single package `handlers_eks`, 37 files, 12,310 lines)
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `access_entry.go` | 255 | resource owner (`access`) | AccessEntry record CRUD/CAS, supported access-policy catalogue, scope validation, ARN/record projection. |
|
||||
| `addon_catalog.go` | 121 | resource owner (`addon`) | Read-only add-on version catalogue backing CreateAddon/DescribeAddonVersions. |
|
||||
| `addon_installer.go` | 97 | resource owner (`addon`) + guest wire contract | Stages manifests in KV; `StagedAddonManifest` is the payload the on-VM addon-sync agent reads, so the type should split to a contract. |
|
||||
| `addon_state.go` | 186 | resource owner (`addon`) | AddonRecord, status enum, CAS update, list/delete. |
|
||||
| `addon_status_reconciler.go` | 79 | resource owner (`addon`), hosted on cluster reconciler | Transition function is addon semantics but is a method on `ClusterReconciler`; split: addon owns `nextAddonStatus`/apply, cluster reconciler only relays reports. |
|
||||
| `addons.go` | 348 | resource owner (`addon`) + guest wire contract | Six AWS add-on actions plus internal `ListStagedAddonManifests` served to the guest. |
|
||||
| `awsgw_endpoint.go` | 42 | resource owner (`cluster`) | Builds the per-cluster OIDC issuer URL. |
|
||||
| `claim.go` | 31 | resource owner (`nodegroup`) | CAS-create helper; only caller is `nodegroup.go:123`. |
|
||||
| `clienttoken.go` | 57 | resource owner (`cluster`) | CreateCluster `clientRequestToken` store (`spinifex-eks-clustertokens`). |
|
||||
| `cluster_reconciler.go` | 1209 | resource owner (`cluster`), mixed | Cluster readiness/health, CP restart, member replacement, etcd reset; also subscribes the addon status subject (`:265-268`, `:447-458`) on behalf of `addon`. |
|
||||
| `cluster_state.go` | 515 | resource owner (`cluster`) | `ClusterMeta` record, status enum, CAS helpers, delete-reap bookkeeping, `DeleteClusterPrefix` (which also erases nodegroup/addon/access keys). |
|
||||
| `cp_vpc.go` | 128 | resource owner (`cluster`) | EKS-specific managed CP VPC realization via `domains/network/systemvpc`. |
|
||||
| `eks_billable_reaper.go` | 133 | resource owner (`cluster`) | Node-local GC backstop for CP VMs whose meta is gone; comment at `:3` cites "ADR-0006 §5", which is not an accepted ADR. |
|
||||
| `eks_deleting_reaper.go` | 159 | resource owner (`cluster`) | Re-drives DELETING clusters with backoff and exhaustion. |
|
||||
| `igw.go` | 32 | resource owner (`cluster`) | Legacy-topology IGW ensure/delete via `systemvpc`. |
|
||||
| `k3s_ha_control_plane.go` | 628 | resource owner (`cluster`), mixed | CP placement, spread, replacement, fresh CP; also contains `natsHostScheduler` (`:461-628`), a NATS fan-out over `contracts/cluster/v1` that is a capability implementation and should move to the composition root or the scheduler owner. |
|
||||
| `k3s_server_role.go` | 51 | resource owner (`cluster`) | Ensures the system CP instance profile through `handlers_iam.EnsureSystemInstanceProfile` (identity dependency). |
|
||||
| `k3s_server_vm.go` | 700 | resource owner (`cluster`) + guest wire contract | CP VM/ENI launch/terminate and AMI lookup; `buildK3sUserData` (`:524+`) is the CP guest boot contract; `lookupEKSGPUNodeAMI` is used by `nodegroup`. |
|
||||
| `nats_bootstrap.go` | 347 | resource owner (`cluster`) + guest wire contract | Bootstrap subject vocabulary and envelope (`:47-57`) are a guest contract; persistence of token/kubeconfig/JWKS/CA is cluster state. |
|
||||
| `nlb.go` | 591 | resource owner (`cluster`) | EKS-specific NLB realization via an ELBv2-shaped capability; also `ReapLBCLoadBalancers` (`:517`) for LBC-created ALBs in the customer VPC. |
|
||||
| `nodegroup.go` | 1409 | resource owner (`nodegroup`), mixed | Nodegroup CRUD, scaling CAS, worker launch/terminate; also creates customer IAM roles/instance profiles (`:768-838`), stages a GPU add-on (`:410`, crosses into `addon`), and decrypts the cluster join token (`:1160`, reads `cluster` secrets). |
|
||||
| `nodegroup_userdata.go` | 171 | resource owner (`nodegroup`) + guest wire contract | Worker cloud-init (K3S_TOKEN, registry mirrors) is the worker guest boot contract. |
|
||||
| `oidc_keypair.go` | 202 | resource owner (`cluster`) | Per-cluster OIDC signing key (encrypted) and JWKS. |
|
||||
| `private_endpoint.go` | 88 | resource owner (`cluster`) | Customer-VPC private-endpoint ENI. |
|
||||
| `reconciler_registry.go` | 183 | resource owner (`cluster`) | In-process registry of per-cluster reconciler goroutines. |
|
||||
| `recovery_directive.go` | 144 | resource owner (`cluster`) + guest wire contract | Per-member etcd recovery directive; `RecoveryDirective` is read by the on-VM recovery agent. |
|
||||
| `restore_snapshot.go` | 355 | resource owner (`cluster`) | Operator-only total-loss DR path; reads etcd snapshots through `providers/objectstore`. |
|
||||
| `security_groups.go` | 505 | resource owner (`cluster`), mixed | Cluster SG realization in system and customer VPCs; `EnsureNodegroupSGRules` (`:351`) is nodegroup realization. |
|
||||
| `service.go` | 73 | temporary residue | `EKSService`, one method per AWS action plus internal methods; it is the NATS RPC surface between gateway and daemon and is also consumed by `accountteardown` and the operator CLI; removal condition: per-resource commands behind `domains/eks/awsapi` registration (ADR-0004 S3) with any remaining process boundary versioned under `contracts/eks/v1`. |
|
||||
| `service_impl.go` | 2518 | mixed: cross-resource orchestration + resource owners | Split: CreateCluster launch sequence and `purgeClusterInfra` (`:504-1013`, `:1268-1452`) are cluster realization orchestration; cluster read/delete (`:1041-1162`) is `cluster`; access-entry actions (`:1527-1816`) are `access`; nodegroup wrappers (`:1465-1525`) are `nodegroup`; Tag actions (`:1842-1992`) span cluster and nodegroup; IdP stubs (`:1818-1832`); `DesiredDNSChanges` (`:2143`) is a DNS projection; `EKSServiceDeps` and capability interfaces (`:39-245`). |
|
||||
| `service_nats.go` | 190 | temporary residue | Gateway-side NATS client implementing `EKSService` over `eks.<Action>` subjects; removal condition as for `service.go`. |
|
||||
| `state_report.go` | 111 | guest/controller wire contract | `eks.state.*` and `eks.addon.*` subjects and payloads published by the CP VM. |
|
||||
| `store.go` | 255 | resource owner (`cluster`), mixed | Bucket names, leader bucket, migrations, key helpers for every resource; split per-resource key helpers into each owner; `AccountWatchBuckets` (`:215`) is effectively a projection consumed by DNS. |
|
||||
| `token_review.go` | 128 | resource owner (`access`) | Authenticates a `get-token` bearer against STS and maps to AccessEntry groups; `ResolveTokenReview` is called from the gateway process and opens EKS KV there. |
|
||||
| `token_webhook.go` | 57 | guest/controller wire contract | `eks.VerifyToken` subject and request/response consumed by `runtime/roles/awsgw`. |
|
||||
| `types.go` | 131 | mixed: `nodegroup` + `access` + residue | `NodegroupRecord` (nodegroup), `AccessEntryRecord`/`AccessScope` (access); `ClusterRecord` (`:19`) and `OIDCProviderConfigRecord` (`:125`) have no production readers or writers (temporary residue; removal: delete). |
|
||||
| `volume_reclaim.go` | 81 | resource owner (`cluster`) | Surfaces (never deletes) CSI-created EBS volumes at cluster delete. |
|
||||
|
||||
Additional dead code observed in `store.go`: `OIDCProviderKey` (`:95`) and `EventKey` (`:132`) have no production callers.
|
||||
|
||||
### 1.2 `gateway/eks.go` and `gateway/eks/` (15 files, 1,473 lines)
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `gateway/eks.go` | 379 | AWS protocol adapter | REST-JSON route table (`:27-219`), dispatch, authorization ordering, `iam:PassRole` check (`:319-347`), caller identity extraction. |
|
||||
| `gateway/eks/access.go` | 89 | AWS protocol adapter | Unmarshal and NATS call for access actions. |
|
||||
| `gateway/eks/addons.go` | 60 | AWS protocol adapter | Same for add-on actions. |
|
||||
| `gateway/eks/authz.go` | 310 | AWS protocol adapter | Per-action resource ARN derivation (`eksScopes` `:45-109`); imports `handlers_eks.PrincipalARNHash`. |
|
||||
| `gateway/eks/cluster.go` | 66 | AWS protocol adapter | Cluster action wrappers. |
|
||||
| `gateway/eks/handler.go` | 47 | AWS protocol adapter | JSON response/error writers. |
|
||||
| `gateway/eks/internal_addons.go` | 38 | guest/controller wire contract | `GET /clusters/{c}/internal-addons/{acct}` for the addon-sync agent. |
|
||||
| `gateway/eks/internal_authz.go` | 129 | AWS protocol adapter + temporary residue | Principal gate for internal routes; `lookupClusterMeta` (`:102-129`) reads `eks-account-*` and `ClusterMeta` directly from the gateway process; removal: owner-published membership query. |
|
||||
| `gateway/eks/internal_publish.go` | 91 | guest/controller wire contract | HTTP-to-NATS relay of bootstrap/state/addon channels. |
|
||||
| `gateway/eks/internal_recovery.go` | 37 | guest/controller wire contract | `GET .../internal-recovery/{acct}/{instance}`. |
|
||||
| `gateway/eks/nodegroup.go` | 65 | AWS protocol adapter | Nodegroup wrappers. |
|
||||
| `gateway/eks/oidc.go` | 45 | AWS protocol adapter | IdP-config wrappers (handlers are stubs). |
|
||||
| `gateway/eks/passrole.go` | 31 | AWS protocol adapter | Extracts passed role ARNs for CreateCluster/CreateNodegroup only. |
|
||||
| `gateway/eks/tags.go` | 35 | AWS protocol adapter | Tag wrappers. |
|
||||
| `gateway/eks/token_review.go` | 51 | guest/controller wire contract | `POST /clusters/{c}/token-review` for the in-VM token webhook. |
|
||||
|
||||
### 1.3 Other production files wiring or reading EKS
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `daemon/eks_deps.go` | 220 | cross-resource orchestration (composition-root binding) | Builds `EKSServiceDeps`, binding VPC, ELBv2, image, EIP, IGW, NAT-GW, route-table, placement-group, IAM, EBS and object-store owners. |
|
||||
| `daemon/eks_adapter.go` | 58 | cross-resource orchestration (composition-root binding) | `SubnetVPCResolver` over the VPC service. |
|
||||
| `daemon/eks_cp_control.go` | 67 | temporary residue | CP describe/start/stop calls `domains/ec2/awsapi/instance.{DescribeInstances,StartInstances,StopInstances}`, i.e. an EC2 AWS adapter used as an internal API (ADR-0004 S3/S4); removal: EC2-owned instance-control capability. |
|
||||
| `daemon/eks_worker_launch.go` | 175 | cross-resource orchestration (composition-root binding) | `WorkerLauncher` via `instanceService.RunInstances`, `ec2.RunInstances.{type}.{node}` and `contracts/ec2/v1` instance commands. |
|
||||
| `daemon/daemon.go` (EKS parts: `:75`, `:167`, `:1085-1128`, `:2018-2019`, `:2175`, `:2302-2304`, `:3017-3018`) | n/a | temporary residue | NATS subscription table for 38 `eks.*` subjects, service init, reconciler spawn, reaper registration, shutdown; comment at `:1085-1087` says every handler returns NotImplemented, which is stale. |
|
||||
| `daemon/dns_reconcile.go` (`:27`, `:64-68`, `:97-98`) | n/a | temporary residue | Watches every `eks-account-*` bucket via `AccountWatchBuckets` and pulls `DesiredDNSChanges`; removal: EKS-published DNS projection. |
|
||||
| `gateway/oidc_discovery.go` | 117 | temporary residue | Public OIDC discovery/JWKS endpoints read `eks-account-*`, `GetClusterMeta` and `OIDCJWKSKey` directly; removal: cluster-owned OIDC projection. |
|
||||
| `handlers/sts/oidc_jwks.go` | 126 | temporary residue (STS-owned file) | `FetchClusterJWKS` reads `eks-account-*`/`OIDCJWKSKey` (`:101-110`); ADR-0001 step 7 names this STS-to-EKS import for removal via an identity-owned contract or EKS projection. |
|
||||
| `runtime/roles/awsgw/eks_token_verify.go` | 75 | guest/controller wire contract (server) | Queue-subscribes `eks.VerifyToken` (`:30`) and verifies presigned STS URLs. |
|
||||
| `accountteardown/reapers_eks.go` | 138 | cross-resource orchestration | Account teardown deletes nodegroups then clusters through the `EKSService` NATS client. |
|
||||
| `operator/cli/admin_eks.go` | 77 | guest/controller wire contract (operator) | `spx admin eks restore-snapshot` over `eks.RestoreSnapshot`. |
|
||||
| `gateway/gateway.go` (`:300`, `:345-346`, `:403`, `:455`, `:513`), `gateway/auth.go` (`:278-279`), `gateway/operations.go` (`:34-36`) | n/a | AWS protocol adapter (legacy registration residue) | Legacy dispatch, OIDC public routes and inventory; removal per ADR-0002 when `domains/eks/awsapi` registers. |
|
||||
| `bootstrap/config/config.go:44` | n/a | AWS protocol adapter (config) | `eks` listed in `AWSGWServiceNames`. |
|
||||
|
||||
Guest counterparties (not inventoried individually): `agents/eks/{gatewaypublish,gatewayfetch,tokenwebhook,credentialprovider,konnectivitycert,webhookcert}` and `scripts/images/eks-node/*`.
|
||||
`agents/eks/tokenwebhook/webhook.go` imports `handlers_eks.WebhookTokenReviewResult`, so a guest binary imports the domain monolith (ADR-0004 S5 debt).
|
||||
|
||||
## 2. Served AWS actions and protocol
|
||||
|
||||
Protocol: REST-JSON, method+path routed by `rest.NewRouter("eks", eksRoutes)` (`gateway/eks.go:234`); unknown routes return `InvalidAction` (`:241`).
|
||||
Authorization order (`gateway/eks.go:237-312`): route lookup, account from auth context, internal-route principal gate, body read, query folding, `ResourceARNs`, `checkPolicyResources`, `checkEKSPassRole`, handler over NATS.
|
||||
Gateway-to-daemon transport: NATS request/reply `eks.<Action>`, queue group `spinifex-workers` (`daemon/daemon.go:1090-1127`).
|
||||
`gateway/operations.go:34-36` declares EKS as `Registered: eksActionNames()` with no `Stubbed` or `Unsupported` list, so stubs and internal routes are indistinguishable in the inventory.
|
||||
|
||||
| Action | Route | Status | Owner |
|
||||
|---|---|---|---|
|
||||
| CreateCluster | POST /clusters | implemented (async) | cluster |
|
||||
| ListClusters | GET /clusters | implemented | cluster |
|
||||
| DescribeCluster | GET /clusters/{c} | implemented | cluster |
|
||||
| DeleteCluster | DELETE /clusters/{c} | implemented (synchronous purge) | cluster |
|
||||
| UpdateClusterConfig | POST /clusters/{c}/update-config | stub NotImplemented (`service_impl.go:1455`) | cluster |
|
||||
| UpdateClusterVersion | POST /clusters/{c}/updates | stub NotImplemented (`service_impl.go:1459`) | cluster |
|
||||
| ListUpdates | GET /clusters/{c}/updates | stub at gateway (`gateway/eks.go:49-52`) | cluster (update records do not exist) |
|
||||
| DescribeUpdate | GET /clusters/{c}/updates/{id} | stub at gateway (`gateway/eks.go:53-56`) | cluster/nodegroup/addon |
|
||||
| CreateNodegroup | POST /clusters/{c}/node-groups | implemented (async) | nodegroup |
|
||||
| ListNodegroups | GET /clusters/{c}/node-groups | implemented | nodegroup |
|
||||
| DescribeNodegroup | GET .../node-groups/{n} | implemented | nodegroup |
|
||||
| UpdateNodegroupConfig | POST .../node-groups/{n}/update-config | implemented (synchronous, returns `Successful`) | nodegroup |
|
||||
| UpdateNodegroupVersion | POST .../node-groups/{n}/update-version | stub NotImplemented (`nodegroup.go:1120-1127`) | nodegroup |
|
||||
| DeleteNodegroup | DELETE .../node-groups/{n} | implemented (synchronous) | nodegroup |
|
||||
| CreateAccessEntry | POST /clusters/{c}/access-entries | implemented (STANDARD type only, `service_impl.go:1561-1564`) | access |
|
||||
| ListAccessEntries | GET /clusters/{c}/access-entries | implemented | access |
|
||||
| DescribeAccessEntry | GET .../access-entries/{p} | implemented | access |
|
||||
| UpdateAccessEntry | POST .../access-entries/{p} | implemented | access |
|
||||
| DeleteAccessEntry | DELETE .../access-entries/{p} | implemented | access |
|
||||
| AssociateAccessPolicy | POST .../access-entries/{p}/access-policies | implemented (cluster scope only, `service_impl.go:1692`) | access |
|
||||
| DisassociateAccessPolicy | DELETE .../access-policies/{policy} | implemented | access |
|
||||
| ListAssociatedAccessPolicies | GET .../access-entries/{p}/access-policies | implemented | access |
|
||||
| ListAccessPolicies | GET /access-policies | implemented (static catalogue) | access |
|
||||
| DescribeAddonVersions | GET /addons/supported-versions | implemented (static catalogue) | addon |
|
||||
| ListAddons | GET /clusters/{c}/addons | implemented | addon |
|
||||
| CreateAddon | POST /clusters/{c}/addons | implemented (async via guest) | addon |
|
||||
| DescribeAddon | GET .../addons/{a} | implemented | addon |
|
||||
| UpdateAddon | POST .../addons/{a}/update | implemented | addon |
|
||||
| DeleteAddon | DELETE .../addons/{a} | implemented | addon |
|
||||
| AssociateIdentityProviderConfig | POST .../identity-provider-configs/associate | stub NotImplemented (`service_impl.go:1818`) | cluster (no IdP resource) |
|
||||
| DescribeIdentityProviderConfig | POST .../identity-provider-configs/describe | stub | cluster |
|
||||
| DisassociateIdentityProviderConfig | POST .../identity-provider-configs/disassociate | stub | cluster |
|
||||
| ListIdentityProviderConfigs | GET .../identity-provider-configs | stub | cluster |
|
||||
| TagResource / UntagResource / ListTagsForResource | POST/DELETE/GET /tags/* | implemented for cluster and nodegroup ARNs; NotImplemented for addon/access-entry ARNs (`service_impl.go:1882,1925,1958`) | cluster, nodegroup |
|
||||
| PublishInternal | POST /clusters/{c}/internal-publish | internal (not AWS) | cluster guest contract |
|
||||
| WebhookTokenReview | POST /clusters/{c}/token-review | internal (not AWS) | access guest contract |
|
||||
| ListInternalAddons | GET /clusters/{c}/internal-addons/{acct} | internal (not AWS) | addon guest contract |
|
||||
| GetRecoveryDirective | GET /clusters/{c}/internal-recovery/{acct}/{id} | internal (not AWS) | cluster guest contract |
|
||||
|
||||
Internal NATS-only methods (no HTTP route): `SetRecoveryDirective`, `RestoreSnapshot`, `ListStagedAddonManifests` (backs ListInternalAddons).
|
||||
Unregistered AWS EKS operations (InvalidAction today) include Fargate profiles, pod identity associations, encryption config, insights, `RegisterCluster`/`DeregisterCluster`, `DescribeAddonConfiguration`, `DescribeClusterVersions` and EKS Anywhere subscriptions (list from memory of the EKS API, not verified against the pinned model).
|
||||
`iam:PassRole` is enforced for CreateCluster `roleArn` and CreateNodegroup `nodeRole` only (`gateway/eks/passrole.go`); CreateAddon/UpdateAddon `serviceAccountRoleArn` is not PassRole-checked (AWS requires `iam:PassRole` for it; verify before treating as a divergence).
|
||||
|
||||
## 3. Durable records and KV buckets
|
||||
|
||||
| Bucket | Key | Record | Owner | Writers | Readers outside EKS |
|
||||
|---|---|---|---|---|---|
|
||||
| `eks-account-{acct}` (history 1, migration v1, `store.go:30-38,235-245`) | `clusters/{c}/meta` | `ClusterMeta` (`cluster_state.go:65`) | cluster | `service_impl.go` create/launch/delete/tags, `cluster_state.go` CAS helpers, `cluster_reconciler.go`, `restore_snapshot.go:150`, `nats_bootstrap.go:297` (CA) | **`gateway/eks/internal_authz.go:110-124`**, **`gateway/oidc_discovery.go:38,60`** |
|
||||
| same | `clusters/{c}/nodegroups/{ng}` | `NodegroupRecord` (`types.go:33`) | nodegroup | `nodegroup.go` (claim, CAS, put, delete), `service_impl.go:1964` (tags) | none observed |
|
||||
| same | `clusters/{c}/access-entries/{sha256(principalARN)}` | `AccessEntryRecord` (`types.go:93`) | access | `service_impl.go:1548-1785`, creator-admin seed `service_impl.go:994-1001` (cluster launch) | **gateway process via `ResolveTokenReview` (`token_review.go:113`)** |
|
||||
| same | `clusters/{c}/addons/{a}` | `AddonRecord` (`addon_state.go:29`) | addon | `addons.go`, `addon_status_reconciler.go`, `nodegroup.go:410` (GPU add-on) | none observed |
|
||||
| same | `clusters/{c}/addons/{a}/manifest` | staged manifest descriptor | addon | `addon_installer.go:80-92` | guest via internal-addons route |
|
||||
| same | `clusters/{c}/oidc-signing-key.pem.enc` | encrypted ECDSA key | cluster | `oidc_keypair.go:86`, zeroized `service_impl.go:1281` | none |
|
||||
| same | `clusters/{c}/oidc-jwks.json` | JWKS | cluster | `oidc_keypair.go:94` | **`handlers/sts/oidc_jwks.go:110`**, **`gateway/oidc_discovery.go:93`** |
|
||||
| same | `clusters/{c}/oidc-jwks-verified` | marker | cluster | `nats_bootstrap.go:291` | none |
|
||||
| same | `clusters/{c}/admin-kubeconfig.enc` | encrypted kubeconfig | cluster | `nats_bootstrap.go:269` | none observed in Go (uncertain whether a CLI reads it) |
|
||||
| same | `clusters/{c}/k3s-node-token.enc` | encrypted join token | cluster | `nats_bootstrap.go:251` | read by `nodegroup.go:1164` (cross-resource) |
|
||||
| same | `clusters/{c}/recovery/{instanceId}` | `RecoveryDirective` | cluster | `recovery_directive.go:102`, reconciler | guest via internal-recovery route |
|
||||
| same (whole bucket) | all keys | watch | cluster | n/a | **`daemon/dns_reconcile.go:68`** via `AccountWatchBuckets` |
|
||||
| `spinifex-eks-leader` (TTL 60s, `store.go:40-42,249`) | `{acct}/{c}` reconciler lease; `.../teardown` lease | kvlease | cluster | `cluster_reconciler.go:391-403`, `service_impl.go:1174-1197` | none |
|
||||
| `spinifex-eks-clustertokens` (TTL 15m, `clienttoken.go:16-21`) | idempotency store keyed by account+token | `idempotency.Store[string]` | cluster | `service_impl.go:529-558,612-619` | none |
|
||||
|
||||
Dead key helpers: `OIDCProviderKey` (`store.go:95`), `EventKey` (`store.go:132`).
|
||||
Secret material in `ClusterMeta`: `ControlPlaneTemplate` is a copy of `K3sServerInput` that clears creds but not `OIDCPrivateKeyPEM` or `JoinToken` (`service_impl.go:945-959`; fields at `k3s_server_vm.go:146,171`), so the OIDC private key and k3s join token appear to be persisted unencrypted in the meta record that the gateway also reads; no scrubbing or custom marshaller was found (verify with a KV dump before acting).
|
||||
`ClusterMeta` combines desired intent (`Version`, `RoleArn`, `ResourcesVpcConfig`, `Logging`, `Tags`), realization refs (CP nodes, NLB, CP VPC, EIP, ENIs), observations (`HealthIssue`, `NodeCount`, `NodegroupNodeCounts`, `CertificateAuthorityB64`) and delete bookkeeping (`DeletingSince`, `DeleteReapAttempts`).
|
||||
Hidden realizations outside EKS stores: CP VMs (system account, `ManagedBy=eks`), CP VPC/subnets/NAT-GW/route tables/IGW, NLB and target groups (ELBv2), SGs, private-endpoint ENI, egress EIP, HA placement group, worker instances (customer account), IAM roles/instance profiles, CSI EBS volumes.
|
||||
|
||||
## 4. Identity and scope
|
||||
|
||||
| Resource | Identifier / ARN | Account | Region | AZ |
|
||||
|---|---|---|---|---|
|
||||
| cluster | name; `arn:aws:eks:{region}:{acct}:cluster/{name}` (`foundation/aws/arn/eks.go:21`) | tenant account owns the record; CP VMs, CP VPC and NLB are realized in the system account `awsidentifiers.GlobalAccountID` (`service_impl.go:738`, `:2240-2248`); customer-VPC SGs and the private-endpoint ENI in the tenant account | single Region from `deps.Region`; not stored as a field | Regional; HA CP spread over hosts by AZ (`k3s_ha_control_plane.go:528`) |
|
||||
| nodegroup | `arn:...:nodegroup/{c}/{ng}/{discriminator}` where the discriminator is derived from (account, cluster, name) (`nodegroup.go:333`), so a recreated nodegroup gets the same ARN, unlike AWS's per-incarnation UUID | tenant | deps.Region | AZs implied by subnets |
|
||||
| addon | `arn:...:addon/{c}/{a}` (`arn/eks.go:51`, `addons.go:100`); no per-incarnation suffix (AWS appends one; verify) | tenant | deps.Region | n/a |
|
||||
| access | `arn:...:access-entry/{c}/{sha256(principal)}` (`access_entry.go:244`); AWS's shape differs (verify) | tenant | deps.Region | n/a |
|
||||
|
||||
Uniqueness is per account bucket and cluster-name prefix; there is no immutable internal UID distinct from the name for any EKS resource.
|
||||
|
||||
## 5. Lifecycle paths per resource
|
||||
|
||||
### cluster
|
||||
Create: `CreateCluster` `service_impl.go:504`; validates, optional token claim (`:529-558`), resolves VPC, builds `ClusterMeta{Status: CREATING}`, `claimClusterName` CAS-create with FAILED-to-CREATING reclaim (`:1200-1265`), finalizes token, then launches `launchClusterInfra` on an in-process goroutine with `context.Background()` (`:625-645`).
|
||||
Launch stages (`:718-1012`): CP VPC, SGs, CP ingress, customer SGs, private ENI, NLB, OIDC keypair, join token, CP placement/launch, persist CP refs, NLB target registration, persist final meta, creator-admin AccessEntry, `spawnBootstrap`, `spawnReconciler`; any failure calls `failClusterLaunch` (`:696`) which marks FAILED.
|
||||
Readiness: `ClusterReconciler.reconcileOnce` (`cluster_reconciler.go:594`) flips CREATING to ACTIVE when bootstrap artifacts exist (`:1173-1189`) and the CP state report is healthy; CREATING becomes FAILED after `createTimeout` (`:1151-1171`).
|
||||
Reconciliation/repair (ACTIVE): health recording, CP restart (`:708`), etcd reset via recovery directives (`:780`), member replacement (`:1010`), all under the per-cluster lease `spinifex-eks-leader/{acct}/{c}` (`:391-418`).
|
||||
Restart: `SpawnRegisteredReconcilers` (`service_impl.go:342-425`) rescans buckets, reclaims orphaned nodegroups, resubscribes missing bootstrap kinds and respawns reconcilers; it does not resume an interrupted `launchClusterInfra`.
|
||||
Delete: `DeleteCluster` (`service_impl.go:1104-1155`) returns success for an absent cluster, takes the teardown lease (losers return DELETING), sets DELETING, runs `purgeClusterInfra` synchronously in the request (`:1268-1452`), and only erases `clusters/{c}/` after all blocking steps succeed; `EKSDeletingReaper` re-drives with backoff and exhaustion (`eks_deleting_reaper.go:46-159`); `EKSBillableReaper` terminates CP VMs whose meta is gone (`eks_billable_reaper.go:46-100`).
|
||||
Fencing: leases only; no desired/observed generation (no `generation` field anywhere in `handlers/eks`).
|
||||
Idempotency: `clientRequestToken` honoured for CreateCluster only, 15-minute TTL, replays current meta by name.
|
||||
|
||||
### nodegroup
|
||||
Create: `createNodegroup` (`nodegroup.go:205`) requires cluster ACTIVE (`:226-229`), `ClaimNodegroupRecord` CAS-create (`:112-126`, ResourceInUse at `:363`), then background `launchNodegroupInfra` (`:373`, `:449-535`).
|
||||
Readiness: `waitWorkersReady` (`:537-563`) polls the cluster's `NodegroupNodeCounts` observation until ready count reaches baseline plus desired, then sets ACTIVE (`:521-523`), else CREATE_FAILED.
|
||||
Update: `UpdateNodegroupConfig` runs `reconcileNodegroup` inline under revision CAS (`:914-979`, `launchWorkersCAS` `:985`, `recordLaunchedWorker` `:1038`) and returns `Update{Status: Successful}` (`:900-906`).
|
||||
Restart: `reclaimOrphanedNodegroups` (`:1189-1228`) terminates workers of CREATING/partial CREATE_FAILED records and marks them CREATE_FAILED; it does not resume.
|
||||
Delete: `deleteNodegroup` (`:1129-1158`) terminates workers then deletes the record synchronously and returns a DELETING projection of a record that no longer exists.
|
||||
Fencing: KV revision CAS; no generation, no lease.
|
||||
Idempotency: no `clientRequestToken`; duplicate create loses the claim and gets ResourceInUse.
|
||||
|
||||
### addon
|
||||
Create: `CreateAddon` (`addons.go:65-111`) validates against the catalogue, get-then-put (`:88-105`, not CAS-create), stages manifest (`addon_installer.go:62-85`), returns CREATING.
|
||||
Readiness: guest addon-sync agent publishes `eks.addon.{acct}.{c}.status` via internal-publish; `applyAddonStatusReport` (`addon_status_reconciler.go:15-47`) maps `ready` to ACTIVE and `failed` to CREATE_FAILED/DEGRADED; `report.Version` is ignored.
|
||||
Update: `UpdateAddon` (`addons.go:203-252`) CAS sets UPDATING, restages, returns `Successful` with `Id = rec.Arn`.
|
||||
Delete: `DeleteAddon` (`addons.go:254-285`) unstages and deletes the record immediately; the DELETING status is only in the response.
|
||||
Fencing: revision CAS on the record; no generation.
|
||||
Idempotency: none (`clientRequestToken` ignored).
|
||||
|
||||
### access
|
||||
Create: `CreateAccessEntry` (`service_impl.go:1548-1581`) get-then-put (`:1570-1578`), synchronous; creator-admin entry seeded during cluster launch (`:994-1001`).
|
||||
Mutations: `casUpdateAccessEntry` (`access_entry.go:141`) for update/associate/disassociate; delete via `DeleteAccessEntryRecord`.
|
||||
Effect: read at authentication time by `Authenticate`/`ResolveTokenReview` (`token_review.go:31-128`); no in-cluster realization or readiness.
|
||||
Delete with cluster: swept by `DeleteClusterPrefix` (`cluster_state.go:493`).
|
||||
Idempotency: none (`clientRequestToken` ignored).
|
||||
|
||||
## 6. NATS subjects and guest/controller messages
|
||||
|
||||
| Subject / route | Direction | Delivery | Defined | Consumer |
|
||||
|---|---|---|---|---|
|
||||
| `eks.<Action>` (38 subjects) | gateway (or teardown/CLI) to daemon | core NATS request/reply, queue `spinifex-workers` | `service_nats.go:30-189`, `daemon/daemon.go:1090-1127` | `EKSServiceImpl` |
|
||||
| `eks.bus.{acct}.{c}.{k3s-bootstrap-token,k3s-admin-kubeconfig,k3s-oidc-jwks,k3s-ca}` | CP VM via gateway relay to daemon | core NATS publish/subscribe, at-most-once | `nats_bootstrap.go:47-57`, subscribe `:197` | `NATSBootstrap` persists to KV with `persistWithRetry` (`:32`) |
|
||||
| `eks.state.{acct}.{c}.server` | CP VM via relay | core NATS, periodic | `state_report.go:13` | `ClusterReconciler` (`cluster_reconciler.go:432`) |
|
||||
| `eks.addon.{acct}.{c}.status` | CP VM via relay | core NATS | `state_report.go:73` | `ClusterReconciler` (`:448`) |
|
||||
| `eks.VerifyToken` | EKS (gateway process) to awsgw STS | request/reply, queue `awsgw-eks-token-verify` | `token_webhook.go:13` | `runtime/roles/awsgw/eks_token_verify.go:30` |
|
||||
| `contracts/cluster/v1` `NodeStatusSubject`, `NodeVMsSubject` | EKS to every node | fan-out request | `k3s_ha_control_plane.go:502,579` | daemons |
|
||||
| `ec2.RunInstances.{type}.{node}` and `contracts/ec2/v1` instance commands | EKS (daemon binding) to EC2 | request/reply | `daemon/eks_worker_launch.go` | EC2 instance owner |
|
||||
| HTTP `internal-publish`, `token-review`, `internal-addons`, `internal-recovery` | guest to gateway (SigV4, system CP role) | HTTP | `gateway/eks.go:59-87` | gateway relays/queries |
|
||||
| HTTP `/oidc/eks/{region}/{acct}/{c}/.well-known/openid-configuration`, `/keys` | public | HTTP, unauthenticated | `gateway/gateway.go:345-346` | `gateway/oidc_discovery.go` |
|
||||
|
||||
Retry and recovery paths: guest `eks-gateway-publish` retries until HTTP 2xx (`agents/eks/gatewaypublish/publish.go:26-31,97-114`), but the gateway returns 2xx after a core-NATS publish even with no subscriber (`gateway/eks/internal_publish.go:84-89`), so a bootstrap message published while the daemon subscriber is absent is lost.
|
||||
On restart, `BootstrapPendingKinds` resubscribes only missing kinds (`nats_bootstrap.go:115-139`, `service_impl.go:410-416`); a lost one-shot message is not replayed, so the cluster reaches FAILED at the create timeout.
|
||||
A bootstrap subscriber that exits with error marks the cluster FAILED (`service_impl.go:2192-2212`).
|
||||
SDK retries of `eks.<Action>` land on any node of the queue group; dedup relies on token store, claim CAS, teardown lease or nothing (addon/access create).
|
||||
|
||||
## 7. Boundary-crossing imports
|
||||
|
||||
### Outbound from `handlers/eks`
|
||||
- IAM/identity: `handlers/iam` `EncryptSecret`/`DecryptSecret` (`nats_bootstrap.go`, `oidc_keypair.go`, `nodegroup.go`), `EnsureSystemInstanceProfile` (`k3s_server_role.go`), `SystemInstanceRoleEnsurer` (`service_impl.go`); `instanceProfileEnsurer` capability with `GetRole/CreateRole/PutRolePolicy/CreateInstanceProfile/AddRoleToInstanceProfile` (`service_impl.go:214`) means EKS creates IAM roles and instance profiles.
|
||||
- Bluebottle: `bluebottle/pkg/auth` `ResolveRoleARN` (`nodegroup.go`) and `ParseRoleARN` (`service_impl.go`); daemon binding uses `bluebottle/pkg/masterkey`; gateway uses `bluebottle/pkg/auth` (`gateway/eks.go`).
|
||||
- STS: token verification over `eks.VerifyToken` (process contract, no import).
|
||||
- EC2 instance: `domains/ec2/systeminstance` (`SystemInstanceInput/Output`, `ExtraENIInput`, `BootAMI`), `domains/ec2/instancetypes`, `domains/ec2/placementgroup` (alias `ec2placementgroup`), `domains/ec2/ebs/metadata`, `runtime/compute/vm` (reapers), plus `WorkerLauncher`, `k3sInstanceLauncher`, `k3sAMIResolver`, `eipProvisioner`, `csiVolumeReclaimer` capabilities.
|
||||
- Network: `domains/network/systemvpc` (`Ensure`, `Delete`, `EnsureIGW`, `DeleteIGW`, provisioner interfaces); `sgProvisioner` and `k3sVPCProvisioner` capabilities (EC2 SDK shapes).
|
||||
- DNS: `domains/dns` `EKSName`, `EKSChanges`, `ResolveBaseDomain`, `Change`, `ActionUpsert` (`service_impl.go`).
|
||||
- ACM/CA: no import; the platform/NATS CA PEM is passed as `GatewayCACert` from `d.config.NATS.CACert` (`daemon/eks_deps.go:64-71`) and baked into CP user data; the k3s cluster CA is self-generated in the guest and returned on `k3s-ca`.
|
||||
- ELBv2: `nlbProvisioner` capability with ELBv2 SDK shapes (`nlb.go:19`), bound to `d.elbv2Service`.
|
||||
- Storage/object: `providers/objectstore` (etcd snapshots), Predastore system creds via deps.
|
||||
- Contracts: `contracts/cluster/v1`.
|
||||
- Foundation: `kvutil`, `kvstore`, `kvlease`, `migrate`, `idempotency`, `reconciler`, `telemetry`, `aws/{arn,errors,tags,identifiers,ami}`, `messaging/nats`.
|
||||
- Bootstrap: `bootstrap/config.Config` in `EKSServiceDeps`.
|
||||
|
||||
### Inbound to `handlers/eks`
|
||||
- `gateway/eks/*`: `NewNATSEKSService`, input/output types, `PrincipalARNHash`, `ClusterMeta`, `ClusterMetaKey`, `AccountBucketName`, `CPInstanceRoleName`, `ResolveTokenReview`, bootstrap/state/addon subject builders.
|
||||
- `gateway/oidc_discovery.go`: `AccountBucketName`, `GetClusterMeta`, `OIDCJWKSKey`.
|
||||
- `handlers/sts/oidc_jwks.go`: `AccountBucketName`, `OIDCJWKSKey` (STS reads EKS private KV).
|
||||
- `runtime/roles/awsgw/eks_token_verify.go`: `TokenVerifySubject`, `TokenVerifyRequest`, `TokenVerifyResponse`.
|
||||
- `agents/eks/tokenwebhook/webhook.go`: `WebhookTokenReviewResult` (guest imports domain).
|
||||
- `daemon/*`: `EKSServiceImpl`, `NewEKSServiceImpl`, `EKSServiceDeps`, `NewNATSHostScheduler`, `SubnetVPCResolver`, `WorkerLauncher`, `AccountWatchBuckets`.
|
||||
- `accountteardown/reapers_eks.go`: `EKSService`, `NewNATSEKSService`.
|
||||
- `operator/cli/admin_eks.go`: `NewNATSEKSService`, `RestoreSnapshotInput`.
|
||||
|
||||
### Reverse knowledge of EKS in other domains (comments or tag conventions, no import)
|
||||
`domains/ec2/instance/service_impl.go:394-396` (worker tag comment), `domains/ec2/systeminstance/launcher.go:4,25`, `handlers/elbv2/{launcher.go:5,service_impl.go:1232-1250,service_impl_nlb_sg.go:196}`, `domains/ec2/vpc/eni.go:54`, `domains/ec2/guestmetadata/resolver.go:37`, `domains/dns/{naming.go:118-124,reconcile.go:58,303}`, `handlers/iam/managed_policies.go` (EKS managed policies), `accountteardown/reapers_iam.go:26-33` (OIDC providers).
|
||||
|
||||
## 8. Known gaps against Q-78 / ADR-0003 (observed)
|
||||
|
||||
1. No desired or observed generation for any EKS resource; fencing is leases (cluster) or record-revision CAS (nodegroup, addon, access).
|
||||
2. Intent and observation share `ClusterMeta` (`cluster_state.go:65-190`), and nodegroup readiness reads a cluster observation (`nodegroup.go:537-549`).
|
||||
3. Cluster realization runs in request-spawned goroutines (`service_impl.go:625-645`); restart does not resume the launch, so an interrupted create ends FAILED after timeout (`cluster_reconciler.go:1151-1171`) rather than converging the same intent.
|
||||
4. CP VMs are live before their IDs are persisted (`service_impl.go:960-970` comment); a crash in that window leaves VMs reachable only by the billable reaper after meta removal.
|
||||
5. Bootstrap artifacts travel on at-most-once core NATS (`nats_bootstrap.go:197`, `gateway/eks/internal_publish.go:84`); a lost message is not replayed.
|
||||
6. DeleteCluster purges synchronously inside the request (`service_impl.go:1141-1154`) rather than accepting asynchronously as AWS does.
|
||||
7. DeleteCluster does not reject a cluster with nodegroups (AWS returns ResourceInUseException) and `DeleteClusterPrefix` erases nodegroup records while their worker instances keep running; `accountteardown/reapers_eks.go:15-18` documents this; workers carry no `ManagedBy` tag (`nodegroup.go:669-670`) and no reaper reclaims them (uncertain whether the e2e suite exercises this order).
|
||||
8. DeleteNodegroup is synchronous and removes the record immediately (`nodegroup.go:1145-1157`), so the resource is absent before EC2 termination completes; no durable deletion intent.
|
||||
9. UpdateNodegroupConfig and UpdateAddon return `Successful` synchronously (`nodegroup.go:900-906`, `addons.go:246-251`) and `DescribeUpdate`/`ListUpdates` are stubs, so there is no truthful pending update.
|
||||
10. CreateAddon and CreateAccessEntry use get-then-put (`addons.go:88-105`, `service_impl.go:1570-1578`), so concurrent creates can both succeed.
|
||||
11. Addon status reports are not fenced by version (`addon_status_reconciler.go:15-47` ignores `AddonStatusReport.Version`), so a stale report can flip UPDATING to ACTIVE.
|
||||
12. DeleteAddon deletes the record before in-cluster removal is observed (`addons.go:270-285`).
|
||||
13. `clientRequestToken` is honoured only by CreateCluster; CreateNodegroup, CreateAddon, CreateAccessEntry, UpdateNodegroupConfig, UpdateAddon ignore it.
|
||||
14. Nodegroup and addon ARNs are deterministic by name, so identifiers are reused across incarnations (Q-78 references Q-43 identifier-reuse rules).
|
||||
15. ADR-0004 S3 inventory: EKS has no `Stubbed`/`Unsupported` classification (`gateway/operations.go:34-36`) although nine actions return NotImplemented, and four internal routes are mixed into the AWS action inventory.
|
||||
16. ADR-0003 S4 / ADR-0004 S4: three non-owner readers of EKS private KV (`gateway/eks/internal_authz.go`, `gateway/oidc_discovery.go`, `handlers/sts/oidc_jwks.go`) plus `daemon/dns_reconcile.go` watching all EKS buckets.
|
||||
17. ADR-0004 S3: `daemon/eks_cp_control.go` uses the EC2 AWS adapter package as an internal API.
|
||||
18. ADR-0004 S5: guest `agents/eks/tokenwebhook` imports `handlers/eks`.
|
||||
19. Observed security risk outside Q-78: plaintext OIDC private key and k3s join token in `ClusterMeta.ControlPlaneTemplate` (Section 3).
|
||||
20. Comment hygiene: `eks_billable_reaper.go:3` cites an unaccepted "ADR-0006"; `daemon/daemon.go:1085-1087` is stale.
|
||||
|
||||
## 9. First-resource candidates
|
||||
|
||||
| Rank | Resource | Approx. size | Private state | Dependencies | Tests | Extractable without EKS identity/IAM/Bluebottle? |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | access | about 670 lines (`access_entry.go` 255, `service_impl.go:1527-1816` about 290, `token_review.go` 128, parts of `types.go`) | `clusters/{c}/access-entries/*` only | cluster existence query (`acctKVForCluster` reads `ClusterMeta`), cluster launch seeds creator admin, gateway authz needs `PrincipalARNHash`, gateway token-review calls `ResolveTokenReview`, STS verify contract | `access_entry_test.go`, `service_impl_test.go` (access cases), `token_review_test.go`, `token_webhook_test.go`, `gateway/eks/authz_test.go`, `gateway/eks/token_review_test.go`, e2e `TestEKS/AccessEntry`, `GetToken` | The record CRUD and policy catalogue are extractable now with a narrow cluster-exists query; the token-review part touches STS identity (`eks.VerifyToken` contract) but not IAM or Bluebottle, so it can stay behind its existing NATS contract. |
|
||||
| 2 | addon | about 830 lines (`addons.go`, `addon_state.go`, `addon_installer.go`, `addon_status_reconciler.go`, `addon_catalog.go`) | `clusters/{c}/addons/*` incl. manifests | cluster existence query, cluster reconciler relays `eks.addon.*` reports, guest internal-addons route, nodegroup GPU staging, IRSA role ARN stored only | `addons_test.go`, `addon_state_test.go`, `addon_status_reconciler_test.go`, `addon_catalog_test.go`, `gateway/eks/wrappers_test.go`, e2e `AddonDelivery` | Yes for IAM/Bluebottle; needs a report-relay seam out of `ClusterReconciler` and a guest contract for `StagedAddonManifest`/`AddonStatusReport`. |
|
||||
| 3 | nodegroup | about 1,650 lines (`nodegroup.go`, `nodegroup_userdata.go`, `claim.go`, part of `types.go`, nodegroup SG rules) | `clusters/{c}/nodegroups/*` | cluster meta (ACTIVE, CP ENI IP, VPC, readiness counts), cluster join token decrypt (`handlers_iam.DecryptSecret`), IAM role/profile creation, Bluebottle `ResolveRoleARN`, EC2 worker launch, AMI lookup, cluster SGs, GPU addon | `nodegroup_test.go` (39 tests), `nodegroup_recreate_test.go`, `nodegroup_instance_profile_test.go`, `nodegroup_userdata_test.go`, `daemon/eks_worker_launch_test.go`, e2e IRSA pod path | No; it needs IAM, Bluebottle and cluster secrets capabilities first. |
|
||||
| 4 | cluster | about 8,500 lines (remainder of `handlers/eks`) | meta, OIDC material, kubeconfig, join token, recovery directives, leader/token buckets | every external owner (EC2, VPC, ELBv2, DNS, IAM, objectstore, placement, scheduler) and all guest contracts | `create_cluster_test.go`, `delete_cluster_test.go`, `cluster_reconciler*_test.go` (5 files), `lifecycle_invariants_test.go` (RLC1-3), reapers, `nats_bootstrap_test.go`, `oidc_keypair_test.go`, `k3s_*_test.go`, `nlb_test.go`, `security_groups_test.go`, `cp_vpc_test.go`, `restore_snapshot_test.go`, e2e `TestEKS` | No; it is the orchestration hub and depends on identity, Bluebottle and the CA. |
|
||||
|
||||
Recommendation: extract `access` first, defining a cluster-owned "cluster exists and is not deleting" query and moving `PrincipalARNHash` and the key helper into the access package.
|
||||
Its shortcomings (gap 10, no client token, no generation) are small and can be recorded in its lifecycle contract without first resolving cluster realization.
|
||||
@@ -0,0 +1,310 @@
|
||||
# ADR-0004 source inventory — RDS
|
||||
|
||||
Code-read inventory of `refactor/spinifex-package-boundaries` at `6c50349bd`.
|
||||
It is not verified against a live system, and the branch is 59 commits behind `dev` at the time of writing, so recheck any file it cites before acting on it.
|
||||
Items marked "verify" or "uncertain" are unconfirmed.
|
||||
Gaps record observed behaviour, not target behaviour.
|
||||
|
||||
|
||||
Repo: `/home/tf-user/Development/mulga/spinifex`, branch `refactor/spinifex-package-boundaries`, paths relative to `spinifex/`.
|
||||
Inputs read: ADR-0001, ADR-0003, ADR-0004 and Q-78 from `origin/main`.
|
||||
Method: static reading of production code only; nothing here was verified against a live cluster.
|
||||
Line references are `file:line` at the time of reading.
|
||||
"Uncertain" marks a claim inferred from code shape rather than traced end to end.
|
||||
|
||||
## 0. Scope findings
|
||||
|
||||
`daemon/instance_records.go` (217) and `daemon/instance_records_migrate.go` (361) are **not RDS**.
|
||||
They own the EC2 per-instance record space `i.<id>` in the EC2 instance-state bucket (`daemon/instance_records.go:25`) and its migrations (`daemon/instance_records_migrate.go:29-70`), and import only `runtime/compute/vm`, `kvstore`, `kvutil`, `migrate`.
|
||||
They are excluded from the RDS inventory and belong to the EC2 instance owner.
|
||||
|
||||
Other production files that wire or consume RDS, found by import search:
|
||||
|
||||
- `daemon/daemon.go` (RDS sections only: `:170-171`, `:1171-1206`, `:2046-2078`, `:2306-2312`, `:2374-2382`).
|
||||
- `daemon/rds_deps.go`.
|
||||
- `daemon/dns_reconcile.go:71-76` and `:99-104` (RDS bucket watch and RDS DNS desired set).
|
||||
- `daemon/bedrock_deps.go:43` (Ochre/Bedrock borrows `handlers_rds.NewNATSVolumeAttacher`).
|
||||
- `handlers/quota/live.go:82-93` (`EnforceRDSInstances`).
|
||||
- `accountteardown/reapers_rds.go` (instance, subnet-group, parameter-group reapers).
|
||||
- `gateway/rds.go`, `gateway/operations.go:43-45`, `gateway/gateway.go:302,525-526` (service map and dispatch).
|
||||
- `bootstrap/config/config.go:338` (`RDSConfig`).
|
||||
- Test-support (non-`_test.go` but not production): `tests/integration/service_daemon_lite.go:51` reflects every `Service` method onto `rds.<Method>`; `tests/e2e/harness/rds*.go`.
|
||||
- Counterparty only: `agents/rds/agent/*.go` and `cmd/rds-agent` import `handlers/rds` for wire types.
|
||||
|
||||
## 1. File table
|
||||
|
||||
Total in-scope production lines: 16,468 (handlers/rds 14,873; gateway/rds 853; gateway/rds.go 95; daemon/rds_deps.go 98; daemon/instance_records* 578, not RDS).
|
||||
|
||||
### handlers/rds
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `store.go` | 451 | temporary residue (prerequisite: per-resource packages exist; removal: each resource owns its own key helpers and bucket view) | One file holds every resource's key space, bucket configs, migrations hook, leader bucket and cross-tenant enumeration (`:20-47`, `:39-50` key list). **Mixed**: split into instance (`db-instances/`, `bootstrap-payloads/`, `instance-index/`, leader bucket), snapshot (`db-snapshots/`, `backups/`, `retained-volumes/`), subnetgroup (`db-subnet-groups/`), parametergroup (`db-parameter-groups/`), events (`events/`), plus a domain-root bucket factory. |
|
||||
| `records.go` | 501 | temporary residue (same prerequisite) | All durable record structs in one file. **Mixed**: `DBInstanceRecord`/`PendingModifiedValues`/`ModifyLease`/`BootstrapState`/`AgentState`/`SnapshotOperation` → instance; `DBSnapshotRecord`/`AutomatedBackupRecord`/`RetainedVolumeRecord` → snapshot; `DBSubnetGroupRecord` → subnetgroup; `DBParameterGroupRecord`/`DBParameterRecord` → parametergroup; `Parameter`, `EngineHealth`, `Heartbeat*` → guest/controller wire contract. |
|
||||
| `service.go` | 314 | temporary residue (prerequisite: resource owners; removal: each owner holds its own deps, domain root holds composition) | Single `Service` with one `Deps` bag for every resource (`:19-45` in stripped view), per-account bucket cache and generic JSON CAS helpers. **Mixed**: `Deps` splits by resource; CAS helpers could move to `foundation/state/kvstore` (already exists) rather than a new common package. |
|
||||
| `service_nats.go` | 189 | temporary residue (prerequisite: awsapi calls owners in-process or via a domain contract; removal: no AWS-SDK-typed NATS client used as internal API) | AWS-SDK-shaped NATS client used by the gateway, quota (`handlers/quota/live.go:89`) and account teardown (`accountteardown/reapers_rds.go:19`). It is currently the de facto internal API, which ADR-0004 S3/S4 forbid for quota. |
|
||||
| `subjects.go` | 90 | **Mixed**: guest/controller wire contract (`rds.bus.*`, `CommandQueueGroup`, `:6-19`, `:65-90`) + temporary residue (Layer-1 `rds.<Action>` gateway→daemon subjects, `:21-63`) | Bus subjects are the real agent-path process boundary; the per-action subjects exist only because the gateway and daemon are separate dispatch hops. |
|
||||
| `arn.go` | 92 | AWS protocol adapter (with per-resource kinds) | `FormatARN`/`ParseARN`/`ResourceKind` (`:15-40`) are used for projection, tags and gateway authz scopes. Kinds could live with each resource; parsing belongs in awsapi. |
|
||||
| `filters.go` | 68 | AWS protocol adapter | Generic AWS `Filter` parsing (`ReadFilters`, `:38`), used by gateway catalogs and describes. |
|
||||
| `paging.go` | 74 | AWS protocol adapter | AWS `MaxRecords`/`Marker` paging (`:23`) and per-type page keys (`:52-74`). |
|
||||
| `validate.go` | 271 | resource owner (instance) — partly AWS protocol adapter | `CreateDBInstance` request validation (`:69`), identifier rules (`:196`), unimplemented-parameter rejection (`:225`). **Mixed**: SDK input parsing → awsapi; identifier/engine/class invariants → instance. |
|
||||
| `status.go` | 69 | resource owner (instance) | DB instance status enum and transition table (`:7-60`). |
|
||||
| `create.go` | 305 | resource owner (instance) | `CreateDBInstance` orchestration, record reservation, rollback (`:29-147`, `:195-305`). |
|
||||
| `describe.go` | 298 | **Mixed**: resource owner (instance) query + AWS protocol adapter (projection to `rds.DBInstance`, `:145-298`) | Projection is awsapi; record read/filter is the owner's query. |
|
||||
| `lifecycle.go` | 454 | resource owner (instance) — **contains an EC2 adapter** | Reboot/Stop/Start (`:57-162`), transition helpers (`:276-405`). `natsInstanceCommander` (`:407-454`) sends `ec2v1` instance commands and `ec2.start`; that is an EC2-owned capability implemented inside RDS (residue). |
|
||||
| `launch.go` | 629 | cross-resource orchestration (instance realization over EC2/VPC/EBS/IAM) — **contains an EC2 adapter** | `LaunchDBInstanceVM` (`:178-366`) composes system instance, two ENIs, data volume, AMI lookup. `natsVolumeAttacher` (`:560-629`) is an EC2 volume-attach NATS client that Bedrock also imports (`daemon/bedrock_deps.go:43`); it is EC2/EBS residue living in RDS. |
|
||||
| `network.go` | 200 | resource owner (instance) with cross-resource read of subnetgroup | Endpoint placement (`:36-75`) reads the subnet-group record directly (`:43`) and EC2 VPC/subnet/SG via `networkResolver` (`:16-20`). |
|
||||
| `vpc.go` | 163 | cross-resource orchestration (system VPC/SG realization for instances) | `EnsureSystemVPC`/`EnsureSystemSecurityGroup` (`:78`, `:104`) over `domains/network/systemvpc`. |
|
||||
| `storage.go` | 155 | resource owner (instance) | Storage grow via `volumeResizer` (EC2 `DescribeVolumes`/`ModifyVolume`, `:23-26`). |
|
||||
| `modify.go` | 528 | resource owner (instance) | `ModifyDBInstance` plan/immediate/pending paths (`:76-198`, `:199-528`). |
|
||||
| `modify_lease.go` | 173 | resource owner (instance) | Per-instance modify lease stored on the instance record (`:24-173`). |
|
||||
| `pending.go` | 256 | resource owner (instance) — with parametergroup orchestration | Applies `PendingModifiedValues` (`:28`), parameter-group apply to an instance (`:149`), filesystem grow finish (`:240`). `applyParameterGroup` is the instance side of parameter-group propagation. |
|
||||
| `replace.go` | 252 | resource owner (instance) — replacement | VM replace for class/storage change (`:43-166`), index rewrite (`:200`). |
|
||||
| `recovery.go` | 220 | resource owner (instance) — failure detection | Health classification (`:54`, `:76`); reports failure, does not repair (`:19-21`). |
|
||||
| `reconciler.go` | 896 | **Mixed**: resource owner (instance reconcile, `:238-660`, `:797-866`) + snapshot reconcile (`:674-796`) + foundation wiring (leader lease, `:83-188`) | Split: instance reconciler, snapshot creating-resolver, domain-root leader/loop. `describeInstanceState` (`:867-896`) is an EC2 projection adapter. |
|
||||
| `window.go` | 356 | resource owner (instance) — maintenance/backup windows | Pure window arithmetic for an instance's backup and maintenance windows. |
|
||||
| `backup.go` | 597 | **Mixed**: resource owner (instance: retention, windows, `runBackupWindow` `:206`, `runMaintenanceWindow` `:340`) + snapshot (automated snapshot identity `:385-394`) + AWS adapter (`DescribeDBInstanceAutomatedBackups` `:396-536`) | Automated backup is an instance obligation that produces snapshot records. |
|
||||
| `backup_sweep.go` | 340 | resource owner (snapshot) — retention reaper | `BackupRetentionReaper` (`:24-112`) deletes automated snapshots and reclaims retained volumes (`:310`); implements `vm.Reaper` from `runtime/compute/vm` (`:31`). |
|
||||
| `snapshot.go` | 768 | resource owner (snapshot) — tightly coupled to instance | Create/Describe/Delete (`:49`, `:224`, `:299`); create CASes the instance into `backing-up` (`:672-723`) and quiesces the engine. |
|
||||
| `restore.go` | 411 | cross-resource orchestration (snapshot → new instance) | `RestoreDBInstanceFromDBSnapshot` (`:24-162`) reads a snapshot record, creates a volume from it and runs the instance create path. |
|
||||
| `delete.go` | 548 | resource owner (instance) — with snapshot orchestration | `DeleteDBInstance` (`:34`), teardown (`:178-227`), final snapshot (`:247-447`), retained data volume (`:449-535`). |
|
||||
| `agent.go` | 667 | **Mixed**: guest/controller wire contract (IO types, `Command`, `CommandReply`, `InstanceIndexEntry`, `:30-160`, `:628-636`) + resource owner (instance agent-path handlers `:161-627`) | Split: `contracts/rds/v1` vocabulary vs instance owner handlers. |
|
||||
| `commands.go` | 239 | **Mixed**: guest/controller wire contract (command types and params, `:12-40`) + resource owner (instance: `issueCommand` issuer, `:86-230`) | Issuer is instance realization; constants are wire vocabulary. |
|
||||
| `bootstrap.go` | 280 | resource owner (instance) — bootstrap secret | Encrypted master-password payload (`:31-280`), encrypted via `handlers_iam.EncryptSecret` (`:105`, `:199`). |
|
||||
| `servingcert.go` | 109 | resource owner (instance) — realization helper | Per-instance serving cert minted from cluster CA (`:46`). Uncertain whether this should sit with a PKI owner instead. |
|
||||
| `userdata.go` | 68 | guest/controller wire contract | Agent user-data rendering (`:36`), the boot-time contract with the DB VM. |
|
||||
| `iam.go` | 83 | **Mixed**: resource owner (instance — instance profile ensure, `:56-83`) + guest/controller wire contract (`InternalAgentActions`, `:24-30`) | The internal-action list is shared by the gateway gate (`gateway/rds/authz.go:52-63`) and the role grant. |
|
||||
| `dns.go` | 72 | resource owner (instance) — publishes a projection | `DesiredDNSChanges` (`:25-51`) is an RDS-owned projection consumed by the daemon DNS reconcile. |
|
||||
| `events.go` | 312 | cross-resource orchestration (event journal) — uncertain final home | Per-source event rings for instance and snapshot (`:88-141`) plus `DescribeEvents` (`:159`). Not in the ADR-0004 resource table; candidate for the domain root, not a `common` package. |
|
||||
| `tags.go` | 280 | cross-resource orchestration + AWS adapter | One tag implementation dispatching on ARN kind over all four record types (`:51-73`, `:202-230`). Split: each owner exposes a tag mutation on its own record; awsapi routes by ARN. |
|
||||
| `engine.go` | 287 | resource owner (engine) — read-only catalogue | Engine registry, version/username/dbname/password rules (`:15-287`); consumed by the agent (`LookupEngine`). |
|
||||
| `engine_postgres.go` | 505 | resource owner (engine) | Postgres engine catalogue and parameter specs/validators. |
|
||||
| `engine_mariadb.go` | 503 | resource owner (engine) | MariaDB engine catalogue and parameter specs/validators. |
|
||||
| `paramcatalog.go` | 449 | resource owner (engine) | Parameter spec model, size-derived defaults, `ResolveEffectiveParameters` (`:354`); imports `domains/ec2/instancetypes` for class memory (`:219`). |
|
||||
| `catalog.go` | 214 | **Mixed**: resource owner (engine — `EngineVersions`, `OrderableOptions` `:102-141`) + AWS protocol adapter (filter value sets, SDK projection `:142-214`) | |
|
||||
| `sizing.go` | 57 | resource owner (engine/instance-class facade) | `db.*` → EC2 instance type map (`:14-21`), validated against `domains/ec2/instancetypes` (`:30`). |
|
||||
| `subnetgroup.go` | 340 | resource owner (subnetgroup) | Create/Describe/Delete (`:39`, `:94`, `:138`), subnet resolution via EC2 (`:176`). Reads instance records for in-use check (`:258-277`). |
|
||||
| `parametergroup.go` | 599 | resource owner (parametergroup) — with instance orchestration | CRUD + DescribeDBParameters (`:29-380`); `propagateParameterGroup` (`:210-236`) pushes changes into instances; `resolveGroupParameters` (`:549`) is called by instance create/modify/restore. |
|
||||
|
||||
### gateway/rds and gateway/rds.go
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `gateway/rds.go` | 95 | AWS protocol adapter (ingress glue) | Query parse, action check, caller derivation, authz order, quota injection and `ExpectedNodes` injection (`:16-75`). Imports `bluebottle/pkg/auth.ParseRoleARN` (`:87`). |
|
||||
| `gateway/rds/handler.go` | 220 | AWS protocol adapter | Action inventory (registered/internal/unsupported, `:96-169`), typed Query→XML adapter (`:47-79`), `Env{ExpectedNodes, QuotaCheck}` (`:33-43`). |
|
||||
| `gateway/rds/authz.go` | 112 | AWS protocol adapter | Principal-class gate and per-action resource ARN scopes (`:20-112`). |
|
||||
| `gateway/rds/instance.go` | 50 | AWS protocol adapter | Instance actions + `DescribeEvents`; runs `QuotaCheck` before forwarding create (`:13-19`). |
|
||||
| `gateway/rds/snapshot.go` | 35 | AWS protocol adapter | Forwards snapshot/restore/automated-backup actions. |
|
||||
| `gateway/rds/subnetgroup.go` | 23 | AWS protocol adapter | Forwards subnet-group actions. |
|
||||
| `gateway/rds/parametergroup.go` | 31 | AWS protocol adapter | Forwards parameter-group actions. |
|
||||
| `gateway/rds/tags.go` | 23 | AWS protocol adapter | Forwards tag actions. |
|
||||
| `gateway/rds/enginecatalog.go` | 184 | **Mixed**: AWS protocol adapter (engine catalogue actions) + temporary residue (prerequisite: EC2-owned instance-type-availability projection; removal: `clusterRunnableTypes` replaced by that projection) | `clusterRunnableTypes` (`:146-174`) calls the EC2 AWS adapter `ec2instanceapi.DescribeInstanceTypes` with `env.ExpectedNodes`. |
|
||||
| `gateway/rds/agent.go` | 287 | **Mixed**: guest/controller wire contract (agent-facing internal actions, long poll, reply relay `:146-287`) + AWS protocol adapter (agent identity gate `:36-111`) | Reads the `rds-system` bucket directly via JetStream (`:72-100`), i.e. the gateway reads RDS private state. |
|
||||
|
||||
### daemon and other wiring
|
||||
|
||||
| File | Lines | Class | Reason |
|
||||
|---|---|---|---|
|
||||
| `daemon/rds_deps.go` | 98 | cross-resource orchestration (composition root) — contains residue | Binds EC2/VPC/IGW/RT/NGW/EIP/image/volume/snapshot services, IAM ensurer, master key and config into `handlers_rds.Deps` (`:20-85`). `describeInstancesFanOut` (`:90-97`) calls the EC2 AWS adapter with `len(clusterConfig.Nodes)` as expected-node count (residue; same blocker family as `ExpectedNodes`). Loads `bluebottle/pkg/masterkey` (`:52`). |
|
||||
| `daemon/daemon.go` (RDS sections) | n/a | cross-resource orchestration (composition root) | Subscriptions (`:1171-1206`), service init (`:2046-2066`), reconciler start (`:2070-2078`), backup reaper registration (`:2310-2312`), and the RDS reconciler lease reused as the cluster-wide GC election (`:2374-2382`). The last is residue: an RDS lease elects unrelated reapers. |
|
||||
| `daemon/dns_reconcile.go` (`:71-76`, `:99-104`) | n/a | cross-resource orchestration | Watches RDS per-account buckets via `handlers_rds.AccountWatchBuckets` and consumes `DesiredDNSChanges`. Bucket watch is a read of RDS private storage by the daemon. |
|
||||
| `daemon/bedrock_deps.go:43` | n/a | temporary residue (prerequisite: EC2/EBS-owned attach capability; removal: Bedrock and RDS both bind the EC2 one) | Another domain imports RDS for an EC2 volume-attach client. |
|
||||
| `handlers/quota/live.go:82-93` | n/a | temporary residue (prerequisite: RDS-owned count projection per ADR-0004 S4; removal: quota no longer calls `NATSService.DescribeDBInstances`) | Quota counts instances through the AWS-shaped RDS API. |
|
||||
| `accountteardown/reapers_rds.go` | n/a | temporary residue / cross-domain consumer | Drives instance/subnet-group/parameter-group deletion through `NATSService` AWS-shaped calls (`:41`, `:76`, `:87`, `:109`, `:125`, `:139`, `:159`). Account teardown is explicitly out of ADR-0004 scope. |
|
||||
| `daemon/instance_records.go`, `daemon/instance_records_migrate.go` | 217 / 361 | not RDS | EC2 instance record space; excluded. |
|
||||
|
||||
## 2. Served AWS actions and protocol
|
||||
|
||||
Protocol: AWS Query in (form `Action=`), XML out using the IAM-style `<ActionResponse><ActionResult>` envelope (`gateway/rds.go:14-26`, `gateway/rds/handler.go:59-75`).
|
||||
Order: unknown action → `InvalidAction` before policy (`gateway/rds.go:29-32`); principal-class gate before policy (`:41-43`); resource ARN derivation and policy (`:45-51`); dispatch (`:57-64`).
|
||||
Unsupported actions return `OperationNotSupported` from `Dispatch` (`gateway/rds/handler.go:182-185`).
|
||||
Path: gateway adapter → NATS request `rds.<Action>` (queue group `spinifex-workers`) → daemon `Service` method (`daemon/daemon.go:1173-1206`), except where noted.
|
||||
|
||||
| Action | Status | Owning resource | Notes |
|
||||
|---|---|---|---|
|
||||
| CreateDBInstance | registered | instance | `QuotaCheck` in gateway before forward (`gateway/rds/instance.go:13-19`); 5 min budget (`service_nats.go:16`). |
|
||||
| DescribeDBInstances | registered | instance | |
|
||||
| ModifyDBInstance | registered, scoped | instance | Also binds parametergroup. |
|
||||
| DeleteDBInstance | registered, scoped | instance | Orchestrates final snapshot. |
|
||||
| RebootDBInstance / StartDBInstance / StopDBInstance | registered, scoped | instance | |
|
||||
| CreateDBSnapshot | registered, scopes instance + snapshot | snapshot (orchestrates instance) | |
|
||||
| DescribeDBSnapshots | registered | snapshot | |
|
||||
| DeleteDBSnapshot | registered, scoped | snapshot | |
|
||||
| RestoreDBInstanceFromDBSnapshot | registered, scopes snapshot + instance | cross-resource (snapshot → instance) | |
|
||||
| DescribeDBInstanceAutomatedBackups | registered | instance (automated backup) / snapshot | |
|
||||
| CreateDBSubnetGroup / DescribeDBSubnetGroups | registered | subnetgroup | |
|
||||
| DeleteDBSubnetGroup | registered, scoped | subnetgroup | |
|
||||
| CreateDBParameterGroup / DescribeDBParameterGroups | registered | parametergroup | |
|
||||
| ModifyDBParameterGroup / DescribeDBParameters / DeleteDBParameterGroup | registered, scoped | parametergroup | Modify propagates to instances. |
|
||||
| AddTagsToResource / RemoveTagsFromResource / ListTagsForResource | registered, ARN scope | per ARN kind (all four) | `tags.go:51-73`. |
|
||||
| DescribeEvents | registered | cross-resource (events) | |
|
||||
| DescribeDBEngineVersions | registered | engine | Served **in the gateway process** with no NATS hop (`gateway/rds/enginecatalog.go:41-73`). |
|
||||
| DescribeOrderableDBInstanceOptions | registered (`typedEnv`) | engine (+ EC2 availability) | Served in the gateway; NATS fan-out to EC2 `DescribeInstanceTypes` using `ExpectedNodes` (`gateway/rds/enginecatalog.go:146-174`). |
|
||||
| RegisterDBInstance, SubmitDBStateChange | registered, internal | instance (agent path) | Gateway → bus `rds.bus.{acct}.{db}.register/health` (`service_nats.go:49-57`). |
|
||||
| GetDBBootstrapConfig, AcknowledgeDBBootstrap | registered, internal | instance (bootstrap) | Gateway → `rds.GetDBBootstrapConfig` / `rds.AcknowledgeDBBootstrap`. |
|
||||
| PollDBCommands | registered, internal | instance (command channel) | Served **in the gateway**: queue-subscribes the bus command subject; no daemon handler (`gateway/rds/agent.go:182-247`). |
|
||||
| CreateDBInstanceReadReplica, PromoteReadReplica, CreateDBCluster, ModifyDBCluster, DeleteDBCluster, DescribeDBClusters, FailoverDBCluster, CreateOptionGroup, ModifyOptionGroup, DeleteOptionGroup, DescribeOptionGroups, RestoreDBInstanceToPointInTime | unsupported (12) | n/a | `gateway/rds/handler.go:157-168`. |
|
||||
|
||||
Totals: 26 customer actions registered, 5 internal agent actions, 12 unsupported, 43 in the inventory (`ActionNames`/`UnsupportedActionNames` feed `gateway/operations.go:43-45`).
|
||||
Daemon subscribes 29 `rds.*` subjects (`daemon/daemon.go:1175-1205`); the two catalogue actions and `PollDBCommands` have no daemon subject.
|
||||
Any action outside the table and not in the map is `InvalidAction`.
|
||||
|
||||
## 3. Durable records and KV buckets
|
||||
|
||||
| Bucket | Key | Record | Owning resource (proposed) | Writers | Readers (flag = outside RDS owner) |
|
||||
|---|---|---|---|---|---|
|
||||
| `rds-account-{accountID}` (history 1, version 1, lazily created; `store.go:20-38`, `:201-213`) | `db-instances/{id}` | `DBInstanceRecord` | instance | create, restore, lifecycle, modify, pending, replace, agent, delete, reconciler, recovery, backup, snapshot (`backing-up` CAS), tags | subnetgroup (`subnetgroup.go:258-277`), parametergroup (`parametergroup.go:211`, `:342`), dns.go, **daemon DNS reconcile via bucket watch** (`daemon/dns_reconcile.go:75`), **quota** indirectly via `DescribeDBInstances` (`handlers/quota/live.go:89`), **account teardown** via `NATSService` |
|
||||
| same | `bootstrap-payloads/{id}` | `BootstrapPayloadEnvelope` (encrypted) | instance | create, restore, replace (via bootstrap.go) | agent.go (`GetDBBootstrapConfig`, `AcknowledgeDBBootstrap`), reconciler (stale pending report) |
|
||||
| same | `db-snapshots/{id}` (colon → `.`, `store.go:84-97`) | `DBSnapshotRecord` (manual and automated) | snapshot | snapshot.go, delete.go (final snapshot), backup.go (automated) | restore.go, backup_sweep.go, reconciler snapshot pass, describe, tags |
|
||||
| same | `backups/{dbId}/automated/{ts}` | `AutomatedBackupRecord` | snapshot (automated-backup index) — instance obligation | backup.go | backup_sweep.go, DescribeDBInstanceAutomatedBackups |
|
||||
| same | `retained-volumes/{volumeID}` | `RetainedVolumeRecord` | snapshot (cleanup obligation for a former instance's volume) | delete.go `retainDataVolume` (`:484`) | snapshot.go `releaseRetainedVolume`/`reclaimRetainedVolume` (`:414`, `:430`), backup_sweep `reclaimOrphanedVolumes` (`:310`) |
|
||||
| same | `db-subnet-groups/{name}` | `DBSubnetGroupRecord` | subnetgroup | subnetgroup.go, tags.go | network.go `resolvePlacement` (`:43`, instance create/restore), describe, **account teardown** via `NATSService` |
|
||||
| same | `db-parameter-groups/{name}/meta` | `DBParameterGroupRecord` | parametergroup | parametergroup.go, tags.go | instance create/modify/restore via `resolveGroupParameters` (`parametergroup.go:549`), pending.go |
|
||||
| same | `db-parameter-groups/{name}/params/{key}` | `DBParameterRecord` | parametergroup | parametergroup.go (`:188`, delete `:361`) | `ListDBParameterOverrides` (`store.go`), instance apply paths |
|
||||
| same | `events/{sourceType}/{sourceId}` | event ring (bounded, 14-day trim) | events (cross-resource) | `RecordEvent` from every resource (`events.go:88`) | `DescribeEvents` |
|
||||
| `rds-system` (history 1, version 1; `store.go:24-26`, `:184-195`) | `instance-index/{ec2InstanceID}` | `InstanceIndexEntry{AccountID, DBInstanceIdentifier, VMGeneration}` (`agent.go:628-636`) | instance (agent identity binding) | create (`create.go:124`), restore (`restore.go:142`), replace (`replace.go:200-215`), delete teardown | **gateway reads JetStream directly** (`gateway/rds/agent.go:72-100`); `Service.LookupInstanceIndex` (`agent.go:637`) |
|
||||
| `spinifex-rds-leader` (TTL 60s; `store.go:28-33`, `:216-223`) | `reconciler` | kvlease record | domain-root reconciler leadership | `kvlease` via `Reconciler` (`reconciler.go:104-118`) | **daemon GC uses it as the cluster-wide reaper election** (`daemon/daemon.go:2374-2382`) |
|
||||
|
||||
Non-durable state that affects decisions: in-memory heartbeat liveness map per daemon (`service.go` `liveness`, `agent.go:284-321`), persisted to KV only on change or every `HeartbeatPersistFloor` (5 beats, `records.go` constants).
|
||||
The reconciler leader reads this node-local map (`recovery.go:191`), so a beat answered by another node's queue-group member is seen only via the KV floor (uncertain impact; observation, not intent).
|
||||
|
||||
## 4. Identity and scope per resource
|
||||
|
||||
| Resource | Identifier | Immutable internal ID | ARN | Account | Region | AZ |
|
||||
|---|---|---|---|---|---|---|
|
||||
| instance | `DBInstanceIdentifier`, account-unique via `createJSONRevision` (`create.go:62`), reusable after delete | `DbiResourceID` `db-…` (`create.go:153`) | `arn:aws:rds:{region}:{acct}:db:{id}` (`arn.go:15-27`) | customer account in bucket name and record; VM, system ENI and IAM profile live in the system account `GlobalAccountID` (`create.go:51-55`, `delete.go:544-546`) | not stored; taken from `Service.region` at projection (`service.go`) | not recorded on the instance; derived only via subnet (uncertain whether projected at all; no `AvailabilityZone` reference in `describe.go`) |
|
||||
| snapshot | `DBSnapshotIdentifier`; automated ones `rds:{db}-{ts}` (`backup.go:385`) | none beyond backing EC2 `SnapshotID` | `…:snapshot:{id}` | customer | from service | none |
|
||||
| subnetgroup | `DBSubnetGroupName` (case-sensitive key; `default` rejected, `subnetgroup.go:318`) | none | `…:subgrp:{name}` | customer | from service | each member subnet carries its AZ (`records.go` `DBSubnetGroupSubnet`) |
|
||||
| parametergroup | `DBParameterGroupName`; `default.*` synthesized per engine, not stored (`parametergroup.go:24`, `:456-482`) | none | `…:pg:{name}` | customer | from service | n/a |
|
||||
| engine | engine name + version | n/a (static) | none | global | global | n/a |
|
||||
| agent identity | EC2 instance ID as `RoleSessionName` of the system instance role (`gateway/rds/agent.go:36-69`) | `VMGeneration` in index | n/a | system account principal mapped to customer account via index | n/a | n/a |
|
||||
|
||||
## 5. Lifecycle paths per resource
|
||||
|
||||
### instance
|
||||
|
||||
- Create: validate (`validate.go:69`) → placement reads subnet group / default VPC (`network.go:36`) → resolve parameter group (`create.go:47`) → ensure IAM instance profile in system account (`create.go:51`) → reserve record at `creating` with `createJSONRevision` (`create.go:62`) → stage encrypted bootstrap payload (`create.go:83`) → `LaunchDBInstanceVM` inline in the request (`create.go:87`) → `recordLaunch` CAS (`create.go:115`, `:195-244`) → write instance index (`create.go:124`) → event, return `creating`.
|
||||
- Create failure: deferred rollback deletes payload and CAS-deletes the record if `DbiResourceID` unchanged (`create.go:246-295`); post-launch failures run `Unwind` (`create.go:297-305`).
|
||||
- Readiness: reconciler `reconcileCreating` moves `creating → available` on first healthy agent beat (`reconciler.go:448-464`); `failed` after `BootstrapTimeout` (default 20 min, `reconciler.go:30`).
|
||||
- Steady state: `reconcileHealth` (`recovery.go:76-101`) records/clears failure; `StatusRecovering` exists in the table (`status.go:22`, `:34`, `:43`, `:47`) but **no code transitions into it** (only `status.go` references it), so there is no automatic repair.
|
||||
- Reboot/Start: `beginTransition` CAS (`lifecycle.go:65`, `:143`) then reconciler `reconcileRestarting` (`reconciler.go:479`).
|
||||
- Stop: CAS to `stopping`, inline engine stop + VM stop, inline `completeTransition` to `stopped` before responding (`lifecycle.go:103-133`); reconciler resumes a dead stop (`reconciler.go:584-605`).
|
||||
- Modify: immediate non-disruptive in-request; disruptive writes `PendingModifiedValues` (`modify.go:153`), then either waits for the maintenance window (`backup.go:340`) or runs under the modify lease (`modify.go:176`); reconciler resumes under the same lease (`reconciler.go:510-583`); class/storage change replaces the VM (`pending.go:80`, `replace.go:43-166`).
|
||||
- Replace: launch new VM (`replace.go:113`), rewrite index (`:125`), CAS record with `VMGeneration++` (`:137`), rollback on error (`:218`).
|
||||
- Delete: validate, deletion protection, reserve final snapshot (`delete.go:68`), CAS to `deleting` (`delete.go:73-91`), inline teardown (`delete.go:178-227`): stop engine, terminate VM, final snapshot, purge automated backups, release or retain data volume, delete ENIs (best-effort), delete index, delete payload, delete record last; reconciler replays teardown (`reconciler.go:611-628`) and marks `failed` after `transitionTimeout` 10 min.
|
||||
- Fencing: KV revision CAS on the record everywhere; per-instance `ModifyLease` on the record (`modify_lease.go:24-173`); cluster `kvlease` for the reconciler (`reconciler.go:104-118`); `VMGeneration` binds bootstrap payloads and acknowledgements (`agent.go:345`, `:470`, `:552-553`) and the index.
|
||||
- No desired generation exists on `DBInstanceRecord` (`records.go`); `VMGeneration` is a realization generation, not desired-intent generation.
|
||||
- Idempotency: AWS `CreateDBInstance` has no client token, and a repeat returns `DBInstanceAlreadyExists` (`create.go:64-66`); delete retry re-runs teardown and must repeat the snapshot choice (`delete.go:57-63`); register is idempotent (`agent.go:159-160`).
|
||||
|
||||
### snapshot
|
||||
|
||||
- Create: check name free (`snapshot.go:92`) → CAS instance to `backing-up` with `SnapshotOperation` (`:672-705`) → create record `creating` (`:109-115`) → quiesce + EC2 snapshot inline (`:144-188`) → `putJSON` record `available` (`:126-130`) → restore instance status (`:706-723`).
|
||||
- Readiness: synchronous; the API returns `available`.
|
||||
- Repair: reconciler resolves records stuck `creating` past 15 min against EC2 (`reconciler.go:674-751`, `findEC2SnapshotFor` `:752`); returns instances stuck in `backing-up` (`reconciler.go:630-652`).
|
||||
- Delete: `removeDBSnapshot` deletes EC2 snapshot then record and releases a retained volume when last holder (`snapshot.go:330-470`).
|
||||
- Automated: window-driven `takeAutomatedBackup` (`backup.go:270`), stamped on instance record so a leader change cannot double-fire (`reconciler.go:654-660` comment); retention sweep `BackupRetentionReaper` via the daemon GC backstop (`backup_sweep.go:45-112`, `daemon/daemon.go:2310`).
|
||||
- Fencing: instance-record CAS into `backing-up`; `createJSON` on snapshot key; no snapshot generation.
|
||||
|
||||
### subnetgroup
|
||||
|
||||
- Create: validate, resolve subnets through EC2 `DescribeSubnets` (`subnetgroup.go:176-239`), `createJSON` (`:78`); synchronous; status always `Complete` (`:31`, `:287`).
|
||||
- Describe: KV read and list (`:94-133`).
|
||||
- Delete: read, scan all instance records for `DBSubnetGroupName == name` (`:154-163`), plain `kv.Delete` (`:165`).
|
||||
- No modify action is served (`ModifyDBSubnetGroup` is neither registered nor unsupported, so it returns `InvalidAction`).
|
||||
- No reconciliation; no generation; tags via `tags.go` CAS.
|
||||
|
||||
### parametergroup
|
||||
|
||||
- Create: validate family against engine registry, `createJSON` meta (`parametergroup.go:29-84`); synchronous.
|
||||
- Modify: validate whole batch, then `putJSON` one key per parameter (`:186-196`), then synchronously push to every attached instance via agent `apply-params` (`:198`, `:210-236`, `pending.go:149`).
|
||||
- Delete: refuse `default.*`, scan instances for current or pending use (`:342-352`), delete params then meta (`:354-367`).
|
||||
- `default.*` groups are synthesized from the engine catalogue (`:456-482`).
|
||||
- No generation; no reconciliation of a failed propagation beyond `ParameterApplyFailed` on the instance (`pending.go:214`).
|
||||
|
||||
### engine
|
||||
|
||||
- Static, process-local catalogue validated at daemon boot (`engine.go:69`, `daemon/daemon.go:2055`); no lifecycle; ADR-0003 S1 excludes pure read-only catalogues from the lifecycle contract.
|
||||
|
||||
## 6. NATS subjects and controller-to-agent messages
|
||||
|
||||
Gateway → daemon (request/reply, queue group `spinifex-workers`): `rds.<Action>` for the 24 customer actions listed in `subjects.go:21-63`, plus `rds.GetDBBootstrapConfig` and `rds.AcknowledgeDBBootstrap` (`subjects.go:17-19`).
|
||||
Budgets: default 30 s, create 5 min, lifecycle 4 min, delete 6 min, modify 6 min, snapshot 6 min, snapshot delete 4 min, restore 8 min (`service_nats.go:12-36`).
|
||||
A gateway timeout does not cancel the daemon's inline orchestration (uncertain; inferred from request/reply shape), so the client may see an error while the create continues.
|
||||
|
||||
Agent path (agent speaks SigV4 Query to awsgw over the mgmt bridge, `daemon/rds_deps.go:71-73`):
|
||||
|
||||
- `RegisterDBInstance` → gateway → `rds.bus.{acct}.{db}.register` → daemon `SubjectRegisterWildcard` (`subjects.go:12-13`, `:65-67`).
|
||||
- `SubmitDBStateChange` (heartbeat folded into state, 30 s interval, stale after 90 s; `records.go` constants) → `rds.bus.{acct}.{db}.health`.
|
||||
- `GetDBBootstrapConfig` / `AcknowledgeDBBootstrap` → Layer-1 subjects; replay is allowed until acknowledged for the bound `VMGeneration` (`gateway/rds/agent.go:150-180`).
|
||||
- `PollDBCommands` long poll (1-20 s) queue-subscribes `rds.bus.{acct}.{db}.command` with group `spinifex-rds-agents` and publishes carried replies to `rds.bus.{acct}.{db}.command-reply` before subscribing (`gateway/rds/agent.go:198-287`).
|
||||
- Command types: `set-password`, `apply-params`, `stop-engine`, `grow-filesystem`, `quiesce`, `unquiesce` (`commands.go:12-23`); params `MasterUsername`, `MasterUserPassword`, `QuiesceLabel`, `QuiesceDeadlineSeconds` (`commands.go:27-35`).
|
||||
- Issuer: `issueCommand` subscribes the reply subject, then republishes the same `CommandID` every 2 s until reply or budget (`commands.go:144-199`, `:72`); core NATS, not durable, by design for the password (`subjects.go:77-80`).
|
||||
- Agent retry: results are retained and re-carried until a poll succeeds (`agents/rds/agent/command.go:100-133`).
|
||||
- Agent dedupe: the agent executes every delivered command and has **no CommandID dedupe** (`agents/rds/agent/command.go:128-131`, `:140-157`); a republish delivered after execution but before the issuer sees the reply can execute twice (uncertain; depends on timing between reply relay and the 2 s republish).
|
||||
- Commands carry no `VMGeneration` (`agent.go:134-139`), so the fence against a superseded VM is the bus subject plus the gateway's index lookup rather than a generation in the message.
|
||||
- `LookupEngine`, `Engine`, `ParamType*`, `ApplyType*`, `Parameter`, `EngineHealth*`, `HeartbeatInterval`, `BootstrapMode*`, `ParameterRollbackMessage`, `SupportedInstanceClasses`, `SmallestInstanceClass` are imported by the agent from `handlers/rds` (full list from import scan); this is the S5 "accidental import of a domain monolith".
|
||||
|
||||
RDS → EC2 subjects used as capabilities: `ec2v1.InstanceCommandSubject(id)` for stop/reboot/attach (`lifecycle.go:434`, `launch.go:583`), `ec2.start` (`lifecycle.go:451`), `ec2.DescribeStoppedInstances` (`launch.go:616`), plus EC2 `DescribeInstances` fan-out (`daemon/rds_deps.go:90-97`) and `DescribeInstanceTypes` fan-out (`gateway/rds/enginecatalog.go:147`).
|
||||
|
||||
Recovery/redelivery: there is no durable work queue; recovery relies on the reconciler re-reading record status (`reconciler.go:238-282`) under `kvlease`, a 5 min resync and KV watch wakeups (`reconciler.go:120-145`).
|
||||
|
||||
## 7. Boundary-crossing imports
|
||||
|
||||
### Outbound from RDS (handlers/rds, gateway/rds, gateway/rds.go, daemon/rds_deps.go)
|
||||
|
||||
- **Quota**: none from `handlers/rds`; the gateway injects `gw.Quota.EnforceRDSInstances` as `Env.QuotaCheck` (`gateway/rds.go:59-62`).
|
||||
- **ExpectedNodes / EC2 instance availability**: `gateway/rds/enginecatalog.go` → `domains/ec2/awsapi/instance.DescribeInstanceTypes` with `env.ExpectedNodes` (`:147`); `daemon/rds_deps.go` → `domains/ec2/awsapi/instance.DescribeInstances` with node count (`:96`).
|
||||
- **EC2 instance**: `domains/ec2/systeminstance` (`launch.go:99-100`, `delete.go:236`), `domains/ec2/instance` (`lifecycle.go`, `StartStoppedInstanceOutput`), `contracts/ec2/v1` (`launch.go`, `lifecycle.go`), `runtime/compute/vm` (`agent.go`, `create.go` `vm.VolumeSerial`, `backup_sweep.go` `vm.Reaper`), `foundation/aws/ami`, `domains/ec2/instancetypes` (`sizing.go:30`, `paramcatalog.go:219`).
|
||||
- **VPC/network**: `domains/network/systemvpc` (`vpc.go`, `launch.go`, `daemon/rds_deps.go`), EC2 SDK VPC/subnet/SG/ENI calls through narrow interfaces `networkResolver` (`network.go:16`) and `launchVPCProvisioner` (`launch.go:76-96`); `foundation/aws/tags` (`ManagedByRDS`).
|
||||
- **Storage/EBS**: EC2 SDK `CreateVolume`/`DeleteVolume` (`launch.go:107-111`), `DescribeVolumes`/`ModifyVolume` (`storage.go:23-26`), EC2 snapshot provider (`snapshot.go:25`), volume attach over EC2 NATS (`launch.go:560-629`).
|
||||
- **IAM**: `handlers/iam` for `SystemInstanceRoleEnsurer`/`EnsureSystemInstanceProfile` (`iam.go:56-77`) and `EncryptSecret`/`DecryptSecret` for bootstrap payloads (`bootstrap.go:105`, `:199`).
|
||||
- **DNS**: `domains/dns` (`dns.go:17`, `dns.RDSName`, `dns.Change`).
|
||||
- **Bluebottle**: `bluebottle/pkg/auth.ParseRoleARN` (`gateway/rds.go:87`), `bluebottle/pkg/masterkey.ReadShared` (`daemon/rds_deps.go:52`); `handlers/rds` imports no Bluebottle package.
|
||||
- **Ingress/foundation**: `ingress/aws/query`, `foundation/aws/xml`, `foundation/aws/errors`, `foundation/aws/identifiers`, `foundation/state/{kvstore,kvlease,kvutil,migrate}`, `foundation/lifecycle/reconciler`, `foundation/messaging/nats`, `foundation/pki`, `bootstrap/config`.
|
||||
|
||||
### Inbound into RDS (who imports `handlers/rds` or `gateway/rds`)
|
||||
|
||||
- `handlers/quota/live.go:14,89` → `NewNATSService(...).DescribeDBInstances` (quota reads RDS through the AWS-shaped API).
|
||||
- `accountteardown/reapers_rds.go:9` → `NATSService` describe/modify/delete for instances, subnet groups, parameter groups.
|
||||
- `daemon/daemon.go:78` → `Service`, `Reconciler`, all `Subject*`, `ValidateEngineRegistry`, `NewService`, `NewReconciler`, `NewBackupRetentionReaper`.
|
||||
- `daemon/rds_deps.go` → `Deps`, `LaunchDeps`, `BackupPolicy`, `NewNATSVolumeAttacher`, `NewDescribeInstanceState`, `NewNATSInstanceCommander`.
|
||||
- `daemon/dns_reconcile.go:16,75` → `AccountWatchBuckets` (raw bucket handles).
|
||||
- `daemon/bedrock_deps.go:10,43` → `NewNATSVolumeAttacher` (Bedrock/Ochre depends on RDS).
|
||||
- `gateway/operations.go:8,44-45` → `gateway_rds.ActionNames`, `UnsupportedActionNames`.
|
||||
- `gateway/rds/*` → `KVBucketRDSSystem`, `InstanceIndexKey`, `InstanceIndexEntry`, `InstanceRoleName`, `InternalAgentActions`, `ResourceKind*`, `FormatARN`, `ParseARN`, catalogue functions, `NATSService`, wire types.
|
||||
- `agents/rds/agent/*` → wire types and engine catalogue (section 6).
|
||||
- `tests/integration/service_daemon_lite.go:51`, `tests/e2e/harness/rds.go` (test support).
|
||||
|
||||
## 8. Known gaps against Q-78 / ADR-0003 (observed in code, not live-verified)
|
||||
|
||||
1. No desired generation on any RDS record (`records.go`); mutations advance only the KV revision, and `PendingModifiedValues` is overwritten by a later modify (`modify.go:153`), so Q-78 "atomically commits a monotonically increasing desired generation" is not met.
|
||||
2. Desired and observed state share one record: `Status`, `Agent`, `UnhealthySince`, `FailureReason` sit beside requested class/storage on `DBInstanceRecord` (`records.go`).
|
||||
3. Create performs all external realization inline in the request (`create.go:87`); if the daemon dies after `LaunchDBInstanceVM` returns and before `recordLaunch` (`create.go:115`), the record stays `creating` with no `InstanceID`, times out to `failed` (`reconciler.go:458-462`), and a later delete has no IDs to clean (`delete.go:229-233`, `:537-547`); the VM, ENIs and volume are tagged `ManagedBy=rds` (`launch.go:260`, `:314`, `:400`) but no RDS orphan reaper for them was found.
|
||||
4. The same interruption window exists for restore (`restore.go:111` → `:136`) and replace (`replace.go:113` → `:137`); replace's rollback runs only on error return, not on process death.
|
||||
5. Delete removes the only ownership record (`delete.go:217`) after ENI deletion that is best-effort and only logs failures (`delete.go:203`, `launch.go:510-520`), contrary to Q-78 "a handler never removes the only reconstructible ownership of required cleanup".
|
||||
6. `deleting` that exceeds 10 min becomes `failed` (`reconciler.go:621-624`); a `failed` instance is then handled by `reconcileHealth` rather than continuing teardown, so incomplete cleanup is not shown as a distinct state (uncertain; inferred from the `switch` at `reconciler.go:256-272`).
|
||||
7. `StatusRecovering` is declared and reachable in the transition table but never entered, so there is no repair path (`status.go:22`; `recovery.go:19-21` says v1.0 reports rather than repairs).
|
||||
8. Synchronous completion where AWS is asynchronous: AWS `CreateDBSnapshot` returns `Status=creating`, here it returns `available` after the snapshot is taken inline (`snapshot.go:126-130`); AWS `StopDBInstance` returns `stopping`, here the handler completes to `stopped` inline (`lifecycle.go:126`); AWS `DeleteDBInstance` leaves the instance describable as `deleting`, here the record is deleted before the response when teardown succeeds inline (`delete.go:97-104`, `:217`).
|
||||
9. DB subnet group and parameter group deletes are check-then-delete across keys with no fence against a concurrent `CreateDBInstance` that has already read the group (`subnetgroup.go:151-165`, `parametergroup.go:339-367`; `network.go:43`, `create.go:47`), so an instance can reference a deleted group.
|
||||
10. `ModifyDBParameterGroup` writes one key per parameter with no batch atomicity (`parametergroup.go:186-196`); a KV failure mid-loop leaves a half-applied batch despite the comment at `:140-143`, and propagation to instances is synchronous best-effort (`:210-236`).
|
||||
11. Quota admission is a non-atomic count before create (`gateway/rds/instance.go:13-19`, `handlers/quota/live.go:85-93`), so concurrent creates can exceed the cap; it counts `len(DescribeDBInstances)` which pages at 100 by default (`paging.go:14`, `describe.go:71`), so an account above 100 instances is under-counted (uncertain whether quota limits ever exceed 100).
|
||||
12. Agent commands are republished every 2 s with no agent-side `CommandID` dedupe and no generation (section 6), so internal redelivery is not proven idempotent for `grow-filesystem`, `apply-params` or `set-password`.
|
||||
13. Heartbeat liveness lives partly in per-node process memory (`agent.go:284-321`) and the leader judges health from its own map plus a KV floor (`recovery.go:191-209`), which Q-78 discourages for recovery decisions (observation-side, lower severity).
|
||||
14. The RDS reconciler lease doubles as the cluster-wide GC election for unrelated reapers (`daemon/daemon.go:2374-2382`); without RDS service init every cluster-wide reaper is skipped.
|
||||
15. No lifecycle contract record exists for any RDS resource (ADR-0003 S1); nothing in `handlers/rds` states readiness predicates or deletion obligations as a contract.
|
||||
|
||||
## 9. First-resource candidates
|
||||
|
||||
| Rank | Resource | Size (prod lines) | Private state | Dependencies | Tests | Extractable without quota / ExpectedNodes / IAM / Bluebottle? |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | **subnetgroup** | ~340 (`subnetgroup.go`) + its slice of `store.go`, `records.go`, `tags.go`, `arn.go`; gateway 23 | `db-subnet-groups/{name}` only | EC2 `DescribeSubnets` through a narrow interface already (`network.go:16-20`, `subnetgroup.go:189`); reverse dependency: needs an instance-owned query "instances referencing subnet group X" to replace `instancesUsingGroup` (`subnetgroup.go:154`); forward dependency: instance placement reads the group record directly (`network.go:43`), which must become a subnetgroup query | `handlers/rds/subnetgroup_test.go` (338), `tags_test.go`, `gateway/rds/handler_test.go`/`authz_test.go`, e2e `tests/e2e/rds/groups_test.go` (`TestSubnetAndParameterGroups`), `isolation_test.go`, `api_matrix_test.go` (uncertain coverage depth for the last two) | **Yes.** No quota, ExpectedNodes, IAM, Bluebottle, agent or EC2-launch dependency. Synchronous AWS semantics (status `Complete`) match the code. Gap 9 (delete race) should be recorded, not fixed, during the move. |
|
||||
| 2 | **parametergroup** | ~600 (`parametergroup.go`) + slice of store/records/tags; gateway 31 | `db-parameter-groups/{name}/meta`, `…/params/{key}` | Engine catalogue (`engineForFamily`, `LookupParameter`, `ResolveEffectiveParameters`) — owned by **engine**; `SmallestInstanceClass` (EC2 `instancetypes`); cluster CA availability (`checkTLSEnforceable`, `parametergroup.go:581`); instance records for in-use check; **instance orchestration** in Modify (`propagateParameterGroup` → `applyParameterGroup` → agent `apply-params`) | `parametergroup_test.go` (806), `paramcatalog_test.go`, `enginefamily_test.go`, `tlsenforcement_test.go` (uncertain scope), `pending_test.go` (propagation), e2e `groups_test.go` | **Mostly.** No quota/ExpectedNodes/IAM/Bluebottle, but Create/Describe/Delete are synchronous while Modify crosses into instance realization and the agent wire contract; extraction requires engine first (or a narrow engine capability) and moving propagation to explicit cross-resource orchestration. |
|
||||
| 3 | **engine** | ~1,960 (`engine.go`, `engine_postgres.go`, `engine_mariadb.go`, `paramcatalog.go`, `catalog.go`, `sizing.go`) | none (static catalogue) | `domains/ec2/instancetypes`; consumed by the agent, parametergroup, instance | `engine_test.go`, `engine_mariadb_test.go`, `enginefamily_test.go`, `paramcatalog_test.go`, `catalog_test.go`, `sizing_test.go`, `gateway/rds/enginecatalog_test.go` | **Package: yes**; it is the natural S5 agent capability. **awsapi action `DescribeOrderableDBInstanceOptions`: no**, blocked on the EC2 instance-type-availability projection (`gateway/rds/enginecatalog.go:146-174`). Not a durable resource under ADR-0003 S1, so it is weak evidence for ADR-0004 rollout item 2. |
|
||||
| 4 | **snapshot** | ~1,700 (`snapshot.go`, `backup_sweep.go`, share of `backup.go`, `restore.go`, `reconciler.go:674-796`) | `db-snapshots/`, `backups/…/automated/`, `retained-volumes/` | Instance record CAS into `backing-up`, engine quiesce over agent commands, EC2 snapshot/volume services, instance delete (final snapshot, retained volumes), restore runs the instance create path | `snapshot_test.go` (879), `restore_test.go`, `backup_test.go` (1013), `backup_sweep_test.go`, `delete_test.go`; e2e `snapshot_restore_test.go`, `automated_backup_test.go` | No quota/ExpectedNodes/IAM/Bluebottle on snapshot itself, but restore inherits instance create (IAM, launch); snapshot creation needs instance-owned quiesce/hold capabilities first. Not a first candidate. |
|
||||
| 5 | **instance** | ~8,000+ (create, describe, lifecycle, launch, network, vpc, storage, modify, modify_lease, pending, replace, recovery, reconciler, window, backup, delete, agent, commands, bootstrap, servingcert, userdata, iam, dns, validate, status) | `db-instances/`, `bootstrap-payloads/`, `rds-system` `instance-index/`, `spinifex-rds-leader` | Quota admission, ExpectedNodes-style EC2 fan-out (`daemon/rds_deps.go:90-97`), IAM ensurer and secret crypto, Bluebottle master key (via composition), EC2 system instance / ENI / volume / snapshot / attach, system VPC, DNS, agent wire contract | `create_test.go`, `launch_test.go`, `lifecycle_test.go`, `delete_test.go`, `modify_test.go`, `modify_lease_test.go`, `pending_test.go`, `replace_test.go`, `reconciler_test.go`, `recovery_test.go`, `revisit_test.go`, `agent_test.go`, `bootstrap_test.go`, `commands_test.go`, `storage_test.go`, `network_test.go`, `vpc_test.go`, `describe_test.go`, `status_test.go`, `iam_test.go`, `userdata_test.go`, `watch_source_test.go`, `gateway/rds/agent_test.go`, `authz_test.go`; e2e `create_describe`, `lifecycle`, `modify`, `delete`, `failure`, `graceful_reboot`, `connect`, `cross_subnet`, `isolation`, `authz`, `mariadb*` | **No.** Every listed blocker applies. Extract last. |
|
||||
|
||||
Cross-resource files to settle alongside the first extraction: `tags.go` (per-ARN dispatch), `events.go` (journal), `arn.go`/`filters.go`/`paging.go` (awsapi), `store.go`/`records.go`/`service.go` (split per owner).
|
||||
|
||||
Recommendation: extract **subnetgroup** first, with two narrow capabilities defined by the consumer: an instance-owned "references to subnet group" query for delete, and a subnetgroup-owned "resolve placement subnets" query for instance create.
|
||||
Then do **engine** (package and agent capability, leaving the orderable action on the legacy path until the EC2 projection exists), then **parametergroup** with propagation made explicit orchestration.
|
||||
Test coverage for subnetgroup is unit plus one e2e test; ADR-0003 S5 failure/recovery evidence for its delete race does not exist yet.
|
||||
Reference in New Issue
Block a user