mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
This commit is contained in:
joshsiv-mulga
@@ -0,0 +1,416 @@
|
||||
# EKS managed add-on: per-bundle readiness and removal inventory
|
||||
|
||||
Code read at 3a55e2ee1; not verified live.
|
||||
|
||||
This is a discovery record for the add-on lifecycle work: for every bundle in the catalogue it lists what is rendered, how identity is wired, what "ready" currently proves, what removal deletes and leaves, and where the bundle collides with other objects.
|
||||
It proposes no design; gaps are stated per bundle and collected at the end.
|
||||
It complements `docs/package-boundary/eks-addon-lifecycle.md` (present contract) and `docs/package-boundary/eks-addon-lifecycle-design.md` (draft correction, which refines open proposal Q-146: delete holds until owned objects and identity associations are observably removed, and `ACTIVE` means workloads applied and ready).
|
||||
Paths starting `spinifex/` are Go sources; `scripts/` and `tests/` are relative to the spinifex repo root; bundle files are under `scripts/images/eks-node/addons/<addon>/<version>/` and are abbreviated to `<addon>/<version>/<file>`.
|
||||
The sync agent `scripts/images/eks-node/mulga-eks-addon-sync.sh` is abbreviated to `sync.sh`.
|
||||
|
||||
## 1. Catalogue
|
||||
|
||||
| Name | Version | `RequiresIRSA` | Hidden | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| `aws-load-balancer-controller` | 2.11.0 | true | no | `spinifex/domains/eks/addon/catalog.go:31-33` |
|
||||
| `argocd` | 3.0.23 | false | no | `catalog.go:34-36` |
|
||||
| `aws-ebs-csi-driver` | 1.40.1 | true | no | `catalog.go:37-39` |
|
||||
| `nvidia-device-plugin` | 0.17.4 | false | yes (auto-staged by GPU node groups) | `catalog.go:42-44`, `spinifex/domains/eks/addon/owner.go:79-89`, `spinifex/handlers/eks/nodegroup.go:408-416` |
|
||||
| `spinifex-noop` | 0.1.0 | false | yes (e2e fixture) | `catalog.go:48-50` |
|
||||
|
||||
`RequiresIRSA` is read only by `DescribeAddonVersions` as `requiresIamPermissions` (`spinifex/handlers/eks/addons.go:255`); `CreateAddon` neither requires a role for the two `true` bundles nor rejects one for the three `false` bundles (`addons.go:61-99`).
|
||||
The bundle manifests are baked into the AMI (`scripts/images/eks-node/manifest.conf:58-67`), but container images are not: every image below is pulled from a public registry at pod start.
|
||||
|
||||
## 2. AWS reference semantics
|
||||
|
||||
Checked against the API reference on 2026-10-09.
|
||||
|
||||
| Item | AWS documents | Source |
|
||||
|---|---|---|
|
||||
| `status` | Enum `CREATING`, `ACTIVE`, `CREATE_FAILED`, `UPDATING`, `DELETING`, `DELETE_FAILED`, `DEGRADED`, `UPDATE_FAILED`; the API reference gives no per-value definition of `ACTIVE` or `DEGRADED`. | API_Addon |
|
||||
| `health` | `AddonHealth.issues`, a list of `AddonIssue`. | API_AddonHealth |
|
||||
| `AddonIssue` | `code` one of `AccessDenied`, `InternalFailure`, `ClusterUnreachable`, `InsufficientNumberOfReplicas`, `ConfigurationConflict`, `AdmissionRequestDenied`, `UnsupportedAddonModification`, `K8sResourceNotFound`, `AddonSubscriptionNeeded`, `AddonPermissionFailure`; plus `message` and `resourceIds`. | API_AddonIssue |
|
||||
| `serviceAccountRoleArn` | "The ARN of the IAM role that's bound to the Kubernetes `ServiceAccount` object that the add-on uses"; on create, "If you don't specify an existing IAM role, then the add-on uses the permissions assigned to the node IAM role", and an IAM OIDC provider must exist for the cluster. | API_Addon, API_CreateAddon |
|
||||
| `configurationValues` | "validated against the schema returned by `DescribeAddonConfiguration`". | API_CreateAddon |
|
||||
| `resolveConflicts` | `NONE`/`PRESERVE` leave a self-managed install's values and may fail creation; `OVERWRITE` replaces them; with no prior install all three behave alike. | API_CreateAddon |
|
||||
| `DeleteAddon` | "When you remove an add-on, it's deleted from the cluster"; `preserve` "preserves the add-on software on your cluster but Amazon EKS stops managing any settings for the add-on. If an IAM account is associated with the add-on, it isn't removed." The example response is `DELETING`. | API_DeleteAddon |
|
||||
|
||||
The issue codes imply AWS distinguishes replica shortfall (`InsufficientNumberOfReplicas`), missing objects (`K8sResourceNotFound`), field conflicts (`ConfigurationConflict`) and admission denials; how EKS computes them, and whether `DEGRADED` is driven by them, is not stated in the API reference and was not verified.
|
||||
Whether AWS offers each of these five names as an EKS add-on was not checked against a live `DescribeAddonVersions`.
|
||||
|
||||
## 3. Shared mechanics (apply to every bundle)
|
||||
|
||||
### 3.1 Render
|
||||
|
||||
| Step | Behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Input | One TSV line per staged add-on: `name`, `version`, `roleArn`, `base64(configurationValues)`. | `spinifex/agents/eks/gatewayfetch/fetch.go` `emitAddonsTSV`, `sync.sh:357` |
|
||||
| Source | Every `*.yaml` in `/usr/share/spinifex-eks/addons/<name>/<version>/`, in glob order, concatenated with `---`. A missing directory reports `failed`. | `sync.sh:41`, `sync.sh:82-89`, `sync.sh:219-242` |
|
||||
| Output | One file per add-on name, `/var/lib/rancher/k3s/server/manifests/spinifex-addon-<name>.yaml`; the name carries no version. Replaced only when content changes. | `sync.sh:42-43`, `sync.sh:83`, `sync.sh:248-253` |
|
||||
| Block markers | `{{IRSA_ENV}}`, `{{IRSA_VOLUME}}`, `{{IRSA_VOLUME_MOUNT}}`, `{{ELB_INGRESS_PARAMS_SPEC}}` replaced by awk when alone on a line. | `sync.sh:224-230` |
|
||||
| Scalar placeholders | `{{SERVICE_ACCOUNT_ROLE_ARN}}`, `{{CLUSTER_NAME}}`, `{{AWS_REGION}}`, `{{AWS_VPC_ID}}`, `{{GATEWAY_ENDPOINT}}`, `{{GATEWAY_CA_PEM_B64}}`, `{{WEBHOOK_CA_B64}}`, `{{WEBHOOK_TLS_CRT_B64}}`, `{{WEBHOOK_TLS_KEY_B64}}` via `sed s|..|..|g`. The role ARN is inserted raw: it is not escaped for sed (`|`, `&`, `\`) or for YAML. | `sync.sh:231-240` |
|
||||
| Webhook cert | Opt-in by `webhook.conf`; minted once per add-on under `/var/lib/spinifex-eks/webhook-certs/<name>`, 10-year validity, reused thereafter. | `sync.sh:98-120`, `spinifex/agents/eks/webhookcert/cert.go:109-110` |
|
||||
| `configurationValues` | Read from the TSV and discarded; no bundle has a hook. | `sync.sh:359` |
|
||||
| Executor | Primary server only; server-join nodes do not run the agent. | `scripts/images/eks-node/eks-node-role.sh:138-149` |
|
||||
|
||||
### 3.2 Identity (IRSA) wiring
|
||||
|
||||
| Element | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Role carrier | The role ARN lives in the add-on record and the staged manifest (`serviceAccountRoleArn`), and after render in the rendered file only. No pod identity association, no EKS-side binding object. | `spinifex/domains/eks/addon/record.go:24-45`, `sync.sh:144-160` |
|
||||
| Credential env | With a role: `AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE`, `AWS_STS_REGIONAL_ENDPOINTS`, region, `AWS_ENDPOINT_URL_STS` (gateway), `AWS_CA_BUNDLE` and `SSL_CERT_FILE`. Without one: region and CA only, so the SDK falls through to node instance-profile credentials over IMDS. | `sync.sh:144-195` |
|
||||
| Token | A projected ServiceAccount token, audience `sts.amazonaws.com`, 86400 s, mounted at `/var/run/secrets/eks.amazonaws.com/serviceaccount`, injected statically by the render. There is no pod-identity mutating webhook in the image, so the `eks.amazonaws.com/role-arn` ServiceAccount annotation is descriptive only; the env and volume are what take effect. | `sync.sh:161-178`; no `pod-identity-webhook` in the tree |
|
||||
| Issuer | The apiserver's `/openid/v1/jwks` is published at first boot; STS `AssumeRoleWithWebIdentity` resolves `iss` to a registered IAM OIDC provider in the role's account, fetches the cluster JWKS, and evaluates the role trust policy (`{iss}:sub`, `{iss}:aud`). | `scripts/images/eks-node/k3s-first-boot.sh:141-162`, `spinifex/handlers/sts/assume_role_with_web_identity.go:33-213`, `spinifex/handlers/sts/web_identity_trust_policy.go:137-170` |
|
||||
| Objects outside the bundle | The IAM role, its trust policy (naming `system:serviceaccount:<ns>:<sa>`) and the IAM OIDC provider are user-owned IAM objects; nothing in the add-on path creates, modifies or deletes them. | `tests/e2e/eks/eks_test.go:658-700` creates them in the test |
|
||||
| Authorisation | `iam:PassRole` on the role at `CreateAddon` and `UpdateAddon`. | `spinifex/gateway/eks/passrole.go:31-37` |
|
||||
| `eks-token-webhook` | Not part of IRSA: it is the apiserver TokenReview authenticator for `aws eks get-token` bearer tokens. | `scripts/images/eks-node/eks-token-webhook.initd:1-8` |
|
||||
|
||||
Because identity association lives only in IAM (user-owned) and in the rendered pod spec, "identity associations observably removed" has no Spinifex-side object to observe for any bundle today: the role is not detached from anything when the add-on is deleted, matching AWS only in that AWS also does not delete the IAM role.
|
||||
|
||||
### 3.3 Readiness
|
||||
|
||||
| Path | Condition | Evidence |
|
||||
|---|---|---|
|
||||
| Primary | The K3s `Addon` CR in `kube-system` whose `.spec.source` is the rendered file carries a non-empty `addon.k3s.cattle.io/gvks` annotation. Reported `ready` on every tick this holds. No pod, replica or CRD condition is consulted. | `sync.sh:266-273`, `sync.sh:361-365` |
|
||||
| Fallback | Only after `ADDON_READY_GRACE` (90 s) with the annotation still empty: `kubectl get -f <rendered> --ignore-not-found`, then require at least one Deployment, StatefulSet or DaemonSet and that every one returned is rolled out (Deployment `availableReplicas >= spec.replicas` and generation observed; StatefulSet `readyReplicas >= spec.replicas`; DaemonSet `desiredNumberScheduled > 0` and `numberReady >= desiredNumberScheduled`). | `sync.sh:56`, `sync.sh:301-320`, `sync.sh:367-372` |
|
||||
| Otherwise | `applied`, which changes no status. | `sync.sh:374-376`, `spinifex/domains/eks/addon/owner.go:186-205` |
|
||||
| `failed` | Emitted only for a missing bundle directory or webhook cert failure; never for apply errors, crash-looping pods or lost replicas. | `sync.sh:87`, `sync.sh:112` |
|
||||
| Status mapping | `ready` lifts any non-`ACTIVE` status (including `CREATE_FAILED`, `DEGRADED`, `UPDATING`) to `ACTIVE` and clears `health`; `failed` maps `ACTIVE` to `DEGRADED`. Reports carry no generation. | `owner.go:140-168`, `owner.go:186-205` |
|
||||
|
||||
Consequences common to all bundles:
|
||||
|
||||
1. On the primary path `ready` means "K3s recorded the GVKs it applied", not "workloads running"; a bundle whose pods never start (image pull failure, unschedulable, crash loop) is `ACTIVE` and stays so.
|
||||
2. `ACTIVE` never degrades on workload loss: the annotation persists, so `ready` is re-sent every tick and no `failed` is ever produced for runtime health.
|
||||
3. `--ignore-not-found` drops absent objects from the fallback set, so a workload that failed to apply is simply not checked; one rolled-out workload is enough.
|
||||
4. After `UpdateAddon` the same tick that rewrites the file reads the annotation from the previous apply; whether K3s clears it before that read is not established, so `UPDATING` may settle `ACTIVE` on the old apply (the agent comment at `sync.sh:244-247` says a re-apply clears it, which suggests a window exists).
|
||||
5. The agent never deletes the K3s `Addon` CR; if K3s keeps that CR after the file is removed, a later re-create would match it by `.spec.source` and could report `ready` from a stale annotation before the new apply (K3s behaviour not verified).
|
||||
|
||||
### 3.4 Removal
|
||||
|
||||
| Step | Behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Trigger | A rendered `spinifex-addon-<name>.yaml` whose name is absent from a successful fetch. A failed fetch skips GC. | `sync.sh:344-348`, `sync.sh:386-391` |
|
||||
| Selection | The object list is the rendered file itself (`kubectl delete -f`), not labels or a GVK inventory. Objects created at runtime by the add-on, or by users through it, are never selected. | `sync.sh:397-401` |
|
||||
| Order | File order, which is glob order of the bundle files then document order. | `sync.sh:219` |
|
||||
| Waiting | `--wait=false`; errors go to syslog; the copy is deleted; nothing is reported. | `sync.sh:400-402` |
|
||||
| Leftovers on the node | `/var/lib/spinifex-eks/webhook-certs/<name>` and the K3s `Addon` CR. The `/tmp` sentinels are cleared. | `sync.sh:47`, `sync.sh:393` |
|
||||
| Control plane | Record and manifest keys are erased before any guest action. | `owner.go:122-137` |
|
||||
| Namespace completion | Namespace deletion needs every aggregated API available; the e2e asserts `v1beta1.metrics.k8s.io` is `Available` first because otherwise the namespace stays `Terminating`. | `tests/e2e/eks/eks_test.go:197-231` |
|
||||
|
||||
`preserve` is ignored (record `eks-addon-lifecycle.md` row Desired state); for every bundle a preserve would need to keep the whole rendered object set and stop only the render and GC, so the per-bundle `preserve` rows below list what in addition would break or dangle.
|
||||
|
||||
## 4. `spinifex-noop` 0.1.0
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| Namespace | `spinifex-noop` | | `spinifex-noop/0.1.0/noop.yaml:5-10` |
|
||||
| ConfigMap | `spinifex-noop` | `spinifex-noop` | `noop.yaml:12-20` |
|
||||
|
||||
Images: none.
|
||||
Both objects carry `app.kubernetes.io/managed-by: spinifex-addon` (`noop.yaml:9-10`, `noop.yaml:17-18`); nothing reads the label.
|
||||
|
||||
### Identity
|
||||
|
||||
No IRSA markers, no ServiceAccount; a supplied `serviceAccountRoleArn` is stored and inert.
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | Ready once K3s records the two GVKs. This is the path the e2e exercises (`tests/e2e/eks/eks_test.go:158-195`). |
|
||||
| Fallback | Never succeeds: no workload kinds, so `length > 0` is false (`sync.sh:308`). |
|
||||
| Meaning of ready | Both objects applied; there is nothing to serve. |
|
||||
| False ready | None material. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | Namespace (cascading the ConfigMap) and ConfigMap. |
|
||||
| Left behind | The K3s `Addon` CR. |
|
||||
| Preserve | Keep both; nothing dangles. |
|
||||
| Completion signal | `kubectl get ns spinifex-noop` NotFound; this is exactly what the e2e asserts (`eks_test.go:221-231`). Blocked indefinitely by an unavailable aggregated API. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
Namespace name could collide with a user namespace of the same name, which GC would then delete.
|
||||
No configuration hook.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. The only bundle with removal proven end-to-end; its completion signal (namespace NotFound) does not generalise to bundles in `kube-system`.
|
||||
|
||||
## 5. `argocd` 3.0.23
|
||||
|
||||
### Objects
|
||||
|
||||
Single file `argocd/3.0.23/argocd.yaml` (26,819 lines), upstream `install.yaml` passed through kustomize with `namespace: argocd` (`argocd.yaml:1-11`).
|
||||
|
||||
| Kind | Count | Names | Namespace | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| Namespace | 1 | `argocd` (first document) | | `argocd.yaml:12-17` |
|
||||
| CustomResourceDefinition | 3 | `applications.argoproj.io`, `applicationsets.argoproj.io`, `appprojects.argoproj.io` | | `argocd.yaml:25`, `:5904`, `:23563` |
|
||||
| ServiceAccount | 7 | `argocd-application-controller`, `-applicationset-controller`, `-dex-server`, `-notifications-controller`, `-redis`, `-repo-server`, `-server` | `argocd` | parsed |
|
||||
| Role / RoleBinding | 6 / 6 | per component | `argocd` | parsed |
|
||||
| ClusterRole / ClusterRoleBinding | 3 / 3 | `argocd-application-controller`, `argocd-applicationset-controller`, `argocd-server` | | `argocd.yaml:24243-24554` |
|
||||
| ConfigMap | 7 | `argocd-cm`, `-cmd-params-cm`, `-gpg-keys-cm`, `-notifications-cm`, `-rbac-cm`, `-ssh-known-hosts-cm`, `-tls-certs-cm` | `argocd` | parsed |
|
||||
| Secret | 2 | `argocd-notifications-secret`, `argocd-secret` | `argocd` | `argocd.yaml:24740`, `:24751` |
|
||||
| Service | 8 | applicationset-controller, dex-server, metrics, notifications-controller-metrics, redis, repo-server, server, server-metrics | `argocd` | parsed |
|
||||
| Deployment | 6 | `argocd-applicationset-controller`, `-dex-server`, `-notifications-controller`, `-redis`, `-repo-server`, `-server` (default 1 replica) | `argocd` | `argocd.yaml:24925-25887` |
|
||||
| StatefulSet | 1 | `argocd-application-controller`, 1 replica | `argocd` | `argocd.yaml:26301-26310` |
|
||||
| NetworkPolicy | 7 | one per component | `argocd` | parsed |
|
||||
|
||||
Images: `quay.io/argoproj/argocd:v3.0.23` (8 containers), `ghcr.io/dexidp/dex:v2.41.1` (`argocd.yaml:25243`), `redis:7.2.11-alpine` (`argocd.yaml:25478`).
|
||||
Only the Namespace carries `app.kubernetes.io/managed-by: spinifex-addon`.
|
||||
|
||||
### Identity
|
||||
|
||||
No IRSA markers (`argocd.yaml:11`); a supplied role is stored and inert.
|
||||
All seven ServiceAccounts are created by the bundle.
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | Per the agent comment, K3s never records the `gvks` annotation for this bundle, so the primary path does not fire (`sync.sh:49-55`); why is not established. |
|
||||
| Fallback | After 90 s: all 6 Deployments available at 1 replica and the StatefulSet at 1 ready replica, among those `kubectl get -f` finds. |
|
||||
| Meaning of ready | Every returned workload rolled out once. CRDs are not checked for `Established`; an absent CRD or workload is dropped by `--ignore-not-found` rather than failing the check. |
|
||||
| False ready | A partial apply that left any document unapplied still reads ready if the remaining workloads roll out; if the missing `gvks` is caused by an apply error on one object, this is the case in practice (hypothesis, not verified). Dex and notifications are counted though unused by default. |
|
||||
| Runtime | Images pulled from three public registries; on an air-gapped site the bundle never rolls out and stays `CREATING`. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | Everything in the file, Namespace first. CRD deletion removes every `Application`, `ApplicationSet` and `AppProject` in the cluster, including the runtime `default` AppProject. |
|
||||
| Left behind | Workloads Argo CD deployed into other namespaces (user applications) unless their `Application` carries the cascade finalizer and the controller survives long enough to process it; runtime Secrets in `argocd` go with the namespace. |
|
||||
| Stuck risk | `Application` objects carrying `resources-finalizer.argocd.argoproj.io` (user-set, upstream behaviour) need the application controller to clear them; it is deleted in the same pass, so the CRs, the `applications` CRD and the `argocd` namespace can stay `Terminating` indefinitely (not verified live). |
|
||||
| Preserve | Keep all objects; the cluster-scoped ClusterRoles and CRDs then remain unowned. |
|
||||
| Completion signal | `argocd` namespace NotFound and the three CRDs NotFound and the three ClusterRoles and ClusterRoleBindings NotFound; namespace and CRD finalizers must be watched explicitly, since NotFound on the namespace alone does not cover cluster-scoped objects. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
A user-installed Argo CD (same CRDs, same ClusterRole names, namespace `argocd`) would be overwritten on create and wholly deleted on DeleteAddon; this is the `resolveConflicts` case.
|
||||
No configuration hook, although `argocd-cm`/`argocd-rbac-cm` are the natural targets.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. Readiness depends entirely on the fallback, which neither checks CRDs nor detects missing objects.
|
||||
2. Removal deletes user-owned `Application` data cluster-wide and can wedge on finalizers; nothing reports it.
|
||||
3. No e2e coverage at all.
|
||||
|
||||
## 6. `aws-ebs-csi-driver` 1.40.1
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| ServiceAccount | `ebs-csi-controller-sa` (role annotation) | `kube-system` | `aws-ebs-csi-driver/1.40.1/00-rbac.yaml:5-14` |
|
||||
| ServiceAccount | `ebs-csi-node-sa` | `kube-system` | `00-rbac.yaml:16-23` |
|
||||
| ClusterRole | `ebs-external-attacher-role`, `ebs-csi-node-role`, `ebs-external-provisioner-role`, `ebs-external-resizer-role`, `ebs-external-snapshotter-role` | | `00-rbac.yaml:25`, `:65`, `:94`, `:184`, `:241` |
|
||||
| ClusterRoleBinding | `ebs-csi-attacher-binding`, `ebs-csi-node-getter-binding`, `ebs-csi-provisioner-binding`, `ebs-csi-resizer-binding`, `ebs-csi-snapshotter-binding` | | `00-rbac.yaml:309-381` |
|
||||
| Role / RoleBinding | `ebs-csi-leases-role` / `ebs-csi-leases-rolebinding` | `kube-system` | `00-rbac.yaml:384-417` |
|
||||
| CSIDriver | `ebs.csi.aws.com` | | `00-rbac.yaml:421-430` |
|
||||
| PodDisruptionBudget | `ebs-csi-controller` (maxUnavailable 1) | `kube-system` | `00-rbac.yaml:432-443` |
|
||||
| Secret | `spinifex-gateway-ca` (shared name, see Conflicts) | `kube-system` | `10-controller.yaml:8-17` |
|
||||
| Deployment | `ebs-csi-controller`, 2 replicas, preferred anti-affinity, tolerates `CriticalAddonsOnly` and `nvidia.com/gpu` | `kube-system` | `10-controller.yaml:19-82` |
|
||||
| DaemonSet | `ebs-csi-node`, tolerates `nvidia.com/gpu` only | `kube-system` | `20-node.yaml:5-58` |
|
||||
| StorageClass | `ebs-gp3`, default class, `WaitForFirstConsumer` | | `30-storageclass.yaml:5-18` |
|
||||
|
||||
Images: `public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.40.1` (`10-controller.yaml:89`, `20-node.yaml:67`), `public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v5.2.0-eks-1-32-7` (`:147`), `external-attacher:v4.8.1-eks-1-32-7` (`:180`), `external-snapshotter/csi-snapshotter:v8.2.1-eks-1-32-7` (`:209`), `external-resizer:v1.13.2-eks-1-32-7` (`:238`), `livenessprobe:v2.14.0-eks-1-32-7` (`:270`, `20-node.yaml:155`), `node-driver-registrar:v2.13.0-eks-1-32-7` (`20-node.yaml:118`).
|
||||
The bundle header says it is vendored from upstream v1.41.0 while the catalogue version is 1.40.1 (`00-rbac.yaml:1-3`); the driver image is v1.40.1.
|
||||
|
||||
### Identity
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Takes `serviceAccountRoleArn` | Yes (`RequiresIRSA`), not required at create. |
|
||||
| Annotated SA | `ebs-csi-controller-sa` (`00-rbac.yaml:12-13`, quoted, so an empty role renders `''`). |
|
||||
| Injected pod | Controller Deployment only: `{{IRSA_ENV}}` (`10-controller.yaml:110`), `{{IRSA_VOLUME_MOUNT}}` (`:114`), `{{IRSA_VOLUME}}` (`:289`). The node DaemonSet gets no credentials and makes no EC2 calls (`20-node.yaml:1-3`). |
|
||||
| SA owner | The bundle creates both ServiceAccounts. |
|
||||
| Without a role | Controller uses node instance-profile credentials over IMDS. |
|
||||
| Evidence | `tests/e2e/eks/eks_test.go:658-800` installs it with a role, creates the OIDC provider and role, and drives a PVC through CreateVolume/AttachVolume. |
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | `ready` as soon as K3s records GVKs; neither replica need be running. The e2e waits for `ACTIVE` and then separately proves the data path (`eks_test.go:767-800`). |
|
||||
| Fallback | Controller Deployment 2 of 2 available and node DaemonSet `desired > 0` and all ready. |
|
||||
| Fallback false negative | A cluster with no worker node that tolerates scheduling (only the control plane, which carries `CriticalAddonsOnly` the DaemonSet does not tolerate) has `desired = 0`, so the fallback never passes. |
|
||||
| False ready | Controller pending or crash-looping (including STS web-identity failures from a bad trust policy), or sidecars failing, still `ACTIVE` via the primary path. The bundle ships no `VolumeSnapshot` CRDs while running `csi-snapshotter`; the sidecar's behaviour without them was not verified and nothing in readiness would show it. |
|
||||
| AWS comparison | `InsufficientNumberOfReplicas` and `AccessDenied` are the codes that would correspond to these states; neither is ever produced. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | All objects above, including the default StorageClass and the CSIDriver, in file order (RBAC and CSIDriver first, then Secret and controller, then node DaemonSet, then StorageClass). |
|
||||
| Left behind (in cluster) | PersistentVolumes and PVCs it provisioned; `VolumeAttachment` objects; runtime leader-election Leases in `kube-system`; `CSINode` driver entries until node plugins unregister. |
|
||||
| Left behind (outside cluster) | Spinifex EBS volumes backing those PVs, still allocated and possibly attached. |
|
||||
| Stuck risk | PVs with `reclaimPolicy: Delete` and VolumeAttachments carry CSI sidecar finalizers that only the deleted controller clears, so later PVC deletion or node drain hangs and the backing volume is never deleted (upstream sidecar behaviour, not verified on this build). |
|
||||
| Preserve | Keep the whole set; existing mounts and new provisioning keep working with the last-rendered role. |
|
||||
| Completion signal | All 20 rendered objects NotFound (all in `kube-system` or cluster-scoped, so there is no namespace to watch); a meaningful "removed" additionally needs zero `VolumeAttachment`s and PVs with `spec.csi.driver == ebs.csi.aws.com`, or an explicit statement that those are user data that outlive the add-on. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
| Object | Collides with |
|
||||
|---|---|
|
||||
| `kube-system/spinifex-gateway-ca` | The identical name in `aws-load-balancer-controller` (`10-aws-load-balancer-controller.yaml:33-41`). Deleting either add-on deletes the Secret the other mounts; K3s re-applies the survivor's file only when it changes, so the survivor's next pod start fails until then (re-apply behaviour not verified). A removal check by NotFound on this Secret also never completes while the other add-on re-creates it. |
|
||||
| `StorageClass/ebs-gp3` with `is-default-class` | Any user default StorageClass, giving two defaults; K3s `local-storage` is disabled (`scripts/images/eks-node/setup.sh:98-101`). |
|
||||
| `CSIDriver/ebs.csi.aws.com`, the five ClusterRoles, `ebs-csi-*` names | A self-managed (Helm) EBS CSI install, the `resolveConflicts` case; DeleteAddon would remove the user's driver too. |
|
||||
|
||||
No configuration hook; upstream exposes replica count, tolerations and StorageClass parameters through Helm values.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. `ACTIVE` does not imply a running controller or working credentials.
|
||||
2. Removal orphans PVs, VolumeAttachments and Spinifex volumes, and can wedge them behind finalizers.
|
||||
3. Shares `spinifex-gateway-ca` with the load balancer controller, so removal of either is not isolated.
|
||||
4. Bundle provenance comment (v1.41.0) disagrees with the catalogue version (1.40.1).
|
||||
|
||||
## 7. `aws-load-balancer-controller` 2.11.0
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| CustomResourceDefinition | `ingressclassparams.elbv2.k8s.aws` | | `aws-load-balancer-controller/2.11.0/00-crds.yaml:1-8` |
|
||||
| CustomResourceDefinition | `targetgroupbindings.elbv2.k8s.aws` | | `00-crds.yaml:256-263` |
|
||||
| ServiceAccount | `aws-load-balancer-controller` (role annotation, unquoted) | `kube-system` | `10-aws-load-balancer-controller.yaml:2-13` |
|
||||
| Secret | `aws-load-balancer-webhook-tls` (node-minted cert) | `kube-system` | `10-…yaml:17-28` |
|
||||
| Secret | `spinifex-gateway-ca` (shared name) | `kube-system` | `10-…yaml:32-41` |
|
||||
| Role / RoleBinding | `aws-load-balancer-controller-leader-election-role` / `-rolebinding` | `kube-system` | `10-…yaml:44-48`, `:242-254` |
|
||||
| ClusterRole / ClusterRoleBinding | `aws-load-balancer-controller-role` / `-rolebinding` | | `10-…yaml:85-89`, `:258-269` |
|
||||
| Service | `aws-load-balancer-webhook-service` | `kube-system` | `10-…yaml:273-277` |
|
||||
| Deployment | `aws-load-balancer-controller`, 1 replica, tolerates `CriticalAddonsOnly` and `nvidia.com/gpu` | `kube-system` | `10-…yaml:288-372` |
|
||||
| List → IngressClassParams | `alb` (subnets injected from `EKS_ELB_SUBNET_IDS`) | | `10-…yaml:375-386`, `sync.sh:197-215` |
|
||||
| List → IngressClass | `alb` | | `10-…yaml:388-400` |
|
||||
| MutatingWebhookConfiguration | `aws-load-balancer-webhook`: `mservice` (Services, all namespaces), `mpod` (Pods in namespaces labelled `elbv2.k8s.aws/pod-readiness-gate-inject`), `mtargetgroupbinding`; all `failurePolicy: Fail` | | `10-…yaml:402-488` |
|
||||
| ValidatingWebhookConfiguration | `aws-load-balancer-webhook`: `vingressclassparams`, `vtargetgroupbinding`, `vingress`; all `failurePolicy: Fail` | | `10-…yaml:489-564` |
|
||||
|
||||
Image: `public.ecr.aws/eks/aws-load-balancer-controller:v2.11.0` (`10-…yaml:320`).
|
||||
Webhook SANs come from `webhook.conf` (`aws-load-balancer-controller/2.11.0/webhook.conf`).
|
||||
|
||||
### Identity
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Takes `serviceAccountRoleArn` | Yes (`RequiresIRSA`), not required at create. |
|
||||
| Annotated SA | `aws-load-balancer-controller` (`10-…yaml:11-12`); the placeholder is unquoted, so the raw ARN becomes a YAML scalar and an empty role renders a null annotation value. |
|
||||
| Injected pod | Controller Deployment: `{{IRSA_ENV}}` (`:319`), `{{IRSA_VOLUME_MOUNT}}` (`:349`), `{{IRSA_VOLUME}}` (`:372`); STS, EC2, ELBv2 and ACM endpoints pinned to the gateway by args (`:309-314`). |
|
||||
| SA owner | The bundle. |
|
||||
| Without a role | Node instance-profile credentials over IMDS. |
|
||||
| Evidence | No e2e installs this add-on; IRSA is proven only for EBS CSI and a hand-written pod (`tests/e2e/eks/eks_test.go:234-247`). |
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | `ready` when GVKs are recorded; the controller need not be running. |
|
||||
| Fallback | The one Deployment at 1 available replica. |
|
||||
| False ready | Controller pending, crash-looping, or failing STS/ELBv2 calls is still `ACTIVE`. Worse than a passive false ready: with the controller absent, the six `failurePolicy: Fail` webhooks reject every Service create or update cluster-wide (`mservice` matches all namespaces except LBC's own objects) and every Ingress, so an `ACTIVE` add-on can break unrelated workloads. CRD `Established` is not checked. |
|
||||
| AWS comparison | `AdmissionRequestDenied` and `InsufficientNumberOfReplicas` are the matching issue codes; neither is produced. |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | CRDs first (00 file sorts first), then ServiceAccount, Secrets, RBAC, Service, Deployment, IngressClassParams, IngressClass, and the webhook configurations last. |
|
||||
| Left behind (in cluster) | User Ingresses and `LoadBalancer` Services it reconciled, with the controller's finalizers still set; the runtime leader Lease. |
|
||||
| Left behind (outside cluster) | Spinifex ELBv2 load balancers, target groups, listeners and any security groups the controller created for those Ingresses and Services. |
|
||||
| Stuck risk | Deleting the `targetgroupbindings` CRD waits on TargetGroupBindings whose finalizer only the controller clears; user Ingresses and Services keep finalizers no one removes, so they cannot be deleted (upstream controller behaviour, not verified on this build). Between the Deployment deletion and the webhook deletion, Service and Ingress writes fail; if the webhook delete errors, that state persists. |
|
||||
| Preserve | Keep the whole set; the webhook cert and gateway CA must stay valid (cert is 10-year). |
|
||||
| Completion signal | All rendered objects NotFound plus both CRDs gone (no `Terminating`); a meaningful "removed" also needs no remaining object carrying the controller's finalizers and no remaining Spinifex ELBv2 resources tagged for this cluster, which only the host side can observe. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
| Object | Collides with |
|
||||
|---|---|
|
||||
| `kube-system/spinifex-gateway-ca` | `aws-ebs-csi-driver` (Section 6). |
|
||||
| `IngressClass/alb`, `IngressClassParams/alb` | Any user IngressClass named `alb`, including a self-managed LBC. |
|
||||
| CRDs, ClusterRole, webhook configuration names | A self-managed LBC (Helm), the `resolveConflicts` case. |
|
||||
|
||||
No configuration hook; `EKS_ELB_SUBNET_IDS` is the only per-cluster input and it comes from the node environment, not from `configurationValues`.
|
||||
The design draft names this bundle as the first configuration target.
|
||||
|
||||
### Gaps
|
||||
|
||||
1. `ACTIVE` can coexist with fail-closed webhooks that have no backend, which breaks Service and Ingress writes cluster-wide.
|
||||
2. Removal orphans user Ingresses/Services behind finalizers and Spinifex ELBv2 resources outside the cluster; nothing in the guest can observe the latter.
|
||||
3. Shares `spinifex-gateway-ca` with EBS CSI.
|
||||
4. Unquoted role placeholder takes the raw ARN as YAML.
|
||||
5. No e2e coverage.
|
||||
|
||||
## 8. `nvidia-device-plugin` 0.17.4
|
||||
|
||||
### Objects
|
||||
|
||||
| Kind | Name | Namespace | Source |
|
||||
|---|---|---|---|
|
||||
| DaemonSet | `nvidia-device-plugin-daemonset`; nodeSelector `nvidia.com/gpu.present=true`; tolerates `nvidia.com/gpu`; `runtimeClassName: nvidia`; hostPath `/` read-only, `/var/lib/kubelet/device-plugins`, `/var/run/cdi` | `kube-system` | `nvidia-device-plugin/0.17.4/10-daemonset.yaml:14-75` |
|
||||
|
||||
Image: `nvcr.io/nvidia/k8s-device-plugin:v0.17.4` (`10-daemonset.yaml:41`).
|
||||
Depends on a `RuntimeClass` named `nvidia` and on the node label written by `mulga-eks-provider-id` (`10-daemonset.yaml:1-12`); neither is in the bundle.
|
||||
|
||||
### Identity
|
||||
|
||||
No IRSA markers and no ServiceAccount object: pods run as the `kube-system/default` ServiceAccount that Kubernetes creates.
|
||||
`EnsureGPUDevicePlugin` stages it with no role (`owner.go:79-89`); a user-created one may carry an inert role.
|
||||
|
||||
### Readiness
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Primary path | `ready` when K3s records the DaemonSet GVK, regardless of GPU nodes. |
|
||||
| Fallback | Requires `desiredNumberScheduled > 0`: never passes on a cluster with no GPU node. |
|
||||
| False ready | `ACTIVE` with zero GPU nodes (nothing serving); `ACTIVE` after every GPU node group is deleted; `FAIL_ON_INIT_ERROR=false` (`10-daemonset.yaml:46-47`) and no readiness probe mean a pod whose NVML init failed is Ready while advertising no `nvidia.com/gpu`. The e2e proves the real signal separately: DaemonSet `1=1` and node allocatable `nvidia.com/gpu=1` (`tests/e2e/gpu/eks_gpu_test.go:260-275`). |
|
||||
|
||||
### Removal
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Deleted | The DaemonSet. |
|
||||
| Left behind | CDI spec files the plugin wrote to host `/var/run/cdi`; node `nvidia.com/gpu` capacity until kubelet reconciles the unregistered plugin (kubelet behaviour not verified). |
|
||||
| Re-staging | Deleting it while GPU node groups exist leaves GPUs unschedulable until the next GPU node group create re-stages it; deleting the last GPU node group never unstages it (`eks-addon-lifecycle.md` row Cross-resource dependency). |
|
||||
| Preserve | Keep the DaemonSet; nothing dangles. |
|
||||
| Completion signal | DaemonSet NotFound; pods gone from GPU nodes. |
|
||||
|
||||
### Conflicts and configuration
|
||||
|
||||
A user-installed NVIDIA device plugin or GPU Operator would run a second plugin on the same nodes and contend for the kubelet device-plugin socket; the DaemonSet name matches the upstream static manifest, so a self-managed upstream install is overwritten.
|
||||
No configuration hook (device-list strategy, time-slicing and MIG are upstream options).
|
||||
|
||||
### Gaps
|
||||
|
||||
1. Ready is indifferent to whether any GPU is exposed; the workload fallback is unreachable on non-GPU clusters.
|
||||
2. Ownership is split between the auto-stager and the user, with no record of which.
|
||||
|
||||
## 9. Cross-bundle findings
|
||||
|
||||
| # | Finding | Bundles | Evidence |
|
||||
|---|---|---|---|
|
||||
| X1 | Readiness is "applied" on the primary path; workload state is never consulted while `gvks` is present, and runtime failure never yields `failed`. | all | `sync.sh:361-365`, `sync.sh:87`, `sync.sh:112` |
|
||||
| X2 | The fallback silently drops absent objects (`--ignore-not-found`) and never checks CRD `Established`. | argocd, LBC, EBS | `sync.sh:304-319` |
|
||||
| X3 | DaemonSet `desired > 0` makes the fallback unreachable for node-dependent bundles on clusters without matching nodes, while the primary path reports ready for the same state. | EBS, nvidia | `sync.sh:317` |
|
||||
| X4 | Removal selects only objects in the rendered file; runtime-created and user-created dependants (PVs, VolumeAttachments, Applications, Ingresses with finalizers, Leases) are not inventoried, and out-of-cluster resources (EBS volumes, ELBv2) are invisible to the guest. | EBS, LBC, argocd | `sync.sh:397-401` |
|
||||
| X5 | `kube-system/spinifex-gateway-ca` is rendered by two bundles under one name; either removal deletes the other's dependency, and a NotFound completion check on it is ill-defined while both exist. | LBC, EBS | `10-aws-load-balancer-controller.yaml:32-41`, `aws-ebs-csi-driver/1.40.1/10-controller.yaml:8-17` |
|
||||
| X6 | No bundle has a stable ownership label set: only noop and the argocd Namespace carry `managed-by: spinifex-addon`; LBC, EBS and nvidia carry only upstream `app.kubernetes.io/name` labels, which a self-managed install of the same software carries too. A label-based inventory or conflict check has nothing reliable to select on. | all | per-bundle sources |
|
||||
| X7 | Identity association is only the rendered pod env plus user-owned IAM objects; there is nothing Spinifex-owned to remove or observe, and the SA annotation is not acted on by any webhook. | LBC, EBS | Section 3.2 |
|
||||
| X8 | `RequiresIRSA` is advisory: no create-time check either way, so a role on argocd, nvidia or noop is stored and inert. | all | `addons.go:61-99`, `addons.go:255` |
|
||||
| X9 | The role ARN is substituted raw into sed and YAML; only `iam:PassRole` (which requires the role to exist) constrains it. | LBC, EBS | `sync.sh:231`, `passrole.go:31-37` |
|
||||
| X10 | `configurationValues` has no schema, no `DescribeAddonConfiguration`, and no render hook for any bundle. | all | `sync.sh:359` |
|
||||
| X11 | Removal runs only on the primary server and only from the local rendered file; a replaced primary has no file and removes nothing. | all | `eks-node-role.sh:138-142`, `sync.sh:386` |
|
||||
| X12 | E2E covers create, ready and delete only for noop, create and data path for EBS (delete only in cleanup, not asserted), and DaemonSet rollout for nvidia; argocd and LBC have none. | all | `tests/e2e/eks/eks_test.go:158-247`, `tests/e2e/gpu/eks_gpu_test.go:260-275` |
|
||||
|
||||
## 10. Per-bundle observable removal signal (summary)
|
||||
|
||||
| Bundle | Objects that must be NotFound | Finalizer or dependant to watch | Out-of-cluster residue |
|
||||
|---|---|---|---|
|
||||
| `spinifex-noop` | Namespace `spinifex-noop` | Namespace `kubernetes` finalizer (aggregated API availability) | none |
|
||||
| `argocd` | Namespace `argocd`, 3 CRDs, 3 ClusterRoles, 3 ClusterRoleBindings | `Application` resources finalizer; CRD and namespace `Terminating` | none from the bundle; user workloads it deployed |
|
||||
| `aws-ebs-csi-driver` | 20 objects, all `kube-system` or cluster-scoped | PV and VolumeAttachment CSI finalizers | Spinifex EBS volumes |
|
||||
| `aws-load-balancer-controller` | 2 CRDs plus 13 objects, webhooks included | Ingress, Service and TargetGroupBinding controller finalizers; CRD `Terminating` | Spinifex ELBv2 load balancers, target groups, security groups |
|
||||
| `nvidia-device-plugin` | 1 DaemonSet | none | host CDI spec files |
|
||||
@@ -0,0 +1,141 @@
|
||||
# EKS managed add-on: delivery executor and fencing inventory
|
||||
|
||||
Code read at 3a55e2ee1; not verified live.
|
||||
|
||||
This inventory supports the "Delivery executor" section (Section 10) and review decision R6 of `docs/package-boundary/eks-addon-lifecycle-design.md`, and the draft proposal PROP-SPINIFEXARCH-007 (`docs/development/proposals/spinifex-architecture/0007-contract-guest-and-persisted-state-evolution.md` in the mulga repo), which is not accepted.
|
||||
It records what the code does today and which evidence exists for choosing and fencing a guest-side delivery executor; it proposes no design.
|
||||
Paths are relative to `spinifex/` unless they start with `scripts/`, `images/`, `contracts/` or `tests/`, which are relative to the spinifex repo root.
|
||||
|
||||
## 1. Which guest process delivers add-ons
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Delivery process | `mulga-eks-addon-sync` (shell loop): fetch staged set, render each bundle into `/var/lib/rancher/k3s/server/manifests/spinifex-addon-<name>.yaml`, report a phase per add-on, GC rendered files whose add-on is no longer staged with `kubectl delete --wait=false`. | `scripts/images/eks-node/mulga-eks-addon-sync.sh:341-405` (sync), `:80-254` (render), `:383-404` (GC) |
|
||||
| Loop and cadence | Every `ADDON_SYNC_INTERVAL` (30 s) forever; one-shot when unset. | `mulga-eks-addon-sync.sh:409-417`, `mulga-eks-addon-sync.initd:18`, `images/mkosi.profiles/eks-server/mkosi.extra/etc/systemd/system/mulga-eks-addon-sync.service:13` |
|
||||
| Fetch failure | Logs and keeps the current render; no GC runs that tick, so already-rendered files stay in the auto-deploy dir. | `mulga-eks-addon-sync.sh:344-348` |
|
||||
| Report | `printf` JSON `{addon,version,phase,message,ts}` piped to `eks-gateway-publish -channel addon`; failure ignored. No executor identity, instance ID or epoch in the payload. | `mulga-eks-addon-sync.sh:68-73`, `contracts/eks/v1/addon_status.go:31-37` |
|
||||
| Local state | Per-add-on sentinels in `/tmp` (`.addon-firstapply-*`, `.addon-diag-*`) and webhook certs in `/var/lib/spinifex-eks/webhook-certs`; no record of which generation or executor rendered a file. | `mulga-eks-addon-sync.sh:47`, `:279-293`, `:326-328` |
|
||||
| Credentials | Sources `/etc/spinifex-eks/first-boot.env`; signs with `EKS_ACCESS_KEY`/`EKS_SECRET_KEY` when present, else the SDK chain reads IMDS instance-role credentials. | `mulga-eks-addon-sync.sh:19-33`, `handlers/eks/k3s_server_vm.go:549-557` |
|
||||
| OpenRC unit | `/etc/init.d/mulga-eks-addon-sync`, `after k3s`, refuses to start without `first-boot.env`. | `scripts/images/eks-node/mulga-eks-addon-sync.initd:1-30`, installed by `scripts/images/eks-node/manifest.conf:43-44` |
|
||||
| systemd unit | `mulga-eks-addon-sync.service`, `After=k3s.service`, `WantedBy=multi-user.target`; baked disabled by preset. | `images/mkosi.profiles/eks-server/mkosi.extra/etc/systemd/system/mulga-eks-addon-sync.service:1-17`, `images/mkosi.profiles/eks-server/mkosi.extra/usr/lib/systemd/system-preset/40-mulga-eks-role.preset:73` |
|
||||
| Bake-time enablement | OpenRC image enables only `crond eks-node-role mulga-ebs-byid mulga-eks-provider-id mulga-vpc-mtu`; systemd image enables only `eks-node-role` and the snapshot/otel units. | `scripts/images/eks-node/manifest.conf:83`, `40-mulga-eks-role.preset:59-75` |
|
||||
|
||||
## 2. How a VM decides it is the primary
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Role source | Host renders `SPINIFEX_K3S_ROLE=server` when `ServerURL` is empty and `server-join` otherwise; workers get `agent`. | `handlers/eks/k3s_server_vm.go:527-545`, `handlers/eks/nodegroup_userdata.go:64` |
|
||||
| Who gets `server` | Create: the first VM of the spread (`nodes[0]`) or the single CP; every other spread member gets `ServerURL` and so `server-join`. | `handlers/eks/k3s_ha_control_plane.go:290-323`, `:272-282`; test `k3s_ha_control_plane_test.go` `TestPlaceControlPlane_SpreadFirstInitsRestJoin` |
|
||||
| Selector | `eks-node-role.sh` runs once at first boot, writes `/etc/spinifex-eks/role`, enables and starts role services, then disables itself; a later boot exits early if the role file exists. | `scripts/images/eks-node/eks-node-role.sh:90-96`, `:118-122`, `:177` |
|
||||
| Primary-only service | Only the `server` branch enables and starts `mulga-eks-addon-sync` (and `k3s-first-boot`); `server-join` does not. | `eks-node-role.sh:124-150` vs `:151-166`; `scripts/images/eks-node/eks-node-role_test.sh:129`, `:162` |
|
||||
| Primacy is static | "Primary" is a first-boot, per-VM, irreversible choice; nothing in the guest re-evaluates it, and no kube Lease, k3s leader or etcd leader is consulted. | `eks-node-role.sh:90-96`; no lease/leader reference in `scripts/images/eks-node/*.sh` |
|
||||
| Can a non-primary run it | Not by any code path read: `server-join` never enables it, the preset and OpenRC runlevel bake it disabled. Only manual enablement inside the VM would start a second instance, and the gateway would admit it (Section 4). | `eks-node-role.sh:151-166`, `40-mulga-eks-role.preset:73`, `manifest.conf:83` |
|
||||
| K3s auto-deploy in HA | Each K3s server watches its own local `server/manifests` dir; whether the auto-deploy controller applies on every server or only on the elected K3s leader is unverified. If it runs only on the leader, a primary that is not the K3s leader would not apply. | not verified; `mulga-eks-addon-sync.sh:261-273` relies on the `Addon` CR the controller creates |
|
||||
|
||||
## 3. Control-plane membership on the host
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Record | `ClusterMeta` at `clusters/{name}/meta` in `eks-account-{account}` (history 1). | `handlers/eks/store.go:28-40`, `handlers/eks/cluster_state.go:64-201` |
|
||||
| Member list | `ControlPlaneNodes []ControlPlaneNode{NodeID, InstanceID, ENIID, ENIIP, MgmtIP}`; scalar `ControlPlaneInstanceID` etc. mirror `[0]`. No role, boot incarnation, executor or epoch field. | `cluster_state.go:109-125`, `:203-212` |
|
||||
| "Primary" in meta | Index `[0]` by comment and by the create path; the scalars mirror it. | `cluster_state.go:119-124`, `handlers/eks/service_impl.go:941-949` |
|
||||
| Membership predicate | `IsControlPlaneMember(instanceID)`: any entry in `ControlPlaneNodes` or the scalar. | `cluster_state.go:238-251` |
|
||||
| Template | `ControlPlaneTemplate` persisted with `ServerURL`, creds and per-node fields cleared, replayed for replacements. | `cluster_state.go:128-133`, `service_impl.go:951-963` |
|
||||
| Meta revision | Written by every `casUpdateMeta` caller, including health and node-count changes, so its KV revision is not a membership counter. | `cluster_state.go:276`, `:416-436`, `cluster_reconciler.go:620`, `:635` |
|
||||
|
||||
## 4. Gateway authentication of the internal routes
|
||||
|
||||
| Item | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| CP principal | Assumed-role session of role `spinifex-eks-server` in the system account; IMDS mints it with `RoleSessionName` = instance ID. | `handlers/eks/k3s_server_role.go:14`, `handlers/sts/assume_role.go:136-161`, `domains/ec2/guestmetadata/metadata.go:476-511` |
|
||||
| Role policy | `eks:PublishInternal`, `eks:ListInternalAddons`, `eks:WebhookTokenReview`, `eks:GetRecoveryDirective` on `Resource: "*"`; one role shared by every cluster's CP VMs. | `k3s_server_role.go:27`, `:42-43` |
|
||||
| Caller mapping | `eksCaller`: `SessionName` from `ctxIdentity`, role name from the underlying role ARN, not the session name. | `gateway/eks.go:307-320` |
|
||||
| `AuthorizeInternal` | Requires the CP principal class, then `meta.IsControlPlaneMember(caller.SessionName)` for the named cluster; `GetRecoveryDirective` additionally requires the path instance ID to equal the caller's. | `gateway/eks/internal_authz.go:44-96`; tests `gateway/eks_authz_test.go` `TestEKSRequest_InternalRoutesAdmitTheCPAgentPrincipal`, `gateway/eks/internal_authz_test.go` `TestAuthorizeInternal_RejectsClusterTheCallerDoesNotServe` |
|
||||
| Which routes it gates | Only actions scoped `sourceInternalCluster`: `ListInternalAddons` and `GetRecoveryDirective`. `PublishInternal` and `WebhookTokenReview` are `sourceBodyAccountCluster`, so `IsInternalAction` is false and only the IAM policy check applies. | `gateway/eks/authz.go:52-60`, `gateway/eks/internal_authz.go:37-39`, `gateway/eks.go:216-220`, `:248` |
|
||||
| Member vs primary | The gateway can tell members apart (instance ID) and bind a fetch to a cluster, but cannot identify the delivery executor: meta has no role field, and `[0]` stops being the `server`-role VM after `[0]` is replaced (Section 6). | `internal_authz.go:74`, `cluster_state.go:463-490` |
|
||||
| Static-cred fallback | When the instance profile cannot be ensured, the VM gets the shared system access key; it has no instance identity, so `ListInternalAddons` is denied for the VM's life while `PublishInternal` remains policy-gated only. | `k3s_server_role.go:30-50`, `k3s_ha_control_plane.go:161-166`, `k3s_server_vm.go:549-557` |
|
||||
| Handler-side identity | `ListStagedAddonManifests` receives only the cluster name; no caller identity reaches the daemon. | `gateway/eks/internal_addons.go:20-33`, `handlers/eks/addons.go:117-128`, `daemon/daemon.go:1116` |
|
||||
|
||||
## 5. Status relay path
|
||||
|
||||
| Hop | Present behaviour | Evidence |
|
||||
|---|---|---|
|
||||
| Agent | `eks-gateway-publish` wraps `{accountId, channel, kind, payload}` and POSTs `/clusters/{cluster}/internal-publish`, 30 attempts. | `agents/eks/gatewaypublish/publish.go:29`, `:57-114` |
|
||||
| Gateway authz | IAM policy against `arn:aws:eks:{region}:{bodyAccount}:cluster/{name}`; with the role's `Resource: "*"`, any CP-role session of any cluster can publish to any cluster's add-on subject. | `gateway/eks/authz.go:155-160`, `k3s_server_role.go:27` |
|
||||
| Relay | `PublishInternal` publishes the raw payload to core NATS `eks.addon.{account}.{cluster}.status`; only a trace header is added. No caller instance ID, session or epoch is attached. | `gateway/eks/internal_publish.go:48-91`, `contracts/eks/v1/addon_status.go:10-12` |
|
||||
| Subscriber | Lease holder's `ClusterReconciler.Run` subscribes plainly; decodes the report and calls `Owner.ApplyReport` with addon, phase and message only (version and ts dropped). | `handlers/eks/cluster_reconciler.go:447-461`, `handlers/eks/addon_status_reconciler.go:22-31` |
|
||||
| Apply | Revision CAS on the add-on record; no check of sender, membership, version or epoch. | `domains/eks/addon/owner.go:140-168`, `domains/eks/addon/store.go:198` |
|
||||
| State reports | `mulga-eks-state-report` runs on both server roles and uses the same unbound relay; the report carries no member identity and the reconciler keeps the newest. | `eks-node-role.sh:137`, `:160`, `handlers/eks/state_report.go:24-42`, `cluster_reconciler.go:535` |
|
||||
|
||||
## 6. Primary failure, replacement and rebuild today
|
||||
|
||||
| Event | What happens to add-on delivery | Evidence |
|
||||
|---|---|---|
|
||||
| Primary stopped or errored | Reconciler restarts it in place after `restartGrace` (same instance ID, same root volume); role file and enabled units persist, so `mulga-eks-addon-sync` resumes with its rendered files. | `cluster_reconciler.go:708-771`, `:927-937` |
|
||||
| Etcd reform (HA, all VMs running) | `members[0]` gets `cluster-reset`, others `wipe-rejoin`; members are stopped and restarted. Wipe removes only `server/db/etcd`, so rendered manifests survive. | `cluster_reconciler.go:780-880`, `scripts/images/eks-node/mulga-eks-k3s-recovery.sh:38`, `:183-186` |
|
||||
| HA member lost (terminated, gone, or describe error) | `ProvisionReplacementCP` launches with `ServerURL` set, so the replacement is `server-join` and never runs `mulga-eks-addon-sync`; `SwapControlPlaneMember` puts it at the dead member's index. | `cluster_reconciler.go:1010-1096`, `k3s_ha_control_plane.go:142-187`, `cluster_state.go:463-490` |
|
||||
| HA primary (`[0]`) lost | Same path: after the swap, `[0]` is a `server-join` VM and no member runs delivery. Staged manifests stay in KV, but nothing fetches, renders, GCs or reports until an operator intervenes; add-ons keep their last status. | as above; no other enable path for the service |
|
||||
| Describe error treated as lost | A member whose state query fails counts as lost and can be replaced after `replaceGrace` while still running; if it was the primary, it keeps rendering and reporting (Section 7). | `cluster_reconciler.go:982-1007`; test `cluster_reconciler_replace_test.go` `TestMaybeReplaceCP_DescribeErrorMemberTreatedAsLost` |
|
||||
| Single-CP total loss | Operator `RestoreSnapshot` launches a fresh `server` VM (`ServerURL` empty), so delivery resumes on the new VM from the KV-staged set; meta is committed before the old CP is terminated, and termination is retried 3 times. | `handlers/eks/restore_snapshot.go:217-353`, `k3s_ha_control_plane.go:206-246`, `restore_snapshot.go:28`, `:185-212` |
|
||||
| Cluster create timeout or bootstrap failure | Marked `FAILED` without purging; CP VMs keep running `mulga-eks-addon-sync`, `ListInternalAddons` still admits them (membership only), and the reconciler has exited, so reports are dropped. | `cluster_reconciler.go:1151-1167`, `service_impl.go:2138`, `cluster_reconciler.go:645-648` |
|
||||
| Async launch failure | `failClusterLaunch` purges infra (terminates CP VMs) and marks `FAILED`, keeping meta. | `service_impl.go:701-717` |
|
||||
| Reclaim of a `FAILED` name | CAS `FAILED`→`CREATING`, purge old infra, overwrite meta; add-on keys are not purged, so the new primary renders the inherited staged set. | `service_impl.go:1203-1261` |
|
||||
| Delete | `purgeClusterInfra` stops the reconciler and terminates the CP VMs. | `service_impl.go:1271-1272` |
|
||||
| Resume from durable state | A fresh `server` VM resumes because the staged set is durable KV and the listing has no per-executor cursor; a `server-join` replacement cannot resume because it never runs the agent. | `handlers/eks/addons.go:117-128`, `eks-node-role.sh:151-166` |
|
||||
|
||||
## 7. Existing leases, epochs and generations
|
||||
|
||||
| Mechanism | Scope | What it proves | Evidence |
|
||||
|---|---|---|---|
|
||||
| ClusterReconciler lease | Host daemon per cluster; key `{account}/{cluster}` in `spinifex-eks-leader`, value = daemon `HolderID`, CAS create, renew by revision CAS every 20 s, bucket TTL 60 s, `Lost()` ends `Run`. | One daemon reconciles at a time. The revision is private (no accessor) and is not passed to report apply or guest routes. Unrelated to the guest executor. | `cluster_reconciler.go:26`, `:391-418`, `:463-470`; `foundation/state/kvlease/lease.go:96-178`, `:201-225`; `store.go:38-40` |
|
||||
| Lease takeover | `SpawnRegisteredReconcilers` runs only at daemon start; a lease loser is not retried elsewhere in the code read. | Reports for a cluster whose lease holder died are dropped until some daemon restarts. | `daemon/daemon.go:2209`, `service_impl.go:385-428`, `handlers/eks/reconciler_registry.go:12-14`, `:166-171` |
|
||||
| Recovery directive epoch | Per member instance, `epoch = prev+1` on each set; guest applies once and records the applied epoch locally. | At-most-once boot-time directive. Not a fence: it is pulled only at boot and has no `disable delivery` action. | `handlers/eks/recovery_directive.go:30-43`, `:90-107`; `mulga-eks-k3s-recovery.sh:131-197` |
|
||||
| Per-cluster epoch or generation | None in `ClusterMeta`; add-on records have no generation. | Nothing. | `cluster_state.go:64-201`, `docs/package-boundary/eks-addon-lifecycle.md` row "Generation" |
|
||||
| K3s server roles | `server` (cluster-init) and `server-join`; fixed at first boot. | Identifies the delivery VM only until `[0]` is replaced. | `k3s_server_vm.go:527-531`, `:585-589` |
|
||||
| Etcd / kube leader election | K3s embedded etcd and kube controllers elect internally; nothing in Spinifex reads them. | Not used. | no reference in `scripts/images/eks-node/*.sh` or `handlers/eks` |
|
||||
|
||||
## 8. Candidate fencing evidence available today
|
||||
|
||||
"Fences" means the old executor provably cannot render or apply, and cannot have its reports accepted.
|
||||
|
||||
| Evidence | Available how | Proves old executor cannot render/apply? | Gap and what would be needed |
|
||||
|---|---|---|---|
|
||||
| EC2 state `terminated` or instance not found | `cpControlAdapter.InstanceState` (system account `DescribeInstances`); `TerminateK3sServerVM` treats not-found as success. | Yes for render and apply once the VM is gone, if the instance record is authoritative; host-partition behaviour of the state record is unverified. | Reports already queued or retried by `eks-gateway-publish` are still accepted. Needs: state proven authoritative under host loss, and report acceptance bound to the current executor. `service_impl.go:153-170`, `k3s_server_vm.go:292-330` |
|
||||
| EC2 state `stopped` / `error` | Same query. | Only momentarily: the reconciler's restart ladder starts stopped/error members automatically, and the VM resumes delivery with the same identity. | Needs a stop that nothing restarts, or a guest-side disable that survives restart. `cluster_reconciler.go:708-771`, `:927-937` |
|
||||
| Describe error | Treated as lost. | No: the VM may be running. | Must not count as fencing evidence. `cluster_reconciler.go:982-1007` |
|
||||
| Removal from `ControlPlaneNodes` | `SwapControlPlaneMember`, `replaceControlPlaneForRestore`. | Partly: `ListInternalAddons` is denied, but the agent keeps its current render on fetch failure, K3s keeps applying those files, and `PublishInternal` still accepts its reports. | Needs: publish gated by membership/executor and identity attached on relay; a guest rule that a denied fetch stops delivery. `internal_authz.go:74`, `mulga-eks-addon-sync.sh:344-348` |
|
||||
| Instance ID / incarnation | Session name = instance ID; stable across stop/start. | No: there is no per-boot incarnation, so a restarted VM is indistinguishable from before. | Needs an executor epoch recorded host-side and checked per request; instance ID alone identifies, it does not fence. `assume_role.go:136-161` |
|
||||
| KV revisions | Meta revision, add-on record revision, lease revision. | No: meta revision churns with health writes, add-on CAS ignores sender, lease revision is host-only and private. | Needs a dedicated, monotonic executor epoch written only on reassignment. `cluster_state.go:416-436`, `kvlease/lease.go:44-52` |
|
||||
| IMDS credential expiry | Instance-role creds last 3600 s, re-minted 5 min early. | No: up to an hour of validity after the last mint, and a running VM keeps re-minting. | Not a fence on its own. `domains/ec2/guestmetadata/credentials.go:14-20` |
|
||||
| Credential revocation | `RevokeRoleSessions` deletes sessions by role (called on role delete); no per-instance revocation. | No per-executor effect: the role is shared by every CP VM of every cluster. | Needs a per-session-name revocation, and proof the gateway rejects a revoked session; a running VM would still re-mint via IMDS unless IMDS also refuses it. `handlers/sts/session_credentials.go:212-219`, `handlers/iam/roles.go:221` |
|
||||
| Static system keys | Shared system access key on fallback. | No: shared, unrevocable per VM, no identity. | Fallback VMs cannot be executors under any identity-based rule. `k3s_server_role.go:30-50` |
|
||||
| K3s node removal / etcd member prune | `EKS_ETCD_PRUNE_PEER_IP` handled in `k3s-first-boot.sh`. | Would stop apply through the shared datastore if the member is removed, but not render to the VM's own disk. The prune path appears unreachable: the variable is set only on replacements, which are `server-join`, and `k3s-first-boot` runs only for `server`. | Needs a reachable prune, verification that a removed member's apiserver cannot write, and still the report gate. `k3s-first-boot.sh:165-187`, `k3s_ha_control_plane.go:153`, `eks-node-role.sh:136`, `:151-166` |
|
||||
| Guest service disable | None remotely; the only host-to-guest instruction channel is the boot-time recovery directive. | No. | Needs an online directive the agent polls, plus an acknowledgement the host can verify; an acknowledgement from the old executor proves only that it was alive to send it. `recovery_directive.go:12-28` |
|
||||
| Restore fence | `confirmOldCPTerminated` after meta commit. | Yes for the old VM once terminate succeeds, but it runs after the new VM was launched and committed. | Ordering is successor-first; R6 requires fence-first. `restore_snapshot.go:262-336` |
|
||||
|
||||
## 9. Evidence each design option would need
|
||||
|
||||
No option is chosen here; each row lists only what it would rely on.
|
||||
|
||||
| Option | Evidence it needs that does not exist today |
|
||||
|---|---|
|
||||
| Host-assigned executor `{instanceId, epoch}` on the cluster record | An executor field and epoch advanced by CAS; `AuthorizeInternal`-style gating on `PublishInternal` (today policy-only); the gateway attaching the caller instance ID (and epoch) to the relayed message; report apply checking it; a reassignment trigger that does not fire on describe errors; fence evidence from Section 8 rows that say "Yes". |
|
||||
| Executor follows meta `[0]` | A guest that can become executor after first boot (today role is fixed and replacements are `server-join`); the same gateway and relay identity checks; `[0]` reassignment tied to fencing of the old `[0]`. |
|
||||
| Run the agent on every server with guest-side leader election (kube Lease) | Host-visible proof of the guest leader (the host cannot read the kube Lease today); verification of K3s auto-deploy behaviour across servers; reports still need gateway-attached identity, since a partitioned old leader can still publish. |
|
||||
| Disable-on-handover directive | An online directive channel and agent polling; a rule that a fetch denial or missing directive stops rendering; an acknowledgement or instance-state proof before the successor acts. |
|
||||
| Credential-based fence | Per-instance session revocation, IMDS refusing to re-mint for a fenced instance, and gateway rejection of revoked sessions; still insufficient for files already rendered to the old VM's disk. |
|
||||
|
||||
## 10. Gaps
|
||||
|
||||
1. Delivery runs only on the first-boot `server` VM; replacing HA `[0]` leaves no member running `mulga-eks-addon-sync`, with no detection or recovery.
|
||||
2. `PublishInternal` is not gated by `AuthorizeInternal`; with the CP role's `Resource: "*"`, any CP-role session can publish add-on (and state) reports for any cluster and account.
|
||||
3. The relay attaches no caller identity, and the payload carries none, so the owner cannot tell which member, executor or epoch produced a report; this conflicts with the direction in draft PROP-SPINIFEXARCH-007 S3, which is not accepted.
|
||||
4. `ClusterMeta` has no role, executor or epoch field; `[0]` as "primary" is a convention that diverges from the guest role after a swap.
|
||||
5. A member removed from meta is denied fetches but keeps applying its last render and keeps having its reports accepted.
|
||||
6. Describe errors count as member loss, so a running primary can be replaced without fencing.
|
||||
7. The restore path launches and commits the successor before fencing the old CP.
|
||||
8. No per-instance credential revocation; the CP role and instance profile are shared across all clusters.
|
||||
9. `FAILED` clusters (create timeout, bootstrap failure) keep running CP VMs whose agent is still admitted while reports are dropped.
|
||||
10. The etcd peer prune for replacements appears unreachable (`server-join` does not run `k3s-first-boot`); not verified live.
|
||||
11. Whether K3s auto-deploy applies from a non-leader server's manifests dir is unverified, which bears on any multi-server option.
|
||||
12. No lease takeover for a cluster whose reconciler holder died until a daemon restarts, so reports are dropped meanwhile.
|
||||
@@ -0,0 +1,86 @@
|
||||
# EKS add-on v1 compatibility: who would drop, erase or ignore added fields
|
||||
|
||||
Code read at 3a55e2ee1; not verified live.
|
||||
|
||||
Question: if a newer binary adds JSON fields (for example `incarnationId`, `generation`, `observedGeneration`, `origin`, `executorEpoch`) to the add-on record `clusters/{c}/addons/{a}`, the staged manifest `clusters/{c}/addons/{a}/manifest`, or a `contracts/eks/v1` payload, what does an older binary still running the code below do with them?
|
||||
|
||||
Verdicts:
|
||||
|
||||
- **DROP**: decodes into a Go struct and writes the re-encoded struct back, erasing undeclared fields (`encoding/json` keeps no unknown-field bag).
|
||||
- **OVERWRITE**: writes a freshly built value with no read of the stored bytes; whatever was stored is replaced.
|
||||
- **ERASE**: deletes the key; content is irrelevant.
|
||||
- **IGNORE**: read-only decode; unknown fields are skipped and nothing is written back.
|
||||
- **FAIL**: decode error when a known field's JSON type changes (not an added field).
|
||||
|
||||
## Version fences that exist today
|
||||
|
||||
| Fence | Where | Effect |
|
||||
|---|---|---|
|
||||
| Record schema/version field | none: `spinifex/domains/eks/addon/record.go:24` | No per-record version, so an old binary cannot tell a newer record from a v1 one. |
|
||||
| Manifest schema/version field | none: `spinifex/domains/eks/addon/record.go:40` | Same for the manifest. |
|
||||
| Per-account bucket schema stamp | `spinifex/handlers/eks/store.go:29` (`KVBucketEKSAccountVersion = 1`), checked at `store.go:203` | Bumping it makes every older binary refuse the whole `eks-account-{id}` bucket with `SchemaAheadError` (`spinifex/foundation/state/migrate/kv.go:87`), i.e. all EKS calls for that account, not just add-ons. |
|
||||
| Registered migrations for the bucket | none | `RegisterKV` is keyed by exact bucket name (`migrate/kv.go:48`, lookup at `kv.go:94`) while EKS buckets are named per account (`store.go:146`). |
|
||||
| Wire version | `contracts/eks/v1/addon_status.go:1-3` package doc | Says generation-aware delivery needs a successor version; nothing on the wire carries a version marker. |
|
||||
|
||||
## Record and manifest writers
|
||||
|
||||
| Writer | file:line | Record | Manifest |
|
||||
|---|---|---|---|
|
||||
| `Owner.Create` (via `CreateAddon` `handlers/eks/addons.go:84`) | `domains/eks/addon/owner.go:48`; `Get` then unconditional `put` `store.go:39,51` | OVERWRITE (the Get/Put pair is not `kv.Create`, so a record created between them is replaced wholesale) | OVERWRITE via `StagingInstaller.Install` `delivery.go:39` → `putManifest` `store.go:153` (built from the four v1 fields) |
|
||||
| `Owner.EnsureGPUDevicePlugin` (via `nodegroup.go:411`) | `owner.go:79` | IGNORE when a record exists (`ErrExists` swallowed); else as Create | as Create |
|
||||
| `Owner.Update` (via `UpdateAddon` `addons.go:181`) | `owner.go:94` → `casUpdate` `store.go:198` (Unmarshal `:209`, Marshal `:215`, `kv.Update` `:219`) | DROP | OVERWRITE via Install `owner.go:113` → `putManifest` |
|
||||
| `Owner.ApplyReport` (via reconciler `handlers/eks/addon_status_reconciler.go:23`) | `owner.go:140` → `casUpdate` | DROP when status or health changes; IGNORE when the report changes nothing (mutate returns false, no write) | untouched |
|
||||
| `Owner.markFailed` (staging failure in Create/Update) | `owner.go:171` → `casUpdate` | DROP | untouched |
|
||||
| `Owner.Delete` (via `DeleteAddon` `addons.go:211`) | `owner.go:122` → Uninstall `delivery.go:51`, `deleteRecord` `store.go:181,189` | ERASE | ERASE (`deleteManifest` `store.go:172`) |
|
||||
| `DeleteClusterPrefix` (DeleteCluster `service_impl.go:1153`, DELETING reaper `eks_deleting_reaper.go:141`, both via `purgeClusterInfra` `service_impl.go:1451`) | `handlers/eks/cluster_state.go:493` | ERASE (every key under `clusters/{c}/`) | ERASE |
|
||||
| `purgeClusterInfra(deleteMeta=false)` (`service_impl.go:712,1251`) | `service_impl.go:1450` | not touched | not touched |
|
||||
| Account teardown | `spinifex/accountteardown/reapers_eks.go:103` | ERASE, only indirectly through `DeleteCluster`; no add-on reaper and no EKS bucket delete | ERASE (same path) |
|
||||
| `TagResource` / `UntagResource` on an add-on ARN | `service_impl.go:1772,1812,1855` | not a writer: add-on ARNs return `NotImplemented`; create-time tags live only in `Record.Tags` | n/a |
|
||||
|
||||
## Record and manifest readers
|
||||
|
||||
| Reader | file:line | Verdict |
|
||||
|---|---|---|
|
||||
| `addon.Get` (DescribeAddon `addons.go:152`, Create existence check, Delete) | `store.go:58,70` | IGNORE; FAIL on a type change |
|
||||
| `addon.List` (ListAddons `addons.go:26`) | `store.go:78,108` | IGNORE; one undecodable record fails the whole listing |
|
||||
| `addon.ListManifests` (`ListStagedAddonManifests` `addons.go:125`) | `store.go:119,141` | IGNORE; one undecodable manifest fails delivery for every add-on in the cluster |
|
||||
| `addonRecordToAWS` | `addons.go:222` | IGNORE (projects v1 fields onto the SDK shape) |
|
||||
| `ListStagedAddonManifests` reply | `addons.go:129-137` | Projects the four v1 fields field-by-field, so unknown manifest fields never leave the daemon even if `addon.Manifest` gained them |
|
||||
|
||||
## contracts/eks/v1 consumers
|
||||
|
||||
| Consumer | file:line | Mechanism | Added field in a newer payload |
|
||||
|---|---|---|---|
|
||||
| Gateway NATS client for `eks.ListStagedAddonManifests` | `handlers/eks/service_nats.go:128`; `foundation/messaging/nats/nats.go:316` | `encoding/json` into `ListStagedAddonManifestsOutput` | DROP on relay: an older gateway re-renders only the fields its `StagedAddonManifest` declares |
|
||||
| Gateway `ListInternalAddons` response | `gateway/eks/internal_addons.go:20,32`; route `gateway/eks.go:55`; `gateway/eks/handler.go:21-22` | `jsonutil.BuildJSON` (Go field names, json tags ignored) | Emits whatever fields its build declares; nothing to ignore |
|
||||
| `eks-gateway-fetch -resource addons` | `agents/eks/gatewayfetch/fetch.go:115-123` | `encoding/json` (case-insensitive) then fixed four-column printf | IGNORE: extra top-level or per-add-on fields never reach the TSV |
|
||||
| `mulga-eks-addon-sync.sh` sync loop | `scripts/images/eks-node/mulga-eks-addon-sync.sh:357` | `IFS=tab read -r _addon _version _role _cfg_b64` | A fifth TSV column would land inside `_cfg_b64` (last variable takes the rest), which is unused today (`:359`); inserting a column before column 4 would shift positions. Fetch binary and script ship in the same image, so they move together. |
|
||||
| `mulga-eks-addon-sync.sh` report | `mulga-eks-addon-sync.sh:67-72` | printf of five fixed keys | N/A for added fields: an older guest never emits them |
|
||||
| Gateway `PublishInternal` relay | `gateway/eks/internal_publish.go:30,82-85` | `Payload json.RawMessage` published verbatim | Passes added report fields through untouched |
|
||||
| Reconciler report subscriber | `handlers/eks/cluster_reconciler.go:449-455`; `state_report.go:69-72`; `addon_status_reconciler.go:22-27` | `encoding/json` into `AddonStatusReport`, then only `Addon`, `Phase`, `Message` reach the owner | IGNORE: a report carrying `incarnationId`/`generation` is applied exactly as today, so an older reconciler cannot reject a stale incarnation's report |
|
||||
| Phase / status enums | `owner.go:186-204` | `nextStatus` default branch | An unknown phase changes nothing; an unknown stored status decodes, and a `failed` report moves it to `CREATE_FAILED` |
|
||||
|
||||
## Consequences for an expand phase
|
||||
|
||||
- Every path that writes the record back (Update, ApplyReport with a change, markFailed) DROPs added fields, so while any older binary can serve an add-on write, added record fields are best-effort and can silently revert to absent.
|
||||
- Every Install rebuilds the manifest from the record, so an older binary's Create/Update erases added manifest fields too.
|
||||
- Delete and cluster deletion ERASE regardless of content, so a newer `origin`/`declined` marker would not stop an older binary deleting the record.
|
||||
- Reads and all wire consumers tolerate added fields; changing the JSON type of an existing field instead fails whole listings on older binaries.
|
||||
- The only hard fence is the per-account bucket stamp, which is all-or-nothing for EKS in that account.
|
||||
|
||||
## Characterisation tests pinning the above
|
||||
|
||||
| Test | file |
|
||||
|---|---|
|
||||
| `TestOwnerWrites_DropUnknownRecordAndManifestFields` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestOwnerApplyReport_NoOpKeepsUnknownFields` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestOwnerDelete_ErasesRecordAndManifestWithUnknownFields` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestReads_IgnoreUnknownFieldsWithoutRewriting` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestList_OneUndecodableEntryFailsTheListing` | `spinifex/domains/eks/addon/v1_compat_test.go` |
|
||||
| `TestAddonStatusReport_IgnoresUnknownFields` | `contracts/eks/v1/v1_compat_test.go` |
|
||||
| `TestInternalAddonsResponse_IgnoresUnknownFields` | `contracts/eks/v1/v1_compat_test.go` |
|
||||
| `TestEmitAddonsTSV_IgnoresUnknownFields` | `spinifex/agents/eks/gatewayfetch/v1_compat_test.go` |
|
||||
| `TestAddonStatusReport_UnknownFieldsAreAppliedAsToday` | `spinifex/handlers/eks/addon_v1_compat_test.go` |
|
||||
| `TestListStagedAddonManifests_ProjectsOnlyV1Fields` | `spinifex/handlers/eks/addon_v1_compat_test.go` |
|
||||
|
||||
Already covered elsewhere: cluster-prefix sweep of add-on keys (`handlers/eks/addon_lifecycle_test.go:306`), hand-written v1 record decode (`addon_lifecycle_test.go:85`), unknown phase is a no-op (`domains/eks/addon/owner_test.go:49`).
|
||||
@@ -0,0 +1,148 @@
|
||||
# Release, Version and Upgrade Mechanism Inventory
|
||||
|
||||
Code read at `3a55e2ee1`; not verified live.
|
||||
|
||||
This is a read-only inventory of the release-membership, rolling-upgrade, rollback, schema-version and guest-image mechanisms that exist today, measured against what the draft Prepare → Activate → Migrate → Finalise sequence and N/N-1 negotiation would need.
|
||||
The draft proposal (`docs/development/proposals/spinifex-architecture/0007-…`, unaccepted) and Q-75/Q-76 in `docs/specs/requirements/resource-lifecycle.md` are the yardstick only; nothing here treats them as decided.
|
||||
The open question it answers is the EKS add-on design's question 4 (`eks-addon-lifecycle-design.md:445`): is there an existing Region-wide release gate that can establish "every daemon runs the new release"?
|
||||
**Short answer: no.**
|
||||
|
||||
Path convention: paths without a prefix are relative to the `spinifex` repo root; `mulga/`, `predastore/`, `viperblock/` and `ansible/` paths are relative to the umbrella repo.
|
||||
Shell scripts, Ansible and docs were read, not run.
|
||||
|
||||
## 1. How a node advertises its build version
|
||||
|
||||
| Mechanism | Evidence | What it carries | Cluster-visible? |
|
||||
|---|---|---|---|
|
||||
| Build-time ldflags | `Makefile:60-62` | `git describe --tags --always --dirty` and short commit into `operator/cli.Version`/`Commit` | No; baked into the binary |
|
||||
| `spx version` | `spinifex/operator/cli/version.go:12-22` | Prints `Version (Commit) os/arch` of the local binary | No; local process only |
|
||||
| Gateway `GetVersion` | `spinifex/operator/cli/service.go:613`, `spinifex/runtime/roles/awsgw/awsgw.go:58-71,487-488`, `spinifex/gateway/spinifex.go:61-62`, `spinifex/gateway/spx/spx.go:27-39` | Version of whichever awsgw process answered the request | Only for the answering gateway; no per-node fan-out |
|
||||
| Daemon heartbeat (KV `spinifex-cluster-state`, key `heartbeat.<node>`) | `spinifex/daemon/jetstream.go:224-236,247`, `spinifex/daemon/heartbeat.go:56-72` | Node, config `Epoch`, services, capacity | Yes, but **no build version field** |
|
||||
| Node health (`spinifex.admin.<node>.health`, HTTPS `/health`) | `contracts/cluster/v1/health.go:15-24` | `ConfigHash`, `Epoch`, uptime, services | Yes, **no build version** |
|
||||
| Node status fan-out (`spinifex.node.status`, `spx get nodes`) | `contracts/cluster/v1/status.go:34-68`, `spinifex/gateway/spx/spx.go:47-60` | Capacity, roles, GPUs | Yes, **no build version** |
|
||||
| Node discovery (`spinifex.nodes.discover`) | `contracts/cluster/v1/discovery.go:14-16` | Node name only | Yes, no version |
|
||||
| Formation join | `spinifex/runtime/formation/formation.go:25-37,70-80` | Name, IPs, region/AZ, services | Join exchange has **no version or protocol field** |
|
||||
| `spinifex.toml` `version` / `epoch` | `spinifex/operator/cli/templates/spinifex.toml:2-3`, `spinifex/bootstrap/config/config.go:19-21`, `spinifex/daemon/daemon.go:2743-2757` | Config-format version ("4") and leader-bumped config epoch, hashed into `ConfigHash` | Indirectly via `ConfigHash`; this is config agreement, not binary version |
|
||||
| Deploy manifest file | `mulga/scripts/update-nodes.sh:165-189,320-321` | Five sub-repo describe strings, written to `/etc/spinifex/deploy-manifest.txt` and the operator's `~/.spinifex/deploy-manifests/` | No; a file on disk, only written by this dev script |
|
||||
|
||||
**Finding:** there is no cluster-wide record of which release each member runs.
|
||||
Every version observation is either local (`spx version`, deploy manifest) or names one process (`GetVersion`).
|
||||
The only cross-node comparison is done by shell scripts over SSH (Section 2).
|
||||
|
||||
## 2. How upgrades are performed today
|
||||
|
||||
| Path | Evidence | Ordering | Drain | Health gate | Version gate |
|
||||
|---|---|---|---|---|---|
|
||||
| Online installer re-run (`curl … \| bash`) | `docs/admin/update/README.md:31-53`, `scripts/setup.sh:828-847,1187-1194` | Per node, operator-driven; no cluster coordination | Whatever `spinifex.target` stop does | None in the script beyond service restart | Optional `--version` pin, SHA-256 verified; no peer comparison |
|
||||
| Manual: install binary, `spx admin upgrade`, restart | `docs/admin/update/README.md:64-100`, `spinifex/operator/cli/admin_upgrade.go:55-120` | Per node | None | None | None; applies local config migrations and systemd unit reconcile only |
|
||||
| `mulga/scripts/update-nodes.sh` (dev/test) | `mulga/scripts/update-nodes.sh:29-38,218-243,329-348,359-377` | **Stop-the-world**: `spx admin cluster shutdown`, update all, start all | Coordinated GATE→DRAIN→STORAGE→PERSIST→INFRA with all-node ACKs | Retries `spx get nodes` until no `NotReady` | Prints `spx version` per host after the fact; header states mixed NATS/predastore/OVN versions are deliberately avoided |
|
||||
| Ansible `cluster-deploy.yml` (dev clusters) | `ansible/playbooks/cluster-deploy.yml:1-37`, `ansible/roles/deploy/tasks/main.yml:97-109,372,390,443-459,516-534` | **Rolling** `serial: 1`, `max_fail_percentage: 0` | `systemctl stop spinifex.target` (drain via `spinifex-shutdown.service`) | awsgw port, target active, predastore Raft quorum rejoin, stale-inode restart | None; no `spx admin upgrade` step, so the cluster runs mixed builds mid-roll with no compatibility check |
|
||||
| `spinifex/scripts/install-node.sh` (formation) | `scripts/install-node.sh:9-11,286-304,347-354` | n/a; forms a cluster | n/a | Smoke test | Refuses to form if `spx version` strings differ (exact match, SSH) |
|
||||
| Coordinated shutdown | `spinifex/operator/cli/admin.go:105-112`, `spinifex/daemon/shutdown.go:29-45` | Phase order with ACKs | Yes | ACK per phase | None |
|
||||
| Local drain | `spinifex/operator/cli/admin.go:120-133` | Local node | GATE+DRAIN only | n/a | None |
|
||||
| `spx admin preflight` | `spinifex/operator/cli/admin_preflight.go:15-28`, `docs/admin/update/README.md:156-183` | Local | n/a | Detects a unit still executing a replaced `spx` inode | Local only; not a peer comparison |
|
||||
|
||||
The two multi-node paths disagree: `update-nodes.sh` avoids mixed versions by stopping the cluster, while `cluster-deploy.yml` rolls one node at a time and so runs N and N-1 together with nothing proving they interoperate.
|
||||
A known mixed-version hazard is already documented: an older daemon ignores the `Target` field of `ShutdownRequest` and treats a single-node drain as cluster-wide (`spinifex/daemon/shutdown.go:34-37`, `docs/admin/host-lifecycle/README.md:123`).
|
||||
RFC2-18 (`mulga/docs/specs/spinifex/rfc2-18_operator-lifecycle.md:34-53,88-104,165-177`, DRAFTING) records the same observations and states that after-the-fact version printing is not a compatibility gate.
|
||||
|
||||
## 3. Feature flags, capability negotiation, minimum-version gates
|
||||
|
||||
| Mechanism | Evidence | Scope | Shape |
|
||||
|---|---|---|---|
|
||||
| EBS provider capabilities | `spinifex/providers/ebs/provider.go:55-70`, `spinifex/services/viperblockd/provider_handlers.go:236-262`, `spinifex/daemon/daemon.go:2397` | Spinifex ↔ block provider | Boolean feature set fetched at runtime; the only real capability exchange found |
|
||||
| EBS provider wire version | `spinifex/providers/ebs/provider.go:13-25`, `spinifex/providers/ebs/errors.go:90-95`, `spinifex/providers/ebs/nats.go:212-316` | Every provider request/response | **Exact match** (`!= SchemaVersion` → `ErrUnsupportedVersion`); no N-1 window |
|
||||
| RDS data-volume contract image tag | `scripts/images/rds-postgres/manifest.conf:49-51`, `scripts/images/rds-mariadb/manifest.conf:59`, `spinifex/handlers/rds/launch.go:39-45,532`, `spinifex/operator/imagecatalog/images.go:666,686` | Control plane → guest image | AMI filter `rds-data-volume-contract=format-auth-v1` excludes older images; a static image-tag capability, not a runtime proof |
|
||||
| RDS bootstrap envelope | `spinifex/handlers/rds/bootstrap.go:23-36,190-192,212` | Host ↔ RDS guest | `envelopeVersion` exact match |
|
||||
| KV schema-ahead refusal | `spinifex/foundation/state/migrate/kv.go:55-67,87-89` | Any process opening a versioned bucket | Older build refuses to open a bucket stamped newer; fail-closed, not negotiation |
|
||||
| systemd unit marker | `spinifex/operator/host/systemd/reconcile.go:28-33,131-140`, `spinifex/operator/host/systemd/units_gen.go` | Host units | `# spinifex-unit-version: N`; replace only when older |
|
||||
|
||||
Searches for `featuregate`, `feature_flag`, `minVersion` (non-TLS), `minimumVersion`, `semver` and version handshakes in Spinifex found nothing else.
|
||||
There is no "all members at version ≥ X" check, no release gate record and no cluster capability set.
|
||||
|
||||
## 4. Persisted-state versions, migrations and rollback
|
||||
|
||||
### 4.1 KV bucket schema versions
|
||||
|
||||
| Item | Evidence | Notes |
|
||||
|---|---|---|
|
||||
| Version stamp | `spinifex/foundation/state/kvutil/state.go:3-4`, `spinifex/foundation/state/kvutil/kvutil.go:211-244` | `_version` key per bucket; `WriteVersion` is read-then-unconditional-put, not CAS |
|
||||
| Migration runner | `spinifex/foundation/state/migrate/kv.go:17-137` | Chain validation, run, stamp after each step; runs at bucket-open time in whichever process opens it |
|
||||
| Bucket-level versions | e.g. `spinifex/daemon/jetstream.go:53-55,139,154,187`, `spinifex/domains/ec2/vpc/service_impl.go:42-44,171-211`, `spinifex/handlers/rds/store.go:27-38,184,209` | About 40 constants, almost all `1`; IPAM, security groups and static pool are at `2` with **no registered migration** (`spinifex/domains/ec2/vpc/ipam.go:17`, `spinifex/domains/ec2/vpc/security_group.go:128`, `spinifex/domains/network/external/static_pool.go:22`) |
|
||||
| Registered migrations | `spinifex/daemon/instance_records_migrate.go:31-72`, `spinifex/migrate/VERSIONS.md` | Only `spinifex-instance-state` (1→5) and `spinifex-terminated-instances` (1→3) |
|
||||
| Catalogue | `spinifex/migrate/VERSIONS.md` | Lists three KV buckets; the other constants are not catalogued |
|
||||
|
||||
A bucket stamped `1` whose target became `2` with no registered step fails `RunKV` with "no migrations registered" (`spinifex/foundation/state/migrate/kv.go:110-111`); `VERSIONS.md` says those installs required `spx admin init --force`, which is a hard cutover with no documented Q-75 exception record.
|
||||
|
||||
### 4.2 Expand/contract precedent
|
||||
|
||||
The instance-record crossing is the only expand–migrate–contract precedent found.
|
||||
`VERSIONS.md` states version 5 of `spinifex-instance-state` froze the legacy `instance.<id>` and `node.<id>` keys and left them in place so the crossing can be rolled back, and the tests cover idempotent re-run and not overwriting fresher records (`spinifex/daemon/instance_records_transition_test.go:221-315`).
|
||||
It is not a full Prepare→Activate→Finalise sequence: activation happens when the first upgraded process opens the bucket, the bump itself makes the older build refuse to start (`SchemaAheadError`), and there is no operator finalisation step that deletes the frozen keys.
|
||||
So "rollback" means a binary downgrade is refused rather than supported; it would need the version stamp to be reset by hand (not documented).
|
||||
|
||||
### 4.3 Record- and file-level schema versions
|
||||
|
||||
| Format | Evidence | Rule |
|
||||
|---|---|---|
|
||||
| EBS volume/snapshot/AMI metadata documents | `spinifex/domains/ec2/ebs/metadata/metadata.go:14-17,166-221` | `schema_version` = 2; decode rejects anything `!= 2` as corrupt, so N-1 cannot read N's documents and vice versa |
|
||||
| Node local state (disk + KV `node.<id>`) | `spinifex/daemon/local_state.go:14-34,131-133` | Exact match `1` |
|
||||
| Config files | `spinifex/migrate/VERSIONS.md`, `spinifex/migrate/migrate.go:191-246,327-332` | Per-file version; `RunConfig` backs up before each step; no config migrations registered |
|
||||
| Object-store migrations | `spinifex/migrate/migrate.go:126-161`, `spinifex/migrate/VERSIONS.md` | Stamp in a KV bucket, read-then-write; none registered |
|
||||
| Predastore cluster config | `predastore/internal/config/config.go:25-28,358` | `Version = 1`, exact match, "no migration between versions" |
|
||||
| Predastore placement record | `predastore/internal/gate/handlers/placement_record.go:17-48,85,125-139` | Readers accept v1–v3 and reject unknown; writer always emits v3 (not "oldest compatible") |
|
||||
| Viperblock WAL/chunk/checkpoint headers | `viperblock/viperblock/snapshot.go:346-349`, `viperblock/viperblock/viperblock.go:3622,4788,6532` | Magic + `Version` uint16, exact match |
|
||||
| Viperblock flat-section version byte | `viperblock/viperblock/snapshot.go:634,726-729` | Reserved, not checked |
|
||||
|
||||
No resource record carries a per-record `schemaVersion` today; the EKS add-on design proposes adding one (`eks-addon-lifecycle-design.md:119,421`).
|
||||
|
||||
### 4.4 Rollback
|
||||
|
||||
No rollback command, procedure or test exists for the platform release.
|
||||
`spx admin upgrade` has no reverse mode (`spinifex/operator/cli/admin_upgrade.go:55-71`); a downgraded unit marker is reported as a conflict rather than restored (`spinifex/operator/host/systemd/reconcile.go:133-140`); config backups are files, not managed restore points (RFC2-18 C-127 at `mulga/docs/specs/spinifex/rfc2-18_operator-lifecycle.md:97-104`).
|
||||
The update runbook covers forward upgrade and stale-binary troubleshooting only (`docs/admin/update/README.md:31-213`).
|
||||
|
||||
## 5. Guest images and agents
|
||||
|
||||
| Guest | Image version source | Agent version | Control plane records | Upgrade path |
|
||||
|---|---|---|---|---|
|
||||
| EKS node (CP and worker) | `scripts/images/eks-node/manifest.conf:2,4,25,87-88`, `scripts/images/eks-node/setup.sh:16` (K3s `v1.32.5+k3s1`), catalogue `spinifex/operator/imagecatalog/images.go:576-590` (Version = Alpine `3.21.7`, mutable URL, sidecar SHA-256) | Guest binaries built without version ldflags; no guest reports one | `ClusterMeta.Version` = requested Kubernetes version, default `1.32`, not validated against the image (`spinifex/handlers/eks/service_impl.go:291,587`); `ControlPlaneInstanceID` (`spinifex/handlers/eks/cluster_state.go:109`); nodegroup `ReleaseVersion` is a fixed placeholder `1.0.0-spinifex-eks-node` (`spinifex/handlers/eks/nodegroup.go:43-46,296-299`); `PlatformVersion` is constant `eks.1` (`spinifex/handlers/eks/service_impl.go:2188-2191`) | None: `UpdateClusterVersion`, `UpdateNodegroupVersion`, `ListUpdates` and `DescribeUpdate` have no gateway route at this revision (`spinifex/gateway/eks.go`, 31 routes, none match); plan only in `mulga/docs/development/feature/eks-cluster-version-upgrade.md` |
|
||||
| EKS add-on bundles | Baked per version into the image under `addons/<name>/<version>` (`scripts/images/eks-node/mulga-eks-addon-sync.sh:38,81-87`) | Add-on status report carries add-on version only (`contracts/eks/v1/addon_status.go:31-37`) | Add-on record `addonVersion` | A new add-on version needs a new image; v1 guest protocol has no protocol version or capability field |
|
||||
| ECS container instance | `scripts/images/ecs-agent/manifest.conf:4,23,40-41` | `version = "dev"` unless ldflags set (`cmd/ecs-agent/main.go:18-24`); `Makefile:78` sets it but the image manifest build does not, so imaged agents likely report `dev` (not verified live) | `AgentVersion` stored and echoed in `VersionInfo` (`spinifex/handlers/ecs/records.go:237`, `spinifex/handlers/ecs/instances.go:283,333`); never compared | Relaunch capacity; no agent-update path |
|
||||
| RDS (postgres, mariadb) | `scripts/images/rds-postgres/manifest.conf:4,19,47-51`, `scripts/images/rds-mariadb/manifest.conf:4,21,55-59`, tags `engine`, `engine-version`, `rds-data-volume-contract` | `version = "dev"` default (`cmd/rds-agent/main.go:21-23,72`); image build sets no ldflags | `Agent.AgentVersion`, `EngineVersion` recorded (`spinifex/handlers/rds/records.go:405-415`, `spinifex/handlers/rds/agent.go:189`); never compared; AMI ID used at launch is logged, not stored on the DB record (`spinifex/handlers/rds/launch.go:189,267,353`) | `ModifyDBInstance EngineVersion` returns unimplemented (`spinifex/handlers/rds/modify.go:477-478`) |
|
||||
|
||||
AMI selection for EKS, RDS and ECS takes the newest image matching tags (`spinifex/handlers/eks/k3s_server_vm.go:353-363`, `spinifex/handlers/rds/launch.go:552-558`, `spinifex/handlers/ecs/capacity.go:108-119`), so importing a new image silently changes what new resources receive while existing ones keep the old one.
|
||||
The control plane can find a guest's AMI only indirectly through the backing EC2 instance; it cannot learn the guest's protocol support from the guest.
|
||||
|
||||
## 6. Offline artefacts and backups
|
||||
|
||||
| Artefact | Evidence | Version field |
|
||||
|---|---|---|
|
||||
| EBS snapshot / AMI metadata | `spinifex/domains/ec2/ebs/metadata/metadata.go:57-58,79-80` | `schema_version` 2, exact match |
|
||||
| Viperblock snapshot checkpoint | `viperblock/viperblock/snapshot.go:346-349` | Header magic + version, exact match |
|
||||
| RDS DB snapshot record | `spinifex/handlers/rds/records.go:291-320` | `EngineVersion` only; no record schema version; data lives in an EC2 snapshot |
|
||||
| EKS etcd backups (`eks-backups-system` bucket) | `spinifex/handlers/eks/restore_snapshot.go:19-22,53-75` | Name-encoded tier and timestamp only; no K3s/etcd version recorded |
|
||||
| Account export | Not found | n/a |
|
||||
|
||||
No artefact declares a support or retirement window, and none records the Spinifex release that wrote it.
|
||||
|
||||
## 7. Gaps against Prepare → Activate → Migrate → Finalise and N/N-1
|
||||
|
||||
Stated as gaps only; no design is implied.
|
||||
|
||||
| # | Need | Current state | Gap |
|
||||
|---|---|---|---|
|
||||
| G1 | Authoritative record of each member's release | Heartbeat, health, status and formation carry no build version (Section 1) | No release-membership data source exists; activation evidence ("every writer can preserve the transitional form") cannot be produced |
|
||||
| G2 | Region-wide activation gate | None; only `install-node.sh` exact string match over SSH at formation | Nothing operator-visible records "activated"; the EKS add-on slice 7 would have to build it |
|
||||
| G3 | Finalisation step and preflight | Frozen instance keys are never retired; no finalise command | No operator-visible end of rollback for any change |
|
||||
| G4 | Controlled rollback | Schema-ahead refusal blocks downgrade; no reverse migration, no unit downgrade, no runbook | Rollback is unsupported once any bucket is bumped |
|
||||
| G5 | N/N-1 wire compatibility | Most contracts unversioned (C-9); versioned ones use exact match (EBS provider, RDS envelope, local state, EBS metadata, viperblock headers) | No contract implements a two-version window or "emit oldest compatible" |
|
||||
| G6 | Capability exchange | Only EBS provider booleans and an RDS image tag | No authenticated per-peer or per-guest capability record; no invalidation on replacement |
|
||||
| G7 | Fenced, owned migration | Migrations run on bucket open in any process; version stamp is read-then-put | No migration owner, fence or per-record progress; concurrent runners rely on idempotence |
|
||||
| G8 | Per-record schema version | Bucket-level only | Lazy per-record migration under a resource owner has no field to key on |
|
||||
| G9 | Coherent rolling procedure | Stop-the-world script vs `serial: 1` Ansible roll with no version or compatibility check; known mixed-version drain hazard | No supported rolling-upgrade contract with a stop-on-skew rule |
|
||||
| G10 | Guest version visibility | Agent versions recorded but default `dev` in images and never compared; EKS guests report none | Control plane cannot tell a v1 from a v2-capable guest except by a protocol it does not yet have |
|
||||
| G11 | Guest upgrade operations | EKS version/nodegroup updates unrouted; RDS engine upgrade unimplemented; newest-AMI-wins selection | No owned path to move a guest to a new image, and image choice is implicit |
|
||||
| G12 | Release manifest | `update-nodes.sh` writes a dev-only text manifest; installer pins one tarball | No immutable release manifest binding binary, contract versions and supported source versions (RFC2-18 Q-183 target) |
|
||||
| G13 | Artefact lifecycle | Snapshot/backup formats lack writer release and support window; etcd backups record no K3s version | Retained artefacts cannot be checked for usability before a format is retired |
|
||||
| G14 | Version catalogue accuracy | `VERSIONS.md` lists 3 of about 40 KV buckets | No single inventory of persisted formats to drive a finalisation preflight |
|
||||
Reference in New Issue
Block a user