# EKS managed add-on: delivery executor and fencing inventory

Code read at 3a55e2ee1; not verified live.

This inventory supports the "Delivery executor" section (Section 10) and review decision R6 of `docs/package-boundary/eks-addon-lifecycle-design.md`, and the draft proposal PROP-SPINIFEXARCH-007 (`docs/development/proposals/spinifex-architecture/0007-contract-guest-and-persisted-state-evolution.md` in the mulga repo), which is not accepted.
It records what the code does today and which evidence exists for choosing and fencing a guest-side delivery executor; it proposes no design.
Paths are relative to `spinifex/` unless they start with `scripts/`, `images/`, `contracts/` or `tests/`, which are relative to the spinifex repo root.

## 1. Which guest process delivers add-ons

| Item | Present behaviour | Evidence |
|---|---|---|
| Delivery process | `mulga-eks-addon-sync` (shell loop): fetch staged set, render each bundle into `/var/lib/rancher/k3s/server/manifests/spinifex-addon-<name>.yaml`, report a phase per add-on, GC rendered files whose add-on is no longer staged with `kubectl delete --wait=false`. | `scripts/images/eks-node/mulga-eks-addon-sync.sh:341-405` (sync), `:80-254` (render), `:383-404` (GC) |
| Loop and cadence | Every `ADDON_SYNC_INTERVAL` (30 s) forever; one-shot when unset. | `mulga-eks-addon-sync.sh:409-417`, `mulga-eks-addon-sync.initd:18`, `images/mkosi.profiles/eks-server/mkosi.extra/etc/systemd/system/mulga-eks-addon-sync.service:13` |
| Fetch failure | Logs and keeps the current render; no GC runs that tick, so already-rendered files stay in the auto-deploy dir. | `mulga-eks-addon-sync.sh:344-348` |
| Report | `printf` JSON `{addon,version,phase,message,ts}` piped to `eks-gateway-publish -channel addon`; failure ignored. No executor identity, instance ID or epoch in the payload. | `mulga-eks-addon-sync.sh:68-73`, `contracts/eks/v1/addon_status.go:31-37` |
| Local state | Per-add-on sentinels in `/tmp` (`.addon-firstapply-*`, `.addon-diag-*`) and webhook certs in `/var/lib/spinifex-eks/webhook-certs`; no record of which generation or executor rendered a file. | `mulga-eks-addon-sync.sh:47`, `:279-293`, `:326-328` |
| Credentials | Sources `/etc/spinifex-eks/first-boot.env`; signs with `EKS_ACCESS_KEY`/`EKS_SECRET_KEY` when present, else the SDK chain reads IMDS instance-role credentials. | `mulga-eks-addon-sync.sh:19-33`, `handlers/eks/k3s_server_vm.go:549-557` |
| OpenRC unit | `/etc/init.d/mulga-eks-addon-sync`, `after k3s`, refuses to start without `first-boot.env`. | `scripts/images/eks-node/mulga-eks-addon-sync.initd:1-30`, installed by `scripts/images/eks-node/manifest.conf:43-44` |
| systemd unit | `mulga-eks-addon-sync.service`, `After=k3s.service`, `WantedBy=multi-user.target`; baked disabled by preset. | `images/mkosi.profiles/eks-server/mkosi.extra/etc/systemd/system/mulga-eks-addon-sync.service:1-17`, `images/mkosi.profiles/eks-server/mkosi.extra/usr/lib/systemd/system-preset/40-mulga-eks-role.preset:73` |
| Bake-time enablement | OpenRC image enables only `crond eks-node-role mulga-ebs-byid mulga-eks-provider-id mulga-vpc-mtu`; systemd image enables only `eks-node-role` and the snapshot/otel units. | `scripts/images/eks-node/manifest.conf:83`, `40-mulga-eks-role.preset:59-75` |

## 2. How a VM decides it is the primary

| Item | Present behaviour | Evidence |
|---|---|---|
| Role source | Host renders `SPINIFEX_K3S_ROLE=server` when `ServerURL` is empty and `server-join` otherwise; workers get `agent`. | `handlers/eks/k3s_server_vm.go:527-545`, `handlers/eks/nodegroup_userdata.go:64` |
| Who gets `server` | Create: the first VM of the spread (`nodes[0]`) or the single CP; every other spread member gets `ServerURL` and so `server-join`. | `handlers/eks/k3s_ha_control_plane.go:290-323`, `:272-282`; test `k3s_ha_control_plane_test.go` `TestPlaceControlPlane_SpreadFirstInitsRestJoin` |
| Selector | `eks-node-role.sh` runs once at first boot, writes `/etc/spinifex-eks/role`, enables and starts role services, then disables itself; a later boot exits early if the role file exists. | `scripts/images/eks-node/eks-node-role.sh:90-96`, `:118-122`, `:177` |
| Primary-only service | Only the `server` branch enables and starts `mulga-eks-addon-sync` (and `k3s-first-boot`); `server-join` does not. | `eks-node-role.sh:124-150` vs `:151-166`; `scripts/images/eks-node/eks-node-role_test.sh:129`, `:162` |
| Primacy is static | "Primary" is a first-boot, per-VM, irreversible choice; nothing in the guest re-evaluates it, and no kube Lease, k3s leader or etcd leader is consulted. | `eks-node-role.sh:90-96`; no lease/leader reference in `scripts/images/eks-node/*.sh` |
| Can a non-primary run it | Not by any code path read: `server-join` never enables it, the preset and OpenRC runlevel bake it disabled. Only manual enablement inside the VM would start a second instance, and the gateway would admit it (Section 4). | `eks-node-role.sh:151-166`, `40-mulga-eks-role.preset:73`, `manifest.conf:83` |
| K3s auto-deploy in HA | Each K3s server watches its own local `server/manifests` dir; whether the auto-deploy controller applies on every server or only on the elected K3s leader is unverified. If it runs only on the leader, a primary that is not the K3s leader would not apply. | not verified; `mulga-eks-addon-sync.sh:261-273` relies on the `Addon` CR the controller creates |

## 3. Control-plane membership on the host

| Item | Present behaviour | Evidence |
|---|---|---|
| Record | `ClusterMeta` at `clusters/{name}/meta` in `eks-account-{account}` (history 1). | `handlers/eks/store.go:28-40`, `handlers/eks/cluster_state.go:64-201` |
| Member list | `ControlPlaneNodes []ControlPlaneNode{NodeID, InstanceID, ENIID, ENIIP, MgmtIP}`; scalar `ControlPlaneInstanceID` etc. mirror `[0]`. No role, boot incarnation, executor or epoch field. | `cluster_state.go:109-125`, `:203-212` |
| "Primary" in meta | Index `[0]` by comment and by the create path; the scalars mirror it. | `cluster_state.go:119-124`, `handlers/eks/service_impl.go:941-949` |
| Membership predicate | `IsControlPlaneMember(instanceID)`: any entry in `ControlPlaneNodes` or the scalar. | `cluster_state.go:238-251` |
| Template | `ControlPlaneTemplate` persisted with `ServerURL`, creds and per-node fields cleared, replayed for replacements. | `cluster_state.go:128-133`, `service_impl.go:951-963` |
| Meta revision | Written by every `casUpdateMeta` caller, including health and node-count changes, so its KV revision is not a membership counter. | `cluster_state.go:276`, `:416-436`, `cluster_reconciler.go:620`, `:635` |

## 4. Gateway authentication of the internal routes

| Item | Present behaviour | Evidence |
|---|---|---|
| CP principal | Assumed-role session of role `spinifex-eks-server` in the system account; IMDS mints it with `RoleSessionName` = instance ID. | `handlers/eks/k3s_server_role.go:14`, `handlers/sts/assume_role.go:136-161`, `domains/ec2/guestmetadata/metadata.go:476-511` |
| Role policy | `eks:PublishInternal`, `eks:ListInternalAddons`, `eks:WebhookTokenReview`, `eks:GetRecoveryDirective` on `Resource: "*"`; one role shared by every cluster's CP VMs. | `k3s_server_role.go:27`, `:42-43` |
| Caller mapping | `eksCaller`: `SessionName` from `ctxIdentity`, role name from the underlying role ARN, not the session name. | `gateway/eks.go:307-320` |
| `AuthorizeInternal` | Requires the CP principal class, then `meta.IsControlPlaneMember(caller.SessionName)` for the named cluster; `GetRecoveryDirective` additionally requires the path instance ID to equal the caller's. | `gateway/eks/internal_authz.go:44-96`; tests `gateway/eks_authz_test.go` `TestEKSRequest_InternalRoutesAdmitTheCPAgentPrincipal`, `gateway/eks/internal_authz_test.go` `TestAuthorizeInternal_RejectsClusterTheCallerDoesNotServe` |
| Which routes it gates | All four internal routes. `ListInternalAddons` and `GetRecoveryDirective` are scoped `sourceInternalCluster`; `PublishInternal` and `WebhookTokenReview` are scoped `sourceInternalBodyCluster`, binding the URL cluster and the body account (read by the same decoder the handler uses) to the caller's membership before policy evaluation. Before that correction both were policy-only. | `gateway/eks/authz.go`, `gateway/eks/internal_authz.go`; tests `gateway/eks_authz_test.go` `TestEKSRequest_PublishInternal*`, `TestEKSRequest_WebhookTokenReview*` |
| Member vs primary | The gateway can tell members apart (instance ID) and bind a fetch to a cluster, but cannot identify the delivery executor: meta has no role field, and `[0]` stops being the `server`-role VM after `[0]` is replaced (Section 6). | `internal_authz.go:74`, `cluster_state.go:463-490` |
| Static-cred fallback | When the instance profile cannot be ensured, the VM gets the shared system access key; it has no instance identity, so every internal route, including `PublishInternal` and `WebhookTokenReview`, is denied for the VM's life; this fallback is unsupported. | `k3s_server_role.go:30-50`, `k3s_ha_control_plane.go:161-166`, `k3s_server_vm.go:549-557` |
| Handler-side identity | `ListStagedAddonManifests` receives only the cluster name; no caller identity reaches the daemon. | `gateway/eks/internal_addons.go:20-33`, `handlers/eks/addons.go:117-128`, `daemon/daemon.go:1116` |

## 5. Status relay path

| Hop | Present behaviour | Evidence |
|---|---|---|
| Agent | `eks-gateway-publish` wraps `{accountId, channel, kind, payload}` and POSTs `/clusters/{cluster}/internal-publish`, 30 attempts. | `agents/eks/gatewaypublish/publish.go:29`, `:57-114` |
| Gateway authz | IAM policy against `arn:aws:eks:{region}:{bodyAccount}:cluster/{name}`; with the role's `Resource: "*"`, any CP-role session of any cluster can publish to any cluster's add-on subject. | `gateway/eks/authz.go:155-160`, `k3s_server_role.go:27` |
| Relay | `PublishInternal` publishes the raw payload to core NATS `eks.addon.{account}.{cluster}.status`; only a trace header is added. No caller instance ID, session or epoch is attached. | `gateway/eks/internal_publish.go:48-91`, `contracts/eks/v1/addon_status.go:10-12` |
| Subscriber | Lease holder's `ClusterReconciler.Run` subscribes plainly; decodes the report and calls `Owner.ApplyReport` with addon, phase and message only (version and ts dropped). | `handlers/eks/cluster_reconciler.go:447-461`, `handlers/eks/addon_status_reconciler.go:22-31` |
| Apply | Revision CAS on the add-on record; no check of sender, membership, version or epoch. | `domains/eks/addon/owner.go:140-168`, `domains/eks/addon/store.go:198` |
| State reports | `mulga-eks-state-report` runs on both server roles and uses the same unbound relay; the report carries no member identity and the reconciler keeps the newest. | `eks-node-role.sh:137`, `:160`, `handlers/eks/state_report.go:24-42`, `cluster_reconciler.go:535` |

## 6. Primary failure, replacement and rebuild today

| Event | What happens to add-on delivery | Evidence |
|---|---|---|
| Primary stopped or errored | Reconciler restarts it in place after `restartGrace` (same instance ID, same root volume); role file and enabled units persist, so `mulga-eks-addon-sync` resumes with its rendered files. | `cluster_reconciler.go:708-771`, `:927-937` |
| Etcd reform (HA, all VMs running) | `members[0]` gets `cluster-reset`, others `wipe-rejoin`; members are stopped and restarted. Wipe removes only `server/db/etcd`, so rendered manifests survive. | `cluster_reconciler.go:780-880`, `scripts/images/eks-node/mulga-eks-k3s-recovery.sh:38`, `:183-186` |
| HA member lost (terminated, gone, or describe error) | `ProvisionReplacementCP` launches with `ServerURL` set, so the replacement is `server-join` and never runs `mulga-eks-addon-sync`; `SwapControlPlaneMember` puts it at the dead member's index. | `cluster_reconciler.go:1010-1096`, `k3s_ha_control_plane.go:142-187`, `cluster_state.go:463-490` |
| HA primary (`[0]`) lost | Same path: after the swap, `[0]` is a `server-join` VM and no member runs delivery. Staged manifests stay in KV, but nothing fetches, renders, GCs or reports until an operator intervenes; add-ons keep their last status. | as above; no other enable path for the service |
| Describe error treated as lost | A member whose state query fails counts as lost and can be replaced after `replaceGrace` while still running; if it was the primary, it keeps rendering and reporting (Section 7). | `cluster_reconciler.go:982-1007`; test `cluster_reconciler_replace_test.go` `TestMaybeReplaceCP_DescribeErrorMemberTreatedAsLost` |
| Single-CP total loss | Operator `RestoreSnapshot` launches a fresh `server` VM (`ServerURL` empty), so delivery resumes on the new VM from the KV-staged set; meta is committed before the old CP is terminated, and termination is retried 3 times. | `handlers/eks/restore_snapshot.go:217-353`, `k3s_ha_control_plane.go:206-246`, `restore_snapshot.go:28`, `:185-212` |
| Cluster create timeout or bootstrap failure | Marked `FAILED` without purging; CP VMs keep running `mulga-eks-addon-sync`, `ListInternalAddons` still admits them (membership only), and the reconciler has exited, so reports are dropped. | `cluster_reconciler.go:1151-1167`, `service_impl.go:2138`, `cluster_reconciler.go:645-648` |
| Async launch failure | `failClusterLaunch` purges infra (terminates CP VMs) and marks `FAILED`, keeping meta. | `service_impl.go:701-717` |
| Reclaim of a `FAILED` name | CAS `FAILED`→`CREATING`, purge old infra, overwrite meta; add-on keys are not purged, so the new primary renders the inherited staged set. | `service_impl.go:1203-1261` |
| Delete | `purgeClusterInfra` stops the reconciler and terminates the CP VMs. | `service_impl.go:1271-1272` |
| Resume from durable state | A fresh `server` VM resumes because the staged set is durable KV and the listing has no per-executor cursor; a `server-join` replacement cannot resume because it never runs the agent. | `handlers/eks/addons.go:117-128`, `eks-node-role.sh:151-166` |

## 7. Existing leases, epochs and generations

| Mechanism | Scope | What it proves | Evidence |
|---|---|---|---|
| ClusterReconciler lease | Host daemon per cluster; key `{account}/{cluster}` in `spinifex-eks-leader`, value = daemon `HolderID`, CAS create, renew by revision CAS every 20 s, bucket TTL 60 s, `Lost()` ends `Run`. | One daemon reconciles at a time. The revision is private (no accessor) and is not passed to report apply or guest routes. Unrelated to the guest executor. | `cluster_reconciler.go:26`, `:391-418`, `:463-470`; `foundation/state/kvlease/lease.go:96-178`, `:201-225`; `store.go:38-40` |
| Lease takeover | `SpawnRegisteredReconcilers` runs only at daemon start; a lease loser is not retried elsewhere in the code read. | Reports for a cluster whose lease holder died are dropped until some daemon restarts. | `daemon/daemon.go:2209`, `service_impl.go:385-428`, `handlers/eks/reconciler_registry.go:12-14`, `:166-171` |
| Recovery directive epoch | Per member instance, `epoch = prev+1` on each set; guest applies once and records the applied epoch locally. | At-most-once boot-time directive. Not a fence: it is pulled only at boot and has no `disable delivery` action. | `handlers/eks/recovery_directive.go:30-43`, `:90-107`; `mulga-eks-k3s-recovery.sh:131-197` |
| Per-cluster epoch or generation | None in `ClusterMeta`; add-on records have no generation. | Nothing. | `cluster_state.go:64-201`, `docs/package-boundary/eks-addon-lifecycle.md` row "Generation" |
| K3s server roles | `server` (cluster-init) and `server-join`; fixed at first boot. | Identifies the delivery VM only until `[0]` is replaced. | `k3s_server_vm.go:527-531`, `:585-589` |
| Etcd / kube leader election | K3s embedded etcd and kube controllers elect internally; nothing in Spinifex reads them. | Not used. | no reference in `scripts/images/eks-node/*.sh` or `handlers/eks` |

## 8. Candidate fencing evidence available today

"Fences" means the old executor provably cannot render or apply, and cannot have its reports accepted.

| Evidence | Available how | Proves old executor cannot render/apply? | Gap and what would be needed |
|---|---|---|---|
| EC2 state `terminated` or instance not found | `cpControlAdapter.InstanceState` (system account `DescribeInstances`); `TerminateK3sServerVM` treats not-found as success. | Yes for render and apply once the VM is gone, if the instance record is authoritative; host-partition behaviour of the state record is unverified. | Reports already queued or retried by `eks-gateway-publish` are still accepted. Needs: state proven authoritative under host loss, and report acceptance bound to the current executor. `service_impl.go:153-170`, `k3s_server_vm.go:292-330` |
| EC2 state `stopped` / `error` | Same query. | Only momentarily: the reconciler's restart ladder starts stopped/error members automatically, and the VM resumes delivery with the same identity. | Needs a stop that nothing restarts, or a guest-side disable that survives restart. `cluster_reconciler.go:708-771`, `:927-937` |
| Describe error | Treated as lost. | No: the VM may be running. | Must not count as fencing evidence. `cluster_reconciler.go:982-1007` |
| Removal from `ControlPlaneNodes` | `SwapControlPlaneMember`, `replaceControlPlaneForRestore`. | Partly: `ListInternalAddons` is denied, but the agent keeps its current render on fetch failure, K3s keeps applying those files; its reports are now denied by the membership check, but reports already in flight are still applied. | Needs: publish gated by executor and identity attached on relay; a guest rule that a denied fetch stops delivery. `internal_authz.go:74`, `mulga-eks-addon-sync.sh:344-348` |
| Instance ID / incarnation | Session name = instance ID; stable across stop/start. | No: there is no per-boot incarnation, so a restarted VM is indistinguishable from before. | Needs an executor epoch recorded host-side and checked per request; instance ID alone identifies, it does not fence. `assume_role.go:136-161` |
| KV revisions | Meta revision, add-on record revision, lease revision. | No: meta revision churns with health writes, add-on CAS ignores sender, lease revision is host-only and private. | Needs a dedicated, monotonic executor epoch written only on reassignment. `cluster_state.go:416-436`, `kvlease/lease.go:44-52` |
| IMDS credential expiry | Instance-role creds last 3600 s, re-minted 5 min early. | No: up to an hour of validity after the last mint, and a running VM keeps re-minting. | Not a fence on its own. `domains/ec2/guestmetadata/credentials.go:14-20` |
| Credential revocation | `RevokeRoleSessions` deletes sessions by role (called on role delete); no per-instance revocation. | No per-executor effect: the role is shared by every CP VM of every cluster. | Needs a per-session-name revocation, and proof the gateway rejects a revoked session; a running VM would still re-mint via IMDS unless IMDS also refuses it. `handlers/sts/session_credentials.go:212-219`, `handlers/iam/roles.go:221` |
| Static system keys | Shared system access key on fallback. | No: shared, unrevocable per VM, no identity. | Fallback VMs cannot be executors under any identity-based rule. `k3s_server_role.go:30-50` |
| K3s node removal / etcd member prune | `EKS_ETCD_PRUNE_PEER_IP` handled in `k3s-first-boot.sh`. | Would stop apply through the shared datastore if the member is removed, but not render to the VM's own disk. The prune path appears unreachable: the variable is set only on replacements, which are `server-join`, and `k3s-first-boot` runs only for `server`. | Needs a reachable prune, verification that a removed member's apiserver cannot write, and still the report gate. `k3s-first-boot.sh:165-187`, `k3s_ha_control_plane.go:153`, `eks-node-role.sh:136`, `:151-166` |
| Guest service disable | None remotely; the only host-to-guest instruction channel is the boot-time recovery directive. | No. | Needs an online directive the agent polls, plus an acknowledgement the host can verify; an acknowledgement from the old executor proves only that it was alive to send it. `recovery_directive.go:12-28` |
| Restore fence | `confirmOldCPTerminated` after meta commit. | Yes for the old VM once terminate succeeds, but it runs after the new VM was launched and committed. | Ordering is successor-first; R6 requires fence-first. `restore_snapshot.go:262-336` |

## 9. Evidence each design option would need

No option is chosen here; each row lists only what it would rely on.

| Option | Evidence it needs that does not exist today |
|---|---|
| Host-assigned executor `{instanceId, epoch}` on the cluster record | An executor field and epoch advanced by CAS; the gateway attaching the caller instance ID (and epoch) to the relayed message; report apply checking it; a reassignment trigger that does not fire on describe errors; fence evidence from Section 8 rows that say "Yes". |
| Executor follows meta `[0]` | A guest that can become executor after first boot (today role is fixed and replacements are `server-join`); the same gateway and relay identity checks; `[0]` reassignment tied to fencing of the old `[0]`. |
| Run the agent on every server with guest-side leader election (kube Lease) | Host-visible proof of the guest leader (the host cannot read the kube Lease today); verification of K3s auto-deploy behaviour across servers; reports still need gateway-attached identity, since a partitioned old leader can still publish. |
| Disable-on-handover directive | An online directive channel and agent polling; a rule that a fetch denial or missing directive stops rendering; an acknowledgement or instance-state proof before the successor acts. |
| Credential-based fence | Per-instance session revocation, IMDS refusing to re-mint for a fenced instance, and gateway rejection of revoked sessions; still insufficient for files already rendered to the old VM's disk. |

## 10. Gaps

1. Delivery runs only on the first-boot `server` VM; replacing HA `[0]` leaves no member running `mulga-eks-addon-sync`, with no detection or recovery.
2. Corrected: `PublishInternal` and `WebhookTokenReview` were policy-only, so any CP-role session, any tenant holding `eks:*`, or static system credentials could publish reports or resolve tokens for any cluster and account. Both now require cluster membership (see Section 3); the binding is to a member, not to the delivery executor.
3. The relay attaches no caller identity, and the payload carries none, so the owner cannot tell which member, executor or epoch produced a report; this conflicts with the direction in draft PROP-SPINIFEXARCH-007 S3, which is not accepted.
4. `ClusterMeta` has no role, executor or epoch field; `[0]` as "primary" is a convention that diverges from the guest role after a swap.
5. A member removed from meta is denied fetches but keeps applying its last render and keeps having its reports accepted.
6. Describe errors count as member loss, so a running primary can be replaced without fencing.
7. The restore path launches and commits the successor before fencing the old CP.
8. No per-instance credential revocation; the CP role and instance profile are shared across all clusters.
9. `FAILED` clusters (create timeout, bootstrap failure) keep running CP VMs whose agent is still admitted while reports are dropped.
10. The etcd peer prune for replacements appears unreachable (`server-join` does not run `k3s-first-boot`); not verified live.
11. Whether K3s auto-deploy applies from a non-leader server's manifests dir is unverified, which bears on any multi-server option.
12. No lease takeover for a cluster whose reconciler holder died until a daemon restarts, so reports are dropped meanwhile.