Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.
The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.
Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d)
Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the
task subnet on placement, hot-plug onto the container instance, plumb ENI
identity through the assign payload + task record, and detach+delete on
STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged.
* feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI
The agent builds a per-task network namespace for awsvpc tasks: it finds the
hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs
the OVN-assigned ENI IP. The container then joins that netns via the OCI network
namespace path; bridge/host tasks keep the host (VM) netns as before.
* build(ecs-node): add iproute2 for awsvpc task netns
The ecs-agent builds awsvpc task network namespaces with `ip netns`, which
busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI;
udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.
Add the ECS v1 scheduler control loop and end-to-end RunTask flow:
- handlers/ecs: per-account KV-backed Service (clusters, task definitions,
container instances, tasks), bin-pack/spread/random placement with CAS
capacity reservation, and a leader-elected Scheduler that owns the Layer-2
bus subscriptions (register/heartbeat/task-state) plus a heartbeat reaper.
- gateway/ecs: wire CreateCluster, Describe/ListClusters,
Register/Describe/ListTaskDefinitions, Register/Describe/ListContainerInstances,
RunTask, Describe/ListTasks onto the daemon over NATS.
- daemon: construct the Service, start the Scheduler goroutine, subscribe the
ECS action subjects.
- ecs-agent: subscribe the per-instance assign subject, pull and run each
container via containerd, and report task state back on the bus.
Josh Siviour