feat(ecs): agent control-plane over the AWS gateway (drop direct NATS) (#419)

* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel

Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.

* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState

Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.

* feat(ecs): durable assign KV inbox + PollAssignments gateway action

Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.

At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.

* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway

The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).

- controlPlane seam over ecsgw.Client replaces the NATS publisher:
  Register (RegisterContainerInstance), SubmitTaskState
  (SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
  LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
  inbox, dispatch each assign once, ack accepted taskIDs on the next
  poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
  ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.

Tests move from the NATS publisher fake to a controlPlane fake.

* feat(ecs): cloud-init cred-seed + ecs-node bring-up script

ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.

The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).

* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring

The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:

- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
  reclaimAssignInbox; both are needed (release the task ENI AND drain the
  KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
  teardownTaskNetns wrappers were lost when the controlPlane Agent struct
  won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
  the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
  fakeCP to match the controlPlane seam.

scripts/ecs-node-bringup.sh

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# ecs-node-bringup.sh — launch a Spinifex ECS container-instance VM.
+#
+# ECS container instances are plain EC2 instances booted from the spinifex-ecs-node
+# AMI (Alpine + containerd + ecs-agent), AWS-faithful: there is no ECS-specific
+# launch API. This script renders the cloud-init user-data that seeds the agent's
+# control-plane config + IAM credentials into /etc/spinifex-ecs/agent.env and the
+# gateway TLS CA into /etc/spinifex-ecs/gateway-ca.pem, then runs the instance.
+#
+# The agent then registers, heartbeats, polls assignments, and reports task state
+# entirely through the AWS gateway over TLS+SigV4 — it never touches NATS.
+#
+# Usage:
+#   AWS_PROFILE=spinifex scripts/ecs-node-bringup.sh <cluster-name>
+#
+# Env overrides:
+#   GATEWAY_IP   gateway/mgmt IP the guest dials   (default 10.15.8.1)
+#   GATEWAY_CA   host path to the gateway CA PEM   (default /etc/spinifex/ca.pem)
+#   AMI_ID       ECS-node AMI                      (default: latest spinifex-ecs-node)
+#   SUBNET_ID / SG_ID / KEY_NAME / INSTANCE_TYPE   (default: first available / t3.small)
+set -euo pipefail
+
+CLUSTER="${1:?usage: ecs-node-bringup.sh <cluster-name>}"
+PROFILE="${AWS_PROFILE:-spinifex}"
+REGION="$(aws configure get region --profile "$PROFILE")"
+GATEWAY_IP="${GATEWAY_IP:-10.15.8.1}"
+GATEWAY_CA="${GATEWAY_CA:-/etc/spinifex/ca.pem}"
+INSTANCE_TYPE="${INSTANCE_TYPE:-t3.small}"
+
+awscli() { aws --profile "$PROFILE" "$@"; }
+
+AMI_ID="${AMI_ID:-$(awscli ec2 describe-images \
+  --filters 'Name=name,Values=spinifex-ecs-node' \
+  --query 'sort_by(Images,&CreationDate)[-1].ImageId' --output text)}"
+SUBNET_ID="${SUBNET_ID:-$(awscli ec2 describe-subnets \
+  --query 'Subnets[0].SubnetId' --output text)}"
+SG_ID="${SG_ID:-$(awscli ec2 describe-security-groups \
+  --query 'SecurityGroups[0].GroupId' --output text)}"
+KEY_NAME="${KEY_NAME:-$(awscli ec2 describe-key-pairs \
+  --query 'KeyPairs[0].KeyName' --output text)}"
+
+ACCESS_KEY="$(aws configure get aws_access_key_id --profile "$PROFILE")"
+SECRET_KEY="$(aws configure get aws_secret_access_key --profile "$PROFILE")"
+
+echo "cluster=$CLUSTER ami=$AMI_ID subnet=$SUBNET_ID sg=$SG_ID key=$KEY_NAME gw=https://$GATEWAY_IP:9999" >&2
+
+# Indent a file's content by six spaces for a cloud-init "content: |" block.
+indent6() { sed 's/^/      /' "$1"; }
+
+USERDATA="$(cat <<EOF
+#cloud-config
+bootcmd:
+  - rm -f /etc/resolv.conf
+  - printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > /etc/resolv.conf
+write_files:
+  - path: /etc/spinifex-ecs/agent.env
+    permissions: '0600'
+    content: |
+      ECS_GATEWAY_URL=https://$GATEWAY_IP:9999
+      ECS_GATEWAY_CA=/etc/spinifex-ecs/gateway-ca.pem
+      ECS_REGION=$REGION
+      ECS_CLUSTER=$CLUSTER
+      ECS_ACCESS_KEY=$ACCESS_KEY
+      ECS_SECRET_KEY=$SECRET_KEY
+  - path: /etc/spinifex-ecs/gateway-ca.pem
+    permissions: '0644'
+    content: |
+$(indent6 "$GATEWAY_CA")
+EOF
+)"
+
+INSTANCE_ID="$(awscli ec2 run-instances \
+  --image-id "$AMI_ID" --instance-type "$INSTANCE_TYPE" \
+  --key-name "$KEY_NAME" --subnet-id "$SUBNET_ID" --security-group-ids "$SG_ID" \
+  --user-data "$USERDATA" \
+  --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=ecs-node-$CLUSTER}]" \
+  --query 'Instances[0].InstanceId' --output text)"
+
+echo "$INSTANCE_ID"

scripts/images/ecs-agent/ecs-agent.initd

@@ -1,10 +1,10 @@
 #!/sbin/openrc-run
 # ecs-agent — Spinifex ECS container-instance agent. Resolves host identity from
-# IMDSv2, connects to the cluster NATS bus, registers + heartbeats on the Layer-2
-# subjects, and pulls task images through containerd.
+# IMDSv2, registers + heartbeats + polls task assignments through the AWS gateway
+# over TLS+SigV4 (no NATS), and pulls task images through containerd.
 #
-# Static config (gateway URL, CA, region, cluster, NATS, containerd socket) comes
-# from /etc/spinifex-ecs/agent.env, dropped by cloud-init user-data at launch.
+# Static config (gateway URL, CA, region, cluster, seeded IAM creds, containerd
+# socket) comes from /etc/spinifex-ecs/agent.env, dropped by cloud-init at launch.
 
 description="Spinifex ECS container-instance agent"
 command="/usr/local/bin/ecs-agent"
@@ -23,11 +23,12 @@ depend() {
 start_pre() {
     # Cloud-init drops the agent config here; OpenRC injects it into the agent's
     # environment. Absent file is tolerated — the agent falls back to its
-    # built-in defaults (cluster=default, local NATS, default IMDS).
+    # built-in defaults (cluster=default, default IMDS).
     if [ -f /etc/spinifex-ecs/agent.env ]; then
         # shellcheck disable=SC1091
         . /etc/spinifex-ecs/agent.env
         export ECS_GATEWAY_URL ECS_GATEWAY_CA ECS_REGION ECS_CLUSTER \
-            ECS_NATS_URL ECS_IMDS_BASE ECS_CONTAINERD_SOCKET ECS_HEARTBEAT_INTERVAL
+            ECS_ACCESS_KEY ECS_SECRET_KEY ECS_IMDS_BASE ECS_CONTAINERD_SOCKET \
+            ECS_HEARTBEAT_INTERVAL ECS_POLL_INTERVAL
     fi
 }

scripts/images/ecs-agent/manifest.conf

@@ -27,7 +27,8 @@ scripts/images/ecs-agent/10-bridge.conflist:/etc/cni/net.d/10-bridge.conflist"
 
 # ecs-ca-install (first boot) installs the cloud-init gateway CA into the system
 # trust before containerd starts; containerd then trusts the ECR registry TLS.
-# ecs-agent registers + heartbeats on the NATS bus once net + containerd are up.
+# ecs-agent registers + heartbeats + polls the AWS gateway (TLS+SigV4, no NATS)
+# once net + containerd are up.
 ENABLE_SERVICES="containerd ecs-ca-install ecs-agent"
 
 SETUP_SCRIPT="scripts/images/ecs-agent/setup.sh"