mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
471e95dac2
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f)
Land the ECS Service controller on top of the 4e scheduler/RunTask flow:
- CreateService/UpdateService/DeleteService/DescribeServices/ListServices.
REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15).
- StartTask (explicit placement) and StopTask.
- Leader-driven replica reconciliation: a 10s scheduler tick converges each
service's running task count on desiredCount, launching the shortfall via
RunTask (tasks tagged group=service:{name}) and force-stopping the surplus.
- ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI
IP on the RUNNING transition and deregisters on STOPPED via the existing
elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer).
- forceStopTask consolidates the control-plane stop (STOPPED state, capacity +
ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and
the heartbeat reaper (the reaper now deregisters targets per Q13).
- Wire the 7 new actions through the gateway action map, NATS adapter, and
daemon subscriptions; add ServiceNotFoundException.
* fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs
The agent minted an ECR authorization token via IMDS for every image, so
public docker.io refs failed with 'IMDS returned empty role name' on hosts
without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other
registries anonymously.
Bare image strings (nginx:alpine) were handed to containerd unmodified, which
parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to
their fully-qualified form (docker.io/library/nginx:alpine, :latest default)
before Pull and GetImage.
Adds StartTask/StopTask service-controller coverage.
* fix: address ECS 4f live-verification findings
ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which
a VPC guest cannot route to — the agent could not reach the gateway. Default to
the host's WAN-bridge IP, which guests reach by NATing out through the WAN.
A target group not forwarded to by any listener reported its targets stuck in
'initial' forever (no health checker advances a detached TG). DescribeTargetHealth
now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.
88 lines
3.6 KiB
Bash
Executable File
88 lines
3.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ecs-node-bringup.sh — launch a Spinifex ECS container-instance VM.
|
|
#
|
|
# ECS container instances are plain EC2 instances booted from the spinifex-ecs-node
|
|
# AMI (Alpine + containerd + ecs-agent), AWS-faithful: there is no ECS-specific
|
|
# launch API. This script renders the cloud-init user-data that seeds the agent's
|
|
# control-plane config + IAM credentials into /etc/spinifex-ecs/agent.env and the
|
|
# gateway TLS CA into /etc/spinifex-ecs/gateway-ca.pem, then runs the instance.
|
|
#
|
|
# The agent then registers, heartbeats, polls assignments, and reports task state
|
|
# entirely through the AWS gateway over TLS+SigV4 — it never touches NATS.
|
|
#
|
|
# Usage:
|
|
# AWS_PROFILE=spinifex scripts/ecs-node-bringup.sh <cluster-name>
|
|
#
|
|
# Env overrides:
|
|
# GATEWAY_IP gateway IP the guest dials (default: host WAN-bridge IP)
|
|
# GATEWAY_CA host path to the gateway CA PEM (default /etc/spinifex/ca.pem)
|
|
# AMI_ID ECS-node AMI (default: latest spinifex-ecs-node)
|
|
# SUBNET_ID / SG_ID / KEY_NAME / INSTANCE_TYPE (default: first available / t3.small)
|
|
set -euo pipefail
|
|
|
|
CLUSTER="${1:?usage: ecs-node-bringup.sh <cluster-name>}"
|
|
PROFILE="${AWS_PROFILE:-spinifex}"
|
|
REGION="$(aws configure get region --profile "$PROFILE")"
|
|
|
|
# VPC guests NAT out through the WAN and reach host-run services on the host's
|
|
# WAN-bridge IP; the mgmt bridge (10.15.8.x) is not routable from a guest netns.
|
|
# Default to the br-wan address, falling back to the mgmt IP if it can't be read.
|
|
default_gateway_ip() {
|
|
ip -4 addr show br-wan 2>/dev/null | awk '/inet /{print $2; exit}' | cut -d/ -f1
|
|
}
|
|
GATEWAY_IP="${GATEWAY_IP:-$(default_gateway_ip)}"
|
|
GATEWAY_IP="${GATEWAY_IP:-10.15.8.1}"
|
|
GATEWAY_CA="${GATEWAY_CA:-/etc/spinifex/ca.pem}"
|
|
INSTANCE_TYPE="${INSTANCE_TYPE:-t3.small}"
|
|
|
|
awscli() { aws --profile "$PROFILE" "$@"; }
|
|
|
|
AMI_ID="${AMI_ID:-$(awscli ec2 describe-images \
|
|
--filters 'Name=name,Values=spinifex-ecs-node' \
|
|
--query 'sort_by(Images,&CreationDate)[-1].ImageId' --output text)}"
|
|
SUBNET_ID="${SUBNET_ID:-$(awscli ec2 describe-subnets \
|
|
--query 'Subnets[0].SubnetId' --output text)}"
|
|
SG_ID="${SG_ID:-$(awscli ec2 describe-security-groups \
|
|
--query 'SecurityGroups[0].GroupId' --output text)}"
|
|
KEY_NAME="${KEY_NAME:-$(awscli ec2 describe-key-pairs \
|
|
--query 'KeyPairs[0].KeyName' --output text)}"
|
|
|
|
ACCESS_KEY="$(aws configure get aws_access_key_id --profile "$PROFILE")"
|
|
SECRET_KEY="$(aws configure get aws_secret_access_key --profile "$PROFILE")"
|
|
|
|
echo "cluster=$CLUSTER ami=$AMI_ID subnet=$SUBNET_ID sg=$SG_ID key=$KEY_NAME gw=https://$GATEWAY_IP:9999" >&2
|
|
|
|
# Indent a file's content by six spaces for a cloud-init "content: |" block.
|
|
indent6() { sed 's/^/ /' "$1"; }
|
|
|
|
USERDATA="$(cat <<EOF
|
|
#cloud-config
|
|
bootcmd:
|
|
- rm -f /etc/resolv.conf
|
|
- printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > /etc/resolv.conf
|
|
write_files:
|
|
- path: /etc/spinifex-ecs/agent.env
|
|
permissions: '0600'
|
|
content: |
|
|
ECS_GATEWAY_URL=https://$GATEWAY_IP:9999
|
|
ECS_GATEWAY_CA=/etc/spinifex-ecs/gateway-ca.pem
|
|
ECS_REGION=$REGION
|
|
ECS_CLUSTER=$CLUSTER
|
|
ECS_ACCESS_KEY=$ACCESS_KEY
|
|
ECS_SECRET_KEY=$SECRET_KEY
|
|
- path: /etc/spinifex-ecs/gateway-ca.pem
|
|
permissions: '0644'
|
|
content: |
|
|
$(indent6 "$GATEWAY_CA")
|
|
EOF
|
|
)"
|
|
|
|
INSTANCE_ID="$(awscli ec2 run-instances \
|
|
--image-id "$AMI_ID" --instance-type "$INSTANCE_TYPE" \
|
|
--key-name "$KEY_NAME" --subnet-id "$SUBNET_ID" --security-group-ids "$SG_ID" \
|
|
--user-data "$USERDATA" \
|
|
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=ecs-node-$CLUSTER}]" \
|
|
--query 'Instances[0].InstanceId' --output text)"
|
|
|
|
echo "$INSTANCE_ID"
|