Compare commits

..

54 Commits

Author SHA1 Message Date
Julian Sommer c2ed729ce1 release: update go.mod for v1.10.0 2026-06-30 10:30:07 +10:00
Julian Sommer 6757f847d2 chore: rebuild ui 2026-06-30 09:50:03 +10:00
dependabot[bot] 99fb65b1ef chore: bump the npm-dependencies group (#437) 2026-06-30 09:48:38 +10:00
Josh Siviour a7241e01f9 feat: run EKS apiserver with anonymous-auth=false (CIS) (#447)
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
2026-06-30 09:21:07 +10:00
dependabot[bot] 6fe9220d6f chore: bump github.com/pelletier/go-toml/v2 in the go-dependencies group (#436) 2026-06-30 09:06:21 +10:00
Josh Siviour 52c0ed1e77 fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT (#446)
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT

bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.

e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.

* test: validate ECS host-netns :80 binding via bridge task

The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
2026-06-30 08:59:52 +10:00
Josh Siviour ab1b1bacf6 fix: auto-restart NATS when JetStream is disabled while the process lives (#445)
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.

Add a JetStream liveness watchdog:

- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
  (http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
  is active and JS fails every sample of a short multi-sample window (so
  transient startup/quorum 503s are ignored) does it systemctl try-restart
  spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
  60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.

The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
2026-06-30 08:41:10 +10:00
Josh Siviour 686a57b4e0 fix: reap orphan LBC ALBs and security groups on EKS cluster teardown (#444)
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.

purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:

- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
  confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
  the DependencyViolation retry budget), in addition to the named cluster
  SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.

Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
2026-06-30 08:03:56 +10:00
Josh Siviour ea38f46392 fix: force-terminate stopped+wedged EKS workers on teardown (#443)
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.

On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
2026-06-29 22:34:41 +10:00
Josh Siviour 746a6f5d05 fix: reap orphan QEMU for terminated instances via node-local reaper (#442)
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.

Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.

Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
2026-06-29 22:33:58 +10:00
Josh Siviour d6cfd22ddb fix: re-assert boot-volume in-use state on daemon-restart reconnect (#441)
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.

Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
2026-06-29 21:05:06 +10:00
Josh Siviour aaef758426 fix: cross-check live BlockDeviceMappings before DeleteVolume (#440)
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.

Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
2026-06-29 15:30:26 +10:00
Josh Siviour 47c9808a96 test(e2e): add ECS end-to-end suite (awsvpc, bridge, host, ELB service) (#439)
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.

bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
2026-06-29 14:16:23 +10:00
Josh Siviour ae800386fb feat: complete EC2 instance health checks (Phase C + E) (#438)
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.

Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
  QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
  on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.

Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
  with ImpairedSince) / initializing (2-min launch grace), non-running to
  not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
  HostUnderMemoryPressure (live MemAvailable below the host reserve).

Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.

Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
2026-06-29 13:16:58 +10:00
Josh Siviour 4e6a2eb46a feat: ECS agent reboot reconciler — re-adopt running containers on restart (#435)
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.

Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.

- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
  containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
  self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.

Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
2026-06-29 12:13:25 +10:00
Josh Siviour 07722e563f docs: ECS service guide + register ecs-quickstart workbook (#434)
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook

Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).

Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.

* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages

Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.

Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
2026-06-29 10:44:19 +10:00
Josh Siviour dd6ed6db2c docs: add ECS quickstart terraform/opentofu workbook (#433)
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
2026-06-29 09:36:00 +10:00
Josh Siviour be7ca0f828 test: lock reaper capacity release + clean re-activation (#432) 2026-06-29 08:30:14 +10:00
Julian Sommer 485cca941f feat: IAM inline role policies (Put/Get/DeleteRolePolicy + real ListRolePolicies) (#429) 2026-06-26 15:38:45 +10:00
Josh Siviour be3af1c6d8 fix: recover reaped container instances when the agent re-registers (#430)
The agent heartbeats by periodically re-registering through the gateway
(RegisterContainerInstance). After a control-plane restart, the heartbeat
reaper marks instances DRAINING for the missed-heartbeat window, but the
re-register path only refreshed LastSeen and never restored the status, so a
reaped instance stayed DRAINING forever even though its agent was healthy and
re-registering.

Mark reaper-induced DRAINING with a Reaped flag and restore the instance to
ACTIVE on the next re-register, clearing the flag. An operator
UpdateContainerInstancesState drain clears Reaped, so an intentional drain is
never undone by the live agent.
2026-06-26 15:32:49 +10:00
Josh Siviour 9a6bf692ca feat(ecs): EC2 capacity provisioning + auto-EIP service tasks + task/service detail pages (#431)
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role

Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node
EC2 instances which register as ECS container instances, fixing
RESOURCE:placement when a cluster has no capacity.

Agent control-plane now SigV4-signs gateway calls with instance-role
credentials read from IMDS (no static keys seeded in user-data). The
gateway already verifies STS/temp creds, so the instance role carries an
ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the
ecsInstanceRole instance profile attached at launch.

Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs
tag, renders keyless user-data, and launches via the customer
RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity
call, a provision-capacity dialog on the container-instances tab, and the
ECS console section nav + system-image gating.

* feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages

Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service
tasks automatically allocate and associate an Elastic IP on the RUNNING
transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP
mirror the target-group registration hooks in recordTaskState and forceStopTask;
the public IP and allocation ID are persisted on the task record and surfaced in
DescribeTasks attachments as publicIPv4Address.

Add AWS-parity ECS task and service detail pages showing private and public
endpoints, container state, and networking configuration, reachable by clicking
a task or service in the cluster tabs.

* test: cover natsEIPManager allocate/associate/release over NATS
2026-06-26 14:58:31 +10:00
Julian Sommer 7ffe12ac82 feat: EC2 instance metadata-options API (IMDSv2-only, faithfully reported) (#428) 2026-06-26 13:36:00 +10:00
Julian Sommer 26167c017e chore: delete orphaned directory 2026-06-26 10:57:50 +10:00
Julian Sommer 8c9561cb1e chore: delete orphaned directory 2026-06-26 10:54:05 +10:00
Julian Sommer 4bc28172d9 feat: per-account service quotas (#427) 2026-06-26 10:52:53 +10:00
Julian Sommer 54d2c8f250 chore: cleanup wrapper functions 2026-06-26 10:47:19 +10:00
Julian Sommer eda5987e0b chore: delete unused files 2026-06-26 10:42:41 +10:00
Josh Siviour 06ea7ecfd1 fix: ECS ListTaskDefinitions honors Status filter (default ACTIVE) (#426)
ListTaskDefinitions appended every revision regardless of the requested
status, so deregistered (INACTIVE) task definitions never dropped off the
default listing and the console could not clear them. Filter by status,
defaulting to ACTIVE to match AWS.
2026-06-26 09:23:03 +10:00
Josh Siviour cb67fbec87 feat: ECS console with deploy-container wizards (#425)
Add an ECS console mirroring the EKS/ECR route shape: a single ECS nav
entry, cluster list and detail with Services/Tasks/Task Definitions/
Container Instances tabs, plus create and teardown paths.

Adds three full-page wizards to drive container deploys end-to-end:
Register Task Definition, Run Task, and Create Service. RunTask surfaces
a 200-with-failures placement result as an error instead of silently
navigating away.

Includes the @aws-sdk/client-ecs client singleton, ecs queries and
mutations, zod form schemas, and co-located tests.
2026-06-26 08:57:54 +10:00
Josh Siviour 873398337a feat: ECS STOPPED task GC (leader sweep, 1h retention) (#424)
STOPPED task records were never pruned: ListTasks enumerated every task a
cluster ever ran and the per-account KV grew unbounded. Add a leader-gated sweep
tick to the scheduler cron (alongside the reaper/reconcile) that deletes task
records STOPPED longer than a 1h retention window — keeping a just-stopped task
describable for DescribeTasks / UI exit reason, matching AWS's STOPPED window.

A task missing its StoppedAt timestamp is never pruned (defensive). Single-writer
(leader only), reuses the reap() bucket walk.
2026-06-25 23:41:44 +10:00
Josh Siviour 30c58ec358 feat: ECS P0 teardown (DeleteCluster cascade, Deregister taskdef/instance, drain) (#423)
Wire the four NotImplemented ECS teardown actions so an IaC stack and the
console can reverse a deployment:

- DeleteCluster cascades: force-stops every task (releasing ENIs, deregistering
  TG targets, returning capacity), then sweeps the cluster's whole KV subtree.
  Unlike AWS it cascades rather than rejecting a non-empty cluster, so terraform
  destroy round-trips regardless of teardown ordering.
- DeregisterTaskDefinition marks a specific family:revision INACTIVE (kept
  describable); a bare family is rejected, matching AWS.
- DeregisterContainerInstance removes an instance, force-stopping its tasks when
  Force is set and rejecting a non-empty instance otherwise.
- UpdateContainerInstancesState sets ACTIVE/DRAINING; draining force-stops the
  instance's service-owned tasks so the reconciler relaunches them elsewhere.

Reuses the existing forceStopTask primitive and the reaper drain pattern; adds a
deleteKeysWithPrefix KV helper and INACTIVE status constants.
2026-06-25 23:21:23 +10:00
Josh Siviour 93a398afb8 feat: ECS task IAM credential endpoint (sprint 4g) (#422)
Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.

The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.

Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
2026-06-25 22:06:19 +10:00
Josh Siviour 2ca655cd39 fix: EBS hot-plug PCIe port allocator (every attach past the first collided) (#421)
* fix: serialize EBS hot-plug per instance to prevent PCIe port race

Concurrent AttachVolume calls to one instance both allocated the lowest free
hot-plug PCIe root port via nextHotplugEBSPort (which reads live QEMU state),
then issued QMP device_add on the same hotplug-ebs bus. The second device_add
failed with 'PCI: slot 0 function 0 already occupied by virtio-blk-pci',
surfacing as nats:timeout -> ServerInternal and a CSI retry-storm
('retry quota exceeded'); volumes never attached.

Add a per-instance attachMu and hold it across the whole AttachVolume and
DetachVolume critical sections, mirroring the ENI hot-plug serialization.
The mutex is non-persisted and shared via the Manager's stable *VM.

* fix: allocate EBS hot-plug PCIe port from in-memory accounting

AttachVolume derived the free PCIe hot-plug root port from a live
query-block scan (nextHotplugEBSPort), extracting the bus with a
hotplug-ebs(\d+) regex on each device's QDev. Real QEMU reports a
hot-plugged blockdev by id (/machine/peripheral/vdisk-vol-xxx/
virtio-backend), with no hotplug-ebs component, so the regex never
matched, the used set stayed empty, and the allocator always returned
port 1. Only the first EBS volume per instance attached; every later
attach collided on "PCI: slot 0 function 0 already occupied by
virtio-blk-pci", rolling back the blockdev, killing NBDKit, and
surfacing as nats: timeout / ServerInternal to the CSI driver, which
then exhausted its retry quota.

Allocate the port from authoritative in-memory accounting instead:
record the chosen port as EBSRequest.HotplugPort (persisted, so it
survives a daemon restart) and pick the lowest unclaimed port with
freeHotplugEBSPort over the instance's recorded requests. Hold
attachMu across allocate->device_add so concurrent CSI attaches cannot
pick the same port. DetachVolume already removes the request, freeing
the port. Drops the query-block round-trip from the attach path.
2026-06-25 21:24:32 +10:00
Josh Siviour 471e95dac2 feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f) (#420)
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel

Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.

* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState

Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.

* feat(ecs): durable assign KV inbox + PollAssignments gateway action

Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.

At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.

* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway

The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).

- controlPlane seam over ecsgw.Client replaces the NATS publisher:
  Register (RegisterContainerInstance), SubmitTaskState
  (SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
  LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
  inbox, dispatch each assign once, ack accepted taskIDs on the next
  poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
  ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.

Tests move from the NATS publisher fake to a controlPlane fake.

* feat(ecs): cloud-init cred-seed + ecs-node bring-up script

ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.

The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).

* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring

The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:

- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
  reclaimAssignInbox; both are needed (release the task ENI AND drain the
  KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
  teardownTaskNetns wrappers were lost when the controlPlane Agent struct
  won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
  the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
  fakeCP to match the controlPlane seam.

* feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f)

Land the ECS Service controller on top of the 4e scheduler/RunTask flow:

- CreateService/UpdateService/DeleteService/DescribeServices/ListServices.
  REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15).
- StartTask (explicit placement) and StopTask.
- Leader-driven replica reconciliation: a 10s scheduler tick converges each
  service's running task count on desiredCount, launching the shortfall via
  RunTask (tasks tagged group=service:{name}) and force-stopping the surplus.
- ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI
  IP on the RUNNING transition and deregisters on STOPPED via the existing
  elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer).
- forceStopTask consolidates the control-plane stop (STOPPED state, capacity +
  ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and
  the heartbeat reaper (the reaper now deregisters targets per Q13).
- Wire the 7 new actions through the gateway action map, NATS adapter, and
  daemon subscriptions; add ServiceNotFoundException.

* fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs

The agent minted an ECR authorization token via IMDS for every image, so
public docker.io refs failed with 'IMDS returned empty role name' on hosts
without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other
registries anonymously.

Bare image strings (nginx:alpine) were handed to containerd unmodified, which
parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to
their fully-qualified form (docker.io/library/nginx:alpine, :latest default)
before Pull and GetImage.

Adds StartTask/StopTask service-controller coverage.

* fix: address ECS 4f live-verification findings

ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which
a VPC guest cannot route to — the agent could not reach the gateway. Default to
the host's WAN-bridge IP, which guests reach by NATing out through the WAN.

A target group not forwarded to by any listener reported its targets stuck in
'initial' forever (no health checker advances a detached TG). DescribeTargetHealth
now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.
2026-06-25 21:06:00 +10:00
Josh Siviour de64d6c63d feat(ecs): agent control-plane over the AWS gateway (drop direct NATS) (#419)
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel

Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.

* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState

Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.

* feat(ecs): durable assign KV inbox + PollAssignments gateway action

Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.

At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.

* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway

The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).

- controlPlane seam over ecsgw.Client replaces the NATS publisher:
  Register (RegisterContainerInstance), SubmitTaskState
  (SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
  LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
  inbox, dispatch each assign once, ack accepted taskIDs on the next
  poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
  ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.

Tests move from the NATS publisher fake to a controlPlane fake.

* feat(ecs): cloud-init cred-seed + ecs-node bring-up script

ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.

The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).

* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring

The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:

- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
  reclaimAssignInbox; both are needed (release the task ENI AND drain the
  KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
  teardownTaskNetns wrappers were lost when the controlPlane Agent struct
  won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
  the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
  fakeCP to match the controlPlane seam.
2026-06-25 18:42:24 +10:00
Josh Siviour 9c76df3373 feat(ecs): awsvpc mode — ENI-per-task allocation + guest datapath (Sprint 4d) (#418)
* feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d)

Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the
task subnet on placement, hot-plug onto the container instance, plumb ENI
identity through the assign payload + task record, and detach+delete on
STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged.

* feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI

The agent builds a per-task network namespace for awsvpc tasks: it finds the
hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs
the OVN-assigned ENI IP. The container then joins that netns via the OCI network
namespace path; bridge/host tasks keep the host (VM) netns as before.

* build(ecs-node): add iproute2 for awsvpc task netns

The ecs-agent builds awsvpc task network namespaces with `ip netns`, which
busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI;
udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.
2026-06-25 16:01:14 +10:00
joshsiv-mulga 3046240915 fix(e2e): ENI reconcile test SSHes as ubuntu to match stock AMI default
PR #411 (cloud-init EC2 datasource) dropped the cloud-init ISO that
created the ec2-user account and baked the harness SSH key; guests now
boot the stock Ubuntu gold AMI (default user ubuntu) bootstrapped via the
Ec2/IMDS datasource. eni_reconcile_test.go was the lone single-suite test
still SSHing as ec2-user, so TestENIHotplugReconcile failed publickey at
the first SSH. Align it with every other single-suite test.
2026-06-25 15:37:11 +10:00
Julian Sommer 82e443f305 chore: more inlines 2026-06-25 14:53:37 +10:00
Julian Sommer 1dfa4a672a chore: remove functions with stdlib alternatives 2026-06-25 14:31:05 +10:00
Julian Sommer 446f93ae8c chore: rename ubuntu-latest to ubuntu-26.04 2026-06-25 13:49:51 +10:00
Julian Sommer 8a94886b13 feat: cloud-init EC2 datasource (#411) 2026-06-25 13:48:08 +10:00
Josh Siviour 5e58645192 feat(eni): hot-plug reconciler + orphan sweep + e2e (Sprint 3d/3e) (#417)
* feat(eni): hot-plug reconciler + orphan sweep (Sprint 3d)

Add eniReconciler, a node-local vm.Reaper that converges ENI hot-plug
state across a daemon restart or crash: KV ENIRecord <-> in-memory PCIe
slot map <-> guest query-pci device list. Resolves records stuck in
attaching/detaching, re-adopts the slot map after restart, and reaps
orphan QMP devices and slot entries.

A staleness gate (new ENIRecord.AttachmentStateAt + 30s threshold) keeps
the reconciler off records a live attach/detach handler is mid-pipeline
on. Wired into the existing GarbageCollector backstop.

* test(eni): unit-cover hot-plug reconciler vm helpers

* test(eni): e2e reconciler convergence across daemon restart (Sprint 3e)
2026-06-25 11:35:52 +10:00
joshsiv-mulga 5fa12ec358 fix(e2e): skip stopped node in multinode NATS peer poll
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that
node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never
converges. Add WithSkipNodes to exclude the stopped node from the poll;
TestMultinodeNodeFailure skips node2.
2026-06-25 08:05:44 +10:00
Josh Siviour d32712b727 feat: ECS scheduler, RunTask placement, and agent task execution (#416)
Add the ECS v1 scheduler control loop and end-to-end RunTask flow:

- handlers/ecs: per-account KV-backed Service (clusters, task definitions,
  container instances, tasks), bin-pack/spread/random placement with CAS
  capacity reservation, and a leader-elected Scheduler that owns the Layer-2
  bus subscriptions (register/heartbeat/task-state) plus a heartbeat reaper.
- gateway/ecs: wire CreateCluster, Describe/ListClusters,
  Register/Describe/ListTaskDefinitions, Register/Describe/ListContainerInstances,
  RunTask, Describe/ListTasks onto the daemon over NATS.
- daemon: construct the Service, start the Scheduler goroutine, subscribe the
  ECS action subjects.
- ecs-agent: subscribe the per-instance assign subject, pull and run each
  container via containerd, and report task state back on the bus.
2026-06-25 07:52:07 +10:00
Josh Siviour 24bd50a53c feat: ecs-node AMI (alpine + containerd + ecs-agent baked) (#415)
Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc +
cni-plugins + ca-certificates with the ecs-agent binary baked to
/usr/local/bin and three OpenRC services wired in dependency order
(ecs-ca-install -> containerd -> ecs-agent).

- manifest.conf drives build-system-image.sh (no host mounts).
- ecs-ca-install.initd: first-boot oneshot installing the cloud-init
  delivered gateway CA into the system trust; no-op when absent.
- ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the
  agent after net + containerd.
- 10-bridge.conflist: CNI bridge for task networking (Sprint 4d).
- setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0
  console reorder + fast boot menu.
- Makefile: build-ecs-node-image / import-ecs-node-image targets.
2026-06-24 21:54:35 +10:00
Josh Siviour 93db7e4d1b feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c) (#414)
* feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c)

Add the ecs-agent binary that runs inside an ECS container instance: it
resolves host identity from IMDS, registers on the Layer-2 NATS bus, and
heartbeats while alive. Image pulls go through a containerd-backed
ImagePuller behind an interface, with an ECR resolver that mints
GetAuthorizationToken creds from the instance role.

Extract the IMDS-credential and ECR-auth flow out of
ecr-credential-provider into internal/imdscreds and internal/ecrauth so
the agent and the kubelet provider share one implementation.

Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2
subject builders and wire payloads, shared by the agent now and the
scheduler in Sprint 4e.

* build: pin genproto via replace to fix containerd/v2 ambiguous import

go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto
into the module graph; it still ships googleapis/rpc/status, which collides
with the split genproto/googleapis/rpc module that containerd/ttrpc needs.

The main module never imports the monolith, so a plain require bump is
dropped by 'go mod tidy'. Pin it with a replace to a post-split version so
the fix survives tidy and contributors can run 'go mod tidy' freely.

* fix(ecs): agent boot failsafes from runtime verify

- Build the ECR gateway HTTP client lazily on first image pull instead of
  at boot. A missing/malformed gateway CA no longer kills the agent before
  it can register + heartbeat, matching the best-effort containerd path.
- Actively probe containerd with IsServing in runtime.New. The client dials
  lazily, so a dead socket previously slipped past the boot-time
  'containerd unavailable' warn and only surfaced on the first pull.
- Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL
  (default 30s) for test/verify and slow-link tuning.
2026-06-24 21:18:03 +10:00
Julian Sommer ea03efe415 fix: reset-dev-env regex 2026-06-24 16:48:06 +10:00
Julian Sommer fa066a64d7 fix 2026-06-24 16:37:58 +10:00
Josh Siviour 78739cc93b feat(ecs): Phase 4 gateway scaffolding + KV constants (Sprint 4a) (#413)
* fix: drop removed predastore s3.Config.Compaction from admin_test

The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.

* feat(ecs): Phase 4 gateway scaffolding + KV constants (Sprint 4a)

ECS v1 control-plane skeleton. Every action is a 501 NotImplemented stub
until its handler lands in a later Phase 4 sprint; this sprint locks the
API contract + JetStream KV schema so sprints 4c-4h can build against it.

- gateway/ecs: X-Amz-Target JSON 1.1 dispatch (AmazonEC2ContainerServiceV20141113),
  full v1 action namespace mapped to NotImplemented (mirrors gateway/ecrapi)
- gateway: register ecs in supportedServices + Request dispatch switch
- handlers/ecs/store.go: per-account bucket ecs-account-{accountID} + shared
  leader bucket spinifex-ecs-leader + Q2 key-path helpers (mirrors handlers/eks)
- NATS_HANDLER_NAMING.md: document the two-layer ecs.* subject convention (Q14)

Ref: docs/development/feature/ecs-v1.md, ecs-4a-gateway-scaffolding.md

* test(ecs): cover store factories + gateway handler/error branches

Lift Sprint 4a diff coverage above the 60% gate (54.5% -> 78%):
- handlers/ecs: JetStream-backed GetOrCreateAccountBucket / InitLeaderBucket
  idempotency + NewStore success (testutil.StartTestJetStream)
- gateway/ecs: in-package NotImplemented + WriteJSONResponse coverage
- gateway: ECS_Request missing-account-ID branch
2026-06-24 15:15:51 +10:00
Julian Sommer 1a4046fd69 chore: delete bad tests 2026-06-24 14:43:52 +10:00
Josh Siviour 9e333c8a6c feat: graceful guest drain on systemctl stop / host shutdown (#412)
* fix: drop removed predastore s3.Config.Compaction from admin_test

The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.

* feat: graceful guest drain on systemctl stop / host shutdown

systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.

Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.

Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.

* fix: ExecStop drain timeout needs duration unit (120 -> 120s)

The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.

* fix: nbdkitInUse read ss state from correct column + cover it

ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
2026-06-24 13:31:14 +10:00
Julian Sommer f45c97215a chore: bump total coverage requirement 2026-06-24 09:32:31 +10:00
Julian Sommer 98db2c4e0c feat(imds): serve network/interfaces/macs/<mac> subtree (single-NIC) (#410) 2026-06-24 09:27:40 +10:00
Julian Sommer e11437cec9 feat(imds): per-tap responder datapath (#408) 2026-06-23 16:44:53 +10:00
552 changed files with 32710 additions and 86279 deletions
+102
View File
@@ -0,0 +1,102 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="470" viewBox="0 0 900 470">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="inet-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb9501f"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="gw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#DC26261f"/>
<stop offset="100%" style="stop-color:#DC262608"/>
</linearGradient>
</defs>
<rect width="900" height="470" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">ECS · EC2 LAUNCH TYPE ON SPINIFEX</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">container instances from spinifex-ecs-node AMI · awsvpc + ALB</text>
<!-- Internet -->
<rect x="120" y="52" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="220" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
<!-- connector to ALB -->
<line x1="220" y1="92" x2="220" y2="112" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="216,108 224,108 220,116" fill="#71717a" opacity="0.7"/>
<!-- ALB -->
<g transform="translate(90, 116)">
<rect width="260" height="60" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">Application Load Balancer</text>
<line x1="16" y1="32" x2="244" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787">target group · target_type = ip</text>
</g>
<!-- Spinifex Gateway (control plane) -->
<g transform="translate(560, 110)">
<rect width="316" height="72" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#F87171" font-weight="700">Spinifex Gateway :9999</text>
<text x="296" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="end">control plane</text>
<line x1="16" y1="32" x2="300" y2="32" stroke="#DC2626" stroke-opacity="0.3" stroke-width="0.5"/>
<text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5">ECS agent registers · TLS + SigV4 (IMDS creds)</text>
<text x="20" y="63" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5">task role creds → container at 169.254.170.2</text>
</g>
<!-- ALB fan-out to subnets (traffic) -->
<line x1="220" y1="176" x2="220" y2="200" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<line x1="220" y1="200" x2="650" y2="200" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<line x1="220" y1="200" x2="220" y2="228" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<line x1="650" y1="200" x2="650" y2="228" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<text x="435" y="195" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="middle">balances to each task's ENI</text>
<!-- agent path: subnets up to gateway -->
<line x1="718" y1="228" x2="718" y2="182" stroke="#F87171" stroke-width="1.1" stroke-dasharray="3,3" opacity="0.6"/>
<polygon points="714,186 722,186 718,178" fill="#F87171" opacity="0.8"/>
<!-- subnet_a -->
<g transform="translate(60, 228)">
<rect width="320" height="156" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">subnet_a</text>
<text x="300" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="304" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<!-- container instance -->
<rect x="20" y="42" width="280" height="98" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.55" stroke-width="0.8"/>
<text x="34" y="60" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" font-weight="700">container instance</text>
<text x="286" y="60" font-family="'Segoe UI', sans-serif" font-size="9" fill="#d8b4fe" text-anchor="end">AMI: spinifex-ecs-node</text>
<text x="34" y="74" font-family="'Segoe UI', sans-serif" font-size="8.5" fill="#a1a1aa">ecsInstanceRole · ECS agent</text>
<rect x="34" y="82" width="118" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="93" y="102" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">task (web)</text>
<text x="93" y="116" font-family="'Segoe UI', sans-serif" font-size="8" fill="#a1a1aa" text-anchor="middle">awsvpc ENI</text>
<rect x="166" y="82" width="118" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="225" y="102" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">task (web)</text>
<text x="225" y="116" font-family="'Segoe UI', sans-serif" font-size="8" fill="#a1a1aa" text-anchor="middle">awsvpc ENI</text>
</g>
<!-- subnet_b -->
<g transform="translate(520, 228)">
<rect width="320" height="156" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">subnet_b</text>
<text x="300" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="304" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="42" width="280" height="98" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.55" stroke-width="0.8"/>
<text x="34" y="60" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" font-weight="700">container instance</text>
<text x="286" y="60" font-family="'Segoe UI', sans-serif" font-size="9" fill="#d8b4fe" text-anchor="end">AMI: spinifex-ecs-node</text>
<text x="34" y="74" font-family="'Segoe UI', sans-serif" font-size="8.5" fill="#a1a1aa">ecsInstanceRole · ECS agent</text>
<rect x="34" y="82" width="118" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="93" y="102" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">task (web)</text>
<text x="93" y="116" font-family="'Segoe UI', sans-serif" font-size="8" fill="#a1a1aa" text-anchor="middle">awsvpc ENI</text>
<rect x="166" y="82" width="118" height="50" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="225" y="103" font-family="'SF Mono', ui-monospace, monospace" font-size="9.5" fill="#7ee787" text-anchor="middle">image pull</text>
<text x="225" y="117" font-family="'Segoe UI', sans-serif" font-size="8" fill="#7ee787" text-anchor="middle">registry / ECR</text>
</g>
<!-- footer note -->
<text x="450" y="410" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#71717a" text-anchor="middle">No ECS launch API — you add capacity by booting EC2 instances from the spinifex-ecs-node AMI (the console's "Provision capacity" wraps RunInstances)</text>
<text x="450" y="426" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#71717a" text-anchor="middle">The service keeps the desired task count running and registers each task's ENI IP with the ALB target group</text>
</svg>

After

Width:  |  Height:  |  Size: 8.4 KiB

+96
View File
@@ -0,0 +1,96 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="470" viewBox="0 0 900 470">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="inet-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb95022"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb9501f"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="cp-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#DC26261f"/>
<stop offset="100%" style="stop-color:#DC262608"/>
</linearGradient>
</defs>
<rect width="900" height="470" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">EKS · MANAGED KUBERNETES ON SPINIFEX</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">control plane VM + NLB · workers from eks-node AMI</text>
<!-- Internet / kubectl -->
<rect x="330" y="52" width="240" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
<text x="450" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Internet · kubectl + aws eks</text>
<!-- connector to control plane -->
<line x1="450" y1="92" x2="450" y2="112" stroke="#71717a" stroke-width="1" opacity="0.5"/>
<polygon points="446,108 454,108 450,116" fill="#71717a" opacity="0.7"/>
<!-- Control plane (Spinifex-managed) -->
<g transform="translate(290, 116)">
<rect width="320" height="76" rx="8" fill="url(#cp-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#F87171" font-weight="700">EKS Control Plane</text>
<text x="300" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="end">Spinifex-managed</text>
<line x1="16" y1="32" x2="304" y2="32" stroke="#DC2626" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="42" width="135" height="22" rx="4" fill="#DC26261a" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="87" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#FCA5A5" text-anchor="middle" font-weight="600">API NLB (public)</text>
<rect x="165" y="42" width="135" height="22" rx="4" fill="#DC26261a" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="232" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#FCA5A5" text-anchor="middle" font-weight="600">apiserver VM</text>
</g>
<!-- authn note -->
<rect x="624" y="138" width="172" height="32" rx="6" fill="#0a0a0a" stroke="#71717a" stroke-width="0.6" stroke-opacity="0.5"/>
<text x="710" y="151" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">authn: API · access entries</text>
<text x="710" y="163" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">aws eks get-token (IAM → RBAC)</text>
<!-- fan-out to node-group subnets -->
<line x1="450" y1="192" x2="450" y2="210" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<line x1="230" y1="210" x2="670" y2="210" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<line x1="230" y1="210" x2="230" y2="228" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<line x1="670" y1="210" x2="670" y2="228" stroke="#3fb950" stroke-width="1" stroke-dasharray="3,3" opacity="0.55"/>
<text x="450" y="206" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="middle">managed node group · workers join the control plane</text>
<!-- subnet_a -->
<g transform="translate(108, 228)">
<rect width="320" height="150" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">subnet_a</text>
<text x="300" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="304" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<!-- worker node -->
<rect x="20" y="42" width="280" height="92" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.55" stroke-width="0.8"/>
<text x="34" y="60" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" font-weight="700">worker node</text>
<text x="286" y="60" font-family="'Segoe UI', sans-serif" font-size="9" fill="#d8b4fe" text-anchor="end">AMI: eks-node</text>
<rect x="34" y="72" width="76" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="72" y="100" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">pod</text>
<rect x="122" y="72" width="76" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="160" y="100" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">pod</text>
<rect x="210" y="72" width="76" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="248" y="100" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">pod</text>
</g>
<!-- subnet_b -->
<g transform="translate(472, 228)">
<rect width="320" height="150" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">subnet_b</text>
<text x="300" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="304" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="42" width="280" height="92" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.55" stroke-width="0.8"/>
<text x="34" y="60" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" font-weight="700">worker node</text>
<text x="286" y="60" font-family="'Segoe UI', sans-serif" font-size="9" fill="#d8b4fe" text-anchor="end">AMI: eks-node</text>
<rect x="34" y="72" width="76" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="72" y="100" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">pod</text>
<rect x="122" y="72" width="76" height="50" rx="4" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="160" y="100" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">pod</text>
<rect x="210" y="72" width="76" height="50" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.6"/>
<text x="248" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="9.5" fill="#7ee787" text-anchor="middle">ECR</text>
<text x="248" y="107" font-family="'Segoe UI', sans-serif" font-size="8" fill="#7ee787" text-anchor="middle">pull</text>
</g>
<!-- footer note -->
<text x="450" y="404" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#71717a" text-anchor="middle">You provide the VPC, subnets, and IAM roles · Spinifex provisions the control plane, node-group VMs, and security groups</text>
<text x="450" y="420" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#71717a" text-anchor="middle">Node groups always boot the registered eks-node image · workers need egress (IGW / NAT) to pull images</text>
</svg>

After

Width:  |  Height:  |  Size: 8.2 KiB

+12 -12
View File
@@ -51,7 +51,7 @@ jobs:
# ---------------------------------------------------------------------------
single_provision:
name: Single env · provision
runs-on: [ self-hosted, ci-single ]
runs-on: [self-hosted, ci-single]
timeout-minutes: 40
outputs:
wan_ip: ${{ steps.vm.outputs.wan_ip }}
@@ -199,7 +199,7 @@ jobs:
ssh "${SSH_OPTS[@]}" "$VM" \
"spx admin images import --file /tmp/eks-node-alpine.raw \
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null 2>&1
ok "eks-node AMI imported"
- name: Install kubectl on VM
@@ -266,7 +266,7 @@ jobs:
single_suite:
name: Single env · suite
needs: single_provision
runs-on: [ self-hosted, ci-single ]
runs-on: [self-hosted, ci-single]
timeout-minutes: 45
strategy:
fail-fast: false
@@ -373,9 +373,9 @@ jobs:
single_teardown:
name: Single env · teardown
needs: [ single_provision, single_suite ]
needs: [single_provision, single_suite]
if: always()
runs-on: [ self-hosted, ci-single ]
runs-on: [self-hosted, ci-single]
timeout-minutes: 20
env:
TOFU_LOG: /tmp/e2e-tofu-single-td-${{ github.run_id }}.log
@@ -468,7 +468,7 @@ jobs:
multi_provision:
name: Multi env · provision
if: ${{ inputs.run_multinode }}
runs-on: [ self-hosted, ci-multi ]
runs-on: [self-hosted, ci-multi]
timeout-minutes: 45
outputs:
node1_ip: ${{ steps.cluster.outputs.node1_ip }}
@@ -615,12 +615,12 @@ jobs:
multi_suite:
name: Multi env · suite
needs: multi_provision
runs-on: [ self-hosted, ci-multi ]
runs-on: [self-hosted, ci-multi]
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
suite: [ multinode ]
suite: [multinode]
env:
NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
@@ -723,9 +723,9 @@ jobs:
multi_teardown:
name: Multi env · teardown
needs: [ multi_provision, multi_suite ]
needs: [multi_provision, multi_suite]
if: always() && inputs.run_multinode
runs-on: [ self-hosted, ci-multi ]
runs-on: [self-hosted, ci-multi]
timeout-minutes: 20
env:
TOFU_LOG: /tmp/e2e-tofu-multi-td-${{ github.run_id }}.log
@@ -786,8 +786,8 @@ jobs:
e2e_suites_status:
name: E2E suite blocks · summary
if: always()
needs: [ single_suite, multi_suite ]
runs-on: ubuntu-latest
needs: [single_suite, multi_suite]
runs-on: ubuntu-26.04
timeout-minutes: 5
steps:
- name: Summarize
+3 -9
View File
@@ -38,7 +38,7 @@ jobs:
# can compare predicted vs actual before flipping it on.
select_suites:
name: Select Suites (shadow)
runs-on: ubuntu-latest
runs-on: ubuntu-26.04
timeout-minutes: 5
steps:
- uses: actions/checkout@v7
@@ -224,13 +224,10 @@ jobs:
scp "${SSH_OPTS[@]}" \
/tmp/eks-node-image-build/eks-node-alpine.raw \
"$VM":/tmp/eks-node-alpine.raw
# Drop the import progress bar (pterm redraws to stdout — thousands of
# lines over the SSH pipe). Errors go to stderr, so they survive; the
# exit code still gates the step.
ssh "${SSH_OPTS[@]}" "$VM" \
"spx admin images import --file /tmp/eks-node-alpine.raw \
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null 2>&1
ok "eks-node AMI imported"
# The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
@@ -668,9 +665,6 @@ jobs:
echo "=== systemctl is-active spinifex-* ==="
ssh "${SSH_OPTS[@]}" "$VM" 'for s in spinifex-nats spinifex-predastore spinifex-awsgw spinifex-vpcd spinifex-daemon; do printf "%-22s %s\n" "$s" "$(systemctl is-active $s 2>&1)"; done'
echo
echo "=== which spx + version ==="
ssh "${SSH_OPTS[@]}" "$VM" 'which spx; spx --version 2>&1 || true'
echo
echo "=== spx get nodes (direct ssh, tf-user) ==="
ssh "${SSH_OPTS[@]}" "$VM" 'spx get nodes --timeout 5s 2>&1; echo "exit=$?"'
echo
@@ -894,7 +888,7 @@ jobs:
name: Report E2E Status
if: always()
needs: [e2e_single, e2e_multi]
runs-on: ubuntu-latest
runs-on: ubuntu-26.04
timeout-minutes: 5
steps:
- name: Report Status to PR
-1
View File
@@ -57,7 +57,6 @@ linters:
check-type-assertions: true
exclude-functions:
- (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
- (*github.com/kdomanski/iso9660.ImageWriter).Cleanup # deferred cleanup, errors not actionable
govet:
enable-all: true
+11 -1
View File
@@ -68,6 +68,10 @@ build-lb-agent:
@echo -e "\n....Building lb-agent (static)"
CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
build-ecs-agent: ## Build the ecs-agent (ships in the ECS-AMI guest; not in host `build`)
@echo -e "\n....Building ecs-agent (static)"
CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w -X main.version=$(VERSION)" -o ./bin/ecs-agent ./cmd/ecs-agent
build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-node)
ifndef IMAGE
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
@@ -90,6 +94,12 @@ import-eks-node-image: ## Build + register the eks-node AMI (requires a running
publish-eks-node-image: ## Build + publish the eks-node AMI to Cloudflare R2 (needs R2_ENDPOINT + AWS_* env)
./scripts/publish-system-image.sh scripts/images/eks-node/manifest.conf --build
build-ecs-node-image: ## Build the spinifex-ecs-node AMI (Alpine + containerd + ecs-agent; IMPORT=1 to register)
$(MAKE) build-system-image IMAGE=ecs-agent
import-ecs-node-image: ## Build + register the ecs-node AMI (requires a running cluster)
$(MAKE) build-system-image IMAGE=ecs-agent IMPORT=1
MICROVM_OUT_DIR := build/microvm
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
@@ -287,7 +297,7 @@ distro-arm64:
distro-clean:
rm -rf dist/
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck \
+50
View File
@@ -0,0 +1,50 @@
#!/bin/sh
# JetStream liveness watchdog for spinifex-nats.
#
# NATS can disable JetStream at runtime (e.g. a filesystem permission error while
# flushing stream state) and keep the process alive serving KV as read-only. That
# is not a process failure, so Restart=on-failure never fires and KV-backed
# services (awsgw IAM bucket init, etc.) stall until NATS is restarted by hand.
#
# This probes the NATS monitoring endpoint's JS health and restarts the service
# only when JetStream is sustainedly unhealthy while the service is running. The
# multi-sample window avoids restarting during transient startup/quorum 503s.
set -eu
HEALTH_URL="http://127.0.0.1:8222/healthz?js-enabled-only=true"
SERVICE="spinifex-nats.service"
SAMPLES=3 # consecutive failing samples required before acting
SAMPLE_GAP=2 # seconds between samples
# Only act on a running service: a stopped/activating unit is systemd's to manage,
# and try-restart on it would either no-op or fight an in-flight (re)start.
if ! systemctl is-active --quiet "$SERVICE"; then
exit 0
fi
# curl is the probe transport; if it is unavailable, do nothing rather than
# false-restart a healthy server.
if ! command -v curl >/dev/null 2>&1; then
echo "nats-js-watchdog: curl not found, skipping JS health probe" >&2
exit 0
fi
# A single 200 means JetStream is healthy — exit fast on the common case.
js_healthy() {
curl -fsS -o /dev/null --max-time 5 "$HEALTH_URL" 2>/dev/null
}
i=1
while [ "$i" -le "$SAMPLES" ]; do
if js_healthy; then
exit 0
fi
if [ "$i" -lt "$SAMPLES" ]; then
sleep "$SAMPLE_GAP"
fi
i=$((i + 1))
done
echo "nats-js-watchdog: JetStream unhealthy on $SAMPLES consecutive probes ($HEALTH_URL); restarting $SERVICE" >&2
systemctl try-restart "$SERVICE"
+3 -1
View File
@@ -34,7 +34,9 @@ DeviceAllow=/dev/net/tun rw
DeviceAllow=char-vfio rw
DeviceAllow=/dev/vfio/vfio rw
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
# /proc/sys/net/ipv4/conf: per-tap IMDS endpoints set rp_filter/accept_local, which
# ProtectKernelTunables=yes would otherwise keep read-only.
ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd /proc/sys/net/ipv4/conf
# RG-9: hardening baseline
ProtectHome=yes
@@ -0,0 +1,22 @@
[Unit]
Description=Spinifex NATS JetStream liveness watchdog
PartOf=spinifex.target
After=spinifex-nats.service
[Service]
Type=oneshot
# Runs as root: the probe is a local curl to the NATS monitoring port, and the
# remedy is `systemctl try-restart spinifex-nats` when JetStream is disabled while
# the process stays alive (which Restart=on-failure cannot catch).
ExecStart=/var/lib/spinifex/nats-js-watchdog.sh
# Hardening (kept compatible with talking to PID1 over /run/systemd).
NoNewPrivileges=yes
ProtectHome=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictRealtime=yes
LockPersonality=yes
RestrictSUIDSGID=yes
@@ -0,0 +1,13 @@
[Unit]
Description=Spinifex NATS JetStream liveness watchdog timer
PartOf=spinifex.target
[Timer]
# Delay the first probe so NATS startup + JetStream cluster quorum settle (a
# forming cluster reports JS not-current transiently, which is not a fault).
OnBootSec=60s
OnUnitActiveSec=30s
AccuracySec=5s
[Install]
WantedBy=timers.target
+18
View File
@@ -0,0 +1,18 @@
[Unit]
Description=Spinifex graceful guest drain
PartOf=spinifex.target
# After= all services so it stops FIRST on target/host shutdown: the drain runs
# while nats/predastore/viperblock/daemon are still up (QMP, unmount, WAL flush).
After=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
# ExecStop only runs when the target/host goes down, not on a single-service
# restart — so a daemon restart still leaves guests running (DDIL).
ExecStop=/usr/local/bin/spx admin node drain --local --timeout=120s
TimeoutStopSec=180
[Install]
WantedBy=spinifex.target
+22 -14
View File
@@ -17,26 +17,34 @@ OOMScoreAdjust=-500
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
# RG-9: network tier. vpcd MUST share the host mount namespace — IMDS enters
# per-VPC netns via setns(), and any filesystem-sandbox directive (ProtectSystem/
# Home, PrivateTmp, ProtectKernel*, ProtectControlGroups/Proc, Read*Paths) forks a
# private mount ns that traps the /run/netns bind-mount and breaks setns(2). Those,
# plus MemoryDenyWriteExecute, are omitted by design — not regressions.
PrivateDevices=no
RestrictSUIDSGID=no
# RG-9: network tier. Per-tap IMDS removed the in-process setns, so vpcd no
# longer enters a per-VPC netns: CAP_SYS_ADMIN is dropped and the filesystem
# sandbox is restored. NoNewPrivileges stays off (RG-10: sudo ip/ovs-vsctl/dhcpcd).
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
NoNewPrivileges=no
ProtectSystem=full
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
# RG-9: hardening baseline
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
# mnt alongside net: in-netns `ip` unshares a mount ns for its own /sys
RestrictNamespaces=net mnt
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=no
MemoryDenyWriteExecute=yes
# AF_PACKET: raw L2 BOOTP for the in-process DHCP client before the iface has an IP
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
# Caps: NET_ADMIN (ovs/ip), NET_RAW (AF_PACKET), NET_BIND_SERVICE (:68), SETUID/SETGID, SYS_ADMIN (setns into VPC netns)
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
NoNewPrivileges=no
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
+1 -1
View File
@@ -2,7 +2,7 @@
Description=Spinifex Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service openvswitch-switch.service ovn-controller.service
Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
[Install]
WantedBy=multi-user.target
+11 -174
View File
@@ -13,9 +13,6 @@ package main
import (
"bufio"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"encoding/json"
"fmt"
"io"
@@ -25,11 +22,8 @@ import (
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ecr"
"github.com/mulgadc/spinifex/internal/tlsconfig"
"github.com/mulgadc/spinifex/internal/ecrauth"
"github.com/mulgadc/spinifex/internal/imdscreds"
_ "github.com/mulgadc/spinifex/internal/fipsboot"
)
@@ -44,7 +38,6 @@ const (
defaultEnvFile = "/etc/spinifex-eks/agent.env"
defaultGatewayCA = "/etc/spinifex-eks/gateway-ca.pem"
defaultIMDSBase = "http://169.254.169.254/latest"
imdsTokenTTL = "21600"
)
// CredentialProviderRequest is the kubelet exec-provider request on STDIN.
@@ -80,7 +73,7 @@ type config struct {
func main() {
cfg := loadConfig(defaultEnvFile)
httpClient, err := gatewayHTTPClient(cfg.GatewayCA)
httpClient, err := ecrauth.GatewayHTTPClient(cfg.GatewayCA)
if err != nil {
slog.Error("ecr-credential-provider: build gateway client", "err", err)
emitEmpty(os.Stdout)
@@ -106,155 +99,35 @@ func run(in io.Reader, out io.Writer, cfg config, httpClient *http.Client) error
if req.Image == "" {
return fmt.Errorf("request image is empty")
}
imageHost := hostFromImage(req.Image)
imageHost := ecrauth.HostFromImage(req.Image)
creds, err := fetchIMDSCredentials(httpClient, cfg.IMDSBase)
creds, err := imdscreds.Fetch(httpClient, cfg.IMDSBase)
if err != nil {
return fmt.Errorf("fetch IMDS credentials: %w", err)
}
username, password, proxyHost, expiresAt, err := getAuthToken(cfg, httpClient, creds)
tok, err := ecrauth.GetAuthorizationToken(cfg.Region, cfg.GatewayURL, httpClient,
creds.AccessKeyID, creds.SecretAccessKey, creds.SessionToken)
if err != nil {
return fmt.Errorf("get authorization token: %w", err)
}
auth := AuthConfig{Username: username, Password: password}
auth := AuthConfig{Username: tok.Username, Password: tok.Password}
authMap := map[string]AuthConfig{imageHost: auth}
if proxyHost != "" && proxyHost != imageHost {
authMap[proxyHost] = auth
if tok.ProxyHost != "" && tok.ProxyHost != imageHost {
authMap[tok.ProxyHost] = auth
}
resp := CredentialProviderResponse{
APIVersion: credProviderAPIVersion,
Kind: "CredentialProviderResponse",
CacheKeyType: "Registry",
CacheDuration: cacheDurationFrom(expiresAt),
CacheDuration: cacheDurationFrom(tok.ExpiresAt),
Auth: authMap,
}
return json.NewEncoder(out).Encode(&resp)
}
// imdsCreds mirrors the IMDS security-credentials JSON shape.
type imdsCreds struct {
Code string `json:"Code"`
AccessKeyID string `json:"AccessKeyId"`
SecretAccessKey string `json:"SecretAccessKey"`
Token string `json:"Token"`
Expiration string `json:"Expiration"`
}
// fetchIMDSCredentials resolves the node role name then its credentials via
// IMDSv2 (PUT token, then GET with the token header).
func fetchIMDSCredentials(client *http.Client, base string) (imdsCreds, error) {
base = strings.TrimRight(base, "/")
token := imdsV2Token(client, base)
roleName, err := imdsGet(client, base+"/meta-data/iam/security-credentials/", token)
if err != nil {
return imdsCreds{}, fmt.Errorf("fetch role name: %w", err)
}
roleName = strings.TrimSpace(roleName)
if roleName == "" {
return imdsCreds{}, fmt.Errorf("IMDS returned empty role name")
}
body, err := imdsGet(client, base+"/meta-data/iam/security-credentials/"+roleName, token)
if err != nil {
return imdsCreds{}, fmt.Errorf("fetch role credentials: %w", err)
}
var creds imdsCreds
if err := json.Unmarshal([]byte(body), &creds); err != nil {
return imdsCreds{}, fmt.Errorf("decode role credentials: %w", err)
}
if creds.AccessKeyID == "" || creds.SecretAccessKey == "" {
return imdsCreds{}, fmt.Errorf("IMDS credentials missing access/secret key")
}
return creds, nil
}
// imdsV2Token requests an IMDSv2 session token; returns "" if the service does
// not enforce v2 (tokenless v1 then applies).
func imdsV2Token(client *http.Client, base string) string {
req, err := http.NewRequest(http.MethodPut, base+"/api/token", nil)
if err != nil {
return ""
}
req.Header.Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", imdsTokenTTL)
resp, err := client.Do(req)
if err != nil {
return ""
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return ""
}
body, _ := io.ReadAll(resp.Body)
return strings.TrimSpace(string(body))
}
func imdsGet(client *http.Client, url, token string) (string, error) {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return "", err
}
if token != "" {
req.Header.Set("X-Aws-Ec2-Metadata-Token", token)
}
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("IMDS %s returned %d: %s", url, resp.StatusCode, strings.TrimSpace(string(body)))
}
return string(body), nil
}
// getAuthToken builds an aws-sdk ECR client pointed at the gateway and calls
// GetAuthorizationToken, decoding the AWS:<jwt> token into Basic-auth parts. It
// returns the username, password, the ProxyEndpoint host, and the token expiry.
func getAuthToken(cfg config, httpClient *http.Client, creds imdsCreds) (username, password, proxyHost string, expiresAt time.Time, err error) {
region := cfg.Region
if region == "" {
region = "us-east-1"
}
sess, serr := session.NewSession(&aws.Config{
Region: aws.String(region),
Endpoint: aws.String(cfg.GatewayURL),
Credentials: credentials.NewStaticCredentials(creds.AccessKeyID, creds.SecretAccessKey, creds.Token),
HTTPClient: httpClient,
DisableSSL: aws.Bool(false),
})
if serr != nil {
return "", "", "", time.Time{}, fmt.Errorf("new session: %w", serr)
}
out, gerr := ecr.New(sess).GetAuthorizationToken(&ecr.GetAuthorizationTokenInput{})
if gerr != nil {
return "", "", "", time.Time{}, fmt.Errorf("GetAuthorizationToken: %w", gerr)
}
if len(out.AuthorizationData) == 0 || out.AuthorizationData[0].AuthorizationToken == nil {
return "", "", "", time.Time{}, fmt.Errorf("GetAuthorizationToken returned no authorization data")
}
data := out.AuthorizationData[0]
raw, derr := base64.StdEncoding.DecodeString(aws.StringValue(data.AuthorizationToken))
if derr != nil {
return "", "", "", time.Time{}, fmt.Errorf("decode authorization token: %w", derr)
}
user, pass, ok := strings.Cut(string(raw), ":")
if !ok {
return "", "", "", time.Time{}, fmt.Errorf("authorization token not in user:password form")
}
if data.ExpiresAt != nil {
expiresAt = aws.TimeValue(data.ExpiresAt)
}
return user, pass, hostFromImage(aws.StringValue(data.ProxyEndpoint)), expiresAt, nil
}
// cacheDurationFrom derives a Go-duration string from the token expiry minus a
// safety margin, falling back to the default when expiry is unknown or too near.
func cacheDurationFrom(expiresAt time.Time) string {
@@ -268,30 +141,6 @@ func cacheDurationFrom(expiresAt time.Time) string {
return d.Round(time.Second).String()
}
// gatewayHTTPClient builds an HTTP client trusting the gateway CA at caPath.
// When caPath is empty it relies on the system trust store.
func gatewayHTTPClient(caPath string) (*http.Client, error) {
tlsCfg := &tls.Config{
MinVersion: tls.VersionTLS13,
CurvePreferences: tlsconfig.Curves,
}
if caPath != "" {
pem, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
}
tlsCfg.RootCAs = pool
}
return &http.Client{
Timeout: 15 * time.Second,
Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
}, nil
}
// loadConfig reads the cloud-init env file then lets real env vars override. The
// IMDS base is fixed (not in the env file) but overridable for tests.
func loadConfig(envFile string) config {
@@ -341,18 +190,6 @@ func parseEnvFile(path string) map[string]string {
return out
}
// hostFromImage returns the host[:port] portion of an image ref or URL. It strips
// any scheme and trims at the first path separator.
func hostFromImage(image string) string {
if i := strings.Index(image, "://"); i >= 0 {
image = image[i+3:]
}
if i := strings.IndexByte(image, '/'); i >= 0 {
image = image[:i]
}
return image
}
// emitEmpty writes a valid CredentialProviderResponse with no auth entries.
func emitEmpty(out io.Writer) {
resp := CredentialProviderResponse{
-14
View File
@@ -122,17 +122,3 @@ func TestRun_ReturnsBasicAuth(t *testing.T) {
t.Errorf("password = %q, want testjwt", auth.Password)
}
}
func TestHostFromImage(t *testing.T) {
cases := map[string]string{
"acct.dkr.ecr.us-east-1.spinifex.internal/demo:latest": "acct.dkr.ecr.us-east-1.spinifex.internal",
"https://acct.dkr.ecr.us-east-1.spinifex.internal": "acct.dkr.ecr.us-east-1.spinifex.internal",
"host:9999/repo:tag": "host:9999",
"plainhost": "plainhost",
}
for in, want := range cases {
if got := hostFromImage(in); got != want {
t.Errorf("hostFromImage(%q) = %q, want %q", in, got, want)
}
}
}
+188
View File
@@ -0,0 +1,188 @@
package main
import (
"bufio"
"context"
"fmt"
"log/slog"
"net/http"
"os"
goruntime "runtime"
"strconv"
"strings"
"time"
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// version is the agent build version, reported in the register message.
// Overridable via -ldflags "-X main.version=...".
var version = "dev"
// Agent wires the ecs-agent's runtime seams: a gateway control-plane client, a
// container runtime, and an ECR resolver. It registers the host as a container
// instance at boot, heartbeats (re-registers) while alive, polls the gateway for
// task assignments, and runs them through containerd, reporting state back over
// the gateway. It never connects to NATS — the bus stays host-internal.
type Agent struct {
cfg config
id identity
cp controlPlane
puller ctrruntime.ImagePuller
runner ctrruntime.Runner
resolver ctrruntime.Resolver
reg *registrar
hb *heartbeater
// netns builds awsvpc task network namespaces (nil-safe; bridge/host skip it).
netns *taskNetns
// cred serves task IAM role credentials at 169.254.170.2 (nil in unit tests).
cred *credEndpoint
closers []func() error
}
// newAgent assembles an Agent from already-built seams. Tests use this directly
// with fakes; New builds the production seams and delegates here. runner may be
// nil when containerd is unavailable; the assign path then reports the task
// STOPPED rather than crashing.
func newAgent(cfg config, id identity, cp controlPlane, puller ctrruntime.ImagePuller, runner ctrruntime.Runner, resolver ctrruntime.Resolver) *Agent {
return &Agent{
cfg: cfg,
id: id,
cp: cp,
puller: puller,
runner: runner,
resolver: resolver,
reg: newRegistrar(cp, id),
hb: newHeartbeater(cp, id, cfg.Heartbeat),
netns: newTaskNetns(execNetRunner{}),
}
}
// New builds an Agent from config: it resolves the host identity from IMDS,
// builds the gateway control-plane client, the ECR resolver and (best-effort)
// the containerd runtime. A containerd connect failure is logged, not fatal —
// registration and heartbeat still run so the instance is visible while the
// runtime recovers. The ECR gateway client is built lazily on first image pull
// (not here), so a missing or malformed gateway CA does not stop registration.
func New(cfg config) (*Agent, error) {
imdsClient := &http.Client{Timeout: 5 * time.Second}
meta, err := fetchInstanceMetadata(imdsClient, cfg.IMDSBase)
if err != nil {
return nil, fmt.Errorf("instance metadata: %w", err)
}
host, _ := os.Hostname()
id := identity{
AccountID: meta.AccountID,
ClusterName: cfg.ClusterName,
InstanceID: meta.InstanceID,
AZ: meta.AZ,
Hostname: host,
Capacity: detectCapacity(),
AgentVersion: version,
}
creds := credentials.NewIMDSProvider(imdsClient, cfg.IMDSBase)
resolver := newLazyECRResolver(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA)
var puller ctrruntime.ImagePuller
var runner ctrruntime.Runner
if rt, perr := ctrruntime.New(cfg.ContainerdSocket); perr != nil {
slog.Warn("ecs-agent: containerd unavailable at boot, image pulls disabled", "err", perr)
} else {
puller = rt
runner = rt
}
cp, err := newGatewayControlPlane(cfg, creds)
if err != nil {
return nil, fmt.Errorf("build gateway control-plane: %w", err)
}
a := newAgent(cfg, id, cp, puller, runner, resolver)
a.cred = newCredEndpoint(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA,
cfg.CredEndpointIP, cfg.CredEndpointPort, execNetRunner{})
if puller != nil {
a.closers = append(a.closers, puller.Close)
}
return a, nil
}
// Run registers the instance, starts the heartbeat and assignment-poll loops,
// and blocks until ctx is cancelled, then tears down.
func (a *Agent) Run(ctx context.Context) error {
if err := a.reg.Register(); err != nil {
slog.Error("ecs-agent: initial registration failed", "err", err)
} else {
slog.Info("ecs-agent: registered", "cluster", a.id.ClusterName, "instance", a.id.InstanceID)
}
if a.cred != nil {
if err := a.cred.Start(); err != nil {
slog.Error("ecs-agent: credential endpoint start failed", "err", err)
}
}
go a.hb.Run(ctx)
// Re-adopt containers still running from before this restart, then poll with
// their tasks pre-seeded so re-delivered assignments are acked, not re-run.
go a.pollAssignments(ctx, a.reconcile(ctx))
<-ctx.Done()
return a.Stop()
}
// Stop closes the runtime. Safe to call once.
func (a *Agent) Stop() error {
var firstErr error
if a.cred != nil {
_ = a.cred.Stop()
}
for _, c := range a.closers {
if err := c(); err != nil && firstErr == nil {
firstErr = err
}
}
return firstErr
}
// detectCapacity reports the host's total CPU units (1 vCPU = 1024) and memory.
func detectCapacity() bus.InstanceCapacity {
return bus.InstanceCapacity{
CPU: goruntime.NumCPU() * 1024,
MemoryMiB: memTotalMiB(),
}
}
// memTotalMiB reads MemTotal from /proc/meminfo, returning 0 if unavailable.
func memTotalMiB() int {
f, err := os.Open("/proc/meminfo")
if err != nil {
return 0
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := scanner.Text()
if !strings.HasPrefix(line, "MemTotal:") {
continue
}
fields := strings.Fields(line)
if len(fields) < 2 {
return 0
}
kb, err := strconv.Atoi(fields[1])
if err != nil {
return 0
}
return kb / 1024
}
return 0
}
+53
View File
@@ -0,0 +1,53 @@
package main
import (
"context"
"testing"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
)
func TestAgent_RunRegistersThenStopsOnContext(t *testing.T) {
cp := &fakeCP{}
cfg := config{Heartbeat: 5 * time.Millisecond, PollInterval: 5 * time.Millisecond}
puller := &ctrruntime.FakePuller{}
a := newAgent(cfg, testIdentity(), cp, puller, puller, nil)
a.closers = append(a.closers, puller.Close)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
// Give it time to register + re-register (heartbeat) a few times.
time.Sleep(30 * time.Millisecond)
cancel()
select {
case err := <-done:
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
case <-time.After(time.Second):
t.Fatal("Run did not return after cancel")
}
// One boot register + at least one heartbeat re-register.
if cp.registerCount() < 2 {
t.Errorf("registers = %d, want >= 2 (boot + heartbeat)", cp.registerCount())
}
if !puller.Closed {
t.Error("Stop did not close the puller")
}
}
func TestDetectCapacity_Positive(t *testing.T) {
c := detectCapacity()
if c.CPU <= 0 {
t.Errorf("CPU = %d, want > 0", c.CPU)
}
// MemoryMiB may be 0 on platforms without /proc/meminfo; just assert non-negative.
if c.MemoryMiB < 0 {
t.Errorf("MemoryMiB = %d, want >= 0", c.MemoryMiB)
}
}
+119
View File
@@ -0,0 +1,119 @@
package main
import (
"bufio"
"os"
"strconv"
"strings"
"time"
)
const (
defaultEnvFile = "/etc/spinifex-ecs/agent.env"
defaultGatewayCA = "/etc/spinifex-ecs/gateway-ca.pem"
defaultIMDSBase = "http://169.254.169.254/latest"
defaultContainerdSocket = "/run/containerd/containerd.sock"
defaultHeartbeat = 30 * time.Second
defaultPollInterval = 5 * time.Second
)
// config holds the static settings the agent reads at boot. Cluster identity
// (account ID, instance ID) is discovered at runtime from IMDS, not configured;
// only the cluster *name* the instance was launched into is static. AccessKey /
// SecretKey are legacy seeded creds, optional now that the agent SigV4-signs
// gateway calls with instance-role credentials from IMDS; the agent never holds
// a NATS token.
type config struct {
GatewayURL string
GatewayCA string
Region string
ClusterName string
AccessKey string
SecretKey string
IMDSBase string
ContainerdSocket string
Heartbeat time.Duration
PollInterval time.Duration
CredEndpointIP string
CredEndpointPort int
}
// loadConfig reads the cloud-init env file then lets real env vars override.
func loadConfig(envFile string) config {
env := parseEnvFile(envFile)
get := func(key string) string {
if v := os.Getenv(key); v != "" {
return v
}
return env[key]
}
cfg := config{
GatewayURL: get("ECS_GATEWAY_URL"),
GatewayCA: get("ECS_GATEWAY_CA"),
Region: get("ECS_REGION"),
ClusterName: get("ECS_CLUSTER"),
AccessKey: get("ECS_ACCESS_KEY"),
SecretKey: get("ECS_SECRET_KEY"),
IMDSBase: get("ECS_IMDS_BASE"),
ContainerdSocket: get("ECS_CONTAINERD_SOCKET"),
}
if cfg.GatewayCA == "" {
cfg.GatewayCA = defaultGatewayCA
}
if cfg.IMDSBase == "" {
cfg.IMDSBase = defaultIMDSBase
}
if cfg.ContainerdSocket == "" {
cfg.ContainerdSocket = defaultContainerdSocket
}
if cfg.ClusterName == "" {
cfg.ClusterName = "default"
}
cfg.Heartbeat = defaultHeartbeat
if v := get("ECS_HEARTBEAT_INTERVAL"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.Heartbeat = d
}
}
cfg.PollInterval = defaultPollInterval
if v := get("ECS_POLL_INTERVAL"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.PollInterval = d
}
}
cfg.CredEndpointIP = get("ECS_CRED_ENDPOINT_IP")
if cfg.CredEndpointIP == "" {
cfg.CredEndpointIP = defaultCredEndpointIP
}
cfg.CredEndpointPort = defaultCredEndpointPort
if v := get("ECS_CRED_ENDPOINT_PORT"); v != "" {
if p, err := strconv.Atoi(v); err == nil && p > 0 {
cfg.CredEndpointPort = p
}
}
return cfg
}
// parseEnvFile reads a simple KEY=value file; missing files yield an empty map.
func parseEnvFile(path string) map[string]string {
out := map[string]string{}
f, err := os.Open(path)
if err != nil {
return out
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, val, ok := strings.Cut(line, "=")
if !ok {
continue
}
out[strings.TrimSpace(key)] = strings.TrimSpace(val)
}
return out
}
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestLoadConfig_Defaults(t *testing.T) {
// Ensure env vars don't leak in from the host.
for _, k := range []string{"ECS_GATEWAY_URL", "ECS_GATEWAY_CA", "ECS_REGION",
"ECS_CLUSTER", "ECS_ACCESS_KEY", "ECS_SECRET_KEY", "ECS_IMDS_BASE",
"ECS_CONTAINERD_SOCKET", "ECS_HEARTBEAT_INTERVAL", "ECS_POLL_INTERVAL"} {
t.Setenv(k, "")
}
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
if cfg.GatewayCA != defaultGatewayCA {
t.Errorf("GatewayCA = %q, want default", cfg.GatewayCA)
}
if cfg.IMDSBase != defaultIMDSBase {
t.Errorf("IMDSBase = %q, want default", cfg.IMDSBase)
}
if cfg.ContainerdSocket != defaultContainerdSocket {
t.Errorf("ContainerdSocket = %q, want default", cfg.ContainerdSocket)
}
if cfg.ClusterName != "default" {
t.Errorf("ClusterName = %q, want default", cfg.ClusterName)
}
if cfg.Heartbeat != defaultHeartbeat {
t.Errorf("Heartbeat = %v, want %v", cfg.Heartbeat, defaultHeartbeat)
}
if cfg.PollInterval != defaultPollInterval {
t.Errorf("PollInterval = %v, want %v", cfg.PollInterval, defaultPollInterval)
}
}
func TestLoadConfig_FileThenEnvOverride(t *testing.T) {
dir := t.TempDir()
envFile := filepath.Join(dir, "agent.env")
body := "# comment\nECS_GATEWAY_URL=https://gw.file\nECS_REGION=us-west-2\nECS_CLUSTER=prod\n"
if err := os.WriteFile(envFile, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("ECS_GATEWAY_URL", "https://gw.env")
t.Setenv("ECS_ACCESS_KEY", "AKIAENV")
cfg := loadConfig(envFile)
if cfg.GatewayURL != "https://gw.env" {
t.Errorf("GatewayURL = %q, want env override", cfg.GatewayURL)
}
if cfg.Region != "us-west-2" {
t.Errorf("Region = %q, want from file", cfg.Region)
}
if cfg.ClusterName != "prod" {
t.Errorf("ClusterName = %q, want from file", cfg.ClusterName)
}
if cfg.AccessKey != "AKIAENV" {
t.Errorf("AccessKey = %q, want env", cfg.AccessKey)
}
}
func TestLoadConfig_HeartbeatOverride(t *testing.T) {
t.Setenv("ECS_HEARTBEAT_INTERVAL", "5s")
if cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env")); cfg.Heartbeat != 5*time.Second {
t.Errorf("Heartbeat = %v, want 5s", cfg.Heartbeat)
}
// Garbage value falls back to the default, not zero.
t.Setenv("ECS_HEARTBEAT_INTERVAL", "not-a-duration")
if cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env")); cfg.Heartbeat != defaultHeartbeat {
t.Errorf("Heartbeat = %v, want default on bad value", cfg.Heartbeat)
}
}
func TestParseEnvFile_SkipsBlankAndComments(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "f.env")
if err := os.WriteFile(p, []byte("\n# c\nA=1\nnokeyval\nB = 2 \n"), 0o600); err != nil {
t.Fatal(err)
}
m := parseEnvFile(p)
if m["A"] != "1" || m["B"] != "2" {
t.Errorf("parse mismatch: %#v", m)
}
if _, ok := m["nokeyval"]; ok {
t.Errorf("line without = should be skipped")
}
}
+129
View File
@@ -0,0 +1,129 @@
package main
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ecs"
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
"github.com/mulgadc/spinifex/internal/ecsgw"
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// controlPlane is the agent's channel to the AWS gateway. It replaces the direct
// NATS publisher: every method is a SigV4-signed gateway request, so the NATS bus
// stays host-internal. Tests inject a fake; production uses gatewayControlPlane.
type controlPlane interface {
// Register (re-)registers the container instance. Called at boot and on every
// heartbeat tick — the gateway's RegisterContainerInstance is idempotent and
// refreshes LastSeen, folding liveness into registration (no bus.Heartbeat).
Register(id identity) error
// SubmitTaskState reports a task's RUNNING/STOPPED transition.
SubmitTaskState(st bus.TaskState) error
// PollAssignments drains this instance's assignment inbox, acking the taskIDs
// accepted on the previous poll. Returns the still-pending assignments.
PollAssignments(cluster, instance string, ack []string) ([]bus.Assign, error)
}
// gatewayControlPlane implements controlPlane over the SigV4 ECS gateway client.
type gatewayControlPlane struct {
client *ecsgw.Client
}
var _ controlPlane = (*gatewayControlPlane)(nil)
// pollTimeout bounds a single PollAssignments request; larger than the register
// timeout to leave room for a future server-side long-poll.
const pollTimeout = 30 * time.Second
// newGatewayControlPlane builds the SigV4 client signing with the instance-role
// credentials from IMDS (fetched per call so rotation is transparent) + pinned CA.
func newGatewayControlPlane(cfg config, creds credentials.CredentialsProvider) (*gatewayControlPlane, error) {
client, err := ecsgw.New(cfg.GatewayURL, cfg.GatewayCA, credentialsFunc(creds), cfg.Region, pollTimeout)
if err != nil {
return nil, err
}
return &gatewayControlPlane{client: client}, nil
}
// credentialsFunc adapts the agent's IMDS CredentialsProvider to the ecsgw
// per-call credentials hook.
func credentialsFunc(p credentials.CredentialsProvider) ecsgw.CredentialsFunc {
return func(ctx context.Context) (string, string, string, error) {
c, err := p.Retrieve(ctx)
if err != nil {
return "", "", "", err
}
return c.AccessKeyID, c.SecretAccessKey, c.SessionToken, nil
}
}
func (g *gatewayControlPlane) Register(id identity) error {
in := &ecs.RegisterContainerInstanceInput{
Cluster: aws.String(id.ClusterName),
InstanceIdentityDocument: aws.String(id.InstanceID),
TotalResources: []*ecs.Resource{
{Name: aws.String("CPU"), Type: aws.String("INTEGER"), IntegerValue: aws.Int64(int64(id.Capacity.CPU))},
{Name: aws.String("MEMORY"), Type: aws.String("INTEGER"), IntegerValue: aws.Int64(int64(id.Capacity.MemoryMiB))},
},
VersionInfo: &ecs.VersionInfo{AgentVersion: aws.String(id.AgentVersion)},
}
body, err := json.Marshal(in)
if err != nil {
return fmt.Errorf("marshal register: %w", err)
}
if _, err := g.client.Call("RegisterContainerInstance", body); err != nil {
return fmt.Errorf("register: %w", err)
}
return nil
}
func (g *gatewayControlPlane) SubmitTaskState(st bus.TaskState) error {
in := &ecs.SubmitTaskStateChangeInput{
Cluster: aws.String(st.ClusterName),
Task: aws.String(st.TaskID),
Status: aws.String(st.LastStatus),
Reason: aws.String(st.Reason),
}
for _, c := range st.Containers {
csc := &ecs.ContainerStateChange{
ContainerName: aws.String(c.Name),
Status: aws.String(c.Status),
RuntimeId: aws.String(c.ContainerID),
}
if c.ExitCode != nil {
csc.ExitCode = aws.Int64(int64(*c.ExitCode))
}
in.Containers = append(in.Containers, csc)
}
body, err := json.Marshal(in)
if err != nil {
return fmt.Errorf("marshal task-state: %w", err)
}
if _, err := g.client.Call("SubmitTaskStateChange", body); err != nil {
return fmt.Errorf("submit task-state: %w", err)
}
return nil
}
func (g *gatewayControlPlane) PollAssignments(cluster, instance string, ack []string) ([]bus.Assign, error) {
in := &handlers_ecs.PollAssignmentsInput{Cluster: cluster, ContainerInstance: instance, AckTaskIDs: ack}
body, err := json.Marshal(in)
if err != nil {
return nil, fmt.Errorf("marshal poll: %w", err)
}
resp, err := g.client.Call("PollAssignments", body)
if err != nil {
return nil, fmt.Errorf("poll: %w", err)
}
var out handlers_ecs.PollAssignmentsOutput
if err := json.Unmarshal(resp, &out); err != nil {
return nil, fmt.Errorf("decode poll: %w", err)
}
return out.Assignments, nil
}
+361
View File
@@ -0,0 +1,361 @@
package main
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"net"
"net/http"
"strconv"
"strings"
"sync"
"time"
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
"github.com/mulgadc/spinifex/internal/ecrauth"
"github.com/mulgadc/spinifex/internal/stsauth"
)
const (
// defaultCredEndpointIP is the AWS-reserved link-local address tasks reach for
// container credentials; defaultCredEndpointPort is the standard HTTP port.
defaultCredEndpointIP = "169.254.170.2"
defaultCredEndpointPort = 80
// defaultCredProxyPort is the high port the listener actually binds; a nat
// DNAT rule rewrites 169.254.170.2:80 to it so nothing holds a socket on :80
// and host/bridge tasks sharing the netns can bind :80 themselves.
defaultCredProxyPort = 51679
// credRefreshMargin refreshes a cached credential set this long before expiry
// so a container never starts a request with a token about to lapse.
credRefreshMargin = 5 * time.Minute
// credDummyIface is the dummy interface the endpoint IP is added to in the host
// netns so the agent can bind it.
credDummyIface = "ecs-cred"
)
// credResponse is the ECS task-credentials JSON the container's AWS SDK expects
// at AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.
type credResponse struct {
RoleArn string `json:"RoleArn"`
AccessKeyID string `json:"AccessKeyId"`
SecretAccessKey string `json:"SecretAccessKey"`
Token string `json:"Token"`
Expiration string `json:"Expiration"`
}
// credEntry caches one task's assumed-role credentials alongside the role it was
// assumed from.
type credEntry struct {
roleARN string
creds stsauth.Credentials
}
// credEndpoint is the host-netns HTTP server that hands a task its IAM role
// credentials. It maps credID -> taskRoleARN (registered as tasks start) and
// mints/caches credentials on demand by assuming the role over the gateway with
// the agent's instance credentials.
type credEndpoint struct {
provider credentials.CredentialsProvider
region string
gatewayURL string
caPath string
ip string
port int
proxyPort int
run netCmdRunner
// assume mints fresh credentials for a role; defaults to AssumeRole over the
// gateway and is overridable in tests.
assume func(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error)
mu sync.Mutex
roles map[string]string // credID -> taskRoleARN
cache map[string]credEntry // credID -> cached creds
httpClient *http.Client
srv *http.Server
ln net.Listener
}
// newCredEndpoint builds the endpoint. An empty ip/port falls back to the
// AWS-reserved 169.254.170.2:80; tests pass loopback + an ephemeral port. run
// adds the link-local address to the dummy interface (real or fake).
func newCredEndpoint(provider credentials.CredentialsProvider, region, gatewayURL, caPath, ip string, port int, run netCmdRunner) *credEndpoint {
if ip == "" {
ip = defaultCredEndpointIP
}
if port == 0 {
port = defaultCredEndpointPort
}
c := &credEndpoint{
provider: provider,
region: region,
gatewayURL: gatewayURL,
caPath: caPath,
ip: ip,
port: port,
proxyPort: defaultCredProxyPort,
run: run,
roles: map[string]string{},
cache: map[string]credEntry{},
}
c.assume = c.assumeOverGateway
return c
}
// Register associates a credID with the task role it serves; Deregister removes
// it and drops any cached credentials. Both are no-ops for an empty credID.
func (c *credEndpoint) Register(credID, roleARN string) {
if credID == "" || roleARN == "" {
return
}
c.mu.Lock()
c.roles[credID] = roleARN
c.mu.Unlock()
}
func (c *credEndpoint) Deregister(credID string) {
if credID == "" {
return
}
c.mu.Lock()
delete(c.roles, credID)
delete(c.cache, credID)
c.mu.Unlock()
}
// relativeURI is the value injected into a container's
// AWS_CONTAINER_CREDENTIALS_RELATIVE_URI for the given credID.
func credRelativeURI(credID string) string { return "/v2/credentials/" + credID }
// ServeHTTP serves GET /v2/credentials/{credID}. Unknown paths/credIDs return
// 404; non-GET returns 405; an AssumeRole failure returns 502.
func (c *credEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
credID, ok := strings.CutPrefix(r.URL.Path, "/v2/credentials/")
if !ok || credID == "" || strings.Contains(credID, "/") {
http.Error(w, "not found", http.StatusNotFound)
return
}
creds, roleARN, err := c.fetch(r.Context(), credID)
if err != nil {
if err == errUnknownCredID {
http.Error(w, "not found", http.StatusNotFound)
return
}
slog.Error("ecs-agent: assume task role failed", "credId", credID, "err", err)
http.Error(w, "credentials unavailable", http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(credResponse{
RoleArn: roleARN,
AccessKeyID: creds.AccessKeyID,
SecretAccessKey: creds.SecretAccessKey,
Token: creds.SessionToken,
Expiration: creds.Expiration.UTC().Format(time.RFC3339),
}); err != nil {
slog.Warn("ecs-agent: encode credential response", "credId", credID, "err", err)
}
}
// errUnknownCredID signals a credID with no registered task role.
var errUnknownCredID = fmt.Errorf("unknown credId")
// fetch returns valid credentials for credID, reusing the cache until within the
// refresh margin and otherwise assuming the task role over the gateway.
func (c *credEndpoint) fetch(ctx context.Context, credID string) (stsauth.Credentials, string, error) {
c.mu.Lock()
roleARN, known := c.roles[credID]
if !known {
c.mu.Unlock()
return stsauth.Credentials{}, "", errUnknownCredID
}
if e, ok := c.cache[credID]; ok && e.roleARN == roleARN && credValid(e.creds, credRefreshMargin) {
c.mu.Unlock()
return e.creds, roleARN, nil
}
c.mu.Unlock()
creds, err := c.assume(ctx, roleARN, sessionName(credID))
if err != nil {
return stsauth.Credentials{}, "", err
}
c.mu.Lock()
c.cache[credID] = credEntry{roleARN: roleARN, creds: creds}
c.mu.Unlock()
return creds, roleARN, nil
}
// assumeOverGateway mints credentials for roleARN by SigV4-signing AssumeRole
// against the gateway with the agent's instance credentials.
func (c *credEndpoint) assumeOverGateway(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {
instCreds, err := c.provider.Retrieve(ctx)
if err != nil {
return stsauth.Credentials{}, fmt.Errorf("instance credentials: %w", err)
}
client, err := c.client()
if err != nil {
return stsauth.Credentials{}, err
}
return stsauth.AssumeRole(c.region, c.gatewayURL, client,
instCreds.AccessKeyID, instCreds.SecretAccessKey, instCreds.SessionToken,
roleARN, sessionName)
}
// client builds the pinned-CA gateway HTTP client on first use and caches it.
func (c *credEndpoint) client() (*http.Client, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.httpClient != nil {
return c.httpClient, nil
}
hc, err := ecrauth.GatewayHTTPClient(c.caPath)
if err != nil {
return nil, err
}
c.httpClient = hc
return hc, nil
}
// Start adds the endpoint IP to a dummy interface in the host netns, serves HTTP
// on the high proxy port, and DNATs the advertised :80 to it. A bind, address,
// or nat failure is returned so the caller can log it without aborting
// register/heartbeat.
func (c *credEndpoint) Start() error {
if err := c.addEndpointAddr(); err != nil {
return err
}
ln, err := net.Listen("tcp", net.JoinHostPort(c.ip, strconv.Itoa(c.listenPort())))
if err != nil {
return fmt.Errorf("listen %s:%d: %w", c.ip, c.listenPort(), err)
}
if err := c.addRedirect(); err != nil {
_ = ln.Close()
return err
}
c.ln = ln
c.srv = &http.Server{Handler: c, ReadHeaderTimeout: 5 * time.Second}
go func() {
if err := c.srv.Serve(ln); err != nil && err != http.ErrServerClosed {
slog.Error("ecs-agent: credential endpoint serve", "err", err)
}
}()
slog.Info("ecs-agent: credential endpoint listening", "addr", ln.Addr().String())
return nil
}
// Stop shuts the server down and removes the dummy interface. Safe on a nil/
// never-started endpoint.
func (c *credEndpoint) Stop() error {
if c == nil {
return nil
}
if c.srv != nil {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
_ = c.srv.Shutdown(ctx)
}
c.delRedirect()
c.delEndpointAddr()
return nil
}
// listenPort is the port the socket binds. Loopback test mode binds the
// caller-supplied (ephemeral) port directly; the real endpoint binds the high
// proxy port and reaches :80 via the DNAT rules.
func (c *credEndpoint) listenPort() int {
if c.ip == "127.0.0.1" {
return c.port
}
return c.proxyPort
}
// addRedirect installs nat DNAT rules rewriting the advertised :80 to the proxy
// port, for both locally generated traffic (host/bridge task containers, OUTPUT)
// and traffic arriving over the awsvpc credential veth (PREROUTING). The dst IP
// is left on the endpoint IP so no route_localnet handling is needed. Skipped in
// loopback test mode, which binds the port directly.
func (c *credEndpoint) addRedirect() error {
if c.run == nil || c.ip == "127.0.0.1" {
return nil
}
for _, chain := range []string{"OUTPUT", "PREROUTING"} {
if _, err := c.run.Run("iptables", c.natArgs("-C", chain)...); err == nil {
continue // rule already present
}
if _, err := c.run.Run("iptables", c.natArgs("-A", chain)...); err != nil {
return fmt.Errorf("add nat redirect %s: %w", chain, err)
}
}
return nil
}
func (c *credEndpoint) delRedirect() {
if c.run == nil || c.ip == "127.0.0.1" {
return
}
for _, chain := range []string{"OUTPUT", "PREROUTING"} {
_, _ = c.run.Run("iptables", c.natArgs("-D", chain)...)
}
}
// natArgs builds the iptables nat rule for op (-A/-C/-D) on chain: DNAT the
// advertised endpoint :80 to the proxy port on the same IP.
func (c *credEndpoint) natArgs(op, chain string) []string {
return []string{
"-t", "nat", op, chain,
"-d", c.ip, "-p", "tcp", "--dport", strconv.Itoa(c.port),
"-j", "DNAT", "--to-destination", net.JoinHostPort(c.ip, strconv.Itoa(c.proxyPort)),
}
}
// addEndpointAddr brings up the dummy interface holding the endpoint IP. Skipped
// for loopback (tests bind 127.0.0.1 directly, no interface plumbing needed).
func (c *credEndpoint) addEndpointAddr() error {
if c.run == nil || c.ip == "127.0.0.1" {
return nil
}
_, _ = c.run.Run("ip", "link", "add", credDummyIface, "type", "dummy")
if _, err := c.run.Run("ip", "addr", "add", c.ip+"/32", "dev", credDummyIface); err != nil &&
!strings.Contains(err.Error(), "exists") {
return fmt.Errorf("add endpoint addr %s: %w", c.ip, err)
}
if _, err := c.run.Run("ip", "link", "set", credDummyIface, "up"); err != nil {
return fmt.Errorf("bring up %s: %w", credDummyIface, err)
}
return nil
}
func (c *credEndpoint) delEndpointAddr() {
if c.run == nil || c.ip == "127.0.0.1" {
return
}
_, _ = c.run.Run("ip", "link", "del", credDummyIface)
}
// credValid reports whether creds are usable now with margin to spare.
func credValid(c stsauth.Credentials, margin time.Duration) bool {
if c.AccessKeyID == "" || c.SecretAccessKey == "" {
return false
}
if c.Expiration.IsZero() {
return true
}
return time.Now().Add(margin).Before(c.Expiration)
}
// sessionName derives an AssumeRole RoleSessionName from credID, satisfying the
// STS [\w+=,.@-]{2,64} constraint.
func sessionName(credID string) string {
name := "ecs-" + credID
if len(name) > 64 {
name = name[:64]
}
return name
}
+197
View File
@@ -0,0 +1,197 @@
package main
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"github.com/mulgadc/spinifex/internal/stsauth"
)
// newTestCredEndpoint builds an endpoint whose assume seam returns canned creds,
// counting calls so cache behaviour is observable. run is nil (no iface plumbing).
func newTestCredEndpoint(creds stsauth.Credentials) (*credEndpoint, *int32) {
c := newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
var calls int32
c.assume = func(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {
atomic.AddInt32(&calls, 1)
return creds, nil
}
return c, &calls
}
func TestCredEndpoint_ServesRegisteredCredID(t *testing.T) {
exp := time.Now().Add(time.Hour).UTC().Truncate(time.Second)
c, _ := newTestCredEndpoint(stsauth.Credentials{
AccessKeyID: "AKIA", SecretAccessKey: "secret", SessionToken: "tok", Expiration: exp,
})
c.Register("cred-1", "arn:aws:iam::111122223333:role/task")
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rr.Code, rr.Body.String())
}
var got credResponse
if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.AccessKeyID != "AKIA" || got.SecretAccessKey != "secret" || got.Token != "tok" {
t.Errorf("bad creds in response: %+v", got)
}
if got.RoleArn != "arn:aws:iam::111122223333:role/task" {
t.Errorf("RoleArn = %q", got.RoleArn)
}
if got.Expiration != exp.Format(time.RFC3339) {
t.Errorf("Expiration = %q, want %q", got.Expiration, exp.Format(time.RFC3339))
}
}
func TestCredEndpoint_UnknownCredIDIs404(t *testing.T) {
c, calls := newTestCredEndpoint(stsauth.Credentials{AccessKeyID: "x", SecretAccessKey: "y"})
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/nope", nil))
if rr.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404", rr.Code)
}
if *calls != 0 {
t.Errorf("assume called %d times for unknown credID, want 0", *calls)
}
}
func TestCredEndpoint_WrongMethodIs405(t *testing.T) {
c, _ := newTestCredEndpoint(stsauth.Credentials{AccessKeyID: "x", SecretAccessKey: "y"})
c.Register("cred-1", "arn:role")
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodPost, "/v2/credentials/cred-1", nil))
if rr.Code != http.StatusMethodNotAllowed {
t.Fatalf("status = %d, want 405", rr.Code)
}
}
func TestCredEndpoint_CachesUntilExpiry(t *testing.T) {
c, calls := newTestCredEndpoint(stsauth.Credentials{
AccessKeyID: "AKIA", SecretAccessKey: "secret", Expiration: time.Now().Add(time.Hour),
})
c.Register("cred-1", "arn:role")
for hit := range 3 {
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
if rr.Code != http.StatusOK {
t.Fatalf("hit %d: status %d", hit, rr.Code)
}
}
if *calls != 1 {
t.Errorf("assume called %d times, want 1 (cached)", *calls)
}
}
func TestCredEndpoint_RefreshesNearExpiry(t *testing.T) {
// Expiry inside the refresh margin -> every hit re-assumes.
c, calls := newTestCredEndpoint(stsauth.Credentials{
AccessKeyID: "AKIA", SecretAccessKey: "secret", Expiration: time.Now().Add(time.Minute),
})
c.Register("cred-1", "arn:role")
for range 2 {
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
}
if *calls != 2 {
t.Errorf("assume called %d times, want 2 (expiry within margin)", *calls)
}
}
func TestCredEndpoint_DeregisterDropsCredID(t *testing.T) {
c, _ := newTestCredEndpoint(stsauth.Credentials{AccessKeyID: "x", SecretAccessKey: "y"})
c.Register("cred-1", "arn:role")
c.Deregister("cred-1")
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
if rr.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404 after deregister", rr.Code)
}
}
func TestCredEndpoint_AssumeFailureIs502(t *testing.T) {
c, _ := newTestCredEndpoint(stsauth.Credentials{})
c.assume = func(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {
return stsauth.Credentials{}, context.DeadlineExceeded
}
c.Register("cred-1", "arn:role")
rr := httptest.NewRecorder()
c.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/v2/credentials/cred-1", nil))
if rr.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502", rr.Code)
}
}
func TestCredEndpoint_StartServesOverLoopback(t *testing.T) {
c, _ := newTestCredEndpoint(stsauth.Credentials{
AccessKeyID: "AKIA", SecretAccessKey: "secret", Expiration: time.Now().Add(time.Hour),
})
c.Register("cred-1", "arn:role")
c.port = 0 // bind an ephemeral port instead of privileged 80
if err := c.Start(); err != nil {
t.Fatalf("start: %v", err)
}
defer c.Stop()
url := "http://" + c.ln.Addr().String() + credRelativeURI("cred-1")
resp, err := http.Get(url)
if err != nil {
t.Fatalf("get: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want 200", resp.StatusCode)
}
}
func TestCredEndpoint_RedirectInstallsAndRemovesDNAT(t *testing.T) {
// failOn the -C existence check so addRedirect falls through to -A.
f := &fakeNetRunner{failOn: "-t nat -C"}
c := newCredEndpoint(nil, "us-east-1", "https://gw", "", "169.254.170.2", 80, f)
if c.listenPort() != defaultCredProxyPort {
t.Fatalf("listenPort = %d, want %d (proxy)", c.listenPort(), defaultCredProxyPort)
}
if err := c.addRedirect(); err != nil {
t.Fatalf("addRedirect: %v", err)
}
target := "-d 169.254.170.2 -p tcp --dport 80 -j DNAT --to-destination 169.254.170.2:51679"
for _, chain := range []string{"-A OUTPUT", "-A PREROUTING"} {
if !f.sawAny("iptables -t nat " + chain + " " + target) {
t.Errorf("missing add rule for %s; calls=%v", chain, f.joined())
}
}
c.delRedirect()
for _, chain := range []string{"-D OUTPUT", "-D PREROUTING"} {
if !f.sawAny("iptables -t nat " + chain + " " + target) {
t.Errorf("missing delete rule for %s; calls=%v", chain, f.joined())
}
}
}
func TestCredEndpoint_RedirectSkippedOnLoopback(t *testing.T) {
f := &fakeNetRunner{}
c := newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, f)
if err := c.addRedirect(); err != nil {
t.Fatalf("addRedirect: %v", err)
}
c.delRedirect()
if f.sawAny("iptables") {
t.Errorf("loopback mode must not touch iptables; calls=%v", f.joined())
}
if c.listenPort() != c.port {
t.Errorf("loopback listenPort = %d, want caller port %d", c.listenPort(), c.port)
}
}
func TestCredRelativeURI(t *testing.T) {
if got := credRelativeURI("abc"); got != "/v2/credentials/abc" {
t.Errorf("credRelativeURI = %q", got)
}
}
+59
View File
@@ -0,0 +1,59 @@
package credentials
import (
"context"
"net/http"
"sync"
"time"
"github.com/mulgadc/spinifex/internal/imdscreds"
)
// refreshMargin is how long before expiry the cache is considered stale.
const refreshMargin = 5 * time.Minute
// IMDSProvider fetches instance-role credentials from IMDS and caches them until
// they near expiry. Safe for concurrent use.
type IMDSProvider struct {
client *http.Client
base string
mu sync.Mutex
cached Credentials
}
var _ CredentialsProvider = (*IMDSProvider)(nil)
// NewIMDSProvider builds a provider against the given IMDS base URL (e.g.
// "http://169.254.169.254/latest"). A nil client uses http.DefaultClient.
func NewIMDSProvider(client *http.Client, base string) *IMDSProvider {
if client == nil {
client = http.DefaultClient
}
return &IMDSProvider{client: client, base: base}
}
// Retrieve returns cached credentials when still valid, else fetches fresh ones.
func (p *IMDSProvider) Retrieve(ctx context.Context) (Credentials, error) {
p.mu.Lock()
defer p.mu.Unlock()
if p.cached.Valid(refreshMargin) {
return p.cached, nil
}
if err := ctx.Err(); err != nil {
return Credentials{}, err
}
raw, err := imdscreds.Fetch(p.client, p.base)
if err != nil {
return Credentials{}, err
}
p.cached = Credentials{
AccessKeyID: raw.AccessKeyID,
SecretAccessKey: raw.SecretAccessKey,
SessionToken: raw.SessionToken,
Expiration: raw.Expiration,
}
return p.cached, nil
}
+104
View File
@@ -0,0 +1,104 @@
package credentials
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
)
func imdsStub(t *testing.T, hits *int32, creds map[string]string) *httptest.Server {
t.Helper()
mux := http.NewServeMux()
mux.HandleFunc("/api/token", func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("v2-token"))
})
mux.HandleFunc("/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/meta-data/iam/security-credentials/" {
_, _ = w.Write([]byte("node-role"))
return
}
if hits != nil {
atomic.AddInt32(hits, 1)
}
_ = json.NewEncoder(w).Encode(creds)
})
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv
}
func TestRetrieve_FetchesAndCaches(t *testing.T) {
var hits int32
exp := time.Now().Add(6 * time.Hour).UTC().Round(time.Second)
srv := imdsStub(t, &hits, map[string]string{
"AccessKeyId": "AKIA",
"SecretAccessKey": "secret",
"Token": "session",
"Expiration": exp.Format(time.RFC3339),
})
p := NewIMDSProvider(srv.Client(), srv.URL)
got, err := p.Retrieve(context.Background())
if err != nil {
t.Fatalf("Retrieve: %v", err)
}
if got.AccessKeyID != "AKIA" || !got.Expiration.Equal(exp) {
t.Errorf("creds mismatch: %+v", got)
}
// Second call within validity window must not re-hit IMDS.
if _, err := p.Retrieve(context.Background()); err != nil {
t.Fatalf("Retrieve 2: %v", err)
}
if hits != 1 {
t.Errorf("IMDS credential hits = %d, want 1 (cached)", hits)
}
}
func TestRetrieve_RefetchesWhenStale(t *testing.T) {
var hits int32
// Expiry inside the refresh margin → never cached, always refetched.
srv := imdsStub(t, &hits, map[string]string{
"AccessKeyId": "AKIA",
"SecretAccessKey": "secret",
"Expiration": time.Now().Add(time.Minute).UTC().Format(time.RFC3339),
})
p := NewIMDSProvider(srv.Client(), srv.URL)
if _, err := p.Retrieve(context.Background()); err != nil {
t.Fatal(err)
}
if _, err := p.Retrieve(context.Background()); err != nil {
t.Fatal(err)
}
if hits != 2 {
t.Errorf("hits = %d, want 2 (stale refetch)", hits)
}
}
func TestRetrieve_CancelledContext(t *testing.T) {
srv := imdsStub(t, nil, map[string]string{"AccessKeyId": "A", "SecretAccessKey": "B"})
p := NewIMDSProvider(srv.Client(), srv.URL)
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := p.Retrieve(ctx); err == nil {
t.Fatal("expected error from cancelled context")
}
}
func TestCredentials_Valid(t *testing.T) {
if (Credentials{}).Valid(0) {
t.Error("empty creds should be invalid")
}
noExp := Credentials{AccessKeyID: "a", SecretAccessKey: "b"}
if !noExp.Valid(time.Hour) {
t.Error("zero expiry should be treated valid")
}
soon := Credentials{AccessKeyID: "a", SecretAccessKey: "b", Expiration: time.Now().Add(time.Minute)}
if soon.Valid(5 * time.Minute) {
t.Error("expiry inside margin should be invalid")
}
}
+36
View File
@@ -0,0 +1,36 @@
// Package credentials supplies AWS SigV4 credentials to the ecs-agent. The agent
// signs gateway calls (ECR GetAuthorizationToken, instance register/heartbeat)
// with the host's instance-role credentials, fetched from IMDS and cached until
// near expiry.
package credentials
import (
"context"
"time"
)
// Credentials is a SigV4 credential set with an expiry.
type Credentials struct {
AccessKeyID string
SecretAccessKey string
SessionToken string
Expiration time.Time
}
// Valid reports whether the credentials are usable now, leaving a margin so a
// caller does not start a request with a token about to expire mid-flight.
func (c Credentials) Valid(margin time.Duration) bool {
if c.AccessKeyID == "" || c.SecretAccessKey == "" {
return false
}
if c.Expiration.IsZero() {
return true
}
return time.Now().Add(margin).Before(c.Expiration)
}
// CredentialsProvider yields current credentials, refreshing as needed. Retrieve
// must be safe for concurrent use.
type CredentialsProvider interface {
Retrieve(ctx context.Context) (Credentials, error)
}
+55
View File
@@ -0,0 +1,55 @@
package main
import (
"testing"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
func TestTaskCredID(t *testing.T) {
cases := []struct {
name string
as bus.Assign
want string
}{
{"no role", bus.Assign{TaskID: "t1"}, ""},
{"role, no credID -> taskID", bus.Assign{TaskID: "t1", TaskRoleARN: "arn:role"}, "t1"},
{"role + credID", bus.Assign{TaskID: "t1", CredID: "c9", TaskRoleARN: "arn:role"}, "c9"},
}
for _, tc := range cases {
if got := taskCredID(&tc.as); got != tc.want {
t.Errorf("%s: taskCredID = %q, want %q", tc.name, got, tc.want)
}
}
}
func TestWithCredEnv(t *testing.T) {
base := map[string]string{"FOO": "bar"}
// No credID -> env returned unchanged (same map).
if got := withCredEnv(base, ""); got["FOO"] != "bar" || got["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"] != "" {
t.Errorf("blank credID mutated env: %v", got)
}
got := withCredEnv(base, "cred-1")
if got["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"] != "/v2/credentials/cred-1" {
t.Errorf("missing relative-uri: %v", got)
}
if got["FOO"] != "bar" {
t.Errorf("dropped existing env: %v", got)
}
// Original map must be untouched (copy semantics).
if _, ok := base["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]; ok {
t.Errorf("source env mutated: %v", base)
}
}
func TestSessionName(t *testing.T) {
if got := sessionName("abc"); got != "ecs-abc" {
t.Errorf("sessionName = %q", got)
}
long := sessionName(string(make([]byte, 100)))
if len(long) != 64 {
t.Errorf("sessionName not truncated to 64: len=%d", len(long))
}
}
+95
View File
@@ -0,0 +1,95 @@
package main
import (
"context"
"net/http"
"strings"
"sync"
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
"github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/internal/ecrauth"
)
// ecrResolver implements runtime.Resolver by minting an ECR authorization token
// per pull: it fetches the host's IMDS credentials, SigV4-signs
// GetAuthorizationToken against the gateway, and returns the AWS:<jwt> as a
// user/password pair for containerd.
//
// The gateway HTTP client is built lazily on first Authorize, not at agent boot:
// a missing or malformed gateway CA only matters when an image is actually
// pulled, so the agent can still register + heartbeat on a host with absent or
// wrong ECR config.
type ecrResolver struct {
creds credentials.CredentialsProvider
region string
gatewayURL string
caPath string
mu sync.Mutex
httpClient *http.Client
}
var _ runtime.Resolver = (*ecrResolver)(nil)
// newECRResolver builds a resolver with an injected HTTP client (tests). The
// client is used as-is and never rebuilt from caPath.
func newECRResolver(creds credentials.CredentialsProvider, region, gatewayURL string, httpClient *http.Client) *ecrResolver {
return &ecrResolver{creds: creds, region: region, gatewayURL: gatewayURL, httpClient: httpClient}
}
// newLazyECRResolver builds a resolver that constructs its gateway HTTP client
// from caPath on first use (production path).
func newLazyECRResolver(creds credentials.CredentialsProvider, region, gatewayURL, caPath string) *ecrResolver {
return &ecrResolver{creds: creds, region: region, gatewayURL: gatewayURL, caPath: caPath}
}
// client returns the gateway HTTP client, building it from caPath on first call
// and caching the result. An injected client (newECRResolver) is returned as-is.
func (e *ecrResolver) client() (*http.Client, error) {
e.mu.Lock()
defer e.mu.Unlock()
if e.httpClient != nil {
return e.httpClient, nil
}
c, err := ecrauth.GatewayHTTPClient(e.caPath)
if err != nil {
return nil, err
}
e.httpClient = c
return c, nil
}
// isECRRef reports whether ref targets an ECR registry, i.e. its registry host
// is "<acct>.dkr.ecr.<region>.<domain>". Only ECR pulls need a minted token;
// public registries (docker.io and friends) pull anonymously, so refs without
// an ECR host must not drag in IMDS host credentials.
func isECRRef(ref string) bool {
host, _, _ := strings.Cut(ref, "/")
if !strings.Contains(host, ".") && !strings.Contains(host, ":") {
return false // no registry host -> docker.io default
}
return strings.Contains(host, ".dkr.ecr.")
}
// Authorize returns the ECR token's user/password and proxy endpoint for ref.
// Non-ECR refs resolve to anonymous pull (empty credentials).
func (e *ecrResolver) Authorize(ctx context.Context, ref string) (user, pass, endpoint string, err error) {
if !isECRRef(ref) {
return "", "", "", nil
}
c, err := e.creds.Retrieve(ctx)
if err != nil {
return "", "", "", err
}
httpClient, err := e.client()
if err != nil {
return "", "", "", err
}
tok, err := ecrauth.GetAuthorizationToken(e.region, e.gatewayURL, httpClient,
c.AccessKeyID, c.SecretAccessKey, c.SessionToken)
if err != nil {
return "", "", "", err
}
return tok.Username, tok.Password, tok.ProxyHost, nil
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
)
// stubCreds is a credentials.CredentialsProvider returning fixed creds or error.
type stubCreds struct {
c credentials.Credentials
err error
}
func (s stubCreds) Retrieve(context.Context) (credentials.Credentials, error) {
return s.c, s.err
}
func gatewayStub(t *testing.T, token, proxy string, status int) *httptest.Server {
t.Helper()
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
w.WriteHeader(http.StatusBadRequest)
return
}
if status != http.StatusOK {
w.WriteHeader(status)
return
}
w.Header().Set("Content-Type", "application/x-amz-json-1.1")
_ = json.NewEncoder(w).Encode(map[string]any{
"authorizationData": []map[string]any{{
"authorizationToken": base64.StdEncoding.EncodeToString([]byte(token)),
"proxyEndpoint": proxy,
}},
})
}))
t.Cleanup(srv.Close)
return srv
}
func trusting(srv *httptest.Server) *http.Client {
pool := x509.NewCertPool()
pool.AddCert(srv.Certificate())
return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
}
func TestECRResolver_Authorize(t *testing.T) {
srv := gatewayStub(t, "AWS:jwt", "https://123456789012.dkr.ecr.us-east-1.spinifex.internal", http.StatusOK)
creds := stubCreds{c: credentials.Credentials{
AccessKeyID: "AKIA", SecretAccessKey: "secret", SessionToken: "sess",
Expiration: time.Now().Add(time.Hour),
}}
r := newECRResolver(creds, "us-east-1", srv.URL, trusting(srv))
user, pass, endpoint, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1")
if err != nil {
t.Fatalf("Authorize: %v", err)
}
if user != "AWS" || pass != "jwt" {
t.Errorf("got %q:%q, want AWS:jwt", user, pass)
}
if endpoint != "123456789012.dkr.ecr.us-east-1.spinifex.internal" {
t.Errorf("endpoint = %q", endpoint)
}
}
func TestECRResolver_CredsError(t *testing.T) {
r := newECRResolver(stubCreds{err: errors.New("no imds")}, "us-east-1", "https://x", http.DefaultClient)
if _, _, _, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1"); err == nil {
t.Fatal("expected creds error to propagate")
}
}
func TestECRResolver_GatewayError(t *testing.T) {
srv := gatewayStub(t, "", "", http.StatusInternalServerError)
creds := stubCreds{c: credentials.Credentials{AccessKeyID: "A", SecretAccessKey: "B"}}
r := newECRResolver(creds, "us-east-1", srv.URL, trusting(srv))
if _, _, _, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1"); err == nil {
t.Fatal("expected gateway error")
}
}
func TestECRResolver_LazyClient_BadCADeferredToAuthorize(t *testing.T) {
creds := stubCreds{c: credentials.Credentials{AccessKeyID: "A", SecretAccessKey: "B"}}
// Construction with a bogus CA path must not fail — only Authorize does.
r := newLazyECRResolver(creds, "us-east-1", "https://gw", filepath.Join(t.TempDir(), "absent.pem"))
if _, _, _, err := r.Authorize(context.Background(), "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1"); err == nil {
t.Fatal("expected lazy gateway-client build to fail on missing CA")
}
}
// TestECRResolver_NonECRAnonymous asserts that a public-registry ref pulls
// anonymously: Authorize returns empty creds without ever touching IMDS, even
// when the credentials provider would error.
func TestECRResolver_NonECRAnonymous(t *testing.T) {
r := newECRResolver(stubCreds{err: errors.New("imds must not be called")}, "us-east-1", "https://x", http.DefaultClient)
for _, ref := range []string{"nginx:alpine", "docker.io/library/nginx:alpine", "ghcr.io/org/app:1"} {
user, pass, endpoint, err := r.Authorize(context.Background(), ref)
if err != nil || user != "" || pass != "" || endpoint != "" {
t.Errorf("ref %q: got (%q,%q,%q,%v), want anonymous", ref, user, pass, endpoint, err)
}
}
}
func TestIsECRRef(t *testing.T) {
cases := map[string]bool{
"nginx:alpine": false,
"docker.io/library/nginx": false,
"ghcr.io/org/app:1": false,
"registry:5000/app:1": false,
"123456789012.dkr.ecr.us-east-1.spinifex.internal/app:1": true,
"123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/x": true,
}
for ref, want := range cases {
if got := isECRRef(ref); got != want {
t.Errorf("isECRRef(%q) = %v, want %v", ref, got, want)
}
}
}
+46
View File
@@ -0,0 +1,46 @@
package main
import (
"context"
"log/slog"
"time"
)
// heartbeater keeps the container instance alive by periodically re-registering
// it through the gateway. Real ECS folds liveness into ACS; v1 folds it into an
// idempotent RegisterContainerInstance, which refreshes the instance's LastSeen
// so the scheduler can tell live instances from dead ones (no bus.Heartbeat).
type heartbeater struct {
cp controlPlane
id identity
interval time.Duration
}
func newHeartbeater(cp controlPlane, id identity, interval time.Duration) *heartbeater {
if interval <= 0 {
interval = defaultHeartbeat
}
return &heartbeater{cp: cp, id: id, interval: interval}
}
// beat re-registers the instance once, refreshing its LastSeen.
func (h *heartbeater) beat() error {
return h.cp.Register(h.id)
}
// Run beats every interval until ctx is cancelled. A failed beat is logged and
// retried on the next tick rather than killing the loop.
func (h *heartbeater) Run(ctx context.Context) {
ticker := time.NewTicker(h.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := h.beat(); err != nil {
slog.Warn("ecs-agent: heartbeat (re-register) failed", "err", err)
}
}
}
}
+39
View File
@@ -0,0 +1,39 @@
// ecs-agent runs inside a Spinifex ECS container instance (a guest VM booted from
// the ECS-AMI, with containerd baked in). It registers the host with the ECS
// scheduler through the AWS gateway over TLS+SigV4 (never NATS), heartbeats by
// re-registering, polls the gateway for task assignments, runs them through
// containerd, and reports state back over the gateway.
//
// Static config (gateway URL, CA, region, cluster, seeded IAM creds, containerd
// socket) is read from the cloud-init env file /etc/spinifex-ecs/agent.env
// (KEY=value); real env vars override it. Host identity (account, instance, AZ)
// comes from IMDS at boot.
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"syscall"
_ "github.com/mulgadc/spinifex/internal/fipsboot"
)
func main() {
cfg := loadConfig(defaultEnvFile)
agent, err := New(cfg)
if err != nil {
slog.Error("ecs-agent: startup failed", "err", err)
os.Exit(1)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
if err := agent.Run(ctx); err != nil {
slog.Error("ecs-agent: run failed", "err", err)
os.Exit(1)
}
}
+97
View File
@@ -0,0 +1,97 @@
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
)
// instanceMetadata is the IMDS-derived identity of the host the agent runs on.
type instanceMetadata struct {
AccountID string
InstanceID string
AZ string
}
// fetchInstanceMetadata reads the instance-id, AZ and account-id from IMDSv2.
// base is the IMDS root, e.g. "http://169.254.169.254/latest".
func fetchInstanceMetadata(client *http.Client, base string) (instanceMetadata, error) {
token, err := imdsToken(client, base)
if err != nil {
return instanceMetadata{}, err
}
instanceID, err := imdsGet(client, base+"/meta-data/instance-id", token)
if err != nil {
return instanceMetadata{}, fmt.Errorf("instance-id: %w", err)
}
az, err := imdsGet(client, base+"/meta-data/placement/availability-zone", token)
if err != nil {
return instanceMetadata{}, fmt.Errorf("availability-zone: %w", err)
}
doc, err := imdsGet(client, base+"/dynamic/instance-identity/document", token)
if err != nil {
return instanceMetadata{}, fmt.Errorf("identity document: %w", err)
}
var ident struct {
AccountID string `json:"accountId"`
}
if err := json.Unmarshal([]byte(doc), &ident); err != nil {
return instanceMetadata{}, fmt.Errorf("parse identity document: %w", err)
}
return instanceMetadata{
AccountID: ident.AccountID,
InstanceID: strings.TrimSpace(instanceID),
AZ: strings.TrimSpace(az),
}, nil
}
// imdsToken fetches an IMDSv2 session token.
func imdsToken(client *http.Client, base string) (string, error) {
req, err := http.NewRequest(http.MethodPut, base+"/api/token", nil)
if err != nil {
return "", err
}
req.Header.Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("imds token: %w", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("imds token: status %d", resp.StatusCode)
}
return string(body), nil
}
// imdsGet performs a token-authenticated IMDS GET.
func imdsGet(client *http.Client, url, token string) (string, error) {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return "", err
}
if token != "" {
req.Header.Set("X-Aws-Ec2-Metadata-Token", token)
}
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("status %d", resp.StatusCode)
}
return string(body), nil
}
+76
View File
@@ -0,0 +1,76 @@
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func metadataStub(t *testing.T, enforceV2 bool) *httptest.Server {
t.Helper()
mux := http.NewServeMux()
mux.HandleFunc("/api/token", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPut {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
_, _ = w.Write([]byte("v2-token"))
})
guard := func(w http.ResponseWriter, r *http.Request) bool {
if enforceV2 && r.Header.Get("X-Aws-Ec2-Metadata-Token") != "v2-token" {
w.WriteHeader(http.StatusUnauthorized)
return false
}
return true
}
mux.HandleFunc("/meta-data/instance-id", func(w http.ResponseWriter, r *http.Request) {
if guard(w, r) {
_, _ = w.Write([]byte("i-0abc123"))
}
})
mux.HandleFunc("/meta-data/placement/availability-zone", func(w http.ResponseWriter, r *http.Request) {
if guard(w, r) {
_, _ = w.Write([]byte("us-east-1a"))
}
})
mux.HandleFunc("/dynamic/instance-identity/document", func(w http.ResponseWriter, r *http.Request) {
if guard(w, r) {
_, _ = w.Write([]byte(`{"accountId":"123456789012","region":"us-east-1"}`))
}
})
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv
}
func TestFetchInstanceMetadata_OK(t *testing.T) {
srv := metadataStub(t, true)
got, err := fetchInstanceMetadata(srv.Client(), srv.URL)
if err != nil {
t.Fatalf("fetch: %v", err)
}
if got.InstanceID != "i-0abc123" || got.AZ != "us-east-1a" || got.AccountID != "123456789012" {
t.Errorf("metadata mismatch: %+v", got)
}
}
func TestFetchInstanceMetadata_TokenRequired(t *testing.T) {
// Server returns 405 for non-PUT token; a GET-only client would 401 downstream.
srv := metadataStub(t, true)
if _, err := fetchInstanceMetadata(srv.Client(), srv.URL); err != nil {
t.Fatalf("v2 path should succeed: %v", err)
}
}
func TestFetchInstanceMetadata_BadDocument(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/api/token", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("t")) })
mux.HandleFunc("/meta-data/instance-id", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("i-1")) })
mux.HandleFunc("/meta-data/placement/availability-zone", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("z")) })
mux.HandleFunc("/dynamic/instance-identity/document", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("not json")) })
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
if _, err := fetchInstanceMetadata(srv.Client(), srv.URL); err == nil {
t.Fatal("expected parse error on bad identity document")
}
}
+167
View File
@@ -0,0 +1,167 @@
package main
import (
"fmt"
"hash/crc32"
"os/exec"
"strings"
"time"
)
// netnsRunDir is the iproute2 netns mount directory; a name here is the netns
// path containerd joins via the OCI network namespace path.
const netnsRunDir = "/var/run/netns"
// Credentials path constants: each task netns gets a /30 veth into the host netns
// from 169.254.172.0/22 and a host-routed /32 to the credential endpoint, keeping
// the endpoint off the task ENI's VPC route. credTaskVeth is the task-side name
// (unique per netns); the host side is per-task (credVethName).
const (
credTaskVeth = "ecsc0"
credEndpointCIDR = defaultCredEndpointIP + "/32"
)
// netCmdRunner executes a host networking command. Abstracted so unit tests can
// assert the ip/netns sequence without touching the kernel.
type netCmdRunner interface {
Run(name string, args ...string) (string, error)
}
// execNetRunner runs real commands.
type execNetRunner struct{}
func (execNetRunner) Run(name string, args ...string) (string, error) {
out, err := exec.Command(name, args...).CombinedOutput()
return string(out), err
}
// taskNetns builds and tears down per-task awsvpc network namespaces. The ENI is
// already hot-plugged into the VM (scheduler did that); the agent finds the NIC
// by MAC, moves it into a task netns, and DHCPs the OVN-assigned ENI IP — the
// same way the primary NIC obtains its address.
type taskNetns struct {
run netCmdRunner
nicWait time.Duration
poll time.Duration
}
func newTaskNetns(run netCmdRunner) *taskNetns {
return &taskNetns{run: run, nicWait: 30 * time.Second, poll: time.Second}
}
func netnsName(taskID string) string { return "ecs-" + taskID }
func netnsPathFor(taskID string) string {
return netnsRunDir + "/" + netnsName(taskID)
}
// Setup creates the task netns, moves the ENI NIC (matched by MAC) into it, and
// brings it up with a DHCP lease. It returns the netns path for containerd. On
// any failure it tears the partial netns down so a retry starts clean.
func (n *taskNetns) Setup(taskID, mac string) (string, error) {
iface, err := n.waitForNIC(mac)
if err != nil {
return "", err
}
name := netnsName(taskID)
if _, err := n.run.Run("ip", "netns", "add", name); err != nil {
return "", fmt.Errorf("netns add %s: %w", name, err)
}
hostVeth := credVethName(taskID)
hostIP, taskIP := credSubnet(taskID)
steps := [][]string{
{"ip", "link", "set", iface, "netns", name},
{"ip", "-n", name, "link", "set", "lo", "up"},
{"ip", "-n", name, "link", "set", iface, "up"},
{"ip", "netns", "exec", name, "udhcpc", "-i", iface, "-q", "-n"},
// Credentials path: a veth into the host netns plus a host-routed /32 to the
// credential endpoint, so the container reaches 169.254.170.2 without it
// leaking onto the task ENI's VPC route.
{"ip", "link", "add", hostVeth, "type", "veth", "peer", "name", credTaskVeth, "netns", name},
{"ip", "addr", "add", hostIP + "/30", "dev", hostVeth},
{"ip", "link", "set", hostVeth, "up"},
{"ip", "-n", name, "addr", "add", taskIP + "/30", "dev", credTaskVeth},
{"ip", "-n", name, "link", "set", credTaskVeth, "up"},
{"ip", "-n", name, "route", "add", credEndpointCIDR, "via", hostIP},
}
for _, s := range steps {
if _, err := n.run.Run(s[0], s[1:]...); err != nil {
_ = n.Teardown(taskID)
return "", fmt.Errorf("netns setup %v: %w", s, err)
}
}
return netnsPathFor(taskID), nil
}
// credVethName is the host-side veth name for a task's credential path. Derived
// from the taskID and kept within the 15-char interface-name limit.
func credVethName(taskID string) string {
return fmt.Sprintf("ecsv%08x", crc32.ChecksumIEEE([]byte(taskID)))
}
// credSubnet derives a unique /30 in 169.254.172.0/22 for a task's credential
// veth, returning the host-side and task-side addresses. The host side is the
// container's gateway to the credential endpoint.
func credSubnet(taskID string) (hostIP, taskIP string) {
slot := crc32.ChecksumIEEE([]byte(taskID)) % 256 // 256 /30s in the /22
third := 172 + (slot*4)/256
base := (slot * 4) % 256
hostIP = fmt.Sprintf("169.254.%d.%d", third, base+1)
taskIP = fmt.Sprintf("169.254.%d.%d", third, base+2)
return hostIP, taskIP
}
// Teardown deletes the task netns (releasing the moved NIC). Idempotent enough
// for the stop path: a missing netns is reported but not fatal to the caller.
func (n *taskNetns) Teardown(taskID string) error {
name := netnsName(taskID)
// Deleting the netns destroys the task-side veth and its host peer; remove the
// host side explicitly too so a leaked half from a partial setup is reaped.
_, _ = n.run.Run("ip", "link", "del", credVethName(taskID))
if _, err := n.run.Run("ip", "netns", "del", name); err != nil {
return fmt.Errorf("netns del %s: %w", name, err)
}
return nil
}
// waitForNIC polls until a NIC with the given MAC appears in the host netns,
// returning its interface name. The hot-plug pipeline is async, so the device
// may lag the assign by a few seconds.
func (n *taskNetns) waitForNIC(mac string) (string, error) {
want := strings.ToLower(strings.TrimSpace(mac))
deadline := time.Now().Add(n.nicWait)
for {
iface, err := n.findNICByMAC(want)
if err == nil {
return iface, nil
}
if time.Now().After(deadline) {
return "", fmt.Errorf("nic with mac %s not found within %s", want, n.nicWait)
}
time.Sleep(n.poll)
}
}
// findNICByMAC parses `ip -o link` and returns the interface whose link/ether
// matches mac.
func (n *taskNetns) findNICByMAC(mac string) (string, error) {
out, err := n.run.Run("ip", "-o", "link")
if err != nil {
return "", fmt.Errorf("ip -o link: %w", err)
}
for line := range strings.SplitSeq(out, "\n") {
if !strings.Contains(strings.ToLower(line), mac) {
continue
}
// "<idx>: <ifname>: <flags> ... link/ether <mac> ..."
fields := strings.Fields(line)
if len(fields) < 2 {
continue
}
name := strings.TrimSuffix(strings.TrimSuffix(fields[1], ":"), "@")
if i := strings.IndexByte(name, '@'); i >= 0 {
name = name[:i]
}
return name, nil
}
return "", fmt.Errorf("no interface with mac %s", mac)
}
+154
View File
@@ -0,0 +1,154 @@
package main
import (
"errors"
"strings"
"sync"
"testing"
"time"
)
// fakeNetRunner records every command and serves canned `ip -o link` output. A
// failOn substring forces the matching command to error so teardown paths run.
type fakeNetRunner struct {
mu sync.Mutex
calls [][]string
linkOut string
failOn string
}
func (f *fakeNetRunner) Run(name string, args ...string) (string, error) {
f.mu.Lock()
defer f.mu.Unlock()
cmd := append([]string{name}, args...)
f.calls = append(f.calls, cmd)
joined := strings.Join(cmd, " ")
if f.failOn != "" && strings.Contains(joined, f.failOn) {
return "", errors.New("boom: " + joined)
}
if name == "ip" && len(args) >= 2 && args[0] == "-o" && args[1] == "link" {
return f.linkOut, nil
}
return "", nil
}
func (f *fakeNetRunner) joined() []string {
f.mu.Lock()
defer f.mu.Unlock()
out := make([]string, len(f.calls))
for i, c := range f.calls {
out[i] = strings.Join(c, " ")
}
return out
}
func (f *fakeNetRunner) sawAny(sub string) bool {
for _, j := range f.joined() {
if strings.Contains(j, sub) {
return true
}
}
return false
}
const linkWithENI = `1: lo: <LOOPBACK,UP> mtu 65536 ... link/loopback 00:00:00:00:00:00 ...
2: eth0: <BROADCAST,UP> mtu 1500 ... link/ether 52:54:00:aa:bb:cc ...
3: eth1: <BROADCAST,UP> mtu 1500 ... link/ether 52:54:00:de:ad:01 ...`
func newTestNetns(f *fakeNetRunner) *taskNetns {
n := newTaskNetns(f)
n.nicWait = 50 * time.Millisecond
n.poll = 5 * time.Millisecond
return n
}
func TestNetns_FindNICByMAC(t *testing.T) {
n := newTestNetns(&fakeNetRunner{linkOut: linkWithENI})
iface, err := n.findNICByMAC("52:54:00:de:ad:01")
if err != nil || iface != "eth1" {
t.Fatalf("want eth1, got %q err=%v", iface, err)
}
if _, err := n.findNICByMAC("52:54:00:ff:ff:ff"); err == nil {
t.Fatal("want error for absent mac")
}
}
func TestNetns_SetupSequence(t *testing.T) {
f := &fakeNetRunner{linkOut: linkWithENI}
n := newTestNetns(f)
path, err := n.Setup("t-001", "52:54:00:DE:AD:01")
if err != nil {
t.Fatalf("setup: %v", err)
}
if path != netnsPathFor("t-001") {
t.Fatalf("want %q, got %q", netnsPathFor("t-001"), path)
}
hostIP, taskIP := credSubnet("t-001")
hostVeth := credVethName("t-001")
want := []string{
"ip netns add ecs-t-001",
"ip link set eth1 netns ecs-t-001",
"ip -n ecs-t-001 link set lo up",
"ip -n ecs-t-001 link set eth1 up",
"ip netns exec ecs-t-001 udhcpc -i eth1 -q -n",
"ip link add " + hostVeth + " type veth peer name ecsc0 netns ecs-t-001",
"ip addr add " + hostIP + "/30 dev " + hostVeth,
"ip -n ecs-t-001 addr add " + taskIP + "/30 dev ecsc0",
"ip -n ecs-t-001 route add 169.254.170.2/32 via " + hostIP,
}
for _, w := range want {
if !f.sawAny(w) {
t.Errorf("missing command %q in %v", w, f.joined())
}
}
}
func TestCredSubnet_UniquePer30(t *testing.T) {
hostA, taskA := credSubnet("task-a")
hostB, taskB := credSubnet("task-b")
if hostA == hostB {
t.Errorf("distinct tasks share a host IP: %s", hostA)
}
for _, ip := range []string{hostA, taskA, hostB, taskB} {
if !strings.HasPrefix(ip, "169.254.17") {
t.Errorf("cred IP %s outside 169.254.172.0/22", ip)
}
}
// host and task ends of one task sit in the same /30 (differ in last octet).
if hostA[:strings.LastIndexByte(hostA, '.')] != taskA[:strings.LastIndexByte(taskA, '.')] {
t.Errorf("host %s and task %s not in same /30", hostA, taskA)
}
}
func TestNetns_SetupNICNotFound(t *testing.T) {
f := &fakeNetRunner{linkOut: linkWithENI}
n := newTestNetns(f)
if _, err := n.Setup("t-001", "52:54:00:00:00:99"); err == nil {
t.Fatal("want timeout error for absent NIC")
}
if f.sawAny("netns add") {
t.Error("should not create netns when NIC never appears")
}
}
func TestNetns_SetupTearsDownOnStepFailure(t *testing.T) {
f := &fakeNetRunner{linkOut: linkWithENI, failOn: "udhcpc"}
n := newTestNetns(f)
if _, err := n.Setup("t-001", "52:54:00:de:ad:01"); err == nil {
t.Fatal("want error when udhcpc fails")
}
if !f.sawAny("ip netns del ecs-t-001") {
t.Errorf("expected teardown after step failure, got %v", f.joined())
}
}
func TestNetns_Teardown(t *testing.T) {
f := &fakeNetRunner{}
n := newTestNetns(f)
if err := n.Teardown("t-001"); err != nil {
t.Fatalf("teardown: %v", err)
}
if !f.sawAny("ip netns del ecs-t-001") {
t.Errorf("expected netns del, got %v", f.joined())
}
}
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"context"
"log/slog"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// reconcile re-adopts containers that survived an agent restart. It lists the
// runtime's containers and, for each task still running in this cluster,
// re-registers its IAM credentials, re-attaches the exit-wait, and refreshes its
// RUNNING state on the gateway. The returned set seeds pollAssignments so the
// gateway's re-delivered assignments for these tasks are acked but not re-run.
//
// It is best-effort: a nil runner or a List error returns an empty set and the
// agent falls back to its pre-reconcile behaviour (poll + run).
func (a *Agent) reconcile(ctx context.Context) map[string]bool {
adopted := map[string]bool{}
if a.runner == nil {
return adopted
}
containers, err := a.runner.List(ctx)
if err != nil {
slog.Warn("ecs-agent: reconcile list failed", "err", err)
return adopted
}
// Group this cluster's running containers back into their tasks. The Assign is
// reconstructed from labels — enough to re-register creds and tear down netns.
type task struct {
as *bus.Assign
containers []ctrruntime.Container
}
tasks := map[string]*task{}
for _, c := range containers {
if !c.Running || c.Labels[labelClusterName] != a.id.ClusterName {
continue
}
taskID := c.Labels[labelTaskID]
if taskID == "" {
continue
}
t := tasks[taskID]
if t == nil {
t = &task{as: &bus.Assign{
TaskID: taskID,
ClusterName: a.id.ClusterName,
CredID: c.Labels[labelCredID],
TaskRoleARN: c.Labels[labelTaskRoleARN],
ENIMacAddress: c.Labels[labelENIMac],
}}
tasks[taskID] = t
}
t.containers = append(t.containers, c)
}
for taskID, t := range tasks {
a.registerTaskCreds(t.as)
statuses := make([]bus.ContainerStatus, 0, len(t.containers))
for _, c := range t.containers {
name := c.Labels[labelContainerName]
go a.waitContainer(ctx, t.as, name, c.ID)
statuses = append(statuses, bus.ContainerStatus{Name: name, Status: bus.TaskStatusRunning, ContainerID: c.ID})
}
a.reportTaskState(t.as, bus.TaskStatusRunning, "", statuses)
adopted[taskID] = true
slog.Info("ecs-agent: re-adopted running task", "task", taskID, "containers", len(t.containers))
}
return adopted
}
+181
View File
@@ -0,0 +1,181 @@
package main
import (
"context"
"errors"
"slices"
"testing"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
const testTaskRole = "arn:aws:iam::111122223333:role/task"
// adoptedContainer builds a runtime.Container as the reconciler would discover it,
// stamping only the mulga.ecs.* labels that are non-empty.
func adoptedContainer(taskID, name, cluster, credID, role, mac string, running bool) ctrruntime.Container {
labels := map[string]string{
labelTaskID: taskID,
labelContainerName: name,
labelClusterName: cluster,
}
if credID != "" {
labels[labelCredID] = credID
}
if role != "" {
labels[labelTaskRoleARN] = role
}
if mac != "" {
labels[labelENIMac] = mac
}
return ctrruntime.Container{ID: containerID(taskID, name), Labels: labels, Running: running}
}
func waitedFor(t *testing.T, rt *ctrruntime.FakePuller, id string) {
t.Helper()
deadline := time.After(time.Second)
for !slices.Contains(rt.Waits(), id) {
select {
case <-deadline:
t.Fatalf("exit-wait never re-attached for %s; waited=%v", id, rt.Waits())
case <-time.After(2 * time.Millisecond):
}
}
}
// A running, correctly-labeled container is re-adopted: its credentials are
// re-registered, its RUNNING state refreshed, and its exit-wait re-attached.
func TestReconcile_AdoptsRunningLabeledContainer(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{
WaitErr: errors.New("blocked"),
Containers: []ctrruntime.Container{
adoptedContainer("t-001", "web", "default", "t-001", testTaskRole, "52:54:00:de:ad:01", true),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
adopted := a.reconcile(context.Background())
if !adopted["t-001"] {
t.Fatalf("t-001 not adopted; got %v", adopted)
}
if got := a.cred.roles["t-001"]; got != testTaskRole {
t.Errorf("cred not re-registered: roles[t-001] = %q, want %q", got, testTaskRole)
}
if n := countStatus(cp.taskStates(), "t-001", bus.TaskStatusRunning); n != 1 {
t.Errorf("RUNNING reports for t-001 = %d, want 1", n)
}
waitedFor(t, rt, containerID("t-001", "web"))
}
// pollAssignments seeded with an adopted task acks its re-delivered assignment
// but does not re-run it, while a genuinely new task still dispatches.
func TestPollAssignments_SeededTaskNotRerun(t *testing.T) {
mk := func(id string) bus.Assign {
return bus.Assign{
AccountID: "123456789012", ClusterName: "default", InstanceID: "i-abc123", TaskID: id,
Containers: []bus.AssignContainer{{Name: "web", Image: "registry/web:1", Command: []string{"/bin/true"}}},
}
}
adopted, fresh := mk("t-001"), mk("t-002")
cp := &fakeCP{pollReplies: [][]bus.Assign{
{adopted, fresh},
{adopted, fresh},
nil,
}}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
go a.pollAssignments(ctx, map[string]bool{"t-001": true})
deadline := time.After(time.Second)
for countStatus(cp.taskStates(), "t-002", bus.TaskStatusRunning) < 1 {
select {
case <-deadline:
t.Fatal("fresh task t-002 was never dispatched")
case <-time.After(2 * time.Millisecond):
}
}
time.Sleep(20 * time.Millisecond)
cancel()
if n := countStatus(cp.taskStates(), "t-001", bus.TaskStatusRunning); n != 0 {
t.Errorf("adopted task t-001 was re-run %d times, want 0", n)
}
if n := countStatus(cp.taskStates(), "t-002", bus.TaskStatusRunning); n != 1 {
t.Errorf("fresh task t-002 dispatched %d times, want 1", n)
}
var t1Acked bool
for _, ack := range cp.acks() {
for _, id := range ack {
if id == "t-001" {
t1Acked = true
}
}
}
if !t1Acked {
t.Errorf("adopted task t-001 was never acked; acks=%v", cp.acks())
}
}
// Containers for another cluster or with a stopped task are not adopted.
func TestReconcile_FiltersOtherClusterAndStopped(t *testing.T) {
rt := &ctrruntime.FakePuller{
WaitErr: errors.New("blocked"),
Containers: []ctrruntime.Container{
adoptedContainer("t-other", "web", "other", "", "", "", true),
adoptedContainer("t-stop", "web", "default", "", "", "", false),
adoptedContainer("t-ok", "web", "default", "", "", "", true),
},
}
a := newAgent(config{}, testIdentity(), &fakeCP{}, rt, rt, nil)
adopted := a.reconcile(context.Background())
if len(adopted) != 1 || !adopted["t-ok"] {
t.Fatalf("adopted = %v, want only t-ok", adopted)
}
}
// A nil runner or a List error degrades to an empty set without panicking.
func TestReconcile_DegradesGracefully(t *testing.T) {
noRunner := newAgent(config{}, testIdentity(), &fakeCP{}, nil, nil, nil)
if got := noRunner.reconcile(context.Background()); len(got) != 0 {
t.Errorf("nil runner: adopted = %v, want empty", got)
}
rt := &ctrruntime.FakePuller{ListErr: errors.New("boom")}
listErr := newAgent(config{}, testIdentity(), &fakeCP{}, rt, rt, nil)
if got := listErr.reconcile(context.Background()); len(got) != 0 {
t.Errorf("list error: adopted = %v, want empty", got)
}
}
// taskLabels emits the optional cred/role/MAC labels only when set.
func TestTaskLabels_OmitsEmptyOptionalLabels(t *testing.T) {
full := taskLabels(&bus.Assign{
TaskID: "t1", ClusterName: "c", CredID: "x", TaskRoleARN: testTaskRole, ENIMacAddress: "mac",
}, "web")
for k, want := range map[string]string{
labelCredID: "x", labelTaskRoleARN: testTaskRole, labelENIMac: "mac",
} {
if full[k] != want {
t.Errorf("full labels[%q] = %q, want %q", k, full[k], want)
}
}
bare := taskLabels(&bus.Assign{TaskID: "t1", ClusterName: "c"}, "web")
if len(bare) != 3 {
t.Errorf("bare labels = %v, want only taskID/containerName/clusterName", bare)
}
for _, k := range []string{labelCredID, labelTaskRoleARN, labelENIMac} {
if _, ok := bare[k]; ok {
t.Errorf("bare labels unexpectedly carry %q", k)
}
}
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"fmt"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// identity is the container instance's stable identity, assembled at boot from
// IMDS (account, instance, AZ) and config (cluster). It is the join key the
// scheduler uses to track this instance.
type identity struct {
AccountID string
ClusterName string
InstanceID string
AZ string
Hostname string
Capacity bus.InstanceCapacity
AgentVersion string
}
// registrar registers the host as a container instance through the gateway.
type registrar struct {
cp controlPlane
id identity
}
func newRegistrar(cp controlPlane, id identity) *registrar {
return &registrar{cp: cp, id: id}
}
// Register calls the gateway's RegisterContainerInstance. The scheduler records
// (or refreshes) the container instance on receipt.
func (r *registrar) Register() error {
if err := r.cp.Register(r.id); err != nil {
return fmt.Errorf("register instance %s: %w", r.id.InstanceID, err)
}
return nil
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"context"
"errors"
"sync"
"testing"
"time"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// fakeCP is a test double for the gateway control-plane. It records registers and
// task-state reports, and replays a scripted queue of poll responses.
type fakeCP struct {
mu sync.Mutex
registers int
states []bus.TaskState
pollAcks [][]string
pollReplies [][]bus.Assign
pollCalls int
registerErr bool
submitErr bool
}
var _ controlPlane = (*fakeCP)(nil)
func (f *fakeCP) Register(_ identity) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.registerErr {
return errors.New("register boom")
}
f.registers++
return nil
}
func (f *fakeCP) SubmitTaskState(st bus.TaskState) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.submitErr {
return errors.New("submit boom")
}
f.states = append(f.states, st)
return nil
}
func (f *fakeCP) PollAssignments(_, _ string, ack []string) ([]bus.Assign, error) {
f.mu.Lock()
defer f.mu.Unlock()
cp := append([]string(nil), ack...)
f.pollAcks = append(f.pollAcks, cp)
var out []bus.Assign
if f.pollCalls < len(f.pollReplies) {
out = f.pollReplies[f.pollCalls]
}
f.pollCalls++
return out, nil
}
func (f *fakeCP) registerCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.registers
}
func (f *fakeCP) taskStates() []bus.TaskState {
f.mu.Lock()
defer f.mu.Unlock()
return append([]bus.TaskState(nil), f.states...)
}
func (f *fakeCP) acks() [][]string {
f.mu.Lock()
defer f.mu.Unlock()
return append([][]string(nil), f.pollAcks...)
}
func testIdentity() identity {
return identity{
AccountID: "123456789012",
ClusterName: "default",
InstanceID: "i-abc123",
AZ: "us-east-1a",
Hostname: "ecs-host",
Capacity: bus.InstanceCapacity{CPU: 2048, MemoryMiB: 4096},
AgentVersion: "test",
}
}
func TestRegistrar_Register(t *testing.T) {
cp := &fakeCP{}
if err := newRegistrar(cp, testIdentity()).Register(); err != nil {
t.Fatalf("Register: %v", err)
}
if cp.registerCount() != 1 {
t.Errorf("registers = %d, want 1", cp.registerCount())
}
}
func TestRegistrar_RegisterError(t *testing.T) {
cp := &fakeCP{registerErr: true}
if err := newRegistrar(cp, testIdentity()).Register(); err == nil {
t.Fatal("expected register error")
}
}
func TestHeartbeater_Beat(t *testing.T) {
cp := &fakeCP{}
if err := newHeartbeater(cp, testIdentity(), time.Second).beat(); err != nil {
t.Fatalf("beat: %v", err)
}
if cp.registerCount() != 1 {
t.Errorf("beat should re-register once, got %d", cp.registerCount())
}
}
func TestHeartbeater_DefaultInterval(t *testing.T) {
h := newHeartbeater(&fakeCP{}, testIdentity(), 0)
if h.interval != defaultHeartbeat {
t.Errorf("interval = %v, want default", h.interval)
}
}
func TestHeartbeater_RunStopsOnContext(t *testing.T) {
cp := &fakeCP{}
h := newHeartbeater(cp, testIdentity(), 5*time.Millisecond)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { h.Run(ctx); close(done) }()
time.Sleep(25 * time.Millisecond)
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("Run did not stop on context cancel")
}
if cp.registerCount() == 0 {
t.Error("expected at least one re-register before cancel")
}
}
+118
View File
@@ -0,0 +1,118 @@
package runtime
import (
"context"
"fmt"
"time"
containerd "github.com/containerd/containerd/v2/client"
"github.com/containerd/containerd/v2/core/remotes"
"github.com/containerd/containerd/v2/core/remotes/docker"
"github.com/containerd/containerd/v2/pkg/namespaces"
"github.com/distribution/reference"
)
// ecsNamespace isolates ecs-agent images/containers in containerd, mirroring the
// "moby"/"k8s.io" convention other runtimes use.
const ecsNamespace = "ecs"
// servingProbeTimeout bounds the boot-time daemon liveness check.
const servingProbeTimeout = 3 * time.Second
// containerdPuller drives a real containerd daemon over its unix socket.
type containerdPuller struct {
client *containerd.Client
}
var _ ImagePuller = (*containerdPuller)(nil)
// New connects to the containerd daemon at socket (e.g.
// "/run/containerd/containerd.sock") and returns an ImagePuller. The containerd
// client dials lazily, so New actively probes the daemon with IsServing and
// returns an error if it is unreachable — otherwise a dead socket would only
// surface on the first pull, and the agent's boot-time "containerd unavailable"
// log would never fire.
func New(socket string) (Runtime, error) {
client, err := containerd.New(socket, containerd.WithDefaultNamespace(ecsNamespace))
if err != nil {
return nil, fmt.Errorf("connect containerd %s: %w", socket, err)
}
ctx, cancel := context.WithTimeout(context.Background(), servingProbeTimeout)
defer cancel()
serving, err := client.IsServing(ctx)
if err != nil || !serving {
_ = client.Close()
return nil, fmt.Errorf("containerd %s not serving: %w", socket, err)
}
return &containerdPuller{client: client}, nil
}
// Pull resolves credentials via r, then pulls and unpacks the image.
func (p *containerdPuller) Pull(ctx context.Context, spec PullSpec, r Resolver) (Image, error) {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
ref, err := normalizeRef(spec.Ref)
if err != nil {
return Image{}, err
}
resolver, err := newDockerResolver(ctx, ref, r)
if err != nil {
return Image{}, err
}
img, err := p.client.Pull(ctx, ref,
containerd.WithPullUnpack,
containerd.WithResolver(resolver),
)
if err != nil {
return Image{}, fmt.Errorf("pull %s: %w", ref, err)
}
target := img.Target()
return Image{
Ref: ref,
Digest: target.Digest.String(),
Size: target.Size,
}, nil
}
// normalizeRef expands a task-definition image string into the fully-qualified
// reference containerd's resolver requires: a bare "nginx:alpine" becomes
// "docker.io/library/nginx:alpine" and an untagged ref gets ":latest". Without
// this, containerd parses the bare ref as "dummy://nginx:alpine" and rejects it.
func normalizeRef(ref string) (string, error) {
named, err := reference.ParseNormalizedNamed(ref)
if err != nil {
return "", fmt.Errorf("parse image ref %q: %w", ref, err)
}
return reference.TagNameOnly(named).String(), nil
}
// Close releases the containerd client connection.
func (p *containerdPuller) Close() error {
if p.client == nil {
return nil
}
return p.client.Close()
}
// newDockerResolver builds a containerd remote resolver whose per-host
// credentials come from r (the ECR token resolver). Credentials are fetched once
// for ref and reused for every host containerd asks about during the pull.
func newDockerResolver(ctx context.Context, ref string, r Resolver) (remotes.Resolver, error) {
creds := func(string) (string, string, error) { return "", "", nil }
if r != nil {
user, pass, _, err := r.Authorize(ctx, ref)
if err != nil {
return nil, fmt.Errorf("authorize %s: %w", ref, err)
}
creds = func(string) (string, string, error) { return user, pass, nil }
}
authorizer := docker.NewDockerAuthorizer(docker.WithAuthCreds(creds))
return docker.NewResolver(docker.ResolverOptions{
Hosts: docker.ConfigureDefaultRegistries(docker.WithAuthorizer(authorizer)),
}), nil
}
+142
View File
@@ -0,0 +1,142 @@
package runtime
import (
"context"
"fmt"
"sort"
containerd "github.com/containerd/containerd/v2/client"
"github.com/containerd/containerd/v2/pkg/cio"
"github.com/containerd/containerd/v2/pkg/namespaces"
"github.com/containerd/containerd/v2/pkg/oci"
specs "github.com/opencontainers/runtime-spec/specs-go"
)
var _ Runner = (*containerdPuller)(nil)
// Run creates a container from an already-pulled image and starts its task. The
// container joins the host network namespace (bridge/CNI task networking is a
// later sprint) and carries spec.Labels for the reboot reconciler. id must be
// unique within the "ecs" namespace.
func (p *containerdPuller) Run(ctx context.Context, id string, spec RunSpec) (string, error) {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
ref, err := normalizeRef(spec.Image)
if err != nil {
return "", err
}
image, err := p.client.GetImage(ctx, ref)
if err != nil {
return "", fmt.Errorf("get image %s: %w", ref, err)
}
specOpts := []oci.SpecOpts{
oci.WithImageConfig(image),
oci.WithHostHostsFile,
oci.WithHostResolvconf,
}
if spec.NetnsPath != "" {
// awsvpc: join the task ENI netns built by the agent.
specOpts = append(specOpts, oci.WithLinuxNamespace(specs.LinuxNamespace{
Type: specs.NetworkNamespace,
Path: spec.NetnsPath,
}))
} else {
// bridge/host: share the VM (host) netns.
specOpts = append(specOpts, oci.WithHostNamespace(specs.NetworkNamespace))
}
if len(spec.Command) > 0 {
specOpts = append(specOpts, oci.WithProcessArgs(spec.Command...))
}
if len(spec.Env) > 0 {
specOpts = append(specOpts, oci.WithEnv(envSlice(spec.Env)))
}
container, err := p.client.NewContainer(ctx, id,
containerd.WithNewSnapshot(id+"-snapshot", image),
containerd.WithNewSpec(specOpts...),
containerd.WithContainerLabels(spec.Labels),
)
if err != nil {
return "", fmt.Errorf("create container %s: %w", id, err)
}
task, err := container.NewTask(ctx, cio.NullIO)
if err != nil {
_ = container.Delete(ctx, containerd.WithSnapshotCleanup)
return "", fmt.Errorf("create task %s: %w", id, err)
}
if err := task.Start(ctx); err != nil {
_, _ = task.Delete(ctx)
_ = container.Delete(ctx, containerd.WithSnapshotCleanup)
return "", fmt.Errorf("start task %s: %w", id, err)
}
return id, nil
}
// Wait blocks until the container's task exits and returns its exit code.
func (p *containerdPuller) Wait(ctx context.Context, containerID string) (RunStatus, error) {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
container, err := p.client.LoadContainer(ctx, containerID)
if err != nil {
return RunStatus{}, fmt.Errorf("load container %s: %w", containerID, err)
}
task, err := container.Task(ctx, nil)
if err != nil {
return RunStatus{}, fmt.Errorf("attach task %s: %w", containerID, err)
}
statusC, err := task.Wait(ctx)
if err != nil {
return RunStatus{}, fmt.Errorf("wait task %s: %w", containerID, err)
}
status := <-statusC
code, _, err := status.Result()
return RunStatus{ExitCode: int(code)}, err
}
// List enumerates every container in the ecs namespace with its labels and
// whether its task is running, so the agent can re-adopt live containers after a
// restart. Per-container label/task errors are treated as "not running" rather
// than failing the whole list.
func (p *containerdPuller) List(ctx context.Context) ([]Container, error) {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
containers, err := p.client.Containers(ctx)
if err != nil {
return nil, fmt.Errorf("list containers: %w", err)
}
out := make([]Container, 0, len(containers))
for _, c := range containers {
labels, _ := c.Labels(ctx)
running := false
if task, terr := c.Task(ctx, nil); terr == nil {
if status, serr := task.Status(ctx); serr == nil {
running = status.Status == containerd.Running
}
}
out = append(out, Container{ID: c.ID(), Labels: labels, Running: running})
}
return out, nil
}
// Remove kills and deletes the container's task, then the container + snapshot.
func (p *containerdPuller) Remove(ctx context.Context, containerID string) error {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
container, err := p.client.LoadContainer(ctx, containerID)
if err != nil {
return nil //nolint:nilerr // load failure means the container is already gone
}
if task, terr := container.Task(ctx, nil); terr == nil {
_, _ = task.Delete(ctx, containerd.WithProcessKill)
}
return container.Delete(ctx, containerd.WithSnapshotCleanup)
}
// envSlice renders an env map as sorted "K=V" entries for deterministic specs.
func envSlice(env map[string]string) []string {
out := make([]string, 0, len(env))
for k, v := range env {
out = append(out, k+"="+v)
}
sort.Strings(out)
return out
}
+82
View File
@@ -0,0 +1,82 @@
// Package runtime is the ecs-agent's container-runtime seam. The agent pulls
// task images and (from Sprint 4e) runs containers through an ImagePuller; the
// real implementation drives containerd, but the interface lets tests and the
// scheduler-only build run against a fake.
package runtime
import "context"
// PullSpec describes one image to pull.
type PullSpec struct {
// Ref is the full image reference, e.g.
// "123456789012.dkr.ecr.us-east-1.spinifex.internal/app:latest".
Ref string
}
// Image identifies a pulled image in the runtime's content store.
type Image struct {
Ref string
Digest string
Size int64
}
// Resolver supplies registry credentials for a given image reference. The
// ecs-agent backs this with ECR GetAuthorizationToken (see ecrresolver.go).
type Resolver interface {
// Authorize returns the username, password and registry endpoint to use for
// ref. An empty user/pass means anonymous pull.
Authorize(ctx context.Context, ref string) (user, pass, endpoint string, err error)
}
// ImagePuller pulls images into the local runtime. Close releases the runtime
// client connection.
type ImagePuller interface {
Pull(ctx context.Context, spec PullSpec, r Resolver) (Image, error)
Close() error
}
// RunSpec describes a single container the agent must create and start. v1 runs
// the container in the host network namespace (bridge/CNI task networking lands
// in a later sprint); Labels carry the mulga.ecs.* task identity for the reboot
// reconciler.
type RunSpec struct {
Image string
Command []string
Env map[string]string
Labels map[string]string
// NetnsPath, when set, joins the container to a pre-built network namespace
// (awsvpc task ENI). Empty keeps the container in the host (VM) netns —
// bridge/host mode behaviour.
NetnsPath string
}
// RunStatus is a finished container's outcome.
type RunStatus struct {
ExitCode int
}
// Container is a container the runtime already knows about, discovered by List.
// Labels carry the mulga.ecs.* task identity so the agent can re-adopt running
// containers after a restart; Running reports whether its task is live.
type Container struct {
ID string
Labels map[string]string
Running bool
}
// Runner creates and starts containers from already-pulled images. id is a
// caller-unique container ID; Wait blocks until the container exits; Remove
// tears down the container + its task; List enumerates known containers so the
// reboot reconciler can re-adopt the ones still running.
type Runner interface {
Run(ctx context.Context, id string, spec RunSpec) (containerID string, err error)
Wait(ctx context.Context, containerID string) (RunStatus, error)
Remove(ctx context.Context, containerID string) error
List(ctx context.Context) ([]Container, error)
}
// Runtime is the full container runtime the ecs-agent drives: pull + run.
type Runtime interface {
ImagePuller
Runner
}
+125
View File
@@ -0,0 +1,125 @@
package runtime
import (
"context"
"fmt"
"sync"
)
// FakePuller is an in-memory ImagePuller for tests and the scheduler-only build.
// It records pull requests and replays a programmed result or error.
type FakePuller struct {
mu sync.Mutex
Pulls []PullSpec
Authzd []string
Result Image
Err error
Closed bool
OnPull func(PullSpec) (Image, error)
// Run bookkeeping.
Runs []RunSpec
RunErr error
WaitCode int
WaitErr error
Waited []string
Removed []string
OnRun func(string, RunSpec) (string, error)
// List bookkeeping: Containers is replayed by List; ListErr forces a failure;
// Listed records that List was called.
Containers []Container
ListErr error
Listed bool
}
var (
_ ImagePuller = (*FakePuller)(nil)
_ Runner = (*FakePuller)(nil)
)
// Pull records the spec, exercises the resolver, and returns the programmed
// result. If OnPull is set it takes precedence over Result/Err.
func (f *FakePuller) Pull(ctx context.Context, spec PullSpec, r Resolver) (Image, error) {
f.mu.Lock()
f.Pulls = append(f.Pulls, spec)
f.mu.Unlock()
if r != nil {
user, _, _, err := r.Authorize(ctx, spec.Ref)
if err != nil {
return Image{}, fmt.Errorf("authorize %s: %w", spec.Ref, err)
}
f.mu.Lock()
f.Authzd = append(f.Authzd, user)
f.mu.Unlock()
}
if f.OnPull != nil {
return f.OnPull(spec)
}
if f.Err != nil {
return Image{}, f.Err
}
res := f.Result
if res.Ref == "" {
res.Ref = spec.Ref
}
return res, nil
}
// Close marks the puller closed.
func (f *FakePuller) Close() error {
f.mu.Lock()
defer f.mu.Unlock()
f.Closed = true
return nil
}
// Run records the spec and returns id (or the programmed result/error).
func (f *FakePuller) Run(_ context.Context, id string, spec RunSpec) (string, error) {
f.mu.Lock()
f.Runs = append(f.Runs, spec)
f.mu.Unlock()
if f.OnRun != nil {
return f.OnRun(id, spec)
}
if f.RunErr != nil {
return "", f.RunErr
}
return id, nil
}
// Wait records the container ID and returns the programmed exit code/error.
func (f *FakePuller) Wait(_ context.Context, containerID string) (RunStatus, error) {
f.mu.Lock()
f.Waited = append(f.Waited, containerID)
f.mu.Unlock()
return RunStatus{ExitCode: f.WaitCode}, f.WaitErr
}
// Remove records the container ID.
func (f *FakePuller) Remove(_ context.Context, containerID string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.Removed = append(f.Removed, containerID)
return nil
}
// Waits returns a copy of the container IDs passed to Wait, for assertions.
func (f *FakePuller) Waits() []string {
f.mu.Lock()
defer f.mu.Unlock()
return append([]string(nil), f.Waited...)
}
// List replays the programmed container set (or error).
func (f *FakePuller) List(_ context.Context) ([]Container, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.Listed = true
if f.ListErr != nil {
return nil, f.ListErr
}
return f.Containers, nil
}
+89
View File
@@ -0,0 +1,89 @@
package runtime
import (
"context"
"errors"
"testing"
)
// staticResolver is a test Resolver returning fixed creds or an error.
type staticResolver struct {
user, pass, endpoint string
err error
}
func (s staticResolver) Authorize(_ context.Context, _ string) (string, string, string, error) {
return s.user, s.pass, s.endpoint, s.err
}
func TestFakePuller_RecordsAndDefaultsRef(t *testing.T) {
f := &FakePuller{}
img, err := f.Pull(context.Background(), PullSpec{Ref: "repo/app:1"}, staticResolver{user: "AWS"})
if err != nil {
t.Fatalf("Pull: %v", err)
}
if img.Ref != "repo/app:1" {
t.Errorf("Ref defaulted wrong: %q", img.Ref)
}
if len(f.Pulls) != 1 || f.Pulls[0].Ref != "repo/app:1" {
t.Errorf("Pulls = %+v", f.Pulls)
}
if len(f.Authzd) != 1 || f.Authzd[0] != "AWS" {
t.Errorf("Authzd = %+v", f.Authzd)
}
}
func TestFakePuller_ResolverError(t *testing.T) {
f := &FakePuller{}
_, err := f.Pull(context.Background(), PullSpec{Ref: "r"}, staticResolver{err: errors.New("boom")})
if err == nil {
t.Fatal("expected resolver error to propagate")
}
}
func TestFakePuller_ProgrammedError(t *testing.T) {
f := &FakePuller{Err: errors.New("pull failed")}
if _, err := f.Pull(context.Background(), PullSpec{Ref: "r"}, nil); err == nil {
t.Fatal("expected programmed error")
}
}
func TestFakePuller_OnPullOverrides(t *testing.T) {
want := Image{Ref: "x", Digest: "sha256:abc", Size: 42}
f := &FakePuller{Err: errors.New("ignored"), OnPull: func(PullSpec) (Image, error) {
return want, nil
}}
got, err := f.Pull(context.Background(), PullSpec{Ref: "r"}, nil)
if err != nil || got != want {
t.Fatalf("got %+v, %v; want %+v", got, err, want)
}
}
func TestFakePuller_Close(t *testing.T) {
f := &FakePuller{}
if err := f.Close(); err != nil || !f.Closed {
t.Fatalf("Close: err=%v closed=%v", err, f.Closed)
}
}
func TestNormalizeRef(t *testing.T) {
cases := map[string]string{
"nginx:alpine": "docker.io/library/nginx:alpine",
"nginx": "docker.io/library/nginx:latest",
"ghcr.io/org/app:1": "ghcr.io/org/app:1",
"123456789012.dkr.ecr.ap-southeast-2.spinifex.internal/app:1": "123456789012.dkr.ecr.ap-southeast-2.spinifex.internal/app:1",
}
for in, want := range cases {
got, err := normalizeRef(in)
if err != nil {
t.Errorf("normalizeRef(%q): %v", in, err)
continue
}
if got != want {
t.Errorf("normalizeRef(%q) = %q, want %q", in, got, want)
}
}
if _, err := normalizeRef("Bad::Ref"); err == nil {
t.Error("normalizeRef accepted invalid ref")
}
}
+234
View File
@@ -0,0 +1,234 @@
package main
import (
"context"
"fmt"
"log/slog"
"maps"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// pollAssignments drains the instance's assignment inbox on a ticker until ctx is
// cancelled. Each assign is dispatched to runTask exactly once; the taskIDs seen
// on a poll are acked on the next poll so the gateway can drop them — a crash
// before ack re-delivers (at-least-once), matching ACS. A failed poll is logged
// and retried on the next tick rather than killing the loop. dispatched is seeded
// by the startup reconcile with tasks already adopted from running containers, so
// their re-delivered assignments are acked but not re-run.
func (a *Agent) pollAssignments(ctx context.Context, dispatched map[string]bool) {
ticker := time.NewTicker(a.cfg.PollInterval)
defer ticker.Stop()
var ackNext []string
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
assigns, err := a.cp.PollAssignments(a.id.ClusterName, a.id.InstanceID, ackNext)
if err != nil {
slog.Warn("ecs-agent: poll assignments failed", "err", err)
continue
}
ackNext = ackNext[:0]
for i := range assigns {
as := assigns[i]
if !dispatched[as.TaskID] {
dispatched[as.TaskID] = true
slog.Info("ecs-agent: task assigned", "task", as.TaskID, "containers", len(as.Containers))
go a.runTask(ctx, &as)
}
ackNext = append(ackNext, as.TaskID)
}
}
}
}
// runTask pulls each container image, starts the containers, and reports task
// state on the bus. A missing runtime or any per-container failure reports the
// task STOPPED with a reason; success reports RUNNING and waits for exit.
func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
if a.puller == nil || a.runner == nil {
a.reportTaskState(as, bus.TaskStatusStopped, "containerd unavailable on agent", nil)
return
}
// awsvpc: build the task netns from the hot-plugged ENI before any container.
netnsPath, err := a.setupTaskNetns(as)
if err != nil {
slog.Error("ecs-agent: task netns setup failed", "task", as.TaskID, "err", err)
a.reportTaskState(as, bus.TaskStatusStopped, "network setup failed: "+err.Error(), nil)
return
}
credID := a.registerTaskCreds(as)
statuses := make([]bus.ContainerStatus, 0, len(as.Containers))
for _, c := range as.Containers {
if _, err := a.puller.Pull(ctx, ctrruntime.PullSpec{Ref: c.Image}, a.resolver); err != nil {
slog.Error("ecs-agent: pull failed", "task", as.TaskID, "image", c.Image, "err", err)
a.teardownTaskNetns(as)
a.reportTaskState(as, bus.TaskStatusStopped, "image pull failed: "+err.Error(), statuses)
return
}
cid := containerID(as.TaskID, c.Name)
spec := ctrruntime.RunSpec{
Image: c.Image,
Command: c.Command,
Env: withCredEnv(c.Environment, credID),
Labels: taskLabels(as, c.Name),
NetnsPath: netnsPath,
}
id, err := a.runner.Run(ctx, cid, spec)
if err != nil {
slog.Error("ecs-agent: run failed", "task", as.TaskID, "container", c.Name, "err", err)
a.teardownTaskNetns(as)
a.reportTaskState(as, bus.TaskStatusStopped, "container start failed: "+err.Error(), statuses)
return
}
statuses = append(statuses, bus.ContainerStatus{Name: c.Name, Status: bus.TaskStatusRunning, ContainerID: id})
go a.waitContainer(ctx, as, c.Name, id)
}
a.reportTaskState(as, bus.TaskStatusRunning, "", statuses)
slog.Info("ecs-agent: task running", "task", as.TaskID, "containers", len(statuses))
}
// waitContainer blocks until a container exits, then reports the task STOPPED
// with the exit code. v1 stops the whole task when its first container exits.
func (a *Agent) waitContainer(ctx context.Context, as *bus.Assign, name, containerID string) {
status, err := a.runner.Wait(ctx, containerID)
if err != nil {
slog.Warn("ecs-agent: wait container failed", "task", as.TaskID, "container", name, "err", err)
return
}
exit := status.ExitCode
a.teardownTaskNetns(as)
statuses := []bus.ContainerStatus{{Name: name, Status: bus.TaskStatusStopped, ContainerID: containerID, ExitCode: &exit}}
a.reportTaskState(as, bus.TaskStatusStopped, fmt.Sprintf("container %s exited (%d)", name, exit), statuses)
slog.Info("ecs-agent: task stopped", "task", as.TaskID, "container", name, "exitCode", exit)
}
// setupTaskNetns builds the awsvpc task netns from the hot-plugged ENI. Bridge/
// host tasks (no ENI MAC) and a nil controller are no-ops returning an empty path.
func (a *Agent) setupTaskNetns(as *bus.Assign) (string, error) {
if as.ENIMacAddress == "" || a.netns == nil {
return "", nil
}
return a.netns.Setup(as.TaskID, as.ENIMacAddress)
}
// teardownTaskNetns releases the task's IAM credential registration, then removes
// the awsvpc task netns (the latter is a no-op for bridge/host tasks). It runs at
// every task-stop path so credentials never outlive the task that owns them.
func (a *Agent) teardownTaskNetns(as *bus.Assign) {
if a.cred != nil {
a.cred.Deregister(taskCredID(as))
}
if as.ENIMacAddress == "" || a.netns == nil {
return
}
if err := a.netns.Teardown(as.TaskID); err != nil {
slog.Warn("ecs-agent: task netns teardown", "task", as.TaskID, "err", err)
}
}
// registerTaskCreds registers the task's credID -> role mapping with the
// credential endpoint and returns the credID (empty when the task has no role).
func (a *Agent) registerTaskCreds(as *bus.Assign) string {
credID := taskCredID(as)
if credID == "" || a.cred == nil {
return credID
}
a.cred.Register(credID, as.TaskRoleARN)
return credID
}
// taskCredID is the credential ID a task's containers fetch credentials under:
// the scheduler-assigned CredID, falling back to the taskID. Empty when the task
// carries no IAM role.
func taskCredID(as *bus.Assign) string {
if as.TaskRoleARN == "" {
return ""
}
if as.CredID != "" {
return as.CredID
}
return as.TaskID
}
// withCredEnv returns env with AWS_CONTAINER_CREDENTIALS_RELATIVE_URI set for
// credID, copying the map so the assign's container env is left untouched. A
// blank credID (no task role) returns env unchanged.
func withCredEnv(env map[string]string, credID string) map[string]string {
if credID == "" {
return env
}
out := make(map[string]string, len(env)+1)
maps.Copy(out, env)
out["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"] = credRelativeURI(credID)
return out
}
// reportTaskState reports a task transition through the gateway's
// SubmitTaskStateChange action.
func (a *Agent) reportTaskState(as *bus.Assign, status, reason string, containers []bus.ContainerStatus) {
if a.cp == nil {
return
}
msg := bus.TaskState{
AccountID: a.id.AccountID,
ClusterName: a.id.ClusterName,
InstanceID: a.id.InstanceID,
TaskID: as.TaskID,
LastStatus: status,
Containers: containers,
Reason: reason,
ReportedAt: time.Now().UTC(),
}
if err := a.cp.SubmitTaskState(msg); err != nil {
slog.Error("ecs-agent: report task-state failed", "task", as.TaskID, "err", err)
}
}
// containerID composes a containerd-valid container ID from a task + container.
func containerID(taskID, name string) string {
return fmt.Sprintf("%s-%s", taskID, name)
}
// mulga.ecs.* label keys stamped on every container. The reboot reconciler reads
// them back to re-associate running containers with their task on restart; the
// cred/role/MAC labels make a container self-describing enough to re-register its
// credentials and tear down its netns without the original assignment.
const (
labelTaskID = "mulga.ecs.taskID"
labelContainerName = "mulga.ecs.containerName"
labelClusterName = "mulga.ecs.clusterName"
labelCredID = "mulga.ecs.credID"
labelTaskRoleARN = "mulga.ecs.taskRoleArn"
labelENIMac = "mulga.ecs.eniMac"
)
// taskLabels are the mulga.ecs.* labels stamped on a container. The cred/role/MAC
// labels are omitted when empty so a container carries only what it needs.
func taskLabels(as *bus.Assign, name string) map[string]string {
labels := map[string]string{
labelTaskID: as.TaskID,
labelContainerName: name,
labelClusterName: as.ClusterName,
}
if credID := taskCredID(as); credID != "" {
labels[labelCredID] = credID
}
if as.TaskRoleARN != "" {
labels[labelTaskRoleARN] = as.TaskRoleARN
}
if as.ENIMacAddress != "" {
labels[labelENIMac] = as.ENIMacAddress
}
return labels
}
+229
View File
@@ -0,0 +1,229 @@
package main
import (
"context"
"errors"
"testing"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
func testAssign() *bus.Assign {
return &bus.Assign{
AccountID: "123456789012",
ClusterName: "default",
InstanceID: "i-abc123",
TaskID: "t-001",
Containers: []bus.AssignContainer{
{Name: "web", Image: "registry/web:1", Command: []string{"/bin/true"},
Environment: map[string]string{"FOO": "bar"}},
},
}
}
func newRunAgent(cp controlPlane, rt *ctrruntime.FakePuller) *Agent {
return newAgent(config{}, testIdentity(), cp, rt, rt, nil)
}
// runTask with no runtime reports the task STOPPED instead of crashing.
func TestRunTask_NoRuntimeReportsStopped(t *testing.T) {
cp := &fakeCP{}
a := newAgent(config{}, testIdentity(), cp, nil, nil, nil)
a.runTask(context.Background(), testAssign())
st := cp.taskStates()
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
t.Fatalf("want one STOPPED state, got %+v", st)
}
if st[0].Reason == "" {
t.Error("STOPPED state missing reason")
}
}
// Happy path: pull + run reports RUNNING with the container ID; Wait blocking
// (forced error) means no STOPPED overwrite, so we can assert the RUNNING frame.
func TestRunTask_PullRunReportsRunning(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
as := testAssign()
a.runTask(context.Background(), as)
if len(rt.Pulls) != 1 || rt.Pulls[0].Ref != "registry/web:1" {
t.Fatalf("expected one pull of registry/web:1, got %+v", rt.Pulls)
}
if len(rt.Runs) != 1 || rt.Runs[0].Image != "registry/web:1" {
t.Fatalf("expected one run, got %+v", rt.Runs)
}
if rt.Runs[0].Labels["mulga.ecs.taskID"] != "t-001" {
t.Errorf("missing task label: %+v", rt.Runs[0].Labels)
}
st := cp.taskStates()
if len(st) == 0 || st[len(st)-1].LastStatus != bus.TaskStatusRunning {
t.Fatalf("want RUNNING final state, got %+v", st)
}
if len(st[len(st)-1].Containers) != 1 || st[len(st)-1].Containers[0].ContainerID != "t-001-web" {
t.Errorf("RUNNING state container wrong: %+v", st[len(st)-1].Containers)
}
}
// A pull failure reports STOPPED and never starts the container.
func TestRunTask_PullFailureReportsStopped(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{Err: errors.New("pull denied")}
a := newRunAgent(cp, rt)
a.runTask(context.Background(), testAssign())
if len(rt.Runs) != 0 {
t.Errorf("container should not run after pull failure, got %+v", rt.Runs)
}
st := cp.taskStates()
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
t.Fatalf("want STOPPED, got %+v", st)
}
}
// A run failure reports STOPPED.
func TestRunTask_RunFailureReportsStopped(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{RunErr: errors.New("start boom")}
a := newRunAgent(cp, rt)
a.runTask(context.Background(), testAssign())
st := cp.taskStates()
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
t.Fatalf("want STOPPED, got %+v", st)
}
}
// On container exit, waitContainer reports STOPPED with the exit code.
func TestRunTask_ExitReportsStoppedWithCode(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitCode: 7}
a := newRunAgent(cp, rt)
a.runTask(context.Background(), testAssign())
deadline := time.After(time.Second)
for {
st := cp.taskStates()
if len(st) > 0 && st[len(st)-1].LastStatus == bus.TaskStatusStopped {
last := st[len(st)-1]
if len(last.Containers) != 1 || last.Containers[0].ExitCode == nil || *last.Containers[0].ExitCode != 7 {
t.Fatalf("want exit code 7, got %+v", last.Containers)
}
return
}
select {
case <-deadline:
t.Fatalf("no STOPPED state after exit; got %+v", cp.taskStates())
case <-time.After(2 * time.Millisecond):
}
}
}
// pollAssignments dispatches each assign once and acks it on the next poll.
func TestPollAssignments_DispatchesOnceAndAcks(t *testing.T) {
as := testAssign()
cp := &fakeCP{pollReplies: [][]bus.Assign{
{*as}, // poll 1: one pending assign
{*as}, // poll 2: still present (not yet acked when poll 2 was issued)
nil, // poll 3: gateway dropped it after the ack
}}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
go a.pollAssignments(ctx, map[string]bool{})
// runTask reports RUNNING through the (mutex-guarded) control plane. With the
// assign returned on two consecutive polls, dedup means exactly one dispatch →
// exactly one RUNNING report.
deadline := time.After(time.Second)
for {
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running >= 1 {
break
}
select {
case <-deadline:
t.Fatal("assign was never dispatched (no RUNNING report)")
case <-time.After(2 * time.Millisecond):
}
}
time.Sleep(20 * time.Millisecond)
cancel()
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running != 1 {
t.Errorf("task dispatched %d times, want exactly 1 (dedup on taskID)", running)
}
// The taskID must appear in a later poll's ack list.
acked := false
for _, ack := range cp.acks() {
for _, id := range ack {
if id == as.TaskID {
acked = true
}
}
}
if !acked {
t.Errorf("task %s was never acked; acks=%v", as.TaskID, cp.acks())
}
}
func countStatus(states []bus.TaskState, taskID, status string) int {
n := 0
for _, st := range states {
if st.TaskID == taskID && st.LastStatus == status {
n++
}
}
return n
}
// awsvpc: an assign carrying an ENI MAC builds the task netns and passes its
// path to the container RunSpec.
func TestRunTask_AwsvpcBuildsNetns(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
f := &fakeNetRunner{linkOut: linkWithENI}
a.netns = newTestNetns(f)
as := testAssign()
as.ENIMacAddress = "52:54:00:de:ad:01"
a.runTask(context.Background(), as)
if !f.sawAny("ip netns add ecs-t-001") {
t.Fatalf("expected task netns build, got %v", f.joined())
}
if len(rt.Runs) != 1 || rt.Runs[0].NetnsPath != netnsPathFor("t-001") {
t.Fatalf("want NetnsPath %q on RunSpec, got %+v", netnsPathFor("t-001"), rt.Runs)
}
}
// bridge/host: no ENI MAC means no netns is built and the container runs in the
// host (VM) netns (empty NetnsPath).
func TestRunTask_NoMacSkipsNetns(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
f := &fakeNetRunner{linkOut: linkWithENI}
a.netns = newTestNetns(f)
a.runTask(context.Background(), testAssign())
if len(f.joined()) != 0 {
t.Fatalf("expected no netns commands, got %v", f.joined())
}
if len(rt.Runs) != 1 || rt.Runs[0].NetnsPath != "" {
t.Fatalf("want empty NetnsPath, got %+v", rt.Runs)
}
}
func TestContainerID(t *testing.T) {
if got := containerID("t-1", "web"); got != "t-1-web" {
t.Errorf("containerID = %q, want t-1-web", got)
}
}
+90 -11
View File
@@ -35,6 +35,7 @@ import (
"os/exec"
"path"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
@@ -115,6 +116,23 @@ Each phase waits for all nodes to ACK before proceeding to the next.`,
Run: runClusterShutdown,
}
var nodeCmd = &cobra.Command{
Use: "node",
Short: "Node-local operations",
Long: `Node-local administrative operations such as a graceful local guest drain.`,
}
var nodeDrainCmd = &cobra.Command{
Use: "drain",
Short: "Gracefully drain guests on the local node",
Long: `Run the GATE and DRAIN shutdown phases against the local node only: power down
its guests via QMP and unmount their volumes (flushing the viperblock WAL) while
every service is still running. STORAGE/PERSIST/INFRA are left to systemd's
ordered unit teardown. This is the ExecStop of spinifex-shutdown.service, so a
systemctl stop or host reboot drains guests before any storage service stops.`,
Run: runNodeDrainLocal,
}
var clusterDrainDHCPCmd = &cobra.Command{
Use: "drain-dhcp",
Short: "Release all upstream DHCP leases held by vpcd",
@@ -258,6 +276,11 @@ func init() {
clusterCmd.AddCommand(clusterDrainDHCPCmd)
clusterDrainDHCPCmd.Flags().Duration("timeout", 30*time.Second, "Reply-collection window for vpcd drain responders")
adminCmd.AddCommand(nodeCmd)
nodeCmd.AddCommand(nodeDrainCmd)
nodeDrainCmd.Flags().Bool("local", false, "Drain the local node only (required)")
nodeDrainCmd.Flags().Duration("timeout", 120*time.Second, "Maximum time to wait per phase")
adminCmd.AddCommand(imagesCmd)
imagesCmd.AddCommand(imagesImportCmd)
imagesCmd.AddCommand(imagesListCmd)
@@ -643,6 +666,13 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
EncryptionEnabled: mkey != nil,
}
// Bake the deployment CA into the image trust store so a stock cloud image
// trusts the gateway from first boot. Best-effort: never fails the import.
// The CA path derives from the --spinifex-dir data root (its config/ symlink
// holds ca.pem), not NodeBaseDir(): node.BaseDir is unset in the import
// command, which would collapse the path to a relative config/ca.pem.
bakeCACertIntoImage(extractedImagePath, filepath.Join(baseDir, "config", "ca.pem"))
err = v_utils.ImportDiskImage(&s3Config, &vbConfig, extractedImagePath)
if err != nil {
@@ -655,6 +685,64 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
}
// caBakeRunCommand installs the uploaded CA into the guest trust store across
// the distro families, then removes the staged copy. Each branch is gated on its
// updater existing so only the matching distro's branch writes anything (install
// -D creates the anchor dir on minimal images). Debian/Ubuntu use
// update-ca-certificates, RHEL/Rocky use update-ca-trust, and stock Alpine ships
// only ca-certificates-bundle (no updater, no anchor dir) so it falls back to
// appending the PEM to the static bundle. The install chain's exit code is
// preserved past the cleanup rm so a failure surfaces instead of a false success.
// The RHEL branch restorecon's only the trust-store paths it touched so the cert
// is correctly labelled without the image-wide --no-selinux-relabel (below)
// leaving an unlabelled bundle; restorecon is best-effort and never fails the bake.
const caBakeRunCommand = `( { command -v update-ca-certificates >/dev/null && install -D -m644 /tmp/spinifex-ca.pem /usr/local/share/ca-certificates/spinifex.crt && update-ca-certificates; } || ` +
`{ command -v update-ca-trust >/dev/null && install -D -m644 /tmp/spinifex-ca.pem /etc/pki/ca-trust/source/anchors/spinifex.crt && update-ca-trust && ` +
`{ command -v restorecon >/dev/null 2>&1 && restorecon -RF /etc/pki/ca-trust/source/anchors/spinifex.crt /etc/pki/ca-trust/extracted >/dev/null 2>&1; true; }; } || ` +
`{ cat /tmp/spinifex-ca.pem >> /etc/ssl/certs/ca-certificates.crt; } ); ` +
`rc=$?; rm -f /tmp/spinifex-ca.pem; exit $rc`
// caBakeTimeout bounds the virt-customize run so a stalled libguestfs appliance
// cannot hang the import indefinitely; on timeout the bake degrades to a skip.
const caBakeTimeout = 5 * time.Minute
// caBakeCmd builds the virt-customize invocation that uploads the deployment CA
// into the disk image at imagePath and installs it into the guest trust store.
// --no-selinux-relabel stops virt-customize flagging the image for a first-boot
// SELinux autorelabel: that relabel+reboot corrupts XFS roots (RHEL/Rocky) on the
// reboot, so the run-command relabels only the touched trust paths instead.
func caBakeCmd(ctx context.Context, imagePath, caCertPath string) *exec.Cmd {
return exec.CommandContext(ctx, "virt-customize", "-a", imagePath,
"--no-selinux-relabel",
"--upload", caCertPath+":/tmp/spinifex-ca.pem",
"--run-command", caBakeRunCommand)
}
// caBakeRunner resolves virt-customize and runs the CA bake; overridable in tests.
var caBakeRunner = func(imagePath, caCertPath string) ([]byte, error) {
if _, err := exec.LookPath("virt-customize"); err != nil {
return nil, fmt.Errorf("virt-customize not found: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), caBakeTimeout)
defer cancel()
return caBakeCmd(ctx, imagePath, caCertPath).CombinedOutput()
}
// bakeCACertIntoImage uploads the deployment CA into the image's trust store via
// virt-customize so an imported stock image trusts the gateway from first boot.
// Best-effort: a missing CA, an absent virt-customize, or an image libguestfs
// cannot inspect logs and continues — image import never fails on the CA.
func bakeCACertIntoImage(imagePath, caCertPath string) {
if _, err := os.Stat(caCertPath); err != nil {
slog.Warn("CA bake skipped: deployment CA not found; imported image will not auto-trust the gateway", "ca", caCertPath, "err", err)
return
}
if out, err := caBakeRunner(imagePath, caCertPath); err != nil {
slog.Warn("CA bake skipped: virt-customize could not customize image; imported image will not auto-trust the gateway", "err", err, "output", string(out))
return
}
}
func runimagesRemoveCmd(cmd *cobra.Command, args []string) {
imageID, _ := cmd.Flags().GetString("image-id")
force, _ := cmd.Flags().GetBool("force")
@@ -793,17 +881,8 @@ func runimagesListCmd(cmd *cobra.Command, args []string) {
{"NAME", "DISTRO", "VERSION", "ARCH", "BOOT"},
}
// Sort A .. Z
// 1. Collect keys
keys := make([]string, 0, len(utils.AvailableImages))
for k := range utils.AvailableImages {
keys = append(keys, k)
}
// 2. Sort keys alphabetically (A→Z)
sort.Strings(keys)
// 3. Iterate in sorted order
// Sort A→Z then iterate.
keys := slices.Sorted(maps.Keys(utils.AvailableImages))
for _, k := range keys {
img := utils.AvailableImages[k]
+67
View File
@@ -0,0 +1,67 @@
package cmd
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestCABakeCmd_Args(t *testing.T) {
cmd := caBakeCmd(context.Background(), "/img/disk.raw", "/data/config/ca.pem")
args := cmd.Args
assert.Equal(t, "virt-customize", filepath.Base(args[0]))
assert.Contains(t, args, "-a")
assert.Contains(t, args, "/img/disk.raw")
assert.Contains(t, args, "--upload")
assert.Contains(t, args, "/data/config/ca.pem:/tmp/spinifex-ca.pem")
assert.Contains(t, args, "--run-command")
// --no-selinux-relabel stops virt-customize forcing a first-boot autorelabel,
// whose relabel+reboot corrupts XFS roots on RHEL/Rocky.
assert.Contains(t, args, "--no-selinux-relabel")
// The run-command covers Debian/Ubuntu (update-ca-certificates), RHEL/Rocky
// (update-ca-trust), and the stock-Alpine fallback that appends to the static
// ca-certificates bundle when no updater is present. The RHEL branch relabels
// only the touched trust paths so the cert is labelled without a full relabel.
joined := strings.Join(args, " ")
assert.Contains(t, joined, "update-ca-certificates")
assert.Contains(t, joined, "update-ca-trust")
assert.Contains(t, joined, "restorecon")
assert.Contains(t, joined, "/etc/ssl/certs/ca-certificates.crt")
}
func TestBakeCACertIntoImage_SkipsWhenCAMissing(t *testing.T) {
called := false
orig := caBakeRunner
caBakeRunner = func(string, string) ([]byte, error) { called = true; return nil, nil }
defer func() { caBakeRunner = orig }()
// A missing CA must not invoke virt-customize and must not panic.
bakeCACertIntoImage("/img/disk.raw", filepath.Join(t.TempDir(), "absent-ca.pem"))
assert.False(t, called, "runner must not be called when the CA is absent")
}
func TestBakeCACertIntoImage_ContinuesOnRunnerError(t *testing.T) {
caPath := filepath.Join(t.TempDir(), "ca.pem")
require.NoError(t, os.WriteFile(caPath, []byte("ca"), 0600))
called := false
orig := caBakeRunner
caBakeRunner = func(string, string) ([]byte, error) {
called = true
return []byte("libguestfs: cannot inspect image"), errors.New("exit status 1")
}
defer func() { caBakeRunner = orig }()
// An image libguestfs cannot inspect must skip-and-continue, never panic/fail.
bakeCACertIntoImage("/img/disk.raw", caPath)
assert.True(t, called, "runner must be invoked when the CA is present")
}
+11 -3
View File
@@ -9,7 +9,6 @@ import (
"strings"
"testing"
s3 "github.com/mulgadc/predastore/s3"
"github.com/mulgadc/spinifex/spinifex/admin"
"github.com/mulgadc/spinifex/spinifex/formation"
"github.com/mulgadc/spinifex/spinifex/utils"
@@ -503,7 +502,7 @@ func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
AccessKey: "AK", SecretKey: "SK", NatsToken: "tok", CompactionIntervalSeconds: interval,
})
assert.Contains(t, single, "[compaction]")
var singleCfg s3.Config
var singleCfg compactionConfig
require.NoError(t, toml.Unmarshal([]byte(single), &singleCfg))
assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
@@ -511,11 +510,20 @@ func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
)
require.NoError(t, err)
var multiCfg s3.Config
var multiCfg compactionConfig
require.NoError(t, toml.Unmarshal([]byte(multi), &multiCfg))
assert.Equal(t, interval, multiCfg.Compaction.IntervalSeconds)
}
// compactionConfig parses just the rendered [compaction] block. Predastore's
// s3.Config dropped its Compaction field, so this asserts the spinifex template
// still emits a valid, parseable block carrying the interval.
type compactionConfig struct {
Compaction struct {
IntervalSeconds int `toml:"interval_seconds"`
} `toml:"compaction"`
}
// The init/join flag must register with an unset (0) default and flow into the
// rendered config via ConfigSettings.
func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
+87
View File
@@ -161,6 +161,93 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
}
// runNodeDrainLocal runs the GATE and DRAIN shutdown phases against the local
// node only, leaving STORAGE/PERSIST/INFRA to systemd's ordered unit teardown.
// It is the ExecStop of spinifex-shutdown.service: guests are powered down and
// their volumes unmounted while every service is still up, so the subsequent
// service stops never tear storage out from under a running guest.
func runNodeDrainLocal(cmd *cobra.Command, args []string) {
local, _ := cmd.Flags().GetBool("local")
timeout, _ := cmd.Flags().GetDuration("timeout")
if !local {
fmt.Fprintln(os.Stderr, "Error: node drain currently supports only --local")
os.Exit(1)
}
cfg, nc, err := loadConfigAndConnect()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
defer nc.Close()
node := cfg.Node
fmt.Printf("Draining local node %q (gate -> drain), timeout %s/phase\n", node, timeout)
for _, phase := range []string{"gate", "drain"} {
topic := "spinifex.cluster.shutdown." + phase
req := daemon.ShutdownRequest{Phase: phase, Timeout: int(timeout.Seconds())}
reqData, err := json.Marshal(req)
if err != nil {
fmt.Fprintf(os.Stderr, "Error marshaling request: %v\n", err)
os.Exit(1)
}
ack, err := collectLocalShutdownACK(nc, topic, reqData, node, timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "[%s] %v\n", strings.ToUpper(phase), err)
os.Exit(1)
}
if ack.Error != "" {
fmt.Fprintf(os.Stderr, "[%s] %s: ERROR - %s\n", strings.ToUpper(phase), ack.Node, ack.Error)
os.Exit(1)
}
if len(ack.Stopped) > 0 {
fmt.Printf("[%s] %s: stopped %s\n", strings.ToUpper(phase), ack.Node, strings.Join(ack.Stopped, ", "))
} else {
fmt.Printf("[%s] %s: OK\n", strings.ToUpper(phase), ack.Node)
}
}
fmt.Printf("Local node %q drained; systemd may now stop storage services\n", node)
}
// collectLocalShutdownACK publishes a shutdown-phase request and returns the ACK
// from the local node, ignoring ACKs from any other node. Returns an error if
// the local node does not respond within timeout.
func collectLocalShutdownACK(nc *nats.Conn, topic string, reqData []byte, node string, timeout time.Duration) (daemon.ShutdownACK, error) {
inbox := nats.NewInbox()
sub, err := nc.SubscribeSync(inbox)
if err != nil {
return daemon.ShutdownACK{}, fmt.Errorf("failed to subscribe to inbox: %w", err)
}
defer sub.Unsubscribe()
if err := nc.PublishRequest(topic, inbox, reqData); err != nil {
return daemon.ShutdownACK{}, fmt.Errorf("failed to publish request: %w", err)
}
nc.Flush()
deadline := time.Now().Add(timeout)
for {
remaining := time.Until(deadline)
if remaining <= 0 {
return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
}
msg, err := sub.NextMsg(remaining)
if err != nil {
return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
}
var ack daemon.ShutdownACK
if err := json.Unmarshal(msg.Data, &ack); err != nil {
continue
}
if ack.Node == node {
return ack, nil
}
}
}
// runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
// it currently holds, returning them to the upstream DHCP server. Intended for
// the teardown path: an env reset otherwise strands held leases until TTL
+10 -6
View File
@@ -19,8 +19,10 @@ package cmd
import (
"encoding/json"
"fmt"
"maps"
"os"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
@@ -206,11 +208,7 @@ func runGetNodes(cmd *cobra.Command, args []string) {
for name := range respondedNodes {
nodeSet[name] = struct{}{}
}
nodeNames := make([]string, 0, len(nodeSet))
for name := range nodeSet {
nodeNames = append(nodeNames, name)
}
sort.Strings(nodeNames)
nodeNames := slices.Sorted(maps.Keys(nodeSet))
for _, name := range nodeNames {
nodeCfg := cfg.Nodes[name]
@@ -291,7 +289,7 @@ func runGetVMs(cmd *cobra.Command, args []string) {
})
tableData := pterm.TableData{
{"INSTANCE", "STATUS", "TYPE", "VCPU", "MEM", "NODE", "IP", "AGE"},
{"INSTANCE", "STATUS", "HEALTH", "CRASHES", "TYPE", "VCPU", "MEM", "NODE", "IP", "AGE"},
}
for _, v := range allVMs {
@@ -299,9 +297,15 @@ func runGetVMs(cmd *cobra.Command, args []string) {
if v.LaunchTime > 0 {
age = formatUptime(time.Now().Unix() - v.LaunchTime)
}
health := v.Health
if health == "" {
health = "-"
}
tableData = append(tableData, []string{
v.InstanceID,
v.Status,
health,
strconv.Itoa(v.CrashCount),
v.InstanceType,
strconv.Itoa(v.VCPU),
formatMemGB(v.MemoryGB),
+12 -3
View File
@@ -1,5 +1,5 @@
# AWS Gateway configuration file in TOML format.
version = "2"
version = "3"
region = "{{.Region}}"
# Enable verbose debugging
@@ -8,8 +8,8 @@ debug = false
# API request throttling (per-account, per-action token bucket)
[ratelimit]
enabled = false
rate = 20 # Default sustained requests per second per account per action
burst = 100 # Default maximum burst capacity
rate = 20 # Default sustained requests per second per account per action
burst = 100 # Default maximum burst capacity
[ratelimit.action.RunInstances]
rate = 2
@@ -26,3 +26,12 @@ burst = 40
[ratelimit.action.StopInstances]
rate = 2
burst = 40
# Per-account service quotas (default off)
[quota]
enabled = false
vcpus = 8
vpcs = 8
subnets = 16
eips = 4
volumes_gib = 100
+4 -11
View File
@@ -19,8 +19,9 @@ package cmd
import (
"encoding/json"
"fmt"
"maps"
"os"
"sort"
"slices"
"strconv"
"time"
@@ -81,11 +82,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
for name := range respondedNodes {
nodeSet[name] = struct{}{}
}
nodeNames := make([]string, 0, len(nodeSet))
for name := range nodeSet {
nodeNames = append(nodeNames, name)
}
sort.Strings(nodeNames)
nodeNames := slices.Sorted(maps.Keys(nodeSet))
nodeTable := pterm.TableData{
{"NAME", "CPU (used/total)", "MEM (used/total)", "GPU (used/total)", "VMs"},
@@ -141,11 +138,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
fmt.Println()
capNames := make([]string, 0, len(capacityMap))
for name := range capacityMap {
capNames = append(capNames, name)
}
sort.Strings(capNames)
capNames := slices.Sorted(maps.Keys(capacityMap))
capTable := pterm.TableData{
{"INSTANCE TYPE", "AVAILABLE", "VCPU", "MEMORY"},
-183
View File
@@ -1,183 +0,0 @@
# Spinifex Systemd Service
This directory contains the systemd service configuration for running Spinifex services on Ubuntu and Debian systems.
## Files
- `spinifex.service` - The systemd service file that manages all Spinifex services
- `install-spinifex-service.sh` - Installation script to set up the service
- `README.md` - This documentation file
## Installation
### Prerequisites
1. Ensure Spinifex is installed in `/opt/spinifex/`
2. Ensure the `spx` binary is available at `/opt/spinifex/bin/spx`
3. Run the installation script as root
### Setup Configuration
```
/opt/spinifex/bin/spx admin init
```
This will init the Spinifex environment and create the default configuration files in the home directory, e.g `~/spinifex/`.
```
/home/username/spinifex:
amis config images logs nats predastore state viperblock volumes
/home/username/spinifex/amis:
/home/username/spinifex/config:
awsgw spinifex.toml nats predastore server.key server.pem
/home/username/spinifex/config/awsgw:
awsgw.toml
/home/username/spinifex/config/nats:
nats.conf
/home/username/spinifex/config/predastore:
predastore.toml
/home/username/spinifex/images:
/home/username/spinifex/logs:
awsgw.log awsgw.pid spinifex.log spinifex.pid nats.log nats.pid predastore.log predastore.pid viperblock.log viperblock.pid
/home/username/spinifex/nats:
/home/username/spinifex/predastore:
data
/home/username/spinifex/predastore/data:
predastore
/home/ben/spinifex/state:
/home/ben/spinifex/viperblock:
/home/ben/spinifex/volumes:
```
### Quick Installation
```bash
sudo ./config/install-spinifex-service.sh
```
### Manual Installation
If you prefer to install manually:
1. Copy the service file:
```bash
sudo cp config/spinifex.service /etc/systemd/system/
```
2. Reload systemd:
```bash
sudo systemctl daemon-reload
```
3. Enable the service:
```bash
sudo systemctl enable spinifex.service
```
## Service Management
### Start the service
```bash
sudo systemctl start spinifex
```
### Stop the service
```bash
sudo systemctl stop spinifex
```
### Check service status
```bash
sudo systemctl status spinifex
```
### Restart the service
```bash
sudo systemctl restart spinifex
```
### View service logs
```bash
sudo journalctl -u spinifex -f
```
### Disable auto-start
```bash
sudo systemctl disable spinifex
```
## Service Details
The `spinifex.service` file:
- Starts services in the correct order: NATS → Predastore → Viperblock
- Includes proper delays between service starts to ensure dependencies are ready
- Stops services in reverse order during shutdown
- Runs as root (required for system services)
- Includes security hardening settings
- Provides proper logging to systemd journal
- Automatically restarts on failure
## Service Order
The service starts the following components in order:
1. **NATS** - Message broker service
2. **Predastore** - Data storage service (waits 3 seconds after NATS)
3. **Viperblock** - Block storage service (waits 2 seconds after Predastore)
## Troubleshooting
### Check if the service is running
```bash
sudo systemctl is-active spinifex
```
### View recent logs
```bash
sudo journalctl -u spinifex --since "1 hour ago"
```
### Check service configuration
```bash
sudo systemctl cat spinifex
```
### Verify binary exists
```bash
ls -la /opt/spinifex/bin/spx
```
### Test service commands manually
```bash
/opt/spinifex/bin/spx service nats start
/opt/spinifex/bin/spx service predastore start
/opt/spinifex/bin/spx service viperblock start
```
## Customization
If you need to modify the service configuration:
1. Edit the service file: `sudo nano /etc/systemd/system/spinifex.service`
2. Reload systemd: `sudo systemctl daemon-reload`
3. Restart the service: `sudo systemctl restart spinifex`
## Security Notes
- The service runs as root, which is necessary for system-level services
- Security hardening is enabled with `ProtectSystem=strict` and `NoNewPrivileges=true`
- Only specific paths are writable: `/opt/spinifex`, `/var/log`, `/var/run`
-74
View File
@@ -1,74 +0,0 @@
#!/bin/bash
# Spinifex Service Installation Script
# This script installs the spinifex systemd service on Ubuntu/Debian systems
set -e
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Function to print colored output
print_status() {
echo -e "${GREEN}[INFO]${NC} $1"
}
print_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
print_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# Check if running as root
if [[ $EUID -ne 0 ]]; then
print_error "This script must be run as root (use sudo)"
exit 1
fi
# Check if systemd is available
if ! command -v systemctl &> /dev/null; then
print_error "systemd is not available on this system"
exit 1
fi
# Get the directory where this script is located
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SERVICE_FILE="$SCRIPT_DIR/spinifex.service"
# Check if service file exists
if [[ ! -f "$SERVICE_FILE" ]]; then
print_error "Service file not found at $SERVICE_FILE"
exit 1
fi
print_status "Installing Spinifex systemd service..."
# Copy service file to systemd directory
cp "$SERVICE_FILE" /etc/systemd/system/
# Reload systemd to recognize the new service
systemctl daemon-reload
# Enable the service to start on boot
systemctl enable spinifex.service
print_status "Service installed successfully!"
print_status "You can now use the following commands:"
echo " systemctl start spinifex # Start the service"
echo " systemctl stop spinifex # Stop the service"
echo " systemctl status spinifex # Check service status"
echo " systemctl restart spinifex # Restart the service"
echo " journalctl -u spinifex -f # View service logs"
# Check if spinifex binary exists
if [[ ! -f "/opt/spinifex/bin/spinifex" ]]; then
print_warning "Spinifex binary not found at /opt/spinifex/bin/spinifex"
print_warning "Please ensure spinifex is installed in /opt/spinifex before starting the service"
fi
print_status "Installation complete!"
-80
View File
@@ -1,80 +0,0 @@
# Predastore S3-compatible storage configuration file in TOML format.
# MULTI NODE used for testing
version = "1.0"
region = "ap-southeast-2"
# Optionally configure port and host, or specify via daemon CLI options
host = "0.0.0.0"
port = 8443
# Enable verbose debugging
debug = false
# Optionally configure base-path if full directory path not defined for "fs" buckets
# base_path = "/home/predastore/data/"
# Specify reed solomon encoding
[rs]
data = 2
parity = 1
[[db]]
id = 1
host = "127.0.0.1"
port = 6660
#raft_port = 7660 # Optional: Raft consensus port (default: port + 1000)
#advertise_host = "" # Optional: Address to advertise to other Raft nodes
path = "distributed/db/node-1/"
access_key_id = "X"
secret_access_key = "X"
leader = true # Bootstrap leader hint
[[db]]
id = 2
host = "127.0.0.1"
port = 6661
path = "distributed/db/node-2/"
access_key_id = "X"
secret_access_key = "X"
[[db]]
id = 3
host = "127.0.0.1"
port = 6662
path = "distributed/db/node-3/"
access_key_id = "X"
secret_access_key = "X"
[[nodes]]
id = 1
host = "0.0.0.0"
port = 9991
path = "distributed/nodes/node-1/"
[[nodes]]
id = 2
host = "0.0.0.0"
port = 9992
path = "distributed/nodes/node-2/"
[[nodes]]
id = 3
host = "0.0.0.0"
port = 9993
path = "distributed/nodes/node-3/"
[[buckets]]
name = "predastore"
region = "ap-southeast-2"
type = "distributed"
public = false
encryption = ""
[[auth]]
access_key_id = "X"
secret_access_key = "X"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
-49
View File
@@ -1,49 +0,0 @@
[Unit]
Description=Spinifex Services
Documentation=https://github.com/mulgadc/spinifex
After=network.target
Wants=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
User=root
Group=root
WorkingDirectory=/opt/spinifex
Environment=PATH=/opt/spinifex/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Start NATS service
ExecStart=/opt/spinifex/bin/spinifex service nats start
ExecStartPost=/bin/sleep 3
# Start Predastore service
ExecStart=/opt/spinifex/bin/spinifex service predastore start
ExecStartPost=/bin/sleep 2
# Start Viperblock service
ExecStart=/opt/spinifex/bin/spinifex service viperblock start
# Stop services in reverse order
ExecStop=/opt/spinifex/bin/spinifex service viperblock stop
ExecStopPost=/bin/sleep 2
ExecStop=/opt/spinifex/bin/spinifex service predastore stop
ExecStopPost=/bin/sleep 2
ExecStop=/opt/spinifex/bin/spinifex service nats stop
# Restart policy
Restart=on-failure
RestartSec=10
# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/spinifex /var/log /var/run
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=spinifex
[Install]
WantedBy=multi-user.target
+23 -5
View File
@@ -66,6 +66,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
|---------|-------|-------------|
| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. |
| `spx admin cluster drain-dhcp` | `--timeout` (reply-collection window, default 30s) | Asks each vpcd to DHCPRELEASE every external-pool DHCP lease it currently holds, returning them to the upstream DHCP server. Run on teardown before stopping services — an env reset otherwise strands held leases upstream until their TTL expires, eventually exhausting the upstream scope. Best-effort: warns and exits 0 if the cluster is already down. |
| `spx admin node drain --local` | `--local` (drain the local node only, required), `--timeout` (max wait per phase, default 120s) | Runs the GATE and DRAIN phases against the local node only: powers down its guests via QMP and unmounts their volumes (flushing the viperblock WAL) while every service is still up. STORAGE/PERSIST/INFRA are left to systemd's ordered teardown. Wired as the `ExecStop` of `spinifex-shutdown.service`, so `systemctl stop spinifex.target` and host reboot/poweroff drain guests before any storage service stops. |
### Certificate Management
@@ -90,7 +91,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. |
| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. On every import a best-effort `virt-customize` step bakes the deployment CA (`<data-root>/config/ca.pem`) into the image's trust store before the block copy, so in-guest SDK-over-TLS calls to Spinifex endpoints trust the gateway from first boot; an image libguestfs cannot customize is imported as-is (CA-free) and logs a skip. Because the CA is fixed into the image, rotating the cluster CA requires re-importing affected images. |
| `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
@@ -113,7 +114,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Implemented Flags | Missing Flags | Status |
|---------|-------------------|---------------|--------|
| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | **DONE** |
| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only), `--metadata-options` (HttpPutResponseHopLimit; IMDSv2-only enforced — rejects `http-tokens=optional`) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--launch-template`, `--hibernate-options` | **DONE** |
| `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
| `start-instances` | `--instance-ids` | `--dry-run`, `--force` | **DONE** |
| `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | **DONE** |
@@ -123,6 +124,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data`, `--disable-api-termination` | `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | **DONE** |
| `get-console-output` | `--instance-id` | `--latest`, `--dry-run` | **DONE** |
| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, disableApiTermination, instanceInitiatedShutdownBehavior, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | **DONE** |
| `modify-instance-metadata-options` | `--instance-id`, `--http-put-response-hop-limit` (1–64), `--http-tokens` (`required`), `--http-endpoint` (`enabled`), `--http-protocol-ipv6`/`--instance-metadata-tags` (`disabled`) — secure values are no-ops, downgrades return `UnsupportedOperation` | `--dry-run` | **DONE** |
| `describe-instance-credit-specifications` | `--instance-ids` | `--filters`, `--max-results`, `--dry-run` | **DONE** (stub — always returns `standard`) |
| `describe-instance-status` | `--instance-ids`, `--include-all-instances`, `--filters` (availability-zone, instance-state-code, instance-state-name, tag:*) | `--max-results`, `--next-token`, `--dry-run`, event/instance-status/system-status filters | **DONE** (static health) |
| `monitor-instances` | — | `--instance-ids` | **NOT STARTED** |
@@ -137,6 +139,13 @@ the node owning the targeted reservation (targeted-by-id only; `open`
auto-matching is not supported). The combination with `--placement` (group) is
rejected. `describe-instances` then reports the instance's `CapacityReservationId`.
`modify-instance-metadata-options` is the only metadata-options mutator — AWS has
no `get-instance-metadata-options` API, so the per-instance block is read through
`describe-instances`. Only the hop limit is mutable; `--http-tokens optional` and
the other posture-weakening values are refused with `UnsupportedOperation`, since
the platform enforces IMDSv2 permanently. `run-instances --metadata-options` runs
the same validation at launch.
### EC2 — IAM Instance Profile Associations
Stored as `vm.VM.IamInstanceProfileArn` + `IamInstanceProfileAssociationId`
@@ -239,6 +248,11 @@ by downstream services.
| `modify-instance-metadata-defaults` | — | `--http-tokens`, `--http-put-response-hop-limit`, `--http-endpoint`, `--instance-metadata-tags` | **NOT STARTED** |
| `get-instance-metadata-defaults` | — | `--dry-run` | **NOT STARTED** |
The account-level `modify-/get-instance-metadata-defaults` are blocked on an
`aws-sdk-go` bump: the `InstanceMetadataDefaultsResponse` type is absent from the
pinned v1.44.266 (these APIs landed ~March 2024), and the gateway's generic
handler needs the typed input struct to route them.
### EC2 — VPC Core
| Command | Implemented Flags | Missing Flags | Status |
@@ -514,8 +528,10 @@ bypasses policy evaluation entirely.
| `attach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
| `detach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
| `list-attached-role-policies` | `--role-name`, `--path-prefix` | `--max-items`, `--marker` | **DONE** |
| `list-role-policies` | `--role-name` | `--max-items`, `--marker` | **DONE** (gateway stub — inline policies unsupported, always returns empty) |
| `get-role-policy` / `put-role-policy` / `delete-role-policy` | — | inline role policies | **NOT STARTED** |
| `list-role-policies` | `--role-name` | `--max-items`, `--marker` | **DONE** |
| `put-role-policy` | `--role-name`, `--policy-name`, `--policy-document` | — | **DONE** |
| `get-role-policy` | `--role-name`, `--policy-name` | — | **DONE** (document returned as raw JSON, not URL-encoded) |
| `delete-role-policy` | `--role-name`, `--policy-name` | — | **DONE** |
### IAM — Instance Profiles
@@ -582,6 +598,8 @@ Available at `169.254.169.254` from inside every running guest VM, matching AWS.
**IMDSv2-only.** Every read requires a session token. A tokenless (v1-style) `GET` returns `401 Unauthorized` with an empty body. Obtain a token with a `PUT /latest/api/token` carrying `X-aws-ec2-metadata-token-ttl-seconds` (1–21600), then send it back in `X-aws-ec2-metadata-token` on every read.
The EC2 control plane reports this posture faithfully: `describe-instances` returns `MetadataOptions.HttpTokens=required`, and `run-instances`/`modify-instance-metadata-options` reject `--http-tokens optional` (and `--http-endpoint disabled`) with `UnsupportedOperation` — exactly as AWS does under account-level IMDSv2 enforcement.
```bash
# Inside the guest VM:
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
@@ -628,7 +646,7 @@ curl -i http://169.254.169.254/latest/meta-data/instance-id
| `/latest/dynamic/instance-identity` | GET | Lists `document` (signed forms listed when the signing key lands) | **DONE** |
| `/latest/dynamic/instance-identity/document` | GET | Unsigned identity document from resolved ENI + instance facts | **DONE** |
| `/latest/dynamic/instance-identity/{signature,pkcs7,rsa2048}` | GET | Signed forms; need a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
| `/latest/meta-data/network/interfaces/macs/<mac>/...` | GET | Multi-ENI rendering (subnet-id, vpc-id, ipv4s, security-group-ids, …) | **NOT STARTED** (404) |
| `/latest/meta-data/network/interfaces/macs/<mac>/...` | GET | Primary ENI subtree: `mac`, `device-number`, `interface-id`, `owner-id`, `subnet-id`, `vpc-id`, `local-ipv4s`, `local-hostname`, `security-group-ids`, `security-groups`, `subnet-ipv4-cidr-block`, `vpc-ipv4-cidr-block(s)`, and `public-ipv4s`/`public-hostname` when an EIP is attached | **DONE** (single-NIC; multi-ENI deferred) |
| `/latest/meta-data/tags/instance/<key>` | GET | Instance-tag metadata; gated on `InstanceMetadataTags` enablement | **NOT STARTED** (404) |
| `/latest/meta-data/block-device-mapping/...` | GET | `ami`/`root`/`ebsN`/`ephemeralN` device map | **NOT STARTED** (404) |
| `/latest/meta-data/placement/{group-name,partition-number,availability-zone-id,host-id}` | GET | Placement extras beyond `availability-zone`/`region` | **NOT STARTED** (404) |
+1 -2
View File
@@ -123,7 +123,7 @@ Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle reques
When a daemon handles `RunInstances`:
1. **Resource Check**: Validates CPU/memory availability via `ResourceManager`
2. **Volume Generation**: Creates boot, cloud-init, and EFI volumes via Viperblock
2. **Volume Generation**: Creates boot and EFI volumes via Viperblock
3. **Volume Mount**: Sends `ebs.mount` request to Viperblock, receives NBD URI
4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks
5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management
@@ -217,7 +217,6 @@ type EBSRequest struct {
VolType string // gp2, io1, etc.
Boot bool // Boot volume flag
EFI bool // EFI boot volume
CloudInit bool // Cloud-init volume
NBDURI string // NBD URI returned from mount
}
```
+1 -1
View File
@@ -137,7 +137,7 @@ In development mode, find the QEMU port forward and connect via localhost:
```bash
ps auxw | grep $INSTANCE_ID
ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p <port>
ssh -i ~/.ssh/spinifex-key ubuntu@127.0.0.1 -p <port>
```
For production SSH via public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).
+270
View File
@@ -0,0 +1,270 @@
---
title: "ECS (Elastic Container Service)"
description: "Run AWS-compatible ECS workloads on Spinifex — create a cluster, register a task definition, boot container instances from the ECS node image, run tasks, and front a long-running service with an Application Load Balancer, all via the AWS CLI, the Spinifex console, or Terraform."
category: "Containers"
tags:
- ecs
- containers
- tasks
- alb
- iam
sections:
- overview
- prerequisites
- instructions
- troubleshooting
resources:
- title: "ECS Quickstart Workbook"
url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/ecs-quickstart"
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
- title: "AWS ECS Task IAM Roles"
url: "https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html"
---
# ECS (Elastic Container Service)
> Spinifex's ECS service gives you an AWS-compatible container orchestrator: you register task definitions and run them — one-off as tasks, or kept-running as a service behind a load balancer — on container instances you launch from the Spinifex ECS node image. Standard tools — `aws ecs`, the Terraform AWS provider — work unchanged.
## Overview
ECS on Spinifex follows the AWS **EC2 launch type**: you supply the compute. A cluster is a logical grouping; the capacity behind it is **container instances** — ordinary EC2 instances booted from Spinifex's `spinifex-ecs-node` image, each running the Spinifex ECS agent. The agent registers the instance with the cluster, reports its CPU/memory, and runs the containers the scheduler places on it.
There is **no ECS-specific launch API** — this is AWS-faithful. You add capacity by launching EC2 instances from the ECS node image (with the `ecsInstanceRole` instance profile), exactly as on AWS. The Spinifex console wraps this in a one-click **Provision capacity** action, but underneath it is just `RunInstances`.
A **task definition** describes one or more containers (image, CPU/memory, ports, environment, and an optional task IAM role). A **task** is a running instantiation of a task definition; the scheduler bin-packs tasks onto instances with free capacity. A **service** keeps a desired number of tasks running, replaces failed ones, and — when configured — registers each task's IP with an Application Load Balancer target group.
<p align="center">
<img src="../../../.github/assets/diagrams/ecs-topology.svg" alt="ECS on Spinifex — an Application Load Balancer fronts awsvpc tasks running on container instances booted from the spinifex-ecs-node AMI; the ECS agent registers with the Spinifex gateway over TLS and SigV4 and serves task-role credentials to containers" width="900">
</p>
**What you'll create:**
| Resource | Purpose |
|---|---|
| VPC + subnets | The network the cluster and tasks run in |
| Internet Gateway | Egress so instances can pull container images |
| `ecsInstanceRole` | Instance profile letting the agent reach the control plane |
| ECS cluster | The logical grouping tasks and instances join |
| Task definition | The container spec (image, CPU/memory, ports, task role) |
| Container instance(s) | EC2 VMs from the ECS node image that run tasks |
| Service + ALB target group | Keeps N tasks running and load-balances them |
**Spinifex specifics**
- **EC2 launch type only.** There is no Fargate-equivalent serverless capacity; you run and pay for the container instances. `RequiresCompatibilities` of `FARGATE` is not honoured.
- **`awsvpc` network mode.** Each task gets its own ENI and private IP in your subnet. Target groups must use `target_type = "ip"`; the service's `network_configuration` selects the subnets and security groups.
- **The container-instance image is fixed.** Instances must boot Spinifex's `spinifex-ecs-node` image (resolve it by the `spinifex:managed-by=ecs` tag) and carry the `ecsInstanceRole` instance profile.
- **Task IAM roles via the credential endpoint.** A task with a `taskRoleArn` gets `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` injected; the agent serves short-lived credentials for that role at `169.254.170.2`, AWS-faithful — your container's SDK picks them up with no static keys.
### Limitations
ECS v1 is deliberately minimal. Keep these in mind:
- **No service discovery.** `serviceRegistries` / Cloud Map integration is not implemented — reach a service through its load balancer, not a DNS name.
- **`awslogs` log driver is discarded.** Container stdout/stderr is not shipped to CloudWatch Logs; the log configuration is accepted but dropped.
- **No rolling deployments.** The reconciler keeps the desired count running but ignores `deploymentConfiguration` (`minimumHealthyPercent` / `maximumPercent`); an `UpdateService` to a new revision replaces tasks without a health-gated rollout or circuit-breaker.
- **No capacity providers or managed scaling.** Capacity is the static total of your registered instances; there is no scale-out/in or ASG binding — you manage the instance count.
- **No distinct execution role.** ECR pulls use the instance role directly; `executionRoleArn` is not honoured separately.
- **`secrets[]` are silently dropped**, and tags set via `TagResource` are not persisted. Health-check settings beyond the target group default are not forwarded.
## Prerequisites
> [!IMPORTANT]
> **Prerequisite — `spinifex-ecs-node` image required.**
>
> Container instances **must** boot Spinifex's **`spinifex-ecs-node`** image — it carries the ECS agent. Import it before provisioning capacity (during `spx admin init` or via the image catalogue); the console's provision-capacity action and the Terraform workbook both resolve it by tag.
>
> **Verify before continuing:**
>
> ```bash
> aws ec2 describe-images \
> --filters 'Name=tag:spinifex:managed-by,Values=ecs' \
> --query 'Images[].[ImageId,Name]' --output text
> ```
>
> No rows means the image is not imported — register it before continuing.
The Terraform workbook builds all of this for you. For the CLI or console paths, have it in place first.
- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)).
- **The `spinifex-ecs-node` image** imported. Confirm it resolves:
```bash
aws ec2 describe-images \
--filters 'Name=tag:spinifex:managed-by,Values=ecs' \
--query 'Images[].[ImageId,Name]' --output text
```
- **A VPC with at least one subnet** and an **Internet Gateway** with a default route (`0.0.0.0/0 → IGW`), so instances can pull container images.
- **The `ecsInstanceRole` instance profile.** It is account-global: a role trusted by `ec2.amazonaws.com` with an `ecs:*` policy, exposed through an instance profile of the same name. The console's provision-capacity action creates it on first use; the Terraform workbook creates it unless you opt out.
- **An optional task IAM role** trusted by `ecs-tasks.amazonaws.com`, if your containers call AWS APIs.
## Instructions
The same workload can be created three ways. Pick your tool — each path reaches the same running service.
:::tabs
@tab AWS CLI
### 1. Create the cluster
```bash
export AWS_PROFILE=spinifex
aws ecs create-cluster --cluster-name demo
```
### 2. Register a task definition
`awsvpc` network mode, EC2 launch type, one nginx container on port 80:
```bash
aws ecs register-task-definition \
--family web \
--network-mode awsvpc \
--requires-compatibilities EC2 \
--cpu 256 --memory 512 \
--container-definitions '[{
"name":"web",
"image":"docker.io/library/nginx:1.27-alpine",
"portMappings":[{"containerPort":80,"protocol":"tcp"}],
"essential":true
}]'
```
### 3. Add capacity
Launch one or more EC2 instances from the ECS node image with the `ecsInstanceRole`
instance profile and a cloud-init that points the agent at the cluster. The
[ECS Quickstart workbook](https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/ecs-quickstart)
produces this user-data for you; the console **Provision capacity** action is the
one-click equivalent. Once an instance boots and its agent registers, it appears here:
```bash
aws ecs list-container-instances --cluster demo
```
### 4. Run a task or create a service
Run a one-off task:
```bash
aws ecs run-task \
--cluster demo --task-definition web --count 1 \
--network-configuration 'awsvpcConfiguration={subnets=[subnet-aaaa]}'
```
Or keep it running behind an ALB target group (`target_type = ip`):
```bash
aws ecs create-service \
--cluster demo --service-name web --task-definition web \
--desired-count 2 \
--network-configuration 'awsvpcConfiguration={subnets=[subnet-aaaa]}' \
--load-balancers 'targetGroupArn=<tg-arn>,containerName=web,containerPort=80'
aws ecs describe-services --cluster demo --services web \
--query 'services[0].[runningCount,desiredCount]'
```
@tab Spinifex UI
The Spinifex console drives the same workflow. From the left navigation open **ECS → Clusters**.
### 1. Create the cluster
Click **Create Cluster**, give it a name, and submit. It appears in the **Clusters** list.
### 2. Provision capacity
Open the cluster and use **Provision capacity** on the **Infrastructure** (container instances) tab. The console launches instances from the `spinifex-ecs-node` image, attaches the `ecsInstanceRole` instance profile (creating it if needed), and injects the gateway URL and CA into the instance's cloud-init — so the agent registers without any manual key handling. The instances appear under **Infrastructure** as they register.
### 3. Register a task definition and run it
- The **Task definitions** view lists families and revisions; register a new revision with the container image, CPU/memory, and ports.
- From a task definition, **Run task** for a one-off, or create a **Service** to keep a desired count running.
- A cluster's **Services** and **Tasks** tabs show what is running; open a task or service for its detail page (containers, networking, and public/private endpoints).
### 4. Attach a load balancer
When creating a service, select an ALB target group (`target_type = ip`) so each task's ENI is registered and traffic is balanced across tasks.
@tab Terraform
The [ECS Quickstart workbook](https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/ecs-quickstart) builds the whole stack in one `apply`: VPC, IAM, cluster, task definition, container instances, and an ALB-fronted service. It is the Terraform-native equivalent of the console's provision-capacity action.
```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/ecs-quickstart
```
The core resources point the AWS provider at Spinifex's `ecs`, `ec2`, `iam`, `sts`, and `elasticloadbalancingv2` endpoints, then create the cluster, an `awsvpc` task definition with a task role, and a service wired to a target group:
```hcl
resource "aws_ecs_cluster" "main" {
name = "ecs-quickstart"
}
resource "aws_ecs_task_definition" "web" {
family = "ecs-quickstart-web"
network_mode = "awsvpc"
requires_compatibilities = ["EC2"]
cpu = "256"
memory = "512"
task_role_arn = aws_iam_role.task.arn
container_definitions = jsonencode([{
name = "web"
image = "docker.io/library/nginx:1.27-alpine"
essential = true
portMappings = [{ containerPort = 80, protocol = "tcp" }]
}])
}
resource "aws_ecs_service" "web" {
name = "ecs-quickstart-web"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.web.arn
desired_count = 1
network_configuration {
subnets = [aws_subnet.a.id, aws_subnet.b.id]
security_groups = [aws_security_group.node.id]
}
load_balancer {
target_group_arn = aws_lb_target_group.web.arn
container_name = "web"
container_port = 80
}
}
```
Because a container instance reaches the control plane over the gateway (TLS + SigV4) rather than a managed AWS endpoint, the workbook makes the two things the console injects explicit: a **LAN-reachable** `gateway_url` (the host's bridge IP, not `127.0.0.1`) and the gateway CA, both baked into the instance's cloud-init. Apply it:
```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply -var 'gateway_url=https://<host-lan-ip>:9999'
```
`ecsInstanceRole` is account-global — if it already exists (from the console), pass `-var 'create_instance_role=false'`. See the workbook README for the full variable list and teardown.
:::
## Troubleshooting
**No container instances appear after launching them.** The agent registers over the gateway, not a managed endpoint, so check its cloud-init injected a correct **LAN-reachable** gateway URL (not `127.0.0.1` — a guest VM cannot reach the host loopback) and the gateway CA. `cloud-init write_files` runs once per instance, so fixing the user-data needs an instance **replacement**, not an in-place modify. Confirm the instance carries the `ecsInstanceRole` instance profile.
**Tasks stay `PENDING`.** No instance has free capacity for the task's CPU/memory reservation. Add capacity, or lower the task definition's reservations. `aws ecs list-container-instances` should show at least one `ACTIVE` instance.
**Service `runningCount` below `desiredCount`.** Either there is not enough capacity (see above), or tasks are failing to start — check the task's containers can pull their image (instances need an egress route) and that the image reference is valid.
**Load balancer target unhealthy / app unreachable.** The target group must be `target_type = ip` (tasks register their ENI IP, not an instance ID). The ALB DNS name ends in `.elb.spinifex.local` and does not resolve from outside; fetch its public IP with `aws elbv2 describe-load-balancers` and curl that. Note health-check settings beyond the target group default are not forwarded.
**Container cannot assume its task role.** Confirm the task definition sets `taskRoleArn` and the role is trusted by `ecs-tasks.amazonaws.com`. The agent injects `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`; a container that overrides this variable, or an SDK too old to honour it, will not pick up the credentials.
**Expected a feature that is missing.** ECS v1 omits service discovery, `awslogs` shipping, rolling deployments, capacity providers, and a distinct execution role. See the Limitations in the Overview.
+19
View File
@@ -36,6 +36,10 @@ An EKS cluster on Spinifex has two parts: a **control plane** (the managed Kuber
Authentication uses **EKS access entries** (the API authentication mode) — IAM principals are mapped to Kubernetes RBAC groups, and `kubectl` authenticates with `aws eks get-token` exactly as it does on AWS.
<p align="center">
<img src="../../../.github/assets/diagrams/eks-topology.svg" alt="EKS on Spinifex — a managed control plane (public API NLB plus apiserver VM) fronts a managed node group of eks-node worker VMs running pods across two subnets, pulling images from ECR" width="900">
</p>
**What you'll create:**
| Resource | Purpose |
@@ -57,6 +61,21 @@ Authentication uses **EKS access entries** (the API authentication mode) — IAM
## Prerequisites
> [!IMPORTANT]
> **Prerequisite — `eks-node` image required.**
>
> Node groups always boot Spinifex's **`eks-node`** image, and it **must** be registered before you create a cluster — the console blocks cluster creation until it is present. Import it during `spx admin init` (or via the image catalogue) ahead of time.
>
> **Verify before continuing:**
>
> ```bash
> aws ec2 describe-images \
> --filters 'Name=tag:spinifex:managed-by,Values=eks' \
> --query 'Images[].[ImageId,Name]' --output text
> ```
>
> No rows means the image is not imported — register it before continuing.
Before creating a cluster you need the supporting infrastructure in place. The Terraform workbooks build all of this for you; if you are using the CLI or console, create it first.
- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)) and `kubectl` installed.
@@ -243,7 +243,7 @@ SSH directly to the instance's public IP:
PUBLIC_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
ssh -i ~/.ssh/spinifex-key ec2-user@$PUBLIC_IP
ssh -i ~/.ssh/spinifex-key ubuntu@$PUBLIC_IP
```
> **Note:** cloud-init takes 30-60 seconds to configure the instance. If SSH is refused, wait and retry.
@@ -271,7 +271,7 @@ MIG is enabled inside each guest VM, not on the host. SSH into each VM and enabl
```bash
# Repeat on each VM IP
ssh -i ~/.ssh/spinifex-key ec2-user@<vm-ip>
ssh -i ~/.ssh/spinifex-key ubuntu@<vm-ip>
sudo nvidia-smi -mig 1
nvidia-smi | grep "MIG M."
+10
View File
@@ -300,6 +300,16 @@ suites:
# spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
# client binary or registry-host resolution is absent.
e2e-ecs:
path: tests/e2e/ecs
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# ECS data plane end-to-end: cluster + a real container instance launched
# from the spinifex-ecs-node AMI (boots, registers over the gateway, runs
# tasks through containerd), standalone tasks in awsvpc/bridge/host modes,
# and an awsvpc service fronted by an ELBv2 target group. Boots a guest VM,
# so vpcd (OVN + awsvpc ENI), viperblockd (root volume), predastore (AMI)
# are all in the path.
e2e-ddil:
path: tests/e2e/ddil
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
@@ -93,13 +93,13 @@ tofu apply
SSH into the bastion:
```bash
ssh -i bastion-demo.pem ec2-user@<bastion_public_ip>
ssh -i bastion-demo.pem ubuntu@<bastion_public_ip>
```
From the bastion, SSH to the private instance. The key is pre-installed at `~/.ssh/bastion-demo.pem` via cloud-init:
```bash
ssh -i ~/.ssh/bastion-demo.pem ec2-user@<private_ip>
ssh -i ~/.ssh/bastion-demo.pem ubuntu@<private_ip>
```
### Step 4. Verify Isolation
@@ -126,8 +126,8 @@ tofu destroy
The private instance has no public IP — it is only reachable from the bastion. SSH to the bastion first, then use the pre-installed key to hop:
```bash
ssh -i bastion-demo.pem ec2-user@<bastion_ip>
ssh -i ~/.ssh/bastion-demo.pem ec2-user@<private_ip>
ssh -i bastion-demo.pem ubuntu@<bastion_ip>
ssh -i ~/.ssh/bastion-demo.pem ubuntu@<private_ip>
```
If the key is missing on the bastion, check that cloud-init completed successfully:
@@ -18,11 +18,11 @@
#
# After apply:
# # SSH to the bastion
# ssh -i bastion-demo.pem ec2-user@<bastion_ip>
# ssh -i bastion-demo.pem ubuntu@<bastion_ip>
#
# # From the bastion, SSH to the private instance
# # (the key is pre-installed at ~/.ssh/bastion-demo.pem via cloud-init)
# ssh -i ~/.ssh/bastion-demo.pem ec2-user@<private_ip>
# ssh -i ~/.ssh/bastion-demo.pem ubuntu@<private_ip>
terraform {
required_version = ">= 1.6.0"
@@ -293,12 +293,12 @@ resource "aws_instance" "bastion" {
user_data_base64 = base64encode(<<-USERDATA
#!/bin/bash
set -euo pipefail
mkdir -p /home/ec2-user/.ssh
cat > /home/ec2-user/.ssh/bastion-demo.pem <<'KEY'
mkdir -p /home/ubuntu/.ssh
cat > /home/ubuntu/.ssh/bastion-demo.pem <<'KEY'
${tls_private_key.bastion.private_key_openssh}
KEY
chmod 600 /home/ec2-user/.ssh/bastion-demo.pem
chown -R ec2-user:ec2-user /home/ec2-user/.ssh
chmod 600 /home/ubuntu/.ssh/bastion-demo.pem
chown -R ubuntu:ubuntu /home/ubuntu/.ssh
USERDATA
)
@@ -342,10 +342,10 @@ output "private_instance_ip" {
output "ssh_to_bastion" {
description = "SSH to the bastion host"
value = "ssh -i bastion-demo.pem ec2-user@${aws_instance.bastion.public_ip}"
value = "ssh -i bastion-demo.pem ubuntu@${aws_instance.bastion.public_ip}"
}
output "ssh_to_private_from_bastion" {
description = "From the bastion, SSH to the private instance (key is pre-installed via cloud-init)"
value = "ssh -i ~/.ssh/bastion-demo.pem ec2-user@${aws_instance.private.private_ip}"
value = "ssh -i ~/.ssh/bastion-demo.pem ubuntu@${aws_instance.private.private_ip}"
}
@@ -0,0 +1,132 @@
---
title: "ECS Quickstart"
description: "Stand up a complete AWS-compatible ECS stack on Spinifex with Terraform — a VPC, IAM roles, a cluster, a task definition with a task role, container instances launched from the ECS node image, and an awsvpc service fronted by an Application Load Balancer."
category: "Terraform Workbooks"
tags:
- terraform
- ecs
- containers
- alb
- iam
- vpc
- workbook
sections:
- overview
- prerequisites
- instructions
- troubleshooting
resources:
- title: "ECS Service Documentation"
url: "/docs/ecs"
- title: "Terraform AWS Provider"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "OpenTofu"
url: "https://opentofu.org/"
---
# Terraform: ECS Quickstart
> The smallest end-to-end ECS example on Spinifex: a VPC, the IAM roles ECS needs, a cluster, a task definition, container instances booted from the ECS node image, and an `awsvpc` service load-balanced behind an Application Load Balancer.
## Overview
This workbook is the Terraform-native equivalent of the console's **Provision capacity** action. It provisions a full ECS stack in one `apply`: a VPC with two public subnets, the `ecsInstanceRole` instance profile, a task IAM role, an ECS cluster, an `awsvpc` task definition running nginx, one or more container instances launched from the `spinifex-ecs-node` AMI, and an Application Load Balancer with a target group the service registers into.
Because a Spinifex container instance reaches the control plane over the **gateway** (TLS + SigV4) rather than a managed AWS endpoint, the workbook makes the two things the console injects for you explicit: a **LAN-reachable** gateway URL and the gateway CA, both baked into the instance's cloud-init user-data. The agent draws its credentials from IMDS via the `ecsInstanceRole` instance profile, so no static keys are written.
## Prerequisites
- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)) and OpenTofu (or Terraform) installed.
- **The `spinifex-ecs-node` AMI imported** — resolved here by the `tag:spinifex:managed-by=ecs` filter:
```bash
aws ec2 describe-images \
--filters 'Name=tag:spinifex:managed-by,Values=ecs' \
--query 'Images[].[ImageId,Name]' --output text
```
- **A LAN-reachable gateway URL** — the host's WAN/bridge IP, not `127.0.0.1` (a guest VM cannot reach the host loopback). For example `https://192.168.1.33:9999`.
- **The gateway CA PEM** readable at `gateway_ca_cert_path` (default `/etc/spinifex/ca.pem`).
## Instructions
### 1. Fetch the workbook
```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/ecs-quickstart
```
### 2. Apply
```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply -var 'gateway_url=https://<host-lan-ip>:9999'
```
`ecsInstanceRole` is account-global. If you have already used the console's provision-capacity action it exists already, so skip re-creating it:
```bash
tofu apply \
-var 'gateway_url=https://<host-lan-ip>:9999' \
-var 'create_instance_role=false'
```
### 3. Verify
Container instances take ~30-60s to boot and register:
```bash
aws ecs list-container-instances --cluster ecs-quickstart
aws ecs describe-services --cluster ecs-quickstart --services ecs-quickstart-web \
--query 'services[0].[runningCount,desiredCount]'
```
The ALB DNS name ends in `.elb.spinifex.local` and does not resolve from your host. Fetch its public IP and curl it:
```bash
aws elbv2 describe-load-balancers --names ecs-quickstart-alb \
--query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
--output text
curl http://<alb_public_ip>
```
### 4. Variables
| Variable | Default | Purpose |
|---|---|---|
| `region` | `ap-southeast-2` | AWS region. |
| `cluster_name` | `ecs-quickstart` | Cluster + resource name prefix. |
| `instance_type` | `t3.small` | Container instance type. |
| `container_count` | `1` | Container instances + service desired count. |
| `task_image` | `docker.io/library/nginx:1.27-alpine` | Image the task runs. |
| `spinifex_endpoint` | `https://127.0.0.1:9999` | Gateway as seen from the host running Terraform. |
| `gateway_url` | _(required)_ | Gateway as seen from a guest VM (LAN-reachable). |
| `gateway_ca_cert_path` | `/etc/spinifex/ca.pem` | Host-readable gateway CA PEM. |
| `create_instance_role` | `true` | Create `ecsInstanceRole`; set `false` if it already exists. |
### 5. Teardown
```bash
tofu destroy -var 'gateway_url=https://<host-lan-ip>:9999'
```
`DeleteCluster` cascades through the service, its tasks, and the container instance registrations, so the destroy round-trips cleanly.
## Troubleshooting
**Container instances never register.** The agent registers over the gateway, not a managed endpoint. Confirm `gateway_url` is the host's **LAN-reachable** IP (not `127.0.0.1`) and that `gateway_ca_cert_path` points at the real gateway CA. Because `cloud-init write_files` runs once per instance, a corrected user-data needs an instance **replacement** — `tofu apply -replace='aws_instance.node[0]'` — not an in-place modify.
**`apply` fails resolving the AMI.** The `spinifex-ecs-node` image is not imported, or its `spinifex:managed-by=ecs` tag is missing. Re-run the `describe-images` check in Prerequisites.
**`MissingParameter` on the instance.** The gateway's `RunInstances` requires a `KeyName`; the workbook generates a key pair for you, so this only appears if you have stripped that out.
**Service `runningCount` stays below `desiredCount`.** No instance has free capacity, or tasks cannot pull the image. Confirm at least one `ACTIVE` container instance and that the subnets have an Internet Gateway route.
**`create_instance_role` conflict.** If `ecsInstanceRole` already exists (from the console), pass `-var 'create_instance_role=false'` so Terraform does not try to recreate it.
@@ -0,0 +1,524 @@
# Example: ECS Quickstart on Spinifex
#
# Provisions a complete ECS stack against the Spinifex AWS gateway: a cluster,
# a task definition with a task role, N container instances launched from the
# spinifex-ecs-node AMI, and an awsvpc service fronted by an Application Load
# Balancer target group.
#
# Unlike the AWS-managed ECS-optimized AMI, a Spinifex container instance reaches
# the control plane over the gateway (TLS + SigV4), so its cloud-init user-data
# must carry a LAN-reachable gateway URL and the gateway CA. The agent draws its
# credentials from IMDS via the ecsInstanceRole instance profile, so no keys are
# written. This is exactly what the console's "provision capacity" action does
# for you; the workbook spells it out so it is reproducible from Terraform.
#
# Usage:
# cd spinifex/docs/terraform-workbooks/ecs-quickstart
# export AWS_PROFILE=spinifex
# tofu init
# tofu apply -var 'gateway_url=https://<host-lan-ip>:9999'
#
# After apply, fetch the ALB's public IP (the *.elb.spinifex.local DNS name does
# not resolve from your host) and curl it:
#
# aws elbv2 describe-load-balancers --names ecs-quickstart-alb \
# --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
# --output text
# curl http://<alb_public_ip>
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
}
}
}
# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------
variable "region" {
type = string
default = "ap-southeast-2"
}
variable "cluster_name" {
type = string
default = "ecs-quickstart"
}
variable "instance_type" {
type = string
default = "t3.small"
}
variable "container_count" {
type = number
default = 1
description = "Container instances to launch (and the service desired count)."
}
variable "task_image" {
type = string
default = "docker.io/library/nginx:1.27-alpine"
description = "Fully-qualified image the task runs; the instance pulls it via containerd."
}
variable "spinifex_endpoint" {
type = string
default = "https://127.0.0.1:9999"
description = "Spinifex AWS gateway endpoint as seen from the host running Terraform."
}
variable "gateway_url" {
type = string
description = "Gateway URL as seen from a guest VM (LAN-reachable, not 127.0.0.1), e.g. https://192.168.1.33:9999."
}
variable "gateway_ca_cert_path" {
type = string
default = "/etc/spinifex/ca.pem"
description = "Host-readable path to the gateway TLS CA PEM, baked into each container instance."
}
variable "create_instance_role" {
type = bool
default = true
description = "Create the ecsInstanceRole role + policy + instance profile. Set false if the ECS console already created it (it is account-global)."
}
# ---------------------------------------------------------------------------
# Provider — point the AWS provider at Spinifex
# ---------------------------------------------------------------------------
provider "aws" {
region = var.region
endpoints {
ec2 = var.spinifex_endpoint
iam = var.spinifex_endpoint
sts = var.spinifex_endpoint
ecs = var.spinifex_endpoint
elasticloadbalancingv2 = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------
data "aws_availability_zones" "available" {
state = "available"
}
# Several spinifex-ecs-node revisions may be imported; take the newest.
data "aws_ami" "ecs_node" {
most_recent = true
owners = ["000000000000"]
filter {
name = "tag:spinifex:managed-by"
values = ["ecs"]
}
}
# ---------------------------------------------------------------------------
# VPC + two public subnets
# ---------------------------------------------------------------------------
resource "aws_vpc" "main" {
cidr_block = "10.40.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "${var.cluster_name}-vpc"
}
}
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
tags = {
Name = "${var.cluster_name}-igw"
}
}
resource "aws_subnet" "a" {
vpc_id = aws_vpc.main.id
cidr_block = "10.40.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
map_public_ip_on_launch = true
tags = {
Name = "${var.cluster_name}-subnet-a"
}
}
resource "aws_subnet" "b" {
vpc_id = aws_vpc.main.id
cidr_block = "10.40.2.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
map_public_ip_on_launch = true
tags = {
Name = "${var.cluster_name}-subnet-b"
}
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = {
Name = "${var.cluster_name}-public-rt"
}
}
resource "aws_route_table_association" "a" {
subnet_id = aws_subnet.a.id
route_table_id = aws_route_table.public.id
}
resource "aws_route_table_association" "b" {
subnet_id = aws_subnet.b.id
route_table_id = aws_route_table.public.id
}
# ---------------------------------------------------------------------------
# Security group — SSH + HTTP inbound, all outbound. Covers both the ALB and
# the awsvpc task ENIs (the ALB reaches each task's ENI on the container port).
# ---------------------------------------------------------------------------
resource "aws_security_group" "ecs" {
name = "${var.cluster_name}-sg"
description = "ECS quickstart: SSH + HTTP inbound, all outbound"
vpc_id = aws_vpc.main.id
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "All outbound"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "${var.cluster_name}-sg"
}
}
# ---------------------------------------------------------------------------
# IAM — the ecsInstanceRole instance profile (instance creds via IMDS) and a
# task role for the task definition's task_role_arn.
#
# ecsInstanceRole is account-global: the name, EC2 trust, and ecs:* policy must
# match what the gateway whitelists for IMDS cred vending. Toggle creation off
# (create_instance_role=false) when the ECS console already created it.
# ---------------------------------------------------------------------------
resource "aws_iam_role" "ecs_instance" {
count = var.create_instance_role ? 1 : 0
name = "ecsInstanceRole"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_policy" "ecs_instance" {
count = var.create_instance_role ? 1 : 0
name = "ecsInstanceRolePolicy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [{ Effect = "Allow", Action = "ecs:*", Resource = "*" }]
})
}
resource "aws_iam_role_policy_attachment" "ecs_instance" {
count = var.create_instance_role ? 1 : 0
role = aws_iam_role.ecs_instance[0].name
policy_arn = aws_iam_policy.ecs_instance[0].arn
}
resource "aws_iam_instance_profile" "ecs_instance" {
count = var.create_instance_role ? 1 : 0
name = "ecsInstanceRole"
role = aws_iam_role.ecs_instance[0].name
}
resource "aws_iam_role" "task" {
name = "${var.cluster_name}-task-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "ecs-tasks.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_policy" "task" {
name = "${var.cluster_name}-task-policy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [{ Effect = "Allow", Action = ["s3:GetObject"], Resource = "*" }]
})
}
resource "aws_iam_role_policy_attachment" "task" {
role = aws_iam_role.task.name
policy_arn = aws_iam_policy.task.arn
}
# ---------------------------------------------------------------------------
# SSH key pair — RunInstances requires a key (the agent itself needs no SSH).
# ---------------------------------------------------------------------------
resource "tls_private_key" "node" {
algorithm = "ED25519"
}
resource "aws_key_pair" "node" {
key_name = "${var.cluster_name}-node"
public_key = tls_private_key.node.public_key_openssh
}
resource "local_file" "node_pem" {
filename = "${path.module}/${var.cluster_name}-node.pem"
content = tls_private_key.node.private_key_openssh
file_permission = "0600"
}
# ---------------------------------------------------------------------------
# ECS cluster
# ---------------------------------------------------------------------------
resource "aws_ecs_cluster" "main" {
name = var.cluster_name
}
# ---------------------------------------------------------------------------
# Task definition — awsvpc, one HTTP container on port 80, with a task role.
# ---------------------------------------------------------------------------
resource "aws_ecs_task_definition" "web" {
family = "${var.cluster_name}-web"
network_mode = "awsvpc"
requires_compatibilities = ["EC2"]
task_role_arn = aws_iam_role.task.arn
container_definitions = jsonencode([{
name = "web"
image = var.task_image
cpu = 128
memory = 256
essential = true
portMappings = [{
containerPort = 80
protocol = "tcp"
}]
}])
}
# ---------------------------------------------------------------------------
# Container instances — spinifex-ecs-node AMI with the ecsInstanceRole profile
# and cloud-config user-data seeding the agent's gateway endpoint + CA.
# ---------------------------------------------------------------------------
resource "aws_instance" "node" {
count = var.container_count
ami = data.aws_ami.ecs_node.id
instance_type = var.instance_type
subnet_id = aws_subnet.a.id
vpc_security_group_ids = [aws_security_group.ecs.id]
iam_instance_profile = "ecsInstanceRole"
key_name = aws_key_pair.node.key_name
# Plain (non-dash) heredoc: leading whitespace is literal, so the YAML
# indentation and the CA's block-scalar indentation are exact. indent(6, ...)
# right-pads the CA's continuation lines to match the first line's 6 spaces.
user_data = <<USERDATA
#cloud-config
write_files:
- path: /etc/spinifex-ecs/agent.env
permissions: '0600'
content: |
ECS_GATEWAY_URL=${var.gateway_url}
ECS_GATEWAY_CA=/etc/spinifex-ecs/gateway-ca.pem
ECS_REGION=${var.region}
ECS_CLUSTER=${var.cluster_name}
- path: /etc/spinifex-ecs/gateway-ca.pem
permissions: '0644'
content: |
${indent(6, file(var.gateway_ca_cert_path))}
USERDATA
tags = {
Name = "ecs-node-${var.cluster_name}-${count.index}"
}
# Instance-profile creation must precede launch so IMDS can vend role creds.
depends_on = [aws_iam_instance_profile.ecs_instance]
}
# ---------------------------------------------------------------------------
# Application Load Balancer + target group (awsvpc → target_type ip) + listener
# ---------------------------------------------------------------------------
resource "aws_lb" "web" {
name = "${var.cluster_name}-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.ecs.id]
subnets = [aws_subnet.a.id, aws_subnet.b.id]
tags = {
Name = "${var.cluster_name}-alb"
}
}
resource "aws_lb_target_group" "web" {
name = "${var.cluster_name}-tg"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.main.id
target_type = "ip"
health_check {
path = "/"
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 5
interval = 10
}
tags = {
Name = "${var.cluster_name}-tg"
}
}
resource "aws_lb_listener" "http" {
load_balancer_arn = aws_lb.web.arn
port = 80
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.web.arn
}
}
# ---------------------------------------------------------------------------
# ECS service — awsvpc, fronted by the target group. The scheduler registers
# each task's ENI IP in the target group on RUNNING.
# ---------------------------------------------------------------------------
resource "aws_ecs_service" "web" {
name = "${var.cluster_name}-web"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.web.arn
desired_count = var.container_count
launch_type = "EC2"
network_configuration {
subnets = [aws_subnet.a.id]
security_groups = [aws_security_group.ecs.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.web.arn
container_name = "web"
container_port = 80
}
# The container instances must be registered before the service can place.
depends_on = [aws_lb_listener.http, aws_instance.node]
}
# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------
output "note" {
value = <<-EOT
Container instances take ~30-60s to boot and register. Track progress with:
aws ecs list-container-instances --cluster ${var.cluster_name}
aws ecs describe-services --cluster ${var.cluster_name} \
--services ${var.cluster_name}-web \
--query 'services[0].[runningCount,desiredCount]'
The ALB DNS name ends in .elb.spinifex.local and will not resolve from your
host. Fetch its public IP and curl it:
aws elbv2 describe-load-balancers --names ${var.cluster_name}-alb \
--query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
--output text
EOT
}
output "cluster_arn" {
value = aws_ecs_cluster.main.arn
}
output "service_name" {
value = aws_ecs_service.web.name
}
output "alb_name" {
value = aws_lb.web.name
}
output "container_instance_ids" {
value = aws_instance.node[*].id
}
+1 -1
View File
@@ -223,7 +223,7 @@ aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
The Nginx instances have no public IP, so you can't SSH in directly from your host. If you need to inspect cloud-init logs, launch a small jump host in the same VPC or run commands via the Spinifex console, then:
```bash
ssh -i nginx-alb-demo.pem ec2-user@<instance_private_ip>
ssh -i nginx-alb-demo.pem ubuntu@<instance_private_ip>
sudo journalctl -u cloud-init --no-pager
sudo systemctl status nginx
```
@@ -81,7 +81,7 @@ After apply completes, use the outputs to test:
```bash
curl http://<public_ip>
ssh -i nginx-demo.pem ec2-user@<public_ip>
ssh -i nginx-demo.pem ubuntu@<public_ip>
```
Open the `web_url` output in your browser to see the Nginx welcome page.
@@ -124,7 +124,7 @@ aws ec2 describe-instances --profile spinifex
SSH into the instance and check cloud-init logs:
```bash
ssh -i nginx-demo.pem ec2-user@<public_ip>
ssh -i nginx-demo.pem ubuntu@<public_ip>
sudo journalctl -u cloud-init --no-pager
sudo systemctl status nginx
```
@@ -11,7 +11,7 @@
#
# After apply:
# curl http://<public_ip> # Nginx welcome page
# ssh -i nginx-demo.pem ec2-user@<public_ip>
# ssh -i nginx-demo.pem ubuntu@<public_ip>
terraform {
required_version = ">= 1.6.0"
@@ -283,7 +283,7 @@ output "public_ip" {
}
output "ssh_command" {
value = "ssh -i nginx-demo.pem ec2-user@${aws_instance.nginx.public_ip}"
value = "ssh -i nginx-demo.pem ubuntu@${aws_instance.nginx.public_ip}"
}
output "web_url" {
+3 -3
View File
@@ -143,7 +143,7 @@ aws s3 ls --profile spinifex --endpoint-url https://localhost:8443
SSH into the instance and check the service:
```bash
ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
ssh -i s3-webapp-demo.pem ubuntu@<public_ip>
sudo systemctl status s3-webapp
sudo journalctl -u s3-webapp --no-pager -n 50
```
@@ -153,7 +153,7 @@ sudo journalctl -u s3-webapp --no-pager -n 50
The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:
```bash
ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
ssh -i s3-webapp-demo.pem ubuntu@<public_ip>
cat /opt/webapp/.env # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
```
@@ -167,7 +167,7 @@ Check that:
boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:
```bash
ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
ssh -i s3-webapp-demo.pem ubuntu@<public_ip>
# IMDSv2: fetch a token, then the role name behind the instance profile
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
+2 -2
View File
@@ -38,7 +38,7 @@
#
# After apply:
# curl http://<public_ip> # File browser UI
# ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
# ssh -i s3-webapp-demo.pem ubuntu@<public_ip>
terraform {
required_version = ">= 1.6.0"
@@ -504,7 +504,7 @@ output "bucket_name" {
}
output "ssh_command" {
value = "ssh -i s3-webapp-demo.pem ec2-user@${aws_instance.webapp.public_ip}"
value = "ssh -i s3-webapp-demo.pem ubuntu@${aws_instance.webapp.public_ip}"
}
output "web_url" {
+53 -12
View File
@@ -7,18 +7,20 @@ require (
github.com/charmbracelet/bubbles v1.0.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/containerd/containerd/v2 v2.3.2
github.com/distribution/reference v0.6.0
github.com/go-chi/chi/v5 v5.3.0
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
github.com/kdomanski/iso9660 v0.4.0
github.com/klauspost/cpuid/v2 v2.3.0
github.com/mulgadc/predastore v1.9.0
github.com/mulgadc/viperblock v1.9.0
github.com/mulgadc/predastore v1.10.0
github.com/mulgadc/viperblock v1.10.0
github.com/nats-io/nats-server/v2 v2.14.2
github.com/nats-io/nats.go v1.52.0
github.com/opencontainers/runtime-spec v1.3.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.4.0
github.com/pelletier/go-toml/v2 v2.4.2
github.com/pterm/pterm v0.12.83
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
@@ -35,6 +37,8 @@ require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/Microsoft/go-winio v0.6.3-0.20251027160822-ad3df93bed29 // indirect
github.com/Microsoft/hcsshim v0.15.0-rc.1 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
@@ -55,13 +59,25 @@ require (
github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/cgroups/v3 v3.1.3 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/containerd/containerd/api v1.11.1 // indirect
github.com/containerd/continuity v0.5.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/fifo v1.1.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v1.0.0-rc.4 // indirect
github.com/containerd/plugin v1.1.0 // indirect
github.com/containerd/ttrpc v1.2.8 // indirect
github.com/containerd/typeurl/v2 v2.2.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgraph-io/badger/v4 v4.9.2 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/fatih/color v1.19.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/go-logr/logr v1.4.3 // indirect
@@ -70,6 +86,8 @@ require (
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/gookit/color v1.6.0 // indirect
@@ -94,9 +112,15 @@ require (
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/mdlayher/packet v1.1.2 // indirect
github.com/mdlayher/socket v0.4.1 // indirect
github.com/mdlayher/socket v0.5.1 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/sys/mountinfo v0.7.2 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/signal v0.7.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
@@ -104,8 +128,11 @@ require (
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/pierrec/lz4/v4 v4.1.14 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
@@ -113,6 +140,7 @@ require (
github.com/quic-go/quic-go v0.60.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
@@ -120,10 +148,12 @@ require (
github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel v1.42.0 // indirect
go.opentelemetry.io/otel/metric v1.42.0 // indirect
go.opentelemetry.io/otel/trace v1.42.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.36.0 // indirect
@@ -134,7 +164,18 @@ require (
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.45.0 // indirect
golang.org/x/vuln v1.1.4 // indirect
google.golang.org/protobuf v1.36.11 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3 // indirect
google.golang.org/grpc v1.81.1 // indirect
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
)
// go.opencensus.io -> grpc@v1.33.2 drags in the pre-split monolithic genproto,
// which still ships google.golang.org/genproto/googleapis/rpc/status and so
// collides with the split genproto/googleapis/rpc module that containerd/ttrpc
// needs ("ambiguous import"). The main module never imports the monolith, so a
// plain require is dropped by `go mod tidy`; pin it via replace to a post-split
// version (no rpc/status) so the fix survives tidy. Remove if grpc/opencensus
// stop requiring the monolith.
replace google.golang.org/genproto => google.golang.org/genproto v0.0.0-20260622175928-b703f567277d
tool golang.org/x/vuln/cmd/govulncheck
+2622 -21
View File
File diff suppressed because it is too large Load Diff
+119
View File
@@ -0,0 +1,119 @@
// Package ecrauth mints ECR registry credentials by SigV4-signing
// GetAuthorizationToken against the Spinifex AWS gateway's ECR endpoint and
// decoding the returned AWS:<jwt> token into Basic-auth parts. It is shared by
// the in-guest binaries that pull images from the internal registry: the EKS
// ecr-credential-provider and the ECS ecs-agent.
package ecrauth
import (
"crypto/tls"
"crypto/x509"
"encoding/base64"
"fmt"
"net/http"
"os"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ecr"
"github.com/mulgadc/spinifex/internal/tlsconfig"
)
// defaultRegion is used when the caller supplies no region; SigV4 still needs a
// scope and the gateway does not enforce a specific value.
const defaultRegion = "us-east-1"
// Token is the decoded result of GetAuthorizationToken: Basic-auth parts plus
// the registry proxy host and token expiry.
type Token struct {
Username string
Password string
ProxyHost string
ExpiresAt time.Time
}
// GetAuthorizationToken builds an aws-sdk ECR client pointed at the gateway and
// calls GetAuthorizationToken with the supplied static credentials, decoding the
// AWS:<jwt> token into Basic-auth parts. httpClient must trust the gateway CA.
func GetAuthorizationToken(region, gatewayURL string, httpClient *http.Client, akid, secret, sessionToken string) (Token, error) {
if region == "" {
region = defaultRegion
}
sess, err := session.NewSession(&aws.Config{
Region: aws.String(region),
Endpoint: aws.String(gatewayURL),
Credentials: credentials.NewStaticCredentials(akid, secret, sessionToken),
HTTPClient: httpClient,
DisableSSL: aws.Bool(false),
})
if err != nil {
return Token{}, fmt.Errorf("new session: %w", err)
}
out, err := ecr.New(sess).GetAuthorizationToken(&ecr.GetAuthorizationTokenInput{})
if err != nil {
return Token{}, fmt.Errorf("GetAuthorizationToken: %w", err)
}
if len(out.AuthorizationData) == 0 || out.AuthorizationData[0].AuthorizationToken == nil {
return Token{}, fmt.Errorf("GetAuthorizationToken returned no authorization data")
}
data := out.AuthorizationData[0]
raw, err := base64.StdEncoding.DecodeString(aws.StringValue(data.AuthorizationToken))
if err != nil {
return Token{}, fmt.Errorf("decode authorization token: %w", err)
}
user, pass, ok := strings.Cut(string(raw), ":")
if !ok {
return Token{}, fmt.Errorf("authorization token not in user:password form")
}
tok := Token{
Username: user,
Password: pass,
ProxyHost: HostFromImage(aws.StringValue(data.ProxyEndpoint)),
}
if data.ExpiresAt != nil {
tok.ExpiresAt = aws.TimeValue(data.ExpiresAt)
}
return tok, nil
}
// GatewayHTTPClient builds an HTTP client trusting the gateway CA at caPath.
// When caPath is empty it relies on the system trust store.
func GatewayHTTPClient(caPath string) (*http.Client, error) {
tlsCfg := &tls.Config{
MinVersion: tls.VersionTLS13,
CurvePreferences: tlsconfig.Curves,
}
if caPath != "" {
pem, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
}
tlsCfg.RootCAs = pool
}
return &http.Client{
Timeout: 15 * time.Second,
Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
}, nil
}
// HostFromImage returns the host[:port] portion of an image ref or URL. It strips
// any scheme and trims at the first path separator.
func HostFromImage(image string) string {
if i := strings.Index(image, "://"); i >= 0 {
image = image[i+3:]
}
if i := strings.IndexByte(image, '/'); i >= 0 {
image = image[:i]
}
return image
}
+107
View File
@@ -0,0 +1,107 @@
package ecrauth
import (
"crypto/tls"
"crypto/x509"
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
func TestHostFromImage(t *testing.T) {
cases := map[string]string{
"acct.dkr.ecr.us-east-1.spinifex.internal/demo:latest": "acct.dkr.ecr.us-east-1.spinifex.internal",
"https://acct.dkr.ecr.us-east-1.spinifex.internal": "acct.dkr.ecr.us-east-1.spinifex.internal",
"host:9999/repo:tag": "host:9999",
"plainhost": "plainhost",
}
for in, want := range cases {
if got := HostFromImage(in); got != want {
t.Errorf("HostFromImage(%q) = %q, want %q", in, got, want)
}
}
}
// gatewayStub serves the ECR GetAuthorizationToken X-Amz-Target dispatch,
// returning a base64 AWS:<jwt> token like the real gateway.
func gatewayStub(t *testing.T, token, proxy string, status int) *httptest.Server {
t.Helper()
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
w.WriteHeader(http.StatusBadRequest)
return
}
if status != http.StatusOK {
w.WriteHeader(status)
return
}
w.Header().Set("Content-Type", "application/x-amz-json-1.1")
_ = json.NewEncoder(w).Encode(map[string]any{
"authorizationData": []map[string]any{{
"authorizationToken": base64.StdEncoding.EncodeToString([]byte(token)),
"proxyEndpoint": proxy,
}},
})
}))
t.Cleanup(srv.Close)
return srv
}
func clientTrusting(srv *httptest.Server) *http.Client {
pool := x509.NewCertPool()
pool.AddCert(srv.Certificate())
return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
}
func TestGetAuthorizationToken_OK(t *testing.T) {
srv := gatewayStub(t, "AWS:testjwt", "https://123456789012.dkr.ecr.us-east-1.spinifex.internal", http.StatusOK)
tok, err := GetAuthorizationToken("us-east-1", srv.URL, clientTrusting(srv), "AKIA", "secret", "session")
if err != nil {
t.Fatalf("GetAuthorizationToken: %v", err)
}
if tok.Username != "AWS" || tok.Password != "testjwt" {
t.Errorf("got %q:%q, want AWS:testjwt", tok.Username, tok.Password)
}
if tok.ProxyHost != "123456789012.dkr.ecr.us-east-1.spinifex.internal" {
t.Errorf("ProxyHost = %q", tok.ProxyHost)
}
}
func TestGetAuthorizationToken_BadTokenForm(t *testing.T) {
srv := gatewayStub(t, "no-colon-token", "https://x", http.StatusOK)
if _, err := GetAuthorizationToken("us-east-1", srv.URL, clientTrusting(srv), "AKIA", "secret", ""); err == nil {
t.Fatal("expected error for token not in user:password form")
}
}
func TestGetAuthorizationToken_ServerError(t *testing.T) {
srv := gatewayStub(t, "", "", http.StatusInternalServerError)
if _, err := GetAuthorizationToken("us-east-1", srv.URL, clientTrusting(srv), "AKIA", "secret", ""); err == nil {
t.Fatal("expected error on gateway 500")
}
}
func TestGatewayHTTPClient(t *testing.T) {
// Empty CA path: system trust store, no error.
if _, err := GatewayHTTPClient(""); err != nil {
t.Fatalf("empty CA path: %v", err)
}
// Missing file: error.
if _, err := GatewayHTTPClient(filepath.Join(t.TempDir(), "nope.pem")); err == nil {
t.Fatal("expected error for missing CA file")
}
// Garbage PEM: error.
bad := filepath.Join(t.TempDir(), "bad.pem")
if err := os.WriteFile(bad, []byte("not a cert"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := GatewayHTTPClient(bad); err == nil {
t.Fatal("expected error for CA file with no certificates")
}
}
+123
View File
@@ -0,0 +1,123 @@
// Package ecsgw is the on-VM SigV4 client the ecs-agent uses to reach the AWS
// gateway over HTTPS instead of connecting to NATS directly. It signs requests
// for service "ecs" with the instance's IMDS instance-role credentials (fetched
// fresh per call so rotation is transparent), keeping the NATS bus host-internal
// (the gateway relays agent calls onto ecs.bus.* host-side).
package ecsgw
import (
"bytes"
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws/credentials"
v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
"github.com/mulgadc/spinifex/internal/tlsconfig"
)
// targetPrefix is the X-Amz-Target service prefix the gateway's ECS dispatch
// strips to the bare action. Any prefix works (ecsActionFromTarget takes the
// suffix); this one mirrors the AWS JSON 1.1 convention.
const targetPrefix = "AmazonEC2ContainerServiceV20141113."
// CredentialsFunc yields the current SigV4 credentials. It is called once per
// request so a provider backed by IMDS can rotate instance-role creds (and their
// session token) transparently. Must be safe for concurrent use.
type CredentialsFunc func(ctx context.Context) (accessKey, secretKey, sessionToken string, err error)
// Client posts SigV4-signed AWS JSON 1.1 requests to the gateway for service
// "ecs". One client is reused for register/heartbeat/state/poll.
type Client struct {
baseURL string
creds CredentialsFunc
region string
httpClient *http.Client
}
// New builds a client. caPath optionally pins the gateway TLS CA; empty relies on
// the system trust store. creds supplies the per-call instance-role credentials.
// region defaults to us-east-1 when empty (SigV4 requires a non-empty region).
// timeout bounds a single call — callers needing a long-poll pass a larger value
// than the default register/heartbeat timeout.
func New(baseURL, caPath string, creds CredentialsFunc, region string, timeout time.Duration) (*Client, error) {
if baseURL == "" {
return nil, fmt.Errorf("ecsgw: baseURL is required")
}
if creds == nil {
return nil, fmt.Errorf("ecsgw: credentials func is required")
}
if region == "" {
region = "us-east-1"
}
if timeout <= 0 {
timeout = 10 * time.Second
}
tlsCfg := &tls.Config{
MinVersion: tls.VersionTLS13,
CurvePreferences: tlsconfig.Curves,
}
if caPath != "" {
pem, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("ecsgw: read gateway CA %q: %w", caPath, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("ecsgw: gateway CA %q has no usable certificates", caPath)
}
tlsCfg.RootCAs = pool
}
return &Client{
baseURL: strings.TrimRight(baseURL, "/"),
creds: creds,
region: region,
httpClient: &http.Client{
Timeout: timeout,
Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
},
}, nil
}
// Call SigV4-signs and POSTs body under X-Amz-Target "<prefix><action>" to the
// gateway root, returning the 2xx response body. Credentials are fetched per call
// and the v4 signer emits X-Amz-Security-Token for instance-role (temporary)
// creds, so the gateway's ASIA path verifies them. No retry; callers wrap.
func (c *Client) Call(action string, body []byte) ([]byte, error) {
req, err := http.NewRequest(http.MethodPost, c.baseURL+"/", bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
req.Header.Set("X-Amz-Target", targetPrefix+action)
akid, secret, token, err := c.creds(req.Context())
if err != nil {
return nil, fmt.Errorf("retrieve credentials: %w", err)
}
signer := v4.NewSigner(credentials.NewStaticCredentials(akid, secret, token))
if _, err := signer.Sign(req, bytes.NewReader(body), "ecs", c.region, time.Now()); err != nil {
return nil, fmt.Errorf("sign request: %w", err)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, fmt.Errorf("send request: %w", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("gateway returned %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
}
return respBody, nil
}
+87
View File
@@ -0,0 +1,87 @@
package ecsgw
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// testCreds returns a fixed temporary credential set (with a session token, as
// IMDS instance-role creds carry).
func testCreds(context.Context) (string, string, string, error) {
return "ASIATEST", "secret", "session-token-xyz", nil
}
func newTestClient(t *testing.T, srv *httptest.Server) *Client {
t.Helper()
c, err := New(srv.URL, "", testCreds, "ap-southeast-2", time.Second)
if err != nil {
t.Fatalf("New: %v", err)
}
c.httpClient = srv.Client()
return c
}
// Call signs for service "ecs", sets the X-Amz-Target action, and returns the
// 2xx body.
func TestClient_CallSignsAndTargets(t *testing.T) {
var gotTarget, gotAuth, gotBody, gotToken string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotTarget = r.Header.Get("X-Amz-Target")
gotAuth = r.Header.Get("Authorization")
gotToken = r.Header.Get("X-Amz-Security-Token")
b, _ := io.ReadAll(r.Body)
gotBody = string(b)
_, _ = w.Write([]byte(`{"ok":true}`))
}))
defer srv.Close()
c := newTestClient(t, srv)
out, err := c.Call("RegisterContainerInstance", []byte(`{"instanceId":"i-1"}`))
if err != nil {
t.Fatalf("Call: %v", err)
}
if string(out) != `{"ok":true}` {
t.Errorf("body = %q", out)
}
if gotTarget != targetPrefix+"RegisterContainerInstance" {
t.Errorf("target = %q", gotTarget)
}
if gotBody != `{"instanceId":"i-1"}` {
t.Errorf("relayed body = %q", gotBody)
}
if !strings.Contains(gotAuth, "AWS4-HMAC-SHA256") || !strings.Contains(gotAuth, "/ecs/aws4_request") {
t.Errorf("auth header not SigV4 for ecs: %q", gotAuth)
}
if gotToken != "session-token-xyz" {
t.Errorf("X-Amz-Security-Token = %q, want the IMDS session token", gotToken)
}
}
// A non-2xx gateway response surfaces as an error carrying status + body.
func TestClient_CallErrorStatus(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusForbidden)
_, _ = w.Write([]byte(`{"__type":"AccessDenied"}`))
}))
defer srv.Close()
c := newTestClient(t, srv)
if _, err := c.Call("Heartbeat", []byte(`{}`)); err == nil ||
!strings.Contains(err.Error(), "403") || !strings.Contains(err.Error(), "AccessDenied") {
t.Fatalf("want 403 AccessDenied error, got %v", err)
}
}
func TestNew_Validation(t *testing.T) {
if _, err := New("", "", testCreds, "r", 0); err == nil {
t.Error("want error for empty baseURL")
}
if _, err := New("https://gw:9999", "", nil, "r", 0); err == nil {
t.Error("want error for nil credentials func")
}
}
+118
View File
@@ -0,0 +1,118 @@
// Package imdscreds fetches AWS instance-role credentials from the in-guest
// IMDS endpoint (169.254.169.254) using IMDSv2. It is shared by the binaries
// that run inside a Spinifex guest and need the node/instance role credentials:
// the EKS ecr-credential-provider and the ECS ecs-agent.
package imdscreds
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// imdsTokenTTL is the IMDSv2 session-token lifetime requested on the PUT.
const imdsTokenTTL = "21600"
// Credentials is a resolved instance-role credential set with parsed expiry.
type Credentials struct {
AccessKeyID string
SecretAccessKey string
SessionToken string
Expiration time.Time
}
// imdsCreds mirrors the IMDS security-credentials JSON shape.
type imdsCreds struct {
Code string `json:"Code"`
AccessKeyID string `json:"AccessKeyId"`
SecretAccessKey string `json:"SecretAccessKey"`
Token string `json:"Token"`
Expiration string `json:"Expiration"`
}
// Fetch resolves the instance role name then its credentials via IMDSv2
// (PUT token, then GET with the token header). base is the IMDS root, e.g.
// http://169.254.169.254/latest.
func Fetch(client *http.Client, base string) (Credentials, error) {
base = strings.TrimRight(base, "/")
token := v2Token(client, base)
roleName, err := get(client, base+"/meta-data/iam/security-credentials/", token)
if err != nil {
return Credentials{}, fmt.Errorf("fetch role name: %w", err)
}
roleName = strings.TrimSpace(roleName)
if roleName == "" {
return Credentials{}, fmt.Errorf("IMDS returned empty role name")
}
body, err := get(client, base+"/meta-data/iam/security-credentials/"+roleName, token)
if err != nil {
return Credentials{}, fmt.Errorf("fetch role credentials: %w", err)
}
var raw imdsCreds
if err := json.Unmarshal([]byte(body), &raw); err != nil {
return Credentials{}, fmt.Errorf("decode role credentials: %w", err)
}
if raw.AccessKeyID == "" || raw.SecretAccessKey == "" {
return Credentials{}, fmt.Errorf("IMDS credentials missing access/secret key")
}
creds := Credentials{
AccessKeyID: raw.AccessKeyID,
SecretAccessKey: raw.SecretAccessKey,
SessionToken: raw.Token,
}
if raw.Expiration != "" {
// IMDS emits RFC3339; a parse failure leaves a zero expiry (treated as
// unknown by callers) rather than failing the whole fetch.
if exp, perr := time.Parse(time.RFC3339, raw.Expiration); perr == nil {
creds.Expiration = exp
}
}
return creds, nil
}
// v2Token requests an IMDSv2 session token; returns "" if the service does not
// enforce v2 (tokenless v1 then applies).
func v2Token(client *http.Client, base string) string {
req, err := http.NewRequest(http.MethodPut, base+"/api/token", nil)
if err != nil {
return ""
}
req.Header.Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", imdsTokenTTL)
resp, err := client.Do(req)
if err != nil {
return ""
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return ""
}
body, _ := io.ReadAll(resp.Body)
return strings.TrimSpace(string(body))
}
// get issues an IMDS GET, attaching the IMDSv2 token header when present.
func get(client *http.Client, url, token string) (string, error) {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return "", err
}
if token != "" {
req.Header.Set("X-Aws-Ec2-Metadata-Token", token)
}
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("IMDS %s returned %d: %s", url, resp.StatusCode, strings.TrimSpace(string(body)))
}
return string(body), nil
}
+80
View File
@@ -0,0 +1,80 @@
package imdscreds
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
)
// imdsStub serves the IMDSv2 token + role + credentials endpoints.
func imdsStub(t *testing.T, creds map[string]string, enforceV2 bool) *httptest.Server {
t.Helper()
mux := http.NewServeMux()
mux.HandleFunc("/api/token", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPut {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
_, _ = w.Write([]byte("v2-token"))
})
mux.HandleFunc("/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
if enforceV2 && r.Header.Get("X-Aws-Ec2-Metadata-Token") != "v2-token" {
w.WriteHeader(http.StatusUnauthorized)
return
}
if r.URL.Path == "/meta-data/iam/security-credentials/" {
_, _ = w.Write([]byte("node-role"))
return
}
_ = json.NewEncoder(w).Encode(creds)
})
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv
}
func TestFetch_OK(t *testing.T) {
exp := time.Now().Add(6 * time.Hour).UTC().Round(time.Second)
srv := imdsStub(t, map[string]string{
"Code": "Success",
"AccessKeyId": "AKIATEST",
"SecretAccessKey": "secrettest",
"Token": "sessiontoken",
"Expiration": exp.Format(time.RFC3339),
}, true)
got, err := Fetch(srv.Client(), srv.URL)
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if got.AccessKeyID != "AKIATEST" || got.SecretAccessKey != "secrettest" || got.SessionToken != "sessiontoken" {
t.Errorf("creds mismatch: %+v", got)
}
if !got.Expiration.Equal(exp) {
t.Errorf("Expiration = %v, want %v", got.Expiration, exp)
}
}
func TestFetch_MissingKeys(t *testing.T) {
srv := imdsStub(t, map[string]string{"Code": "Success"}, false)
if _, err := Fetch(srv.Client(), srv.URL); err == nil {
t.Fatal("expected error when access/secret key absent")
}
}
func TestFetch_BadExpirationIsZero(t *testing.T) {
srv := imdsStub(t, map[string]string{
"AccessKeyId": "AKIA",
"SecretAccessKey": "secret",
"Expiration": "not-a-time",
}, false)
got, err := Fetch(srv.Client(), srv.URL)
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if !got.Expiration.IsZero() {
t.Errorf("Expiration = %v, want zero on parse failure", got.Expiration)
}
}
+70
View File
@@ -0,0 +1,70 @@
// Package stsauth mints task-role credentials by SigV4-signing sts:AssumeRole
// against the Spinifex AWS gateway. It mirrors internal/ecrauth: the in-guest
// ecs-agent already holds instance-role credentials and signs gateway calls with
// them, so it assumes a task's IAM role the same way rather than reaching STS
// over NATS (the agent has no NATS connection).
package stsauth
import (
"fmt"
"net/http"
"time"
"github.com/aws/aws-sdk-go/aws"
awscreds "github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/sts"
)
// defaultRegion is used when the caller supplies no region; SigV4 still needs a
// scope and the gateway does not enforce a specific value.
const defaultRegion = "us-east-1"
// Credentials is the decoded result of AssumeRole: a SigV4 credential set bound
// to the assumed role, with its expiry.
type Credentials struct {
AccessKeyID string
SecretAccessKey string
SessionToken string
Expiration time.Time
}
// AssumeRole builds an aws-sdk STS client pointed at the gateway and calls
// AssumeRole with the supplied static (instance-role) credentials, returning the
// assumed-role credential set. httpClient must trust the gateway CA.
func AssumeRole(region, gatewayURL string, httpClient *http.Client, akid, secret, sessionToken, roleARN, sessionName string) (Credentials, error) {
if region == "" {
region = defaultRegion
}
if roleARN == "" || sessionName == "" {
return Credentials{}, fmt.Errorf("stsauth: roleARN and sessionName are required")
}
sess, err := session.NewSession(&aws.Config{
Region: aws.String(region),
Endpoint: aws.String(gatewayURL),
Credentials: awscreds.NewStaticCredentials(akid, secret, sessionToken),
HTTPClient: httpClient,
DisableSSL: aws.Bool(false),
})
if err != nil {
return Credentials{}, fmt.Errorf("new session: %w", err)
}
out, err := sts.New(sess).AssumeRole(&sts.AssumeRoleInput{
RoleArn: aws.String(roleARN),
RoleSessionName: aws.String(sessionName),
})
if err != nil {
return Credentials{}, fmt.Errorf("AssumeRole: %w", err)
}
if out.Credentials == nil {
return Credentials{}, fmt.Errorf("AssumeRole returned no credentials")
}
c := out.Credentials
return Credentials{
AccessKeyID: aws.StringValue(c.AccessKeyId),
SecretAccessKey: aws.StringValue(c.SecretAccessKey),
SessionToken: aws.StringValue(c.SessionToken),
Expiration: aws.TimeValue(c.Expiration),
}, nil
}
+63
View File
@@ -0,0 +1,63 @@
package stsauth
import (
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// assumeRoleXML is a minimal AWS AssumeRole response the aws-sdk STS client
// decodes into Credentials.
const assumeRoleXML = `<AssumeRoleResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
<AssumeRoleResult>
<Credentials>
<AccessKeyId>ASIATESTKEY</AccessKeyId>
<SecretAccessKey>testsecret</SecretAccessKey>
<SessionToken>testtoken</SessionToken>
<Expiration>2031-01-02T03:04:05Z</Expiration>
</Credentials>
</AssumeRoleResult>
</AssumeRoleResponse>`
func TestAssumeRole_DecodesCredentials(t *testing.T) {
var gotBody string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b, _ := io.ReadAll(r.Body)
gotBody = string(b)
w.Header().Set("Content-Type", "text/xml")
_, _ = w.Write([]byte(assumeRoleXML))
}))
defer srv.Close()
creds, err := AssumeRole("us-east-1", srv.URL, srv.Client(),
"AKIAINSTANCE", "instancesecret", "", "arn:aws:iam::111122223333:role/task", "ecs-cred-1")
if err != nil {
t.Fatalf("AssumeRole: %v", err)
}
if creds.AccessKeyID != "ASIATESTKEY" || creds.SecretAccessKey != "testsecret" || creds.SessionToken != "testtoken" {
t.Errorf("bad creds: %+v", creds)
}
want, _ := time.Parse(time.RFC3339, "2031-01-02T03:04:05Z")
if !creds.Expiration.Equal(want) {
t.Errorf("Expiration = %v, want %v", creds.Expiration, want)
}
// The request must carry the AssumeRole action and the role ARN we asked for.
if !strings.Contains(gotBody, "Action=AssumeRole") {
t.Errorf("request not an AssumeRole call: %s", gotBody)
}
if !strings.Contains(gotBody, "role%2Ftask") && !strings.Contains(gotBody, "role/task") {
t.Errorf("request missing role ARN: %s", gotBody)
}
}
func TestAssumeRole_RequiresRoleAndSession(t *testing.T) {
if _, err := AssumeRole("us-east-1", "https://gw", http.DefaultClient, "a", "b", "", "", "sess"); err == nil {
t.Error("want error for empty roleARN")
}
if _, err := AssumeRole("us-east-1", "https://gw", http.DefaultClient, "a", "b", "", "arn:role", ""); err == nil {
t.Error("want error for empty sessionName")
}
}
-3
View File
@@ -1,3 +0,0 @@
Forked from https://github.com/aws/aws-sdk-go
APACHE 2.0 LICENSE
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
@@ -1,339 +0,0 @@
{
"version": "1.0",
"parameters": {
"Region": {
"builtIn": "AWS::Region",
"required": false,
"documentation": "The AWS region used to dispatch the request.",
"type": "String"
},
"UseDualStack": {
"builtIn": "AWS::UseDualStack",
"required": true,
"default": false,
"documentation": "When true, use the dual-stack endpoint. If the configured endpoint does not support dual-stack, dispatching the request MAY return an error.",
"type": "Boolean"
},
"UseFIPS": {
"builtIn": "AWS::UseFIPS",
"required": true,
"default": false,
"documentation": "When true, send this request to the FIPS-compliant regional endpoint. If the configured endpoint does not have a FIPS compliant endpoint, dispatching the request will return an error.",
"type": "Boolean"
},
"Endpoint": {
"builtIn": "SDK::Endpoint",
"required": false,
"documentation": "Override the endpoint used to send this request",
"type": "String"
}
},
"rules": [
{
"conditions": [
{
"fn": "isSet",
"argv": [
{
"ref": "Endpoint"
}
]
}
],
"rules": [
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
{
"ref": "UseFIPS"
},
true
]
}
],
"error": "Invalid Configuration: FIPS and custom endpoint are not supported",
"type": "error"
},
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
{
"ref": "UseDualStack"
},
true
]
}
],
"error": "Invalid Configuration: Dualstack and custom endpoint are not supported",
"type": "error"
},
{
"conditions": [],
"endpoint": {
"url": {
"ref": "Endpoint"
},
"properties": {},
"headers": {}
},
"type": "endpoint"
}
],
"type": "tree"
},
{
"conditions": [
{
"fn": "isSet",
"argv": [
{
"ref": "Region"
}
]
}
],
"rules": [
{
"conditions": [
{
"fn": "aws.partition",
"argv": [
{
"ref": "Region"
}
],
"assign": "PartitionResult"
}
],
"rules": [
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
{
"ref": "UseFIPS"
},
true
]
},
{
"fn": "booleanEquals",
"argv": [
{
"ref": "UseDualStack"
},
true
]
}
],
"rules": [
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
true,
{
"fn": "getAttr",
"argv": [
{
"ref": "PartitionResult"
},
"supportsFIPS"
]
}
]
},
{
"fn": "booleanEquals",
"argv": [
true,
{
"fn": "getAttr",
"argv": [
{
"ref": "PartitionResult"
},
"supportsDualStack"
]
}
]
}
],
"rules": [
{
"conditions": [],
"endpoint": {
"url": "https://ec2-fips.{Region}.{PartitionResult#dualStackDnsSuffix}",
"properties": {},
"headers": {}
},
"type": "endpoint"
}
],
"type": "tree"
},
{
"conditions": [],
"error": "FIPS and DualStack are enabled, but this partition does not support one or both",
"type": "error"
}
],
"type": "tree"
},
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
{
"ref": "UseFIPS"
},
true
]
}
],
"rules": [
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
{
"fn": "getAttr",
"argv": [
{
"ref": "PartitionResult"
},
"supportsFIPS"
]
},
true
]
}
],
"rules": [
{
"conditions": [
{
"fn": "stringEquals",
"argv": [
{
"fn": "getAttr",
"argv": [
{
"ref": "PartitionResult"
},
"name"
]
},
"aws-us-gov"
]
}
],
"endpoint": {
"url": "https://ec2.{Region}.amazonaws.com",
"properties": {},
"headers": {}
},
"type": "endpoint"
},
{
"conditions": [],
"endpoint": {
"url": "https://ec2-fips.{Region}.{PartitionResult#dnsSuffix}",
"properties": {},
"headers": {}
},
"type": "endpoint"
}
],
"type": "tree"
},
{
"conditions": [],
"error": "FIPS is enabled but this partition does not support FIPS",
"type": "error"
}
],
"type": "tree"
},
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
{
"ref": "UseDualStack"
},
true
]
}
],
"rules": [
{
"conditions": [
{
"fn": "booleanEquals",
"argv": [
true,
{
"fn": "getAttr",
"argv": [
{
"ref": "PartitionResult"
},
"supportsDualStack"
]
}
]
}
],
"rules": [
{
"conditions": [],
"endpoint": {
"url": "https://ec2.{Region}.{PartitionResult#dualStackDnsSuffix}",
"properties": {},
"headers": {}
},
"type": "endpoint"
}
],
"type": "tree"
},
{
"conditions": [],
"error": "DualStack is enabled but this partition does not support DualStack",
"type": "error"
}
],
"type": "tree"
},
{
"conditions": [],
"endpoint": {
"url": "https://ec2.{Region}.{PartitionResult#dnsSuffix}",
"properties": {},
"headers": {}
},
"type": "endpoint"
}
],
"type": "tree"
}
],
"type": "tree"
},
{
"conditions": [],
"error": "Invalid Configuration: Missing Region",
"type": "error"
}
]
}
@@ -1,743 +0,0 @@
{
"testCases": [
{
"documentation": "For region af-south-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.af-south-1.amazonaws.com"
}
},
"params": {
"Region": "af-south-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-east-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-east-1.amazonaws.com"
}
},
"params": {
"Region": "ap-east-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-northeast-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-northeast-1.amazonaws.com"
}
},
"params": {
"Region": "ap-northeast-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-northeast-2 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-northeast-2.amazonaws.com"
}
},
"params": {
"Region": "ap-northeast-2",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-northeast-3 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-northeast-3.amazonaws.com"
}
},
"params": {
"Region": "ap-northeast-3",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-south-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-south-1.amazonaws.com"
}
},
"params": {
"Region": "ap-south-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-south-1 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-south-1.api.aws"
}
},
"params": {
"Region": "ap-south-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region ap-southeast-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-southeast-1.amazonaws.com"
}
},
"params": {
"Region": "ap-southeast-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-southeast-2 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-southeast-2.amazonaws.com"
}
},
"params": {
"Region": "ap-southeast-2",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ap-southeast-3 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ap-southeast-3.amazonaws.com"
}
},
"params": {
"Region": "ap-southeast-3",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ca-central-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.ca-central-1.amazonaws.com"
}
},
"params": {
"Region": "ca-central-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region ca-central-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.ca-central-1.amazonaws.com"
}
},
"params": {
"Region": "ca-central-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region eu-central-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-central-1.amazonaws.com"
}
},
"params": {
"Region": "eu-central-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region eu-north-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-north-1.amazonaws.com"
}
},
"params": {
"Region": "eu-north-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region eu-south-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-south-1.amazonaws.com"
}
},
"params": {
"Region": "eu-south-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region eu-west-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-west-1.amazonaws.com"
}
},
"params": {
"Region": "eu-west-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region eu-west-1 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-west-1.api.aws"
}
},
"params": {
"Region": "eu-west-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region eu-west-2 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-west-2.amazonaws.com"
}
},
"params": {
"Region": "eu-west-2",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region eu-west-3 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.eu-west-3.amazonaws.com"
}
},
"params": {
"Region": "eu-west-3",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region me-south-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.me-south-1.amazonaws.com"
}
},
"params": {
"Region": "me-south-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region sa-east-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.sa-east-1.amazonaws.com"
}
},
"params": {
"Region": "sa-east-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region sa-east-1 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.sa-east-1.api.aws"
}
},
"params": {
"Region": "sa-east-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-east-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-east-1.amazonaws.com"
}
},
"params": {
"Region": "us-east-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-east-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-east-1.amazonaws.com"
}
},
"params": {
"Region": "us-east-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-east-1 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-east-1.api.aws"
}
},
"params": {
"Region": "us-east-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-east-2 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-east-2.amazonaws.com"
}
},
"params": {
"Region": "us-east-2",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-east-2 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-east-2.amazonaws.com"
}
},
"params": {
"Region": "us-east-2",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-east-2 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-east-2.api.aws"
}
},
"params": {
"Region": "us-east-2",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-west-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-west-1.amazonaws.com"
}
},
"params": {
"Region": "us-west-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-west-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-west-1.amazonaws.com"
}
},
"params": {
"Region": "us-west-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-west-2 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-west-2.amazonaws.com"
}
},
"params": {
"Region": "us-west-2",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-west-2 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-west-2.amazonaws.com"
}
},
"params": {
"Region": "us-west-2",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-west-2 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-west-2.api.aws"
}
},
"params": {
"Region": "us-west-2",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-east-1 with FIPS enabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-east-1.api.aws"
}
},
"params": {
"Region": "us-east-1",
"UseFIPS": true,
"UseDualStack": true
}
},
{
"documentation": "For region cn-north-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.cn-north-1.amazonaws.com.cn"
}
},
"params": {
"Region": "cn-north-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region cn-northwest-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.cn-northwest-1.amazonaws.com.cn"
}
},
"params": {
"Region": "cn-northwest-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region cn-north-1 with FIPS enabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.cn-north-1.api.amazonwebservices.com.cn"
}
},
"params": {
"Region": "cn-north-1",
"UseFIPS": true,
"UseDualStack": true
}
},
{
"documentation": "For region cn-north-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.cn-north-1.amazonaws.com.cn"
}
},
"params": {
"Region": "cn-north-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region cn-north-1 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.cn-north-1.api.amazonwebservices.com.cn"
}
},
"params": {
"Region": "cn-north-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-gov-east-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-gov-east-1.amazonaws.com"
}
},
"params": {
"Region": "us-gov-east-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-gov-west-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-gov-west-1.amazonaws.com"
}
},
"params": {
"Region": "us-gov-west-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-gov-east-1 with FIPS enabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-gov-east-1.api.aws"
}
},
"params": {
"Region": "us-gov-east-1",
"UseFIPS": true,
"UseDualStack": true
}
},
{
"documentation": "For region us-gov-east-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-gov-east-1.amazonaws.com"
}
},
"params": {
"Region": "us-gov-east-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-gov-east-1 with FIPS disabled and DualStack enabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-gov-east-1.api.aws"
}
},
"params": {
"Region": "us-gov-east-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-iso-east-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-iso-east-1.c2s.ic.gov"
}
},
"params": {
"Region": "us-iso-east-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-iso-west-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-iso-west-1.c2s.ic.gov"
}
},
"params": {
"Region": "us-iso-west-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-iso-east-1 with FIPS enabled and DualStack enabled",
"expect": {
"error": "FIPS and DualStack are enabled, but this partition does not support one or both"
},
"params": {
"Region": "us-iso-east-1",
"UseFIPS": true,
"UseDualStack": true
}
},
{
"documentation": "For region us-iso-east-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-iso-east-1.c2s.ic.gov"
}
},
"params": {
"Region": "us-iso-east-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-iso-east-1 with FIPS disabled and DualStack enabled",
"expect": {
"error": "DualStack is enabled but this partition does not support DualStack"
},
"params": {
"Region": "us-iso-east-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For region us-isob-east-1 with FIPS disabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2.us-isob-east-1.sc2s.sgov.gov"
}
},
"params": {
"Region": "us-isob-east-1",
"UseFIPS": false,
"UseDualStack": false
}
},
{
"documentation": "For region us-isob-east-1 with FIPS enabled and DualStack enabled",
"expect": {
"error": "FIPS and DualStack are enabled, but this partition does not support one or both"
},
"params": {
"Region": "us-isob-east-1",
"UseFIPS": true,
"UseDualStack": true
}
},
{
"documentation": "For region us-isob-east-1 with FIPS enabled and DualStack disabled",
"expect": {
"endpoint": {
"url": "https://ec2-fips.us-isob-east-1.sc2s.sgov.gov"
}
},
"params": {
"Region": "us-isob-east-1",
"UseFIPS": true,
"UseDualStack": false
}
},
{
"documentation": "For region us-isob-east-1 with FIPS disabled and DualStack enabled",
"expect": {
"error": "DualStack is enabled but this partition does not support DualStack"
},
"params": {
"Region": "us-isob-east-1",
"UseFIPS": false,
"UseDualStack": true
}
},
{
"documentation": "For custom endpoint with region set and fips disabled and dualstack disabled",
"expect": {
"endpoint": {
"url": "https://example.com"
}
},
"params": {
"Region": "us-east-1",
"UseFIPS": false,
"UseDualStack": false,
"Endpoint": "https://example.com"
}
},
{
"documentation": "For custom endpoint with region not set and fips disabled and dualstack disabled",
"expect": {
"endpoint": {
"url": "https://example.com"
}
},
"params": {
"UseFIPS": false,
"UseDualStack": false,
"Endpoint": "https://example.com"
}
},
{
"documentation": "For custom endpoint with fips enabled and dualstack disabled",
"expect": {
"error": "Invalid Configuration: FIPS and custom endpoint are not supported"
},
"params": {
"Region": "us-east-1",
"UseFIPS": true,
"UseDualStack": false,
"Endpoint": "https://example.com"
}
},
{
"documentation": "For custom endpoint with fips disabled and dualstack enabled",
"expect": {
"error": "Invalid Configuration: Dualstack and custom endpoint are not supported"
},
"params": {
"Region": "us-east-1",
"UseFIPS": false,
"UseDualStack": true,
"Endpoint": "https://example.com"
}
},
{
"documentation": "Missing region",
"expect": {
"error": "Invalid Configuration: Missing Region"
}
}
],
"version": "1.0"
}
File diff suppressed because it is too large Load Diff
-885
View File
@@ -1,885 +0,0 @@
{
"pagination": {
"DescribeAccountAttributes": {
"result_key": "AccountAttributes"
},
"DescribeAddressTransfers": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "AddressTransfers"
},
"DescribeAddresses": {
"result_key": "Addresses"
},
"DescribeAddressesAttribute": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Addresses"
},
"DescribeAvailabilityZones": {
"result_key": "AvailabilityZones"
},
"DescribeAwsNetworkPerformanceMetricSubscriptions": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Subscriptions"
},
"DescribeBundleTasks": {
"result_key": "BundleTasks"
},
"DescribeByoipCidrs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ByoipCidrs"
},
"DescribeCapacityBlockOfferings": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "CapacityBlockOfferings"
},
"DescribeCapacityReservationFleets": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "CapacityReservationFleets"
},
"DescribeCapacityReservations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "CapacityReservations"
},
"DescribeCarrierGateways": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "CarrierGateways"
},
"DescribeClassicLinkInstances": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Instances"
},
"DescribeClientVpnAuthorizationRules": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "AuthorizationRules"
},
"DescribeClientVpnConnections": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Connections"
},
"DescribeClientVpnEndpoints": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ClientVpnEndpoints"
},
"DescribeClientVpnRoutes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Routes"
},
"DescribeClientVpnTargetNetworks": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ClientVpnTargetNetworks"
},
"DescribeCoipPools": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "CoipPools"
},
"DescribeConversionTasks": {
"result_key": "ConversionTasks"
},
"DescribeCustomerGateways": {
"result_key": "CustomerGateways"
},
"DescribeDhcpOptions": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "DhcpOptions"
},
"DescribeEgressOnlyInternetGateways": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "EgressOnlyInternetGateways"
},
"DescribeExportImageTasks": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ExportImageTasks"
},
"DescribeExportTasks": {
"result_key": "ExportTasks"
},
"DescribeFastLaunchImages": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "FastLaunchImages"
},
"DescribeFastSnapshotRestores": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "FastSnapshotRestores"
},
"DescribeFleets": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Fleets"
},
"DescribeFlowLogs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "FlowLogs"
},
"DescribeFpgaImages": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "FpgaImages"
},
"DescribeHostReservationOfferings": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "OfferingSet"
},
"DescribeHostReservations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "HostReservationSet"
},
"DescribeHosts": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Hosts"
},
"DescribeIamInstanceProfileAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IamInstanceProfileAssociations"
},
"DescribeImages": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Images"
},
"DescribeImportImageTasks": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ImportImageTasks"
},
"DescribeImportSnapshotTasks": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ImportSnapshotTasks"
},
"DescribeInstanceConnectEndpoints": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceConnectEndpoints"
},
"DescribeInstanceCreditSpecifications": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceCreditSpecifications"
},
"DescribeInstanceEventWindows": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceEventWindows"
},
"DescribeInstanceStatus": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceStatuses"
},
"DescribeInstanceTopology": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Instances"
},
"DescribeInstanceTypeOfferings": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceTypeOfferings"
},
"DescribeInstanceTypes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceTypes"
},
"DescribeInstances": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Reservations"
},
"DescribeInternetGateways": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InternetGateways"
},
"DescribeIpamPools": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamPools"
},
"DescribeIpamResourceDiscoveries": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamResourceDiscoveries"
},
"DescribeIpamResourceDiscoveryAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamResourceDiscoveryAssociations"
},
"DescribeIpamScopes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamScopes"
},
"DescribeIpams": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Ipams"
},
"DescribeIpv6Pools": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Ipv6Pools"
},
"DescribeKeyPairs": {
"result_key": "KeyPairs"
},
"DescribeLaunchTemplateVersions": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LaunchTemplateVersions"
},
"DescribeLaunchTemplates": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LaunchTemplates"
},
"DescribeLocalGatewayRouteTableVirtualInterfaceGroupAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LocalGatewayRouteTableVirtualInterfaceGroupAssociations"
},
"DescribeLocalGatewayRouteTableVpcAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LocalGatewayRouteTableVpcAssociations"
},
"DescribeLocalGatewayRouteTables": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LocalGatewayRouteTables"
},
"DescribeLocalGatewayVirtualInterfaceGroups": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LocalGatewayVirtualInterfaceGroups"
},
"DescribeLocalGatewayVirtualInterfaces": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LocalGatewayVirtualInterfaces"
},
"DescribeLocalGateways": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LocalGateways"
},
"DescribeMacHosts": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "MacHosts"
},
"DescribeManagedPrefixLists": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "PrefixLists"
},
"DescribeMovingAddresses": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "MovingAddressStatuses"
},
"DescribeNatGateways": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NatGateways"
},
"DescribeNetworkAcls": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkAcls"
},
"DescribeNetworkInsightsAccessScopeAnalyses": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkInsightsAccessScopeAnalyses"
},
"DescribeNetworkInsightsAccessScopes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkInsightsAccessScopes"
},
"DescribeNetworkInsightsAnalyses": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkInsightsAnalyses"
},
"DescribeNetworkInsightsPaths": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkInsightsPaths"
},
"DescribeNetworkInterfacePermissions": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkInterfacePermissions"
},
"DescribeNetworkInterfaces": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "NetworkInterfaces"
},
"DescribePlacementGroups": {
"result_key": "PlacementGroups"
},
"DescribePrefixLists": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "PrefixLists"
},
"DescribePrincipalIdFormat": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Principals"
},
"DescribePublicIpv4Pools": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "PublicIpv4Pools"
},
"DescribeRegions": {
"result_key": "Regions"
},
"DescribeReplaceRootVolumeTasks": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ReplaceRootVolumeTasks"
},
"DescribeReservedInstances": {
"result_key": "ReservedInstances"
},
"DescribeReservedInstancesListings": {
"result_key": "ReservedInstancesListings"
},
"DescribeReservedInstancesModifications": {
"input_token": "NextToken",
"output_token": "NextToken",
"result_key": "ReservedInstancesModifications"
},
"DescribeReservedInstancesOfferings": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ReservedInstancesOfferings"
},
"DescribeRouteTables": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "RouteTables"
},
"DescribeScheduledInstanceAvailability": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ScheduledInstanceAvailabilitySet"
},
"DescribeScheduledInstances": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ScheduledInstanceSet"
},
"DescribeSecurityGroupRules": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SecurityGroupRules"
},
"DescribeSecurityGroups": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SecurityGroups"
},
"DescribeSnapshotTierStatus": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SnapshotTierStatuses"
},
"DescribeSnapshots": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Snapshots"
},
"DescribeSpotFleetRequests": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SpotFleetRequestConfigs"
},
"DescribeSpotInstanceRequests": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SpotInstanceRequests"
},
"DescribeSpotPriceHistory": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SpotPriceHistory"
},
"DescribeStaleSecurityGroups": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "StaleSecurityGroupSet"
},
"DescribeStoreImageTasks": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "StoreImageTaskResults"
},
"DescribeSubnets": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Subnets"
},
"DescribeTags": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Tags"
},
"DescribeTrafficMirrorFilters": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TrafficMirrorFilters"
},
"DescribeTrafficMirrorSessions": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TrafficMirrorSessions"
},
"DescribeTrafficMirrorTargets": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TrafficMirrorTargets"
},
"DescribeTransitGatewayAttachments": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayAttachments"
},
"DescribeTransitGatewayConnectPeers": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayConnectPeers"
},
"DescribeTransitGatewayConnects": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayConnects"
},
"DescribeTransitGatewayMulticastDomains": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayMulticastDomains"
},
"DescribeTransitGatewayPeeringAttachments": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayPeeringAttachments"
},
"DescribeTransitGatewayPolicyTables": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayPolicyTables"
},
"DescribeTransitGatewayRouteTableAnnouncements": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayRouteTableAnnouncements"
},
"DescribeTransitGatewayRouteTables": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayRouteTables"
},
"DescribeTransitGatewayVpcAttachments": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayVpcAttachments"
},
"DescribeTransitGateways": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGateways"
},
"DescribeTrunkInterfaceAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InterfaceAssociations"
},
"DescribeVerifiedAccessEndpoints": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VerifiedAccessEndpoints"
},
"DescribeVerifiedAccessGroups": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VerifiedAccessGroups"
},
"DescribeVerifiedAccessInstanceLoggingConfigurations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "LoggingConfigurations"
},
"DescribeVerifiedAccessInstances": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VerifiedAccessInstances"
},
"DescribeVerifiedAccessTrustProviders": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VerifiedAccessTrustProviders"
},
"DescribeVolumeStatus": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VolumeStatuses"
},
"DescribeVolumes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Volumes"
},
"DescribeVolumesModifications": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VolumesModifications"
},
"DescribeVpcClassicLinkDnsSupport": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Vpcs"
},
"DescribeVpcEndpointConnectionNotifications": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ConnectionNotificationSet"
},
"DescribeVpcEndpointConnections": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VpcEndpointConnections"
},
"DescribeVpcEndpointServiceConfigurations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "ServiceConfigurations"
},
"DescribeVpcEndpointServicePermissions": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "AllowedPrincipals"
},
"DescribeVpcEndpoints": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VpcEndpoints"
},
"DescribeVpcPeeringConnections": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VpcPeeringConnections"
},
"DescribeVpcs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Vpcs"
},
"DescribeVpnConnections": {
"result_key": "VpnConnections"
},
"DescribeVpnGateways": {
"result_key": "VpnGateways"
},
"GetAssociatedIpv6PoolCidrs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Ipv6CidrAssociations"
},
"GetAwsNetworkPerformanceData": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "DataResponses"
},
"GetGroupsForCapacityReservation": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "CapacityReservationGroups"
},
"GetInstanceTypesFromInstanceRequirements": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "InstanceTypes"
},
"GetIpamAddressHistory": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "HistoryRecords"
},
"GetIpamDiscoveredAccounts": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamDiscoveredAccounts"
},
"GetIpamDiscoveredResourceCidrs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamDiscoveredResourceCidrs"
},
"GetIpamPoolAllocations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamPoolAllocations"
},
"GetIpamPoolCidrs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamPoolCidrs"
},
"GetIpamResourceCidrs": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "IpamResourceCidrs"
},
"GetManagedPrefixListAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "PrefixListAssociations"
},
"GetManagedPrefixListEntries": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Entries"
},
"GetNetworkInsightsAccessScopeAnalysisFindings": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "AnalysisFindings"
},
"GetSecurityGroupsForVpc": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SecurityGroupForVpcs"
},
"GetSpotPlacementScores": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "SpotPlacementScores"
},
"GetTransitGatewayAttachmentPropagations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayAttachmentPropagations"
},
"GetTransitGatewayMulticastDomainAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "MulticastDomainAssociations"
},
"GetTransitGatewayPolicyTableAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Associations"
},
"GetTransitGatewayPrefixListReferences": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayPrefixListReferences"
},
"GetTransitGatewayRouteTableAssociations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Associations"
},
"GetTransitGatewayRouteTablePropagations": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "TransitGatewayRouteTablePropagations"
},
"GetVpnConnectionDeviceTypes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "VpnConnectionDeviceTypes"
},
"ListImagesInRecycleBin": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Images"
},
"ListSnapshotsInRecycleBin": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Snapshots"
},
"SearchLocalGatewayRoutes": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "Routes"
},
"SearchTransitGatewayMulticastGroups": {
"input_token": "NextToken",
"limit_key": "MaxResults",
"output_token": "NextToken",
"result_key": "MulticastGroups"
}
}
}
-31
View File
@@ -1,31 +0,0 @@
{
"version": 2,
"testCases": [
{
"id": "DescribeRegionsSuccess",
"operationName": "DescribeRegions",
"input": {},
"expectation": {
"success": {}
},
"config": {
"region": "us-west-2"
}
},
{
"id": "DescribeInstancesFailure",
"operationName": "DescribeInstances",
"input": {
"InstanceIds": [
"i-12345678"
]
},
"expectation": {
"failure": {}
},
"config": {
"region": "us-west-2"
}
}
]
}
-20
View File
@@ -1,20 +0,0 @@
{
"version": 1,
"defaultRegion": "us-west-2",
"testCases": [
{
"operationName": "DescribeRegions",
"input": {},
"errorExpectedFromService": false
},
{
"operationName": "DescribeInstances",
"input": {
"InstanceIds": [
"i-12345678"
]
},
"errorExpectedFromService": true
}
]
}
-726
View File
@@ -1,726 +0,0 @@
{
"version": 2,
"waiters": {
"InstanceExists": {
"delay": 5,
"maxAttempts": 40,
"operation": "DescribeInstances",
"acceptors": [
{
"matcher": "path",
"expected": true,
"argument": "length(Reservations[]) > `0`",
"state": "success"
},
{
"matcher": "error",
"expected": "InvalidInstanceID.NotFound",
"state": "retry"
}
]
},
"BundleTaskComplete": {
"delay": 15,
"operation": "DescribeBundleTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "complete",
"matcher": "pathAll",
"state": "success",
"argument": "BundleTasks[].State"
},
{
"expected": "failed",
"matcher": "pathAny",
"state": "failure",
"argument": "BundleTasks[].State"
}
]
},
"ConversionTaskCancelled": {
"delay": 15,
"operation": "DescribeConversionTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "cancelled",
"matcher": "pathAll",
"state": "success",
"argument": "ConversionTasks[].State"
}
]
},
"ConversionTaskCompleted": {
"delay": 15,
"operation": "DescribeConversionTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "completed",
"matcher": "pathAll",
"state": "success",
"argument": "ConversionTasks[].State"
},
{
"expected": "cancelled",
"matcher": "pathAny",
"state": "failure",
"argument": "ConversionTasks[].State"
},
{
"expected": "cancelling",
"matcher": "pathAny",
"state": "failure",
"argument": "ConversionTasks[].State"
}
]
},
"ConversionTaskDeleted": {
"delay": 15,
"operation": "DescribeConversionTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "deleted",
"matcher": "pathAll",
"state": "success",
"argument": "ConversionTasks[].State"
}
]
},
"CustomerGatewayAvailable": {
"delay": 15,
"operation": "DescribeCustomerGateways",
"maxAttempts": 40,
"acceptors": [
{
"expected": "available",
"matcher": "pathAll",
"state": "success",
"argument": "CustomerGateways[].State"
},
{
"expected": "deleted",
"matcher": "pathAny",
"state": "failure",
"argument": "CustomerGateways[].State"
},
{
"expected": "deleting",
"matcher": "pathAny",
"state": "failure",
"argument": "CustomerGateways[].State"
}
]
},
"ExportTaskCancelled": {
"delay": 15,
"operation": "DescribeExportTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "cancelled",
"matcher": "pathAll",
"state": "success",
"argument": "ExportTasks[].State"
}
]
},
"ExportTaskCompleted": {
"delay": 15,
"operation": "DescribeExportTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "completed",
"matcher": "pathAll",
"state": "success",
"argument": "ExportTasks[].State"
}
]
},
"ImageExists": {
"operation": "DescribeImages",
"maxAttempts": 40,
"delay": 15,
"acceptors": [
{
"matcher": "path",
"expected": true,
"argument": "length(Images[]) > `0`",
"state": "success"
},
{
"matcher": "error",
"expected": "InvalidAMIID.NotFound",
"state": "retry"
}
]
},
"ImageAvailable": {
"operation": "DescribeImages",
"maxAttempts": 40,
"delay": 15,
"acceptors": [
{
"state": "success",
"matcher": "pathAll",
"argument": "Images[].State",
"expected": "available"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "Images[].State",
"expected": "failed"
}
]
},
"InstanceRunning": {
"delay": 15,
"operation": "DescribeInstances",
"maxAttempts": 40,
"acceptors": [
{
"expected": "running",
"matcher": "pathAll",
"state": "success",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "shutting-down",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "terminated",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "stopping",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
},
{
"matcher": "error",
"expected": "InvalidInstanceID.NotFound",
"state": "retry"
}
]
},
"InstanceStatusOk": {
"operation": "DescribeInstanceStatus",
"maxAttempts": 40,
"delay": 15,
"acceptors": [
{
"state": "success",
"matcher": "pathAll",
"argument": "InstanceStatuses[].InstanceStatus.Status",
"expected": "ok"
},
{
"matcher": "error",
"expected": "InvalidInstanceID.NotFound",
"state": "retry"
}
]
},
"InstanceStopped": {
"delay": 15,
"operation": "DescribeInstances",
"maxAttempts": 40,
"acceptors": [
{
"expected": "stopped",
"matcher": "pathAll",
"state": "success",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "pending",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "terminated",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
}
]
},
"InstanceTerminated": {
"delay": 15,
"operation": "DescribeInstances",
"maxAttempts": 40,
"acceptors": [
{
"expected": "terminated",
"matcher": "pathAll",
"state": "success",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "pending",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
},
{
"expected": "stopping",
"matcher": "pathAny",
"state": "failure",
"argument": "Reservations[].Instances[].State.Name"
}
]
},
"InternetGatewayExists": {
"operation": "DescribeInternetGateways",
"delay": 5,
"maxAttempts": 6,
"acceptors": [
{
"expected": true,
"matcher": "path",
"state": "success",
"argument": "length(InternetGateways[].InternetGatewayId) > `0`"
},
{
"expected": "InvalidInternetGateway.NotFound",
"matcher": "error",
"state": "retry"
}
]
},
"KeyPairExists": {
"operation": "DescribeKeyPairs",
"delay": 5,
"maxAttempts": 6,
"acceptors": [
{
"expected": true,
"matcher": "path",
"state": "success",
"argument": "length(KeyPairs[].KeyName) > `0`"
},
{
"expected": "InvalidKeyPair.NotFound",
"matcher": "error",
"state": "retry"
}
]
},
"NatGatewayAvailable": {
"operation": "DescribeNatGateways",
"delay": 15,
"maxAttempts": 40,
"acceptors": [
{
"state": "success",
"matcher": "pathAll",
"argument": "NatGateways[].State",
"expected": "available"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "NatGateways[].State",
"expected": "failed"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "NatGateways[].State",
"expected": "deleting"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "NatGateways[].State",
"expected": "deleted"
},
{
"state": "retry",
"matcher": "error",
"expected": "NatGatewayNotFound"
}
]
},
"NatGatewayDeleted": {
"operation": "DescribeNatGateways",
"delay": 15,
"maxAttempts": 40,
"acceptors": [
{
"state": "success",
"matcher": "pathAll",
"argument": "NatGateways[].State",
"expected": "deleted"
},
{
"state": "success",
"matcher": "error",
"expected": "NatGatewayNotFound"
}
]
},
"NetworkInterfaceAvailable": {
"operation": "DescribeNetworkInterfaces",
"delay": 20,
"maxAttempts": 10,
"acceptors": [
{
"expected": "available",
"matcher": "pathAll",
"state": "success",
"argument": "NetworkInterfaces[].Status"
},
{
"expected": "InvalidNetworkInterfaceID.NotFound",
"matcher": "error",
"state": "failure"
}
]
},
"PasswordDataAvailable": {
"operation": "GetPasswordData",
"maxAttempts": 40,
"delay": 15,
"acceptors": [
{
"state": "success",
"matcher": "path",
"argument": "length(PasswordData) > `0`",
"expected": true
}
]
},
"SnapshotCompleted": {
"delay": 15,
"operation": "DescribeSnapshots",
"maxAttempts": 40,
"acceptors": [
{
"expected": "completed",
"matcher": "pathAll",
"state": "success",
"argument": "Snapshots[].State"
},
{
"expected": "error",
"matcher": "pathAny",
"state": "failure",
"argument": "Snapshots[].State"
}
]
},
"SnapshotImported": {
"delay": 15,
"operation": "DescribeImportSnapshotTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "completed",
"matcher": "pathAll",
"state": "success",
"argument": "ImportSnapshotTasks[].SnapshotTaskDetail.Status"
},
{
"expected": "error",
"matcher": "pathAny",
"state": "failure",
"argument": "ImportSnapshotTasks[].SnapshotTaskDetail.Status"
}
]
},
"SecurityGroupExists": {
"operation": "DescribeSecurityGroups",
"delay": 5,
"maxAttempts": 6,
"acceptors": [
{
"expected": true,
"matcher": "path",
"state": "success",
"argument": "length(SecurityGroups[].GroupId) > `0`"
},
{
"expected": "InvalidGroup.NotFound",
"matcher": "error",
"state": "retry"
}
]
},
"SpotInstanceRequestFulfilled": {
"operation": "DescribeSpotInstanceRequests",
"maxAttempts": 40,
"delay": 15,
"acceptors": [
{
"state": "success",
"matcher": "pathAll",
"argument": "SpotInstanceRequests[].Status.Code",
"expected": "fulfilled"
},
{
"state": "success",
"matcher": "pathAll",
"argument": "SpotInstanceRequests[].Status.Code",
"expected": "request-canceled-and-instance-running"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "SpotInstanceRequests[].Status.Code",
"expected": "schedule-expired"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "SpotInstanceRequests[].Status.Code",
"expected": "canceled-before-fulfillment"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "SpotInstanceRequests[].Status.Code",
"expected": "bad-parameters"
},
{
"state": "failure",
"matcher": "pathAny",
"argument": "SpotInstanceRequests[].Status.Code",
"expected": "system-error"
},
{
"state": "retry",
"matcher": "error",
"expected": "InvalidSpotInstanceRequestID.NotFound"
}
]
},
"StoreImageTaskComplete": {
"delay": 5,
"operation": "DescribeStoreImageTasks",
"maxAttempts": 40,
"acceptors": [
{
"expected": "Completed",
"matcher": "pathAll",
"state": "success",
"argument": "StoreImageTaskResults[].StoreTaskState"
},
{
"expected": "Failed",
"matcher": "pathAny",
"state": "failure",
"argument": "StoreImageTaskResults[].StoreTaskState"
},
{
"expected": "InProgress",
"matcher": "pathAny",
"state": "retry",
"argument": "StoreImageTaskResults[].StoreTaskState"
}
]
},
"SubnetAvailable": {
"delay": 15,
"operation": "DescribeSubnets",
"maxAttempts": 40,
"acceptors": [
{
"expected": "available",
"matcher": "pathAll",
"state": "success",
"argument": "Subnets[].State"
}
]
},
"SystemStatusOk": {
"operation": "DescribeInstanceStatus",
"maxAttempts": 40,
"delay": 15,
"acceptors": [
{
"state": "success",
"matcher": "pathAll",
"argument": "InstanceStatuses[].SystemStatus.Status",
"expected": "ok"
}
]
},
"VolumeAvailable": {
"delay": 15,
"operation": "DescribeVolumes",
"maxAttempts": 40,
"acceptors": [
{
"expected": "available",
"matcher": "pathAll",
"state": "success",
"argument": "Volumes[].State"
},
{
"expected": "deleted",
"matcher": "pathAny",
"state": "failure",
"argument": "Volumes[].State"
}
]
},
"VolumeDeleted": {
"delay": 15,
"operation": "DescribeVolumes",
"maxAttempts": 40,
"acceptors": [
{
"expected": "deleted",
"matcher": "pathAll",
"state": "success",
"argument": "Volumes[].State"
},
{
"matcher": "error",
"expected": "InvalidVolume.NotFound",
"state": "success"
}
]
},
"VolumeInUse": {
"delay": 15,
"operation": "DescribeVolumes",
"maxAttempts": 40,
"acceptors": [
{
"expected": "in-use",
"matcher": "pathAll",
"state": "success",
"argument": "Volumes[].State"
},
{
"expected": "deleted",
"matcher": "pathAny",
"state": "failure",
"argument": "Volumes[].State"
}
]
},
"VpcAvailable": {
"delay": 15,
"operation": "DescribeVpcs",
"maxAttempts": 40,
"acceptors": [
{
"expected": "available",
"matcher": "pathAll",
"state": "success",
"argument": "Vpcs[].State"
}
]
},
"VpcExists": {
"operation": "DescribeVpcs",
"delay": 1,
"maxAttempts": 5,
"acceptors": [
{
"matcher": "status",
"expected": 200,
"state": "success"
},
{
"matcher": "error",
"expected": "InvalidVpcID.NotFound",
"state": "retry"
}
]
},
"VpnConnectionAvailable": {
"delay": 15,
"operation": "DescribeVpnConnections",
"maxAttempts": 40,
"acceptors": [
{
"expected": "available",
"matcher": "pathAll",
"state": "success",
"argument": "VpnConnections[].State"
},
{
"expected": "deleting",
"matcher": "pathAny",
"state": "failure",
"argument": "VpnConnections[].State"
},
{
"expected": "deleted",
"matcher": "pathAny",
"state": "failure",
"argument": "VpnConnections[].State"
}
]
},
"VpnConnectionDeleted": {
"delay": 15,
"operation": "DescribeVpnConnections",
"maxAttempts": 40,
"acceptors": [
{
"expected": "deleted",
"matcher": "pathAll",
"state": "success",
"argument": "VpnConnections[].State"
},
{
"expected": "pending",
"matcher": "pathAny",
"state": "failure",
"argument": "VpnConnections[].State"
}
]
},
"VpcPeeringConnectionExists": {
"delay": 15,
"operation": "DescribeVpcPeeringConnections",
"maxAttempts": 40,
"acceptors": [
{
"matcher": "status",
"expected": 200,
"state": "success"
},
{
"matcher": "error",
"expected": "InvalidVpcPeeringConnectionID.NotFound",
"state": "retry"
}
]
},
"VpcPeeringConnectionDeleted": {
"delay": 15,
"operation": "DescribeVpcPeeringConnections",
"maxAttempts": 40,
"acceptors": [
{
"expected": "deleted",
"matcher": "pathAll",
"state": "success",
"argument": "VpcPeeringConnections[].Status.Code"
},
{
"matcher": "error",
"expected": "InvalidVpcPeeringConnectionID.NotFound",
"state": "success"
}
]
}
}
}
+1 -1
View File
@@ -5,7 +5,7 @@ set -euo pipefail
# Usage: scripts/check-coverage.sh <coverprofile> [quiet]
PROFILE="${1:?Usage: check-coverage.sh <coverprofile> [quiet]}"
MIN=68.0
MIN=70.0
QUIET="${2:-}"
if [[ ! -s "$PROFILE" ]]; then
+2 -2
View File
@@ -15,7 +15,7 @@
#
# Optional:
# SSH_KEY (default: ~/.ssh/spinifex-key)
# SSH_USER (default: ec2-user)
# SSH_USER (default: ubuntu)
set -euo pipefail
@@ -26,7 +26,7 @@ QWEN_IP="${QWEN_IP:?set QWEN_IP}"
VIDEO_PATH="${VIDEO_PATH:?set VIDEO_PATH}"
SSH_KEY="${SSH_KEY:-$HOME/.ssh/spinifex-key}"
SSH_USER="${SSH_USER:-ec2-user}"
SSH_USER="${SSH_USER:-ubuntu}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env bash
# ecs-node-bringup.sh — launch a Spinifex ECS container-instance VM.
#
# ECS container instances are plain EC2 instances booted from the spinifex-ecs-node
# AMI (Alpine + containerd + ecs-agent), AWS-faithful: there is no ECS-specific
# launch API. This script renders the cloud-init user-data that seeds the agent's
# control-plane config + IAM credentials into /etc/spinifex-ecs/agent.env and the
# gateway TLS CA into /etc/spinifex-ecs/gateway-ca.pem, then runs the instance.
#
# The agent then registers, heartbeats, polls assignments, and reports task state
# entirely through the AWS gateway over TLS+SigV4 — it never touches NATS.
#
# Usage:
# AWS_PROFILE=spinifex scripts/ecs-node-bringup.sh <cluster-name>
#
# Env overrides:
# GATEWAY_IP gateway IP the guest dials (default: host WAN-bridge IP)
# GATEWAY_CA host path to the gateway CA PEM (default /etc/spinifex/ca.pem)
# AMI_ID ECS-node AMI (default: latest spinifex-ecs-node)
# SUBNET_ID / SG_ID / KEY_NAME / INSTANCE_TYPE (default: first available / t3.small)
set -euo pipefail
CLUSTER="${1:?usage: ecs-node-bringup.sh <cluster-name>}"
PROFILE="${AWS_PROFILE:-spinifex}"
REGION="$(aws configure get region --profile "$PROFILE")"
# VPC guests NAT out through the WAN and reach host-run services on the host's
# WAN-bridge IP; the mgmt bridge (10.15.8.x) is not routable from a guest netns.
# Default to the br-wan address, falling back to the mgmt IP if it can't be read.
default_gateway_ip() {
ip -4 addr show br-wan 2>/dev/null | awk '/inet /{print $2; exit}' | cut -d/ -f1
}
GATEWAY_IP="${GATEWAY_IP:-$(default_gateway_ip)}"
GATEWAY_IP="${GATEWAY_IP:-10.15.8.1}"
GATEWAY_CA="${GATEWAY_CA:-/etc/spinifex/ca.pem}"
INSTANCE_TYPE="${INSTANCE_TYPE:-t3.small}"
awscli() { aws --profile "$PROFILE" "$@"; }
AMI_ID="${AMI_ID:-$(awscli ec2 describe-images \
--filters 'Name=name,Values=spinifex-ecs-node' \
--query 'sort_by(Images,&CreationDate)[-1].ImageId' --output text)}"
SUBNET_ID="${SUBNET_ID:-$(awscli ec2 describe-subnets \
--query 'Subnets[0].SubnetId' --output text)}"
SG_ID="${SG_ID:-$(awscli ec2 describe-security-groups \
--query 'SecurityGroups[0].GroupId' --output text)}"
KEY_NAME="${KEY_NAME:-$(awscli ec2 describe-key-pairs \
--query 'KeyPairs[0].KeyName' --output text)}"
ACCESS_KEY="$(aws configure get aws_access_key_id --profile "$PROFILE")"
SECRET_KEY="$(aws configure get aws_secret_access_key --profile "$PROFILE")"
echo "cluster=$CLUSTER ami=$AMI_ID subnet=$SUBNET_ID sg=$SG_ID key=$KEY_NAME gw=https://$GATEWAY_IP:9999" >&2
# Indent a file's content by six spaces for a cloud-init "content: |" block.
indent6() { sed 's/^/ /' "$1"; }
USERDATA="$(cat <<EOF
#cloud-config
bootcmd:
- rm -f /etc/resolv.conf
- printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > /etc/resolv.conf
write_files:
- path: /etc/spinifex-ecs/agent.env
permissions: '0600'
content: |
ECS_GATEWAY_URL=https://$GATEWAY_IP:9999
ECS_GATEWAY_CA=/etc/spinifex-ecs/gateway-ca.pem
ECS_REGION=$REGION
ECS_CLUSTER=$CLUSTER
ECS_ACCESS_KEY=$ACCESS_KEY
ECS_SECRET_KEY=$SECRET_KEY
- path: /etc/spinifex-ecs/gateway-ca.pem
permissions: '0644'
content: |
$(indent6 "$GATEWAY_CA")
EOF
)"
INSTANCE_ID="$(awscli ec2 run-instances \
--image-id "$AMI_ID" --instance-type "$INSTANCE_TYPE" \
--key-name "$KEY_NAME" --subnet-id "$SUBNET_ID" --security-group-ids "$SG_ID" \
--user-data "$USERDATA" \
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=ecs-node-$CLUSTER}]" \
--query 'Instances[0].InstanceId' --output text)"
echo "$INSTANCE_ID"
@@ -0,0 +1,21 @@
{
"cniVersion": "1.0.0",
"name": "ecs-bridge",
"plugins": [
{
"type": "bridge",
"bridge": "ecs0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"ranges": [
[{ "subnet": "172.31.0.0/16" }]
],
"routes": [{ "dst": "0.0.0.0/0" }]
}
},
{ "type": "portmap", "capabilities": { "portMappings": true } },
{ "type": "firewall" }
]
}
+34
View File
@@ -0,0 +1,34 @@
#!/sbin/openrc-run
# ecs-agent — Spinifex ECS container-instance agent. Resolves host identity from
# IMDSv2, registers + heartbeats + polls task assignments through the AWS gateway
# over TLS+SigV4 (no NATS), and pulls task images through containerd.
#
# Static config (gateway URL, CA, region, cluster, seeded IAM creds, containerd
# socket) comes from /etc/spinifex-ecs/agent.env, dropped by cloud-init at launch.
description="Spinifex ECS container-instance agent"
command="/usr/local/bin/ecs-agent"
command_background="yes"
pidfile="/run/ecs-agent.pid"
output_log="/var/log/ecs-agent.log"
error_log="/var/log/ecs-agent.log"
start_stop_daemon_args="--wait 2000"
depend() {
need net containerd
after firewall ecs-ca-install
}
start_pre() {
# Cloud-init drops the agent config here; OpenRC injects it into the agent's
# environment. Absent file is tolerated — the agent falls back to its
# built-in defaults (cluster=default, default IMDS).
if [ -f /etc/spinifex-ecs/agent.env ]; then
# shellcheck disable=SC1091
. /etc/spinifex-ecs/agent.env
export ECS_GATEWAY_URL ECS_GATEWAY_CA ECS_REGION ECS_CLUSTER \
ECS_ACCESS_KEY ECS_SECRET_KEY ECS_IMDS_BASE ECS_CONTAINERD_SOCKET \
ECS_HEARTBEAT_INTERVAL ECS_POLL_INTERVAL
fi
}
@@ -0,0 +1,33 @@
#!/sbin/openrc-run
# ecs-ca-install — first-boot oneshot that installs the cloud-init-delivered
# gateway CA into the system trust store, so containerd trusts the internal ECR
# registry's TLS when pulling task images. Ordered before containerd + ecs-agent.
#
# The CA is delivered per-instance (not baked) at /etc/spinifex-ecs/gateway-ca.pem
# by cloud-init user-data, mirroring how EKS delivers EKS_GATEWAY_CA. No-op when
# the file is absent — containerd then relies on the stock system trust.
description="Install Spinifex gateway CA into the system trust (first boot)"
GATEWAY_CA="/etc/spinifex-ecs/gateway-ca.pem"
TRUST_DST="/usr/local/share/ca-certificates/spinifex-gateway.crt"
depend() {
before containerd ecs-agent
after localmount
}
start() {
if [ ! -f "${GATEWAY_CA}" ]; then
einfo "no gateway CA at ${GATEWAY_CA}; using stock system trust"
return 0
fi
ebegin "Installing gateway CA into system trust"
install -D -m 0644 "${GATEWAY_CA}" "${TRUST_DST}"
update-ca-certificates >/dev/null 2>&1
eend $?
}
stop() {
return 0
}
+37
View File
@@ -0,0 +1,37 @@
IMAGE_NAME="ecs-agent"
IMAGE_DESCRIPTION="Spinifex ECS container instance — Alpine + containerd + runc + cni-plugins + ecs-agent"
DISTRO="alpine"
ALPINE_VERSION="3.21.7"
# 4G root: containerd content store (pulled task images) lives under
# /var/lib/containerd on the root disk. Instances can attach a larger root EBS at
# RunInstances; this is the baked floor.
IMAGE_SIZE="4G"
# containerd-openrc ships the OpenRC service for containerd (the containerd apk
# itself has no init script). cni-plugins + the baked bridge conflist ready the
# image for Sprint 4d task networking. curl is used by the first-boot CA hook and
# IMDS probes. iproute2 supplies the full `ip` (busybox lacks `ip netns`) the
# agent uses to build awsvpc task netns; udhcpc is busybox built-in.
APK_PACKAGES="containerd containerd-openrc runc cni-plugins ca-certificates curl openrc iproute2"
# ecs-agent is the only built binary (static, FIPS boot like the host build).
BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags \"-s -w\" -o bin/ecs-agent ./cmd/ecs-agent"
INSTALL_BINARIES="bin/ecs-agent:/usr/local/bin/ecs-agent"
INSTALL_FILES="\
scripts/images/ecs-agent/ecs-agent.initd:/etc/init.d/ecs-agent \
scripts/images/ecs-agent/ecs-ca-install.initd:/etc/init.d/ecs-ca-install \
scripts/images/ecs-agent/10-bridge.conflist:/etc/cni/net.d/10-bridge.conflist"
# ecs-ca-install (first boot) installs the cloud-init gateway CA into the system
# trust before containerd starts; containerd then trusts the ECR registry TLS.
# ecs-agent registers + heartbeats + polls the AWS gateway (TLS+SigV4, no NATS)
# once net + containerd are up.
ENABLE_SERVICES="containerd ecs-ca-install ecs-agent"
SETUP_SCRIPT="scripts/images/ecs-agent/setup.sh"
AMI_NAME="spinifex-ecs-node"
SYSTEM_TAG="spinifex:managed-by=ecs"

Some files were not shown because too many files have changed in this diff Show More