Files
spinifex/internal/ecsgw/client.go
T
Josh Siviour de64d6c63d feat(ecs): agent control-plane over the AWS gateway (drop direct NATS) (#419)
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel

Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.

* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState

Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.

* feat(ecs): durable assign KV inbox + PollAssignments gateway action

Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.

At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.

* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway

The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).

- controlPlane seam over ecsgw.Client replaces the NATS publisher:
  Register (RegisterContainerInstance), SubmitTaskState
  (SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
  LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
  inbox, dispatch each assign once, ack accepted taskIDs on the next
  poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
  ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.

Tests move from the NATS publisher fake to a controlPlane fake.

* feat(ecs): cloud-init cred-seed + ecs-node bring-up script

ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.

The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).

* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring

The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:

- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
  reclaimAssignInbox; both are needed (release the task ENI AND drain the
  KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
  teardownTaskNetns wrappers were lost when the controlPlane Agent struct
  won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
  the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
  fakeCP to match the controlPlane seam.
2026-06-25 18:42:24 +10:00

113 lines
3.6 KiB
Go

// Package ecsgw is the on-VM SigV4 client the ecs-agent uses to reach the AWS
// gateway over HTTPS instead of connecting to NATS directly. It signs requests
// for service "ecs" with the instance's seeded IAM credentials, keeping the NATS
// bus host-internal (the gateway relays agent calls onto ecs.bus.* host-side).
package ecsgw
import (
"bytes"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
"github.com/mulgadc/predastore/auth"
"github.com/mulgadc/spinifex/internal/tlsconfig"
)
// targetPrefix is the X-Amz-Target service prefix the gateway's ECS dispatch
// strips to the bare action. Any prefix works (ecsActionFromTarget takes the
// suffix); this one mirrors the AWS JSON 1.1 convention.
const targetPrefix = "AmazonEC2ContainerServiceV20141113."
// Client posts SigV4-signed AWS JSON 1.1 requests to the gateway for service
// "ecs". One client is reused for register/heartbeat/state/poll.
type Client struct {
baseURL string
accessKey string
secretKey string
region string
httpClient *http.Client
}
// New builds a client. caPath optionally pins the gateway TLS CA; empty relies on
// the system trust store. region defaults to us-east-1 when empty (SigV4 requires
// a non-empty region). timeout bounds a single call — callers needing a long-poll
// pass a larger value than the default register/heartbeat timeout.
func New(baseURL, caPath, accessKey, secretKey, region string, timeout time.Duration) (*Client, error) {
if baseURL == "" {
return nil, fmt.Errorf("ecsgw: baseURL is required")
}
if accessKey == "" || secretKey == "" {
return nil, fmt.Errorf("ecsgw: access key and secret key are required")
}
if region == "" {
region = "us-east-1"
}
if timeout <= 0 {
timeout = 10 * time.Second
}
tlsCfg := &tls.Config{
MinVersion: tls.VersionTLS13,
CurvePreferences: tlsconfig.Curves,
}
if caPath != "" {
pem, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("ecsgw: read gateway CA %q: %w", caPath, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("ecsgw: gateway CA %q has no usable certificates", caPath)
}
tlsCfg.RootCAs = pool
}
return &Client{
baseURL: strings.TrimRight(baseURL, "/"),
accessKey: accessKey,
secretKey: secretKey,
region: region,
httpClient: &http.Client{
Timeout: timeout,
Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
},
}, nil
}
// Call SigV4-signs and POSTs body under X-Amz-Target "<prefix><action>" to the
// gateway root, returning the 2xx response body. Errors carry the gateway status
// and body. No retry; callers wrap as needed.
func (c *Client) Call(action string, body []byte) ([]byte, error) {
req, err := http.NewRequest(http.MethodPost, c.baseURL+"/", bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
req.Header.Set("X-Amz-Target", targetPrefix+action)
sum := sha256.Sum256(body)
if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "ecs", c.region); err != nil {
return nil, fmt.Errorf("sign request: %w", err)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, fmt.Errorf("send request: %w", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("gateway returned %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
}
return respBody, nil
}