ec2rolecreds implements aws.HandleFailRefreshCredentialsCacheStrategy, which
aws.CredentialsCache prefers over returning the refresh error. When the cached
credential is expired or within five minutes of expiring and the refresh fails,
it hands back the same access key, secret and session token with Expires pushed
a random five to fifteen minutes out, reported as a success. Every later
refresh fails the same way, so ecs-agent signs with a credential the control
plane has already rejected, indefinitely, and nothing surfaces it.
The cache picks its strategies by asserting on the provider's dynamic type, so
returning aws.CredentialsProvider from NewProvider does not disarm it. Wrap the
provider in a type whose method set omits HandleFailToRefresh, and the cache
falls through to returning the error. AdjustExpiresBy is forwarded on purpose:
it only fires under an ExpiryWindow, which no caller sets today, but dropping
it silently would change behaviour for whoever sets one.
Fixed in imdscreds rather than at the one exposed call site, so a future caller
that wraps this provider in a cache is safe by construction.
ecr-credential-provider was never affected: it calls Retrieve uncached.
Two comments claimed the hazard was ec2rolecreds capping Expires to an hour.
That cap only ever shortens a too-far-future expiry and is unrelated.
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c)
Add the ecs-agent binary that runs inside an ECS container instance: it
resolves host identity from IMDS, registers on the Layer-2 NATS bus, and
heartbeats while alive. Image pulls go through a containerd-backed
ImagePuller behind an interface, with an ECR resolver that mints
GetAuthorizationToken creds from the instance role.
Extract the IMDS-credential and ECR-auth flow out of
ecr-credential-provider into internal/imdscreds and internal/ecrauth so
the agent and the kubelet provider share one implementation.
Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2
subject builders and wire payloads, shared by the agent now and the
scheduler in Sprint 4e.
* build: pin genproto via replace to fix containerd/v2 ambiguous import
go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto
into the module graph; it still ships googleapis/rpc/status, which collides
with the split genproto/googleapis/rpc module that containerd/ttrpc needs.
The main module never imports the monolith, so a plain require bump is
dropped by 'go mod tidy'. Pin it with a replace to a post-split version so
the fix survives tidy and contributors can run 'go mod tidy' freely.
* fix(ecs): agent boot failsafes from runtime verify
- Build the ECR gateway HTTP client lazily on first image pull instead of
at boot. A missing/malformed gateway CA no longer kills the agent before
it can register + heartbeat, matching the best-effort containerd path.
- Actively probe containerd with IsServing in runtime.New. The client dials
lazily, so a dead socket previously slipped past the boot-time
'containerd unavailable' warn and only surfaced on the first pull.
- Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL
(default 30s) for test/verify and slow-link tuning.
Josh Siviour