CreateTags is a read-modify-write of one resource's whole tag map. It was
serialised by a sync.Mutex on the service struct, which exists once per
process, while the handler answers on the spinifex-workers queue group and so
runs on whichever node NATS picks. Two nodes read the same map, each added its
own key, and the later write won. Every call returned success.
Six parallel writers against one security group on a three-node cluster left
four tags. On a single node the mutex hides it entirely, which is why no test
caught it.
The tag index moves from one S3 object per resource to a JetStream KV bucket,
where the merge runs inside kvutil.Update and commits at the revision it read.
A writer that loses the race re-reads the winner's keys and adds its own on
top. S3 could not host the fix: predastore's PutObject has no If-Match, so
there is nothing to compare and swap against.
Tags written before the move are not migrated in a batch. A batch that
half-failed would leave a resource looking untagged, and the next write would
commit that as the truth. Instead the CAS seeds a missing entry from the old
object inside the attempt, so a resource migrates on its first write and a
failure to read the old object fails that one call. DescribeTags reads the
union of both key sets, preferring the bucket, and DeleteAllTags clears both so
old tags cannot reappear through the fallback.
The process-local mutex is removed rather than left beside the CAS.
The regression test drives six separate service values over one bucket,
because that is what makes it able to fail: with the old code every caller
inside one process was already serialised, and only a second node could lose an
update.
kvutil.BucketNames ranged the KeyValueStoreNames lister channel using whatever context the caller supplied, and the EKS desired-set builder supplies one that lives as long as the daemon. A lost stream-names reply therefore left the channel open with nothing ever sent on it and parked the caller for the life of the process. kvutil.Keys, immediately below it, already bounds exactly this case and says why; BucketNames never got the same treatment.
The DNS reconcile made that hang cluster-wide. A pass takes the reconcile lease before computing its batch and releases it on a deferred call, and the lease renews on a background timer for as long as the holder lives, so a pass that never returns holds it forever and the other nodes never take over. Observed on a three-node cluster: the zone object stopped being written after a rolling deploy, and every name published afterwards stayed NXDOMAIN until the holder was restarted.
The listing now carries its own deadline and reports a timeout as an error, never as a short list, because callers prune on absence. The pass bounds the batch computation as well, so any other deadline-free call reached from the desired-set builders costs a cycle rather than the cluster. Only the read side is bounded: it has no side effects, so abandoning it is safe, while the apply stays inline under the lease.
ListTasks read only the cluster and returned every task record under it, discarding DesiredStatus, Family, ServiceName, StartedBy and ContainerInstance. On a cluster with one running task and a hundred stopped ones, both status filters returned the same hundred-and-one ARNs.
A taskFilter now carries the input's matchable fields and is applied per record. An omitted desiredStatus means RUNNING, as it does on AWS, so an unfiltered call no longer hands back everything the cluster has ever run.
recordTaskState moves a self-exited task's desired status to STOPPED alongside its last status. Without that the filter would be honoured and still wrong: a task that died on its own would answer the RUNNING filter for as long as its record survived.
Container-instance counts came from len(PlacedTasks), which is the set of tasks still owing capacity rather than a status tally, so a PENDING task counted as running and a task whose release did not complete counted as running forever. PendingTasksCount was never set at all. Both are now tallied from the task records, the way the cluster-level counts already were. DeregisterContainerInstance passes zero deliberately — the instance is gone, and counting there would report the tasks it just force-stopped.
launchType is not filtered because TaskRecord has no launch-type field to match on, and maxResults/nextToken are left alone because no List call in this package paginates; adding a token scheme to one would leave the other four inconsistent.
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
* fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host
DescribeRegions hardcoded https://localhost:9999 as the Region's endpoint
regardless of where the gateway actually runs, so a workload doing endpoint
discovery through this API was pointed at its own loopback instead of a
reachable address. The endpoint is now resolved by the caller from
GatewayConfig.RegistryHost/RegistryPort, the same advertised host:port ECR
already uses (concrete AWSGW bind host, else AdvertiseIP), and passed into
DescribeRegions rather than invented there. Falls back to localhost:9999
only when no concrete host is configured, which is also the only case where
that value is actually correct.
Closes mulga-0bqih.
* style: trim DescribeRegions endpoint comments
* feat(imds): serve block-device-mapping from instance block device data
The aws-ebs-csi-driver's IMDS client fails outright on this 404,
aborting IMDSInstanceInfo before it reaches the Kubernetes fallback the
driver otherwise uses. Serve ami, root, and one ebsN per attached EBS
volume from the instance's existing BlockDeviceMappings, root device
first, matching how the driver counts "ebs" occurrences in the listing.
* style: trim block-device-mapping comments
AuthorizeSecurityGroupIngress and AuthorizeSecurityGroupEgress rejected any
IpPermission carrying an Ipv6Ranges entry. The security-group module defaults
egress_ipv6_cidr_blocks to ::/0 and the provider revokes the group's own
default egress before authorizing the replacement, so the rejection left the
group with no egress rules at all, which is treated as deny.
SGRule gains CidrIpv6. The rule is stored, deduped, revoked and projected like
any other, and the two policy.Rule adapters drop it before the ACL builder,
which is IPv4-only. No interface here has an IPv6 address, so the rule matches
nothing - which is what an IPv6 rule does on AWS in a VPC with no IPv6 CIDR.
Revoking an IPv6 rule that does not exist now returns InvalidPermission.NotFound
rather than succeeding as a no-op. The no-op existed only because authorize
refused to store the rule, so a caller's state could never match.
Closes mulga-6bu36.
DescribeTasks returned an empty attachments array for every STOPPED awsvpc task. On env19, 134 of 135 tasks in a cluster reported attachments: [], and only the one PENDING task carried an ElasticNetworkInterface block. On AWS a stopped task keeps its attachment for the retention window, with status DELETED and the networkInterfaceId still readable, because it is part of the forensic record of a task that has stopped.
taskToAWS gates the attachment on ENIID being set, and the ENI release retry had made a successful release clear that field so a non-empty ENIID would mean "still owed" and the sweep could retry from it. The signal was right; carrying it in the emptiness of ENIID was not, because the same field is the projection's only source.
The state moves to an ENIReleased flag of its own. ENIID stays put, so the attachment survives as long as the record does, and the sweep's owed test, its refusal to prune, and the placement rollback all read the flag instead. A released record ages out on retention exactly as it did before the retry landed. The attachment reports DELETED once released rather than echoing the task's own status.
This also restores the evidence path the ENI-leak probe depends on: it follows a stopped task to its ENI, and with no attachment to follow it had been reporting skipped on every run.
Closes mulga-5bnx3.
Releasing an awsvpc task ENI was best effort: reclaimTaskENI logged a failure and returned, and nothing ever tried again. Its own comment said the ENI fields stayed on the record for a later retry, but no later retry existed, and the stopped-task sweep pruned the record an hour later without looking at those fields — deleting the only pointer to the leaked interface. An orphaned task ENI still references its security group, so DeleteSecurityGroup fails with DependencyViolation and no awsvpc stack can be destroyed without deleting the ENIs by hand.
A non-empty ENIID on a stopped record now means exactly one thing: this ENI is still owed. reclaimTaskENI clears the identity when Release succeeds and leaves it when it does not, forceStopTask reclaims before its single write so the cleared fields land in it, and the stopped-task sweep gains the job of paying the debt off — retrying with a backoff that doubles from 30s to a 15m ceiling, capped at twenty attempts per pass because Release is a 30s NATS round trip and a bucket full of them would stall the sweep for the whole fleet.
A record that still owes an ENI is never pruned, however far past retention it sits. It is the only pointer to the interface, so dropping it converts a recoverable leak into a permanent one.
The placement rollback had the same hole from the other end: an attach failure cleared the ENI fields unconditionally and RunTask never persisted the record, so a failed release there leaked with no trace at all. It now clears only when the release succeeded, and otherwise persists the record as STOPPED so the sweep owns it like any other.
Closes mulga-1ysjx.
An AMI imported on a node whose timezone is not UTC was stamped with local wall-clock time carrying a Z suffix, so on a +1000 node an 11:42Z import reported 21:42Z — ten hours in the future. The layout "2006-01-02T15:04:05.000Z" treats the trailing Z as a literal character rather than a zone directive, so Format printed whatever zone the value carried and then asserted UTC regardless.
DescribeImages now converts before formatting, which also corrects records already stored, since the stored time.Time still carries its original zone. The four sites that stamp a new image now store time.Now().UTC() as well, so the value is right on both sides of the round trip.
Terraform's most_recent sorts on this field, so before this an AMI imported on a +1000 node outranked a genuinely newer one imported on a UTC node for ten hours.
Closes mulga-il7w1.
DescribeServices returned no availabilityZoneRebalancing field, so the terraform provider read the computed attribute as empty and planned an in-place update to DISABLED on every plan, including one taken immediately after a clean apply.
The pinned aws-sdk-go v1.55.8 is the last v1 release and its ECS shapes predate the field, so it can be neither parsed from an input nor set on an output through the SDK types. Embedding ecs.Service in a wrapper does not work either: jsonutil nests an embedded struct rather than flattening it, so the service arrives under its own key.
The field is read off the raw request body and written onto the encoded response instead. checkAZRebalancing refuses anything but DISABLED with InvalidParameterException, because nothing here rebalances tasks across zones and a silently dropped ENABLED would report a service as rebalancing when nothing does. addSDKGapFields runs from WriteJSONResponse so every service-carrying action is covered from one place, decodes with UseNumber to keep the epoch-second times jsonutil emitted, and leaves a value already present alone.
Closes mulga-t239t.
oci.WithTTY put Terminal in the OCI spec while the task was created with a cio
creator that asked for no terminal, so the shim made no console socket and runc
refused to start the container at all. The terminal flag now reaches the FIFO
set, and a terminal carries no separate stderr stream.
* fix(ecs): enforce or refuse container runtime fields (TFC-0040)
user, mountPoints, volumesFrom, readonlyRootFilesystem, privileged,
pseudoTerminal, interactive, linuxParameters, systemControls,
startTimeout and stopTimeout registered successfully but were never
stored, sent to the agent, enforced, or echoed back on describe.
Store and echo the fields the agent can enforce (user, readonly root
filesystem, privileged, pseudo terminal, interactive, systemControls,
capability add/drop, start/stop timeout), apply them through containerd
via OCI spec options and a graceful Stop, and refuse at
RegisterTaskDefinition the fields the agent cannot honor (mountPoints,
volumesFrom, linuxParameters.devices/initProcessEnabled/
sharedMemorySize/tmpfs). Refusal only triggers on a value that asks
for the restriction; empty lists and false/unset flags are accepted
and echoed as before.
* test: move the bus wire-contract test to an external package
* fix(ecs-agent): release an interactive container's stdin so its copier exits
* fix(ecs): return a caller-supplied empty collection instead of omitting it
CreateNetworkInterface accepted a caller-supplied PrivateIpAddress
verbatim, with no subnet-range check and no IPAM reservation, so a
direct call could collide with a concurrent AllocateIP handing out the
same address. Add IPAM.ClaimIP, a CAS-guarded sibling of AllocateIP
that validates the address against the subnet and rejects it if
already allocated, and wire it into CreateNetworkInterface with the
matching AWS error codes.
RunInstances --private-ip-address silently returned a different,
pool-assigned address instead of honouring the request. Forward
PrivateIpAddress (including the NetworkInterfaces[].PrivateIpAddresses
primary entry) into the auto-created ENI's CreateNetworkInterfaceInput,
and reject a pinned address on a launch with MaxCount > 1 since AWS
only allows it for a single instance.
* fix: decompress single-file zst/bz2/xz/gz disk images into tmpdir
ExtractDiskImageFromFile only handled tar archives and a broken plain
.xz branch that ran xz -dk beside the source and then tried to parse
tar chatter for a path, so it silently returned an empty string. Talos
(.raw.zst, .raw.xz), Fedora CoreOS (.qcow2.xz, .raw.xz) and Flatcar
(.img.bz2) publish only single-file compressed artifacts, so none of
them could be imported.
Add streaming decompression for .zst, .bz2, .xz and .gz that writes
into the caller's tmpdir and returns that path directly, instead of
inferring it from tar output. A decompressed .qcow2 payload now flows
through the existing qcow2-to-raw qemu-img conversion via a shared
finalizeDiskImage helper. zstd uses klauspost/compress, already a
transitive dependency via containerd, for a pure-Go streaming decoder.
bz2 and gzip use the standard library. xz shells out to the xz binary
since no pure-Go xz decoder is already a dependency, and the existing
.tar.xz branch already requires that binary on the host via tar's -J
flag, so this adds no new host requirement.
* fix: write sparse holes when decompressing single-file disk images
Verified live on env13: a Talos raw.zst imported successfully but the
decompressed output was written literally rather than sparsely, an
11 GiB file on disk for content that decompresses to 183 MiB with
xz -dc | dd conv=sparse. This is a regression relative to the
qcow2-to-raw path beside it, which already writes sparse via
qemu-img convert.
decompressToTemp now reads in 1 MiB blocks, seeks past all-zero
blocks instead of writing them, and truncates to the final size so a
trailing zero run is still recorded as a hole. 1 MiB amortises the
read/seek syscall overhead over a multi-GiB image while still being
small enough to catch the large contiguous zero runs that dominate a
sparse raw disk; a zero run shorter than one block is written
literally, which is the safe direction for a false negative to fail
in.
zstd, bzip2 and gzip all route through decompressToTemp, so all three
gain sparse output from this change. The plain-.xz path shells out to
the xz binary with output piped straight to a file and does not go
through decompressToTemp, so it is unaffected by this fix and remains
a known follow-up.
* fix: make plain-.xz decompression sparse too
decompressXz shelled its output straight to a file, bypassing the
sparse-hole handling just added to decompressToTemp for zstd, bzip2
and gzip. .raw.xz is the format Talos and Flatcar actually publish
alongside .zst, so this path is the one most likely hit in practice,
and left dense the exact disk-size regression this fix targets still
fires for it.
exec.Cmd already runs its own copier goroutine when Stdout is not an
*os.File, and Wait joins it. decompressXz now hands the xz process an
io.Pipe write end and reuses decompressToTemp on the read end, so
plain .xz gets identical sparse-hole handling to the other codecs. A
separate goroutine closes the pipe with the process's exit error once
Wait returns, so a read past the end of a failed decompression
surfaces that error, stderr included, instead of a bare EOF; the
error path also closes the read end explicitly so a stuck copier
cannot wedge Wait forever if decompressToTemp gives up early.
AssociateAccessPolicy recorded a policy association but never granted
it in the cluster. Bake static ClusterRoleBindings for the mulga:
groups into the eks-node image, project a principal's cluster-scoped
associated policies onto those groups in the TokenReview webhook, and
reject namespace scope at association time since there is no
namespace-scoped RBAC writer.
* feat(iam): evaluate trust-policy conditions and accept aws:SourceAccount
evalTrustPolicy read Effect, Action and Principal but never Condition, so a
stored aws:SourceAccount clause was a control nothing enforced. Give it a
condition arm evaluated in both the Deny and Allow passes, following the
StringEquals-only, fail-closed-on-unknown-key shape
evalTrustPolicyForWebIdentity already uses. The value is the caller ARN's own
account, derived server-side rather than caller-asserted.
Relax validateWebIdentityCondition from a single-action gate to a per-action
allowlist: sts:AssumeRoleWithWebIdentity keeps its existing IRSA keys, and
sts:AssumeRole now accepts aws:SourceAccount under StringEquals. Everything
else still fails at write time, naming the operator, key and action rather
than a bare MalformedPolicyDocument, and CreateRole/UpdateAssumeRolePolicy now
carry that reason to the caller instead of swallowing it into slog.Debug.
Extend the identity-policy refusal at validateStatementRestrictions with the
reason for aws:SourceArn and aws:RequestTag/*, the two keys a companion change
records as divergences in rfc0-7_aws-divergence.md.
* test(iam): fix trust-policy reason tests after aws:SourceAccount now valid
PR #1046 merged into dev with fixtures asserting aws:SourceAccount plus
StringEquals on sts:AssumeRole is refused. Part B of mulga-yxd9p makes
that combination valid, so the fixtures now exercise an unsupported
action and an unsupported key instead.
* docs: drop bead and plan-decision references from test comments
Bump bluebottle for the new key. requestConditionKeys now maps this
door's internal principal-type spelling to the canonical AWS value
through the bluebottle constants, and omits the key for an
unrecognized type rather than supplying an empty string.
ValidateTrustPolicyDocument builds precise errors naming the statement index
and the offending element, and both callers discarded them into slog.Debug
and returned a bare MalformedPolicyDocument. Debug is below the default
level, so the reason reached nobody: not the caller, and not the operator
reading the gateway log.
The contrast with the identity-policy path was stark. CreatePolicy names the
operator and the key exactly; CreateRole answered only 'The policy document
is malformed.', so an operator could not tell whether Principal, Action, a
size limit or a Condition block was at fault. The trust gate is the one the
terraform ECS module hits, which is what made this worth fixing on its own.
Both sites now use awserrors.Errorf, the same mechanism the identity path
uses. The separate Debug line is dropped rather than duplicated: the gateway
already logs a returned error at Error level, so one message now serves both
the client and the operator.
* feat(ecs): populate service event ring on DescribeServices
Events was always projected as null since ServiceRecord carried no event
history. Add a capped, newest-first event ring populated at the rollout
transitions the reconciler already computes: steady state reached, a
circuit-breaker trip, and a placement shortfall it cannot resolve this pass.
Emission is edge-triggered on the state change already guarding each site, so
the repeated reconcile loop does not refill the ring with one repeated
message every pass.
* feat(ecs): project createdAt on services and deployments
CreatedAt was already computed and persisted on ServiceRecord and Deployment
at creation time, but serviceToAWS never echoed it, so DescribeServices could
never answer which build wrote a given record. Project ServiceRecord.CreatedAt
and Deployment.CreatedAt/UpdatedAt, omitting the key entirely for a legacy
record whose timestamp predates the field rather than emitting the 1970 epoch.
CreateCluster refused accessConfig.authenticationMode = API_AND_CONFIG_MAP,
which is both AWS's default since 1.23 and the terraform-aws-eks module
default, and named no parameter in the refusal. The mode is now stored on
the cluster record and echoed by DescribeCluster, so a caller does not see
a permanent access_config diff on every subsequent plan; bare CONFIG_MAP
still fails, since Spinifex has no aws-auth ConfigMap path. The refusal
now names accessConfig.authenticationMode via awserrors.Errorf.
nodegroupRecordToAWS also only set Health when a node group had a status
reason, so a healthy node group described back with health: null instead
of the empty-issues object AWS always returns. Health is now always
constructed with a non-nil issues slice.
The unserved-service gate ran before signature verification but answered
SignatureDoesNotMatch, falsely blaming credentials that were never checked.
It now returns InvalidAction, the same code an unimplemented action on a
served service already gets, so the behaviour is consistent rather than
novel.
The envelope was also always XML, because jsonErrorService only lists
scopes the gateway serves and an unserved scope is by definition absent
from it. A JSON-protocol caller such as sqs or dynamodb therefore could
not deserialize the body and never saw the true error. writeSigV4Error now
falls back to the request's own X-Amz-Target and Content-Type headers to
pick JSON when the scope is not in that list, leaving every served scope's
envelope selection unchanged.
RegisterTargets read the target group, appended in memory and wrote the whole record back with no revision guard, so two concurrent registrations lost one. Terraform parallelises by default, so any target group with more than one member hit it, and the apply exited 0 with half the fleet not behind the load balancer.
Every writer of the record now goes through the store's existing CAS rather than a wholesale Put, because a CAS loop on one writer is defeated by a non-CAS writer on the same key. Contention stays per target group; no global lock is introduced.
RequestCertificate returned InternalErrorException for a missing tenant CA, which is a client-actionable precondition rather than a server fault. Worse, that code is classified retryable, so the provider backed off and retried instead of failing, and a single certificate sat at Still creating for over nine minutes. It now returns ACM's not-found code, which is terminal, and carries a message naming the missing CA and how to create one.
ListCertificates omitted CertificateSummaryList entirely on an empty account, since the SDK marshaller drops a nil slice unconditionally.
Closes: mulga-c6khe
Closes: mulga-wbjal
Closes: mulga-l6yft
Closes: mulga-61h3h
* fix(iam): persist policy tags and count attachments across all principal kinds
CreatePolicy built its record with an empty tag list and never read input.Tags, and GetPolicy, ListPolicies and CreatePolicy's own response projected no Tags field at all. Both halves are fixed, matching the role path.
buildAttachmentCounts scanned only the users bucket, so a policy attached to a role or group counted as unattached. That is not just a misleading number: DeletePolicy gates its conflict check on the same count, so an attached policy could be deleted, leaving the principal holding a dangling ARN.
The count and the reverse lookup are the same traversal with the names kept or discarded, so both now derive from one buildPolicyAttachments over users, roles and groups. Nothing is stored, so no counter can drift from the attachments it counts.
ListEntitiesForPolicy is implemented on that traversal, honouring EntityFilter and PathPrefix.
Closes: mulga-fvhlc
Closes: mulga-9i6uh
Closes: mulga-ryu30
Closes: mulga-8ranb
* test(iam): account for ListEntitiesForPolicy in the cross-tenant and policy-ARN gates
The action resolves PolicyArn through the same branch as GetPolicy, so it
belongs with that helper's siblings; it was in neither gate, which is what
the accounting test exists to catch.
* fix(ecr): store and project repository encryption, scanning and tags
CreateRepository parsed none of encryptionConfiguration, imageScanningConfiguration or tags, so none were stored and none came back. Both blocks are now parsed, persisted on RepoMeta and projected, defaulting to AES256 and scanOnPush false as AWS does.
encryptionType KMS is rejected rather than echoed. Predastore seals every fragment with AES-256-GCM under a required master key, so reporting AES256 states something true, but no customer-managed key material is plumbed anywhere and accepting a KMS key ARN would assert a protection that does not exist.
Repository tags are stored on the same record, and TagResource and UntagResource now work instead of returning NotImplemented.
PutImageScanningConfiguration gets its own handler: asking for scanning off succeeds as a no-op, which is the already-true state, while asking for it on still returns OperationNotSupportedException because no scanner exists. Reporting scanning as enabled when nothing scans would read as a false all-clear.
The AWS Repository shape was built in three places with no shared helper, so every new field would have been added three times. They now share one buildRepository projection.
Closes: mulga-ywvtu
Closes: mulga-j2y54
Closes: mulga-3whvb
* fix: accept and store repository tags on ECR CreateRepository
The AWS API takes a tags parameter on CreateRepository; the gateway accepted
the request and discarded them, leaving the repository untagged with no error.
Decode them into the RepoMeta tag map at create time, rejecting an empty key
the same way TagResource does.
DescribeTaskDefinition now echoes portMappings[].name and runtimePlatform,
DescribeServices projects the stored networkConfiguration and
enableECSManagedTags, DescribeClusters returns the four instance, task and
service counts, and the five list operations return their ARN key as a
present empty list instead of omitting it.
UpdateService now honours networkConfiguration and enableECSManagedTags so a
changed value converges instead of producing a permanent diff.
The eleven container runtime fields the task definition also drops are
deliberately left absent: nothing between the daemon and the on-host agent
carries them, and reporting a container as privileged or read-only-root when
nothing enforces it is worse than reporting nothing.
Closes: mulga-0t9q2
Closes: mulga-7thn7
Closes: mulga-d8kif
Closes: mulga-5z8z9
Three defects at the gateway's HTTP error boundary, none of which change an authorisation outcome.
xmlErrorBody defaulted to the EC2 query envelope, so any scope it did not enumerate, including every service the gateway does not serve, received a shape a REST-XML client cannot deserialize and the real code never surfaced. The generic ErrorResponse is now the default and the query-protocol services are named explicitly. An unparsed signature keeps the EC2 shape, since no service is known there and that path is not what this changes.
The supportedServices gate logged the truthful reason and then emitted the canned credentials text, because writeSigV4Error had no parameter to carry a message. It takes one now, and the gate names the service it refused. The signature is never verified at that point, so blaming the caller's credentials was actively misleading.
ctxAction was set only for the query protocol, so every JSON-1.1 request shared the throttle bucket "<account>:unknown" and no per-action limit could ever match. It now resolves from X-Amz-Target, and for the path-routed rest-json services from the route's own lookup function.
Also sets X-Amzn-Errortype on the four JSON emitters, which JSON-protocol SDKs read to resolve the modelled exception type.
Recovery tested whether the QEMU process was still running before it looked at the instance's state, so a live process short-circuited the transitional-state handling and reconnectInstance then set the instance back to running. A stop or terminate the caller had already been told was under way was silently discarded.
Check for StateStopping and StateShuttingDown first, kill a QEMU that outlived the transition, and finalize. reconnectInstance now refuses a transitional instance outright rather than promoting it.
Whether br-mgmt is a per-host island or one L2 segment spanning the cluster
decides whether a system instance can reach the gateway address it was baked
with. No artifact recorded which one a run built, and the environment is
destroyed before it can be asked, so the decisive evidence for an LB agent
heartbeat failure was gone by the time anyone looked.
Records ports, address and fail_mode per node, then probes every node's
br-mgmt address from every node. Islands reach only themselves and spanned
segments reach everything; a mixed result is the failure this catches.
* fix: carry a handler's own refusal message to the client without leaking wrapper context
A daemon that explains a refusal had that explanation discarded one step short of the client, so every NATS-backed error surfaced as the generic ErrorLookup blurb for its code. The message crossed NATS intact; resolveErrorDetail then dropped it because decodedNATSError was not the shape it recognises.
Rebuild the decoded response through awserrors.Errorf so the walk finds the message, and send only ResolveErrorDetail's curated message from the two encode sites instead of err.Error(). The latter rendered every %w layer, including context a call site added purely for its own logs, so restoring the message without also fixing the encode side would have turned a fidelity bug into an information disclosure.
Two existing tests pinned that wrapper context onto the wire, directly contradicting the gateway's own no-leak assertion; both are rewritten to pin the sanitized contract instead.
Also name the rejected filter in the twenty-four ParseFilters sites that logged the detail and returned a bare code, and give the ACM force-renew refusals a registered code so their reason still reaches the caller.
* fix: stop the filter rejection repeating its own error code to the client
ParseFilters spelled the code into its message text, so once the code began travelling separately the client saw it twice: "InvalidParameterValue: InvalidParameterValue: The filter 'x' is invalid".
Give ParseFilters the code directly via awserrors.Errorf and have its twenty-five callers return that error unwrapped rather than re-wrapping it. All of them already mapped it to InvalidParameterValue, so nothing changes but the duplication.
* fix: surface why a security group rule was rejected
ipPermissionsToSGRules already names the reason, "IPv6 rules are not supported" or "IpPermission must specify at least one IpRange or UserIdGroupPair", and both Authorize call sites logged it and returned a bare InvalidParameterValue, so the caller saw only the canned blurb.
This is the case mulga-e9qqx names specifically. It does not make IPv6 rules work; it makes the refusal legible.
CreateCluster and CreateNodegroup accepted fields that DescribeCluster and DescribeNodegroup never returned. The unmodified terraform-aws-modules/eks/aws module reads those fields back, finds them empty, and either aborts the apply or proposes a destroy of a cluster that is already correct.
Two of the omissions aborted the apply outright. A module precondition on data.aws_eks_cluster failed with cluster_service_cidr is "" because kubernetesNetworkConfig was absent, and the tag resource address built from clusterSecurityGroupId came out as (,suite) because that field was empty.
Store what the caller asked for on ClusterMeta and NodegroupRecord at create time, and project it back. Both records are encoding/json with omitempty, so records already in KV stay readable and project exactly as they did before.
The read-backs echo the request rather than synthesising an AWS default, because UpdateClusterConfig is not implemented — a value the caller did not ask for would be a diff they could never converge. Where a field does report a default it is the value genuinely in force: 10.43.0.0/16 is k3s's own service CIDR, API is the only authentication mode implemented, and logging reports every type disabled because no control-plane log export exists.
resourcesVpcConfig.securityGroupIds previously overwrote the caller's own groups with the platform's pair, which live in a VPC the caller cannot see. It now echoes the caller. The customer-facing cluster security group is created in the caller's account and VPC at cluster-create time, so the id reported by clusterSecurityGroupId resolves for the tenant that owns it. That creation is idempotent with the one launchNodegroupInfra already makes, and matches AWS, which creates the group with the cluster.
updateNodegroupConfig accepted UpdateConfig and Taints, ignored both, and returned SUCCESSFUL. It now applies them under the existing compare-and-swap, so a plan converges instead of asking forever. Caller-supplied labels and taints still never reach kubelet; that gap predates this change and is tracked separately.
Verified on env19: the module applies clean and a re-plan reports no changes.
Six defects in the EC2 describe path, found by running unmodified Terraform Registry modules against a live cluster and planning immediately after a clean apply. They share one shape: the write succeeds, the read-back is incomplete, and the caller cannot tell.
Populate Association on the primary network interface. The public IP was projected onto the instance but never onto the NIC, so the AWS provider read associate_public_ip_address as false for an instance that had a working public address. That attribute is force-new, so a no-op plan proposed destroying and recreating the instance.
Stamp EnclaveOptions. AWS always returns it, and its absence rendered as a permanent enclave_options block in every plan.
Stop emitting empty Reservations. The reservation map entry was created before the filter predicate ran, so a filter that excluded every instance in a reservation still emitted the reservation with no instances. Projection and filtering now run before the insert, in both the live and the KV path.
Merge reservations by ID in the gateway. Response frames were concatenated rather than merged, so one reservation could appear twice in a single response and which copy landed first varied between calls.
Mint the reservation ID once per RunInstances call. Each node minted its own, so a launch spread across nodes came back as several reservations. The gateway now allocates the ID and passes it to every node in a NATS header; a node that does not read the header mints its own as before, so the change is additive in both directions.
Answer an incomplete DescribeInstances sweep with a retryable ServiceUnavailable. Suppressing a false InvalidInstanceID.NotFound when a node had not answered was correct, but no branch was ever written for that case, so the caller received an empty 200 that read as authoritative. A filters-only query with no matches is still a legitimate empty result.
Accept the AWS spellings of the DescribeVpcs filters. The filters were implemented but named cidr-block and is-default where AWS uses cidr and isDefault, so Terraform's aws_vpc data source was rejected outright. Both spellings now work, since an internal caller depends on the old one. The parse error also names the offending filter to the client instead of only the log.
Resolve image owner aliases before filtering on them. describe-images --owners self compared an alias against a numeric account ID and matched nothing; the correct resolution already existed but ran after the filter.
Also builds the segscan test oracle with -buildvcs=false. That is unrelated to the above and fails on current toolchains regardless of this change, but it blocks preflight.
Josh Siviour