Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own.
- Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence.
- The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering.
- "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs.
- "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade.
- Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install.
install-usb and vpc-networking followed the two moved sections to their new paths.
Three omissions left the launched root volume reading back as something the caller never asked for.
DescribeInstances returned a null RootDeviceName and RootDeviceType while DescribeImages populated both for the same AMI, so the two answers disagreed about the same image. The launch path now projects both onto the instance.
The root volume's EBS request carried no device name, so the launcher wrote an empty string to the volume's metadata and DescribeVolumes reported an attached volume with an empty Attachments[].Device. The name parseVolumeParams already resolved now travels with the request.
A volume-scoped TagSpecification was discarded: only instance-scoped tags were read, and the comment claimed volume specs were handled elsewhere, which nothing did. They are now written to the root volume's metadata document.
BlockDeviceMappings[].DeviceName still reports the guest path rather than the requested name. That rewrite is deliberate and is left alone here.
* fix: route the EKS update reads so they refuse instead of reading as a typo
UpdateClusterVersion was registered at POST /clusters/{name}/update-version, which is the nodegroup pattern and not a path the SDK ever calls; AWS puts it at POST /clusters/{name}/updates. The refusal written for it was therefore unreachable, and the router answered InvalidAction, which tells a caller they typed something that does not exist.
ListUpdates and DescribeUpdate had no route at all and answered the same way. Both now refuse with NotImplemented at the gateway rather than through a service method: the update surface they would read is absent, so a round-trip to reach a constant refusal is surface for its own sake.
The absence of the cluster-level update surface is unchanged. This only stops the platform blaming the caller's spelling for its own routing.
* test: account for the two EKS update reads in the cross-tenant table
Both name their cluster by a path segment that carries no account segment, so
neither can name another tenant's resource and neither needs a case of its own.
The agent acked a stop in the same loop iteration that launched the reap, so
the directive was deleted before the reap had done anything. When the reap
could not run — no container runtime, or a failing List — nothing re-posted it,
and the task held its ENI, its capacity reservation and its target-group
registrations forever. The in-memory `stopping` map made that permanent: it was
set at dispatch and never cleared, so the agent would not re-dispatch even if
the directive did come back.
A stop ledger now separates a reap in flight from a reap that reported STOPPED.
Only the second is acked, so a failed reap leaves the directive in the inbox to
be redelivered and retried, while the assign loop keeps reading both halves and
so still refuses to start a task that is stopping or stopped.
* fix: persist security group rule tags and report -1 ports for all-protocol rules
A security group rule accepted a security-group-rule TagSpecification and kept
none of it, so DescribeSecurityGroupRules answered with no tags and every
subsequent plan offered to add them again. SGRule now carries a Tags map,
populated on both authorize paths and projected into the central tag store, and
DescribeSecurityGroupRules evaluates tag: and tag-key filters against it instead
of excluding every rule.
CreateTags and DeleteTags on an sgr- id were a silent no-op because
updateRecordTags routed only subnet-, vpc-, sg- and eni-. It now finds the rule
inside its parent group's record and mutates it there, and a revoked rule's
central entry is cleared with it.
An all-protocol rule reported the ports it was stored with, 0 and 0, where AWS
reports -1 and -1, which is the same never-converging plan by another route.
* fix: recognise sgr- as a security-group-rule resource id
DescribeTags reported ResourceType "unknown" for a security group rule, so a
resource-type filter naming security-group-rule matched none of them, and the
authorizer could build no ARN for a rule id and fell back to the any-resource
wildcard. Both read the same prefix table, which had no entry for sgr-.
RunInstances read BlockDeviceMappings[0] and nothing else, so a request carrying a root mapping and a data mapping created one volume sized from whichever happened to be first, and the first mapping became the root whatever device it named. A module emitting the data mapping first produced a root volume of the data volume's size and no data volume at all, with no error.
The root is now the mapping naming the AMI root device, with position deciding only when nothing names it, so the single /dev/vda mapping ECS and EKS launch with still works. Any further EBS mapping is refused in ValidateRunInstancesInput with InvalidParameterValue naming BlockDeviceMappings and the devices it cannot attach, rather than being discarded silently. Ephemeral mappings carry no volume and are ignored.
Creating the additional volumes at launch remains unimplemented; this makes its absence visible.
The deploy builds with GOWORK=off, so spx links the pin rather than the sibling
checkout, and the versioning handlers are absent from every deployed binary
until it moves.
GitHub registers a schedule: only from a repository's default branch, so this cron could drive exactly one branch and always executed main's copy of this file whatever it tested. main was never exercised by the nightly, and a fix to the workflow itself could not be proved until it had already shipped — which is how cell-24's unguarded pkill stayed live on the schedule after its fix merged to dev.
A Cloudflare Worker now dispatches this workflow once on dev and once on main. A dispatch takes both the workflow file and the source under test from its ref, so GIT_BRANCH is simply github.ref_name and every schedule-only expression goes with the cron.
cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d
-config'`, which matches every predastore node on the runner and not only the
ones this run left behind. The predastore repository runs the same harnesses on
the same machine, so "still running" does not mean "stale" — on 2026-09-16 this
step killed a predastore benchmark 32 seconds into its workload, and the run
failed with 61 connection-refused errors that look nothing like their cause.
Both kills now run under the host benchmark lock the harnesses themselves take,
so anything alive while it is held really is a leftover. The second one needs it
as much as the first: by the time cleanup runs the harness has released the
lock, so another repository's benchmark can already be up.
Every deployed binary links whatever go.mod pins, so the routing fix for bucket sub-resources is absent from the gateway until this moves. Without it a provider-level default_tags block still leaves no bucket creatable by Terraform.
DescribeTags kept answering for resources that no longer exist: the central tag store's DeleteAllTags had one production caller, the instance terminate path, so every other resource type left its entry behind forever.
CentralTagWriter becomes CentralTagStore and gains DeleteAllTags, which TagsServiceImpl already implemented, so the daemon wiring is a rename rather than new plumbing. clearRecordTags mirrors projectRecordTags: a nil store is a no-op and a failure is logged rather than returned.
The clear runs after the record delete has succeeded, never before, so a delete that fails leaves a live resource with its tags intact. deleteNetworkInterface covers both the public delete and the forced instance teardown, and deleteSecurityGroupInternal covers the default group that only the DeleteVpc cascade ever removes.
An AWS SDK's default endpoint resolution builds
{service}.{region}.{services/domain}. IMDS previously answered that
leaf with the literal amazonaws.com, so a workload that forgot its
endpoint override silently reached real AWS instead of failing.
Now that service-endpoint DNS resolves cluster-side, serve the
cluster's configured AWS.ServicesDomain instead, threaded from
config through vpcd.Config into NewIMDSServiceImpl. An empty value
fails closed with a 404 rather than ever falling back to
amazonaws.com. services/partition is unchanged: it still returns
aws, since AWS SDKs carry hardcoded partition tables.
A tag set at creation time through TagSpecifications was written only onto the resource's own KV record. DescribeTags reads a separate central index at tags/{accountID}/{resourceID}.json, which only CreateTags writes, so a resource born with tags described as having none until something tagged it again.
Terraform finds this immediately: terraform-aws-modules/eks reads a cluster security group's tags back after creating it, and aws_ec2_tag fails against a resource whose tags the tag API cannot see.
The four VPC create paths now project their stored tags into the central store through an optional CentralTagWriter, injected after construction like the external IPAM. A nil writer leaves create behaviour unchanged, and a write failure is logged rather than failing the create — the resource exists either way, and failing a create over a tag index would be worse than the divergence it repairs.
Only vpc, subnet, security group and network interface are covered here. Image, volume, snapshot, route table, internet gateway, egress-only internet gateway, elastic IP, NAT gateway, key pair and launch template have the same split and are follow-up work.
* fix(ecs): make awsvpc ENI slots a placement resource
A container instance carries a finite number of awsvpc task ENIs — the hot-plug PCIe root ports the daemon pre-allocates from the instance type's cap — and nothing outside the daemon knew the limit existed. The scheduler's capacity model was CPU, memory and GPU, so every over-capacity task was placed, created an ENI, failed to hot-plug it and rolled it back. A service asking for more tasks than the cluster has slots churned ENI create/delete indefinitely instead of reporting a placement failure once. On a t3.micro the third task and everything after it failed with "attach task ENI eni-… -> i-…: ServerInternal".
The instance record now carries the instance type, reported the way AWS carries it, as the ecs.instance-type container-instance attribute; the ENI cap is derived from it rather than stored, so a corrected limit table applies on the next read. An instance whose type has not been reported is not gated at all: a recognised type always has at least one slot, so zero can only mean unknown, and refusing every awsvpc placement against an older agent would turn a mixed-version cluster into an outage.
Both placement paths gate on it. A task declined for want of an interface is reported as RESOURCE:ENI, spelled as AWS spells it, and separately from a plain capacity refusal. The caller-visible detail is a fixed sentence; the wrapped error keeps its identifiers and server error codes in the log, where someone can act on them.
DescribeInstanceTypes now reports NetworkInfo.MaximumNetworkInterfaces, from the same table the slot allocator enforces, so a caller sizing an autoscaling group has something to size against. Fields this stack does not implement are left unset rather than guessed.
Finally, the daemon records the slot it actually allocated as the ENI's device index. The requested index was stored verbatim, so every task ENI on an instance reported index 1 while landing on a different PCIe slot; on AWS a device index is unique per instance.
* fix(ecs): send an ENI capacity refusal as the reason alone
AWS returns RESOURCE:* placement failures with no detail. Carrying a sentence
there gives a caller a second string to parse that says nothing the reason does
not already say.
CreateTags is a read-modify-write of one resource's whole tag map. It was
serialised by a sync.Mutex on the service struct, which exists once per
process, while the handler answers on the spinifex-workers queue group and so
runs on whichever node NATS picks. Two nodes read the same map, each added its
own key, and the later write won. Every call returned success.
Six parallel writers against one security group on a three-node cluster left
four tags. On a single node the mutex hides it entirely, which is why no test
caught it.
The tag index moves from one S3 object per resource to a JetStream KV bucket,
where the merge runs inside kvutil.Update and commits at the revision it read.
A writer that loses the race re-reads the winner's keys and adds its own on
top. S3 could not host the fix: predastore's PutObject has no If-Match, so
there is nothing to compare and swap against.
Tags written before the move are not migrated in a batch. A batch that
half-failed would leave a resource looking untagged, and the next write would
commit that as the truth. Instead the CAS seeds a missing entry from the old
object inside the attempt, so a resource migrates on its first write and a
failure to read the old object fails that one call. DescribeTags reads the
union of both key sets, preferring the bucket, and DeleteAllTags clears both so
old tags cannot reappear through the fallback.
The process-local mutex is removed rather than left beside the CAS.
The regression test drives six separate service values over one bucket,
because that is what makes it able to fail: with the old code every caller
inside one process was already serialised, and only a second node could lose an
update.
kvutil.BucketNames ranged the KeyValueStoreNames lister channel using whatever context the caller supplied, and the EKS desired-set builder supplies one that lives as long as the daemon. A lost stream-names reply therefore left the channel open with nothing ever sent on it and parked the caller for the life of the process. kvutil.Keys, immediately below it, already bounds exactly this case and says why; BucketNames never got the same treatment.
The DNS reconcile made that hang cluster-wide. A pass takes the reconcile lease before computing its batch and releases it on a deferred call, and the lease renews on a background timer for as long as the holder lives, so a pass that never returns holds it forever and the other nodes never take over. Observed on a three-node cluster: the zone object stopped being written after a rolling deploy, and every name published afterwards stayed NXDOMAIN until the holder was restarted.
The listing now carries its own deadline and reports a timeout as an error, never as a short list, because callers prune on absence. The pass bounds the batch computation as well, so any other deadline-free call reached from the desired-set builders costs a cycle rather than the cluster. Only the read side is bounded: it has no side effects, so abandoning it is safe, while the apply stays inline under the lease.
ListTasks read only the cluster and returned every task record under it, discarding DesiredStatus, Family, ServiceName, StartedBy and ContainerInstance. On a cluster with one running task and a hundred stopped ones, both status filters returned the same hundred-and-one ARNs.
A taskFilter now carries the input's matchable fields and is applied per record. An omitted desiredStatus means RUNNING, as it does on AWS, so an unfiltered call no longer hands back everything the cluster has ever run.
recordTaskState moves a self-exited task's desired status to STOPPED alongside its last status. Without that the filter would be honoured and still wrong: a task that died on its own would answer the RUNNING filter for as long as its record survived.
Container-instance counts came from len(PlacedTasks), which is the set of tasks still owing capacity rather than a status tally, so a PENDING task counted as running and a task whose release did not complete counted as running forever. PendingTasksCount was never set at all. Both are now tallied from the task records, the way the cluster-level counts already were. DeregisterContainerInstance passes zero deliberately — the instance is gone, and counting there would report the tasks it just force-stopped.
launchType is not filtered because TaskRecord has no launch-type field to match on, and maxResults/nextToken are left alone because no List call in this package paginates; adding a token scheme to one would leave the other four inconsistent.
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
* fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host
DescribeRegions hardcoded https://localhost:9999 as the Region's endpoint
regardless of where the gateway actually runs, so a workload doing endpoint
discovery through this API was pointed at its own loopback instead of a
reachable address. The endpoint is now resolved by the caller from
GatewayConfig.RegistryHost/RegistryPort, the same advertised host:port ECR
already uses (concrete AWSGW bind host, else AdvertiseIP), and passed into
DescribeRegions rather than invented there. Falls back to localhost:9999
only when no concrete host is configured, which is also the only case where
that value is actually correct.
Closes mulga-0bqih.
* style: trim DescribeRegions endpoint comments
* feat(imds): serve block-device-mapping from instance block device data
The aws-ebs-csi-driver's IMDS client fails outright on this 404,
aborting IMDSInstanceInfo before it reaches the Kubernetes fallback the
driver otherwise uses. Serve ami, root, and one ebsN per attached EBS
volume from the instance's existing BlockDeviceMappings, root device
first, matching how the driver counts "ebs" occurrences in the listing.
* style: trim block-device-mapping comments
AuthorizeSecurityGroupIngress and AuthorizeSecurityGroupEgress rejected any
IpPermission carrying an Ipv6Ranges entry. The security-group module defaults
egress_ipv6_cidr_blocks to ::/0 and the provider revokes the group's own
default egress before authorizing the replacement, so the rejection left the
group with no egress rules at all, which is treated as deny.
SGRule gains CidrIpv6. The rule is stored, deduped, revoked and projected like
any other, and the two policy.Rule adapters drop it before the ACL builder,
which is IPv4-only. No interface here has an IPv6 address, so the rule matches
nothing - which is what an IPv6 rule does on AWS in a VPC with no IPv6 CIDR.
Revoking an IPv6 rule that does not exist now returns InvalidPermission.NotFound
rather than succeeding as a no-op. The no-op existed only because authorize
refused to store the rule, so a caller's state could never match.
Closes mulga-6bu36.
DescribeTasks returned an empty attachments array for every STOPPED awsvpc task. On env19, 134 of 135 tasks in a cluster reported attachments: [], and only the one PENDING task carried an ElasticNetworkInterface block. On AWS a stopped task keeps its attachment for the retention window, with status DELETED and the networkInterfaceId still readable, because it is part of the forensic record of a task that has stopped.
taskToAWS gates the attachment on ENIID being set, and the ENI release retry had made a successful release clear that field so a non-empty ENIID would mean "still owed" and the sweep could retry from it. The signal was right; carrying it in the emptiness of ENIID was not, because the same field is the projection's only source.
The state moves to an ENIReleased flag of its own. ENIID stays put, so the attachment survives as long as the record does, and the sweep's owed test, its refusal to prune, and the placement rollback all read the flag instead. A released record ages out on retention exactly as it did before the retry landed. The attachment reports DELETED once released rather than echoing the task's own status.
This also restores the evidence path the ENI-leak probe depends on: it follows a stopped task to its ENI, and with no attachment to follow it had been reporting skipped on every run.
Closes mulga-5bnx3.
Releasing an awsvpc task ENI was best effort: reclaimTaskENI logged a failure and returned, and nothing ever tried again. Its own comment said the ENI fields stayed on the record for a later retry, but no later retry existed, and the stopped-task sweep pruned the record an hour later without looking at those fields — deleting the only pointer to the leaked interface. An orphaned task ENI still references its security group, so DeleteSecurityGroup fails with DependencyViolation and no awsvpc stack can be destroyed without deleting the ENIs by hand.
A non-empty ENIID on a stopped record now means exactly one thing: this ENI is still owed. reclaimTaskENI clears the identity when Release succeeds and leaves it when it does not, forceStopTask reclaims before its single write so the cleared fields land in it, and the stopped-task sweep gains the job of paying the debt off — retrying with a backoff that doubles from 30s to a 15m ceiling, capped at twenty attempts per pass because Release is a 30s NATS round trip and a bucket full of them would stall the sweep for the whole fleet.
A record that still owes an ENI is never pruned, however far past retention it sits. It is the only pointer to the interface, so dropping it converts a recoverable leak into a permanent one.
The placement rollback had the same hole from the other end: an attach failure cleared the ENI fields unconditionally and RunTask never persisted the record, so a failed release there leaked with no trace at all. It now clears only when the release succeeded, and otherwise persists the record as STOPPED so the sweep owns it like any other.
Closes mulga-1ysjx.
An AMI imported on a node whose timezone is not UTC was stamped with local wall-clock time carrying a Z suffix, so on a +1000 node an 11:42Z import reported 21:42Z — ten hours in the future. The layout "2006-01-02T15:04:05.000Z" treats the trailing Z as a literal character rather than a zone directive, so Format printed whatever zone the value carried and then asserted UTC regardless.
DescribeImages now converts before formatting, which also corrects records already stored, since the stored time.Time still carries its original zone. The four sites that stamp a new image now store time.Now().UTC() as well, so the value is right on both sides of the round trip.
Terraform's most_recent sorts on this field, so before this an AMI imported on a +1000 node outranked a genuinely newer one imported on a UTC node for ten hours.
Closes mulga-il7w1.
DescribeServices returned no availabilityZoneRebalancing field, so the terraform provider read the computed attribute as empty and planned an in-place update to DISABLED on every plan, including one taken immediately after a clean apply.
The pinned aws-sdk-go v1.55.8 is the last v1 release and its ECS shapes predate the field, so it can be neither parsed from an input nor set on an output through the SDK types. Embedding ecs.Service in a wrapper does not work either: jsonutil nests an embedded struct rather than flattening it, so the service arrives under its own key.
The field is read off the raw request body and written onto the encoded response instead. checkAZRebalancing refuses anything but DISABLED with InvalidParameterException, because nothing here rebalances tasks across zones and a silently dropped ENABLED would report a service as rebalancing when nothing does. addSDKGapFields runs from WriteJSONResponse so every service-carrying action is covered from one place, decodes with UseNumber to keep the epoch-second times jsonutil emitted, and leaves a value already present alone.
Closes mulga-t239t.
oci.WithTTY put Terminal in the OCI spec while the task was created with a cio
creator that asked for no terminal, so the shim made no console socket and runc
refused to start the container at all. The terminal flag now reaches the FIFO
set, and a terminal carries no separate stderr stream.
* fix(ecs): enforce or refuse container runtime fields (TFC-0040)
user, mountPoints, volumesFrom, readonlyRootFilesystem, privileged,
pseudoTerminal, interactive, linuxParameters, systemControls,
startTimeout and stopTimeout registered successfully but were never
stored, sent to the agent, enforced, or echoed back on describe.
Store and echo the fields the agent can enforce (user, readonly root
filesystem, privileged, pseudo terminal, interactive, systemControls,
capability add/drop, start/stop timeout), apply them through containerd
via OCI spec options and a graceful Stop, and refuse at
RegisterTaskDefinition the fields the agent cannot honor (mountPoints,
volumesFrom, linuxParameters.devices/initProcessEnabled/
sharedMemorySize/tmpfs). Refusal only triggers on a value that asks
for the restriction; empty lists and false/unset flags are accepted
and echoed as before.
* test: move the bus wire-contract test to an external package
* fix(ecs-agent): release an interactive container's stdin so its copier exits
* fix(ecs): return a caller-supplied empty collection instead of omitting it
CreateNetworkInterface accepted a caller-supplied PrivateIpAddress
verbatim, with no subnet-range check and no IPAM reservation, so a
direct call could collide with a concurrent AllocateIP handing out the
same address. Add IPAM.ClaimIP, a CAS-guarded sibling of AllocateIP
that validates the address against the subnet and rejects it if
already allocated, and wire it into CreateNetworkInterface with the
matching AWS error codes.
RunInstances --private-ip-address silently returned a different,
pool-assigned address instead of honouring the request. Forward
PrivateIpAddress (including the NetworkInterfaces[].PrivateIpAddresses
primary entry) into the auto-created ENI's CreateNetworkInterfaceInput,
and reject a pinned address on a launch with MaxCount > 1 since AWS
only allows it for a single instance.
* fix: decompress single-file zst/bz2/xz/gz disk images into tmpdir
ExtractDiskImageFromFile only handled tar archives and a broken plain
.xz branch that ran xz -dk beside the source and then tried to parse
tar chatter for a path, so it silently returned an empty string. Talos
(.raw.zst, .raw.xz), Fedora CoreOS (.qcow2.xz, .raw.xz) and Flatcar
(.img.bz2) publish only single-file compressed artifacts, so none of
them could be imported.
Add streaming decompression for .zst, .bz2, .xz and .gz that writes
into the caller's tmpdir and returns that path directly, instead of
inferring it from tar output. A decompressed .qcow2 payload now flows
through the existing qcow2-to-raw qemu-img conversion via a shared
finalizeDiskImage helper. zstd uses klauspost/compress, already a
transitive dependency via containerd, for a pure-Go streaming decoder.
bz2 and gzip use the standard library. xz shells out to the xz binary
since no pure-Go xz decoder is already a dependency, and the existing
.tar.xz branch already requires that binary on the host via tar's -J
flag, so this adds no new host requirement.
* fix: write sparse holes when decompressing single-file disk images
Verified live on env13: a Talos raw.zst imported successfully but the
decompressed output was written literally rather than sparsely, an
11 GiB file on disk for content that decompresses to 183 MiB with
xz -dc | dd conv=sparse. This is a regression relative to the
qcow2-to-raw path beside it, which already writes sparse via
qemu-img convert.
decompressToTemp now reads in 1 MiB blocks, seeks past all-zero
blocks instead of writing them, and truncates to the final size so a
trailing zero run is still recorded as a hole. 1 MiB amortises the
read/seek syscall overhead over a multi-GiB image while still being
small enough to catch the large contiguous zero runs that dominate a
sparse raw disk; a zero run shorter than one block is written
literally, which is the safe direction for a false negative to fail
in.
zstd, bzip2 and gzip all route through decompressToTemp, so all three
gain sparse output from this change. The plain-.xz path shells out to
the xz binary with output piped straight to a file and does not go
through decompressToTemp, so it is unaffected by this fix and remains
a known follow-up.
* fix: make plain-.xz decompression sparse too
decompressXz shelled its output straight to a file, bypassing the
sparse-hole handling just added to decompressToTemp for zstd, bzip2
and gzip. .raw.xz is the format Talos and Flatcar actually publish
alongside .zst, so this path is the one most likely hit in practice,
and left dense the exact disk-size regression this fix targets still
fires for it.
exec.Cmd already runs its own copier goroutine when Stdout is not an
*os.File, and Wait joins it. decompressXz now hands the xz process an
io.Pipe write end and reuses decompressToTemp on the read end, so
plain .xz gets identical sparse-hole handling to the other codecs. A
separate goroutine closes the pipe with the process's exit error once
Wait returns, so a read past the end of a failed decompression
surfaces that error, stderr included, instead of a bare EOF; the
error path also closes the read end explicitly so a stuck copier
cannot wedge Wait forever if decompressToTemp gives up early.
AssociateAccessPolicy recorded a policy association but never granted
it in the cluster. Bake static ClusterRoleBindings for the mulga:
groups into the eks-node image, project a principal's cluster-scoped
associated policies onto those groups in the TokenReview webhook, and
reject namespace scope at association time since there is no
namespace-scoped RBAC writer.
* feat(iam): evaluate trust-policy conditions and accept aws:SourceAccount
evalTrustPolicy read Effect, Action and Principal but never Condition, so a
stored aws:SourceAccount clause was a control nothing enforced. Give it a
condition arm evaluated in both the Deny and Allow passes, following the
StringEquals-only, fail-closed-on-unknown-key shape
evalTrustPolicyForWebIdentity already uses. The value is the caller ARN's own
account, derived server-side rather than caller-asserted.
Relax validateWebIdentityCondition from a single-action gate to a per-action
allowlist: sts:AssumeRoleWithWebIdentity keeps its existing IRSA keys, and
sts:AssumeRole now accepts aws:SourceAccount under StringEquals. Everything
else still fails at write time, naming the operator, key and action rather
than a bare MalformedPolicyDocument, and CreateRole/UpdateAssumeRolePolicy now
carry that reason to the caller instead of swallowing it into slog.Debug.
Extend the identity-policy refusal at validateStatementRestrictions with the
reason for aws:SourceArn and aws:RequestTag/*, the two keys a companion change
records as divergences in rfc0-7_aws-divergence.md.
* test(iam): fix trust-policy reason tests after aws:SourceAccount now valid
PR #1046 merged into dev with fixtures asserting aws:SourceAccount plus
StringEquals on sts:AssumeRole is refused. Part B of mulga-yxd9p makes
that combination valid, so the fixtures now exercise an unsupported
action and an unsupported key instead.
* docs: drop bead and plan-decision references from test comments
Bump bluebottle for the new key. requestConditionKeys now maps this
door's internal principal-type spelling to the canonical AWS value
through the bluebottle constants, and omits the key for an
unrecognized type rather than supplying an empty string.
ValidateTrustPolicyDocument builds precise errors naming the statement index
and the offending element, and both callers discarded them into slog.Debug
and returned a bare MalformedPolicyDocument. Debug is below the default
level, so the reason reached nobody: not the caller, and not the operator
reading the gateway log.
The contrast with the identity-policy path was stark. CreatePolicy names the
operator and the key exactly; CreateRole answered only 'The policy document
is malformed.', so an operator could not tell whether Principal, Action, a
size limit or a Condition block was at fault. The trust gate is the one the
terraform ECS module hits, which is what made this worth fixing on its own.
Both sites now use awserrors.Errorf, the same mechanism the identity path
uses. The separate Debug line is dropped rather than duplicated: the gateway
already logs a returned error at Error level, so one message now serves both
the client and the operator.
* feat(ecs): populate service event ring on DescribeServices
Events was always projected as null since ServiceRecord carried no event
history. Add a capped, newest-first event ring populated at the rollout
transitions the reconciler already computes: steady state reached, a
circuit-breaker trip, and a placement shortfall it cannot resolve this pass.
Emission is edge-triggered on the state change already guarding each site, so
the repeated reconcile loop does not refill the ring with one repeated
message every pass.
* feat(ecs): project createdAt on services and deployments
CreatedAt was already computed and persisted on ServiceRecord and Deployment
at creation time, but serviceToAWS never echoed it, so DescribeServices could
never answer which build wrote a given record. Project ServiceRecord.CreatedAt
and Deployment.CreatedAt/UpdatedAt, omitting the key entirely for a legacy
record whose timestamp predates the field rather than emitting the 1970 epoch.
CreateCluster refused accessConfig.authenticationMode = API_AND_CONFIG_MAP,
which is both AWS's default since 1.23 and the terraform-aws-eks module
default, and named no parameter in the refusal. The mode is now stored on
the cluster record and echoed by DescribeCluster, so a caller does not see
a permanent access_config diff on every subsequent plan; bare CONFIG_MAP
still fails, since Spinifex has no aws-auth ConfigMap path. The refusal
now names accessConfig.authenticationMode via awserrors.Errorf.
nodegroupRecordToAWS also only set Health when a node group had a status
reason, so a healthy node group described back with health: null instead
of the empty-issues object AWS always returns. Health is now always
constructed with a non-nil issues slice.
The unserved-service gate ran before signature verification but answered
SignatureDoesNotMatch, falsely blaming credentials that were never checked.
It now returns InvalidAction, the same code an unimplemented action on a
served service already gets, so the behaviour is consistent rather than
novel.
The envelope was also always XML, because jsonErrorService only lists
scopes the gateway serves and an unserved scope is by definition absent
from it. A JSON-protocol caller such as sqs or dynamodb therefore could
not deserialize the body and never saw the true error. writeSigV4Error now
falls back to the request's own X-Amz-Target and Content-Type headers to
pick JSON when the scope is not in that list, leaving every served scope's
envelope selection unchanged.
Ben Duncan