Treat the ECR registry JWT as a signed identity pointer rather than a
permission grant. Every /v2/* request now rehydrates the token's access
key against current IAM/STS state, maps the OCI route to an ecr: action
and repository ARN, and runs the identity-policy evaluator before
dispatch. PullOnly/ReadOnly principals can no longer push, overwrite
tags, or delete manifests, and key/session/user/role/account/policy
revocation takes effect on the next request instead of at token expiry.
Cross-repository blob mounts now require source-read permission and
proof the digest is referenced by a manifest in the named source repo,
closing an account-wide blob-fishing gap.
A CreateLoadBalancerSync whose data-plane launch failed returned without
rolling back the NLB managed security group. The orphaned SG sat in the EKS
managed control-plane VPC and blocked DeleteVpc with DependencyViolation
indefinitely, wedging the cluster in DELETING. The failure was also flattened
to ServerInternal, discarding the only useful diagnostic.
The sync path now unwinds fully on launch failure (ENIs, managed SG, name
claim, record) and maps the launch error onto its real cause: capacity
shortfalls pass through as their own code, anything else stays ServerInternal.
The async path deliberately keeps its failed record — the caller holds the ARN
and can reclaim it via DeleteLoadBalancer.
The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin
with the operator box's gateway identity, which has no S3 read grant on the
bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a
whole run. It now HEADs the object on the node over PeerSSH using predastore's
own [[auth]] credentials from predastore.toml (root-owned, so read via
passwordless sudo; the CA is readable through the spinifex group). The
credentials are read into shell locals, used inline for one aws s3api
head-object, then unset — never echoed, never leaving the node. A probe failure
that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe
error) rather than reporting a false zero, so a dead instrument can never look
like no growth. The dead operator-side S3 client path is removed.
Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
* fix(admin): round the AMI root volume up so the image fits
`spx admin images import` sized the root volume with integer division, which
floors: a 4.94 GiB image registered a 4 GiB volume. The image is then copied
into a volume too small to hold it, is truncated, and the guest boots with no
root partition — it stalls on the root device until systemd gives up, and
nothing anywhere reports an undersized volume.
floorVolumeSizeToAMI already exists to prevent exactly this ("Under-sizing
hangs the guest in dracut"), but it raises the caller's requested size to the
AMI's recorded VolumeSizeGiB — the same floored value — so it inherited the
mistake instead of catching it.
This stayed hidden because every current system image is an exact multiple of
1 GiB, where flooring and rounding up agree. It surfaced on a 5306470400-byte
image, which would not boot.
Round up instead, via a helper so the boundaries are testable: an exact
multiple must not gain a spare GiB, and one byte over must.
* fix(eks): default a nodegroup's root volume to 20 GiB, as AWS does
CreateNodegroup stored diskSize verbatim, so omitting it persisted 0 and
the launch skipped its block device mapping entirely, dropping the worker
onto whatever volume the AMI itself declares.
That was harmless while the node AMIs baked generous virtual disks — the
GPU image declares 16 GiB for ~2 GiB of content, so the fall-through
happened to leave plenty of room. It stops being harmless as soon as an
image is sized to its contents: the worker then boots onto a volume with
barely a spare GiB and cannot pull the container images it exists to run.
Sizing the node is the nodegroup's decision either way, not a property
the image should be able to dictate by accident.
20 GiB matches the AWS EKS managed-nodegroup default. It is resolved at
create rather than at launch so DescribeNodegroup reports the size the
node actually gets.
TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping asserted the old
fall-through and is replaced by its inverse; the explicit-size case is
already covered by TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping.
* fix(ecs): size a container instance's root volume to 30 GiB by default
ProvisionCapacity built its RunInstances input without a block device
mapping, so a container instance inherited whatever volume its AMI
declares. EC2 has no default of its own to fall back on — the AMI is the
only thing that decides — and a node AMI is sized to hold the image, which
leaves the agent no room for the container images it exists to pull.
30 GiB mirrors what the AWS ECS-optimized AMI declares. ProvisionCapacity
is spinifex's own API rather than an AWS one, so DiskSize is added to its
input for callers that want to size a cluster explicitly, defaulting when
omitted the same way InstanceType and Count already do.
* fix(images): pull the Ubuntu catalog images from releases, not a pinned daily
Both Ubuntu catalog entries pointed at a dated build under the codename
dailies tree, which Canonical prunes as builds age out. That path now 404s,
so "spx admin images import --name ubuntu-26.04-x86_64" fails outright for
x86_64 and arm64 alike — the entries had rotted with nothing to catch it.
Release builds are retained rather than pruned, so the pin stays resolvable
while keeping a specific, reproducible build. The filenames under releases/
carry the version instead of the codename, which matters because the sums
entry is matched on the URL's basename.
CI is unaffected and could not have caught this: it fetches resolute/current/,
a symlink that never rots, so nothing exercised the pinned path.
Debian pins a dated directory too but is fine — those builds are retained,
and its entries still resolve.
Pulls a container image inside a guest onto a viperblock volume and samples
guest used bytes, host backing-store bytes, nbdkit RSS and the live
checkpoint size as it goes, so backing-store cost can be fitted against the
bytes the guest actually wrote.
Complements the N-pass overwrite workload, which rewrites one extent and so
pins the block map size — that shape cannot show whether cost grows in
proportion to bytes written or faster than them.
Redirects both dockerd's data-root and containerd's root onto the target
volume. Redirecting data-root alone leaves containerd extracting layers into
/var/lib/containerd on the AMI's small root disk, which fails with ENOSPC
while the target volume sits empty.
The storagegrowth gate called ResolveCACert unconditionally before
building the AWS client, even though newAWSClient itself skips CA
resolution entirely when SPINIFEX_AWS_INSECURE=1. That blocked the
documented runner-resident remote path (env12 and similar clusters
with no local CA cert on disk) and the skip message falsely claimed
no node was running when the real cause was a missing CA. Skip the
CA check under SPINIFEX_AWS_INSECURE=1 and state the actual cause
plus the escape hatch in the skip message.
Rewrites one logical extent N times through full attach/format-write/detach
cycles, so the guest-visible footprint stays constant while total bytes written
scales with N. That shape isolates the leak where viperblock mints chunk objects
under unique monotonic keys and never deletes the superseded ones: a sequential
write-once supersedes nothing and so cannot exercise it.
The suite drives the workload only and deliberately does not measure the
backend. Measuring on-disk bytes from here would mean either a host du (which
cannot separate live from dead bytes, and so cannot distinguish this leak from
the unrelated one where an overwritten key's old extent is orphaned without a
tombstone) or a dependency on tooling outside this repository, which would stop
the suite building standalone. It instead reports what it did to a caller-named
result file and asserts only what a guest can observe: a sha256 round-trip
through a fresh mount per pass, and a footprint that stays constant across them.
The volume is retained on request, because deleting it removes the evidence:
DeleteVolume enumerates the bucket by prefix and drops every key under it,
including the leaked chunks.
Phase 8 of TestNATUplink checked the second VPC router's lr-nat-list with a
single non-polling assert; the igw-default-snat row commits a beat after the
gateway LRP appears, so the read raced the OVN commit and intermittently saw
an empty list (flaky CI red). Wrap it in EventuallyErr, matching the sibling
LRP and host-route polls in the same phase.
AddSNAT logged only on the idempotent-skip path, so the first install was
silent and RCA could not tell a late SNAT from a missing one. Emit an install
log keyed by router + vpc_cidr + external_ip.
Server-cert IP SANs were built solely from live interface enumeration
(DiscoverLocalIPs) plus bindIP, so the canonical mgmt-bridge IP entered
the SAN set only when br-mgmt happened to be up at cert-generation time.
On a host where br-mgmt is down when the cert is minted, 10.15.8.1 was
silently dropped and the EKS control-plane publish to https://10.15.8.1
failed cert verify, leaving the cluster stuck CREATING.
Pin config.DefaultMgmtBridgeIP into the extra SANs for both the init and
join cert paths so publish succeeds regardless of interface state at
generation time. Add a regression test asserting the SAN is present when
br-mgmt is not a live interface.
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected
ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN /
AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the
addon-sync agent injects — and asserts the pod's in-cluster
aws sts get-caller-identity resolves to the assumed IRSA role. This closes the
gap left by IRSAWebIdentity, which only exchanges the token host-side.
Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a
1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted)
and reaches awsgw STS over the worker subnet NAT.
The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI
publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but
the publish step hardcoded /tmp and could not find the freshly built raw.
GPU legs failed 'Raw image not found'; Alpine legs silently reused stale
pre-existing /tmp raws on the persistent runner. Mirror the build script's
derivation so publish looks where the build actually wrote.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Josh Siviour