Six defects in the EC2 describe path, found by running unmodified Terraform Registry modules against a live cluster and planning immediately after a clean apply. They share one shape: the write succeeds, the read-back is incomplete, and the caller cannot tell.
Populate Association on the primary network interface. The public IP was projected onto the instance but never onto the NIC, so the AWS provider read associate_public_ip_address as false for an instance that had a working public address. That attribute is force-new, so a no-op plan proposed destroying and recreating the instance.
Stamp EnclaveOptions. AWS always returns it, and its absence rendered as a permanent enclave_options block in every plan.
Stop emitting empty Reservations. The reservation map entry was created before the filter predicate ran, so a filter that excluded every instance in a reservation still emitted the reservation with no instances. Projection and filtering now run before the insert, in both the live and the KV path.
Merge reservations by ID in the gateway. Response frames were concatenated rather than merged, so one reservation could appear twice in a single response and which copy landed first varied between calls.
Mint the reservation ID once per RunInstances call. Each node minted its own, so a launch spread across nodes came back as several reservations. The gateway now allocates the ID and passes it to every node in a NATS header; a node that does not read the header mints its own as before, so the change is additive in both directions.
Answer an incomplete DescribeInstances sweep with a retryable ServiceUnavailable. Suppressing a false InvalidInstanceID.NotFound when a node had not answered was correct, but no branch was ever written for that case, so the caller received an empty 200 that read as authoritative. A filters-only query with no matches is still a legitimate empty result.
Accept the AWS spellings of the DescribeVpcs filters. The filters were implemented but named cidr-block and is-default where AWS uses cidr and isDefault, so Terraform's aws_vpc data source was rejected outright. Both spellings now work, since an internal caller depends on the old one. The parse error also names the offending filter to the client instead of only the log.
Resolve image owner aliases before filtering on them. describe-images --owners self compared an alias against a numeric account ID and matched nothing; the correct resolution already existed but ran after the filter.
Also builds the segscan test oracle with -buildvcs=false. That is unrelated to the above and fails on current toolchains regardless of this change, but it blocks preflight.
Host reservations, ECR image scanning and the flow-logs integration template
were declared not applicable, and the reserved instance family with them. None
of those are ruled out: reservations and scheduled instances can be mocked over
capacity reservations the way spot instances already are, scanning needs only a
scanner behind the API, and the template needs CloudFormation.
What stays is what the platform cannot have: retired AWS features, operations
whose content is an AWS price or counterparty, and AWS-hosted services with
nothing behind them here.
The roadmap carries IPv4 pools, local zones as an installation option, and an
S3 gap closure naming inventory and intelligent tiering. All three were
published as operations this platform will never serve.
Twenty-nine EC2 operations — local gateways, CoIP pools, BYOIP and the zone
group opt-in — and nine S3 operations are no longer declared. Carrier gateways
stay, as a route into a telco's 5G network in a Wavelength Zone.
The internal report listed every unimplemented operation together, so the 87
EC2 operations declared not applicable sat in the same list as the work, where
anything reading it for candidates would pick them up.
They are now listed apart, grouped under the reason they were declared with,
and the counts line reports them as not applicable rather than as gaps.
EC2 published one not-applicable operation against 625 in the model, so the
page was silent about a large surface that is absent for a structural reason
rather than an unfinished one.
Eighty-six more are declared across nine families: EC2-Classic, the 2016
resource ID migration, FPGA images, Elastic Graphics, Outposts and the edge
zone gateways, capacity purchasing, AWS backbone telemetry, Marketplace
product codes and BYOIP. S3 loses its replication and analytics declarations,
which named work that is not ruled out.
Multi-AZ clusters, read replicas and point-in-time restore were declared not
applicable, which says the platform will never serve them. All three are
planned, so the pages claimed something the roadmap contradicts.
Only a declaration now publishes that claim. A handler that refuses an
operation today establishes nothing about tomorrow, so it reads as the gap it
is rather than a permanent one.
Splitting them into a second table repeated a reason on every row that shared
it and separated an operation from the rest of the surface it belongs to. One
table carries every operation the platform answers, served or refused, and the
reasons return to numbered footnotes below it.
S3 was the one served surface with no coverage page. The gateway has no
operation-name dispatch table for it, so there was nothing to compare against
the model.
Predastore now declares the operations each of its S3 routes answers, so the
generator compares that inventory the same way it compares the other nine
services. The page credits Predastore rather than Spinifex, and its intro
separates what is routed from what conforms, pointing at the ceph/s3-tests
report for the second.
The index listed a count per service and left the reader to add them up. It
now opens with the total and closes the table with it, counting only the
services it links.
The coverage pages led with a percentage and an "x of y" count, and every
table listed the modelled operations Spinifex does not implement, which reads
as a scorecard against AWS rather than a statement of what the platform serves.
A published page now opens with the operation count alone, tables the
operations the gateway serves, and carries a separate not-applicable table with
each reason inline instead of footnotes. The unimplemented operations and the
counts move to docs/coverage/internal-report.md, which no docs-site slug points
at. A service with no implemented operations has no publishable page.
Every node's startup pass, the account.created event and the gateway's
EnsureDefaultVpc request can each build an account's defaults on a different
node. Each checked for an existing record, then wrote a fresh random ID with
Put, so racing nodes built a VPC, security group, route table and internet
gateway each. A per-account claim, written with Create, now fixes the IDs
before anything is built; every record is created under those IDs with
Create, and only the creator announces it to vpcd. nextVNI retries its CAS
instead of failing the loser.
spx admin init now discards the node's pre-formation JetStream store by default for --nodes >= 2; --discard-jetstream=false restores the refusal. spx admin kv digest gains --json with stream replica, retention and limit settings and per-sequence subject, time and TTL. The new spx admin kv compare reads every node's digest and fails only on real divergence: different content at one sequence, a missing message nothing explains, or wrong replica placement. install-node.sh gates formation on it and gains --ipsec on|off, verified on every node. The multi-node guide documents both.
A weekend gap is two runs of evidence lost on a suite whose remaining
failures are a rate rather than a bug. GitHub queues scheduled runs behind
every other top-of-hour cron and sheds them under load, which is why
2026-09-14 never fired and why every run started 3h15m-3h45m late.
* fix: reboot a guest by asking it, not by pulling its reset line
RebootInstances was QMP system_reset alone. The guest kernel is never told,
so it does not sync, does not unmount and loses every dirty page - which is
how a reboot issued seconds after a write loses that write, and how a guest
rebooted during cloud-init can fail to come back at all.
Guests now launch with -action shutdown=pause, so a clean guest poweroff
leaves QEMU paused rather than gone. Reboot presses the power button, waits
for the guest to reach that state, then resets and resumes the same process:
no volume, tap or state churn, and no window where the guest is missing. A
guest that will not go quietly is hard-reset, which is what AWS documents.
The button is pressed again every 10s while the guest ignores it. One signal
is not a shutdown request on a guest whose acpid has not started, which is
the same boot race that produced the failure.
Two consequences handled here: a guest that powers itself off now leaves
QEMU paused, so the heartbeat quits it; and a guest launched before the flag
exits on powerdown rather than pausing, so it is reset without being asked.
* fix: answer a reboot when it is accepted, not when the guest is back
Asking a guest to shut down takes as long as the guest takes. RebootInstances
gives the daemon five seconds to reply, so a reboot that waited for the guest
reported a reboot that had in fact succeeded as InvalidInstanceID.NotFound.
Manager.Reboot now admits the reboot — the instance exists and is running — and
runs the powerdown and reset in the background, which is the contract EC2
documents. The two admission failures are still reported; nothing after them
can be, so a reset that fails is logged and left to the heartbeat.
The gateway also stops calling a NATS timeout NotFound. Only the absence of a
subscriber says anything about where the instance is, which is how the sibling
start and terminate paths already read it.
* test: prove the churn reboot by its boot ID, not by how fast it happened
The subtest waited ten seconds and then read /proc/uptime. A reboot is
asynchronous, so the guest answering at that point may be the one that has
not shut down yet, and its uptime is whatever it was before — which the test
reads as "reboot may not have occurred".
Waiting for the boot ID to change is positive proof the guest rebooted and
does not depend on how long the guest takes to go down. The uptime assertion
stays, now measured against a boot that is known to be new.
* test: prove the graceful reboot against real guests and a real engine
Two tests the reboot design asked for and the unit tests cannot give.
Cell 18 launches two guests that differ only in how they answer the power
button. One installs a shutdown hook writing an fsynced line a second, so the
file that survives measures how much of its shutdown it was allowed to run —
a guest reset instead of asked leaves none of it. The other ignores the power
key entirely and must be reset once its budget expires, with the daemon saying
so. Each variant asserts the evidence only it can produce, and the slow one
asserts the absence of the line the wedged one requires.
Cell 27 loads a PostgreSQL instance past its shared buffers with pgbench and
reboots it dirty. The probe is an UNLOGGED table: PostgreSQL writes no WAL for
one, so a clean shutdown checkpoints it and crash recovery truncates it
unconditionally. Rows committed normally survive a hard reset too, so the
obvious assertion would pass against the bug.
The rds suite and cell 27 both gain budget for the load phase.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly
* fix: stop probing and nudging the EIP datapath of a stopped instance's port
* fix: log the real client IP, request ID and duration_ms in the awsgw access line
* test: mark the UI proxy header test as in-package
* fix: discard pre-formation JetStream data when forming a multi-node cluster
A node that ran as its own single-node cluster carries KV streams into
formation. nats-server adopts a local stream with the same name as the
clustered one, and raft catch-up only fetches sequences beyond a
follower's last, so the conflicting prefix is never repaired and replicas
diverge permanently.
- spx admin join removes the local JetStream store once formation
succeeds. A store holding streams now needs --force, and the
identity-discard warning names it.
- spx admin init --nodes >= 2 refuses a store that holds streams unless
--discard-jetstream is given, and removes it once every node has joined.
Keeping it as the seed is unsafe: stream leadership ignores who holds
data, and unassigned local streams are deleted as orphans.
- Neither removes a store while `spx service nats` runs, --force included.
- install-node.sh passes --discard-jetstream and waits for every
`spx service` process to exit before formation.
* feat: add spx admin kv digest for per-replica JetStream consistency checks
Replicas of a stream can hold different content while raft reports them
current, and nothing in the cluster detects it. A direct get over NATS is
no way to inspect one replica: a client's request goes to a random member
of the direct-get queue group, routed replicas included.
kv digest copies this node's $G streams tree (without consumer state) to a
temporary directory, opens the copy in a private in-process server with no
listener, and prints first/last sequence, message count and a SHA-256 over
every stored message per stream. Running it on each node and diffing the
output shows which streams diverge; --seqs locates the first differing
sequence. The live store is only ever read.
* fix: let spx admin kv digest read replicated streams
A replicated stream's meta.inf carries num_replicas > 1, which a standalone
server refuses to recover, so the digest could not read any production store.
Rewrite the copy's config to one replica with no expiry and a recomputed
checksum, and test it against a three-node cluster and an expired bucket.
* test: use assert.ErrorIs for the removed JetStream store
* test: mark the JetStream store and digest tests as in-package
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: retry a recovery relaunch that lost a race with a starting dependency
A guest drain-stopped by a coordinated cluster shutdown could be lost for good.
The relaunch ran before JetStream was serving, the volume lease claim timed out,
and the failure was recorded as permanent.
The backoff this needed already existed. relaunchWithRetry retries five times,
but only on ErrMountRetryable, and mountErrRetryable recognised exactly two
viperblock state-load sentinels. A lease claim returning context deadline
exceeded matched neither, so the mount was reported as permanently failed and
the backoff never engaged once.
- Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and
include it in mountErrRetryable. A lease held by another node is deliberately
excluded: that is an answer, not an outage.
- Probe readiness rather than connectedness. checkViperblockReady tested
natsConn.IsConnected(), which is true against a server whose JetStream is
still starting; it now calls AccountInfo with a timeout.
- Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked
in error by a dependency-unavailable reason, once the backing store is
confirmed ready, for a bounded number of passes, then leaves them alone.
Only mount-state and launch failures qualify; a failed reconnect or
pre-relaunch hook can mean the guest itself needs looking at.
Closes the e2e gap that made this feel covered. The node-failure tests stop the
service units directly so guests keep running and recovery is a QMP re-attach to
a live process. Nothing stopped a guest, so nothing relaunched one from a sealed
volume against dependencies that were still starting.
TestMultinodeClusterRestartResumesGuests does, including the wait for every spx
process to exit that makes it a real restart rather than a stop that returned
early.
Beads mulga-v7kp4, mulga-kyb1d.
* fix: stop every spx process by unit in the cluster-restart helper
The first CI run of TestMultinodeClusterRestartResumesGuests failed at the
process wait with "node1 still has 2 spx process(es)", and the same thing
happened by hand on a prod node.
Every service is the same binary, and the stop was driven from a hard-coded
list. StopNode names seven units and omits spinifex-northstar and
spinifex-qmp-collector, which is exactly the two survivors. The unit name now
comes from the surviving process's cgroup, so the set that stops is whoever is
still running the binary and the list cannot go stale when a service is added.
StopNode itself is left alone: its list is deliberate for the hard outage it
simulates, where the target's drain must not fire.
Also marks recovery_retry_test.go as //test:in-package — the retry policy is
unexported and the tests drive it through the seams restore_test.go uses.
* fix: stop spinifex.target in the cluster-restart helper, not the units
The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and
never started again.
StopNode stops the service units directly and leaves spinifex.target active.
That is right for the hard outage it simulates and wrong for a real shutdown —
the target then reads as satisfied while its units are dead, and the later start
raced a stop job that was still queued.
The helper now runs what the operator procedure and update-nodes.sh run: stop
spinifex.target, wait for every spx process to exit, and additionally wait for
systemd to have no spinifex job left queued before starting. That last wait is
the one that closes this race.
* fix: wait for the right NATS peer count in the cluster-restart helper
WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two.
Asking for three could never be satisfied, and the run that hit it had in fact
restarted all three NATS correctly.
The QEMU guard never fired. pgrep -x and pkill -x match comm, which the kernel
truncates to 15 characters, so qemu-system-x86_64 is stored as qemu-system-x86
and neither the -x name nor the -x alternation could ever match. The script
therefore never signalled a guest and never waited, and wiped viperblock state
out from under live guests. Match the 11-character prefix against comm, the same
rule qemuProcessPrefix already applies in vm/orphan_scan.go, and scope it to the
user the daemon runs guests as so a host carrying libvirt guests alongside
Spinifex ones does not lose the wrong set. Not -f: that matches the whole
command line, where argv[0] may carry a path.
Nothing waited for spx either. Stopping spinifex.target returns once the target
is inactive, but spinifex-shutdown.service is ordered After= the storage
services, so their stop jobs queue behind its ExecStop and NATS can still be
running. JetStream then rewrites its index after the rm and before the
verification, failing the reset on a file the script had already deleted. Add
the wait update-nodes.sh has carried since d9bc6e35.
The firewall restore was gated on a mode file the failed run had already
deleted, so a retry read the absence as "never armed", skipped the restore, and
install-node.sh honoured that and formed the cluster around an open node. Stash
the mode outside every wiped tree, and fail rather than warn when a node that
arrived armed cannot be put back.
install-node.sh had the other half of that cascade: it disarms every host to let
the OVN raft ports through and only re-arms at the end, so any abort in between
left them all open. Re-arm from an EXIT trap, and say so loudly when a host
arrives with no policy.
The wipe loop deletes files but keeps every directory, so one empty
$G/streams/KV_<name> directory per stream survived the reset. NATS enumerates
those on start and reports them as HA assets it must restore, leaving the meta
group at size=4 with no leader over hundreds of assets that have no data behind
them. Every service then fails to reach JetStream and the daemon aborts its
bootstrap.
Remove the tree outright, mirroring the predastore/cluster removal above it.
Josh Siviour