An internet-facing ALB's address is now tagged managed-by at allocation, and
ReleaseAddressByInstanceID leaves any untagged record alone with a warning, so
a wrong instance ID can no longer release a customer's own address.
Resolves GHSA-7rv5-mrxv-xcr9, GHSA-wp97-x4hv-qvhp, GHSA-97r2-qh9m-474h,
GHSA-9wx9-5pv9-3349, GHSA-f86p-h4j2-7qgj (seroval) and GHSA-g3hv-75wg-m249
(minimatch, dev-only via eslint, forced with a pnpm override).
The conformance and NBD client suites, the fence, lease, dirty-marker,
abandon, takeover and stale-mount tests each stand up their own embedded
JetStream and Config, so they now overlap instead of running back to back.
Tests that touch package or process-wide state stay serial.
The six account-teardown tests and the unanswered network-test case each
wait about 5s on a production timer. Calling t.Parallel() first queued all
seven through the parallel slots; starting the teardown or network test
before t.Parallel() lets the wait run during the serial phase instead.
Also generate 1024-bit RSA keys in TestMain and shorten the test-only
context in TestQueryMetaNodeStatus_Unreachable.
go test -coverprofile at GOMAXPROCS=4: 16.9s -> 7.1-8.0s.
The four tests that wait out the fixed 3s describe fan-out deadline for a
silent node (9s, 6s, 6s, 6s) were queued behind the package's short
parallel tests, two of them as subtests of a table test. Hoist the two
slow table cases into top-level tests and move all four into a file that
sorts first, so they take parallel slots before the short tests do. The
table body becomes a shared helper; no case or assertion changes.
Each test stands up its own embedded JetStream server, so top-level tests
and independent subtests now call t.Parallel. Tests that redeclare the
process-wide cluster size or capture the default logger stay sequential.
Package wall time drops from ~23.5s to ~4.4s locally.
Six account-delete tests each wait one 5s teardown drain poll, and the
network-test and storage-status tests wait out fixed 5s and 2s timeouts.
Each uses its own embedded NATS, so they run concurrently: the package
drops from about 37s to 8s.
* test: pin instance inventory gaps G1-G3 as current behaviour
* feat(ec2): strict account instance inventory from the record store
* feat(ec2): strict spot-request and volume inventories
* fix(ec2): strict instance inventory fails on node-only instances
* fix(ec2): strict instance inventory sees error and late strays and rejects malformed records
The ecs-agent now holds an aws.CredentialsProvider (a CredentialsCache over
imdscreds.NewProvider) and ecsgw.New takes a *gwsign.Signer, matching eksgw,
rdsgw and elbv2. imdscreds.Fetch and stsauth.AssumeRole return aws.Credentials,
so agents/ecs/credentials and the field-copy adapters are removed.
* feat(uefi): delete a terminated instance's variable store through its store record and report unowned stores
Terminate, on both the running and stopped paths, moves the store record
to deleting and stamps a uefi teardown entry, then returns. The
cluster-wide uefi-store reaper deletes the objects under the instance
prefix and then the record, so the deletion converges even after the
terminated record's TTL has expired. CompareAndSet refuses to return a
deleting record to live.
spx admin volumes orphans now also lists instance prefixes with no
owner, image prefixes with no AMI document, and store records with no
instance record. It deletes nothing.
* refactor(uefi): read instance state for the store reaper and orphan report through daemon capabilities
The reaper and the orphan report named the raw instance-state records,
buckets and key prefixes, which ADR-0007:S6 forbids a new consumer to do.
They now ask TerminateDecided, OpenTeardowns, MarkTeardownDone and
KnownInstanceIDs, each stating its absent-record and failure behaviour.
* fix(uefi): keep a stopped instance's store when its terminate left the stopped record
TerminateDecided read the terminated record first, so a stopped terminate
that wrote it but failed to remove the stopped record let the reaper delete
the store of an instance that can still start. The live record now decides,
and the reaper no longer marks a live store deleting while one exists.
* feat(uefi): launch a UEFI instance from its variable store pointer
A launch whose record points at a variable store keeps the live copy at
<DataDir>/uefi/<instance-id>/vars.fd and opens it as pflash unit 1 with
format=raw. It reuses the copy when seeded.json beside it names the
pointer, and otherwise gets the object, decompresses it, checks it is
exactly the node's VARS size and writes vars.fd then seeded.json by
temporary file and rename. A failed get is retryable like a mount; a
missing or malformed object is not.
A store whose firmware arch or VARS size differs from the node's fails
the launch before any volume is mounted, with a Server.InternalError
state reason and a log naming both firmwares. A different CODE hash
boots, and is logged and counted.
A UEFI instance with no pointer and no EFI volume, or with both, is
refused with InternalError before its record is changed, in launch,
Start and the claimed-instance start. An instance whose variables are on
an EFI volume still launches through it.
* test(uefi): separate the embedded store from the injected error in the failing-get fake
* test(uefi): call startQEMU with its one argument after the dev merge
* fix(ec2): classify a volume held elsewhere by the provider code, not lease text
The instance service and the EKS cluster reconciler recognised a start
refused because another node holds a volume only by viperblockd's lease
text, so a provider that words the refusal differently was reported as
ServerInternal and counted against the reconciler's restart budget.
The instance service now matches the EBS provider's volume_held_elsewhere
code, on both the stopped-instance claim and the live-owner start. The
owning node marks such a refusal with an X-Volume-Held-Elsewhere reply
header, the ec2.start forwarder relays it, and StartInstances wraps
ec2v1.ErrVolumeHeldElsewhere around the rebuilt error, which the
reconciler matches with errors.Is. The AWS code stays IncorrectState.
The lease text stays as a fallback for replies that carry no code: the
legacy ebs.mount reply, a publish below schema version 3 (where
volume_in_use also means a read-only mismatch), and an owner node that
predates the header.
* fix(daemon): relay every header of the owner's start reply on both forwarder paths
The ec2.start forwarder copied only the held-elsewhere header onto a
relayed reply, and its unsealed-writes branch answered with the body
alone, so any other header the owning node set was lost in the relay.
Both relay branches now go through relayStartReply, which forwards the
owner's reply body and every header unchanged.
* test(daemon): pin that only a held volume gets the held-elsewhere header
The owner's reply header is the one signal that a start was refused
because another node holds a volume, and every other start refusal in
the same IncorrectState code must reach the caller without it.
* refactor(ec2): classify a volume held elsewhere only by the provider code
Every node sends provider requests at the one contract version, at which
every provider refuses a volume held by another node with the
volume_held_elsewhere code. The lease-text fallbacks in the instance
service and the EKS cluster reconciler matched replies that no longer
exist, so they are removed: a start is a held-volume refusal only when
the mount error carries the provider code, or when the start error wraps
ec2v1.ErrVolumeHeldElsewhere, which StartInstances rebuilds from the
owner's header.
The ec2instance.ErrVolumeHeldElsewhere alias is removed, so the sentinel
has one name, ec2v1.ErrVolumeHeldElsewhere. The reconciler's budget test
now proves an IncorrectState without the sentinel still counts, whether
its message is unrelated or carries the lease text.
* fix(ec2): log a held-volume start refusal at Warn with the instance ID
A start refused because another node holds a volume is returned as
IncorrectState, a client error, yet was logged at Error without the
instance it concerned. It is now logged at Warn, and every vmMgr.Start
failure log names the instance.
Tags are checked against the ELBv2 model's key and value length and
pattern (ValidationError), repeated keys are refused with DuplicateTagKeys,
and more than 50 tags on a resource with TooManyTags, counting a resource's
existing tags on AddTags. A tag with no Value is stored as an empty value
and a tag with no Key is refused rather than dropped.
Ochre's two streaming pumps now record through one streamRecordCtx.record and
frame upstream faults through one frameWriter.writeUpstreamFault. The EKS and
Lambda route tables list their actions through rest.Router.Actions,
toDetails converts toSummary's identically-shaped struct, Lambda's
environment response uses aws.StringMap, and the e2e suites call
harness.DiscoverNanoInstanceType and harness.EnsureKeyPair directly.
* test: pin guest poweroff handling and stopped-instance volume state
* test: gofmt the poweroff and stopped-volume tests
* test: pin start after detach, one-poll poweroff, reattach of released volumes and the delete refusal
* test: separate the embedded mounter from the recorder fields
* fix: end a guest poweroff as a stop and keep stopped instances' data volumes attached
A clean QEMU exit after the heartbeat saw the guest power off now stops the
instance instead of waiting for crash detection. Unmount no longer releases
data volumes. DetachVolume on a stopped instance edits the record, then
releases the volume. Starting an instance stopped on an older build drops data
volumes that are attached elsewhere or deleted and fails on a missing boot
volume.
* fix: keep the stopped-instance detach inside the instance service file and format start reconcile
* docs: describe guest poweroff, stopped-instance volumes and detach on a stopped instance
* fix: allow detaching the root volume of a stopped instance and refuse a start with no root volume as AWS does
* feat: attach volumes to a stopped instance and fix root-detach start fixtures
* docs: describe attach and root detach on a stopped instance
* test: Unmount seals only and leaves every attached volume in-use
Changed with the person's explicit approval, relayed by dev-design on 11-03.
EBS root and data volumes persist across stop on AWS, so stop and crash
unmount no longer release any attached volume.
* docs: note that the EFI variable store cannot be detached from a stopped instance
* test: unmount contract tests expect no volume state write on stop
* test: order imports in stopped-volume daemon tests
* refactor: drop the unused volume state updater from the volume mounter
* test: console log trim, removal by instance state and append-mode chardevs
* fix: cap and reap instance console logs, always append
* test: expect append mode on microvm file chardevs
* fix: retire console logs on terminate instead of reading instance state
* docs: describe console log location, caps and retirement on terminate
* test: drop removed append argument from startQEMU calls
* docs: state that GetConsoleOutput returns NotFound once an instance has been torn down
* docs: drop the unverified AWS retention claim for console output
* feat: remove idle console logs that no QEMU on the node holds open
* fix: keep idle console logs when a QEMU pidfile names a pid the scan cannot see
* test: eks node log forwarder filter, redaction, caps and role wiring
* feat(eks-node): forward k3s agent warnings and errors to the serial console
A redacting, rate-capped forwarder tails the agent log and writes warning,
error and node-status lines to the console so they reach the sink. Secrets are
redacted before the 512-byte cut.
InstanceStatus and VM carry an optional UEFIVarStoreRef naming the store
object and the firmware it was written under. The new uefistore package
holds FirmwareIdentity, UEFIStoreRecord and UEFIWriteBack, and a Store over
the spinifex-uefi-stores bucket (history 1, no TTL, schema 1) whose writes
are all revision-guarded and validated. The daemon creates the bucket at
startup. Nothing reads or writes either yet.
* feat(ebs): carry the fence notice over the provider contract subject
viperblockd now sends the contract FenceNotice on
ebs.provider.v1.<node>.fenced when it revokes a live export, and the daemon
subscribes to that subject whatever the EBS provider, stopping the guest
using the fenced volume as before. The legacy ebs.<node>.fenced emission,
subscription and their viperblockd legacy contract types are removed.
NotifyFenced is the shared single-attempt sender; viperblockd keeps its
retry-until-answered loop around it. MemoryProvider can revoke a
publication and announce it, and the conformance suite gains a
RevokeExport case that viperblockd and MemoryProvider both run.
* fix(ebs): send the fence notice through the contract request path
NotifyFenced now shares requestReply with NATSProvider.request, so a
notice gets the connected check, the X-Account-ID header and the
account-attributed client span the other contract requests do. A
client span also fails for a reply it cannot read.
A notice that can never be sent (no volume, an unaddressable node, no
connection) is ErrInvalidArgument and terminal, and FenceNoticeRetryable
names the errors worth sending again: unheard, or refused as
unavailable. MemoryProvider.RevokePublication retries until heard or
its context ends, and shares unpublish with UnpublishVolume.
* fix(viperblock): refuse an unaddressable node and retry an unavailable fence notice
viperblockd refuses to start, and RegisterProviderSubjects to register,
when the node name is not one contract subject token: there would be no
fence-notice subject, so a fenced guest would keep running. Nothing is
subscribed before the refusal.
publishVolumeFenced keeps sending a notice the daemon refused as
unavailable, stops on one that can never be sent, and no longer checks
the subject itself. The late fence completion is shown to send its
notice.
* fix(daemon): refuse an unaddressable node and trace fence notices as the contract does
NewDaemon refuses a cluster config node that cannot address the fence
notice subject, instead of starting with no fence-notice subscription.
handleFenceNotice opens the EBS provider server span, so the notice is
attributed to the sender's account, and every refusal goes through one
path that records its code on the span and answers in the notice's
version. An unsupported version names the version it got.
Tests hold the guest's stop open to show no ack leaves before it ends,
and drive the notice through subscribeAll's real subscription table.
The user reaper removes each group membership before DeleteUser and the group reaper detaches attached and deletes inline policies before DeleteGroup, so a user in a group or a group carrying policies no longer strands the identity stage. DeleteUser now ignores a group name with no group record, as it already does for an access key ID with no key entry.
* feat(ebs): drain the volume inside each provider's CreateSnapshot
viperblockd now drains a mounted volume's export over snapshot.sock in the
background snapshot work, before reading the live checkpoint. A refused or
stale socket fails the snapshot; no socket with the volume lock free means no
client is connected and nothing is buffered; no socket with the lock held is a
client between connections and is reported as retry-later.
Write-through providers drain nothing: the control-plane DrainVolume and the
daemon's drain handler skip the snapshot.sock drain when the node's EBS
provider is not viperblockd, so a snapshot or CreateImage of an attached
volume on qemunbd no longer fails. The control-plane drain stays for
viperblockd nodes that predate the provider drain.
The conformance suite gains a case that CreateSnapshot answered on the queue
subject by a node other than the volume's holder is refused, while the
holder's own snapshot succeeds, run against viperblockd through a third node
reachable only on the queue subjects.
* fix(ebs): refuse a snapshot whose last export client left unsealed WAL
The plugin releases its volume lock even when the seal on disconnect fails,
so a free lock alone did not show the last client's writes reached the
backend. With no drain socket and the lock free, the snapshot now also
requires that the volume holds no local WAL, or that a seal receipt attests
to it.
A socket path of 108 bytes or more cannot be listened on, so a held lock
there is reported as a failed drain rather than as retry-later, and the path
bound is now inclusive. The drain logs with the request context.
* fix(qemunbdd): snapshot a published volume through its export
qemu-nbd holds a published volume's qcow2 open with the write lock, so
qemu-img convert of the file failed with "Failed to get shared write lock",
and reading it with -U would race the metadata qemu-nbd caches. A published
volume is now copied from the NBD export, which serves exactly what the
attached client wrote; an unpublished volume keeps copying the file.
The reader needs a second connection beside the attached client. Any
--shared limit above 1 makes qemu-nbd advertise multi-conn, after which a
client opening several connections blocks on those over the limit, so the
export now accepts any number of clients.
* fix(ebs): require retry-later from the queue-subject snapshot refusal
The conformance case now requires the refusal to carry ErrUnavailable rather
than any error. The viperblockd harness returns a struct instead of five
positional values and stands its queue-only node up only for the case that
asks for it.
* docs(ebs): say only the snapshot accept reply is freed of the drain
* fix(ec2): warn when a drain is skipped for a provider not known to write through
Only viperblockd serves the drain socket, so the skip itself is unchanged, but
it is logged at Info only for qemunbd and at Warn for any other provider. The
daemon logs the host-side skip under its own message.
* fix(ec2): log a provider snapshot refusal and surface retry-later as ServiceUnavailable
CreateSnapshot flattened every provider failure into an unlogged
ServerInternal. It now logs the provider error with the volume and snapshot
ids, and maps a provider ErrUnavailable to ServiceUnavailable as the delete
path already does. CreateImage's snapshot step shares the mapping.
* fix(qemunbdd): refuse a snapshot of a published volume
Reading a published volume through its export needed qemu-nbd to accept more
than one client, which drops the guard against a second writer, and the copy
it made of a live export was not point-in-time. CreateSnapshot now refuses a
published volume with volume_in_use before running qemu-img, and the export
is back to a single client. An unpublished volume still snapshots from its
file.
* refactor(ec2): drop the control-plane snapshot drain
Every viperblockd in a cluster drains the volume inside its own
CreateSnapshot, so CreateSnapshot and CreateImage no longer dial
snapshot.sock or route a DrainVolume command to the instance's host.
Removed: DrainVolume, DrainVolumeSocket, DrainsThroughSocket and the
skipped-drain logging in the snapshot domain; the state.json read and
the volumestate package only it used; the daemon's DrainVolume handler
and the DrainVolume command, payload and response in ec2v1. CreateImage
now snapshots the root volume through one provider path whether or not
the instance is running.
Tests assert that both EC2 paths snapshot an attached volume through the
provider without opening the drain socket or messaging the host, and
that viperblockd drains a mounted volume exactly once.
* fix(viperblock): prove no local writer before an unmounted snapshot
The unmounted branch of CreateSnapshot opened its own engine once the lease
was held, but the lease is reentrant on the volume's own host, so an nbdkit
left running unregistered after a failed recovery rebuild kept taking writes
the snapshot never saw. The branch now takes the volume lock before the open
and holds it until the snapshot is read; a held lock is drained through the
export's socket, or refused as retry-later when it serves none.
A seal receipt now approves local WAL only if it is newer than every change
under the WAL, so a receipt from an earlier connection no longer covers WAL a
later one failed to seal. A refusal for a held lock with no drain socket is
logged at WARN with the socket path.
* fix(ec2): map a provider volume-in-use snapshot refusal to IncorrectState
A provider that cannot snapshot an attached volume, as qemunbdd refuses a
published one, surfaced as ServerInternal. CreateSnapshot and the CreateImage
snapshot step now return IncorrectState telling the caller to detach the
volume first.
* docs(e2e): describe the stopped CreateImage path as the single snapshotVolume path
* chore(viperblock): order the io/fs import as gofmt does
* feat(daemon): mount and unmount through PublishVolume behind the local provider's version 3
The mounter asks its own node's provider for capabilities on the per-node
subject and, when version 3 is listed, mounts and unmounts every volume
through PublishVolume and UnpublishVolume at version 1, recording the
published export on the request. Otherwise it keeps the legacy
ebs.<node>.mount and unmount subjects.
A retry-later or not_found refusal and a publish with no responder back off
as ErrMountRetryable, so restore keeps the legacy retry. No responder on
any mount or unmount drops the cached route so the next call re-reads
capabilities. An unpublish takes the provider's data-movement budget.
* fix(daemon): treat a publish reply with no usable export as an ambiguous mount
NATSProvider.PublishVolume now wraps ErrNoPublishedVolume and
ErrInvalidExport, keeping its messages. The mounter turns either, and an
export it cannot apply, into vm.ErrMountAmbiguous, so hot attach runs its
defensive unmount and Mount's rollback unpublishes the failing volume as
well as those already published.
* fix(daemon): give the contract unpublish its own seal budget
DetachVolume bounds its context for the unplug and then seals, so an
unpublish on that context got whatever the unplug left. The mounter now
detaches the unpublish from the caller's cancellation, keeping the trace,
so NATSProvider's data-movement timeout governs the seal as
unmountSealTimeout does on the legacy path.
* fix(ebs): let NATSProvider requests buffer through a reconnect
request refused whenever the connection was not connected, which includes
reconnecting, while the legacy subjects' plain nats request buffers and
sends once the connection is back. Only a nil or closed connection now
refuses.
* fix(daemon): roll back a mount over the route the mount took
Mount's rollback re-read the route, so after a failure that dropped the
cached one a contract-published volume could be unmounted over legacy
ebs.unmount. The rollback now takes the route the mounts took. Unmount,
unmountOne and the rollback share one seal path that is given the route,
replacing the per-route helpers and their error out-parameter.
* test(daemon): cover a not_found unpublish and a legacy no-responder
An unpublish answered not_found counts as sealed for Unmount and
UnmountOne, and a mount that finds no responder on the legacy subject
re-reads capabilities, so a provider restarted at version 3 takes the next
mount over the contract.
* refactor(daemon): reuse endOpSpan in the mounter and correct its comments
The contract spans end through the existing endOpSpan, the adapter's doc
comment fits three lines, and configureEBSProvider no longer claims its
provider is the only one wired, since the mounter builds its own.
* refactor(daemon): mount, unmount and abandon over the contract at version 3
A cluster runs one EBS contract version and every provider in it serves
version 3, so the volume mounter no longer reads its node's capabilities
to choose a route. Mount, MountOne, Unmount and UnmountOne always send
PublishVolume and UnpublishVolume at version 3, and Abandon is an
UnpublishVolume with NoSeal whose torn-down field replaces the legacy
abandon reply. The mounter sends nothing on a legacy ebs.<node> subject.
Export mapping, the retry-later and ambiguous-mount mappings, the seal
budget, the detached rollback context and the provider error chain are
unchanged. Tests that drove the legacy route or route selection are
removed or moved onto the contract subjects.
* refactor(ebs): remove the per-node capabilities subject
The volume mounter was the only consumer of ebs.provider.v1.<node>.capabilities,
and it no longer reads capabilities to choose a route. NodeCapabilitiesSubject,
its wildcard and verb, NATSProvider.GetNodeCapabilities and the natsserve and
viperblockd subscriptions serving it go. The cluster-wide GetCapabilities and
its contract version list stay.
* fix(ebs): refuse requests on a disconnected connection and carry the account
NATSProvider refuses a request again whenever its connection is not
connected. A request buffered through a reconnect is flushed after its
caller may have given up, so the provider would act with nobody waiting,
such as creating a volume the caller reported as failed.
A reply that cannot be decoded, or that carries a schema version this
build does not serve, now wraps ErrUnreadableReply, so a caller can tell
a request the provider received from one it never saw.
WithAccountID puts an account on a context. Every request made under it
carries the X-Account-ID header and names the account on its client
span, and the server span reads it back from the header.
* fix(daemon): retry a refused publish, tear down an unknown one, attribute every call
A publish refused because the connection is down was never sent, so it is
now vm.ErrMountRetryable and restore backs off. A publish that timed out
or whose reply could not be read may have started an export, so it is
vm.ErrMountAmbiguous and Mount and AttachVolume unpublish it.
Publish, unpublish and abandon put the instance's account on the context,
so the provider request carries it on the wire and on its span. The
mounter's own per-call span is gone, leaving one client span per call.
Abandon opens one span naming the instance, account and reason, and a
failed volume's error names the reason. Its bound is a field so a test
can check that an unanswered abandon gives up within it.
The mounter takes the daemon's provider rather than building a second
one, sends at NewLatestVersioned, calls unpublish from UnmountOne and
leaves export validation to the provider client.
* refactor: describe the single-version cluster and the publish path
NewVersioned and NewLatestVersioned no longer describe an N/N-1 rollout
or a capability exchange, since a cluster runs one contract version. The
volume path names publish and unpublish rather than the legacy ebs.mount
and ebs.unmount subjects, and ErrMountAmbiguous describes a publish reply
with no usable export.
* test(viperblock): pin the held-elsewhere text on a version 3 refusal
Callers still recognise a volume held by another node by its message, so
the held_elsewhere refusal must keep ec2v1.VolumeHeldElsewhereText.
* fix(daemon): tear down a publish whose caller cancelled after sending
A publish the provider received and the caller then stopped waiting for has
an unknown outcome, like a timeout: the provider may have started the export.
Classify context.Canceled from a sent publish as ErrMountAmbiguous so Mount
unpublishes it and hot attach rolls it back, both under a context the caller's
cancellation does not reach.
A caller already gone before the publish starts sends nothing, so publish
returns its context error without calling the provider and without the
ambiguous classification; a disconnected connection stays retryable.
The PXE build rebuilt spinifex as root inside the workspace, and a cancelled
job's root make kept running for an hour, so the next job on the runner could
not write bin/spx. The microVM image is now built as the runner user, the PXE
build skips its spinifex rebuild, and both root steps run in named systemd
scopes that are stopped when the job ends and before the next one starts.
* feat(compute): boot the root volume from a -blockdev node that takes NBD or host_device
The boot volume moves from -drive if=none to a -blockdev node named os, so
a block export reaches QEMU as host_device as data volumes already do. The
NBD boot shape keeps its cache, reconnect, iothread and bootindex settings;
werror and rerror move onto the virtio-blk device. A block EFI volume is
still refused because pflash has no host_device form.
* test(viperblock): pin the boot -blockdev shape in the nbdkit orphan reaper
The boot volume's NBD server options now carry a file. prefix, which the
referencer scan matches only because it compares by substring. Cover the
unix and TCP boot strings so a stricter match cannot reap a live root
export, and stop describing the joined host:port form as a boot drive.
* fix(compute): refuse a disk-boot launch with no boot volume
Data-volume -blockdev nodes satisfy Execute's drive check on their own, so
a disk-boot request list without a boot volume would start a QEMU with
nothing to boot. The disk-boot path now refuses it before launch.
* refactor(compute): share the NBD node options and export dispatch
The data and boot -blockdev builders share one helper for the nbd server,
export and reconnect-delay options, buildDrives dispatches both branches
through one export helper, and hot attach parses its URI with the shared
parser. Emitted arguments are unchanged. The BlkDeviceMapped doc now states
when it falls back to the scalar form.
* test(compute): cover an unnamed boot device with no medium in the device map
* feat(ec2): delete live launch templates in platform cleanup when no consumer names them
A cleanup request may ask for live templates to be deleted. It then reads which
templates records outside EC2 (nodegroups) name, deletes the rest first, and
checks key pair and placement group references against what remains. A failed,
partial or absent consumer read deletes nothing and blocks every live template.
Requests that do not ask behave as before.
* feat(accountlifecycle): clean L09 kinds through the EC2 platform cleanup on ordinary delete
Ordinary delete evaluates quiescence again under the same fence before the
platform stage, cleans only the kinds that record as quiesced (live launch
templates only when EKS names none), and re-reads the owner inventory before
the fenced DeleteAccount. Any remaining L09 record or incomplete re-read keeps
the account TERMINATING; Force is not forwarded. The key-pair, placement-group
and launch-template reapers and the dry-run shadow comparison are removed.
* test(accountlifecycle): Force never bypasses a reference on an ordinary delete
Until versions can be published, the only version is $LATEST: the list holds
its configuration with the FunctionArn qualified as :$LATEST and no
NextMarker. MaxItems outside 1 to 10000 is refused, and Marker is refused
outright rather than guessing AWS's paging behaviour. It is authorized as
lambda:ListVersionsByFunction on the function ARN and does not need the code
store.
* test(eks): pin async nodegroup delete and per-nodegroup instance profile
* feat(eks): delete nodegroups asynchronously and own one instance profile per nodegroup
* fix: confirm nodegroup workers are gone before deleting their profile, and kick the delete after the request
* fix: end a stuck nodegroup delete as DELETE_FAILED, and cover nodegroup delete in the EKS e2e
* docs: describe asynchronous nodegroup delete and the per-group instance profile
* test: keep the EKS e2e bystander instance profile until the parent test ends
* test: cover nodegroup delete failure paths, profile ownership refusals and worker absence checks
* feat(admin): non-finalizing DeleteAccount exercises the quiesced EC2 platform kinds
A non-finalizing DeleteAccount quiesces the account, records the S9 evaluation, cleans only the key-pair, placement-group and launch-template kinds it recorded as quiesced, then re-reads the owner inventory and accounts for what remains. The job ends EXERCISED with the account still TERMINATING; a later ordinary delete takes the same operation over.
Only an exercise records a quiesced scope, so ordinary deletes still record none. The cleanup capability gains a kinds selector, awsgw binds the in-process owners, and the instance reference reader fails closed.
* refactor(admin): name the takeover of a finished exercise exercise_finished
Code signing is unsupported, so the read only proves the function exists and
returns AWS's body for a function with no code signing config: both members
present and null. A missing function is ResourceNotFoundException. It is
authorized as lambda:GetFunctionCodeSigningConfig on the function ARN and
needs neither the code store nor environment decryption.
Serves GET /2015-03-31/functions/{FunctionName}/configuration with the
same name, qualifier, ownership and environment handling as GetFunction,
authorized as lambda:GetFunctionConfiguration on the function ARN. It is
a metadata read, so it neither opens nor presigns the code store.
Dev-only code delivery behind the lambdatestlink tag: the host attaches the
ZIP as a read-only disk and passes its exact size and SHA-256 through
fw_cfg. The agent reads those values before sealing fw_cfg, and installs
the package only after authenticating and receiving the host's first
frame: it checks size and hash, validates every entry before writing any,
extracts through os.Root into a nosuid,nodev tmpfs at /var/task, and makes
it read-only. Any failure exits without starting the bootstrap.
Entry names that could escape the task root, entries other than regular
files and directories, and sizes over the dev bounds are refused; declared
sizes cannot wrap the running total, and actual bytes are counted whatever
the header claims. The bounds are dev limits, not AWS limits.
StartBootstrap now starts the bootstrap in its own directory, so packaged
functions can open their files by relative path.
* feat(config): refuse a spinifex.toml whose nodes resolve to different EBS providers
* feat(daemon): lock a cluster to one EBS provider in spinifex-cluster-config
The first daemon records its provider at ebs-provider with Create, which
also backfills an existing cluster. A daemon whose provider differs from
the recorded one refuses to start, naming both providers and the recovery.
* fix(config): compare only explicit EBS providers across nodes
A generated spinifex.toml writes no ebs stanza for peers, so treating an
absent peer provider as viperblockd refused every non-default cluster.
* fix(daemon): exit on an EBS provider lock refusal and refuse a malformed lock
A mismatch now exits 1 so the unit shows failed. A recorded value that
does not decode or names no supported provider is a distinct permanent
refusal quoting the raw value. Permanence is a sentinel matched with
errors.Is, and the cluster-config bucket is declared beside the others.
* style(daemon): align JetStreamManager fields
* feat(ebs): contract schema version 3 for the provider data path
Servers accept schema versions 1 to 3 and answer every request in the
version it was sent at, while every client still sends version 1.
Version 3 adds no_seal to UnpublishVolumeRequest, refused below 3, and
torn_down to UnpublishVolumeResponse. UnpublishVolume now returns an
UnpublishedVolume so the torn-down result reaches callers.
A publish refused because another node holds the volume returns
volume_held_elsewhere at version 3 and volume_in_use below it, which is
what v1.23.0 returns. A read-only mismatch on republish stays
volume_in_use at every version.
GetCapabilities lists the contract versions served and is also answered
on a per-node subject. The unsealed-writes subject is part of
EBSProvider, and write-through providers answer it with nothing held. A
fence-notice event subject and message are defined. Unpublish gets the
data-movement timeout.
The memory provider, natsserve and qemunbdd implement all of this. A
qemunbdd republish whose qemu-nbd has died re-exports the volume instead
of returning the dead socket.
* feat(viperblock): serve contract schema version 3 from viperblockd
Every provider subject replies in its request's version, and
GetCapabilities lists the contract versions and is also served on the
node's own capabilities subject.
PublishVolume no longer returns an existing entry early. It goes through
mountVolume, which hands back a live export and drops and remounts one
whose nbdkit has died. Mount failures map to not_found, held_elsewhere
(version 3 only, volume_in_use below it), volume_in_use for a mode
mismatch, unavailable for a backing or lease store that is not ready,
and internal otherwise.
UnpublishVolume with no_seal abandons the export without sealing it,
and reports whether an export was torn down.
* test(ebs): conformance cases for contract schema version 3
Every provider run through the suite now has to:
- list ascending contract versions including the legacy one, answer each
in that version, and refuse an unlisted one without changing state;
- refuse a publish elsewhere with volume_held_elsewhere only at version 3
and volume_in_use at versions 1 and 2;
- refuse a republish in the other access mode as volume_in_use, never
held elsewhere;
- report torn_down truthfully, refuse no_seal below version 3, and either
honour no_seal at version 3 or refuse it as unsupported;
- answer the unsealed-writes query with a subset of what was asked.
SuiteConfig.KillExport lets a provider whose export is a process prove
that a republish after the export died returns a serving export.
qemunbdd and viperblockd supply it.
* fix(ebs): stop a running export before remounting over its lost socket
An export whose socket is gone was treated as dead even while its
process ran, and a republish started a second export beside it. A
client connected before the socket went keeps writing through the old
process, so that made two writers on one volume.
qemunbdd now SIGTERMs that qemu-nbd and waits for it to exit before
exporting again. viperblockd stops the nbdkit through unmountVolume, so
the old copy is sealed as a normal unmount would seal it. If the process
will not stop, the publish is refused as unavailable and the entry is
kept for the next attempt.
* test(viperblock): give the no-seal tests a seal they could reach
The no-seal unpublish and abandon tests left BaseDir empty, so
volumeNeedsSeal was false and no path could seal whatever the code did.
Both now create local state under BaseDir first, and check that a seal
is reachable before asserting none ran.
* refactor(ebs): leave version rules to the provider and trim the v3 surface
The provider decides what a request's version allows, so natsserve no
longer gates TornDown or held-elsewhere on it: classifyError takes no
version, and the reply carries what the provider returned. viperblockd
applies the TornDown rule in unpublishVolume, its provider half.
The unpublish handlers check the version once, through
CheckUnpublishVersion, and classify its refusal. Each handler binds its
reply version once.
UnpublishedVolume is reduced to TornDown. movesData classifies a subject
by its parsed verb, which replaces isNodeSubject and the owner-verb
list. nodeSubjectWildcard is unexported. The conformance suite reads
capabilities once.
* test(ebs): check v3 success replies, timeouts and refusals end to end
- Unpublish is sized as data movement, and publish, unsealed writes and
per-node capabilities keep the caller's timeout.
- A mount whose lease store or backing store is not ready reaches the
wire as unavailable.
- A successful publish replies in the request's version at every served
version, in natsserve and in viperblockd. Every natsserve case now gets
its own seeded provider, so none relies on an earlier case's state.
The conformance republish runs at every listed version.
- viperblockd's per-node capabilities list the versions it serves.
- A no_seal unpublish above the served version is refused, and the
export is neither torn down nor sealed.
- The dead-entry republish test asserts unconditionally. The snapshot
lease test runs against real volume state and asserts nothing was
written. The conformance torn-down case skips a provider that does
not list version 3.
* fix(ebs): wrap the stop failure behind an unavailable publish
The refusal now wraps the stop error as well as ErrUnavailable, so a
caller can still inspect why the old export did not stop. Also fixes
import order and spacing that the linter flagged in the new tests.
* fix(admin): read the newest network test by its claim key, so a listing never leaves it out
* fix(admin): keep the newest-test lookup additive, so it never fails a listing
The lambdatestlink agent now loads its identity, seals fw_cfg and dials the
host exactly as the normal agent does, then carries the test stream over
that mutual-TLS connection instead of a device path. Nothing tenant-facing
starts until the host's first valid frame: a refusal after the handshake,
an early close or a shutdown never starts the bootstrap. Each VM serves one
invocation and then stops.
The normal agent is unchanged and still accepts no invocations. The tagged
tests join the race tier for the two packages that carry the build tag.
* fix: find node planes from the node's configuration, and admit only a node's own address as a network test target
Node facts looked for planes only on br-wan, br-lan and br-vpc, so a host
whose addresses sit on plain NICs (a Vultr or OCI VM) reported no planes:
the topology had no links, every port read unused, and Run test stayed
disabled. Each plane now resolves to the device holding the node's
configured address: ovn-encap-ip for vpc, the node's own host for lan,
and vpcd.external_interface (or the bridge it is a port of) for wan, with
the br-* names as the fallback.
A run request now names its target node, and the client admits only the
address that node reports for the plane. The old subnet rule refused
legitimate WAN tests on Vultr, where each VM has its own /23, and admitted
strangers sharing a cloud subnet.
The network test's selects showed their raw values (__all__) because Base
UI's Select needs its options as items to render a label.
* fix(ui): list a plane on a plain NIC without an empty bridge, and note a shared LAN port as well as a shared bridge
* fix(vultr): turn on public addresses in the default subnets once the Vultr pool is live
* fix(ui): say where an RDS database goes, and how to use another VPC, when no DB subnet group exists
A stop/start that leaves the guest stuck in cloud-init-local waiting on
IMDS fails the restart SSH probe, but the diagnostics only captured the
EIP/OVN side, which was healthy. Capture br-imds flows, the endpoint's
link, addresses, sysctls, reply rule and route, the responder listener
and conntrack too, so the next occurrence names the broken hop.
* fix(rds-agent): stop a command blocked on a backup hold from keeping out its release
The agent ran commands one at a time inside its poll loop. A MariaDB
ALTER USER issued while an automated backup held BACKUP STAGE blocked at
its commit, so the unquiesce that would release the hold could never be
delivered: the rotation timed out at the API while the new password was
already live in memory, and the instance stayed backing-up.
Commands now run on their own goroutines while the agent keeps polling,
re-published copies of a running or answered command are ignored, and a
finished reply cuts the current poll short so it is carried at once.
MariaDB SetPassword also waits for any hold to end before it runs, for
less than the control plane's budget, so a rotation that cannot finish
fails before touching the engine.
* fix(rds-agent): keep engine commands serial and bound every command by its budget
Engine-mutating commands go back to running one at a time on a single
lane, as before; only unquiesce runs beside it, so a command blocked on a
backup hold still cannot keep out the release.
Each command is bounded by its type's control-plane budget, now exported
as rdsinstance.CommandBudget and measured from receipt. A command whose
budget ran out while it queued is answered as expired and never run, so
a stale quiesce cannot put the engine in backup mode. Quiesce takes its
bookkeeping lock within its context, and the MariaDB ALTER USER carries a
lock_wait_timeout from what is left of the budget.
* test(rds): rotate the MariaDB master password through an automated backup
Opens the backup window on purpose and rotates the password back and
forth until one rotation is in flight while the instance reports
backing-up, asserting each outcome is whole and that the backup releases
the engine. Every suite create now pins its backup and maintenance
windows clear of the run, so no other subtest's outcome depends on the
hour the cell runs.
* fix(rds): cap an agent command by the budget its issuer has left
The agent measured each command's deadline from receipt, so a first copy
delayed by a gateway or bus outage was granted a whole fresh budget when
its issuer had nearly given up; a late quiesce could then take a hold
nobody would release. The issuer now stamps every republished copy with
the budget it has left, a duration that needs no clock agreement, and the
agent uses the smaller of that and its own type budget.
* feat(ui): console feedback round: state tones, inline tags, filter bar and form fixes
Settled states go green and transitions yellow with a spinner, status text is
lowercase, tags are edited inline row by row, every list filters through one
filter bar, long cells truncate with a tooltip and times are relative.
EC2 launch takes a Name and offers ICMP echo, and leaves Placement Group out
when no group is available. RDS create is reordered with engine tiles and
window pickers. Route 53 records get bulk delete, a TTL select, value labels
that follow the type, and NS. Tooltips use the popover tokens, and the console
is sized over a 110% root.
* feat(ui): size RDS instance classes and give each record value its own input
An RDS class is a db.-prefixed EC2 type, so the class picker shows its vCPUs
and memory in every option and under the field, as the EC2 launch form does,
on create, modify and restore.
A Route 53 record's values were one textarea, which let a line break into a
value. Each value is now a single-line input with Add value; Enter starts the
next one, a pasted list splits into rows, and CNAME takes exactly one.
* fix(ui): open a DNS record for editing from the keyboard
* feat: admin node inventory, topology, live host metrics and network test
Each node collects its hardware facts as root at daemon start (dmidecode,
ethtool, networkctl LLDP where available) into /etc/spinifex/node-facts, and
the daemon serves them with its plane wiring over NATS.
New admin-only actions, refused for every account but the system admin
account regardless of policy:
- GetNodeFacts: hardware, NICs, LLDP neighbours and planes per node
- GetNodeMetrics: live CPU, memory and per-plane traffic for one node
- StartNetworkTest / DescribeNetworkTests: iperf2 between nodes over a
chosen plane, both directions, one test cluster-wide at a time, recorded
in KV and converged asynchronously
The iperf listener runs only for the duration of a run, without
capabilities, on TCP 5001, which the peers firewall rule now allows. A run
refuses any target outside the node's own plane subnet.
The console gains a node panel (hardware, network, metrics, network test
with a dial scaled to the port's line rate) and a topology view built with
@xyflow/react. setup.sh installs dmidecode and iperf.
* test: cover the node insight handlers and the facts file mode
* feat: report every CPU flag in node facts
* feat(ui): node table on the dashboard, every CPU flag and a readable topology
The dashboard's nodes section is the Nodes page table, open by default, and a
row opens that node. Ready reads green, capacity sorts by most available, and
the processor's features tip lists every flag. Topology draws one row per
port with switches in a muted card, highlights a node's links on hover, and
offers a left-to-right layout beside top-down.
* chore(ui): rebuild dist
* fix: align the CPU facts fields
* fix: bound the network test server start and read ARM CPU features and cores
* chore(ui): rebuild dist
* test(ui): pin the instances baseline clock and take the screenshots from CI's Playwright image
* fix: retry a network test claim whose holder released it mid-check
StartBootstrap sets no_new_privs on a dedicated locked OS thread and
starts the bootstrap from it, as uid/gid 993 with no supplementary groups
and only the environment it is given. The thread is never unlocked, so
the flag never reaches the agent's other threads, and no dynamic loader
runs before no_new_privs is set. The test-only agent start uses it
instead of running the bootstrap as root. The normal agent still accepts
no invocations.
* feat(lambda): open the authenticated control connection at guest startup
A normal lambda-agent build now reads its per-generation identity from
fw_cfg, unloads qemu_fw_cfg and refuses to start unless the fw_cfg tree is
gone, then dials the host under the pinned mutual TLS and holds the
connection. It accepts no invocations and never starts the bootstrap: any
data from the host fails closed until the message protocol exists.
The image loads qemu_fw_cfg and the virtio_net chain and configures the
guest interface from opt/spinifex/netcfg, as other system VMs do. The
lambdatestlink build also seals fw_cfg before starting the bootstrap.
* test(lambda): bound startup test waits and report host accept errors
* feat(iam): record quiescence evidence on the deletion job, with the L09 scope unresolved until the final inventory re-read
* fix(iam): cite ADR-0008:S9 as the basis for the STS consumer
* feat(lambda): add the Lambda runtime image and guest agent
* fix(lambda): stop the guest agent after a timed-out invocation
* docs(lambda): declare the L-GUEST executor-to-agent contract
* refactor(vm): remove the unused virtio-serial port builder
Lambda guest control moves to TCP over the microVM's network, so no
caller needs SerialPorts. This reverts d7905107b.
* fix(lambda): drop the serial carrier from guest startup and keep virtio-net
run-agent no longer waits for a carrier device named on the kernel
command line, init loads virtio_net instead of virtio_console, and the
image keeps virtio_net's failover and net_failover dependencies. The
normal agent still refuses to start without an authenticated transport.
* feat(lambda): authenticate the guest's TCP control connection with pinned mutual TLS
hostlink loads the per-generation identity from fw_cfg and dials the host
over TLS 1.3 with ALPN spinifex-lambda-guest/1. VerifyPeer requires exactly
the pinned certificate, validates it, and checks its single URI SAN names
the expected scope and role. The L-GUEST contract now describes this
transport; message framing and fencing stay open, and the normal agent
still refuses to start.
* feat(lambda): add the host end of the guest's TCP mutual TLS connection
guestconn creates fresh host and guest identities for each environment
generation, produces the fw_cfg entries the guest reads, and accepts only
that generation's exact guest certificate. Peers on the untrusted bridge
that fail the handshake are closed and the listener keeps waiting until
its context ends; the cancel callback finishes before the deadline is
cleared so the listener can be reused.
The pinned-certificate check and the L-GUEST vocabulary move to
foundation/pki and contracts/lambda/v1 so both ends share them.
* docs(lambda): describe the shared connection vocabulary in the contract README
* test(eks): specify the AWS-style cluster security group layout
* feat(eks): give clusters one AWS-style cluster security group
Create the eks-cluster-sg-<name>-<id> SG in the customer VPC with the EKS tags and a self rule, report it as clusterSecurityGroupId, launch workers with it (or the launch template's SGs), and attach it to the private-endpoint ENI. New clusters no longer get customer-VPC control-plane and nodegroup SGs; clusters created before keep theirs, and delete finds the new SG by tag.
* fix(eks): own the cluster SG by its recorded id and scope the delete sweep
Workers and teardown use the SG id recorded on the cluster; a create retry only adopts an SG carrying tags the service alone can set. DeleteClusterSGs now revokes every rule only on cluster-owned SGs and, on any other SG in the VPC, only the rules that reference one.
* docs(eks): use the cluster security group in workbooks and e2e
Workbooks read vpc_config[0].cluster_security_group_id and drop the hand
NodePort rules. E2E covers an internet-facing instance-target ALB with no
hand rule, exec and logs, and a bystander security group surviving
DeleteCluster.
* docs(eks): drop stale node-group SG sentence
* feat(operator): resume account deletions whose holder died
Each awsgw scans the deletion job bucket and takes over RUNNING jobs that
carry an accepted operation, a fence of at least 1 and an expired lease,
through the same fenced takeover a retried request uses. A resumed job runs
on a fresh timeout budget, so recovery no longer depends on the requester.
* chore: trim resumer comments to three lines
* refactor(cli): delete accounts only through the admin API
* docs: describe account delete as admin API only
* feat(iam): quiesce an account under a fenced deletion operation
* fix(iam): keep the termination capabilities off the shared interface and fence the account removal
* test(awsgw): guard the IAM termination binding on the operator deps
* docs: say which credentials the account delete signs with
* feat(lambda): add unregistered CreateFunction and GetFunction
Function records live in a daemon-owned per-account KV bucket, keyed by
Region and name, claimed with one conditional create that carries the
function ID and a Pending state. A claim whose code copy never completes
expires, so a later create can take the name over; the original gateway
then fails to mark its code stored. Code copies are never deleted here.
CreateFunction checks the ZIP is readable, refuses an execution role that
does not trust lambda.amazonaws.com (using the STS trust evaluation), and
encrypts environment values in the gateway with the master key, bound to
the account and function ID. GetFunction returns Code.Location as a
10-minute presigned URL for the off-host Predastore address, signed by a
session that outlasts it; with no off-host address it fails instead of
naming an internal one. Lambda uses the REST-JSON error envelope.
The handlers are built in the gateway but not routed: no public Lambda API
is registered until a runtime image exists.
* fix(lambda): return and accept only the default LoggingConfig
Every function configuration now carries LoggingConfig with LogFormat Text
and LogGroup /aws/lambda/<name>. CreateFunction accepts the default when
supplied and refuses a JSON format, another log group or a log level.
Adds an integration test that drives a signed CreateFunction and
GetFunction through the gateway, the NATS record owner and the code store,
then downloads Code.Location and checks the bytes. The STS trust check
moves to stsdomain.RoleTrusts so the test uses the production evaluation.
* fix(lambda): render the stored record in the CreateFunction response
CodeStored now returns the record the owner wrote, and CreateFunction
renders it instead of stamping its own time, so CreatedAt, LastModified
and RevisionId match an immediate GetFunction.
* fix(lambda): match IAM error codes, not messages, in code-store provisioning
IAM errors now carry a message, so comparing err.Error() with the bare
code no longer matched EntityAlreadyExists or NoSuchEntity. Provisioning
then failed on every start after the first and could not adopt its own
role. Resolve the AWS error code instead.
* test(iam): check the resolved NoSuchEntity code in UpdateUser tests
IAM errors now carry a message, so err.Error() no longer equals the bare
code. Assert the resolved AWS error code instead.
* test(iam): account for UpdateUser as named by bare name
UpdateUser addresses its user by bare name, so the cross-tenant ARN table
cannot target another account; list it with the other byName IAM actions.
Add a focused test that a rename stays within the caller's account.
Lambda code copies live in a system-account Predastore bucket reached
only through a short-lived session on a Regional role scoped to that
bucket. STS gains an in-process AssumeCodeStoreRole that derives the
Region's code-store role itself and assumes it as lambda.amazonaws.com;
it is not on the STS interface, NATS or HTTPS.
Provisioning creates or verifies the owner-tagged role, inline policy
and bucket on first use only, retries Lambda's own partial create,
and refuses an untagged or foreign resource with an operator-visible
reason. Copies are keyed by their SHA-256, so different bytes can never
replace a recorded revision. The gateway holds the root client and IAM;
the Lambda package receives only the bounded provisioning operations
and the scoped store.
* test: pin AWS-format EKS add-on versions
* feat: return EKS add-on versions in AWS format and accept them as input
* test: DescribeAddonVersions query parameters, compatibilities and paging
* feat: honour DescribeAddonVersions filters and report compatibilities
* test: EKS parameter refusals must carry a code the operation declares
* fix: EKS parameter refusals use a code the operation declares
* docs: describe AWS-format EKS add-on versions
Document the v<x.y.z>-eksbuild.<n> form, the bare-form leniency, the
DescribeAddonVersions filters and compatibilities, and how to pin a
version in Terraform. Replace the leave-addon_version-unset guidance.
The four crash cases each restarted the store, which on the three-node
cluster means a full Raft re-election every time. They now set up in
separate namespaces, crash once, and run the same post-crash checks as
named subtests, cutting TestR3/Conformance from ~39s to ~11s.
* refactor(compute): extract a shared direct-boot microVM builder
runtime/compute/vm gains MicroVMSpec, NewMicroVMConfig and
MicroVMMachineType, with CPU mitigations on unless a caller sets
MitigationsOff. The ELB and SQS system-instance launches pass their own
settings to it; only the load balancer opts out of mitigations. A golden
test pins the full QEMU argv for both launches, unchanged by the move.
* test(daemon): stub QEMU on PATH in the microVM argv golden test
Execute resolves a QEMU binary before returning the command, and CI
runners have none installed. The test now puts an empty
qemu-system-x86_64 on PATH, so it checks the real argument construction
without QEMU.
TestR3 restarts real three-node JetStream clusters and waits out Raft
elections, adding ~40s to the slowest preflight package while covering
~0.1% of its statements. It now skips unless SPINIFEX_SQS_R3 is set, and
make test-race sets it, so CI's Race Detection job runs it on every PR.
* fix(awsgw): admit accounts with a synchronous in-process status read
* fix(iam): refuse to move the system and admin accounts off ACTIVE
* fix(awsgw): deny a missing account record instead of failing the request
install.mulgadc.com serves the latest release's setup.sh, so a --channel dev
or --version install ran an installer older than the tarball it unpacked and
missed files it did not know about, such as lib/distro.sh. setup.sh now hands
off to the tarball's own setup.sh, with the same arguments, before it changes
anything on the host.
The Vultr guide now installs from the latest release instead of dev.
* test: pod identity webhook mutation, publisher and webhook config contracts
* feat: pod identity webhook mutator, CA bundle publisher and webhook configuration
* feat(eks): run the pod identity webhook on every control-plane server
Add the eks-pod-identity-webhook binary and its OpenRC service. It listens on
127.0.0.1:8444 with a serving certificate signed by the k3s server CA, authenticates
to the API server with a client certificate signed by the k3s client CA, and runs
under its own least-privilege ClusterRole. Each server stages the RBAC and the
MutatingWebhookConfiguration (caBundle is the server CA) into the k3s manifests
directory, so server-join nodes get them too.
Match upstream's container skip rule: skip only when the regional STS key is
present, and add the token mount otherwise.
* fix(eks): issue pod identity webhook certs for one year and renew in the running process
Both the serving and client certificates were valid for ten years. Kubernetes
cannot revoke a client certificate, so a copy taken off a control-plane disk
would stay a valid identity for a decade. Issue both for one year with the
existing 30-day renew window.
The process loaded the serving certificate once. A daily check now exits
with ErrCertRenewal when either certificate enters the renew window, so the
supervisor restarts it and EnsureCerts reissues.
Add tests for KubeClient against a TLS fake API server and for Run, which the
package had none of.
* test: e2e covers pod-identity webhook and unmodified LBC chart
IRSAPod now uses a ServiceAccount annotation only and asserts the injected
env and volumes plus an untouched unannotated pod. LBCChart installs the
upstream chart 1.11.0 and reconciles an Ingress into an active ALB.
* test: pass the tagging endpoint to the LBC chart and mark Shield/WAF flags Spinifex-only
* fix: make the pod identity webhook init script executable in the eks-node image
setup.sh chmods each shipped init script to 0755 explicitly, and the new
service was missing from the list. rc-update then refused it, and
eks-node-role, running under set -eu, aborted before starting k3s.
Add a guard test that every init script in INSTALL_FILES is chmod'd.
* fix: start the pod identity webhook last and keep its failures non-fatal
eks-node-role runs under set -eu, so a fault enabling or starting the
optional webhook aborted the control plane boot before k3s-first-boot,
state-report and addon-sync. Enable and start it at the end of the server
and server-join roles and log a warning on failure instead.
The role test now asserts the webhook is the last call and that an enable
or start failure still exits 0 and finishes the role.
* fix: move the pod identity webhook off the token webhook readiness port
eks-token-webhook already serves its plain HTTP readiness endpoint on 8444,
which the API server target group health-checks. The pod identity webhook
bound the same port, failed with address already in use and respawned
forever, so no pod was ever injected.
Listen on 127.0.0.1:8445 and add a test that the two ports differ.
* docs: describe pod identity webhook for user workloads in the EKS guide
* fix: split STUB_FAIL assignment from export in the role test
* feat(eks): converge DeleteCluster's teardown behind the recorded DELETING
DeleteCluster ran the whole cluster teardown inside the NATS request. Each
control-plane VM that ignores ACPI costs 40 s (20 s graceful powerdown, then
20 s waiting for the PID file), so a two-member cluster overran the 30 s
budget: the caller got a 500, and its retry queued behind the first handler
on the serial subscription and got a 404.
The request now validates, answers a DELETING cluster from its record,
records DELETING and returns. The teardown runs in a background converger
started by the request and backed by the deleting reaper:
- An in-process per-cluster gate sits in front of the teardown lease, which
re-adopts a key its own node holds; one per-node bound covers kicks and
reaper hand-offs.
- The lease renews on the service context, not the request's, and the
converger stops at a step boundary if it is lost.
- The converger acts only on a record that is DELETING when re-read, refuses
to remove a record that still has a node group, deletes the cluster's
child records before the meta, and deletes the meta only if no ownership
ref appeared that its pass did not tear down.
- DELETING is sticky for every status writer, and the create launch's meta
writes add their refs to a DELETING record instead of overwriting it, and
never recreate a swept one.
- The reaper hands work to the converger instead of running it inline,
retries at a backoff capped at 30 minutes without ever abandoning a
record, and resumes records an older build flagged exhausted.
- A stuck delete is surfaced in DescribeCluster as one INTERNAL_FAILURE
issue naming the step, never the internal error.
* fix(eks): refuse DeleteCluster while node groups are attached
AWS answers ResourceInUseException ("Cluster has nodegroups attached") when
managed node groups remain. DeleteCluster went ahead instead, and its sweep
erased the node group records, leaving their workers running in the
customer account with no record to own them and no control plane.
The request now decides that error before recording anything. The e2e
suites, the terraform workbooks and the account teardown engine all delete
node groups first; the console shows the refusal's message.
* fix(eks): refuse new add-ons and access entries on a deleting cluster
CreateAddon and CreateAccessEntry checked only that the cluster record
existed, so a dependant could be written to a cluster whose delete was
already recorded, and the teardown would then sweep it with the record.
Both now answer ResourceInUseException for a DELETING cluster, a code both
operations declare in the EKS model.
* test(e2e): DeleteCluster returns DELETING at once and never reads ACTIVE after
The first delete-cluster must return within 5 s with status DELETING, as
AWS does. While the cluster is still listed every DescribeCluster must read
DELETING, and a repeat delete must answer DELETING from the record; once it
is gone a further delete answers ResourceNotFoundException.
* docs(eks): record the cluster delete lifecycle contract
The contract ADR-0003 S1 asks for before a conformance claim: what the
request decides, the recorded obligation, how the teardown converges and
recovers, and the S8 and S7 tests that pin each item, with the remaining
gaps listed.
* test(eks): wait for the delete request before counting terminates
On a build whose DeleteCluster is synchronous the request goroutine is still
recording terminates when the assertions read, so the tests must wait for it
to return as well as for the background teardown.
* test(eks): classify every eks request subject by its handler's work
The ADR-0003 S8 subject table for EKS. Every eks.* subject the gateway sends
and the daemon serves is classified in eksSubjects, and the table fails for
one that is not, so a new subject is classified deliberately rather than by
inheriting the 30 s default. DeleteCluster is request-scaled. Four handlers
still hold the request for work that grows with the resource and are named
as not conformant: CreateCluster's FAILED-name reclaim, DeleteNodegroup,
UpdateNodegroupConfig and RestoreSnapshot. Their budget stays 30 s, since a
larger one would hide the defect.
* fix(eks): leave no child record behind a launch overtaken by a delete
A create launch still running when its cluster was deleted and swept could
seed the creator's access entry afterwards, leaving it under a cluster with
no record. The launch now re-checks the record after seeding and, finding it
gone, removes the cluster's child keys with the infra it holds. The teardown
also sweeps child keys again after deleting the meta, for a write that lands
between its two deletes.
* fix(eks): fence the create launch on its generation and keep its infra owned
Review findings on the async delete, each with a test that fails before:
- The launch's meta write checked only the status, so a launch overtaken by
a delete could overwrite a newer cluster of the same name and then release
by name. Writes are now fenced on CreatedAt and the ARN; a superseded
launch releases only the instances it started, by ID.
- A launch whose record was swept released its infra once, best effort. It
now recreates the record as DELETING with its refs, so the teardown and
the reaper own and retry the release.
- A record marked FAILED under a running launch was treated as owned by a
teardown; the launch now merges its refs and releases what it built.
- Child records: a fresh claim clears keys an earlier cluster of the name
left, the teardown stops the cluster's bootstrap subscriber, bootstrap
refuses to write into a record that is gone or DELETING, the creator's
access entry and the GPU device add-on are gated on the record.
- Deleting the last node group of a DELETING cluster resumes its delete,
and waiting on node groups no longer counts as a failed pass.
- One converger run is bounded at 15 minutes, so a hung step cannot hold
the gate and lease for ever; the timed-out pass is recorded.
- The FAILED-name reclaim purge runs under the teardown gate and lease.
- The meta-last test drives the converger with a failing KV instead of
DeleteClusterPrefix, which production no longer called and is removed.
- The S7 test asserts the teardown issues no volume delete and terminates
no worker.
* fix(eks): keep a deleted cluster's record while its create launch runs
Second review of the async delete, each behavioural change with a test that
fails before:
- A create launch now heartbeats its owner and time on the record, and the
teardown keeps the record while that heartbeat is fresh, so DescribeCluster
never answers NotFound and then DELETING again when an overtaken launch
recreates the record. The launch hands its refs over at its next write and
starts the teardown when it returns; a heartbeat silent for 90 s is taken
as a crashed launch, and recreating a swept record stays as the backstop.
- The launch's OIDC keypair writes are fenced on the record's generation, so
an overtaken launch cannot replace a newer same-name cluster's signing key.
- Re-creating a FAILED cluster's name is no longer refused by the per-node
teardown bound.
- A deadline reached while removing the record is still recorded; an exited
bootstrap subscriber removes its own entry; a record that is not DELETING
is refused as not recorded even when it has node groups.
The guest's SQS nats-server listens on 127.0.0.1 and the worker dials it
there, but the kernel leaves lo down, so the store failed to bind and the
worker never polled. Also correct the stale CA comment in init.
DescribeTargetGroups rejects a malformed target group ARN with ValidationError
and reports a missing group with AWS's message, DescribeLoadBalancers fails with
LoadBalancerNotFound for an unknown name, and DescribeListeners fails with
LoadBalancerNotFound for an unknown load balancer ARN. The EKS cluster NLB lookup
treats the new not-found as absent.
* feat(sqs): run queue and message owners in the worker by signed pull
The SQS worker's ready step now opens the queue and message owners on
its dedicated store and serves owner calls it pulls from its node's
gateway over the existing instance-role channel (PollWork,
CompleteWork). Each poll and post is authorized against the current
worker identity, and a result is accepted only from the identity that
took the work. No payload crosses platform NATS.
A receipt-signing key is created on the worker disk only under the
first-store decision that creates the store key, and its fingerprint
is bound in the store; a missing or changed key refuses to serve.
Received messages are released only when the gateway proves the caller
left; receives are capped so a worst-case result fits one post.
The public relay is not registered in production.
* fix: harden SQS relay results and bound pending bytes
Unavailable outranks client faults when encoding owner errors, and wire
errors carry fixed text only, never owner or backend text. The ingress
rejects incomplete or mismatched success results as unavailable, and the
relay bounds total pending input bytes as well as request count.
Adds tests of every relay action against real owners, the receive claim
limit, release, faulting work, the receipt-key fingerprint marker and the
poll loop through gateway failures.
* fix: refuse SQS serving outside a single-node host cluster
DescribeStore now reports the gateway's declared cluster node count, and
the worker refuses to serve, before reading keys, connecting or creating
any store state, unless that count is exactly one. A gateway that reports
no count reads as zero and is refused.
* feat(sqs): record a store server key pin once per store instance
Add the store role and its internal actions so the bound store instance
can learn its claim and record the public pin of its TLS server key,
once, without the platform holding the private key. The worker sees the
store as bound only after a pin exists and must receive a valid pin.
* refactor(sqs): serve store server keys from a separate capability
Keep the worker Stores interface unchanged and give the store guest's
actions their own ServerKeys dependency, so existing Stores fakes need
no store-key methods.
* feat(sqs): launch the store with a mgmt NIC, CA and scoped store role
The store now boots with the region, gateway URL and the daemon's own
cluster CA, a mgmt NIC to reach the gateway, and an sqsStoreRole profile
granted only the store's two internal actions, so it can authenticate
its server key pin. A store record that predates these fields refuses
to relaunch; its volume is kept.
* feat(sqs): keep the store TLS server key on its disk and record its pin
The store image gains sqs-store-key, run after the data disk is mounted.
It loads or creates a P-256 server key under the disk's tls directory,
records only the key's public pin with the gateway as the bound store
instance, and writes a self-signed certificate for it to the run
directory. nats-server then requires TLS 1.3 with that key.
A record for another volume, a recorded pin with a missing key, or a key
that does not match the recorded pin refuses the store; no key is
regenerated and nothing on the disk is deleted. Launch-not-yet-bound and
lost replies are retried. The store role vocabulary moves to
contracts/sqs/v1 so the guest agent does not import a domain.
* feat(sqs): pin the worker's store connection to the recorded server key
The worker's readiness check and serving connection now use TLS 1.3 and
accept only the store server key whose pin the bound store recorded with
the gateway. The pin is checked before the worker key or any queue or
message data is sent, and on every reconnect. A store with another key,
a plaintext store or a malformed pin is refused as final.
The worker NKey client authentication is unchanged. The host still
holds the store's data disk key, so the pin does not protect the store
from a privileged host.
* fix(sqs): retry store key bootstrap and track guest binary sources
The store's server key check moves from sysinit into the respawned run
step: a gateway or IMDS outage exits after a pause so init retries it,
without remounting or creating a new key, while a missing, changed or
wrong-store key is recorded as refused and never retried. Stop also ends
a running key helper before unmounting.
The guest helper and worker binaries now depend on every main-module
package dir and file they link, from go list, so edits, additions,
deletions and renames all rebuild them.
* feat(sqs): operator enable and status with a signed ingress test
Add a per-Region SQS service record (disabled by default, pinned to one
node, refused on multi-node clusters before any write), gate the relay on
it, and expose `spx admin sqs enable` and `spx admin sqs status`. The
gateway now hands queued work to the polling worker and reports whether
one is polling.
The public SQS service stays unregistered in production until its
capability profile is reviewed. An integration test registers it through
the harness and drives real SigV4 and IAM requests to a real worker for
create, send, receive and delete, asserting no SQS body crosses platform
NATS.
* fix(sqs): report readiness only from observed component state
Status now reports whether the tenant SQS API is registered on this
gateway and is never Ready while it is not. The worker stops polling while
its store connection is down, so a worker that lost its store drops out of
the gateway's poller window instead of looking healthy.
* wip: sqs model inventory
* wip: sqs public registration
* wip: sqs registration review fixes
* fix(sqs): refuse pre-API store and worker records on enable
A store or worker record this release creates now carries an explicit
contract. enable reads both records without creating a bucket and refuses
any record without it before it writes desired state or launches anything;
status names the record and is never Ready. A worker replacement keeps its
predecessor's contract, since it reuses the same data volume.
Missing metadata is only ever grounds to refuse: nothing is upgraded,
inspected, emptied, replaced or deleted. Upgrading a pre-API dev install is
not supported.
* fix(sqs): describe the enable refusal as an unsupported launch contract
A missing or unknown contract proves neither when a record was written nor
that it predates authenticated bootstrap: an older writer can drop a newer
field, and a future contract is not pre-API. The refusal is unchanged; its
message, status field and docs now name a missing or unsupported launch
contract and say any existing volume is kept.
* wip: port reviewed SQS serving path onto dev without the store VM
* wip: single SQS system VM
* fix(sqs): retry receipt bootstrap under supervision and drop store install path
Run the receipt-key boot step from the respawned store service so a
transient gateway or IMDS outage recovers without a reboot; final key
and volume refusals stay refused. Remove the separate store image
install and staging path from setup.
* fix(sqs): align worker ready config fields
* fix(sqs): stop an in-progress boot before unmounting the worker disk
The receipt-key boot step ran as an unrecorded child of the store shell,
so shutdown stopped only the shell and could unmount the disk under a
boot still talking to the gateway. Record the boot child like the other
guest processes and stop it before unmounting; a survivor keeps the disk
mounted.
* fix(sqs): record the boot child before its stopping check
The parent wrote boot.pid only after starting the child, so a shutdown
in that gap could unmount under an unrecorded boot. The child now
records itself and checks stopping before it runs sqs-worker boot.
Bumps the docker-images group with 1 update in the /build directory: golang.
Bumps the docker-images group with 2 updates in the /docs/terraform-workbooks/demo-app directory: golang and distroless/static.
Bumps the docker-images group with 1 update in the /scripts directory: debian.
Updates `golang` from `972b411` to `d9b6a1b`
Updates `golang` from `f2c872b` to `d200d2d`
Updates `distroless/static` from `18f4cc2` to `25ac8f0`
Updates `debian` from `62bd423` to `f6b08b0`
---
updated-dependencies:
- dependency-name: golang
dependency-version: 1.27.3-bookworm
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: docker-images
- dependency-name: golang
dependency-version: 1.27.3-alpine
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: docker-images
- dependency-name: distroless/static
dependency-version: nonroot
dependency-type: direct:production
dependency-group: docker-images
- dependency-name: debian
dependency-version: trixie-slim
dependency-type: direct:production
dependency-group: docker-images
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(test): never hand KillProcess a PID a real process can hold
* fix(iam): queue same-record writers so a looping self-write cannot starve DetachUserPolicy
* fix(ecs-agent): report RUNNING before starting container exit-waits
* fix(rds): wait for the lease-loss cancellation instead of sleeping in the takeover test
* fix(cli): name the account when the follow deadline lands mid-poll
* test(imagecatalog): never answer from the deadline fixture
* test(nats): order Gather StopOnFirst frames causally, not by sleep
* fix(ec2/key): find a key pair by name past an unreadable metadata record
* fix(accountlifecycle): re-list at the drain deadline so a just-deleted resource is not stuck
* fix(eks): give only the exhausting subtest the short SG delete budget
* fix(eks): stop healthz and nodegroup Ready-gate tests racing fixed wall-clock windows
* fix(test): let in-process NATS clusters bind their own route ports
* fix(objectstore): list the memory store in S3 key order with real paging
* test(viperblock): create a fresh volume on every run of the local-state test
* test(ochre): close router streams so drained tests release their admission slot
* test(cli): restore shared cobra flags after each ochre command test
* test(daemon): keep shared-JetStream, metric and RSS tests correct on repeat runs
* fix(ochre): let a warm-up probe stop so a test's poller cannot hit a later test's server
* test(cli): pin the package's cluster size so a multi-node config load cannot strand later fixtures
* feat(admission): add the quota override seal capability contract
* feat(awsgw): serve the quota override seal capability per node
* refactor(accountlifecycle): parameterise the seal capability probe and add the quota one
* test(accountlifecycle): pin the probe wire shape against both capability contracts
* feat(accountlifecycle): seal the quota override record during account teardown
* docs: record the quota override seal contract and teardown behaviour
* docs: fill in the quota override seal PR number in the package boundary migration
The ISO cells build the rootfs as root inside the workspace and only cleaned
it at the start of their own job, so the next job on the same runner from any
workflow failed its checkout on files it could not remove.
* fix(eks): delete the Node of a terminated nodegroup worker
K3s runs no AWS cloud provider, and its own reports every instance as
existing, so a worker terminated by DeleteNodegroup, scale-down or reclaim
stayed a NotReady Node forever. Its pods stayed bound to it, and once most
Nodes were NotReady the node lifecycle controller stopped evicting, leaving
metrics-server on a dead Node.
Add an internal-workers route the control plane pulls the cluster's live
nodegroup workers from, and a mulga-eks-node-gc agent on every server that
deletes a nodegroup Node only when it is not Ready and its instance is no
longer one of those workers, as the AWS cloud controller does.
* test(e2e): check the live worker's Node and wait out nodegroup deletes
WorkerNodeReady accepted any Ready Node, the tolerate-all DaemonSet check
dropped pods with no Ready condition, and nodegroup cleanup returned before
the nodegroup was gone. Scope the Ready check to the CPIsolation nodegroup,
count a pod with no Ready condition as not Ready, wait for each nodegroup
delete, and assert that deleted nodegroups leave no Node behind.
* test(integration): account for the internal ListInternalWorkers route in the cross-tenant table
NetworkManager restarted while /etc/resolv.conf is a plain file runs in
rc-manager=symlink mode and writes resolv.conf on its first DNS update, which
can land after the installer links the stub. Write a conf.d drop-in
(dns=systemd-resolved, rc-manager=unmanaged) before the restart, refuse a
later file that overrides rc-manager, wait for NetworkManager to settle, and
assert the link holds with bounded relinks. A rerun after an older handover
adds the drop-in, reloads NetworkManager and relinks the stub.
* test: pin the ENI owner stamp and failed-LB cleanup contracts
Failing tests: an ENI update after the caller's own write must survive a
lagging replica read, an unwritable owner stamp must fail the system
launch, and the reaper must terminate the failed LB's own VM only.
* fix: stamp the ENI owner in the attach write and release a failed LB's VM
A lagging-replica read before the owner stamp's compare-and-swap lost the
stamp, leaving a system VM that IMDS could never resolve.
- AttachENIWithOwner records the instance owner in the attach write.
- ENI read-modify-writes re-read and re-apply on a revision conflict,
bounded, and report the store error to the caller's log.
- A system launch whose ENI cannot be attached or stamped fails and cleans
up the VM.
- The reaper that marks an LB failed terminates that LB's own VM and
detaches its ENIs.
* fix: log ENI write retries and exhaustion
An Info line on a write that succeeds after a revision conflict and an
Error line when the bounded retries run out, so the sink can count both.
* test: cover an internal ALB in a private subnet and report the ENI attachment owner
DescribeNetworkInterfaces now returns Attachment.InstanceOwnerId when the record has one, as AWS does, so the owner stamp is observable. The e2e launches an internal ALB in a VPC with no internet gateway and checks the owner, the private address and in-VPC traffic. The ENI write error log now reports the real attempt count, so a non-conflict store error is not counted as an exhausted retry.
* revert: stop reporting the ENI attachment owner through Describe
AWS always returns Attachment.InstanceOwnerId, so reporting it only for system-owned ENIs adds a divergence. The e2e now relies on the LB reaching active with a healthy target, which needs the owner stamp to resolve.
* test: direct-boot LB and SQS worker NICs must be virtio-net-pci on microvm pcie=on
* fix: boot LB and SQS worker microvms on PCIe with virtio-net-pci NICs
virtio-mmio devices share level-triggered IRQ lines and a NIC's RX path
stalled on them. The direct-boot machine gets pcie=on, so NICs are
virtio-net-pci with MSI-X, and the virtio_mmio.device= kernel args go.
IsMMIO is now the transport check (false with pcie=on) and IsMicrovm the
machine check. Data volumes keep their serial and ID on virtio-blk-pci.
The LB guest console loglevel is raised to 7.
* refactor: trim microvmMachineType comment
* docs: describe direct-boot microVM NICs and data volumes on PCIe
* refactor: move the gateway into ingress/aws/pipeline and retire the gateway layering rows
The gateway's request pipeline joins the packages it already consumed.
Operation inventory moves to ingress/aws/dispatch. Files keep their names;
only the package line, the import paths and the now-redundant pipeline
qualifier change. The layering rows for the gateway root are removed: the
package is under ingress, so R2 binds it, and the daemon-direction test now
asserts R2 rejects runtime, operator and domain imports.
An empty registry_helper_test.go was dropped rather than moved.
* refactor: rename pipeline.GatewayConfig to pipeline.Config
* docs: update the design and package boundary docs for the gateway move
* docs: point the EKS add-on lifecycle evidence at the tests' current files
RebootInstances powered the guest down under -action shutdown=pause and
then ran system_reset + cont in the same QEMU process. On some hosts the
reset left the guest silent in firmware with QMP still answering, while a
fresh QEMU on the same volumes always booted.
A reboot now shuts the guest down over ACPI (forcing it off after 60s, as
before), ends that QEMU, and starts a fresh one on the same mounted volumes,
taps, IMDS datapath and identity, the way nova's soft reboot and Proxmox's
qm reboot do. The instance stays running in Describe, no hooks fire, the
console log is appended rather than truncated, and the existing QMP client
is reconnected in place so the heartbeat carries on.
A relaunched guest that writes nothing to its console within 60s has its
run state, vCPUs, vCPU thread states and registers (sampled twice) logged,
and if it has printed before, its QEMU is replaced once more. A relaunch
that cannot start goes to crash recovery. Stop, terminate and hot-plug
wait for a relaunch in flight so volumes are never sealed under a process
being started. Guest-initiated reboots stay an in-process reset.
The daemon now logs the host kernel, QEMU version and the OVMF package
and checksum at start, and every reboot relaunch carries them.
* fix(vm): release an auto-assigned public IP before a stop publishes stopped
* fix(e2e): skip a TryReady caller once the suite's SSH datapath is marked broken
* fix(e2e): skip a catalog entry whose third-party mirror is unreachable, still fail on a non-2xx
* fix(e2e): retry the diskperf fio install with backoff before calling it an infra fault
* fix(e2e): make the storagefault root-load check observe the load it started
* fix(ci): give the single suite a 40m budget and the nightly cells that run it 60m
* fix(ci): fail a nightly cell fast when a hypervisor it provisions on is out of disk
Three defects in the fence path made partition recovery fail:
- onVolumeLeaseLost read the lease holder before killing nbdkit, with no
deadline, spending the 5s margin between local validity and the server
TTL. The kill now comes first and the read is bounded at 2s.
- The fence notice was one NATS publish, lost when the node's NATS
restarts mid-fence. It is now a request repeated until the daemon
acknowledges, dropped once the volume is exported here again.
- A fenced guest got a 20s ACPI powerdown, a 20s pidfile wait and a
SIGTERM, keeping QEMU alive while a survivor relaunched it. It is now
SIGKILLed, and resumeAfterIOError never resumes a fenced guest.
The routed-NAT host route to a VPC CIDR was installed only by the vpcd
that received the IGW attach event, which goes through a queue group,
and by whichever node won a leased drift pass. Every other node waited
for the five-minute resync, and with two VPCs on one CIDR each node kept
whichever route it met first.
Each node now runs a host ingress loop outside the drift lease. It reads
intent and the gateway LRPs on every intent write, on a nudge from the
local IGW hooks and every 30s, installs or repoints the routes, and
prunes tagged routes no attached VPC holds. A shared CIDR goes to the
bootstrap VPC, then to the lowest VPC ID.
The kernel tracks connections only while some rule uses conntrack. With
inet spinifex_filter unloaded nothing tracked, so OVN RAFT links formed
while disarmed (or before the daemon's first arm) were first seen
mid-stream at arm time. With tcp_loose a packet from the server end became
a new flow from peer:6643/6644 to an ephemeral port and hit the drop rule,
and RAFT lost its leader.
Add table inet spinifex_conntrack (accept-only) and keep it loaded on
every node: setup.sh loads it at install, the boot unit loads it
unconditionally, firstboot loads it before OVN, and disable swaps the
policy for it in one transaction instead of turning conntrack off.
* test: pin the order of the unauthenticated mounts against SigV4 and the throttle
* refactor: mount the OIDC and registry routes through a public-routes hook
* refactor: take the admin registration name and request-id context as gateway config
* refactor: move node discovery out of the gateway into the awsgw role
* refactor: drop the gateway config fields production never reads (Config, RootCAs, InternalSuffix)
* refactor: drop the gateway Version and Commit config fields
* refactor: compose the production gateway config in one tested function
* refactor: drop the gateway IAMService, STSService, NATSConn and ExpectedNodes config fields
Production never reads them: composeGateway builds Policy and Readiness from
its own inputs. Tests that used the config as a carrier now keep those
collaborators in gwfix.Deps, recorded per config with gwfix.WithDeps and read
back with gwfix.DepsOf. Gateway non-test code now imports nothing under
domains/, runtime/ or operator/.
Test edits are setup only: 153 GatewayConfig literals in 45 files had the four
keys moved into gwfix.WithDeps(..., gwfix.Deps{...}), and 113 field reads
became gwfix.DepsOf(x).{IAM,STS,NATS,Nodes}. No assertion changed.
Script 1, literal rewrite (balanced-brace scanner, keys moved out of the
literal):
import re,sys,subprocess,os
KEYS={'IAMService':'IAM','STSService':'STS','NATSConn':'NATS','ExpectedNodes':'Nodes'}
def scan_close(s,i):
# s[i]=='{' ; return index of matching '}' ; skips strings, runes, comments
d=0;n=len(s);k=i
while k<n:
c=s[k]
if c=='"':
k+=1
while s[k]!='"':
k+= 2 if s[k]=='\\' else 1
elif c=='`':
k=s.index('`',k+1)
elif c=="'":
k+=1
while s[k]!="'":
k+= 2 if s[k]=='\\' else 1
elif s.startswith('//',k):
k=s.index('\n',k)
continue
elif s.startswith('/*',k):
k=s.index('*/',k)+1
elif c in '{([': d+=1
elif c in '})]':
d-=1
if d==0: return k
k+=1
raise Exception('unbalanced')
def split_top(body):
parts=[];d=0;start=0;k=0;n=len(body)
while k<n:
c=body[k]
if c=='"':
k+=1
while body[k]!='"': k+= 2 if body[k]=='\\' else 1
elif c=='`': k=body.index('`',k+1)
elif c=="'":
k+=1
while body[k]!="'": k+= 2 if body[k]=='\\' else 1
elif body.startswith('//',k):
k=body.index('\n',k); continue
elif body.startswith('/*',k): k=body.index('*/',k)+1
elif c in '{([': d+=1
elif c in '})]': d-=1
elif c==',' and d==0:
parts.append(body[start:k]); start=k+1
k+=1
parts.append(body[start:])
return parts
changed={}
files=[l for l in subprocess.check_output(['git','ls-files','*.go']).decode().split() if os.path.exists(l)]
SKIP={'spinifex/gateway/gateway.go','spinifex/runtime/roles/awsgw/compose.go','spinifex/runtime/roles/awsgw/awsgw.go'}
lit=re.compile(r'(&?)((?:gateway\.)?GatewayConfig)\{')
total=0
for f in files:
if f in SKIP: continue
s=open(f).read()
if 'GatewayConfig{' not in s: continue
out=[];pos=0;cnt=0
for m in lit.finditer(s):
if m.start()<pos: continue
o=m.end()-1
c=scan_close(s,o)
body=s[o+1:c]
parts=split_top(body)
keep=[];moved=[]
for p in parts:
km=re.match(r'\s*(IAMService|STSService|NATSConn|ExpectedNodes)\s*:\s*(.*)$',p,re.S)
if km: moved.append((KEYS[km.group(1)],km.group(2).strip()))
else: keep.append(p)
if not moved: continue
# rebuild
kb=','.join(keep)
if kb.strip()=='' : kb=''
elif not kb.rstrip().endswith('\n') and '\n' in body: pass
d=', '.join(f'{k}: {v}' for k,v in moved)
new=f'gwfix.WithDeps({m.group(1)}{m.group(2)}{{{kb}}}, gwfix.Deps{{{d}}})'
if not m.group(1): new=f'gwfix.WithDeps(&{m.group(2)}{{{kb}}}, gwfix.Deps{{{d}}})' # value literals flagged below
out.append((m.start(),c+1,new)); pos=c+1; cnt+=1
if not m.group(1): print('VALUE LITERAL',f,s.count('\n',0,m.start())+1)
if not out: continue
r=[];last=0
for a,b,n in out:
r.append(s[last:a]);r.append(n);last=b
r.append(s[last:])
s2=''.join(r)
if 'testkit/gwfix"' not in s2:
s2=re.sub(r'import \(\n','import (\n\t"github.com/mulgadc/spinifex/internal/testkit/gwfix"\n',s2,1)
open(f,'w').write(s2); total+=cnt; changed[f]=cnt
print('literals rewritten',total,'files',len(changed))
Script 2, read rewrite (driven by the compiler's "has no field or method"
errors for the type, so only GatewayConfig receivers are touched):
import re,subprocess,sys,collections
M={'IAMService':'IAM','STSService':'STS','NATSConn':'NATS','ExpectedNodes':'Nodes'}
def run():
env=dict(__import__('os').environ,GOWORK='off')
out=''
for tags,pk in [('',['./spinifex/gateway','./spinifex/runtime/roles/awsgw/...']),('integration',['./tests/integration/...'])]:
cmd=['go','test','-count=1','-run','NONE_X','-gcflags=-e']+(['-tags',tags] if tags else [])+pk
r=subprocess.run(cmd,capture_output=True,text=True,env=env)
out+=r.stdout+r.stderr
return out
fixed=0
for it in range(8):
out=run()
errs=re.findall(r'^(\S+\.go):(\d+):(\d+): (.*?) undefined \(type (\*?(?:gateway\.)?GatewayConfig) has no field or method (\w+)\)',out,re.M)
if not errs:
print(out[-3000:]); break
byfile=collections.defaultdict(list)
for f,l,c,expr,t,fld in errs: byfile[f].append((int(l),int(c),expr,fld))
for f,es in byfile.items():
lines=open(f).read().split('\n')
for l,c,expr,fld in sorted(set(es),reverse=True):
line=lines[l-1]
# expr like 'gw.IAMService' or 'x.y.IAMService'
new=f'gwfix.DepsOf({expr[:-len(fld)-1]}).{M[fld]}'
if expr not in line: print('MISS',f,l,expr); continue
lines[l-1]=line.replace(expr,new)
fixed+=1
s='\n'.join(lines)
if 'testkit/gwfix"' not in s:
s=re.sub(r'import \(\n','import (\n\t"github.com/mulgadc/spinifex/internal/testkit/gwfix"\n',s,count=1)
open(f,'w').write(s)
print('read sites fixed',fixed)
* docs: record the gateway mount hooks and config-field removal in the package boundary migration
* docs: fill in the 11d-1 PR number in the package boundary migration
* test: pin service-keyed gateway behaviour before it moves to registration metadata
* refactor: declare the XML throttle code on the registration
The pipeline picked Throttling for iam, sts, eks and route53 by name. EKS is a JSON registration and never reached that arm, so it is not carried over.
* refactor: declare the XML error shape on the registration
EC2 and the operator surface declare the EC2 shape. S3 is not served here, so the composition declares its scope traits and a rejected S3 request keeps S3's body.
* refactor: declare the global signing region on the registration
* refactor: supply the unsigned web-identity exchange as a pre-auth handler
STS composition now hands the pipeline a dispatch.PreAuth. The gateway no longer recognises STS or the action by name, and the layering allowlist row for gateway -> sts/awsapi is paid off.
* refactor: declare payload-body scopes as traits
The action recovery for a rejected request skipped S3 by name. The composition now declares the S3 scope as carrying object data, and the gateway reads the trait.
* test: wire the integration harness through PreAuth
* test: pin control-plane no-agent config, etcd member removal and CP isolation e2e
* test: tighten CP isolation e2e and assert a logged prune failure
* fix: run EKS control planes without a k3s agent and prune etcd members directly
Control-plane VMs no longer register as Nodes, so pods cannot schedule onto a
network that cannot reach the workers or the advertised endpoint. With no Node
to delete, a replaced peer's etcd member is removed by a new eks-etcd-member
helper that matches exactly one member by peer-URL host.
* fix: refuse to remove the local etcd member by ID, not only by IP
Self-protection no longer depends on first boot deriving its own address: the
helper asks the connected endpoint for its member ID and refuses a match on it.
* test(e2e): align EKS suite with control planes that are not Nodes
Assert zero Nodes and delivered addon objects before any nodegroup. Check Ready worker, metrics APIService, addon deletion and pod logs once a worker exists, scoping logs to Ready nodes and waiting 10m for metrics-server to reschedule.
* docs: EKS control-plane hosts are not Kubernetes Nodes
Document that kubectl get nodes lists workers only, add upgrade notes for existing clusters, and list the eks-etcd-member helper.
* docs: explain why addon manifests keep the CriticalAddonsOnly toleration
Comment-only. The toleration stays for clusters created before control-plane hosts stopped registering as Nodes.
* test: cover the etcd member client and CLI against a TLS fake etcd
The real etcd client and the command had no tests, leaving the PR below the
diff coverage threshold. A fake etcd serving the Cluster and Maintenance APIs
over mutual TLS drives the client with the K3s certificate layout, and main
becomes a testable run function.
* fix: bound the console's instance-type payload and stop polling it
DescribeInstanceTypes with capacity=true returned one entry per free slot per
node: 3,528,765 bytes of XML for 50 distinct types on prod, refetched every
five seconds. The response was sized by free capacity, so the emptier the
cluster the larger it got, and at 44,346 slots it exceeded the NATS maximum
payload and the console reported no compute at all.
The listing now names each admissible type once. The free-slot figure crosses
as a number through a new GetInstanceTypeCapacity action, open to any account
because the launch form a tenant uses is what needs it. The catalogue query no
longer polls, and the EKS and ECS dialogs read it only when open.
The gateway negotiates gzip inside the SigV4 group, which excludes the OCI
Distribution registry structurally. A response is compressed only if the client
offered gzip, the handler declared neither a length nor an encoding, the
content type is neither streaming nor already compressed, the status carries a
body, and 1 KiB was written before any flush.
An unknown capacity figure no longer caps the count at 1. The launch decides,
and RunInstances answers InsufficientInstanceCapacity if it cannot.
* test: read the free slot count from its own action in the console e2e
Two e2e tests encoded capacity as repeated instance-type entries, so one
expected a caption the deduplicated listing can no longer produce. The fixture
answers GetInstanceTypeCapacity instead, which lets a test give the catalogue
and the figure different answers.
A third test covers the case the change introduces: a session whose policy does
not grant the action reads no figure, so the form offers the type with no
capacity caption and leaves the launch enabled.
The committed bundle is rebuilt, which rehashes every chunk reaching the two
query modules.
* fix: bound the capacity describe reply by instance-type count, not free slots
A node answering `DescribeInstanceTypes` with `capacity=true` sent one full copy of an instance type's description per free slot, so the reply grew with how much capacity the host had free.
At 1344 bytes per slot the four-node production cluster, holding no guests, produced 874 slots and a reply over the 1 MB NATS maximum payload.
Every node's reply was dropped, the gateway collected zero frames, and it returned an empty list as a success.
The console reads an empty capacity list as no capacity and refuses every launch, which is how a cluster with hundreds of free vCPU reported "No compute available".
The emptier the cluster, the more likely the failure, which is the opposite of what the shape should be.
A node now returns each admissible type once with the number of instances of it the node can take, so its reply is bounded by the size of the instance-type catalogue and not by free capacity.
On the production catalogue that is 84 KB for 62 types whatever the slot count, against 59 MB for the same reply on a large idle host under the old shape.
The gateway sums the per-type counts across nodes and expands them on the last hop, because the client SDK drops response fields it does not model and the console displays the figure to the customer.
A capacity fan-out that collected no frames now returns a retryable ServiceUnavailable.
Reporting an empty list as a success is what turned a transport failure into a false statement about the cluster.
Two tests pin the shape: the node reply for an idle 4096-vCPU host stays under the NATS payload limit, and a capacity fan-out with no responders does not return an empty success.
* test: cover the console against a real cluster, and pin the capacity banner
The console has never been exercised against a real control plane.
`playwright.config.ts` pins `localhost:3101` and `:3102` and intercepts every API call, so the suite cannot be pointed at a cluster and cannot see a defect whose whole content is what the control plane actually sends.
That is why an idle production cluster came to tell customers it had no compute: the capacity listing was dropped in transport, the empty result read as no capacity, and nothing in CI could reach the page that says so.
`playwright.live.config.ts` runs the same console against a cluster named by `SPX_CONSOLE_URL`, signing in with `SPX_ACCESS_KEY_ID` and `SPX_SECRET_ACCESS_KEY`.
It starts no server, mocks nothing, runs one worker because it launches real guests, and is never part of `pnpm e2e`.
The suite signs in, asserts a cluster with free capacity offers instance types rather than reporting none, reads the per-type capacity figure, launches an instance and waits for `running`, reads its live metric graphs, terminates it in a `finally` block, and loads the nine listing pages a customer reaches from the sidebar.
Three tests in the mocked suite pin the banner's own logic, which is cheap and catches the UI side of the same failure: the form offers what the capacity listing reports, it shows the free slot count for the selected type, and an account with genuinely no capacity is told so and cannot submit.
The third passes on the current code and is the one that documents why the banner exists at all.
* test: correct three live-console selectors found by running it against prod
The first run of the live suite was 11 passed and 2 failed, both in the suite rather than in the console.
The dashboard check looked for node status on `/`, where it is not shown.
It now loads `/nodes` and asserts the `Resource usage by node` and `Instance type capacity` tables, which is where the daemon reports a node's own capacity.
The OS tile's radio is `sr-only` behind the distribution logo, so `check()` failed its actionability test on the `<img>` covering it.
The click goes to the enclosing `<label>` instead and the browser forwards it.
The terminate dialog holds its confirm disabled until the phrase is typed, so the teardown waited out its timeout on a disabled button and left the instance running.
The helper types `terminate` first.
With those three corrected the launch, running, live-metrics and terminate path passes against the production console in 34 s.
* fix: let a snapshot go once every volume cloned from it is already deleting
DeleteVolume now records State=deleting and leaves the teardown to the
reclaim reaper, so a clone stays listed after the tenant has deleted it.
DeleteSnapshot counted that clone as still reading the snapshot and refused
with InvalidSnapshot.InUse, which broke DeleteDBSnapshot after deleting the
DB instance restored from it and left the retained source volume behind.
A deleting volume's teardown removes only its own objects and nothing moves
it out of deleting, so it no longer holds the snapshot's chunks.
* test(e2e/rds): delete a snapshot straight after the instance restored from it
The subtest deleted the source instance between the two, which gave the
volume reclaim reaper time to finish the restored volume on a fast node and
hid the refusal the nightly hit. Deleting the snapshot immediately makes the
regression deterministic.
The release wait now covers a full reclaim sweep: since deletion became
asynchronous a released volume is gone only after the reaper's next pass.
* refactor: one authorization gate in ingress/aws/pipeline
The key merge and the caller guards move out of the gateway so tests that
drive a registration directly use the same gate as the pipeline.
* test: run ACM handler and policy tests in domains/acm/awsapi
Adds internal/testkit/gwtest, which invokes a registration through the
pipeline's own authorization gate with real policy evaluation.
* test: share policy assertion helpers through gwtest
* test: run ELBv2 handler and policy tests in domains/elbv2/awsapi
* refactor: one actor projection in ingress/aws/pipeline
* test: run RDS handler and policy tests in domains/rds/awsapi
* test: run ECS handler and policy tests in domains/ecs/awsapi
* test: assert the exact policy gate verdict in gwtest
* test: run EKS authz and wire tests in domains/eks/awsapi
* test: run IAM authorization tests in domains/iam/awsapi
The policy-driven DeleteRole and AddRoleToInstanceProfile tests run in the IAM package through gwtest, with PassRoleTo shared. The HTTP-handler, examples and signed-pipeline tests stay in gateway.
* test: run ECR authorization tests in domains/ecr/awsapi
* test: run ECR repository, image and lifecycle-preview tests in domains/ecr/awsapi
* test: run ECR dispatch error tests in domains/ecr/awsapi
* test: run Bedrock authorization tests in domains/ochre/awsapi
* refactor: move EC2 authz tests into domains/ec2/awsapi
* test: restore the ECS scope-table completeness gate in domains/ecs/awsapi
* Revert "test: restore the ECS scope-table completeness gate in domains/ecs/awsapi"
This reverts commit 0211a0a1bf.
Shows deploy-vultr.sh building a private Vultr VPC with one node or three,
the nodes clustering over the private network for the control plane, EBS
and S3, Spinifex calling the Vultr API for reserved IPs, and the AWS
services running across the fleet.
Drops the horizontal rules the docs site rendered as stray text, links
the AWS CLI install page, captions each screenshot, shows the expected
spx get nodes output for three nodes, explains what a three-node
deployment protects against, and simplifies the limits to the sizing
and quorum a reader needs.
* feat(sqs): queue and Lambda-facing message API foundation
Adds the SQS JSON 1.0 adapter (unregistered in production) with queue
create, lookup, attributes, list, tags and the message operations a
Lambda poller uses: SendMessage(Batch), ReceiveMessage with long
polling and attribute selection, DeleteMessage(Batch) and
ChangeMessageVisibility(Batch), backed by the queue record owner and
the JetStream message store. String and Binary message attributes are
validated, stored and checksummed; Number attributes are refused until
AWS behaviour is confirmed.
* test(sqs): poller lifecycle through the message owners
Covers receive, release, redelivery, delete, visibility extension, queue
binding of receipt handles and long polling against the real queue and
JetStream message stores, from the message package itself.
* test(sqs): move SDK coverage from the gateway to the SQS domain
Drives the registration through dispatch.Invocation over the real queue
and message owners, so the gateway package carries no SQS changes.
* test: pin gateway pipeline order, request ids and region
* refactor: carry request facts in one ingress request record
* fix: seal the request principal once
* refactor: build the caller ARN once in credverify
* refactor: gateway takes policy and admission as capabilities
* refactor: gateway takes cluster readiness as a capability
* chore: gofmt GatewayConfig field alignment
* docs: record slice 11a request record and capabilities
* docs: add PR number to slice 11a migration row
* refactor: move pipeline capability interfaces to ingress/aws/pipeline
* feat: Vultr reserved-IP provider and floating external-address pool
Adds a source="vultr" external pool backed by Vultr reserved IPs: an API
client pinned by captured responses, an in-memory fake with the measured
semantics, and a floating allocator that creates, claims, reconciles and
releases labelled addresses. OCI and Vultr claims now share one
AddressClaimer pass.
Also fixes the host firewall dropping guest IMDS when imds_host_meta_ip is
set, by matching the pre-DNAT destination, and arms the Spinifex firewall in
place of an active ufw.
* fix: detach a Vultr reserved IP before deleting it
Vultr was measured refusing delete of an attached address with 400 "Unable to detach subnet", so release, rollback and reconcile now detach first. A first claim after allocation logs at info rather than warn.
* fix: release the auto-assigned public IP an Elastic IP displaces
AssociateAddress restamped the ENI with the EIP and never handed back the
address launch had assigned. Nothing referred to it afterwards, so it stayed
allocated for good, and on a cloud pool every associate left a billed address.
AWS releases it on associate; withdraw its NAT rule and release it, scoped to
the ENI, leaving any address another associated EIP holds alone.
* fix: retry Vultr's refused detach on reserved-IP detach and delete
Vultr has answered a detach with 400 "Unable to detach subnet" and accepted
the same detach moments later. Surfacing it left the address for reconcile
to collect, up to twenty billed minutes later; retry it like a 5xx instead.
* fix: stop reporting the released launch address after an EIP is disassociated
Associating an EIP now releases the auto-assigned public IP, but the stored
instance kept its launch stamp, and DescribeInstances fell back to it once the
EIP was disassociated. The public-IP change handler now moves the stamp with
the runtime address.
* feat: add a Vultr deploy script for a single-node Spinifex on a VX VM
deploy.sh up/destroy/status drives Terraform for the VPC and VM, installs and
forms Spinifex over SSH, copies the API key to the node rather than through
user-data, and sweeps boot volumes and reserved IPs Terraform does not own.
Cloud-init keeps needrestart off the Spinifex, OVS and OVN units and disables
unattended upgrades on these disposable hosts.
* feat: form a three-node Vultr deployment with install-node.sh over the VPC
Each node comes up as an armed single node, the state install-node.sh forms
from, with the cluster and Geneve planes on the VPC NIC. The pool and every
node's provider_host_id go on each node after formation.
* fix: read the pinned host key with ssh-keygen -F when forming a Vultr cluster
ssh_config hashes known_hosts entries by default on Debian, so a grep for the
address found nothing and set -e ended the run without a message.
* docs: add the Vultr guide, one command to deploy one or three nodes and one to remove them
* docs: drop the failover write note from the Vultr guide
* fix: keep needrestart from restarting the Spinifex, OVS and OVN units
Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's
replaced libraries to the daemon and restarted it after every dpkg run. The
restart never cleared them, so an unattended upgrade ran the unit into its
start limit and left the control plane down.
setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting
spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists
them, so restarting after an upgrade stays the operator's choice.
* chore: drop the Vultr cloud-init needrestart override now setup.sh writes it
* docs: replace the Vultr guide's apt upgrade warning now the installer keeps needrestart off the stack
* docs: install the Vultr guide's build from the dev channel until Vultr support is released
* fix: answer ARP for Vultr reserved IPs on bare metal
Vultr routes a reserved IP to a VM by MAC but ARPs for it on a bare-metal
server's public segment, and a Vultr pool gave vpcd neither a gateway nor a
bind bridge to proxy-ARP on, so nothing answered and no traffic arrived. A
Vultr pool now proxy-ARPs on the interface the node routes the internet
through. setup.sh installs arping, which the gratuitous ARP already called.
* feat: deploy Spinifex on Vultr bare metal, renamed deploy-vultr.sh
A vbm- plan creates bare-metal servers, one or three. Without --name the
deployment writes the AWS profile spinifex, up prints every node's address
and the API key allowlist, credentials and ssh subcommands are added, no AMI
is imported unless --image is given, and OpenTofu or Terraform is used. The
host key is pinned only once cloud-init is done, since bare metal first
answers SSH from Vultr's installer with a different key.
* docs: rework the Vultr guide into the OCI guide's flow, with bare metal
* fix: restamp LaunchTime when a stopped or recovered instance relaunches
The pending watchdog times a launch from LaunchTime, which still held the
original launch. An instance recovered from a failed node, or started more
than five minutes after it was first launched, was terminated as stuck in
pending seconds after QEMU started. AWS restamps LaunchTime on every start.
* feat: spread the nginx-alb workbook across nodes, and power Vultr nodes off and on from deploy-vultr.sh
The workbook takes server_count and placement_strategy, so three web servers
can run one per node. Its README now matches the template (public subnets,
no NAT gateway) and the diagram shows the spread. deploy-vultr.sh gains
stop and start, which halt and start a node through the Vultr API.
* fix: let a terminate finish tearing down before the reaper calls it wedged
OrphanQEMUReaper finalized any shutting-down instance whose QEMU had exited,
on the premise that a live terminate still has a live QEMU. A terminate is
still deleting volumes, releasing its IP and leaving its placement group for
tens of seconds after QEMU exits, so the instance was reported terminated
first. Terraform then deleted the placement group and got
InvalidPlacementGroup.InUse. The reconcile now waits two minutes from
shutting-down, well inside the ten-minute stuck-terminate backstop.
* fix: release an ovs-vswitchd left in flow-restore-wait after a node boots
ovn-controller clears flow-restore-wait once its flows are installed and
sets it again straight after. When both writes reach ovs-vswitchd as one
update, it only sees true, starts no upcall threads and forwards nothing.
Seen on Vultr bare metal after a hard power-off: instances recovered onto
the rebooted node ran but were unreachable. vpcd now clears the flag on any
node held that way for a minute, longer than ovs-ctl's own restore takes.
* docs: add a resiliency test to the Vultr guide and the nginx-alb workbook
The Vultr guide gains step 8, which spreads three web servers across a
three-node deployment, powers a node off and back on, and shows what was
measured on VMs and bare metal. The workbook gains the same test as step 6.
The guide's table of contents is repaired, and its limits now cover losing
two nodes and recovery inside a spread placement group.
* fix: never terminate an instance that has run because a relaunch failed
The pending watchdog terminated any instance pending past five minutes, and
terminating deletes DeleteOnTermination volumes. A start of a stopped
instance, a recovery from a failed node and a daemon relaunch all pass
through pending, so a slow one destroyed an existing instance and its root
disk. On Vultr a recovered instance was terminated seconds after QEMU
started and its root volume deleted.
An instance is now marked FirstLaunch from RunInstances until it first
reaches running, cleared before the running state is written. The watchdog
acts only on first launches, as AWS terminates only a launch that never
ran. MarkFailed refuses to terminate any other instance and marks it
recovery-failed instead, which keeps its volumes, ENI and addresses and
shows it as stopped. A claimed relaunch is never a first launch.
Invariant tests pin ADR-0003:S7. The instancerecovery e2e suite gains a
test that ages its guest past the watchdog before recovering it, then
stops and starts it, and requires the instance, its root volume and the
data on it to survive both.
* feat: run the Terraform workbooks against a Vultr deployment on demand, and show the Vultr portal in the guide
e2e-cloudvendor-vultr.yml is dispatch-only. It builds the ref, deploys it
with deploy-vultr.sh on one or three VMs or bare-metal servers, runs the
published workbooks from node1 with the same driver as the hypervisor
nightly and OCI, and destroys the deployment, failing if anything is left
billing. It needs one secret, VULTR_API_KEY, and generates its SSH key per
run. The guide gains screenshots of the API key's access control list and
of the nodes in the Vultr portal, and its portal navigation now matches.
* fix: install setup.sh on Vultr nodes when the tarball does not carry it
build-install-artifacts.sh writes setup.sh beside the tarball, so a node
installed from its tarball had no /usr/local/share/spinifex/setup.sh and
arming the firewall failed.
* fix: pass the remote command after the destination in the Vultr e2e driver
* fix: run each workbook in the cluster's own region
The workbook driver passed no region, so every workbook used its default,
ap-southeast-2, and on a cluster in any other region every request failed
SignatureDoesNotMatch. It now passes the region of the node's spinifex
profile.
* feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015)
A node counted physical cores, collapsing SMT threads, and held back a
fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is
now one online CPU, as AWS counts it: a thread with SMT on, a core with
SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4
up to 63 and 8 from 64, and the rest is allocated one to one. The
existing overrides stay operator-only. Invariant tests pin every tier
boundary.
* docs: state the Vultr guide's sizing under the online-CPU count
* docs: drop an unmeasured EKS sizing claim from the Vultr README
* fix: let a guest reach an Elastic IP hosted on its own node
A guest's packet to an EIP on its own node left its VPC from the transit
range, was masqueraded to the transit host address, and went back into
OVN, which exempts that range from EIP DNAT, so it was never answered.
On Vultr an EKS worker could not reach its control plane's endpoint, so
it never joined. AWS lets an instance reach a public address in its own
VPC.
Hairpinned transit traffic now jumps to SPX-EIP-HAIRPIN, above the egress
MASQUERADE, where each EIP's rule sends it from the uplink address, as the
host's own traffic to the EIP already is. The rule follows the EIP's
ingress plumbing in and out, and anything else returns to the MASQUERADE.
* fix: resolve Spinifex's LB and EKS names on Vultr nodes
Vultr's Ubuntu image writes a static resolv.conf naming Vultr's resolvers,
so lookups bypassed the split DNS Spinifex configures in systemd-resolved
and an EKS endpoint name did not resolve on the node. deploy-vultr.sh
points resolv.conf at the resolved stub, as stock Ubuntu does.
* test: prove a guest reaches an EIP on its own node in the natuplink e2e
The nat-single cells run it with a public pool, so the hairpin path that
left EKS workers unable to reach their control plane on Vultr is now
covered nightly.
* docs: record the natuplink hairpin check in the e2e coverage index
* fix: keep each Vultr node's logs when an e2e run fails
The driver destroyed a node whose install failed together with the log
that said why. It now copies /root/*.log from every node into the run's
artifacts before the teardown, and the workflow uploads them.
---------
* test: pin EKS worker IMDS metadata options and launch template resolution
* fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options
* fix: default unset launch template metadata fields for EKS workers
* test: e2e coverage for EKS worker IMDS hop limit and launch template metadata options
* test: drop the unneeded GetCallerIdentity policy from the worker IMDS e2e
* docs: describe the EKS worker IMDS defaults and launch template overrides
* test: drop the stale GetCallerIdentity policy from the IRSA pod e2e
A node counted physical cores, collapsing SMT threads, and held back a
fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is
now one online CPU, as AWS counts it: a thread with SMT on, a core with
SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4
up to 63 and 8 from 64, and the rest is allocated one to one. The
existing overrides stay operator-only. Invariant tests pin every tier
boundary.
* feat: lay the Spinifex UI design-system foundation
Brand tokens in OKLCH with semantic status and a colour-blind-checked
chart palette, Archivo headings and a 13px type scale, and a pre-paint
theme script so dark mode never flashes white under the console CSP.
Adds the missing Base UI primitives, a /design catalogue route, a route
ratchet against hand-written tables, a ban on chart-library imports in
routes, and lets the content column shrink so wide tables scroll in
place. The dev server now forwards /api and /proxy to the Go console.
Sonner is removed: its inline style element is blocked by the CSP.
* feat(ui): side panels, TanStack table, uPlot charts and e2e gate
- DataTable renders @tanstack/react-table, with column hiding as container
queries so columns drop when a side panel narrows the list.
- UniFi-style DetailPanel and ResourceView: Instances, VPCs and Internet
gateways open in a panel beside the list from 1024px and as a full page
below; selected, tab and create live in the URL. Create flows for
instances, VPCs and gateways open in the same panel.
- Instance monitoring charts on uPlot (CPU, network, packets), on simulated
data and badged as such.
- Dates in the AWS console form, e.g. 2026/06/04 02:00 GMT-5.
- Tabs style Base UI's data-active, so the selected tab now shows.
- Playwright suite gates CI as the UI End-to-End job in ui.yml.
* feat(ui): one details component for panel and page, UniFi-style overlay panel
The side panel now lies over the list at full height with a shaded edge, and a
click on the list's background closes it. Instances, VPCs and internet
gateways each render one details component as either the panel or the
resource's own page, so the page gains the monitoring graphs and the panel
gains IAM role, GPU, console output, change type and create image. SSH moves
into Instance information as rows. Internet gateways attach and detach from
the panel.
* feat(ui): roll the panel pattern out to every service and gate screenshots in CI
Every EC2, IAM, Route 53, Bedrock, EKS, ECS and RDS resource is now a
list plus a side panel, with one details component serving as both the
panel and the one-column page. DetailCard, DetailRow, ListCard,
ResourceList and DeleteResourceButton are deleted.
- Panel: slide-out close, row selection closes it, expand keeps the tab
and fetched console output, PanelSpinner while loading.
- Instances: info tips, theme-aware coloured console output, power
button group plus Actions menu, Stop and Reboot confirms, security
group, Elastic IP and network interface sections, and a live
one-second monitoring window on simulated data.
- Launch instance: OS tiles picking the newest version, architecture
choice, location tile, expanded root volume, honest default subnet.
- On/off settings are switches app-wide; disabled controls show
cursor: not-allowed.
- The e2e job runs in the pinned Playwright image with screenshots
gating; make ui-baselines and make ui-e2e-container use the same image.
* feat: bulk select and delete on every resource list, OS marks and instance type sizes
Every list with a delete action gets row checkboxes and a selection bar.
Delete confirms in a dialog naming each resource, makes one call per
resource, keeps each refusal against its row and retries only failures.
The Launch form shows distribution marks for Ubuntu, Debian, Rocky,
Alpine and CentOS, served as files under the CSP, and states the
selected instance type's vCPUs and memory.
* feat: sign-in overlay, larger type scale, typed confirms on irreversible deletes and dialog shortcuts
The sign-in card sits over an inert shell. Any sidebar group folds, and entering one unfolds it. Instance DNS names copy. Escape cancels and Cmd/Ctrl+Enter confirms every dialog. Volume, snapshot, RDS, EKS, ECS and hosted zone deletes need "delete" typed, and terminate needs "terminate".
* feat: fade sign-in and sign-out in place, and show the confirm shortcut
Signing in no longer reloads the page: the card and a lighter veil fade out and the console routes to the dashboard. Signing out cross-fades into the sign-in card with a Signed out notice. Confirm buttons show Cmd or Ctrl Enter for the platform.
* feat(ui): generic sidebar group names and a paged S3 object browser
The sidebar names its groups Compute, Databases, Authentication and Object
Storage; the command palette still finds each by its AWS service name.
Buckets get row selection and delete. Inside a bucket, folders and objects
share one table paged by ListObjectsV2 continuation tokens, with a find box
that queries by prefix, multi-select delete through DeleteObjects (folders
with everything under them), and an object panel showing HeadObject
properties, user metadata and a download.
* feat(ui): show upload progress and stop the S3 proxy cutting off slow transfers
The Upload button shows a running batch's percentage, and a large or slow
batch opens a dialog with bytes sent, bit-rate, time left and per-file
cancel and retry. Uploads run in a store outside React, so they continue
while the user browses.
The console's S3 proxy now allows an hour per request. The server's 15 s
ReadTimeout cancelled every upload part that took longer, so large uploads
failed on links below roughly 18 Mbit/s.
* feat(ui): stream live EC2 metrics to the Monitoring tab
The Monitoring tab drew simulated data. It now streams CPU, network
bytes and packets from the owning node's qmp-collector every 500 ms
while someone is watching, with nothing stored.
- qmp-collector: a watched sampler on schedstat and tap counters, on a
30 s lease, keeping five minutes in memory for reconnects. It answers
only watches for the account in the VM's discovery file.
- awsgw: spinifex:GetInstanceMetrics, checked against the instance ARN,
answered as text/event-stream in 60 s segments. Another account's
instance is InvalidInstanceID.NotFound.
- console: a signed stream reader that reconnects with Since and backs
off on failure or an empty segment, stops on InvalidAction, and a
chart axis that scrolls between samples. The simulator and its badge
are removed.
* feat: catch refused creates in every form, answer the third review, and guard tables
Every create form now catches a refused mutation, so the failure shows beside
the submit button and no longer escapes as an unhandled rejection. Three EKS
tabs that swallowed a refused create now show it.
Third-review answers: the attach dialog offers running or stopped instances in
the volume's zone; the default security group's delete is disabled with its
reason; the load balancer panel splits into Listeners and Attributes tabs; a
main route table lists its implicit subnets; IAM single delete dialogs name the
same dependants as the bulk ones; a snapshot can be deleted in any state.
Every list in a route renders through DataTable, enforced by a unit test with a
named allowance, and the /design baselines are redrawn in the pinned container.
* feat: cap live metrics streams, prune hidden selections, and move panel focus
Each gateway holds at most 10 live metrics streams per account and 200 in
total, refusing past either with RequestLimitExceeded, and counts opened and
refused streams. A row that leaves a list leaves its selection, the instances
page uses the shared selection hook, the detail panel takes focus on open and
returns it on close, and Escape yields only to the popup it was pressed in or
a modal. The UI workflow also runs on live metrics backend changes.
* test: authorise an ECR token from the live role once its session is gone
* feat: carry the role ID in an ECR token minted for an assumed role
* feat: rehydrate an assumed-role caller from the live role
* fix: authorise an ECR token from its live role after the session expires
* docs: note that an ECR token outlives its role session
* test: characterise EC2 request labels and the cluster-unavailable gate order
Pins the span, audit, throttle and policy-gate labels an EC2 request carries through the signed pipeline, and that the NATS gate answers before policy, action lookup and gateway-local actions.
* refactor(gateway): register EC2 through the ADR-0002 seam
EC2 is composed in awsgw from domains/ec2/awsapi and served by the
registration seam. The legacy service switch and its tables are removed,
along with the two paid-off gateway allowlist rows.
* docs: record the EC2 seam registration in the package boundary migration
* docs: record the EC2 seam registration PR number
DeleteVolume returned only once every object behind the volume was gone, which
made the call's latency a function of the volume's size. Object removal costs
one round trip per object, so no timeout is both short enough for a control
plane request and long enough for a large volume: on a 100 GiB volume the
provider request overran its 30 s budget at 52 s, the tenant was told the
delete failed, and the teardown that had already started was left with no
record accountable for it.
AWS does not work this way. DeleteVolume's output in the EC2 model is
smithy.api#Unit, VolumeState includes deleting, and the documentation says a
volume can remain in that state for several minutes.
DeleteVolume now validates, decides every AWS-returnable error, records
State=deleting and returns. VolumeReclaimReaper converges the teardown and
removes the record only once it is complete, so an unreachable or saturated
blob node delays reclamation instead of failing a delete the tenant has already
been told succeeded. The reaper is cluster-wide and leader-gated, which is also
what serialises the teardown: three concurrent destructive handlers ran one
volume's objects on prod because a retry had nothing to collapse against.
A volume already recorded as deleting answers a second delete from the record,
which collapses both the tenant's retry and DeleteOnTermination's own delete.
Two ordering fixes behind it. The provider handler now tears down local state
before deleting objects, because the local half is fixed-cost and sat behind the
half that scales, so a timeout left this node's WAL and dirty marker behind for
good. And volume.delete and snapshot.delete are classified as data movement,
since both reference real data and neither answers from state already held.
Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's
replaced libraries to the daemon and restarted it after every dpkg run. The
restart never cleared them, so an unattended upgrade ran the unit into its
start limit and left the control plane down.
setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting
spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists
them, so restarting after an upgrade stays the operator's choice.
* test: characterise EC2 through the signed gateway pipeline
* refactor: serve EC2 actions from domains/ec2/awsapi over composed capabilities
The EC2 action table, idempotency and validation move out of the gateway behind a Deps struct. Quota is a consumer-side interface and the instance adapters take a narrow InstanceProfiles interface instead of the whole IAM service. The gateway only composes Deps and calls Dispatch. The 18 gateway to EC2 awsapi allowlist rows are deleted.
* test: drive every EC2 action through Dispatch in its own package
Covers the request errors, the gateway-local actions without NATS and each routed action reaching its handler against a stub daemon.
* docs: record the EC2 capability move in the package boundary migration
* docs: record the EC2 capability move PR number
* feat: persist the SQS worker's store key on its own disk
The shared SQS worker generates its store-client NKey inside the guest and
keeps it in an owner-only directory on its own persistent disk, so no platform
component creates or unseals the seed. The key is synced and linked into place
without replacing anything; a missing, mismatched, unreadable or loosely
permissioned key fails closed instead of being regenerated.
* fix: make the SQS worker key durably named before returning it
Every successful call now syncs the key directory and its entry in the parent
before handing out a key, so a retry after a failed directory sync cannot
return a key whose name is not yet durable. Only the key directory is created,
inside a parent that must exist, so an unmounted worker disk fails closed. The
first-start decision is explicit: an existing store's authorized key is
required and never defaults to generating a new one.
* docs: state that workerkey cannot detect an unmounted worker disk
An existing parent directory does not prove the worker disk is mounted, since
the mountpoint can exist on the initramfs after a failed mount. The error now
says only that the parent is missing, and the package documents that its
caller must verify the expected data volume is mounted first.
* feat: add the SQS worker guest bootstrap
The shared SQS worker microVM boots with sqs-worker boot, which selects
exactly the disk whose serial fw_cfg names, checks and mounts it at /data,
and proves from the mount table that this disk is what is mounted before
loading or creating its store key there. A missing, empty or unknown store
status, a contradictory or invalid store key, a missing or ambiguous disk, a
failed check or mount, an empty mountpoint or a foreign mount refuses without
creating a key. Only the public key is logged and published.
The image is built by the store image script under SQS_MICROVM_PROFILE=worker,
with the worker binary, no nats-server and an empty /data mountpoint; the
store image's contents are unchanged.
* fix: refuse replayable new-store status at SQS worker boot
A static fw_cfg store-status of "new" survives a guest reset, so after the
store exists it could still reach key creation and replace a lost worker key.
Boot now accepts only "existing" with the store's authorized public key and
refuses "new" before touching any disk, until a current authoritative
first-store decision exists. workerkey.ForNewStore remains, uncalled.
* feat: add SQS first-store claim and worker key acquisition
The SQS domain records one first-store decision per Region in its own KV
bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound
to the worker's instance ID, claim ID and generation. Key creation is
authorized only while no public key is recorded; afterwards only the
recorded key is returned, and a launch intent requires it. Older
generations, same-generation rivals and unreadable records authorize
nothing, and records hold no private key material.
workerkey.Acquire syncs a new key to disk and returns it only after its
public key is recorded, reusing the on-disk key after a crash before
recording and failing closed on a missing or different key afterwards.
No transport carries the decision yet.
* fix: read SQS first-store decisions from the stream leader
KV.Get uses a direct get, which a lagging replica can answer, so a stale
claimed read could authorize key creation after the leader had recorded
the worker's public key, or make a superseded generation look current.
Reads now use the SQS store's leader-answered stream message get, and
every authorizing answer is also confirmed by a compare-and-swap at the
revision read, so a stale or unavailable read authorizes nothing.
* feat(sqs): add regional worker identity record and instance index
The SQS domain keeps one worker identity record per Region with a monotonic
generation, claim ID, pinned node and encrypted data volume, moving from
launching to bound or failed under compare-and-swap. A launch intent takes
the next generation before any instance exists, so the previous worker stops
being current; binding writes the instance index entry first.
Authorize accepts an instance only while the Region's leader-read record is
bound to exactly that instance, generation and claim, confirmed by a
compare-and-swap at the read revision, so an old, orphaned, rewritten or
unbound index entry, or a lagging replica, authorizes nothing.
Both this bucket and the first-store bucket now open through the kvutil
chokepoint and refuse a stored configuration that could expire, cap or evict
records or holds fewer replicas than the cluster needs. Nothing calls the
store yet.
* fix(sqs): make worker launch exclusive and harden durable buckets
BeginLaunch now refuses a new claim while the previous attempt is launching
or bound, including one whose outcome is unknown, so only one caller ever
holds an intent to start a VM on the pinned volume. A replacement takes the
next generation only after FailLaunch or the new RecordStopped confirms that
nothing holds the volume, and each request names the generation it
supersedes, so a replayed request cannot take a generation it never saw.
The durable bucket opener also refuses non-limits retention, per-message
TTLs, asynchronous flush, a message-size cap, any discard policy other than
discard-new or a per-subject discard, unacknowledged writes, a sealed
stream, and a mirror or sources.
* feat: add internal SQS worker first-store gateway actions
Serve ClaimFirstStore and RecordWorkerPublicKey under the internal
spinifex-sqs signing service, apart from the public AmazonSQS inventory.
Only an sqsWorkerRole session in the system account passes the class
gate, then the role policy; the worker identity is resolved from the
current workerident record by the STS-set session name, never from the
request, and any extra parameter is refused. Seeds and other non-public
keys are refused without being recorded, logged or echoed.
The gateway binds the worker-identity and first-store buckets per
request and never creates them, so it provisions nothing and answers
ServiceUnavailable until a worker launch has created them.
* fix(sqs): fence replaced worker generations out of first-store
A request that passed worker authorization before a confirmed stop and
replacement could reach an unclaimed first-store record ahead of the
successor and create or record the old generation's key. The launch path
now raises a monotonic first-store fence past the replaced generation
before BeginLaunch, which refuses a replacement until a leader read shows
it; firststore refuses every call from a generation below the fence.
The internal adapter no longer logs caller-chosen parameter names.
* feat(sqs): launch the shared SQS worker on its pinned node
Add the SQS worker system-instance kind and a domain launcher that
realizes the Region's worker from its identity record without ever
starting a second VM:
- domains/sqs/workerlaunch returns a bound worker as is, prepares the
data volume on first launch only, refuses a volume whose own record
is not encrypted, marks the intent dispatched before launching and
never resends a dispatched launch: an ambiguous outcome stays
launching until reconciled. A replacement starts only after a
confirmed stop or failure and the first-store fence, reusing the
pinned volume.
- workerident gains MarkDispatched and Current; firststore gains a
leader-read RecordedKey.
- The daemon launches the worker through the shared direct-boot path
with its encrypted volume attached at boot and kept on terminate, a
required instance profile and a mandatory mgmt NIC. Its firmware
config carries only public values: the data serial, the store
decision, the gateway URL on this node's bridge and the daemon's own
cluster CA. A relaunch is refused on any other node.
- sqsWorkerRole is ensured in the system account granting exactly the
two internal actions; an existing role whose trust admits more than
EC2 is refused. The role contract lives in domains/sqs/workerrole so
the composition root does not import the gateway adapter.
The worker network allocator and any caller remain unwired, and the
guest still refuses to create a key for a new store.
* fix: verify SQS worker role trust before granting and report unclaimed volumes
Check that an existing or racing-created sqsWorkerRole trusts only EC2
before writing its policy, and accept the instance profile only when a
re-read shows it carries exactly that role. Refuse the first launch
before allocating when the cluster storage key is not configured, and
return the ID of any volume created but not confirmed in a launch
intent instead of dropping it.
* feat: add private SQS system VPC and worker ENI allocator
Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet
from the VPC and subnet surface only, so it cannot create an internet
gateway, route table, NAT gateway or EIP. The daemon's SQS worker network
adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and
finds or creates one ENI per launch claim in the VPC default security
group, refusing a duplicated claim or an address outside the subnet.
Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have
recorded the volume.
* feat: worker takes its first-store decision from the gateway at boot
The SQS worker guest no longer trusts a static fw_cfg store status. After
selecting, checking and verifying the mount of its own data disk, it asks
the internal spinifex-sqs gateway service for its current first-store
decision, signed with instance-metadata credentials only, over TLS 1.3
trusting only the cluster CA. It then creates and records or loads the
key on that disk; a key lost after recording is never replaced.
- internal/sqsworkergw: SigV4 client for ClaimFirstStore and
RecordWorkerPublicKey; refuses non-public keys, non-decisions and
redirects; errors carry only the parsed code
- gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback
- workerboot: retries only lost replies and ServiceUnavailable/Throttling
- launch contract carries the Region instead of a store status or key
* fix: worker retries a denial while its launch is still binding it
The worker VM can reach the gateway before its launch binds the instance
identity, and that early call is legitimately AccessDenied. The guest
treated the denial as final and stayed unprepared even after binding.
AccessDenied is now a pending authorization the guest asks about again
within its existing bound; a denied claim creates nothing, and a worker
that is never bound, or was replaced, gives up at the bound.
* feat(sqs): add operator EnsureSQSWorker action and node launch handler
Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the
internal spinifex operator service. It validates the node against the
cluster and as a single subject token, then requests
sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers
and the worker stays pinned.
The node handler re-checks the named node and instance type and runs one
composed workerlaunch.Launcher (real volume preparer, role, private network
and direct-boot launch), one launch at a time, in the daemon's own context
with a 30-minute bound. A caller timeout does not cancel the launch; a
retry reports the bound worker, or IncorrectState for a dispatched launch
whose outcome is unknown, which is never resent. The reply names the bound
VM and does not claim the worker or SQS is ready. Nothing starts a worker
at daemon startup.
* feat: ship the SQS worker image and check it before allocating
Build the SQS worker microVM image in both release workflows, stage it into
the tarball as microvm-sqs-worker, and install it with setup.sh under the
store image's rules: a release without it removes an earlier one and an
incomplete one is refused. Add make install-sqs-worker-microvm.
The worker launcher now asks a Host capability whether this node can boot a
worker before it prepares a volume, records or resumes an intent, fences a
replaced generation, allocates the network or dispatches. The daemon checks
for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and
the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a
call after the image is installed launches instead of leaving an ambiguous
dispatched intent.
Cover missing and incomplete images through the admin action and handler,
a stopped worker left untouched, and an ENI create whose reply is lost.
* fix: never repeat an uncertain SQS worker ENI create
The launcher now looks for the claim's ENI without creating one, marks the
launch intent NetworkAttempted, and only then sends the single create. Once
marked, a missing ENI fails closed with IncorrectState instead of a second
create that could give the claim two interfaces; a visible ENI is reused.
Adds a test that buildSQSWorkerLauncher wires the daemon's real host check.
* feat(sqs): operator-triggered SQS store launch on the worker's node
Add the super-admin EnsureSQSStore action and its node handler. The
store is launched only on the node the Region's worker is bound to,
with the public key that worker recorded in the first-store decision;
the daemon never receives the worker's seed.
A per-Region store launch record marks each step before its uncertain
create (volume, ENI in the worker's private subnet, VM launch), so a
step whose result is not visible fails closed and is never resent.
Volumes are refused unless their own record is encrypted, and their
IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and
instance types at or below that are refused; this is a floor, not a
validated size.
* fix: retry uncreated SQS store volumes and fix the store instance type
A volume preparation that failed before sending any create (a missing
tool, scratch or format failure, or no cluster storage key) now carries
storevolume.ErrNotCreated. The store launcher withdraws its volume mark
and returns ServiceUnavailable, so a retry may try again once the host
is fixed; any create that was sent stays one-shot.
The store launch record now keeps the operator's instance type. A
request for a different type is refused as IncorrectState before
dispatch and after binding, and the reply names the launched type,
which the admin action checks.
* feat: SQS worker store readiness and store size check
The worker's one-shot ready step mounts and verifies its disk, asks the
internal spinifex-sqs DescribeStore action for the bound store, loads its
key only if it matches the recorded public key, and checks the store's
JetStream with it. It retries within a bound while the store is pending or
not answering, and stops on a mismatched or refused key. The NKey
authenticates the worker only; the store server is not authenticated.
EnsureSQSStore now also refuses a different volume size once the store is
bound, instead of returning the existing store.
* fix(sqs): keep worker store readiness current until shutdown
The readiness step now runs as a guest-local watch loop with no overall
deadline: each pass is bounded, a pending or unreachable store is retried
every few seconds, and a ready store is rechecked periodically. The
store-ready marker is removed on any failed pass, cleared at start and
removed on exit; shutdown stops the watcher by its recorded PID. Terminal
refusals such as a rejected or mismatched key back off for minutes rather
than spinning. The NKey still authenticates only the worker to the store.
* feat(sqs): supervise the worker readiness watcher and add spx admin sqs
Run the worker guest's readiness watcher under init respawn, rate-limited,
and remove any stale store-ready marker before each start, so a killed or
crashed watcher cannot leave a false readiness marker. Shutdown blocks
respawn, removes the marker, stops the recorded watcher by PID and start
time with SIGKILL escalation, and unmounts /data only once it has exited;
a surviving watcher leaves every mount in place.
Add spx admin sqs ensure-worker and ensure-store, which call the existing
EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex
gateway path with the standard AWS credential chain. Both refuse missing
inputs before sending anything and report only the bound VM, never
readiness.
* fix: park SQS worker readiness and fix the worker instance type
A readiness start that finds shutdown in progress or no worker key now
logs once and waits in place instead of exiting, so init's respawn does
not restart and log it in a loop; shutdown stops it like a watcher.
The worker identity record keeps each generation's instance type. A
request for another type is refused as IncorrectState before dispatch
and after binding, and only a replacement may change it. The ensure
reply, admin action and spx admin sqs ensure-worker report and check
the recorded type.
* fix: record the SQS worker volume intent before creating it
The first worker launch prepared its data volume before any record
existed, so a create that applied but returned no ID left nothing for the
next call to find, and it could prepare a second volume.
The launch intent is now recorded first, and its one volume create is
marked before it is sent. A Prepare proven to have sent nothing withdraws
the mark and is retryable; any other outcome is never resent and returns
IncorrectState, naming the volume when its ID is known. An intent that
has sent no create may still be re-typed.
* docs: document spx admin sqs ensure-worker and ensure-store
Record the flags, the caller and owning accounts, the bound-versus-ready
distinction and the current retry and recovery limits.
* docs: distinguish SQS ensure timeouts from refusals that created nothing
A gateway timeout means only that no reply arrived; the launch may still be
running. The worker type locks once any step has been attempted.
* docs: state what each SQS ensure ServiceUnavailable establishes
No refusal proves earlier attempts created nothing; a retry checks the
durable attempt and continues only when that is safe.
* test: relay a capacity refusal from an owner holding unsealed writes
* fix: refuse a start elsewhere when the owner holds unsealed writes
* fix: bound the unsealed-writes check well under the start forward timeout
* test: refuse a volume takeover while the marker owner is live
* fix: refuse a volume takeover while the previous owner is live
* fix: retry EKS control-plane starts refused for capacity or a held volume
* docs: describe the interim unsealed-writes takeover guards
* test: give the viperblockd conformance harness a cluster-state bucket
The takeover guard reads node liveness from the cluster-state bucket the
daemon creates. The harness had none, so the real reader errored and the
guard refused every takeover. Create the bucket the way the daemon does so
the real reader runs and finds no heartbeat.
* fix: leave no dirty marker behind when opening a volume that does not exist
An open takes the lease and writes the dirty marker before the volume's
state is read. When the volume turned out to be absent the marker stayed,
so a later open on another node met the takeover guard and was refused
as held elsewhere instead of answering not found. Clear the marker this
open wrote when the state load reports the volume missing.
* test: characterise the four Bedrock services through the SigV4 pipeline
* refactor: serve the four Bedrock services from one ochre registration
The Bedrock adapters move from gateway to domains/ochre/awsapi behind a
single dispatch registration. The sub-service scopes are aliases, a
sub-service scope selects its dispatcher outright and only the shared
bedrock scope is split by path. The thirteen Bedrock fields leave
GatewayConfig for awsapi.Deps, and the bodyscope allowlist row for
domains/ochre is deleted.
* docs: record the Bedrock registration in the package boundary migration
* docs: record the Bedrock registration PR number
* test: pin the ECR /v2 pipeline through the gateway handler
* feat: rehydrate a principal from an access key id in credverify
Adds credential.Rehydrator and CallerARN on Principal, implemented beside Authenticate and Resolve on the same key, session and user lookups, with a parity table against the gateway's ECR rehydrate.
* refactor: share the credverify key and session lookups with Rehydrate
Authenticate, Resolve and Rehydrate now call one lookup per credential kind, so the checks cannot drift. The parity table also asserts the gateway and credverify caller ARN builders agree.
* refactor: move registry host matching to ingress/http
The matcher takes its service labels from the caller and exposes the parsed account and region through accessors, so ingress names no service.
* refactor: extract the identity-policy evaluator into policyauth
The SigV4 gateway path and the ECR registry authorize through one evaluator and one set of request condition keys, over the credential principal.
* refactor: extract account admission into accountadmission
The SigV4 gateway path asks one Admitter for account status, with the ACTIVE-only cache moved with it, so the ECR registry can use the same capability.
* refactor: move the ECR /v2 bridge out of the gateway
The bridge, token exchange and per-operation authorization now live in
domains/ecr/registry behind registry-owned Principal, Rehydrator and
Authorizer types, so the package imports no ingress code. The awsgw role
adapts credverify, policyauth and host matching to them, and the gateway
keeps only a RegistryHandler mounted at /v2. The bridge refuses to
compose without a verifier, issuer, rehydrator, authorizer or admission,
so the open-registry mode for a nil verifier is gone.
* test: carry the retired ECR principal parity rows into credverify
Global root, a verifier with no IAM, a record with no principal type and
the gone, replaced and legacy session verdicts for user and role sessions
were only exercised by the gateway tests that went with the old path.
* docs: record the ECR /v2 bridge move in the package boundary migration
* docs: record the ECR /v2 bridge PR number
* test: cover AWS-form IRSA condition keys and StringLike on web identity
* fix: match AWS-form IRSA condition keys and allow StringLike on web identity
* fix: keep the provider ARN host-path unnormalised for web identity
* feat: log the reason a web identity trust policy denies
* fix: satisfy errname, modernize and gofmt on the denial error
* test: cover the stock AWS IRSA trust policy in the EKS e2e
* test: make the IRSA denial prove the policy was evaluated
* test: pin ECS and EKS dispatch through the SigV4 pipeline
* refactor: register ECS and EKS on the ingress seam
EKS resolves the cluster and nodegroup role through passrole.Check with the
IAM role resolver, in the order it had. Responses and check order are unchanged.
* fix: match AWS role handling for ECS and EKS PassRole cases
ECS refuses a foreign task role before PassRole, accepts a role that is not an ARN or does not exist after PassRole, and reports a RunTask override role it cannot assume. EKS CreateCluster and CreateNodegroup refuse a foreign or non-role ARN with AccessDeniedException before PassRole.
* test: cover ECS and EKS awsapi registration dispatch
* fix: report EKS parameter faults as InvalidParameterException
* docs: record ECS and EKS gateway migration and EKS divergences
* test: pin the current role-resolution gaps in EC2, ECS and EKS
* feat: add the shared iam:PassRole flow with classified role resolution
* feat: resolve same-account roles for iam:PassRole in domains/iam
* docs: point the /admin migration row at spinifex#1345
* test: characterise IAM and STS dispatch through the SigV4 pipeline
* refactor: register IAM through the ingress seam
* refactor: register STS through the ingress seam
* docs: point the IAM and STS migration rows at spinifex#1346
* refactor: move the /admin handlers and wire types into operator/admin
* refactor: serve /admin through the spinifex registration
* docs: record the /admin move into the spinifex registration
* refactor(gateway): register RDS through the ingress seam
RDS dispatch moves from gateway/rds.go to domains/rds/awsapi.NewRegistration, composed by awsgw and the integration harness.
The caller comes from dispatch.Actor and the principal-class gate still runs before the policy check.
* docs: point the RDS migration row at spinifex#1344
* chore: group the imports merged in the actor projection test
* feat: build a separate SQS store microVM image
Package the go.mod-pinned nats-server, ext4 support and a clean ACPI
power-down into a microVM kernel and initramfs of their own. The build
checks the packed archive's contents and refuses a nats-server whose
version differs from the pin. The image does not start the store and
nothing installs or releases it; the LB image is unchanged.
* fix: publish the SQS store image only after its gates pass
Stage the kernel and initramfs and move them into place only once the
content, size and guest-RAM gates pass, so a failed build cannot leave a
pair make takes as up to date. The old initramfs is removed first and the
new one lands last, so an interrupted publish never leaves a mixed pair.
Check the module list without piping into grep -q, which under pipefail
reported a present module as missing.
* feat: start the SQS store guest on its checked data disk
The SQS store image now prepares, runs and stops a dedicated JetStream
server. It reads a narrow fw_cfg contract (data disk serial, listen
address, one user NKey public key), refuses anything absent, malformed
or ambiguous, has the pinned server test the config, then checks and
mounts exactly one matching ext4 disk. It never formats and has no
ephemeral fallback. The server listens only on the supplied guest NIC
address, admits only that NKey and syncs every write.
Shutdown stops the recorded run process first, whether it is the server
or a start not yet exec'd, and runs the generic read-only remount only
once it is gone. A store that survives SIGKILL leaves every mount alone.
* fix: report the SQS store as prepared, not listening, before it starts
* feat: launch the SQS store guest on its own versioned subject
An explicit SQS store kind carries an existing system-account data volume
and the one NATS user public key the store accepts. It is served only on
system.LaunchSQSStore.v1, so a daemon without this code has no responder
and the launch fails instead of booting a load balancer without its disk;
the legacy launch path refuses the kind on both send and receive.
Every refusal happens before capacity, an ENI or a VM record exists. The
volume goes through the normal EBS mount and terminate detach and is
never deleted on termination. The store boots its own image with CPU
mitigations on, gets only its network, disk serial, listen address and
key, and cannot be relaunched after a host restart yet.
* feat: ship the SQS store microVM image in releases and installs
Release builds now build the SQS store image and stage it beside the load
balancer's as microvm-sqs-store/, so an older installer that copies
microvm/* without -r still works. setup.sh installs it to
/usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it.
A release without a complete image never leaves an older one usable:
staging drops a previously staged image, and the installer removes an
earlier installed pair before warning (missing) or failing (incomplete).
Only the two image files are removed. The initramfs is installed last, so
an interrupted install leaves no bootable mismatched pair.
* feat: restart the SQS store guest on its launching node
Persist the non-secret guest config of an SQS store instance (launching
node, data volume, listen address, client public NKey and rendered
network config) in the instance record, and rebuild its fw_cfg files
from it on relaunch. The relaunch refuses on any other node, or when the
record is missing or malformed, before writing any file; the data volume
is remounted through the normal EBS path and kept on terminate.
Cross-node recovery now never selects or claims an SQS store record, so
no other node spends retry budget on it or relaunches it. Same-node
restore is unchanged, as are load balancer restarts. Older daemons know
neither the exclusion nor the record field, so this does not prevent a
cross-node relaunch by a mixed-version cluster.
* feat: hold prepared system volumes in creating until released
Add an internal CreateVolumeForPreparation path that writes a new
volume's first record as creating, and CompleteVolumePreparation to
release an unattached creating volume in the caller's account to
available. Ordinary CreateVolume is unchanged.
Attach and delete already refuse a creating volume. ModifyVolume,
ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState
so nothing can expand, release or capture a volume mid-preparation.
* feat: prepare fresh SQS store volumes with an ext4 filesystem
The SQS store guest never formats a disk, so a new volume gets its
filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img,
builds an ext4 image in private scratch, creates the volume in
"creating", publishes it on this node, copies the image with the NBD
import helper, unpublishes, and only then releases it as available.
It never writes a volume it did not just create. A volume is deleted
only when no export can still be live: after a confirmed unpublish, or
when the export was not NBD. Any publish error, a failed or ambiguous
unpublish, or a failed release retains the volume and reports its ID.
A publish refused as in-use or already-exists sends no unpublish, since
another export may exist. Cleanup ignores caller cancellation but is
bounded at 30s.
The raw-image copy moves to runtime/compute/nbd unchanged, and the
domain receives it as an injected writer. The volume service gains a
discard path for an unattached volume still in "creating".
* fix: report unknown state and match qemu-img lookup in volume prep
A failed release may still have made the volume available, so the
retained-volume error now says its state is unknown rather than
claiming it is still creating.
The copy runs qemu-img by name through PATH, so the pre-allocation
check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin
fallback because it runs from the resolved path.
* docs: describe the unwired SQS store system instance
Records what the guest image, versioned launch, same-node restart and
fresh-volume preparation provide, that nothing calls them, and what is
untested or missing before production use.
A dispatch failure was logged at ERROR whatever the caller was about to
be given, so DryRunOperation, which AWS documents as the success answer
to a dry run, read as a fault of ours along with every NotFound and
malformed parameter. The level now comes from the status the caller
receives, resolved the way ErrorHandler resolves it.
QEMU announces its own exit on stderr, so a guest stopping on the
SIGTERM the daemon sent it was logged at ERROR too.
* fix: count volume lease validity from the write, not its reply
The server starts a lease entry's TTL when it applies the write, but the
holder started its validity window when the acknowledgement arrived, so a
slow reply stretched validity past the server's expiry and a peer could
acquire the volume while the holder still considered it valid.
Validity now counts from when each create, generation publish and renewal
was sent. An acquisition acknowledged after validity lapsed is refused as
retryable, and a lost or lapsed lease still awaiting its fence is no
longer shared with another open on the same node. A renewal that lands
after the lease was given up, or after its validity elapsed, cannot revive
it: the lapse check and the update run under the lease lock, and fencing
runs once. The renew loop also fences at the validity deadline itself
rather than at the next check, which could consume the whole margin.
This bounds local validity only; it is not backend-enforced fencing, and a
paused host can still write after its validity ends.
* docs: describe volume lease validity, fencing and their limits
* refactor(gateway): verify SigV4 credentials behind a CredentialVerifier
The authenticator no longer imports the IAM and STS domains. It hands the
signature check to a verifier as a callback, so the secret never leaves the
adapter, and runs the account-active check between Authenticate and Resolve.
A nil verifier fails every request closed.
The adapter lives in runtime/roles/awsgw/credverify. ResolveCallerUser and the
principal-type constants move from domains/sts/awsapi to domains/sts so the
adapter can use them without importing an awsapi package.
* fix(gateway): guard a nil credential verifier and tighten its tests
A typed-nil credverify.Verifier now fails closed instead of panicking.
The session verdict test goes through the real verifier into the lockout boundary.
Test setups build the verifier from the same mock as the gateway services.
* test(gateway): commit the verifier test helper
* test: pin the caller facts the gateway gives a registered service per principal kind
* refactor(gateway): seal the verified caller into one request-context value
* feat(dispatch): give registered services the caller's principal ARN, user ID, user name and session name
Actor gains PrincipalARN, UserID, UserName and SessionName, filled from the
sealed principal, and RoleName(), which keeps the account-checked role ARN
rule in one place. No consumer reads them yet.
* refactor(gateway): read the caller from dispatch.Actor in the RDS, EKS, STS, IAM and admin handlers
callerActor builds the Actor the seam hands services, and the seven readers that
took fields off the sealed principal now take them from it. RDS and EKS take the
role name from Actor.RoleName(). No behaviour change.
* test: characterize the spinifex Query and /admin surfaces through the full gateway pipeline
* refactor: register the spinifex Query service from operator/admin
* docs: point the spinifex registration migration row at the PR
The ELBv2 move to the ingress seam added its inventory to the coverage tool but not to TestRequestConformance, so 36 operations and 45 findings dropped out while the suite passed. A served service with no implemented operations now fails the sweep.
* test: characterize Tagging dispatch, authorization and GetResources responses through the gateway pipeline
* refactor: register Resource Groups Tagging through the ingress seam
* test: characterize ELBv2 through the full gateway pipeline
* refactor: register ELBv2 through the ingress seam
* test: cover the ELBv2 registration, query parser and load balancer limit
* test: characterize RDS through the full gateway pipeline
* docs: point the Tagging and ELBv2 migration rows at the PR
* test: register Tagging and ELBv2 on the integration harness gateway
AWS's EC2 error reference documents it, and DeleteEgressOnlyInternetGateway and ReplaceRoute answer it, so the integration suite reported two false violations.
* refactor: move gateway/spx to operator/admin
* test: pin the admin wire types and method list in golden JSON
* refactor: move the admin wire types and method list to contracts/operator/v1
* refactor: serve EKS OIDC discovery and JWKS from the EKS domain
* refactor: assert the EKS OIDC publisher satisfies the gateway interface
* docs: cite the pull request in the gateway root-move rows
The captures are about 90% of each E2E artifact and nothing reads them unless
a run failed, so a clean single-node or multi-node pass drops them before the
artifact upload.
* feat: accept EnaSupport on RegisterImage and name the refused parameter in its errors
EnaSupport true and false are stored on the image record, returned by DescribeImages when set and carried through CopyImage. Every validation error in ValidateRegisterImageInput keeps its code and now says which parameter and value it refused.
* feat: set EnaSupport on images created from an instance
AWS takes an image's EnaSupport from the instance it was made from, and every instance here reports it, so the source AMI's value is not inherited.
* docs: record RegisterImage EnaSupport support and the refused parameters
* fix: report EnaSupport on DescribeInstances and RunInstances
* refactor: remove the host SQS store systemd unit
The SQS message store will run inside system-owned instances rather than
as a host daemon, so spinifex.target no longer starts it. Bump the target
to version 4 and drop the unit from the embedded set and its invariants.
* refactor: remove host SQS store config provisioning
The SQS store will run in system-owned instances, so admin init and join
no longer generate or distribute a host store token, routes or config,
and the formation join payload drops the store token field.
* refactor: remove host SQS store service command and role
The SQS store will run in system-owned instances, so drop the unused
spx service sqs-store command, its service registration and the host
sqsstore role. The domains/sqs/message JetStream store is unaffected.
* refactor: remove host SQS store provisioning from install scripts
The SQS store will run in system-owned instances, so setup no longer
creates the spinifex-sqsstore user, its directories or the 4223/4249
peer firewall ports, and install, reset and uninstall stop handling the
retired unit, user and data directory.
* docs: drop the host SQS store from firewall and connection tables
Spinifex no longer ships a host SQS store service, so remove its ports
from the host firewall and network connection references and say in
DESIGN.md that no host service, unit, user or firewall rule exists for it.
The targeted suite selector (select-suites) has not run in CI since its shadow-mode step was removed in #556.
The service-interfaces manifest, manifest-check and manifest-lint existed only to feed it, so they go too.
Also drops the now-meaningless e2e:allow-create markers, keeping their rationale as plain comments.
* feat: render X-Forwarded-For, -Proto and -Port headers in ALB HAProxy frontends
Append joins every client X-Forwarded-For line and adds the client address as AWS does, preserve and remove follow the load balancer attribute, and Proto and Port are always overwritten.
* feat: validate the ALB forwarding attributes and re-render config when they change
ModifyLoadBalancerAttributes rejects bad xff_header_processing.mode and xff_client_port.enabled values with ValidationError before storing anything, and regenerates the stored config when either key changes.
* test: assert the ALB X-Forwarded headers and the remove mode in the lb e2e
The app instances echo the request headers on /__headers. The internal ALB is driven from the probe client and the HTTPS ALB from the runner, covering the caller address, a client value, spoofed Proto and Port, mode remove and a rejected mode.
* test: let the lb probe client trigger accept a null header list
* docs: document the ALB forwarding headers and their attributes
* feat: add the store-neutral SQS message store contract and conformance suite
Defines message.Store over opaque shard and queue ID namespaces with no
create call, plus a reusable conformance runner exercised against a
test-only in-memory reference store.
* feat: add the SQS message store's JetStream layout, append and state CAS
Provisions the fixed set of shared message streams and state buckets,
publishes each queue on sqs.m.<shard>.<queue>, dedupes idempotent appends
with a queue-scoped Nats-Msg-Id, and maps state revision CAS onto KV.
* fix: give each SQS queue immutable message and state shard allocations
A changed stream or bucket count applies only to new queues: the namespace
names its message shard and state shard directly, retained streams and
buckets stay addressable, and the store refuses cluster sizes other than
one node (R1) or three (R3).
* fix: enumerate complete SQS namespaces and backstop state buckets
Enumeration now returns each namespace with both allocations, so a
state-only namespace can be counted and cleaned without its queue record,
and state buckets expire records after 14 days plus 1 hour.
* feat: add the SQS JetStream store's reads, counts and cleanup
Get and Delete are scoped to the namespace's subject, DeleteThrough and
DeleteNamespace are subject-filtered purges that leave shard neighbours
intact, and Count reads the per-subject message total and live state keys.
* fix: count distinct SQS state keys and validate before namespace purge
The key lister can repeat a key under concurrent writes, so Count tallies
distinct names; DeleteNamespace resolves both the message stream and the
state bucket before purging either, so a wrong state shard deletes nothing.
* feat: add the SQS JetStream store's ordered message-shard feed
Append records the namespace's state shard in an internal header that is
never returned as metadata. Feed reads one shared message stream through
an ephemeral, headers-only ordered consumer with no acknowledgement state,
rebuilds each change's complete namespace, refuses a malformed header with
ErrCorruptRecord, and deletes its consumer on Close.
* feat: enumerate SQS namespaces and run conformance on the JetStream store
Namespaces joins per-subject body counts, whose internal header names the
state shard, with live state keys from every state bucket that exists,
found by listing so a historical bucket counts and none is created. An
allocation conflict is reported as ErrCorruptRecord. The store now
implements message.Store and runs the conformance suite on R1, per case
with crash reopen and on one shared instance.
* fix: check every SQS body's state shard when enumerating namespaces
Sampling the first and last body let a middle body naming another state
shard pass. Namespaces now reads every body's header in one headers-only
ordered pass over the shard's stream, counting each queue's bodies from
that pass, and deletes its consumer when done.
* test: run SQS store conformance on R3 and guard against consumer state
TestR3Conformance runs the conformance suite on a real three-node
JetStream cluster in process, with Reopen restarting every node over the
same data, and a check that every stream and state bucket is R3.
TestS5_NoConsumerDeliveryStateInSQS fails on any acknowledgement,
redelivery, durable-consumer or ack-retention mechanism in SQS source.
* test: bar legacy NATS acknowledgement helpers from SQS source
Adds AckSync, ManualAck, AckAll and AckExplicit to the ADR-0011:S5
consumer-state guard.
* feat: add a bounded SQS store scan and the tombstoned-body sweeper
Store gains Scan: a paginated, headers-only pass over a message shard,
bounded at the shard's last position when it starts, skipping deleted
positions and never returning a later append. The conformance suite and
fake cover continuation, deleted runs and the bound.
The JetStream adapter reads every bounded range (scan, body enumeration
and state keys) through one reader whose completion rests on a direct get
of the next match, not on consumer state; the client's key lister and
pending counts are no longer trusted to end a pass.
message.Sweep walks a shard with Scan and, for each body whose record the
domain reports terminal, deletes the body then the record by revision.
* test: add an opt-in SQS store conformance run on an external R3 cluster
TestR3ExternalConformance runs the conformance suite against the
dedicated three-node SQS JetStream cluster named by SQS_STORE_R3_URLS,
checks every stream is R3 with two peers, and uses SQS_STORE_R3_RESTART
to restart every node for the crash cases. It skips when unset.
* test: run the SQS sweeper against the reference store fake
Covers live records kept, tombstones swept, another shard untouched, a
no-op rerun, and a retry after the body delete succeeds and the state
delete fails.
* test: harden the SQS store R3 harnesses against restart and placement races
The external R3 harness reused one connection across restarts, so a case's
cleanup closed it for every later case; each case and restart now opens its
own connection and waits for stream leaders.
The in-process cluster released its free ports before the servers bound them,
and R3 placement can fail until the meta leader has every peer's config and
stats. Client ports are now server-chosen, route ports are reserved together,
and provisioning retries the idempotent New until placement succeeds.
* fix: stop the SQS store feed idling for a whole pull expiry
Under load an ordered feed on a three-node cluster sat for nats.go's 30s
default pull expiry, once 60s, with stored changes pending and no pull
waiting on its consumer, both on feeds opened after a full restart and on
feeds kept open across one. Feed now sets a 2s pull expiry.
Adds restart tests for a feed opened after a full-cluster restart and one
open across it, each replaying a shared-shard backlog then a live append and
failing on any 5s gap. The in-process cluster keeps client ports across
restarts so open connections reconnect, and can write node logs.
* fix: read SQS message state from the stream leader
GetState used nats.go's KV Get, which switches to a direct get whenever
the bucket allows it. On a replicated bucket a follower can answer that
from state it has not applied yet, so a record the leader had already
deleted or purged could still be returned.
Read the key's last message with a stream message get instead, which
only the stream leader answers, and apply KV Get's delete and purge
marker rules to the result. Bucket configuration is unchanged. Missing,
deleted and purged records still return ErrNotFound; values, revisions,
context errors and the API prefix or domain of the caller's JetStream
context are preserved.
* docs: narrow the SQS GetState leader-read comment to what it avoids
* fix: enumerate SQS state keys from the stream leader
Count and Namespaces found live state keys with a direct-get probe and a
headers-only ordered consumer on the state bucket's stream. On a
replicated bucket a follower can answer either from state it has not
applied, so keys the leader had already deleted could still be counted.
Walk the key prefix with stream message gets by next subject, which
only the stream leader answers, through the stream's last sequence at
the call. Delete and purge markers drop a key, as GetState reads them.
This costs one leader request per record under the prefix. Bucket
configuration, historical state shards, the message-body scans and the
feed are unchanged.
* test: keep SQS API recorder deliveries in arrival order
The recorder fed API subjects and its sentinel to two async
subscriptions, which nats.go delivers on separate goroutines, so drain
could return before the last request was recorded. Both now share one
channel that the connection fills in arrival order.
* fix: read SQS body enumerations from the stream leader
Namespaces and Scan read bodies through an ephemeral ordered consumer,
which the server may place on a follower that has not applied a purge,
so a deleted queue could still be listed or scanned. Each stored match
through the bound is now a leader stream message get. That transfers
every full body, where the consumer read headers only.
* test: stop the SQS API recorder goroutine with its test
The recorder ranged over a caller-owned ChanSubscribe channel, which
nats.go never closes, so its goroutine outlived every test. Cleanup now
unsubscribes both subscriptions, then signals the goroutine and waits
for it to return; the sentinel send also watches that signal.
* fix: recover a stalled SQS feed from its cursor
An ordered feed was seen to idle with stored changes pending after a
cluster restart. Next now waits at most an idle check on the consumer,
then asks the stream leader whether a change follows the cursor; if one
does, the consumer is deleted and reopened just after the cursor, so no
change is skipped or repeated. The root cause of the stall is unknown.
* fix: make closing an SQS feed terminal and safe against Next
Close now marks the feed closed under a lock, cancels any in-flight
Next, stops the iterator and deletes the consumer. A later or in-flight
Next returns ErrFeedClosed and never opens a consumer; one opened by a
recovery racing Close is deleted rather than adopted. Repeated Close
returns nil.
* fix: define SQS feed close in the store contract
message.Feed now documents one Next reader with Close callable at any
time and more than once, and message.ErrFeedClosed replaces the
adapter's own error. Once Close has begun, Next returns ErrFeedClosed
and withholds any change it had received; a feed from the last cursor
still delivers it. Shared conformance cases and the reference fake
enforce this.
The JetStream feed's Close also waits, within a bound, for a consumer
open already in progress, so a consumer created during Close is deleted
before Close returns nil; past the bound, or if that delete fails, Close
returns an error and a repeated Close waits again.
* fix: track SQS feed consumer cleanup until Close succeeds
Every consumer open and release is now tracked work recorded under the
lock that takes the consumer, and a delete that fails is kept for Close
to retry. Close returns nil only once no such work is in progress and no
failed delete remains, waiting within a bound and otherwise returning an
error; concurrent and repeated Close calls wait the same way.
Next claims a change under that lock and advances its cursor there, so a
change is either returned before Close begins or withheld for a feed
resumed from the last cursor.
A consumer create no longer inherits the caller's cancellation, which
was seen to leave most cancelled creates on the server with no owner;
it has its own timeout, and the name of one that fails is deleted and
kept for Close to delete again.
* fix: release SQS feed consumers opened after the caller's context ends
Feed and recovery now check the caller's context before starting a
consumer open and after it completes. A create that finishes after the
context ended is released as tracked work and the context's error is
returned, or ErrFeedClosed when Close won; no change is returned.
The feed contract and adapter docs now describe cleanup as complete for
consumers the feed knows of, with the server's one-minute inactive
threshold removing one an ambiguous create lands after its last delete.
* fix: report cancellation when an SQS feed open fails after it
A first consumer open that fails after the caller's context ended now
returns the context's error joined with the failure, so cancellation
stays detectable. The feed contract's cancellation text is narrowed to
opening and reopening, whose cleanup is best effort.
* docs: describe the internal SQS message store and its limits
DESIGN.md gains a section on the store-neutral SQS message store, its
JetStream adapter and conformance suite: an internal primitive with no
service wired to it, its shared stream and bucket layout, leader-directed
reads and their full-body cost, feed recovery and its best-effort
consumer cleanup, and the absence of end-to-end or multi-host
validation. The Scan contract says metadata only, not headers only.
* feat: record manifest subject and artifactType and return OCI-Subject on push
* fix: resolve the manifest media type once so a parameterised Content-Type cannot misclassify an index
* feat: index referrers per subject and serve them through the metadata service
* feat: serve the OCI referrers API for ECR repositories
* test: cover the ECR referrers route in the gateway authorization matrices
* feat: delete a subject image's referrers with it
* fix: satisfy nilerr in the referrers manifest parser
* fix: return the manifest parse error and format the referrers code
* docs: describe the ECR OCI referrers API and cosign support
* docs: name oras cp and crane copy as the way to carry v3 signatures to an ECR mirror
* docs: present oras cp and crane copy as one remedy for copied v3 signatures
* feat: add a gateway readiness endpoint to the EKS token webhook
* feat: serve token webhook readiness beyond loopback and admit it to the control-plane SG
* feat: health-check the EKS apiserver target group on the token webhook readiness port
* style: order the strconv import
* fix: export the readiness listener address to the EKS token webhook
* docs: regenerate the ELB coverage README
* docs: describe EKS control plane readiness and what happens when a gateway stops
* test: range over SplitSeq in the webhook initd guard
* feat: add routableTargets helper that skips unhealthy targets and fails open
* fix: drop unhealthy targets from the NLB nginx upstream, failing open when none are healthy
* fix: re-render load balancer config when a target health state changes
* fix: write load balancer config and heartbeat fields under CAS so a re-render is never rolled back
* fix: write live load balancer modifications under CAS so they cannot roll back a re-rendered config
* fix: let ALB backends fail open to unchecked backup servers when every target is down
* test: cover NLB unhealthy-target routing and ALB fail open in the lb e2e suite
* test: flip ALB backend health per instance and log how long each health phase takes
* style: gofmt haproxy backend struct and the attributes fetch table
* fix: re-render load balancer config when a target group's health check changes
* fix: return the health-target marshal error instead of discarding it
* test: mark the health-check target group create as an intentional create-path test
* docs: describe target health, fail open and time to effect in the ELBv2 guide
* feat: render a data volume as an MMIO virtio-blk-device
A package-private helper reuses the PCI volume device's id, drive,
iothread, serial and stop-on-error options, without a PCI bus.
* feat: cold-boot microvm data volumes on MMIO without a PCI hot-plug port
microvm has no PCIe root ports, so its data volumes render as
virtio-blk-device and take no hot-plug port; a recorded port is kept.
NBD and host-block exports and q35 PCI allocation are unchanged.
* feat: compose a launch-only microvm config with EBS data volumes
directBootLaunchConfig copies a clean direct-boot base and appends the
mounted data volumes after its devices, rejecting boot and EFI volumes
and non-microvm bases. Neither input is modified, so repeated launches
never accumulate entries or keep a stale endpoint.
* feat: launch direct-boot microvms with mounted data volumes from a launch-only config
startQEMU composes the direct-boot command from the saved base and one
snapshot of the mounted requests, which also feeds the readiness probes,
so the launched endpoint is the one checked and is never saved.
* fix: refuse volume hot-attach on direct-boot microvms before any mount
A microvm's MMIO bus has no PCIe root port to hot-plug into, so
AttachVolume returns ErrHotAttachUnsupported before mounting the volume
or changing instance or volume state.
* fix: return UnsupportedOperation for microvm volume hot-attach
AttachVolume's ErrHotAttachUnsupported is a caller-fixable refusal, so
the attach/detach error boundary maps it to a 400 UnsupportedOperation
instead of a generic server error.
* docs: describe direct-boot microvm data volumes and refused hot-attach
Enables the rule with functions: false, since hoisted declarations are safe, and defines CONDITION_FIELD_TEXT and the handoff message listener before their first use.
Turns on oxlint's eslint/complexity rule (max 20) and brings the 18 offending functions under the limit by extracting helpers, hooks and subcomponents with no behaviour change.
Shares the RDS subnet and parameter group option builders through rds-form-fields, and rebuilds dist.
* refactor: share the ECR registry address precedence in domains/ecr
* feat: expose every gateway's ECR registry host to the EKS service
* fix: skip wildcard peer hosts when listing ECR registry hosts
* feat: trust every gateway's ECR registry host on EKS workers
* docs: describe how EKS workers trust every gateway's ECR registry host
* docs: split the ECR gateway limits to one sentence per line
* refactor: format the registry URI host in domains/ecr so the daemon avoids the awsapi adapter
* docs: state that a refused gateway fails over at once and record the 3-node ECR measurements
* fix: add GatewayMTU capping the gateway MTU at the internet path MTU
* fix: set gateway_mtu on the VPC gateway router port so oversized packets get ICMP frag-needed
* fix: wire the gateway MTU into vpcd and cover reconcile convergence
* feat: add host MSS clamp tables for the veth and routed uplinks
* fix: shorten MSS clamp comments and log the periodic ensure at debug
* feat: run the host MSS clamp loop in vpcd
* fix: run nft under the ambient CAP_NET_ADMIN instead of sudo
* docs: describe the gateway MTU and the host MSS clamp for the internet path
Bound the zone reload wait with a test context, generate 2048-bit keys in
cert renew tests via the existing seam, replace 100ms negative waits with
deterministic checks, and run tests bound by production timeouts in parallel.
Ubuntu 26.04's openvswitch-switch declares Alias=openvswitch.service, so
enabling it leaves an /etc symlink that distro_ovs_unit read as a second,
EL-style umbrella unit. setup-ovn.sh then refused to pick one and every
ubuntu-26.04 bootstrap failed. A symlink back to openvswitch-switch.service
now counts as the Debian unit; a real or masked openvswitch.service still
does not.
ResourcesPerPage outside 1-100 and a malformed PaginationToken now return
InvalidParameterException. An unbounded page size let start+size overflow
negative and panic on the slice.
The test checked IPsec config and that AES-GCM SAs exist, and only warned when
a 5s idle capture saw no ESP, so a cluster sending plaintext Geneve passed.
It now launches an instance pair on distinct nodes, pings 200 x 1000 bytes
between them, and for each direction fails unless the outbound SA counters to
the peer's ovn-encap-ip rose by the payload, an outbound-only capture on the
interface routing to the peer saw ESP or UDP/4500 ESP, and saw no UDP/6081.
The distinct-node launch loop moves from the baseline cross-host test into a
shared helper. The test is no longer parallel since it now launches instances.
Vendor the route-53 2013-04-01 Smithy model at the pinned commit, add the
Route53 service, and pass route53awsapi.OperationInventory() to the
generator. coverage.json and docs/coverage/route53 now list the 13 served
operations; the Spinifex activation extensions stay out of the model count.
The generator now fails when a served service has no embedded model, so
the next one cannot drop out of the report silently. Request encoding
skips rest-xml by protocol, which covers Route 53 as well as S3.
ensure_clustered_db_storage() collected only databases in standalone
format, so --recreate-db returned early when the NB/SB databases were
already clustered and removed nothing. ovn-ctl then served the stale
database and silently ignored the RAFT flags, which is the failure the
function exists to prevent, reached from the other direction. Because
the old database carries the previous member list and server ID, the
node rejoined the earlier cluster instead of replacing it: re-forming a
production cluster brought the create node back as a follower, and
ovn-nbctl over the leader-only unix socket failed with "database
connection failed ()".
install-node.sh passes --recreate-db on every database node on every
run, so formation already intended a purge. Remove whatever NB/SB
state is present, standalone or clustered, and keep refusing a
standalone database when the flag is absent.
Two further gaps in the same path. systemctl stop returns once the unit
is inactive rather than once the processes are gone, so the purge now
waits for the database servers and escalates to SIGKILL, failing the
run if one survives rather than unlinking files under it. The pattern
is anchored on the OVN database directory so it can never reap the OVS
conf.db server. The lock files, the compute-node move-asides and an
earlier run's standalone backup are cleared alongside the databases.
* feat: add a dedicated SQS message store JetStream server separate from control-plane NATS
* test: cover the SQS store server start, validation and memory limit
HAProxy's short 'option httpchk GET <path>' form sends HTTP/1.0 with no Host header, which Envoy rejects with 426 (and 400 for HTTP/1.1 without Host), so Envoy-fronted targets never became healthy. Match the AWS request shape: GET <path> HTTP/1.1, Host set to the target IP (with the port only when it is not the scheme default), and User-Agent ELB-HealthChecker/2.0.
MemoryObjectStore.DeleteObject silently succeeded for an absent key while
Predastore answers NoSuchKey, so handlers that treated a missing key as fatal
passed unit tests and failed in production. The fake now returns
NoSuchKeyError, and delete paths that must be idempotent (DeleteSnapshot's
sweep and document delete, DeleteVolume, DeregisterImage, the admin image
removal sweep, and ECR upload and GC cleanup) treat it as already deleted.
* feat: let a registration select REST-JSON or JSON 1.0 query-compatible error envelopes
* feat: add a per-registration internal error code and decode envelopes through the AWS SDK v2
* feat: let a JSON 1.0 registration namespace the __type of its errors
* feat: match captured SQS error bytes on the JSON 1.0 envelope
2026-10-27 13:11:43 +10:00
3368 changed files with 225887 additions and 65309 deletions
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
<textx="772"y="394"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#d8b4fe"text-anchor="end">placement group · strategy spread · one server per node</text>
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">SPINIFEX ON VULTR · ONE NODE OR THREE</text>
<textx="876"y="30"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#71717a"text-anchor="end">--nodes 1 or 3 · vx1-g-16c-64g or bare metal</text>
<textx="450.0"y="94"font-family="'Segoe UI', system-ui, sans-serif"font-size="10"fill="#7ee787"text-anchor="middle">customers reach your EC2 and load balancers</text>
<textx="450"y="348"font-family="'Segoe UI', system-ui, sans-serif"font-size="10"fill="#79c0ff"text-anchor="middle">nodes cluster over the private network: control plane, EBS replicas and S3 shards</text>
<textx="80"y="394"font-family="'SF Mono', ui-monospace, monospace"font-size="12"fill="#c084fc"font-weight="700">AWS services across the fleet</text>
<textx="820"y="394"font-family="'Segoe UI', system-ui, sans-serif"font-size="9.5"fill="#d8b4fe"text-anchor="end">spread across the nodes; with three, moved to a survivor if one fails</text>
<textx="450"y="500"font-family="'Segoe UI', system-ui, sans-serif"font-size="10.5"fill="#a1a1aa"text-anchor="middle">Use the AWS CLI, SDKs and Terraform against your own endpoint, unmodified.</text>
<textx="450"y="520"font-family="'Segoe UI', system-ui, sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">Public addresses are Vultr reserved IPs, attached to the node each resource runs on.</text>
<!-- Complete this section only when the PR changes account-scoped resource lifecycle, account termination, resource cleanup, or a lifecycle boundary. Otherwise write: N/A. -->
- Lifecycle owner and resource family:
- Owner-local ordering, recovery, retained-state, or force impact:
- [ ] All community contributions in this pull request are licensed to Mulga under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). By creating this pull request I represent that I have the right to license these contributions to Mulga under the Apache License 2.0.
echo"::error::hypervisor $host has $((root_kb /1024)) MiB free on the filesystem holding /etc/libvirt/qemu (need ${HV_MIN_ROOT_FREE_MB:-1024} MiB): libvirt cannot write a domain definition there. Free space on $host before re-running." >&2
echo"::error::hypervisor $host has $((pool_kb /1024/1024)) GiB free in storage pool $pool at $pool_path (need ${HV_MIN_POOL_FREE_GB:-20} GiB): the node's disk cannot grow. Free space on $host before re-running." >&2
description:"Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
description:"Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
@@ -54,7 +54,7 @@ For Terraform, start from the [example workbooks](https://docs.mulgadc.com/docs/
## Services
The platform serves **392 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
The platform serves **405 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
| Service | What it is | Built on | Status |
| --- | --- | --- | --- |
@@ -68,8 +68,10 @@ The platform serves **392 AWS API operations** across the services below. See ex
| **ECR** | Container registry | Predastore, deduplicated per account | Available |
| **ECS** | Container service | Spinifex instances | Available |
| **RDS** | Databases | PostgreSQL and MariaDB, with automated backups | Available |
| **Route 53** | DNS | [Northstar](https://github.com/mulgadc/northstar): authoritative DNS with hosted zones | Available |
| **Bedrock** | AI deployment | vLLM for text generation and Hugging Face TEI for embeddings and reranking | Q4 2026 |
| **SQS** | Message queues | A dedicated NATS JetStream store in a per-Region QEMU `microvm` guest | Q1 2027 |
Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console dashboard" width="900">
<img src=".github/assets/dashboard.webp" alt="Spinifex web console dashboard" width="900">
</p>
Spinifex ships with a built-in web console for your instances, volumes, buckets, networks, and IAM. See [Launching the Web UI](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui).
{From:"agents/rds/agent",To:"domains/rds/instance",Debt:"agents/rds/agent (dependency, narrowed)",Reason:"command protocol, wire and record types; needs a guest/controller wire contract"},
{From:"cmd/aws-model-coverage",To:"domains/acm/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared ECR/ACM inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/ecr/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared ECR/ACM inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/acm/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/ecr/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/route53/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/elbv2/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/rds/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/ec2/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/ecs/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/eks/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/iam/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/sqs/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/sts/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"runtime/daemon",To:"domains/ec2/awsapi/instance",Debt:"Instance adapter consumers outside EC2",Reason:"runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
@@ -52,9 +58,6 @@ var allowlist = []allowance{
{From:"domains/ec2/awsapi/instance",To:"runtime/compute/cache",Debt:"domains/ec2/awsapi/instance (dependency)",Reason:"node liveness and VM state; needs an EC2-owned projection"},
{From:"domains/ec2/awsapi/instance",To:"runtime/compute/vm",Debt:"domains/ec2/awsapi/instance (dependency)",Reason:"node liveness and VM state; needs an EC2-owned projection"},
{From:"domains/ec2/awsapi/spotinstance",To:"domains/admission/quota",Debt:"domains/ec2/awsapi/spotinstance (dependency)",Reason:"RequestSpotInstances.go uses IAMService and quota.Service"},
{From:"domains/ec2/awsapi/spotinstance",To:"domains/iam",Debt:"domains/ec2/awsapi/spotinstance (dependency)",Reason:"RequestSpotInstances.go uses IAMService and quota.Service"},
{From:"domains/ec2/eip",To:"domains/network/topology",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ec2/guestmetadata",To:"domains/dns",Debt:"Guest metadata: the cross-domain and runtime imports above",Reason:"guest-metadata relocation residual; needs capabilities, projections or contracts"},
@@ -91,42 +94,14 @@ var allowlist = []allowance{
{From:"domains/ochre",To:"domains/ec2/systeminstance",Debt:"Cross-domain",Reason:"consumes another domain's implementation rather than a capability"},
{From:"domains/ochre",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"secret encrypt/decrypt helpers; needs a shared crypto boundary"},
{From:"domains/ochre",To:"domains/network/systemvpc",Debt:"Cross-domain",Reason:"consumes another domain's implementation rather than a capability"},
{From:"domains/ochre",To:"ingress/aws/bodyscope",Debt:"domains/ochre (action adapters)",Reason:"Bedrock action adapters still live in the Ochre package; they move to domains/ochre/awsapi"},
{From:"domains/ochre",To:"runtime/compute/gpu",Debt:"Domain → runtime",Reason:"GPU runtime state read directly; needs a domain-owned projection"},
{From:"domains/ochre/vector",To:"domains/iam",Debt:"Cross-domain",Reason:"consumes another domain's implementation rather than a capability"},
{From:"gateway",To:"domains/ec2/awsapi",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/account",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/capacityreservation",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/eigw",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/eip",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/idem",Debt:"gateway/ec2.go (registration; ec2_idempotency.go)",Reason:"dispatch-time idempotency wrapper; moves with the EC2 dispatch map"},
{From:"gateway",To:"domains/ec2/awsapi/igw",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/image",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/instance",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/key",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/launchtemplate",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/natgw",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/placementgroup",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/routetable",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/snapshot",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/spotinstance",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/tags",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/volume",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/vpc",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/zone",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"domains/rds/awsapi",To:"domains/ec2/awsapi/instance",Debt:"Instance adapter consumers outside EC2",Reason:"enginecatalog.go calls the instance adapter; needs an EC2 capability"},
{From:"domains/sts/awsapi",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"GetCallerIdentity takes IAMService; needs an IAM capability"},
{From:"gateway",To:"domains/eks/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/iam/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/rds/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/sts/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ecs/awsapi",Debt:"gateway (service registration)",Reason:"ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/elbv2/awsapi",Debt:"gateway (service registration)",Reason:"ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
{From:"domains/ecs",To:"domains/ec2/ebs/metadata",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/ecs",To:"domains/ec2/eip",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/ecs",To:"domains/ec2/instancetypes",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
@@ -159,6 +134,4 @@ var allowlist = []allowance{
{From:"domains/rds/instance",To:"runtime/compute/vm",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/sts",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"STS reads IAM service, trust-policy and OIDC-provider state; needs an IAM capability"},
{From:"operator/cli",To:"gateway",Debt:"operator/cli (dependency)",Reason:"command tree imports implementations; needs public capabilities"},
clause:"ADR-0002:S7: the request record is private to ingress, so no domain, runtime or operator package reads it; only packages under ingress/aws may import ingress/aws/reqctx",
hint:"receive what you need through dispatch.Invocation instead of the request record",
a request/reply route on which a gateway, in the `awsgw-quota-seal` queue
group, irreversibly seals one account's quota override record.
`domains/admission/quota` owns it: `awsgw` serves it.
`accountteardown` is the consumer: it calls this subject to fence every
`operator/accountlifecycle` is the consumer: it calls this subject to fence every
override writer at the quota owner before the account record is removed.
The request names the account in its body (`account_id`) and may carry an
@@ -31,10 +31,20 @@ Any current gateway may answer; the leader-accepted revision is the evidence
whichever one does, and the override bucket is opened at seal time even on a
gateway whose quotas were disabled at startup.
## Seal-quota-override capability
`AccountQuotaOverrideSealCapabilitySubject(node)` (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) is a per-node, read-only request/reply route on which `operator/accountlifecycle` asks one configured gateway whether it supports the quota override seal, immediately before every seal attempt.
The quota owner (`domains/admission/quota`) owns it: `awsgw` serves it with no queue group, so a request addressed to one node reaches only subscribers configured as that node, and two gateways sharing a node ID both answer.
The consumer's quota seal reaper probes every configured node that runs `awsgw` and sends no seal request unless every one answers compatible.
`AccountQuotaOverrideSealCapabilityVersion` is a claim about the whole binary's writers, not about the responder: version 1 means every quota override writer in the binary is revision-guarded and refuses a sealed record, and the binary serves the seal subject.
A gateway binary without this responder has no subscriber, so the probe for that node reads as incomplete (`no-reply`), the same as a stopped or partitioned node.
The reply proves nothing beyond the moment it was sent: a gateway downgraded or replaced after a successful probe is outside the guarantee.
### Compatibility boundary
The subject name carries the contract major; `account_quota_seal_test.go`
preserves the version-1 JSON shape.
The subject name carries the contract major; `account_quota_seal_test.go` and `account_quota_seal_capability_test.go`
preserve the version-1 JSON shapes and the subject builder.
Producer and consumer are in the coordinated Spinifex release, so the window
is N and N-1.
A gateway that predates the subject has no subscription, so the request gets
@@ -23,7 +23,7 @@ subject shape are wire compatibility. Callers and receivers must use this
package rather than rebuild those details. `commands_test.go` preserves the
version-1 JSON shape and subject form.
This package owns shared command payloads and the drain-volume response. It
This package owns shared command payloads. It
does not define every action's reply body, authorize the request, store EC2
state or implement NATS transport. Those remain responsibilities of the EC2
domain and its runtime composition.
@@ -37,6 +37,13 @@ contract or an approved compatibility plan; it is not a package refactor.
Only the subject is owned here today.
The request (`account_id`) and reply (`vpc_id`, `error`) bodies are still declared as anonymous structs on each side; moving them into this package is a contract change that needs its own characterization first.
## Volume held elsewhere
A start refused because another node holds one of the instance's volumes is an `IncorrectState` error payload on the `ec2.start`, `ec2.start.<node>` and `ec2.cmd.<instance-id>` replies, and the AWS body cannot say why.
The owning node adds the `X-Volume-Held-Elsewhere: true` header to that reply, the `ec2.start` forwarder relays it, and `StartInstances` wraps `ErrVolumeHeldElsewhere` around the error it rebuilds, so a caller matches it with `errors.Is` rather than reading the message.
The owning node sets the header only from the EBS provider's `volume_held_elsewhere` code, which every provider returns for this refusal; no reader matches the message text.
`VolumeHeldElsewhereText` is the wording viperblockd puts in that refusal's message, not a signal.
## Platform inventory
`PlatformInventorySubject` (`ec2.v1.PlatformInventory`) is a request/reply route on which a daemon in the `spinifex-workers` queue group reports one account's key pairs, placement groups, launch templates and their `ec2-tags` entries from the EC2 owner stores.
This package will own the Lambda process contracts declared by ADR-0009:S13.
It holds the L-GUEST declaration and, in `guest.go`, the connection vocabulary both ends share: the scope, the URI SAN roles, the ALPN identifier and the fw_cfg entry names. The transport and authentication are decided (ADR-0009:S17); the invocation message types and framing are not chosen, so no message types are defined here.
It is not an AWS tenant-facing API.
## L-GUEST: node executor to guest agent
**Owner:** the Lambda domain.
**Producers and consumers:** the node-local executor (`runtime/compute/lambda`) and the in-guest Lambda agent (`agents/lambda`), in both directions.
**Transport:** TCP over the microVM's network (ADR-0009:S17).
The agent dials one connection to a listener the executor owns for the environment, and the guest exposes no listener for host control.
At most one authenticated control connection is active per environment, and the agent may reconnect; how a prior connection and its in-flight work are handled is open (ADR-0009 M1).
The network is shared and untrusted: no source address, port or bridge membership is identity or authority.
Tenant code may be able to reach the listener, but it cannot open the agent's connection or key material, so any connection it makes fails authentication.
**Authentication:** mutual TLS 1.3, bound to the environment ID and realization generation, using material the user runtime cannot read (ADR-0009:S17).
For each generation the executor creates ephemeral self-signed host and guest certificates whose single URI SAN is `spinifex-lambda://<Region>/<account>/<function ID>/<environment ID>/<generation>/<role>`, with role `host` or `guest`.
Each end validates the peer certificate against the one pinned peer certificate and separately requires that URI SAN with the opposite role; hostname verification is not relied on.
ALPN carries the L-GUEST major version, `spinifex-lambda-guest/1`, and an unsupported version fails the handshake.
Under TLS 1.3 the guest's handshake can complete before the host refuses its certificate, so the guest treats the connection as accepted only after the host's first message.
**Bootstrap material:** the executor passes these fw_cfg entries under `opt/spinifex/lambda/`, and the agent reads them as root and unloads `qemu_fw_cfg` before user code starts.
| Entry | Content |
|---|---|
| `guest-key` | Guest private key, PEM |
| `guest-cert` | Guest certificate, PEM |
| `host-cert` | The one pinned host certificate, PEM |
The guest's own interface settings come from `opt/spinifex/netcfg`, the entry other system VMs use.
The agent unloads `qemu_fw_cfg` even when an entry is missing, and refuses to start if the fw_cfg tree is still readable afterwards.
Until the message format exists, the agent authenticates and holds the connection but accepts no host message and starts no user code.
**Authority:** one environment's attempt only.
The user runtime never holds these keys, and sharing the guest with the agent gives user code no platform authority.
**Idempotency and fencing:** an invoke carries the attempt-grant identity and the attempt ID.
The agent accepts a run command only under a current grant, and a completion under a superseded grant is discarded and reported.
How a grant is enforced and proven ended is open (ADR-0009 M1).
**Errors and versions:** the agent and executor fail closed on an unknown protocol version or an identity mismatch.
**Readiness:** environment readiness needs authenticated, current-generation agent evidence; a running hypervisor process does not count.
**Payloads:** the contract must carry a synchronous payload and response of up to 6 MB and an asynchronous payload of up to 1 MB, plus framing, over a path whose size bound is declared.
How payloads move is open (ADR-0009 M3).
### Compatibility boundary
The agent ships inside the runtime image pinned by each published version, so it is outside the coordinated release train.
The executor supports every L-GUEST major carried by a still-supported runtime image.
Retiring an image follows a published runtime-deprecation lifecycle that blocks create and update before it blocks invoke.
Evidence required before the first agent image ships: a function on an old image invoked by a new executor, and the reverse during rollback.
## L-RUNTIME
The guest agent serves the AWS Lambda Runtime API, version `2018-06-01`, to the user runtime at `AWS_LAMBDA_RUNTIME_API`.
Spinifex does not version this AWS contract, and conformance is measured against AWS.
@@ -8,3 +8,8 @@ administration. It is separate from AWS tenant-facing service contracts.
roles and Reed-Solomon configuration. It is an operational status contract,
not a generic provider abstraction; do not change its subject or JSON shape
incompatibly. Add a new version when that is required.
`admin.go` holds the `/admin/<Method>` request and response bodies the private admin surface shares with the CLI.
`AdminMethodNames` is the list of callable methods, and the gateway router builds its accept set from it.
`AccountQuotaRequest`, `AccountQuotaResponse` and `DeleteAccountResponse` stay in the gateway until the `/admin` handlers move, because they embed types the contracts layer may not import.
Do not change a field name, JSON tag or omitempty incompatibly; add a new version when that is required.
@@ -26,7 +26,7 @@ Every service is `PartOf=spinifex.target` and ordered by its own `After=`/`Wants
| `spinifex-qmp-collector.service` | `spx service qmp-collector start` | Reads the `contracts/telemetry/v1` per-VM sidecar, polls telemetry QMP sockets and tap counters, publishes to NATS `metrics.ec2.*` |
| `spinifex-shutdown.service` | `/bin/true`; the work is its `ExecStop` | Runs `spx admin node drain --local --unless-restarting`. `After=` every service, so it stops first and drains guests while the stack is still up |
| `spinifex-nats-watchdog.service` / `.timer` | `nats-js-watchdog.sh` | JetStream liveness probe every 30s. `try-restart`s NATS when JetStream stops accepting writes while the process stays alive — what `Restart=on-failure` cannot catch |
| `spinifex-firewall.service` | `spinifex-firewall-apply` | Oneshot nftables input policy, `WantedBy=sysinit.target` rather than the target. No `ExecStop`: remove it with `nft delete table inet spinifex_filter` |
| `spinifex-firewall.service` | `spinifex-firewall-apply boot` | Oneshot nftables input policy, `WantedBy=sysinit.target` rather than the target. Loads the `inet spinifex_conntrack` tracking table on every node, and the policy only once the daemon has written a peer file. No `ExecStop`: remove the policy with `spinifex-firewall-apply disable` |
| `regenerate-ssh-host-keys.service` | `ssh-keygen -A` | Oneshot for AMI-built nodes, conditional on the host key being absent |
#### Operations
@@ -95,7 +95,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--advertise` (defaults to `--bind`, or the auto-detected WAN IP when `--bind` is 0.0.0.0), `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns`, `--email`, `--no-telemetry`, `--predastore-compaction-interval`, `--join-timeout` (how long to keep retrying while the formation server is unreachable, default 20m), `--force` (join even though this node is already initialized or holds JetStream streams, **discarding its own CA, master key and JetStream store**; the store is removed only after formation succeeds, and the join is refused while NATS is running even with `--force`) | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
### Version
@@ -127,7 +127,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. Exits non-zero if any asset is Missing, Stale or Ungranted. |
| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. On x86_64 it also checks that the QEMU binary the daemon will run lists the `microvm` machine type and that the microvm firmware `/usr/share/spinifex/microvm/qboot.rom` is installed, since load balancers boot on both. It runs the host network check `spinifex-daemon` and `spinifex-vpcd` run before they start, and fails while firewalld or ufw is active, since the Spinifex nftables policy is the only supported node firewall. It reports a Warning, without failing, when systemd-resolved is active but `/etc/resolv.conf` is not its stub, since lookups then bypass the route that sends Spinifex names to northstar. Exits non-zero if any asset is Missing, Stale, Ungranted, Unsupported or Failed. |
| `spx admin node js-probe` | `--timeout` (default: 10s) | Round-trips a canary key through the cluster-state KV bucket and exits non-zero if the write or the read-back fails. JetStream can latch a write error per message block and keep serving reads — nats-server's filestore records the failure on the block and returns it on every later write, and nothing clears it short of a restart. The HTTP health endpoint reports the server-wide JetStream state, which that latch never touches, so only an actual write observes it. |
| `spx admin banner` | `--boot-check` | Writes the node information banner to `/etc/issue` (shown before login on the physical/serial console) and `/etc/motd` (shown after SSH login). With `--boot-check` it also detects a management IP that has changed since last boot and updates `/etc/spinifex/node.conf` accordingly. |
@@ -135,8 +135,8 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. |
| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is allowed, and is preferred over leaving the instance unable to run, but it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. It also lists UEFI variable stores nothing owns: a `uefi/i-*/` prefix with no instance record and no store record, a `uefi/ami-*/` prefix with no AMI document, and a store record that is not `deleting` and has no instance record. These are reported and never deleted, because an absent instance record is not proof of a terminate. |
| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is refused while that node is alive (its daemon heartbeat is under 60 seconds old, or cannot be read) and is otherwise allowed, in which case it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
| `spx admin storage status` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Sends the signed `GetStorageStatus` action to the AWS gateway with credentials from the standard AWS chain (e.g. `AWS_PROFILE=spinifex`) and prints what the console's Service Metrics page shows: predastore encoding, meta and blob node counts, cluster health, a meta node table (health, raft state, leader, term, commit and applied index) and the blob nodes. Cluster health is `Degraded` when any meta node is unreachable or the nodes reporting raft state do not agree on one leader. A meta node read only through its host's `/readyz` shows healthy with blank raft fields. |
### Upgrade Management
@@ -152,12 +152,12 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| `spx admin account create` | `--name` (required) | Connects to NATS → refuses if an account already holds the name (case- and whitespace-insensitive) → reserves the name in `spinifex-account-names` KV → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates default VPC → creates `admin` user, AdministratorAccess policy (Action:*, Resource:*) and one access key → prints credentials and writes a `spinifex-<name>` AWS CLI profile |
| `spx admin account create --remote` | `--name` (required), `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA), `--client-token` (default: generated), `--source` (default: `spx-cli`) | Same account, created through `POST /admin/CreateAccount` over SigV4 instead of NATS — the path the public signup form uses, so a shell can exercise it. Credentials come from the standard AWS chain (env vars or `AWS_PROFILE`), never from the cluster master key; the caller must be an IAM user in the super-admin account holding `spinifex:CreateAccount`. Reuse the printed `--client-token` to retry: a fresh token would create a second account. |
| `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |
| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS credential chain: on a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command fails to find credentials), or equivalent credentials; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
| `spx admin account quota get <account-id>` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Reads the limits in force for one account through `POST /admin/GetAccountQuota` over SigV4, and reports for each dimension whether it came from the `[quota]` block in `awsgw.toml` (`config`) or from an override stored against the account (`override`). Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:GetAccountQuota`. |
| `spx admin account quota set <account-id>` | `--vcpus`, `--vpcs`, `--subnets`, `--eips`, `--volumes`, `--volumes-gib`, `--rds-instances`, `--load-balancers`, `--clear`, plus `--endpoint`, `--region`, `--ca-bundle` | Overrides individual limits for one account through `POST /admin/PutAccountQuota`, leaving every dimension not named inheriting the configured baseline. `-1` uncaps a dimension; `0` is a real limit that denies every request on it. `--clear` removes the whole override record and cannot be combined with a limit flag. The account must already exist. Prints the resulting limits, so a caller never has to guess what it applied. The caller must hold `spinifex:PutAccountQuota`. |
| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record) | Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record), `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Always runs through `POST /admin/DeleteAccount` over SigV4 and follows `DescribeAccountDeletion` to completion; the CLI never tears an account down in its own process, so a CLI delete and an API delete cannot race. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → seals the account's quota override record so no later write can recreate it, which needs every gateway node the cluster configures to answer a read-only probe first (otherwise nothing is sealed and the account stays `TERMINATING`) → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
| `spx admin account delete --force` | as above | Escalates only for resources the ordinary API refuses: it clears a volume attachment in the control plane without the guest's cooperation (`ec2.ForceDetachVolume`), hard-destroys after a graceful attempt times out, and treats an already-missing resource as deleted. It never reorders the stages and never widens scope beyond the account. Use it once the guest is being destroyed — this is the escape from a volume stuck attached to an instance that will not terminate, where neither can otherwise be deleted. |
| `spx admin account delete --remote` | as above, plus `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Same teardown driven through `POST /admin/DeleteAccount` over SigV4 instead of NATS, then follows `DescribeAccountDeletion` to completion, printing each stage as it lands. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. |
| `spx admin account delete --remote` | as above | Accepted for existing scripts and ignored: every delete already goes through the admin API. |
| `spx admin principal create <name>` | `--grant` (repeatable or comma-separated; default: every admin method — `CreateAccount`, `DeleteAccount`, `DescribeAccountDeletion`, `GetAccountQuota`, `ListAccounts`, `PutAccountQuota`) | Creates an IAM user in the super-admin account with an inline `spx-admin-methods` policy allowing the requested methods, revokes any access key left from a previous run, and mints one new key. Each method is named rather than granted as `spinifex:*`, so a later addition to the admin surface is not authorised by an existing key. The principal's grants are exactly that one policy: re-running with fewer `--grant` values narrows it, and any other inline policy on the user is removed, so a rename cannot leave a wider grant behind. The secret is printed once; store it in a secret manager or an AWS profile, never in a config file. `admin` is refused as a name — it is the account's own administrator and rotating its key would revoke the bootstrap credential. |
| `spx admin principal list` | — | Lists every principal in the super-admin account with its access-key count and what it may call. An attached policy is reported by name (`AdministratorAccess (attached)`) rather than expanded, which is how an unscoped principal is spotted. |
| `spx admin principal revoke <name>` | — | Deletes every access key the named principal holds, leaving the user and its policy in place. Immediate and cluster-wide; re-issue with `principal create`. |
@@ -181,7 +181,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| `spx admin gpu status` | `--node` (default: local node) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. |
| `spx admin gpu enable` | — | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. |
| `spx admin gpu disable` | — | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. |
| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. |
| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Resolves its boot tools first (`update-grub` or `grubby`, `update-initramfs` or `dracut`) and stops before any change if either pair has neither or both. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → if IOMMU is inactive, add `intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt` to `GRUB_CMDLINE_LINUX_DEFAULT` and run `update-grub`, or run `grubby --update-kernel=ALL --args=...` and check the default entry and `/etc/kernel/cmdline` carry them → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to `/etc/initramfs-tools/modules`, or write `/etc/dracut.conf.d/spinifex-vfio.conf` (`force_drivers` for the vfio modules, `install_items` for the `modprobe.d` files). If any change requires a reboot: runs `update-initramfs -u`, or `dracut -f --regenerate-all` and checks every kernel's image with `lsinitrd`, then exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies every PCI function in each GPU's slot is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound, fails naming any function on another driver) → calls `gpu enable` to activate passthrough. |
| `spx admin gpu mig status` | — (must be run on target host) | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
| `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
| `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
@@ -192,6 +192,35 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
|---------|-------|-------------|
| `spx admin eks restore-snapshot` | `--cluster` (required), `--snapshot` (optional, defaults to the latest snapshot in predastore), `--account` (optional, defaults to the bootstrap account) | Single-CP total-loss DR path (fail-safe): validates the snapshot exists in predastore BEFORE any mutation (a typo'd/missing key hard-fails, never resets into an empty datastore) → launches a fresh control-plane VM as a cluster-init seed (replaying the persisted create-time launch template) → sets a required-snapshot `RecoveryDirective` (`cluster-reset`) so the boot-time recovery agent aborts rather than resets-into-empty if it cannot fetch the snapshot → persists the replacement in cluster meta BEFORE re-pointing the NLB (so an NLB failure is convergeable by the reconciler, returned as a provisional status, not a hard error) → re-points the cluster NLB's apiserver and konnectivity target groups from the old CP's ENI to the new one → fences the old CP with retries, failing loudly if it cannot be confirmed terminated (split-brain guard). Any failure before the meta commit unwinds the fresh CP (terminate + clear directive) so a re-run does not stack a second resetting control plane. The returned status is provisional — success means the sequence completed, not that etcd is restored and serving; verify cluster health. HA clusters (a spread with a potentially surviving quorum) are rejected — recover those via quorum reformation instead. |
### SQS System Instances
These commands enable SQS on a single-node cluster and start its one system VM.
A bound VM is not a ready service: `spx admin sqs status` reports readiness.
The SQS VM runs two processes on one encrypted data volume: the Region's dedicated SQS NATS server, which listens only on the guest's loopback address and is separate from platform NATS, and the worker, which serves SQS owner calls from that local store. The worker starts serving only after the local store answers and the receipt-signing key on the volume matches the fingerprint the platform recorded.
SQS supports only a fresh single-node install. A worker record without a supported launch contract, or any record of the separate SQS store VM from an earlier dev build, cannot be upgraded: `enable` refuses it, and any existing volume is kept, not inspected, emptied or replaced. An existing separate store VM is refused on relaunch with its volume kept, and an existing worker VM is not restarted into a supported mode; upgrading such an install is unsupported.
**Do not put tenant production data in SQS.** SQS-managed per-message encryption is not implemented: `GetQueueAttributes` reports `SqsManagedSseEnabled=false`, and a request to enable SSE-SQS or SSE-KMS is rejected.
The SQS data volume must be encrypted, which is a separate layer under every queue and not SSE.
There is no `DeleteQueue` or `PurgeQueue`, so a queue and its unexpired messages cannot be removed once created.
Retention is checked when a message is received, against the queue's retention at that moment, and nothing deletes an expired message: the store drops a body only at its age backstop of 14 days and 1 hour.
So a message past a short retention can be delivered again if `MessageRetentionPeriod` is later raised while its body is still stored.
Account termination does not yet remove SQS queues or messages, and there is no disable: the VM's volume, receipt key and data are kept.
Each sends a signed `spinifex` admin action (`EnableSQS`, `DescribeSQS`, `EnsureSQSWorker`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VM, volume, private VPC and ENI belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. `ensure-worker` and `enable` wait up to 11 minutes, and `status` up to 1 minute.
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin sqs enable` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Records SQS as enabled for the Region, pinned to `--node`, then launches or reports the SQS VM there. The desired state is recorded first, so rerunning the command resumes an interrupted enable. More than one node in the cluster is refused before anything is recorded. A worker record with a missing or unsupported launch contract, or a record of the separate store VM, is also refused before anything is recorded or launched; any existing volume is kept, and upgrading it is not supported. |
| `spx admin sqs status` | `--endpoint`, `--region`, `--ca-bundle` | Shows the requested state, node, generation, VM state, receipt key state, whether a worker is polling this gateway, whether this gateway serves the SQS API, and `Ready`, which is true only when all of them hold. `Unsupported` names a record whose missing or unsupported launch contract blocks enable. |
| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The VM is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted system-account data volume of `--volume-size-gib` that holds the queue data and the receipt-signing key the guest creates on first boot. The platform records only the key's fingerprint. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. A different `--volume-size-gib` than the recorded one is `IncorrectState`. |
`ServiceUnavailable` never proves that no SQS resources exist, because an earlier attempt may already have created some. Its message says what it does establish. When no node serves the subject, this request was not delivered. When a volume create was not sent, that particular create did not happen. When the node cannot boot the guest yet or its launcher lacks a capability, this request stopped before allocating or recording anything. When the gateway times out waiting for the node's reply, nothing is known: the launch may still be running, so never assume it created nothing. In every case, call the command again with the same arguments. A retry first checks the durable record of the attempt and continues only when that is safe; it can proceed to the first launch after a create that was proven unsent, and otherwise reports the recorded outcome as bound or `IncorrectState`, or returns `ServiceUnavailable` again. `IncorrectState` from a volume, network interface or launch step means that step was recorded as sent but its outcome was never observed. It is never resent, the message names the volume when its ID is known, and no tool reconciles it yet: recovery is manual.
`bound` says only that the VM was launched for the current record. The worker is ready only when the guest's local store answers, and it serves only while its receipt key matches the recorded fingerprint; a missing or different key once one is recorded refuses serving and is never replaced.
Real VPC routing, IMDS and gateway TLS from the guest have not been tested on a live host.
## Private Admin API
Not an AWS API: a Spinifex-internal surface on the AWS gateway, JSON in and JSON out, one method per path segment at `POST /admin/<Method>`. It exists so a self-service signup form, a provisioning system or a load-test harness can create and remove tenant accounts without an operator shell on a node.
3. **Service Detection**: Extracts service name from the Authorization header
4. **Action Dispatch**: Routes to service-specific handler. A service registered through the ingress dispatch registry (`ingress/aws/dispatch`, wired by `runtime/roles/awsgw`) is dispatched by its registration; currently those are the ECR and ACM control planes, registered from `domains/ecr/awsapi` and `domains/acm/awsapi`. Every other service uses the legacy switch:
4. **Service Detection**: Extracts service name from the Authorization header
5. **Action Dispatch**: Every service is a registration (`ingress/aws/dispatch`) built by its domain's `awsapi.NewRegistration` and wired by `runtime/roles/awsgw`. The pipeline dispatches by registered service name and holds no per-service code. The private `/admin/<Method>` surface dispatches to the registration `awsgw` names as `Admin.Service`.
### 3. EC2 Handler
The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and delegates to specific handlers via the generic `ec2Handler` and `ec2HandlerWithReq` wrappers. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal:
The EC2 registration (`spinifex/domains/ec2/awsapi`, `NewRegistration`) parses the `Action` parameter and looks the handler up in its action table. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal. Each action's handler is built from the owning EC2 resource package (`domains/ec2/instance`, `volume`, `vpc` and so on), with PassRole, quota and client-token idempotency wrapped on at the action table rather than inside the resource.
### 4. NATS Messaging
The gateway communicates with daemons via NATS request/response. Most calls go through `natsmsg.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
// QEMU launch with NBD-backed drives in args (runtime/compute/vm/vm.go)
cmd := exec.Command(qemuArchitecture, args...) // qemu-system-x86_64 or qemu-system-aarch64
@@ -181,12 +147,17 @@ type Daemon struct {
Tracks available and allocated CPU/memory to prevent overcommit, and drives the dynamic `ec2.RunInstances.{type}` subscriptions:
One instance vCPU is one CPU the kernel has online: a thread with SMT on, a core with SMT off or on ARM, as AWS counts a vCPU.
The OS and Spinifex's own services keep 1 CPU on a host with up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and every other CPU is schedulable one-to-one with no overcommit.
[ADR-0015](https://github.com/mulgadc/mulga/blob/main/docs/adr/0015-host-cpu-capacity-and-reservation.md) fixes this rule.
```go
type ResourceManager struct {
mu sync.RWMutex
hostVCPU int // raw runtime.NumCPU
hostVCPU int // online CPUs (ADR-0015:S1)
hostMemGB float64 // raw /proc/meminfo MemTotal
reservedVCPU int // held back for spinifex services
reservedVCPU int // ADR-0015:S2 tier: 1, 2, 4 or 8
reservedMem float64 // held back for spinifex services
allocatedVCPU int
allocatedMem float64
@@ -229,6 +200,55 @@ type EBSRequest struct {
Viperblock responds with an NBD URI that QEMU uses to access the block device.
#### Direct-boot microVM data volumes
A direct-boot microVM keeps the launcher's saved `vm.Config` as a clean base, and `startQEMU` adds data volumes only to a launch-only copy of it (`runtime/compute/vm/lifecycle.go`).
At each start the daemon takes one snapshot of the instance's already-mounted data requests, probes each NBD or host block export from that snapshot, and builds the command from the same snapshot, so the launched endpoint is the one that was checked and is never written back into the saved base.
The microVM boots with `pcie=on`, so its NICs are `virtio-net-pci` devices with MSI-X interrupts instead of `virtio-net-device` MMIO devices.
Each data volume becomes a `virtio-blk-pci` device on the default PCIe bus with the same id, node name, I/O thread, serial and `werror=stop,rerror=stop` policy as the q35 device, and takes no PCIe hot-plug port.
A microVM started without `pcie=on` keeps the `virtio-*-device` MMIO transports.
Boot and EFI requests are rejected on this path, and a direct-boot instance with no data requests launches its base unchanged.
Disk-boot (q35) instances keep the existing PCI hot-plug path.
Hot-attaching a volume to a running direct-boot microVM is not supported, because the microVM has no PCIe hot-plug root port to plug into.
`AttachVolume` refuses it before mounting the volume or changing any instance or volume state, and the EC2 API returns `UnsupportedOperation`.
This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
#### Volume leases
Before opening an engine on a volume, the Viperblock worker takes a lease: one entry per volume in the `VIPERBLOCK_VOLUME_LEASES` JetStream KV bucket, which expires 45 seconds after its last write (`providers/ebs/viperblock/volume_lease.go`).
A renewal becomes eligible once the last confirmed write is 10 seconds old, and the holder checks for that periodically, so the rewrite lands some time after that point.
Each rewrite is conditioned on the revision the holder last wrote, and a refused write that finds the entry gone or naming another owner or generation fences the export as a takeover.
It may keep writing for 40 seconds after the last write the store acknowledged, measured from when that write was sent, because the server starts the entry's expiry when it applies the write and never earlier.
A slow acknowledgement therefore shortens the remaining validity rather than extending it, and an acquisition acknowledged after its validity has already run out is refused as retryable.
For a holder that cannot renew, a timer is armed for the validity deadline, leaving a nominal 5 seconds before the entry can expire; when it fires, the export's nbdkit is fenced with SIGKILL (`volume_fence.go`).
The fence runs on its own goroutine, so Go scheduling, that goroutine and process termination all delay the kill, and enough delay can use up the whole margin; a kill held in uninterruptible sleep, which the fence waits up to 30 seconds for, is the extreme case.
Once a lease is lost or has lapsed it stays lost: a renewal that succeeds afterwards is ignored, and another open on the same node cannot share it while the fence runs.
These rules bound only the holder's own clock.
The storage backend does not reject a stale writer, so a paused or partitioned host that resumes after its validity can still write; generation enforcement at the write boundary is required by Q-24 in the umbrella repository's `docs/specs/viperblock/rfc3-1_overview.md` and is not implemented.
An export that survives a daemon restart writes without a renewed lease until recovery adopts it, and a survivor that recovery cannot adopt while a guest still uses it is left running without a lease or a fence.
#### Unsealed writes and start elsewhere (interim)
A volume is unsealed while its node holds writes the backend has not confirmed; the node records this in a dirty marker naming itself as owner (`providers/ebs/viperblock/volume_dirty.go`).
Opening such a volume on another node loses those writes, so two guards stop it while the previous owner may still hold them.
This is an interim guard until the instance lifecycle work defines ownership, and it is not a fence.
When `ec2.start` is forwarded to the node that last ran a stopped instance and that node refuses for capacity, the forwarding node normally starts the instance locally.
If the refusing node still holds unsealed writes for any of the instance's volumes, it sets the `X-Unsealed-Writes` NATS reply header and the forwarder relays `InsufficientInstanceCapacity` instead of starting elsewhere.
The check asks the owner's viperblockd over the per-node `unsealed` verb with a 5 second bound, and a check that fails or times out counts as holding, so the refusal is kept.
Independently, a node that finds another node's dirty marker on a volume refuses to open it when that node's daemon heartbeat (`heartbeat.<node>` in the cluster-state bucket) is under 60 seconds old.
It also refuses when the heartbeat cannot be read or no heartbeat source is wired, and the refusal maps to the held-elsewhere error that recovery retries.
A previous owner that has stopped heartbeating is opened over with a warning, as before.
The heartbeat age is compared across node clocks, so the 60 second window is also the clock-skew allowance.
EKS control-plane recovery treats both refusals, capacity and held-elsewhere, as retryable: it keeps retrying the stopped member and does not count them against its restart attempts.
### Predastore (S3)
Object storage used for:
@@ -256,6 +276,60 @@ file. It does not connect to Predastore to do so, and no Predastore node
exposes a status endpoint. This is an operator status interface, not a generic
storage-provider contract.
### SQS message store (internal)
`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
It is used only by the SQS worker inside the SQS system instance and does not itself deliver an SQS API.
Spinifex ships no host service, unit, user or firewall rule for the dedicated JetStream server; the system instance that runs it is described under *SQS system instance (operator-launched)*.
The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
The adapter provisions a fixed set of shared message streams (`SQS_M_<shard>`, one subject per queue) and shared state buckets (`sqs_state_<shard>`), keeps historical ones, and applies a 14-day-plus-one-hour retention backstop to both.
It supports one node (R1) and three nodes (R3) only and refuses any other declared cluster size before provisioning.
Reads that must observe a completed write or delete go to the stream leader: `GetState`, the state-key pass behind `Count` and `Namespaces`, and the body pass behind `Namespaces` and `Scan`.
This avoids the follower-served direct gets and follower-placed consumers that returned purged data in local three-node fault-injection tests; it is not a general linearizability guarantee.
Each leader read is one request per record, and the body pass transfers each full body internally even though `Scan` returns metadata only, which is a material cost for a large backlog.
A state-key pass is bounded but is not a snapshot when a key is overwritten during it.
The feed reads through an in-memory, headers-only ordered consumer.
If that consumer delivers nothing for about two seconds while the stream holds a change after the feed's cursor, the feed replaces it from the cursor, so a stored change does not wait on a stalled consumer; the cause of those stalls is not established.
`Close` is terminal, waits for consumer opens and deletes in progress and retries failed deletes, but cleanup is best effort for a create whose outcome is unknown: a consumer that lands after its last delete is removed by the server's one-minute inactive threshold.
Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
### SQS system instance (operator-launched)
One system-owned microVM per Region runs both the SQS store and the SQS worker as separate processes, on one encrypted data volume pinned to one node (ADR-0011 Amendment 5, ADR-0012 Amendment 6).
Only an operator activates it with the super-admin `EnsureSQSWorker` or `EnableSQS` actions; nothing starts it at daemon startup, and platform NATS is neither used nor changed for SQS data.
This first profile is fresh single-node only: a host cluster of more than one node is refused before anything is recorded or allocated, and there is no disable.
`make build-sqs-worker-microvm-image` builds the one `sqs-worker` image: the static `sqs-worker` binary, the pinned `nats-server`, the ext4 tools and a network `/init`; releases ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker`.
At boot the guest reads three firmware config values under `opt/spinifex/sqs-worker/` (the data disk's serial, its Region and the gateway URL on this node's mgmt bridge) and refuses before touching any disk if one is missing or malformed.
It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and confirms from `/proc/self/mountinfo` that the whole disk is what is mounted; the guest never formats.
The store is a dedicated `nats-server` listening only on `127.0.0.1`, with its JetStream directory at `/data/jetstream` and `GOMEMLIMIT=512MiB` (ADR-0011:S11), so nothing outside the VM can reach it and it needs no network identity of its own.
The store starts only after the receipt key is settled, and the worker connects only after the store answers JetStream account info, so an unavailable store never appears ready.
The receipt-signing key lives on the data disk under `/data/sqs-worker-key`, written to a temporary file, synced, renamed and its directory synced before its fingerprint is recorded.
`domains/sqs/workerident` holds the platform's receipt-key state on the worker's identity record, beside its generation, claim, pinned node and volume: `pending` is set only when the record first creates its volume, and `recorded` holds the fingerprint.
Under `pending` the guest may create a key, or reuse a valid key it already wrote before a crash, only while no `/data/jetstream` directory exists; any store data under `pending` refuses, because it proves the volume is not new.
Under `recorded` a missing, unreadable or different key refuses the boot and is never regenerated; a record with no receipt-key state is refused as not proven new.
`RecordReceiptKey` writes the fingerprint by compare-and-swap that re-checks the current instance, generation, claim and volume, and a lost reply is answered by reading the record again, never by a new key.
A replacement worker reuses the pinned volume and keeps its predecessor's receipt-key state.
The worker reaches the gateway over TLS 1.3 trusting only the cluster CA, signing with instance-role credentials from the metadata service only.
The internal `spinifex-sqs` actions are `ClaimReceiptKey`, `RecordReceiptKey`, `PollWork` and `CompleteWork`; the gateway admits only an assumed-role session of `sqsWorkerRole` in the system account and resolves the instance through `workerident.Authorize`, a point-in-time check against the current bound record.
The worker pulls tenant work from its own node's gateway relay and posts results back, so no SQS payload crosses platform NATS; the relay holds a bounded, in-memory set of waiting requests and answers `ServiceUnavailable` when it is full or no worker is polling.
`EnsureSQSWorker` takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB`; the instance type's memory must exceed the store's 512 MiB limit, and a later request for a different type or size is refused as `IncorrectState`.
`EnableSQS` records the Region's desired state, pinned to one node, and then ensures the worker; `status` reports it ready only when the worker is bound, polling and the tenant API is registered.
A store launch record or VM from the earlier separate-store design, or a worker record without a supported launch contract, is refused with its volume kept: enable stops before recording anything, and a relaunch of an old store VM is marked failed rather than started.
Evidence is local: unit tests, the guest shell test and an AWS SDK test that creates a queue and sends, receives and deletes a message through the real worker loop against a real local JetStream store and a restart.
A real guest boot, its metadata route on a real ENI and the signed path on a live node are unproven, Viperblock's at-rest key is held by the platform, and SQS-managed message encryption is not implemented, so the profile must not hold tenant production data.
| `cmd/aws-model-coverage` | Qualification generator | Thin main; reusable code in `internal/tooling` | Produces the dispatch-versus-model inventory; never imported by product packages. |
| `cmd/aws-model-coverage` | Qualification generator | Thin main; reusable code in `internal/tooling` | Produces the dispatch-versus-model inventory; never imported by product packages. Debt: it reads each domain's declared `OperationInventory` from `domains/<service>/awsapi` (ACM, EC2, ECR, ECS, EKS, ELBv2, IAM, RDS, Route 53, SQS, STS) and never constructs a registration. |
| `cmd/dhcptest` | Development diagnostic | Thin main; reusable code in `internal/tooling` if extraction is useful | It is a DHCP probe, not a supported `spx admin` command. |
| `cmd/spx-loadgen` | Qualification/load tool | Thin main; reusable code in `internal/testkit` | It drives test load and is not shipped to nodes. |
| `tests/e2e/*/cmd/*`, `tests/bench/*/cmd/*` | Test and benchmark-private tools | Remain test-private | They are invoked by a test or benchmark harness, not a product process. |
@@ -351,8 +351,27 @@ while the branch is in flight; they do not describe the umbrella repository's
| `spinifex#1269` | `spinifex/handlers/elbv2` | `spinifex/domains/elbv2` (package `elbv2`) | A mechanical whole-package relocation under the 2026-10-26 ADR-0001 amendment: the ELBv2 control plane keeps its files, tests and current dependencies, and importers alias it `elbv2domain`. No identifier, API, NATS subject, KV bucket, metric, error string or behaviour changes. Its allowlist entries are now sourced from `domains/elbv2`. Q-30's data-plane boundary stays deferred, recorded as `agents/elbv2` (Q-30 data-plane boundary) debt. |
| `spinifex#1271` | `spinifex/handlers/ecs` | `spinifex/domains/ecs` (package `ecs`) | A mechanical whole-package relocation: ADR-0001's `handlers/ecs` row targets `domains/ecs`, the files are already organised by resource, and resource packages are introduced only where callers need a compiler boundary. The ECS control plane keeps its files, tests and current dependencies alongside the already-extracted `domains/ecs/taskdefinition`, and importers alias it `ecsdomain`. No identifier, API, NATS subject, KV bucket, metric, error string or behaviour changes. Its allowlist entries are now sourced from `domains/ecs`. `handlers` has no package left and is a retired legacy root; R5 holds `domains/ecs` to the interim rule's ban on importing the daemon. |
| `spinifex#1273` | `spinifex/gateway/{ecs,elbv2}` | `spinifex/domains/{ecs,elbv2}/awsapi` (`awsapi`, imported as `ecsawsapi`, `elbv2awsapi`) | A mechanical relocation of the ECS and ELBv2 AWS action adapters to their ADR-0001 domain homes, as `spinifex#1265` did for IAM, STS, EKS and RDS; APIs, NATS subjects, KV buckets and behaviour are unchanged. The root `gateway` dispatch still imports each package, recorded as `gateway` (service registration) debt, and `domains/admission/quota` keeps its live ELBv2 counter import, now of `domains/elbv2/awsapi`. |
| `spinifex#1336` | `spinifex/gateway/tagging.go` (`Tagging_Request`, `taggingActions`, `taggingActionFromTarget`) and the `GatewayConfig.TaggingResources` field | `spinifex/domains/tagging/awsapi` (`NewRegistration`, `Deps`, `Actions`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | Resource Groups Tagging registers through the ADR-0002 seam like ACM and Route 53. The check order is kept: action, scope, authorization, account, body, aggregator, so a denied caller still gets `AccessDenied` before any body error and an unwired aggregator answers `ServerInternal`. The `checkPolicyResources` stand-in of `*` for every action is unchanged. The legacy switch case, the `supportedServices` entry and the `jsonErrorService` case are removed, and the registration declares the JSON 1.1 envelope. No allowlist row existed for this path. The nil-NATS check moves to the gateway-wide gate, which answers `ServiceUnavailableException` before dispatch as it did for every service. `cmd/aws-model-coverage` output is byte-identical because there is no embedded Tagging model. The characterization tests that pin the pipeline behaviour were committed first and pass unchanged against the registration; only the `withTagging` test helper changed. |
| `spinifex#1336` | `spinifex/gateway/elbv2.go` (`ELBv2_Request`, `elbv2Actions`, `elbv2Handler`, `accountLoadBalancerLimit`) | `spinifex/domains/elbv2/awsapi` (`NewRegistration`, `Deps`, `Quota`, `ActionNames`, `OperationInventory`), `spinifex/domains/admission/quota` (`Service.LoadBalancerLimit`), `spinifex/ingress/aws/query` (`ParseArgs`), composed by `spinifex/runtime/roles/awsgw` | ELBv2 registers through the ADR-0002 seam as the first AWS Query service. The check order is kept: query parse, action, account, input parse, scope, authorization, NATS, dispatch, so a denied caller still gets `AccessDenied` before the handler runs. The gateway members ELBv2 read are now supplied as `Deps`: `NATSConn` as `Deps.NATS`, `Quota` as the `Deps.Quota` interface declared in `awsapi` because `domains/admission/quota` imports `awsapi`, `Region` as `Invocation.Region`, and the policy gate as `Invocation.Authorize`. The `DescribeAccountLimits` cap resolution moves to `Service.LoadBalancerLimit` unchanged, and a nil `Deps.Quota` enforces nothing and reports the AWS default. AWSGW now wires the registry after it builds the quota service, because the registration captures it. The registration declares the XML envelope and the inventory name `elasticloadbalancingv2`, so `cmd/aws-model-coverage` keys it explicitly and its output is byte-identical. `gateway.ParseAWSQueryArgs` delegates to `query.ParseArgs`. The legacy switch case, the `supportedServices` entry, the inventory entry and the `gateway -> domains/elbv2/awsapi` allowlist row are removed. `cmd/aws-model-coverage` imports `domains/elbv2/awsapi` for the inventory, under a fourth named-exception allowlist row whose reason is now generic for all four. Finding, not fixed here: the XML for `DescribeAccountLimits` and `DescribeLoadBalancers` lists member fields in a different order from run to run because the marshal is map-ordered, so the characterization tests pin decoded content rather than bytes. The characterization tests that pin the pipeline behaviour were committed first and pass unchanged against the registration; only the `withELBv2` and `serveELBv2` test helpers and the setup of two older gateway tests changed. |
| `spinifex#1337` | `spinifex/gateway/spinifex.go` (`Spinifex_Request`, `spinifexAdminActions`) | `spinifex/operator/admin` (`NewRegistration`, `Deps`, `ModelAccess`, `ActionNames`), composed by `spinifex/runtime/roles/awsgw` | The `spinifex` Query service registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, authorization with the wildcard resource, account, the super-admin account gate, then the action, so a denied caller learns nothing about which actions exist. The gateway members the actions read are now `Deps`: `Version` and `Commit` strings, `NATSConn` as `Deps.NATS`, `RootCAs`, `DiscoverActiveNodes` as `Deps.ActiveNodes`, `BedrockAccessAdmin` as the `ModelAccess` interface, and the credential and access resolvers as `Deps.Credentials` and `Deps.Access`. The query actions need only the caller account, so the actor is not widened. The cluster-unavailable answer stays in the shared request path ahead of authorization. `/admin/{method}` moved onto the same registration in `mulga-3nbla.7`, recorded below. |
| `spinifex#1344` | `spinifex/gateway/rds.go` (`RDS_Request`, `rdsCaller`) | `spinifex/domains/rds/awsapi` (`NewRegistration`, `Deps`, `Quota`, `CallerFromActor`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | RDS registers through the ADR-0002 seam with the XML envelope it had. The check order is kept: query parse, action, account, the principal-class gate for the internal agent actions, scope, authorization, NATS, quota and dispatch, so a denied caller still gets `AccessDenied` before the handler runs and no customer grant is evaluated against an agent action. The gateway members RDS read are now supplied as `Deps`: `NATSConn` as `Deps.NATS`, `ExpectedNodes` as `Deps.ExpectedNodes`, `Quota.EnforceRDSInstances` as the `Deps.Quota` interface, `Region` as `Invocation.Region` and the policy gate as `Invocation.Authorize`. A nil `Deps.Quota` skips the DB-instance cap. The caller comes from `dispatch.Actor`: the session name for an assumed-role session and the user name otherwise, and the role name from `Actor.RoleName()`, which parses the underlying role ARN and never the session name. The account comes from `Invocation.AccountID`, so a caller with no composable ARN still reaches the policy check and is refused there as before. The legacy switch case, the `supportedServices` entry, the inventory entry and the `gateway -> domains/rds/awsapi` allowlist row are removed, and `cmd/aws-model-coverage` reads the inventory under a fifth named-exception allowlist row. The characterization tests that pin the pipeline behaviour were committed first with spinifex#1336 and pass unchanged against the registration; only the `withRDS` and `serveRDS` test helpers and the setup of the direct-call RDS tests changed. |
| `#1277` | `spinifex/accountteardown` | `spinifex/operator/accountlifecycle` (package `accountlifecycle`, imported as `accountteardown`) | A mechanical whole-package relocation under the 2026-10-27 ADR-0001 amendment: the account deletion coordinator keeps its APIs, NATS subjects, KV access and behaviour; importers keep the `accountteardown` name as an alias, so only their import paths change. Replacing its direct domain and adapter cleanup calls with owner capabilities stays deferred debt. |
| `#1285` | `spinifex/gateway/requestaudit.go` and `slogRequestLogger` from `spinifex/gateway/gateway.go` | `spinifex/ingress/aws/audit` (package `audit`) | The first generic-ingress extraction from the root gateway: the request-audit middleware and access-log middleware move unchanged in behaviour. The gateway's authentication flow and dispatch still populate the record, now through the exported `audit.From` and `Record` setters at the same call sites; the package imports only `foundation/telemetry` and `ingress/http`. Its unit tests moved with it; the tests that run it behind SigV4 stay in `gateway`. |
| `spinifex#1334` | `spinifex/gateway/spx` (`spx.go`, `storage.go`, `promote_image.go` and their tests) | `spinifex/operator/admin` (package `admin`) | The version, node, VM and storage-status gather helpers and the image-promote NATS request move unchanged with no shim; the `spinifex` service dispatch that calls them stays in `gateway/spinifex.go` until the `spinifex` registration slice. `operator/admin` already imports `domains/ec2/image` and `contracts/operator/v1`, so the move adds no edge and the layering ratchet needed no allowlist row. The `gateway/spx` frozen-legacy entry is removed. |
| `spinifex#1334` | `spinifex/gateway` (`CreateAccountRequest`/`Response`, `DeleteAccountRequest`, `DescribeAccountDeletionRequest`, `DeletionState*`, `AccountSummary`, `ListAccountsResponse`, `AdminMethodNames`) | `contracts/operator/v1` (`admin.go`) | The `/admin` bodies the CLI shares with the gateway move unchanged in field names, JSON tags and omitempty, pinned by golden JSON written against the gateway types first and moved with them. `AdminMethodNames` is now owned by the contract and the gateway builds its `adminMethods` accept set from it, with a test that the router accepts exactly that list. `AccountQuotaRequest`/`AccountQuotaResponse` (embed `quota.Overrides` from `domains/admission/quota`) and `DeleteAccountResponse` (embeds `accountlifecycle.Result`) stay in `gateway`: `contracts` may import only `foundation` and `contracts`, so they are settled when the `/admin` handlers move to `operator/admin`. `accountlifecycle.Stage` serialises as an integer, not its name; the golden test pins that. |
| `spinifex#1345` | `spinifex/gateway/admin*.go` (the `/admin/{method}` handlers, gates and body cap) and the `AccountQuotaRequest`/`AccountQuotaResponse`/`DeleteAccountResponse` wire types | `spinifex/operator/admin` (`rest.go`, `account_*.go`), served by the same `spinifex` registration | `/admin/{method}` stays a chi route in `gateway`, ahead of the service selector, but `Admin_Request` now only mints the request ID and header and hands the request to the registered entry through `dispatchRegistered`, the path every Query request takes. `dispatch.Invocation` gains `Service`, the credential-scope service, so the scope gate is the first identity gate in the dispatcher and a key signed for another service is denied exactly as before. The order is unchanged: verb and method, then the five fail-closed gates (scope, super-admin account, IAM user, identity, `spinifex:<Method>` policy), then the cluster check, then the 8 KiB body cap. The dispatcher returns errors and the route renders them as the JSON envelope with `requestId`, then logs once from the real result, so a denial or a 5xx is never logged as a completed request; the registration's `ResolveAction` labels the method, which removes the `auth.go` special case. The caller is read from `dispatch.Actor` (`UserName` for the gate and `updatedBy`). `BucketStore` and `SignupMaxAccounts` leave `GatewayConfig` for `operator/admin.Deps`, composed in `awsgw` beside `IAM`, `Quota`, `ExpectedNodes` and `NodeIDs`. The three wire types that could not enter `contracts` move to `operator/admin` with their handlers, `operator/cli` imports them from there, and the last `operator/cli` → `gateway` allowlist entry is deleted. |
| `spinifex#1346` | `spinifex/gateway/iam.go` (`IAM_Request`, the IAM action table, `getUserForCaller`, the PassRole check) | `spinifex/domains/iam/awsapi` (`NewRegistration`, `Deps`, `ActionNames`), composed by `spinifex/runtime/roles/awsgw` | IAM registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, service and account checks, then one parse of the input, the `GetUser` default for a missing `UserName`, scope resolution, authorization, the `iam:PassRole` check for `AddRoleToInstanceProfile`, then the handler. The caller is read from `dispatch.Actor` (`PrincipalType`, `UserName`), with no ARN parsing. The dependencies are `Deps.IAM` and `Deps.CountInstanceProfileAssociations`, composed in `awsgw`, and the gateway keeps no IAM fallback. The `cmd/aws-model-coverage` named exception gains an `iam` row. |
| `spinifex#1346` | `spinifex/gateway/sts.go` (`STS_Request`, the STS action table, the anonymous STS dispatch) | `spinifex/domains/sts/awsapi` (`NewRegistration`, `Deps`, `ActionNames`, `NewAnonymousAssumeRoleWithWebIdentity`), composed by `spinifex/runtime/roles/awsgw` | STS registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, service check, caller, the `AssumeRole` identity gate scoped to the stored role ARN, then the handler. The caller is read from `dispatch.Actor` (`CallerARN`, `UserName`, `SessionName`, `UserID`, `AccessKeyID`), with no ARN parsing. The gateway's authorization failure wraps a generic `dispatch.PolicyDenialError`, and STS matches it to answer with the caller, action and resource, so the gateway holds no STS-specific denial code. An unsigned `AssumeRoleWithWebIdentity` stays ahead of SigV4 in the gateway and reaches the handler through `GatewayConfig.AnonymousSTS`, a func composed in `awsgw`, never the STS service. The `cmd/aws-model-coverage` named exception gains an `sts` row. |
| `spinifex#1348` | `spinifex/gateway/ecs.go` (`ECS_Request`) and `spinifex/gateway/eks.go` (`EKS_Request`, the EKS route table) | `spinifex/domains/ecs/awsapi` and `spinifex/domains/eks/awsapi` (`NewRegistration`, `Deps`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | ECS (AWS JSON 1.1) and EKS (REST JSON) register through the ADR-0002 seam with the JSON error envelopes they had, both declared as AWS JSON 1.1 (see the envelope divergence below). The check order and every response are kept, pinned by the pipeline characterisation tests committed first. EKS and ECS resolve roles through `foundation/aws/passrole``Check` with `domains/iam``RoleResolver`: a malformed or foreign role is refused first, a missing role is authorized as written before it is reported, and a resolver outage is returned unchanged. Role handling now follows the AWS capture of 2026-10-30. ECS RegisterTaskDefinition refuses a foreign role with `ClientException` "Role is not valid" before PassRole and accepts a nonexistent, invented-path or non-ARN role from a caller who can pass roles. ECS RunTask refuses a foreign `taskRoleArn` or `executionRoleArn` override the same way and answers a nonexistent or invented-path one with "ECS was unable to assume the role". EKS CreateCluster and CreateNodegroup answer a non-ARN or foreign role with `AccessDeniedException` "Cross-account pass role is not allowed.". EKS public routes now report every parameter fault as `InvalidParameterException`, the code the EKS model declares, while the internal control-plane routes keep `InvalidParameterValue` because the guest agents match on it. Two deliberate EKS divergences remain. First, CreateCluster still refuses a nonexistent or invented-path role with `InvalidParameterException` "role ... does not exist", where AWS accepts it and the cluster later ends FAILED. Our EKS only stores, validates and echoes the cluster role and never assumes it, so accepting would leave an ACTIVE cluster bound to a role that does not exist. Second, CreateNodegroup keeps its existing refusal of a nonexistent role because AWS was not captured for it. A third divergence is the error envelope. EKS is restJson1 but registers `ErrorEnvelopeJSON`, so errors carry `application/x-amz-json-1.1` and a `__type` body where AWS sends `application/json` and a `message` body. The `X-Amzn-Errortype` header is correct and SDKs read the code from it, so impact is low. Switching to `ErrorEnvelopeRESTJSON` also changes the throttle code and the default internal-error code (`ServiceException` where the EKS model declares `ServerException`), so it is left for its own change. Verified live on env14: the ECS and EKS role cases above, and a valid CreateCluster reaching ACTIVE with the control-plane agent publishing through the gateway. The EKS route table and its tests move with the registration, and the gateway keeps no ECS or EKS fallback. The `cmd/aws-model-coverage` named exception gains an `ecs` and an `eks` row, and the two `gateway` allowlist rows are removed. |
| `spinifex#1350` | `spinifex/gateway` (`ecr.go`, `ecrauth_middleware.go`, the `/v2/token` exchange, `ecr_principal.go` with `ResolveECRPrincipal`, the per-operation `/v2` authorization, the registry host matcher, `ctxECRPrincipal`) | `spinifex/domains/ecr/registry` (`Bridge`, `NewBridge`, `BridgeConfig`, and its own `Principal`, `Rehydrator`, `Authorizer`, `Admission` types with `ErrInvalidCredential` and `ErrUnavailable`), `spinifex/ingress/http` (`HostMatcher`, `TargetAccount`, `RequestClientIP`; names no service), `spinifex/ingress/aws/credential` (`Rehydrator`, `Principal.CallerARN`), and under `spinifex/runtime/roles/awsgw`: `credverify` (`Rehydrate`, sharing the SigV4 key and session lookups), `policyauth`, `accountadmission`, `registryadapt` | ADR-0001 amendment 2026-10-29 and ADR-0005:S2: the OCI `/v2` bridge and its token auth move out of the gateway, and ECR resolves its principal through the credential capability instead of reading `IAMService` and `STSService` directly. ADR-0002:S5 keeps `/v2` outside the seam, so the gateway keeps only a `RegistryHandler` mount that answers 501 when unset. `domains/ecr/registry` imports no ingress package (`TestADR0001_Layering` unchanged): the principal, rehydrator and authorizer are registry-local types and `registryadapt` maps `credential.Failure` codes onto `ErrInvalidCredential` (401) and `ErrUnavailable` (503). No widening of `dispatch.Actor` or `dispatch.Invocation`. The SigV4 path and the bridge share one policy evaluator (`policyauth`) and one account-admission check. Deliberate change: the open-registry mode for a nil token verifier is deleted and `NewBridge` refuses a nil Verifier, Issuer, Rehydrator, Authorizer or Admission, so tests that want an unauthenticated registry mount `registry.Handler` directly. The pre-move pipeline characterisation (status codes, error bodies, the `WWW-Authenticate` challenge, referrers, token exchange, revoked and deleted principals) passes unchanged apart from setup, and the retired `ResolveECRPrincipal` parity rows live on as `credverify``Rehydrate` rows and `Bridge` claim-check rows. Live on env14 (DHCP external IP): crane, skopeo and docker push and pull (`tests/e2e/ecr`), a cosign sign and verify in the legacy tag mode and in OCI 1.1 referrers mode with the signature returned by `GET /v2/{repo}/referrers/{digest}`, `/v2/token` refresh from fresh logins, and a deactivated access key answering 401 on `/v2/token`, manifest GET, crane pull and crane push, then 200 again once reactivated. The ECR signing-key deadlock bead (`mulga-6wqce`) is untouched. |
| `spinifex#1351` | `spinifex/gateway/{bedrock,bedrockruntime,bedrockagent,bedrockagentruntime}.go` (`Bedrock_Request`, `BedrockRuntime_Request`, `BedrockAgent_Request`, `BedrockAgentRuntime_Request`, the four route tables), `spinifex/domains/ochre/authz.go`, and the 13 `Bedrock*` fields of `GatewayConfig` | `spinifex/domains/ochre/awsapi` (`NewRegistration`, `Deps`, `Quota`, `OperationInventory`, `ServiceName`), composed by `spinifex/runtime/roles/awsgw` | The four Bedrock services register through the ADR-0002 seam as one registration, per ADR-0002:S1 (one per signing service) and the ADR-0001 amendment of 2026-10-06: the service is `bedrock` and `bedrock-runtime`, `bedrock-agent` and `bedrock-agent-runtime` are aliases. A request signed with a sub-service scope is dispatched to that sub-service whatever its path, so a path owned by another sub-service answers `InvalidActionException` as before; only the shared `bedrock` scope is split by path. The pipeline characterisation tests committed first (status, error type and envelope per service, metering order, nil-capability answers) pass unchanged apart from setup. The 13 gateway fields become `awsapi.Deps`: `Credentials`, `Endpoints`, `Access`, `Recorder`, `Logging`, `Provisioned`, `Guardrails`, `Embedder`, `KB`, `DataSources`, `Vector` and `Quota`, with the documented nil fallbacks (no grants denies every model, a nil store errors rather than panics); `BedrockAccessAdmin` had no reader outside one test and is gone, and `Deps.NATS` is dropped because the nil-NATS answer is given by the gateway-wide gate before dispatch. `Logging`, `Provisioned`, `Guardrails`, `KB` and `DataSources` are the concrete exported store types, a temporary edge until Ochre exposes capability interfaces for them. The wire-typed action functions stay in the `domains/ochre` root and the SDK-typed adapters move; `authz.go` moves with them, which removes the `domains/ochre` to `ingress/aws/bodyscope` allowlist row, the only allowlist change. The legacy switch cases, the `supportedServices` and `jsonErrorService` entries and the path-based sub-service rewrite in `GetService` are removed, so `ec2` is the only service left on the legacy table; the rewrite's rows are pinned in `awsapi` as `resolveSubservice` and dispatch tests. The registration declares the AWS JSON 1.1 envelope it had and `cmd/aws-model-coverage` is not wired, so no coverage allowlist row is added. Recorded divergences, not changed by this move: Bedrock is restJson1 in AWS but answers with the JSON 1.1 envelope (`application/x-amz-json-1.1`, `__type` body), and CreateGuardrail answers 200 where AWS answers 202; both are tracked by `mulga-xug8b`. No change to `dispatch.Actor` or `dispatch.Invocation`. |
| `spinifex#1354` | `spinifex/gateway/ec2.go` (the EC2 action table, `ec2Handler`, `ec2Actions`, the dispatch pipeline), `ec2_idempotency.go`, `ec2_validation.go`, and the EC2 token-bucket fields of `GatewayConfig` | `spinifex/domains/ec2/awsapi` (`Handler`, `NewHandler`, `Deps`, `Quota`, `Dispatch`, `ActionNames`, `InputHasField`), composed by `spinifex/gateway/ec2.go` | Slice 9a of the EC2 move: the capabilities first, with no behaviour change. The EC2 action table, idempotency and validation move to `domains/ec2/awsapi` behind `Deps{NATS, IAM, Quota, ActiveNodes, ExpectedNodes, NodeIDs, AZ, Endpoint, InstanceStatus}`; the gateway keeps the composition and `EC2_Request`, still on the legacy switch until slice 9b registers EC2 through the ADR-0002 seam. `Quota` is declared in `awsapi` because `domains/admission/quota` imports the EC2 action packages and cannot be imported back, and the gateway wraps `quota.Service` to supply the NATS instance-type resolver its retype gate takes. The instance and spot-instance adapters take the two-method `InstanceProfiles` interface (`ResolveInstanceProfile`, `CanonicalResourceARN`) instead of `iamdomain.IAMService`, which pays off the `spotinstance` to `domains/admission/quota` and `domains/iam` allowlist rows. Allowlist changes are 20 deletions and no additions: the 18 `gateway` to `domains/ec2/awsapi/*` rows and those 2. The pipeline characterisation tests committed first (error envelopes, check order, gateway-local actions, idempotent replay, PassRole, ImportKeyPair padding, launch-template expansion before authorization, RunInstances instance-profile refusal order) pass unchanged. Package-local tests drive every routed action through `Dispatch` against a stub daemon. No change to `dispatch.Actor` or `dispatch.Invocation`. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and MaxResults errors, a no-policy user refused with UnauthorizedOperation (including on the gateway-local DescribeRegions), an assumed-role session allowed and refused per its policy, and one RunInstances to running then terminated. |
| `spinifex#1355` | `spinifex/gateway/ec2.go` (`EC2_Request` in production, the legacy service switch, its `supportedServices` and JSON tables) and the `Quota` field of `GatewayConfig` | `spinifex/domains/ec2/awsapi` (`NewRegistration`, `OperationInventory`, `ServiceName`), composed by `spinifex/runtime/roles/awsgw` (`ec2.go`: `newEC2Registration`, `advertisedEndpoint`; `ec2quota`) | Slice 9b of the EC2 move: EC2 registers through the ADR-0002 seam as service `ec2` and the legacy gateway switch is empty. The EC2 query-protocol error envelope is kept through `ErrorEnvelopeXML`, and the policy denial match is now `*dispatch.PolicyDenialError`. The characterisation tests committed first also pin the Authorize service name and action, the trace span and audit action, the throttle key (account plus action) and the cluster-unavailable gate, which runs before policy and action lookup for gateway-local and NATS-backed actions alike. They and the 9a tests pass unchanged apart from setup: STS and EC2 test setups now register the services they sign as, and the two tests that claimed a legacy service name are removed because no legacy name remains. `EC2_Request` survives only as a test shim. Allowlist changes are two deletions (the `gateway` to `domains/ec2/awsapi` and `domains/ec2/awsapi/instance` rows) and one addition, the per-service model-coverage row `cmd/aws-model-coverage` to `domains/ec2/awsapi`. `startInstanceStatus` stays in `awsgw.go` because ADR-0007:S6 records the raw instance-state uses by file. No change to `dispatch.Actor` or `dispatch.Invocation`. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and InvalidVolume.NotFound errors, a no-policy user refused with UnauthorizedOperation (including on the gateway-local DescribeRegions), an assumed-role session allowed and refused per its policy, and one RunInstances to running then terminated. The gateway request logs carry `service=ec2` and the per-action label. Not exercised live: a throttled request (the default limits are not reachable from one client; the throttle key is pinned by the characterisation tests) and an agent-signed EC2 call. |
| `spinifex#1363` | `spinifex/gateway` context keys (`ctxService`, `ctxRegion`, `ctxAction`, the principal and access key id keys, and the unread `ctxQueryArgs`), the duplicate `callerPrincipal` and `principalContext` types, `gateway/caller_arn.go` (`buildCallerARN`), and the direct `policyauth`, `accountadmission` and NATS-connection reads in the auth pipeline | `spinifex/ingress/aws/reqctx` (the request record), `credverify.CallerARN`, and the gateway-consumed capabilities `PolicyEvaluator`, `AccountAdmission` and `Readiness`, composed by `spinifex/runtime/roles/awsgw` (`clusterready`) | Slice 11a of the gateway removal: the request-context record and the ADR-0002:S8 capabilities, with no behaviour change. The record is a copy-on-write value under one context key; the principal is sealed once and a second `WithPrincipal` keeps the first caller. The layering rule `S7_RequestRecordPrivateToIngress` lets only `gateway` and `ingress/aws/*` import `reqctx`, with a fault-injection test, and slice 11c moves the package to `ingress/aws/internal/reqctx` and drops the rule. The caller ARN is built once in `credverify` (root cases kept) and filled on `Principal.CallerARN`. The policy evaluator, account admission (`credential.ErrAccountNotActive`, with `accountadmission.ErrNotActive` an alias) and cluster readiness (`Available() bool`) are interfaces the gateway consumes and fails closed on when nil. The step 1 characterisation tests (throttle against both readiness gates and the policy gate for ECS and IAM, `/admin` skipping `GetService` and the second gate, the nil-connection difference between the two gates, five error writers each minting a request id that differs from the audit record, and the credential-scope region reaching only the span) pass unchanged apart from setup. No allowlist rows change, and there is no change to `dispatch.Actor` or `dispatch.Invocation`. Debt for slice 11c: the test-only `defaultCapabilities` seam (a package variable set by a test `init` so about 60 gateway test files can build a config from `IAMService` alone), the nil-connection case of the early readiness gate, the remaining `NATSConn` readers (`DiscoverActiveNodes` and the field that `awsgw` and the integration harness set), and the `IAMService`, `STSService`, `Config`, `RootCAs` and `InternalSuffix` fields of `GatewayConfig`, which no production code reads. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and InvalidVolume.NotFound errors, a bad access key refused with InvalidClientTokenId on a Query service and InvalidClientTokenIdException on a JSON service, a no-policy user refused with UnauthorizedOperation, and an assumed-role session allowed and refused per its policy. Not exercised live: a suspended-account refusal, a throttled request and the NATS-down 503, which the step 1 tests pin. |
| `spinifex#1378` | `spinifex/gateway``oidc_discovery.go` and `registry.go` (the `/oidc/eks` routes and the `/v2` mount), the `DiscoverActiveNodes` method and its cache, and the `GatewayConfig` fields `Config`, `RootCAs`, `InternalSuffix`, `Version`, `Commit`, `IAMService`, `STSService`, `NATSConn` and `ExpectedNodes` | `spinifex/runtime/roles/awsgw` (`PublicRoutes`, `NodeDiscovery.ActiveNodes`, `composeGateway`); the gateway keeps one `PublicRoutes` mount hook and the `Admin` registration name supplied by the role |
| `spinifex#1384` | `spinifex/gateway` (`GatewayConfig`, the SigV4 and policy middleware, `Request`, the admin dispatch, `AWSOperationInventory`) | `spinifex/ingress/aws/pipeline` (`Config`) and `spinifex/ingress/aws/dispatch` (`AWSOperationInventory`); the gateway layering rows are retired and R2 binds the pipeline |
| `spinifex#1334` | `spinifex/gateway/oidc_discovery.go` (`eksAccountKV`, the discovery-document and JWKS read and construction) | `spinifex/domains/eks/cluster` (`OIDCPublisher`, `OIDCDiscoveryDocument`), composed by `spinifex/runtime/roles/awsgw` | The EKS domain now reads its own account bucket and builds the per-cluster discovery document and JWKS responses (ADR-0005:S4), and the gateway only maps the unauthenticated `/oidc/eks/...` routes onto an `OIDCPublisher` interface it is handed. Responses are byte-identical, pinned by `oidc_discovery_current_behaviour_test.go`, whose assertions are unchanged. The gateway no longer imports `domains/eks/cluster` in production code. `STS` keeps its own `ClusterJWKSResolver` (`OIDCJWKSReader`), untouched. Recorded debt, not repaired here: a stored JWKS is served verbatim with 200 even when it is not valid JSON. |
| `spinifex#1245`, `spinifex#1390` | — | `contracts/admission/v1` | The quota override seal route (`admission.v1.SealAccountQuotaOverride`, served by `awsgw`) and the per-gateway read-only capability probe (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) are an explicit versioned cross-process contract between the quota owner and `operator/accountlifecycle`. The consumer wiring is new: `NewClusterEngine` takes a `ClusterNodes` value with separate daemon and gateway node sets, `ClusterConfig.NodesRunning` is the single definition of which nodes run a service, and the seal reaper runs in `StageAccount` after the EC2 settings seal. A termination whose gateway probe is incomplete seals nothing and leaves the account `TERMINATING`. Contract, responder, probe and reaper tests pin the subjects, JSON shapes and fail-closed cases; `TestAccountTermination_QuotaOverrideEndsSealed` replaces the former characterization that overrides survive teardown. |
The EC2 contract row is intentionally different from the directory moves: it creates
a compatibility boundary. `ec2.cmd.*` retains its deployed one-token NATS
subscription and permission shape, while callers use
@@ -387,9 +406,9 @@ the canonical IAM/STS principal before dispatch, because ECR must not depend on
gateway identity context or ARN construction.
All implemented ECR control-plane actions now dispatch through explicit ECR
capabilities. The remaining ECR-named gateway code is generic `/v2/*` route
assembly, SigV4/registry authentication and policy enforcement; it remains in
gateway rather than inverting those dependencies into the ECR domain.
capabilities. The OCI `/v2` bridge, its token exchange and its principal
resolution now live in `domains/ecr/registry`; the gateway keeps only the
`RegistryHandler` mount, and `awsgw` composes the bridge.
### ACM relocation residuals
@@ -438,7 +457,7 @@ These residuals were recorded by individual slices and must be resolved once the
- `runtime/roles/vpcd` (placement): the role still mixes process/role supervision with network-controller responsibilities. `lease_reaper.go` (orphaned external-pool DHCP lease sweep), `ovn_controller_watchdog.go` (OVN controller liveness and restart), `host_eip_loop.go` (host-EIP reconcile loop) and `lease_datapath_reconcile.go` (lease datapath reconcile) are periodic network-control loops, not role wiring; `vpcd.go` itself also carries bridge-mode detection and verification, chassis-list parsing, external-CIDR resolution, DHCP manager startup and lease/IP-change handling, and gateway/host-EIP allocator and neighbour flush/prime construction alongside `Service.Start`. The future placement question is whether these belong in `domains/network`. Removal condition: an accepted decision on network-controller placement, and extraction of those loops and helpers behind the role.
- ECR control-plane registration: `awsapi.Deps.NATS` is a raw `*nats.Conn`, not a narrow capability. Every NATS-backed `Handler` passes it to `domains/ecr.NewNATSMetaStore`, which calls `natsmsg.NATSRequest`, so narrowing it means changing the exported `Handler` type, every handler and the meta store; it must not be broadened further meanwhile.
- ACM control-plane registration: `domains/acm/awsapi.Deps.NATS` (`registration.go`) is likewise a raw `*nats.Conn` carried by every `Handler`, for the same reason; it must not be broadened further meanwhile.
- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi` and `gateway` to read ECR's and ACM's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `gateway` and the `awsapi` package of each registered service (`domains/ecr/awsapi`, `domains/acm/awsapi`, `domains/route53/awsapi`, `domains/elbv2/awsapi`, `domains/rds/awsapi`, `domains/ecs/awsapi`, `domains/eks/awsapi`) to read their declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
- Ingress pipeline (ADR-0002 phase 4): the request-audit record, the trace action enricher, the throttle key functions and the gateway request-context keys stay in `gateway` until the auth/IAM and request-context boundary is designed. SigV4 authentication writes the audit record and context keys from four places, so moving them now would export exactly the setters ADR-0002 says must not be exported solely to make a directory move possible.
- NAT projection delivery is inconsistent and is preserved as found: EIP `AssociateAddress` sends `vpc.add-nat` unacknowledged (`AnnounceNATAdd`), skipping the 45s OVN barrier every other add path waits on, and `DeleteNetworkInterface` sends `vpc.delete-nat` unacknowledged (`AnnounceNATDelete`), skipping the ack and the ErrNoResponders retry. The reconciler's orphan sweep is the only backstop for both. Clears when those call sites use `AddNAT`/`AddNATBestEffort` and `RemoveNAT`, a reliability change of its own.
- `vpc.add-system-egress` has a `vpcd` subscriber and a contract entry but no producer (its former `domains/ec2/eip` publisher is gone); `domains/eks/cluster` still publishes the delete. Either a producer is missing or the add route is dead; it needs an owner decision.
@@ -446,7 +465,7 @@ These residuals were recorded by individual slices and must be resolved once the
- `foundation/messaging/nats/nats_trace.go` still names its OpenTelemetry tracer `github.com/mulgadc/spinifex/spinifex/utils`. It is an emitted instrumentation-scope name that dashboards may filter on, so renaming it is a telemetry change, not a path fix.
- `runtime/compute/cache.New`: two construction-time panics carry a lint exemption pending a constructor that returns an error.
- `domains/ec2/systeminstance`: managed-service parent metadata is only the `ManagedBy` tag value.
- `domains/ochre` (action adapters): since `spinifex#1266` the Bedrock action layer (handlers, action authorization, event-stream framing and the CRUD, catalogue and Converse/Invoke entry points) shares package `ochre` with the domain core, stores and inference engine, and imports `ingress/aws/bodyscope`. Moving it to `domains/ochre/awsapi` is not mechanical: the adapters reach package-private store internals (the guardrail and provisioned-throughput store handles and get/put/update methods), and adapter tests override the package-private model catalogue. It needs core capabilities first, then the move; the top-level `gateway/bedrock*.go` HTTP handlers additionally wait for the ingress registration seam.
- `domains/ochre` (action adapters): cleared by the Bedrock registration in `spinifex#1351`. The SDK-typed adapters and `authz.go` moved to `domains/ochre/awsapi`, the `ingress/aws/bodyscope` allowlist row is deleted, and the adapters take the concrete exported Ochre stores through `awsapi.Deps` as a temporary edge until Ochre exposes capability interfaces for them.
- `domains/network`: `reconcile/intent.go` builds desired network state by reading EC2's KV buckets and decoding EC2's private VPC, subnet, ENI, route-table, gateway, EIP, NAT and security-group records, and `reconcile/drift.go` watches those buckets. ADR-0001 names this pair explicitly: EC2 and network communicate through a capability/projection or versioned contract, never by one being the other's KV reader. Step 7 fix: an EC2-owned `domains/ec2/projection` exposing network intent with list and watch (it may stay KV-backed, so no data moves); a narrow intent interface declared by `reconcile`; wiring at the composition root; and the duplicated `PurposeEIP`/`PurposeIGWLRP` constants and mirrored `SGRule`/`SGEvent` folded into the projection or `contracts/ec2/v1`. The drift-watch coverage test and reconcile scenario tests are the evidence the projection must keep. `domains/network` also imports `runtime/compute/vm` and `runtime/host/command` (domain-to-runtime).
- `domains/ec2/instance`: imports `runtime/compute/vm` and `runtime/compute/gpu` (domain-to-runtime) and `domains/dns` (cross-domain implementation import).
- `domains/ec2/instance/state` (dependency): the vpcd IMDS local reader imports `runtime/compute/vm` to return `vm.VM`. Removal condition: that type moves into a domain-owned package that `runtime/compute/vm` consumes (runtime to domain is allowed).
@@ -456,14 +475,12 @@ These residuals were recorded by individual slices and must be resolved once the
- `operator/installer/firstboot` (security): `Config.InstallCallback` is rendered into `spinifex-firstboot.sh`, which is world-readable (0755). The field is currently unused (`fireInstallCallback` reads `SPINIFEX_INSTALL_CALLBACK` from the environment instead), but any future wiring must not place secrets or tokenised callback URLs in the generated script; sensitive callback credentials should reach first boot through a protected channel such as a 0600 file or a credential store.
- `operator/cli` (known security risk): `spx admin join` contacts the formation server with `InsecureSkipVerify`, sends the join token as a bearer credential and receives the cluster CA certificate and key, master key, Viperblock key and shared credentials, so an on-path attacker during formation can capture the token and the cluster root secrets or substitute a CA. This predates the move; lint surfaced it when the tree left `cmd/`. It is accepted for now and carries a line-level gosec annotation stating it is not a false positive. Removal condition: authenticate formation TLS with a pinned key or certificate delivered with the join material, or an out-of-band CA, with tests proving a mismatched server certificate fails before credentials are sent.
- `domains/eks/cluster` (potential security risk): `ClusterMeta.ControlPlaneTemplate` (`domains/eks/cluster/service_impl.go`) clears the template's credentials but keeps `OIDCPrivateKeyPEM` and the k3s `JoinToken`, so both are persisted in the `eks-account-{acct}` cluster record, which the gateway also reads. Whether that record is encrypted at rest is unconfirmed; if it is not, a reader of the bucket can forge service-account tokens for the cluster's OIDC issuer and join nodes to it. This predates the refactor and is not fixed in it. Removal condition: the key and token are held by a dedicated secret store (a future KMS or Secrets Manager capability is the likely owner) and the cluster record keeps only a reference, with a test proving the persisted record contains neither secret.
- `operator/cli` (dependency): the command tree still imports domain and service implementations directly (`handlers/{iam,eks}`, `domains/admission/quota`,`gateway`,`domains/ochre`, `vpcd`, the Viperblock adapter and runtime roles). The ADR's `cmd/spinifex` current-to-target map row requires command packages to reach domains through public capabilities; splitting supported use cases into `operator/*` packages is later work.
- `operator/cli` (dependency): the command tree still imports domain and service implementations directly (`handlers/{iam,eks}`, `domains/admission/quota`, `domains/ochre`, `vpcd`, the Viperblock adapter and runtime roles). The ADR's `cmd/spinifex` current-to-target map row requires command packages to reach domains through public capabilities; splitting supported use cases into `operator/*` packages is later work.
- `providers/ebs/viperblock` → `runtime/compute/nbd` and `runtime/host/process` (dependency, moved unchanged from `services/viperblockd`): the adapter formats NBD endpoints and manages nbdkit processes through runtime packages. ADR-0001 does not list providers → runtime as forbidden and the layering test does not enforce it (same category as `providers/ebs/conformance` → `runtime/compute/nbd`), but a provider should not depend on node runtime. Clears when NBD endpoint naming has a provider-facing home (a candidate for `contracts/ebs/v1`) and process supervision is injected by the role. The adapter also imports `providers/objectstore`, a provider-to-provider edge that is allowed.
- `gateway/ec2.go` (registration): the EC2 dispatch map stays a`GatewayConfig` method in `gateway` and imports every `domains/ec2/awsapi/*` package; `gateway/ec2_idempotency.go` also imports `domains/ec2/awsapi/idem` (`KVBucket`, `TTL`, `TokenAndParams`) for the dispatch-time idempotency wrapper. ADR-0001 makes the domain own its action registration; moving the map waits on the generic ingress registration seam and must not be done as a package move.
- `gateway` (service registration): the root `gateway` dispatch for IAM, STS, EKS, RDS, ECS and ELBv2 imports `domains/{iam,sts,eks,rds,ecs,elbv2}/awsapi`, which only AWSGW may wire. It clears when each service registers through the ADR-0002 ingress registration seam; as with EC2, moving the dispatch must not be done as a package move.
- `gateway` (service registration): the root`gateway` dispatch for IAM, STS, EKS, RDS and ECS imports `domains/{iam,sts,eks,rds,ecs}/awsapi`, which only AWSGW may wire. It clears when each service registers through the ADR-0002 ingress registration seam; as with EC2, moving the dispatch must not be done as a package move.
- `operator/accountlifecycle` (dependency): its EC2 reapers call `domains/ec2/awsapi/*` action adapters directly, as they did under `gateway/ec2`. The amended ADR-0001 `accountteardown` row keeps replacing these with domain-owned cleanup capabilities as deferred debt; it was not a relocation prerequisite. It also holds a read-only consumer of the EC2 platform-inventory contract (`contracts/ec2/v1`, `ec2.v1.PlatformInventory`), served by `domains/ec2/accountinventory`; only the explicit dry run (`Engine.DryRun`) calls it, to log a shadow comparison, and the key-pair, placement-group and launch-template reapers still use the `awsapi` adapters. A domain-owned cleanup capability for the proven-safe subset of those three kinds (key pairs and empty placement groups no instance, spot request or live launch-template version names, and launch-template residue; never live launch templates) exists as `domains/ec2/accountcleanup` (a Go interface with injected owners and reference readers, no NATS subject); nothing calls it yet. It also holds the StageAccount settings-seal reaper, a consumer of `contracts/ec2/v1`'s `ec2.v1.SealAccountSettings` and per-node `ec2.v1.AccountSettingsSealCapability.<node>` subjects, served by `domains/ec2/account` through the daemon; the reaper reads no KV and imports no domain package, reaching both subjects only through NATS and the configured node set `NewClusterEngine` already receives.
- `domains/admission/quota` (dependency, moved from `handlers/quota` unchanged): `live.go` imports `domains/elbv2/awsapi` and `domains/rds/instance` for its live ELBv2 and RDS counters; `records.go` imports `runtime/compute/vm` for the instance-record vCPU sweep; `live.go`, `vcpu.go` and `storage.go` call `domains/ec2/awsapi/{vpc,eip,volume}` describe adapters to count VPCs, subnets, addresses and volumes, as they did under `gateway/ec2`. The ADR's `handlers/quota` current-to-target map row replaces these with reader capabilities once their final locations exist.
- `domains/ec2/awsapi/instance` (dependency): imports `domains/iam`(`IAMService` in `DescribeInstances.go`, `RunInstances.go` and`IamInstanceProfileAssociation.go`; `InstanceProfile` in the latter two), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge needs an IAM capability; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
- `domains/ec2/awsapi/spotinstance` (dependency): imports `domains/iam.IAMService` and `domains/admission/quota.Service` in `RequestSpotInstances.go`.
- `domains/ec2/awsapi/instance` (dependency): imports `domains/iam`for the `InstanceProfile` type that `InstanceProfiles.ResolveInstanceProfile` returns (`RunInstances.go`,`IamInstanceProfileAssociation.go`), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge is now the two-method `InstanceProfiles` interface plus that type; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
- Instance adapter consumers outside EC2: `domains/rds/awsapi/enginecatalog.go`, `runtime/daemon/eks_cp_control.go`, `runtime/daemon/rds_deps.go`, `domains/admission/quota`'s `vcpu.go` and `gateway/iam.go` (`CountInstanceProfileAssociations`) call `domains/ec2/awsapi/instance` directly; each needs an EC2 capability rather than the AWS action adapter.
- `runtime/daemon` (Q-28 ownership and deployment): the daemon moved whole, keeping its current ownership, deployment and `daemon → gateway → handlers` dependencies. Q-28's write-side consolidation and any daemon decomposition or split remain deferred by ADR-0001 and are not paid by the move.
@@ -472,8 +489,8 @@ Enforcement: the owner decided the layering test is blocking now, with a ratchet
- `domains/rds/instance` → `domains/rds/parametergroup` (legacy edges): `parameterGroupReferences` binds `Dependants` by scanning instance records; this clears when the RDS instance gains its own owner and answers `Dependants` from it. `tags.go` writes the meta record through its generic tag registry; this clears when tagging invokes a parameter-group-owned tag mutation capability, and moving the tagging adapter to `domains/rds/awsapi` alone does not clear it. The Modify handler still calls `propagateParameterGroup` after the owner's write; propagation is instance-side realization and stays legacy until it is designed as asynchronous realization (group generation, instance staleness, retry). `parameterGroupValues` returns the persisted `Record` and overrides to instance code; when the instance owner is extracted, replace them with a consumer-owned resolved-parameters projection.
- `domains/ecs` → `domains/ecs/taskdefinition` (owner bypass): `tags.go` reads and writes the revision record through its generic tag registry; this clears when ECS tagging invokes an owner-provided tag mutation capability, and moving `tags.go` to `domains/ecs/awsapi` alone does not clear it. `taskDefinitionResolver` returns the persisted `Record` to task and service launch code; when the task and service owners are extracted, replace the record crossing that boundary with a consumer-owned resolved task-definition (launch-spec) projection. Moving packages has not made the record private.
- `domains/ecs/taskdefinition` (deliberate duplication, not debt): private `getJSON`, `putJSON` and `keysWithPrefix` copy the raw-KV helpers in `domains/ecs` so the extraction keeps ECS's get/put/list error and recovery semantics. Revisit only if a later, tested state abstraction proves it preserves those semantics; do not migrate ECS to `kvstore` merely to remove them.
- `domains/eks/cluster` → `domains/eks/access` (legacy edges): `ResolveTokenReview` opens the EKS bucket in the gateway process and reads records through the exported `access.Get`; this clears when token review asks the owner across a contract rather than opening the bucket itself. `DeleteClusterPrefix` erases access-entry keys as part of its blanket cluster sweep; this clears when cluster delete invokes an owner-provided removal of the cluster's entries. The owner returns the persisted `Record` to the AWS adapter and token review; replace it with consumer-owned projections when those consumers are extracted.
- `domains/eks/cluster` → `domains/eks/addon` (legacy edges): `DeleteClusterPrefix` erases add-on records and manifests in its blanket cluster sweep; this clears when cluster delete invokes an add-on-owned teardown capability. `stageGPUDeviceAddon` is a temporary node-group-to-add-on orchestration edge: best-effort, errors logged and swallowed, never undone on node-group delete; it clears when the node-group owner is extracted and the dependency is designed. `ClusterReconciler` hosts the add-on status subscription; it clears when add-on delivery observation has its own host under the lifecycle correction. Cluster existence is checked in the handler, not through an owner-declared capability; it moves into the owner when cluster admission is defined (today status is ignored). Guest wire types (`AddonStatusReport`, `AddonDeliveryPhase`, `AddonStatusSubject`, `StagedAddonManifest`, `InternalAddonsResponse`) are owned by `contracts/eks/v1`, which `domains/eks/cluster`, `gateway/eks` and `agents/eks/gatewayfetch` import directly.
- `domains/eks/cluster` → `domains/eks/access` (legacy edges): `ResolveTokenReview` opens the EKS bucket in the gateway process and reads records through the exported `access.Get`; this clears when token review asks the owner across a contract rather than opening the bucket itself. `DeleteClusterChildren` erases access-entry keys as part of its blanket cluster sweep; this clears when cluster delete invokes an owner-provided removal of the cluster's entries. The owner returns the persisted `Record` to the AWS adapter and token review; replace it with consumer-owned projections when those consumers are extracted.
- `domains/eks/cluster` → `domains/eks/addon` (legacy edges): `DeleteClusterChildren` erases add-on records and manifests in its blanket cluster sweep; this clears when cluster delete invokes an add-on-owned teardown capability. `stageGPUDeviceAddon` is a temporary node-group-to-add-on orchestration edge: best-effort, errors logged and swallowed, never undone on node-group delete; it clears when the node-group owner is extracted and the dependency is designed. `ClusterReconciler` hosts the add-on status subscription; it clears when add-on delivery observation has its own host under the lifecycle correction. Cluster existence is checked in the handler, not through an owner-declared capability; it moves into the owner when cluster admission is defined (today status is ignored). Guest wire types (`AddonStatusReport`, `AddonDeliveryPhase`, `AddonStatusSubject`, `StagedAddonManifest`, `InternalAddonsResponse`) are owned by `contracts/eks/v1`, which `domains/eks/cluster`, `gateway/eks` and `agents/eks/gatewayfetch` import directly.
- `domains/eks/cluster` and `domains/eks/awsapi` → `domains/eks/addon` (further edges): the AWS adapter reads the persisted `Record` through `addon.Get`/`addon.List`; this clears when the owner returns a view and the add-on adapter moves to `domains/eks/awsapi`. The guest route reaches the owner through the `handlers_eks` NATS RPC (`ListStagedAddonManifests`); this clears with per-resource commands behind EKS registration. The add-on ARN is derived in both the owner and `domains/eks/awsapi/authz.go`; this clears when authorization obtains the ARN from the owner, which an incarnation suffix would require. The owner's `Manifest`/`Report` beside the `contracts/eks/v1` types is deliberate duplication, not debt.
- `agents/rds/agent` (dependency, narrowed): engine metadata now arrives through the agent-declared `EngineCatalog`, but the agent still imports `domains/rds/instance` for the command protocol, wire and record types (`Command*`, `Parameter`, `EngineHealth*`, `BootstrapMode*`) and API output types. This clears when those move behind a guest/controller wire contract.
- `domains/rds/awsapi` → `domains/rds/instance` (`InstanceSizing`): the gateway obtains its `Sizing` through the RDS handler package's adapter rather than its own composition root. This clears when the gateway's RDS composition moves to `runtime/roles/awsgw` and builds `Sizing` there.
@@ -672,12 +689,12 @@ series above; "needs decision" is open.
| `handlers/rds` | — | Done. The remaining RDS instance resource moved to `domains/rds/instance` (package `instance`) in `spinifex#1251` with no API, NATS subject, storage or behaviour change, fulfilling ADR-0001's organise-by-resource condition alongside the already-extracted subnet group, parameter group and engine catalogue. The ADR-0004 blocked-dependency imports (quota admission, the duplicated expected-node fan-out, the IAM ensurer and the Bluebottle master key) remain dependency-rule debt, with their allowlist entries now sourced from `domains/rds/instance`. |
| `handlers/elbv2` | — | Done. The implementation moved to `domains/elbv2` (package `elbv2`) with no API, NATS subject, storage or behaviour change. Q-30's data-plane boundary remains deferred debt; the amended ADR-0001 row makes it not a relocation prerequisite. Its cross-domain, IAM and `agents/elbv2` imports remain dependency-rule debt, with their allowlist entries now sourced from `domains/elbv2`. |
| `handlers/iam/mock` | — | Done. Moved with the implementation to `domains/iam/mock`. |
| `gateway` (root) | Generic HTTP route assembly, SigV4/registry authentication, the policy gate, request-audit middleware, request-context keys, and the legacy per-service dispatch switch for every service not yet registered. | Deferred: the ADR-0002 phases above (auth/IAM and request-context boundary design) gate moving the audit, context-key and legacy-dispatch mechanics. |
| `gateway/bedrock` | — | Done. The whole package moved into `domains/ochre` in `spinifex#1266` with its package-private core and tests intact and no behaviour change. Separating the action adapters into `domains/ochre/awsapi` remains deferred, recorded as `domains/ochre` (action adapters) debt above. |
| `gateway` (root) | Generic HTTP route assembly, SigV4 authentication, the policy gate, request-audit middleware and request-context keys. | Done and deleted. The request pipeline moved into `ingress/aws/pipeline` (as `pipeline.Config`) and the operation inventory into `ingress/aws/dispatch` in `spinifex#1384`; the `/oidc`, `/v2` and node-discovery code had already moved to `runtime/roles/awsgw` in `spinifex#1378`. The layering test binds the pipeline with R2 and `gateway` is a retired legacy root. |
| `gateway/bedrock` | — | Done. The whole package moved into `domains/ochre` in `spinifex#1266` with its package-private core and tests intact and no behaviour change. Separating the action adapters into `domains/ochre/awsapi` remains deferred, recorded as `domains/ochre` (action adapters) debt above. The action adapters then moved to `domains/ochre/awsapi` behind one ADR-0002 registration in `spinifex#1351`. |
| `gateway/{iam,sts,eks,rds}` | — | Done. The action adapters moved to `domains/{iam,sts,eks,rds}/awsapi` in `spinifex#1265` with no API, NATS subject, storage or behaviour change. ADR-0001 does not make ADR-0002 registration a prerequisite for relocating a domain's adapters, as the EC2 adapter moves already showed; the root `gateway` dispatch that still imports them is recorded `gateway` (service registration) debt. |
| `gateway/{ecs,elbv2}` | — | Done. The action adapters moved to `domains/{ecs,elbv2}/awsapi` in `spinifex#1273` with no API, NATS subject, storage or behaviour change; the root `gateway` dispatch that still imports them is recorded `gateway` (service registration) debt. |
| `gateway/spx` | `promote_image.go` and `spx.go` dispatch (`storage.go` already consumes `contracts/operator/v1`). | Needs decision: which package owns operator-plane action dispatch registration — an `ingress/aws`-style seam for the operator plane, or `gateway` stays its home. |
| `gateway/tagging` | — | Done. `domains/tagging` owns the read-only cross-resource `GetResources` aggregation and its API semantics (`0d830e901`, `aedd0b2c4`); AWSGW composes the owner listers, the EC2 tag index and ELBv2 (`f3545e3ea`); the root `gateway` keeps only the legacy dispatch in `gateway/tagging.go`. The package is removed, and its two allowlist rows and frozen-legacy entry are gone. Owner tag mutation capabilities remain future direction, so the RDS and ECS tag-writing edges above stay open. |
| `gateway/spx` | — | Done. The gather helpers, storage-status config and image-promote request moved to `operator/admin` with no behaviour change and the package is removed. The `spinifex` Query dispatch and the `/admin/{method}` handlers that call them now live in `operator/admin` as one seam registration. |
| `gateway/tagging` | — | Done. `domains/tagging` owns the read-only cross-resource `GetResources` aggregation and its API semantics (`0d830e901`, `aedd0b2c4`); AWSGW composes the owner listers, the EC2 tag index and ELBv2 (`f3545e3ea`); the dispatch moved to `domains/tagging/awsapi` and the root `gateway` no longer serves Tagging. The package is removed, and its two allowlist rows and frozen-legacy entry are gone. Owner tag mutation capabilities remain future direction, so the RDS and ECS tag-writing edges above stay open. |
| `services/viperblockd` | — | Done. The adapter is `providers/ebs/viperblock` (with the read-only `dirty` reader), the shared vocabulary is `contracts/viperblockd/legacy/v1`, and the role is `runtime/roles/viperblockd`. `services/` is empty. |
| `utils` | — | Done and deleted. `ec2_subject.go` went to `contracts/ec2/v1`, the vpcd protocol to `contracts/network/v1` with the typed client in `domains/network/projection`, and the master-key loader to `domains/ec2/ebs/encryption`. |
| `vpcd` | — | Done. The IMDS local-file reader and record-space opener moved to `domains/ec2/instance/state` and are injected through `guestmetadata.LocalStateReader`/`RecordLoader` (recorded above), so `vpcd` no longer imports `daemon` directly; its inter-process boundary now exists (`contracts/network/v1`, with `domains/network/subscribers` as the consumer and `domains/network/projection` as the producer client). The package moved unchanged to `runtime/roles/vpcd` (recorded below), and the legacy directory is removed. |
@@ -696,7 +713,7 @@ ADR-0001 completion is structural, so none of these is cleared by remediating se
| One narrow missing boundary | Done: `handlers/sts` EKS JWKS lookup → `handlers/eks``ClusterJWKSResolver` (`254d898a3`); `gateway/tagging` → `domains/tagging` with AWSGW-composed listers (`0d830e901`, `f3545e3ea`, `aedd0b2c4`). Open:`gateway/spx`(owner of operator-plane dispatch registration) | Each needs one decision or one capability, then a mechanical move. |
| One narrow missing boundary | Done: `handlers/sts` EKS JWKS lookup → `handlers/eks``ClusterJWKSResolver` (`254d898a3`); `gateway/tagging` → `domains/tagging` with AWSGW-composed listers (`0d830e901`, `f3545e3ea`, `aedd0b2c4`);`gateway/spx`→ `operator/admin`. Done: the `spinifex` service and `/admin` dispatch registration, owned by `operator/admin` | Each needs one decision or one capability, then a mechanical move. |
| Broader boundary or follows another package | `gateway` root (ADR-0002 auth and request-context phase) | Recorded debt with removal conditions above. |
Per-consumer IAM capabilities are the widest unblocker: they gate STS's IAM reads, Ochre's IAM reads, the ELBv2 role path and the ECS, EKS and RDS ensurers.
@@ -236,12 +236,12 @@ Each stage runs to empty before the next begins, so nothing is deleted while som
A large account takes minutes to tear down. The API does not hold the connection open for it: `DeleteAccount` starts the job, returns a `deletionId` immediately, and progress is read from `DescribeAccountDeletion`.
`spx --remote` starts the job and then polls, printing each stage as it lands. Your own client can do the same, or start the job and check back later — the deletion record outlives the account, so `DescribeAccountDeletion` stays answerable after the account is gone.
`spx admin account delete` always goes through the admin API, so it signs with the standard AWS credential chain. On a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command cannot find credentials), or equivalent credentials for an IAM user holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`, and a reachable gateway. The older `--remote` flag is accepted and ignored. The command starts the job and then polls, printing each stage as it lands. Your own client can do the same, or start the job and check back later — the deletion record outlives the account, so `DescribeAccountDeletion` stays answerable after the account is gone.
One job runs per account. A second `DeleteAccount` with a different token while one is running returns `OperationInProgress`; with the *same* token it returns the running job, which is what makes a retry safe.
@@ -155,8 +155,11 @@ Restarting the daemon is what re-arms it: the node rebuilds its peer list from t
`disable` removes Spinifex's own table and the peer file and leaves every other nftables table alone. It is idempotent, and it is the recovery path — it runs even on a node with no policy installed, so it cannot leave you with a node you can neither arm nor disarm.
`disable` leaves `table inet spinifex_conntrack` loaded, and the boot-time unit loads it on every node whether or not the policy is armed. It accepts everything and exists only to keep connection tracking running. Without it, a connection opened while the policy is off, such as an OVN RAFT link formed during that window, is first seen mid-stream when the policy comes back, and its replies are dropped as new connections. Do not delete it.
> [!WARNING]
> Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
> On Oracle Linux with IPsec on, a flush also removes `inet spinifex_ipsec_guard`, so Geneve to a peer without an SA goes out in clear until the daemon puts the table back, within 60 seconds.
## The Files
@@ -170,6 +173,7 @@ Everything lives under `/etc/spinifex/firewall/`. Which files are safe to edit f
| `open-ports.nft` | The apply helper — the formation window only | No. Reset closed on install and upgrade |
| `custom.nft` | **You** — `trusted_ssh_peers` and anything else you add | **Yes.** Created once, never rewritten |
| `mode` | `setup.sh` — whether the install path armed the policy | No |
| `conntrack.nft` | `setup.sh` — the tracking table, loaded armed or not | No. Edits are overwritten |
The daemon cannot write `/etc` itself, and a blanket `nft` grant would be root-equivalent, so changes go through `/usr/local/lib/spinifex/spinifex-firewall-apply` — a root helper that accepts a fixed set of verbs and re-validates every address handed to it.
@@ -189,10 +193,10 @@ Dropped packets are logged, rate-limited to five a minute, so this tells you whe
sudo journalctl -k | grep 'spinifex-fw drop'
```
An empty log with a connection still failing means the policy is not the cause. A node with no table loaded at all is not armed:
An empty log with a connection still failing means the policy is not the cause. A node with no peer file is not armed, and its boot-time unit loads only the tracking table:
```bash
systemctl is-enabled spinifex-firewall.service
sudo test -e /etc/spinifex/firewall/peers.nft && echo armed || echo not armed
```
## Troubleshooting
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
joshsiv-mulga
