An ocibsd API-key run seeds spinifex.toml with the credential before init,
so a non-empty file no longer shows that formation ran, and a checkpoint
left by an earlier run skipped init on fresh nodes.
spx get nodes draws its columns with box characters, so the Ready count
split on '|' never found a node. A multi-node leader's config lists the
other nodes without EBS tables, so read the cluster_id only from tables
that carry one.
The node policy and the API-key policy statements granted manage
volume-family and use instances to every deployment, although only
ocibsd touches OCI volumes. They now follow ebs_provider, and
setup-identity.sh takes --ebs-provider so the shared policy can carry
them; validate-topology.sh passes the topology's provider.
A publish on a remote block backend waits for the attachment and the
device it yields, which can take minutes, so the 30s request floor turned
a slow but successful attach into an ambiguous mount failure. Publish now
gets PublishTimeout (5m) on the NATS client, and ocibsd bounds a whole
publish 30s short of it so it answers retry-later before the client gives up.
The ocibsd topologies set no EBS provider and check that init detected it on
the OCI VM, with the placement the state reports; the other topologies set
SPINIFEX_EBS_PROVIDER=viperblockd. The ocibsd unit install and the post-init
auth write are gone; an API-key run installs the credential and seeds each
node's config before init.
install-node.sh drops --ebs-provider, --oci-compartment-id and --oci-availability-domain, which spx admin init no longer takes. A SPINIFEX_EBS_PROVIDER in its environment is passed to init on the first host through env, after sudo, and joining nodes take the provider from formation. The value is joined into a remote sh -c, so anything but lower-case letters is refused.
When the OCI instance metadata service answers within a second with a shape that is not bare metal, the check spx admin init detects ocibsd by, setup.sh installs open-iscsi (apt) or iscsi-initiator-utils (dnf), enables iscsid.socket, and stages the tarball's ocibsd/ under /usr/local/share/spinifex/ocibsd for init or join to install.
On a node whose existing config already names ocibsd, the new ocibsd stage also reinstalls the unit, daemon drop-in and udev rule from the new stage, masks the viperblock unit install_systemd has just rewritten, and reloads systemd and udev. Any other node gets none of them.
spx admin init loses --ebs-provider, --oci-compartment-id and --oci-availability-domain. It resolves the provider once: the one an existing node config resolves to, an omitted provider counting as viperblockd; then SPINIFEX_EBS_PROVIDER; then detection, which picks ocibsd when the OCI instance metadata service answers within one second with a shape that is not BM.*, and viperblockd otherwise. Init prints the provider and where it came from.
For ocibsd the compartment and availability domain come from the metadata and auth defaults to instance_principal, with an existing config's [nodes.<name>.ebs.oci] values winning; the cluster ID is generated once as before. Before any config is written, init makes the read ocibsd makes at start with the node's credential and stops, naming the manage volume-family and use instances statements, when it is refused. spx admin join takes the provider from formation and checks its own credential the same way.
When the provider is ocibsd, init and join install the unit, udev rule and daemon drop-in staged under /usr/local/share/spinifex/ocibsd, mask spinifex-viperblock.service by a link to /dev/null, enable spinifex-ocibsd.service, and reload systemd and udev. A viperblock node is untouched.
Adds the vm-single-ocibsd and vm-multi-ocibsd topologies. They build nodes with the Block Volume Management plugin disabled, grant the node principal manage volume-family and use instances, install the ocibsd units, and init with --ebs-provider ocibsd and the state's compartment and availability domain.
An ebs provider gate checks the units and masks on every node, boots from an AMI imported onto OCI, and drives stop and start, attach and detach, snapshot and restore with a guest-written marker, a cross-node start on multi-node, and a terminate that leaves no tagged OCI volume or backup.
Each run records the cluster_id init generated, and e2e-oci.sh deletes every volume and backup tagged with it after the destroy and in --sweep-only, failing the run if any remain.
install-node.sh takes --ebs-provider, --oci-compartment-id and --oci-availability-domain and adds them to spx admin init. The values are joined into a remote sh -c, so each is checked against the characters an OCID or availability domain can hold.
An OCI backup takes 40-49 s, past the 30 s ceiling. The poll now backs off from 10 ms to 2 s so fast providers still settle at once and a slow backend is not read in a tight loop.
In a live run OCI's ListVolumeBackups returned one new backup twice as it
left REQUEST_RECEIVED, and ocibsd refused the snapshot read because two
backups appeared to carry its snapshot ID. The SDK client's ListVolumes and
ListVolumeBackups now keep the first entry for each OCID.
t.Context is cancelled before cleanups run, so the suite's teardown, sent over NATS, reached the provider with a cancelled context and left every volume, backup and attachment behind.
The live test runs the unmodified EBS provider conformance suite, in process
and over NATS, against the real SDK block client and one OCI instance, with
iscsiadm and the device check run on the instance over SSH. It also checks
that OCI's GB is 2^30 bytes on the attached device, and that a volume
recreated under a deleted volume's ID is never answered with the deleted one.
It skips unless SPINIFEX_OCIBSD_LIVE is set; the README lists its inputs.
spx service viperblock start exits with an error when the node's [ebs] provider resolves to anything but viperblockd, so a viperblock unit reinstalled over its mask on an ocibsd or qemunbd node does not answer the provider subjects; an unset provider still resolves to viperblockd and passes. ocibsd start shares the same check.
formation.NetworkConfig carries the EBS provider as a nil pointer with omitempty, set only when the provider is not viperblockd, so a viperblock cluster's payload is the same JSON as before the provider choice existed. A joining node that receives no ebs object writes the default provider; one that receives ocibsd still gets the provider, OCI fields and cluster_id.
build/ocibsd holds spinifex-ocibsd.service, which runs spx service ocibsd start as root with no capabilities once NATS is up, a spinifex-daemon.service drop-in allowing block-sd so QEMU opens the iSCSI disk, and a udev rule giving group spinifex read-write on disks whose by-path names an OCI attachment IQN. The release image and dev-install.sh package the directory as ocibsd/, which setup.sh never installs; the install role copies it onto ocibsd nodes only. uninstall-spx.sh removes the drop-in, which its unit globs miss.
spx admin init with --ebs-provider ocibsd generates a random 128-bit cluster ID once, writes it as [nodes.<name>.ebs.oci] cluster_id, keeps an existing one across a --force re-init, and formation carries it to every joining node with the provider choice. Any other provider writes no cluster ID and renders the same spinifex.toml as before. ocibsd tags every OCI resource with it in place of a value derived from the cluster CA, which would change if the CA were reissued, and refuses to start without it; config validation refuses nodes that carry different IDs.
runtime/roles/ocibsd reads the node's OCI instance, refuses a node outside the cluster's availability domain, starts the provider and serves it with this node's per-node subjects; the cluster tag is derived from the cluster CA. service.New and spx service gain an ocibsd arm. [nodes.<name>.ebs.oci] carries compartment_id, availability_domain and the credential keys, validated with the external pool's OCI auth rules and required to agree across nodes. spx admin init gains --ebs-provider, --oci-compartment-id and --oci-availability-domain, and formation carries the choice to every joining node.
The real Initiator runs iscsiadm through an injectable command runner, writes every node record with manual startup and a 120 s replacement timeout before logging in, and waits for the by-path device. Provider.Start reads the compartment's volumes, then logs out only sessions to this cluster's OCI volumes that OCI no longer attaches to this node; PublishVolume is refused until it has run.
The image import and the SQS store volume publish at the latest contract
version, since a block export has no version 1 form, and write through a
shared imagewrite.Writer. It dispatches on the export's kind: an nbd export
goes to the NBD writer, and a block export is written to its device path with
qemu-img convert -n, refusing an image larger than the export. The SQS
preparer's writer takes the published export and is supplied by the daemon.
Lifecycle publishes use the latest contract version. Per-version publish
cases require a provider whose latest export is a block device to refuse a
caller below the descriptor version, and ocibsd now refuses such a publish
before attaching. The ocibsd suite runs over NATS against the fakes.
BlockClient covers what an EBS provider on OCI block volumes needs: volume
create, get, list by tag, update and delete; iSCSI attach, get, list and
detach; and incremental backup create, get, list by tag and delete. Creates
carry an opc-retry-token, and attachments are never shareable.
BlockFake models the behaviours the provider has to handle: the 50 GB
floor, retry tokens, replies lost after the call took effect, the 409s on
deleting a volume or a backup while a backup is in progress, backups that
stay CREATING until a test settles them, attachment state transitions and
the 32 iSCSI attachments per instance.
CopySnapshot is now optional behaviour. A provider that leaves SnapshotCopy
false refuses it as unsupported_capability, the conformance suite runs its
copy cases only when it is advertised, and AWS CopySnapshot answers
UnsupportedOperation on the refusal. viperblockd, qemunbd and the memory
reference set advertise it.
AWS CreateVolume and the root volume create send the volume type in
CreateVolumeRequest.Parameters, which the existing providers ignore.
* feat: add EBS contract version 4 with the snapshot filter and state
CreateSnapshot may answer pending, and a filtered ListSnapshots reads one
snapshot and its state on every provider. The NATS client returns a pending
completion, and a viperblock repeat of CreateSnapshot returns the first
snapshot instead of retaking it.
* test: cover pending snapshots and the snapshot filter in conformance
The conformance suite polls the filtered read until a snapshot completes,
covers idempotent CreateSnapshot, an absent snapshot and the filter below v4,
and runs the memory provider in pending mode in process and over NATS.
* fix: answer a repeated viperblock CreateSnapshot from the stored snapshot
A repeat is answered from the stored snapshot before the volume is opened,
so it still returns the first snapshot after the source volume is deleted,
and it reports the creation time the snapshot config records. An identity
that cannot be read is now internal rather than already_exists. The
conformance suite repeats CreateSnapshot after deleting the source volume.
* feat: record a pending snapshot before the provider call and settle it in a reaper
CreateSnapshot takes the snapshot's claim, writes the volume reference and a pending document, then calls the provider. A definite refusal of the first call removes both; an ambiguous failure repeats the call once, and nothing after the first call deletes.
PendingSnapshotReaper settles each pending document from the provider, or repeats the call under the claim, taking it over only from a node whose heartbeat has stopped. A starting daemon releases its own node's claims on pending documents before serving requests.
* test: cover pending snapshot creation, the reaper and restart release
Includes the ADR-0003:S7 invariant that the pending-snapshot reaper deletes no snapshot, volume or AMI.
* fix: write the pending snapshot document before its reference and repeat with the volume handle
CreateSnapshot writes the pending document before the volume reference, so a crash between them leaves a document the reaper discovers rather than a reference that blocks DeleteVolume. The reaper restores a missing reference before it settles a document, and drops it again if the document was deleted meanwhile.
The reaper's repeat CreateSnapshot sends the source volume's provider handle, or none when the volume document is gone. A starting daemon releases its node's claims on snapshot documents in any state.
* feat: keep RDS snapshots creating while their EC2 snapshot is pending
A DB snapshot whose EC2 snapshot answers pending stays creating, and the reconciler settles every creating record that names its EC2 snapshot: completed makes it available, error makes it failed with an event. resolveCreatingSnapshot adopts the snapshot ID and applies the same outcomes.
A deletion stays in deleting while its final snapshot is pending, and the wait does not count toward transitionTimeout. A final snapshot that errors keeps the instance in deleting with an event and its data volume kept. A teardown waiting on a cut final snapshot no longer stops the engine again.
* feat: wait for a pending snapshot before registering an imported image
When the provider answers the import's CreateSnapshot pending, the importer reads the snapshot with the filtered ListSnapshots every five seconds within imageImportTimeout, and fails on error, on its disappearance or on the timeout. The snapshot and AMI documents are written only once it completes.
* refactor: give the pending snapshot reaper its scope in the daemon
The snapshot domain imported runtime/compute/vm only to name its reaper's
scope, which ADR-0001 rule 5 forbids and which made the vm tests an import
cycle. The daemon now wraps the reaper as cluster-wide.
* fix: fail a creating DB snapshot whose EC2 snapshot is gone
A DB snapshot, or an instance's final snapshot, whose EC2 snapshot document
no longer exists now settles failed with an event naming the missing
snapshot, instead of staying creating to be retried on every pass.
* feat: defer the delete of a snapshot the provider may still hold
DeleteSnapshot on a pending snapshot, or one whose create went unanswered,
marks it under the snapshot's claim and returns. The snapshot is hidden from
then on, and the reaper deletes it at the provider once it settles, then its
reference and document, leaving a seven-day tombstone when a late create may
still land. Every create call now records AmbiguousCreate first, and the
reaper's settle and tag mirroring write under the claim.
CreateVolume, CopySnapshot and RegisterImage refuse a snapshot that has not
completed, and RunInstances and CopyImage an AMI that is not available.
CreateImage records its snapshot and AMI pending before the provider call,
through the CreateSnapshot path, and the reaper settles a pending AMI from
its snapshot.
* chore: fix lint findings on the pending snapshots branch
* fix: publish a qemunbdd snapshot file only once its copy is complete
CreateSnapshot and CopySnapshot write the snapshot to a partial file in the
snapshots directory, sync it, rename it to the snapshot's name and sync the
directory. The partial name does not end in .qcow2, so a copy interrupted by
a failure or a killed process is never listed, looked up or reported
completed, and the next create of that ID replaces it.
* fix: roll back a refused snapshot create in the reaper's order
The rollback marks the pending document DeleteRequested with AmbiguousCreate
cleared, then removes the volume reference, then deletes the document. A
failure or crash after the mark leaves a marked document, which the reaper
finishes by removing the reference and deleting the document, without
repeating the refused create. No reference is left without its document.
* fix: exclude a pending final snapshot's wait from the deletion timeout
finalSnapshotWait counts the time since the final snapshot was reserved
while it has not settled, not only once SettledAt is written, and returns a
failed read of its record instead of a zero wait. reconcileDeleting fails a
deletion past transitionTimeout only when that wait was read, so a transient
failure to read a pending backup's state retries the teardown instead.
* docs: specify pending snapshots in the EBSProvider contract
Snapshots.tla states that a snapshot's data is fixed when CreateSnapshot
returns, even while pending, adds the snapshot_id filter and the state a
ListSnapshots ref carries, and models a pending snapshot settling as
completed or error without a call. The README drops the two open questions
this answers, records that the filter and state arrive at contract version
4, and notes that the control plane defers a delete of a pending snapshot.
e2e-coverage.md records that no suite reaches a pending snapshot and where
the pending paths are tested.
* fix: serialise AMI settlement with DeregisterImage under the image's claim
The pending-snapshot reaper reread a pending AMI and then wrote it back
unconditionally, so a DeregisterImage between the two had its AMI document
recreated, pointing at UEFI objects it had deleted. The reaper now settles
each AMI under a claim keyed by image ID in the snapshot claim group,
rereading under it and writing only a still-pending AMI. DeregisterImage
takes the same claim across both deletes, with the bounded retry
writeUnderClaim uses, and answers IncorrectState while it stays held.
ReleaseNodeClaims now also frees this node's stale AMI claims at start-up.
* fix: settle CreateImage's AMI under the image's claim
CreateImage's Completed hook wrote its in-memory AMI as available without
rereading it, so a DeregisterImage after the AMI was recorded pending had
its document recreated. The hook now takes the image's claim, rereads the
AMI under it and writes available only while it still exists and is still
pending; a claim it cannot take leaves the AMI to the reaper.
* refactor: settle a pending AMI through one snapshot-service function
CreateImage's completion hook and the pending-snapshot reaper each moved a
pending AMI to its settled state under the image claim. Both now call the
snapshot service's settlePendingAMI; the request path keeps LockImage's claim
retries and the reaper keeps a single take.
* refactor: treat a CreateSnapshot answer with no state as not completed
Every provider sets Snapshot.State on a CreateSnapshot answer, and the NATS
client rejects a completion without one, so an empty state no longer counts as
completed. It leaves the document pending for the reaper, as any other answer
that is not completed does.
* refactor: settle a DB snapshot's status from one table and one emitter
EC2 snapshot state maps to a DB snapshot status in rdsSnapshotStatus, and
recordSnapshotSettled emits every created and failed event. A final snapshot
whose record already names its EC2 snapshot is settled only by
reconcileSnapshots; takeFinalSnapshot reads its status, so a deletion may wait
one more reconciler pass.
* refactor: share the reaper's claim, provider delete and settled cleanup
withClaim takes and releases a claim around every reaper and AMI settlement
step, deleteAtProvider is the one provider delete that tolerates not_found, and
cleanUpIfSettled carries out an accepted delete from a provider read made under
the snapshot's claim.
* refactor: take a snapshot claim through one acquire
lock and take share the reservation, the TryLock and its unwinding, and differ
only in whether a dead holder's claim is taken over.
* refactor: decide a repeated CreateSnapshot's outcome in one place
repeatOutcome maps an answer after an unanswered call to the provider's
answer, error, or pending for both the request and the reaper. createSnapshot
sends its call in a loop of at most two attempts, keeping the rollback for a
refusal of the first.
* refactor: build updatePending and writeUnderClaim on one rewrite
rewrite rereads a snapshot document, lets mutate decide whether to write it,
and writes it back; updatePending adds the pending check and writeUnderClaim
the claim.
* refactor: wire CreateImage's snapshot service explicitly
The image service no longer builds a standalone snapshot service when none is
wired, and the unused snapshotVolume is gone. CreateImage on an unwired service
fails with InternalError; test fixtures and the integration helper wire one.
* refactor: name AMI and snapshot states and readiness errors in ebsmetadata
AMI pending and failed sit beside available in ebsmetadata, the snapshot
domain uses its completed state, and Snapshot.NotReadyError and
AMI.UnavailableError give the IncorrectState and InvalidAMIID.Unavailable
answers the snapshot, image, volume and instance domains share.
* refactor: read one provider snapshot through ebsprovider.ReadSnapshot
The reaper, image import's wait and the conformance suite read a snapshot by
ID through one helper. Viperblock builds its completed snapshot answer once,
and qemunbdd clears a partial snapshot file with removeIfExists.
* refactor: send the snapshot domain's CreateSnapshot at the latest version
CreateSnapshot and the reaper's repeat now go at NewLatestVersioned, as the
reaper's reads do. The conformance suite's create and repeat cases are sent at
that version too, so every provider is judged on the creates it now receives.
* test: share the deregister-stays-deleted assertions of the AMI race tests
* test: table the final snapshot failures that keep an RDS deletion
A final snapshot whose EC2 snapshot ended in error and one whose EC2 snapshot
is gone are two rows of one test.
* test: seed pending-snapshot test documents through one seedDoc
seedDoc fills the defaults of a fresh pending document of the fixture's
volume and owner, and writes the reference when asked; it replaces
seedPending, the reference-writing seedDoc and the inline document literals.
* test: drop a doubled blank line from the pending-snapshot delete tests
* test: wire a snapshot service into the UEFI image S7 fixtures' CreateImage
* fix: answer a repeated CreateSnapshot by its snapshot ID when the provider has lost its source
The control plane mints each snapshot ID once, so the ID names the request. qemunbdd answers a repeat after a restart with the snapshot on disk instead of already_exists, which settled a held snapshot as error.
* fix: allow deleting a failed DB snapshot and retake a deleted final snapshot
DeleteDBSnapshot and the retention sweep remove an available or failed DB snapshot, as AWS does, deleting the record only at the revision read. A deletion's timeout counts from the later of its start and its final snapshot's creation.
* feat(lambda): add UpdateFunctionConfiguration
Change an Active function's role, handler, description, timeout, memory
and environment by compare-and-swap through the function owner, keeping
its identity, code and event source mappings. A new role is checked for
PassRole and Lambda trust, a supplied environment is sealed to the
function and bounded as a serialized object, and the handler must meet
the test-link guest's constraint.
* fix(lambda): refuse unknown tracing modes on configuration update
v7's only breaking change is the TypeBox resolver; the zod resolver code is unchanged from 6.0.2, and every form uses zod. Held at 7.0.3 by pnpm minimumReleaseAge.
* fix(ochre): stop the Bedrock reaper reclaiming after it loses its lease
A sweep now carries the Lost channel of the lease acquisition it began under
and checks it before each endpoint and before a relaunch. A reclaim is fenced
under the key's mutex: it tears down only a READY record of the same
incarnation and generation the sweep read, while the acquisition still holds,
and the READY-to-DRAINING CAS keeps the teardown exclusive. Observation writes
also require the same incarnation, so a stale sweep cannot overwrite a
relaunched endpoint at the same generation.
* fix(ochre): yield a fenced reclaim to a record moved after admission
* fix(elbv2): make the delete suspend-before-terminate test deterministic
TestLBHealth_DeleteLoadBalancerSuspendsBeforeTerminatingTheVM failed about
1% of runs because its terminate callback could run after the handler had
already deleted lbhealth.<id>, so it saw a nil record despite a durable
suspension.
The test now gates writes to lb.<id> and lbhealth.<id> after the listener
cascade until the terminate callback has run the repair reconciler against
an eligible candidate, and asserts no repair started and no reboot was sent.
Moving SuspendLBHealth below the terminate goroutine now fails every run.
* fix(elbv2): fail the delete health test when the write gate's failsafe fires
The test waited for the NATS client to report disconnected, but nats.go marks the client reconnecting before it runs the disconnect callback, so the daemon could still report cluster mode and the count was read before any reconnect. Wait for standalone mode, then for the retry count, then assert cluster mode.
* refactor: retire the viperblockd legacy/v1 subjects
viperblockd no longer serves ebs.<node>.mount, ebs.<node>.unmount,
ebs.<node>.abandon, ebs.delete, ebs.sync, ebs.config or the per-volume
ebs.config.<volume> subscription; the provider contract is its only request
route. The control plane skips a leftover EFI request on terminate instead
of sending ebs.delete for it.
mountVolume and unmountVolume return unexported result structs, keeping
Reaped so an unpublish that reaps an unregistered export still reports
TornDown. EBSRequest moves into runtime/compute/vm with its JSON tags
unchanged, the dirty-marker bucket vocabulary moves into the dirty package,
and contracts/viperblockd/legacy/v1 is deleted.
* test: pin the persisted EBSRequest JSON shape in the vm package
* fix(build): fail the microVM build on an incomplete initramfs
* test(build): cover microvm_pack_initramfs rejecting and retrying an incomplete archive
* test(ui): cover the Nodes topology's direction, node click and look
The map moves onto a shared canvas next, so its layout toggle, the click
that opens a node's panel, and both themes' screenshots are pinned first.
* refactor(ui): move the Nodes topology's React Flow shell into TopologyCanvas
The theme, fit, zoom controls, hidden attribution, hover tracing, labelled
lines and the direction toggle become one shared canvas, so other maps can
draw on it. The Nodes map renders as before.
* feat(ui): add a VPC topology tab drawn from the AWS API
The VPC panel's Topology tab maps the internet, gateways, Elastic IPs,
the VPC and its databases, subnets by availability zone and the instances
and load balancers in each, with one labelled line per route. The map is
read-only, bounded to 120 cards and 240 lines, and says when a read was
truncated, incomplete or failed.
* fix(ui): bound the VPC map's reads by page size and show the canvas in the catalogue
Every paged read the map makes asks for 100 a page, the budget fill
indexes each card's lines, the subnet click e2e checks the panel opens,
and the design catalogue shows the topology canvas.
* feat(ui): route-based subnet type, internet reachability and boxed resources on the topology maps
Subnets show public or private from their route table on the map, the list and the panel.
The Internet card is green, red or grey by whether a route reaches an attached gateway, and the
NAT gateway's outbound-only path is drawn. Instances and load balancers sit in dashed boxes, the
full-page map lays out left to right or top down, and lines stay beneath the cards.
* fix(ui): bound the topology address read, confirm routes and node answers before drawing them
Addresses are asked for by the instances and NAT gateways the map read, in page-sized batches.
A partial route table read draws only routes from a subnet's own table, the Nodes Internet card is
red only when every node in the cluster answered, the endpoint index is linear, and a side panel
can zoom out far enough to fit a wide map.
* fix(ui): fit the topology map, its controls and legend in the side panel
* fix(ui): leave room for a line's label between neighbouring gateway cards
* feat(ui): topology tabs for subnets, route tables, internet and NAT gateways, and instances
Each panel draws its resource's VPC with the network map, the resource and what it touches
highlighted and the rest dimmed. An instance's map always holds the instance and its subnet,
even past the read's page limit, and a detached internet gateway says it has no map.
* fix(ui): keep a resource's map frame and its own resource past capped reads
* fix(ui): set a route's label clear of the cards its line passes
* feat(ui): add a security group topology tab
* fix(ui): read a security group's instances by VPC and keep stacked labels apart
* feat(ui): show each security group rule's direction and the rules among its members
* fix(ui): say a peer group's VPC was not read rather than place it in this VPC
* feat(lambda): update function code from an inline ZIP
Add UpdateFunctionCode (PUT /2015-03-31/functions/{name}/code) for an
inline ZipFile. The gateway checks the function is Active with stored
code and honours RevisionId (PreconditionFailedException) and DryRun.
The owner records an owed upload, the gateway stores the new immutable
copy, and the owner swaps only the code reference and LastModified with
a compare-and-swap, so configuration, state and event source mappings
are unchanged. Competing updates lose with ResourceConflictException and
leave their copy owed. S3 code, images, Publish and arm64 are refused.
A daemon that predates the new subjects yields a retryable error and
records nothing.
* test(lambda): cover interleaved code update commits
Hold one commit after the owner has read the record and checked its
upload, let a competing commit win, then release it: the held commit
must lose its conditional write and leave its upload owed.
* test(lambda): add code update methods to the seam fake
* refactor: remove images through the EBS provider and stop deleting {imageID}/config.json
DeregisterImage deletes only the UEFI store and the AMI document. spx admin
images remove previews the image volume's capacity with GetVolume, reporting
unknown rather than failing on a provider error, and deletes the snapshot and
then the volume through the provider instead of sweeping their prefixes.
* chore: separate the embedded provider in the images remove test double
The report no longer knows the -efi suffix: UEFI variable stores live in the
UEFI store, so a leftover <root>-efi provider volume has no owner and is
reported, never deleted. OrphanVolume.Derived and its CLI note go.
* feat: carry the UEFI variable store through CreateImage, CopyImage and DeregisterImage
CreateImage copies the object a UEFI instance's store record points at to the
image's own prefix and names it in the AMI document's uefi_data. CopyImage gives
the copy its own object, DeregisterImage deletes the image's prefix before its
document, and RunInstances seeds a new instance's store from the AMI's store
when its firmware matches the node's.
* fix: copy the UEFI store before CreateImage's snapshot and refuse a store deleted mid-request
A refused UEFI copy no longer leaves a root snapshot behind, and a record
that vanishes after a live pointer was read is IncorrectInstanceState, not
an image without a store. CopyImage's rollback log names the UEFI key.
* fix: delete a system image's UEFI store before its AMI document
RemoveSystemImage deletes uefi/<ami>/ first and keeps the document when
that fails, so no store object outlives the image that names it.
* fix: drive the UEFI image-store S7 tests through the prune and the store reaper
The instance prune, the reaper after a terminate of an imaged instance, and
the reaper after a failed launch seeded from an image each leave the
image's store object in place.
* fix: say RemoveSystemImage deletes the UEFI store before config.json
* docs: describe how images carry the UEFI variable store
The startup readiness wait and the recovery retry now also require this
node's EBS provider to answer an empty HoldsUnsealedWrites request on its
per-node subject, which no other node's provider can answer.
checkViperblockReady becomes checkJetStreamReady, since it only checks
JetStream. vm.Deps gains EBSProviderReady, read by RetryRecoveryFailed
alone; BackingStoreReady and the restore orphan check are unchanged.
The provider contract states that an empty unsealed-writes request answers
none and that per-node subjects are served after startup recovery, and a
conformance case holds every provider to the empty request.
* fix: leave a deleted snapshot's storage to the EBS provider
DeleteSnapshot no longer lists and deletes the snap-xxx/ object prefix after
the provider delete. viperblockd's own DeleteSnapshot already removes that
prefix, so the control plane touched viperblock's object-store layout for no
effect. A provider failure still returns before the document delete, so the
snapshot stays visible and the delete can be retried.
The sweep's tests are retired, and a new test asserts DeleteSnapshot touches
only metadata-store keys. recordingstore now also records heads, puts and
deletes.
* test: cover viperblockd's prefix delete across listing pages
DeleteSnapshot now relies on viperblockd's prefix delete to remove a
snapshot's chunks, so the multi-page case the control-plane sweep's test
covered moves to deleteObjectPrefix.
* chore: group the snapshot partition test imports
ExpandVolume persists the new size before it returns and refuses a
mounted volume, so the best-effort reload request had nothing to reload.
viperblockd keeps its ebs.sync subscriber until the legacy subjects are
retired.
* feat: seed a UEFI instance's variable store at RunInstances instead of an EFI volume
RunInstances no longer creates a <root>-efi volume or appends an EFI
request. The launching node takes the instance's absent store claim,
creates the store record live at the claim's generation, puts its VARS
template gzip-compressed at uefi/<instance-id>/<uuid> with source=template,
and moves the pointer to it by the fenced write. It keeps the region as
vars.fd beside seeded.json, so the launch boots it without a get.
Seeding refuses an ID whose store record or claim already exists and
changes neither. A failed RunInstances terminates the instance, which
sets its store record to deleting and now releases the claim this node
holds, so the store reaper removes the store.
The EFI request paths are gone from buildDrives and the launch's in-use
marking; a UEFI instance that still carries an EFI request is refused
before anything changes.
* feat: write a UEFI instance's variable store back on write and at stop
A running UEFI guest's vars.fd is written back as a new object and the store
record's pointer moved under the instance's claim generation. The stop writes
back between QEMU's exit and the unmount and then unlocks the claim; a failed
write-back keeps the claim and marks seeded.json owed for the write-back
reaper, which retries it, or drops it when another node owns the store.
Superseded objects are pruned, keeping the pointed object and anything under
10 minutes old.
* fix: decide every UEFI write-back skip from vars.fd and leader-read records
A stop, a drop and the reaper compared only the events the writer had
consumed, so a final guest write still queued at the stop was left behind an
unlocked claim. seeded.json and the writer now carry the SHA-256 of the
region the pointer holds, and each skip compares vars.fd with it. A writer
that cannot watch vars.fd polls it every 10 s instead of waiting for the stop.
An owed copy is dropped only on proof another node owns the store: a claim
at a later generation, or a fenced or deleting record. MovePointer and the
prune read the record from the stream leader, and an older claim generation
is a lagging read that keeps the copy.
* chore: separate embedded fields in the UEFI write-back test fakes
* fix: never mark a store deleting that a refused seed did not create
A seed refused because a store already exists under the instance's ID left
RunInstances' failure path to mark that store deleting. The instance now
records that its seed has not created the store, and terminate leaves such a
store alone.
* fix: release UEFI claims over clean copies and recover lost writers
The write-back reaper unlocks a claim this node holds over a stopped
instance's clean copy, so a failed unlock at stop or a restart that left no
writer no longer blocks starts elsewhere. It also adopts a running guest
whose reconnect could not start a writer. A launch that fails after QEMU
started settles its store as a stop does once that QEMU has exited.
* docs: name the missing store record among the UEFI launch refusals
* docs: add the UEFI variable store operator guide
* fix: keep a stopped UEFI store's claim until its QEMU is proven gone
* test: add the uefistore e2e suite for guest UEFI variable durability
A real UEFI guest writes EFI variables through efivarfs, and the suite proves
each one reaches the object the store record points at, survives stop and
start and a daemon restart under the running guest, and that the prune keeps
the pointed object and every object younger than 10 minutes until the prefix
returns to one object (ADR-0003:S7).
* fix: keep UEFI write-back running until a stopped guest's QEMU is gone
A stop whose QEMU is not proven gone now leaves the instance's writer
registered and running with its claim held, so a live guest's variable
changes keep landing. The write-back reaper settles that writer as a stop
does once QEMU is proven gone and no guest may run here.
A reconnect that cannot read the store claim now retries every 5 s on the
UEFI clock until a writer is registered, the claim is held elsewhere, the
guest stops running here or the daemon shuts down, so a running guest is
back under write-back within the loss budget. Reaper adoption stays the
backstop.
* fix: keep the UEFI store e2e suite's NATS connection and write variables without seeking
The store view's connection closed with the first test, failing the next two, and dd's seek is rejected by efivarfs.
* feat: claim the UEFI variable store and fence its pointer by generation
The store record now carries the boot pointer and the claim generation that
may move it. A launch claims the store through kvsync, takes it over only once
the holder's heartbeat is LivenessWindow old, stamps the record with its
generation, and boots from the pointer it read after the stamp. A record above
the claim's generation refuses the start as retryable and changes nothing.
The store reaper removes the claim before it closes the record.
* fix: refuse a start whose UEFI store a live node holds as InsufficientInstanceCapacity
* fix: keep a UEFI claim guarding a local copy and refuse on an undated heartbeat
A launch that fails after adopting its claim keeps it while seeded.json
names the claim's generation, or cannot be read. A heartbeat that exists but
cannot be dated refuses the takeover. A stamp at the record's own
generation is written by CAS, so a follower's stale pointer is never booted.
* fix: check UEFI firmware before the claim and never overwrite a copy the claim guards
A start checks the pointer's firmware before it touches the claim, reuses
vars.fd whenever seeded.json names the claim's generation, and refuses as
retryable, keeping the file and the claim, when seeded.json cannot be read.
A rollback releases the claim only when QEMU never started and no local
copy may owe a write-back. An EFI-volume instance reads no store record,
and a pointer move to the current key is still confirmed by CAS.
* fix: confirm a stale or absent UEFI holder heartbeat with the stream leader before takeover
* docs: say only a decided terminate marks a UEFI store deleting
* fix: keep a UEFI copy and its claim when vars.fd cannot be checked
* feat: add kvsync claims and leases over a compare-and-swap store
kvsync provides Claim and Lease, each with a group for per-resource keys,
and RunElection, over a three-method KVStore interface. Every acquisition
returns a token whose generation is the revision of its own write. A lease
expires by its contenders' local observation on a monotonic clock, renews
every ttl/3 and is lost after 2*ttl/3 without a landed renewal. Keys are
registered per store value and a collision panics.
kvstore.Bucket implements KVStore with Load and CompareAndSwap, and
OpenSyncBucket returns the spinifex-sync bucket every lock is built on.
* refactor: port the leader leases to kvsync and delete kvlease
Every leader election now runs a kvsync Lease in the shared sync bucket,
constructed once by the role that owns it. The two vpcd sites and the two
DNS sites share one Lease, so they exclude each other on one node too.
The daemon deletes the nine per-caller lease buckets at start, and
kvlease is removed.
* fix: never recreate a lost bucket through the first-use open
A failed recovery dropped the handle, and the next call's first-use open
created the bucket again, which RecreateIfMissing=false forbids. For the
sync bucket that restarted revisions under live tokens.
* fix: count a lease's validity from the write's send
A peer can see a revision as soon as it commits, so validity measured from
the reply could outlast a takeover. A grant whose reply took two thirds of
the TTL is now refused.
* fix: forget EKS cluster leases before a same-name cluster can take them
The reconciler lease is forgotten while the DELETING record still blocks a
create, and the teardown lease before the teardown gate opens, so cleanup
can no longer release a recreated cluster's lease.
* test: check the sync bucket in the KV quorum suite
The suite looked for the per-caller lease buckets, which the daemon now
deletes. It now requires the sync bucket at full replication and none of
the old lease buckets.
* fix: open a bucket without holding its lock across the network
KV and Reopen held the bucket mutex across the open, so a caller behind a
stuck open waited with no context to stop it. One open is now in flight at a
time and each waiter stops at its own context.
* fix: lose a lease at its deadline even when the renewal is stuck
A renewal CAS that blocked past its context kept the lease reported held
after a peer could take over. Each renewal is now given up at 2*ttl/3 from
the last send, and every renewal is scheduled from that send rather than from
install.
* fix: retry a held per-pass lease once a dead holder can be taken over
A pass that lost the election waited for the loop's next resync, so a dead
holder blocked DNS, vpcd drift and Route 53 for up to two intervals. A lost
pass now asks to run again just past one TTL. vpcd's one-shot startup
election waits up to a minute for JetStream quorum, as it did before.
* fix: keep contending for an EKS cluster's lease while it is live
The reconciler asked for its lease once, and a fresh lease only takes over a
foreign key after watching it for a TTL, so a dead holder's key left the
cluster unreconciled for good. It now asks every ttl/3 while the cluster is
CREATING or ACTIVE, contends again after a loss, and watches the Lost channel
of the acquisition it was given.
* test: per-service leaders stop on a taken-over lease and resume after re-election
* test: cover lease gating, teardown-lease retirement, obsolete buckets and kvsync edge cases
* fix: confirm a claim's adoption by CAS so a stale replica read cannot adopt a taken-over key
* fix: Unlock abandons an in-flight lease renewal instead of waiting it out
* fix: wait out a forming JetStream at start and retry a held pass within a renewal interval
* chore: sort the reconcile lock imports for gofmt
* refactor: trim the kvsync lease internals and their callers
Drop the lease's separate stop channel, the election's held flag, the
reconcile try-leader glue, the unused ErrLeaseHeld, Lease.Holder and the
EKS lease-group fallback, and route constructor checks through one panic.
* refactor: port the ELBv2 data-path leader to kvsync
The data-path reconciler that landed on dev elected its leader through
kvlease in its own spinifex-elbv2-leader bucket. It now holds the
elbv2.datapath lease in the sync bucket, constructed with the daemon's
other leases, and the old bucket joins the set deleted at start.
The RDS delete tests that landed alongside it build their reconciler
through the lease-taking constructor.
* test: give the integration service daemon's EKS service its lease group
* refactor: fold kvutil into kvstore and rename Claim to Take
kvutil was the layer under kvstore, so its bucket, replica, version, key
and CAS helpers now live in kvstore and every importer changes only its
import path and qualifier. kvutil.Claim becomes kvstore.Take so that it
does not read as a lock claim, and the kvutil package is deleted.
The bucket-creation invariant now exempts kvstore/buckets.go, the one
file holding the chokepoint, rather than the whole package.
* refactor: merge the kvstore test exports into one file
* refactor: move the kvutil callers added on dev to kvstore
* fix(eks): alarm on a control-plane VM with no cluster record instead of terminating it
* chore(eks): order test imports
* feat(eks): add spx admin eks orphans and share the cluster-record judgement with the billable reaper
* fix(eks): list orphan control planes through an EC2-owned inventory capability
* test(eks): cover the orphan report ENI reader and the inventory fail-closed path
* fix(eks): report only running control planes and treat a missing ENI as gone
* fix(eks): open DNS watch buckets attach-only so a deleted account bucket is not recreated
* fix(eks): read control-plane ENIs from their record so a corrupt one is unreadable, and report the zone
* feat(lambda): record SQS event-source mappings behind Create, Get, List and Delete
Create refuses every option this build does not implement, reads the
queue as the function's execution role and requires the queue's
visibility window to cover the function's run limit. Delete records
Deleting for the poller to finish. Get and Delete authorize against
the mapping ARN.
* feat(lambda): poll mapped SQS queues in test-link gateways
The poller receives and deletes as the function's execution role and
invokes through the internal executor. A batch is deleted only after
the function succeeds; an error, refusal or unknown outcome leaves the
messages to reappear.
* test(lambda): give the poller tests their in-package reason
* fix: derive the mapping UUID so duplicate creates collide atomically
Create no longer scans every mapping. The UUID is derived from the
account, Region, function and queue, so the keyed create refuses a
duplicate pair in one atomic write with no second record to orphan.
* fix: mark the poller in-process SQS request as received over TLS
The request never leaves the TLS-terminating gateway, so an execution
role that requires aws:SecureTransport must still authorize it.
* fix: renew the poller session per event and back off on failed batches
Each event now runs on a session that outlives its invocation and its
delete, so a later split event is never acknowledged on expired
credentials. A failed, refused or unknown invocation makes the worker
back off before the next receive.
* feat: page ListEventSourceMappings with Marker and NextMarker
* test: drive the poller through InvokeEventSource on invalid and busy outcomes
* fix: skip locally held management addresses when allocating
The shared record can lose an entry an instance on this node still holds. Allocate now treats every address in the local cache as taken, refuses cached or recorded hits that another local instance holds, and resets its results on each CAS attempt.
* fix: refuse ListEventSourceMappings until listing can be bounded
The mapping store can only be listed whole, so a list request would
wait on the whole inventory. Refuse it, and skip repeated keys when the
gateway lists mappings at startup.
* feat: add RebootSystemInstance for load balancer VMs
Add a reboot call to SystemInstanceLauncher with a NATS client and a
daemon local and routed implementation, subscribed next to terminate.
It refuses unless the VM is an ELBv2 system VM in the system account,
reusing the ownership classifier, and never terminates.
* test: cover the system reboot client and routing failures
Add tests for the NATS reboot client and the routed reboot paths: a
missing connection, an undecodable reply, a refusing owner, a malformed
subject and the idempotent subscription.
* feat: probe each LB data NIC's gateway and report it in the heartbeat
lb-agent sends an ARP request for each data NIC's gateway on an AF_PACKET
socket bound to it, counts consecutive misses (failing from the third),
and adds the instance ID and per-NIC probe result and counters to the
heartbeat as optional fields. The daemon decodes an older agent's
heartbeat unchanged. The microVM image keeps af_packet, loads it before
lb-agent starts, and fails the build when the module is absent.
* fix: bound the lb-agent instance-id lookup and ARP receive wait
The metadata lookup now runs under one 1s deadline, waits at most that
long inside a tick, and after a failure is retried once every 6 ticks, so
an unreachable metadata service cannot push a heartbeat past one tick.
The ARP probe resets its receive timeout to the time left before each
read, so a non-matching frame cannot extend one exchange to twice the
wait.
* feat: record LB data-path evidence and report active_impaired
Heartbeat NIC evidence is stored under lbhealth.<lbId>, written by CAS only
when the set of failing NICs changes and only when the agent's instance ID
matches the LB's current VM. DeleteLoadBalancer and SetSubnets suspend the
record before touching the VM or ENIs; SetSubnets lifts it on the new VM and
DeleteLoadBalancer removes it. DescribeLoadBalancers reports active_impaired
while a NIC of the current VM is failing.
* fix: resume LB repair when DeleteLoadBalancer fails before the VM is touched
* feat: repair an LB VM whose data NIC fails its gateway probe
* test: e2e subtests for LB data-NIC self-heal and the EKS endpoint NIC
Add a lbrecovery subtest that downs a data NIC on an internet-facing NLB and an internal ALB through QMP and watches the repair, with S7 checks on the target and an unrelated customer instance, plus an EKS endpoint NIC case. List both in the e2e coverage index.
* test: give the endpoint NIC self-heal check its own private-endpoint cluster
The shared cluster has no private endpoint, so the check had no endpoint NIC
to disable. It now builds its own cluster with private and public access,
finds the guest NIC by the endpoint ENI's MAC through query-rx-filter, fails
if no guest NIC matches, and deletes its cluster on cleanup.
* fix: make data-path repair reservation, leadership and suspension converge safely
The reservation callback now resets its admission state on every CAS retry,
leadership is checked before the reservation and again before dispatch, and a
suspension left by a failed resume converges through the heartbeat of the VM
the load balancer names, with a 30-minute backstop. An established failure
clears only on a clean probe, and a heartbeat never rebinds a bound record.
* test: bound post-repair traffic by one deadline from fault injection
* test: check the internal ALB self-heal from a client inside the VPC
* test: log the guest clock offset in the internal ALB self-heal check
* test: link the LB repair invariant tests to ADR-0003:S7
* test: require a client failure after the fault before the recovery counts
* feat(lambda): deliver guest configuration on a disk the agent erases
* feat(lambda): boot test-link attempts under Firecracker with a kernel hard kill
* fix(lambda): detect the configuration disk by its missing fw_cfg entry
* test(lambda): cover the configuration-disk seal from the agent and state in-package reasons
* fix(lambda): release a Firecracker attempt whose supervisor never started
* feat(lambda): activate test-link functions once the node serves the pinned image
In lambdatestlink builds the function owner moves a Pending function with
stored code to Active when this node's daemon serves the pinned runtime
image and the cluster has one node. The pass runs at start, after each
code-stored completion and periodically, and writes by compare-and-swap.
Release builds set no readiness, so nothing leaves Pending there.
* build: install the test-link Lambda daemon, image and initializer
install-lambda-testlink puts the outputs of build-lambda-testlink-image and
build-lambda-testlink in /usr/share/spinifex/microvm/lambda-testlink and
/usr/local/bin, where the tagged daemon and gateway read them, and restarts
the services. Controlled-test only; nothing default changes.
* fix(lambda): keep test-link functions the executor cannot run Pending
A function with environment variables or a code copy over the test-link
bound is refused by the executor on every Invoke, so readiness no longer
activates it and names the unsupported setting as its StateReason.
* build: drop install-lambda-testlink in favour of the deploy role
A target stop and restart can leave members running the old spx. The
Ansible deploy role already stops, waits, swaps and restarts any member
still on a stale binary, so the test-link install goes through it.
* docs: describe test-link activation and its install path in the Lambda contract README
* test(lambda): run the activation tests from an external package
The wake test now waits for the start pass before storing code, so only
the code-stored wake can activate the second function.
* test(lambda): mark the readiness test as in-package with its reason
* fix(lambda): refuse handlers AWS refuses and keep ones the test link cannot run Pending
* fix(lambda): stop reporting the test link as serving once it stops accepting
* test(lambda): enable readiness after the first function is stored so no wake is buffered
* revert(lambda): leave CreateFunction handler validation unchanged in this slice
* test(lambda): call ServeControl with the fixed frame limit from dev
* test(lambda): hold a daemon invocation while serving ends on listener failure
* fix(rds): run one teardown of a DB instance delete at a time
The handler that accepted a delete now owns its teardown for its request
budget, a repeat of an accepted delete answers deleting without a second
teardown, and the reconciler resumes only a delete older than that budget.
Two teardowns racing let the handler see the VM mid-shutdown as gone and
answer 500 on the still-attached data volume.
* fix(rds): end the delete handler's teardown exactly when its claim does
The handler's deadline now derives from the persisted transition stamp the
reconciler measures from, and a teardown past that deadline starts no
terminate, since the graceful engine stop swallows the cancellation.
* feat: add owner-routed commands for instance metadata options and termination protection to the ec2.cmd contract
* feat: dispatch metadata-options and termination-protection commands on the owner
* feat: route ModifyInstanceMetadataOptions to the instance owner first
* test: pin the N-1 rejection, account isolation and timeout-after-apply for owner routing
* feat: route DisableApiTermination changes to the instance owner first
* test: pin the N-1 rejection with the previous release's command type and exercise the real stopped fallback
* feat(lambda): declare test-link control-socket payload and frame bounds
Replace the NATS-derived control-socket limits with a raw 6 MiB payload
bound and a frame bound that leaves 64 KiB beside a full base64 payload.
The daemon admits at most two connections from accept to close, closes
admitted connections on shutdown, and the gateway's dial stops at its
reply deadline. The control name moves to /3 so mixed limits never meet.
* fix(lambda): join test-link close-on-cancel callbacks before releasing
context.AfterFunc's stop does not wait for a callback already running, so
a connection's close could still be in flight after its admission slot
was released and Serve returned. Join a started callback first, and test
the join, a stalled peer at shutdown and cancellation racing completion.
* fix(lambda): refuse test-link requests whose guest frame cannot fit
The control frame and the guest invocation frame bound different fields,
so a request the control socket accepts could still overflow HostSend
after the grant started. The daemon now encodes the guest frame at its
longest deadline before Start and refuses one that does not fit.
* fix(lambda): check the test-link guest frame after taking the slot
Encode the guest frame at MinInt64, the longest int64 deadline, and only
once the busy slot is held, so a busy refusal never encodes the payload;
a fit refusal releases the slot. Bound every wait in the lifecycle and
guest-frame tests so a regression fails rather than hangs.
* test: cover test-link control frame bounds, admission and late dispatch
* test: prove 6 MiB test-link invoke through the gateway and real daemon refusals
* test: join held exchanges and check the echo runtime's own result
* chore(lambda): describe the executor request bound as the control frame
* test(lambda): prove test-link sender bounds, accepted connections and full lifetimes
The sender test uses a real presigned code URL, the longest function name
and qualifier, and the real frame limit; a 6 MiB + 1 payload is refused
before encoding, shown by bytes allocated. Admission reads the kernel socket
table, and the memory measurement covers both lifetimes through full replies.
* docs(lambda): document the /3 control socket and its bounds
* test(lambda): sample heap only across the measured lifetimes and join the sampler
The sampler now starts after the post-GC baseline and stops after both
replies are read, and cleanup joins it if the test fails first.
* test(lambda): check the /2 control socket cannot reach /3 in either direction
* feat(ec2): read what compute and storage an account still holds, and serve teardown-outstanding over NATS
* feat(accountlifecycle): repeat compute and storage cleanup inside the teardown job within a bounded loop
* test(ec2): AccountTeardownOutstanding fails closed on no responder, error reply, bad reply and timeout
* refactor(teardown): remove the legacy instance, volume and spot reapers
* feat(teardown): bind the EC2 compute and storage cleanup into account deletion
* chore(teardown): drop dead test helpers and fix lint after the reaper removal
* chore(layering): drop the paid accountlifecycle to awsapi/volume allowlist entry
* fix(teardown): re-read compute and storage before account removal, list them in the dry run, leave instances alone while a compute owner is stuck
* test(teardown): engine-level S7 over the real cleanups for an expired record with an attached volume; bound the lost-fence test
* feat(teardown): log each convergence pass
* fix: hold the test-link slot when the grant start outcome is unknown
* feat: bind the pinned runtime image digest to each test-link Invoke
The gateway carries the daemon's image digest pin and sends it in every
request; the daemon refuses before Start when it is missing or differs. The
control socket moves to a versioned name so mixed gateway and daemon builds
never connect.
* test: tie the mixed-version control test to the production socket name
The gateway logs a missing or malformed image pin once at startup.
* test: cover withdraw faults, image replacement and the digest's share of the request bound
* test: assert the at-limit request is exactly the probed size
* test: send the prepared digest so owner-refusal cases reach the owner
* docs: catch up DESIGN, COMMANDS, scripts, networking, security and EC2 coverage READMEs with merged behaviour
* docs: qualify host route convergence by network mode and shared CIDR
* docs: correct the account exercise, node-facts and UEFI store descriptions
* docs: describe CreateSnapshot by provider error mapping, not as refusing attached volumes
Tests that touch no process-wide state now call t.Parallel(). Tests that
set package vars, env vars, the slog default, OTel providers or the
hot-plug seams stay serial. Each createTestDaemon fixture gets its own
node ID, and tests serving fixed API subjects use a private NATS server.
* feat(sts): revoke one Lambda environment session by scope and revision
* fix(sts): keep the access key out of Lambda session revoke errors
* feat(lambda): add the test-link control socket with peer UID checks
* fix(lambda): bound the test-link reply wait and keep cancellation sticky
* fix(lambda): cap the test-link write by the reply bound and refuse a zero bound
* test(lambda): bound the control-socket peer waits and surface peer errors
* feat(lambda): issue and revoke the execution session around each test-link Invoke
* feat(lambda): serve test-link requests on the control socket and deliver the execution credential to the guest
The daemon answers the gateway on the abstract control socket instead of
NATS and hands the guest the execution-role credential in its first frame.
Test-link builds negotiate spinifex-lambda-testlink/2 on both ends, so a
mismatched host and guest fail the handshake before any request is sent,
and the agent refuses a frame without a credential.
* test(lambda): run the composed test-link seam on real STS and a real control socket
The seam now issues and revokes sessions through the real STS service and
reaches the daemon's attempt steps over a real abstract control socket.
New cases show the guest's credential resolving as the role session with
its Lambda labels under the base ARN until Invoke returns, both orders of a
give-up revoke against the first frame, a failed revoke staying unknown
without the key in logs, a write error after full delivery being unknown
and revoked, and legacy NATS forgeries never reaching the caller.
* test(lambda): show the test-link daemon refuses only before the VM starts
A started process whose launch identity cannot be recorded replies unknown,
proven with a stub VM that is alive when its identity is read and an
injected write failure. A source check pins every refused reply in the
launch path before the VM start. The files the attempt leaves, its reply
and the log carry neither the credential nor the code URL.
* test(lambda): fail a delivered write at the control client and bound the seam waits
A test-only writer at the control client's write boundary forwards both
frames, waits until the daemon has read them, then fails the write; the
client reports an outcome that is not ErrNotSent. Seam tests now bound every
wait, open their gates and cancel their contexts at cleanup, and join the
inner exchange they start.
* docs(lambda): note the test-link ALPN protocol in the guest contract README
* test(sts): satisfy lint in the Lambda session revoke tests
* test(lambda): mark the new in-package test-link tests with their reason
* fix(lambda): report a refused test-link attempt with a failed revoke as unknown and run the handshake tests in CI
* fix(ec2): a block device mapping that does not name the root adds a volume instead of replacing it
* fix(ec2): refuse a launch block device mapping that names no device
* fix(ec2): refuse an unnamed launch device mapping as InvalidBlockDeviceMapping
Start the embedded NATS server in the calling goroutine instead of a new one. Start returns once JetStream is enabled and the client listener is bound, so ReadyForConnections passes on its first check rather than busy-spinning (timeouts up to 25ms) or sleeping 25ms per attempt.
* fix(vm): delete volumes and the UEFI store on terminate only once QEMU is proven gone
* test: lay out the terminate gate cases the way the formatter wants
* docs: say a terminate only runs on the node holding the instance record
* fix: answer an absent NAT gateway with NatGatewayNotFound
DescribeNatGateways and DeleteNatGateway returned InvalidNatGatewayID.NotFound
for an absent gateway; AWS answers NatGatewayNotFound. The account-deletion NAT
reaper now treats the new code as already gone, the code is added to the EC2
error catalogue, and CreateRoute keeps InvalidNatGatewayID.NotFound as AWS does.
* fix: answer an absent EC2 resource with HTTP 400 and ACM's absent resource likewise
AWS answers 400, not 404, for the 25 EC2 not-found codes captured on 2026-11-04.
The 24 that were 404 here now carry 400 in ErrorLookup, so the status on the wire
and the status IsTerminal and HTTPStatusForError see stay one value. Codes that
were not captured keep their 404.
ACM answers an absent certificate with 400 ResourceNotFoundException. The change
is an ACM-scoped override, so the shared ResourceNotFound code is unchanged for
any other service.
* feat: pin AWS-shaped availability-zone IDs per Region
Add a zone ID resolver: AWS's published short-code table, a derived code for
custom Regions, an az_short_code node override refused at load when invalid,
and a zoneids.<region> record written by CAS so an ID is assigned once and
never changes. Daemon start pins the node's zone and every zone already named
by a subnet record, off the request path.
* feat: report AWS-shaped zone IDs from one pinned source
DescribeAvailabilityZones, DescribeSubnets and DescribeInstanceTypeOfferings
now read the stored zone ID instead of the fixed spinifexz1 constant, so the
three cannot disagree under an az_short_code override. The old constant is
removed and is now refused as an unknown zone ID.
* feat: filter subnets by availability-zone-id
DescribeSubnets accepts the availability-zone-id filter and matches it
against the pinned zone ID of each subnet's zone. An unknown ID matches
nothing and is not an error.
* fix: cap a Region's pinned availability zones at 64
A pin that would add a 65th distinct zone is refused, and CreateSubnet
pins the zone before writing anything so a refused zone leaves no subnet
behind. Pinned zones are never removed or renumbered.
* fix: pin a subnet's zone ID before the subnet is saved or announced
* test: drive the zone ID backfill through the daemon wiring
* test: pin ACM absent-certificate status through the signed gateway
* test: use a well-formed NAT gateway ID for absent-resource cases
A run records its result before its deferred release of the cluster-wide
claim, so starting the next test the moment the status changes can still
see the claim held on a slow runner.
Every test stood up its own embedded JetStream server and IAM service and
ran serially. Run the independent ones in parallel; tests that pin the
presign clock or capture the default logger stay serial.
Seed the 1,100 zones of the realizer scale test straight into the store, concurrently, instead of through CreateHostedZone. Each create listed the account's zones and changes, so the setup was quadratic and took about 90% of the package's run time.
Measure per-op allocation over a fixed number of runs instead of through testing.Benchmark, which spent its default one-second benchtime on every assertion.
Long-poll tests waited on fixed sleeps and multi-second WaitTimeSeconds; wait on the receiver's first scan instead, shorten waits that only need to expire, and run the two held-scan receives side by side. Run the R1 conformance cases in parallel, as each has its own server.
The fake daemon in newTestHandler replied without a node ID header, so the
DescribeInstances and DescribeInstanceStatus identity fan-outs never saw the
configured node answer and each waited out its 3s deadline. Reply as node-1.
* feat: issue Lambda execution-environment role sessions in-process
AssumeRoleForLambdaEnvironment mints a customer execution-role session for
lambda.amazonaws.com, scoped to one environment. It resolves the role once,
evaluates its trust with the function's account as aws:SourceAccount, and
writes the function and environment labels in the create-only record. The
role's MaxSessionDuration does not apply to service-assumed sessions. Nothing
calls it yet.
* fix: hold Lambda environment sessions to the function owner's identity syntax
Require the aws partition, a valid region and function name in the
unqualified base function ARN, and accept the owner's function ID form. A
conformance test checks the local syntax against the owner's Create. Tests
now observe the first Create directly and check the secret of a failed mint.
* test: name the failed-mint test for what it proves
* style: keep the session record's original field groups and separate the embedded test field
* test: put the credverify session test in the external package and mark the STS issuer tests in-package
* test: cover revocation and the janitor for Lambda environment sessions
The spy recorded requests on an async subscription callback, so the
test could read it before the callback ran and see fewer than eight
requests under load. Subscribe synchronously and, on each read, flush
the connection and drain every queued request first.
* fix(vm): free a leaked volume only once its QEMU is proven gone
* fix(vm): require PID-file proof of QEMU exit even when QEMU teardown reads done
* test: drop the unused PID path helper from the leak gate tests
* fix: refuse system terminate for any instance that is not a system VM
TerminateSystemInstance and terminateSystemInstanceLocal now classify the ID
first (live, stopped or terminated record) and only proceed for a system VM.
A customer or unknown ID is answered as not found; an unreadable record returns
ErrSystemInstanceOwnershipUnknown, and every caller stops and retries on it.
* fix: keep ALB VM termination asynchronous in DeleteLoadBalancer
Restores the background terminate so a delete does not wait on VM shutdown.
* fix: release only the recorded launch allocation in the system terminate backstop
The backstop no longer releases every address recorded against an instance ID.
It releases the allocation recorded on the system VM at launch, only while that
allocation is still associated with the VM's ENI.
* fix: keep the launch address when the owner cannot read the instance record
The system terminate wrapper skips the address backstop when the owning node
answers ownership unknown, since the VM may still be running. The customer
address fixture now carries the system manager tag to prove it is never released.
* fix: refuse a customer instance as not a system instance instead of gone
A customer ID reaching the system terminate path was answered as not found,
and callers that treat not found as gone then acted on that ID (RDS ended a
stopped record from the shared store). Customer now returns
ErrSystemInstanceNotSystem, distinct from gone and from ownership unknown,
mapped back across the routed path, and every caller stops on it.
* fix: stop EKS teardown on a refused terminate and classify legacy control planes as system
EKS cluster teardown now returns at the control-plane step when the terminate is
refused as not a system instance or as ownership unknown, so the egress address,
ENIs and network a possibly running VM leans on stay for the retry.
A VM this node runs whose record is absent answers ownership unknown instead of
not found, so a caller retries rather than cleaning up under a live guest and the
backstop keeps its launch address.
An EKS control-plane VM in a customer account, as launched before EKS moved to
the system account, classifies as a system VM. Other managers in a customer
account stay customer. No customer path sets the manager: RunInstances
TagSpecifications and CreateTags on running and stopped instances only write
tags, each proved by a test.
The age-based trim kept every entry during a burst of dev pushes, so the saved cache grew to 16.5 GB unpacked and restores reached 99s. Restored entries are now backdated past Go's one-hour mtime refresh, and the trim keeps only what the run read or wrote, plus outputs a kept action entry names. A red run saves untrimmed.
* feat(ec2): add CleanupStorage to delete free volumes and clear instance, volume and spot tag residue
* fix(ec2): read teardown before deleting volumes and report volumes that appear during cleanup
* test: align the fake deleter fields
* feat(ec2): add CleanupStorage to delete free volumes and clear instance, volume and spot tag residue
* fix(ec2): read teardown before deleting volumes and report volumes that appear during cleanup
* test: align the fake deleter fields
* feat(ec2): add CleanupCompute to cancel spot requests and terminate an account's instances
* fix(ec2): do not read a missing teardown record as completion; cancel disabled spot requests
* fix(ec2): terminate nothing when the instance inventory holds an unidentified entry
* feat(ui): add a Disks tab to the Nodes panel with drives and filesystem usage
The daemon reads drives from /sys/block and filesystems from its mountinfo when it builds node facts. Usage comes from statfs on a 30 s loop outside the facts lock, each mount bounded by a 2 s deadline, so a hung mount is reported as not answering and never holds up GetNodeFacts or network-test setup.
* fix(nodes): key disk usage by mount, show hidden mounts, keep zero space
A filesystem replaced at the same path is read afresh and a stuck read of
the old one no longer blocks it. A mount covered in the daemon's namespace,
as ProtectHome covers /home, is listed as not visible instead of left out.
Usage is nested and sent whole, so a full filesystem reports 0 available.
* fix(nodes): leave the daemon's own sandbox mounts out of its filesystems
On dev-prod the daemon listed /var/lib/spinifex/spinifex, which
ReadWritePaths binds over itself, and /var/tmp, which PrivateTmp binds from
a systemd-private directory. Neither is a mount on the host.
* fix(ui): fit the Disks tables to the panel without scrolling
Drives carry the media beside the transport instead of in a column of its
own. Filesystems show one Use% cell, the bar with the used and total sizes
beneath it, in place of the size, used and available columns.
* fix(nodes): keep a filesystem mounted over another visible
Only the covered mount and what lies beneath it are hidden. The covering
mount, and anything mounted on it, inherited the hidden flag and was never
read.
* feat(ui): group Spinifex's mounts apart from the system's on the Disks tab
Spinifex mounts are its data directory, what is mounted under it, and any
path binding the same directory. Usage reads as used / size so the cell
fits the panel, and rows are keyed by mount and source.
* fix(lambda): admit one Runtime API response body and cap it early
* feat(lambda): stream the test-link result frame without whole-frame copies
* fix(lambda): bound the result write and send nothing for a lost response
* fix(lambda): end the invocation at once when an admitted body is lost
* fix(lambda): keep a posted response when the bootstrap exits and wait for host shutdown
* test(lambda): host-side outcomes for a response lost or kept across bootstrap exit
* test(lambda): bound the exit tests' waits and release the write gate on cleanup
* fix(lambda): let the test carrier's reader finish after an unread frame
* test(lambda): host-session outcomes for partial frames, late results and slow bodies
* revert(lambda): drop the test carrier's input buffer
* test(lambda): drain the harness stream and prove admission before the deadline passes
* test(lambda): order admission before the host deadline and show the guest reader finishes
* test(lambda): confirm admission preceded the host outcome
* test(lambda): bound every channel wait in the new tests
* test(lambda): mark the new in-package tests that need unexported access
* test(lambda): write the expired invocation's timeout frame over a real connection
* test(lambda): drive a bootstrap exit before any invocation through the host
* test(lambda): drain the guest stream in the before-invocation exit test
* refactor: move the durable KV contract into kvutil
* feat: add the unwired durable management IP record store
* docs: mark the store test in-package and state the freshness boundary
* test: cover stale Lookup confirmation and in-place snapshot entry mutation
* test: drive the stale Lookup case through a wrapped KV instead of a production seam
* test: satisfy lint on the wrapped KV double
* feat(ui): group launch subnets as public or private by their route tables
A subnet is public when its explicit route table, or else its VPC's main table, routes to an internet gateway. The NAT gateway form filters by the same rule instead of MapPublicIpOnLaunch.
* fix(ui): keep the subnet forms usable when route tables cannot be read
* fix(ui): encode ARN path parameters once so breadcrumbs show the ARN
Links passed encodeURIComponent output as route params, which the router encodes again, so the breadcrumb showed arn%3Aaws.... The test router mock now encodes params like the real router.
* test(ui): prove ARN routing through the real router and old bookmarks
* test(ui): type the routing test's SDK mock and keep its hooks in the describe
* fix(rds): grow and delete a DB instance whose VM is stopped with its volume attached
* test(rds): run the stopped-terminate commander test in parallel
* fix(rds): hold a delete behind an active modify and detach on terminate only the terminated instance's volume
* fix(rds): repeat the guarded detach when a stopped terminate left the data volume attached
* fix(rds): take the data volume out of a stopped DB VM's record before releasing it and terminating the VM
* fix(rds): tear down from the re-read record and keep an accepted delete when a modify fails
* fix(rds): release a stopped DB VM's data volume the way a stopped-instance detach does
* feat(lambda): add the unregistered RequestResponse Invoke path
Invoke resolves the function and qualifier with the existing lookup,
authorizes lambda:InvokeFunction on the function ARN, and hands one
synchronous attempt to an injected Executor. With no executor it fails
closed; a Pending function is ResourceConflictException. An accepted
attempt is not cancelled by a caller disconnect: only the wait is bounded.
Event, DryRun, LogType Tail and other unimplemented options are refused
explicitly. Lambda stays unregistered and no user code runs.
Also makes two flaky tests deterministic: the workerboot key-mismatch test
could pick an identical key 1 time in 256, and the loadgen first-breach
test depended on a 2ms op staying under a 30ms wall-clock p99.
* feat: track each Lambda code upload before storing it
CreateFunction now writes a per-Region upload record before the name
claim and before any code Put, so a failed upload, rollback, expired
claim or late original writer cannot leave an untracked copy. Refused
or ambiguous outcomes keep the record; nothing is deleted or inferred
complete from an expired claim.
CodeStored validates the upload record before committing and refuses a
missing, unreadable, future-format, mismatched or non-owed record
without changing either record. Attach CASes only the checked record.
* feat(lambda): run test-link Invoke in a daemon-owned microVM
Behind the lambdatestlink build tag only. The gateway executor presigns
the exact code digest and sends one NATS request to this node's daemon;
the daemon fetches and verifies the code, then runs one fresh microVM
per invocation through a runner that holds QEMU's creating thread,
kills only through a verified pidfd, and counts a stop only on a
recorded exit status. The host enforces the function timeout from
dispatch, and a late result is a timeout.
Each attempt writes a marker before allocating an address or tap and
refuses further test invocations while any marker remains, including
after a restart. Taps are created exclusively and removed only while
their ifindex and owner alias still match. Anything whose stop or
release is not proven stays held.
The default build is unchanged: Invoke refuses before presigning,
sending or allocating, and Lambda stays unrouted.
Limits: functions never leave Pending (the env15 run activated its
test record by hand); no grant mechanism; a hung daemon leaves the hard
timeout unenforced; held attempts are not recovered after a restart;
requests are bounded by the 1 MiB NATS message; a timeout is an unknown
outcome rather than AWS's Unhandled error.
* feat(lambda): enforce attempt grants and a QEMU hard-kill backstop on the test link
Test-only and local: everything here is behind the lambdatestlink build tag
except the new domains/lambda/execution owner package. The default gateway
still refuses Invoke before presigning, sending or allocating, the normal
guest agent stays closed, and Lambda stays unregistered.
Attempt grants (domains/lambda/execution):
- One owner record per node holds the current attempt; every change is a
conditional write, and an unrecognised format is left untouched.
- The gateway issues a grant binding request ID, account, Region, function,
revision and code digest before anything is sent; a second attempt is
refused while one is unresolved.
- The daemon starts only under the current matching grant, before it
downloads code or allocates anything; stale, replayed or mismatched
grants are refused with the record unchanged.
- A result counts only if the owner accepts it under the dispatch deadline
while the attempt is running; a late or uncertain acceptance is reported
as an unknown outcome, never success.
- An attempt ends only after the VM stop is proven and resources are
released, or when nothing was launched; otherwise it is held as
ambiguous. Time alone never releases it. An uncertain stop write is an
unknown outcome: it may already have landed.
QEMU hard-kill backstop (runtime/compute/lambda/testlink):
- A preloaded initializer arms a SIGKILL timer inside QEMU before main and
acknowledges with QEMU's PID, the deadline and a per-attempt nonce. Any
failure after it loads kills the process before main.
- The daemon checks the initializer against a pinned SHA-256 before launch,
builds QEMU's environment without inherited LD_* or backstop settings,
and sends no invocation frame without the exact acknowledgement; a
missing or wrong acknowledgement kills QEMU and keeps state until the
stop is proven.
Limits, not production readiness:
- Not the M1 decision. The grant binds request identity and code digest,
not payload, memory or function ARN, and the guest does not yet bind
generation or sequence.
- If the preload never loads (missing file, static or setuid QEMU), nothing
arms; a daemon hung before refusing can leave QEMU alive, and the only
guarantee is that no invocation frame was sent.
- Hash-then-load has a replacement race and is not a trust boundary; the
initializer is built and installed by hand, with no packaging.
- A killed QEMU stays a zombie until the daemon reaps it; held attempts are
not recovered after restart; duplicate attempt IDs are checked only
against the current attempt.
* feat(lambda): seal the test-link timer library and bind the guest to its grant
Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and
the normal guest agent stays fail-closed.
Timer library: the daemon reads the hard-kill initializer once, checks those
bytes against the pinned digest, copies them into a sealed memfd, verifies the
seals and re-hashes the sealed copy, then passes it to QEMU as an inherited fd
loaded through /proc/self/fd. The initializer closes that fd before it
acknowledges. A kernel that rejects MFD_EXEC, or a memfd_noexec policy that
refuses it, refuses the attempt; there is no fallback.
Guest binding: the daemon writes a test-grant fw_cfg entry with the owner's
grant ID, attempt ID, environment ID and generation, and the guest's scope now
comes from that grant. Before installing code or starting the bootstrap, the
guest refuses unless the grant names the authenticated environment generation,
the first frame carries that grant and attempt ID, and the frame deadline is
present and still ahead on the guest clock.
Limits:
- The digest is only as trustworthy as the build input that pins it; the memfd
closes the swap window between check and load, it does not authenticate it.
- Not yet shown under the real spinifex-daemon systemd sandbox
(MemoryDenyWriteExecute, ProtectProc) or with memfd_noexec set to 1 or 2.
- fw_cfg and the request frame both come from the executor, so matching them
does not independently authenticate owner authority. This is not the M1
decision.
- The guest clock check is a local guard only; it cannot detect clock skew or a
delayed, superseded grant. The guest has no channel to the owner and checks
the generation only on the first frame.
- Held attempts are still not recovered after a daemon restart.
* feat(lambda): recover held test-link attempts after a daemon restart
Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and
the normal guest agent stays fail-closed.
Launch identity: attempt.Run passes the session callback its own duplicate of
the clone-time pidfd and closes SessionDone before publishing the result.
Before the guest is sent anything, the daemon takes the VM's pidfs inode from
that exact handle (refusing if the pidfd is not on pidfs or the VM has
exited), with the host machine ID, boot ID, PID namespace and the launching
daemon's own pidfs identity, and records it in the owner record with a
conditional Launched transition. A failed write holds the attempt as
ambiguous.
Recovery: RecoverLoop retries for a bounded number of rounds on the daemon
context, each round holding the node slot so it never runs beside a request.
It holds anything it cannot scope to the recorded boot and PID namespace with
the launcher gone, and holds an attempt with no launch identity. A PID with no
process, or one now held by a process with another pidfs inode, proves the
stop without signalling; the recorded VM still running is killed through its
own pidfd and counted stopped only once the pidfd reports the exit.
Cleanup order: files, tap and address (MgmtIPAllocator.ReleaseChecked, which
reports an unconfirmed release; Release is unchanged for its callers), then a
new owner state released that still blocks Issue, then the marker, then
ended. Recovery finishes from either side of each step.
Limits:
- Reboot and partition recovery stay held until host identity or fencing is
established; a machine ID can be copied, so it is not host proof.
- A launch that crashes before its identity is recorded stays held.
- No recovery if the daemon cannot restart or hangs.
- Not yet run under the real spinifex-daemon systemd unit, and no kernel
without pidfs was tested.
- This is not the M1 decision.
* fix(lambda): refuse ownerless checked mgmt IP releases
ReleaseChecked now refuses an entry that names no node, or an allocator
with no node bound, and leaves both the cluster and local entries in place.
The checked path deletes its local entry only after a confirmed release
and only if it still holds the same address. Legacy Release is unchanged.
Callers must serialize Allocate and ReleaseChecked for one instance ID;
the Lambda test-link slot does.
Recovery tests that need real pidfs now skip only when the kernel's pidfds
are not on pidfs, and a non-pidfs descriptor is shown to be refused. The
runtime still fails closed. Adds an owner test for Store.Launched.
* fix(lambda): retry test-link recovery until it settles
RecoverLoop now retries on the daemon context until the held attempt
settles, starting at 5s and doubling to a 5 minute cap, instead of
giving up after ten rounds. Each round still takes the node slot, so it
never runs beside Handle. It covers only what a previous daemon left:
an attempt a later Handle in the same daemon holds stays held until the
next restart.
The serving log now says the guest does not verify owner grant
authority.
* feat(lambda): build the test-link daemon and initializer as one pair
Add make build-lambda-testlink. It compiles the hard-kill initializer,
checks it against the 2.34 glibc ceiling, pins its SHA-256 into a
lambdatestlink spx build, and proves with TestLambdaHardKillPin that the
daemon's own pin accepts that file and refuses a changed one. Both files
are built in a staging directory and moved together only on success; any
earlier pair is removed first, so a failed rebuild leaves no pair. The
target refuses to build unless Go targets linux on the host architecture.
The initializer now uses _POSIX_C_SOURCE with -std=c17, so it no longer
needs __isoc23_strtol (glibc 2.38). The daemon refuses an initializer
built for another ELF machine or class.
Startup now begins recovery with only the owner store, tap and address
cleanup. A missing bridge address, pin, trust store or matching
initializer stops new invocations but not recovery.
Limits: test-only and unregistered. The pin is only as trustworthy as
the build host and toolchain; the initializer is still installed by
hand and ships in no release artefact. linux/arm64 is untested. Recovery
needs pidfs, which older kernels may lack. This is not the M1 decision.
* feat(lambda): fix one CLOCK_BOOTTIME backstop deadline per test-link attempt
Test-only (lambdatestlink); the default build and public Lambda route are
unchanged.
- The daemon reads CLOCK_BOOTTIME once before the grant Start and fixes one
absolute deadline: bounded code fetch + function timeout + boot allowance +
kill margin. Nothing recomputes or extends it; Handle never retries and
recovery never relaunches.
- The hard-kill initializer arms its SIGKILL timer at that absolute deadline
(SPX_HARDKILL_DEADLINE_NS, TIMER_ABSTIME) instead of relative to QEMU start.
- Dispatch is refused once the deadline has passed: before QEMU starts, when
the initializer ack is read at or after it (VM killed, stop proven), and
immediately before the invocation frame is sent. A clock read failure counts
as expired.
Limits: tap/IP setup, image read, initializer seal and QEMU start have no caps
of their own beyond the boot allowance. An overdue timer is not shown to fire
before vCPUs resume after host suspend. Owner-side retention, cross-node
ordering, clock-rate assumptions and M1 remain unsolved; nothing is released by
time and held attempts still need stop proof.
* feat(lambda): boot the test-link runtime image from a pinned digest
The test-link daemon now prepares its runtime image once at startup: each
boot file is opened without following symlinks or blocking, capped by size,
copied into a sealed memfd and hashed there, and the image digest over those
sealed copies must equal LAMBDA_IMAGE_DIGEST pinned into the build. QEMU boots
only the sealed copies (fds 3-5), and firmware comes from the image directory
rather than the shared microvm qboot.rom. The digest identifies the bytes; it
does not authenticate where they came from.
build-lambda-testlink-image builds a separately tagged guest into its own
outputs and prints the digest with the daemon's own file selection. Both
test-link targets refuse anything but an x86_64 host building linux/amd64,
and the digest input is read raw and validated before it reaches ldflags.
* test: cover test-link image load failures and daemon startup release
Exercise each sealed-copy failure path through the memfd hook and check
partial copies are closed. Extract the daemon's readiness and release
steps so startup tests can show the image is released when serving fails
and that Handle refuses without a prepared image.
* fix: align test-link Invoke errors with AWS and verify guest build tag
Malformed JSON payloads return 400 InvalidRequestContentException and
oversized requests return 413 with X-Amzn-Errortype
RequestEntityTooLargeException, both before the executor runs. Timeouts
and lost or failed sends are reported as unknown rather than as a
function timeout, and any Sandbox.Timedout result is treated as unknown.
Oversized function responses report Unhandled with
Function.ResponseSizeTooLarge in the body.
Add lambda-image-agent-tag, which reads the agent's build information
from each built initramfs and refuses an image whose lambdatestlink tag
does not match the target. Add tests that the default gateway cannot
invoke and that management IP Release drops the local entry when the
address is held elsewhere or the KV is down.
* test: use the shared RequireAWSCode helper in Lambda tests
The per-package requireCode helpers were replaced by
testutil.RequireAWSCode, which makes the same assertions; the Lambda
Invoke and upload tests still called the removed helper.
* fix: validate and authorize qualified Lambda Invoke requests
Invoke now runs every deterministic check before authorization and
authorizes the qualified function ARN derived from the same validated
target. Qualifiers follow the documented FunctionName and Qualifier
patterns and length limits, and a malformed query string is refused.
Code download errors no longer carry transport detail, so a presigned
credential in a bad redirect cannot reach the reply or the gateway log.
A seam built only with lambdatestlink and lambdatestlinkseam drives real
session outcomes through the executor and composed gateway in one test.
* test: restore the loadgen and workerboot tests to dev
* test: drive a bootstrap-posted timeout through the composed Invoke path
* fix: carry handler and qualifier to the test-link runtime and refuse function environments
The test link now passes the stored handler to the guest as _HANDLER and the
validated qualifier into the invoked-function ARN. Functions with environment
variables are refused before any presign, grant or dispatch, since delivering
them needs a separate secret-transport design.
* test: stub QEMU for the test-link package so it runs on hosts without it
* fix(vm): delete volumes only once QEMU is proven gone on a forced terminate
* fix(vm): treat an unreadable QEMU pid file as unknown and keep the UEFI obligation on a failed kill
* fix: judge QEMU exit only on the node that ran the guest
* fix(vm): keep unknown PID state and the local UEFI store obligation across reapers
* fix(vm): treat a non-positive QEMU pid as unknown state in both exit gates
* fix(vm): keep PID files that do not name a positive process at every deletion site
* fix(vm): treat only ESRCH as proof a QEMU exited, not any failed signal probe
* refactor: expose the signal probe as a plain setter instead of importing testing
* fix(daemon): complete a terminated UEFI store mark only once QEMU teardown is done
* fix(daemon): treat an expired terminated record as unproven QEMU exit for the UEFI store reaper
* test: separate the embedded field in the expired-record fake
* feat: run Spinifex on latitude.sh bare metal
Add a latitude.sh provider and deploy script. One node uses a static Elastic IP pool. Three nodes use a latitude floating pool over a fixed Elastic IP set, with the cluster and Geneve planes on a latitude.sh VLAN.
The floating engine gains a FixedSet provider interface. Its allocation token, re-read before a move and holder label apply to fixed sets only, so Vultr records and behaviour are unchanged.
Bead mulga-c43h3.3
* fix: check a server's project before destroy and refuse mismatched response IDs
A recorded server must sit in the project this deployment's Terraform holds, so a recycled ID with the same hostname elsewhere stops destroy. A missing project or ID refuses too.
The client rejects an Elastic IP response whose ID is not the requested one, without showing it.
Bead mulga-c43h3.3
* feat: cap a latitude pool at five Elastic IPs
Config validation rejects a source="latitude" pool listing more than MaxLatitudeElasticIPs addresses, so an allocation makes at most five bounded provider reads.
Bead mulga-c43h3.3
* fix: remember the project ID before destroy for the cleanup sweep
The Terraform output that names the project is gone once destroy has run, so the sweep could not tell a matching server from a foreign one and kept both.
Bead mulga-c43h3.3
* feat(ebs): record an unpublish obligation on force detach and terminate
A force detach, or a terminate whose unmount did not complete, clears a
volume's attachment while the node that served it may still export it.
The request now records an UnpublishObligation naming that publish in the
new spinifex-ebs-unpublish-obligations bucket before the clear, and
returns without waiting on the node.
The node-local unpublishObligationReaper settles its own node's
obligations under the mounter's per-volume lock, through the existing
UnpublishVolume client: it seals a surviving volume, drops a deleting or
absent one unsealed, and closes without a provider call when a newer
attachment or a newer publish on the node owns the export. A cluster-wide
pass closes the obligations of nodes removed from the cluster, and
spx admin volumes obligations lists the open ones.
The force-complete terminate and a stopped-instance terminate after a
failed seal record the same obligation; a terminate whose unmount
completed records none. ForceDetachVolume now answers with the attachment
in state detaching, as AWS answers a detach.
* fix(ebs): refuse a publish while the obligation reaper seals the volume
A publish waited on the volume lock while the unpublish obligation reaper
held it through a seal, so AttachVolume waited on work that scales with
the volume. A publish now refuses at once with ErrVolumeSettling, which
AttachVolume answers as IncorrectState, and the reaper takes the lock only
when nobody holds or awaits it.
* fix(ebs): refuse a publish behind any seal of the volume, a stop's included
A stop's or detach's sealing unpublish now marks the volume lock sealing,
so an attach after a force detach answers IncorrectState instead of
waiting out the seal. A waiting seal lets waiting publishes go first.
* fix(ebs): close a superseded unpublish obligation past a newer publish here
After a force detach and an attach again on the same node, the reaper
returned on the newer publish before it read the volume, so the old
obligation stayed open until the new attachment ended.
* fix(vm): recover a wedged terminate without waiting on its stalled unmount
The unmount records every volume as owed before it starts, so recovery
keeps that record rather than wait for the request lock the stalled
seal holds.
Resolves GHSA-7rv5-mrxv-xcr9, GHSA-wp97-x4hv-qvhp, GHSA-97r2-qh9m-474h,
GHSA-9wx9-5pv9-3349, GHSA-f86p-h4j2-7qgj (seroval) and GHSA-g3hv-75wg-m249
(minimatch, dev-only via eslint, forced with a pnpm override).
The conformance and NBD client suites, the fence, lease, dirty-marker,
abandon, takeover and stale-mount tests each stand up their own embedded
JetStream and Config, so they now overlap instead of running back to back.
Tests that touch package or process-wide state stay serial.
The six account-teardown tests and the unanswered network-test case each
wait about 5s on a production timer. Calling t.Parallel() first queued all
seven through the parallel slots; starting the teardown or network test
before t.Parallel() lets the wait run during the serial phase instead.
Also generate 1024-bit RSA keys in TestMain and shorten the test-only
context in TestQueryMetaNodeStatus_Unreachable.
go test -coverprofile at GOMAXPROCS=4: 16.9s -> 7.1-8.0s.
The four tests that wait out the fixed 3s describe fan-out deadline for a
silent node (9s, 6s, 6s, 6s) were queued behind the package's short
parallel tests, two of them as subtests of a table test. Hoist the two
slow table cases into top-level tests and move all four into a file that
sorts first, so they take parallel slots before the short tests do. The
table body becomes a shared helper; no case or assertion changes.
Each test stands up its own embedded JetStream server, so top-level tests
and independent subtests now call t.Parallel. Tests that redeclare the
process-wide cluster size or capture the default logger stay sequential.
Package wall time drops from ~23.5s to ~4.4s locally.
Six account-delete tests each wait one 5s teardown drain poll, and the
network-test and storage-status tests wait out fixed 5s and 2s timeouts.
Each uses its own embedded NATS, so they run concurrently: the package
drops from about 37s to 8s.
* test: pin instance inventory gaps G1-G3 as current behaviour
* feat(ec2): strict account instance inventory from the record store
* feat(ec2): strict spot-request and volume inventories
* fix(ec2): strict instance inventory fails on node-only instances
* fix(ec2): strict instance inventory sees error and late strays and rejects malformed records
The ecs-agent now holds an aws.CredentialsProvider (a CredentialsCache over
imdscreds.NewProvider) and ecsgw.New takes a *gwsign.Signer, matching eksgw,
rdsgw and elbv2. imdscreds.Fetch and stsauth.AssumeRole return aws.Credentials,
so agents/ecs/credentials and the field-copy adapters are removed.
* feat(uefi): delete a terminated instance's variable store through its store record and report unowned stores
Terminate, on both the running and stopped paths, moves the store record
to deleting and stamps a uefi teardown entry, then returns. The
cluster-wide uefi-store reaper deletes the objects under the instance
prefix and then the record, so the deletion converges even after the
terminated record's TTL has expired. CompareAndSet refuses to return a
deleting record to live.
spx admin volumes orphans now also lists instance prefixes with no
owner, image prefixes with no AMI document, and store records with no
instance record. It deletes nothing.
* refactor(uefi): read instance state for the store reaper and orphan report through daemon capabilities
The reaper and the orphan report named the raw instance-state records,
buckets and key prefixes, which ADR-0007:S6 forbids a new consumer to do.
They now ask TerminateDecided, OpenTeardowns, MarkTeardownDone and
KnownInstanceIDs, each stating its absent-record and failure behaviour.
* fix(uefi): keep a stopped instance's store when its terminate left the stopped record
TerminateDecided read the terminated record first, so a stopped terminate
that wrote it but failed to remove the stopped record let the reaper delete
the store of an instance that can still start. The live record now decides,
and the reaper no longer marks a live store deleting while one exists.
* feat(uefi): launch a UEFI instance from its variable store pointer
A launch whose record points at a variable store keeps the live copy at
<DataDir>/uefi/<instance-id>/vars.fd and opens it as pflash unit 1 with
format=raw. It reuses the copy when seeded.json beside it names the
pointer, and otherwise gets the object, decompresses it, checks it is
exactly the node's VARS size and writes vars.fd then seeded.json by
temporary file and rename. A failed get is retryable like a mount; a
missing or malformed object is not.
A store whose firmware arch or VARS size differs from the node's fails
the launch before any volume is mounted, with a Server.InternalError
state reason and a log naming both firmwares. A different CODE hash
boots, and is logged and counted.
A UEFI instance with no pointer and no EFI volume, or with both, is
refused with InternalError before its record is changed, in launch,
Start and the claimed-instance start. An instance whose variables are on
an EFI volume still launches through it.
* test(uefi): separate the embedded store from the injected error in the failing-get fake
* test(uefi): call startQEMU with its one argument after the dev merge
* fix(ec2): classify a volume held elsewhere by the provider code, not lease text
The instance service and the EKS cluster reconciler recognised a start
refused because another node holds a volume only by viperblockd's lease
text, so a provider that words the refusal differently was reported as
ServerInternal and counted against the reconciler's restart budget.
The instance service now matches the EBS provider's volume_held_elsewhere
code, on both the stopped-instance claim and the live-owner start. The
owning node marks such a refusal with an X-Volume-Held-Elsewhere reply
header, the ec2.start forwarder relays it, and StartInstances wraps
ec2v1.ErrVolumeHeldElsewhere around the rebuilt error, which the
reconciler matches with errors.Is. The AWS code stays IncorrectState.
The lease text stays as a fallback for replies that carry no code: the
legacy ebs.mount reply, a publish below schema version 3 (where
volume_in_use also means a read-only mismatch), and an owner node that
predates the header.
* fix(daemon): relay every header of the owner's start reply on both forwarder paths
The ec2.start forwarder copied only the held-elsewhere header onto a
relayed reply, and its unsealed-writes branch answered with the body
alone, so any other header the owning node set was lost in the relay.
Both relay branches now go through relayStartReply, which forwards the
owner's reply body and every header unchanged.
* test(daemon): pin that only a held volume gets the held-elsewhere header
The owner's reply header is the one signal that a start was refused
because another node holds a volume, and every other start refusal in
the same IncorrectState code must reach the caller without it.
* refactor(ec2): classify a volume held elsewhere only by the provider code
Every node sends provider requests at the one contract version, at which
every provider refuses a volume held by another node with the
volume_held_elsewhere code. The lease-text fallbacks in the instance
service and the EKS cluster reconciler matched replies that no longer
exist, so they are removed: a start is a held-volume refusal only when
the mount error carries the provider code, or when the start error wraps
ec2v1.ErrVolumeHeldElsewhere, which StartInstances rebuilds from the
owner's header.
The ec2instance.ErrVolumeHeldElsewhere alias is removed, so the sentinel
has one name, ec2v1.ErrVolumeHeldElsewhere. The reconciler's budget test
now proves an IncorrectState without the sentinel still counts, whether
its message is unrelated or carries the lease text.
* fix(ec2): log a held-volume start refusal at Warn with the instance ID
A start refused because another node holds a volume is returned as
IncorrectState, a client error, yet was logged at Error without the
instance it concerned. It is now logged at Warn, and every vmMgr.Start
failure log names the instance.
Tags are checked against the ELBv2 model's key and value length and
pattern (ValidationError), repeated keys are refused with DuplicateTagKeys,
and more than 50 tags on a resource with TooManyTags, counting a resource's
existing tags on AddTags. A tag with no Value is stored as an empty value
and a tag with no Key is refused rather than dropped.
Ochre's two streaming pumps now record through one streamRecordCtx.record and
frame upstream faults through one frameWriter.writeUpstreamFault. The EKS and
Lambda route tables list their actions through rest.Router.Actions,
toDetails converts toSummary's identically-shaped struct, Lambda's
environment response uses aws.StringMap, and the e2e suites call
harness.DiscoverNanoInstanceType and harness.EnsureKeyPair directly.
* test: pin guest poweroff handling and stopped-instance volume state
* test: gofmt the poweroff and stopped-volume tests
* test: pin start after detach, one-poll poweroff, reattach of released volumes and the delete refusal
* test: separate the embedded mounter from the recorder fields
* fix: end a guest poweroff as a stop and keep stopped instances' data volumes attached
A clean QEMU exit after the heartbeat saw the guest power off now stops the
instance instead of waiting for crash detection. Unmount no longer releases
data volumes. DetachVolume on a stopped instance edits the record, then
releases the volume. Starting an instance stopped on an older build drops data
volumes that are attached elsewhere or deleted and fails on a missing boot
volume.
* fix: keep the stopped-instance detach inside the instance service file and format start reconcile
* docs: describe guest poweroff, stopped-instance volumes and detach on a stopped instance
* fix: allow detaching the root volume of a stopped instance and refuse a start with no root volume as AWS does
* feat: attach volumes to a stopped instance and fix root-detach start fixtures
* docs: describe attach and root detach on a stopped instance
* test: Unmount seals only and leaves every attached volume in-use
Changed with the person's explicit approval, relayed by dev-design on 11-03.
EBS root and data volumes persist across stop on AWS, so stop and crash
unmount no longer release any attached volume.
* docs: note that the EFI variable store cannot be detached from a stopped instance
* test: unmount contract tests expect no volume state write on stop
* test: order imports in stopped-volume daemon tests
* refactor: drop the unused volume state updater from the volume mounter
* test: console log trim, removal by instance state and append-mode chardevs
* fix: cap and reap instance console logs, always append
* test: expect append mode on microvm file chardevs
* fix: retire console logs on terminate instead of reading instance state
* docs: describe console log location, caps and retirement on terminate
* test: drop removed append argument from startQEMU calls
* docs: state that GetConsoleOutput returns NotFound once an instance has been torn down
* docs: drop the unverified AWS retention claim for console output
* feat: remove idle console logs that no QEMU on the node holds open
* fix: keep idle console logs when a QEMU pidfile names a pid the scan cannot see
* test: eks node log forwarder filter, redaction, caps and role wiring
* feat(eks-node): forward k3s agent warnings and errors to the serial console
A redacting, rate-capped forwarder tails the agent log and writes warning,
error and node-status lines to the console so they reach the sink. Secrets are
redacted before the 512-byte cut.
InstanceStatus and VM carry an optional UEFIVarStoreRef naming the store
object and the firmware it was written under. The new uefistore package
holds FirmwareIdentity, UEFIStoreRecord and UEFIWriteBack, and a Store over
the spinifex-uefi-stores bucket (history 1, no TTL, schema 1) whose writes
are all revision-guarded and validated. The daemon creates the bucket at
startup. Nothing reads or writes either yet.
* feat(ebs): carry the fence notice over the provider contract subject
viperblockd now sends the contract FenceNotice on
ebs.provider.v1.<node>.fenced when it revokes a live export, and the daemon
subscribes to that subject whatever the EBS provider, stopping the guest
using the fenced volume as before. The legacy ebs.<node>.fenced emission,
subscription and their viperblockd legacy contract types are removed.
NotifyFenced is the shared single-attempt sender; viperblockd keeps its
retry-until-answered loop around it. MemoryProvider can revoke a
publication and announce it, and the conformance suite gains a
RevokeExport case that viperblockd and MemoryProvider both run.
* fix(ebs): send the fence notice through the contract request path
NotifyFenced now shares requestReply with NATSProvider.request, so a
notice gets the connected check, the X-Account-ID header and the
account-attributed client span the other contract requests do. A
client span also fails for a reply it cannot read.
A notice that can never be sent (no volume, an unaddressable node, no
connection) is ErrInvalidArgument and terminal, and FenceNoticeRetryable
names the errors worth sending again: unheard, or refused as
unavailable. MemoryProvider.RevokePublication retries until heard or
its context ends, and shares unpublish with UnpublishVolume.
* fix(viperblock): refuse an unaddressable node and retry an unavailable fence notice
viperblockd refuses to start, and RegisterProviderSubjects to register,
when the node name is not one contract subject token: there would be no
fence-notice subject, so a fenced guest would keep running. Nothing is
subscribed before the refusal.
publishVolumeFenced keeps sending a notice the daemon refused as
unavailable, stops on one that can never be sent, and no longer checks
the subject itself. The late fence completion is shown to send its
notice.
* fix(daemon): refuse an unaddressable node and trace fence notices as the contract does
NewDaemon refuses a cluster config node that cannot address the fence
notice subject, instead of starting with no fence-notice subscription.
handleFenceNotice opens the EBS provider server span, so the notice is
attributed to the sender's account, and every refusal goes through one
path that records its code on the span and answers in the notice's
version. An unsupported version names the version it got.
Tests hold the guest's stop open to show no ack leaves before it ends,
and drive the notice through subscribeAll's real subscription table.
The user reaper removes each group membership before DeleteUser and the group reaper detaches attached and deletes inline policies before DeleteGroup, so a user in a group or a group carrying policies no longer strands the identity stage. DeleteUser now ignores a group name with no group record, as it already does for an access key ID with no key entry.
* feat(ebs): drain the volume inside each provider's CreateSnapshot
viperblockd now drains a mounted volume's export over snapshot.sock in the
background snapshot work, before reading the live checkpoint. A refused or
stale socket fails the snapshot; no socket with the volume lock free means no
client is connected and nothing is buffered; no socket with the lock held is a
client between connections and is reported as retry-later.
Write-through providers drain nothing: the control-plane DrainVolume and the
daemon's drain handler skip the snapshot.sock drain when the node's EBS
provider is not viperblockd, so a snapshot or CreateImage of an attached
volume on qemunbd no longer fails. The control-plane drain stays for
viperblockd nodes that predate the provider drain.
The conformance suite gains a case that CreateSnapshot answered on the queue
subject by a node other than the volume's holder is refused, while the
holder's own snapshot succeeds, run against viperblockd through a third node
reachable only on the queue subjects.
* fix(ebs): refuse a snapshot whose last export client left unsealed WAL
The plugin releases its volume lock even when the seal on disconnect fails,
so a free lock alone did not show the last client's writes reached the
backend. With no drain socket and the lock free, the snapshot now also
requires that the volume holds no local WAL, or that a seal receipt attests
to it.
A socket path of 108 bytes or more cannot be listened on, so a held lock
there is reported as a failed drain rather than as retry-later, and the path
bound is now inclusive. The drain logs with the request context.
* fix(qemunbdd): snapshot a published volume through its export
qemu-nbd holds a published volume's qcow2 open with the write lock, so
qemu-img convert of the file failed with "Failed to get shared write lock",
and reading it with -U would race the metadata qemu-nbd caches. A published
volume is now copied from the NBD export, which serves exactly what the
attached client wrote; an unpublished volume keeps copying the file.
The reader needs a second connection beside the attached client. Any
--shared limit above 1 makes qemu-nbd advertise multi-conn, after which a
client opening several connections blocks on those over the limit, so the
export now accepts any number of clients.
* fix(ebs): require retry-later from the queue-subject snapshot refusal
The conformance case now requires the refusal to carry ErrUnavailable rather
than any error. The viperblockd harness returns a struct instead of five
positional values and stands its queue-only node up only for the case that
asks for it.
* docs(ebs): say only the snapshot accept reply is freed of the drain
* fix(ec2): warn when a drain is skipped for a provider not known to write through
Only viperblockd serves the drain socket, so the skip itself is unchanged, but
it is logged at Info only for qemunbd and at Warn for any other provider. The
daemon logs the host-side skip under its own message.
* fix(ec2): log a provider snapshot refusal and surface retry-later as ServiceUnavailable
CreateSnapshot flattened every provider failure into an unlogged
ServerInternal. It now logs the provider error with the volume and snapshot
ids, and maps a provider ErrUnavailable to ServiceUnavailable as the delete
path already does. CreateImage's snapshot step shares the mapping.
* fix(qemunbdd): refuse a snapshot of a published volume
Reading a published volume through its export needed qemu-nbd to accept more
than one client, which drops the guard against a second writer, and the copy
it made of a live export was not point-in-time. CreateSnapshot now refuses a
published volume with volume_in_use before running qemu-img, and the export
is back to a single client. An unpublished volume still snapshots from its
file.
* refactor(ec2): drop the control-plane snapshot drain
Every viperblockd in a cluster drains the volume inside its own
CreateSnapshot, so CreateSnapshot and CreateImage no longer dial
snapshot.sock or route a DrainVolume command to the instance's host.
Removed: DrainVolume, DrainVolumeSocket, DrainsThroughSocket and the
skipped-drain logging in the snapshot domain; the state.json read and
the volumestate package only it used; the daemon's DrainVolume handler
and the DrainVolume command, payload and response in ec2v1. CreateImage
now snapshots the root volume through one provider path whether or not
the instance is running.
Tests assert that both EC2 paths snapshot an attached volume through the
provider without opening the drain socket or messaging the host, and
that viperblockd drains a mounted volume exactly once.
* fix(viperblock): prove no local writer before an unmounted snapshot
The unmounted branch of CreateSnapshot opened its own engine once the lease
was held, but the lease is reentrant on the volume's own host, so an nbdkit
left running unregistered after a failed recovery rebuild kept taking writes
the snapshot never saw. The branch now takes the volume lock before the open
and holds it until the snapshot is read; a held lock is drained through the
export's socket, or refused as retry-later when it serves none.
A seal receipt now approves local WAL only if it is newer than every change
under the WAL, so a receipt from an earlier connection no longer covers WAL a
later one failed to seal. A refusal for a held lock with no drain socket is
logged at WARN with the socket path.
* fix(ec2): map a provider volume-in-use snapshot refusal to IncorrectState
A provider that cannot snapshot an attached volume, as qemunbdd refuses a
published one, surfaced as ServerInternal. CreateSnapshot and the CreateImage
snapshot step now return IncorrectState telling the caller to detach the
volume first.
* docs(e2e): describe the stopped CreateImage path as the single snapshotVolume path
* chore(viperblock): order the io/fs import as gofmt does
* feat(daemon): mount and unmount through PublishVolume behind the local provider's version 3
The mounter asks its own node's provider for capabilities on the per-node
subject and, when version 3 is listed, mounts and unmounts every volume
through PublishVolume and UnpublishVolume at version 1, recording the
published export on the request. Otherwise it keeps the legacy
ebs.<node>.mount and unmount subjects.
A retry-later or not_found refusal and a publish with no responder back off
as ErrMountRetryable, so restore keeps the legacy retry. No responder on
any mount or unmount drops the cached route so the next call re-reads
capabilities. An unpublish takes the provider's data-movement budget.
* fix(daemon): treat a publish reply with no usable export as an ambiguous mount
NATSProvider.PublishVolume now wraps ErrNoPublishedVolume and
ErrInvalidExport, keeping its messages. The mounter turns either, and an
export it cannot apply, into vm.ErrMountAmbiguous, so hot attach runs its
defensive unmount and Mount's rollback unpublishes the failing volume as
well as those already published.
* fix(daemon): give the contract unpublish its own seal budget
DetachVolume bounds its context for the unplug and then seals, so an
unpublish on that context got whatever the unplug left. The mounter now
detaches the unpublish from the caller's cancellation, keeping the trace,
so NATSProvider's data-movement timeout governs the seal as
unmountSealTimeout does on the legacy path.
* fix(ebs): let NATSProvider requests buffer through a reconnect
request refused whenever the connection was not connected, which includes
reconnecting, while the legacy subjects' plain nats request buffers and
sends once the connection is back. Only a nil or closed connection now
refuses.
* fix(daemon): roll back a mount over the route the mount took
Mount's rollback re-read the route, so after a failure that dropped the
cached one a contract-published volume could be unmounted over legacy
ebs.unmount. The rollback now takes the route the mounts took. Unmount,
unmountOne and the rollback share one seal path that is given the route,
replacing the per-route helpers and their error out-parameter.
* test(daemon): cover a not_found unpublish and a legacy no-responder
An unpublish answered not_found counts as sealed for Unmount and
UnmountOne, and a mount that finds no responder on the legacy subject
re-reads capabilities, so a provider restarted at version 3 takes the next
mount over the contract.
* refactor(daemon): reuse endOpSpan in the mounter and correct its comments
The contract spans end through the existing endOpSpan, the adapter's doc
comment fits three lines, and configureEBSProvider no longer claims its
provider is the only one wired, since the mounter builds its own.
* refactor(daemon): mount, unmount and abandon over the contract at version 3
A cluster runs one EBS contract version and every provider in it serves
version 3, so the volume mounter no longer reads its node's capabilities
to choose a route. Mount, MountOne, Unmount and UnmountOne always send
PublishVolume and UnpublishVolume at version 3, and Abandon is an
UnpublishVolume with NoSeal whose torn-down field replaces the legacy
abandon reply. The mounter sends nothing on a legacy ebs.<node> subject.
Export mapping, the retry-later and ambiguous-mount mappings, the seal
budget, the detached rollback context and the provider error chain are
unchanged. Tests that drove the legacy route or route selection are
removed or moved onto the contract subjects.
* refactor(ebs): remove the per-node capabilities subject
The volume mounter was the only consumer of ebs.provider.v1.<node>.capabilities,
and it no longer reads capabilities to choose a route. NodeCapabilitiesSubject,
its wildcard and verb, NATSProvider.GetNodeCapabilities and the natsserve and
viperblockd subscriptions serving it go. The cluster-wide GetCapabilities and
its contract version list stay.
* fix(ebs): refuse requests on a disconnected connection and carry the account
NATSProvider refuses a request again whenever its connection is not
connected. A request buffered through a reconnect is flushed after its
caller may have given up, so the provider would act with nobody waiting,
such as creating a volume the caller reported as failed.
A reply that cannot be decoded, or that carries a schema version this
build does not serve, now wraps ErrUnreadableReply, so a caller can tell
a request the provider received from one it never saw.
WithAccountID puts an account on a context. Every request made under it
carries the X-Account-ID header and names the account on its client
span, and the server span reads it back from the header.
* fix(daemon): retry a refused publish, tear down an unknown one, attribute every call
A publish refused because the connection is down was never sent, so it is
now vm.ErrMountRetryable and restore backs off. A publish that timed out
or whose reply could not be read may have started an export, so it is
vm.ErrMountAmbiguous and Mount and AttachVolume unpublish it.
Publish, unpublish and abandon put the instance's account on the context,
so the provider request carries it on the wire and on its span. The
mounter's own per-call span is gone, leaving one client span per call.
Abandon opens one span naming the instance, account and reason, and a
failed volume's error names the reason. Its bound is a field so a test
can check that an unanswered abandon gives up within it.
The mounter takes the daemon's provider rather than building a second
one, sends at NewLatestVersioned, calls unpublish from UnmountOne and
leaves export validation to the provider client.
* refactor: describe the single-version cluster and the publish path
NewVersioned and NewLatestVersioned no longer describe an N/N-1 rollout
or a capability exchange, since a cluster runs one contract version. The
volume path names publish and unpublish rather than the legacy ebs.mount
and ebs.unmount subjects, and ErrMountAmbiguous describes a publish reply
with no usable export.
* test(viperblock): pin the held-elsewhere text on a version 3 refusal
Callers still recognise a volume held by another node by its message, so
the held_elsewhere refusal must keep ec2v1.VolumeHeldElsewhereText.
* fix(daemon): tear down a publish whose caller cancelled after sending
A publish the provider received and the caller then stopped waiting for has
an unknown outcome, like a timeout: the provider may have started the export.
Classify context.Canceled from a sent publish as ErrMountAmbiguous so Mount
unpublishes it and hot attach rolls it back, both under a context the caller's
cancellation does not reach.
A caller already gone before the publish starts sends nothing, so publish
returns its context error without calling the provider and without the
ambiguous classification; a disconnected connection stays retryable.
The PXE build rebuilt spinifex as root inside the workspace, and a cancelled
job's root make kept running for an hour, so the next job on the runner could
not write bin/spx. The microVM image is now built as the runner user, the PXE
build skips its spinifex rebuild, and both root steps run in named systemd
scopes that are stopped when the job ends and before the next one starts.
* feat(compute): boot the root volume from a -blockdev node that takes NBD or host_device
The boot volume moves from -drive if=none to a -blockdev node named os, so
a block export reaches QEMU as host_device as data volumes already do. The
NBD boot shape keeps its cache, reconnect, iothread and bootindex settings;
werror and rerror move onto the virtio-blk device. A block EFI volume is
still refused because pflash has no host_device form.
* test(viperblock): pin the boot -blockdev shape in the nbdkit orphan reaper
The boot volume's NBD server options now carry a file. prefix, which the
referencer scan matches only because it compares by substring. Cover the
unix and TCP boot strings so a stricter match cannot reap a live root
export, and stop describing the joined host:port form as a boot drive.
* fix(compute): refuse a disk-boot launch with no boot volume
Data-volume -blockdev nodes satisfy Execute's drive check on their own, so
a disk-boot request list without a boot volume would start a QEMU with
nothing to boot. The disk-boot path now refuses it before launch.
* refactor(compute): share the NBD node options and export dispatch
The data and boot -blockdev builders share one helper for the nbd server,
export and reconnect-delay options, buildDrives dispatches both branches
through one export helper, and hot attach parses its URI with the shared
parser. Emitted arguments are unchanged. The BlkDeviceMapped doc now states
when it falls back to the scalar form.
* test(compute): cover an unnamed boot device with no medium in the device map
* feat(ec2): delete live launch templates in platform cleanup when no consumer names them
A cleanup request may ask for live templates to be deleted. It then reads which
templates records outside EC2 (nodegroups) name, deletes the rest first, and
checks key pair and placement group references against what remains. A failed,
partial or absent consumer read deletes nothing and blocks every live template.
Requests that do not ask behave as before.
* feat(accountlifecycle): clean L09 kinds through the EC2 platform cleanup on ordinary delete
Ordinary delete evaluates quiescence again under the same fence before the
platform stage, cleans only the kinds that record as quiesced (live launch
templates only when EKS names none), and re-reads the owner inventory before
the fenced DeleteAccount. Any remaining L09 record or incomplete re-read keeps
the account TERMINATING; Force is not forwarded. The key-pair, placement-group
and launch-template reapers and the dry-run shadow comparison are removed.
* test(accountlifecycle): Force never bypasses a reference on an ordinary delete
Until versions can be published, the only version is $LATEST: the list holds
its configuration with the FunctionArn qualified as :$LATEST and no
NextMarker. MaxItems outside 1 to 10000 is refused, and Marker is refused
outright rather than guessing AWS's paging behaviour. It is authorized as
lambda:ListVersionsByFunction on the function ARN and does not need the code
store.
* test(eks): pin async nodegroup delete and per-nodegroup instance profile
* feat(eks): delete nodegroups asynchronously and own one instance profile per nodegroup
* fix: confirm nodegroup workers are gone before deleting their profile, and kick the delete after the request
* fix: end a stuck nodegroup delete as DELETE_FAILED, and cover nodegroup delete in the EKS e2e
* docs: describe asynchronous nodegroup delete and the per-group instance profile
* test: keep the EKS e2e bystander instance profile until the parent test ends
* test: cover nodegroup delete failure paths, profile ownership refusals and worker absence checks
* feat(admin): non-finalizing DeleteAccount exercises the quiesced EC2 platform kinds
A non-finalizing DeleteAccount quiesces the account, records the S9 evaluation, cleans only the key-pair, placement-group and launch-template kinds it recorded as quiesced, then re-reads the owner inventory and accounts for what remains. The job ends EXERCISED with the account still TERMINATING; a later ordinary delete takes the same operation over.
Only an exercise records a quiesced scope, so ordinary deletes still record none. The cleanup capability gains a kinds selector, awsgw binds the in-process owners, and the instance reference reader fails closed.
* refactor(admin): name the takeover of a finished exercise exercise_finished
Code signing is unsupported, so the read only proves the function exists and
returns AWS's body for a function with no code signing config: both members
present and null. A missing function is ResourceNotFoundException. It is
authorized as lambda:GetFunctionCodeSigningConfig on the function ARN and
needs neither the code store nor environment decryption.
Serves GET /2015-03-31/functions/{FunctionName}/configuration with the
same name, qualifier, ownership and environment handling as GetFunction,
authorized as lambda:GetFunctionConfiguration on the function ARN. It is
a metadata read, so it neither opens nor presigns the code store.
Dev-only code delivery behind the lambdatestlink tag: the host attaches the
ZIP as a read-only disk and passes its exact size and SHA-256 through
fw_cfg. The agent reads those values before sealing fw_cfg, and installs
the package only after authenticating and receiving the host's first
frame: it checks size and hash, validates every entry before writing any,
extracts through os.Root into a nosuid,nodev tmpfs at /var/task, and makes
it read-only. Any failure exits without starting the bootstrap.
Entry names that could escape the task root, entries other than regular
files and directories, and sizes over the dev bounds are refused; declared
sizes cannot wrap the running total, and actual bytes are counted whatever
the header claims. The bounds are dev limits, not AWS limits.
StartBootstrap now starts the bootstrap in its own directory, so packaged
functions can open their files by relative path.
* feat(config): refuse a spinifex.toml whose nodes resolve to different EBS providers
* feat(daemon): lock a cluster to one EBS provider in spinifex-cluster-config
The first daemon records its provider at ebs-provider with Create, which
also backfills an existing cluster. A daemon whose provider differs from
the recorded one refuses to start, naming both providers and the recovery.
* fix(config): compare only explicit EBS providers across nodes
A generated spinifex.toml writes no ebs stanza for peers, so treating an
absent peer provider as viperblockd refused every non-default cluster.
* fix(daemon): exit on an EBS provider lock refusal and refuse a malformed lock
A mismatch now exits 1 so the unit shows failed. A recorded value that
does not decode or names no supported provider is a distinct permanent
refusal quoting the raw value. Permanence is a sentinel matched with
errors.Is, and the cluster-config bucket is declared beside the others.
* style(daemon): align JetStreamManager fields
* feat(ebs): contract schema version 3 for the provider data path
Servers accept schema versions 1 to 3 and answer every request in the
version it was sent at, while every client still sends version 1.
Version 3 adds no_seal to UnpublishVolumeRequest, refused below 3, and
torn_down to UnpublishVolumeResponse. UnpublishVolume now returns an
UnpublishedVolume so the torn-down result reaches callers.
A publish refused because another node holds the volume returns
volume_held_elsewhere at version 3 and volume_in_use below it, which is
what v1.23.0 returns. A read-only mismatch on republish stays
volume_in_use at every version.
GetCapabilities lists the contract versions served and is also answered
on a per-node subject. The unsealed-writes subject is part of
EBSProvider, and write-through providers answer it with nothing held. A
fence-notice event subject and message are defined. Unpublish gets the
data-movement timeout.
The memory provider, natsserve and qemunbdd implement all of this. A
qemunbdd republish whose qemu-nbd has died re-exports the volume instead
of returning the dead socket.
* feat(viperblock): serve contract schema version 3 from viperblockd
Every provider subject replies in its request's version, and
GetCapabilities lists the contract versions and is also served on the
node's own capabilities subject.
PublishVolume no longer returns an existing entry early. It goes through
mountVolume, which hands back a live export and drops and remounts one
whose nbdkit has died. Mount failures map to not_found, held_elsewhere
(version 3 only, volume_in_use below it), volume_in_use for a mode
mismatch, unavailable for a backing or lease store that is not ready,
and internal otherwise.
UnpublishVolume with no_seal abandons the export without sealing it,
and reports whether an export was torn down.
* test(ebs): conformance cases for contract schema version 3
Every provider run through the suite now has to:
- list ascending contract versions including the legacy one, answer each
in that version, and refuse an unlisted one without changing state;
- refuse a publish elsewhere with volume_held_elsewhere only at version 3
and volume_in_use at versions 1 and 2;
- refuse a republish in the other access mode as volume_in_use, never
held elsewhere;
- report torn_down truthfully, refuse no_seal below version 3, and either
honour no_seal at version 3 or refuse it as unsupported;
- answer the unsealed-writes query with a subset of what was asked.
SuiteConfig.KillExport lets a provider whose export is a process prove
that a republish after the export died returns a serving export.
qemunbdd and viperblockd supply it.
* fix(ebs): stop a running export before remounting over its lost socket
An export whose socket is gone was treated as dead even while its
process ran, and a republish started a second export beside it. A
client connected before the socket went keeps writing through the old
process, so that made two writers on one volume.
qemunbdd now SIGTERMs that qemu-nbd and waits for it to exit before
exporting again. viperblockd stops the nbdkit through unmountVolume, so
the old copy is sealed as a normal unmount would seal it. If the process
will not stop, the publish is refused as unavailable and the entry is
kept for the next attempt.
* test(viperblock): give the no-seal tests a seal they could reach
The no-seal unpublish and abandon tests left BaseDir empty, so
volumeNeedsSeal was false and no path could seal whatever the code did.
Both now create local state under BaseDir first, and check that a seal
is reachable before asserting none ran.
* refactor(ebs): leave version rules to the provider and trim the v3 surface
The provider decides what a request's version allows, so natsserve no
longer gates TornDown or held-elsewhere on it: classifyError takes no
version, and the reply carries what the provider returned. viperblockd
applies the TornDown rule in unpublishVolume, its provider half.
The unpublish handlers check the version once, through
CheckUnpublishVersion, and classify its refusal. Each handler binds its
reply version once.
UnpublishedVolume is reduced to TornDown. movesData classifies a subject
by its parsed verb, which replaces isNodeSubject and the owner-verb
list. nodeSubjectWildcard is unexported. The conformance suite reads
capabilities once.
* test(ebs): check v3 success replies, timeouts and refusals end to end
- Unpublish is sized as data movement, and publish, unsealed writes and
per-node capabilities keep the caller's timeout.
- A mount whose lease store or backing store is not ready reaches the
wire as unavailable.
- A successful publish replies in the request's version at every served
version, in natsserve and in viperblockd. Every natsserve case now gets
its own seeded provider, so none relies on an earlier case's state.
The conformance republish runs at every listed version.
- viperblockd's per-node capabilities list the versions it serves.
- A no_seal unpublish above the served version is refused, and the
export is neither torn down nor sealed.
- The dead-entry republish test asserts unconditionally. The snapshot
lease test runs against real volume state and asserts nothing was
written. The conformance torn-down case skips a provider that does
not list version 3.
* fix(ebs): wrap the stop failure behind an unavailable publish
The refusal now wraps the stop error as well as ErrUnavailable, so a
caller can still inspect why the old export did not stop. Also fixes
import order and spacing that the linter flagged in the new tests.
* fix(admin): read the newest network test by its claim key, so a listing never leaves it out
* fix(admin): keep the newest-test lookup additive, so it never fails a listing
The lambdatestlink agent now loads its identity, seals fw_cfg and dials the
host exactly as the normal agent does, then carries the test stream over
that mutual-TLS connection instead of a device path. Nothing tenant-facing
starts until the host's first valid frame: a refusal after the handshake,
an early close or a shutdown never starts the bootstrap. Each VM serves one
invocation and then stops.
The normal agent is unchanged and still accepts no invocations. The tagged
tests join the race tier for the two packages that carry the build tag.
* fix: find node planes from the node's configuration, and admit only a node's own address as a network test target
Node facts looked for planes only on br-wan, br-lan and br-vpc, so a host
whose addresses sit on plain NICs (a Vultr or OCI VM) reported no planes:
the topology had no links, every port read unused, and Run test stayed
disabled. Each plane now resolves to the device holding the node's
configured address: ovn-encap-ip for vpc, the node's own host for lan,
and vpcd.external_interface (or the bridge it is a port of) for wan, with
the br-* names as the fallback.
A run request now names its target node, and the client admits only the
address that node reports for the plane. The old subnet rule refused
legitimate WAN tests on Vultr, where each VM has its own /23, and admitted
strangers sharing a cloud subnet.
The network test's selects showed their raw values (__all__) because Base
UI's Select needs its options as items to render a label.
* fix(ui): list a plane on a plain NIC without an empty bridge, and note a shared LAN port as well as a shared bridge
* fix(vultr): turn on public addresses in the default subnets once the Vultr pool is live
* fix(ui): say where an RDS database goes, and how to use another VPC, when no DB subnet group exists
A stop/start that leaves the guest stuck in cloud-init-local waiting on
IMDS fails the restart SSH probe, but the diagnostics only captured the
EIP/OVN side, which was healthy. Capture br-imds flows, the endpoint's
link, addresses, sysctls, reply rule and route, the responder listener
and conntrack too, so the next occurrence names the broken hop.
* fix(rds-agent): stop a command blocked on a backup hold from keeping out its release
The agent ran commands one at a time inside its poll loop. A MariaDB
ALTER USER issued while an automated backup held BACKUP STAGE blocked at
its commit, so the unquiesce that would release the hold could never be
delivered: the rotation timed out at the API while the new password was
already live in memory, and the instance stayed backing-up.
Commands now run on their own goroutines while the agent keeps polling,
re-published copies of a running or answered command are ignored, and a
finished reply cuts the current poll short so it is carried at once.
MariaDB SetPassword also waits for any hold to end before it runs, for
less than the control plane's budget, so a rotation that cannot finish
fails before touching the engine.
* fix(rds-agent): keep engine commands serial and bound every command by its budget
Engine-mutating commands go back to running one at a time on a single
lane, as before; only unquiesce runs beside it, so a command blocked on a
backup hold still cannot keep out the release.
Each command is bounded by its type's control-plane budget, now exported
as rdsinstance.CommandBudget and measured from receipt. A command whose
budget ran out while it queued is answered as expired and never run, so
a stale quiesce cannot put the engine in backup mode. Quiesce takes its
bookkeeping lock within its context, and the MariaDB ALTER USER carries a
lock_wait_timeout from what is left of the budget.
* test(rds): rotate the MariaDB master password through an automated backup
Opens the backup window on purpose and rotates the password back and
forth until one rotation is in flight while the instance reports
backing-up, asserting each outcome is whole and that the backup releases
the engine. Every suite create now pins its backup and maintenance
windows clear of the run, so no other subtest's outcome depends on the
hour the cell runs.
* fix(rds): cap an agent command by the budget its issuer has left
The agent measured each command's deadline from receipt, so a first copy
delayed by a gateway or bus outage was granted a whole fresh budget when
its issuer had nearly given up; a late quiesce could then take a hold
nobody would release. The issuer now stamps every republished copy with
the budget it has left, a duration that needs no clock agreement, and the
agent uses the smaller of that and its own type budget.
* feat(ui): console feedback round: state tones, inline tags, filter bar and form fixes
Settled states go green and transitions yellow with a spinner, status text is
lowercase, tags are edited inline row by row, every list filters through one
filter bar, long cells truncate with a tooltip and times are relative.
EC2 launch takes a Name and offers ICMP echo, and leaves Placement Group out
when no group is available. RDS create is reordered with engine tiles and
window pickers. Route 53 records get bulk delete, a TTL select, value labels
that follow the type, and NS. Tooltips use the popover tokens, and the console
is sized over a 110% root.
* feat(ui): size RDS instance classes and give each record value its own input
An RDS class is a db.-prefixed EC2 type, so the class picker shows its vCPUs
and memory in every option and under the field, as the EC2 launch form does,
on create, modify and restore.
A Route 53 record's values were one textarea, which let a line break into a
value. Each value is now a single-line input with Add value; Enter starts the
next one, a pasted list splits into rows, and CNAME takes exactly one.
* fix(ui): open a DNS record for editing from the keyboard
* feat: admin node inventory, topology, live host metrics and network test
Each node collects its hardware facts as root at daemon start (dmidecode,
ethtool, networkctl LLDP where available) into /etc/spinifex/node-facts, and
the daemon serves them with its plane wiring over NATS.
New admin-only actions, refused for every account but the system admin
account regardless of policy:
- GetNodeFacts: hardware, NICs, LLDP neighbours and planes per node
- GetNodeMetrics: live CPU, memory and per-plane traffic for one node
- StartNetworkTest / DescribeNetworkTests: iperf2 between nodes over a
chosen plane, both directions, one test cluster-wide at a time, recorded
in KV and converged asynchronously
The iperf listener runs only for the duration of a run, without
capabilities, on TCP 5001, which the peers firewall rule now allows. A run
refuses any target outside the node's own plane subnet.
The console gains a node panel (hardware, network, metrics, network test
with a dial scaled to the port's line rate) and a topology view built with
@xyflow/react. setup.sh installs dmidecode and iperf.
* test: cover the node insight handlers and the facts file mode
* feat: report every CPU flag in node facts
* feat(ui): node table on the dashboard, every CPU flag and a readable topology
The dashboard's nodes section is the Nodes page table, open by default, and a
row opens that node. Ready reads green, capacity sorts by most available, and
the processor's features tip lists every flag. Topology draws one row per
port with switches in a muted card, highlights a node's links on hover, and
offers a left-to-right layout beside top-down.
* chore(ui): rebuild dist
* fix: align the CPU facts fields
* fix: bound the network test server start and read ARM CPU features and cores
* chore(ui): rebuild dist
* test(ui): pin the instances baseline clock and take the screenshots from CI's Playwright image
* fix: retry a network test claim whose holder released it mid-check
StartBootstrap sets no_new_privs on a dedicated locked OS thread and
starts the bootstrap from it, as uid/gid 993 with no supplementary groups
and only the environment it is given. The thread is never unlocked, so
the flag never reaches the agent's other threads, and no dynamic loader
runs before no_new_privs is set. The test-only agent start uses it
instead of running the bootstrap as root. The normal agent still accepts
no invocations.
* feat(lambda): open the authenticated control connection at guest startup
A normal lambda-agent build now reads its per-generation identity from
fw_cfg, unloads qemu_fw_cfg and refuses to start unless the fw_cfg tree is
gone, then dials the host under the pinned mutual TLS and holds the
connection. It accepts no invocations and never starts the bootstrap: any
data from the host fails closed until the message protocol exists.
The image loads qemu_fw_cfg and the virtio_net chain and configures the
guest interface from opt/spinifex/netcfg, as other system VMs do. The
lambdatestlink build also seals fw_cfg before starting the bootstrap.
* test(lambda): bound startup test waits and report host accept errors
* feat(iam): record quiescence evidence on the deletion job, with the L09 scope unresolved until the final inventory re-read
* fix(iam): cite ADR-0008:S9 as the basis for the STS consumer
* feat(lambda): add the Lambda runtime image and guest agent
* fix(lambda): stop the guest agent after a timed-out invocation
* docs(lambda): declare the L-GUEST executor-to-agent contract
* refactor(vm): remove the unused virtio-serial port builder
Lambda guest control moves to TCP over the microVM's network, so no
caller needs SerialPorts. This reverts d7905107b.
* fix(lambda): drop the serial carrier from guest startup and keep virtio-net
run-agent no longer waits for a carrier device named on the kernel
command line, init loads virtio_net instead of virtio_console, and the
image keeps virtio_net's failover and net_failover dependencies. The
normal agent still refuses to start without an authenticated transport.
* feat(lambda): authenticate the guest's TCP control connection with pinned mutual TLS
hostlink loads the per-generation identity from fw_cfg and dials the host
over TLS 1.3 with ALPN spinifex-lambda-guest/1. VerifyPeer requires exactly
the pinned certificate, validates it, and checks its single URI SAN names
the expected scope and role. The L-GUEST contract now describes this
transport; message framing and fencing stay open, and the normal agent
still refuses to start.
* feat(lambda): add the host end of the guest's TCP mutual TLS connection
guestconn creates fresh host and guest identities for each environment
generation, produces the fw_cfg entries the guest reads, and accepts only
that generation's exact guest certificate. Peers on the untrusted bridge
that fail the handshake are closed and the listener keeps waiting until
its context ends; the cancel callback finishes before the deadline is
cleared so the listener can be reused.
The pinned-certificate check and the L-GUEST vocabulary move to
foundation/pki and contracts/lambda/v1 so both ends share them.
* docs(lambda): describe the shared connection vocabulary in the contract README
* test(eks): specify the AWS-style cluster security group layout
* feat(eks): give clusters one AWS-style cluster security group
Create the eks-cluster-sg-<name>-<id> SG in the customer VPC with the EKS tags and a self rule, report it as clusterSecurityGroupId, launch workers with it (or the launch template's SGs), and attach it to the private-endpoint ENI. New clusters no longer get customer-VPC control-plane and nodegroup SGs; clusters created before keep theirs, and delete finds the new SG by tag.
* fix(eks): own the cluster SG by its recorded id and scope the delete sweep
Workers and teardown use the SG id recorded on the cluster; a create retry only adopts an SG carrying tags the service alone can set. DeleteClusterSGs now revokes every rule only on cluster-owned SGs and, on any other SG in the VPC, only the rules that reference one.
* docs(eks): use the cluster security group in workbooks and e2e
Workbooks read vpc_config[0].cluster_security_group_id and drop the hand
NodePort rules. E2E covers an internet-facing instance-target ALB with no
hand rule, exec and logs, and a bystander security group surviving
DeleteCluster.
* docs(eks): drop stale node-group SG sentence
* feat(operator): resume account deletions whose holder died
Each awsgw scans the deletion job bucket and takes over RUNNING jobs that
carry an accepted operation, a fence of at least 1 and an expired lease,
through the same fenced takeover a retried request uses. A resumed job runs
on a fresh timeout budget, so recovery no longer depends on the requester.
* chore: trim resumer comments to three lines
* refactor(cli): delete accounts only through the admin API
* docs: describe account delete as admin API only
* feat(iam): quiesce an account under a fenced deletion operation
* fix(iam): keep the termination capabilities off the shared interface and fence the account removal
* test(awsgw): guard the IAM termination binding on the operator deps
* docs: say which credentials the account delete signs with
* feat(lambda): add unregistered CreateFunction and GetFunction
Function records live in a daemon-owned per-account KV bucket, keyed by
Region and name, claimed with one conditional create that carries the
function ID and a Pending state. A claim whose code copy never completes
expires, so a later create can take the name over; the original gateway
then fails to mark its code stored. Code copies are never deleted here.
CreateFunction checks the ZIP is readable, refuses an execution role that
does not trust lambda.amazonaws.com (using the STS trust evaluation), and
encrypts environment values in the gateway with the master key, bound to
the account and function ID. GetFunction returns Code.Location as a
10-minute presigned URL for the off-host Predastore address, signed by a
session that outlasts it; with no off-host address it fails instead of
naming an internal one. Lambda uses the REST-JSON error envelope.
The handlers are built in the gateway but not routed: no public Lambda API
is registered until a runtime image exists.
* fix(lambda): return and accept only the default LoggingConfig
Every function configuration now carries LoggingConfig with LogFormat Text
and LogGroup /aws/lambda/<name>. CreateFunction accepts the default when
supplied and refuses a JSON format, another log group or a log level.
Adds an integration test that drives a signed CreateFunction and
GetFunction through the gateway, the NATS record owner and the code store,
then downloads Code.Location and checks the bytes. The STS trust check
moves to stsdomain.RoleTrusts so the test uses the production evaluation.
* fix(lambda): render the stored record in the CreateFunction response
CodeStored now returns the record the owner wrote, and CreateFunction
renders it instead of stamping its own time, so CreatedAt, LastModified
and RevisionId match an immediate GetFunction.
* fix(lambda): match IAM error codes, not messages, in code-store provisioning
IAM errors now carry a message, so comparing err.Error() with the bare
code no longer matched EntityAlreadyExists or NoSuchEntity. Provisioning
then failed on every start after the first and could not adopt its own
role. Resolve the AWS error code instead.
* test(iam): check the resolved NoSuchEntity code in UpdateUser tests
IAM errors now carry a message, so err.Error() no longer equals the bare
code. Assert the resolved AWS error code instead.
* test(iam): account for UpdateUser as named by bare name
UpdateUser addresses its user by bare name, so the cross-tenant ARN table
cannot target another account; list it with the other byName IAM actions.
Add a focused test that a rename stays within the caller's account.
Lambda code copies live in a system-account Predastore bucket reached
only through a short-lived session on a Regional role scoped to that
bucket. STS gains an in-process AssumeCodeStoreRole that derives the
Region's code-store role itself and assumes it as lambda.amazonaws.com;
it is not on the STS interface, NATS or HTTPS.
Provisioning creates or verifies the owner-tagged role, inline policy
and bucket on first use only, retries Lambda's own partial create,
and refuses an untagged or foreign resource with an operator-visible
reason. Copies are keyed by their SHA-256, so different bytes can never
replace a recorded revision. The gateway holds the root client and IAM;
the Lambda package receives only the bounded provisioning operations
and the scoped store.
* test: pin AWS-format EKS add-on versions
* feat: return EKS add-on versions in AWS format and accept them as input
* test: DescribeAddonVersions query parameters, compatibilities and paging
* feat: honour DescribeAddonVersions filters and report compatibilities
* test: EKS parameter refusals must carry a code the operation declares
* fix: EKS parameter refusals use a code the operation declares
* docs: describe AWS-format EKS add-on versions
Document the v<x.y.z>-eksbuild.<n> form, the bare-form leniency, the
DescribeAddonVersions filters and compatibilities, and how to pin a
version in Terraform. Replace the leave-addon_version-unset guidance.
The four crash cases each restarted the store, which on the three-node
cluster means a full Raft re-election every time. They now set up in
separate namespaces, crash once, and run the same post-crash checks as
named subtests, cutting TestR3/Conformance from ~39s to ~11s.
* refactor(compute): extract a shared direct-boot microVM builder
runtime/compute/vm gains MicroVMSpec, NewMicroVMConfig and
MicroVMMachineType, with CPU mitigations on unless a caller sets
MitigationsOff. The ELB and SQS system-instance launches pass their own
settings to it; only the load balancer opts out of mitigations. A golden
test pins the full QEMU argv for both launches, unchanged by the move.
* test(daemon): stub QEMU on PATH in the microVM argv golden test
Execute resolves a QEMU binary before returning the command, and CI
runners have none installed. The test now puts an empty
qemu-system-x86_64 on PATH, so it checks the real argument construction
without QEMU.
TestR3 restarts real three-node JetStream clusters and waits out Raft
elections, adding ~40s to the slowest preflight package while covering
~0.1% of its statements. It now skips unless SPINIFEX_SQS_R3 is set, and
make test-race sets it, so CI's Race Detection job runs it on every PR.
* fix(awsgw): admit accounts with a synchronous in-process status read
* fix(iam): refuse to move the system and admin accounts off ACTIVE
* fix(awsgw): deny a missing account record instead of failing the request
install.mulgadc.com serves the latest release's setup.sh, so a --channel dev
or --version install ran an installer older than the tarball it unpacked and
missed files it did not know about, such as lib/distro.sh. setup.sh now hands
off to the tarball's own setup.sh, with the same arguments, before it changes
anything on the host.
The Vultr guide now installs from the latest release instead of dev.
* test: pod identity webhook mutation, publisher and webhook config contracts
* feat: pod identity webhook mutator, CA bundle publisher and webhook configuration
* feat(eks): run the pod identity webhook on every control-plane server
Add the eks-pod-identity-webhook binary and its OpenRC service. It listens on
127.0.0.1:8444 with a serving certificate signed by the k3s server CA, authenticates
to the API server with a client certificate signed by the k3s client CA, and runs
under its own least-privilege ClusterRole. Each server stages the RBAC and the
MutatingWebhookConfiguration (caBundle is the server CA) into the k3s manifests
directory, so server-join nodes get them too.
Match upstream's container skip rule: skip only when the regional STS key is
present, and add the token mount otherwise.
* fix(eks): issue pod identity webhook certs for one year and renew in the running process
Both the serving and client certificates were valid for ten years. Kubernetes
cannot revoke a client certificate, so a copy taken off a control-plane disk
would stay a valid identity for a decade. Issue both for one year with the
existing 30-day renew window.
The process loaded the serving certificate once. A daily check now exits
with ErrCertRenewal when either certificate enters the renew window, so the
supervisor restarts it and EnsureCerts reissues.
Add tests for KubeClient against a TLS fake API server and for Run, which the
package had none of.
* test: e2e covers pod-identity webhook and unmodified LBC chart
IRSAPod now uses a ServiceAccount annotation only and asserts the injected
env and volumes plus an untouched unannotated pod. LBCChart installs the
upstream chart 1.11.0 and reconciles an Ingress into an active ALB.
* test: pass the tagging endpoint to the LBC chart and mark Shield/WAF flags Spinifex-only
* fix: make the pod identity webhook init script executable in the eks-node image
setup.sh chmods each shipped init script to 0755 explicitly, and the new
service was missing from the list. rc-update then refused it, and
eks-node-role, running under set -eu, aborted before starting k3s.
Add a guard test that every init script in INSTALL_FILES is chmod'd.
* fix: start the pod identity webhook last and keep its failures non-fatal
eks-node-role runs under set -eu, so a fault enabling or starting the
optional webhook aborted the control plane boot before k3s-first-boot,
state-report and addon-sync. Enable and start it at the end of the server
and server-join roles and log a warning on failure instead.
The role test now asserts the webhook is the last call and that an enable
or start failure still exits 0 and finishes the role.
* fix: move the pod identity webhook off the token webhook readiness port
eks-token-webhook already serves its plain HTTP readiness endpoint on 8444,
which the API server target group health-checks. The pod identity webhook
bound the same port, failed with address already in use and respawned
forever, so no pod was ever injected.
Listen on 127.0.0.1:8445 and add a test that the two ports differ.
* docs: describe pod identity webhook for user workloads in the EKS guide
* fix: split STUB_FAIL assignment from export in the role test
* feat(eks): converge DeleteCluster's teardown behind the recorded DELETING
DeleteCluster ran the whole cluster teardown inside the NATS request. Each
control-plane VM that ignores ACPI costs 40 s (20 s graceful powerdown, then
20 s waiting for the PID file), so a two-member cluster overran the 30 s
budget: the caller got a 500, and its retry queued behind the first handler
on the serial subscription and got a 404.
The request now validates, answers a DELETING cluster from its record,
records DELETING and returns. The teardown runs in a background converger
started by the request and backed by the deleting reaper:
- An in-process per-cluster gate sits in front of the teardown lease, which
re-adopts a key its own node holds; one per-node bound covers kicks and
reaper hand-offs.
- The lease renews on the service context, not the request's, and the
converger stops at a step boundary if it is lost.
- The converger acts only on a record that is DELETING when re-read, refuses
to remove a record that still has a node group, deletes the cluster's
child records before the meta, and deletes the meta only if no ownership
ref appeared that its pass did not tear down.
- DELETING is sticky for every status writer, and the create launch's meta
writes add their refs to a DELETING record instead of overwriting it, and
never recreate a swept one.
- The reaper hands work to the converger instead of running it inline,
retries at a backoff capped at 30 minutes without ever abandoning a
record, and resumes records an older build flagged exhausted.
- A stuck delete is surfaced in DescribeCluster as one INTERNAL_FAILURE
issue naming the step, never the internal error.
* fix(eks): refuse DeleteCluster while node groups are attached
AWS answers ResourceInUseException ("Cluster has nodegroups attached") when
managed node groups remain. DeleteCluster went ahead instead, and its sweep
erased the node group records, leaving their workers running in the
customer account with no record to own them and no control plane.
The request now decides that error before recording anything. The e2e
suites, the terraform workbooks and the account teardown engine all delete
node groups first; the console shows the refusal's message.
* fix(eks): refuse new add-ons and access entries on a deleting cluster
CreateAddon and CreateAccessEntry checked only that the cluster record
existed, so a dependant could be written to a cluster whose delete was
already recorded, and the teardown would then sweep it with the record.
Both now answer ResourceInUseException for a DELETING cluster, a code both
operations declare in the EKS model.
* test(e2e): DeleteCluster returns DELETING at once and never reads ACTIVE after
The first delete-cluster must return within 5 s with status DELETING, as
AWS does. While the cluster is still listed every DescribeCluster must read
DELETING, and a repeat delete must answer DELETING from the record; once it
is gone a further delete answers ResourceNotFoundException.
* docs(eks): record the cluster delete lifecycle contract
The contract ADR-0003 S1 asks for before a conformance claim: what the
request decides, the recorded obligation, how the teardown converges and
recovers, and the S8 and S7 tests that pin each item, with the remaining
gaps listed.
* test(eks): wait for the delete request before counting terminates
On a build whose DeleteCluster is synchronous the request goroutine is still
recording terminates when the assertions read, so the tests must wait for it
to return as well as for the background teardown.
* test(eks): classify every eks request subject by its handler's work
The ADR-0003 S8 subject table for EKS. Every eks.* subject the gateway sends
and the daemon serves is classified in eksSubjects, and the table fails for
one that is not, so a new subject is classified deliberately rather than by
inheriting the 30 s default. DeleteCluster is request-scaled. Four handlers
still hold the request for work that grows with the resource and are named
as not conformant: CreateCluster's FAILED-name reclaim, DeleteNodegroup,
UpdateNodegroupConfig and RestoreSnapshot. Their budget stays 30 s, since a
larger one would hide the defect.
* fix(eks): leave no child record behind a launch overtaken by a delete
A create launch still running when its cluster was deleted and swept could
seed the creator's access entry afterwards, leaving it under a cluster with
no record. The launch now re-checks the record after seeding and, finding it
gone, removes the cluster's child keys with the infra it holds. The teardown
also sweeps child keys again after deleting the meta, for a write that lands
between its two deletes.
* fix(eks): fence the create launch on its generation and keep its infra owned
Review findings on the async delete, each with a test that fails before:
- The launch's meta write checked only the status, so a launch overtaken by
a delete could overwrite a newer cluster of the same name and then release
by name. Writes are now fenced on CreatedAt and the ARN; a superseded
launch releases only the instances it started, by ID.
- A launch whose record was swept released its infra once, best effort. It
now recreates the record as DELETING with its refs, so the teardown and
the reaper own and retry the release.
- A record marked FAILED under a running launch was treated as owned by a
teardown; the launch now merges its refs and releases what it built.
- Child records: a fresh claim clears keys an earlier cluster of the name
left, the teardown stops the cluster's bootstrap subscriber, bootstrap
refuses to write into a record that is gone or DELETING, the creator's
access entry and the GPU device add-on are gated on the record.
- Deleting the last node group of a DELETING cluster resumes its delete,
and waiting on node groups no longer counts as a failed pass.
- One converger run is bounded at 15 minutes, so a hung step cannot hold
the gate and lease for ever; the timed-out pass is recorded.
- The FAILED-name reclaim purge runs under the teardown gate and lease.
- The meta-last test drives the converger with a failing KV instead of
DeleteClusterPrefix, which production no longer called and is removed.
- The S7 test asserts the teardown issues no volume delete and terminates
no worker.
* fix(eks): keep a deleted cluster's record while its create launch runs
Second review of the async delete, each behavioural change with a test that
fails before:
- A create launch now heartbeats its owner and time on the record, and the
teardown keeps the record while that heartbeat is fresh, so DescribeCluster
never answers NotFound and then DELETING again when an overtaken launch
recreates the record. The launch hands its refs over at its next write and
starts the teardown when it returns; a heartbeat silent for 90 s is taken
as a crashed launch, and recreating a swept record stays as the backstop.
- The launch's OIDC keypair writes are fenced on the record's generation, so
an overtaken launch cannot replace a newer same-name cluster's signing key.
- Re-creating a FAILED cluster's name is no longer refused by the per-node
teardown bound.
- A deadline reached while removing the record is still recorded; an exited
bootstrap subscriber removes its own entry; a record that is not DELETING
is refused as not recorded even when it has node groups.
The guest's SQS nats-server listens on 127.0.0.1 and the worker dials it
there, but the kernel leaves lo down, so the store failed to bind and the
worker never polled. Also correct the stale CA comment in init.
DescribeTargetGroups rejects a malformed target group ARN with ValidationError
and reports a missing group with AWS's message, DescribeLoadBalancers fails with
LoadBalancerNotFound for an unknown name, and DescribeListeners fails with
LoadBalancerNotFound for an unknown load balancer ARN. The EKS cluster NLB lookup
treats the new not-found as absent.
* feat(sqs): run queue and message owners in the worker by signed pull
The SQS worker's ready step now opens the queue and message owners on
its dedicated store and serves owner calls it pulls from its node's
gateway over the existing instance-role channel (PollWork,
CompleteWork). Each poll and post is authorized against the current
worker identity, and a result is accepted only from the identity that
took the work. No payload crosses platform NATS.
A receipt-signing key is created on the worker disk only under the
first-store decision that creates the store key, and its fingerprint
is bound in the store; a missing or changed key refuses to serve.
Received messages are released only when the gateway proves the caller
left; receives are capped so a worst-case result fits one post.
The public relay is not registered in production.
* fix: harden SQS relay results and bound pending bytes
Unavailable outranks client faults when encoding owner errors, and wire
errors carry fixed text only, never owner or backend text. The ingress
rejects incomplete or mismatched success results as unavailable, and the
relay bounds total pending input bytes as well as request count.
Adds tests of every relay action against real owners, the receive claim
limit, release, faulting work, the receipt-key fingerprint marker and the
poll loop through gateway failures.
* fix: refuse SQS serving outside a single-node host cluster
DescribeStore now reports the gateway's declared cluster node count, and
the worker refuses to serve, before reading keys, connecting or creating
any store state, unless that count is exactly one. A gateway that reports
no count reads as zero and is refused.
* feat(sqs): record a store server key pin once per store instance
Add the store role and its internal actions so the bound store instance
can learn its claim and record the public pin of its TLS server key,
once, without the platform holding the private key. The worker sees the
store as bound only after a pin exists and must receive a valid pin.
* refactor(sqs): serve store server keys from a separate capability
Keep the worker Stores interface unchanged and give the store guest's
actions their own ServerKeys dependency, so existing Stores fakes need
no store-key methods.
* feat(sqs): launch the store with a mgmt NIC, CA and scoped store role
The store now boots with the region, gateway URL and the daemon's own
cluster CA, a mgmt NIC to reach the gateway, and an sqsStoreRole profile
granted only the store's two internal actions, so it can authenticate
its server key pin. A store record that predates these fields refuses
to relaunch; its volume is kept.
* feat(sqs): keep the store TLS server key on its disk and record its pin
The store image gains sqs-store-key, run after the data disk is mounted.
It loads or creates a P-256 server key under the disk's tls directory,
records only the key's public pin with the gateway as the bound store
instance, and writes a self-signed certificate for it to the run
directory. nats-server then requires TLS 1.3 with that key.
A record for another volume, a recorded pin with a missing key, or a key
that does not match the recorded pin refuses the store; no key is
regenerated and nothing on the disk is deleted. Launch-not-yet-bound and
lost replies are retried. The store role vocabulary moves to
contracts/sqs/v1 so the guest agent does not import a domain.
* feat(sqs): pin the worker's store connection to the recorded server key
The worker's readiness check and serving connection now use TLS 1.3 and
accept only the store server key whose pin the bound store recorded with
the gateway. The pin is checked before the worker key or any queue or
message data is sent, and on every reconnect. A store with another key,
a plaintext store or a malformed pin is refused as final.
The worker NKey client authentication is unchanged. The host still
holds the store's data disk key, so the pin does not protect the store
from a privileged host.
* fix(sqs): retry store key bootstrap and track guest binary sources
The store's server key check moves from sysinit into the respawned run
step: a gateway or IMDS outage exits after a pause so init retries it,
without remounting or creating a new key, while a missing, changed or
wrong-store key is recorded as refused and never retried. Stop also ends
a running key helper before unmounting.
The guest helper and worker binaries now depend on every main-module
package dir and file they link, from go list, so edits, additions,
deletions and renames all rebuild them.
* feat(sqs): operator enable and status with a signed ingress test
Add a per-Region SQS service record (disabled by default, pinned to one
node, refused on multi-node clusters before any write), gate the relay on
it, and expose `spx admin sqs enable` and `spx admin sqs status`. The
gateway now hands queued work to the polling worker and reports whether
one is polling.
The public SQS service stays unregistered in production until its
capability profile is reviewed. An integration test registers it through
the harness and drives real SigV4 and IAM requests to a real worker for
create, send, receive and delete, asserting no SQS body crosses platform
NATS.
* fix(sqs): report readiness only from observed component state
Status now reports whether the tenant SQS API is registered on this
gateway and is never Ready while it is not. The worker stops polling while
its store connection is down, so a worker that lost its store drops out of
the gateway's poller window instead of looking healthy.
* wip: sqs model inventory
* wip: sqs public registration
* wip: sqs registration review fixes
* fix(sqs): refuse pre-API store and worker records on enable
A store or worker record this release creates now carries an explicit
contract. enable reads both records without creating a bucket and refuses
any record without it before it writes desired state or launches anything;
status names the record and is never Ready. A worker replacement keeps its
predecessor's contract, since it reuses the same data volume.
Missing metadata is only ever grounds to refuse: nothing is upgraded,
inspected, emptied, replaced or deleted. Upgrading a pre-API dev install is
not supported.
* fix(sqs): describe the enable refusal as an unsupported launch contract
A missing or unknown contract proves neither when a record was written nor
that it predates authenticated bootstrap: an older writer can drop a newer
field, and a future contract is not pre-API. The refusal is unchanged; its
message, status field and docs now name a missing or unsupported launch
contract and say any existing volume is kept.
* wip: port reviewed SQS serving path onto dev without the store VM
* wip: single SQS system VM
* fix(sqs): retry receipt bootstrap under supervision and drop store install path
Run the receipt-key boot step from the respawned store service so a
transient gateway or IMDS outage recovers without a reboot; final key
and volume refusals stay refused. Remove the separate store image
install and staging path from setup.
* fix(sqs): align worker ready config fields
* fix(sqs): stop an in-progress boot before unmounting the worker disk
The receipt-key boot step ran as an unrecorded child of the store shell,
so shutdown stopped only the shell and could unmount the disk under a
boot still talking to the gateway. Record the boot child like the other
guest processes and stop it before unmounting; a survivor keeps the disk
mounted.
* fix(sqs): record the boot child before its stopping check
The parent wrote boot.pid only after starting the child, so a shutdown
in that gap could unmount under an unrecorded boot. The child now
records itself and checks stopping before it runs sqs-worker boot.
Bumps the docker-images group with 1 update in the /build directory: golang.
Bumps the docker-images group with 2 updates in the /docs/terraform-workbooks/demo-app directory: golang and distroless/static.
Bumps the docker-images group with 1 update in the /scripts directory: debian.
Updates `golang` from `972b411` to `d9b6a1b`
Updates `golang` from `f2c872b` to `d200d2d`
Updates `distroless/static` from `18f4cc2` to `25ac8f0`
Updates `debian` from `62bd423` to `f6b08b0`
---
updated-dependencies:
- dependency-name: golang
dependency-version: 1.27.3-bookworm
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: docker-images
- dependency-name: golang
dependency-version: 1.27.3-alpine
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: docker-images
- dependency-name: distroless/static
dependency-version: nonroot
dependency-type: direct:production
dependency-group: docker-images
- dependency-name: debian
dependency-version: trixie-slim
dependency-type: direct:production
dependency-group: docker-images
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(test): never hand KillProcess a PID a real process can hold
* fix(iam): queue same-record writers so a looping self-write cannot starve DetachUserPolicy
* fix(ecs-agent): report RUNNING before starting container exit-waits
* fix(rds): wait for the lease-loss cancellation instead of sleeping in the takeover test
* fix(cli): name the account when the follow deadline lands mid-poll
* test(imagecatalog): never answer from the deadline fixture
* test(nats): order Gather StopOnFirst frames causally, not by sleep
* fix(ec2/key): find a key pair by name past an unreadable metadata record
* fix(accountlifecycle): re-list at the drain deadline so a just-deleted resource is not stuck
* fix(eks): give only the exhausting subtest the short SG delete budget
* fix(eks): stop healthz and nodegroup Ready-gate tests racing fixed wall-clock windows
* fix(test): let in-process NATS clusters bind their own route ports
* fix(objectstore): list the memory store in S3 key order with real paging
* test(viperblock): create a fresh volume on every run of the local-state test
* test(ochre): close router streams so drained tests release their admission slot
* test(cli): restore shared cobra flags after each ochre command test
* test(daemon): keep shared-JetStream, metric and RSS tests correct on repeat runs
* fix(ochre): let a warm-up probe stop so a test's poller cannot hit a later test's server
* test(cli): pin the package's cluster size so a multi-node config load cannot strand later fixtures
* feat(admission): add the quota override seal capability contract
* feat(awsgw): serve the quota override seal capability per node
* refactor(accountlifecycle): parameterise the seal capability probe and add the quota one
* test(accountlifecycle): pin the probe wire shape against both capability contracts
* feat(accountlifecycle): seal the quota override record during account teardown
* docs: record the quota override seal contract and teardown behaviour
* docs: fill in the quota override seal PR number in the package boundary migration
The ISO cells build the rootfs as root inside the workspace and only cleaned
it at the start of their own job, so the next job on the same runner from any
workflow failed its checkout on files it could not remove.
* fix(eks): delete the Node of a terminated nodegroup worker
K3s runs no AWS cloud provider, and its own reports every instance as
existing, so a worker terminated by DeleteNodegroup, scale-down or reclaim
stayed a NotReady Node forever. Its pods stayed bound to it, and once most
Nodes were NotReady the node lifecycle controller stopped evicting, leaving
metrics-server on a dead Node.
Add an internal-workers route the control plane pulls the cluster's live
nodegroup workers from, and a mulga-eks-node-gc agent on every server that
deletes a nodegroup Node only when it is not Ready and its instance is no
longer one of those workers, as the AWS cloud controller does.
* test(e2e): check the live worker's Node and wait out nodegroup deletes
WorkerNodeReady accepted any Ready Node, the tolerate-all DaemonSet check
dropped pods with no Ready condition, and nodegroup cleanup returned before
the nodegroup was gone. Scope the Ready check to the CPIsolation nodegroup,
count a pod with no Ready condition as not Ready, wait for each nodegroup
delete, and assert that deleted nodegroups leave no Node behind.
* test(integration): account for the internal ListInternalWorkers route in the cross-tenant table
NetworkManager restarted while /etc/resolv.conf is a plain file runs in
rc-manager=symlink mode and writes resolv.conf on its first DNS update, which
can land after the installer links the stub. Write a conf.d drop-in
(dns=systemd-resolved, rc-manager=unmanaged) before the restart, refuse a
later file that overrides rc-manager, wait for NetworkManager to settle, and
assert the link holds with bounded relinks. A rerun after an older handover
adds the drop-in, reloads NetworkManager and relinks the stub.
2026-11-01 10:30:43 -06:00
2927 changed files with 160322 additions and 36986 deletions
<!-- Complete this section only when the PR changes account-scoped resource lifecycle, account termination, resource cleanup, or a lifecycle boundary. Otherwise write: N/A. -->
- Lifecycle owner and resource family:
- Owner-local ordering, recovery, retained-state, or force impact:
<!-- One line, so a reviewer knows whether the design records in the private mulga repository need to change. Write either "Updated: <mulga PR or bead id>" or "Not needed: <reason>". This is a reminder, not a gate. -->
## Community Contribution License
- [ ] All community contributions in this pull request are licensed to Mulga under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). By creating this pull request I represent that I have the right to license these contributions to Mulga under the Apache License 2.0.
description:"Space-separated topologies (blank = vm-single vm-multi bm vm-single-principal). One name runs one topology."
description:"Space-separated topologies (blank = vm-single vm-multi bm vm-single-principal vm-single-ocibsd vm-multi-ocibsd). One name runs one topology."
required:false
workbooks:
description:"Space-separated workbooks (blank = the driver's default list)"
@@ -62,9 +62,10 @@ jobs:
# a dedicated label keeps a two-hour cloud wait off the hypervisor labels and
# `main` and `dev` as separate nightlies, and it would always execute `main`'s
# copy of this file — leaving a fix to the workflow itself untestable until it
# had already shipped. A dispatch takes both the file and the source from its ref.
#
# Cells (matrix cells are defined in .github/e2e-cells.json):
#
# Single node Three nodes
# 1 Debian 13, DHCP 8 Debian 13, DHCP
# 2 Ubuntu 26.04, static 9 Ubuntu 26.04, static
# 3 Ubuntu 26.04, DHCP 10 Ubuntu 26.04, DHCP
# 17 tofu examples 29 storage fault injection
# 18 reboot 30 routed NAT
# 19 routed NAT 31 instance recovery
# 25 eks 32 load balancer recovery
# 26 ecs 33 KV bucket quorum under node loss
# 27 rds 37 Oracle Linux 10, DHCP
# 34 Oracle Linux 10, DHCP 38 Oracle Linux 10, routed NAT
# 35 Oracle Linux 10, reboot
# 36 Oracle Linux 10, routed NAT
#
# Standalone jobs
# 20 bare metal (PXE) 23 predastore throughput
# 21 ISO, single node 24 predastore fault injection
# 22 ISO, three nodes 28 guest disk performance (bare metal)
on:
workflow_dispatch:
inputs:
cells:
description:"Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
description:"Comma-separated cell numbers, no spaces, e.g. 1,25,27 (blank = all). The cell list is at the top of .github/workflows/e2e-nightly.yml."
required:false
permissions:
@@ -557,6 +578,8 @@ jobs:
echo \"::group::suite ${SUITE}\"; \
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
test"$$host"= x86_64 &&test"$$goos/$$goarch"= linux/amd64 ||{echo"the test link supports only an x86_64 host building linux/amd64, not $$host building $$goos/$$goarch";exit 1;}
test"$$host"= x86_64 &&test"$$goos/$$goarch"= linux/amd64 ||{echo"the test link supports only an x86_64 host building linux/amd64, not $$host building $$goos/$$goarch";exit 1;}
| **SQS** | Message queues | A dedicated NATS JetStream store in a per-Region QEMU `microvm` guest | Q1 2027 |
Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console dashboard" width="900">
<img src=".github/assets/dashboard.webp" alt="Spinifex web console dashboard" width="900">
</p>
Spinifex ships with a built-in web console for your instances, volumes, buckets, networks, and IAM. See [Launching the Web UI](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui).
{From:"agents/ecs/agent",To:"domains/ecs",Debt:"agents/ecs/agent",Reason:"assignment/state wire types from the ECS implementation; needs a guest/controller contract"},
@@ -38,6 +34,7 @@ var allowlist = []allowance{
{From:"cmd/aws-model-coverage",To:"domains/ecs/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/eks/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/iam/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/sqs/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/sts/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared service inventories; named exception, must not construct a registration"},
{From:"runtime/daemon",To:"domains/ec2/awsapi/instance",Debt:"Instance adapter consumers outside EC2",Reason:"runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
a request/reply route on which a gateway, in the `awsgw-quota-seal` queue
group, irreversibly seals one account's quota override record.
`domains/admission/quota` owns it: `awsgw` serves it.
`accountteardown` is the consumer: it calls this subject to fence every
`operator/accountlifecycle` is the consumer: it calls this subject to fence every
override writer at the quota owner before the account record is removed.
The request names the account in its body (`account_id`) and may carry an
@@ -31,10 +31,20 @@ Any current gateway may answer; the leader-accepted revision is the evidence
whichever one does, and the override bucket is opened at seal time even on a
gateway whose quotas were disabled at startup.
## Seal-quota-override capability
`AccountQuotaOverrideSealCapabilitySubject(node)` (`admission.v1.AccountQuotaOverrideSealCapability.<node>`) is a per-node, read-only request/reply route on which `operator/accountlifecycle` asks one configured gateway whether it supports the quota override seal, immediately before every seal attempt.
The quota owner (`domains/admission/quota`) owns it: `awsgw` serves it with no queue group, so a request addressed to one node reaches only subscribers configured as that node, and two gateways sharing a node ID both answer.
The consumer's quota seal reaper probes every configured node that runs `awsgw` and sends no seal request unless every one answers compatible.
`AccountQuotaOverrideSealCapabilityVersion` is a claim about the whole binary's writers, not about the responder: version 1 means every quota override writer in the binary is revision-guarded and refuses a sealed record, and the binary serves the seal subject.
A gateway binary without this responder has no subscriber, so the probe for that node reads as incomplete (`no-reply`), the same as a stopped or partitioned node.
The reply proves nothing beyond the moment it was sent: a gateway downgraded or replaced after a successful probe is outside the guarantee.
### Compatibility boundary
The subject name carries the contract major; `account_quota_seal_test.go`
preserves the version-1 JSON shape.
The subject name carries the contract major; `account_quota_seal_test.go` and `account_quota_seal_capability_test.go`
preserve the version-1 JSON shapes and the subject builder.
Producer and consumer are in the coordinated Spinifex release, so the window
is N and N-1.
A gateway that predates the subject has no subscription, so the request gets
@@ -23,7 +23,7 @@ subject shape are wire compatibility. Callers and receivers must use this
package rather than rebuild those details. `commands_test.go` preserves the
version-1 JSON shape and subject form.
This package owns shared command payloads and the drain-volume response. It
This package owns shared command payloads. It
does not define every action's reply body, authorize the request, store EC2
state or implement NATS transport. Those remain responsibilities of the EC2
domain and its runtime composition.
@@ -37,6 +37,13 @@ contract or an approved compatibility plan; it is not a package refactor.
Only the subject is owned here today.
The request (`account_id`) and reply (`vpc_id`, `error`) bodies are still declared as anonymous structs on each side; moving them into this package is a contract change that needs its own characterization first.
## Volume held elsewhere
A start refused because another node holds one of the instance's volumes is an `IncorrectState` error payload on the `ec2.start`, `ec2.start.<node>` and `ec2.cmd.<instance-id>` replies, and the AWS body cannot say why.
The owning node adds the `X-Volume-Held-Elsewhere: true` header to that reply, the `ec2.start` forwarder relays it, and `StartInstances` wraps `ErrVolumeHeldElsewhere` around the error it rebuilds, so a caller matches it with `errors.Is` rather than reading the message.
The owning node sets the header only from the EBS provider's `volume_held_elsewhere` code, which every provider returns for this refusal; no reader matches the message text.
`VolumeHeldElsewhereText` is the wording viperblockd puts in that refusal's message, not a signal.
## Platform inventory
`PlatformInventorySubject` (`ec2.v1.PlatformInventory`) is a request/reply route on which a daemon in the `spinifex-workers` queue group reports one account's key pairs, placement groups, launch templates and their `ec2-tags` entries from the EC2 owner stores.
This package will own the Lambda process contracts declared by ADR-0009:S13.
It holds the L-GUEST declaration and, in `guest.go`, the connection vocabulary both ends share: the scope, the URI SAN roles, the ALPN identifier and the fw_cfg entry names. The transport and authentication are decided (ADR-0009:S17); the invocation message types and framing are not chosen, so no message types are defined here.
It is not an AWS tenant-facing API.
## L-GUEST: node executor to guest agent
**Owner:** the Lambda domain.
**Producers and consumers:** the node-local executor (`runtime/compute/lambda`) and the in-guest Lambda agent (`agents/lambda`), in both directions.
**Transport:** TCP over the microVM's network (ADR-0009:S17).
The agent dials one connection to a listener the executor owns for the environment, and the guest exposes no listener for host control.
At most one authenticated control connection is active per environment, and the agent may reconnect; how a prior connection and its in-flight work are handled is open (ADR-0009 M1).
The network is shared and untrusted: no source address, port or bridge membership is identity or authority.
Tenant code may be able to reach the listener, but it cannot open the agent's connection or key material, so any connection it makes fails authentication.
**Authentication:** mutual TLS 1.3, bound to the environment ID and realization generation, using material the user runtime cannot read (ADR-0009:S17).
For each generation the executor creates ephemeral self-signed host and guest certificates whose single URI SAN is `spinifex-lambda://<Region>/<account>/<function ID>/<environment ID>/<generation>/<role>`, with role `host` or `guest`.
Each end validates the peer certificate against the one pinned peer certificate and separately requires that URI SAN with the opposite role; hostname verification is not relied on.
ALPN carries the L-GUEST major version, `spinifex-lambda-guest/1`, and an unsupported version fails the handshake. Builds with the `lambdatestlink` tag offer only `spinifex-lambda-testlink/3` on both host and guest, whose first frame carries the execution-role credential and function environment, so a test-link end and a release end fail the handshake; release builds are unchanged.
In the same builds, the gateway reaches the host daemon on the local abstract socket `spinifex-lambda-testlink-control/4`, and each request carries the runtime-image digest the gateway was built with; the daemon refuses a missing or different digest before starting the attempt, and a gateway and daemon on different socket names never connect.
On that socket a request's raw payload is at most 6 MiB, checked by the gateway before encoding and by the daemon after decoding.
Its encoded frame is at most the base64 length of 6 MiB plus 64 KiB of metadata headroom and 64 bytes of field slack, and a receiver checks the frame length before allocating; metadata has no separate bound within the frame.
The credential and environment travel as their own frame of at most 40 KiB, and the reply has the same payload and frame bounds as the request.
The daemon holds at most two connections at once, from accept to close, and leaves the rest unaccepted in the listen backlog.
Under TLS 1.3 the guest's handshake can complete before the host refuses its certificate, so the guest treats the connection as accepted only after the host's first message.
In the same builds, the function owner moves a created function with stored code from `Pending` to `Active` once this node's daemon serves the pinned image and the cluster configuration has exactly one node; a code copy over the test-link bound stays `Pending`, and `StateReason` names the unmet condition.
Release builds never leave `Pending`.
The Mulga deploy role installs a test-link build with `deploy_lambda_testlink=true`; this profile does not meet Q-199 fail-closed revocation, and its guests take management addresses whose record can expire (mulga-0hsub).
**Bootstrap material:** the executor passes these fw_cfg entries under `opt/spinifex/lambda/`, and the agent reads them as root and unloads `qemu_fw_cfg` before user code starts.
| Entry | Content |
|---|---|
| `guest-key` | Guest private key, PEM |
| `guest-cert` | Guest certificate, PEM |
| `host-cert` | The one pinned host certificate, PEM |
The guest's own interface settings come from `opt/spinifex/netcfg`, the entry other system VMs use.
The agent unloads `qemu_fw_cfg` even when an entry is missing, and refuses to start if the fw_cfg tree is still readable afterwards.
On a VMM without fw_cfg (the test-link Firecracker executor), the executor attaches the same entries as a tar archive on the second, writable virtio-blk disk, laid out as `by_name/<entry>/raw`.
The runtime image's init extracts it to a root-only tmpfs at `/spinifex-cfg`, and the agent reads the entries there, then overwrites the whole disk with zeros and removes the tree; it refuses to start if the tree is still present.
Until the message format exists, the agent authenticates and holds the connection but accepts no host message and starts no user code.
**Authority:** one environment's attempt only.
The user runtime never holds these keys, and sharing the guest with the agent gives user code no platform authority.
**Idempotency and fencing:** an invoke carries the attempt-grant identity and the attempt ID.
The agent accepts a run command only under a current grant, and a completion under a superseded grant is discarded and reported.
How a grant is enforced and proven ended is open (ADR-0009 M1).
**Errors and versions:** the agent and executor fail closed on an unknown protocol version or an identity mismatch.
**Readiness:** environment readiness needs authenticated, current-generation agent evidence; a running hypervisor process does not count.
**Payloads:** the contract must carry a synchronous payload and response of up to 6 MB and an asynchronous payload of up to 1 MB, plus framing, over a path whose size bound is declared.
How payloads move is open (ADR-0009 M3).
### Compatibility boundary
The agent ships inside the runtime image pinned by each published version, so it is outside the coordinated release train.
The executor supports every L-GUEST major carried by a still-supported runtime image.
Retiring an image follows a published runtime-deprecation lifecycle that blocks create and update before it blocks invoke.
Evidence required before the first agent image ships: a function on an old image invoked by a new executor, and the reverse during rollback.
## L-RUNTIME
The guest agent serves the AWS Lambda Runtime API, version `2018-06-01`, to the user runtime at `AWS_LAMBDA_RUNTIME_API`.
Spinifex does not version this AWS contract, and conformance is measured against AWS.
This package owns the deployed, transitional configuration-update and
state-reload request/reply protocols between Spinifex's legacy control plane
and `viperblockd`. It makes those existing compatibility boundaries explicit
while the generic `providers/ebs` contract is adopted; it is not the EBS
provider abstraction and new provider integrations must not depend on it.
## Configuration update routes
`ConfigUpdateSubject` is `ebs.config`. Every `viperblockd` daemon subscribes
to it in the `spinifex-workers` queue group. It is the fallback path for an
encrypted volume with no live mount: the selected daemon opens the volume
exclusively and reseals its state.
`VolumeConfigUpdateSubject(volume)` is `ebs.config.<volume>`. The daemon with
the live mounted volume subscribes directly so the sole holder of the volume's
state sequence number applies the update.
Both routes carry `EBSConfigUpdateRequest` and respond with
`EBSConfigUpdateResponse`. The JSON field spelling, route shapes, and
request/reply semantics are deployed compatibility behaviour. Incompatible
changes require a new contract version or an explicit migration plan.
## State reload route
`SyncSubject` is `ebs.sync`. After a successful EC2 `ModifyVolume`, the daemon
uses request/reply to ask a `spinifex-workers` queue-group member to reload the
volume's state from its backend. The selected Viperblockd worker responds with
`EBSSyncResponse`; it returns an error when it does not own a live mount of the
volume. The daemon records that error as a warning after the API response has
already been returned. This is deployed legacy behaviour, not a guarantee that
every modification has reached its owning mount.
The route carries `EBSSyncRequest` and `EBSSyncResponse`. Their JSON shape and
the queue-group route are compatibility behaviour.
## Volume lifecycle routes
`MountSubject(node)` and `UnmountSubject(node)` retain the deployed mount and
unmount routes. With a node name they address that node's Viperblockd process
(`ebs.<node>.mount` or `ebs.<node>.unmount`); without one they retain the
single-node `spinifex-workers` queue-group routes. Both carry `EBSRequest`.
`EBSMountResponse` supplies the exported NBD URI, while
`EBSUnMountResponse` reports the seal result. The response broadcasts remain
`ebs.mount.response` and `ebs.unmount.response` alongside ordinary
request/reply replies.
`DeleteSubject` retains the `ebs.delete` queue-group route. It carries
`EBSDeleteRequest` and responds with `EBSDeleteResponse` after Viperblockd has
removed its local state and export. These routes and JSON shapes are deployed
legacy compatibility behaviour, not a generic EBS-provider interface.
## Fence route
`VolumeFencedSubject(node)` is `ebs.<node>.fenced`, or `ebs.fenced` with no
node for a single-node daemon. The Viperblock adapter publishes
`VolumeFencedEvent` on it after tearing down an export whose volume lease
moved to another node; the local daemon subscribes to stop the guest that
was using it. Node-addressed, like mount and unmount: the guest is local, and
no other node's daemon has anything to stop.
The event is sent as a request and repeated until the daemon replies with `VolumeFencedAck`, because the fence often runs while the node's NATS server is restarting and a single publish can reach no subscriber.
The daemon replies only when the event carries a reply subject, so a plain publish is still accepted.
Repeats are harmless, since fencing a guest that is already fenced changes nothing.
## Abandon route
`VolumeAbandonSubject(node)` is `ebs.<node>.abandon`. The daemon uses
request/reply to ask the Viperblock adapter on that node to tear an export
down without sealing, because the instance using it belongs to another node
now. The route carries `VolumeAbandonRequest` and responds with
`VolumeAbandonResponse`. `Abandoned` is false with no error when the volume
was not exported there, which is the ordinary case: the request is
idempotent by design, since the caller is reconciling rather than
transacting.
## Dirty-marker bucket
`DirtyBucket` is the JetStream KV bucket `VIPERBLOCK_VOLUME_DIRTY`, keyed by
volume name under `VolumeKeyPattern`, storing `DirtyRecord` JSON. It records
which node holds writes for a volume that the backend may not have.
Owner: `providers/ebs/viperblock` owns the dirty-volume state and its
transitions. `services/viperblockd` was only its transitional custodian.
Producers: only the Viperblock adapter writes, clears or changes a dirty
record.
Consumers: Viperblock recovery and mount logic on any node. `spx admin` may
obtain a read-only operational view.
Storage contract: the bucket name, `VolumeKeyPattern` and the `DirtyRecord`
JSON shape are this package's legacy v1 contract. There is no TTL; a record
persists until an explicit adapter-owned recovery or unseal transition
removes it.
Compatibility: this is a deployed legacy v1 contract, and coordinated
writers and readers support N/N-1. No renames, type changes or semantic
reinterpretation of existing fields; a breaking change needs a successor
contract and a migration.
CLI authority: the CLI is never a writer and must not perform arbitrary
bucket operations.
This package does not authorize requests, validate a volume ID, store volume
state, or define the generic storage-provider API. Those concerns remain with
the consuming daemon, the EC2 domain, and `providers/ebs` respectively.
@@ -19,7 +19,7 @@ Every service is `PartOf=spinifex.target` and ordered by its own `After=`/`Wants
| `spinifex-predastore.service` | `spx service predastore start` | S3-compatible object storage, serving that host's nodes and the S3 gate among them |
| `spinifex-northstar.service` | `spx service northstar start` | Authoritative DNS for `*.spx3.net`, recursive via upstream forwarders, syncing zones from its S3 bucket |
| `spinifex-viperblock.service` | `spx service viperblock start` | Block storage with the NBD plugin. `KillMode=process`: nbdkit serving a guest survives a restart of this unit |
| `spinifex-daemon.service` | `spx service spinifex start` | VM orchestration, NATS subscriptions, health endpoint. `KillMode=process`: guest QEMU survives a restart and the new daemon reattaches. `SIGHUP` reloads config |
| `spinifex-daemon.service` | `spx service spinifex start` | VM orchestration, NATS subscriptions, health endpoint. Two `ExecStartPre` steps run first: `spx admin hostnet check`, and `spx admin node-facts collect`, which runs as root and whose failure does not stop the daemon.`KillMode=process`: guest QEMU survives a restart and the new daemon reattaches. `SIGHUP` reloads config |
| `spinifex-awsgw.service` | `spx service awsgw start` | AWS-compatible gateway: SigV4 auth, IAM policy enforcement, TLS. Also serves the Private Admin API |
| `spinifex-vpcd.service` | `spx service vpcd start` | VPC daemon: translates `vpc.*` NATS events to OVN logical switches, ports and routers |
| `spinifex-ui.service` | `spx service spinifex-ui start` | Web UI serving the React frontend |
@@ -95,7 +95,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab; on a host whose IPsec stack is libreswan it defaults to false and `--ipsec=true` is refused), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-dns` (comma-separated IPv4 addresses, at most 3; the DNS servers written to the external pool's `dns_servers` and handed to guests, verbatim; when omitted they are detected from the host resolver at init time, so set it to give every node of a cluster the same value), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--advertise` (defaults to `--bind`, or the auto-detected WAN IP when `--bind` is 0.0.0.0), `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns`, `--email`, `--no-telemetry`, `--predastore-compaction-interval`, `--join-timeout` (how long to keep retrying while the formation server is unreachable, default 20m), `--force` (join even though this node is already initialized or holds JetStream streams, **discarding its own CA, master key and JetStream store**; the store is removed only after formation succeeds, and the join is refused while NATS is running even with `--force`) | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
### Version
@@ -129,13 +129,15 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
|---------|-------|-------------|
| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. On x86_64 it also checks that the QEMU binary the daemon will run lists the `microvm` machine type and that the microvm firmware `/usr/share/spinifex/microvm/qboot.rom` is installed, since load balancers boot on both. It runs the host network check `spinifex-daemon` and `spinifex-vpcd` run before they start, and fails while firewalld or ufw is active, since the Spinifex nftables policy is the only supported node firewall. It reports a Warning, without failing, when systemd-resolved is active but `/etc/resolv.conf` is not its stub, since lookups then bypass the route that sends Spinifex names to northstar. Exits non-zero if any asset is Missing, Stale, Ungranted, Unsupported or Failed. |
| `spx admin node js-probe` | `--timeout` (default: 10s) | Round-trips a canary key through the cluster-state KV bucket and exits non-zero if the write or the read-back fails. JetStream can latch a write error per message block and keep serving reads — nats-server's filestore records the failure on the block and returns it on every later write, and nothing clears it short of a restart. The HTTP health endpoint reports the server-wide JetStream state, which that latch never touches, so only an actual write observes it. |
| `spx admin node-facts collect` | — | Runs `dmidecode`, `ethtool`, `udevadm`, `networkctl lldp` and `systemd-detect-virt`, and writes their raw output to `/etc/spinifex/node-facts/`, readable by the spinifex group. `spinifex-daemon.service` runs it as a root `ExecStartPre` because the daemon itself is unprivileged; the daemon parses the files when the operator console asks. A probe that fails writes no new file, so a file from an earlier successful run can remain and the console reads it as available; a source with no file is shown as unavailable. |
| `spx admin banner` | `--boot-check` | Writes the node information banner to `/etc/issue` (shown before login on the physical/serial console) and `/etc/motd` (shown after SSH login). With `--boot-check` it also detects a management IP that has changed since last boot and updates `/etc/spinifex/node.conf` accordingly. |
### Storage Inspection
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. |
| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. It also lists UEFI variable stores nothing owns: a `uefi/i-*/` prefix with no instance record and no store record, a `uefi/ami-*/` prefix with no AMI document, and a store record that is not `deleting` and has no instance record. These are reported and never deleted, because an absent instance record is not proof of a terminate. |
| `spx admin volumes obligations` | — | Lists the open unpublish obligations, oldest first: a volume export that a force detach, or a terminate whose unmount did not complete, left on the node that held the attachment. Each row names the volume, the node, the instance and the age, plus attempts and the last error once the node has tried. The node settles each one once it answers; an obligation stays open while its node is down and closes without the node only once the node has been removed from the cluster. **Reports only.** |
| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is refused while that node is alive (its daemon heartbeat is under 60 seconds old, or cannot be read) and is otherwise allowed, in which case it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
| `spx admin storage status` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Sends the signed `GetStorageStatus` action to the AWS gateway with credentials from the standard AWS chain (e.g. `AWS_PROFILE=spinifex`) and prints what the console's Service Metrics page shows: predastore encoding, meta and blob node counts, cluster health, a meta node table (health, raft state, leader, term, commit and applied index) and the blob nodes. Cluster health is `Degraded` when any meta node is unreachable or the nodes reporting raft state do not agree on one leader. A meta node read only through its host's `/readyz` shows healthy with blank raft fields. |
@@ -152,12 +154,13 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| `spx admin account create` | `--name` (required) | Connects to NATS → refuses if an account already holds the name (case- and whitespace-insensitive) → reserves the name in `spinifex-account-names` KV → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates default VPC → creates `admin` user, AdministratorAccess policy (Action:*, Resource:*) and one access key → prints credentials and writes a `spinifex-<name>` AWS CLI profile |
| `spx admin account create --remote` | `--name` (required), `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA), `--client-token` (default: generated), `--source` (default: `spx-cli`) | Same account, created through `POST /admin/CreateAccount` over SigV4 instead of NATS — the path the public signup form uses, so a shell can exercise it. Credentials come from the standard AWS chain (env vars or `AWS_PROFILE`), never from the cluster master key; the caller must be an IAM user in the super-admin account holding `spinifex:CreateAccount`. Reuse the printed `--client-token` to retry: a fresh token would create a second account. |
| `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |
| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS credential chain: on a cluster node that means `AWS_PROFILE=spinifex` (there is no default profile, so without it the command fails to find credentials), or equivalent credentials; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
| `spx admin account quota get <account-id>` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Reads the limits in force for one account through `POST /admin/GetAccountQuota` over SigV4, and reports for each dimension whether it came from the `[quota]` block in `awsgw.toml` (`config`) or from an override stored against the account (`override`). Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:GetAccountQuota`. |
| `spx admin account quota set <account-id>` | `--vcpus`, `--vpcs`, `--subnets`, `--eips`, `--volumes`, `--volumes-gib`, `--rds-instances`, `--load-balancers`, `--clear`, plus `--endpoint`, `--region`, `--ca-bundle` | Overrides individual limits for one account through `POST /admin/PutAccountQuota`, leaving every dimension not named inheriting the configured baseline. `-1` uncaps a dimension; `0` is a real limit that denies every request on it. `--clear` removes the whole override record and cannot be combined with a limit flag. The account must already exist. Prints the resulting limits, so a caller never has to guess what it applied. The caller must hold `spinifex:PutAccountQuota`. |
| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record) | Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--non-finalizing` (hidden), `--yes` (requires `--name`), `--name` (account name, must match the stored record), `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Always runs through `POST /admin/DeleteAccount` over SigV4 and follows `DescribeAccountDeletion` to completion; the CLI never tears an account down in its own process, so a CLI delete and an API delete cannot race. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → seals the account's quota override record so no later write can recreate it, which needs every gateway node the cluster configures to answer a read-only probe first (otherwise nothing is sealed and the account stays `TERMINATING`) → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
| `spx admin account delete --force` | as above | Escalates only for resources the ordinary API refuses: it clears a volume attachment in the control plane without the guest's cooperation (`ec2.ForceDetachVolume`), hard-destroys after a graceful attempt times out, and treats an already-missing resource as deleted. It never reorders the stages and never widens scope beyond the account. Use it once the guest is being destroyed — this is the escape from a volume stuck attached to an instance that will not terminate, where neither can otherwise be deleted. |
| `spx admin account delete --remote` | as above, plus `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Same teardown driven through `POST /admin/DeleteAccount` over SigV4 instead of NATS, then follows `DescribeAccountDeletion` to completion, printing each stage as it lands. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. |
| `spx admin account delete --non-finalizing` | as above | An exercise run, not a finalising delete, but the cleanup is real. Marks the account `TERMINATING`, deletes the resources of each EC2 platform kind the evaluation found quiesced (for example key pairs, empty placement groups and orphaned tags), re-reads the owner inventory against that cleanup, and ends `EXERCISED` with the account still `TERMINATING`. No other stage runs and the account record, name reservation and quota are left in place. It refuses `--force` and `--dry-run`. |
| `spx admin account delete --remote` | as above | Accepted for existing scripts and ignored: every delete already goes through the admin API. |
| `spx admin principal create <name>` | `--grant` (repeatable or comma-separated; default: every admin method — `CreateAccount`, `DeleteAccount`, `DescribeAccountDeletion`, `GetAccountQuota`, `ListAccounts`, `PutAccountQuota`) | Creates an IAM user in the super-admin account with an inline `spx-admin-methods` policy allowing the requested methods, revokes any access key left from a previous run, and mints one new key. Each method is named rather than granted as `spinifex:*`, so a later addition to the admin surface is not authorised by an existing key. The principal's grants are exactly that one policy: re-running with fewer `--grant` values narrows it, and any other inline policy on the user is removed, so a rename cannot leave a wider grant behind. The secret is printed once; store it in a secret manager or an AWS profile, never in a config file. `admin` is refused as a name — it is the account's own administrator and rotating its key would revoke the bootstrap credential. |
| `spx admin principal list` | — | Lists every principal in the super-admin account with its access-key count and what it may call. An attached policy is reported by name (`AdministratorAccess (attached)`) rather than expanded, which is how an unscoped principal is spotted. |
| `spx admin principal revoke <name>` | — | Deletes every access key the named principal holds, leaving the user and its policy in place. Immediate and cluster-wide; re-issue with `principal create`. |
@@ -171,8 +174,8 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify`, `--checksum`, `--ami-name` (override the registered `DescribeImages` name, which otherwise defaults to `ami-{distro}-{version}-{arch}`; used for locally-built appliances such as `--ami-name spinifex-eks-node`), `--tmp-dir` (default: `<spinifex-dir>/images`) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports require an explicit `--boot-mode` because there is no catalog metadata to inherit from, and are verified only when `--checksum <sums-file>` is passed. `--checksum` takes the sums file the image was published with (`SHA256SUMS`, `SHA512SUMS`, `<image>.sha256`/`.sha512`, coreutils, BSD or bare-digest format), looks up the entry for the image's basename, and infers sha256 or sha512 from the digest length. A mismatch exits 1 before anything is written to storage and leaves the file in place. `--checksum` proves only that the image matches the sums file supplied with it, not that either is authentic; verify the sums file's vendor signature separately. `--checksum` is refused without `--file` or with `--skip-verify`. A `--file` import without `--checksum` prints `Importing <file> without checksum validation` to stderr and proceeds; `--name --file` does not fall back to the catalog digest. Every import records the source artifact's digest on the AMI with who verified it (`operator`, `catalog` or `unverified`, the unverified cases hashed as sha256), prints it on success, and shows it in `spx admin images describe`. The digest is of the file as supplied, not of the stored volume. `--skip-verify` bypasses verification for catalog imports and prints a warning to stderr; use only for debugging or when upstream mirrors are confirmed-broken. Import never modifies the image: the bytes written to storage are the bytes that were verified. Compressed and qcow2 images are extracted into a hidden `.import-*` scratch directory under `--tmp-dir`, which is removed when the import succeeds or fails, so the catalog directory keeps only the downloaded source (for `--name`) and the AMI manifest. Point `--tmp-dir` at a disk-backed filesystem with room for the full raw image, never at a tmpfs such as `/tmp`. A guest that needs to trust the deployment CA for in-guest SDK-over-TLS calls fetches it at runtime from `http://169.254.169.254/spinifex/ca.pem` (see the IMDS section), so a CA rotation reaches guests without re-importing anything. |
| `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
| `spx admin images describe` | `--image-id` (required) | Read-only. Prints the AMI's name, owner, creation date, boot mode and architecture, then its source digest: algorithm and value, verification (`operator` with the sums file name, `catalog` with the sums URL, or `unverified`), and the source file name. AMIs imported before digest recording print `Source digest: not recorded`. The digest is not exposed through `DescribeImages`. |
| `spx admin images promote` | `--image-id` (required), `--yes` | Reads `ami-<id>/config.json`, validates the AMI is account-owned, then rewrites `ImageOwnerAlias` to `"system"`in-place. No block data is copied. The change takes effect immediately — the AMI becomes visible to all accounts via `DescribeImages`. Prompts for confirmation (skipped with `--yes`). Already-system AMIs are refused. |
| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
| `spx admin images promote` | `--image-id` (required), `--yes` | Reads the AMI's metadata document, validates the AMI is account-owned, then rewrites its `ImageOwnerAlias` to `"system"`and re-keys its snapshot document under the global account. No block data is copied. The change takes effect immediately — the AMI becomes visible to all accounts via `DescribeImages`. Prompts for confirmation (skipped with `--yes`). Already-system AMIs are refused. |
| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Connects to NATS and predastore. Loads the AMI's metadata document, asks the EBS provider for the capacity of the image's volume (`absent` when the provider answers not found, `unknown` with the provider's error on any other failure, which never aborts the preview), and walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap. It then prompts (skipped with `--yes`) before deleting the image's UEFI variable store, then its metadata document (the DescribeImages barrier), then asking the provider to delete snapshot `snap-ami-<id>` and then volume `ami-<id>`. Both provider deletes succeed on an absent target, so a rerun finishes a half-done removal; the provider refuses to delete a volume that is still published. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and corrupt-document checks for salvage of an orphaned volume and snapshot. |
### GPU Management
@@ -194,18 +197,32 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
### SQS System Instances
These commands start the first SQS system instances on one node. They do not serve the SQS API, and a bound VM is not a ready service.
These commands enable SQS on a single-node cluster and start its one system VM.
A bound VM is not a ready service: `spx admin sqs status` reports readiness.
Both send a signed `spinifex` admin action (`EnsureSQSWorker`, `EnsureSQSStore`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VMs, volumes, private VPC and ENIs belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. Each call waits up to 11 minutes.
The SQS VM runs two processes on one encrypted data volume: the Region's dedicated SQS NATS server, which listens only on the guest's loopback address and is separate from platform NATS, and the worker, which serves SQS owner calls from that local store. The worker starts serving only after the local store answers and the receipt-signing key on the volume matches the fingerprint the platform recorded.
SQS supports only a fresh single-node install. A worker record without a supported launch contract, or any record of the separate SQS store VM from an earlier dev build, cannot be upgraded: `enable` refuses it, and any existing volume is kept, not inspected, emptied or replaced. An existing separate store VM is refused on relaunch with its volume kept, and an existing worker VM is not restarted into a supported mode; upgrading such an install is unsupported.
**Do not put tenant production data in SQS.** SQS-managed per-message encryption is not implemented: `GetQueueAttributes` reports `SqsManagedSseEnabled=false`, and a request to enable SSE-SQS or SSE-KMS is rejected.
The SQS data volume must be encrypted, which is a separate layer under every queue and not SSE.
There is no `DeleteQueue` or `PurgeQueue`, so a queue and its unexpired messages cannot be removed once created.
Retention is checked when a message is received, against the queue's retention at that moment, and nothing deletes an expired message: the store drops a body only at its age backstop of 14 days and 1 hour.
So a message past a short retention can be delivered again if `MessageRetentionPeriod` is later raised while its body is still stored.
Account termination does not yet remove SQS queues or messages, and there is no disable: the VM's volume, receipt key and data are kept.
Each sends a signed `spinifex` admin action (`EnableSQS`, `DescribeSQS`, `EnsureSQSWorker`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VM, volume, private VPC and ENI belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. `ensure-worker` and `enable` wait up to 11 minutes, and `status` up to 1 minute.
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS worker VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The worker is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted 1 GiB system-account data volume on which the guest generates its own store key. The platform records only the public key. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. |
| `spx admin sqs ensure-store` | `--node` (required, the worker's node), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's SQS store VM on the worker's node with an encrypted system-account data volume, or reports the VM already bound there: node, region, instance, instance type and address. Needs the worker bound with its public key recorded, else `IncorrectState`. The store accepts only that key. A different size, type or key than the first call is `IncorrectState`. |
| `spx admin sqs enable` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Records SQS as enabled for the Region, pinned to `--node`, then launches or reports the SQS VM there. The desired state is recorded first, so rerunning the command resumes an interrupted enable. More than one node in the cluster is refused before anything is recorded. A worker record with a missing or unsupported launch contract, or a record of the separate store VM, is also refused before anything is recorded or launched; any existing volume is kept, and upgrading it is not supported. |
| `spx admin sqs status` | `--endpoint`, `--region`, `--ca-bundle` | Shows the requested state, node, generation, VM state, receipt key state, whether a worker is polling this gateway, whether this gateway serves the SQS API, and `Ready`, which is true only when all of them hold. `Unsupported` names a record whose missing or unsupported launch contract blocks enable. |
| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The VM is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted system-account data volume of `--volume-size-gib` that holds the queue data and the receipt-signing key the guest creates on first boot. The platform records only the key's fingerprint. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. A different `--volume-size-gib` than the recorded one is `IncorrectState`. |
`ServiceUnavailable` never proves that no SQS resources exist, because an earlier attempt may already have created some. Its message says what it does establish. When no node serves the subject, this request was not delivered. When a volume create was not sent, that particular create did not happen. When the node cannot boot the guest yet or its launcher lacks a capability, this request stopped before allocating or recording anything. When the gateway times out waiting for the node's reply, nothing is known: the launch may still be running, so never assume it created nothing. In every case, call the command again with the same arguments. A retry first checks the durable record of the attempt and continues only when that is safe; it can proceed to the first launch after a create that was proven unsent, and otherwise reports the recorded outcome as bound or `IncorrectState`, or returns `ServiceUnavailable` again. `IncorrectState` from a volume, network interface or launch step means that step was recorded as sent but its outcome was never observed. It is never resent, the message names the volume when its ID is known, and no tool reconciles it yet: recovery is manual.
`bound` says only that the VM was launched for the current record. The worker decides store readiness itself, by checking that the store is bound to its own public key and that a NATS connection with that key succeeds; nothing reports that readiness to the operator. The worker-to-store connection does not authenticate the store server. Real VPC routing, IMDS and gateway TLS from the guest, and worker-to-store traffic on a real interface have not been tested.
`bound` says only that the VM was launched for the current record. The worker is ready only when the guest's local store answers, and it serves only while its receipt key matches the recorded fingerprint; a missing or different key once one is recorded refuses serving and is never replaced.
Real VPC routing, IMDS and gateway TLS from the guest have not been tested on a live host.
@@ -65,9 +65,10 @@ The EC2 registration (`spinifex/domains/ec2/awsapi`, `NewRegistration`) parses t
The gateway communicates with daemons via NATS request/response. Most calls go through `natsmsg.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
@@ -110,15 +111,17 @@ Daemons (`spinifex/runtime/daemon/daemon.go`) subscribe to NATS topics and handl
When a daemon handles `RunInstances`:
1. **Resource Check**: Validates CPU/memory availability via `ResourceManager`
2. **Volume Generation**: Creates boot and EFI volumes via Viperblock
3. **Volume Mount**: Sends a mount request to the colocated Viperblock worker on `ebs.{node}.mount`, receives NBD URI
2. **Volume Generation**: Creates the boot volume through the EBS provider and, for a UEFI guest, seeds its variable store as an object under `uefi/<instance-id>/` with a store record in `spinifex-uefi-stores` that points at it
At launch a UEFI instance's `vars.fd` is fetched from the object its store pointer names (or reused when `seeded.json` on this node already names the same object) after a firmware architecture and VARS size check, and a UEFI instance with no live store record naming a pointer, or one that still carries an EFI volume request, is refused.
The store is written back to the object store on write and at stop, and a terminated instance's store is removed by a cluster-wide reaper.
3. **Volume Mount**: Publishes each volume through the EBS provider contract (`PublishVolume` on `ebs.provider.v1.{node}.mount`), receives its export
4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks
5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management
6. **Command subscription**: Subscribes to `ec2.cmd.{instanceID}` for subsequent commands on this VM
```go
// Volume mounting via NATS (runtime/daemon/vm_adapters.go)
// QEMU launch with NBD-backed drives in args (runtime/compute/vm/vm.go)
cmd := exec.Command(qemuArchitecture, args...) // qemu-system-x86_64 or qemu-system-aarch64
@@ -224,6 +227,9 @@ It may keep writing for 40 seconds after the last write the store acknowledged,
A slow acknowledgement therefore shortens the remaining validity rather than extending it, and an acquisition acknowledged after its validity has already run out is refused as retryable.
For a holder that cannot renew, a timer is armed for the validity deadline, leaving a nominal 5 seconds before the entry can expire; when it fires, the export's nbdkit is fenced with SIGKILL (`volume_fence.go`).
The kill comes first: the read that names the winner (bounded at 2 seconds) and the notice to the daemon follow it, so a hung lease store cannot hold the writer alive.
The notice is a request that viperblockd repeats every 2 seconds until the local daemon replies with `VolumeFencedAck`, because the fence often runs while the node's NATS server is restarting.
On the notice the daemon moves the guest using the volume to `error` with state reason `Server.VolumeFenced` and SIGKILLs its QEMU immediately rather than shutting it down gracefully, because there is no disk left to flush to; it only stops the guest process and removes its taps, and never terminates the instance or deletes a volume.
The fence runs on its own goroutine, so Go scheduling, that goroutine and process termination all delay the kill, and enough delay can use up the whole margin; a kill held in uninterruptible sleep, which the fence waits up to 30 seconds for, is the extreme case.
Once a lease is lost or has lapsed it stays lost: a renewal that succeeds afterwards is ignored, and another open on the same node cannot share it while the fence runs.
@@ -278,8 +284,8 @@ storage-provider contract.
### SQS message store (internal)
`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
It is an internal primitive: nothing outside the SQS domain uses it, no daemon or service is wired to it, and it does not itself deliver an SQS API.
The dedicated JetStream server ADR-0011 describes is not deployed by this code, and Spinifex ships no host service, unit, user or firewall rule for it; the guest that could run it is described under *SQS store system instance (not wired)*.
It is used only by the SQS worker inside the SQS system instance and does not itself deliver an SQS API.
Spinifex ships no host service, unit, user or firewall rule for the dedicated JetStream server; the system instance that runs it is described under *SQS system instance (operator-launched)*.
The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
@@ -298,124 +304,36 @@ If that consumer delivers nothing for about two seconds while the stream holds a
Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
### SQS store system instance (not wired)
### SQS system instance (operator-launched)
Spinifex can build, install, launch, restart and give storage to a dedicated SQS store microVM, but nothing calls any of it.
No SQS API, domain service, admin command or daemon start-up path creates a store, so none of this is in production use.
One system-owned microVM per Region runs both the SQS store and the SQS worker as separate processes, on one encrypted data volume pinned to one node (ADR-0011 Amendment 5, ADR-0012 Amendment 6).
Only an operator activates it with the super-admin `EnsureSQSWorker` or `EnableSQS` actions; nothing starts it at daemon startup, and platform NATS is neither used nor changed for SQS data.
This first profile is fresh single-node only: a host cluster of more than one node is refused before anything is recorded or allocated, and there is no disable.
The guest image is built by `make build-sqs-store-microvm-image`, separate from the load balancer image, and bundles the `nats-server` version pinned in `go.mod` plus the ext4 tools.
Release tarballs carry it as `microvm-sqs-store/`, and `setup.sh` installs it to `/usr/share/spinifex/microvm/sqs-store/`.
A release without a complete image removes any previously installed one, so a launch is then refused rather than booting a stale image.
`make build-sqs-worker-microvm-image` builds the one `sqs-worker` image: the static `sqs-worker` binary, the pinned `nats-server`, the ext4 tools and a network `/init`; releases ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker`.
At boot the guest reads three firmware config values under `opt/spinifex/sqs-worker/` (the data disk's serial, its Region and the gateway URL on this node's mgmt bridge) and refuses before touching any disk if one is missing or malformed.
It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and confirms from `/proc/self/mountinfo` that the whole disk is what is mounted; the guest never formats.
At boot the guest reads three firmware config values: the data disk's serial, an IPv4 address on one of its interfaces, and one NATS user NKey public key.
It mounts exactly one matching disk at `/data` after `e2fsck -p`, refuses a missing, duplicate, unformatted or unrepairable disk, and never formats.
JetStream keeps its data on `/data` with `sync_interval: always` (ADR-0011:S3), `max_payload: 2MiB` admits a full 1 MiB SQS message with its headers (ADR-0011:S4), and the server listens only on that address and accepts only that key.
On shutdown the store is stopped before `/data` is unmounted; a store that survives SIGKILL leaves every mount in place.
The store is a dedicated `nats-server` listening only on `127.0.0.1`, with its JetStream directory at `/data/jetstream` and `GOMEMLIMIT=512MiB` (ADR-0011:S11), so nothing outside the VM can reach it and it needs no network identity of its own.
The store starts only after the receipt key is settled, and the worker connects only after the store answers JetStream account info, so an unavailable store never appears ready.
A launch uses the versioned subject `system.LaunchSQSStore.v1`, which a daemon without this code does not serve, so it fails with no responder instead of booting another guest.
The legacy `system.LaunchInstance` path refuses the store kind on both send and receive.
Every input check runs before capacity is reserved, and the data volume must be an available system-account volume.
It is mounted through the normal EBS path and detached, never deleted, on terminate, which is the protection standard EBS gives today and nothing stronger.
CPU mitigations stay enabled for the store, unlike the load balancer.
A successful launch means the VM is running, not that NATS is ready.
The receipt-signing key lives on the data disk under `/data/sqs-worker-key`, written to a temporary file, synced, renamed and its directory synced before its fingerprint is recorded.
`domains/sqs/workerident` holds the platform's receipt-key state on the worker's identity record, beside its generation, claim, pinned node and volume: `pending` is set only when the record first creates its volume, and `recorded` holds the fingerprint.
Under `pending` the guest may create a key, or reuse a valid key it already wrote before a crash, only while no `/data/jetstream` directory exists; any store data under `pending` refuses, because it proves the volume is not new.
Under `recorded` a missing, unreadable or different key refuses the boot and is never regenerated; a record with no receipt-key state is refused as not proven new.
`RecordReceiptKey` writes the fingerprint by compare-and-swap that re-checks the current instance, generation, claim and volume, and a lostreply is answered by reading the record again, never by a new key.
A replacement worker reuses the pinned volume and keeps its predecessor's receipt-key state.
The instance record keeps the store's non-secret guest settings so a restart on the node that launched it rewrites the firmware config files.
A relaunch on any other node is refused, and cross-node recovery never claims a store.
Both checks exist only in daemons with this code: an older daemon can still claim and relaunch a store elsewhere, and an explicit start served by another node claims the instance before its relaunch is refused.
The worker reaches the gateway over TLS 1.3 trusting only the cluster CA, signing with instance-role credentials from the metadata service only.
The internal `spinifex-sqs` actions are `ClaimReceiptKey`, `RecordReceiptKey`, `PollWork` and `CompleteWork`; the gateway admits only an assumed-role session of `sqsWorkerRole` in the system account and resolves the instance through `workerident.Authorize`, a point-in-time check against the current bound record.
The worker pulls tenant work from its own node's gateway relay and posts results back, so no SQS payload crosses platform NATS; therelay holds a bounded, in-memory set of waiting requests and answers `ServiceUnavailable` when it is full or no worker is polling.
`domains/sqs/storevolume` prepares a fresh data volume, because the guest never formats.
It checks for `mkfs.ext4` and `qemu-img` before allocating, creates the volume in `creating`, writes an empty ext4 image to it over NBD on the local node, unpublishes it, and only then makes it `available`.
While a volume is `creating`, attach, delete, modify, force-detach and snapshot are refused.
A volume is deleted only after a confirmed unpublish; any publish error, an unconfirmed unpublish or a failed release keeps it and reports its ID, with its state reported as unknown after a failed release.
`EnsureSQSWorker` takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB`; the instance type's memory must exceed the store's 512 MiB limit, and a later request for a different type or size is refused as `IncorrectState`.
`EnableSQS` records the Region's desired state, pinned to one node, and then ensures the worker; `status` reports it ready only when the worker is bound, polling and the tenant API is registered.
A store launch record or VM from the earlier separate-store design, or a worker record without a supported launch contract, is refused with its volume kept: enable stops before recording anything, and a relaunch of an old store VM is marked failed rather than started.
Evidence is local only: unit tests, real QEMU boots against disposable scratch disks, and one run of the preparer against the `qemunbdd` provider whose volume kept acknowledged writes across a reboot.
Viperblock, volume encryption, the EC2 volume record path, multiple nodes and host reachability of the guest address are not tested.
Credential issuance and rotation, store placement and NATS readiness checks do not exist, and there is no failover: a lost node leaves its store unavailable.
### SQS worker guest (operator-launched)
The shared SQS worker guest holds the SQS store's client credential, so that no platform component does (ADR-0011:S2).
Only an operator activates it, with the super-admin `EnsureSQSWorker` action described below; nothing starts it at daemon startup, and no worker service runs after boot.
`make build-sqs-worker-microvm-image` builds it with the store image's script and network `/init` under `SQS_MICROVM_PROFILE=worker`; it carries the static `sqs-worker` binary and the ext4 tools, no `nats-server`, and an empty `/data` mountpoint.
Releases build it beside the store image and ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker` with the store image's rules: a release without it removes an earlier one, and an incomplete one is refused.
At boot `sqs-worker boot` reads three firmware config values under `opt/spinifex/sqs-worker/`: the worker disk's serial, its Region and the gateway URL; none is a first-store decision, because a static value survives a guest reset.
A missing, empty or malformed value refuses before any disk is touched.
It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and then confirms from `/proc/self/mountinfo` that this disk, as a whole ext4 filesystem, is what is mounted there; an empty mountpoint, a foreign mount or a failed check or mount refuses, and the guest never formats.
Only then does it ask the gateway for the current decision with `ClaimFirstStore`, signed by `internal/sqsworkergw` with credentials from the instance metadata service only (`gwsign.NewInstanceRole`, no environment, profile or static-key fallback), over TLS 1.3 that trusts only the cluster CA from `opt/spinifex/ca-cert` and never follows a redirect.
Under that decision `workerkey.Acquire` creates the key on `/data/sqs-worker-key`, owner-only and synced, records only its public key and then uses it, or loads the recorded key; a missing or different key once one is recorded refuses and is never replaced.
The VM can start before its launch binds the instance, so its first calls are legitimately `AccessDenied`; that denial creates nothing and is asked again, as are a lost reply and a `ServiceUnavailable` or `Throttling` refusal, all within one bound (60 attempts 5 seconds apart), after which a worker that was never bound, or was replaced, refuses the boot.
A signature or other refusal, or an answer that is neither "create" nor one valid public key, refuses the boot at once.
`domains/sqs/firststore` is the SQS domain's first-store decision: one record per Region in the `spinifex-sqs-first-store` KV bucket moves from `claimed` to `key-recorded` to `launch-intent` under compare-and-swap, bound to the worker's instance ID, claim ID and generation.
A claim authorizes key creation only while no public key is recorded; once one is, every claim returns only that key, and a store launch intent needs it recorded first.
An older generation, a different identity at the same generation or an unreadable record authorizes nothing, and the record holds no private key material.
Reads are stream message gets answered by the stream leader, never direct gets a lagging replica can answer, and every authorizing answer is also confirmed by a compare-and-swap at the revision it read, so a stale or unavailable read authorizes nothing.
The authenticated caller that will carry this decision must derive the worker's instance ID and generation from authoritative instance state; a generation supplied in a request cannot itself authorize a takeover.
`domains/sqs/workerident` is that authoritative instance state: one record per Region in the `spinifex-sqs-worker-identity` KV bucket holds a monotonic generation, claim ID, pinned node and encrypted data volume, and moves from `launching` to `bound` to `stopped`, or from `launching` to `failed`, under compare-and-swap.
A launch intent is refused while the previous attempt is `launching` or `bound`, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume.
A replacement takes the next generation only after `FailLaunch` or `RecordStopped` records that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never observed; there is no automatic takeover.
The launch path then raises the first-store fence to the successor's generation with `firststore.Fence` before `BeginLaunch`, which refuses a replacement until a leader read shows that fence; the fence only rises, and `firststore` refuses every call from a generation below it.
Only the current claim ID is recorded, so claim IDs are minted fresh per attempt and a VM launch must be idempotent on generation and claim together.
Binding writes the instance's index entry first and then binds the record to that instance.
The index is only a lookup: `Authorize` accepts an instance only while the Region's record is `bound` to exactly that instance, generation and claim, confirmed by a compare-and-swap at the leader-read revision, so an old, orphaned, rewritten or unbound entry authorizes nothing.
A same-instance reboot keeps its instance ID and identity; a replacement takes a new generation, and the first cut refuses another node or data volume and an unencrypted volume, although the encryption flag is the caller's claim and the launch wiring must verify it from the volume's metadata.
Both buckets are opened through `OpenDurableKV`, which refuses an existing bucket whose effective stream configuration could expire, evict, lose, lag or refuse records: non-limits retention, an age, per-message or delete-marker TTL, memory storage, asynchronous flush, a size, count or message-size cap, a discard policy other than discard-new, unacknowledged writes, a sealed stream, a mirror or sources, or fewer replicas than the cluster needs.
`workerkey.Acquire` drives that decision: it syncs a new key to disk, records its public key and only then returns it, so a crash before recording reuses the key already on disk.
`domains/sqs/awsapi/worker` carries the decision to the worker as `ClaimFirstStore` and `RecordWorkerPublicKey`, served under the internal `spinifex-sqs` signing service, which no AWS SDK signs for; the public AmazonSQS action inventory is untouched.
The gateway admits only an assumed-role session of `sqsWorkerRole` in the system account, then the role's policy, and only then resolves the session name, the instance ID STS set, through `workerident.Authorize`; the request carries no identity, and any parameter beyond the action's own is refused.
The first-store identity is the current record's instance, claim and generation, and a worker bound to another Region is refused.
A request that passed `Authorize` just before a confirmed stop and replacement still carries the old generation; the fence was raised before the successor began, so `firststore` refuses it whether or not the successor has claimed yet.
The adapter never logs a caller-chosen parameter name, because one could carry a secret.
`RecordWorkerPublicKey` accepts only a valid public user key, so a seed is refused without being recorded, logged or echoed.
The gateway binds both buckets per request and never creates them, so it provisions nothing and answers `ServiceUnavailable` until a worker launch has created them; the worker launch below ensures the role.
The private key stays on the worker disk; the guest logs and publishes only the public key to `/run/sqs-worker/public-key`.
`domains/sqs/workerlaunch` realizes the worker on the node it runs on, its only node: a bound record is returned as is; otherwise it takes the intent, then on the first launch marks the intent's one volume create, prepares the volume and records its ID, reads the volume's own record and refuses it unless encrypted, ensures the role, finds or creates the network for the claim and reads the recorded store key, and marks the intent dispatched before launching.
A dispatched launch is never sent again: a lost reply or error leaves the intent `launching` and dispatched, and every later call returns `ErrOutcomeUnknown` until the outcome is reconciled, since the launch path is not idempotent.
A failure before dispatch leaves an undispatched intent that the next call resumes with the same claim; a confirmed stop or failure is replaced only after `firststore.Fence` excludes the old generation, reusing the pinned volume.
The launch passes no store decision or key: the guest asks the gateway at boot.
The intent exists before any volume, so concurrent first launches allocate nothing for the losers. A Prepare proven to have sent no create withdraws the mark and returns `ServiceUnavailable`, or `UnsupportedOperation` when the cluster storage key is not configured, and a retry may prepare the volume; a create that may have applied is never sent again, and every later call returns `IncorrectState` until it is reconciled. A volume the call knows of but cannot use is reported with its ID in a `VolumeError` and left in place, never deleted. An intent that has sent no create may be re-typed; one that has keeps its type until replaced. The worker role is created with EC2-only trust, and an existing role is checked for that trust before any policy is granted; the instance profile must carry exactly the worker role, and one holding any other role is refused unchanged.
The daemon's `LaunchSQSWorkerInstance` boots the `sqs-worker` image through the same direct-boot path as the store, with the data volume attached at boot and kept on terminate, a required instance profile, and a required mgmt NIC; it re-reads the volume's own record and refuses one that is not available and encrypted, before anything is allocated.
Its firmware config adds the gateway URL on this node's mgmt bridge and the cluster CA from the daemon's own trust store; the caller cannot supply either, and no value written is secret.
A relaunch rebuilds that config from the VM record on the node that launched it and is refused anywhere else, and membership takeover already skips SQS-managed instances.
`sqsWorkerRole` is ensured in the system account granting exactly `spinifex-sqs:ClaimFirstStore` and `spinifex-sqs:RecordWorkerPublicKey`, from the `domains/sqs/workerrole` contract the gateway gate also reads; an existing role whose trust admits more than the EC2 service is refused.
Evidence is local only: unit tests with a fake firmware config, sysfs and mount table, tests running the boot through the real client against the real `spinifex-sqs` registration and records, and QEMU boots of the built image against disposable scratch disks and a host-side test gateway and metadata service, showing the first boot creates the key, reboots reuse it, a key lost after recording is never replaced, and the seed reaches no log or request.
The worker's network is a private SQS system VPC per Region, built by `systemvpc.EnsurePrivate` from the `[sqs]` supernet (default `10.240.0.0/14`): one VPC and one /24 subnet, with no internet gateway, NAT gateway, EIP, route table or daemon-to-store node port. Each launch claim gets one ENI in that subnet, in the VPC default security group, found again by its claim on a retry; two ENIs for one claim are refused. The launch intent is marked before the one create is sent, so a create whose ENI is not yet visible is never sent again: the launch fails closed with `IncorrectState` until the ENI appears or the outcome is reconciled. Worker-to-store TCP and the guest's route to the metadata service are not yet proven by any traffic test.
The super-admin `EnsureSQSWorker` action on the internal `spinifex` operator service takes a `NodeId`, which must be a node of this cluster and a single subject token, and an `InstanceType`, and sends a request to `sqs.worker.Ensure.v1.<node>` with no queue group, so only that node answers and the worker stays pinned.
That node's daemon re-checks the node the request names and the instance type, and refuses as `ServiceUnavailable` before allocating a volume or network or recording an intent unless the worker image's kernel and initramfs are non-empty files, its mgmt bridge exists and the cluster CA is readable; a call after the image is installed then launches.
It then runs its one composed launcher for its own Region, one launch at a time, in the daemon's own context with a 30-minute bound, so a caller's timeout does not cancel it.
A reply lost to that timeout is reported as `ServiceUnavailable`; calling again reports the bound worker, or `IncorrectState` for a dispatched launch whose outcome is unknown, which is never resent; a Region pinned to another node is refused as `InvalidParameterValue`.
The reply names the bound VM's node, Region, instance ID and generation; bound is not ready, and it says nothing about the guest's boot, its key or the store.
Stop/detach evidence for `RecordStopped` is not wired, the guest's metadata route and gateway TLS on a real ENI and worker-to-store TCP are unproven, and Viperblock's at-rest key is held by the platform.
The worker identity record holds the instance type of its generation; a request for another type is refused as `IncorrectState`, before dispatch and after binding, the reply names the recorded type, and only a replacement after a confirmed stop may change it.
Operators call both actions with `spx admin sqs ensure-worker --node --instance-type` and `spx admin sqs ensure-store --node --instance-type --volume-size-gib`, signed for the `spinifex` service with the standard AWS credential chain like the other `spx admin` gateway commands.
Each command refuses missing inputs before sending anything and reports only the bound VM; a bound VM is not a ready worker or store.
The super-admin `EnsureSQSStore` action takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB` and sends a request to `sqs.store.Ensure.v1.<node>` with no queue group; only the node the Region's worker is bound to can launch the store.
That node refuses an instance type whose memory does not exceed 512 MiB, the store's `GOMEMLIMIT` under ADR-0011:S11; this is a floor, not a validated size, and no type is defaulted.
It refuses before recording or allocating anything unless the store image and the cluster storage key are present, the worker is bound on this node, and that worker's identity has recorded its public key in the first-store decision, which it then advances to a launch intent.
The store's own launch record in `spinifex-sqs-store-launch` holds only that public key, the claim, the node and the operator's instance type and size, and marks each step before its uncertain create: the volume, the ENI in the worker's private subnet, and the launch.
A later request for a different instance type or size is refused as `IncorrectState`, before dispatch and after binding, and the reply names the type the store was launched with.
A volume preparation that failed before sending any create, such as a missing `mkfs.ext4` or a scratch or format failure, withdraws its mark and returns `ServiceUnavailable`, so a retry may try again once the host is fixed.
Any other marked step whose result is not visible is reported as `IncorrectState` and never resent; a volume that is not encrypted by its own record is refused as `UnsupportedOperation` with its ID kept, and the daemon re-checks the volume's encryption and the guest's size at launch.
The daemon never receives the worker's seed; the store's NATS server accepts only that public key, and the reply's bound VM and address are not readiness, which only the worker can check.
After boot, the worker's long-running `sqs-worker ready` step mounts and verifies its disk again and calls the internal `spinifex-sqs``DescribeStore` action, which admits only the current worker and returns the bound store's address and recorded public key, or pending.
It loads its key only if it matches that public key, then connects to the store's NATS on port 4222 with it and asks JetStream for account info, each pass bounded to 30 seconds.
The step runs until shutdown with no overall deadline, so a store the operator launches at any time is picked up: it asks again every 5 seconds while the store is pending, not answering or the reply was lost, and re-checks a ready store every 30 seconds.
A pass writes only the address and public key to `/run/sqs-worker/store-ready`; any failed pass removes that file, and so does the step's own exit.
A start that finds shutdown in progress or no worker key logs once and waits in place rather than exiting, so init does not respawn it in a loop; shutdown stops it like a running watcher.
Init respawns the step, at most once every 5 seconds, and every start removes a marker left by a watcher that was killed or crashed before the new watcher runs.
A ready store that fails is noticed up to one re-check interval plus one bounded call later, so the marker can be up to about a minute stale.
Shutdown blocks further respawns, removes the marker, and stops the watcher by its recorded PID and start time, escalating to SIGKILL after a budget; it unmounts the worker disk only once the watcher is confirmed gone, and a survivor leaves every mount in place.
A mismatched key, a store that refuses the key, a key missing from the disk or an unclassified gateway refusal is asked again only every 5 minutes, so it never spins and still recovers if the operator fixes it; a missing key is never replaced.
The NKey authenticates the worker to the store, but nothing authenticates the store to the worker: there is no TLS, so a pass means only that something at that address accepted the key.
This is store readiness for the worker alone; there is no SQS API, and no production sizing or support is claimed.
Evidence is local: unit tests, the guest shell test and an AWS SDK test that creates a queue and sends, receives and deletes a message through the real worker loop against a real local JetStream store and a restart.
A real guest boot, its metadata route on a real ENI and the signed path on a live node are unproven, Viperblock's at-rest key is held by the platform, and SQS-managed message encryption is not implemented, so the profile must not hold tenant production data.
## Configuration
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
brynmailer-mdc
