* feat(ui): group launch subnets as public or private by their route tables
A subnet is public when its explicit route table, or else its VPC's main table, routes to an internet gateway. The NAT gateway form filters by the same rule instead of MapPublicIpOnLaunch.
* fix(ui): keep the subnet forms usable when route tables cannot be read
* fix(ui): encode ARN path parameters once so breadcrumbs show the ARN
Links passed encodeURIComponent output as route params, which the router encodes again, so the breadcrumb showed arn%3Aaws.... The test router mock now encodes params like the real router.
* test(ui): prove ARN routing through the real router and old bookmarks
* test(ui): type the routing test's SDK mock and keep its hooks in the describe
* fix(rds): grow and delete a DB instance whose VM is stopped with its volume attached
* test(rds): run the stopped-terminate commander test in parallel
* fix(rds): hold a delete behind an active modify and detach on terminate only the terminated instance's volume
* fix(rds): repeat the guarded detach when a stopped terminate left the data volume attached
* fix(rds): take the data volume out of a stopped DB VM's record before releasing it and terminating the VM
* fix(rds): tear down from the re-read record and keep an accepted delete when a modify fails
* fix(rds): release a stopped DB VM's data volume the way a stopped-instance detach does
* feat(lambda): add the unregistered RequestResponse Invoke path
Invoke resolves the function and qualifier with the existing lookup,
authorizes lambda:InvokeFunction on the function ARN, and hands one
synchronous attempt to an injected Executor. With no executor it fails
closed; a Pending function is ResourceConflictException. An accepted
attempt is not cancelled by a caller disconnect: only the wait is bounded.
Event, DryRun, LogType Tail and other unimplemented options are refused
explicitly. Lambda stays unregistered and no user code runs.
Also makes two flaky tests deterministic: the workerboot key-mismatch test
could pick an identical key 1 time in 256, and the loadgen first-breach
test depended on a 2ms op staying under a 30ms wall-clock p99.
* feat: track each Lambda code upload before storing it
CreateFunction now writes a per-Region upload record before the name
claim and before any code Put, so a failed upload, rollback, expired
claim or late original writer cannot leave an untracked copy. Refused
or ambiguous outcomes keep the record; nothing is deleted or inferred
complete from an expired claim.
CodeStored validates the upload record before committing and refuses a
missing, unreadable, future-format, mismatched or non-owed record
without changing either record. Attach CASes only the checked record.
* feat(lambda): run test-link Invoke in a daemon-owned microVM
Behind the lambdatestlink build tag only. The gateway executor presigns
the exact code digest and sends one NATS request to this node's daemon;
the daemon fetches and verifies the code, then runs one fresh microVM
per invocation through a runner that holds QEMU's creating thread,
kills only through a verified pidfd, and counts a stop only on a
recorded exit status. The host enforces the function timeout from
dispatch, and a late result is a timeout.
Each attempt writes a marker before allocating an address or tap and
refuses further test invocations while any marker remains, including
after a restart. Taps are created exclusively and removed only while
their ifindex and owner alias still match. Anything whose stop or
release is not proven stays held.
The default build is unchanged: Invoke refuses before presigning,
sending or allocating, and Lambda stays unrouted.
Limits: functions never leave Pending (the env15 run activated its
test record by hand); no grant mechanism; a hung daemon leaves the hard
timeout unenforced; held attempts are not recovered after a restart;
requests are bounded by the 1 MiB NATS message; a timeout is an unknown
outcome rather than AWS's Unhandled error.
* feat(lambda): enforce attempt grants and a QEMU hard-kill backstop on the test link
Test-only and local: everything here is behind the lambdatestlink build tag
except the new domains/lambda/execution owner package. The default gateway
still refuses Invoke before presigning, sending or allocating, the normal
guest agent stays closed, and Lambda stays unregistered.
Attempt grants (domains/lambda/execution):
- One owner record per node holds the current attempt; every change is a
conditional write, and an unrecognised format is left untouched.
- The gateway issues a grant binding request ID, account, Region, function,
revision and code digest before anything is sent; a second attempt is
refused while one is unresolved.
- The daemon starts only under the current matching grant, before it
downloads code or allocates anything; stale, replayed or mismatched
grants are refused with the record unchanged.
- A result counts only if the owner accepts it under the dispatch deadline
while the attempt is running; a late or uncertain acceptance is reported
as an unknown outcome, never success.
- An attempt ends only after the VM stop is proven and resources are
released, or when nothing was launched; otherwise it is held as
ambiguous. Time alone never releases it. An uncertain stop write is an
unknown outcome: it may already have landed.
QEMU hard-kill backstop (runtime/compute/lambda/testlink):
- A preloaded initializer arms a SIGKILL timer inside QEMU before main and
acknowledges with QEMU's PID, the deadline and a per-attempt nonce. Any
failure after it loads kills the process before main.
- The daemon checks the initializer against a pinned SHA-256 before launch,
builds QEMU's environment without inherited LD_* or backstop settings,
and sends no invocation frame without the exact acknowledgement; a
missing or wrong acknowledgement kills QEMU and keeps state until the
stop is proven.
Limits, not production readiness:
- Not the M1 decision. The grant binds request identity and code digest,
not payload, memory or function ARN, and the guest does not yet bind
generation or sequence.
- If the preload never loads (missing file, static or setuid QEMU), nothing
arms; a daemon hung before refusing can leave QEMU alive, and the only
guarantee is that no invocation frame was sent.
- Hash-then-load has a replacement race and is not a trust boundary; the
initializer is built and installed by hand, with no packaging.
- A killed QEMU stays a zombie until the daemon reaps it; held attempts are
not recovered after restart; duplicate attempt IDs are checked only
against the current attempt.
* feat(lambda): seal the test-link timer library and bind the guest to its grant
Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and
the normal guest agent stays fail-closed.
Timer library: the daemon reads the hard-kill initializer once, checks those
bytes against the pinned digest, copies them into a sealed memfd, verifies the
seals and re-hashes the sealed copy, then passes it to QEMU as an inherited fd
loaded through /proc/self/fd. The initializer closes that fd before it
acknowledges. A kernel that rejects MFD_EXEC, or a memfd_noexec policy that
refuses it, refuses the attempt; there is no fallback.
Guest binding: the daemon writes a test-grant fw_cfg entry with the owner's
grant ID, attempt ID, environment ID and generation, and the guest's scope now
comes from that grant. Before installing code or starting the bootstrap, the
guest refuses unless the grant names the authenticated environment generation,
the first frame carries that grant and attempt ID, and the frame deadline is
present and still ahead on the guest clock.
Limits:
- The digest is only as trustworthy as the build input that pins it; the memfd
closes the swap window between check and load, it does not authenticate it.
- Not yet shown under the real spinifex-daemon systemd sandbox
(MemoryDenyWriteExecute, ProtectProc) or with memfd_noexec set to 1 or 2.
- fw_cfg and the request frame both come from the executor, so matching them
does not independently authenticate owner authority. This is not the M1
decision.
- The guest clock check is a local guard only; it cannot detect clock skew or a
delayed, superseded grant. The guest has no channel to the owner and checks
the generation only on the first frame.
- Held attempts are still not recovered after a daemon restart.
* feat(lambda): recover held test-link attempts after a daemon restart
Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and
the normal guest agent stays fail-closed.
Launch identity: attempt.Run passes the session callback its own duplicate of
the clone-time pidfd and closes SessionDone before publishing the result.
Before the guest is sent anything, the daemon takes the VM's pidfs inode from
that exact handle (refusing if the pidfd is not on pidfs or the VM has
exited), with the host machine ID, boot ID, PID namespace and the launching
daemon's own pidfs identity, and records it in the owner record with a
conditional Launched transition. A failed write holds the attempt as
ambiguous.
Recovery: RecoverLoop retries for a bounded number of rounds on the daemon
context, each round holding the node slot so it never runs beside a request.
It holds anything it cannot scope to the recorded boot and PID namespace with
the launcher gone, and holds an attempt with no launch identity. A PID with no
process, or one now held by a process with another pidfs inode, proves the
stop without signalling; the recorded VM still running is killed through its
own pidfd and counted stopped only once the pidfd reports the exit.
Cleanup order: files, tap and address (MgmtIPAllocator.ReleaseChecked, which
reports an unconfirmed release; Release is unchanged for its callers), then a
new owner state released that still blocks Issue, then the marker, then
ended. Recovery finishes from either side of each step.
Limits:
- Reboot and partition recovery stay held until host identity or fencing is
established; a machine ID can be copied, so it is not host proof.
- A launch that crashes before its identity is recorded stays held.
- No recovery if the daemon cannot restart or hangs.
- Not yet run under the real spinifex-daemon systemd unit, and no kernel
without pidfs was tested.
- This is not the M1 decision.
* fix(lambda): refuse ownerless checked mgmt IP releases
ReleaseChecked now refuses an entry that names no node, or an allocator
with no node bound, and leaves both the cluster and local entries in place.
The checked path deletes its local entry only after a confirmed release
and only if it still holds the same address. Legacy Release is unchanged.
Callers must serialize Allocate and ReleaseChecked for one instance ID;
the Lambda test-link slot does.
Recovery tests that need real pidfs now skip only when the kernel's pidfds
are not on pidfs, and a non-pidfs descriptor is shown to be refused. The
runtime still fails closed. Adds an owner test for Store.Launched.
* fix(lambda): retry test-link recovery until it settles
RecoverLoop now retries on the daemon context until the held attempt
settles, starting at 5s and doubling to a 5 minute cap, instead of
giving up after ten rounds. Each round still takes the node slot, so it
never runs beside Handle. It covers only what a previous daemon left:
an attempt a later Handle in the same daemon holds stays held until the
next restart.
The serving log now says the guest does not verify owner grant
authority.
* feat(lambda): build the test-link daemon and initializer as one pair
Add make build-lambda-testlink. It compiles the hard-kill initializer,
checks it against the 2.34 glibc ceiling, pins its SHA-256 into a
lambdatestlink spx build, and proves with TestLambdaHardKillPin that the
daemon's own pin accepts that file and refuses a changed one. Both files
are built in a staging directory and moved together only on success; any
earlier pair is removed first, so a failed rebuild leaves no pair. The
target refuses to build unless Go targets linux on the host architecture.
The initializer now uses _POSIX_C_SOURCE with -std=c17, so it no longer
needs __isoc23_strtol (glibc 2.38). The daemon refuses an initializer
built for another ELF machine or class.
Startup now begins recovery with only the owner store, tap and address
cleanup. A missing bridge address, pin, trust store or matching
initializer stops new invocations but not recovery.
Limits: test-only and unregistered. The pin is only as trustworthy as
the build host and toolchain; the initializer is still installed by
hand and ships in no release artefact. linux/arm64 is untested. Recovery
needs pidfs, which older kernels may lack. This is not the M1 decision.
* feat(lambda): fix one CLOCK_BOOTTIME backstop deadline per test-link attempt
Test-only (lambdatestlink); the default build and public Lambda route are
unchanged.
- The daemon reads CLOCK_BOOTTIME once before the grant Start and fixes one
absolute deadline: bounded code fetch + function timeout + boot allowance +
kill margin. Nothing recomputes or extends it; Handle never retries and
recovery never relaunches.
- The hard-kill initializer arms its SIGKILL timer at that absolute deadline
(SPX_HARDKILL_DEADLINE_NS, TIMER_ABSTIME) instead of relative to QEMU start.
- Dispatch is refused once the deadline has passed: before QEMU starts, when
the initializer ack is read at or after it (VM killed, stop proven), and
immediately before the invocation frame is sent. A clock read failure counts
as expired.
Limits: tap/IP setup, image read, initializer seal and QEMU start have no caps
of their own beyond the boot allowance. An overdue timer is not shown to fire
before vCPUs resume after host suspend. Owner-side retention, cross-node
ordering, clock-rate assumptions and M1 remain unsolved; nothing is released by
time and held attempts still need stop proof.
* feat(lambda): boot the test-link runtime image from a pinned digest
The test-link daemon now prepares its runtime image once at startup: each
boot file is opened without following symlinks or blocking, capped by size,
copied into a sealed memfd and hashed there, and the image digest over those
sealed copies must equal LAMBDA_IMAGE_DIGEST pinned into the build. QEMU boots
only the sealed copies (fds 3-5), and firmware comes from the image directory
rather than the shared microvm qboot.rom. The digest identifies the bytes; it
does not authenticate where they came from.
build-lambda-testlink-image builds a separately tagged guest into its own
outputs and prints the digest with the daemon's own file selection. Both
test-link targets refuse anything but an x86_64 host building linux/amd64,
and the digest input is read raw and validated before it reaches ldflags.
* test: cover test-link image load failures and daemon startup release
Exercise each sealed-copy failure path through the memfd hook and check
partial copies are closed. Extract the daemon's readiness and release
steps so startup tests can show the image is released when serving fails
and that Handle refuses without a prepared image.
* fix: align test-link Invoke errors with AWS and verify guest build tag
Malformed JSON payloads return 400 InvalidRequestContentException and
oversized requests return 413 with X-Amzn-Errortype
RequestEntityTooLargeException, both before the executor runs. Timeouts
and lost or failed sends are reported as unknown rather than as a
function timeout, and any Sandbox.Timedout result is treated as unknown.
Oversized function responses report Unhandled with
Function.ResponseSizeTooLarge in the body.
Add lambda-image-agent-tag, which reads the agent's build information
from each built initramfs and refuses an image whose lambdatestlink tag
does not match the target. Add tests that the default gateway cannot
invoke and that management IP Release drops the local entry when the
address is held elsewhere or the KV is down.
* test: use the shared RequireAWSCode helper in Lambda tests
The per-package requireCode helpers were replaced by
testutil.RequireAWSCode, which makes the same assertions; the Lambda
Invoke and upload tests still called the removed helper.
* fix: validate and authorize qualified Lambda Invoke requests
Invoke now runs every deterministic check before authorization and
authorizes the qualified function ARN derived from the same validated
target. Qualifiers follow the documented FunctionName and Qualifier
patterns and length limits, and a malformed query string is refused.
Code download errors no longer carry transport detail, so a presigned
credential in a bad redirect cannot reach the reply or the gateway log.
A seam built only with lambdatestlink and lambdatestlinkseam drives real
session outcomes through the executor and composed gateway in one test.
* test: restore the loadgen and workerboot tests to dev
* test: drive a bootstrap-posted timeout through the composed Invoke path
* fix: carry handler and qualifier to the test-link runtime and refuse function environments
The test link now passes the stored handler to the guest as _HANDLER and the
validated qualifier into the invoked-function ARN. Functions with environment
variables are refused before any presign, grant or dispatch, since delivering
them needs a separate secret-transport design.
* test: stub QEMU for the test-link package so it runs on hosts without it
* fix(vm): delete volumes only once QEMU is proven gone on a forced terminate
* fix(vm): treat an unreadable QEMU pid file as unknown and keep the UEFI obligation on a failed kill
* fix: judge QEMU exit only on the node that ran the guest
* fix(vm): keep unknown PID state and the local UEFI store obligation across reapers
* fix(vm): treat a non-positive QEMU pid as unknown state in both exit gates
* fix(vm): keep PID files that do not name a positive process at every deletion site
* fix(vm): treat only ESRCH as proof a QEMU exited, not any failed signal probe
* refactor: expose the signal probe as a plain setter instead of importing testing
* fix(daemon): complete a terminated UEFI store mark only once QEMU teardown is done
* fix(daemon): treat an expired terminated record as unproven QEMU exit for the UEFI store reaper
* test: separate the embedded field in the expired-record fake
* feat: run Spinifex on latitude.sh bare metal
Add a latitude.sh provider and deploy script. One node uses a static Elastic IP pool. Three nodes use a latitude floating pool over a fixed Elastic IP set, with the cluster and Geneve planes on a latitude.sh VLAN.
The floating engine gains a FixedSet provider interface. Its allocation token, re-read before a move and holder label apply to fixed sets only, so Vultr records and behaviour are unchanged.
Bead mulga-c43h3.3
* fix: check a server's project before destroy and refuse mismatched response IDs
A recorded server must sit in the project this deployment's Terraform holds, so a recycled ID with the same hostname elsewhere stops destroy. A missing project or ID refuses too.
The client rejects an Elastic IP response whose ID is not the requested one, without showing it.
Bead mulga-c43h3.3
* feat: cap a latitude pool at five Elastic IPs
Config validation rejects a source="latitude" pool listing more than MaxLatitudeElasticIPs addresses, so an allocation makes at most five bounded provider reads.
Bead mulga-c43h3.3
* fix: remember the project ID before destroy for the cleanup sweep
The Terraform output that names the project is gone once destroy has run, so the sweep could not tell a matching server from a foreign one and kept both.
Bead mulga-c43h3.3
* feat(ebs): record an unpublish obligation on force detach and terminate
A force detach, or a terminate whose unmount did not complete, clears a
volume's attachment while the node that served it may still export it.
The request now records an UnpublishObligation naming that publish in the
new spinifex-ebs-unpublish-obligations bucket before the clear, and
returns without waiting on the node.
The node-local unpublishObligationReaper settles its own node's
obligations under the mounter's per-volume lock, through the existing
UnpublishVolume client: it seals a surviving volume, drops a deleting or
absent one unsealed, and closes without a provider call when a newer
attachment or a newer publish on the node owns the export. A cluster-wide
pass closes the obligations of nodes removed from the cluster, and
spx admin volumes obligations lists the open ones.
The force-complete terminate and a stopped-instance terminate after a
failed seal record the same obligation; a terminate whose unmount
completed records none. ForceDetachVolume now answers with the attachment
in state detaching, as AWS answers a detach.
* fix(ebs): refuse a publish while the obligation reaper seals the volume
A publish waited on the volume lock while the unpublish obligation reaper
held it through a seal, so AttachVolume waited on work that scales with
the volume. A publish now refuses at once with ErrVolumeSettling, which
AttachVolume answers as IncorrectState, and the reaper takes the lock only
when nobody holds or awaits it.
* fix(ebs): refuse a publish behind any seal of the volume, a stop's included
A stop's or detach's sealing unpublish now marks the volume lock sealing,
so an attach after a force detach answers IncorrectState instead of
waiting out the seal. A waiting seal lets waiting publishes go first.
* fix(ebs): close a superseded unpublish obligation past a newer publish here
After a force detach and an attach again on the same node, the reaper
returned on the newer publish before it read the volume, so the old
obligation stayed open until the new attachment ended.
* fix(vm): recover a wedged terminate without waiting on its stalled unmount
The unmount records every volume as owed before it starts, so recovery
keeps that record rather than wait for the request lock the stalled
seal holds.
Resolves GHSA-7rv5-mrxv-xcr9, GHSA-wp97-x4hv-qvhp, GHSA-97r2-qh9m-474h,
GHSA-9wx9-5pv9-3349, GHSA-f86p-h4j2-7qgj (seroval) and GHSA-g3hv-75wg-m249
(minimatch, dev-only via eslint, forced with a pnpm override).
The conformance and NBD client suites, the fence, lease, dirty-marker,
abandon, takeover and stale-mount tests each stand up their own embedded
JetStream and Config, so they now overlap instead of running back to back.
Tests that touch package or process-wide state stay serial.
The six account-teardown tests and the unanswered network-test case each
wait about 5s on a production timer. Calling t.Parallel() first queued all
seven through the parallel slots; starting the teardown or network test
before t.Parallel() lets the wait run during the serial phase instead.
Also generate 1024-bit RSA keys in TestMain and shorten the test-only
context in TestQueryMetaNodeStatus_Unreachable.
go test -coverprofile at GOMAXPROCS=4: 16.9s -> 7.1-8.0s.
The four tests that wait out the fixed 3s describe fan-out deadline for a
silent node (9s, 6s, 6s, 6s) were queued behind the package's short
parallel tests, two of them as subtests of a table test. Hoist the two
slow table cases into top-level tests and move all four into a file that
sorts first, so they take parallel slots before the short tests do. The
table body becomes a shared helper; no case or assertion changes.
Each test stands up its own embedded JetStream server, so top-level tests
and independent subtests now call t.Parallel. Tests that redeclare the
process-wide cluster size or capture the default logger stay sequential.
Package wall time drops from ~23.5s to ~4.4s locally.
Six account-delete tests each wait one 5s teardown drain poll, and the
network-test and storage-status tests wait out fixed 5s and 2s timeouts.
Each uses its own embedded NATS, so they run concurrently: the package
drops from about 37s to 8s.
* test: pin instance inventory gaps G1-G3 as current behaviour
* feat(ec2): strict account instance inventory from the record store
* feat(ec2): strict spot-request and volume inventories
* fix(ec2): strict instance inventory fails on node-only instances
* fix(ec2): strict instance inventory sees error and late strays and rejects malformed records
The ecs-agent now holds an aws.CredentialsProvider (a CredentialsCache over
imdscreds.NewProvider) and ecsgw.New takes a *gwsign.Signer, matching eksgw,
rdsgw and elbv2. imdscreds.Fetch and stsauth.AssumeRole return aws.Credentials,
so agents/ecs/credentials and the field-copy adapters are removed.
Ben Duncan