9 Commits

Author	SHA1	Message	Date
Josh Siviour	b9bce93877	feat(lambda): test-only Invoke path on a digest-pinned microVM (draft) (#1582) ... * feat(lambda): add the unregistered RequestResponse Invoke path Invoke resolves the function and qualifier with the existing lookup, authorizes lambda:InvokeFunction on the function ARN, and hands one synchronous attempt to an injected Executor. With no executor it fails closed; a Pending function is ResourceConflictException. An accepted attempt is not cancelled by a caller disconnect: only the wait is bounded. Event, DryRun, LogType Tail and other unimplemented options are refused explicitly. Lambda stays unregistered and no user code runs. Also makes two flaky tests deterministic: the workerboot key-mismatch test could pick an identical key 1 time in 256, and the loadgen first-breach test depended on a 2ms op staying under a 30ms wall-clock p99. * feat: track each Lambda code upload before storing it CreateFunction now writes a per-Region upload record before the name claim and before any code Put, so a failed upload, rollback, expired claim or late original writer cannot leave an untracked copy. Refused or ambiguous outcomes keep the record; nothing is deleted or inferred complete from an expired claim. CodeStored validates the upload record before committing and refuses a missing, unreadable, future-format, mismatched or non-owed record without changing either record. Attach CASes only the checked record. * feat(lambda): run test-link Invoke in a daemon-owned microVM Behind the lambdatestlink build tag only. The gateway executor presigns the exact code digest and sends one NATS request to this node's daemon; the daemon fetches and verifies the code, then runs one fresh microVM per invocation through a runner that holds QEMU's creating thread, kills only through a verified pidfd, and counts a stop only on a recorded exit status. The host enforces the function timeout from dispatch, and a late result is a timeout. Each attempt writes a marker before allocating an address or tap and refuses further test invocations while any marker remains, including after a restart. Taps are created exclusively and removed only while their ifindex and owner alias still match. Anything whose stop or release is not proven stays held. The default build is unchanged: Invoke refuses before presigning, sending or allocating, and Lambda stays unrouted. Limits: functions never leave Pending (the env15 run activated its test record by hand); no grant mechanism; a hung daemon leaves the hard timeout unenforced; held attempts are not recovered after a restart; requests are bounded by the 1 MiB NATS message; a timeout is an unknown outcome rather than AWS's Unhandled error. * feat(lambda): enforce attempt grants and a QEMU hard-kill backstop on the test link Test-only and local: everything here is behind the lambdatestlink build tag except the new domains/lambda/execution owner package. The default gateway still refuses Invoke before presigning, sending or allocating, the normal guest agent stays closed, and Lambda stays unregistered. Attempt grants (domains/lambda/execution): - One owner record per node holds the current attempt; every change is a conditional write, and an unrecognised format is left untouched. - The gateway issues a grant binding request ID, account, Region, function, revision and code digest before anything is sent; a second attempt is refused while one is unresolved. - The daemon starts only under the current matching grant, before it downloads code or allocates anything; stale, replayed or mismatched grants are refused with the record unchanged. - A result counts only if the owner accepts it under the dispatch deadline while the attempt is running; a late or uncertain acceptance is reported as an unknown outcome, never success. - An attempt ends only after the VM stop is proven and resources are released, or when nothing was launched; otherwise it is held as ambiguous. Time alone never releases it. An uncertain stop write is an unknown outcome: it may already have landed. QEMU hard-kill backstop (runtime/compute/lambda/testlink): - A preloaded initializer arms a SIGKILL timer inside QEMU before main and acknowledges with QEMU's PID, the deadline and a per-attempt nonce. Any failure after it loads kills the process before main. - The daemon checks the initializer against a pinned SHA-256 before launch, builds QEMU's environment without inherited LD_* or backstop settings, and sends no invocation frame without the exact acknowledgement; a missing or wrong acknowledgement kills QEMU and keeps state until the stop is proven. Limits, not production readiness: - Not the M1 decision. The grant binds request identity and code digest, not payload, memory or function ARN, and the guest does not yet bind generation or sequence. - If the preload never loads (missing file, static or setuid QEMU), nothing arms; a daemon hung before refusing can leave QEMU alive, and the only guarantee is that no invocation frame was sent. - Hash-then-load has a replacement race and is not a trust boundary; the initializer is built and installed by hand, with no packaging. - A killed QEMU stays a zombie until the daemon reaps it; held attempts are not recovered after restart; duplicate attempt IDs are checked only against the current attempt. * feat(lambda): seal the test-link timer library and bind the guest to its grant Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Timer library: the daemon reads the hard-kill initializer once, checks those bytes against the pinned digest, copies them into a sealed memfd, verifies the seals and re-hashes the sealed copy, then passes it to QEMU as an inherited fd loaded through /proc/self/fd. The initializer closes that fd before it acknowledges. A kernel that rejects MFD_EXEC, or a memfd_noexec policy that refuses it, refuses the attempt; there is no fallback. Guest binding: the daemon writes a test-grant fw_cfg entry with the owner's grant ID, attempt ID, environment ID and generation, and the guest's scope now comes from that grant. Before installing code or starting the bootstrap, the guest refuses unless the grant names the authenticated environment generation, the first frame carries that grant and attempt ID, and the frame deadline is present and still ahead on the guest clock. Limits: - The digest is only as trustworthy as the build input that pins it; the memfd closes the swap window between check and load, it does not authenticate it. - Not yet shown under the real spinifex-daemon systemd sandbox (MemoryDenyWriteExecute, ProtectProc) or with memfd_noexec set to 1 or 2. - fw_cfg and the request frame both come from the executor, so matching them does not independently authenticate owner authority. This is not the M1 decision. - The guest clock check is a local guard only; it cannot detect clock skew or a delayed, superseded grant. The guest has no channel to the owner and checks the generation only on the first frame. - Held attempts are still not recovered after a daemon restart. * feat(lambda): recover held test-link attempts after a daemon restart Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Launch identity: attempt.Run passes the session callback its own duplicate of the clone-time pidfd and closes SessionDone before publishing the result. Before the guest is sent anything, the daemon takes the VM's pidfs inode from that exact handle (refusing if the pidfd is not on pidfs or the VM has exited), with the host machine ID, boot ID, PID namespace and the launching daemon's own pidfs identity, and records it in the owner record with a conditional Launched transition. A failed write holds the attempt as ambiguous. Recovery: RecoverLoop retries for a bounded number of rounds on the daemon context, each round holding the node slot so it never runs beside a request. It holds anything it cannot scope to the recorded boot and PID namespace with the launcher gone, and holds an attempt with no launch identity. A PID with no process, or one now held by a process with another pidfs inode, proves the stop without signalling; the recorded VM still running is killed through its own pidfd and counted stopped only once the pidfd reports the exit. Cleanup order: files, tap and address (MgmtIPAllocator.ReleaseChecked, which reports an unconfirmed release; Release is unchanged for its callers), then a new owner state released that still blocks Issue, then the marker, then ended. Recovery finishes from either side of each step. Limits: - Reboot and partition recovery stay held until host identity or fencing is established; a machine ID can be copied, so it is not host proof. - A launch that crashes before its identity is recorded stays held. - No recovery if the daemon cannot restart or hangs. - Not yet run under the real spinifex-daemon systemd unit, and no kernel without pidfs was tested. - This is not the M1 decision. * fix(lambda): refuse ownerless checked mgmt IP releases ReleaseChecked now refuses an entry that names no node, or an allocator with no node bound, and leaves both the cluster and local entries in place. The checked path deletes its local entry only after a confirmed release and only if it still holds the same address. Legacy Release is unchanged. Callers must serialize Allocate and ReleaseChecked for one instance ID; the Lambda test-link slot does. Recovery tests that need real pidfs now skip only when the kernel's pidfds are not on pidfs, and a non-pidfs descriptor is shown to be refused. The runtime still fails closed. Adds an owner test for Store.Launched. * fix(lambda): retry test-link recovery until it settles RecoverLoop now retries on the daemon context until the held attempt settles, starting at 5s and doubling to a 5 minute cap, instead of giving up after ten rounds. Each round still takes the node slot, so it never runs beside Handle. It covers only what a previous daemon left: an attempt a later Handle in the same daemon holds stays held until the next restart. The serving log now says the guest does not verify owner grant authority. * feat(lambda): build the test-link daemon and initializer as one pair Add make build-lambda-testlink. It compiles the hard-kill initializer, checks it against the 2.34 glibc ceiling, pins its SHA-256 into a lambdatestlink spx build, and proves with TestLambdaHardKillPin that the daemon's own pin accepts that file and refuses a changed one. Both files are built in a staging directory and moved together only on success; any earlier pair is removed first, so a failed rebuild leaves no pair. The target refuses to build unless Go targets linux on the host architecture. The initializer now uses _POSIX_C_SOURCE with -std=c17, so it no longer needs __isoc23_strtol (glibc 2.38). The daemon refuses an initializer built for another ELF machine or class. Startup now begins recovery with only the owner store, tap and address cleanup. A missing bridge address, pin, trust store or matching initializer stops new invocations but not recovery. Limits: test-only and unregistered. The pin is only as trustworthy as the build host and toolchain; the initializer is still installed by hand and ships in no release artefact. linux/arm64 is untested. Recovery needs pidfs, which older kernels may lack. This is not the M1 decision. * feat(lambda): fix one CLOCK_BOOTTIME backstop deadline per test-link attempt Test-only (lambdatestlink); the default build and public Lambda route are unchanged. - The daemon reads CLOCK_BOOTTIME once before the grant Start and fixes one absolute deadline: bounded code fetch + function timeout + boot allowance + kill margin. Nothing recomputes or extends it; Handle never retries and recovery never relaunches. - The hard-kill initializer arms its SIGKILL timer at that absolute deadline (SPX_HARDKILL_DEADLINE_NS, TIMER_ABSTIME) instead of relative to QEMU start. - Dispatch is refused once the deadline has passed: before QEMU starts, when the initializer ack is read at or after it (VM killed, stop proven), and immediately before the invocation frame is sent. A clock read failure counts as expired. Limits: tap/IP setup, image read, initializer seal and QEMU start have no caps of their own beyond the boot allowance. An overdue timer is not shown to fire before vCPUs resume after host suspend. Owner-side retention, cross-node ordering, clock-rate assumptions and M1 remain unsolved; nothing is released by time and held attempts still need stop proof. * feat(lambda): boot the test-link runtime image from a pinned digest The test-link daemon now prepares its runtime image once at startup: each boot file is opened without following symlinks or blocking, capped by size, copied into a sealed memfd and hashed there, and the image digest over those sealed copies must equal LAMBDA_IMAGE_DIGEST pinned into the build. QEMU boots only the sealed copies (fds 3-5), and firmware comes from the image directory rather than the shared microvm qboot.rom. The digest identifies the bytes; it does not authenticate where they came from. build-lambda-testlink-image builds a separately tagged guest into its own outputs and prints the digest with the daemon's own file selection. Both test-link targets refuse anything but an x86_64 host building linux/amd64, and the digest input is read raw and validated before it reaches ldflags. * test: cover test-link image load failures and daemon startup release Exercise each sealed-copy failure path through the memfd hook and check partial copies are closed. Extract the daemon's readiness and release steps so startup tests can show the image is released when serving fails and that Handle refuses without a prepared image. * fix: align test-link Invoke errors with AWS and verify guest build tag Malformed JSON payloads return 400 InvalidRequestContentException and oversized requests return 413 with X-Amzn-Errortype RequestEntityTooLargeException, both before the executor runs. Timeouts and lost or failed sends are reported as unknown rather than as a function timeout, and any Sandbox.Timedout result is treated as unknown. Oversized function responses report Unhandled with Function.ResponseSizeTooLarge in the body. Add lambda-image-agent-tag, which reads the agent's build information from each built initramfs and refuses an image whose lambdatestlink tag does not match the target. Add tests that the default gateway cannot invoke and that management IP Release drops the local entry when the address is held elsewhere or the KV is down. * test: use the shared RequireAWSCode helper in Lambda tests The per-package requireCode helpers were replaced by testutil.RequireAWSCode, which makes the same assertions; the Lambda Invoke and upload tests still called the removed helper. * fix: validate and authorize qualified Lambda Invoke requests Invoke now runs every deterministic check before authorization and authorizes the qualified function ARN derived from the same validated target. Qualifiers follow the documented FunctionName and Qualifier patterns and length limits, and a malformed query string is refused. Code download errors no longer carry transport detail, so a presigned credential in a bad redirect cannot reach the reply or the gateway log. A seam built only with lambdatestlink and lambdatestlinkseam drives real session outcomes through the executor and composed gateway in one test. * test: restore the loadgen and workerboot tests to dev * test: drive a bootstrap-posted timeout through the composed Invoke path * fix: carry handler and qualifier to the test-link runtime and refuse function environments The test link now passes the stored handler to the guest as _HANDLER and the validated qualifier into the invoked-function ARN. Functions with environment variables are refused before any presign, grant or dispatch, since delivering them needs a separate secret-transport design. * test: stub QEMU for the test-link package so it runs on hosts without it	2026-11-04 22:51:49 +10:00
Josh Siviour	a541ee312d	feat(lambda): add the runtime image, guest agent and authenticated TCP control connection (not wired) (#1448) ... * feat(lambda): add the Lambda runtime image and guest agent * fix(lambda): stop the guest agent after a timed-out invocation * docs(lambda): declare the L-GUEST executor-to-agent contract * refactor(vm): remove the unused virtio-serial port builder Lambda guest control moves to TCP over the microVM's network, so no caller needs SerialPorts. This reverts d7905107b. * fix(lambda): drop the serial carrier from guest startup and keep virtio-net run-agent no longer waits for a carrier device named on the kernel command line, init loads virtio_net instead of virtio_console, and the image keeps virtio_net's failover and net_failover dependencies. The normal agent still refuses to start without an authenticated transport. * feat(lambda): authenticate the guest's TCP control connection with pinned mutual TLS hostlink loads the per-generation identity from fw_cfg and dials the host over TLS 1.3 with ALPN spinifex-lambda-guest/1. VerifyPeer requires exactly the pinned certificate, validates it, and checks its single URI SAN names the expected scope and role. The L-GUEST contract now describes this transport; message framing and fencing stay open, and the normal agent still refuses to start. * feat(lambda): add the host end of the guest's TCP mutual TLS connection guestconn creates fresh host and guest identities for each environment generation, produces the fw_cfg entries the guest reads, and accepts only that generation's exact guest certificate. Peers on the untrusted bridge that fail the handshake are closed and the listener keeps waiting until its context ends; the cancel callback finishes before the deadline is cleared so the listener can be reused. The pinned-certificate check and the L-GUEST vocabulary move to foundation/pki and contracts/lambda/v1 so both ends share them. * docs(lambda): describe the shared connection vocabulary in the contract README	2026-11-02 22:09:58 +10:00
Josh Siviour	b034f3b2e9	fix(sqs): bring up loopback in the SQS guest before its store starts (#1409) ... The guest's SQS nats-server listens on 127.0.0.1 and the worker dials it there, but the kernel leaves lo down, so the store failed to bind and the worker never polled. Also correct the stale CA comment in init.	2026-11-02 09:40:28 +10:00
Josh Siviour	671a8a9ff5	feat(sqs): fresh single-node SQS in one system microVM (#1389) ... * feat(sqs): run queue and message owners in the worker by signed pull The SQS worker's ready step now opens the queue and message owners on its dedicated store and serves owner calls it pulls from its node's gateway over the existing instance-role channel (PollWork, CompleteWork). Each poll and post is authorized against the current worker identity, and a result is accepted only from the identity that took the work. No payload crosses platform NATS. A receipt-signing key is created on the worker disk only under the first-store decision that creates the store key, and its fingerprint is bound in the store; a missing or changed key refuses to serve. Received messages are released only when the gateway proves the caller left; receives are capped so a worst-case result fits one post. The public relay is not registered in production. * fix: harden SQS relay results and bound pending bytes Unavailable outranks client faults when encoding owner errors, and wire errors carry fixed text only, never owner or backend text. The ingress rejects incomplete or mismatched success results as unavailable, and the relay bounds total pending input bytes as well as request count. Adds tests of every relay action against real owners, the receive claim limit, release, faulting work, the receipt-key fingerprint marker and the poll loop through gateway failures. * fix: refuse SQS serving outside a single-node host cluster DescribeStore now reports the gateway's declared cluster node count, and the worker refuses to serve, before reading keys, connecting or creating any store state, unless that count is exactly one. A gateway that reports no count reads as zero and is refused. * feat(sqs): record a store server key pin once per store instance Add the store role and its internal actions so the bound store instance can learn its claim and record the public pin of its TLS server key, once, without the platform holding the private key. The worker sees the store as bound only after a pin exists and must receive a valid pin. * refactor(sqs): serve store server keys from a separate capability Keep the worker Stores interface unchanged and give the store guest's actions their own ServerKeys dependency, so existing Stores fakes need no store-key methods. * feat(sqs): launch the store with a mgmt NIC, CA and scoped store role The store now boots with the region, gateway URL and the daemon's own cluster CA, a mgmt NIC to reach the gateway, and an sqsStoreRole profile granted only the store's two internal actions, so it can authenticate its server key pin. A store record that predates these fields refuses to relaunch; its volume is kept. * feat(sqs): keep the store TLS server key on its disk and record its pin The store image gains sqs-store-key, run after the data disk is mounted. It loads or creates a P-256 server key under the disk's tls directory, records only the key's public pin with the gateway as the bound store instance, and writes a self-signed certificate for it to the run directory. nats-server then requires TLS 1.3 with that key. A record for another volume, a recorded pin with a missing key, or a key that does not match the recorded pin refuses the store; no key is regenerated and nothing on the disk is deleted. Launch-not-yet-bound and lost replies are retried. The store role vocabulary moves to contracts/sqs/v1 so the guest agent does not import a domain. * feat(sqs): pin the worker's store connection to the recorded server key The worker's readiness check and serving connection now use TLS 1.3 and accept only the store server key whose pin the bound store recorded with the gateway. The pin is checked before the worker key or any queue or message data is sent, and on every reconnect. A store with another key, a plaintext store or a malformed pin is refused as final. The worker NKey client authentication is unchanged. The host still holds the store's data disk key, so the pin does not protect the store from a privileged host. * fix(sqs): retry store key bootstrap and track guest binary sources The store's server key check moves from sysinit into the respawned run step: a gateway or IMDS outage exits after a pause so init retries it, without remounting or creating a new key, while a missing, changed or wrong-store key is recorded as refused and never retried. Stop also ends a running key helper before unmounting. The guest helper and worker binaries now depend on every main-module package dir and file they link, from go list, so edits, additions, deletions and renames all rebuild them. * feat(sqs): operator enable and status with a signed ingress test Add a per-Region SQS service record (disabled by default, pinned to one node, refused on multi-node clusters before any write), gate the relay on it, and expose `spx admin sqs enable` and `spx admin sqs status`. The gateway now hands queued work to the polling worker and reports whether one is polling. The public SQS service stays unregistered in production until its capability profile is reviewed. An integration test registers it through the harness and drives real SigV4 and IAM requests to a real worker for create, send, receive and delete, asserting no SQS body crosses platform NATS. * fix(sqs): report readiness only from observed component state Status now reports whether the tenant SQS API is registered on this gateway and is never Ready while it is not. The worker stops polling while its store connection is down, so a worker that lost its store drops out of the gateway's poller window instead of looking healthy. * wip: sqs model inventory * wip: sqs public registration * wip: sqs registration review fixes * fix(sqs): refuse pre-API store and worker records on enable A store or worker record this release creates now carries an explicit contract. enable reads both records without creating a bucket and refuses any record without it before it writes desired state or launches anything; status names the record and is never Ready. A worker replacement keeps its predecessor's contract, since it reuses the same data volume. Missing metadata is only ever grounds to refuse: nothing is upgraded, inspected, emptied, replaced or deleted. Upgrading a pre-API dev install is not supported. * fix(sqs): describe the enable refusal as an unsupported launch contract A missing or unknown contract proves neither when a record was written nor that it predates authenticated bootstrap: an older writer can drop a newer field, and a future contract is not pre-API. The refusal is unchanged; its message, status field and docs now name a missing or unsupported launch contract and say any existing volume is kept. * wip: port reviewed SQS serving path onto dev without the store VM * wip: single SQS system VM * fix(sqs): retry receipt bootstrap under supervision and drop store install path Run the receipt-key boot step from the respawned store service so a transient gateway or IMDS outage recovers without a reboot; final key and volume refusals stay refused. Remove the separate store image install and staging path from setup. * fix(sqs): align worker ready config fields * fix(sqs): stop an in-progress boot before unmounting the worker disk The receipt-key boot step ran as an unrecorded child of the store shell, so shutdown stopped only the shell and could unmount the disk under a boot still talking to the gateway. Record the boot child like the other guest processes and stop it before unmounting; a survivor keeps the disk mounted. * fix(sqs): record the boot child before its stopping check The parent wrote boot.pid only after starting the child, so a shutdown in that gap could unmount under an unrecorded boot. The child now records itself and checks stopping before it runs sqs-worker boot.	2026-11-02 09:02:06 +10:00
Josh Siviour	e155bcf155	feat(sqs): queue and Lambda-facing message API foundation (#1364) ... * feat(sqs): queue and Lambda-facing message API foundation Adds the SQS JSON 1.0 adapter (unregistered in production) with queue create, lookup, attributes, list, tags and the message operations a Lambda poller uses: SendMessage(Batch), ReceiveMessage with long polling and attribute selection, DeleteMessage(Batch) and ChangeMessageVisibility(Batch), backed by the queue record owner and the JetStream message store. String and Binary message attributes are validated, stored and checksummed; Number attributes are refused until AWS behaviour is confirmed. * test(sqs): poller lifecycle through the message owners Covers receive, release, redelivery, delete, visibility extension, queue binding of receipt handles and long polling against the real queue and JetStream message stores, from the message package itself. * test(sqs): move SDK coverage from the gateway to the SQS domain Drives the registration through dispatch.Invocation over the real queue and message owners, so the gateway package carries no SQS changes.	2026-10-31 22:27:11 +10:00
Josh Siviour	d066786faa	feat: operator-launched SQS worker and store system VMs (#1353) ... * feat: persist the SQS worker's store key on its own disk The shared SQS worker generates its store-client NKey inside the guest and keeps it in an owner-only directory on its own persistent disk, so no platform component creates or unseals the seed. The key is synced and linked into place without replacing anything; a missing, mismatched, unreadable or loosely permissioned key fails closed instead of being regenerated. * fix: make the SQS worker key durably named before returning it Every successful call now syncs the key directory and its entry in the parent before handing out a key, so a retry after a failed directory sync cannot return a key whose name is not yet durable. Only the key directory is created, inside a parent that must exist, so an unmounted worker disk fails closed. The first-start decision is explicit: an existing store's authorized key is required and never defaults to generating a new one. * docs: state that workerkey cannot detect an unmounted worker disk An existing parent directory does not prove the worker disk is mounted, since the mountpoint can exist on the initramfs after a failed mount. The error now says only that the parent is missing, and the package documents that its caller must verify the expected data volume is mounted first. * feat: add the SQS worker guest bootstrap The shared SQS worker microVM boots with sqs-worker boot, which selects exactly the disk whose serial fw_cfg names, checks and mounts it at /data, and proves from the mount table that this disk is what is mounted before loading or creating its store key there. A missing, empty or unknown store status, a contradictory or invalid store key, a missing or ambiguous disk, a failed check or mount, an empty mountpoint or a foreign mount refuses without creating a key. Only the public key is logged and published. The image is built by the store image script under SQS_MICROVM_PROFILE=worker, with the worker binary, no nats-server and an empty /data mountpoint; the store image's contents are unchanged. * fix: refuse replayable new-store status at SQS worker boot A static fw_cfg store-status of "new" survives a guest reset, so after the store exists it could still reach key creation and replace a lost worker key. Boot now accepts only "existing" with the store's authorized public key and refuses "new" before touching any disk, until a current authoritative first-store decision exists. workerkey.ForNewStore remains, uncalled. * feat: add SQS first-store claim and worker key acquisition The SQS domain records one first-store decision per Region in its own KV bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound to the worker's instance ID, claim ID and generation. Key creation is authorized only while no public key is recorded; afterwards only the recorded key is returned, and a launch intent requires it. Older generations, same-generation rivals and unreadable records authorize nothing, and records hold no private key material. workerkey.Acquire syncs a new key to disk and returns it only after its public key is recorded, reusing the on-disk key after a crash before recording and failing closed on a missing or different key afterwards. No transport carries the decision yet. * fix: read SQS first-store decisions from the stream leader KV.Get uses a direct get, which a lagging replica can answer, so a stale claimed read could authorize key creation after the leader had recorded the worker's public key, or make a superseded generation look current. Reads now use the SQS store's leader-answered stream message get, and every authorizing answer is also confirmed by a compare-and-swap at the revision read, so a stale or unavailable read authorizes nothing. * feat(sqs): add regional worker identity record and instance index The SQS domain keeps one worker identity record per Region with a monotonic generation, claim ID, pinned node and encrypted data volume, moving from launching to bound or failed under compare-and-swap. A launch intent takes the next generation before any instance exists, so the previous worker stops being current; binding writes the instance index entry first. Authorize accepts an instance only while the Region's leader-read record is bound to exactly that instance, generation and claim, confirmed by a compare-and-swap at the read revision, so an old, orphaned, rewritten or unbound index entry, or a lagging replica, authorizes nothing. Both this bucket and the first-store bucket now open through the kvutil chokepoint and refuse a stored configuration that could expire, cap or evict records or holds fewer replicas than the cluster needs. Nothing calls the store yet. * fix(sqs): make worker launch exclusive and harden durable buckets BeginLaunch now refuses a new claim while the previous attempt is launching or bound, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume. A replacement takes the next generation only after FailLaunch or the new RecordStopped confirms that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never saw. The durable bucket opener also refuses non-limits retention, per-message TTLs, asynchronous flush, a message-size cap, any discard policy other than discard-new or a per-subject discard, unacknowledged writes, a sealed stream, and a mirror or sources. * feat: add internal SQS worker first-store gateway actions Serve ClaimFirstStore and RecordWorkerPublicKey under the internal spinifex-sqs signing service, apart from the public AmazonSQS inventory. Only an sqsWorkerRole session in the system account passes the class gate, then the role policy; the worker identity is resolved from the current workerident record by the STS-set session name, never from the request, and any extra parameter is refused. Seeds and other non-public keys are refused without being recorded, logged or echoed. The gateway binds the worker-identity and first-store buckets per request and never creates them, so it provisions nothing and answers ServiceUnavailable until a worker launch has created them. * fix(sqs): fence replaced worker generations out of first-store A request that passed worker authorization before a confirmed stop and replacement could reach an unclaimed first-store record ahead of the successor and create or record the old generation's key. The launch path now raises a monotonic first-store fence past the replaced generation before BeginLaunch, which refuses a replacement until a leader read shows it; firststore refuses every call from a generation below the fence. The internal adapter no longer logs caller-chosen parameter names. * feat(sqs): launch the shared SQS worker on its pinned node Add the SQS worker system-instance kind and a domain launcher that realizes the Region's worker from its identity record without ever starting a second VM: - domains/sqs/workerlaunch returns a bound worker as is, prepares the data volume on first launch only, refuses a volume whose own record is not encrypted, marks the intent dispatched before launching and never resends a dispatched launch: an ambiguous outcome stays launching until reconciled. A replacement starts only after a confirmed stop or failure and the first-store fence, reusing the pinned volume. - workerident gains MarkDispatched and Current; firststore gains a leader-read RecordedKey. - The daemon launches the worker through the shared direct-boot path with its encrypted volume attached at boot and kept on terminate, a required instance profile and a mandatory mgmt NIC. Its firmware config carries only public values: the data serial, the store decision, the gateway URL on this node's bridge and the daemon's own cluster CA. A relaunch is refused on any other node. - sqsWorkerRole is ensured in the system account granting exactly the two internal actions; an existing role whose trust admits more than EC2 is refused. The role contract lives in domains/sqs/workerrole so the composition root does not import the gateway adapter. The worker network allocator and any caller remain unwired, and the guest still refuses to create a key for a new store. * fix: verify SQS worker role trust before granting and report unclaimed volumes Check that an existing or racing-created sqsWorkerRole trusts only EC2 before writing its policy, and accept the instance profile only when a re-read shows it carries exactly that role. Refuse the first launch before allocating when the cluster storage key is not configured, and return the ID of any volume created but not confirmed in a launch intent instead of dropping it. * feat: add private SQS system VPC and worker ENI allocator Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet from the VPC and subnet surface only, so it cannot create an internet gateway, route table, NAT gateway or EIP. The daemon's SQS worker network adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and finds or creates one ENI per launch claim in the VPC default security group, refusing a duplicated claim or an address outside the subnet. Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have recorded the volume. * feat: worker takes its first-store decision from the gateway at boot The SQS worker guest no longer trusts a static fw_cfg store status. After selecting, checking and verifying the mount of its own data disk, it asks the internal spinifex-sqs gateway service for its current first-store decision, signed with instance-metadata credentials only, over TLS 1.3 trusting only the cluster CA. It then creates and records or loads the key on that disk; a key lost after recording is never replaced. - internal/sqsworkergw: SigV4 client for ClaimFirstStore and RecordWorkerPublicKey; refuses non-public keys, non-decisions and redirects; errors carry only the parsed code - gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback - workerboot: retries only lost replies and ServiceUnavailable/Throttling - launch contract carries the Region instead of a store status or key * fix: worker retries a denial while its launch is still binding it The worker VM can reach the gateway before its launch binds the instance identity, and that early call is legitimately AccessDenied. The guest treated the denial as final and stayed unprepared even after binding. AccessDenied is now a pending authorization the guest asks about again within its existing bound; a denied claim creates nothing, and a worker that is never bound, or was replaced, gives up at the bound. * feat(sqs): add operator EnsureSQSWorker action and node launch handler Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the internal spinifex operator service. It validates the node against the cluster and as a single subject token, then requests sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers and the worker stays pinned. The node handler re-checks the named node and instance type and runs one composed workerlaunch.Launcher (real volume preparer, role, private network and direct-boot launch), one launch at a time, in the daemon's own context with a 30-minute bound. A caller timeout does not cancel the launch; a retry reports the bound worker, or IncorrectState for a dispatched launch whose outcome is unknown, which is never resent. The reply names the bound VM and does not claim the worker or SQS is ready. Nothing starts a worker at daemon startup. * feat: ship the SQS worker image and check it before allocating Build the SQS worker microVM image in both release workflows, stage it into the tarball as microvm-sqs-worker, and install it with setup.sh under the store image's rules: a release without it removes an earlier one and an incomplete one is refused. Add make install-sqs-worker-microvm. The worker launcher now asks a Host capability whether this node can boot a worker before it prepares a volume, records or resumes an intent, fences a replaced generation, allocates the network or dispatches. The daemon checks for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a call after the image is installed launches instead of leaving an ambiguous dispatched intent. Cover missing and incomplete images through the admin action and handler, a stopped worker left untouched, and an ENI create whose reply is lost. * fix: never repeat an uncertain SQS worker ENI create The launcher now looks for the claim's ENI without creating one, marks the launch intent NetworkAttempted, and only then sends the single create. Once marked, a missing ENI fails closed with IncorrectState instead of a second create that could give the claim two interfaces; a visible ENI is reused. Adds a test that buildSQSWorkerLauncher wires the daemon's real host check. * feat(sqs): operator-triggered SQS store launch on the worker's node Add the super-admin EnsureSQSStore action and its node handler. The store is launched only on the node the Region's worker is bound to, with the public key that worker recorded in the first-store decision; the daemon never receives the worker's seed. A per-Region store launch record marks each step before its uncertain create (volume, ENI in the worker's private subnet, VM launch), so a step whose result is not visible fails closed and is never resent. Volumes are refused unless their own record is encrypted, and their IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and instance types at or below that are refused; this is a floor, not a validated size. * fix: retry uncreated SQS store volumes and fix the store instance type A volume preparation that failed before sending any create (a missing tool, scratch or format failure, or no cluster storage key) now carries storevolume.ErrNotCreated. The store launcher withdraws its volume mark and returns ServiceUnavailable, so a retry may try again once the host is fixed; any create that was sent stays one-shot. The store launch record now keeps the operator's instance type. A request for a different type is refused as IncorrectState before dispatch and after binding, and the reply names the launched type, which the admin action checks. * feat: SQS worker store readiness and store size check The worker's one-shot ready step mounts and verifies its disk, asks the internal spinifex-sqs DescribeStore action for the bound store, loads its key only if it matches the recorded public key, and checks the store's JetStream with it. It retries within a bound while the store is pending or not answering, and stops on a mismatched or refused key. The NKey authenticates the worker only; the store server is not authenticated. EnsureSQSStore now also refuses a different volume size once the store is bound, instead of returning the existing store. * fix(sqs): keep worker store readiness current until shutdown The readiness step now runs as a guest-local watch loop with no overall deadline: each pass is bounded, a pending or unreachable store is retried every few seconds, and a ready store is rechecked periodically. The store-ready marker is removed on any failed pass, cleared at start and removed on exit; shutdown stops the watcher by its recorded PID. Terminal refusals such as a rejected or mismatched key back off for minutes rather than spinning. The NKey still authenticates only the worker to the store. * feat(sqs): supervise the worker readiness watcher and add spx admin sqs Run the worker guest's readiness watcher under init respawn, rate-limited, and remove any stale store-ready marker before each start, so a killed or crashed watcher cannot leave a false readiness marker. Shutdown blocks respawn, removes the marker, stops the recorded watcher by PID and start time with SIGKILL escalation, and unmounts /data only once it has exited; a surviving watcher leaves every mount in place. Add spx admin sqs ensure-worker and ensure-store, which call the existing EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex gateway path with the standard AWS credential chain. Both refuse missing inputs before sending anything and report only the bound VM, never readiness. * fix: park SQS worker readiness and fix the worker instance type A readiness start that finds shutdown in progress or no worker key now logs once and waits in place instead of exiting, so init's respawn does not restart and log it in a loop; shutdown stops it like a watcher. The worker identity record keeps each generation's instance type. A request for another type is refused as IncorrectState before dispatch and after binding, and only a replacement may change it. The ensure reply, admin action and spx admin sqs ensure-worker report and check the recorded type. * fix: record the SQS worker volume intent before creating it The first worker launch prepared its data volume before any record existed, so a create that applied but returned no ID left nothing for the next call to find, and it could prepare a second volume. The launch intent is now recorded first, and its one volume create is marked before it is sent. A Prepare proven to have sent nothing withdraws the mark and is retryable; any other outcome is never resent and returns IncorrectState, naming the volume when its ID is known. An intent that has sent no create may still be re-typed. * docs: document spx admin sqs ensure-worker and ensure-store Record the flags, the caller and owning accounts, the bound-versus-ready distinction and the current retry and recovery limits. * docs: distinguish SQS ensure timeouts from refusals that created nothing A gateway timeout means only that no reply arrived; the launch may still be running. The worker type locks once any step has been attempted. * docs: state what each SQS ensure ServiceUnavailable establishes No refusal proves earlier attempts created nothing; a retry checks the durable attempt and continues only when that is safe.	2026-10-31 00:17:25 +10:00
Josh Siviour	488f1c7bf9	feat: SQS store guest on a prepared cold-attached EBS volume (not wired) (#1343) ... * feat: build a separate SQS store microVM image Package the go.mod-pinned nats-server, ext4 support and a clean ACPI power-down into a microVM kernel and initramfs of their own. The build checks the packed archive's contents and refuses a nats-server whose version differs from the pin. The image does not start the store and nothing installs or releases it; the LB image is unchanged. * fix: publish the SQS store image only after its gates pass Stage the kernel and initramfs and move them into place only once the content, size and guest-RAM gates pass, so a failed build cannot leave a pair make takes as up to date. The old initramfs is removed first and the new one lands last, so an interrupted publish never leaves a mixed pair. Check the module list without piping into grep -q, which under pipefail reported a present module as missing. * feat: start the SQS store guest on its checked data disk The SQS store image now prepares, runs and stops a dedicated JetStream server. It reads a narrow fw_cfg contract (data disk serial, listen address, one user NKey public key), refuses anything absent, malformed or ambiguous, has the pinned server test the config, then checks and mounts exactly one matching ext4 disk. It never formats and has no ephemeral fallback. The server listens only on the supplied guest NIC address, admits only that NKey and syncs every write. Shutdown stops the recorded run process first, whether it is the server or a start not yet exec'd, and runs the generic read-only remount only once it is gone. A store that survives SIGKILL leaves every mount alone. * fix: report the SQS store as prepared, not listening, before it starts * feat: launch the SQS store guest on its own versioned subject An explicit SQS store kind carries an existing system-account data volume and the one NATS user public key the store accepts. It is served only on system.LaunchSQSStore.v1, so a daemon without this code has no responder and the launch fails instead of booting a load balancer without its disk; the legacy launch path refuses the kind on both send and receive. Every refusal happens before capacity, an ENI or a VM record exists. The volume goes through the normal EBS mount and terminate detach and is never deleted on termination. The store boots its own image with CPU mitigations on, gets only its network, disk serial, listen address and key, and cannot be relaunched after a host restart yet. * feat: ship the SQS store microVM image in releases and installs Release builds now build the SQS store image and stage it beside the load balancer's as microvm-sqs-store/, so an older installer that copies microvm/* without -r still works. setup.sh installs it to /usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it. A release without a complete image never leaves an older one usable: staging drops a previously staged image, and the installer removes an earlier installed pair before warning (missing) or failing (incomplete). Only the two image files are removed. The initramfs is installed last, so an interrupted install leaves no bootable mismatched pair. * feat: restart the SQS store guest on its launching node Persist the non-secret guest config of an SQS store instance (launching node, data volume, listen address, client public NKey and rendered network config) in the instance record, and rebuild its fw_cfg files from it on relaunch. The relaunch refuses on any other node, or when the record is missing or malformed, before writing any file; the data volume is remounted through the normal EBS path and kept on terminate. Cross-node recovery now never selects or claims an SQS store record, so no other node spends retry budget on it or relaunches it. Same-node restore is unchanged, as are load balancer restarts. Older daemons know neither the exclusion nor the record field, so this does not prevent a cross-node relaunch by a mixed-version cluster. * feat: hold prepared system volumes in creating until released Add an internal CreateVolumeForPreparation path that writes a new volume's first record as creating, and CompleteVolumePreparation to release an unattached creating volume in the caller's account to available. Ordinary CreateVolume is unchanged. Attach and delete already refuse a creating volume. ModifyVolume, ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState so nothing can expand, release or capture a volume mid-preparation. * feat: prepare fresh SQS store volumes with an ext4 filesystem The SQS store guest never formats a disk, so a new volume gets its filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img, builds an ext4 image in private scratch, creates the volume in "creating", publishes it on this node, copies the image with the NBD import helper, unpublishes, and only then releases it as available. It never writes a volume it did not just create. A volume is deleted only when no export can still be live: after a confirmed unpublish, or when the export was not NBD. Any publish error, a failed or ambiguous unpublish, or a failed release retains the volume and reports its ID. A publish refused as in-use or already-exists sends no unpublish, since another export may exist. Cleanup ignores caller cancellation but is bounded at 30s. The raw-image copy moves to runtime/compute/nbd unchanged, and the domain receives it as an injected writer. The volume service gains a discard path for an unattached volume still in "creating". * fix: report unknown state and match qemu-img lookup in volume prep A failed release may still have made the volume available, so the retained-volume error now says its state is unknown rather than claiming it is still creating. The copy runs qemu-img by name through PATH, so the pre-allocation check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin fallback because it runs from the resolved path. * docs: describe the unwired SQS store system instance Records what the guest image, versioned launch, same-node restart and fresh-volume preparation provide, that nothing calls them, and what is untested or missing before production use.	2026-10-30 08:16:43 +10:00
Julian Sommer	d953d0a973	fix: treat Ubuntu's openvswitch.service alias as the Debian OVS unit ... Ubuntu 26.04's openvswitch-switch declares Alias=openvswitch.service, so enabling it leaves an /etc symlink that distro_ovs_unit read as a second, EL-style umbrella unit. setup-ovn.sh then refused to pick one and every ubuntu-26.04 bootstrap failed. A symlink back to openvswitch-switch.service now counts as the Debian unit; a real or masked openvswitch.service still does not.	2026-10-28 09:46:45 +10:00
Julian Sommer	7dde151766	feat: Oracle Linux 10 single-node host support (#1270)	2026-10-27 16:13:42 +11:00