mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-04 02:03:39 +00:00
Josh Siviour
d066786faa
* feat: persist the SQS worker's store key on its own disk The shared SQS worker generates its store-client NKey inside the guest and keeps it in an owner-only directory on its own persistent disk, so no platform component creates or unseals the seed. The key is synced and linked into place without replacing anything; a missing, mismatched, unreadable or loosely permissioned key fails closed instead of being regenerated. * fix: make the SQS worker key durably named before returning it Every successful call now syncs the key directory and its entry in the parent before handing out a key, so a retry after a failed directory sync cannot return a key whose name is not yet durable. Only the key directory is created, inside a parent that must exist, so an unmounted worker disk fails closed. The first-start decision is explicit: an existing store's authorized key is required and never defaults to generating a new one. * docs: state that workerkey cannot detect an unmounted worker disk An existing parent directory does not prove the worker disk is mounted, since the mountpoint can exist on the initramfs after a failed mount. The error now says only that the parent is missing, and the package documents that its caller must verify the expected data volume is mounted first. * feat: add the SQS worker guest bootstrap The shared SQS worker microVM boots with sqs-worker boot, which selects exactly the disk whose serial fw_cfg names, checks and mounts it at /data, and proves from the mount table that this disk is what is mounted before loading or creating its store key there. A missing, empty or unknown store status, a contradictory or invalid store key, a missing or ambiguous disk, a failed check or mount, an empty mountpoint or a foreign mount refuses without creating a key. Only the public key is logged and published. The image is built by the store image script under SQS_MICROVM_PROFILE=worker, with the worker binary, no nats-server and an empty /data mountpoint; the store image's contents are unchanged. * fix: refuse replayable new-store status at SQS worker boot A static fw_cfg store-status of "new" survives a guest reset, so after the store exists it could still reach key creation and replace a lost worker key. Boot now accepts only "existing" with the store's authorized public key and refuses "new" before touching any disk, until a current authoritative first-store decision exists. workerkey.ForNewStore remains, uncalled. * feat: add SQS first-store claim and worker key acquisition The SQS domain records one first-store decision per Region in its own KV bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound to the worker's instance ID, claim ID and generation. Key creation is authorized only while no public key is recorded; afterwards only the recorded key is returned, and a launch intent requires it. Older generations, same-generation rivals and unreadable records authorize nothing, and records hold no private key material. workerkey.Acquire syncs a new key to disk and returns it only after its public key is recorded, reusing the on-disk key after a crash before recording and failing closed on a missing or different key afterwards. No transport carries the decision yet. * fix: read SQS first-store decisions from the stream leader KV.Get uses a direct get, which a lagging replica can answer, so a stale claimed read could authorize key creation after the leader had recorded the worker's public key, or make a superseded generation look current. Reads now use the SQS store's leader-answered stream message get, and every authorizing answer is also confirmed by a compare-and-swap at the revision read, so a stale or unavailable read authorizes nothing. * feat(sqs): add regional worker identity record and instance index The SQS domain keeps one worker identity record per Region with a monotonic generation, claim ID, pinned node and encrypted data volume, moving from launching to bound or failed under compare-and-swap. A launch intent takes the next generation before any instance exists, so the previous worker stops being current; binding writes the instance index entry first. Authorize accepts an instance only while the Region's leader-read record is bound to exactly that instance, generation and claim, confirmed by a compare-and-swap at the read revision, so an old, orphaned, rewritten or unbound index entry, or a lagging replica, authorizes nothing. Both this bucket and the first-store bucket now open through the kvutil chokepoint and refuse a stored configuration that could expire, cap or evict records or holds fewer replicas than the cluster needs. Nothing calls the store yet. * fix(sqs): make worker launch exclusive and harden durable buckets BeginLaunch now refuses a new claim while the previous attempt is launching or bound, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume. A replacement takes the next generation only after FailLaunch or the new RecordStopped confirms that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never saw. The durable bucket opener also refuses non-limits retention, per-message TTLs, asynchronous flush, a message-size cap, any discard policy other than discard-new or a per-subject discard, unacknowledged writes, a sealed stream, and a mirror or sources. * feat: add internal SQS worker first-store gateway actions Serve ClaimFirstStore and RecordWorkerPublicKey under the internal spinifex-sqs signing service, apart from the public AmazonSQS inventory. Only an sqsWorkerRole session in the system account passes the class gate, then the role policy; the worker identity is resolved from the current workerident record by the STS-set session name, never from the request, and any extra parameter is refused. Seeds and other non-public keys are refused without being recorded, logged or echoed. The gateway binds the worker-identity and first-store buckets per request and never creates them, so it provisions nothing and answers ServiceUnavailable until a worker launch has created them. * fix(sqs): fence replaced worker generations out of first-store A request that passed worker authorization before a confirmed stop and replacement could reach an unclaimed first-store record ahead of the successor and create or record the old generation's key. The launch path now raises a monotonic first-store fence past the replaced generation before BeginLaunch, which refuses a replacement until a leader read shows it; firststore refuses every call from a generation below the fence. The internal adapter no longer logs caller-chosen parameter names. * feat(sqs): launch the shared SQS worker on its pinned node Add the SQS worker system-instance kind and a domain launcher that realizes the Region's worker from its identity record without ever starting a second VM: - domains/sqs/workerlaunch returns a bound worker as is, prepares the data volume on first launch only, refuses a volume whose own record is not encrypted, marks the intent dispatched before launching and never resends a dispatched launch: an ambiguous outcome stays launching until reconciled. A replacement starts only after a confirmed stop or failure and the first-store fence, reusing the pinned volume. - workerident gains MarkDispatched and Current; firststore gains a leader-read RecordedKey. - The daemon launches the worker through the shared direct-boot path with its encrypted volume attached at boot and kept on terminate, a required instance profile and a mandatory mgmt NIC. Its firmware config carries only public values: the data serial, the store decision, the gateway URL on this node's bridge and the daemon's own cluster CA. A relaunch is refused on any other node. - sqsWorkerRole is ensured in the system account granting exactly the two internal actions; an existing role whose trust admits more than EC2 is refused. The role contract lives in domains/sqs/workerrole so the composition root does not import the gateway adapter. The worker network allocator and any caller remain unwired, and the guest still refuses to create a key for a new store. * fix: verify SQS worker role trust before granting and report unclaimed volumes Check that an existing or racing-created sqsWorkerRole trusts only EC2 before writing its policy, and accept the instance profile only when a re-read shows it carries exactly that role. Refuse the first launch before allocating when the cluster storage key is not configured, and return the ID of any volume created but not confirmed in a launch intent instead of dropping it. * feat: add private SQS system VPC and worker ENI allocator Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet from the VPC and subnet surface only, so it cannot create an internet gateway, route table, NAT gateway or EIP. The daemon's SQS worker network adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and finds or creates one ENI per launch claim in the VPC default security group, refusing a duplicated claim or an address outside the subnet. Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have recorded the volume. * feat: worker takes its first-store decision from the gateway at boot The SQS worker guest no longer trusts a static fw_cfg store status. After selecting, checking and verifying the mount of its own data disk, it asks the internal spinifex-sqs gateway service for its current first-store decision, signed with instance-metadata credentials only, over TLS 1.3 trusting only the cluster CA. It then creates and records or loads the key on that disk; a key lost after recording is never replaced. - internal/sqsworkergw: SigV4 client for ClaimFirstStore and RecordWorkerPublicKey; refuses non-public keys, non-decisions and redirects; errors carry only the parsed code - gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback - workerboot: retries only lost replies and ServiceUnavailable/Throttling - launch contract carries the Region instead of a store status or key * fix: worker retries a denial while its launch is still binding it The worker VM can reach the gateway before its launch binds the instance identity, and that early call is legitimately AccessDenied. The guest treated the denial as final and stayed unprepared even after binding. AccessDenied is now a pending authorization the guest asks about again within its existing bound; a denied claim creates nothing, and a worker that is never bound, or was replaced, gives up at the bound. * feat(sqs): add operator EnsureSQSWorker action and node launch handler Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the internal spinifex operator service. It validates the node against the cluster and as a single subject token, then requests sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers and the worker stays pinned. The node handler re-checks the named node and instance type and runs one composed workerlaunch.Launcher (real volume preparer, role, private network and direct-boot launch), one launch at a time, in the daemon's own context with a 30-minute bound. A caller timeout does not cancel the launch; a retry reports the bound worker, or IncorrectState for a dispatched launch whose outcome is unknown, which is never resent. The reply names the bound VM and does not claim the worker or SQS is ready. Nothing starts a worker at daemon startup. * feat: ship the SQS worker image and check it before allocating Build the SQS worker microVM image in both release workflows, stage it into the tarball as microvm-sqs-worker, and install it with setup.sh under the store image's rules: a release without it removes an earlier one and an incomplete one is refused. Add make install-sqs-worker-microvm. The worker launcher now asks a Host capability whether this node can boot a worker before it prepares a volume, records or resumes an intent, fences a replaced generation, allocates the network or dispatches. The daemon checks for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a call after the image is installed launches instead of leaving an ambiguous dispatched intent. Cover missing and incomplete images through the admin action and handler, a stopped worker left untouched, and an ENI create whose reply is lost. * fix: never repeat an uncertain SQS worker ENI create The launcher now looks for the claim's ENI without creating one, marks the launch intent NetworkAttempted, and only then sends the single create. Once marked, a missing ENI fails closed with IncorrectState instead of a second create that could give the claim two interfaces; a visible ENI is reused. Adds a test that buildSQSWorkerLauncher wires the daemon's real host check. * feat(sqs): operator-triggered SQS store launch on the worker's node Add the super-admin EnsureSQSStore action and its node handler. The store is launched only on the node the Region's worker is bound to, with the public key that worker recorded in the first-store decision; the daemon never receives the worker's seed. A per-Region store launch record marks each step before its uncertain create (volume, ENI in the worker's private subnet, VM launch), so a step whose result is not visible fails closed and is never resent. Volumes are refused unless their own record is encrypted, and their IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and instance types at or below that are refused; this is a floor, not a validated size. * fix: retry uncreated SQS store volumes and fix the store instance type A volume preparation that failed before sending any create (a missing tool, scratch or format failure, or no cluster storage key) now carries storevolume.ErrNotCreated. The store launcher withdraws its volume mark and returns ServiceUnavailable, so a retry may try again once the host is fixed; any create that was sent stays one-shot. The store launch record now keeps the operator's instance type. A request for a different type is refused as IncorrectState before dispatch and after binding, and the reply names the launched type, which the admin action checks. * feat: SQS worker store readiness and store size check The worker's one-shot ready step mounts and verifies its disk, asks the internal spinifex-sqs DescribeStore action for the bound store, loads its key only if it matches the recorded public key, and checks the store's JetStream with it. It retries within a bound while the store is pending or not answering, and stops on a mismatched or refused key. The NKey authenticates the worker only; the store server is not authenticated. EnsureSQSStore now also refuses a different volume size once the store is bound, instead of returning the existing store. * fix(sqs): keep worker store readiness current until shutdown The readiness step now runs as a guest-local watch loop with no overall deadline: each pass is bounded, a pending or unreachable store is retried every few seconds, and a ready store is rechecked periodically. The store-ready marker is removed on any failed pass, cleared at start and removed on exit; shutdown stops the watcher by its recorded PID. Terminal refusals such as a rejected or mismatched key back off for minutes rather than spinning. The NKey still authenticates only the worker to the store. * feat(sqs): supervise the worker readiness watcher and add spx admin sqs Run the worker guest's readiness watcher under init respawn, rate-limited, and remove any stale store-ready marker before each start, so a killed or crashed watcher cannot leave a false readiness marker. Shutdown blocks respawn, removes the marker, stops the recorded watcher by PID and start time with SIGKILL escalation, and unmounts /data only once it has exited; a surviving watcher leaves every mount in place. Add spx admin sqs ensure-worker and ensure-store, which call the existing EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex gateway path with the standard AWS credential chain. Both refuse missing inputs before sending anything and report only the bound VM, never readiness. * fix: park SQS worker readiness and fix the worker instance type A readiness start that finds shutdown in progress or no worker key now logs once and waits in place instead of exiting, so init's respawn does not restart and log it in a loop; shutdown stops it like a watcher. The worker identity record keeps each generation's instance type. A request for another type is refused as IncorrectState before dispatch and after binding, and only a replacement may change it. The ensure reply, admin action and spx admin sqs ensure-worker report and check the recorded type. * fix: record the SQS worker volume intent before creating it The first worker launch prepared its data volume before any record existed, so a create that applied but returned no ID left nothing for the next call to find, and it could prepare a second volume. The launch intent is now recorded first, and its one volume create is marked before it is sent. A Prepare proven to have sent nothing withdraws the mark and is retryable; any other outcome is never resent and returns IncorrectState, naming the volume when its ID is known. An intent that has sent no create may still be re-typed. * docs: document spx admin sqs ensure-worker and ensure-store Record the flags, the caller and owning accounts, the bound-versus-ready distinction and the current retry and recovery limits. * docs: distinguish SQS ensure timeouts from refusals that created nothing A gateway timeout means only that no reply arrived; the launch may still be running. The worker type locks once any step has been attempted. * docs: state what each SQS ensure ServiceUnavailable establishes No refusal proves earlier attempts created nothing; a retry checks the durable attempt and continues only when that is safe.