482 Commits

Author	SHA1	Message	Date
Josh Siviour	671a8a9ff5	feat(sqs): fresh single-node SQS in one system microVM (#1389) ... * feat(sqs): run queue and message owners in the worker by signed pull The SQS worker's ready step now opens the queue and message owners on its dedicated store and serves owner calls it pulls from its node's gateway over the existing instance-role channel (PollWork, CompleteWork). Each poll and post is authorized against the current worker identity, and a result is accepted only from the identity that took the work. No payload crosses platform NATS. A receipt-signing key is created on the worker disk only under the first-store decision that creates the store key, and its fingerprint is bound in the store; a missing or changed key refuses to serve. Received messages are released only when the gateway proves the caller left; receives are capped so a worst-case result fits one post. The public relay is not registered in production. * fix: harden SQS relay results and bound pending bytes Unavailable outranks client faults when encoding owner errors, and wire errors carry fixed text only, never owner or backend text. The ingress rejects incomplete or mismatched success results as unavailable, and the relay bounds total pending input bytes as well as request count. Adds tests of every relay action against real owners, the receive claim limit, release, faulting work, the receipt-key fingerprint marker and the poll loop through gateway failures. * fix: refuse SQS serving outside a single-node host cluster DescribeStore now reports the gateway's declared cluster node count, and the worker refuses to serve, before reading keys, connecting or creating any store state, unless that count is exactly one. A gateway that reports no count reads as zero and is refused. * feat(sqs): record a store server key pin once per store instance Add the store role and its internal actions so the bound store instance can learn its claim and record the public pin of its TLS server key, once, without the platform holding the private key. The worker sees the store as bound only after a pin exists and must receive a valid pin. * refactor(sqs): serve store server keys from a separate capability Keep the worker Stores interface unchanged and give the store guest's actions their own ServerKeys dependency, so existing Stores fakes need no store-key methods. * feat(sqs): launch the store with a mgmt NIC, CA and scoped store role The store now boots with the region, gateway URL and the daemon's own cluster CA, a mgmt NIC to reach the gateway, and an sqsStoreRole profile granted only the store's two internal actions, so it can authenticate its server key pin. A store record that predates these fields refuses to relaunch; its volume is kept. * feat(sqs): keep the store TLS server key on its disk and record its pin The store image gains sqs-store-key, run after the data disk is mounted. It loads or creates a P-256 server key under the disk's tls directory, records only the key's public pin with the gateway as the bound store instance, and writes a self-signed certificate for it to the run directory. nats-server then requires TLS 1.3 with that key. A record for another volume, a recorded pin with a missing key, or a key that does not match the recorded pin refuses the store; no key is regenerated and nothing on the disk is deleted. Launch-not-yet-bound and lost replies are retried. The store role vocabulary moves to contracts/sqs/v1 so the guest agent does not import a domain. * feat(sqs): pin the worker's store connection to the recorded server key The worker's readiness check and serving connection now use TLS 1.3 and accept only the store server key whose pin the bound store recorded with the gateway. The pin is checked before the worker key or any queue or message data is sent, and on every reconnect. A store with another key, a plaintext store or a malformed pin is refused as final. The worker NKey client authentication is unchanged. The host still holds the store's data disk key, so the pin does not protect the store from a privileged host. * fix(sqs): retry store key bootstrap and track guest binary sources The store's server key check moves from sysinit into the respawned run step: a gateway or IMDS outage exits after a pause so init retries it, without remounting or creating a new key, while a missing, changed or wrong-store key is recorded as refused and never retried. Stop also ends a running key helper before unmounting. The guest helper and worker binaries now depend on every main-module package dir and file they link, from go list, so edits, additions, deletions and renames all rebuild them. * feat(sqs): operator enable and status with a signed ingress test Add a per-Region SQS service record (disabled by default, pinned to one node, refused on multi-node clusters before any write), gate the relay on it, and expose `spx admin sqs enable` and `spx admin sqs status`. The gateway now hands queued work to the polling worker and reports whether one is polling. The public SQS service stays unregistered in production until its capability profile is reviewed. An integration test registers it through the harness and drives real SigV4 and IAM requests to a real worker for create, send, receive and delete, asserting no SQS body crosses platform NATS. * fix(sqs): report readiness only from observed component state Status now reports whether the tenant SQS API is registered on this gateway and is never Ready while it is not. The worker stops polling while its store connection is down, so a worker that lost its store drops out of the gateway's poller window instead of looking healthy. * wip: sqs model inventory * wip: sqs public registration * wip: sqs registration review fixes * fix(sqs): refuse pre-API store and worker records on enable A store or worker record this release creates now carries an explicit contract. enable reads both records without creating a bucket and refuses any record without it before it writes desired state or launches anything; status names the record and is never Ready. A worker replacement keeps its predecessor's contract, since it reuses the same data volume. Missing metadata is only ever grounds to refuse: nothing is upgraded, inspected, emptied, replaced or deleted. Upgrading a pre-API dev install is not supported. * fix(sqs): describe the enable refusal as an unsupported launch contract A missing or unknown contract proves neither when a record was written nor that it predates authenticated bootstrap: an older writer can drop a newer field, and a future contract is not pre-API. The refusal is unchanged; its message, status field and docs now name a missing or unsupported launch contract and say any existing volume is kept. * wip: port reviewed SQS serving path onto dev without the store VM * wip: single SQS system VM * fix(sqs): retry receipt bootstrap under supervision and drop store install path Run the receipt-key boot step from the respawned store service so a transient gateway or IMDS outage recovers without a reboot; final key and volume refusals stay refused. Remove the separate store image install and staging path from setup. * fix(sqs): align worker ready config fields * fix(sqs): stop an in-progress boot before unmounting the worker disk The receipt-key boot step ran as an unrecorded child of the store shell, so shutdown stopped only the shell and could unmount the disk under a boot still talking to the gateway. Record the boot child like the other guest processes and stop it before unmounting; a survivor keeps the disk mounted. * fix(sqs): record the boot child before its stopping check The parent wrote boot.pid only after starting the child, so a shutdown in that gap could unmount under an unrecorded boot. The child now records itself and checks stopping before it runs sqs-worker boot.	2026-11-02 09:02:06 +10:00
dependabot[bot]	bbd4e84f15	chore: bump the docker-images group across 3 directories with 3 updates (#1375) ... Bumps the docker-images group with 1 update in the /build directory: golang. Bumps the docker-images group with 2 updates in the /docs/terraform-workbooks/demo-app directory: golang and distroless/static. Bumps the docker-images group with 1 update in the /scripts directory: debian. Updates `golang` from `972b411` to `d9b6a1b` Updates `golang` from `f2c872b` to `d200d2d` Updates `distroless/static` from `18f4cc2` to `25ac8f0` Updates `debian` from `62bd423` to `f6b08b0` --- updated-dependencies: - dependency-name: golang dependency-version: 1.27.3-bookworm dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: golang dependency-version: 1.27.3-alpine dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: distroless/static dependency-version: nonroot dependency-type: direct:production dependency-group: docker-images - dependency-name: debian dependency-version: trixie-slim dependency-type: direct:production dependency-group: docker-images ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 09:05:45 +11:00
Josh Siviour	30b8c24776	feat(accountlifecycle): seal quota overrides during account teardown (#1390) ... * feat(admission): add the quota override seal capability contract * feat(awsgw): serve the quota override seal capability per node * refactor(accountlifecycle): parameterise the seal capability probe and add the quota one * test(accountlifecycle): pin the probe wire shape against both capability contracts * feat(accountlifecycle): seal the quota override record during account teardown * docs: record the quota override seal contract and teardown behaviour * docs: fill in the quota override seal PR number in the package boundary migration	2026-11-02 06:59:06 +10:00
Ben Duncan	6c3d6c6be3	fix(setup): keep NetworkManager from replacing the resolved stub link (#1391) ... NetworkManager restarted while /etc/resolv.conf is a plain file runs in rc-manager=symlink mode and writes resolv.conf on its first DNS update, which can land after the installer links the stub. Write a conf.d drop-in (dns=systemd-resolved, rc-manager=unmanaged) before the restart, refuse a later file that overrides rc-manager, wait for NetworkManager to settle, and assert the link holds with bounded relinks. A rerun after an older handover adds the drop-in, reloads NetworkManager and relinks the stub.	2026-11-01 10:30:43 -06:00
Josh Siviour	2466a9c8af	fix: stamp the ENI owner in the attach write and release a failed LB's VM (#1388) ... * test: pin the ENI owner stamp and failed-LB cleanup contracts Failing tests: an ENI update after the caller's own write must survive a lagging replica read, an unwritable owner stamp must fail the system launch, and the reaper must terminate the failed LB's own VM only. * fix: stamp the ENI owner in the attach write and release a failed LB's VM A lagging-replica read before the owner stamp's compare-and-swap lost the stamp, leaving a system VM that IMDS could never resolve. - AttachENIWithOwner records the instance owner in the attach write. - ENI read-modify-writes re-read and re-apply on a revision conflict, bounded, and report the store error to the caller's log. - A system launch whose ENI cannot be attached or stamped fails and cleans up the VM. - The reaper that marks an LB failed terminates that LB's own VM and detaches its ENIs. * fix: log ENI write retries and exhaustion An Info line on a write that succeeds after a revision conflict and an Error line when the bounded retries run out, so the sink can count both. * test: cover an internal ALB in a private subnet and report the ENI attachment owner DescribeNetworkInterfaces now returns Attachment.InstanceOwnerId when the record has one, as AWS does, so the owner stamp is observable. The e2e launches an internal ALB in a VPC with no internet gateway and checks the owner, the private address and in-VPC traffic. The ENI write error log now reports the real attempt count, so a non-conflict store error is not counted as an exhausted retry. * revert: stop reporting the ENI attachment owner through Describe AWS always returns Attachment.InstanceOwnerId, so reporting it only for system-owned ENIs adds a divergence. The e2e now relies on the LB reaching active with a healthy target, which needs the owner stamp to resolve.	2026-11-01 22:39:34 +10:00
Josh Siviour	f37776384c	docs: fill in the 11d-2 PR number in the package boundary migration (#1386)	2026-11-01 20:53:26 +10:00
Josh Siviour	ea1ca38c52	fix: boot LB and SQS worker microvms with PCIe NICs (#1385) ... * test: direct-boot LB and SQS worker NICs must be virtio-net-pci on microvm pcie=on * fix: boot LB and SQS worker microvms on PCIe with virtio-net-pci NICs virtio-mmio devices share level-triggered IRQ lines and a NIC's RX path stalled on them. The direct-boot machine gets pcie=on, so NICs are virtio-net-pci with MSI-X, and the virtio_mmio.device= kernel args go. IsMMIO is now the transport check (false with pcie=on) and IsMicrovm the machine check. Data volumes keep their serial and ID on virtio-blk-pci. The LB guest console loglevel is raised to 7. * refactor: trim microvmMachineType comment * docs: describe direct-boot microVM NICs and data volumes on PCIe	2026-11-01 20:51:40 +10:00
Josh Siviour	8add7c03b3	refactor: move the gateway into ingress/aws/pipeline and delete it (11d-2) (#1384) ... * refactor: move the gateway into ingress/aws/pipeline and retire the gateway layering rows The gateway's request pipeline joins the packages it already consumed. Operation inventory moves to ingress/aws/dispatch. Files keep their names; only the package line, the import paths and the now-redundant pipeline qualifier change. The layering rows for the gateway root are removed: the package is under ingress, so R2 binds it, and the daemon-direction test now asserts R2 rejects runtime, operator and domain imports. An empty registry_helper_test.go was dropped rather than moved. * refactor: rename pipeline.GatewayConfig to pipeline.Config * docs: update the design and package boundary docs for the gateway move * docs: point the EKS add-on lifecycle evidence at the tests' current files	2026-11-01 20:42:27 +10:00
Ben Duncan	34f7b4d8e5	fix(vm): reboot by relaunching QEMU after a clean guest shutdown (#1383) ... RebootInstances powered the guest down under -action shutdown=pause and then ran system_reset + cont in the same QEMU process. On some hosts the reset left the guest silent in firmware with QMP still answering, while a fresh QEMU on the same volumes always booted. A reboot now shuts the guest down over ACPI (forcing it off after 60s, as before), ends that QEMU, and starts a fresh one on the same mounted volumes, taps, IMDS datapath and identity, the way nova's soft reboot and Proxmox's qm reboot do. The instance stays running in Describe, no hooks fire, the console log is appended rather than truncated, and the existing QMP client is reconnected in place so the heartbeat carries on. A relaunched guest that writes nothing to its console within 60s has its run state, vCPUs, vCPU thread states and registers (sampled twice) logged, and if it has printed before, its QEMU is replaced once more. A relaunch that cannot start goes to crash recovery. Stop, terminate and hot-plug wait for a relaunch in flight so volumes are never sealed under a process being started. Guest-initiated reboots stay an in-process reset. The daemon now logs the host kernel, QEMU version and the OVMF package and checksum at start, and every reboot relaunch carries them.	2026-11-01 02:48:00 -06:00
Ben Duncan	956b830f18	fix(firewall): keep conntrack running while the host policy is disarmed (#1379) ... The kernel tracks connections only while some rule uses conntrack. With inet spinifex_filter unloaded nothing tracked, so OVN RAFT links formed while disarmed (or before the daemon's first arm) were first seen mid-stream at arm time. With tcp_loose a packet from the server end became a new flow from peer:6643/6644 to an ephemeral port and hit the drop rule, and RAFT lost its leader. Add table inet spinifex_conntrack (accept-only) and keep it loaded on every node: setup.sh loads it at install, the boot unit loads it unconditionally, firstboot loads it before OVN, and disable swaps the policy for it in one transaction instead of turning conntrack off.	2026-11-01 01:49:12 -05:00
Josh Siviour	f0a704cfcd	refactor: move the gateway's public routes, admin wiring and unread config out (11d-1) (#1378) ... * test: pin the order of the unauthenticated mounts against SigV4 and the throttle * refactor: mount the OIDC and registry routes through a public-routes hook * refactor: take the admin registration name and request-id context as gateway config * refactor: move node discovery out of the gateway into the awsgw role * refactor: drop the gateway config fields production never reads (Config, RootCAs, InternalSuffix) * refactor: drop the gateway Version and Commit config fields * refactor: compose the production gateway config in one tested function * refactor: drop the gateway IAMService, STSService, NATSConn and ExpectedNodes config fields Production never reads them: composeGateway builds Policy and Readiness from its own inputs. Tests that used the config as a carrier now keep those collaborators in gwfix.Deps, recorded per config with gwfix.WithDeps and read back with gwfix.DepsOf. Gateway non-test code now imports nothing under domains/, runtime/ or operator/. Test edits are setup only: 153 GatewayConfig literals in 45 files had the four keys moved into gwfix.WithDeps(..., gwfix.Deps{...}), and 113 field reads became gwfix.DepsOf(x).{IAM,STS,NATS,Nodes}. No assertion changed. Script 1, literal rewrite (balanced-brace scanner, keys moved out of the literal): import re,sys,subprocess,os KEYS={'IAMService':'IAM','STSService':'STS','NATSConn':'NATS','ExpectedNodes':'Nodes'} def scan_close(s,i): # s[i]=='{' ; return index of matching '}' ; skips strings, runes, comments d=0;n=len(s);k=i while k<n: c=s[k] if c=='"': k+=1 while s[k]!='"': k+= 2 if s[k]=='\\' else 1 elif c=='`': k=s.index('`',k+1) elif c=="'": k+=1 while s[k]!="'": k+= 2 if s[k]=='\\' else 1 elif s.startswith('//',k): k=s.index('\n',k) continue elif s.startswith('/*',k): k=s.index('*/',k)+1 elif c in '{([': d+=1 elif c in '})]': d-=1 if d==0: return k k+=1 raise Exception('unbalanced') def split_top(body): parts=[];d=0;start=0;k=0;n=len(body) while k<n: c=body[k] if c=='"': k+=1 while body[k]!='"': k+= 2 if body[k]=='\\' else 1 elif c=='`': k=body.index('`',k+1) elif c=="'": k+=1 while body[k]!="'": k+= 2 if body[k]=='\\' else 1 elif body.startswith('//',k): k=body.index('\n',k); continue elif body.startswith('/*',k): k=body.index('*/',k)+1 elif c in '{([': d+=1 elif c in '})]': d-=1 elif c==',' and d==0: parts.append(body[start:k]); start=k+1 k+=1 parts.append(body[start:]) return parts changed={} files=[l for l in subprocess.check_output(['git','ls-files','*.go']).decode().split() if os.path.exists(l)] SKIP={'spinifex/gateway/gateway.go','spinifex/runtime/roles/awsgw/compose.go','spinifex/runtime/roles/awsgw/awsgw.go'} lit=re.compile(r'(&?)((?:gateway\.)?GatewayConfig)\{') total=0 for f in files: if f in SKIP: continue s=open(f).read() if 'GatewayConfig{' not in s: continue out=[];pos=0;cnt=0 for m in lit.finditer(s): if m.start()<pos: continue o=m.end()-1 c=scan_close(s,o) body=s[o+1:c] parts=split_top(body) keep=[];moved=[] for p in parts: km=re.match(r'\s*(IAMService|STSService|NATSConn|ExpectedNodes)\s*:\s*(.*)$',p,re.S) if km: moved.append((KEYS[km.group(1)],km.group(2).strip())) else: keep.append(p) if not moved: continue # rebuild kb=','.join(keep) if kb.strip()=='' : kb='' elif not kb.rstrip().endswith('\n') and '\n' in body: pass d=', '.join(f'{k}: {v}' for k,v in moved) new=f'gwfix.WithDeps({m.group(1)}{m.group(2)}{{{kb}}}, gwfix.Deps{{{d}}})' if not m.group(1): new=f'gwfix.WithDeps(&{m.group(2)}{{{kb}}}, gwfix.Deps{{{d}}})' # value literals flagged below out.append((m.start(),c+1,new)); pos=c+1; cnt+=1 if not m.group(1): print('VALUE LITERAL',f,s.count('\n',0,m.start())+1) if not out: continue r=[];last=0 for a,b,n in out: r.append(s[last:a]);r.append(n);last=b r.append(s[last:]) s2=''.join(r) if 'testkit/gwfix"' not in s2: s2=re.sub(r'import \(\n','import (\n\t"github.com/mulgadc/spinifex/internal/testkit/gwfix"\n',s2,1) open(f,'w').write(s2); total+=cnt; changed[f]=cnt print('literals rewritten',total,'files',len(changed)) Script 2, read rewrite (driven by the compiler's "has no field or method" errors for the type, so only GatewayConfig receivers are touched): import re,subprocess,sys,collections M={'IAMService':'IAM','STSService':'STS','NATSConn':'NATS','ExpectedNodes':'Nodes'} def run(): env=dict(__import__('os').environ,GOWORK='off') out='' for tags,pk in [('',['./spinifex/gateway','./spinifex/runtime/roles/awsgw/...']),('integration',['./tests/integration/...'])]: cmd=['go','test','-count=1','-run','NONE_X','-gcflags=-e']+(['-tags',tags] if tags else [])+pk r=subprocess.run(cmd,capture_output=True,text=True,env=env) out+=r.stdout+r.stderr return out fixed=0 for it in range(8): out=run() errs=re.findall(r'^(\S+\.go):(\d+):(\d+): (.*?) undefined \(type (\*?(?:gateway\.)?GatewayConfig) has no field or method (\w+)\)',out,re.M) if not errs: print(out[-3000:]); break byfile=collections.defaultdict(list) for f,l,c,expr,t,fld in errs: byfile[f].append((int(l),int(c),expr,fld)) for f,es in byfile.items(): lines=open(f).read().split('\n') for l,c,expr,fld in sorted(set(es),reverse=True): line=lines[l-1] # expr like 'gw.IAMService' or 'x.y.IAMService' new=f'gwfix.DepsOf({expr[:-len(fld)-1]}).{M[fld]}' if expr not in line: print('MISS',f,l,expr); continue lines[l-1]=line.replace(expr,new) fixed+=1 s='\n'.join(lines) if 'testkit/gwfix"' not in s: s=re.sub(r'import \(\n','import (\n\t"github.com/mulgadc/spinifex/internal/testkit/gwfix"\n',s,count=1) open(f,'w').write(s) print('read sites fixed',fixed) * docs: record the gateway mount hooks and config-field removal in the package boundary migration * docs: fill in the 11d-1 PR number in the package boundary migration	2026-11-01 13:04:31 +10:00
Josh Siviour	1cfaf15c9d	fix: EKS control-plane VMs are not Kubernetes Nodes (#1373) ... * test: pin control-plane no-agent config, etcd member removal and CP isolation e2e * test: tighten CP isolation e2e and assert a logged prune failure * fix: run EKS control planes without a k3s agent and prune etcd members directly Control-plane VMs no longer register as Nodes, so pods cannot schedule onto a network that cannot reach the workers or the advertised endpoint. With no Node to delete, a replaced peer's etcd member is removed by a new eks-etcd-member helper that matches exactly one member by peer-URL host. * fix: refuse to remove the local etcd member by ID, not only by IP Self-protection no longer depends on first boot deriving its own address: the helper asks the connected endpoint for its member ID and refuses a match on it. * test(e2e): align EKS suite with control planes that are not Nodes Assert zero Nodes and delivered addon objects before any nodegroup. Check Ready worker, metrics APIService, addon deletion and pod logs once a worker exists, scoping logs to Ready nodes and waiting 10m for metrics-server to reschedule. * docs: EKS control-plane hosts are not Kubernetes Nodes Document that kubectl get nodes lists workers only, add upgrade notes for existing clusters, and list the eks-etcd-member helper. * docs: explain why addon manifests keep the CriticalAddonsOnly toleration Comment-only. The toleration stays for clusters created before control-plane hosts stopped registering as Nodes. * test: cover the etcd member client and CLI against a TLS fake etcd The real etcd client and the command had no tests, leaving the PR below the diff coverage threshold. A fake etcd serving the Cluster and Maintenance APIs over mutual TLS drives the client with the K3s certificate layout, and main becomes a testable run function.	2026-11-01 10:55:57 +10:00
Ben Duncan	e3a44893fa	fix: bound the console's instance-type payload and stop polling it (#1371) ... * fix: bound the console's instance-type payload and stop polling it DescribeInstanceTypes with capacity=true returned one entry per free slot per node: 3,528,765 bytes of XML for 50 distinct types on prod, refetched every five seconds. The response was sized by free capacity, so the emptier the cluster the larger it got, and at 44,346 slots it exceeded the NATS maximum payload and the console reported no compute at all. The listing now names each admissible type once. The free-slot figure crosses as a number through a new GetInstanceTypeCapacity action, open to any account because the launch form a tenant uses is what needs it. The catalogue query no longer polls, and the EKS and ECS dialogs read it only when open. The gateway negotiates gzip inside the SigV4 group, which excludes the OCI Distribution registry structurally. A response is compressed only if the client offered gzip, the handler declared neither a length nor an encoding, the content type is neither streaming nor already compressed, the status carries a body, and 1 KiB was written before any flush. An unknown capacity figure no longer caps the count at 1. The launch decides, and RunInstances answers InsufficientInstanceCapacity if it cannot. * test: read the free slot count from its own action in the console e2e Two e2e tests encoded capacity as repeated instance-type entries, so one expected a caption the deduplicated listing can no longer produce. The fixture answers GetInstanceTypeCapacity instead, which lets a test give the catalogue and the figure different answers. A third test covers the case the change introduces: a session whose policy does not grant the action reads no figure, so the form offers the type with no capacity caption and leaves the launch enabled. The committed bundle is rebuilt, which rehashes every chunk reaching the two query modules.	2026-10-31 14:03:19 -05:00
Ben Duncan	9c99312f74	feat(ui): enforce ADR-0016 with guard tests and move dialog forms into FormPanel (#1369)	2026-10-31 13:41:53 -05:00
Ben Duncan	4468f13e53	docs: add a diagram of a Vultr deployment to the guide (#1367) ... Shows deploy-vultr.sh building a private Vultr VPC with one node or three, the nodes clustering over the private network for the control plane, EBS and S3, Spinifex calling the Vultr API for reserved IPs, and the AWS services running across the fleet.	2026-10-31 08:36:10 -05:00
Ben Duncan	8a71854463	docs: tidy the Vultr guide for publication (#1365) ... Drops the horizontal rules the docs site rendered as stray text, links the AWS CLI install page, captions each screenshot, shows the expected spx get nodes output for three nodes, explains what a three-node deployment protects against, and simplifies the limits to the sizing and quorum a reader needs.	2026-10-31 08:20:53 -05:00
Josh Siviour	e155bcf155	feat(sqs): queue and Lambda-facing message API foundation (#1364) ... * feat(sqs): queue and Lambda-facing message API foundation Adds the SQS JSON 1.0 adapter (unregistered in production) with queue create, lookup, attributes, list, tags and the message operations a Lambda poller uses: SendMessage(Batch), ReceiveMessage with long polling and attribute selection, DeleteMessage(Batch) and ChangeMessageVisibility(Batch), backed by the queue record owner and the JetStream message store. String and Binary message attributes are validated, stored and checksummed; Number attributes are refused until AWS behaviour is confirmed. * test(sqs): poller lifecycle through the message owners Covers receive, release, redelivery, delete, visibility extension, queue binding of receipt handles and long polling against the real queue and JetStream message stores, from the message package itself. * test(sqs): move SDK coverage from the gateway to the SQS domain Drives the registration through dispatch.Invocation over the real queue and message owners, so the gateway package carries no SQS changes.	2026-10-31 22:27:11 +10:00
Josh Siviour	7aa76c5bd6	refactor: gateway request record and capabilities (ADR-0002:S7, S8) (#1363) ... * test: pin gateway pipeline order, request ids and region * refactor: carry request facts in one ingress request record * fix: seal the request principal once * refactor: build the caller ARN once in credverify * refactor: gateway takes policy and admission as capabilities * refactor: gateway takes cluster readiness as a capability * chore: gofmt GatewayConfig field alignment * docs: record slice 11a request record and capabilities * docs: add PR number to slice 11a migration row * refactor: move pipeline capability interfaces to ingress/aws/pipeline	2026-10-31 17:57:15 +10:00
Ben Duncan	b4dd606104	feat: run Spinifex on Vultr, with failover, workbooks and the fixes they needed (#1361) ... * feat: Vultr reserved-IP provider and floating external-address pool Adds a source="vultr" external pool backed by Vultr reserved IPs: an API client pinned by captured responses, an in-memory fake with the measured semantics, and a floating allocator that creates, claims, reconciles and releases labelled addresses. OCI and Vultr claims now share one AddressClaimer pass. Also fixes the host firewall dropping guest IMDS when imds_host_meta_ip is set, by matching the pre-DNAT destination, and arms the Spinifex firewall in place of an active ufw. * fix: detach a Vultr reserved IP before deleting it Vultr was measured refusing delete of an attached address with 400 "Unable to detach subnet", so release, rollback and reconcile now detach first. A first claim after allocation logs at info rather than warn. * fix: release the auto-assigned public IP an Elastic IP displaces AssociateAddress restamped the ENI with the EIP and never handed back the address launch had assigned. Nothing referred to it afterwards, so it stayed allocated for good, and on a cloud pool every associate left a billed address. AWS releases it on associate; withdraw its NAT rule and release it, scoped to the ENI, leaving any address another associated EIP holds alone. * fix: retry Vultr's refused detach on reserved-IP detach and delete Vultr has answered a detach with 400 "Unable to detach subnet" and accepted the same detach moments later. Surfacing it left the address for reconcile to collect, up to twenty billed minutes later; retry it like a 5xx instead. * fix: stop reporting the released launch address after an EIP is disassociated Associating an EIP now releases the auto-assigned public IP, but the stored instance kept its launch stamp, and DescribeInstances fell back to it once the EIP was disassociated. The public-IP change handler now moves the stamp with the runtime address. * feat: add a Vultr deploy script for a single-node Spinifex on a VX VM deploy.sh up/destroy/status drives Terraform for the VPC and VM, installs and forms Spinifex over SSH, copies the API key to the node rather than through user-data, and sweeps boot volumes and reserved IPs Terraform does not own. Cloud-init keeps needrestart off the Spinifex, OVS and OVN units and disables unattended upgrades on these disposable hosts. * feat: form a three-node Vultr deployment with install-node.sh over the VPC Each node comes up as an armed single node, the state install-node.sh forms from, with the cluster and Geneve planes on the VPC NIC. The pool and every node's provider_host_id go on each node after formation. * fix: read the pinned host key with ssh-keygen -F when forming a Vultr cluster ssh_config hashes known_hosts entries by default on Debian, so a grep for the address found nothing and set -e ended the run without a message. * docs: add the Vultr guide, one command to deploy one or three nodes and one to remove them * docs: drop the failover write note from the Vultr guide * fix: keep needrestart from restarting the Spinifex, OVS and OVN units Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's replaced libraries to the daemon and restarted it after every dpkg run. The restart never cleared them, so an unattended upgrade ran the unit into its start limit and left the control plane down. setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists them, so restarting after an upgrade stays the operator's choice. * chore: drop the Vultr cloud-init needrestart override now setup.sh writes it * docs: replace the Vultr guide's apt upgrade warning now the installer keeps needrestart off the stack * docs: install the Vultr guide's build from the dev channel until Vultr support is released * fix: answer ARP for Vultr reserved IPs on bare metal Vultr routes a reserved IP to a VM by MAC but ARPs for it on a bare-metal server's public segment, and a Vultr pool gave vpcd neither a gateway nor a bind bridge to proxy-ARP on, so nothing answered and no traffic arrived. A Vultr pool now proxy-ARPs on the interface the node routes the internet through. setup.sh installs arping, which the gratuitous ARP already called. * feat: deploy Spinifex on Vultr bare metal, renamed deploy-vultr.sh A vbm- plan creates bare-metal servers, one or three. Without --name the deployment writes the AWS profile spinifex, up prints every node's address and the API key allowlist, credentials and ssh subcommands are added, no AMI is imported unless --image is given, and OpenTofu or Terraform is used. The host key is pinned only once cloud-init is done, since bare metal first answers SSH from Vultr's installer with a different key. * docs: rework the Vultr guide into the OCI guide's flow, with bare metal * fix: restamp LaunchTime when a stopped or recovered instance relaunches The pending watchdog times a launch from LaunchTime, which still held the original launch. An instance recovered from a failed node, or started more than five minutes after it was first launched, was terminated as stuck in pending seconds after QEMU started. AWS restamps LaunchTime on every start. * feat: spread the nginx-alb workbook across nodes, and power Vultr nodes off and on from deploy-vultr.sh The workbook takes server_count and placement_strategy, so three web servers can run one per node. Its README now matches the template (public subnets, no NAT gateway) and the diagram shows the spread. deploy-vultr.sh gains stop and start, which halt and start a node through the Vultr API. * fix: let a terminate finish tearing down before the reaper calls it wedged OrphanQEMUReaper finalized any shutting-down instance whose QEMU had exited, on the premise that a live terminate still has a live QEMU. A terminate is still deleting volumes, releasing its IP and leaving its placement group for tens of seconds after QEMU exits, so the instance was reported terminated first. Terraform then deleted the placement group and got InvalidPlacementGroup.InUse. The reconcile now waits two minutes from shutting-down, well inside the ten-minute stuck-terminate backstop. * fix: release an ovs-vswitchd left in flow-restore-wait after a node boots ovn-controller clears flow-restore-wait once its flows are installed and sets it again straight after. When both writes reach ovs-vswitchd as one update, it only sees true, starts no upcall threads and forwards nothing. Seen on Vultr bare metal after a hard power-off: instances recovered onto the rebooted node ran but were unreachable. vpcd now clears the flag on any node held that way for a minute, longer than ovs-ctl's own restore takes. * docs: add a resiliency test to the Vultr guide and the nginx-alb workbook The Vultr guide gains step 8, which spreads three web servers across a three-node deployment, powers a node off and back on, and shows what was measured on VMs and bare metal. The workbook gains the same test as step 6. The guide's table of contents is repaired, and its limits now cover losing two nodes and recovery inside a spread placement group. * fix: never terminate an instance that has run because a relaunch failed The pending watchdog terminated any instance pending past five minutes, and terminating deletes DeleteOnTermination volumes. A start of a stopped instance, a recovery from a failed node and a daemon relaunch all pass through pending, so a slow one destroyed an existing instance and its root disk. On Vultr a recovered instance was terminated seconds after QEMU started and its root volume deleted. An instance is now marked FirstLaunch from RunInstances until it first reaches running, cleared before the running state is written. The watchdog acts only on first launches, as AWS terminates only a launch that never ran. MarkFailed refuses to terminate any other instance and marks it recovery-failed instead, which keeps its volumes, ENI and addresses and shows it as stopped. A claimed relaunch is never a first launch. Invariant tests pin ADR-0003:S7. The instancerecovery e2e suite gains a test that ages its guest past the watchdog before recovering it, then stops and starts it, and requires the instance, its root volume and the data on it to survive both. * feat: run the Terraform workbooks against a Vultr deployment on demand, and show the Vultr portal in the guide e2e-cloudvendor-vultr.yml is dispatch-only. It builds the ref, deploys it with deploy-vultr.sh on one or three VMs or bare-metal servers, runs the published workbooks from node1 with the same driver as the hypervisor nightly and OCI, and destroys the deployment, failing if anything is left billing. It needs one secret, VULTR_API_KEY, and generates its SSH key per run. The guide gains screenshots of the API key's access control list and of the nodes in the Vultr portal, and its portal navigation now matches. * fix: install setup.sh on Vultr nodes when the tarball does not carry it build-install-artifacts.sh writes setup.sh beside the tarball, so a node installed from its tarball had no /usr/local/share/spinifex/setup.sh and arming the firewall failed. * fix: pass the remote command after the destination in the Vultr e2e driver * fix: run each workbook in the cluster's own region The workbook driver passed no region, so every workbook used its default, ap-southeast-2, and on a cluster in any other region every request failed SignatureDoesNotMatch. It now passes the region of the node's spinifex profile. * feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) A node counted physical cores, collapsing SMT threads, and held back a fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is now one online CPU, as AWS counts it: a thread with SMT on, a core with SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and the rest is allocated one to one. The existing overrides stay operator-only. Invariant tests pin every tier boundary. * docs: state the Vultr guide's sizing under the online-CPU count * docs: drop an unmeasured EKS sizing claim from the Vultr README * fix: let a guest reach an Elastic IP hosted on its own node A guest's packet to an EIP on its own node left its VPC from the transit range, was masqueraded to the transit host address, and went back into OVN, which exempts that range from EIP DNAT, so it was never answered. On Vultr an EKS worker could not reach its control plane's endpoint, so it never joined. AWS lets an instance reach a public address in its own VPC. Hairpinned transit traffic now jumps to SPX-EIP-HAIRPIN, above the egress MASQUERADE, where each EIP's rule sends it from the uplink address, as the host's own traffic to the EIP already is. The rule follows the EIP's ingress plumbing in and out, and anything else returns to the MASQUERADE. * fix: resolve Spinifex's LB and EKS names on Vultr nodes Vultr's Ubuntu image writes a static resolv.conf naming Vultr's resolvers, so lookups bypassed the split DNS Spinifex configures in systemd-resolved and an EKS endpoint name did not resolve on the node. deploy-vultr.sh points resolv.conf at the resolved stub, as stock Ubuntu does. * test: prove a guest reaches an EIP on its own node in the natuplink e2e The nat-single cells run it with a public pool, so the hairpin path that left EKS workers unable to reach their control plane on Vultr is now covered nightly. * docs: record the natuplink hairpin check in the e2e coverage index * fix: keep each Vultr node's logs when an e2e run fails The driver destroyed a node whose install failed together with the log that said why. It now copies /root/*.log from every node into the run's artifacts before the teardown, and the workflow uploads them. ---------	2026-10-30 23:12:58 -05:00
Josh Siviour	e709717fde	fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options (#1360) ... * test: pin EKS worker IMDS metadata options and launch template resolution * fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options * fix: default unset launch template metadata fields for EKS workers * test: e2e coverage for EKS worker IMDS hop limit and launch template metadata options * test: drop the unneeded GetCallerIdentity policy from the worker IMDS e2e * docs: describe the EKS worker IMDS defaults and launch template overrides * test: drop the stale GetCallerIdentity policy from the IRSA pod e2e	2026-10-31 14:07:05 +10:00
Ben Duncan	f09250d2bb	feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) (#1359) ... A node counted physical cores, collapsing SMT threads, and held back a fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is now one online CPU, as AWS counts it: a thread with SMT on, a core with SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and the rest is allocated one to one. The existing overrides stay operator-only. Invariant tests pin every tier boundary.	2026-10-30 22:53:45 -05:00
Ben Duncan	8839407c71	feat(ui): console design system, side-panel resource views and live EC2 metrics (#1358) ... * feat: lay the Spinifex UI design-system foundation Brand tokens in OKLCH with semantic status and a colour-blind-checked chart palette, Archivo headings and a 13px type scale, and a pre-paint theme script so dark mode never flashes white under the console CSP. Adds the missing Base UI primitives, a /design catalogue route, a route ratchet against hand-written tables, a ban on chart-library imports in routes, and lets the content column shrink so wide tables scroll in place. The dev server now forwards /api and /proxy to the Go console. Sonner is removed: its inline style element is blocked by the CSP. * feat(ui): side panels, TanStack table, uPlot charts and e2e gate - DataTable renders @tanstack/react-table, with column hiding as container queries so columns drop when a side panel narrows the list. - UniFi-style DetailPanel and ResourceView: Instances, VPCs and Internet gateways open in a panel beside the list from 1024px and as a full page below; selected, tab and create live in the URL. Create flows for instances, VPCs and gateways open in the same panel. - Instance monitoring charts on uPlot (CPU, network, packets), on simulated data and badged as such. - Dates in the AWS console form, e.g. 2026/06/04 02:00 GMT-5. - Tabs style Base UI's data-active, so the selected tab now shows. - Playwright suite gates CI as the UI End-to-End job in ui.yml. * feat(ui): one details component for panel and page, UniFi-style overlay panel The side panel now lies over the list at full height with a shaded edge, and a click on the list's background closes it. Instances, VPCs and internet gateways each render one details component as either the panel or the resource's own page, so the page gains the monitoring graphs and the panel gains IAM role, GPU, console output, change type and create image. SSH moves into Instance information as rows. Internet gateways attach and detach from the panel. * feat(ui): roll the panel pattern out to every service and gate screenshots in CI Every EC2, IAM, Route 53, Bedrock, EKS, ECS and RDS resource is now a list plus a side panel, with one details component serving as both the panel and the one-column page. DetailCard, DetailRow, ListCard, ResourceList and DeleteResourceButton are deleted. - Panel: slide-out close, row selection closes it, expand keeps the tab and fetched console output, PanelSpinner while loading. - Instances: info tips, theme-aware coloured console output, power button group plus Actions menu, Stop and Reboot confirms, security group, Elastic IP and network interface sections, and a live one-second monitoring window on simulated data. - Launch instance: OS tiles picking the newest version, architecture choice, location tile, expanded root volume, honest default subnet. - On/off settings are switches app-wide; disabled controls show cursor: not-allowed. - The e2e job runs in the pinned Playwright image with screenshots gating; make ui-baselines and make ui-e2e-container use the same image. * feat: bulk select and delete on every resource list, OS marks and instance type sizes Every list with a delete action gets row checkboxes and a selection bar. Delete confirms in a dialog naming each resource, makes one call per resource, keeps each refusal against its row and retries only failures. The Launch form shows distribution marks for Ubuntu, Debian, Rocky, Alpine and CentOS, served as files under the CSP, and states the selected instance type's vCPUs and memory. * feat: sign-in overlay, larger type scale, typed confirms on irreversible deletes and dialog shortcuts The sign-in card sits over an inert shell. Any sidebar group folds, and entering one unfolds it. Instance DNS names copy. Escape cancels and Cmd/Ctrl+Enter confirms every dialog. Volume, snapshot, RDS, EKS, ECS and hosted zone deletes need "delete" typed, and terminate needs "terminate". * feat: fade sign-in and sign-out in place, and show the confirm shortcut Signing in no longer reloads the page: the card and a lighter veil fade out and the console routes to the dashboard. Signing out cross-fades into the sign-in card with a Signed out notice. Confirm buttons show Cmd or Ctrl Enter for the platform. * feat(ui): generic sidebar group names and a paged S3 object browser The sidebar names its groups Compute, Databases, Authentication and Object Storage; the command palette still finds each by its AWS service name. Buckets get row selection and delete. Inside a bucket, folders and objects share one table paged by ListObjectsV2 continuation tokens, with a find box that queries by prefix, multi-select delete through DeleteObjects (folders with everything under them), and an object panel showing HeadObject properties, user metadata and a download. * feat(ui): show upload progress and stop the S3 proxy cutting off slow transfers The Upload button shows a running batch's percentage, and a large or slow batch opens a dialog with bytes sent, bit-rate, time left and per-file cancel and retry. Uploads run in a store outside React, so they continue while the user browses. The console's S3 proxy now allows an hour per request. The server's 15 s ReadTimeout cancelled every upload part that took longer, so large uploads failed on links below roughly 18 Mbit/s. * feat(ui): stream live EC2 metrics to the Monitoring tab The Monitoring tab drew simulated data. It now streams CPU, network bytes and packets from the owning node's qmp-collector every 500 ms while someone is watching, with nothing stored. - qmp-collector: a watched sampler on schedstat and tap counters, on a 30 s lease, keeping five minutes in memory for reconnects. It answers only watches for the account in the VM's discovery file. - awsgw: spinifex:GetInstanceMetrics, checked against the instance ARN, answered as text/event-stream in 60 s segments. Another account's instance is InvalidInstanceID.NotFound. - console: a signed stream reader that reconnects with Since and backs off on failure or an empty segment, stops on InvalidAction, and a chart axis that scrolls between samples. The simulator and its badge are removed. * feat: catch refused creates in every form, answer the third review, and guard tables Every create form now catches a refused mutation, so the failure shows beside the submit button and no longer escapes as an unhandled rejection. Three EKS tabs that swallowed a refused create now show it. Third-review answers: the attach dialog offers running or stopped instances in the volume's zone; the default security group's delete is disabled with its reason; the load balancer panel splits into Listeners and Attributes tabs; a main route table lists its implicit subnets; IAM single delete dialogs name the same dependants as the bulk ones; a snapshot can be deleted in any state. Every list in a route renders through DataTable, enforced by a unit test with a named allowance, and the /design baselines are redrawn in the pinned container. * feat: cap live metrics streams, prune hidden selections, and move panel focus Each gateway holds at most 10 live metrics streams per account and 200 in total, refusing past either with RequestLimitExceeded, and counts opened and refused streams. A row that leaves a list leaves its selection, the instances page uses the shared selection hook, the detail panel takes focus on open and returns it on close, and Escape yields only to the popup it was pressed in or a modal. The UI workflow also runs on live metrics backend changes.	2026-10-30 22:40:47 -05:00
Josh Siviour	dee37b976a	fix: keep an ECR token valid after its role session expires (#1357) ... * test: authorise an ECR token from the live role once its session is gone * feat: carry the role ID in an ECR token minted for an assumed role * feat: rehydrate an assumed-role caller from the live role * fix: authorise an ECR token from its live role after the session expires * docs: note that an ECR token outlives its role session	2026-10-31 12:20:24 +10:00
Josh Siviour	ce8272cf5e	refactor: register EC2 through the ingress seam and delete the legacy service switch (#1355) ... * test: characterise EC2 request labels and the cluster-unavailable gate order Pins the span, audit, throttle and policy-gate labels an EC2 request carries through the signed pipeline, and that the NATS gate answers before policy, action lookup and gateway-local actions. * refactor(gateway): register EC2 through the ADR-0002 seam EC2 is composed in awsgw from domains/ec2/awsapi and served by the registration seam. The legacy service switch and its tables are removed, along with the two paid-off gateway allowlist rows. * docs: record the EC2 seam registration in the package boundary migration * docs: record the EC2 seam registration PR number	2026-10-31 12:09:59 +10:00
Josh Siviour	0fc93e975a	refactor: serve EC2 actions from domains/ec2/awsapi (#1354) ... * test: characterise EC2 through the signed gateway pipeline * refactor: serve EC2 actions from domains/ec2/awsapi over composed capabilities The EC2 action table, idempotency and validation move out of the gateway behind a Deps struct. Quota is a consumer-side interface and the instance adapters take a narrow InstanceProfiles interface instead of the whole IAM service. The gateway only composes Deps and calls Dispatch. The 18 gateway to EC2 awsapi allowlist rows are deleted. * test: drive every EC2 action through Dispatch in its own package Covers the request errors, the gateway-local actions without NATS and each routed action reaching its handler against a stub daemon. * docs: record the EC2 capability move in the package boundary migration * docs: record the EC2 capability move PR number	2026-10-31 00:30:51 +10:00
Josh Siviour	d066786faa	feat: operator-launched SQS worker and store system VMs (#1353) ... * feat: persist the SQS worker's store key on its own disk The shared SQS worker generates its store-client NKey inside the guest and keeps it in an owner-only directory on its own persistent disk, so no platform component creates or unseals the seed. The key is synced and linked into place without replacing anything; a missing, mismatched, unreadable or loosely permissioned key fails closed instead of being regenerated. * fix: make the SQS worker key durably named before returning it Every successful call now syncs the key directory and its entry in the parent before handing out a key, so a retry after a failed directory sync cannot return a key whose name is not yet durable. Only the key directory is created, inside a parent that must exist, so an unmounted worker disk fails closed. The first-start decision is explicit: an existing store's authorized key is required and never defaults to generating a new one. * docs: state that workerkey cannot detect an unmounted worker disk An existing parent directory does not prove the worker disk is mounted, since the mountpoint can exist on the initramfs after a failed mount. The error now says only that the parent is missing, and the package documents that its caller must verify the expected data volume is mounted first. * feat: add the SQS worker guest bootstrap The shared SQS worker microVM boots with sqs-worker boot, which selects exactly the disk whose serial fw_cfg names, checks and mounts it at /data, and proves from the mount table that this disk is what is mounted before loading or creating its store key there. A missing, empty or unknown store status, a contradictory or invalid store key, a missing or ambiguous disk, a failed check or mount, an empty mountpoint or a foreign mount refuses without creating a key. Only the public key is logged and published. The image is built by the store image script under SQS_MICROVM_PROFILE=worker, with the worker binary, no nats-server and an empty /data mountpoint; the store image's contents are unchanged. * fix: refuse replayable new-store status at SQS worker boot A static fw_cfg store-status of "new" survives a guest reset, so after the store exists it could still reach key creation and replace a lost worker key. Boot now accepts only "existing" with the store's authorized public key and refuses "new" before touching any disk, until a current authoritative first-store decision exists. workerkey.ForNewStore remains, uncalled. * feat: add SQS first-store claim and worker key acquisition The SQS domain records one first-store decision per Region in its own KV bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound to the worker's instance ID, claim ID and generation. Key creation is authorized only while no public key is recorded; afterwards only the recorded key is returned, and a launch intent requires it. Older generations, same-generation rivals and unreadable records authorize nothing, and records hold no private key material. workerkey.Acquire syncs a new key to disk and returns it only after its public key is recorded, reusing the on-disk key after a crash before recording and failing closed on a missing or different key afterwards. No transport carries the decision yet. * fix: read SQS first-store decisions from the stream leader KV.Get uses a direct get, which a lagging replica can answer, so a stale claimed read could authorize key creation after the leader had recorded the worker's public key, or make a superseded generation look current. Reads now use the SQS store's leader-answered stream message get, and every authorizing answer is also confirmed by a compare-and-swap at the revision read, so a stale or unavailable read authorizes nothing. * feat(sqs): add regional worker identity record and instance index The SQS domain keeps one worker identity record per Region with a monotonic generation, claim ID, pinned node and encrypted data volume, moving from launching to bound or failed under compare-and-swap. A launch intent takes the next generation before any instance exists, so the previous worker stops being current; binding writes the instance index entry first. Authorize accepts an instance only while the Region's leader-read record is bound to exactly that instance, generation and claim, confirmed by a compare-and-swap at the read revision, so an old, orphaned, rewritten or unbound index entry, or a lagging replica, authorizes nothing. Both this bucket and the first-store bucket now open through the kvutil chokepoint and refuse a stored configuration that could expire, cap or evict records or holds fewer replicas than the cluster needs. Nothing calls the store yet. * fix(sqs): make worker launch exclusive and harden durable buckets BeginLaunch now refuses a new claim while the previous attempt is launching or bound, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume. A replacement takes the next generation only after FailLaunch or the new RecordStopped confirms that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never saw. The durable bucket opener also refuses non-limits retention, per-message TTLs, asynchronous flush, a message-size cap, any discard policy other than discard-new or a per-subject discard, unacknowledged writes, a sealed stream, and a mirror or sources. * feat: add internal SQS worker first-store gateway actions Serve ClaimFirstStore and RecordWorkerPublicKey under the internal spinifex-sqs signing service, apart from the public AmazonSQS inventory. Only an sqsWorkerRole session in the system account passes the class gate, then the role policy; the worker identity is resolved from the current workerident record by the STS-set session name, never from the request, and any extra parameter is refused. Seeds and other non-public keys are refused without being recorded, logged or echoed. The gateway binds the worker-identity and first-store buckets per request and never creates them, so it provisions nothing and answers ServiceUnavailable until a worker launch has created them. * fix(sqs): fence replaced worker generations out of first-store A request that passed worker authorization before a confirmed stop and replacement could reach an unclaimed first-store record ahead of the successor and create or record the old generation's key. The launch path now raises a monotonic first-store fence past the replaced generation before BeginLaunch, which refuses a replacement until a leader read shows it; firststore refuses every call from a generation below the fence. The internal adapter no longer logs caller-chosen parameter names. * feat(sqs): launch the shared SQS worker on its pinned node Add the SQS worker system-instance kind and a domain launcher that realizes the Region's worker from its identity record without ever starting a second VM: - domains/sqs/workerlaunch returns a bound worker as is, prepares the data volume on first launch only, refuses a volume whose own record is not encrypted, marks the intent dispatched before launching and never resends a dispatched launch: an ambiguous outcome stays launching until reconciled. A replacement starts only after a confirmed stop or failure and the first-store fence, reusing the pinned volume. - workerident gains MarkDispatched and Current; firststore gains a leader-read RecordedKey. - The daemon launches the worker through the shared direct-boot path with its encrypted volume attached at boot and kept on terminate, a required instance profile and a mandatory mgmt NIC. Its firmware config carries only public values: the data serial, the store decision, the gateway URL on this node's bridge and the daemon's own cluster CA. A relaunch is refused on any other node. - sqsWorkerRole is ensured in the system account granting exactly the two internal actions; an existing role whose trust admits more than EC2 is refused. The role contract lives in domains/sqs/workerrole so the composition root does not import the gateway adapter. The worker network allocator and any caller remain unwired, and the guest still refuses to create a key for a new store. * fix: verify SQS worker role trust before granting and report unclaimed volumes Check that an existing or racing-created sqsWorkerRole trusts only EC2 before writing its policy, and accept the instance profile only when a re-read shows it carries exactly that role. Refuse the first launch before allocating when the cluster storage key is not configured, and return the ID of any volume created but not confirmed in a launch intent instead of dropping it. * feat: add private SQS system VPC and worker ENI allocator Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet from the VPC and subnet surface only, so it cannot create an internet gateway, route table, NAT gateway or EIP. The daemon's SQS worker network adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and finds or creates one ENI per launch claim in the VPC default security group, refusing a duplicated claim or an address outside the subnet. Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have recorded the volume. * feat: worker takes its first-store decision from the gateway at boot The SQS worker guest no longer trusts a static fw_cfg store status. After selecting, checking and verifying the mount of its own data disk, it asks the internal spinifex-sqs gateway service for its current first-store decision, signed with instance-metadata credentials only, over TLS 1.3 trusting only the cluster CA. It then creates and records or loads the key on that disk; a key lost after recording is never replaced. - internal/sqsworkergw: SigV4 client for ClaimFirstStore and RecordWorkerPublicKey; refuses non-public keys, non-decisions and redirects; errors carry only the parsed code - gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback - workerboot: retries only lost replies and ServiceUnavailable/Throttling - launch contract carries the Region instead of a store status or key * fix: worker retries a denial while its launch is still binding it The worker VM can reach the gateway before its launch binds the instance identity, and that early call is legitimately AccessDenied. The guest treated the denial as final and stayed unprepared even after binding. AccessDenied is now a pending authorization the guest asks about again within its existing bound; a denied claim creates nothing, and a worker that is never bound, or was replaced, gives up at the bound. * feat(sqs): add operator EnsureSQSWorker action and node launch handler Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the internal spinifex operator service. It validates the node against the cluster and as a single subject token, then requests sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers and the worker stays pinned. The node handler re-checks the named node and instance type and runs one composed workerlaunch.Launcher (real volume preparer, role, private network and direct-boot launch), one launch at a time, in the daemon's own context with a 30-minute bound. A caller timeout does not cancel the launch; a retry reports the bound worker, or IncorrectState for a dispatched launch whose outcome is unknown, which is never resent. The reply names the bound VM and does not claim the worker or SQS is ready. Nothing starts a worker at daemon startup. * feat: ship the SQS worker image and check it before allocating Build the SQS worker microVM image in both release workflows, stage it into the tarball as microvm-sqs-worker, and install it with setup.sh under the store image's rules: a release without it removes an earlier one and an incomplete one is refused. Add make install-sqs-worker-microvm. The worker launcher now asks a Host capability whether this node can boot a worker before it prepares a volume, records or resumes an intent, fences a replaced generation, allocates the network or dispatches. The daemon checks for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a call after the image is installed launches instead of leaving an ambiguous dispatched intent. Cover missing and incomplete images through the admin action and handler, a stopped worker left untouched, and an ENI create whose reply is lost. * fix: never repeat an uncertain SQS worker ENI create The launcher now looks for the claim's ENI without creating one, marks the launch intent NetworkAttempted, and only then sends the single create. Once marked, a missing ENI fails closed with IncorrectState instead of a second create that could give the claim two interfaces; a visible ENI is reused. Adds a test that buildSQSWorkerLauncher wires the daemon's real host check. * feat(sqs): operator-triggered SQS store launch on the worker's node Add the super-admin EnsureSQSStore action and its node handler. The store is launched only on the node the Region's worker is bound to, with the public key that worker recorded in the first-store decision; the daemon never receives the worker's seed. A per-Region store launch record marks each step before its uncertain create (volume, ENI in the worker's private subnet, VM launch), so a step whose result is not visible fails closed and is never resent. Volumes are refused unless their own record is encrypted, and their IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and instance types at or below that are refused; this is a floor, not a validated size. * fix: retry uncreated SQS store volumes and fix the store instance type A volume preparation that failed before sending any create (a missing tool, scratch or format failure, or no cluster storage key) now carries storevolume.ErrNotCreated. The store launcher withdraws its volume mark and returns ServiceUnavailable, so a retry may try again once the host is fixed; any create that was sent stays one-shot. The store launch record now keeps the operator's instance type. A request for a different type is refused as IncorrectState before dispatch and after binding, and the reply names the launched type, which the admin action checks. * feat: SQS worker store readiness and store size check The worker's one-shot ready step mounts and verifies its disk, asks the internal spinifex-sqs DescribeStore action for the bound store, loads its key only if it matches the recorded public key, and checks the store's JetStream with it. It retries within a bound while the store is pending or not answering, and stops on a mismatched or refused key. The NKey authenticates the worker only; the store server is not authenticated. EnsureSQSStore now also refuses a different volume size once the store is bound, instead of returning the existing store. * fix(sqs): keep worker store readiness current until shutdown The readiness step now runs as a guest-local watch loop with no overall deadline: each pass is bounded, a pending or unreachable store is retried every few seconds, and a ready store is rechecked periodically. The store-ready marker is removed on any failed pass, cleared at start and removed on exit; shutdown stops the watcher by its recorded PID. Terminal refusals such as a rejected or mismatched key back off for minutes rather than spinning. The NKey still authenticates only the worker to the store. * feat(sqs): supervise the worker readiness watcher and add spx admin sqs Run the worker guest's readiness watcher under init respawn, rate-limited, and remove any stale store-ready marker before each start, so a killed or crashed watcher cannot leave a false readiness marker. Shutdown blocks respawn, removes the marker, stops the recorded watcher by PID and start time with SIGKILL escalation, and unmounts /data only once it has exited; a surviving watcher leaves every mount in place. Add spx admin sqs ensure-worker and ensure-store, which call the existing EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex gateway path with the standard AWS credential chain. Both refuse missing inputs before sending anything and report only the bound VM, never readiness. * fix: park SQS worker readiness and fix the worker instance type A readiness start that finds shutdown in progress or no worker key now logs once and waits in place instead of exiting, so init's respawn does not restart and log it in a loop; shutdown stops it like a watcher. The worker identity record keeps each generation's instance type. A request for another type is refused as IncorrectState before dispatch and after binding, and only a replacement may change it. The ensure reply, admin action and spx admin sqs ensure-worker report and check the recorded type. * fix: record the SQS worker volume intent before creating it The first worker launch prepared its data volume before any record existed, so a create that applied but returned no ID left nothing for the next call to find, and it could prepare a second volume. The launch intent is now recorded first, and its one volume create is marked before it is sent. A Prepare proven to have sent nothing withdraws the mark and is retryable; any other outcome is never resent and returns IncorrectState, naming the volume when its ID is known. An intent that has sent no create may still be re-typed. * docs: document spx admin sqs ensure-worker and ensure-store Record the flags, the caller and owning accounts, the bound-versus-ready distinction and the current retry and recovery limits. * docs: distinguish SQS ensure timeouts from refusals that created nothing A gateway timeout means only that no reply arrived; the launch may still be running. The worker type locks once any step has been attempted. * docs: state what each SQS ensure ServiceUnavailable establishes No refusal proves earlier attempts created nothing; a retry checks the durable attempt and continues only when that is safe.	2026-10-31 00:17:25 +10:00
Josh Siviour	55054ad0ac	fix: interim guard against opening a volume behind a live owner holding unsealed writes (#1352) ... * test: relay a capacity refusal from an owner holding unsealed writes * fix: refuse a start elsewhere when the owner holds unsealed writes * fix: bound the unsealed-writes check well under the start forward timeout * test: refuse a volume takeover while the marker owner is live * fix: refuse a volume takeover while the previous owner is live * fix: retry EKS control-plane starts refused for capacity or a held volume * docs: describe the interim unsealed-writes takeover guards * test: give the viperblockd conformance harness a cluster-state bucket The takeover guard reads node liveness from the cluster-state bucket the daemon creates. The harness had none, so the real reader errored and the guard refused every takeover. Create the bucket the way the daemon does so the real reader runs and finds no heartbeat. * fix: leave no dirty marker behind when opening a volume that does not exist An open takes the lease and writes the dirty marker before the volume's state is read. When the volume turned out to be absent the marker stayed, so a later open on another node met the takeover guard and was refused as held elsewhere instead of answering not found. Clear the marker this open wrote when the state load reports the volume missing.	2026-10-30 23:53:41 +10:00
Josh Siviour	27304bbfd6	refactor: serve the four Bedrock services from one Ochre registration (#1351) ... * test: characterise the four Bedrock services through the SigV4 pipeline * refactor: serve the four Bedrock services from one ochre registration The Bedrock adapters move from gateway to domains/ochre/awsapi behind a single dispatch registration. The sub-service scopes are aliases, a sub-service scope selects its dispatcher outright and only the shared bedrock scope is split by path. The thirteen Bedrock fields leave GatewayConfig for awsapi.Deps, and the bodyscope allowlist row for domains/ochre is deleted. * docs: record the Bedrock registration in the package boundary migration * docs: record the Bedrock registration PR number	2026-10-30 23:06:24 +10:00
Josh Siviour	a52b078678	refactor: move the ECR /v2 bridge out of the gateway (#1350) ... * test: pin the ECR /v2 pipeline through the gateway handler * feat: rehydrate a principal from an access key id in credverify Adds credential.Rehydrator and CallerARN on Principal, implemented beside Authenticate and Resolve on the same key, session and user lookups, with a parity table against the gateway's ECR rehydrate. * refactor: share the credverify key and session lookups with Rehydrate Authenticate, Resolve and Rehydrate now call one lookup per credential kind, so the checks cannot drift. The parity table also asserts the gateway and credverify caller ARN builders agree. * refactor: move registry host matching to ingress/http The matcher takes its service labels from the caller and exposes the parsed account and region through accessors, so ingress names no service. * refactor: extract the identity-policy evaluator into policyauth The SigV4 gateway path and the ECR registry authorize through one evaluator and one set of request condition keys, over the credential principal. * refactor: extract account admission into accountadmission The SigV4 gateway path asks one Admitter for account status, with the ACTIVE-only cache moved with it, so the ECR registry can use the same capability. * refactor: move the ECR /v2 bridge out of the gateway The bridge, token exchange and per-operation authorization now live in domains/ecr/registry behind registry-owned Principal, Rehydrator and Authorizer types, so the package imports no ingress code. The awsgw role adapts credverify, policyauth and host matching to them, and the gateway keeps only a RegistryHandler mounted at /v2. The bridge refuses to compose without a verifier, issuer, rehydrator, authorizer or admission, so the open-registry mode for a nil verifier is gone. * test: carry the retired ECR principal parity rows into credverify Global root, a verifier with no IAM, a record with no principal type and the gone, replaced and legacy session verdicts for user and role sessions were only exercised by the gateway tests that went with the old path. * docs: record the ECR /v2 bridge move in the package boundary migration * docs: record the ECR /v2 bridge PR number	2026-10-30 15:57:37 +10:00
Josh Siviour	26d53602a3	fix: accept AWS-format IRSA trust-policy condition keys (#1349) ... * test: cover AWS-form IRSA condition keys and StringLike on web identity * fix: match AWS-form IRSA condition keys and allow StringLike on web identity * fix: keep the provider ARN host-path unnormalised for web identity * feat: log the reason a web identity trust policy denies * fix: satisfy errname, modernize and gofmt on the denial error * test: cover the stock AWS IRSA trust policy in the EKS e2e * test: make the IRSA denial prove the policy was evaluated	2026-10-30 14:34:08 +10:00
Josh Siviour	060204514e	refactor: register ECS and EKS through the ingress seam with shared PassRole (#1348) ... * test: pin ECS and EKS dispatch through the SigV4 pipeline * refactor: register ECS and EKS on the ingress seam EKS resolves the cluster and nodegroup role through passrole.Check with the IAM role resolver, in the order it had. Responses and check order are unchanged. * fix: match AWS role handling for ECS and EKS PassRole cases ECS refuses a foreign task role before PassRole, accepts a role that is not an ARN or does not exist after PassRole, and reports a RunTask override role it cannot assume. EKS CreateCluster and CreateNodegroup refuse a foreign or non-role ARN with AccessDeniedException before PassRole. * test: cover ECS and EKS awsapi registration dispatch * fix: report EKS parameter faults as InvalidParameterException * docs: record ECS and EKS gateway migration and EKS divergences	2026-10-30 14:18:53 +10:00
Josh Siviour	099a69b23b	refactor: register IAM and STS through the ingress seam (#1346) ... * docs: point the /admin migration row at spinifex#1345 * test: characterise IAM and STS dispatch through the SigV4 pipeline * refactor: register IAM through the ingress seam * refactor: register STS through the ingress seam * docs: point the IAM and STS migration rows at spinifex#1346	2026-10-30 12:08:15 +10:00
Josh Siviour	6462ccc41f	refactor: serve the /admin operator API from the spinifex registration (#1345) ... * refactor: move the /admin handlers and wire types into operator/admin * refactor: serve /admin through the spinifex registration * docs: record the /admin move into the spinifex registration	2026-10-30 10:50:47 +10:00
Josh Siviour	ae8ee1e5fe	refactor: register RDS through the ingress seam (#1344) ... * refactor(gateway): register RDS through the ingress seam RDS dispatch moves from gateway/rds.go to domains/rds/awsapi.NewRegistration, composed by awsgw and the integration harness. The caller comes from dispatch.Actor and the principal-class gate still runs before the policy check. * docs: point the RDS migration row at spinifex#1344 * chore: group the imports merged in the actor projection test	2026-10-30 09:59:15 +10:00
Josh Siviour	488f1c7bf9	feat: SQS store guest on a prepared cold-attached EBS volume (not wired) (#1343) ... * feat: build a separate SQS store microVM image Package the go.mod-pinned nats-server, ext4 support and a clean ACPI power-down into a microVM kernel and initramfs of their own. The build checks the packed archive's contents and refuses a nats-server whose version differs from the pin. The image does not start the store and nothing installs or releases it; the LB image is unchanged. * fix: publish the SQS store image only after its gates pass Stage the kernel and initramfs and move them into place only once the content, size and guest-RAM gates pass, so a failed build cannot leave a pair make takes as up to date. The old initramfs is removed first and the new one lands last, so an interrupted publish never leaves a mixed pair. Check the module list without piping into grep -q, which under pipefail reported a present module as missing. * feat: start the SQS store guest on its checked data disk The SQS store image now prepares, runs and stops a dedicated JetStream server. It reads a narrow fw_cfg contract (data disk serial, listen address, one user NKey public key), refuses anything absent, malformed or ambiguous, has the pinned server test the config, then checks and mounts exactly one matching ext4 disk. It never formats and has no ephemeral fallback. The server listens only on the supplied guest NIC address, admits only that NKey and syncs every write. Shutdown stops the recorded run process first, whether it is the server or a start not yet exec'd, and runs the generic read-only remount only once it is gone. A store that survives SIGKILL leaves every mount alone. * fix: report the SQS store as prepared, not listening, before it starts * feat: launch the SQS store guest on its own versioned subject An explicit SQS store kind carries an existing system-account data volume and the one NATS user public key the store accepts. It is served only on system.LaunchSQSStore.v1, so a daemon without this code has no responder and the launch fails instead of booting a load balancer without its disk; the legacy launch path refuses the kind on both send and receive. Every refusal happens before capacity, an ENI or a VM record exists. The volume goes through the normal EBS mount and terminate detach and is never deleted on termination. The store boots its own image with CPU mitigations on, gets only its network, disk serial, listen address and key, and cannot be relaunched after a host restart yet. * feat: ship the SQS store microVM image in releases and installs Release builds now build the SQS store image and stage it beside the load balancer's as microvm-sqs-store/, so an older installer that copies microvm/* without -r still works. setup.sh installs it to /usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it. A release without a complete image never leaves an older one usable: staging drops a previously staged image, and the installer removes an earlier installed pair before warning (missing) or failing (incomplete). Only the two image files are removed. The initramfs is installed last, so an interrupted install leaves no bootable mismatched pair. * feat: restart the SQS store guest on its launching node Persist the non-secret guest config of an SQS store instance (launching node, data volume, listen address, client public NKey and rendered network config) in the instance record, and rebuild its fw_cfg files from it on relaunch. The relaunch refuses on any other node, or when the record is missing or malformed, before writing any file; the data volume is remounted through the normal EBS path and kept on terminate. Cross-node recovery now never selects or claims an SQS store record, so no other node spends retry budget on it or relaunches it. Same-node restore is unchanged, as are load balancer restarts. Older daemons know neither the exclusion nor the record field, so this does not prevent a cross-node relaunch by a mixed-version cluster. * feat: hold prepared system volumes in creating until released Add an internal CreateVolumeForPreparation path that writes a new volume's first record as creating, and CompleteVolumePreparation to release an unattached creating volume in the caller's account to available. Ordinary CreateVolume is unchanged. Attach and delete already refuse a creating volume. ModifyVolume, ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState so nothing can expand, release or capture a volume mid-preparation. * feat: prepare fresh SQS store volumes with an ext4 filesystem The SQS store guest never formats a disk, so a new volume gets its filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img, builds an ext4 image in private scratch, creates the volume in "creating", publishes it on this node, copies the image with the NBD import helper, unpublishes, and only then releases it as available. It never writes a volume it did not just create. A volume is deleted only when no export can still be live: after a confirmed unpublish, or when the export was not NBD. Any publish error, a failed or ambiguous unpublish, or a failed release retains the volume and reports its ID. A publish refused as in-use or already-exists sends no unpublish, since another export may exist. Cleanup ignores caller cancellation but is bounded at 30s. The raw-image copy moves to runtime/compute/nbd unchanged, and the domain receives it as an injected writer. The volume service gains a discard path for an unattached volume still in "creating". * fix: report unknown state and match qemu-img lookup in volume prep A failed release may still have made the volume available, so the retained-volume error now says its state is unknown rather than claiming it is still creating. The copy runs qemu-img by name through PATH, so the pre-allocation check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin fallback because it runs from the resolved path. * docs: describe the unwired SQS store system instance Records what the guest image, versioned launch, same-node restart and fresh-volume preparation provide, that nothing calls them, and what is untested or missing before production use.	2026-10-30 08:16:43 +10:00
Josh Siviour	24281cb860	fix: count volume lease validity from the write, not its reply (#1340) ... * fix: count volume lease validity from the write, not its reply The server starts a lease entry's TTL when it applies the write, but the holder started its validity window when the acknowledgement arrived, so a slow reply stretched validity past the server's expiry and a peer could acquire the volume while the holder still considered it valid. Validity now counts from when each create, generation publish and renewal was sent. An acquisition acknowledged after validity lapsed is refused as retryable, and a lost or lapsed lease still awaiting its fence is no longer shared with another open on the same node. A renewal that lands after the lease was given up, or after its validity elapsed, cannot revive it: the lapse check and the update run under the lease lock, and fencing runs once. The renew loop also fences at the validity deadline itself rather than at the next check, which could consume the whole margin. This bounds local validity only; it is not backend-enforced fencing, and a paused host can still write after its validity ends. * docs: describe volume lease validity, fencing and their limits	2026-10-29 21:38:22 +10:00
Josh Siviour	18adea9486	refactor: register the spinifex operator service through the ingress seam (#1337) ... * test: characterize the spinifex Query and /admin surfaces through the full gateway pipeline * refactor: register the spinifex Query service from operator/admin * docs: point the spinifex registration migration row at the PR	2026-10-29 14:55:50 +10:00
Julian Sommer	ef7d30578f	chore(awsmodel): sync AWS models to api-models-aws e10d6a1c (2026-10-28) ... Seven models changed and add 26 unimplemented operations. The request sweep gains 33 requests with no new findings.	2026-10-29 13:35:39 +10:00
Josh Siviour	d7d922e963	refactor: register Tagging and ELBv2 through the ingress seam (#1336) ... * test: characterize Tagging dispatch, authorization and GetResources responses through the gateway pipeline * refactor: register Resource Groups Tagging through the ingress seam * test: characterize ELBv2 through the full gateway pipeline * refactor: register ELBv2 through the ingress seam * test: cover the ELBv2 registration, query parser and load balancer limit * test: characterize RDS through the full gateway pipeline * docs: point the Tagging and ELBv2 migration rows at the PR * test: register Tagging and ELBv2 on the integration harness gateway	2026-10-29 13:27:43 +10:00
Josh Siviour	ad9568a6c6	refactor: move gateway/spx, admin wire types and EKS OIDC discovery out of gateway (#1334) ... * refactor: move gateway/spx to operator/admin * test: pin the admin wire types and method list in golden JSON * refactor: move the admin wire types and method list to contracts/operator/v1 * refactor: serve EKS OIDC discovery and JWKS from the EKS domain * refactor: assert the EKS OIDC publisher satisfies the gateway interface * docs: cite the pull request in the gateway root-move rows	2026-10-29 12:22:16 +10:00
Josh Siviour	c94b0f22f6	feat: accept EnaSupport on RegisterImage, report it on instances, and name refused parameters (#1332) ... * feat: accept EnaSupport on RegisterImage and name the refused parameter in its errors EnaSupport true and false are stored on the image record, returned by DescribeImages when set and carried through CopyImage. Every validation error in ValidateRegisterImageInput keeps its code and now says which parameter and value it refused. * feat: set EnaSupport on images created from an instance AWS takes an image's EnaSupport from the instance it was made from, and every instance here reports it, so the source AMI's value is not inherited. * docs: record RegisterImage EnaSupport support and the refused parameters * fix: report EnaSupport on DescribeInstances and RunInstances	2026-10-29 10:53:30 +10:00
Josh Siviour	11e8b68ddc	refactor: remove the unused host SQS message store service (#1333) ... * refactor: remove the host SQS store systemd unit The SQS message store will run inside system-owned instances rather than as a host daemon, so spinifex.target no longer starts it. Bump the target to version 4 and drop the unit from the embedded set and its invariants. * refactor: remove host SQS store config provisioning The SQS store will run in system-owned instances, so admin init and join no longer generate or distribute a host store token, routes or config, and the formation join payload drops the store token field. * refactor: remove host SQS store service command and role The SQS store will run in system-owned instances, so drop the unused spx service sqs-store command, its service registration and the host sqsstore role. The domains/sqs/message JetStream store is unaffected. * refactor: remove host SQS store provisioning from install scripts The SQS store will run in system-owned instances, so setup no longer creates the spinifex-sqsstore user, its directories or the 4223/4249 peer firewall ports, and install, reset and uninstall stop handling the retired unit, user and data directory. * docs: drop the host SQS store from firewall and connection tables Spinifex no longer ships a host SQS store service, so remove its ports from the host firewall and network connection references and say in DESIGN.md that no host service, unit, user or firewall rule exists for it.	2026-10-29 10:48:05 +10:00
Julian Sommer	2feb264863	feat: Oracle Linux multi-node clusters with OVN IPsec (#1327)	2026-10-29 11:17:22 +11:00
Josh Siviour	ddadfa2812	chore: remove unused e2e suite-selection manifest and drift lints (#1331) ... The targeted suite selector (select-suites) has not run in CI since its shadow-mode step was removed in #556. The service-interfaces manifest, manifest-check and manifest-lint existed only to feed it, so they go too. Also drops the now-meaningless e2e:allow-create markers, keeping their rationale as plain comments.	2026-10-29 10:04:31 +10:00
Josh Siviour	5923143062	feat: set X-Forwarded-For, -Proto and -Port on ALB requests (#1330) ... * feat: render X-Forwarded-For, -Proto and -Port headers in ALB HAProxy frontends Append joins every client X-Forwarded-For line and adds the client address as AWS does, preserve and remove follow the load balancer attribute, and Proto and Port are always overwritten. * feat: validate the ALB forwarding attributes and re-render config when they change ModifyLoadBalancerAttributes rejects bad xff_header_processing.mode and xff_client_port.enabled values with ValidationError before storing anything, and regenerates the stored config when either key changes. * test: assert the ALB X-Forwarded headers and the remove mode in the lb e2e The app instances echo the request headers on /__headers. The internal ALB is driven from the probe client and the HTTPS ALB from the runner, covering the caller address, a client value, spoofed Proto and Port, mode remove and a rejected mode. * test: let the lb probe client trigger accept a null header list * docs: document the ALB forwarding headers and their attributes	2026-10-29 09:33:39 +10:00
Josh Siviour	c42a352064	feat: add store-neutral SQS message store with a JetStream adapter (#1323) ... * feat: add the store-neutral SQS message store contract and conformance suite Defines message.Store over opaque shard and queue ID namespaces with no create call, plus a reusable conformance runner exercised against a test-only in-memory reference store. * feat: add the SQS message store's JetStream layout, append and state CAS Provisions the fixed set of shared message streams and state buckets, publishes each queue on sqs.m.<shard>.<queue>, dedupes idempotent appends with a queue-scoped Nats-Msg-Id, and maps state revision CAS onto KV. * fix: give each SQS queue immutable message and state shard allocations A changed stream or bucket count applies only to new queues: the namespace names its message shard and state shard directly, retained streams and buckets stay addressable, and the store refuses cluster sizes other than one node (R1) or three (R3). * fix: enumerate complete SQS namespaces and backstop state buckets Enumeration now returns each namespace with both allocations, so a state-only namespace can be counted and cleaned without its queue record, and state buckets expire records after 14 days plus 1 hour. * feat: add the SQS JetStream store's reads, counts and cleanup Get and Delete are scoped to the namespace's subject, DeleteThrough and DeleteNamespace are subject-filtered purges that leave shard neighbours intact, and Count reads the per-subject message total and live state keys. * fix: count distinct SQS state keys and validate before namespace purge The key lister can repeat a key under concurrent writes, so Count tallies distinct names; DeleteNamespace resolves both the message stream and the state bucket before purging either, so a wrong state shard deletes nothing. * feat: add the SQS JetStream store's ordered message-shard feed Append records the namespace's state shard in an internal header that is never returned as metadata. Feed reads one shared message stream through an ephemeral, headers-only ordered consumer with no acknowledgement state, rebuilds each change's complete namespace, refuses a malformed header with ErrCorruptRecord, and deletes its consumer on Close. * feat: enumerate SQS namespaces and run conformance on the JetStream store Namespaces joins per-subject body counts, whose internal header names the state shard, with live state keys from every state bucket that exists, found by listing so a historical bucket counts and none is created. An allocation conflict is reported as ErrCorruptRecord. The store now implements message.Store and runs the conformance suite on R1, per case with crash reopen and on one shared instance. * fix: check every SQS body's state shard when enumerating namespaces Sampling the first and last body let a middle body naming another state shard pass. Namespaces now reads every body's header in one headers-only ordered pass over the shard's stream, counting each queue's bodies from that pass, and deletes its consumer when done. * test: run SQS store conformance on R3 and guard against consumer state TestR3Conformance runs the conformance suite on a real three-node JetStream cluster in process, with Reopen restarting every node over the same data, and a check that every stream and state bucket is R3. TestS5_NoConsumerDeliveryStateInSQS fails on any acknowledgement, redelivery, durable-consumer or ack-retention mechanism in SQS source. * test: bar legacy NATS acknowledgement helpers from SQS source Adds AckSync, ManualAck, AckAll and AckExplicit to the ADR-0011:S5 consumer-state guard. * feat: add a bounded SQS store scan and the tombstoned-body sweeper Store gains Scan: a paginated, headers-only pass over a message shard, bounded at the shard's last position when it starts, skipping deleted positions and never returning a later append. The conformance suite and fake cover continuation, deleted runs and the bound. The JetStream adapter reads every bounded range (scan, body enumeration and state keys) through one reader whose completion rests on a direct get of the next match, not on consumer state; the client's key lister and pending counts are no longer trusted to end a pass. message.Sweep walks a shard with Scan and, for each body whose record the domain reports terminal, deletes the body then the record by revision. * test: add an opt-in SQS store conformance run on an external R3 cluster TestR3ExternalConformance runs the conformance suite against the dedicated three-node SQS JetStream cluster named by SQS_STORE_R3_URLS, checks every stream is R3 with two peers, and uses SQS_STORE_R3_RESTART to restart every node for the crash cases. It skips when unset. * test: run the SQS sweeper against the reference store fake Covers live records kept, tombstones swept, another shard untouched, a no-op rerun, and a retry after the body delete succeeds and the state delete fails. * test: harden the SQS store R3 harnesses against restart and placement races The external R3 harness reused one connection across restarts, so a case's cleanup closed it for every later case; each case and restart now opens its own connection and waits for stream leaders. The in-process cluster released its free ports before the servers bound them, and R3 placement can fail until the meta leader has every peer's config and stats. Client ports are now server-chosen, route ports are reserved together, and provisioning retries the idempotent New until placement succeeds. * fix: stop the SQS store feed idling for a whole pull expiry Under load an ordered feed on a three-node cluster sat for nats.go's 30s default pull expiry, once 60s, with stored changes pending and no pull waiting on its consumer, both on feeds opened after a full restart and on feeds kept open across one. Feed now sets a 2s pull expiry. Adds restart tests for a feed opened after a full-cluster restart and one open across it, each replaying a shared-shard backlog then a live append and failing on any 5s gap. The in-process cluster keeps client ports across restarts so open connections reconnect, and can write node logs. * fix: read SQS message state from the stream leader GetState used nats.go's KV Get, which switches to a direct get whenever the bucket allows it. On a replicated bucket a follower can answer that from state it has not applied yet, so a record the leader had already deleted or purged could still be returned. Read the key's last message with a stream message get instead, which only the stream leader answers, and apply KV Get's delete and purge marker rules to the result. Bucket configuration is unchanged. Missing, deleted and purged records still return ErrNotFound; values, revisions, context errors and the API prefix or domain of the caller's JetStream context are preserved. * docs: narrow the SQS GetState leader-read comment to what it avoids * fix: enumerate SQS state keys from the stream leader Count and Namespaces found live state keys with a direct-get probe and a headers-only ordered consumer on the state bucket's stream. On a replicated bucket a follower can answer either from state it has not applied, so keys the leader had already deleted could still be counted. Walk the key prefix with stream message gets by next subject, which only the stream leader answers, through the stream's last sequence at the call. Delete and purge markers drop a key, as GetState reads them. This costs one leader request per record under the prefix. Bucket configuration, historical state shards, the message-body scans and the feed are unchanged. * test: keep SQS API recorder deliveries in arrival order The recorder fed API subjects and its sentinel to two async subscriptions, which nats.go delivers on separate goroutines, so drain could return before the last request was recorded. Both now share one channel that the connection fills in arrival order. * fix: read SQS body enumerations from the stream leader Namespaces and Scan read bodies through an ephemeral ordered consumer, which the server may place on a follower that has not applied a purge, so a deleted queue could still be listed or scanned. Each stored match through the bound is now a leader stream message get. That transfers every full body, where the consumer read headers only. * test: stop the SQS API recorder goroutine with its test The recorder ranged over a caller-owned ChanSubscribe channel, which nats.go never closes, so its goroutine outlived every test. Cleanup now unsubscribes both subscriptions, then signals the goroutine and waits for it to return; the sentinel send also watches that signal. * fix: recover a stalled SQS feed from its cursor An ordered feed was seen to idle with stored changes pending after a cluster restart. Next now waits at most an idle check on the consumer, then asks the stream leader whether a change follows the cursor; if one does, the consumer is deleted and reopened just after the cursor, so no change is skipped or repeated. The root cause of the stall is unknown. * fix: make closing an SQS feed terminal and safe against Next Close now marks the feed closed under a lock, cancels any in-flight Next, stops the iterator and deletes the consumer. A later or in-flight Next returns ErrFeedClosed and never opens a consumer; one opened by a recovery racing Close is deleted rather than adopted. Repeated Close returns nil. * fix: define SQS feed close in the store contract message.Feed now documents one Next reader with Close callable at any time and more than once, and message.ErrFeedClosed replaces the adapter's own error. Once Close has begun, Next returns ErrFeedClosed and withholds any change it had received; a feed from the last cursor still delivers it. Shared conformance cases and the reference fake enforce this. The JetStream feed's Close also waits, within a bound, for a consumer open already in progress, so a consumer created during Close is deleted before Close returns nil; past the bound, or if that delete fails, Close returns an error and a repeated Close waits again. * fix: track SQS feed consumer cleanup until Close succeeds Every consumer open and release is now tracked work recorded under the lock that takes the consumer, and a delete that fails is kept for Close to retry. Close returns nil only once no such work is in progress and no failed delete remains, waiting within a bound and otherwise returning an error; concurrent and repeated Close calls wait the same way. Next claims a change under that lock and advances its cursor there, so a change is either returned before Close begins or withheld for a feed resumed from the last cursor. A consumer create no longer inherits the caller's cancellation, which was seen to leave most cancelled creates on the server with no owner; it has its own timeout, and the name of one that fails is deleted and kept for Close to delete again. * fix: release SQS feed consumers opened after the caller's context ends Feed and recovery now check the caller's context before starting a consumer open and after it completes. A create that finishes after the context ended is released as tracked work and the context's error is returned, or ErrFeedClosed when Close won; no change is returned. The feed contract and adapter docs now describe cleanup as complete for consumers the feed knows of, with the server's one-minute inactive threshold removing one an ambiguous create lands after its last delete. * fix: report cancellation when an SQS feed open fails after it A first consumer open that fails after the caller's context ended now returns the context's error joined with the failure, so cancellation stays detectable. The feed contract's cancellation text is narrowed to opening and reopening, whose cleanup is best effort. * docs: describe the internal SQS message store and its limits DESIGN.md gains a section on the store-neutral SQS message store, its JetStream adapter and conformance suite: an internal primitive with no service wired to it, its shared stream and bucket layout, leader-directed reads and their full-body cost, feed recovery and its best-effort consumer cleanup, and the absence of end-to-end or multi-host validation. The Scan contract says metadata only, not headers only.	2026-10-29 08:43:00 +10:00
Josh Siviour	d5f4d48956	feat: serve the OCI referrers API for ECR repositories (#1319) ... * feat: record manifest subject and artifactType and return OCI-Subject on push * fix: resolve the manifest media type once so a parameterised Content-Type cannot misclassify an index * feat: index referrers per subject and serve them through the metadata service * feat: serve the OCI referrers API for ECR repositories * test: cover the ECR referrers route in the gateway authorization matrices * feat: delete a subject image's referrers with it * fix: satisfy nilerr in the referrers manifest parser * fix: return the manifest parse error and format the referrers code * docs: describe the ECR OCI referrers API and cosign support * docs: name oras cp and crane copy as the way to carry v3 signatures to an ECR mirror * docs: present oras cp and crane copy as one remedy for copied v3 signatures	2026-10-28 22:52:47 +10:00
Josh Siviour	acfa078586	fix: take an EKS control plane out of rotation when its gateway is down (#1318) ... * feat: add a gateway readiness endpoint to the EKS token webhook * feat: serve token webhook readiness beyond loopback and admit it to the control-plane SG * feat: health-check the EKS apiserver target group on the token webhook readiness port * style: order the strconv import * fix: export the readiness listener address to the EKS token webhook * docs: regenerate the ELB coverage README * docs: describe EKS control plane readiness and what happens when a gateway stops * test: range over SplitSeq in the webhook initd guard	2026-10-28 21:12:29 +10:00
Josh Siviour	17dd3f325b	fix: stop routing to unhealthy load balancer targets and apply health-check changes (#1317) ... * feat: add routableTargets helper that skips unhealthy targets and fails open * fix: drop unhealthy targets from the NLB nginx upstream, failing open when none are healthy * fix: re-render load balancer config when a target health state changes * fix: write load balancer config and heartbeat fields under CAS so a re-render is never rolled back * fix: write live load balancer modifications under CAS so they cannot roll back a re-rendered config * fix: let ALB backends fail open to unchecked backup servers when every target is down * test: cover NLB unhealthy-target routing and ALB fail open in the lb e2e suite * test: flip ALB backend health per instance and log how long each health phase takes * style: gofmt haproxy backend struct and the attributes fetch table * fix: re-render load balancer config when a target group's health check changes * fix: return the health-target marshal error instead of discarding it * test: mark the health-check target group create as an intentional create-path test * docs: describe target health, fail open and time to effect in the ELBv2 guide	2026-10-28 19:54:49 +10:00
Josh Siviour	d2b02414b5	feat: direct-boot microvm data volumes and refused hot-attach (#1316) ... * feat: render a data volume as an MMIO virtio-blk-device A package-private helper reuses the PCI volume device's id, drive, iothread, serial and stop-on-error options, without a PCI bus. * feat: cold-boot microvm data volumes on MMIO without a PCI hot-plug port microvm has no PCIe root ports, so its data volumes render as virtio-blk-device and take no hot-plug port; a recorded port is kept. NBD and host-block exports and q35 PCI allocation are unchanged. * feat: compose a launch-only microvm config with EBS data volumes directBootLaunchConfig copies a clean direct-boot base and appends the mounted data volumes after its devices, rejecting boot and EFI volumes and non-microvm bases. Neither input is modified, so repeated launches never accumulate entries or keep a stale endpoint. * feat: launch direct-boot microvms with mounted data volumes from a launch-only config startQEMU composes the direct-boot command from the saved base and one snapshot of the mounted requests, which also feeds the readiness probes, so the launched endpoint is the one checked and is never saved. * fix: refuse volume hot-attach on direct-boot microvms before any mount A microvm's MMIO bus has no PCIe root port to hot-plug into, so AttachVolume returns ErrHotAttachUnsupported before mounting the volume or changing instance or volume state. * fix: return UnsupportedOperation for microvm volume hot-attach AttachVolume's ErrHotAttachUnsupported is a caller-fixable refusal, so the attach/detach error boundary maps it to a 400 UnsupportedOperation instead of a generic server error. * docs: describe direct-boot microvm data volumes and refused hot-attach	2026-10-28 19:45:47 +10:00