mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-06 02:51:30 +00:00
cd2618bebf424b779b0a329de80821998a078055
495 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
00943e9789 | chore: update ui images | ||
|
Julian Sommer
|
29de13586f | chore: coverage | ||
|
Ben Duncan
|
69df9ea9bf |
feat(ui): console feedback v1 and the admin Nodes view (#1454)
* feat(ui): console feedback round: state tones, inline tags, filter bar and form fixes Settled states go green and transitions yellow with a spinner, status text is lowercase, tags are edited inline row by row, every list filters through one filter bar, long cells truncate with a tooltip and times are relative. EC2 launch takes a Name and offers ICMP echo, and leaves Placement Group out when no group is available. RDS create is reordered with engine tiles and window pickers. Route 53 records get bulk delete, a TTL select, value labels that follow the type, and NS. Tooltips use the popover tokens, and the console is sized over a 110% root. * feat(ui): size RDS instance classes and give each record value its own input An RDS class is a db.-prefixed EC2 type, so the class picker shows its vCPUs and memory in every option and under the field, as the EC2 launch form does, on create, modify and restore. A Route 53 record's values were one textarea, which let a line break into a value. Each value is now a single-line input with Add value; Enter starts the next one, a pasted list splits into rows, and CNAME takes exactly one. * fix(ui): open a DNS record for editing from the keyboard * feat: admin node inventory, topology, live host metrics and network test Each node collects its hardware facts as root at daemon start (dmidecode, ethtool, networkctl LLDP where available) into /etc/spinifex/node-facts, and the daemon serves them with its plane wiring over NATS. New admin-only actions, refused for every account but the system admin account regardless of policy: - GetNodeFacts: hardware, NICs, LLDP neighbours and planes per node - GetNodeMetrics: live CPU, memory and per-plane traffic for one node - StartNetworkTest / DescribeNetworkTests: iperf2 between nodes over a chosen plane, both directions, one test cluster-wide at a time, recorded in KV and converged asynchronously The iperf listener runs only for the duration of a run, without capabilities, on TCP 5001, which the peers firewall rule now allows. A run refuses any target outside the node's own plane subnet. The console gains a node panel (hardware, network, metrics, network test with a dial scaled to the port's line rate) and a topology view built with @xyflow/react. setup.sh installs dmidecode and iperf. * test: cover the node insight handlers and the facts file mode * feat: report every CPU flag in node facts * feat(ui): node table on the dashboard, every CPU flag and a readable topology The dashboard's nodes section is the Nodes page table, open by default, and a row opens that node. Ready reads green, capacity sorts by most available, and the processor's features tip lists every flag. Topology draws one row per port with switches in a muted card, highlights a node's links on hover, and offers a left-to-right layout beside top-down. * chore(ui): rebuild dist * fix: align the CPU facts fields * fix: bound the network test server start and read ARM CPU features and cores * chore(ui): rebuild dist * test(ui): pin the instances baseline clock and take the screenshots from CI's Playwright image * fix: retry a network test claim whose holder released it mid-check |
||
|
Josh Siviour
|
8e3ccf45c4 |
feat(eks): give clusters one AWS-style cluster security group (#1445)
* test(eks): specify the AWS-style cluster security group layout * feat(eks): give clusters one AWS-style cluster security group Create the eks-cluster-sg-<name>-<id> SG in the customer VPC with the EKS tags and a self rule, report it as clusterSecurityGroupId, launch workers with it (or the launch template's SGs), and attach it to the private-endpoint ENI. New clusters no longer get customer-VPC control-plane and nodegroup SGs; clusters created before keep theirs, and delete finds the new SG by tag. * fix(eks): own the cluster SG by its recorded id and scope the delete sweep Workers and teardown use the SG id recorded on the cluster; a create retry only adopts an SG carrying tags the service alone can set. DeleteClusterSGs now revokes every rule only on cluster-owned SGs and, on any other SG in the VPC, only the rules that reference one. * docs(eks): use the cluster security group in workbooks and e2e Workbooks read vpc_config[0].cluster_security_group_id and drop the hand NodePort rules. E2E covers an internet-facing instance-target ALB with no hand rule, exec and logs, and a bystander security group surviving DeleteCluster. * docs(eks): drop stale node-group SG sentence |
||
|
Josh Siviour
|
a293d4e983 |
feat(iam): write TERMINATING only under a fenced deletion operation (#1442)
* refactor(cli): delete accounts only through the admin API * docs: describe account delete as admin API only * feat(iam): quiesce an account under a fenced deletion operation * fix(iam): keep the termination capabilities off the shared interface and fence the account removal * test(awsgw): guard the IAM termination binding on the operator deps * docs: say which credentials the account delete signs with |
||
|
Julian Sommer
|
70f3543de7 | feat(iam): implement UpdateUser so a user can be renamed or moved to a new path (#1439) | ||
|
Josh Siviour
|
7b9260f8f7 |
feat: AWS-format EKS add-on versions, DescribeAddonVersions filters and declared refusal codes (#1426)
* test: pin AWS-format EKS add-on versions * feat: return EKS add-on versions in AWS format and accept them as input * test: DescribeAddonVersions query parameters, compatibilities and paging * feat: honour DescribeAddonVersions filters and report compatibilities * test: EKS parameter refusals must carry a code the operation declares * fix: EKS parameter refusals use a code the operation declares * docs: describe AWS-format EKS add-on versions Document the v<x.y.z>-eksbuild.<n> form, the bare-form leniency, the DescribeAddonVersions filters and compatibilities, and how to pin a version in Terraform. Replace the leave-addon_version-unset guidance. |
||
|
Julian Sommer
|
0eeed40cb6 | chore: cleanup oracle linux install doc | ||
|
Julian Sommer
|
a6459ba7e4 | fix: default deploy-vultr.sh to the latest release channel | ||
|
Julian Sommer
|
805a71a63c |
fix: run the setup.sh shipped in the downloaded tarball
install.mulgadc.com serves the latest release's setup.sh, so a --channel dev or --version install ran an installer older than the tarball it unpacked and missed files it did not know about, such as lib/distro.sh. setup.sh now hands off to the tarball's own setup.sh, with the same arguments, before it changes anything on the host. The Vultr guide now installs from the latest release instead of dev. |
||
|
Julian Sommer
|
581be4067a | chore: use default base-ui for cmdk search | ||
|
Josh Siviour
|
8fa33f07ce |
feat(eks): inject IRSA credentials into pods of annotated ServiceAccounts (#1396)
* test: pod identity webhook mutation, publisher and webhook config contracts * feat: pod identity webhook mutator, CA bundle publisher and webhook configuration * feat(eks): run the pod identity webhook on every control-plane server Add the eks-pod-identity-webhook binary and its OpenRC service. It listens on 127.0.0.1:8444 with a serving certificate signed by the k3s server CA, authenticates to the API server with a client certificate signed by the k3s client CA, and runs under its own least-privilege ClusterRole. Each server stages the RBAC and the MutatingWebhookConfiguration (caBundle is the server CA) into the k3s manifests directory, so server-join nodes get them too. Match upstream's container skip rule: skip only when the regional STS key is present, and add the token mount otherwise. * fix(eks): issue pod identity webhook certs for one year and renew in the running process Both the serving and client certificates were valid for ten years. Kubernetes cannot revoke a client certificate, so a copy taken off a control-plane disk would stay a valid identity for a decade. Issue both for one year with the existing 30-day renew window. The process loaded the serving certificate once. A daily check now exits with ErrCertRenewal when either certificate enters the renew window, so the supervisor restarts it and EnsureCerts reissues. Add tests for KubeClient against a TLS fake API server and for Run, which the package had none of. * test: e2e covers pod-identity webhook and unmodified LBC chart IRSAPod now uses a ServiceAccount annotation only and asserts the injected env and volumes plus an untouched unannotated pod. LBCChart installs the upstream chart 1.11.0 and reconciles an Ingress into an active ALB. * test: pass the tagging endpoint to the LBC chart and mark Shield/WAF flags Spinifex-only * fix: make the pod identity webhook init script executable in the eks-node image setup.sh chmods each shipped init script to 0755 explicitly, and the new service was missing from the list. rc-update then refused it, and eks-node-role, running under set -eu, aborted before starting k3s. Add a guard test that every init script in INSTALL_FILES is chmod'd. * fix: start the pod identity webhook last and keep its failures non-fatal eks-node-role runs under set -eu, so a fault enabling or starting the optional webhook aborted the control plane boot before k3s-first-boot, state-report and addon-sync. Enable and start it at the end of the server and server-join roles and log a warning on failure instead. The role test now asserts the webhook is the last call and that an enable or start failure still exits 0 and finishes the role. * fix: move the pod identity webhook off the token webhook readiness port eks-token-webhook already serves its plain HTTP readiness endpoint on 8444, which the API server target group health-checks. The pod identity webhook bound the same port, failed with address already in use and respawned forever, so no pod was ever injected. Listen on 127.0.0.1:8445 and add a test that the two ports differ. * docs: describe pod identity webhook for user workloads in the EKS guide * fix: split STUB_FAIL assignment from export in the role test |
||
|
Ben Duncan
|
8cded67776 |
feat(eks): DeleteCluster returns DELETING and converges the teardown in the background (#1394)
* feat(eks): converge DeleteCluster's teardown behind the recorded DELETING
DeleteCluster ran the whole cluster teardown inside the NATS request. Each
control-plane VM that ignores ACPI costs 40 s (20 s graceful powerdown, then
20 s waiting for the PID file), so a two-member cluster overran the 30 s
budget: the caller got a 500, and its retry queued behind the first handler
on the serial subscription and got a 404.
The request now validates, answers a DELETING cluster from its record,
records DELETING and returns. The teardown runs in a background converger
started by the request and backed by the deleting reaper:
- An in-process per-cluster gate sits in front of the teardown lease, which
re-adopts a key its own node holds; one per-node bound covers kicks and
reaper hand-offs.
- The lease renews on the service context, not the request's, and the
converger stops at a step boundary if it is lost.
- The converger acts only on a record that is DELETING when re-read, refuses
to remove a record that still has a node group, deletes the cluster's
child records before the meta, and deletes the meta only if no ownership
ref appeared that its pass did not tear down.
- DELETING is sticky for every status writer, and the create launch's meta
writes add their refs to a DELETING record instead of overwriting it, and
never recreate a swept one.
- The reaper hands work to the converger instead of running it inline,
retries at a backoff capped at 30 minutes without ever abandoning a
record, and resumes records an older build flagged exhausted.
- A stuck delete is surfaced in DescribeCluster as one INTERNAL_FAILURE
issue naming the step, never the internal error.
* fix(eks): refuse DeleteCluster while node groups are attached
AWS answers ResourceInUseException ("Cluster has nodegroups attached") when
managed node groups remain. DeleteCluster went ahead instead, and its sweep
erased the node group records, leaving their workers running in the
customer account with no record to own them and no control plane.
The request now decides that error before recording anything. The e2e
suites, the terraform workbooks and the account teardown engine all delete
node groups first; the console shows the refusal's message.
* fix(eks): refuse new add-ons and access entries on a deleting cluster
CreateAddon and CreateAccessEntry checked only that the cluster record
existed, so a dependant could be written to a cluster whose delete was
already recorded, and the teardown would then sweep it with the record.
Both now answer ResourceInUseException for a DELETING cluster, a code both
operations declare in the EKS model.
* test(e2e): DeleteCluster returns DELETING at once and never reads ACTIVE after
The first delete-cluster must return within 5 s with status DELETING, as
AWS does. While the cluster is still listed every DescribeCluster must read
DELETING, and a repeat delete must answer DELETING from the record; once it
is gone a further delete answers ResourceNotFoundException.
* docs(eks): record the cluster delete lifecycle contract
The contract ADR-0003 S1 asks for before a conformance claim: what the
request decides, the recorded obligation, how the teardown converges and
recovers, and the S8 and S7 tests that pin each item, with the remaining
gaps listed.
* test(eks): wait for the delete request before counting terminates
On a build whose DeleteCluster is synchronous the request goroutine is still
recording terminates when the assertions read, so the tests must wait for it
to return as well as for the background teardown.
* test(eks): classify every eks request subject by its handler's work
The ADR-0003 S8 subject table for EKS. Every eks.* subject the gateway sends
and the daemon serves is classified in eksSubjects, and the table fails for
one that is not, so a new subject is classified deliberately rather than by
inheriting the 30 s default. DeleteCluster is request-scaled. Four handlers
still hold the request for work that grows with the resource and are named
as not conformant: CreateCluster's FAILED-name reclaim, DeleteNodegroup,
UpdateNodegroupConfig and RestoreSnapshot. Their budget stays 30 s, since a
larger one would hide the defect.
* fix(eks): leave no child record behind a launch overtaken by a delete
A create launch still running when its cluster was deleted and swept could
seed the creator's access entry afterwards, leaving it under a cluster with
no record. The launch now re-checks the record after seeding and, finding it
gone, removes the cluster's child keys with the infra it holds. The teardown
also sweeps child keys again after deleting the meta, for a write that lands
between its two deletes.
* fix(eks): fence the create launch on its generation and keep its infra owned
Review findings on the async delete, each with a test that fails before:
- The launch's meta write checked only the status, so a launch overtaken by
a delete could overwrite a newer cluster of the same name and then release
by name. Writes are now fenced on CreatedAt and the ARN; a superseded
launch releases only the instances it started, by ID.
- A launch whose record was swept released its infra once, best effort. It
now recreates the record as DELETING with its refs, so the teardown and
the reaper own and retry the release.
- A record marked FAILED under a running launch was treated as owned by a
teardown; the launch now merges its refs and releases what it built.
- Child records: a fresh claim clears keys an earlier cluster of the name
left, the teardown stops the cluster's bootstrap subscriber, bootstrap
refuses to write into a record that is gone or DELETING, the creator's
access entry and the GPU device add-on are gated on the record.
- Deleting the last node group of a DELETING cluster resumes its delete,
and waiting on node groups no longer counts as a failed pass.
- One converger run is bounded at 15 minutes, so a hung step cannot hold
the gate and lease for ever; the timed-out pass is recorded.
- The FAILED-name reclaim purge runs under the teardown gate and lease.
- The meta-last test drives the converger with a failing KV instead of
DeleteClusterPrefix, which production no longer called and is removed.
- The S7 test asserts the teardown issues no volume delete and terminates
no worker.
* fix(eks): keep a deleted cluster's record while its create launch runs
Second review of the async delete, each behavioural change with a test that
fails before:
- A create launch now heartbeats its owner and time on the record, and the
teardown keeps the record while that heartbeat is fresh, so DescribeCluster
never answers NotFound and then DELETING again when an overtaken launch
recreates the record. The launch hands its refs over at its next write and
starts the teardown when it returns; a heartbeat silent for 90 s is taken
as a crashed launch, and recreating a swept record stays as the backstop.
- The launch's OIDC keypair writes are fenced on the record's generation, so
an overtaken launch cannot replace a newer same-name cluster's signing key.
- Re-creating a FAILED cluster's name is no longer refused by the per-node
teardown bound.
- A deadline reached while removing the record is still recorded; an exited
bootstrap subscriber removes its own entry; a record that is not DELETING
is refused as not recorded even when it has node groups.
|
||
|
Josh Siviour
|
671a8a9ff5 |
feat(sqs): fresh single-node SQS in one system microVM (#1389)
* feat(sqs): run queue and message owners in the worker by signed pull The SQS worker's ready step now opens the queue and message owners on its dedicated store and serves owner calls it pulls from its node's gateway over the existing instance-role channel (PollWork, CompleteWork). Each poll and post is authorized against the current worker identity, and a result is accepted only from the identity that took the work. No payload crosses platform NATS. A receipt-signing key is created on the worker disk only under the first-store decision that creates the store key, and its fingerprint is bound in the store; a missing or changed key refuses to serve. Received messages are released only when the gateway proves the caller left; receives are capped so a worst-case result fits one post. The public relay is not registered in production. * fix: harden SQS relay results and bound pending bytes Unavailable outranks client faults when encoding owner errors, and wire errors carry fixed text only, never owner or backend text. The ingress rejects incomplete or mismatched success results as unavailable, and the relay bounds total pending input bytes as well as request count. Adds tests of every relay action against real owners, the receive claim limit, release, faulting work, the receipt-key fingerprint marker and the poll loop through gateway failures. * fix: refuse SQS serving outside a single-node host cluster DescribeStore now reports the gateway's declared cluster node count, and the worker refuses to serve, before reading keys, connecting or creating any store state, unless that count is exactly one. A gateway that reports no count reads as zero and is refused. * feat(sqs): record a store server key pin once per store instance Add the store role and its internal actions so the bound store instance can learn its claim and record the public pin of its TLS server key, once, without the platform holding the private key. The worker sees the store as bound only after a pin exists and must receive a valid pin. * refactor(sqs): serve store server keys from a separate capability Keep the worker Stores interface unchanged and give the store guest's actions their own ServerKeys dependency, so existing Stores fakes need no store-key methods. * feat(sqs): launch the store with a mgmt NIC, CA and scoped store role The store now boots with the region, gateway URL and the daemon's own cluster CA, a mgmt NIC to reach the gateway, and an sqsStoreRole profile granted only the store's two internal actions, so it can authenticate its server key pin. A store record that predates these fields refuses to relaunch; its volume is kept. * feat(sqs): keep the store TLS server key on its disk and record its pin The store image gains sqs-store-key, run after the data disk is mounted. It loads or creates a P-256 server key under the disk's tls directory, records only the key's public pin with the gateway as the bound store instance, and writes a self-signed certificate for it to the run directory. nats-server then requires TLS 1.3 with that key. A record for another volume, a recorded pin with a missing key, or a key that does not match the recorded pin refuses the store; no key is regenerated and nothing on the disk is deleted. Launch-not-yet-bound and lost replies are retried. The store role vocabulary moves to contracts/sqs/v1 so the guest agent does not import a domain. * feat(sqs): pin the worker's store connection to the recorded server key The worker's readiness check and serving connection now use TLS 1.3 and accept only the store server key whose pin the bound store recorded with the gateway. The pin is checked before the worker key or any queue or message data is sent, and on every reconnect. A store with another key, a plaintext store or a malformed pin is refused as final. The worker NKey client authentication is unchanged. The host still holds the store's data disk key, so the pin does not protect the store from a privileged host. * fix(sqs): retry store key bootstrap and track guest binary sources The store's server key check moves from sysinit into the respawned run step: a gateway or IMDS outage exits after a pause so init retries it, without remounting or creating a new key, while a missing, changed or wrong-store key is recorded as refused and never retried. Stop also ends a running key helper before unmounting. The guest helper and worker binaries now depend on every main-module package dir and file they link, from go list, so edits, additions, deletions and renames all rebuild them. * feat(sqs): operator enable and status with a signed ingress test Add a per-Region SQS service record (disabled by default, pinned to one node, refused on multi-node clusters before any write), gate the relay on it, and expose `spx admin sqs enable` and `spx admin sqs status`. The gateway now hands queued work to the polling worker and reports whether one is polling. The public SQS service stays unregistered in production until its capability profile is reviewed. An integration test registers it through the harness and drives real SigV4 and IAM requests to a real worker for create, send, receive and delete, asserting no SQS body crosses platform NATS. * fix(sqs): report readiness only from observed component state Status now reports whether the tenant SQS API is registered on this gateway and is never Ready while it is not. The worker stops polling while its store connection is down, so a worker that lost its store drops out of the gateway's poller window instead of looking healthy. * wip: sqs model inventory * wip: sqs public registration * wip: sqs registration review fixes * fix(sqs): refuse pre-API store and worker records on enable A store or worker record this release creates now carries an explicit contract. enable reads both records without creating a bucket and refuses any record without it before it writes desired state or launches anything; status names the record and is never Ready. A worker replacement keeps its predecessor's contract, since it reuses the same data volume. Missing metadata is only ever grounds to refuse: nothing is upgraded, inspected, emptied, replaced or deleted. Upgrading a pre-API dev install is not supported. * fix(sqs): describe the enable refusal as an unsupported launch contract A missing or unknown contract proves neither when a record was written nor that it predates authenticated bootstrap: an older writer can drop a newer field, and a future contract is not pre-API. The refusal is unchanged; its message, status field and docs now name a missing or unsupported launch contract and say any existing volume is kept. * wip: port reviewed SQS serving path onto dev without the store VM * wip: single SQS system VM * fix(sqs): retry receipt bootstrap under supervision and drop store install path Run the receipt-key boot step from the respawned store service so a transient gateway or IMDS outage recovers without a reboot; final key and volume refusals stay refused. Remove the separate store image install and staging path from setup. * fix(sqs): align worker ready config fields * fix(sqs): stop an in-progress boot before unmounting the worker disk The receipt-key boot step ran as an unrecorded child of the store shell, so shutdown stopped only the shell and could unmount the disk under a boot still talking to the gateway. Record the boot child like the other guest processes and stop it before unmounting; a survivor keeps the disk mounted. * fix(sqs): record the boot child before its stopping check The parent wrote boot.pid only after starting the child, so a shutdown in that gap could unmount under an unrecorded boot. The child now records itself and checks stopping before it runs sqs-worker boot. |
||
|
dependabot[bot]
|
bbd4e84f15 |
chore: bump the docker-images group across 3 directories with 3 updates (#1375)
Bumps the docker-images group with 1 update in the /build directory: golang. Bumps the docker-images group with 2 updates in the /docs/terraform-workbooks/demo-app directory: golang and distroless/static. Bumps the docker-images group with 1 update in the /scripts directory: debian. Updates `golang` from `972b411` to `d9b6a1b` Updates `golang` from `f2c872b` to `d200d2d` Updates `distroless/static` from `18f4cc2` to `25ac8f0` Updates `debian` from `62bd423` to `f6b08b0` --- updated-dependencies: - dependency-name: golang dependency-version: 1.27.3-bookworm dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: golang dependency-version: 1.27.3-alpine dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: distroless/static dependency-version: nonroot dependency-type: direct:production dependency-group: docker-images - dependency-name: debian dependency-version: trixie-slim dependency-type: direct:production dependency-group: docker-images ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Josh Siviour
|
30b8c24776 |
feat(accountlifecycle): seal quota overrides during account teardown (#1390)
* feat(admission): add the quota override seal capability contract * feat(awsgw): serve the quota override seal capability per node * refactor(accountlifecycle): parameterise the seal capability probe and add the quota one * test(accountlifecycle): pin the probe wire shape against both capability contracts * feat(accountlifecycle): seal the quota override record during account teardown * docs: record the quota override seal contract and teardown behaviour * docs: fill in the quota override seal PR number in the package boundary migration |
||
|
Ben Duncan
|
6c3d6c6be3 |
fix(setup): keep NetworkManager from replacing the resolved stub link (#1391)
NetworkManager restarted while /etc/resolv.conf is a plain file runs in rc-manager=symlink mode and writes resolv.conf on its first DNS update, which can land after the installer links the stub. Write a conf.d drop-in (dns=systemd-resolved, rc-manager=unmanaged) before the restart, refuse a later file that overrides rc-manager, wait for NetworkManager to settle, and assert the link holds with bounded relinks. A rerun after an older handover adds the drop-in, reloads NetworkManager and relinks the stub. |
||
|
Josh Siviour
|
2466a9c8af |
fix: stamp the ENI owner in the attach write and release a failed LB's VM (#1388)
* test: pin the ENI owner stamp and failed-LB cleanup contracts Failing tests: an ENI update after the caller's own write must survive a lagging replica read, an unwritable owner stamp must fail the system launch, and the reaper must terminate the failed LB's own VM only. * fix: stamp the ENI owner in the attach write and release a failed LB's VM A lagging-replica read before the owner stamp's compare-and-swap lost the stamp, leaving a system VM that IMDS could never resolve. - AttachENIWithOwner records the instance owner in the attach write. - ENI read-modify-writes re-read and re-apply on a revision conflict, bounded, and report the store error to the caller's log. - A system launch whose ENI cannot be attached or stamped fails and cleans up the VM. - The reaper that marks an LB failed terminates that LB's own VM and detaches its ENIs. * fix: log ENI write retries and exhaustion An Info line on a write that succeeds after a revision conflict and an Error line when the bounded retries run out, so the sink can count both. * test: cover an internal ALB in a private subnet and report the ENI attachment owner DescribeNetworkInterfaces now returns Attachment.InstanceOwnerId when the record has one, as AWS does, so the owner stamp is observable. The e2e launches an internal ALB in a VPC with no internet gateway and checks the owner, the private address and in-VPC traffic. The ENI write error log now reports the real attempt count, so a non-conflict store error is not counted as an exhausted retry. * revert: stop reporting the ENI attachment owner through Describe AWS always returns Attachment.InstanceOwnerId, so reporting it only for system-owned ENIs adds a divergence. The e2e now relies on the LB reaching active with a healthy target, which needs the owner stamp to resolve. |
||
|
Josh Siviour
|
f37776384c | docs: fill in the 11d-2 PR number in the package boundary migration (#1386) | ||
|
Josh Siviour
|
ea1ca38c52 |
fix: boot LB and SQS worker microvms with PCIe NICs (#1385)
* test: direct-boot LB and SQS worker NICs must be virtio-net-pci on microvm pcie=on * fix: boot LB and SQS worker microvms on PCIe with virtio-net-pci NICs virtio-mmio devices share level-triggered IRQ lines and a NIC's RX path stalled on them. The direct-boot machine gets pcie=on, so NICs are virtio-net-pci with MSI-X, and the virtio_mmio.device= kernel args go. IsMMIO is now the transport check (false with pcie=on) and IsMicrovm the machine check. Data volumes keep their serial and ID on virtio-blk-pci. The LB guest console loglevel is raised to 7. * refactor: trim microvmMachineType comment * docs: describe direct-boot microVM NICs and data volumes on PCIe |
||
|
Josh Siviour
|
8add7c03b3 |
refactor: move the gateway into ingress/aws/pipeline and delete it (11d-2) (#1384)
* refactor: move the gateway into ingress/aws/pipeline and retire the gateway layering rows The gateway's request pipeline joins the packages it already consumed. Operation inventory moves to ingress/aws/dispatch. Files keep their names; only the package line, the import paths and the now-redundant pipeline qualifier change. The layering rows for the gateway root are removed: the package is under ingress, so R2 binds it, and the daemon-direction test now asserts R2 rejects runtime, operator and domain imports. An empty registry_helper_test.go was dropped rather than moved. * refactor: rename pipeline.GatewayConfig to pipeline.Config * docs: update the design and package boundary docs for the gateway move * docs: point the EKS add-on lifecycle evidence at the tests' current files |
||
|
Ben Duncan
|
34f7b4d8e5 |
fix(vm): reboot by relaunching QEMU after a clean guest shutdown (#1383)
RebootInstances powered the guest down under -action shutdown=pause and then ran system_reset + cont in the same QEMU process. On some hosts the reset left the guest silent in firmware with QMP still answering, while a fresh QEMU on the same volumes always booted. A reboot now shuts the guest down over ACPI (forcing it off after 60s, as before), ends that QEMU, and starts a fresh one on the same mounted volumes, taps, IMDS datapath and identity, the way nova's soft reboot and Proxmox's qm reboot do. The instance stays running in Describe, no hooks fire, the console log is appended rather than truncated, and the existing QMP client is reconnected in place so the heartbeat carries on. A relaunched guest that writes nothing to its console within 60s has its run state, vCPUs, vCPU thread states and registers (sampled twice) logged, and if it has printed before, its QEMU is replaced once more. A relaunch that cannot start goes to crash recovery. Stop, terminate and hot-plug wait for a relaunch in flight so volumes are never sealed under a process being started. Guest-initiated reboots stay an in-process reset. The daemon now logs the host kernel, QEMU version and the OVMF package and checksum at start, and every reboot relaunch carries them. |
||
|
Ben Duncan
|
956b830f18 |
fix(firewall): keep conntrack running while the host policy is disarmed (#1379)
The kernel tracks connections only while some rule uses conntrack. With inet spinifex_filter unloaded nothing tracked, so OVN RAFT links formed while disarmed (or before the daemon's first arm) were first seen mid-stream at arm time. With tcp_loose a packet from the server end became a new flow from peer:6643/6644 to an ephemeral port and hit the drop rule, and RAFT lost its leader. Add table inet spinifex_conntrack (accept-only) and keep it loaded on every node: setup.sh loads it at install, the boot unit loads it unconditionally, firstboot loads it before OVN, and disable swaps the policy for it in one transaction instead of turning conntrack off. |
||
|
Josh Siviour
|
f0a704cfcd |
refactor: move the gateway's public routes, admin wiring and unread config out (11d-1) (#1378)
* test: pin the order of the unauthenticated mounts against SigV4 and the throttle
* refactor: mount the OIDC and registry routes through a public-routes hook
* refactor: take the admin registration name and request-id context as gateway config
* refactor: move node discovery out of the gateway into the awsgw role
* refactor: drop the gateway config fields production never reads (Config, RootCAs, InternalSuffix)
* refactor: drop the gateway Version and Commit config fields
* refactor: compose the production gateway config in one tested function
* refactor: drop the gateway IAMService, STSService, NATSConn and ExpectedNodes config fields
Production never reads them: composeGateway builds Policy and Readiness from
its own inputs. Tests that used the config as a carrier now keep those
collaborators in gwfix.Deps, recorded per config with gwfix.WithDeps and read
back with gwfix.DepsOf. Gateway non-test code now imports nothing under
domains/, runtime/ or operator/.
Test edits are setup only: 153 GatewayConfig literals in 45 files had the four
keys moved into gwfix.WithDeps(..., gwfix.Deps{...}), and 113 field reads
became gwfix.DepsOf(x).{IAM,STS,NATS,Nodes}. No assertion changed.
Script 1, literal rewrite (balanced-brace scanner, keys moved out of the
literal):
import re,sys,subprocess,os
KEYS={'IAMService':'IAM','STSService':'STS','NATSConn':'NATS','ExpectedNodes':'Nodes'}
def scan_close(s,i):
# s[i]=='{' ; return index of matching '}' ; skips strings, runes, comments
d=0;n=len(s);k=i
while k<n:
c=s[k]
if c=='"':
k+=1
while s[k]!='"':
k+= 2 if s[k]=='\\' else 1
elif c=='`':
k=s.index('`',k+1)
elif c=="'":
k+=1
while s[k]!="'":
k+= 2 if s[k]=='\\' else 1
elif s.startswith('//',k):
k=s.index('\n',k)
continue
elif s.startswith('/*',k):
k=s.index('*/',k)+1
elif c in '{([': d+=1
elif c in '})]':
d-=1
if d==0: return k
k+=1
raise Exception('unbalanced')
def split_top(body):
parts=[];d=0;start=0;k=0;n=len(body)
while k<n:
c=body[k]
if c=='"':
k+=1
while body[k]!='"': k+= 2 if body[k]=='\\' else 1
elif c=='`': k=body.index('`',k+1)
elif c=="'":
k+=1
while body[k]!="'": k+= 2 if body[k]=='\\' else 1
elif body.startswith('//',k):
k=body.index('\n',k); continue
elif body.startswith('/*',k): k=body.index('*/',k)+1
elif c in '{([': d+=1
elif c in '})]': d-=1
elif c==',' and d==0:
parts.append(body[start:k]); start=k+1
k+=1
parts.append(body[start:])
return parts
changed={}
files=[l for l in subprocess.check_output(['git','ls-files','*.go']).decode().split() if os.path.exists(l)]
SKIP={'spinifex/gateway/gateway.go','spinifex/runtime/roles/awsgw/compose.go','spinifex/runtime/roles/awsgw/awsgw.go'}
lit=re.compile(r'(&?)((?:gateway\.)?GatewayConfig)\{')
total=0
for f in files:
if f in SKIP: continue
s=open(f).read()
if 'GatewayConfig{' not in s: continue
out=[];pos=0;cnt=0
for m in lit.finditer(s):
if m.start()<pos: continue
o=m.end()-1
c=scan_close(s,o)
body=s[o+1:c]
parts=split_top(body)
keep=[];moved=[]
for p in parts:
km=re.match(r'\s*(IAMService|STSService|NATSConn|ExpectedNodes)\s*:\s*(.*)$',p,re.S)
if km: moved.append((KEYS[km.group(1)],km.group(2).strip()))
else: keep.append(p)
if not moved: continue
# rebuild
kb=','.join(keep)
if kb.strip()=='' : kb=''
elif not kb.rstrip().endswith('\n') and '\n' in body: pass
d=', '.join(f'{k}: {v}' for k,v in moved)
new=f'gwfix.WithDeps({m.group(1)}{m.group(2)}{{{kb}}}, gwfix.Deps{{{d}}})'
if not m.group(1): new=f'gwfix.WithDeps(&{m.group(2)}{{{kb}}}, gwfix.Deps{{{d}}})' # value literals flagged below
out.append((m.start(),c+1,new)); pos=c+1; cnt+=1
if not m.group(1): print('VALUE LITERAL',f,s.count('\n',0,m.start())+1)
if not out: continue
r=[];last=0
for a,b,n in out:
r.append(s[last:a]);r.append(n);last=b
r.append(s[last:])
s2=''.join(r)
if 'testkit/gwfix"' not in s2:
s2=re.sub(r'import \(\n','import (\n\t"github.com/mulgadc/spinifex/internal/testkit/gwfix"\n',s2,1)
open(f,'w').write(s2); total+=cnt; changed[f]=cnt
print('literals rewritten',total,'files',len(changed))
Script 2, read rewrite (driven by the compiler's "has no field or method"
errors for the type, so only GatewayConfig receivers are touched):
import re,subprocess,sys,collections
M={'IAMService':'IAM','STSService':'STS','NATSConn':'NATS','ExpectedNodes':'Nodes'}
def run():
env=dict(__import__('os').environ,GOWORK='off')
out=''
for tags,pk in [('',['./spinifex/gateway','./spinifex/runtime/roles/awsgw/...']),('integration',['./tests/integration/...'])]:
cmd=['go','test','-count=1','-run','NONE_X','-gcflags=-e']+(['-tags',tags] if tags else [])+pk
r=subprocess.run(cmd,capture_output=True,text=True,env=env)
out+=r.stdout+r.stderr
return out
fixed=0
for it in range(8):
out=run()
errs=re.findall(r'^(\S+\.go):(\d+):(\d+): (.*?) undefined \(type (\*?(?:gateway\.)?GatewayConfig) has no field or method (\w+)\)',out,re.M)
if not errs:
print(out[-3000:]); break
byfile=collections.defaultdict(list)
for f,l,c,expr,t,fld in errs: byfile[f].append((int(l),int(c),expr,fld))
for f,es in byfile.items():
lines=open(f).read().split('\n')
for l,c,expr,fld in sorted(set(es),reverse=True):
line=lines[l-1]
# expr like 'gw.IAMService' or 'x.y.IAMService'
new=f'gwfix.DepsOf({expr[:-len(fld)-1]}).{M[fld]}'
if expr not in line: print('MISS',f,l,expr); continue
lines[l-1]=line.replace(expr,new)
fixed+=1
s='\n'.join(lines)
if 'testkit/gwfix"' not in s:
s=re.sub(r'import \(\n','import (\n\t"github.com/mulgadc/spinifex/internal/testkit/gwfix"\n',s,count=1)
open(f,'w').write(s)
print('read sites fixed',fixed)
* docs: record the gateway mount hooks and config-field removal in the package boundary migration
* docs: fill in the 11d-1 PR number in the package boundary migration
|
||
|
Josh Siviour
|
1cfaf15c9d |
fix: EKS control-plane VMs are not Kubernetes Nodes (#1373)
* test: pin control-plane no-agent config, etcd member removal and CP isolation e2e * test: tighten CP isolation e2e and assert a logged prune failure * fix: run EKS control planes without a k3s agent and prune etcd members directly Control-plane VMs no longer register as Nodes, so pods cannot schedule onto a network that cannot reach the workers or the advertised endpoint. With no Node to delete, a replaced peer's etcd member is removed by a new eks-etcd-member helper that matches exactly one member by peer-URL host. * fix: refuse to remove the local etcd member by ID, not only by IP Self-protection no longer depends on first boot deriving its own address: the helper asks the connected endpoint for its member ID and refuses a match on it. * test(e2e): align EKS suite with control planes that are not Nodes Assert zero Nodes and delivered addon objects before any nodegroup. Check Ready worker, metrics APIService, addon deletion and pod logs once a worker exists, scoping logs to Ready nodes and waiting 10m for metrics-server to reschedule. * docs: EKS control-plane hosts are not Kubernetes Nodes Document that kubectl get nodes lists workers only, add upgrade notes for existing clusters, and list the eks-etcd-member helper. * docs: explain why addon manifests keep the CriticalAddonsOnly toleration Comment-only. The toleration stays for clusters created before control-plane hosts stopped registering as Nodes. * test: cover the etcd member client and CLI against a TLS fake etcd The real etcd client and the command had no tests, leaving the PR below the diff coverage threshold. A fake etcd serving the Cluster and Maintenance APIs over mutual TLS drives the client with the K3s certificate layout, and main becomes a testable run function. |
||
|
Ben Duncan
|
e3a44893fa |
fix: bound the console's instance-type payload and stop polling it (#1371)
* fix: bound the console's instance-type payload and stop polling it DescribeInstanceTypes with capacity=true returned one entry per free slot per node: 3,528,765 bytes of XML for 50 distinct types on prod, refetched every five seconds. The response was sized by free capacity, so the emptier the cluster the larger it got, and at 44,346 slots it exceeded the NATS maximum payload and the console reported no compute at all. The listing now names each admissible type once. The free-slot figure crosses as a number through a new GetInstanceTypeCapacity action, open to any account because the launch form a tenant uses is what needs it. The catalogue query no longer polls, and the EKS and ECS dialogs read it only when open. The gateway negotiates gzip inside the SigV4 group, which excludes the OCI Distribution registry structurally. A response is compressed only if the client offered gzip, the handler declared neither a length nor an encoding, the content type is neither streaming nor already compressed, the status carries a body, and 1 KiB was written before any flush. An unknown capacity figure no longer caps the count at 1. The launch decides, and RunInstances answers InsufficientInstanceCapacity if it cannot. * test: read the free slot count from its own action in the console e2e Two e2e tests encoded capacity as repeated instance-type entries, so one expected a caption the deduplicated listing can no longer produce. The fixture answers GetInstanceTypeCapacity instead, which lets a test give the catalogue and the figure different answers. A third test covers the case the change introduces: a session whose policy does not grant the action reads no figure, so the form offers the type with no capacity caption and leaves the launch enabled. The committed bundle is rebuilt, which rehashes every chunk reaching the two query modules. |
||
|
Ben Duncan
|
9c99312f74 | feat(ui): enforce ADR-0016 with guard tests and move dialog forms into FormPanel (#1369) | ||
|
Ben Duncan
|
4468f13e53 |
docs: add a diagram of a Vultr deployment to the guide (#1367)
Shows deploy-vultr.sh building a private Vultr VPC with one node or three, the nodes clustering over the private network for the control plane, EBS and S3, Spinifex calling the Vultr API for reserved IPs, and the AWS services running across the fleet. |
||
|
Ben Duncan
|
8a71854463 |
docs: tidy the Vultr guide for publication (#1365)
Drops the horizontal rules the docs site rendered as stray text, links the AWS CLI install page, captions each screenshot, shows the expected spx get nodes output for three nodes, explains what a three-node deployment protects against, and simplifies the limits to the sizing and quorum a reader needs. |
||
|
Josh Siviour
|
e155bcf155 |
feat(sqs): queue and Lambda-facing message API foundation (#1364)
* feat(sqs): queue and Lambda-facing message API foundation Adds the SQS JSON 1.0 adapter (unregistered in production) with queue create, lookup, attributes, list, tags and the message operations a Lambda poller uses: SendMessage(Batch), ReceiveMessage with long polling and attribute selection, DeleteMessage(Batch) and ChangeMessageVisibility(Batch), backed by the queue record owner and the JetStream message store. String and Binary message attributes are validated, stored and checksummed; Number attributes are refused until AWS behaviour is confirmed. * test(sqs): poller lifecycle through the message owners Covers receive, release, redelivery, delete, visibility extension, queue binding of receipt handles and long polling against the real queue and JetStream message stores, from the message package itself. * test(sqs): move SDK coverage from the gateway to the SQS domain Drives the registration through dispatch.Invocation over the real queue and message owners, so the gateway package carries no SQS changes. |
||
|
Josh Siviour
|
7aa76c5bd6 |
refactor: gateway request record and capabilities (ADR-0002:S7, S8) (#1363)
* test: pin gateway pipeline order, request ids and region * refactor: carry request facts in one ingress request record * fix: seal the request principal once * refactor: build the caller ARN once in credverify * refactor: gateway takes policy and admission as capabilities * refactor: gateway takes cluster readiness as a capability * chore: gofmt GatewayConfig field alignment * docs: record slice 11a request record and capabilities * docs: add PR number to slice 11a migration row * refactor: move pipeline capability interfaces to ingress/aws/pipeline |
||
|
Ben Duncan
|
b4dd606104 |
feat: run Spinifex on Vultr, with failover, workbooks and the fixes they needed (#1361)
* feat: Vultr reserved-IP provider and floating external-address pool Adds a source="vultr" external pool backed by Vultr reserved IPs: an API client pinned by captured responses, an in-memory fake with the measured semantics, and a floating allocator that creates, claims, reconciles and releases labelled addresses. OCI and Vultr claims now share one AddressClaimer pass. Also fixes the host firewall dropping guest IMDS when imds_host_meta_ip is set, by matching the pre-DNAT destination, and arms the Spinifex firewall in place of an active ufw. * fix: detach a Vultr reserved IP before deleting it Vultr was measured refusing delete of an attached address with 400 "Unable to detach subnet", so release, rollback and reconcile now detach first. A first claim after allocation logs at info rather than warn. * fix: release the auto-assigned public IP an Elastic IP displaces AssociateAddress restamped the ENI with the EIP and never handed back the address launch had assigned. Nothing referred to it afterwards, so it stayed allocated for good, and on a cloud pool every associate left a billed address. AWS releases it on associate; withdraw its NAT rule and release it, scoped to the ENI, leaving any address another associated EIP holds alone. * fix: retry Vultr's refused detach on reserved-IP detach and delete Vultr has answered a detach with 400 "Unable to detach subnet" and accepted the same detach moments later. Surfacing it left the address for reconcile to collect, up to twenty billed minutes later; retry it like a 5xx instead. * fix: stop reporting the released launch address after an EIP is disassociated Associating an EIP now releases the auto-assigned public IP, but the stored instance kept its launch stamp, and DescribeInstances fell back to it once the EIP was disassociated. The public-IP change handler now moves the stamp with the runtime address. * feat: add a Vultr deploy script for a single-node Spinifex on a VX VM deploy.sh up/destroy/status drives Terraform for the VPC and VM, installs and forms Spinifex over SSH, copies the API key to the node rather than through user-data, and sweeps boot volumes and reserved IPs Terraform does not own. Cloud-init keeps needrestart off the Spinifex, OVS and OVN units and disables unattended upgrades on these disposable hosts. * feat: form a three-node Vultr deployment with install-node.sh over the VPC Each node comes up as an armed single node, the state install-node.sh forms from, with the cluster and Geneve planes on the VPC NIC. The pool and every node's provider_host_id go on each node after formation. * fix: read the pinned host key with ssh-keygen -F when forming a Vultr cluster ssh_config hashes known_hosts entries by default on Debian, so a grep for the address found nothing and set -e ended the run without a message. * docs: add the Vultr guide, one command to deploy one or three nodes and one to remove them * docs: drop the failover write note from the Vultr guide * fix: keep needrestart from restarting the Spinifex, OVS and OVN units Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's replaced libraries to the daemon and restarted it after every dpkg run. The restart never cleared them, so an unattended upgrade ran the unit into its start limit and left the control plane down. setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists them, so restarting after an upgrade stays the operator's choice. * chore: drop the Vultr cloud-init needrestart override now setup.sh writes it * docs: replace the Vultr guide's apt upgrade warning now the installer keeps needrestart off the stack * docs: install the Vultr guide's build from the dev channel until Vultr support is released * fix: answer ARP for Vultr reserved IPs on bare metal Vultr routes a reserved IP to a VM by MAC but ARPs for it on a bare-metal server's public segment, and a Vultr pool gave vpcd neither a gateway nor a bind bridge to proxy-ARP on, so nothing answered and no traffic arrived. A Vultr pool now proxy-ARPs on the interface the node routes the internet through. setup.sh installs arping, which the gratuitous ARP already called. * feat: deploy Spinifex on Vultr bare metal, renamed deploy-vultr.sh A vbm- plan creates bare-metal servers, one or three. Without --name the deployment writes the AWS profile spinifex, up prints every node's address and the API key allowlist, credentials and ssh subcommands are added, no AMI is imported unless --image is given, and OpenTofu or Terraform is used. The host key is pinned only once cloud-init is done, since bare metal first answers SSH from Vultr's installer with a different key. * docs: rework the Vultr guide into the OCI guide's flow, with bare metal * fix: restamp LaunchTime when a stopped or recovered instance relaunches The pending watchdog times a launch from LaunchTime, which still held the original launch. An instance recovered from a failed node, or started more than five minutes after it was first launched, was terminated as stuck in pending seconds after QEMU started. AWS restamps LaunchTime on every start. * feat: spread the nginx-alb workbook across nodes, and power Vultr nodes off and on from deploy-vultr.sh The workbook takes server_count and placement_strategy, so three web servers can run one per node. Its README now matches the template (public subnets, no NAT gateway) and the diagram shows the spread. deploy-vultr.sh gains stop and start, which halt and start a node through the Vultr API. * fix: let a terminate finish tearing down before the reaper calls it wedged OrphanQEMUReaper finalized any shutting-down instance whose QEMU had exited, on the premise that a live terminate still has a live QEMU. A terminate is still deleting volumes, releasing its IP and leaving its placement group for tens of seconds after QEMU exits, so the instance was reported terminated first. Terraform then deleted the placement group and got InvalidPlacementGroup.InUse. The reconcile now waits two minutes from shutting-down, well inside the ten-minute stuck-terminate backstop. * fix: release an ovs-vswitchd left in flow-restore-wait after a node boots ovn-controller clears flow-restore-wait once its flows are installed and sets it again straight after. When both writes reach ovs-vswitchd as one update, it only sees true, starts no upcall threads and forwards nothing. Seen on Vultr bare metal after a hard power-off: instances recovered onto the rebooted node ran but were unreachable. vpcd now clears the flag on any node held that way for a minute, longer than ovs-ctl's own restore takes. * docs: add a resiliency test to the Vultr guide and the nginx-alb workbook The Vultr guide gains step 8, which spreads three web servers across a three-node deployment, powers a node off and back on, and shows what was measured on VMs and bare metal. The workbook gains the same test as step 6. The guide's table of contents is repaired, and its limits now cover losing two nodes and recovery inside a spread placement group. * fix: never terminate an instance that has run because a relaunch failed The pending watchdog terminated any instance pending past five minutes, and terminating deletes DeleteOnTermination volumes. A start of a stopped instance, a recovery from a failed node and a daemon relaunch all pass through pending, so a slow one destroyed an existing instance and its root disk. On Vultr a recovered instance was terminated seconds after QEMU started and its root volume deleted. An instance is now marked FirstLaunch from RunInstances until it first reaches running, cleared before the running state is written. The watchdog acts only on first launches, as AWS terminates only a launch that never ran. MarkFailed refuses to terminate any other instance and marks it recovery-failed instead, which keeps its volumes, ENI and addresses and shows it as stopped. A claimed relaunch is never a first launch. Invariant tests pin ADR-0003:S7. The instancerecovery e2e suite gains a test that ages its guest past the watchdog before recovering it, then stops and starts it, and requires the instance, its root volume and the data on it to survive both. * feat: run the Terraform workbooks against a Vultr deployment on demand, and show the Vultr portal in the guide e2e-cloudvendor-vultr.yml is dispatch-only. It builds the ref, deploys it with deploy-vultr.sh on one or three VMs or bare-metal servers, runs the published workbooks from node1 with the same driver as the hypervisor nightly and OCI, and destroys the deployment, failing if anything is left billing. It needs one secret, VULTR_API_KEY, and generates its SSH key per run. The guide gains screenshots of the API key's access control list and of the nodes in the Vultr portal, and its portal navigation now matches. * fix: install setup.sh on Vultr nodes when the tarball does not carry it build-install-artifacts.sh writes setup.sh beside the tarball, so a node installed from its tarball had no /usr/local/share/spinifex/setup.sh and arming the firewall failed. * fix: pass the remote command after the destination in the Vultr e2e driver * fix: run each workbook in the cluster's own region The workbook driver passed no region, so every workbook used its default, ap-southeast-2, and on a cluster in any other region every request failed SignatureDoesNotMatch. It now passes the region of the node's spinifex profile. * feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) A node counted physical cores, collapsing SMT threads, and held back a fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is now one online CPU, as AWS counts it: a thread with SMT on, a core with SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and the rest is allocated one to one. The existing overrides stay operator-only. Invariant tests pin every tier boundary. * docs: state the Vultr guide's sizing under the online-CPU count * docs: drop an unmeasured EKS sizing claim from the Vultr README * fix: let a guest reach an Elastic IP hosted on its own node A guest's packet to an EIP on its own node left its VPC from the transit range, was masqueraded to the transit host address, and went back into OVN, which exempts that range from EIP DNAT, so it was never answered. On Vultr an EKS worker could not reach its control plane's endpoint, so it never joined. AWS lets an instance reach a public address in its own VPC. Hairpinned transit traffic now jumps to SPX-EIP-HAIRPIN, above the egress MASQUERADE, where each EIP's rule sends it from the uplink address, as the host's own traffic to the EIP already is. The rule follows the EIP's ingress plumbing in and out, and anything else returns to the MASQUERADE. * fix: resolve Spinifex's LB and EKS names on Vultr nodes Vultr's Ubuntu image writes a static resolv.conf naming Vultr's resolvers, so lookups bypassed the split DNS Spinifex configures in systemd-resolved and an EKS endpoint name did not resolve on the node. deploy-vultr.sh points resolv.conf at the resolved stub, as stock Ubuntu does. * test: prove a guest reaches an EIP on its own node in the natuplink e2e The nat-single cells run it with a public pool, so the hairpin path that left EKS workers unable to reach their control plane on Vultr is now covered nightly. * docs: record the natuplink hairpin check in the e2e coverage index * fix: keep each Vultr node's logs when an e2e run fails The driver destroyed a node whose install failed together with the log that said why. It now copies /root/*.log from every node into the run's artifacts before the teardown, and the workflow uploads them. --------- |
||
|
Josh Siviour
|
e709717fde |
fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options (#1360)
* test: pin EKS worker IMDS metadata options and launch template resolution * fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options * fix: default unset launch template metadata fields for EKS workers * test: e2e coverage for EKS worker IMDS hop limit and launch template metadata options * test: drop the unneeded GetCallerIdentity policy from the worker IMDS e2e * docs: describe the EKS worker IMDS defaults and launch template overrides * test: drop the stale GetCallerIdentity policy from the IRSA pod e2e |
||
|
Ben Duncan
|
f09250d2bb |
feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) (#1359)
A node counted physical cores, collapsing SMT threads, and held back a fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is now one online CPU, as AWS counts it: a thread with SMT on, a core with SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and the rest is allocated one to one. The existing overrides stay operator-only. Invariant tests pin every tier boundary. |
||
|
Ben Duncan
|
8839407c71 |
feat(ui): console design system, side-panel resource views and live EC2 metrics (#1358)
* feat: lay the Spinifex UI design-system foundation Brand tokens in OKLCH with semantic status and a colour-blind-checked chart palette, Archivo headings and a 13px type scale, and a pre-paint theme script so dark mode never flashes white under the console CSP. Adds the missing Base UI primitives, a /design catalogue route, a route ratchet against hand-written tables, a ban on chart-library imports in routes, and lets the content column shrink so wide tables scroll in place. The dev server now forwards /api and /proxy to the Go console. Sonner is removed: its inline style element is blocked by the CSP. * feat(ui): side panels, TanStack table, uPlot charts and e2e gate - DataTable renders @tanstack/react-table, with column hiding as container queries so columns drop when a side panel narrows the list. - UniFi-style DetailPanel and ResourceView: Instances, VPCs and Internet gateways open in a panel beside the list from 1024px and as a full page below; selected, tab and create live in the URL. Create flows for instances, VPCs and gateways open in the same panel. - Instance monitoring charts on uPlot (CPU, network, packets), on simulated data and badged as such. - Dates in the AWS console form, e.g. 2026/06/04 02:00 GMT-5. - Tabs style Base UI's data-active, so the selected tab now shows. - Playwright suite gates CI as the UI End-to-End job in ui.yml. * feat(ui): one details component for panel and page, UniFi-style overlay panel The side panel now lies over the list at full height with a shaded edge, and a click on the list's background closes it. Instances, VPCs and internet gateways each render one details component as either the panel or the resource's own page, so the page gains the monitoring graphs and the panel gains IAM role, GPU, console output, change type and create image. SSH moves into Instance information as rows. Internet gateways attach and detach from the panel. * feat(ui): roll the panel pattern out to every service and gate screenshots in CI Every EC2, IAM, Route 53, Bedrock, EKS, ECS and RDS resource is now a list plus a side panel, with one details component serving as both the panel and the one-column page. DetailCard, DetailRow, ListCard, ResourceList and DeleteResourceButton are deleted. - Panel: slide-out close, row selection closes it, expand keeps the tab and fetched console output, PanelSpinner while loading. - Instances: info tips, theme-aware coloured console output, power button group plus Actions menu, Stop and Reboot confirms, security group, Elastic IP and network interface sections, and a live one-second monitoring window on simulated data. - Launch instance: OS tiles picking the newest version, architecture choice, location tile, expanded root volume, honest default subnet. - On/off settings are switches app-wide; disabled controls show cursor: not-allowed. - The e2e job runs in the pinned Playwright image with screenshots gating; make ui-baselines and make ui-e2e-container use the same image. * feat: bulk select and delete on every resource list, OS marks and instance type sizes Every list with a delete action gets row checkboxes and a selection bar. Delete confirms in a dialog naming each resource, makes one call per resource, keeps each refusal against its row and retries only failures. The Launch form shows distribution marks for Ubuntu, Debian, Rocky, Alpine and CentOS, served as files under the CSP, and states the selected instance type's vCPUs and memory. * feat: sign-in overlay, larger type scale, typed confirms on irreversible deletes and dialog shortcuts The sign-in card sits over an inert shell. Any sidebar group folds, and entering one unfolds it. Instance DNS names copy. Escape cancels and Cmd/Ctrl+Enter confirms every dialog. Volume, snapshot, RDS, EKS, ECS and hosted zone deletes need "delete" typed, and terminate needs "terminate". * feat: fade sign-in and sign-out in place, and show the confirm shortcut Signing in no longer reloads the page: the card and a lighter veil fade out and the console routes to the dashboard. Signing out cross-fades into the sign-in card with a Signed out notice. Confirm buttons show Cmd or Ctrl Enter for the platform. * feat(ui): generic sidebar group names and a paged S3 object browser The sidebar names its groups Compute, Databases, Authentication and Object Storage; the command palette still finds each by its AWS service name. Buckets get row selection and delete. Inside a bucket, folders and objects share one table paged by ListObjectsV2 continuation tokens, with a find box that queries by prefix, multi-select delete through DeleteObjects (folders with everything under them), and an object panel showing HeadObject properties, user metadata and a download. * feat(ui): show upload progress and stop the S3 proxy cutting off slow transfers The Upload button shows a running batch's percentage, and a large or slow batch opens a dialog with bytes sent, bit-rate, time left and per-file cancel and retry. Uploads run in a store outside React, so they continue while the user browses. The console's S3 proxy now allows an hour per request. The server's 15 s ReadTimeout cancelled every upload part that took longer, so large uploads failed on links below roughly 18 Mbit/s. * feat(ui): stream live EC2 metrics to the Monitoring tab The Monitoring tab drew simulated data. It now streams CPU, network bytes and packets from the owning node's qmp-collector every 500 ms while someone is watching, with nothing stored. - qmp-collector: a watched sampler on schedstat and tap counters, on a 30 s lease, keeping five minutes in memory for reconnects. It answers only watches for the account in the VM's discovery file. - awsgw: spinifex:GetInstanceMetrics, checked against the instance ARN, answered as text/event-stream in 60 s segments. Another account's instance is InvalidInstanceID.NotFound. - console: a signed stream reader that reconnects with Since and backs off on failure or an empty segment, stops on InvalidAction, and a chart axis that scrolls between samples. The simulator and its badge are removed. * feat: catch refused creates in every form, answer the third review, and guard tables Every create form now catches a refused mutation, so the failure shows beside the submit button and no longer escapes as an unhandled rejection. Three EKS tabs that swallowed a refused create now show it. Third-review answers: the attach dialog offers running or stopped instances in the volume's zone; the default security group's delete is disabled with its reason; the load balancer panel splits into Listeners and Attributes tabs; a main route table lists its implicit subnets; IAM single delete dialogs name the same dependants as the bulk ones; a snapshot can be deleted in any state. Every list in a route renders through DataTable, enforced by a unit test with a named allowance, and the /design baselines are redrawn in the pinned container. * feat: cap live metrics streams, prune hidden selections, and move panel focus Each gateway holds at most 10 live metrics streams per account and 200 in total, refusing past either with RequestLimitExceeded, and counts opened and refused streams. A row that leaves a list leaves its selection, the instances page uses the shared selection hook, the detail panel takes focus on open and returns it on close, and Escape yields only to the popup it was pressed in or a modal. The UI workflow also runs on live metrics backend changes. |
||
|
Josh Siviour
|
dee37b976a |
fix: keep an ECR token valid after its role session expires (#1357)
* test: authorise an ECR token from the live role once its session is gone * feat: carry the role ID in an ECR token minted for an assumed role * feat: rehydrate an assumed-role caller from the live role * fix: authorise an ECR token from its live role after the session expires * docs: note that an ECR token outlives its role session |
||
|
Josh Siviour
|
ce8272cf5e |
refactor: register EC2 through the ingress seam and delete the legacy service switch (#1355)
* test: characterise EC2 request labels and the cluster-unavailable gate order Pins the span, audit, throttle and policy-gate labels an EC2 request carries through the signed pipeline, and that the NATS gate answers before policy, action lookup and gateway-local actions. * refactor(gateway): register EC2 through the ADR-0002 seam EC2 is composed in awsgw from domains/ec2/awsapi and served by the registration seam. The legacy service switch and its tables are removed, along with the two paid-off gateway allowlist rows. * docs: record the EC2 seam registration in the package boundary migration * docs: record the EC2 seam registration PR number |
||
|
Josh Siviour
|
0fc93e975a |
refactor: serve EC2 actions from domains/ec2/awsapi (#1354)
* test: characterise EC2 through the signed gateway pipeline * refactor: serve EC2 actions from domains/ec2/awsapi over composed capabilities The EC2 action table, idempotency and validation move out of the gateway behind a Deps struct. Quota is a consumer-side interface and the instance adapters take a narrow InstanceProfiles interface instead of the whole IAM service. The gateway only composes Deps and calls Dispatch. The 18 gateway to EC2 awsapi allowlist rows are deleted. * test: drive every EC2 action through Dispatch in its own package Covers the request errors, the gateway-local actions without NATS and each routed action reaching its handler against a stub daemon. * docs: record the EC2 capability move in the package boundary migration * docs: record the EC2 capability move PR number |
||
|
Josh Siviour
|
d066786faa |
feat: operator-launched SQS worker and store system VMs (#1353)
* feat: persist the SQS worker's store key on its own disk The shared SQS worker generates its store-client NKey inside the guest and keeps it in an owner-only directory on its own persistent disk, so no platform component creates or unseals the seed. The key is synced and linked into place without replacing anything; a missing, mismatched, unreadable or loosely permissioned key fails closed instead of being regenerated. * fix: make the SQS worker key durably named before returning it Every successful call now syncs the key directory and its entry in the parent before handing out a key, so a retry after a failed directory sync cannot return a key whose name is not yet durable. Only the key directory is created, inside a parent that must exist, so an unmounted worker disk fails closed. The first-start decision is explicit: an existing store's authorized key is required and never defaults to generating a new one. * docs: state that workerkey cannot detect an unmounted worker disk An existing parent directory does not prove the worker disk is mounted, since the mountpoint can exist on the initramfs after a failed mount. The error now says only that the parent is missing, and the package documents that its caller must verify the expected data volume is mounted first. * feat: add the SQS worker guest bootstrap The shared SQS worker microVM boots with sqs-worker boot, which selects exactly the disk whose serial fw_cfg names, checks and mounts it at /data, and proves from the mount table that this disk is what is mounted before loading or creating its store key there. A missing, empty or unknown store status, a contradictory or invalid store key, a missing or ambiguous disk, a failed check or mount, an empty mountpoint or a foreign mount refuses without creating a key. Only the public key is logged and published. The image is built by the store image script under SQS_MICROVM_PROFILE=worker, with the worker binary, no nats-server and an empty /data mountpoint; the store image's contents are unchanged. * fix: refuse replayable new-store status at SQS worker boot A static fw_cfg store-status of "new" survives a guest reset, so after the store exists it could still reach key creation and replace a lost worker key. Boot now accepts only "existing" with the store's authorized public key and refuses "new" before touching any disk, until a current authoritative first-store decision exists. workerkey.ForNewStore remains, uncalled. * feat: add SQS first-store claim and worker key acquisition The SQS domain records one first-store decision per Region in its own KV bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound to the worker's instance ID, claim ID and generation. Key creation is authorized only while no public key is recorded; afterwards only the recorded key is returned, and a launch intent requires it. Older generations, same-generation rivals and unreadable records authorize nothing, and records hold no private key material. workerkey.Acquire syncs a new key to disk and returns it only after its public key is recorded, reusing the on-disk key after a crash before recording and failing closed on a missing or different key afterwards. No transport carries the decision yet. * fix: read SQS first-store decisions from the stream leader KV.Get uses a direct get, which a lagging replica can answer, so a stale claimed read could authorize key creation after the leader had recorded the worker's public key, or make a superseded generation look current. Reads now use the SQS store's leader-answered stream message get, and every authorizing answer is also confirmed by a compare-and-swap at the revision read, so a stale or unavailable read authorizes nothing. * feat(sqs): add regional worker identity record and instance index The SQS domain keeps one worker identity record per Region with a monotonic generation, claim ID, pinned node and encrypted data volume, moving from launching to bound or failed under compare-and-swap. A launch intent takes the next generation before any instance exists, so the previous worker stops being current; binding writes the instance index entry first. Authorize accepts an instance only while the Region's leader-read record is bound to exactly that instance, generation and claim, confirmed by a compare-and-swap at the read revision, so an old, orphaned, rewritten or unbound index entry, or a lagging replica, authorizes nothing. Both this bucket and the first-store bucket now open through the kvutil chokepoint and refuse a stored configuration that could expire, cap or evict records or holds fewer replicas than the cluster needs. Nothing calls the store yet. * fix(sqs): make worker launch exclusive and harden durable buckets BeginLaunch now refuses a new claim while the previous attempt is launching or bound, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume. A replacement takes the next generation only after FailLaunch or the new RecordStopped confirms that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never saw. The durable bucket opener also refuses non-limits retention, per-message TTLs, asynchronous flush, a message-size cap, any discard policy other than discard-new or a per-subject discard, unacknowledged writes, a sealed stream, and a mirror or sources. * feat: add internal SQS worker first-store gateway actions Serve ClaimFirstStore and RecordWorkerPublicKey under the internal spinifex-sqs signing service, apart from the public AmazonSQS inventory. Only an sqsWorkerRole session in the system account passes the class gate, then the role policy; the worker identity is resolved from the current workerident record by the STS-set session name, never from the request, and any extra parameter is refused. Seeds and other non-public keys are refused without being recorded, logged or echoed. The gateway binds the worker-identity and first-store buckets per request and never creates them, so it provisions nothing and answers ServiceUnavailable until a worker launch has created them. * fix(sqs): fence replaced worker generations out of first-store A request that passed worker authorization before a confirmed stop and replacement could reach an unclaimed first-store record ahead of the successor and create or record the old generation's key. The launch path now raises a monotonic first-store fence past the replaced generation before BeginLaunch, which refuses a replacement until a leader read shows it; firststore refuses every call from a generation below the fence. The internal adapter no longer logs caller-chosen parameter names. * feat(sqs): launch the shared SQS worker on its pinned node Add the SQS worker system-instance kind and a domain launcher that realizes the Region's worker from its identity record without ever starting a second VM: - domains/sqs/workerlaunch returns a bound worker as is, prepares the data volume on first launch only, refuses a volume whose own record is not encrypted, marks the intent dispatched before launching and never resends a dispatched launch: an ambiguous outcome stays launching until reconciled. A replacement starts only after a confirmed stop or failure and the first-store fence, reusing the pinned volume. - workerident gains MarkDispatched and Current; firststore gains a leader-read RecordedKey. - The daemon launches the worker through the shared direct-boot path with its encrypted volume attached at boot and kept on terminate, a required instance profile and a mandatory mgmt NIC. Its firmware config carries only public values: the data serial, the store decision, the gateway URL on this node's bridge and the daemon's own cluster CA. A relaunch is refused on any other node. - sqsWorkerRole is ensured in the system account granting exactly the two internal actions; an existing role whose trust admits more than EC2 is refused. The role contract lives in domains/sqs/workerrole so the composition root does not import the gateway adapter. The worker network allocator and any caller remain unwired, and the guest still refuses to create a key for a new store. * fix: verify SQS worker role trust before granting and report unclaimed volumes Check that an existing or racing-created sqsWorkerRole trusts only EC2 before writing its policy, and accept the instance profile only when a re-read shows it carries exactly that role. Refuse the first launch before allocating when the cluster storage key is not configured, and return the ID of any volume created but not confirmed in a launch intent instead of dropping it. * feat: add private SQS system VPC and worker ENI allocator Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet from the VPC and subnet surface only, so it cannot create an internet gateway, route table, NAT gateway or EIP. The daemon's SQS worker network adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and finds or creates one ENI per launch claim in the VPC default security group, refusing a duplicated claim or an address outside the subnet. Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have recorded the volume. * feat: worker takes its first-store decision from the gateway at boot The SQS worker guest no longer trusts a static fw_cfg store status. After selecting, checking and verifying the mount of its own data disk, it asks the internal spinifex-sqs gateway service for its current first-store decision, signed with instance-metadata credentials only, over TLS 1.3 trusting only the cluster CA. It then creates and records or loads the key on that disk; a key lost after recording is never replaced. - internal/sqsworkergw: SigV4 client for ClaimFirstStore and RecordWorkerPublicKey; refuses non-public keys, non-decisions and redirects; errors carry only the parsed code - gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback - workerboot: retries only lost replies and ServiceUnavailable/Throttling - launch contract carries the Region instead of a store status or key * fix: worker retries a denial while its launch is still binding it The worker VM can reach the gateway before its launch binds the instance identity, and that early call is legitimately AccessDenied. The guest treated the denial as final and stayed unprepared even after binding. AccessDenied is now a pending authorization the guest asks about again within its existing bound; a denied claim creates nothing, and a worker that is never bound, or was replaced, gives up at the bound. * feat(sqs): add operator EnsureSQSWorker action and node launch handler Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the internal spinifex operator service. It validates the node against the cluster and as a single subject token, then requests sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers and the worker stays pinned. The node handler re-checks the named node and instance type and runs one composed workerlaunch.Launcher (real volume preparer, role, private network and direct-boot launch), one launch at a time, in the daemon's own context with a 30-minute bound. A caller timeout does not cancel the launch; a retry reports the bound worker, or IncorrectState for a dispatched launch whose outcome is unknown, which is never resent. The reply names the bound VM and does not claim the worker or SQS is ready. Nothing starts a worker at daemon startup. * feat: ship the SQS worker image and check it before allocating Build the SQS worker microVM image in both release workflows, stage it into the tarball as microvm-sqs-worker, and install it with setup.sh under the store image's rules: a release without it removes an earlier one and an incomplete one is refused. Add make install-sqs-worker-microvm. The worker launcher now asks a Host capability whether this node can boot a worker before it prepares a volume, records or resumes an intent, fences a replaced generation, allocates the network or dispatches. The daemon checks for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a call after the image is installed launches instead of leaving an ambiguous dispatched intent. Cover missing and incomplete images through the admin action and handler, a stopped worker left untouched, and an ENI create whose reply is lost. * fix: never repeat an uncertain SQS worker ENI create The launcher now looks for the claim's ENI without creating one, marks the launch intent NetworkAttempted, and only then sends the single create. Once marked, a missing ENI fails closed with IncorrectState instead of a second create that could give the claim two interfaces; a visible ENI is reused. Adds a test that buildSQSWorkerLauncher wires the daemon's real host check. * feat(sqs): operator-triggered SQS store launch on the worker's node Add the super-admin EnsureSQSStore action and its node handler. The store is launched only on the node the Region's worker is bound to, with the public key that worker recorded in the first-store decision; the daemon never receives the worker's seed. A per-Region store launch record marks each step before its uncertain create (volume, ENI in the worker's private subnet, VM launch), so a step whose result is not visible fails closed and is never resent. Volumes are refused unless their own record is encrypted, and their IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and instance types at or below that are refused; this is a floor, not a validated size. * fix: retry uncreated SQS store volumes and fix the store instance type A volume preparation that failed before sending any create (a missing tool, scratch or format failure, or no cluster storage key) now carries storevolume.ErrNotCreated. The store launcher withdraws its volume mark and returns ServiceUnavailable, so a retry may try again once the host is fixed; any create that was sent stays one-shot. The store launch record now keeps the operator's instance type. A request for a different type is refused as IncorrectState before dispatch and after binding, and the reply names the launched type, which the admin action checks. * feat: SQS worker store readiness and store size check The worker's one-shot ready step mounts and verifies its disk, asks the internal spinifex-sqs DescribeStore action for the bound store, loads its key only if it matches the recorded public key, and checks the store's JetStream with it. It retries within a bound while the store is pending or not answering, and stops on a mismatched or refused key. The NKey authenticates the worker only; the store server is not authenticated. EnsureSQSStore now also refuses a different volume size once the store is bound, instead of returning the existing store. * fix(sqs): keep worker store readiness current until shutdown The readiness step now runs as a guest-local watch loop with no overall deadline: each pass is bounded, a pending or unreachable store is retried every few seconds, and a ready store is rechecked periodically. The store-ready marker is removed on any failed pass, cleared at start and removed on exit; shutdown stops the watcher by its recorded PID. Terminal refusals such as a rejected or mismatched key back off for minutes rather than spinning. The NKey still authenticates only the worker to the store. * feat(sqs): supervise the worker readiness watcher and add spx admin sqs Run the worker guest's readiness watcher under init respawn, rate-limited, and remove any stale store-ready marker before each start, so a killed or crashed watcher cannot leave a false readiness marker. Shutdown blocks respawn, removes the marker, stops the recorded watcher by PID and start time with SIGKILL escalation, and unmounts /data only once it has exited; a surviving watcher leaves every mount in place. Add spx admin sqs ensure-worker and ensure-store, which call the existing EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex gateway path with the standard AWS credential chain. Both refuse missing inputs before sending anything and report only the bound VM, never readiness. * fix: park SQS worker readiness and fix the worker instance type A readiness start that finds shutdown in progress or no worker key now logs once and waits in place instead of exiting, so init's respawn does not restart and log it in a loop; shutdown stops it like a watcher. The worker identity record keeps each generation's instance type. A request for another type is refused as IncorrectState before dispatch and after binding, and only a replacement may change it. The ensure reply, admin action and spx admin sqs ensure-worker report and check the recorded type. * fix: record the SQS worker volume intent before creating it The first worker launch prepared its data volume before any record existed, so a create that applied but returned no ID left nothing for the next call to find, and it could prepare a second volume. The launch intent is now recorded first, and its one volume create is marked before it is sent. A Prepare proven to have sent nothing withdraws the mark and is retryable; any other outcome is never resent and returns IncorrectState, naming the volume when its ID is known. An intent that has sent no create may still be re-typed. * docs: document spx admin sqs ensure-worker and ensure-store Record the flags, the caller and owning accounts, the bound-versus-ready distinction and the current retry and recovery limits. * docs: distinguish SQS ensure timeouts from refusals that created nothing A gateway timeout means only that no reply arrived; the launch may still be running. The worker type locks once any step has been attempted. * docs: state what each SQS ensure ServiceUnavailable establishes No refusal proves earlier attempts created nothing; a retry checks the durable attempt and continues only when that is safe. |
||
|
Josh Siviour
|
55054ad0ac |
fix: interim guard against opening a volume behind a live owner holding unsealed writes (#1352)
* test: relay a capacity refusal from an owner holding unsealed writes * fix: refuse a start elsewhere when the owner holds unsealed writes * fix: bound the unsealed-writes check well under the start forward timeout * test: refuse a volume takeover while the marker owner is live * fix: refuse a volume takeover while the previous owner is live * fix: retry EKS control-plane starts refused for capacity or a held volume * docs: describe the interim unsealed-writes takeover guards * test: give the viperblockd conformance harness a cluster-state bucket The takeover guard reads node liveness from the cluster-state bucket the daemon creates. The harness had none, so the real reader errored and the guard refused every takeover. Create the bucket the way the daemon does so the real reader runs and finds no heartbeat. * fix: leave no dirty marker behind when opening a volume that does not exist An open takes the lease and writes the dirty marker before the volume's state is read. When the volume turned out to be absent the marker stayed, so a later open on another node met the takeover guard and was refused as held elsewhere instead of answering not found. Clear the marker this open wrote when the state load reports the volume missing. |
||
|
Josh Siviour
|
27304bbfd6 |
refactor: serve the four Bedrock services from one Ochre registration (#1351)
* test: characterise the four Bedrock services through the SigV4 pipeline * refactor: serve the four Bedrock services from one ochre registration The Bedrock adapters move from gateway to domains/ochre/awsapi behind a single dispatch registration. The sub-service scopes are aliases, a sub-service scope selects its dispatcher outright and only the shared bedrock scope is split by path. The thirteen Bedrock fields leave GatewayConfig for awsapi.Deps, and the bodyscope allowlist row for domains/ochre is deleted. * docs: record the Bedrock registration in the package boundary migration * docs: record the Bedrock registration PR number |
||
|
Josh Siviour
|
a52b078678 |
refactor: move the ECR /v2 bridge out of the gateway (#1350)
* test: pin the ECR /v2 pipeline through the gateway handler * feat: rehydrate a principal from an access key id in credverify Adds credential.Rehydrator and CallerARN on Principal, implemented beside Authenticate and Resolve on the same key, session and user lookups, with a parity table against the gateway's ECR rehydrate. * refactor: share the credverify key and session lookups with Rehydrate Authenticate, Resolve and Rehydrate now call one lookup per credential kind, so the checks cannot drift. The parity table also asserts the gateway and credverify caller ARN builders agree. * refactor: move registry host matching to ingress/http The matcher takes its service labels from the caller and exposes the parsed account and region through accessors, so ingress names no service. * refactor: extract the identity-policy evaluator into policyauth The SigV4 gateway path and the ECR registry authorize through one evaluator and one set of request condition keys, over the credential principal. * refactor: extract account admission into accountadmission The SigV4 gateway path asks one Admitter for account status, with the ACTIVE-only cache moved with it, so the ECR registry can use the same capability. * refactor: move the ECR /v2 bridge out of the gateway The bridge, token exchange and per-operation authorization now live in domains/ecr/registry behind registry-owned Principal, Rehydrator and Authorizer types, so the package imports no ingress code. The awsgw role adapts credverify, policyauth and host matching to them, and the gateway keeps only a RegistryHandler mounted at /v2. The bridge refuses to compose without a verifier, issuer, rehydrator, authorizer or admission, so the open-registry mode for a nil verifier is gone. * test: carry the retired ECR principal parity rows into credverify Global root, a verifier with no IAM, a record with no principal type and the gone, replaced and legacy session verdicts for user and role sessions were only exercised by the gateway tests that went with the old path. * docs: record the ECR /v2 bridge move in the package boundary migration * docs: record the ECR /v2 bridge PR number |
||
|
Josh Siviour
|
26d53602a3 |
fix: accept AWS-format IRSA trust-policy condition keys (#1349)
* test: cover AWS-form IRSA condition keys and StringLike on web identity * fix: match AWS-form IRSA condition keys and allow StringLike on web identity * fix: keep the provider ARN host-path unnormalised for web identity * feat: log the reason a web identity trust policy denies * fix: satisfy errname, modernize and gofmt on the denial error * test: cover the stock AWS IRSA trust policy in the EKS e2e * test: make the IRSA denial prove the policy was evaluated |
||
|
Josh Siviour
|
060204514e |
refactor: register ECS and EKS through the ingress seam with shared PassRole (#1348)
* test: pin ECS and EKS dispatch through the SigV4 pipeline * refactor: register ECS and EKS on the ingress seam EKS resolves the cluster and nodegroup role through passrole.Check with the IAM role resolver, in the order it had. Responses and check order are unchanged. * fix: match AWS role handling for ECS and EKS PassRole cases ECS refuses a foreign task role before PassRole, accepts a role that is not an ARN or does not exist after PassRole, and reports a RunTask override role it cannot assume. EKS CreateCluster and CreateNodegroup refuse a foreign or non-role ARN with AccessDeniedException before PassRole. * test: cover ECS and EKS awsapi registration dispatch * fix: report EKS parameter faults as InvalidParameterException * docs: record ECS and EKS gateway migration and EKS divergences |
||
|
Josh Siviour
|
099a69b23b |
refactor: register IAM and STS through the ingress seam (#1346)
* docs: point the /admin migration row at spinifex#1345 * test: characterise IAM and STS dispatch through the SigV4 pipeline * refactor: register IAM through the ingress seam * refactor: register STS through the ingress seam * docs: point the IAM and STS migration rows at spinifex#1346 |
||
|
Josh Siviour
|
6462ccc41f |
refactor: serve the /admin operator API from the spinifex registration (#1345)
* refactor: move the /admin handlers and wire types into operator/admin * refactor: serve /admin through the spinifex registration * docs: record the /admin move into the spinifex registration |
||
|
Josh Siviour
|
ae8ee1e5fe |
refactor: register RDS through the ingress seam (#1344)
* refactor(gateway): register RDS through the ingress seam RDS dispatch moves from gateway/rds.go to domains/rds/awsapi.NewRegistration, composed by awsgw and the integration harness. The caller comes from dispatch.Actor and the principal-class gate still runs before the policy check. * docs: point the RDS migration row at spinifex#1344 * chore: group the imports merged in the actor projection test |
||
|
Josh Siviour
|
488f1c7bf9 |
feat: SQS store guest on a prepared cold-attached EBS volume (not wired) (#1343)
* feat: build a separate SQS store microVM image Package the go.mod-pinned nats-server, ext4 support and a clean ACPI power-down into a microVM kernel and initramfs of their own. The build checks the packed archive's contents and refuses a nats-server whose version differs from the pin. The image does not start the store and nothing installs or releases it; the LB image is unchanged. * fix: publish the SQS store image only after its gates pass Stage the kernel and initramfs and move them into place only once the content, size and guest-RAM gates pass, so a failed build cannot leave a pair make takes as up to date. The old initramfs is removed first and the new one lands last, so an interrupted publish never leaves a mixed pair. Check the module list without piping into grep -q, which under pipefail reported a present module as missing. * feat: start the SQS store guest on its checked data disk The SQS store image now prepares, runs and stops a dedicated JetStream server. It reads a narrow fw_cfg contract (data disk serial, listen address, one user NKey public key), refuses anything absent, malformed or ambiguous, has the pinned server test the config, then checks and mounts exactly one matching ext4 disk. It never formats and has no ephemeral fallback. The server listens only on the supplied guest NIC address, admits only that NKey and syncs every write. Shutdown stops the recorded run process first, whether it is the server or a start not yet exec'd, and runs the generic read-only remount only once it is gone. A store that survives SIGKILL leaves every mount alone. * fix: report the SQS store as prepared, not listening, before it starts * feat: launch the SQS store guest on its own versioned subject An explicit SQS store kind carries an existing system-account data volume and the one NATS user public key the store accepts. It is served only on system.LaunchSQSStore.v1, so a daemon without this code has no responder and the launch fails instead of booting a load balancer without its disk; the legacy launch path refuses the kind on both send and receive. Every refusal happens before capacity, an ENI or a VM record exists. The volume goes through the normal EBS mount and terminate detach and is never deleted on termination. The store boots its own image with CPU mitigations on, gets only its network, disk serial, listen address and key, and cannot be relaunched after a host restart yet. * feat: ship the SQS store microVM image in releases and installs Release builds now build the SQS store image and stage it beside the load balancer's as microvm-sqs-store/, so an older installer that copies microvm/* without -r still works. setup.sh installs it to /usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it. A release without a complete image never leaves an older one usable: staging drops a previously staged image, and the installer removes an earlier installed pair before warning (missing) or failing (incomplete). Only the two image files are removed. The initramfs is installed last, so an interrupted install leaves no bootable mismatched pair. * feat: restart the SQS store guest on its launching node Persist the non-secret guest config of an SQS store instance (launching node, data volume, listen address, client public NKey and rendered network config) in the instance record, and rebuild its fw_cfg files from it on relaunch. The relaunch refuses on any other node, or when the record is missing or malformed, before writing any file; the data volume is remounted through the normal EBS path and kept on terminate. Cross-node recovery now never selects or claims an SQS store record, so no other node spends retry budget on it or relaunches it. Same-node restore is unchanged, as are load balancer restarts. Older daemons know neither the exclusion nor the record field, so this does not prevent a cross-node relaunch by a mixed-version cluster. * feat: hold prepared system volumes in creating until released Add an internal CreateVolumeForPreparation path that writes a new volume's first record as creating, and CompleteVolumePreparation to release an unattached creating volume in the caller's account to available. Ordinary CreateVolume is unchanged. Attach and delete already refuse a creating volume. ModifyVolume, ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState so nothing can expand, release or capture a volume mid-preparation. * feat: prepare fresh SQS store volumes with an ext4 filesystem The SQS store guest never formats a disk, so a new volume gets its filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img, builds an ext4 image in private scratch, creates the volume in "creating", publishes it on this node, copies the image with the NBD import helper, unpublishes, and only then releases it as available. It never writes a volume it did not just create. A volume is deleted only when no export can still be live: after a confirmed unpublish, or when the export was not NBD. Any publish error, a failed or ambiguous unpublish, or a failed release retains the volume and reports its ID. A publish refused as in-use or already-exists sends no unpublish, since another export may exist. Cleanup ignores caller cancellation but is bounded at 30s. The raw-image copy moves to runtime/compute/nbd unchanged, and the domain receives it as an injected writer. The volume service gains a discard path for an unattached volume still in "creating". * fix: report unknown state and match qemu-img lookup in volume prep A failed release may still have made the volume available, so the retained-volume error now says its state is unknown rather than claiming it is still creating. The copy runs qemu-img by name through PATH, so the pre-allocation check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin fallback because it runs from the resolved path. * docs: describe the unwired SQS store system instance Records what the guest image, versioned launch, same-node restart and fresh-volume preparation provide, that nothing calls them, and what is untested or missing before production use. |
||
|
Josh Siviour
|
24281cb860 |
fix: count volume lease validity from the write, not its reply (#1340)
* fix: count volume lease validity from the write, not its reply The server starts a lease entry's TTL when it applies the write, but the holder started its validity window when the acknowledgement arrived, so a slow reply stretched validity past the server's expiry and a peer could acquire the volume while the holder still considered it valid. Validity now counts from when each create, generation publish and renewal was sent. An acquisition acknowledged after validity lapsed is refused as retryable, and a lost or lapsed lease still awaiting its fence is no longer shared with another open on the same node. A renewal that lands after the lease was given up, or after its validity elapsed, cannot revive it: the lapse check and the update run under the lease lock, and fencing runs once. The renew loop also fences at the validity deadline itself rather than at the next check, which could consume the whole margin. This bounds local validity only; it is not backend-enforced fencing, and a paused host can still write after its validity ends. * docs: describe volume lease validity, fencing and their limits |
||
|
Josh Siviour
|
18adea9486 |
refactor: register the spinifex operator service through the ingress seam (#1337)
* test: characterize the spinifex Query and /admin surfaces through the full gateway pipeline * refactor: register the spinifex Query service from operator/admin * docs: point the spinifex registration migration row at the PR |