mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-08 03:33:13 +00:00
2d9eadbf237c47dfdafe75494c170e9930f24266
355 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
8fa33f07ce |
feat(eks): inject IRSA credentials into pods of annotated ServiceAccounts (#1396)
* test: pod identity webhook mutation, publisher and webhook config contracts * feat: pod identity webhook mutator, CA bundle publisher and webhook configuration * feat(eks): run the pod identity webhook on every control-plane server Add the eks-pod-identity-webhook binary and its OpenRC service. It listens on 127.0.0.1:8444 with a serving certificate signed by the k3s server CA, authenticates to the API server with a client certificate signed by the k3s client CA, and runs under its own least-privilege ClusterRole. Each server stages the RBAC and the MutatingWebhookConfiguration (caBundle is the server CA) into the k3s manifests directory, so server-join nodes get them too. Match upstream's container skip rule: skip only when the regional STS key is present, and add the token mount otherwise. * fix(eks): issue pod identity webhook certs for one year and renew in the running process Both the serving and client certificates were valid for ten years. Kubernetes cannot revoke a client certificate, so a copy taken off a control-plane disk would stay a valid identity for a decade. Issue both for one year with the existing 30-day renew window. The process loaded the serving certificate once. A daily check now exits with ErrCertRenewal when either certificate enters the renew window, so the supervisor restarts it and EnsureCerts reissues. Add tests for KubeClient against a TLS fake API server and for Run, which the package had none of. * test: e2e covers pod-identity webhook and unmodified LBC chart IRSAPod now uses a ServiceAccount annotation only and asserts the injected env and volumes plus an untouched unannotated pod. LBCChart installs the upstream chart 1.11.0 and reconciles an Ingress into an active ALB. * test: pass the tagging endpoint to the LBC chart and mark Shield/WAF flags Spinifex-only * fix: make the pod identity webhook init script executable in the eks-node image setup.sh chmods each shipped init script to 0755 explicitly, and the new service was missing from the list. rc-update then refused it, and eks-node-role, running under set -eu, aborted before starting k3s. Add a guard test that every init script in INSTALL_FILES is chmod'd. * fix: start the pod identity webhook last and keep its failures non-fatal eks-node-role runs under set -eu, so a fault enabling or starting the optional webhook aborted the control plane boot before k3s-first-boot, state-report and addon-sync. Enable and start it at the end of the server and server-join roles and log a warning on failure instead. The role test now asserts the webhook is the last call and that an enable or start failure still exits 0 and finishes the role. * fix: move the pod identity webhook off the token webhook readiness port eks-token-webhook already serves its plain HTTP readiness endpoint on 8444, which the API server target group health-checks. The pod identity webhook bound the same port, failed with address already in use and respawned forever, so no pod was ever injected. Listen on 127.0.0.1:8445 and add a test that the two ports differ. * docs: describe pod identity webhook for user workloads in the EKS guide * fix: split STUB_FAIL assignment from export in the role test |
||
|
Josh Siviour
|
b034f3b2e9 |
fix(sqs): bring up loopback in the SQS guest before its store starts (#1409)
The guest's SQS nats-server listens on 127.0.0.1 and the worker dials it there, but the kernel leaves lo down, so the store failed to bind and the worker never polled. Also correct the stale CA comment in init. |
||
|
Josh Siviour
|
671a8a9ff5 |
feat(sqs): fresh single-node SQS in one system microVM (#1389)
* feat(sqs): run queue and message owners in the worker by signed pull The SQS worker's ready step now opens the queue and message owners on its dedicated store and serves owner calls it pulls from its node's gateway over the existing instance-role channel (PollWork, CompleteWork). Each poll and post is authorized against the current worker identity, and a result is accepted only from the identity that took the work. No payload crosses platform NATS. A receipt-signing key is created on the worker disk only under the first-store decision that creates the store key, and its fingerprint is bound in the store; a missing or changed key refuses to serve. Received messages are released only when the gateway proves the caller left; receives are capped so a worst-case result fits one post. The public relay is not registered in production. * fix: harden SQS relay results and bound pending bytes Unavailable outranks client faults when encoding owner errors, and wire errors carry fixed text only, never owner or backend text. The ingress rejects incomplete or mismatched success results as unavailable, and the relay bounds total pending input bytes as well as request count. Adds tests of every relay action against real owners, the receive claim limit, release, faulting work, the receipt-key fingerprint marker and the poll loop through gateway failures. * fix: refuse SQS serving outside a single-node host cluster DescribeStore now reports the gateway's declared cluster node count, and the worker refuses to serve, before reading keys, connecting or creating any store state, unless that count is exactly one. A gateway that reports no count reads as zero and is refused. * feat(sqs): record a store server key pin once per store instance Add the store role and its internal actions so the bound store instance can learn its claim and record the public pin of its TLS server key, once, without the platform holding the private key. The worker sees the store as bound only after a pin exists and must receive a valid pin. * refactor(sqs): serve store server keys from a separate capability Keep the worker Stores interface unchanged and give the store guest's actions their own ServerKeys dependency, so existing Stores fakes need no store-key methods. * feat(sqs): launch the store with a mgmt NIC, CA and scoped store role The store now boots with the region, gateway URL and the daemon's own cluster CA, a mgmt NIC to reach the gateway, and an sqsStoreRole profile granted only the store's two internal actions, so it can authenticate its server key pin. A store record that predates these fields refuses to relaunch; its volume is kept. * feat(sqs): keep the store TLS server key on its disk and record its pin The store image gains sqs-store-key, run after the data disk is mounted. It loads or creates a P-256 server key under the disk's tls directory, records only the key's public pin with the gateway as the bound store instance, and writes a self-signed certificate for it to the run directory. nats-server then requires TLS 1.3 with that key. A record for another volume, a recorded pin with a missing key, or a key that does not match the recorded pin refuses the store; no key is regenerated and nothing on the disk is deleted. Launch-not-yet-bound and lost replies are retried. The store role vocabulary moves to contracts/sqs/v1 so the guest agent does not import a domain. * feat(sqs): pin the worker's store connection to the recorded server key The worker's readiness check and serving connection now use TLS 1.3 and accept only the store server key whose pin the bound store recorded with the gateway. The pin is checked before the worker key or any queue or message data is sent, and on every reconnect. A store with another key, a plaintext store or a malformed pin is refused as final. The worker NKey client authentication is unchanged. The host still holds the store's data disk key, so the pin does not protect the store from a privileged host. * fix(sqs): retry store key bootstrap and track guest binary sources The store's server key check moves from sysinit into the respawned run step: a gateway or IMDS outage exits after a pause so init retries it, without remounting or creating a new key, while a missing, changed or wrong-store key is recorded as refused and never retried. Stop also ends a running key helper before unmounting. The guest helper and worker binaries now depend on every main-module package dir and file they link, from go list, so edits, additions, deletions and renames all rebuild them. * feat(sqs): operator enable and status with a signed ingress test Add a per-Region SQS service record (disabled by default, pinned to one node, refused on multi-node clusters before any write), gate the relay on it, and expose `spx admin sqs enable` and `spx admin sqs status`. The gateway now hands queued work to the polling worker and reports whether one is polling. The public SQS service stays unregistered in production until its capability profile is reviewed. An integration test registers it through the harness and drives real SigV4 and IAM requests to a real worker for create, send, receive and delete, asserting no SQS body crosses platform NATS. * fix(sqs): report readiness only from observed component state Status now reports whether the tenant SQS API is registered on this gateway and is never Ready while it is not. The worker stops polling while its store connection is down, so a worker that lost its store drops out of the gateway's poller window instead of looking healthy. * wip: sqs model inventory * wip: sqs public registration * wip: sqs registration review fixes * fix(sqs): refuse pre-API store and worker records on enable A store or worker record this release creates now carries an explicit contract. enable reads both records without creating a bucket and refuses any record without it before it writes desired state or launches anything; status names the record and is never Ready. A worker replacement keeps its predecessor's contract, since it reuses the same data volume. Missing metadata is only ever grounds to refuse: nothing is upgraded, inspected, emptied, replaced or deleted. Upgrading a pre-API dev install is not supported. * fix(sqs): describe the enable refusal as an unsupported launch contract A missing or unknown contract proves neither when a record was written nor that it predates authenticated bootstrap: an older writer can drop a newer field, and a future contract is not pre-API. The refusal is unchanged; its message, status field and docs now name a missing or unsupported launch contract and say any existing volume is kept. * wip: port reviewed SQS serving path onto dev without the store VM * wip: single SQS system VM * fix(sqs): retry receipt bootstrap under supervision and drop store install path Run the receipt-key boot step from the respawned store service so a transient gateway or IMDS outage recovers without a reboot; final key and volume refusals stay refused. Remove the separate store image install and staging path from setup. * fix(sqs): align worker ready config fields * fix(sqs): stop an in-progress boot before unmounting the worker disk The receipt-key boot step ran as an unrecorded child of the store shell, so shutdown stopped only the shell and could unmount the disk under a boot still talking to the gateway. Record the boot child like the other guest processes and stop it before unmounting; a survivor keeps the disk mounted. * fix(sqs): record the boot child before its stopping check The parent wrote boot.pid only after starting the child, so a shutdown in that gap could unmount under an unrecorded boot. The child now records itself and checks stopping before it runs sqs-worker boot. |
||
|
dependabot[bot]
|
bbd4e84f15 |
chore: bump the docker-images group across 3 directories with 3 updates (#1375)
Bumps the docker-images group with 1 update in the /build directory: golang. Bumps the docker-images group with 2 updates in the /docs/terraform-workbooks/demo-app directory: golang and distroless/static. Bumps the docker-images group with 1 update in the /scripts directory: debian. Updates `golang` from `972b411` to `d9b6a1b` Updates `golang` from `f2c872b` to `d200d2d` Updates `distroless/static` from `18f4cc2` to `25ac8f0` Updates `debian` from `62bd423` to `f6b08b0` --- updated-dependencies: - dependency-name: golang dependency-version: 1.27.3-bookworm dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: golang dependency-version: 1.27.3-alpine dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: distroless/static dependency-version: nonroot dependency-type: direct:production dependency-group: docker-images - dependency-name: debian dependency-version: trixie-slim dependency-type: direct:production dependency-group: docker-images ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Ben Duncan
|
d420bff09c |
fix(eks): delete the Node of a terminated nodegroup worker (#1392)
* fix(eks): delete the Node of a terminated nodegroup worker K3s runs no AWS cloud provider, and its own reports every instance as existing, so a worker terminated by DeleteNodegroup, scale-down or reclaim stayed a NotReady Node forever. Its pods stayed bound to it, and once most Nodes were NotReady the node lifecycle controller stopped evicting, leaving metrics-server on a dead Node. Add an internal-workers route the control plane pulls the cluster's live nodegroup workers from, and a mulga-eks-node-gc agent on every server that deletes a nodegroup Node only when it is not Ready and its instance is no longer one of those workers, as the AWS cloud controller does. * test(e2e): check the live worker's Node and wait out nodegroup deletes WorkerNodeReady accepted any Ready Node, the tolerate-all DaemonSet check dropped pods with no Ready condition, and nodegroup cleanup returned before the nodegroup was gone. Scope the Ready check to the CPIsolation nodegroup, count a pod with no Ready condition as not Ready, wait for each nodegroup delete, and assert that deleted nodegroups leave no Node behind. * test(integration): account for the internal ListInternalWorkers route in the cross-tenant table |
||
|
Ben Duncan
|
6c3d6c6be3 |
fix(setup): keep NetworkManager from replacing the resolved stub link (#1391)
NetworkManager restarted while /etc/resolv.conf is a plain file runs in rc-manager=symlink mode and writes resolv.conf on its first DNS update, which can land after the installer links the stub. Write a conf.d drop-in (dns=systemd-resolved, rc-manager=unmanaged) before the restart, refuse a later file that overrides rc-manager, wait for NetworkManager to settle, and assert the link holds with bounded relinks. A rerun after an older handover adds the drop-in, reloads NetworkManager and relinks the stub. |
||
|
Ben Duncan
|
956b830f18 |
fix(firewall): keep conntrack running while the host policy is disarmed (#1379)
The kernel tracks connections only while some rule uses conntrack. With inet spinifex_filter unloaded nothing tracked, so OVN RAFT links formed while disarmed (or before the daemon's first arm) were first seen mid-stream at arm time. With tcp_loose a packet from the server end became a new flow from peer:6643/6644 to an ephemeral port and hit the drop rule, and RAFT lost its leader. Add table inet spinifex_conntrack (accept-only) and keep it loaded on every node: setup.sh loads it at install, the boot unit loads it unconditionally, firstboot loads it before OVN, and disable swaps the policy for it in one transaction instead of turning conntrack off. |
||
|
Josh Siviour
|
1cfaf15c9d |
fix: EKS control-plane VMs are not Kubernetes Nodes (#1373)
* test: pin control-plane no-agent config, etcd member removal and CP isolation e2e * test: tighten CP isolation e2e and assert a logged prune failure * fix: run EKS control planes without a k3s agent and prune etcd members directly Control-plane VMs no longer register as Nodes, so pods cannot schedule onto a network that cannot reach the workers or the advertised endpoint. With no Node to delete, a replaced peer's etcd member is removed by a new eks-etcd-member helper that matches exactly one member by peer-URL host. * fix: refuse to remove the local etcd member by ID, not only by IP Self-protection no longer depends on first boot deriving its own address: the helper asks the connected endpoint for its member ID and refuses a match on it. * test(e2e): align EKS suite with control planes that are not Nodes Assert zero Nodes and delivered addon objects before any nodegroup. Check Ready worker, metrics APIService, addon deletion and pod logs once a worker exists, scoping logs to Ready nodes and waiting 10m for metrics-server to reschedule. * docs: EKS control-plane hosts are not Kubernetes Nodes Document that kubectl get nodes lists workers only, add upgrade notes for existing clusters, and list the eks-etcd-member helper. * docs: explain why addon manifests keep the CriticalAddonsOnly toleration Comment-only. The toleration stays for clusters created before control-plane hosts stopped registering as Nodes. * test: cover the etcd member client and CLI against a TLS fake etcd The real etcd client and the command had no tests, leaving the PR below the diff coverage threshold. A fake etcd serving the Cluster and Maintenance APIs over mutual TLS drives the client with the K3s certificate layout, and main becomes a testable run function. |
||
|
Josh Siviour
|
e155bcf155 |
feat(sqs): queue and Lambda-facing message API foundation (#1364)
* feat(sqs): queue and Lambda-facing message API foundation Adds the SQS JSON 1.0 adapter (unregistered in production) with queue create, lookup, attributes, list, tags and the message operations a Lambda poller uses: SendMessage(Batch), ReceiveMessage with long polling and attribute selection, DeleteMessage(Batch) and ChangeMessageVisibility(Batch), backed by the queue record owner and the JetStream message store. String and Binary message attributes are validated, stored and checksummed; Number attributes are refused until AWS behaviour is confirmed. * test(sqs): poller lifecycle through the message owners Covers receive, release, redelivery, delete, visibility extension, queue binding of receipt handles and long polling against the real queue and JetStream message stores, from the message package itself. * test(sqs): move SDK coverage from the gateway to the SQS domain Drives the registration through dispatch.Invocation over the real queue and message owners, so the gateway package carries no SQS changes. |
||
|
Ben Duncan
|
b4dd606104 |
feat: run Spinifex on Vultr, with failover, workbooks and the fixes they needed (#1361)
* feat: Vultr reserved-IP provider and floating external-address pool Adds a source="vultr" external pool backed by Vultr reserved IPs: an API client pinned by captured responses, an in-memory fake with the measured semantics, and a floating allocator that creates, claims, reconciles and releases labelled addresses. OCI and Vultr claims now share one AddressClaimer pass. Also fixes the host firewall dropping guest IMDS when imds_host_meta_ip is set, by matching the pre-DNAT destination, and arms the Spinifex firewall in place of an active ufw. * fix: detach a Vultr reserved IP before deleting it Vultr was measured refusing delete of an attached address with 400 "Unable to detach subnet", so release, rollback and reconcile now detach first. A first claim after allocation logs at info rather than warn. * fix: release the auto-assigned public IP an Elastic IP displaces AssociateAddress restamped the ENI with the EIP and never handed back the address launch had assigned. Nothing referred to it afterwards, so it stayed allocated for good, and on a cloud pool every associate left a billed address. AWS releases it on associate; withdraw its NAT rule and release it, scoped to the ENI, leaving any address another associated EIP holds alone. * fix: retry Vultr's refused detach on reserved-IP detach and delete Vultr has answered a detach with 400 "Unable to detach subnet" and accepted the same detach moments later. Surfacing it left the address for reconcile to collect, up to twenty billed minutes later; retry it like a 5xx instead. * fix: stop reporting the released launch address after an EIP is disassociated Associating an EIP now releases the auto-assigned public IP, but the stored instance kept its launch stamp, and DescribeInstances fell back to it once the EIP was disassociated. The public-IP change handler now moves the stamp with the runtime address. * feat: add a Vultr deploy script for a single-node Spinifex on a VX VM deploy.sh up/destroy/status drives Terraform for the VPC and VM, installs and forms Spinifex over SSH, copies the API key to the node rather than through user-data, and sweeps boot volumes and reserved IPs Terraform does not own. Cloud-init keeps needrestart off the Spinifex, OVS and OVN units and disables unattended upgrades on these disposable hosts. * feat: form a three-node Vultr deployment with install-node.sh over the VPC Each node comes up as an armed single node, the state install-node.sh forms from, with the cluster and Geneve planes on the VPC NIC. The pool and every node's provider_host_id go on each node after formation. * fix: read the pinned host key with ssh-keygen -F when forming a Vultr cluster ssh_config hashes known_hosts entries by default on Debian, so a grep for the address found nothing and set -e ended the run without a message. * docs: add the Vultr guide, one command to deploy one or three nodes and one to remove them * docs: drop the failover write note from the Vultr guide * fix: keep needrestart from restarting the Spinifex, OVS and OVN units Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's replaced libraries to the daemon and restarted it after every dpkg run. The restart never cleared them, so an unattended upgrade ran the unit into its start limit and left the control plane down. setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists them, so restarting after an upgrade stays the operator's choice. * chore: drop the Vultr cloud-init needrestart override now setup.sh writes it * docs: replace the Vultr guide's apt upgrade warning now the installer keeps needrestart off the stack * docs: install the Vultr guide's build from the dev channel until Vultr support is released * fix: answer ARP for Vultr reserved IPs on bare metal Vultr routes a reserved IP to a VM by MAC but ARPs for it on a bare-metal server's public segment, and a Vultr pool gave vpcd neither a gateway nor a bind bridge to proxy-ARP on, so nothing answered and no traffic arrived. A Vultr pool now proxy-ARPs on the interface the node routes the internet through. setup.sh installs arping, which the gratuitous ARP already called. * feat: deploy Spinifex on Vultr bare metal, renamed deploy-vultr.sh A vbm- plan creates bare-metal servers, one or three. Without --name the deployment writes the AWS profile spinifex, up prints every node's address and the API key allowlist, credentials and ssh subcommands are added, no AMI is imported unless --image is given, and OpenTofu or Terraform is used. The host key is pinned only once cloud-init is done, since bare metal first answers SSH from Vultr's installer with a different key. * docs: rework the Vultr guide into the OCI guide's flow, with bare metal * fix: restamp LaunchTime when a stopped or recovered instance relaunches The pending watchdog times a launch from LaunchTime, which still held the original launch. An instance recovered from a failed node, or started more than five minutes after it was first launched, was terminated as stuck in pending seconds after QEMU started. AWS restamps LaunchTime on every start. * feat: spread the nginx-alb workbook across nodes, and power Vultr nodes off and on from deploy-vultr.sh The workbook takes server_count and placement_strategy, so three web servers can run one per node. Its README now matches the template (public subnets, no NAT gateway) and the diagram shows the spread. deploy-vultr.sh gains stop and start, which halt and start a node through the Vultr API. * fix: let a terminate finish tearing down before the reaper calls it wedged OrphanQEMUReaper finalized any shutting-down instance whose QEMU had exited, on the premise that a live terminate still has a live QEMU. A terminate is still deleting volumes, releasing its IP and leaving its placement group for tens of seconds after QEMU exits, so the instance was reported terminated first. Terraform then deleted the placement group and got InvalidPlacementGroup.InUse. The reconcile now waits two minutes from shutting-down, well inside the ten-minute stuck-terminate backstop. * fix: release an ovs-vswitchd left in flow-restore-wait after a node boots ovn-controller clears flow-restore-wait once its flows are installed and sets it again straight after. When both writes reach ovs-vswitchd as one update, it only sees true, starts no upcall threads and forwards nothing. Seen on Vultr bare metal after a hard power-off: instances recovered onto the rebooted node ran but were unreachable. vpcd now clears the flag on any node held that way for a minute, longer than ovs-ctl's own restore takes. * docs: add a resiliency test to the Vultr guide and the nginx-alb workbook The Vultr guide gains step 8, which spreads three web servers across a three-node deployment, powers a node off and back on, and shows what was measured on VMs and bare metal. The workbook gains the same test as step 6. The guide's table of contents is repaired, and its limits now cover losing two nodes and recovery inside a spread placement group. * fix: never terminate an instance that has run because a relaunch failed The pending watchdog terminated any instance pending past five minutes, and terminating deletes DeleteOnTermination volumes. A start of a stopped instance, a recovery from a failed node and a daemon relaunch all pass through pending, so a slow one destroyed an existing instance and its root disk. On Vultr a recovered instance was terminated seconds after QEMU started and its root volume deleted. An instance is now marked FirstLaunch from RunInstances until it first reaches running, cleared before the running state is written. The watchdog acts only on first launches, as AWS terminates only a launch that never ran. MarkFailed refuses to terminate any other instance and marks it recovery-failed instead, which keeps its volumes, ENI and addresses and shows it as stopped. A claimed relaunch is never a first launch. Invariant tests pin ADR-0003:S7. The instancerecovery e2e suite gains a test that ages its guest past the watchdog before recovering it, then stops and starts it, and requires the instance, its root volume and the data on it to survive both. * feat: run the Terraform workbooks against a Vultr deployment on demand, and show the Vultr portal in the guide e2e-cloudvendor-vultr.yml is dispatch-only. It builds the ref, deploys it with deploy-vultr.sh on one or three VMs or bare-metal servers, runs the published workbooks from node1 with the same driver as the hypervisor nightly and OCI, and destroys the deployment, failing if anything is left billing. It needs one secret, VULTR_API_KEY, and generates its SSH key per run. The guide gains screenshots of the API key's access control list and of the nodes in the Vultr portal, and its portal navigation now matches. * fix: install setup.sh on Vultr nodes when the tarball does not carry it build-install-artifacts.sh writes setup.sh beside the tarball, so a node installed from its tarball had no /usr/local/share/spinifex/setup.sh and arming the firewall failed. * fix: pass the remote command after the destination in the Vultr e2e driver * fix: run each workbook in the cluster's own region The workbook driver passed no region, so every workbook used its default, ap-southeast-2, and on a cluster in any other region every request failed SignatureDoesNotMatch. It now passes the region of the node's spinifex profile. * feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) A node counted physical cores, collapsing SMT threads, and held back a fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is now one online CPU, as AWS counts it: a thread with SMT on, a core with SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and the rest is allocated one to one. The existing overrides stay operator-only. Invariant tests pin every tier boundary. * docs: state the Vultr guide's sizing under the online-CPU count * docs: drop an unmeasured EKS sizing claim from the Vultr README * fix: let a guest reach an Elastic IP hosted on its own node A guest's packet to an EIP on its own node left its VPC from the transit range, was masqueraded to the transit host address, and went back into OVN, which exempts that range from EIP DNAT, so it was never answered. On Vultr an EKS worker could not reach its control plane's endpoint, so it never joined. AWS lets an instance reach a public address in its own VPC. Hairpinned transit traffic now jumps to SPX-EIP-HAIRPIN, above the egress MASQUERADE, where each EIP's rule sends it from the uplink address, as the host's own traffic to the EIP already is. The rule follows the EIP's ingress plumbing in and out, and anything else returns to the MASQUERADE. * fix: resolve Spinifex's LB and EKS names on Vultr nodes Vultr's Ubuntu image writes a static resolv.conf naming Vultr's resolvers, so lookups bypassed the split DNS Spinifex configures in systemd-resolved and an EKS endpoint name did not resolve on the node. deploy-vultr.sh points resolv.conf at the resolved stub, as stock Ubuntu does. * test: prove a guest reaches an EIP on its own node in the natuplink e2e The nat-single cells run it with a public pool, so the hairpin path that left EKS workers unable to reach their control plane on Vultr is now covered nightly. * docs: record the natuplink hairpin check in the e2e coverage index * fix: keep each Vultr node's logs when an e2e run fails The driver destroyed a node whose install failed together with the log that said why. It now copies /root/*.log from every node into the run's artifacts before the teardown, and the workflow uploads them. --------- |
||
|
Ben Duncan
|
8839407c71 |
feat(ui): console design system, side-panel resource views and live EC2 metrics (#1358)
* feat: lay the Spinifex UI design-system foundation Brand tokens in OKLCH with semantic status and a colour-blind-checked chart palette, Archivo headings and a 13px type scale, and a pre-paint theme script so dark mode never flashes white under the console CSP. Adds the missing Base UI primitives, a /design catalogue route, a route ratchet against hand-written tables, a ban on chart-library imports in routes, and lets the content column shrink so wide tables scroll in place. The dev server now forwards /api and /proxy to the Go console. Sonner is removed: its inline style element is blocked by the CSP. * feat(ui): side panels, TanStack table, uPlot charts and e2e gate - DataTable renders @tanstack/react-table, with column hiding as container queries so columns drop when a side panel narrows the list. - UniFi-style DetailPanel and ResourceView: Instances, VPCs and Internet gateways open in a panel beside the list from 1024px and as a full page below; selected, tab and create live in the URL. Create flows for instances, VPCs and gateways open in the same panel. - Instance monitoring charts on uPlot (CPU, network, packets), on simulated data and badged as such. - Dates in the AWS console form, e.g. 2026/06/04 02:00 GMT-5. - Tabs style Base UI's data-active, so the selected tab now shows. - Playwright suite gates CI as the UI End-to-End job in ui.yml. * feat(ui): one details component for panel and page, UniFi-style overlay panel The side panel now lies over the list at full height with a shaded edge, and a click on the list's background closes it. Instances, VPCs and internet gateways each render one details component as either the panel or the resource's own page, so the page gains the monitoring graphs and the panel gains IAM role, GPU, console output, change type and create image. SSH moves into Instance information as rows. Internet gateways attach and detach from the panel. * feat(ui): roll the panel pattern out to every service and gate screenshots in CI Every EC2, IAM, Route 53, Bedrock, EKS, ECS and RDS resource is now a list plus a side panel, with one details component serving as both the panel and the one-column page. DetailCard, DetailRow, ListCard, ResourceList and DeleteResourceButton are deleted. - Panel: slide-out close, row selection closes it, expand keeps the tab and fetched console output, PanelSpinner while loading. - Instances: info tips, theme-aware coloured console output, power button group plus Actions menu, Stop and Reboot confirms, security group, Elastic IP and network interface sections, and a live one-second monitoring window on simulated data. - Launch instance: OS tiles picking the newest version, architecture choice, location tile, expanded root volume, honest default subnet. - On/off settings are switches app-wide; disabled controls show cursor: not-allowed. - The e2e job runs in the pinned Playwright image with screenshots gating; make ui-baselines and make ui-e2e-container use the same image. * feat: bulk select and delete on every resource list, OS marks and instance type sizes Every list with a delete action gets row checkboxes and a selection bar. Delete confirms in a dialog naming each resource, makes one call per resource, keeps each refusal against its row and retries only failures. The Launch form shows distribution marks for Ubuntu, Debian, Rocky, Alpine and CentOS, served as files under the CSP, and states the selected instance type's vCPUs and memory. * feat: sign-in overlay, larger type scale, typed confirms on irreversible deletes and dialog shortcuts The sign-in card sits over an inert shell. Any sidebar group folds, and entering one unfolds it. Instance DNS names copy. Escape cancels and Cmd/Ctrl+Enter confirms every dialog. Volume, snapshot, RDS, EKS, ECS and hosted zone deletes need "delete" typed, and terminate needs "terminate". * feat: fade sign-in and sign-out in place, and show the confirm shortcut Signing in no longer reloads the page: the card and a lighter veil fade out and the console routes to the dashboard. Signing out cross-fades into the sign-in card with a Signed out notice. Confirm buttons show Cmd or Ctrl Enter for the platform. * feat(ui): generic sidebar group names and a paged S3 object browser The sidebar names its groups Compute, Databases, Authentication and Object Storage; the command palette still finds each by its AWS service name. Buckets get row selection and delete. Inside a bucket, folders and objects share one table paged by ListObjectsV2 continuation tokens, with a find box that queries by prefix, multi-select delete through DeleteObjects (folders with everything under them), and an object panel showing HeadObject properties, user metadata and a download. * feat(ui): show upload progress and stop the S3 proxy cutting off slow transfers The Upload button shows a running batch's percentage, and a large or slow batch opens a dialog with bytes sent, bit-rate, time left and per-file cancel and retry. Uploads run in a store outside React, so they continue while the user browses. The console's S3 proxy now allows an hour per request. The server's 15 s ReadTimeout cancelled every upload part that took longer, so large uploads failed on links below roughly 18 Mbit/s. * feat(ui): stream live EC2 metrics to the Monitoring tab The Monitoring tab drew simulated data. It now streams CPU, network bytes and packets from the owning node's qmp-collector every 500 ms while someone is watching, with nothing stored. - qmp-collector: a watched sampler on schedstat and tap counters, on a 30 s lease, keeping five minutes in memory for reconnects. It answers only watches for the account in the VM's discovery file. - awsgw: spinifex:GetInstanceMetrics, checked against the instance ARN, answered as text/event-stream in 60 s segments. Another account's instance is InvalidInstanceID.NotFound. - console: a signed stream reader that reconnects with Since and backs off on failure or an empty segment, stops on InvalidAction, and a chart axis that scrolls between samples. The simulator and its badge are removed. * feat: catch refused creates in every form, answer the third review, and guard tables Every create form now catches a refused mutation, so the failure shows beside the submit button and no longer escapes as an unhandled rejection. Three EKS tabs that swallowed a refused create now show it. Third-review answers: the attach dialog offers running or stopped instances in the volume's zone; the default security group's delete is disabled with its reason; the load balancer panel splits into Listeners and Attributes tabs; a main route table lists its implicit subnets; IAM single delete dialogs name the same dependants as the bulk ones; a snapshot can be deleted in any state. Every list in a route renders through DataTable, enforced by a unit test with a named allowance, and the /design baselines are redrawn in the pinned container. * feat: cap live metrics streams, prune hidden selections, and move panel focus Each gateway holds at most 10 live metrics streams per account and 200 in total, refusing past either with RequestLimitExceeded, and counts opened and refused streams. A row that leaves a list leaves its selection, the instances page uses the shared selection hook, the detail panel takes focus on open and returns it on close, and Escape yields only to the popup it was pressed in or a modal. The UI workflow also runs on live metrics backend changes. |
||
|
Ben Duncan
|
9d9e3aab6a |
fix: keep needrestart from restarting the Spinifex, OVS and OVN units (#1341)
Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's replaced libraries to the daemon and restarted it after every dpkg run. The restart never cleared them, so an unattended upgrade ran the unit into its start limit and left the control plane down. setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists them, so restarting after an upgrade stays the operator's choice. |
||
|
Josh Siviour
|
d066786faa |
feat: operator-launched SQS worker and store system VMs (#1353)
* feat: persist the SQS worker's store key on its own disk The shared SQS worker generates its store-client NKey inside the guest and keeps it in an owner-only directory on its own persistent disk, so no platform component creates or unseals the seed. The key is synced and linked into place without replacing anything; a missing, mismatched, unreadable or loosely permissioned key fails closed instead of being regenerated. * fix: make the SQS worker key durably named before returning it Every successful call now syncs the key directory and its entry in the parent before handing out a key, so a retry after a failed directory sync cannot return a key whose name is not yet durable. Only the key directory is created, inside a parent that must exist, so an unmounted worker disk fails closed. The first-start decision is explicit: an existing store's authorized key is required and never defaults to generating a new one. * docs: state that workerkey cannot detect an unmounted worker disk An existing parent directory does not prove the worker disk is mounted, since the mountpoint can exist on the initramfs after a failed mount. The error now says only that the parent is missing, and the package documents that its caller must verify the expected data volume is mounted first. * feat: add the SQS worker guest bootstrap The shared SQS worker microVM boots with sqs-worker boot, which selects exactly the disk whose serial fw_cfg names, checks and mounts it at /data, and proves from the mount table that this disk is what is mounted before loading or creating its store key there. A missing, empty or unknown store status, a contradictory or invalid store key, a missing or ambiguous disk, a failed check or mount, an empty mountpoint or a foreign mount refuses without creating a key. Only the public key is logged and published. The image is built by the store image script under SQS_MICROVM_PROFILE=worker, with the worker binary, no nats-server and an empty /data mountpoint; the store image's contents are unchanged. * fix: refuse replayable new-store status at SQS worker boot A static fw_cfg store-status of "new" survives a guest reset, so after the store exists it could still reach key creation and replace a lost worker key. Boot now accepts only "existing" with the store's authorized public key and refuses "new" before touching any disk, until a current authoritative first-store decision exists. workerkey.ForNewStore remains, uncalled. * feat: add SQS first-store claim and worker key acquisition The SQS domain records one first-store decision per Region in its own KV bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound to the worker's instance ID, claim ID and generation. Key creation is authorized only while no public key is recorded; afterwards only the recorded key is returned, and a launch intent requires it. Older generations, same-generation rivals and unreadable records authorize nothing, and records hold no private key material. workerkey.Acquire syncs a new key to disk and returns it only after its public key is recorded, reusing the on-disk key after a crash before recording and failing closed on a missing or different key afterwards. No transport carries the decision yet. * fix: read SQS first-store decisions from the stream leader KV.Get uses a direct get, which a lagging replica can answer, so a stale claimed read could authorize key creation after the leader had recorded the worker's public key, or make a superseded generation look current. Reads now use the SQS store's leader-answered stream message get, and every authorizing answer is also confirmed by a compare-and-swap at the revision read, so a stale or unavailable read authorizes nothing. * feat(sqs): add regional worker identity record and instance index The SQS domain keeps one worker identity record per Region with a monotonic generation, claim ID, pinned node and encrypted data volume, moving from launching to bound or failed under compare-and-swap. A launch intent takes the next generation before any instance exists, so the previous worker stops being current; binding writes the instance index entry first. Authorize accepts an instance only while the Region's leader-read record is bound to exactly that instance, generation and claim, confirmed by a compare-and-swap at the read revision, so an old, orphaned, rewritten or unbound index entry, or a lagging replica, authorizes nothing. Both this bucket and the first-store bucket now open through the kvutil chokepoint and refuse a stored configuration that could expire, cap or evict records or holds fewer replicas than the cluster needs. Nothing calls the store yet. * fix(sqs): make worker launch exclusive and harden durable buckets BeginLaunch now refuses a new claim while the previous attempt is launching or bound, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume. A replacement takes the next generation only after FailLaunch or the new RecordStopped confirms that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never saw. The durable bucket opener also refuses non-limits retention, per-message TTLs, asynchronous flush, a message-size cap, any discard policy other than discard-new or a per-subject discard, unacknowledged writes, a sealed stream, and a mirror or sources. * feat: add internal SQS worker first-store gateway actions Serve ClaimFirstStore and RecordWorkerPublicKey under the internal spinifex-sqs signing service, apart from the public AmazonSQS inventory. Only an sqsWorkerRole session in the system account passes the class gate, then the role policy; the worker identity is resolved from the current workerident record by the STS-set session name, never from the request, and any extra parameter is refused. Seeds and other non-public keys are refused without being recorded, logged or echoed. The gateway binds the worker-identity and first-store buckets per request and never creates them, so it provisions nothing and answers ServiceUnavailable until a worker launch has created them. * fix(sqs): fence replaced worker generations out of first-store A request that passed worker authorization before a confirmed stop and replacement could reach an unclaimed first-store record ahead of the successor and create or record the old generation's key. The launch path now raises a monotonic first-store fence past the replaced generation before BeginLaunch, which refuses a replacement until a leader read shows it; firststore refuses every call from a generation below the fence. The internal adapter no longer logs caller-chosen parameter names. * feat(sqs): launch the shared SQS worker on its pinned node Add the SQS worker system-instance kind and a domain launcher that realizes the Region's worker from its identity record without ever starting a second VM: - domains/sqs/workerlaunch returns a bound worker as is, prepares the data volume on first launch only, refuses a volume whose own record is not encrypted, marks the intent dispatched before launching and never resends a dispatched launch: an ambiguous outcome stays launching until reconciled. A replacement starts only after a confirmed stop or failure and the first-store fence, reusing the pinned volume. - workerident gains MarkDispatched and Current; firststore gains a leader-read RecordedKey. - The daemon launches the worker through the shared direct-boot path with its encrypted volume attached at boot and kept on terminate, a required instance profile and a mandatory mgmt NIC. Its firmware config carries only public values: the data serial, the store decision, the gateway URL on this node's bridge and the daemon's own cluster CA. A relaunch is refused on any other node. - sqsWorkerRole is ensured in the system account granting exactly the two internal actions; an existing role whose trust admits more than EC2 is refused. The role contract lives in domains/sqs/workerrole so the composition root does not import the gateway adapter. The worker network allocator and any caller remain unwired, and the guest still refuses to create a key for a new store. * fix: verify SQS worker role trust before granting and report unclaimed volumes Check that an existing or racing-created sqsWorkerRole trusts only EC2 before writing its policy, and accept the instance profile only when a re-read shows it carries exactly that role. Refuse the first launch before allocating when the cluster storage key is not configured, and return the ID of any volume created but not confirmed in a launch intent instead of dropping it. * feat: add private SQS system VPC and worker ENI allocator Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet from the VPC and subnet surface only, so it cannot create an internet gateway, route table, NAT gateway or EIP. The daemon's SQS worker network adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and finds or creates one ENI per launch claim in the VPC default security group, refusing a duplicated claim or an address outside the subnet. Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have recorded the volume. * feat: worker takes its first-store decision from the gateway at boot The SQS worker guest no longer trusts a static fw_cfg store status. After selecting, checking and verifying the mount of its own data disk, it asks the internal spinifex-sqs gateway service for its current first-store decision, signed with instance-metadata credentials only, over TLS 1.3 trusting only the cluster CA. It then creates and records or loads the key on that disk; a key lost after recording is never replaced. - internal/sqsworkergw: SigV4 client for ClaimFirstStore and RecordWorkerPublicKey; refuses non-public keys, non-decisions and redirects; errors carry only the parsed code - gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback - workerboot: retries only lost replies and ServiceUnavailable/Throttling - launch contract carries the Region instead of a store status or key * fix: worker retries a denial while its launch is still binding it The worker VM can reach the gateway before its launch binds the instance identity, and that early call is legitimately AccessDenied. The guest treated the denial as final and stayed unprepared even after binding. AccessDenied is now a pending authorization the guest asks about again within its existing bound; a denied claim creates nothing, and a worker that is never bound, or was replaced, gives up at the bound. * feat(sqs): add operator EnsureSQSWorker action and node launch handler Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the internal spinifex operator service. It validates the node against the cluster and as a single subject token, then requests sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers and the worker stays pinned. The node handler re-checks the named node and instance type and runs one composed workerlaunch.Launcher (real volume preparer, role, private network and direct-boot launch), one launch at a time, in the daemon's own context with a 30-minute bound. A caller timeout does not cancel the launch; a retry reports the bound worker, or IncorrectState for a dispatched launch whose outcome is unknown, which is never resent. The reply names the bound VM and does not claim the worker or SQS is ready. Nothing starts a worker at daemon startup. * feat: ship the SQS worker image and check it before allocating Build the SQS worker microVM image in both release workflows, stage it into the tarball as microvm-sqs-worker, and install it with setup.sh under the store image's rules: a release without it removes an earlier one and an incomplete one is refused. Add make install-sqs-worker-microvm. The worker launcher now asks a Host capability whether this node can boot a worker before it prepares a volume, records or resumes an intent, fences a replaced generation, allocates the network or dispatches. The daemon checks for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a call after the image is installed launches instead of leaving an ambiguous dispatched intent. Cover missing and incomplete images through the admin action and handler, a stopped worker left untouched, and an ENI create whose reply is lost. * fix: never repeat an uncertain SQS worker ENI create The launcher now looks for the claim's ENI without creating one, marks the launch intent NetworkAttempted, and only then sends the single create. Once marked, a missing ENI fails closed with IncorrectState instead of a second create that could give the claim two interfaces; a visible ENI is reused. Adds a test that buildSQSWorkerLauncher wires the daemon's real host check. * feat(sqs): operator-triggered SQS store launch on the worker's node Add the super-admin EnsureSQSStore action and its node handler. The store is launched only on the node the Region's worker is bound to, with the public key that worker recorded in the first-store decision; the daemon never receives the worker's seed. A per-Region store launch record marks each step before its uncertain create (volume, ENI in the worker's private subnet, VM launch), so a step whose result is not visible fails closed and is never resent. Volumes are refused unless their own record is encrypted, and their IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and instance types at or below that are refused; this is a floor, not a validated size. * fix: retry uncreated SQS store volumes and fix the store instance type A volume preparation that failed before sending any create (a missing tool, scratch or format failure, or no cluster storage key) now carries storevolume.ErrNotCreated. The store launcher withdraws its volume mark and returns ServiceUnavailable, so a retry may try again once the host is fixed; any create that was sent stays one-shot. The store launch record now keeps the operator's instance type. A request for a different type is refused as IncorrectState before dispatch and after binding, and the reply names the launched type, which the admin action checks. * feat: SQS worker store readiness and store size check The worker's one-shot ready step mounts and verifies its disk, asks the internal spinifex-sqs DescribeStore action for the bound store, loads its key only if it matches the recorded public key, and checks the store's JetStream with it. It retries within a bound while the store is pending or not answering, and stops on a mismatched or refused key. The NKey authenticates the worker only; the store server is not authenticated. EnsureSQSStore now also refuses a different volume size once the store is bound, instead of returning the existing store. * fix(sqs): keep worker store readiness current until shutdown The readiness step now runs as a guest-local watch loop with no overall deadline: each pass is bounded, a pending or unreachable store is retried every few seconds, and a ready store is rechecked periodically. The store-ready marker is removed on any failed pass, cleared at start and removed on exit; shutdown stops the watcher by its recorded PID. Terminal refusals such as a rejected or mismatched key back off for minutes rather than spinning. The NKey still authenticates only the worker to the store. * feat(sqs): supervise the worker readiness watcher and add spx admin sqs Run the worker guest's readiness watcher under init respawn, rate-limited, and remove any stale store-ready marker before each start, so a killed or crashed watcher cannot leave a false readiness marker. Shutdown blocks respawn, removes the marker, stops the recorded watcher by PID and start time with SIGKILL escalation, and unmounts /data only once it has exited; a surviving watcher leaves every mount in place. Add spx admin sqs ensure-worker and ensure-store, which call the existing EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex gateway path with the standard AWS credential chain. Both refuse missing inputs before sending anything and report only the bound VM, never readiness. * fix: park SQS worker readiness and fix the worker instance type A readiness start that finds shutdown in progress or no worker key now logs once and waits in place instead of exiting, so init's respawn does not restart and log it in a loop; shutdown stops it like a watcher. The worker identity record keeps each generation's instance type. A request for another type is refused as IncorrectState before dispatch and after binding, and only a replacement may change it. The ensure reply, admin action and spx admin sqs ensure-worker report and check the recorded type. * fix: record the SQS worker volume intent before creating it The first worker launch prepared its data volume before any record existed, so a create that applied but returned no ID left nothing for the next call to find, and it could prepare a second volume. The launch intent is now recorded first, and its one volume create is marked before it is sent. A Prepare proven to have sent nothing withdraws the mark and is retryable; any other outcome is never resent and returns IncorrectState, naming the volume when its ID is known. An intent that has sent no create may still be re-typed. * docs: document spx admin sqs ensure-worker and ensure-store Record the flags, the caller and owning accounts, the bound-versus-ready distinction and the current retry and recovery limits. * docs: distinguish SQS ensure timeouts from refusals that created nothing A gateway timeout means only that no reply arrived; the launch may still be running. The worker type locks once any step has been attempted. * docs: state what each SQS ensure ServiceUnavailable establishes No refusal proves earlier attempts created nothing; a retry checks the durable attempt and continues only when that is safe. |
||
|
Josh Siviour
|
488f1c7bf9 |
feat: SQS store guest on a prepared cold-attached EBS volume (not wired) (#1343)
* feat: build a separate SQS store microVM image Package the go.mod-pinned nats-server, ext4 support and a clean ACPI power-down into a microVM kernel and initramfs of their own. The build checks the packed archive's contents and refuses a nats-server whose version differs from the pin. The image does not start the store and nothing installs or releases it; the LB image is unchanged. * fix: publish the SQS store image only after its gates pass Stage the kernel and initramfs and move them into place only once the content, size and guest-RAM gates pass, so a failed build cannot leave a pair make takes as up to date. The old initramfs is removed first and the new one lands last, so an interrupted publish never leaves a mixed pair. Check the module list without piping into grep -q, which under pipefail reported a present module as missing. * feat: start the SQS store guest on its checked data disk The SQS store image now prepares, runs and stops a dedicated JetStream server. It reads a narrow fw_cfg contract (data disk serial, listen address, one user NKey public key), refuses anything absent, malformed or ambiguous, has the pinned server test the config, then checks and mounts exactly one matching ext4 disk. It never formats and has no ephemeral fallback. The server listens only on the supplied guest NIC address, admits only that NKey and syncs every write. Shutdown stops the recorded run process first, whether it is the server or a start not yet exec'd, and runs the generic read-only remount only once it is gone. A store that survives SIGKILL leaves every mount alone. * fix: report the SQS store as prepared, not listening, before it starts * feat: launch the SQS store guest on its own versioned subject An explicit SQS store kind carries an existing system-account data volume and the one NATS user public key the store accepts. It is served only on system.LaunchSQSStore.v1, so a daemon without this code has no responder and the launch fails instead of booting a load balancer without its disk; the legacy launch path refuses the kind on both send and receive. Every refusal happens before capacity, an ENI or a VM record exists. The volume goes through the normal EBS mount and terminate detach and is never deleted on termination. The store boots its own image with CPU mitigations on, gets only its network, disk serial, listen address and key, and cannot be relaunched after a host restart yet. * feat: ship the SQS store microVM image in releases and installs Release builds now build the SQS store image and stage it beside the load balancer's as microvm-sqs-store/, so an older installer that copies microvm/* without -r still works. setup.sh installs it to /usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it. A release without a complete image never leaves an older one usable: staging drops a previously staged image, and the installer removes an earlier installed pair before warning (missing) or failing (incomplete). Only the two image files are removed. The initramfs is installed last, so an interrupted install leaves no bootable mismatched pair. * feat: restart the SQS store guest on its launching node Persist the non-secret guest config of an SQS store instance (launching node, data volume, listen address, client public NKey and rendered network config) in the instance record, and rebuild its fw_cfg files from it on relaunch. The relaunch refuses on any other node, or when the record is missing or malformed, before writing any file; the data volume is remounted through the normal EBS path and kept on terminate. Cross-node recovery now never selects or claims an SQS store record, so no other node spends retry budget on it or relaunches it. Same-node restore is unchanged, as are load balancer restarts. Older daemons know neither the exclusion nor the record field, so this does not prevent a cross-node relaunch by a mixed-version cluster. * feat: hold prepared system volumes in creating until released Add an internal CreateVolumeForPreparation path that writes a new volume's first record as creating, and CompleteVolumePreparation to release an unattached creating volume in the caller's account to available. Ordinary CreateVolume is unchanged. Attach and delete already refuse a creating volume. ModifyVolume, ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState so nothing can expand, release or capture a volume mid-preparation. * feat: prepare fresh SQS store volumes with an ext4 filesystem The SQS store guest never formats a disk, so a new volume gets its filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img, builds an ext4 image in private scratch, creates the volume in "creating", publishes it on this node, copies the image with the NBD import helper, unpublishes, and only then releases it as available. It never writes a volume it did not just create. A volume is deleted only when no export can still be live: after a confirmed unpublish, or when the export was not NBD. Any publish error, a failed or ambiguous unpublish, or a failed release retains the volume and reports its ID. A publish refused as in-use or already-exists sends no unpublish, since another export may exist. Cleanup ignores caller cancellation but is bounded at 30s. The raw-image copy moves to runtime/compute/nbd unchanged, and the domain receives it as an injected writer. The volume service gains a discard path for an unattached volume still in "creating". * fix: report unknown state and match qemu-img lookup in volume prep A failed release may still have made the volume available, so the retained-volume error now says its state is unknown rather than claiming it is still creating. The copy runs qemu-img by name through PATH, so the pre-allocation check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin fallback because it runs from the resolved path. * docs: describe the unwired SQS store system instance Records what the guest image, versioned launch, same-node restart and fresh-volume preparation provide, that nothing calls them, and what is untested or missing before production use. |
||
|
Julian Sommer
|
cc1186fa48 | fix: grant the build user passwordless sudo for the appliance build when it lacks it, and remove the grant at seal | ||
|
Julian Sommer
|
dbdb827a69 | fix: install tar on a minimal Oracle Linux build system before copying the appliance payload | ||
|
Julian Sommer
|
71567d0998 | fix: show the appliance password rules before the first-login change | ||
|
Josh Siviour
|
11e8b68ddc |
refactor: remove the unused host SQS message store service (#1333)
* refactor: remove the host SQS store systemd unit The SQS message store will run inside system-owned instances rather than as a host daemon, so spinifex.target no longer starts it. Bump the target to version 4 and drop the unit from the embedded set and its invariants. * refactor: remove host SQS store config provisioning The SQS store will run in system-owned instances, so admin init and join no longer generate or distribute a host store token, routes or config, and the formation join payload drops the store token field. * refactor: remove host SQS store service command and role The SQS store will run in system-owned instances, so drop the unused spx service sqs-store command, its service registration and the host sqsstore role. The domains/sqs/message JetStream store is unaffected. * refactor: remove host SQS store provisioning from install scripts The SQS store will run in system-owned instances, so setup no longer creates the spinifex-sqsstore user, its directories or the 4223/4249 peer firewall ports, and install, reset and uninstall stop handling the retired unit, user and data directory. * docs: drop the host SQS store from firewall and connection tables Spinifex no longer ships a host SQS store service, so remove its ports from the host firewall and network connection references and say in DESIGN.md that no host service, unit, user or firewall rule exists for it. |
||
|
Julian Sommer
|
8f4754da13 | fix: install the appliance with the setup.sh from the release tarball, since install.mulgadc.com serves the stable one | ||
|
Julian Sommer
|
ac2620df58 | fix: report an SSH failure in build-oracle-appliance.sh as one, not as missing sudo | ||
|
Julian Sommer
|
2feb264863 | feat: Oracle Linux multi-node clusters with OVN IPsec (#1327) | ||
|
Josh Siviour
|
acfa078586 |
fix: take an EKS control plane out of rotation when its gateway is down (#1318)
* feat: add a gateway readiness endpoint to the EKS token webhook * feat: serve token webhook readiness beyond loopback and admit it to the control-plane SG * feat: health-check the EKS apiserver target group on the token webhook readiness port * style: order the strconv import * fix: export the readiness listener address to the EKS token webhook * docs: regenerate the ELB coverage README * docs: describe EKS control plane readiness and what happens when a gateway stops * test: range over SplitSeq in the webhook initd guard |
||
|
Julian Sommer
|
d953d0a973 |
fix: treat Ubuntu's openvswitch.service alias as the Debian OVS unit
Ubuntu 26.04's openvswitch-switch declares Alias=openvswitch.service, so enabling it leaves an /etc symlink that distro_ovs_unit read as a second, EL-style umbrella unit. setup-ovn.sh then refused to pick one and every ubuntu-26.04 bootstrap failed. A symlink back to openvswitch-switch.service now counts as the Debian unit; a real or masked openvswitch.service still does not. |
||
|
Julian Sommer
|
920f0487a3 |
feat: include the Route 53 API in the AWS operation coverage report
Vendor the route-53 2013-04-01 Smithy model at the pinned commit, add the Route53 service, and pass route53awsapi.OperationInventory() to the generator. coverage.json and docs/coverage/route53 now list the 13 served operations; the Spinifex activation extensions stay out of the model count. The generator now fails when a served service has no embedded model, so the next one cannot drop out of the report silently. Request encoding skips rest-xml by protocol, which covers Route 53 as well as S3. |
||
|
Ben Duncan
|
2989a55309 |
fix: purge a clustered OVN database under --recreate-db (#1301)
ensure_clustered_db_storage() collected only databases in standalone format, so --recreate-db returned early when the NB/SB databases were already clustered and removed nothing. ovn-ctl then served the stale database and silently ignored the RAFT flags, which is the failure the function exists to prevent, reached from the other direction. Because the old database carries the previous member list and server ID, the node rejoined the earlier cluster instead of replacing it: re-forming a production cluster brought the create node back as a follower, and ovn-nbctl over the leader-only unix socket failed with "database connection failed ()". install-node.sh passes --recreate-db on every database node on every run, so formation already intended a purge. Remove whatever NB/SB state is present, standalone or clustered, and keep refusing a standalone database when the flag is absent. Two further gaps in the same path. systemctl stop returns once the unit is inactive rather than once the processes are gone, so the purge now waits for the database servers and escalates to SIGKILL, failing the run if one survives rather than unlinking files under it. The pattern is anchored on the OVN database directory so it can never reap the OVS conf.db server. The lock files, the compute-node move-asides and an earlier run's standalone backup are cleared alongside the databases. |
||
|
Julian Sommer
|
7dde151766 | feat: Oracle Linux 10 single-node host support (#1270) | ||
|
Josh Siviour
|
e7a3be3b4e |
feat: add a dedicated SQS message store JetStream server (#1299)
* feat: add a dedicated SQS message store JetStream server separate from control-plane NATS * test: cover the SQS store server start, validation and memory limit |
||
|
dependabot[bot]
|
8de934767d |
chore: bump the docker-images group across 3 directories with 3 updates (#1246)
Bumps the docker-images group with 1 update in the /build directory: golang. Bumps the docker-images group with 1 update in the /docs/terraform-workbooks/demo-app directory: distroless/static. Bumps the docker-images group with 1 update in the /scripts directory: debian. Updates `golang` from `5c3e9b4` to `972b411` Updates `distroless/static` from `e2e927e` to `18f4cc2` Updates `debian` from `a29215f` to `62bd423` --- updated-dependencies: - dependency-name: golang dependency-version: 1.27.3-bookworm dependency-type: direct:production update-type: version-update:semver-patch dependency-group: docker-images - dependency-name: distroless/static dependency-version: nonroot dependency-type: direct:production dependency-group: docker-images - dependency-name: debian dependency-version: trixie-slim dependency-type: direct:production dependency-group: docker-images ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Adam Szynkowski
|
8b3b23deff | feat(oci): support resumable OCI topology validation with template (#1231) | ||
|
Julian Sommer
|
eccbf48273 | test: stub sudo so the uninstall multi-node refusal check runs on any host | ||
|
Julian Sommer
|
29770dc798 | test: stub ip and ovs-vsctl so the setup-ovn teardown order check runs on any host | ||
|
Julian Sommer
|
be84b73abe |
fix: rebuild the initramfs after upgrading the kernel in system AMIs
On Alpine 3.21 apk does not run the mkinitfs trigger inside the libguestfs appliance, so apk upgrade left the old initramfs beside the new kernel. The guest could not load virtio_blk and dropped to the emergency shell, which stalled EKS cluster creation. Run mkinitfs for the installed kernel and fail the build if the initramfs lacks its virtio_blk module. |
||
|
Julian Sommer
|
3dbe352d1b |
fix: ship patched guest kernels in system AMIs and the LB microVM (#1222)
build-system-image.sh never upgraded the base cloud image, so every Alpine system AMI kept the kernel its base image was released with. Run apk upgrade before installing manifest packages so rebuilt AMIs carry the branch's current linux-virt (6.18.57-r0 on 3.24, 6.12.113-r0 on 3.21; CVE-2026-99999). The ELBv2 microVM defaulted to Alpine 3.20, which is EOL and will never get a fixed kernel. Default to 3.24. |
||
|
Julian Sommer
|
efdd255435 | chore: drop the temporary vuln-scan acceptance list now Go is current | ||
|
Josh Siviour
|
b5879f44ff |
ADR-0001 package-boundary migration checkpoint (#1190)
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in
|
||
|
Julian Sommer
|
02c02f6632 | chore: pin base images by digest and let dependabot refresh them | ||
|
Julian Sommer
|
ebc36d6f0c | chore: cleanup stale scripts | ||
|
Julian Sommer
|
de4be25d80 | chore: go 1.27.2 | ||
|
Ben Duncan
|
c4f92608e1 |
fix: let the OCI principal topology pick a shape its tenancy has quota for (#1185)
Service limits are per tenancy. vm-single-principal now builds in its own, so the shape the other topologies share can have zero quota there, and the apply then fails before anything about the instance principal is exercised. Measured in ours: standard-e6-core-count is 0 available, against 13 for e5 and 10 for standard3. The first principal run died on 400-LimitExceeded for Core Instance, which reads as a capacity problem and is a shape with no quota. OCI_PRINCIPAL_SHAPE overrides it for that topology alone and defaults to the shape every other topology uses, so nothing changes where quota already exists. |
||
|
Ben Duncan
|
8409799cbf |
fix: default the OCI helper to the DEFAULT profile (#1174)
* fix: default the OCI helper to the DEFAULT profile oci_env.py shipped our own internal Oracle compartment profile as the default for every reader, with a second internal name as a fallback. A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name nobody else can have, and the fallback chain could land a deployment in whichever tenancy a different profile happened to point at. The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a named profile the config does not hold is an error rather than a fallback. `DEFAULT` needs its own existence test, because configparser treats it as the fallback for every other section and `has_section` denies it exists even when `oci setup config` wrote nothing else. `eip-move-test.sh` and the config table in the README carried the same name. CI reads its credential from the environment rather than from a profile, so nothing there referenced it. * docs: look the OCI compartment OCID up by name The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and the `oci` CLI it already requires can resolve the name instead. The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0 and prints nothing when the name matches nothing, so an unset `$CID` would otherwise reach Terraform and fail with an OCI error naming neither the compartment nor the cause. Two compartments in one tenancy can share a name, which the listing command shows and the lookup cannot resolve, so the paste form stays as the answer for that case. Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to whatever `$CID` holds. * Revert "docs: look the OCI compartment OCID up by name" This reverts commit |
||
|
Ben Duncan
|
7c56855526 |
fix(oci): both credential routes in the guide, and a nightly topology that proves the principal one (#1173)
* docs(oci): offer both credential routes, and default the guide to the release Step 2 was one route only, the scoped API key, with the instance principal mentioned in a TIP at the end of it. It is now "2. OCI Credentials for Spinifex" with the two routes as 2.1 and 2.2, a table to choose between them, and the instance principal first because it leaves no key on any node. The guide also assumed the dev branch throughout. `git switch dev` and `--channel dev` are now an optional note for installing a pre-release, and the documented path is the published release. * test(oci): add a vm-single-principal topology, and cover the credential branch Instance-principal authentication was broken for a release while the OCI nightly stayed green, because the run's two axes never crossed: every topology that built the address allocator authenticated with an API key, and every run that authenticated as the instance was dispatched with no external pool, so the allocator was never constructed and the credential never used. vm-single-principal closes that. It is vm-single with the credential fixed by the topology rather than by a separate switch, so the name alone decides it, and pairing it with --no-external-pool is refused rather than run: without the allocator there is nothing for it to exercise. It joins the nightly's default list, and naming it alone is the way to test this one thing in a quarter of the full run. A unit test covers the same branch without a tenancy. newClient now reaches its two constructors through package variables a test can swap, which is the only way to observe the choice off an OCI instance - both real constructors need either an instance identity or a key file on disk. * fix(oci): let the tfvars choose the credential, and stop the guide hardcoding paths The deploy driver passed instance_principal as a -var, which outranks a tfvars file, so a deployment that set it there had it silently replaced with off and the nodes went looking for a key. Same defect class as the shape and node count, and the same fix: the flag is a choice about one run and still outranks everything, and without it the deployment's own tfvars decides. The driver now reads instance_principal from terraform.tfvars and *.auto.tfvars for its pre-apply gates, taking the last assignment as Terraform does, and checks that guess against a new Terraform output once the apply has run. They have to agree, because the credential handoff is skipped on the strength of it and a wrong answer leaves a node with no credential at all. The guide now carries one export rather than two more steps. SPX_PRINCIPAL is set in step 2.1, expanded into the tfvars step 4 already writes, and read back from there in step 5, so an instance-principal deployment runs the same command as a key-based one. Getting into the node needed an address copied out of the log. Every topology now writes its hosts file, not only the multi-node one that hands it to install-node.sh, and the run prints the ssh and destroy commands when --keep leaves the infrastructure up. The guide exports TOPOLOGY once, so the paths read the same for one VM, three, or bare metal. * fix(oci): print the ssh key path as ~ rather than expanded The --keep line read ssh -i /home/you/.ssh/oci-spx, which is the invoking user's home and differs on every machine. Collapsed to ~ when the key is under $HOME, left alone when it is not. |
||
|
Ben Duncan
|
9a8b63b127 |
fix(oci): run the deploy driver on the bash macOS ships, and finish the guide at the handoff (#1168)
* fix(oci): run the deploy driver on the bash macOS ships `validate-topology.sh` died on line 41 with `bm: unbound variable` on a Mac. macOS ships bash 3.2, where `declare -A` is an indexed assignment and the subscript is read as an arithmetic expression, so `[bm]` dereferenced an unset variable and `set -u` ended the run. `mapfile` further down would have failed next. The guide names macOS as a supported workstation, so the driver now uses neither: the per-topology defaults are a `case` in a function, and the host list is a read loop over a captured string. The output is captured before the loop so a failed `terraform output` fails there rather than silently producing an empty host list. Verified: all three topologies resolve their shape and node count, and an unknown topology still names the valid set. * docs(oci): end the OCI deploy where the install guides end The deploy command now carries --skip-workload, so the guide stops at a formed cluster and a working AWS call rather than at a workbook run, and hands off to Setting Up Your Cluster the way every other install guide does. A step to SSH to the node was missing between the deploy and the verification call. Three things this corrects while the section was open. The expected output block now matches the command it documents, including the --skip-workload line that ends the run. The --channel dev callout said OCI support was missing from the deploy driver, when what is missing is an OCI address allocator in a published release. The verification output was fenced as json while holding a shell session, which is neither valid json nor the thing being shown. The note on probing a guest's public address from outside the VCN moves to the handoff section, since that is now where a guest first gets launched. |
||
|
Ben Duncan
|
7d190fc1f1 |
fix(oci): let terraform.auto.tfvars decide the shape and the node count (#1167)
`validate-topology.sh` passed the topology's shape and node count as `-var`, which is the strongest source Terraform reads, so a `compute_shape` or `node_count` in `terraform.auto.tfvars` was silently discarded. The OCI guide told the reader to write both into that file and then to run the driver, so a deployment asking for `VM.Standard.E5.Flex` got `VM.Standard.E6.Flex` and said nothing about it. Three of the five shapes the guide recommends were unreachable by the command it gives, including both bare-metal ones other than the shape CI happens to use. They now go in as `TF_VAR_compute_shape` and `TF_VAR_node_count`, the weakest source, so the tfvars file wins and a deployment needs no flag to size itself. CI is unaffected: `*.auto.tfvars` is gitignored, so a runner's checkout holds no such file and a named topology still resolves to the same values every run. `instance_principal` stays a `-var`, because `--instance-principal` is a choice about the run rather than about the infrastructure's size and a stale tfvars must not quietly contradict it. Because the count can now come from the file, the driver no longer presumes it. The number of addresses in the `hosts_file` output is the authority for the single-node init path and the readiness gate, a divergence from the topology's default is logged rather than fatal, and a new `compute_shape` output lets the run report the shape it built instead of the one it asked for. In the guide: the tfvars heredoc is unquoted so `$CID` expands, with a `grep` to confirm it did, and `node_client_cidr_allow_list` is bracketed, since it is a `list(string)` and the bare CIDR was an HCL syntax error. |
||
|
Ben Duncan
|
4fbde4a665 |
docs(oci): ship the credential installer, and rewrite both OCI guides in the house voice (#1166)
* docs(oci): rewrite both OCI guides in the house voice Seventy-five em dashes across the two guides, with four of them in headings where GitHub and the docs site slugify them differently and a cross-link resolves on neither. Recast as commas, colons and separate sentences. The silent-credential failure was stated five times as the same triple, "forms, passes every health check, and then refuses every launch". It now reads in full once, under Credentials, and the other four refer back to it instead of echoing the skeleton. Also broken up: the parallel stack in "Why run Spinifex on OCI", several five-sentence paragraphs, and a handful of clauses that leaned on a dash where a full stop was clearer. No technical content changed. Every inline-code token is byte-identical before and after, every fenced block is unchanged apart from one comment line, and the four reworded headings carry no inbound anchor links. The docs build reports zero warnings for either page and both render at the same size as before. * docs(oci): open the install guide with what Spinifex is, as the other install guides do * docs(oci): ship the credential installer instead of asking for a hook The guide told the reader to write their own executable to put the OCI API key on each node, which is work nobody should have to do to install a product, and the step was unclear about where the scoping actually happens. `scripts/terraform/oci-spx/spx-oci-config.sh` now does it. It takes `validate-topology.sh`'s hook contract, `hook <ssh-key> <host>...`, so it drops straight into `--credential-hook` and is also runnable on its own against an existing cluster. It resolves the credential in four steps and takes the first complete one: the `OCI_SPX_*` environment variables, the `[spinifex]` profile in `~/.oci/config`, `~/.oci/oci_api_key_spx.pem` with the Terraform profile's identifiers, then the Terraform credential itself with a warning that it is wider than the allocator needs. Steps 2 to 4 go through `scripts/oci_env.py`, the same resolver the apply uses, so the credential on the nodes and the one that built the infrastructure cannot come from different profiles. The fingerprint is checked against the key before anything is written, because a mismatched pair is otherwise an OCI 401 at the first allocation, long after the deploy has reported success. The node's filename stays `oci_api_key.pem` while the workstation's is `oci_api_key_spx.pem`: the suffix exists only to keep the Spinifex key off whatever `[DEFAULT]` uses, and a node has no other OCI key. Nothing reads the name anyway, since the daemon learns the path from `key_file` inside `/etc/spinifex/oci/config`. Key material reaches the nodes over SSH, in the remote shell's stdin rather than its arguments where `ps` would expose it, and goes into neither user-data nor Terraform state. `--credential-hook` still accepts any executable with that argument shape, so a tenancy that keeps credentials in a vault can substitute its own. Also in the guide: the policy statements name the compartment by `compartment id` rather than by name, which is the form that resolves from a compartment the policy is not attached at; `$CID` is consistently a compartment OCID; and the table of contents matches the headings again. |
||
|
Ben Duncan
|
46c0c44cc2 |
feat: run every Terraform workbook on OCI, and split the guide in two (#1163)
* feat: run every Terraform workbook on OCI, and split the guide in two The OCI topology driver ran the shared driver's default five workbooks, which left ECS and all three EKS variants untested on every platform even though their assertions already existed. It now passes an explicit nine and imports the three appliance images they need — RDS, ECS and EKS each boot their own, found by a spinifex:managed-by tag and never by name. demo-app stays out because it is not a workbook: it has no .tf at all, being the container image the EKS workbooks' nested workloads modules deploy, so listing it would fail on a missing root module rather than test anything. The guide is now two documents. oci-integration/README.md is the end-user path — six steps, deploy first, and it closes the three prerequisites nothing documented before: the clone, the ref, and the SSH keypair the driver refuses to start without. architecture.md holds the sizing, the IAM policy, every variable, the three filtering layers and the troubleshooting. Its firewall advice was also backwards. node_client_cidr_allow_list is an OCI security-list rule over the whole public subnet, and a guest's public address is a secondary private IP on the node's own VNIC, so narrowing it to the operator's address cuts internet ingress to every guest rather than protecting the node. The hardening section now arms the nft policy and narrows SSH through custom.nft, which is the layer that can tell a node's listener from a guest's. The deploy step names --channel dev rather than a pinned tag. The dev tag string grows with every cut, so a pinned example goes stale within the week. * fix(oci): the guide could not work on a fresh tenancy, because the profile name was ours oci_env.py bridges ~/.oci/config into TF_VAR_* — the provider takes the tenancy, user, fingerprint, key and region as Terraform variables and reads no config file itself — but the profile it looked for was the reference tenancy's, and validate-topology.sh passed no way to change it. A fresh account's config holds [DEFAULT], so every deploy would have stopped at "OCI profile 'apacanzset03child03' was not found" before building anything. The profile is now resolved as --oci-profile, then $OCI_CLI_PROFILE, then the reference tenancy, then DEFAULT. A name passed explicitly and absent is an error listing the profiles present, never a fallback: the failure worth preventing is building a cluster in whichever tenancy happened to be first. The reference tenancy still outranks DEFAULT, so existing runs are unchanged, and CI is untouched because it takes the environment route rather than a profile. has_profile exists because configparser treats DEFAULT as the fallback for every other section rather than as a section, so has_section denies it exists even when it is the only thing in the file — which made the first version of this fallback do nothing for exactly the case it was written for. Three documentation corrections found while checking the guide against the code. The IAM policy is ten operations and needs "use vnics", not "read vnics": registering a secondary private IP is an operation on a VNIC, so OCI checks VNIC_ASSIGN, which lives in the use verb, and with read the node forms and looks healthy and every allocation fails NotAuthorizedOrNotFound. These are now the same three statements instance-principal.tf grants, which is the only version of the policy that has been exercised. The guide also promised a worked example of the credential hook that does not exist and should not: nothing here distributes a credential. It now states the contract instead — the argument shape, the two files, and that a non-zero exit stops the deploy before anything is built. And the dev channel's rate limit is documented with the tag as its remedy, rather than left in the driver's help where a reader will not see it. * fix(ci): a failing workbook's log never reached the artifact, so only the runner had it run-tofu-examples-e2e.sh writes its assertion output to the per-topology workbooks.log in the state directory. "Sweep anything left behind" deletes those directories and runs before "Upload the run", so the upload's three oci-state/.validate-* patterns matched nothing and said so to nobody. The first nine-workbook run lost the one thing it was dispatched to find: ecs-quickstart failed on vm-single and the reason existed only on the runner, which does not resolve from a workstation. A collect step now copies *.log, *.txt and *.tsv into the artifact directory before the sweep, prefixed by topology so two topologies cannot overwrite each other. The workflow already records that "Publish the result tables" sits ahead of the sweep for this reason; the logs were simply never given the same treatment. The dead patterns are removed rather than left. A path that cannot match is worse than no path, because it reads as protection that is present — which applies to the credential-hook exclusion too: it now lives in the collect step, which is the only thing that writes into the artifact tree, so the file cannot reach an artifact rather than being filtered out of one. * fix(ci): a state directory kept the last run's logs, and the artifact published them as this run's Collecting the per-topology logs surfaced the next defect immediately: a vm-single-only run produced an artifact holding bm-workbooks.log and nine vm-multi node logs, because those state directories still held the previous run's output. Anyone reading that artifact would conclude three topologies ran when one did, which is worse than the missing log it replaced. Two causes, one each side. validate-topology.sh never cleared its own logs, so a topology's directory accumulated them; it now removes *.log, nodes.txt and cloudinit-*.txt as it starts, before the first line of the new run, and never the tfstate, which is how the sweep finds hosts to destroy. The globs are named rather than a find, so what goes is readable at a glance, and :? stops the shell instead of expanding to /*.log if STATE_DIR is ever unset. That alone does not cover a topology the run skipped, whose directory is never entered. So the collect step now takes only the topologies this run drove, read back from the per-topology log the harness writes rather than from the dispatch input -- a run that names no topology gets the harness's default, and reading that here would be a second copy of the list to disagree with the first. Also proved by this: ecs-quickstart passes on a freshly built vm-single, 172s, where it failed at 293s as the seventh of nine. The failure is order or load dependent on one 32 GB node, not a property of the workbook. * fix(ci): a failed workbook run left no node journal, so the cause was unreadable The workbook log says which API call failed; only the journal says why, and the nodes are destroyed minutes later. An IAM teardown failure on the nightly came down to one swallowed warning that existed nowhere else, and by the time the artifact was read the host it was logged on had been torn down. Warnings and above first, since that is where a swallowed error surfaces, then a bounded tail for context. Unbounded, a multi-hour suite's journal would dwarf the rest of the artifact. The existing state-directory clear keeps these from going stale between runs, and the collect step already globs *.log. |
||
|
Ben Duncan
|
c6841ce1f0 |
feat(oci): separate using an instance principal from creating one (#1157)
* feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts. |
||
|
Ben Duncan
|
70007405c1 |
fix(oci): install the dev channel by environment variable, not --channel (#1155)
The installer served by install.mulgadc.com is the latest release's, so it predates the --channel flag the harness was passing and would have died with an unknown option. Document the same form in both install guides. |
||
|
Ben Duncan
|
1743b60246 |
feat: publish a dev channel, so cloud contacts can install dev without a release (#1154)
install.mulgadc.com has served a dev channel since it was written -- setup.sh understands INSTALL_SPINIFEX_CHANNEL=dev and the Worker resolves it to the newest GitHub prerelease -- but nothing ever produced one, so /download/dev/amd64 returned 404. This is the producer. A separate workflow rather than conditionals in release.yml. A dev build is a different artifact, not a release with flags off: amd64 only, no ISO, no system images, no bare-metal gate, and nothing on a self-hosted runner. The sibling repositories are taken at dev rather than their default branch, because dev of spinifex against main of the others is a pairing nobody runs. The tag is the describe string plus -dev, which release.yml's own convention already treats as a prerelease, and which makes the commit recoverable from the version alone. The flag is set at creation and then the published state is asserted: a prerelease must never become the latest release, because latest is what an unqualified curl | bash installs. setup.sh gains --channel and --version so the instruction is a flag rather than an environment variable, and the OCI harness gains OCI_CHANNEL so a nightly can test the path the contacts are actually given. |
||
|
Ben Duncan
|
ade060f384 |
fix(oci): make bare metal work, remove the per-node API key, and validate all three topologies (#1146)
* fix(oci): make a bare-metal node form and mount its data volume
Bare metal presents VNICs differently from a VM: the first VNIC on a physical
port is the port, and a second one sharing that port is a tagged overlay with no
Linux device at all. The external VNIC was landing in that second slot, so no
interface ever carried it and the WAN bridge could not be built. Attach it at
nic_index 1 on a BM shape, which makes it the first VNIC on the second port and
therefore untagged and present at boot, exactly as it is on a VM.
Two consequences of that followed. The bridge script picked the first non-primary
VNIC from instance metadata, which on bare metal can be one with no interface, so
it now selects the VNIC the kernel actually has a device for and waits for that.
And OCI makes a bare-metal volume attachment multipath, where the Oracle agent
refuses to log in unless multipathd is running -- probing the unit name Debian
retired, so on Ubuntu 26.04 it never sees it. Alias the retired name and assert
the alias on every poll rather than once, because the first run had it created and
then vanish.
Finally, the three cloud-init steps become ordered systemd units. runcmd
continues past a failed step, so a node whose volume never appeared still opened
its service ports with nothing behind them; Requires= makes that impossible and
lets a boot that raced the iSCSI login retry on the next one.
* feat(oci): authenticate the address allocator as the instance principal
The OCI allocator could only authenticate from an API-key config file, so every
node needed a private key copied to it and a config file hand-written beside it.
A node without them forms, passes every health check, and then refuses every
launch that wants a public address with InsufficientAddressCapacity -- the cause
appears only in the journal.
NewInstancePrincipalClient already existed and was called from nowhere. Wire it
in behind a new oci_auth key on the pool, so a node can authenticate with the
certificate its own metadata service serves and hold no key material at all.
Nothing to copy, nothing to rotate, and nothing in Terraform state.
The reason recorded for preferring the config file was that it reads no instance
metadata, because Spinifex's own IMDS endpoints claim 169.254.169.254 and take
the cloud's metadata service with them. That no longer distinguishes the two:
oci_vnic_iface is what the guide recommends and it resolves the VNIC through a
metadata lookup already, so such a node depends on the IMDS remap either way.
The two constructors' comments also contradicted each other on which was
preferred; both now say what they cost.
config_file stays the default, because instance principal needs a dynamic group
and a policy only a tenancy admin can write. The Terraform creates both behind
enable_instance_principal, off by default for the same reason the module creates
no other tenancy-root resource, and the policy grants three verbs on vnics,
private-ips and public-ips in one compartment rather than anything broader.
Naming the method explicitly rather than inferring it is deliberate: a node meant
to use its own identity now fails at config load instead of falling back to a key
file nobody installed. Pairing oci_auth="instance_principal" with a config file
is refused for the same reason, since the certificate is already the credential
and the file would be the half silently ignored.
Terraform also stages the pool fragment on each node, which under instance
principal contains no secret at all.
* feat(oci): validate all three topologies from one script, and document both auth routes
validate-topology.sh builds one topology from nothing, installs the published
release, forms the cluster, configures the pool and the IMDS remap, runs a
workbook against it, proves the workbook actually serves traffic, and destroys
everything. Three topologies because each breaks differently: bare metal presents
VNICs unlike a VM, a single node has no Geneve underlay to get wrong, and only a
cluster exercises RAFT, the gateway chassis and cross-node allocation.
--topology has no default, deliberately. A command aimed at the wrong topology is
the easiest expensive mistake available here, and each keeps its own Terraform
state so two can be built from one checkout without either destroying the other's
instances.
The teardown decides the verdict rather than tidying up after it. A topology or a
workbook that cannot be destroyed is half proved, so destroy runs from an EXIT
trap and a teardown failure fails the run. It also checks the two things a green
apply does not: that vpcd logged resolving the external VNIC, which is the only
evidence the credentials work and br-wan matched a real VNIC, and that the load
balancer returns 200 -- a workbook can create cleanly and carry no traffic.
The guide gains the instance-principal route beside the key file, and loses its
claim that bare metal changes nothing else in the document. That claim was wrong
in exactly two places, and both are now handled rather than left to the reader:
the external VNIC's nic_index and the Oracle agent's multipath unit name. It now
also says plainly that the external-address path on bare metal is unproven.
* test(oci): let a topology be validated without an OCI credential
The external pool is the one phase that needs a key or a dynamic group, and
the VM topologies have to be regression-tested regardless. --no-external-pool
forms and verifies without it, and implies --skip-workload so the run cannot
claim a guest path it never exercised.
* test(oci): one entry point that proves a ref works on OCI, and cleans up after itself
e2e-oci.sh drives every topology from environment variables alone, so a
workstation, a Worker dispatch and e2e-oci-nightly.yml all take the same path
rather than drifting into three answers about whether OCI works.
Three things it deliberately does not do. It does not test the published
release: --distro installs the tarball built from the ref under test, because a
nightly that cannot fail on a bad commit is decoration. It does not carry its
own workbook assertions, running tests/e2e/run-tofu-examples-e2e.sh over the
published workbooks instead, so a workbook that passes on a hypervisor and fails
here is a difference in OCI. And it cannot quietly leave cloud resources
running: --sweep-only destroys whatever an earlier run's state still holds, and
it runs before the topologies, after them, and from an always() step.
The credential now reaches Terraform as a value through OCI_PRIVATE_KEY, so no
runner writes key material to disk, and oci_env.py refuses --shell on that route
because in CI stdout is the job log.
Four harness defects the first real runs exposed, each of which had been giving
a wrong answer:
- oci_env.py announced itself on stdout, corrupting every captured
terraform output. That is also why a single node came back as two hosts.
- terraform output was passed -var, which it rejects, and a silent fallback
then read the default state file instead of the topology's.
- grep -c Ready matches NotReady, so a single-node run whose only node was
NotReady on 0.0.0.0 reported PASS. The status column is parsed now, and
membership is waited for rather than sampled once.
- install-node.sh refuses fewer than two hosts, so a single node forms with
spx admin init the way the guide documents.
Also waits for the dpkg lock: cloud-init being finished does not mean apt is.
* fix(oci): resolve mulga's path explicitly, and refuse an unpushed ref
Two assumptions in the new harness that a run would not have caught quickly.
Deriving mulga's root from spinifex's is right for a submodule checkout and
wrong for CI, where the two are siblings under the workspace, so MULGA_ROOT is
explicit and the workflow sets it.
build-install-artifacts.sh clones each repo from GitHub rather than using the
checkout beside it, and falls back to dev when the ref is not there. A nightly
on an unpushed branch would then pass or fail on dev's code while reporting the
branch. Check the ref is on origin first, and skip the Go suite build: the
workbook driver is a shell script, so those binaries prove nothing here.
* fix(oci): do not read $? after a command when set -e is on
The workbook phase read the driver's exit status on the line after the ssh that
produced it, which under set -e is never reached: a failing run aborted before
printing the driver's own per-workbook diagnostics, losing them exactly when
they are the thing you need. An `if !` keeps them, and names which workbooks
passed and which did not.
* feat: let an OCI e2e dispatch form without the external allocator
A tenancy without the nodes' dynamic group cannot authenticate the
allocator, so every topology failed at it rather than reporting what it
had proved. The no_external_pool input forms with no allocator and
records the allocator and the workbooks as SKIPPED, never PASS.
Also rebuilds the dry-run plan message imperatively: it still referenced
a variable renamed to WORKBOOKS, and under set -u the substitution died
and the text silently vanished from the line.
* fix: name the offending input when an OCI credential is unusable
A credential from the environment reached the SDK unvalidated, so a
secret with a trailing newline failed as an uncaught InvalidConfig
traceback naming an SDK config field that nobody set by that name.
Strip the scalar values, since a secret store written with echo carries
a newline, and translate both InvalidConfig and InvalidPrivateKey into
the variable or config key to go and fix.
* fix: make the OCI nightly able to build, and stop it leaking a dispatch input into the shell
Four defects, found reviewing the workflow rather than running it.
The builder clones all five sub-repos from GitHub over https and they are
all private, so a hosted runner had no credential and the tree build
could never have worked. The app token now covers six repositories and
git carries it for our organisation only.
A dispatch input was interpolated into a run block, where a quote ends
the string and the remainder executes. It goes through the environment
now, which is the only place a caller-supplied value belongs.
The artifact upload took the whole directory, which meant the distro
tarball and the test archive, and would have taken terraform.tfstate had
the state directories sat under it. It is a named list now, and it also
collects the per-topology logs that were never uploaded at all.
* fix: make the OCI nightly's runner a setting, and stop widening the app token
The five sub-repos are public, so build-install-artifacts.sh clones them
without a credential and the token has no reason to reach them. Only
mulga is private, which is what it was scoped to before.
The runner is now inputs.runner, then the OCI_E2E_RUNNER variable, then
hosted. A per-provider self-hosted agent is a reasonable thing to want --
it lifts the six-hour cap a hosted job carries and keeps a long cloud
wait off the hypervisor labels -- and it should not need a workflow edit.
Removing the SSH key is load bearing on that path: a self-hosted runner
keeps $HOME between jobs, so the key would sit there between nightlies.
* fix: install the OCI SDK the harness needs, and keep state out of the checkout
Nothing installed the oci Python module, on any runner: validate-topology.sh
called bare python3 and oci_env.py imports oci, so every topology would have
failed at the first terraform call. The harness now uses a system python3 that
has it and otherwise builds its own virtualenv from requirements.txt, which is
the same behaviour on a workstation, a hosted runner and a self-hosted one.
The state directories move under OCI_STATE_ROOT, defaulting to the script
directory. A persistent runner needs them outside the checkout, because
actions/checkout wipes untracked files there and would remove the state of a
killed run before the sweep could read it, leaving OCI instances running that
nothing is able to find.
* fix: drop the OCI SDK dependency, so the harness runs on any host
python3-venv is not installed on banksia, and the harness needed it: the
environment helper re-executed itself from a virtualenv to import the OCI
SDK. It used the SDK for two things -- parsing an INI file, which
configparser already does, and discovering the tenancy home region, which
no Terraform file consumed. Neither justifies a dependency on every host
that runs this, so the helper is standard library only now.
The credential is checked for shape instead, locally and before any OCI
call, which is a better failure than a 401 forty seconds into a run.
Two things the removal uncovered. The home region is a fixed property of
a tenancy rather than something to rediscover every run, so it is
OCI_HOME_REGION. And OCI serves IAM writes from the home region while the
dynamic group and policy were using the default provider for the build
region, which would have failed when the tenancy applied them: they go
through an aliased provider that defaults to the build region, so a
tenancy whose home region is where it builds sees no change.
|
||
|
Julian Sommer
|
0c3403889c | feat: load AWS conformance models from api-models-aws (#1121) |