Compare commits

...

239 Commits

Author SHA1 Message Date
joshsiv-mulga 784e1f909a refactor: declare payload-body scopes as traits
The action recovery for a rejected request skipped S3 by name. The composition now declares the S3 scope as carrying object data, and the gateway reads the trait.
2026-11-01 10:04:45 +10:00
joshsiv-mulga 60cf4ca314 refactor: supply the unsigned web-identity exchange as a pre-auth handler
STS composition now hands the pipeline a dispatch.PreAuth. The gateway no longer recognises STS or the action by name, and the layering allowlist row for gateway -> sts/awsapi is paid off.
2026-11-01 10:00:18 +10:00
joshsiv-mulga fab4036972 refactor: declare the global signing region on the registration 2026-11-01 09:48:59 +10:00
joshsiv-mulga 2eb0f5ec83 refactor: declare the XML error shape on the registration
EC2 and the operator surface declare the EC2 shape. S3 is not served here, so the composition declares its scope traits and a rejected S3 request keeps S3's body.
2026-11-01 09:42:00 +10:00
joshsiv-mulga 561582b971 refactor: declare the XML throttle code on the registration
The pipeline picked Throttling for iam, sts, eks and route53 by name. EKS is a JSON registration and never reached that arm, so it is not carried over.
2026-11-01 09:19:39 +10:00
joshsiv-mulga 68c9a9123a test: pin service-keyed gateway behaviour before it moves to registration metadata 2026-11-01 09:00:42 +10:00
Josh Siviour 0fd85d117b test: move remaining service and verdict tests out of gateway (#1372)
* refactor: move gateway auth fixtures to runtime-free gwfix package

* refactor: move ACM gateway tests to awsgw

* refactor: keep gateway-internal ACM tests in gateway and drop test-only exports

* refactor: move RDS gateway tests to awsgw

* refactor: move ELBv2 gateway tests to awsgw

* refactor: move IAM gateway tests to awsgw

* refactor: move STS gateway tests to awsgw

* refactor: move tagging gateway tests to awsgw

* refactor: move operator and bedrock gateway tests to awsgw

* refactor: move ECR gateway tests to awsgw

* refactor: move ECS and EKS gateway tests to awsgw

* refactor: move EC2 gateway tests to awsgw

* refactor: move credential verdict tests to awsgw

* refactor: move policy-gate dispatch tests to their service packages

* refactor: move checkPolicy verdict tests to policyauth

* refactor: run gateway tests on a fake verifier, real auth only for listed verdict tests
2026-11-01 08:42:07 +10:00
Ben Duncan 51d7ac91f5 fix: bound the console's instance-type payload and stop polling it (#1371)
* fix: bound the console's instance-type payload and stop polling it

DescribeInstanceTypes with capacity=true returned one entry per free slot per
node: 3,528,765 bytes of XML for 50 distinct types on prod, refetched every
five seconds. The response was sized by free capacity, so the emptier the
cluster the larger it got, and at 44,346 slots it exceeded the NATS maximum
payload and the console reported no compute at all.

The listing now names each admissible type once. The free-slot figure crosses
as a number through a new GetInstanceTypeCapacity action, open to any account
because the launch form a tenant uses is what needs it. The catalogue query no
longer polls, and the EKS and ECS dialogs read it only when open.

The gateway negotiates gzip inside the SigV4 group, which excludes the OCI
Distribution registry structurally. A response is compressed only if the client
offered gzip, the handler declared neither a length nor an encoding, the
content type is neither streaming nor already compressed, the status carries a
body, and 1 KiB was written before any flush.

An unknown capacity figure no longer caps the count at 1. The launch decides,
and RunInstances answers InsufficientInstanceCapacity if it cannot.

* test: read the free slot count from its own action in the console e2e

Two e2e tests encoded capacity as repeated instance-type entries, so one
expected a caption the deduplicated listing can no longer produce. The fixture
answers GetInstanceTypeCapacity instead, which lets a test give the catalogue
and the figure different answers.

A third test covers the case the change introduces: a session whose policy does
not grant the action reads no figure, so the form offers the type with no
capacity caption and leaves the launch enabled.

The committed bundle is rebuilt, which rehashes every chunk reaching the two
query modules.
2026-10-31 14:03:19 -05:00
Ben Duncan 707dbefa13 fix: bound the capacity describe reply by instance-type count, not free slots (#1368)
* fix: bound the capacity describe reply by instance-type count, not free slots

A node answering `DescribeInstanceTypes` with `capacity=true` sent one full copy of an instance type's description per free slot, so the reply grew with how much capacity the host had free.
At 1344 bytes per slot the four-node production cluster, holding no guests, produced 874 slots and a reply over the 1 MB NATS maximum payload.
Every node's reply was dropped, the gateway collected zero frames, and it returned an empty list as a success.
The console reads an empty capacity list as no capacity and refuses every launch, which is how a cluster with hundreds of free vCPU reported "No compute available".

The emptier the cluster, the more likely the failure, which is the opposite of what the shape should be.

A node now returns each admissible type once with the number of instances of it the node can take, so its reply is bounded by the size of the instance-type catalogue and not by free capacity.
On the production catalogue that is 84 KB for 62 types whatever the slot count, against 59 MB for the same reply on a large idle host under the old shape.
The gateway sums the per-type counts across nodes and expands them on the last hop, because the client SDK drops response fields it does not model and the console displays the figure to the customer.

A capacity fan-out that collected no frames now returns a retryable ServiceUnavailable.
Reporting an empty list as a success is what turned a transport failure into a false statement about the cluster.

Two tests pin the shape: the node reply for an idle 4096-vCPU host stays under the NATS payload limit, and a capacity fan-out with no responders does not return an empty success.

* test: cover the console against a real cluster, and pin the capacity banner

The console has never been exercised against a real control plane.
`playwright.config.ts` pins `localhost:3101` and `:3102` and intercepts every API call, so the suite cannot be pointed at a cluster and cannot see a defect whose whole content is what the control plane actually sends.
That is why an idle production cluster came to tell customers it had no compute: the capacity listing was dropped in transport, the empty result read as no capacity, and nothing in CI could reach the page that says so.

`playwright.live.config.ts` runs the same console against a cluster named by `SPX_CONSOLE_URL`, signing in with `SPX_ACCESS_KEY_ID` and `SPX_SECRET_ACCESS_KEY`.
It starts no server, mocks nothing, runs one worker because it launches real guests, and is never part of `pnpm e2e`.
The suite signs in, asserts a cluster with free capacity offers instance types rather than reporting none, reads the per-type capacity figure, launches an instance and waits for `running`, reads its live metric graphs, terminates it in a `finally` block, and loads the nine listing pages a customer reaches from the sidebar.

Three tests in the mocked suite pin the banner's own logic, which is cheap and catches the UI side of the same failure: the form offers what the capacity listing reports, it shows the free slot count for the selected type, and an account with genuinely no capacity is told so and cannot submit.
The third passes on the current code and is the one that documents why the banner exists at all.

* test: correct three live-console selectors found by running it against prod

The first run of the live suite was 11 passed and 2 failed, both in the suite rather than in the console.

The dashboard check looked for node status on `/`, where it is not shown.
It now loads `/nodes` and asserts the `Resource usage by node` and `Instance type capacity` tables, which is where the daemon reports a node's own capacity.

The OS tile's radio is `sr-only` behind the distribution logo, so `check()` failed its actionability test on the `<img>` covering it.
The click goes to the enclosing `<label>` instead and the browser forwards it.

The terminate dialog holds its confirm disabled until the phrase is typed, so the teardown waited out its timeout on a disabled button and left the instance running.
The helper types `terminate` first.

With those three corrected the launch, running, live-metrics and terminate path passes against the production console in 34 s.
2026-10-31 13:42:13 -05:00
Ben Duncan 4050b63ada feat(ui): enforce ADR-0016 with guard tests and move dialog forms into FormPanel (#1369) 2026-10-31 13:41:53 -05:00
Ben Duncan 955714f557 fix: let a snapshot go once every volume cloned from it is already deleting (#1370)
* fix: let a snapshot go once every volume cloned from it is already deleting

DeleteVolume now records State=deleting and leaves the teardown to the
reclaim reaper, so a clone stays listed after the tenant has deleted it.
DeleteSnapshot counted that clone as still reading the snapshot and refused
with InvalidSnapshot.InUse, which broke DeleteDBSnapshot after deleting the
DB instance restored from it and left the retained source volume behind.

A deleting volume's teardown removes only its own objects and nothing moves
it out of deleting, so it no longer holds the snapshot's chunks.

* test(e2e/rds): delete a snapshot straight after the instance restored from it

The subtest deleted the source instance between the two, which gave the
volume reclaim reaper time to finish the restored volume on a fast node and
hid the refusal the nightly hit. Deleting the snapshot immediately makes the
regression deterministic.

The release wait now covers a full reclaim sweep: since deletion became
asynchronous a released volume is gone only after the reaper's next pass.
2026-10-31 13:27:01 -05:00
Josh Siviour 2e770e8859 test: move per-service gateway tests into their domains (#1366)
* refactor: one authorization gate in ingress/aws/pipeline

The key merge and the caller guards move out of the gateway so tests that
drive a registration directly use the same gate as the pipeline.

* test: run ACM handler and policy tests in domains/acm/awsapi

Adds internal/testkit/gwtest, which invokes a registration through the
pipeline's own authorization gate with real policy evaluation.

* test: share policy assertion helpers through gwtest

* test: run ELBv2 handler and policy tests in domains/elbv2/awsapi

* refactor: one actor projection in ingress/aws/pipeline

* test: run RDS handler and policy tests in domains/rds/awsapi

* test: run ECS handler and policy tests in domains/ecs/awsapi

* test: assert the exact policy gate verdict in gwtest

* test: run EKS authz and wire tests in domains/eks/awsapi

* test: run IAM authorization tests in domains/iam/awsapi

The policy-driven DeleteRole and AddRoleToInstanceProfile tests run in the IAM package through gwtest, with PassRoleTo shared. The HTTP-handler, examples and signed-pipeline tests stay in gateway.

* test: run ECR authorization tests in domains/ecr/awsapi

* test: run ECR repository, image and lifecycle-preview tests in domains/ecr/awsapi

* test: run ECR dispatch error tests in domains/ecr/awsapi

* test: run Bedrock authorization tests in domains/ochre/awsapi

* refactor: move EC2 authz tests into domains/ec2/awsapi

* test: restore the ECS scope-table completeness gate in domains/ecs/awsapi

* Revert "test: restore the ECS scope-table completeness gate in domains/ecs/awsapi"

This reverts commit 0211a0a1bf.
2026-10-31 23:45:35 +10:00
Ben Duncan f3ee105ed4 docs: add a diagram of a Vultr deployment to the guide (#1367)
Shows deploy-vultr.sh building a private Vultr VPC with one node or three,
the nodes clustering over the private network for the control plane, EBS
and S3, Spinifex calling the Vultr API for reserved IPs, and the AWS
services running across the fleet.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 08:36:10 -05:00
Ben Duncan 00070b1ecd docs: tidy the Vultr guide for publication (#1365)
Drops the horizontal rules the docs site rendered as stray text, links
the AWS CLI install page, captions each screenshot, shows the expected
spx get nodes output for three nodes, explains what a three-node
deployment protects against, and simplifies the limits to the sizing
and quorum a reader needs.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 08:20:53 -05:00
Josh Siviour 9405d2c233 feat(sqs): queue and Lambda-facing message API foundation (#1364)
* feat(sqs): queue and Lambda-facing message API foundation

Adds the SQS JSON 1.0 adapter (unregistered in production) with queue
create, lookup, attributes, list, tags and the message operations a
Lambda poller uses: SendMessage(Batch), ReceiveMessage with long
polling and attribute selection, DeleteMessage(Batch) and
ChangeMessageVisibility(Batch), backed by the queue record owner and
the JetStream message store. String and Binary message attributes are
validated, stored and checksummed; Number attributes are refused until
AWS behaviour is confirmed.

* test(sqs): poller lifecycle through the message owners

Covers receive, release, redelivery, delete, visibility extension, queue
binding of receipt handles and long polling against the real queue and
JetStream message stores, from the message package itself.

* test(sqs): move SDK coverage from the gateway to the SQS domain

Drives the registration through dispatch.Invocation over the real queue
and message owners, so the gateway package carries no SQS changes.
2026-10-31 22:27:11 +10:00
Josh Siviour a431fd3385 refactor: gateway request record and capabilities (ADR-0002:S7, S8) (#1363)
* test: pin gateway pipeline order, request ids and region

* refactor: carry request facts in one ingress request record

* fix: seal the request principal once

* refactor: build the caller ARN once in credverify

* refactor: gateway takes policy and admission as capabilities

* refactor: gateway takes cluster readiness as a capability

* chore: gofmt GatewayConfig field alignment

* docs: record slice 11a request record and capabilities

* docs: add PR number to slice 11a migration row

* refactor: move pipeline capability interfaces to ingress/aws/pipeline
2026-10-31 17:57:15 +10:00
Josh Siviour 3c285b2f37 ci: ask PR authors whether Mulga design needs to change (#1362) 2026-10-31 15:56:46 +10:00
Ben Duncan da3e4e5b73 feat: run Spinifex on Vultr, with failover, workbooks and the fixes they needed (#1361)
* feat: Vultr reserved-IP provider and floating external-address pool

Adds a source="vultr" external pool backed by Vultr reserved IPs: an API
client pinned by captured responses, an in-memory fake with the measured
semantics, and a floating allocator that creates, claims, reconciles and
releases labelled addresses. OCI and Vultr claims now share one
AddressClaimer pass.

Also fixes the host firewall dropping guest IMDS when imds_host_meta_ip is
set, by matching the pre-DNAT destination, and arms the Spinifex firewall in
place of an active ufw.

* fix: detach a Vultr reserved IP before deleting it

Vultr was measured refusing delete of an attached address with 400 "Unable to detach subnet", so release, rollback and reconcile now detach first. A first claim after allocation logs at info rather than warn.

* fix: release the auto-assigned public IP an Elastic IP displaces

AssociateAddress restamped the ENI with the EIP and never handed back the
address launch had assigned. Nothing referred to it afterwards, so it stayed
allocated for good, and on a cloud pool every associate left a billed address.
AWS releases it on associate; withdraw its NAT rule and release it, scoped to
the ENI, leaving any address another associated EIP holds alone.

* fix: retry Vultr's refused detach on reserved-IP detach and delete

Vultr has answered a detach with 400 "Unable to detach subnet" and accepted
the same detach moments later. Surfacing it left the address for reconcile
to collect, up to twenty billed minutes later; retry it like a 5xx instead.

* fix: stop reporting the released launch address after an EIP is disassociated

Associating an EIP now releases the auto-assigned public IP, but the stored
instance kept its launch stamp, and DescribeInstances fell back to it once the
EIP was disassociated. The public-IP change handler now moves the stamp with
the runtime address.

* feat: add a Vultr deploy script for a single-node Spinifex on a VX VM

deploy.sh up/destroy/status drives Terraform for the VPC and VM, installs and
forms Spinifex over SSH, copies the API key to the node rather than through
user-data, and sweeps boot volumes and reserved IPs Terraform does not own.
Cloud-init keeps needrestart off the Spinifex, OVS and OVN units and disables
unattended upgrades on these disposable hosts.

* feat: form a three-node Vultr deployment with install-node.sh over the VPC

Each node comes up as an armed single node, the state install-node.sh forms
from, with the cluster and Geneve planes on the VPC NIC. The pool and every
node's provider_host_id go on each node after formation.

* fix: read the pinned host key with ssh-keygen -F when forming a Vultr cluster

ssh_config hashes known_hosts entries by default on Debian, so a grep for the
address found nothing and set -e ended the run without a message.

* docs: add the Vultr guide, one command to deploy one or three nodes and one to remove them

* docs: drop the failover write note from the Vultr guide

* fix: keep needrestart from restarting the Spinifex, OVS and OVN units

Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's
replaced libraries to the daemon and restarted it after every dpkg run. The
restart never cleared them, so an unattended upgrade ran the unit into its
start limit and left the control plane down.

setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting
spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists
them, so restarting after an upgrade stays the operator's choice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop the Vultr cloud-init needrestart override now setup.sh writes it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: replace the Vultr guide's apt upgrade warning now the installer keeps needrestart off the stack

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: install the Vultr guide's build from the dev channel until Vultr support is released

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: answer ARP for Vultr reserved IPs on bare metal

Vultr routes a reserved IP to a VM by MAC but ARPs for it on a bare-metal
server's public segment, and a Vultr pool gave vpcd neither a gateway nor a
bind bridge to proxy-ARP on, so nothing answered and no traffic arrived. A
Vultr pool now proxy-ARPs on the interface the node routes the internet
through. setup.sh installs arping, which the gratuitous ARP already called.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: deploy Spinifex on Vultr bare metal, renamed deploy-vultr.sh

A vbm- plan creates bare-metal servers, one or three. Without --name the
deployment writes the AWS profile spinifex, up prints every node's address
and the API key allowlist, credentials and ssh subcommands are added, no AMI
is imported unless --image is given, and OpenTofu or Terraform is used. The
host key is pinned only once cloud-init is done, since bare metal first
answers SSH from Vultr's installer with a different key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: rework the Vultr guide into the OCI guide's flow, with bare metal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: restamp LaunchTime when a stopped or recovered instance relaunches

The pending watchdog times a launch from LaunchTime, which still held the
original launch. An instance recovered from a failed node, or started more
than five minutes after it was first launched, was terminated as stuck in
pending seconds after QEMU started. AWS restamps LaunchTime on every start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: spread the nginx-alb workbook across nodes, and power Vultr nodes off and on from deploy-vultr.sh

The workbook takes server_count and placement_strategy, so three web servers
can run one per node. Its README now matches the template (public subnets,
no NAT gateway) and the diagram shows the spread. deploy-vultr.sh gains
stop and start, which halt and start a node through the Vultr API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let a terminate finish tearing down before the reaper calls it wedged

OrphanQEMUReaper finalized any shutting-down instance whose QEMU had exited,
on the premise that a live terminate still has a live QEMU. A terminate is
still deleting volumes, releasing its IP and leaving its placement group for
tens of seconds after QEMU exits, so the instance was reported terminated
first. Terraform then deleted the placement group and got
InvalidPlacementGroup.InUse. The reconcile now waits two minutes from
shutting-down, well inside the ten-minute stuck-terminate backstop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: release an ovs-vswitchd left in flow-restore-wait after a node boots

ovn-controller clears flow-restore-wait once its flows are installed and
sets it again straight after. When both writes reach ovs-vswitchd as one
update, it only sees true, starts no upcall threads and forwards nothing.
Seen on Vultr bare metal after a hard power-off: instances recovered onto
the rebooted node ran but were unreachable. vpcd now clears the flag on any
node held that way for a minute, longer than ovs-ctl's own restore takes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add a resiliency test to the Vultr guide and the nginx-alb workbook

The Vultr guide gains step 8, which spreads three web servers across a
three-node deployment, powers a node off and back on, and shows what was
measured on VMs and bare metal. The workbook gains the same test as step 6.
The guide's table of contents is repaired, and its limits now cover losing
two nodes and recovery inside a spread placement group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: never terminate an instance that has run because a relaunch failed

The pending watchdog terminated any instance pending past five minutes, and
terminating deletes DeleteOnTermination volumes. A start of a stopped
instance, a recovery from a failed node and a daemon relaunch all pass
through pending, so a slow one destroyed an existing instance and its root
disk. On Vultr a recovered instance was terminated seconds after QEMU
started and its root volume deleted.

An instance is now marked FirstLaunch from RunInstances until it first
reaches running, cleared before the running state is written. The watchdog
acts only on first launches, as AWS terminates only a launch that never
ran. MarkFailed refuses to terminate any other instance and marks it
recovery-failed instead, which keeps its volumes, ENI and addresses and
shows it as stopped. A claimed relaunch is never a first launch.

Invariant tests pin ADR-0003:S7. The instancerecovery e2e suite gains a
test that ages its guest past the watchdog before recovering it, then
stops and starts it, and requires the instance, its root volume and the
data on it to survive both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: run the Terraform workbooks against a Vultr deployment on demand, and show the Vultr portal in the guide

e2e-cloudvendor-vultr.yml is dispatch-only. It builds the ref, deploys it
with deploy-vultr.sh on one or three VMs or bare-metal servers, runs the
published workbooks from node1 with the same driver as the hypervisor
nightly and OCI, and destroys the deployment, failing if anything is left
billing. It needs one secret, VULTR_API_KEY, and generates its SSH key per
run. The guide gains screenshots of the API key's access control list and
of the nodes in the Vultr portal, and its portal navigation now matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: install setup.sh on Vultr nodes when the tarball does not carry it

build-install-artifacts.sh writes setup.sh beside the tarball, so a node
installed from its tarball had no /usr/local/share/spinifex/setup.sh and
arming the firewall failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: pass the remote command after the destination in the Vultr e2e driver

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: run each workbook in the cluster's own region

The workbook driver passed no region, so every workbook used its default,
ap-southeast-2, and on a cluster in any other region every request failed
SignatureDoesNotMatch. It now passes the region of the node's spinifex
profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015)

A node counted physical cores, collapsing SMT threads, and held back a
fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is
now one online CPU, as AWS counts it: a thread with SMT on, a core with
SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4
up to 63 and 8 from 64, and the rest is allocated one to one. The
existing overrides stay operator-only. Invariant tests pin every tier
boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: state the Vultr guide's sizing under the online-CPU count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: drop an unmeasured EKS sizing claim from the Vultr README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let a guest reach an Elastic IP hosted on its own node

A guest's packet to an EIP on its own node left its VPC from the transit
range, was masqueraded to the transit host address, and went back into
OVN, which exempts that range from EIP DNAT, so it was never answered.
On Vultr an EKS worker could not reach its control plane's endpoint, so
it never joined. AWS lets an instance reach a public address in its own
VPC.

Hairpinned transit traffic now jumps to SPX-EIP-HAIRPIN, above the egress
MASQUERADE, where each EIP's rule sends it from the uplink address, as the
host's own traffic to the EIP already is. The rule follows the EIP's
ingress plumbing in and out, and anything else returns to the MASQUERADE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: resolve Spinifex's LB and EKS names on Vultr nodes

Vultr's Ubuntu image writes a static resolv.conf naming Vultr's resolvers,
so lookups bypassed the split DNS Spinifex configures in systemd-resolved
and an EKS endpoint name did not resolve on the node. deploy-vultr.sh
points resolv.conf at the resolved stub, as stock Ubuntu does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: prove a guest reaches an EIP on its own node in the natuplink e2e

The nat-single cells run it with a public pool, so the hairpin path that
left EKS workers unable to reach their control plane on Vultr is now
covered nightly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: record the natuplink hairpin check in the e2e coverage index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep each Vultr node's logs when an e2e run fails

The driver destroyed a node whose install failed together with the log
that said why. It now copies /root/*.log from every node into the run's
artifacts before the teardown, and the workflow uploads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 23:12:58 -05:00
Josh Siviour 23be7c201a fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options (#1360)
* test: pin EKS worker IMDS metadata options and launch template resolution

* fix: launch EKS workers with IMDS hop limit 2 and honour launch template metadata options

* fix: default unset launch template metadata fields for EKS workers

* test: e2e coverage for EKS worker IMDS hop limit and launch template metadata options

* test: drop the unneeded GetCallerIdentity policy from the worker IMDS e2e

* docs: describe the EKS worker IMDS defaults and launch template overrides

* test: drop the stale GetCallerIdentity policy from the IRSA pod e2e
2026-10-31 14:07:05 +10:00
Ben Duncan e5b9fef0ae feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) (#1359)
A node counted physical cores, collapsing SMT threads, and held back a
fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is
now one online CPU, as AWS counts it: a thread with SMT on, a core with
SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4
up to 63 and 8 from 64, and the rest is allocated one to one. The
existing overrides stay operator-only. Invariant tests pin every tier
boundary.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 22:53:45 -05:00
Ben Duncan 5271b16d2a feat(ui): console design system, side-panel resource views and live EC2 metrics (#1358)
* feat: lay the Spinifex UI design-system foundation

Brand tokens in OKLCH with semantic status and a colour-blind-checked
chart palette, Archivo headings and a 13px type scale, and a pre-paint
theme script so dark mode never flashes white under the console CSP.

Adds the missing Base UI primitives, a /design catalogue route, a route
ratchet against hand-written tables, a ban on chart-library imports in
routes, and lets the content column shrink so wide tables scroll in
place. The dev server now forwards /api and /proxy to the Go console.

Sonner is removed: its inline style element is blocked by the CSP.

* feat(ui): side panels, TanStack table, uPlot charts and e2e gate

- DataTable renders @tanstack/react-table, with column hiding as container
  queries so columns drop when a side panel narrows the list.
- UniFi-style DetailPanel and ResourceView: Instances, VPCs and Internet
  gateways open in a panel beside the list from 1024px and as a full page
  below; selected, tab and create live in the URL. Create flows for
  instances, VPCs and gateways open in the same panel.
- Instance monitoring charts on uPlot (CPU, network, packets), on simulated
  data and badged as such.
- Dates in the AWS console form, e.g. 2026/06/04 02:00 GMT-5.
- Tabs style Base UI's data-active, so the selected tab now shows.
- Playwright suite gates CI as the UI End-to-End job in ui.yml.

* feat(ui): one details component for panel and page, UniFi-style overlay panel

The side panel now lies over the list at full height with a shaded edge, and a
click on the list's background closes it. Instances, VPCs and internet
gateways each render one details component as either the panel or the
resource's own page, so the page gains the monitoring graphs and the panel
gains IAM role, GPU, console output, change type and create image. SSH moves
into Instance information as rows. Internet gateways attach and detach from
the panel.

* feat(ui): roll the panel pattern out to every service and gate screenshots in CI

Every EC2, IAM, Route 53, Bedrock, EKS, ECS and RDS resource is now a
list plus a side panel, with one details component serving as both the
panel and the one-column page. DetailCard, DetailRow, ListCard,
ResourceList and DeleteResourceButton are deleted.

- Panel: slide-out close, row selection closes it, expand keeps the tab
  and fetched console output, PanelSpinner while loading.
- Instances: info tips, theme-aware coloured console output, power
  button group plus Actions menu, Stop and Reboot confirms, security
  group, Elastic IP and network interface sections, and a live
  one-second monitoring window on simulated data.
- Launch instance: OS tiles picking the newest version, architecture
  choice, location tile, expanded root volume, honest default subnet.
- On/off settings are switches app-wide; disabled controls show
  cursor: not-allowed.
- The e2e job runs in the pinned Playwright image with screenshots
  gating; make ui-baselines and make ui-e2e-container use the same image.

* feat: bulk select and delete on every resource list, OS marks and instance type sizes

Every list with a delete action gets row checkboxes and a selection bar.
Delete confirms in a dialog naming each resource, makes one call per
resource, keeps each refusal against its row and retries only failures.

The Launch form shows distribution marks for Ubuntu, Debian, Rocky,
Alpine and CentOS, served as files under the CSP, and states the
selected instance type's vCPUs and memory.

* feat: sign-in overlay, larger type scale, typed confirms on irreversible deletes and dialog shortcuts

The sign-in card sits over an inert shell. Any sidebar group folds, and entering one unfolds it. Instance DNS names copy. Escape cancels and Cmd/Ctrl+Enter confirms every dialog. Volume, snapshot, RDS, EKS, ECS and hosted zone deletes need "delete" typed, and terminate needs "terminate".

* feat: fade sign-in and sign-out in place, and show the confirm shortcut

Signing in no longer reloads the page: the card and a lighter veil fade out and the console routes to the dashboard. Signing out cross-fades into the sign-in card with a Signed out notice. Confirm buttons show Cmd or Ctrl Enter for the platform.

* feat(ui): generic sidebar group names and a paged S3 object browser

The sidebar names its groups Compute, Databases, Authentication and Object
Storage; the command palette still finds each by its AWS service name.

Buckets get row selection and delete. Inside a bucket, folders and objects
share one table paged by ListObjectsV2 continuation tokens, with a find box
that queries by prefix, multi-select delete through DeleteObjects (folders
with everything under them), and an object panel showing HeadObject
properties, user metadata and a download.

* feat(ui): show upload progress and stop the S3 proxy cutting off slow transfers

The Upload button shows a running batch's percentage, and a large or slow
batch opens a dialog with bytes sent, bit-rate, time left and per-file
cancel and retry. Uploads run in a store outside React, so they continue
while the user browses.

The console's S3 proxy now allows an hour per request. The server's 15 s
ReadTimeout cancelled every upload part that took longer, so large uploads
failed on links below roughly 18 Mbit/s.

* feat(ui): stream live EC2 metrics to the Monitoring tab

The Monitoring tab drew simulated data. It now streams CPU, network
bytes and packets from the owning node's qmp-collector every 500 ms
while someone is watching, with nothing stored.

- qmp-collector: a watched sampler on schedstat and tap counters, on a
  30 s lease, keeping five minutes in memory for reconnects. It answers
  only watches for the account in the VM's discovery file.
- awsgw: spinifex:GetInstanceMetrics, checked against the instance ARN,
  answered as text/event-stream in 60 s segments. Another account's
  instance is InvalidInstanceID.NotFound.
- console: a signed stream reader that reconnects with Since and backs
  off on failure or an empty segment, stops on InvalidAction, and a
  chart axis that scrolls between samples. The simulator and its badge
  are removed.

* feat: catch refused creates in every form, answer the third review, and guard tables

Every create form now catches a refused mutation, so the failure shows beside
the submit button and no longer escapes as an unhandled rejection. Three EKS
tabs that swallowed a refused create now show it.

Third-review answers: the attach dialog offers running or stopped instances in
the volume's zone; the default security group's delete is disabled with its
reason; the load balancer panel splits into Listeners and Attributes tabs; a
main route table lists its implicit subnets; IAM single delete dialogs name the
same dependants as the bulk ones; a snapshot can be deleted in any state.

Every list in a route renders through DataTable, enforced by a unit test with a
named allowance, and the /design baselines are redrawn in the pinned container.

* feat: cap live metrics streams, prune hidden selections, and move panel focus

Each gateway holds at most 10 live metrics streams per account and 200 in
total, refusing past either with RequestLimitExceeded, and counts opened and
refused streams. A row that leaves a list leaves its selection, the instances
page uses the shared selection hook, the detail panel takes focus on open and
returns it on close, and Escape yields only to the popup it was pressed in or
a modal. The UI workflow also runs on live metrics backend changes.
2026-10-30 22:40:47 -05:00
Josh Siviour e06baa3aef fix: keep an ECR token valid after its role session expires (#1357)
* test: authorise an ECR token from the live role once its session is gone

* feat: carry the role ID in an ECR token minted for an assumed role

* feat: rehydrate an assumed-role caller from the live role

* fix: authorise an ECR token from its live role after the session expires

* docs: note that an ECR token outlives its role session
2026-10-31 12:20:24 +10:00
Josh Siviour ded44feedb refactor: register EC2 through the ingress seam and delete the legacy service switch (#1355)
* test: characterise EC2 request labels and the cluster-unavailable gate order

Pins the span, audit, throttle and policy-gate labels an EC2 request carries through the signed pipeline, and that the NATS gate answers before policy, action lookup and gateway-local actions.

* refactor(gateway): register EC2 through the ADR-0002 seam

EC2 is composed in awsgw from domains/ec2/awsapi and served by the
registration seam. The legacy service switch and its tables are removed,
along with the two paid-off gateway allowlist rows.

* docs: record the EC2 seam registration in the package boundary migration

* docs: record the EC2 seam registration PR number
2026-10-31 12:09:59 +10:00
Ben Duncan cf4a6abafb fix: make volume deletion asynchronous so a large volume cannot fail its own delete (#1356)
DeleteVolume returned only once every object behind the volume was gone, which
made the call's latency a function of the volume's size. Object removal costs
one round trip per object, so no timeout is both short enough for a control
plane request and long enough for a large volume: on a 100 GiB volume the
provider request overran its 30 s budget at 52 s, the tenant was told the
delete failed, and the teardown that had already started was left with no
record accountable for it.

AWS does not work this way. DeleteVolume's output in the EC2 model is
smithy.api#Unit, VolumeState includes deleting, and the documentation says a
volume can remain in that state for several minutes.

DeleteVolume now validates, decides every AWS-returnable error, records
State=deleting and returns. VolumeReclaimReaper converges the teardown and
removes the record only once it is complete, so an unreachable or saturated
blob node delays reclamation instead of failing a delete the tenant has already
been told succeeded. The reaper is cluster-wide and leader-gated, which is also
what serialises the teardown: three concurrent destructive handlers ran one
volume's objects on prod because a retry had nothing to collapse against.

A volume already recorded as deleting answers a second delete from the record,
which collapses both the tenant's retry and DeleteOnTermination's own delete.

Two ordering fixes behind it. The provider handler now tears down local state
before deleting objects, because the local half is fixed-cost and sat behind the
half that scales, so a timeout left this node's WAL and dirty marker behind for
good. And volume.delete and snapshot.delete are classified as data movement,
since both reference real data and neither answers from state already held.
2026-10-30 15:49:33 -05:00
Ben Duncan 06f93802b8 fix: keep needrestart from restarting the Spinifex, OVS and OVN units (#1341)
Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's
replaced libraries to the daemon and restarted it after every dpkg run. The
restart never cleared them, so an unattended upgrade ran the unit into its
start limit and left the control plane down.

setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting
spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists
them, so restarting after an upgrade stays the operator's choice.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 15:17:15 -05:00
Josh Siviour 0fc93e975a refactor: serve EC2 actions from domains/ec2/awsapi (#1354)
* test: characterise EC2 through the signed gateway pipeline

* refactor: serve EC2 actions from domains/ec2/awsapi over composed capabilities

The EC2 action table, idempotency and validation move out of the gateway behind a Deps struct. Quota is a consumer-side interface and the instance adapters take a narrow InstanceProfiles interface instead of the whole IAM service. The gateway only composes Deps and calls Dispatch. The 18 gateway to EC2 awsapi allowlist rows are deleted.

* test: drive every EC2 action through Dispatch in its own package

Covers the request errors, the gateway-local actions without NATS and each routed action reaching its handler against a stub daemon.

* docs: record the EC2 capability move in the package boundary migration

* docs: record the EC2 capability move PR number
2026-10-31 00:30:51 +10:00
Josh Siviour d066786faa feat: operator-launched SQS worker and store system VMs (#1353)
* feat: persist the SQS worker's store key on its own disk

The shared SQS worker generates its store-client NKey inside the guest and
keeps it in an owner-only directory on its own persistent disk, so no platform
component creates or unseals the seed. The key is synced and linked into place
without replacing anything; a missing, mismatched, unreadable or loosely
permissioned key fails closed instead of being regenerated.

* fix: make the SQS worker key durably named before returning it

Every successful call now syncs the key directory and its entry in the parent
before handing out a key, so a retry after a failed directory sync cannot
return a key whose name is not yet durable. Only the key directory is created,
inside a parent that must exist, so an unmounted worker disk fails closed. The
first-start decision is explicit: an existing store's authorized key is
required and never defaults to generating a new one.

* docs: state that workerkey cannot detect an unmounted worker disk

An existing parent directory does not prove the worker disk is mounted, since
the mountpoint can exist on the initramfs after a failed mount. The error now
says only that the parent is missing, and the package documents that its
caller must verify the expected data volume is mounted first.

* feat: add the SQS worker guest bootstrap

The shared SQS worker microVM boots with sqs-worker boot, which selects
exactly the disk whose serial fw_cfg names, checks and mounts it at /data,
and proves from the mount table that this disk is what is mounted before
loading or creating its store key there. A missing, empty or unknown store
status, a contradictory or invalid store key, a missing or ambiguous disk, a
failed check or mount, an empty mountpoint or a foreign mount refuses without
creating a key. Only the public key is logged and published.

The image is built by the store image script under SQS_MICROVM_PROFILE=worker,
with the worker binary, no nats-server and an empty /data mountpoint; the
store image's contents are unchanged.

* fix: refuse replayable new-store status at SQS worker boot

A static fw_cfg store-status of "new" survives a guest reset, so after the
store exists it could still reach key creation and replace a lost worker key.
Boot now accepts only "existing" with the store's authorized public key and
refuses "new" before touching any disk, until a current authoritative
first-store decision exists. workerkey.ForNewStore remains, uncalled.

* feat: add SQS first-store claim and worker key acquisition

The SQS domain records one first-store decision per Region in its own KV
bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound
to the worker's instance ID, claim ID and generation. Key creation is
authorized only while no public key is recorded; afterwards only the
recorded key is returned, and a launch intent requires it. Older
generations, same-generation rivals and unreadable records authorize
nothing, and records hold no private key material.

workerkey.Acquire syncs a new key to disk and returns it only after its
public key is recorded, reusing the on-disk key after a crash before
recording and failing closed on a missing or different key afterwards.
No transport carries the decision yet.

* fix: read SQS first-store decisions from the stream leader

KV.Get uses a direct get, which a lagging replica can answer, so a stale
claimed read could authorize key creation after the leader had recorded
the worker's public key, or make a superseded generation look current.
Reads now use the SQS store's leader-answered stream message get, and
every authorizing answer is also confirmed by a compare-and-swap at the
revision read, so a stale or unavailable read authorizes nothing.

* feat(sqs): add regional worker identity record and instance index

The SQS domain keeps one worker identity record per Region with a monotonic
generation, claim ID, pinned node and encrypted data volume, moving from
launching to bound or failed under compare-and-swap. A launch intent takes
the next generation before any instance exists, so the previous worker stops
being current; binding writes the instance index entry first.

Authorize accepts an instance only while the Region's leader-read record is
bound to exactly that instance, generation and claim, confirmed by a
compare-and-swap at the read revision, so an old, orphaned, rewritten or
unbound index entry, or a lagging replica, authorizes nothing.

Both this bucket and the first-store bucket now open through the kvutil
chokepoint and refuse a stored configuration that could expire, cap or evict
records or holds fewer replicas than the cluster needs. Nothing calls the
store yet.

* fix(sqs): make worker launch exclusive and harden durable buckets

BeginLaunch now refuses a new claim while the previous attempt is launching
or bound, including one whose outcome is unknown, so only one caller ever
holds an intent to start a VM on the pinned volume. A replacement takes the
next generation only after FailLaunch or the new RecordStopped confirms that
nothing holds the volume, and each request names the generation it
supersedes, so a replayed request cannot take a generation it never saw.

The durable bucket opener also refuses non-limits retention, per-message
TTLs, asynchronous flush, a message-size cap, any discard policy other than
discard-new or a per-subject discard, unacknowledged writes, a sealed
stream, and a mirror or sources.

* feat: add internal SQS worker first-store gateway actions

Serve ClaimFirstStore and RecordWorkerPublicKey under the internal
spinifex-sqs signing service, apart from the public AmazonSQS inventory.
Only an sqsWorkerRole session in the system account passes the class
gate, then the role policy; the worker identity is resolved from the
current workerident record by the STS-set session name, never from the
request, and any extra parameter is refused. Seeds and other non-public
keys are refused without being recorded, logged or echoed.

The gateway binds the worker-identity and first-store buckets per
request and never creates them, so it provisions nothing and answers
ServiceUnavailable until a worker launch has created them.

* fix(sqs): fence replaced worker generations out of first-store

A request that passed worker authorization before a confirmed stop and
replacement could reach an unclaimed first-store record ahead of the
successor and create or record the old generation's key. The launch path
now raises a monotonic first-store fence past the replaced generation
before BeginLaunch, which refuses a replacement until a leader read shows
it; firststore refuses every call from a generation below the fence.

The internal adapter no longer logs caller-chosen parameter names.

* feat(sqs): launch the shared SQS worker on its pinned node

Add the SQS worker system-instance kind and a domain launcher that
realizes the Region's worker from its identity record without ever
starting a second VM:

- domains/sqs/workerlaunch returns a bound worker as is, prepares the
  data volume on first launch only, refuses a volume whose own record
  is not encrypted, marks the intent dispatched before launching and
  never resends a dispatched launch: an ambiguous outcome stays
  launching until reconciled. A replacement starts only after a
  confirmed stop or failure and the first-store fence, reusing the
  pinned volume.
- workerident gains MarkDispatched and Current; firststore gains a
  leader-read RecordedKey.
- The daemon launches the worker through the shared direct-boot path
  with its encrypted volume attached at boot and kept on terminate, a
  required instance profile and a mandatory mgmt NIC. Its firmware
  config carries only public values: the data serial, the store
  decision, the gateway URL on this node's bridge and the daemon's own
  cluster CA. A relaunch is refused on any other node.
- sqsWorkerRole is ensured in the system account granting exactly the
  two internal actions; an existing role whose trust admits more than
  EC2 is refused. The role contract lives in domains/sqs/workerrole so
  the composition root does not import the gateway adapter.

The worker network allocator and any caller remain unwired, and the
guest still refuses to create a key for a new store.

* fix: verify SQS worker role trust before granting and report unclaimed volumes

Check that an existing or racing-created sqsWorkerRole trusts only EC2
before writing its policy, and accept the instance profile only when a
re-read shows it carries exactly that role. Refuse the first launch
before allocating when the cluster storage key is not configured, and
return the ID of any volume created but not confirmed in a launch
intent instead of dropping it.

* feat: add private SQS system VPC and worker ENI allocator

Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet
from the VPC and subnet surface only, so it cannot create an internet
gateway, route table, NAT gateway or EIP. The daemon's SQS worker network
adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and
finds or creates one ENI per launch claim in the VPC default security
group, refusing a duplicated claim or an address outside the subnet.

Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have
recorded the volume.

* feat: worker takes its first-store decision from the gateway at boot

The SQS worker guest no longer trusts a static fw_cfg store status. After
selecting, checking and verifying the mount of its own data disk, it asks
the internal spinifex-sqs gateway service for its current first-store
decision, signed with instance-metadata credentials only, over TLS 1.3
trusting only the cluster CA. It then creates and records or loads the
key on that disk; a key lost after recording is never replaced.

- internal/sqsworkergw: SigV4 client for ClaimFirstStore and
  RecordWorkerPublicKey; refuses non-public keys, non-decisions and
  redirects; errors carry only the parsed code
- gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback
- workerboot: retries only lost replies and ServiceUnavailable/Throttling
- launch contract carries the Region instead of a store status or key

* fix: worker retries a denial while its launch is still binding it

The worker VM can reach the gateway before its launch binds the instance
identity, and that early call is legitimately AccessDenied. The guest
treated the denial as final and stayed unprepared even after binding.
AccessDenied is now a pending authorization the guest asks about again
within its existing bound; a denied claim creates nothing, and a worker
that is never bound, or was replaced, gives up at the bound.

* feat(sqs): add operator EnsureSQSWorker action and node launch handler

Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the
internal spinifex operator service. It validates the node against the
cluster and as a single subject token, then requests
sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers
and the worker stays pinned.

The node handler re-checks the named node and instance type and runs one
composed workerlaunch.Launcher (real volume preparer, role, private network
and direct-boot launch), one launch at a time, in the daemon's own context
with a 30-minute bound. A caller timeout does not cancel the launch; a
retry reports the bound worker, or IncorrectState for a dispatched launch
whose outcome is unknown, which is never resent. The reply names the bound
VM and does not claim the worker or SQS is ready. Nothing starts a worker
at daemon startup.

* feat: ship the SQS worker image and check it before allocating

Build the SQS worker microVM image in both release workflows, stage it into
the tarball as microvm-sqs-worker, and install it with setup.sh under the
store image's rules: a release without it removes an earlier one and an
incomplete one is refused. Add make install-sqs-worker-microvm.

The worker launcher now asks a Host capability whether this node can boot a
worker before it prepares a volume, records or resumes an intent, fences a
replaced generation, allocates the network or dispatches. The daemon checks
for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and
the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a
call after the image is installed launches instead of leaving an ambiguous
dispatched intent.

Cover missing and incomplete images through the admin action and handler,
a stopped worker left untouched, and an ENI create whose reply is lost.

* fix: never repeat an uncertain SQS worker ENI create

The launcher now looks for the claim's ENI without creating one, marks the
launch intent NetworkAttempted, and only then sends the single create. Once
marked, a missing ENI fails closed with IncorrectState instead of a second
create that could give the claim two interfaces; a visible ENI is reused.

Adds a test that buildSQSWorkerLauncher wires the daemon's real host check.

* feat(sqs): operator-triggered SQS store launch on the worker's node

Add the super-admin EnsureSQSStore action and its node handler. The
store is launched only on the node the Region's worker is bound to,
with the public key that worker recorded in the first-store decision;
the daemon never receives the worker's seed.

A per-Region store launch record marks each step before its uncertain
create (volume, ENI in the worker's private subnet, VM launch), so a
step whose result is not visible fails closed and is never resent.
Volumes are refused unless their own record is encrypted, and their
IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and
instance types at or below that are refused; this is a floor, not a
validated size.

* fix: retry uncreated SQS store volumes and fix the store instance type

A volume preparation that failed before sending any create (a missing
tool, scratch or format failure, or no cluster storage key) now carries
storevolume.ErrNotCreated. The store launcher withdraws its volume mark
and returns ServiceUnavailable, so a retry may try again once the host
is fixed; any create that was sent stays one-shot.

The store launch record now keeps the operator's instance type. A
request for a different type is refused as IncorrectState before
dispatch and after binding, and the reply names the launched type,
which the admin action checks.

* feat: SQS worker store readiness and store size check

The worker's one-shot ready step mounts and verifies its disk, asks the
internal spinifex-sqs DescribeStore action for the bound store, loads its
key only if it matches the recorded public key, and checks the store's
JetStream with it. It retries within a bound while the store is pending or
not answering, and stops on a mismatched or refused key. The NKey
authenticates the worker only; the store server is not authenticated.

EnsureSQSStore now also refuses a different volume size once the store is
bound, instead of returning the existing store.

* fix(sqs): keep worker store readiness current until shutdown

The readiness step now runs as a guest-local watch loop with no overall
deadline: each pass is bounded, a pending or unreachable store is retried
every few seconds, and a ready store is rechecked periodically. The
store-ready marker is removed on any failed pass, cleared at start and
removed on exit; shutdown stops the watcher by its recorded PID. Terminal
refusals such as a rejected or mismatched key back off for minutes rather
than spinning. The NKey still authenticates only the worker to the store.

* feat(sqs): supervise the worker readiness watcher and add spx admin sqs

Run the worker guest's readiness watcher under init respawn, rate-limited,
and remove any stale store-ready marker before each start, so a killed or
crashed watcher cannot leave a false readiness marker. Shutdown blocks
respawn, removes the marker, stops the recorded watcher by PID and start
time with SIGKILL escalation, and unmounts /data only once it has exited;
a surviving watcher leaves every mount in place.

Add spx admin sqs ensure-worker and ensure-store, which call the existing
EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex
gateway path with the standard AWS credential chain. Both refuse missing
inputs before sending anything and report only the bound VM, never
readiness.

* fix: park SQS worker readiness and fix the worker instance type

A readiness start that finds shutdown in progress or no worker key now
logs once and waits in place instead of exiting, so init's respawn does
not restart and log it in a loop; shutdown stops it like a watcher.

The worker identity record keeps each generation's instance type. A
request for another type is refused as IncorrectState before dispatch
and after binding, and only a replacement may change it. The ensure
reply, admin action and spx admin sqs ensure-worker report and check
the recorded type.

* fix: record the SQS worker volume intent before creating it

The first worker launch prepared its data volume before any record
existed, so a create that applied but returned no ID left nothing for the
next call to find, and it could prepare a second volume.

The launch intent is now recorded first, and its one volume create is
marked before it is sent. A Prepare proven to have sent nothing withdraws
the mark and is retryable; any other outcome is never resent and returns
IncorrectState, naming the volume when its ID is known. An intent that
has sent no create may still be re-typed.

* docs: document spx admin sqs ensure-worker and ensure-store

Record the flags, the caller and owning accounts, the bound-versus-ready
distinction and the current retry and recovery limits.

* docs: distinguish SQS ensure timeouts from refusals that created nothing

A gateway timeout means only that no reply arrived; the launch may still be
running. The worker type locks once any step has been attempted.

* docs: state what each SQS ensure ServiceUnavailable establishes

No refusal proves earlier attempts created nothing; a retry checks the
durable attempt and continues only when that is safe.
2026-10-31 00:17:25 +10:00
Josh Siviour 55054ad0ac fix: interim guard against opening a volume behind a live owner holding unsealed writes (#1352)
* test: relay a capacity refusal from an owner holding unsealed writes

* fix: refuse a start elsewhere when the owner holds unsealed writes

* fix: bound the unsealed-writes check well under the start forward timeout

* test: refuse a volume takeover while the marker owner is live

* fix: refuse a volume takeover while the previous owner is live

* fix: retry EKS control-plane starts refused for capacity or a held volume

* docs: describe the interim unsealed-writes takeover guards

* test: give the viperblockd conformance harness a cluster-state bucket

The takeover guard reads node liveness from the cluster-state bucket the
daemon creates. The harness had none, so the real reader errored and the
guard refused every takeover. Create the bucket the way the daemon does so
the real reader runs and finds no heartbeat.

* fix: leave no dirty marker behind when opening a volume that does not exist

An open takes the lease and writes the dirty marker before the volume's
state is read. When the volume turned out to be absent the marker stayed,
so a later open on another node met the takeover guard and was refused
as held elsewhere instead of answering not found. Clear the marker this
open wrote when the state load reports the volume missing.
2026-10-30 23:53:41 +10:00
Josh Siviour 27304bbfd6 refactor: serve the four Bedrock services from one Ochre registration (#1351)
* test: characterise the four Bedrock services through the SigV4 pipeline

* refactor: serve the four Bedrock services from one ochre registration

The Bedrock adapters move from gateway to domains/ochre/awsapi behind a
single dispatch registration. The sub-service scopes are aliases, a
sub-service scope selects its dispatcher outright and only the shared
bedrock scope is split by path. The thirteen Bedrock fields leave
GatewayConfig for awsapi.Deps, and the bodyscope allowlist row for
domains/ochre is deleted.

* docs: record the Bedrock registration in the package boundary migration

* docs: record the Bedrock registration PR number
2026-10-30 23:06:24 +10:00
Josh Siviour a52b078678 refactor: move the ECR /v2 bridge out of the gateway (#1350)
* test: pin the ECR /v2 pipeline through the gateway handler

* feat: rehydrate a principal from an access key id in credverify

Adds credential.Rehydrator and CallerARN on Principal, implemented beside Authenticate and Resolve on the same key, session and user lookups, with a parity table against the gateway's ECR rehydrate.

* refactor: share the credverify key and session lookups with Rehydrate

Authenticate, Resolve and Rehydrate now call one lookup per credential kind, so the checks cannot drift. The parity table also asserts the gateway and credverify caller ARN builders agree.

* refactor: move registry host matching to ingress/http

The matcher takes its service labels from the caller and exposes the parsed account and region through accessors, so ingress names no service.

* refactor: extract the identity-policy evaluator into policyauth

The SigV4 gateway path and the ECR registry authorize through one evaluator and one set of request condition keys, over the credential principal.

* refactor: extract account admission into accountadmission

The SigV4 gateway path asks one Admitter for account status, with the ACTIVE-only cache moved with it, so the ECR registry can use the same capability.

* refactor: move the ECR /v2 bridge out of the gateway

The bridge, token exchange and per-operation authorization now live in
domains/ecr/registry behind registry-owned Principal, Rehydrator and
Authorizer types, so the package imports no ingress code. The awsgw role
adapts credverify, policyauth and host matching to them, and the gateway
keeps only a RegistryHandler mounted at /v2. The bridge refuses to
compose without a verifier, issuer, rehydrator, authorizer or admission,
so the open-registry mode for a nil verifier is gone.

* test: carry the retired ECR principal parity rows into credverify

Global root, a verifier with no IAM, a record with no principal type and
the gone, replaced and legacy session verdicts for user and role sessions
were only exercised by the gateway tests that went with the old path.

* docs: record the ECR /v2 bridge move in the package boundary migration

* docs: record the ECR /v2 bridge PR number
2026-10-30 15:57:37 +10:00
Josh Siviour 26d53602a3 fix: accept AWS-format IRSA trust-policy condition keys (#1349)
* test: cover AWS-form IRSA condition keys and StringLike on web identity

* fix: match AWS-form IRSA condition keys and allow StringLike on web identity

* fix: keep the provider ARN host-path unnormalised for web identity

* feat: log the reason a web identity trust policy denies

* fix: satisfy errname, modernize and gofmt on the denial error

* test: cover the stock AWS IRSA trust policy in the EKS e2e

* test: make the IRSA denial prove the policy was evaluated
2026-10-30 14:34:08 +10:00
Josh Siviour 060204514e refactor: register ECS and EKS through the ingress seam with shared PassRole (#1348)
* test: pin ECS and EKS dispatch through the SigV4 pipeline

* refactor: register ECS and EKS on the ingress seam

EKS resolves the cluster and nodegroup role through passrole.Check with the
IAM role resolver, in the order it had. Responses and check order are unchanged.

* fix: match AWS role handling for ECS and EKS PassRole cases

ECS refuses a foreign task role before PassRole, accepts a role that is not an ARN or does not exist after PassRole, and reports a RunTask override role it cannot assume. EKS CreateCluster and CreateNodegroup refuse a foreign or non-role ARN with AccessDeniedException before PassRole.

* test: cover ECS and EKS awsapi registration dispatch

* fix: report EKS parameter faults as InvalidParameterException

* docs: record ECS and EKS gateway migration and EKS divergences
2026-10-30 14:18:53 +10:00
Josh Siviour 12cea4f6f7 feat: add IAM role resolution and the shared iam:PassRole flow (#1347)
* test: pin the current role-resolution gaps in EC2, ECS and EKS

* feat: add the shared iam:PassRole flow with classified role resolution

* feat: resolve same-account roles for iam:PassRole in domains/iam
2026-10-30 12:54:44 +10:00
Josh Siviour 099a69b23b refactor: register IAM and STS through the ingress seam (#1346)
* docs: point the /admin migration row at spinifex#1345

* test: characterise IAM and STS dispatch through the SigV4 pipeline

* refactor: register IAM through the ingress seam

* refactor: register STS through the ingress seam

* docs: point the IAM and STS migration rows at spinifex#1346
2026-10-30 12:08:15 +10:00
Josh Siviour 6462ccc41f refactor: serve the /admin operator API from the spinifex registration (#1345)
* refactor: move the /admin handlers and wire types into operator/admin

* refactor: serve /admin through the spinifex registration

* docs: record the /admin move into the spinifex registration
2026-10-30 10:50:47 +10:00
Josh Siviour ae8ee1e5fe refactor: register RDS through the ingress seam (#1344)
* refactor(gateway): register RDS through the ingress seam

RDS dispatch moves from gateway/rds.go to domains/rds/awsapi.NewRegistration, composed by awsgw and the integration harness.
The caller comes from dispatch.Actor and the principal-class gate still runs before the policy check.

* docs: point the RDS migration row at spinifex#1344

* chore: group the imports merged in the actor projection test
2026-10-30 09:59:15 +10:00
Josh Siviour 488f1c7bf9 feat: SQS store guest on a prepared cold-attached EBS volume (not wired) (#1343)
* feat: build a separate SQS store microVM image

Package the go.mod-pinned nats-server, ext4 support and a clean ACPI
power-down into a microVM kernel and initramfs of their own. The build
checks the packed archive's contents and refuses a nats-server whose
version differs from the pin. The image does not start the store and
nothing installs or releases it; the LB image is unchanged.

* fix: publish the SQS store image only after its gates pass

Stage the kernel and initramfs and move them into place only once the
content, size and guest-RAM gates pass, so a failed build cannot leave a
pair make takes as up to date. The old initramfs is removed first and the
new one lands last, so an interrupted publish never leaves a mixed pair.
Check the module list without piping into grep -q, which under pipefail
reported a present module as missing.

* feat: start the SQS store guest on its checked data disk

The SQS store image now prepares, runs and stops a dedicated JetStream
server. It reads a narrow fw_cfg contract (data disk serial, listen
address, one user NKey public key), refuses anything absent, malformed
or ambiguous, has the pinned server test the config, then checks and
mounts exactly one matching ext4 disk. It never formats and has no
ephemeral fallback. The server listens only on the supplied guest NIC
address, admits only that NKey and syncs every write.

Shutdown stops the recorded run process first, whether it is the server
or a start not yet exec'd, and runs the generic read-only remount only
once it is gone. A store that survives SIGKILL leaves every mount alone.

* fix: report the SQS store as prepared, not listening, before it starts

* feat: launch the SQS store guest on its own versioned subject

An explicit SQS store kind carries an existing system-account data volume
and the one NATS user public key the store accepts. It is served only on
system.LaunchSQSStore.v1, so a daemon without this code has no responder
and the launch fails instead of booting a load balancer without its disk;
the legacy launch path refuses the kind on both send and receive.

Every refusal happens before capacity, an ENI or a VM record exists. The
volume goes through the normal EBS mount and terminate detach and is
never deleted on termination. The store boots its own image with CPU
mitigations on, gets only its network, disk serial, listen address and
key, and cannot be relaunched after a host restart yet.

* feat: ship the SQS store microVM image in releases and installs

Release builds now build the SQS store image and stage it beside the load
balancer's as microvm-sqs-store/, so an older installer that copies
microvm/* without -r still works. setup.sh installs it to
/usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it.

A release without a complete image never leaves an older one usable:
staging drops a previously staged image, and the installer removes an
earlier installed pair before warning (missing) or failing (incomplete).
Only the two image files are removed. The initramfs is installed last, so
an interrupted install leaves no bootable mismatched pair.

* feat: restart the SQS store guest on its launching node

Persist the non-secret guest config of an SQS store instance (launching
node, data volume, listen address, client public NKey and rendered
network config) in the instance record, and rebuild its fw_cfg files
from it on relaunch. The relaunch refuses on any other node, or when the
record is missing or malformed, before writing any file; the data volume
is remounted through the normal EBS path and kept on terminate.

Cross-node recovery now never selects or claims an SQS store record, so
no other node spends retry budget on it or relaunches it. Same-node
restore is unchanged, as are load balancer restarts. Older daemons know
neither the exclusion nor the record field, so this does not prevent a
cross-node relaunch by a mixed-version cluster.

* feat: hold prepared system volumes in creating until released

Add an internal CreateVolumeForPreparation path that writes a new
volume's first record as creating, and CompleteVolumePreparation to
release an unattached creating volume in the caller's account to
available. Ordinary CreateVolume is unchanged.

Attach and delete already refuse a creating volume. ModifyVolume,
ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState
so nothing can expand, release or capture a volume mid-preparation.

* feat: prepare fresh SQS store volumes with an ext4 filesystem

The SQS store guest never formats a disk, so a new volume gets its
filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img,
builds an ext4 image in private scratch, creates the volume in
"creating", publishes it on this node, copies the image with the NBD
import helper, unpublishes, and only then releases it as available.

It never writes a volume it did not just create. A volume is deleted
only when no export can still be live: after a confirmed unpublish, or
when the export was not NBD. Any publish error, a failed or ambiguous
unpublish, or a failed release retains the volume and reports its ID.
A publish refused as in-use or already-exists sends no unpublish, since
another export may exist. Cleanup ignores caller cancellation but is
bounded at 30s.

The raw-image copy moves to runtime/compute/nbd unchanged, and the
domain receives it as an injected writer. The volume service gains a
discard path for an unattached volume still in "creating".

* fix: report unknown state and match qemu-img lookup in volume prep

A failed release may still have made the volume available, so the
retained-volume error now says its state is unknown rather than
claiming it is still creating.

The copy runs qemu-img by name through PATH, so the pre-allocation
check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin
fallback because it runs from the resolved path.

* docs: describe the unwired SQS store system instance

Records what the guest image, versioned launch, same-node restart and
fresh-volume preparation provide, that nothing calls them, and what is
untested or missing before production use.
2026-10-30 08:16:43 +10:00
Ben Duncan 3617e4a045 fix: log an expected outcome at its own level, not ERROR (#1342)
A dispatch failure was logged at ERROR whatever the caller was about to
be given, so DryRunOperation, which AWS documents as the success answer
to a dry run, read as a fault of ours along with every NotFound and
malformed parameter. The level now comes from the status the caller
receives, resolved the way ErrorHandler resolves it.

QEMU announces its own exit on stderr, so a guest stopping on the
SIGTERM the daemon sent it was logged at ERROR too.
2026-10-29 14:38:41 -05:00
Josh Siviour 24281cb860 fix: count volume lease validity from the write, not its reply (#1340)
* fix: count volume lease validity from the write, not its reply

The server starts a lease entry's TTL when it applies the write, but the
holder started its validity window when the acknowledgement arrived, so a
slow reply stretched validity past the server's expiry and a peer could
acquire the volume while the holder still considered it valid.

Validity now counts from when each create, generation publish and renewal
was sent. An acquisition acknowledged after validity lapsed is refused as
retryable, and a lost or lapsed lease still awaiting its fence is no
longer shared with another open on the same node. A renewal that lands
after the lease was given up, or after its validity elapsed, cannot revive
it: the lapse check and the update run under the lease lock, and fencing
runs once. The renew loop also fences at the validity deadline itself
rather than at the next check, which could consume the whole margin.

This bounds local validity only; it is not backend-enforced fencing, and a
paused host can still write after its validity ends.

* docs: describe volume lease validity, fencing and their limits
2026-10-29 21:38:22 +10:00
Josh Siviour e58838bc67 refactor: verify SigV4 credentials through an ingress capability (#1339)
* refactor(gateway): verify SigV4 credentials behind a CredentialVerifier

The authenticator no longer imports the IAM and STS domains. It hands the
signature check to a verifier as a callback, so the secret never leaves the
adapter, and runs the account-active check between Authenticate and Resolve.
A nil verifier fails every request closed.

The adapter lives in runtime/roles/awsgw/credverify. ResolveCallerUser and the
principal-type constants move from domains/sts/awsapi to domains/sts so the
adapter can use them without importing an awsapi package.

* fix(gateway): guard a nil credential verifier and tighten its tests

A typed-nil credverify.Verifier now fails closed instead of panicking.
The session verdict test goes through the real verifier into the lockout boundary.
Test setups build the verifier from the same mock as the gateway services.

* test(gateway): commit the verifier test helper
2026-10-29 21:30:06 +10:00
Josh Siviour f0e4ffe70c refactor: give seam services the verified caller's identity facts (#1338)
* test: pin the caller facts the gateway gives a registered service per principal kind

* refactor(gateway): seal the verified caller into one request-context value

* feat(dispatch): give registered services the caller's principal ARN, user ID, user name and session name

Actor gains PrincipalARN, UserID, UserName and SessionName, filled from the
sealed principal, and RoleName(), which keeps the account-checked role ARN
rule in one place. No consumer reads them yet.

* refactor(gateway): read the caller from dispatch.Actor in the RDS, EKS, STS, IAM and admin handlers

callerActor builds the Actor the seam hands services, and the seven readers that
took fields off the sealed principal now take them from it. RDS and EKS take the
role name from Actor.RoleName(). No behaviour change.
2026-10-29 16:13:41 +10:00
Josh Siviour 18adea9486 refactor: register the spinifex operator service through the ingress seam (#1337)
* test: characterize the spinifex Query and /admin surfaces through the full gateway pipeline

* refactor: register the spinifex Query service from operator/admin

* docs: point the spinifex registration migration row at the PR
2026-10-29 14:55:50 +10:00
Julian Sommer f06def6894 chore: update operation count 2026-10-29 13:50:59 +10:00
Julian Sommer ef7d30578f chore(awsmodel): sync AWS models to api-models-aws e10d6a1c (2026-10-28)
Seven models changed and add 26 unimplemented operations. The request sweep gains 33 requests with no new findings.
2026-10-29 13:35:39 +10:00
Julian Sommer b76679e571 fix(integration): restore ELBv2 to the request conformance sweep
The ELBv2 move to the ingress seam added its inventory to the coverage tool but not to TestRequestConformance, so 36 operations and 45 findings dropped out while the suite passed. A served service with no implemented operations now fails the sweep.
2026-10-29 13:34:12 +10:00
Josh Siviour d7d922e963 refactor: register Tagging and ELBv2 through the ingress seam (#1336)
* test: characterize Tagging dispatch, authorization and GetResources responses through the gateway pipeline

* refactor: register Resource Groups Tagging through the ingress seam

* test: characterize ELBv2 through the full gateway pipeline

* refactor: register ELBv2 through the ingress seam

* test: cover the ELBv2 registration, query parser and load balancer limit

* test: characterize RDS through the full gateway pipeline

* docs: point the Tagging and ELBv2 migration rows at the PR

* test: register Tagging and ELBv2 on the integration harness gateway
2026-10-29 13:27:43 +10:00
Julian Sommer 64e3711893 fix(awsmodel): accept InvalidGatewayID.NotFound in the EC2 error catalog
AWS's EC2 error reference documents it, and DeleteEgressOnlyInternetGateway and ReplaceRoute answer it, so the integration suite reported two false violations.
2026-10-29 13:27:09 +10:00
Josh Siviour ad9568a6c6 refactor: move gateway/spx, admin wire types and EKS OIDC discovery out of gateway (#1334)
* refactor: move gateway/spx to operator/admin

* test: pin the admin wire types and method list in golden JSON

* refactor: move the admin wire types and method list to contracts/operator/v1

* refactor: serve EKS OIDC discovery and JWKS from the EKS domain

* refactor: assert the EKS OIDC publisher satisfies the gateway interface

* docs: cite the pull request in the gateway root-move rows
2026-10-29 12:22:16 +10:00
Josh Siviour dedb3d8d75 ci: keep the E2E WAN packet captures only for runs that did not pass (#1335)
The captures are about 90% of each E2E artifact and nothing reads them unless
a run failed, so a clean single-node or multi-node pass drops them before the
artifact upload.
2026-10-29 12:21:30 +10:00
Julian Sommer cc1186fa48 fix: grant the build user passwordless sudo for the appliance build when it lacks it, and remove the grant at seal 2026-10-29 11:10:51 +10:00
Julian Sommer dbdb827a69 fix: install tar on a minimal Oracle Linux build system before copying the appliance payload 2026-10-29 11:05:35 +10:00
Josh Siviour c94b0f22f6 feat: accept EnaSupport on RegisterImage, report it on instances, and name refused parameters (#1332)
* feat: accept EnaSupport on RegisterImage and name the refused parameter in its errors

EnaSupport true and false are stored on the image record, returned by DescribeImages when set and carried through CopyImage. Every validation error in ValidateRegisterImageInput keeps its code and now says which parameter and value it refused.

* feat: set EnaSupport on images created from an instance

AWS takes an image's EnaSupport from the instance it was made from, and every instance here reports it, so the source AMI's value is not inherited.

* docs: record RegisterImage EnaSupport support and the refused parameters

* fix: report EnaSupport on DescribeInstances and RunInstances
2026-10-29 10:53:30 +10:00
Julian Sommer 71567d0998 fix: show the appliance password rules before the first-login change 2026-10-29 10:51:06 +10:00
Josh Siviour 11e8b68ddc refactor: remove the unused host SQS message store service (#1333)
* refactor: remove the host SQS store systemd unit

The SQS message store will run inside system-owned instances rather than
as a host daemon, so spinifex.target no longer starts it. Bump the target
to version 4 and drop the unit from the embedded set and its invariants.

* refactor: remove host SQS store config provisioning

The SQS store will run in system-owned instances, so admin init and join
no longer generate or distribute a host store token, routes or config,
and the formation join payload drops the store token field.

* refactor: remove host SQS store service command and role

The SQS store will run in system-owned instances, so drop the unused
spx service sqs-store command, its service registration and the host
sqsstore role. The domains/sqs/message JetStream store is unaffected.

* refactor: remove host SQS store provisioning from install scripts

The SQS store will run in system-owned instances, so setup no longer
creates the spinifex-sqsstore user, its directories or the 4223/4249
peer firewall ports, and install, reset and uninstall stop handling the
retired unit, user and data directory.

* docs: drop the host SQS store from firewall and connection tables

Spinifex no longer ships a host SQS store service, so remove its ports
from the host firewall and network connection references and say in
DESIGN.md that no host service, unit, user or firewall rule exists for it.
2026-10-29 10:48:05 +10:00
Julian Sommer 8f4754da13 fix: install the appliance with the setup.sh from the release tarball, since install.mulgadc.com serves the stable one 2026-10-29 10:30:48 +10:00
Julian Sommer 8b965d24ce fix: refuse Route 53 requests that break the model's bounds with InvalidInput (#1329) 2026-10-29 11:29:45 +11:00
Julian Sommer ac2620df58 fix: report an SSH failure in build-oracle-appliance.sh as one, not as missing sudo 2026-10-29 10:26:09 +10:00
Julian Sommer 2feb264863 feat: Oracle Linux multi-node clusters with OVN IPsec (#1327) 2026-10-29 11:17:22 +11:00
Josh Siviour ddadfa2812 chore: remove unused e2e suite-selection manifest and drift lints (#1331)
The targeted suite selector (select-suites) has not run in CI since its shadow-mode step was removed in #556.
The service-interfaces manifest, manifest-check and manifest-lint existed only to feed it, so they go too.
Also drops the now-meaningless e2e:allow-create markers, keeping their rationale as plain comments.
2026-10-29 10:04:31 +10:00
Josh Siviour 5923143062 feat: set X-Forwarded-For, -Proto and -Port on ALB requests (#1330)
* feat: render X-Forwarded-For, -Proto and -Port headers in ALB HAProxy frontends

Append joins every client X-Forwarded-For line and adds the client address as AWS does, preserve and remove follow the load balancer attribute, and Proto and Port are always overwritten.

* feat: validate the ALB forwarding attributes and re-render config when they change

ModifyLoadBalancerAttributes rejects bad xff_header_processing.mode and xff_client_port.enabled values with ValidationError before storing anything, and regenerates the stored config when either key changes.

* test: assert the ALB X-Forwarded headers and the remove mode in the lb e2e

The app instances echo the request headers on /__headers. The internal ALB is driven from the probe client and the HTTPS ALB from the runner, covering the caller address, a client value, spoofed Proto and Port, mode remove and a rejected mode.

* test: let the lb probe client trigger accept a null header list

* docs: document the ALB forwarding headers and their attributes
2026-10-29 09:33:39 +10:00
Julian Sommer 46ca3dfba1 chore: comment 2026-10-29 09:13:32 +10:00
Julian Sommer 596b5b1733 fix: refuse DeleteVpc while an egress-only internet gateway is attached (#1325) 2026-10-29 09:53:48 +11:00
Josh Siviour 1c5eaaf852 fix: use canonical OCI header names in the ECR registry (#1328) 2026-10-29 08:53:39 +10:00
Julian Sommer 49117cf7b0 fix: return InvalidIPAddress.InUse when releasing an associated elastic IP (#1326) 2026-10-29 09:53:31 +11:00
Julian Sommer 3a9985c79d fix: return ResourceNotFoundException from EKS DeleteCluster for a nonexistent cluster (#1324) 2026-10-29 09:53:16 +11:00
Josh Siviour c42a352064 feat: add store-neutral SQS message store with a JetStream adapter (#1323)
* feat: add the store-neutral SQS message store contract and conformance suite

Defines message.Store over opaque shard and queue ID namespaces with no
create call, plus a reusable conformance runner exercised against a
test-only in-memory reference store.

* feat: add the SQS message store's JetStream layout, append and state CAS

Provisions the fixed set of shared message streams and state buckets,
publishes each queue on sqs.m.<shard>.<queue>, dedupes idempotent appends
with a queue-scoped Nats-Msg-Id, and maps state revision CAS onto KV.

* fix: give each SQS queue immutable message and state shard allocations

A changed stream or bucket count applies only to new queues: the namespace
names its message shard and state shard directly, retained streams and
buckets stay addressable, and the store refuses cluster sizes other than
one node (R1) or three (R3).

* fix: enumerate complete SQS namespaces and backstop state buckets

Enumeration now returns each namespace with both allocations, so a
state-only namespace can be counted and cleaned without its queue record,
and state buckets expire records after 14 days plus 1 hour.

* feat: add the SQS JetStream store's reads, counts and cleanup

Get and Delete are scoped to the namespace's subject, DeleteThrough and
DeleteNamespace are subject-filtered purges that leave shard neighbours
intact, and Count reads the per-subject message total and live state keys.

* fix: count distinct SQS state keys and validate before namespace purge

The key lister can repeat a key under concurrent writes, so Count tallies
distinct names; DeleteNamespace resolves both the message stream and the
state bucket before purging either, so a wrong state shard deletes nothing.

* feat: add the SQS JetStream store's ordered message-shard feed

Append records the namespace's state shard in an internal header that is
never returned as metadata. Feed reads one shared message stream through
an ephemeral, headers-only ordered consumer with no acknowledgement state,
rebuilds each change's complete namespace, refuses a malformed header with
ErrCorruptRecord, and deletes its consumer on Close.

* feat: enumerate SQS namespaces and run conformance on the JetStream store

Namespaces joins per-subject body counts, whose internal header names the
state shard, with live state keys from every state bucket that exists,
found by listing so a historical bucket counts and none is created. An
allocation conflict is reported as ErrCorruptRecord. The store now
implements message.Store and runs the conformance suite on R1, per case
with crash reopen and on one shared instance.

* fix: check every SQS body's state shard when enumerating namespaces

Sampling the first and last body let a middle body naming another state
shard pass. Namespaces now reads every body's header in one headers-only
ordered pass over the shard's stream, counting each queue's bodies from
that pass, and deletes its consumer when done.

* test: run SQS store conformance on R3 and guard against consumer state

TestR3Conformance runs the conformance suite on a real three-node
JetStream cluster in process, with Reopen restarting every node over the
same data, and a check that every stream and state bucket is R3.
TestS5_NoConsumerDeliveryStateInSQS fails on any acknowledgement,
redelivery, durable-consumer or ack-retention mechanism in SQS source.

* test: bar legacy NATS acknowledgement helpers from SQS source

Adds AckSync, ManualAck, AckAll and AckExplicit to the ADR-0011:S5
consumer-state guard.

* feat: add a bounded SQS store scan and the tombstoned-body sweeper

Store gains Scan: a paginated, headers-only pass over a message shard,
bounded at the shard's last position when it starts, skipping deleted
positions and never returning a later append. The conformance suite and
fake cover continuation, deleted runs and the bound.

The JetStream adapter reads every bounded range (scan, body enumeration
and state keys) through one reader whose completion rests on a direct get
of the next match, not on consumer state; the client's key lister and
pending counts are no longer trusted to end a pass.

message.Sweep walks a shard with Scan and, for each body whose record the
domain reports terminal, deletes the body then the record by revision.

* test: add an opt-in SQS store conformance run on an external R3 cluster

TestR3ExternalConformance runs the conformance suite against the
dedicated three-node SQS JetStream cluster named by SQS_STORE_R3_URLS,
checks every stream is R3 with two peers, and uses SQS_STORE_R3_RESTART
to restart every node for the crash cases. It skips when unset.

* test: run the SQS sweeper against the reference store fake

Covers live records kept, tombstones swept, another shard untouched, a
no-op rerun, and a retry after the body delete succeeds and the state
delete fails.

* test: harden the SQS store R3 harnesses against restart and placement races

The external R3 harness reused one connection across restarts, so a case's
cleanup closed it for every later case; each case and restart now opens its
own connection and waits for stream leaders.

The in-process cluster released its free ports before the servers bound them,
and R3 placement can fail until the meta leader has every peer's config and
stats. Client ports are now server-chosen, route ports are reserved together,
and provisioning retries the idempotent New until placement succeeds.

* fix: stop the SQS store feed idling for a whole pull expiry

Under load an ordered feed on a three-node cluster sat for nats.go's 30s
default pull expiry, once 60s, with stored changes pending and no pull
waiting on its consumer, both on feeds opened after a full restart and on
feeds kept open across one. Feed now sets a 2s pull expiry.

Adds restart tests for a feed opened after a full-cluster restart and one
open across it, each replaying a shared-shard backlog then a live append and
failing on any 5s gap. The in-process cluster keeps client ports across
restarts so open connections reconnect, and can write node logs.

* fix: read SQS message state from the stream leader

GetState used nats.go's KV Get, which switches to a direct get whenever
the bucket allows it. On a replicated bucket a follower can answer that
from state it has not applied yet, so a record the leader had already
deleted or purged could still be returned.

Read the key's last message with a stream message get instead, which
only the stream leader answers, and apply KV Get's delete and purge
marker rules to the result. Bucket configuration is unchanged. Missing,
deleted and purged records still return ErrNotFound; values, revisions,
context errors and the API prefix or domain of the caller's JetStream
context are preserved.

* docs: narrow the SQS GetState leader-read comment to what it avoids

* fix: enumerate SQS state keys from the stream leader

Count and Namespaces found live state keys with a direct-get probe and a
headers-only ordered consumer on the state bucket's stream. On a
replicated bucket a follower can answer either from state it has not
applied, so keys the leader had already deleted could still be counted.

Walk the key prefix with stream message gets by next subject, which
only the stream leader answers, through the stream's last sequence at
the call. Delete and purge markers drop a key, as GetState reads them.
This costs one leader request per record under the prefix. Bucket
configuration, historical state shards, the message-body scans and the
feed are unchanged.

* test: keep SQS API recorder deliveries in arrival order

The recorder fed API subjects and its sentinel to two async
subscriptions, which nats.go delivers on separate goroutines, so drain
could return before the last request was recorded. Both now share one
channel that the connection fills in arrival order.

* fix: read SQS body enumerations from the stream leader

Namespaces and Scan read bodies through an ephemeral ordered consumer,
which the server may place on a follower that has not applied a purge,
so a deleted queue could still be listed or scanned. Each stored match
through the bound is now a leader stream message get. That transfers
every full body, where the consumer read headers only.

* test: stop the SQS API recorder goroutine with its test

The recorder ranged over a caller-owned ChanSubscribe channel, which
nats.go never closes, so its goroutine outlived every test. Cleanup now
unsubscribes both subscriptions, then signals the goroutine and waits
for it to return; the sentinel send also watches that signal.

* fix: recover a stalled SQS feed from its cursor

An ordered feed was seen to idle with stored changes pending after a
cluster restart. Next now waits at most an idle check on the consumer,
then asks the stream leader whether a change follows the cursor; if one
does, the consumer is deleted and reopened just after the cursor, so no
change is skipped or repeated. The root cause of the stall is unknown.

* fix: make closing an SQS feed terminal and safe against Next

Close now marks the feed closed under a lock, cancels any in-flight
Next, stops the iterator and deletes the consumer. A later or in-flight
Next returns ErrFeedClosed and never opens a consumer; one opened by a
recovery racing Close is deleted rather than adopted. Repeated Close
returns nil.

* fix: define SQS feed close in the store contract

message.Feed now documents one Next reader with Close callable at any
time and more than once, and message.ErrFeedClosed replaces the
adapter's own error. Once Close has begun, Next returns ErrFeedClosed
and withholds any change it had received; a feed from the last cursor
still delivers it. Shared conformance cases and the reference fake
enforce this.

The JetStream feed's Close also waits, within a bound, for a consumer
open already in progress, so a consumer created during Close is deleted
before Close returns nil; past the bound, or if that delete fails, Close
returns an error and a repeated Close waits again.

* fix: track SQS feed consumer cleanup until Close succeeds

Every consumer open and release is now tracked work recorded under the
lock that takes the consumer, and a delete that fails is kept for Close
to retry. Close returns nil only once no such work is in progress and no
failed delete remains, waiting within a bound and otherwise returning an
error; concurrent and repeated Close calls wait the same way.

Next claims a change under that lock and advances its cursor there, so a
change is either returned before Close begins or withheld for a feed
resumed from the last cursor.

A consumer create no longer inherits the caller's cancellation, which
was seen to leave most cancelled creates on the server with no owner;
it has its own timeout, and the name of one that fails is deleted and
kept for Close to delete again.

* fix: release SQS feed consumers opened after the caller's context ends

Feed and recovery now check the caller's context before starting a
consumer open and after it completes. A create that finishes after the
context ended is released as tracked work and the context's error is
returned, or ErrFeedClosed when Close won; no change is returned.

The feed contract and adapter docs now describe cleanup as complete for
consumers the feed knows of, with the server's one-minute inactive
threshold removing one an ambiguous create lands after its last delete.

* fix: report cancellation when an SQS feed open fails after it

A first consumer open that fails after the caller's context ended now
returns the context's error joined with the failure, so cancellation
stays detectable. The feed contract's cancellation text is narrowed to
opening and reopening, whose cleanup is best effort.

* docs: describe the internal SQS message store and its limits

DESIGN.md gains a section on the store-neutral SQS message store, its
JetStream adapter and conformance suite: an internal primitive with no
service wired to it, its shared stream and bucket layout, leader-directed
reads and their full-body cost, feed recovery and its best-effort
consumer cleanup, and the absence of end-to-end or multi-host
validation. The Scan contract says metadata only, not headers only.
2026-10-29 08:43:00 +10:00
Josh Siviour 61686f2530 test: run the LB health suites serially after the parallel internal group on multi-node (#1320) 2026-10-29 07:53:49 +10:00
Josh Siviour d5f4d48956 feat: serve the OCI referrers API for ECR repositories (#1319)
* feat: record manifest subject and artifactType and return OCI-Subject on push

* fix: resolve the manifest media type once so a parameterised Content-Type cannot misclassify an index

* feat: index referrers per subject and serve them through the metadata service

* feat: serve the OCI referrers API for ECR repositories

* test: cover the ECR referrers route in the gateway authorization matrices

* feat: delete a subject image's referrers with it

* fix: satisfy nilerr in the referrers manifest parser

* fix: return the manifest parse error and format the referrers code

* docs: describe the ECR OCI referrers API and cosign support

* docs: name oras cp and crane copy as the way to carry v3 signatures to an ECR mirror

* docs: present oras cp and crane copy as one remedy for copied v3 signatures
2026-10-28 22:52:47 +10:00
Josh Siviour acfa078586 fix: take an EKS control plane out of rotation when its gateway is down (#1318)
* feat: add a gateway readiness endpoint to the EKS token webhook

* feat: serve token webhook readiness beyond loopback and admit it to the control-plane SG

* feat: health-check the EKS apiserver target group on the token webhook readiness port

* style: order the strconv import

* fix: export the readiness listener address to the EKS token webhook

* docs: regenerate the ELB coverage README

* docs: describe EKS control plane readiness and what happens when a gateway stops

* test: range over SplitSeq in the webhook initd guard
2026-10-28 21:12:29 +10:00
Josh Siviour 17dd3f325b fix: stop routing to unhealthy load balancer targets and apply health-check changes (#1317)
* feat: add routableTargets helper that skips unhealthy targets and fails open

* fix: drop unhealthy targets from the NLB nginx upstream, failing open when none are healthy

* fix: re-render load balancer config when a target health state changes

* fix: write load balancer config and heartbeat fields under CAS so a re-render is never rolled back

* fix: write live load balancer modifications under CAS so they cannot roll back a re-rendered config

* fix: let ALB backends fail open to unchecked backup servers when every target is down

* test: cover NLB unhealthy-target routing and ALB fail open in the lb e2e suite

* test: flip ALB backend health per instance and log how long each health phase takes

* style: gofmt haproxy backend struct and the attributes fetch table

* fix: re-render load balancer config when a target group's health check changes

* fix: return the health-target marshal error instead of discarding it

* test: mark the health-check target group create as an intentional create-path test

* docs: describe target health, fail open and time to effect in the ELBv2 guide
2026-10-28 19:54:49 +10:00
Josh Siviour d2b02414b5 feat: direct-boot microvm data volumes and refused hot-attach (#1316)
* feat: render a data volume as an MMIO virtio-blk-device

A package-private helper reuses the PCI volume device's id, drive,
iothread, serial and stop-on-error options, without a PCI bus.

* feat: cold-boot microvm data volumes on MMIO without a PCI hot-plug port

microvm has no PCIe root ports, so its data volumes render as
virtio-blk-device and take no hot-plug port; a recorded port is kept.
NBD and host-block exports and q35 PCI allocation are unchanged.

* feat: compose a launch-only microvm config with EBS data volumes

directBootLaunchConfig copies a clean direct-boot base and appends the
mounted data volumes after its devices, rejecting boot and EFI volumes
and non-microvm bases. Neither input is modified, so repeated launches
never accumulate entries or keep a stale endpoint.

* feat: launch direct-boot microvms with mounted data volumes from a launch-only config

startQEMU composes the direct-boot command from the saved base and one
snapshot of the mounted requests, which also feeds the readiness probes,
so the launched endpoint is the one checked and is never saved.

* fix: refuse volume hot-attach on direct-boot microvms before any mount

A microvm's MMIO bus has no PCIe root port to hot-plug into, so
AttachVolume returns ErrHotAttachUnsupported before mounting the volume
or changing instance or volume state.

* fix: return UnsupportedOperation for microvm volume hot-attach

AttachVolume's ErrHotAttachUnsupported is a caller-fixable refusal, so
the attach/detach error boundary maps it to a 400 UnsupportedOperation
instead of a generic server error.

* docs: describe direct-boot microvm data volumes and refused hot-attach
2026-10-28 19:45:47 +10:00
Julian Sommer af3fe610f9 refactor: enable eslint no-use-before-define in spinifexui
Enables the rule with functions: false, since hoisted declarations are safe, and defines CONDITION_FIELD_TEXT and the handoff message listener before their first use.
2026-10-28 15:30:07 +10:00
Julian Sommer c8092975c2 refactor: enable eslint complexity in spinifexui and split the components that exceeded it (#1315)
Turns on oxlint's eslint/complexity rule (max 20) and brings the 18 offending functions under the limit by extracting helpers, hooks and subcomponents with no behaviour change.
Shares the RDS subnet and parameter group option builders through rds-form-fields, and rebuilds dist.
2026-10-28 16:18:57 +11:00
Josh Siviour 49a4a07e00 fix: trust every gateway's ECR registry host on EKS workers (#1314)
* refactor: share the ECR registry address precedence in domains/ecr

* feat: expose every gateway's ECR registry host to the EKS service

* fix: skip wildcard peer hosts when listing ECR registry hosts

* feat: trust every gateway's ECR registry host on EKS workers

* docs: describe how EKS workers trust every gateway's ECR registry host

* docs: split the ECR gateway limits to one sentence per line

* refactor: format the registry URI host in domains/ecr so the daemon avoids the awsapi adapter

* docs: state that a refused gateway fails over at once and record the 3-node ECR measurements
2026-10-28 14:28:12 +10:00
Julian Sommer 02fd49be7e fix: block DeleteSubnet when an ENI record in the account cannot be decoded (#1313) 2026-10-28 14:14:25 +11:00
Josh Siviour 2aac488fa7 fix: stop TCP stalling through the VPC gateway by setting gateway MTU and clamping MSS on the host hop (#1312)
* fix: add GatewayMTU capping the gateway MTU at the internet path MTU

* fix: set gateway_mtu on the VPC gateway router port so oversized packets get ICMP frag-needed

* fix: wire the gateway MTU into vpcd and cover reconcile convergence

* feat: add host MSS clamp tables for the veth and routed uplinks

* fix: shorten MSS clamp comments and log the periodic ensure at debug

* feat: run the host MSS clamp loop in vpcd

* fix: run nft under the ambient CAP_NET_ADMIN instead of sudo

* docs: describe the gateway MTU and the host MSS clamp for the internet path
2026-10-28 12:01:50 +10:00
Julian Sommer 11ef3b63b8 chore: parallelise silent-node tests in operator/accountlifecycle (#1311)
Each silent node costs the full fan-out deadline, which the test cannot
shorten, so let those tests and their subtests overlap.
2026-10-28 11:32:20 +11:00
Julian Sommer d5fbb7b628 chore: cut wall time of dns, ochre, gateway and operator/cli test packages (#1310)
Bound the zone reload wait with a test context, generate 2048-bit keys in
cert renew tests via the existing seam, replace 100ms negative waits with
deterministic checks, and run tests bound by production timeouts in parallel.
2026-10-28 11:10:05 +11:00
Julian Sommer d953d0a973 fix: treat Ubuntu's openvswitch.service alias as the Debian OVS unit
Ubuntu 26.04's openvswitch-switch declares Alias=openvswitch.service, so
enabling it leaves an /etc symlink that distro_ovs_unit read as a second,
EL-style umbrella unit. setup-ovn.sh then refused to pick one and every
ubuntu-26.04 bootstrap failed. A symlink back to openvswitch-switch.service
now counts as the Debian unit; a real or masked openvswitch.service still
does not.
2026-10-28 09:46:45 +10:00
Julian Sommer 43e84874df fix: validate GetResources page size and token as AWS does (#1309)
ResourcesPerPage outside 1-100 and a malformed PaginationToken now return
InvalidParameterException. An unbounded page size let start+size overflow
negative and panic on the slice.
2026-10-28 10:17:18 +11:00
Julian Sommer 3f7664ddf9 feat: send Route 53 through the request conformance suite (#1308) 2026-10-28 10:08:28 +11:00
Julian Sommer 7528c063cb feat: make TestMultinodeIPSec fail unless cross-node guest traffic leaves as ESP (#1307)
The test checked IPsec config and that AES-GCM SAs exist, and only warned when
a 5s idle capture saw no ESP, so a cluster sending plaintext Geneve passed.

It now launches an instance pair on distinct nodes, pings 200 x 1000 bytes
between them, and for each direction fails unless the outbound SA counters to
the peer's ovn-encap-ip rose by the payload, an outbound-only capture on the
interface routing to the peer saw ESP or UDP/4500 ESP, and saw no UDP/6081.

The distinct-node launch loop moves from the baseline cross-host test into a
shared helper. The test is no longer parallel since it now launches instances.
2026-10-28 10:00:47 +11:00
Julian Sommer 772e1b4c37 fix: report a missing EKS node group with the AWS node-group message instead of the cluster message (#1306) 2026-10-28 09:19:58 +11:00
Julian Sommer 05256f1cfc fix: let DeleteSnapshot name the unreadable record or provider failure instead of a bare ServerInternal (#1305) 2026-10-28 09:16:37 +11:00
Julian Sommer 920f0487a3 feat: include the Route 53 API in the AWS operation coverage report
Vendor the route-53 2013-04-01 Smithy model at the pinned commit, add the
Route53 service, and pass route53awsapi.OperationInventory() to the
generator. coverage.json and docs/coverage/route53 now list the 13 served
operations; the Spinifex activation extensions stay out of the model count.

The generator now fails when a served service has no embedded model, so
the next one cannot drop out of the report silently. Request encoding
skips rest-xml by protocol, which covers Route 53 as well as S3.
2026-10-28 08:12:55 +10:00
Julian Sommer 919d083bbf chore: name the Route 53 inventory in the aws-model-coverage layering exception
cmd/aws-model-coverage reads route53awsapi.OperationInventory() as it does
ECR's and ACM's, under the same recorded ADR-0001 rule 5 exception.
2026-10-28 08:12:55 +10:00
Julian Sommer 816b3168b7 docs: use --node-role in the EKS guide's create-nodegroup example (#1304) 2026-10-28 09:10:21 +11:00
Julian Sommer a33fbcc696 fix: serve the node's configured region in IMDS instead of deriving it from the zone name (#1303) 2026-10-28 09:09:23 +11:00
Ben Duncan 2989a55309 fix: purge a clustered OVN database under --recreate-db (#1301)
ensure_clustered_db_storage() collected only databases in standalone
format, so --recreate-db returned early when the NB/SB databases were
already clustered and removed nothing. ovn-ctl then served the stale
database and silently ignored the RAFT flags, which is the failure the
function exists to prevent, reached from the other direction. Because
the old database carries the previous member list and server ID, the
node rejoined the earlier cluster instead of replacing it: re-forming a
production cluster brought the create node back as a follower, and
ovn-nbctl over the leader-only unix socket failed with "database
connection failed ()".

install-node.sh passes --recreate-db on every database node on every
run, so formation already intended a purge. Remove whatever NB/SB
state is present, standalone or clustered, and keep refusing a
standalone database when the flag is absent.

Two further gaps in the same path. systemctl stop returns once the unit
is inactive rather than once the processes are gone, so the purge now
waits for the database servers and escalates to SIGKILL, failing the
run if one survives rather than unlinking files under it. The pattern
is anchored on the OVN database directory so it can never reap the OVS
conf.db server. The lock files, the compute-node move-asides and an
earlier run's standalone backup are cleared alongside the databases.
2026-10-27 14:36:31 -05:00
Josh Siviour f12a0d8e98 docs: qualify legacy lifecycle ADR citations (#1300)
* docs: qualify legacy lifecycle ADR citations

* docs: qualify ADR-0003:S2 citations with their historic mechanism

* docs: reword possessive legacy citations
2026-10-27 17:22:34 +10:00
Julian Sommer 7dde151766 feat: Oracle Linux 10 single-node host support (#1270) 2026-10-27 16:13:42 +11:00
Josh Siviour e7a3be3b4e feat: add a dedicated SQS message store JetStream server (#1299)
* feat: add a dedicated SQS message store JetStream server separate from control-plane NATS

* test: cover the SQS store server start, validation and memory limit
2026-10-27 15:05:27 +10:00
Julian Sommer f92d73d6af fix: send ALB health checks as HTTP/1.1 with Host and ELB-HealthChecker User-Agent (#1296)
HAProxy's short 'option httpchk GET <path>' form sends HTTP/1.0 with no Host header, which Envoy rejects with 426 (and 400 for HTTP/1.1 without Host), so Envoy-fronted targets never became healthy. Match the AWS request shape: GET <path> HTTP/1.1, Host set to the target IP (with the port only when it is not the scheme default), and User-Agent ELB-HealthChecker/2.0.
2026-10-27 15:41:44 +11:00
Julian Sommer c02cfb806b fix: default an omitted container essential flag to true in RegisterTaskDefinition (#1295) 2026-10-27 15:41:33 +11:00
Julian Sommer e1916c99f8 fix: make MemoryObjectStore report NoSuchKey on delete like Predastore (#1297)
MemoryObjectStore.DeleteObject silently succeeded for an absent key while
Predastore answers NoSuchKey, so handlers that treated a missing key as fatal
passed unit tests and failed in production. The fake now returns
NoSuchKeyError, and delete paths that must be idempotent (DeleteSnapshot's
sweep and document delete, DeleteVolume, DeregisterImage, the admin image
removal sweep, and ECR upload and GC cleanup) treat it as already deleted.
2026-10-27 15:40:03 +11:00
Julian Sommer 8b589c99c7 fix: reject CreateTags resource ids with an unrecognised prefix as InvalidID (#1298) 2026-10-27 15:39:51 +11:00
Josh Siviour 5216b18f88 feat: render AWS errors in each registration's declared protocol (REST-JSON, JSON 1.0 query-compatible) (#1294)
* feat: let a registration select REST-JSON or JSON 1.0 query-compatible error envelopes

* feat: add a per-registration internal error code and decode envelopes through the AWS SDK v2

* feat: let a JSON 1.0 registration namespace the __type of its errors

* feat: match captured SQS error bytes on the JSON 1.0 envelope
2026-10-27 13:11:43 +10:00
Julian Sommer 7aaf6bf5e7 release: update go.mod for v1.23.0 2026-10-27 12:34:01 +10:00
Ben Duncan 6f9875abb9 feat: Route 53 public hosted zones, activation by delegation, and a DNS console (#1274) 2026-10-27 13:24:57 +11:00
Josh Siviour 3503215bae refactor: move gateway request audit into ingress/aws/audit (#1285)
* refactor: move gateway request audit into ingress/aws/audit

First generic-ingress extraction from the root gateway under ADR-0001.
requestaudit.go and slogRequestLogger move unchanged in behaviour to
package audit. The gateway's auth flow and dispatch populate the record
through the exported From and Record setters at the same call sites.
Record.Fields is exported for the SigV4-path tests that stay in gateway.
The migration-record deferral bullet is replaced by a move row.

* docs: cite #1285 in the request-audit move row
2026-10-27 10:49:02 +10:00
Julian Sommer e5509443b3 fix: return IpAddressType and LoadBalancerArns on ELBv2 target groups (#1284) 2026-10-27 11:03:05 +11:00
Julian Sommer ee4f571687 fix: validate every ELBv2 AddTags/RemoveTags ARN before tagging any resource (#1282) 2026-10-27 11:00:47 +11:00
Julian Sommer 49aea1b365 fix: commit EC2 KV tag-mirror writes at the read revision so concurrent record changes survive (#1281)
UpdateKVRecordTags now goes through kvutil.Update, which commits at the
revision it read and retries on conflict, instead of a plain put that
reverted any tag or non-tag write landing between its read and write.

Volume and AMI tag mirrors write ebsmetadata documents to the object
store, which has no conditional put; characterisation tests pin that
they still lose interleaved writes.
2026-10-27 11:00:15 +11:00
Julian Sommer d9f19be941 fix: default IAM GetUser without UserName to the calling principal (#1283)
aws iam get-user sends no UserName and got MissingParameter. The gateway now
applies the AWS default before authorization: an IAM user is treated as naming
themselves, root gets the account root shape, and a role session gets
ValidationError.

Refs mulgadc/spinifex#1275
2026-10-27 10:59:58 +11:00
Josh Siviour 598e3aec8d refactor: move accountteardown to operator/accountlifecycle (#1277)
* refactor: move accountteardown to operator/accountlifecycle

Mechanical relocation under the amended ADR-0001 accountteardown row.
Package renamed to accountlifecycle; importers keep the accountteardown
alias. No API, NATS subject, KV access or behaviour change. Replacing
direct domain/adapter cleanup calls with owner capabilities stays
recorded as deferred debt.

* docs: cite #1277 in the accountteardown move row
2026-10-27 09:55:10 +10:00
Julian Sommer 7957186998 fix: stop IAM role updates from reattaching a concurrently detached policy (#1280) 2026-10-27 10:54:50 +11:00
Julian Sommer 3a778d579e fix: serve current SSH key material from IMDS after a key pair is re-imported under the same name (#1279) 2026-10-27 10:50:37 +11:00
Julian Sommer fffa2e4a47 fix: serve IMDS ENI facts from the current ENI record so public-ipv4 follows an EIP association (#1278) 2026-10-27 10:45:27 +11:00
Julian Sommer 31b5356109 fix: move the Alpine catalog entries to 3.24.3 for CVE-2026-99999 (#1276)
3.24.1 ships linux-virt 6.18.35; 3.24.3 ships 6.18.58-r0. From 3.24.2 the
cloud image names drop the generic_ prefix and the uefi part.
2026-10-27 10:29:05 +11:00
Josh Siviour 6eac30217f refactor: relocate gateway/{ecs,elbv2} to domains/{ecs,elbv2}/awsapi (#1273)
* refactor: relocate gateway/{ecs,elbv2} to domains/{ecs,elbv2}/awsapi

Mechanical ADR-0001 relocation of the ECS and ELBv2 AWS action adapters
to their domain homes, as #1265 did for IAM, STS, EKS and RDS. APIs, NATS
subjects, KV buckets and behaviour are unchanged. The root gateway
dispatch still imports both packages, recorded as gateway (service
registration) debt; the quota live ELBv2 counter import moves with it.

* docs: cite spinifex#1273 in the migration record
2026-10-27 09:18:16 +10:00
Julian Sommer 2511cc4cc3 chore: bump Node.js to 24.22.1 and point docs at .nvmrc 2026-10-27 08:52:42 +10:00
Josh Siviour a6b6bb73a9 fix: restore ADR-0007 invariant scan coverage for relocated domains (#1272)
The IAM, STS, RDS, EKS and ELBv2 relocations moved their packages out of
spinifex/handlers, which the S4, S6 and S7 network invariant scans still
walked, so those packages silently dropped out of the scans. Re-add their
new homes alongside domains/ecs. The checked rules are unchanged.
2026-10-26 16:35:15 +10:00
Josh Siviour 744a73ef1e refactor: relocate handlers/ecs to domains/ecs (#1271)
* refactor: relocate handlers/ecs to domains/ecs

Mechanical move per the ADR-0001 handlers/ecs row (files already
organised by resource). Package renamed handlers_ecs -> ecs; importers
alias ecsdomain. APIs, NATS subjects, storage and behaviour unchanged.
handlers is now a retired legacy root; ECS allowlist debt retargeted to
domains/ecs and R5 holds it to the interim daemon-import ban.

* docs: cite spinifex#1271 in the ECS relocation row

* fix: point network invariant scans at domains/ecs

The ADR-0007 S4, S6 and S7 scans walked spinifex/handlers, which no
longer exists after the ECS move. Scan domains/ecs, the same code.
2026-10-26 16:09:59 +10:00
Josh Siviour 2df87981f2 refactor: relocate handlers/elbv2 to domains/elbv2 (#1269)
* refactor: relocate handlers/elbv2 to domains/elbv2

Mechanical move per the amended ADR-0001 handlers/elbv2 row. Package
renamed handlers_elbv2 -> elbv2; importers alias elbv2domain. APIs,
NATS subjects, storage and behaviour unchanged. Layering debt entries
retargeted; Q-30 data-plane boundary stays deferred debt.

* docs: cite spinifex#1269 in the ELBv2 relocation row

* docs: match citation style in the ELBv2 relocation row
2026-10-26 15:23:17 +10:00
Josh Siviour 7fc79d95bd refactor: relocate lbagent to agents/elbv2 (#1268)
Mechanical move permitted by the ADR-0001 lbagent amendment. The package
becomes elbv2 under agents/; importers alias it lbagent. APIs, on-guest
paths, NATS subjects and behaviour are unchanged. Q-30's data-plane
boundary is recorded as deferred debt in the migration record.

Also brings docs/DESIGN.md up to date with the current code: gateway
dispatch and registrations, EC2 handler and NATS request signatures,
RunInstances placement, daemon subscription form, volume mount subject,
DescribeInstances scatter-gather, and the package layout.
2026-10-26 14:34:13 +10:00
Josh Siviour 0c440a6717 refactor: relocate daemon to runtime/daemon (#1267)
Mechanical whole-package move under the amended ADR-0001 daemon row. Package name, APIs, NATS subjects, storage and behaviour are unchanged; only import paths and path comments outside the package change.

The interim daemon -> gateway -> handlers layering guardrail, the ADR-0007 network invariant scans and the instance-state ratchet locate the daemon by directory, so they now match runtime/daemon. Q-28's ownership and deployment refactor is recorded as deferred debt.
2026-10-26 13:45:35 +10:00
Josh Siviour 00e38f9e76 refactor: relocate gateway/bedrock into domains/ochre (#1266)
* refactor: relocate gateway/bedrock into domains/ochre

Mechanical ADR-0001 whole-package relocation. The Ochre domain core,
stores, inference engine and Bedrock action layer join package ochre
with their package-private code and tests intact; no identifier, API,
NATS subject, KV bucket, metric, error string or behaviour changes.
Separating the action adapters into domains/ochre/awsapi is deferred
and recorded as debt.

* docs: cite spinifex#1266 in the migration record
2026-10-26 12:57:16 +10:00
Josh Siviour 57ea08be2a refactor: move IAM, STS, EKS and RDS AWS adapters to domains/<svc>/awsapi (#1265)
* refactor: move IAM, STS, EKS and RDS AWS adapters to domains/<svc>/awsapi

Mechanical ADR-0001 relocation of gateway/{iam,sts,eks,rds} to
domains/{iam,sts,eks,rds}/awsapi. APIs, NATS subjects, storage and
behaviour are unchanged. The root gateway dispatch still imports each
package and is recorded as service-registration debt. gateway/bedrock
stays: ADR-0001 moves only its action adapters, and the package also
holds the Ochre domain core, stores and inference engine.

* docs: cite spinifex#1265 in the migration record
2026-10-26 11:58:10 +10:00
Josh Siviour 10fc0af2c0 refactor(eks): move handlers/eks to domains/eks/cluster (#1264)
* refactor(eks): move handlers/eks to domains/eks/cluster

Mechanical ADR-0001 relocation of the remaining EKS implementation,
nodegroups included, alongside the access and addon packages. APIs, NATS
subjects, KV buckets and behaviour are unchanged; only the package name
changes from handlers_eks to cluster (imported as ekscluster). The
frozen-legacy entry is removed, allowlist entries now name
domains/eks/cluster, and the stale operator/cli entry is removed.
Nodegroup service extraction is recorded as deferred.

* docs: name the EKS cluster relocation PR in the migration record
2026-10-26 11:08:37 +10:00
Bryn Mailer 524c20db8b feat(ebsprovider): typed export descriptor (nbd | block) with host_device attach (#1243)
* feat(vm): attach host block device exports via raw over host_device

EBSRequest gains optional BlockDevice and BlockSizeBytes. A single
classifier decides whether a request is an NBD export, a block export or
invalid, and buildDrives, AttachVolume, launch readiness, the QMP greeting
deadline and restore all use it. A block data volume is attached as a
sized raw node over a host_device file node with the same node name,
iothread and virtio-blk device as an NBD volume; boot and EFI volumes
reject block exports. NBD output is unchanged.

* feat(ebsprovider): replace PublishedVolume.NBDURI with a typed export descriptor

PublishedVolume now carries an Export of kind nbd or block, checked by VolumeExport.Validate.
The NATS client rejects an invalid export in a PublishVolume reply, and the server refuses to relay one, reporting it as internal.
SchemaVersion moves to 2.

viperblockd, qemunbdd and the memory provider publish nbd exports exactly as before.
Image import refuses a non-nbd export and still unpublishes.
Conformance requires a valid export, and the NBD-dialling suites fail explicitly on a non-nbd export.

* fix(ebsprovider): preserve v1 export compatibility

---------

Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
2026-10-26 10:57:35 +10:00
Julian Sommer 2ed2a1c9dc fix: re-install br-imds flows lost on an ovs-vswitchd restart (#1237) 2026-10-26 11:45:28 +11:00
Julian Sommer ac403f57bc fix: refuse a ModifyInstanceAttribute retype when a node is silent (#1262) 2026-10-26 10:43:36 +11:00
Josh Siviour 6a41d269b9 refactor(rds): move handlers/rds to domains/rds/instance (#1251)
* refactor(rds): move handlers/rds to domains/rds/instance

Mechanical ADR-0001 relocation of the remaining RDS instance resource to
its resource package, alongside engine, parametergroup and subnetgroup.
APIs, NATS subjects, KV buckets and behaviour are unchanged; only the
package name changes from handlers_rds to instance (imported as
rdsinstance). The frozen-legacy entry is removed and the allowlist
entries now name domains/rds/instance; the recorded debt remains.

* docs: name the RDS instance relocation PR in the migration record
2026-10-26 09:39:33 +10:00
Julian Sommer 1bb470517c fix: keep the services.internal SANs and mgmt-bridge IP when spx admin cert renew reissues the server cert (#1263) 2026-10-26 10:38:18 +11:00
Julian Sommer c545aa6220 feat: add spx admin init --external-dns to set the external pool's dns_servers (#1261) 2026-10-26 10:37:50 +11:00
Julian Sommer e006837e1c test: prove a failed ochre vector account grant rolls back its schema (#1260) 2026-10-26 10:37:17 +11:00
Julian Sommer f9dbe9526d fix: commit launch template version deletes through the header revision so a concurrent default change cannot name a deleted version (#1257)
* fix: commit launch template version deletes through the header revision so a concurrent default change cannot name a deleted version

* fix: keep version-delete rollbacks from leaving headerless bodies or a false default-version refusal
2026-10-26 10:29:12 +11:00
Julian Sommer 738f1e6d4e fix: retry the OCI allocator's startup reconcile until the bindings KV answers (#1258) 2026-10-26 10:28:25 +11:00
Julian Sommer 42ca999e0b test: let CI tune the multinode DNS convergence budgets (#1259) 2026-10-26 10:27:42 +11:00
Julian Sommer f35853c1d3 fix: return the template NotFound code when ModifyLaunchTemplate races a delete (#1256) 2026-10-26 10:23:50 +11:00
Julian Sommer b586b7a1f9 feat: add spx admin storage status to read predastore meta node raft state from the CLI (#1255) 2026-10-26 10:22:21 +11:00
Julian Sommer 24e2572ebd fix: resolve a GetSessionToken session as its IAM user in presigned GetCallerIdentity, and reject an identity with no ARN (#1254)
* fix: resolve a GetSessionToken session as its IAM user in presigned GetCallerIdentity, and reject an identity with no ARN

* fix: report the account ID as root's UserId for a root session in presigned GetCallerIdentity
2026-10-26 10:21:03 +11:00
Julian Sommer 0ac0a106c0 fix: let DeleteUser succeed when the user record lists an access key ID whose key entry is gone (#1252) 2026-10-26 10:15:58 +11:00
Julian Sommer b2b7822f6a test: wait for the mount reply in TestIntegration_EBSMountClearsStaleSealReceipt (#1253) 2026-10-26 10:09:41 +11:00
Josh Siviour fbf76cbedc refactor(sts): move handlers/sts to domains/sts (#1250)
* refactor(sts): move handlers/sts to domains/sts

Mechanical ADR-0001 relocation of the STS implementation. APIs, NATS
subjects, KV buckets and behaviour are unchanged; only the package name
changes from handlers_sts to sts (imported as stsdomain). The frozen-legacy
entry is removed and the allowlist entries now name domains/sts; the
per-consumer IAM capability debt remains.

* docs: name the STS relocation PR in the migration record
2026-10-26 08:28:10 +10:00
Julian Sommer b557386c4b chore: rebuild frontend 2026-10-26 08:24:17 +10:00
dependabot[bot] e04a551185 chore: bump the npm-dependencies group (#1248)
Bumps the npm-dependencies group in /spinifex/runtime/roles/spinifexui/frontend with 28 updates:

| Package | From | To |
| --- | --- | --- |
| [@aws-sdk/client-acm](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-acm) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-bedrock](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-bedrock-agent](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-agent) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-bedrock-agent-runtime](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-agent-runtime) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-bedrock-runtime](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-runtime) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-ec2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ec2) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-ecs](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecs) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-eks](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-eks) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-elastic-load-balancing-v2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-elastic-load-balancing-v2) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-iam](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-iam) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-rds](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-rds) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/client-sts](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-sts) | `3.1152.0` | `3.1155.0` |
| [@aws-sdk/lib-storage](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/lib/lib-storage) | `3.1152.0` | `3.1155.0` |
| [@hookform/resolvers](https://github.com/react-hook-form/resolvers) | `5.9.1` | `6.0.0` |
| [@tanstack/react-query](https://github.com/TanStack/query/tree/HEAD/packages/react-query) | `5.104.3` | `5.104.5` |
| [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.44` | `1.170.45` |
| [cn](https://github.com/shadcn-ui/cn/tree/HEAD/packages/cn) | `0.4.0` | `0.5.0` |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.57.0` | `1.61.0` |
| [@shadcn/lint](https://github.com/shadcn-ui/lint/tree/HEAD/packages/lint) | `0.2.1` | `0.2.3` |
| [@tanstack/eslint-plugin-query](https://github.com/TanStack/query/tree/HEAD/packages/eslint-plugin-query) | `5.104.3` | `5.104.5` |
| [@tanstack/router-plugin](https://github.com/TanStack/router/tree/HEAD/packages/router-plugin) | `1.168.46` | `1.168.47` |
| [happy-dom](https://github.com/capricorn86/happy-dom) | `21.0.1` | `21.0.4` |
| [oxc-transform-react](https://github.com/oxc-project/oxc/tree/HEAD/napi/transform-react) | `0.154.0` | `0.155.0` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) | `0.73.0` | `0.74.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.88.0` | `1.89.0` |
| [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.3.4` | `8.3.5` |


Updates `@aws-sdk/client-acm` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-acm/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-acm)

Updates `@aws-sdk/client-bedrock` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-bedrock)

Updates `@aws-sdk/client-bedrock-agent` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-agent/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-bedrock-agent)

Updates `@aws-sdk/client-bedrock-agent-runtime` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-agent-runtime/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-bedrock-agent-runtime)

Updates `@aws-sdk/client-bedrock-runtime` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-runtime/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-bedrock-runtime)

Updates `@aws-sdk/client-ec2` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ec2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-ec2)

Updates `@aws-sdk/client-ecr` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-ecr)

Updates `@aws-sdk/client-ecs` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecs/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-ecs)

Updates `@aws-sdk/client-eks` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-eks/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-eks)

Updates `@aws-sdk/client-elastic-load-balancing-v2` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-elastic-load-balancing-v2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-elastic-load-balancing-v2)

Updates `@aws-sdk/client-iam` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-iam/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-iam)

Updates `@aws-sdk/client-rds` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-rds/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-rds)

Updates `@aws-sdk/client-s3` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-s3)

Updates `@aws-sdk/client-sts` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/clients/client-sts)

Updates `@aws-sdk/lib-storage` from 3.1152.0 to 3.1155.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1155.0/lib/lib-storage)

Updates `@hookform/resolvers` from 5.9.1 to 6.0.0
- [Release notes](https://github.com/react-hook-form/resolvers/releases)
- [Commits](https://github.com/react-hook-form/resolvers/compare/v5.9.1...v6.0.0)

Updates `@tanstack/react-query` from 5.104.3 to 5.104.5
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/react-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/react-query@5.104.5/packages/react-query)

Updates `@tanstack/react-router` from 1.170.44 to 1.170.45
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.45/packages/react-router)

Updates `cn` from 0.4.0 to 0.5.0
- [Release notes](https://github.com/shadcn-ui/cn/releases)
- [Changelog](https://github.com/shadcn-ui/cn/blob/main/packages/cn/CHANGELOG.md)
- [Commits](https://github.com/shadcn-ui/cn/commits/cn@0.5.0/packages/cn)

Updates `lucide-react` from 1.57.0 to 1.61.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.61.0/packages/lucide-react)

Updates `@shadcn/lint` from 0.2.1 to 0.2.3
- [Release notes](https://github.com/shadcn-ui/lint/releases)
- [Changelog](https://github.com/shadcn-ui/lint/blob/main/packages/lint/CHANGELOG.md)
- [Commits](https://github.com/shadcn-ui/lint/commits/@shadcn/lint@0.2.3/packages/lint)

Updates `@tanstack/eslint-plugin-query` from 5.104.3 to 5.104.5
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/eslint-plugin-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/eslint-plugin-query@5.104.5/packages/eslint-plugin-query)

Updates `@tanstack/router-plugin` from 1.168.46 to 1.168.47
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/router-plugin/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/router-plugin@1.168.47/packages/router-plugin)

Updates `happy-dom` from 21.0.1 to 21.0.4
- [Release notes](https://github.com/capricorn86/happy-dom/releases)
- [Commits](https://github.com/capricorn86/happy-dom/compare/v21.0.1...v21.0.4)

Updates `oxc-transform-react` from 0.154.0 to 0.155.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/napi/transform-react/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/crates_v0.155.0/napi/transform-react)

Updates `oxfmt` from 0.73.0 to 0.74.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.74.0/npm/oxfmt)

Updates `oxlint` from 1.88.0 to 1.89.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.89.0/npm/oxlint)

Updates `vite` from 8.3.4 to 8.3.5
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.3.5/packages/vite)

---
updated-dependencies:
- dependency-name: "@aws-sdk/client-acm"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-agent"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-agent-runtime"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-runtime"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ec2"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ecr"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ecs"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-eks"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-elastic-load-balancing-v2"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-iam"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-rds"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-s3"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-sts"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/lib-storage"
  dependency-version: 3.1155.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@hookform/resolvers"
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/react-query"
  dependency-version: 5.104.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/react-router"
  dependency-version: 1.170.45
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: cn
  dependency-version: 0.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: lucide-react
  dependency-version: 1.61.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@shadcn/lint"
  dependency-version: 0.2.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/eslint-plugin-query"
  dependency-version: 5.104.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/router-plugin"
  dependency-version: 1.168.47
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: happy-dom
  dependency-version: 21.0.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: oxc-transform-react
  dependency-version: 0.155.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxfmt
  dependency-version: 0.74.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxlint
  dependency-version: 1.89.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: vite
  dependency-version: 8.3.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
2026-10-26 09:23:33 +11:00
dependabot[bot] 0e81918345 chore: bump the go-dependencies group with 3 updates (#1247)
Bumps the go-dependencies group with 3 updates: [github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream](https://github.com/aws/aws-sdk-go-v2), [github.com/oracle/oci-go-sdk/v65](https://github.com/oracle/oci-go-sdk) and [github.com/ovn-kubernetes/libovsdb](https://github.com/ovn-kubernetes/libovsdb).


Updates `github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream` from 1.7.22 to 1.7.23
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/notifications/v1.7.22...aws/protocol/eventstream/v1.7.23)

Updates `github.com/oracle/oci-go-sdk/v65` from 65.127.0 to 65.128.0
- [Release notes](https://github.com/oracle/oci-go-sdk/releases)
- [Changelog](https://github.com/oracle/oci-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/oracle/oci-go-sdk/compare/v65.127.0...v65.128.0)

Updates `github.com/ovn-kubernetes/libovsdb` from 0.8.1 to 0.9.0
- [Release notes](https://github.com/ovn-kubernetes/libovsdb/releases)
- [Commits](https://github.com/ovn-kubernetes/libovsdb/compare/v0.8.1...v0.9.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream
  dependency-version: 1.7.23
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/oracle/oci-go-sdk/v65
  dependency-version: 65.128.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/ovn-kubernetes/libovsdb
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:03:35 +11:00
dependabot[bot] 8de934767d chore: bump the docker-images group across 3 directories with 3 updates (#1246)
Bumps the docker-images group with 1 update in the /build directory: golang.
Bumps the docker-images group with 1 update in the /docs/terraform-workbooks/demo-app directory: distroless/static.
Bumps the docker-images group with 1 update in the /scripts directory: debian.


Updates `golang` from `5c3e9b4` to `972b411`

Updates `distroless/static` from `e2e927e` to `18f4cc2`

Updates `debian` from `a29215f` to `62bd423`

---
updated-dependencies:
- dependency-name: golang
  dependency-version: 1.27.3-bookworm
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: docker-images
- dependency-name: distroless/static
  dependency-version: nonroot
  dependency-type: direct:production
  dependency-group: docker-images
- dependency-name: debian
  dependency-version: trixie-slim
  dependency-type: direct:production
  dependency-group: docker-images
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:03:22 +11:00
Adam Szynkowski 8b3b23deff feat(oci): support resumable OCI topology validation with template (#1231) 2026-10-26 09:01:45 +11:00
Josh Siviour 8a4ab4438c refactor(iam): move handlers/iam to domains/iam (#1249)
* refactor(iam): move handlers/iam to domains/iam

Relocate IAM's implementation and its test double to the ADR-0001 home
with git mv. The package is renamed from handlers_iam to iam and its
importers use the iamdomain alias. APIs, NATS subjects, KV buckets and
behaviour are unchanged.

The cross-domain allowlist entries now target domains/iam; the two
frozen-legacy entries and the operator/cli entry, which no longer breaks
the legacy-implementation rule, are removed. The migration record states
that per-consumer IAM capabilities remain debt but were not a
prerequisite to the move.

* docs: record the IAM move against its PR
2026-10-25 21:58:05 +10:00
Josh Siviour 6daa340044 feat(quota): seal per-account quota overrides behind a revision-guarded owner action (#1245)
Store the override record with a state field kept separate from the admin
wire type. Every set and clear is revision-guarded and refuses a sealed
record with Blocked, and every read of a sealed record, including
admission, returns Blocked. The read-failure fallback to configured limits
is unchanged.

Add Service.SealAccountOverrides, accepted only when IAM already holds the
account as TERMINATING. It creates the override bucket if needed after that
check, creates or updates the record as sealed, and succeeds only on a
leader-accepted write, returning its revision. awsgw serves it on
admission.v1.SealAccountQuotaOverride. Nothing calls it yet.

Replace the unwired delete-based CleanupAccount.
2026-10-23 18:22:54 +10:00
Josh Siviour b4d4fa65f8 fix(awserrors): attach Retry-After without duplicating the error text (#1244)
Joining a coded error with RetryAfter rendered the bare code on a second
line, so logged and stored reasons read "...: ServiceUnavailable
ServiceUnavailable". Add WithRetryAfter, which wraps the error and keeps
its text while ResolveRetryAfter and ResolveErrorDetail still resolve the
hint and the code. Use it for the seal-capability refusal and the
incomplete DescribeInstances and DescribeInstanceTypes fan-out errors.
2026-10-23 16:08:26 +10:00
Josh Siviour 2e2465dd25 feat(accountteardown): seal EC2 account settings during termination (#1242)
Wire the EC2 account-settings seal into account termination as a
StageAccount reaper. Immediately before every seal attempt the reaper
probes each configured node's new read-only, per-node seal-capability
subject and requires exactly one identified reply per node at capability
version 1 or later. Missing, duplicate, wrong-identity, malformed, error,
older-version or unreachable responders send no seal and return a
retryable ServiceUnavailable, so the account stays TERMINATING until the
cluster is compatible.

The reaper completes only on a sealed reply for the account with a
non-zero revision; a fresh termination run seals again. Daemons validate
their node ID before subscribing, so an invalid ID serves no capability
and reads as incompatible.
2026-10-23 15:20:33 +10:00
Josh Siviour 5d2c3e749f feat(ec2): seal account settings with revision-guarded writes (#1241)
Replace the unwired delete-based CleanupAccount with an owner-local sealed
state in the existing account-settings KV value. All settings mutations now
use revision-guarded create/update; a sealed record rejects every read and
mutation with Blocked and cannot be recreated. Records without a state
field remain active.

SealAccount CAS-writes the sealed value whether the key is present, absent
or tombstoned, and succeeds only on a leader-accepted write, returning its
revision. It first checks, through a read-only IAM account-status reader,
that the account is already TERMINATING; otherwise it refuses with
IncorrectState and writes nothing.

The daemon serves the internal subject ec2.v1.SealAccountSettings. Nothing
calls it yet.
2026-10-23 13:34:53 +10:00
Josh Siviour fb0f8e9291 feat(quota): add unwired owner-side quota override cleanup (#1240)
Pin that account teardown removes the usage counter but leaves the
per-account quota override, so a reused account ID inherits raised or
Unlimited limits. Add an unwired, idempotent CleanupAccount on the quota
service: a present override key is removed by a revision-guarded delete
and reported deleted with its revision; a missing key is reported only as
observed-absent-at-scan.
2026-10-23 11:56:08 +10:00
Josh Siviour bdfa01459c feat: add unwired EC2 account-settings owner cleanup (#1239)
Characterise that EC2 account settings survive today's account teardown,
and add an owner-side CleanupAccount in domains/ec2/account that deletes
one account's settings key.

A present key is removed with a revision-guarded delete, which the stream
leader orders regardless of which replica served the preceding read, and
is reported deleted with that revision. A key that a direct get does not
find is reported only as observed absent at scan time, because KV direct
gets can be answered by a lagging follower. Store errors and exhausted
revision conflicts are failed, never absent. The capability refuses empty,
malformed and global account IDs and never falls back to the global key.

Nothing calls it yet: no NATS subject, no accountteardown wiring.
2026-10-23 11:03:38 +10:00
Julian Sommer f707df92e7 chore: go 1.27.3 2026-10-21 10:57:02 +10:00
Julian Sommer f1ebb74acc fix: flush daemon subscriptions before platform inventory test requests
The inventory and disclosure test helpers subscribed on one connection and
requested on another, so a request could reach the server before its SUBs
and fail with no responders under load.
2026-10-21 10:49:16 +10:00
Julian Sommer bde54ffd2f fix: report a max_age stream whose replicas turned over between digest copies as skipped, not DIVERGED, in spx admin kv compare (#1238) 2026-10-21 11:46:22 +11:00
Julian Sommer d01e9b400d test: assert a TTL kvstore bucket is created at the cluster's replica count on a three-node cluster (#1236) 2026-10-21 10:10:00 +11:00
Julian Sommer 77b46481f1 fix: report InsufficientInstanceCapacity instead of a raw NATS no-responders error when RunInstances finds no launch subscriber on a node (#1235) 2026-10-21 09:56:49 +11:00
Julian Sommer a16b667cbb fix: log the local node as node, not holder, in reconcile/lock and kvlease lease lines (#1234)
A node that lost the election logged holder=<itself>, which reads as though it holds the lock. Lease lines now carry the local node as node, and a refused claim logs the node that owns the key, read from the Get the claim already makes, as holder at debug level.
2026-10-21 09:31:57 +11:00
Julian Sommer 7f634496b6 fix: count only the first shutdown ACK from each node so a duplicate cannot stand in for a silent node (#1233) 2026-10-21 09:30:36 +11:00
Julian Sommer 3a849c4c18 fix: name the blocking volume in DeleteSnapshot's InUse error instead of blaming AMIs (#1232) 2026-10-21 09:20:37 +11:00
Julian Sommer 2f609efa14 fix: hide platform-managed instances from DescribeInstanceStatus (#1230) 2026-10-20 13:03:18 +11:00
Julian Sommer eccbf48273 test: stub sudo so the uninstall multi-node refusal check runs on any host 2026-10-20 10:21:23 +10:00
Julian Sommer 84df82c26f fix: accept the snapshot-id filter on DescribeVolumes and match it against the volume's source snapshot (#1229) 2026-10-20 11:21:02 +11:00
Julian Sommer 29770dc798 test: stub ip and ovs-vsctl so the setup-ovn teardown order check runs on any host 2026-10-20 10:09:28 +10:00
Julian Sommer 09392d18a1 test: wait for the consumer span to end before asserting ebs trace linkage (#1228)
TestEbsRequestWithTrace_ProducerConsumerLinked read the span recorder as
soon as UnmountOne returned, but the handler replies before its deferred
span.End runs, so under -race the recorder could hold only the producer
span. The handler now signals after its span ends and the test waits for
that signal instead of racing it.
2026-10-20 10:57:52 +11:00
Josh Siviour 142ca6eec3 fix: refuse DeleteSnapshot while a registered AMI uses the snapshot (#1227)
DeleteSnapshot checked only volume documents, so a snapshot backing a
registered AMI deleted anyway and left the AMI pointing at nothing.
Add a strict, whole-cluster AMI read next to the existing volume-clone
check and refuse with InvalidSnapshot.InUse when a registered AMI
names the snapshot; an unreadable or undecodable AMI document fails
the delete closed with ServerInternal.

The e2e teardown sweep now deregisters images before deleting
snapshots, since a snapshot sweep that runs first would now be
blocked by its AMI.

Known residual, out of scope here: RegisterImage reading the snapshot
and DeleteSnapshot's clone/AMI checks are not atomic, so a concurrent
RegisterImage can still race a DeleteSnapshot that passed its checks.
2026-10-20 09:54:35 +10:00
Josh Siviour 36949804e8 fix: unreadable volume records are never NotFound in DeleteVolume or ForceDetachVolume (#1226)
* fix: report an unreadable volume document as InternalError, not NotFound

* fix: ForceDetachVolume refuses an unreadable volume record

ForceDetachVolume mapped every metadata read error to InvalidVolume.NotFound.
An unreadable (present but undecodable or unreconstructable) document now
returns InternalError and leaves the attachment untouched; a genuinely
absent document or another account's volume still returns NotFound.

This is the call spx admin account delete --force uses, so an unreadable
volume no longer reads as already detached.
2026-10-20 09:41:28 +10:00
Josh Siviour a43febc187 test: characterise EC2 image, snapshot and volume deletion safety blockers (#1225)
Pin current behaviour only; no production change. The tests document why
owner cleanup of images and snapshots is not yet safe:
- a CreateImage snapshot does not pin its source volume;
- DeleteSnapshot ignores a registered AMI;
- a snapshot deleted inside the clone window strands the new volume;
- CreateImage of a retained instance needs its launch AMI;
- a snapshotted DeleteOnTermination volume survives terminate.
2026-10-20 09:37:20 +10:00
Josh Siviour e7cb02dedf feat: add EC2-owned platform cleanup capability for the proven-safe subset (#1224)
Add domains/ec2/accountcleanup, an unwired EC2-owned capability that
inventories key pairs, placement groups and launch templates for one
account, applies their owner delete rules, and proves each disposition
(deleted, blocked, failed, residue) by owner re-scan, never by Describe.

Cleanup deletes only the proven-safe subset: placement groups with no
members and key pairs that no non-terminated instance, open spot request
or live launch-template version references, plus unreachable
launch-template residue. Live launch templates are always blocked while
the EKS nodegroup reference is unresolved. Unknown or unreadable
reference state fails closed. Reference readers are injected; no action
adapter, NATS subject, IAM or accountteardown wiring is added.

Owner delete corrections shared by the public Delete* actions:
- DeleteKeyPair keeps metadata when the public-key delete fails, removes
  leftover public-key objects by name, and fails closed on unreadable or
  undecodable metadata.
- DeleteLaunchTemplate removes all version bodies and the name index
  before the revision-guarded header delete, and returns InternalError
  when a step fails.
- CreateLaunchTemplateVersion commits through the header revision, so a
  concurrent delete leaves no headerless body.
- DeletePlacementGroup is revision-guarded against concurrent
  reservations and maps store read errors to InternalError.
- Placement-group and launch-template deletes clear their tag-index
  entries.
2026-10-19 15:59:15 +10:00
Julian Sommer be84b73abe fix: rebuild the initramfs after upgrading the kernel in system AMIs
On Alpine 3.21 apk does not run the mkinitfs trigger inside the libguestfs
appliance, so apk upgrade left the old initramfs beside the new kernel. The
guest could not load virtio_blk and dropped to the emergency shell, which
stalled EKS cluster creation. Run mkinitfs for the installed kernel and fail
the build if the initramfs lacks its virtio_blk module.
2026-10-19 13:11:03 +10:00
Julian Sommer 3dbe352d1b fix: ship patched guest kernels in system AMIs and the LB microVM (#1222)
build-system-image.sh never upgraded the base cloud image, so every Alpine
system AMI kept the kernel its base image was released with. Run apk upgrade
before installing manifest packages so rebuilt AMIs carry the branch's current
linux-virt (6.18.57-r0 on 3.24, 6.12.113-r0 on 3.21; CVE-2026-99999).

The ELBv2 microVM defaulted to Alpine 3.20, which is EOL and will never get a
fixed kernel. Default to 3.24.
2026-10-19 14:04:27 +11:00
Josh Siviour fcfcc1dcff test: characterise AWSGW and ECR account admission (#1223) 2026-10-19 12:28:12 +10:00
Julian Sommer cd4363ba53 refactor: adopt shared helpers and remove local duplication in spinifexui (#1221)
Sidebar built from a typed NAV array with links rendered by the menu
button; one dashboard ResourceCard; signOut owns the clear-everything rule.
SystemImageRequired rechecks itself, wizards read mutation state, and
dialogs reset by unmounting. IAM policy hooks take a principal kind; ECS,
RDS, Nodes, S3 and ELBv2 share their duplicated tables and fields; EC2 CLI
builders use the flag helpers; CIDR fields validate with isValidCidr and
AWS prefix ranges.
2026-10-19 11:49:50 +11:00
Julian Sommer 90739c889b refactor: add shared helpers to spinifexui (#1220)
Signed fetch: signedAdminFetch wraps signedProxyFetch, which resolves
credentials itself, removing 11 per-caller checks.

Tags: one tagSchema, toSdkTags and diffTags; the tags editor moves to
components/ on useForm and TagsFieldArray.

Utilities: arnResourceId, byName, sameSet and shellSingleQuote replace
22 local copies.

Components: DeleteResourceButton (13 EC2 detail pages), ResourceList
(23 list pages), ResourceDeleteDialog (RDS, ECR), SelectField,
TextField, VpcSelectField, WizardFailureBanner, EditorShell and a
mode-driven ListenerDialog.

Tests: test/render.tsx with renderMutationHook, mockRouter,
createFakeSdk and mockAwsClients; the RDS and ELBv2 hook flows move to
mutations/*-flow.test.tsx.
2026-10-19 11:28:20 +11:00
Julian Sommer a4e5494e66 refactor: remove dead code and tautological tests from spinifexui (#1219)
Deletes unreferenced components, hook fields, types and query options,
drops exports used only in their own file and redundant Outlet route
components, and removes tests that assert their own fakes, restate query
key literals or repeat vitest's clearMocks. Query-key tests become
invalidation-prefix behaviour tests, and the Go compression tests now
drive the real handler chain.
2026-10-19 10:37:37 +11:00
Julian Sommer efdd255435 chore: drop the temporary vuln-scan acceptance list now Go is current 2026-10-19 09:27:09 +10:00
Julian Sommer ed8a6f2fb2 fix: Bedrock badges, VPC wizard tags, SG labels and LB protocol defaults in the UI (#1218) 2026-10-19 10:23:31 +11:00
Julian Sommer 0fb5c83a26 chore: fix spinifexui script 2026-10-19 08:56:54 +10:00
Julian Sommer 07bda82f1e chore: rebuild ui 2026-10-19 08:56:26 +10:00
Josh Siviour 49b94444ba feat: compare the EC2 platform inventory with legacy listings on explicit account-deletion dry runs (#1216) 2026-10-19 08:49:39 +10:00
Julian Sommer 1c21123c68 chore: fix lint issues 2026-10-19 08:46:10 +10:00
dependabot[bot] 15c9308e41 chore: bump the npm-dependencies group (#1217)
Bumps the npm-dependencies group in /spinifex/runtime/roles/spinifexui/frontend with 36 updates:

| Package | From | To |
| --- | --- | --- |
| [@aws-sdk/client-acm](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-acm) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-bedrock](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-bedrock-agent](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-agent) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-bedrock-agent-runtime](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-agent-runtime) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-bedrock-runtime](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-runtime) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-ec2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ec2) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-ecs](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecs) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-eks](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-eks) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-elastic-load-balancing-v2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-elastic-load-balancing-v2) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-iam](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-iam) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-rds](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-rds) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/client-sts](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-sts) | `3.1147.0` | `3.1152.0` |
| [@aws-sdk/lib-storage](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/lib/lib-storage) | `3.1147.0` | `3.1152.0` |
| [@base-ui/react](https://github.com/mui/base-ui/tree/HEAD/packages/react) | `1.8.0` | `1.9.0` |
| [@tanstack/react-query](https://github.com/TanStack/query/tree/HEAD/packages/react-query) | `5.104.1` | `5.104.3` |
| [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.41` | `1.170.44` |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.52.0` | `1.57.0` |
| [react](https://github.com/react/react/tree/HEAD/packages/react) | `19.3.0` | `19.3.1` |
| [react-dom](https://github.com/react/react/tree/HEAD/packages/react-dom) | `19.3.0` | `19.3.1` |
| [react-hook-form](https://github.com/react-hook-form/react-hook-form) | `7.89.0` | `7.90.0` |
| [zod](https://github.com/colinhacks/zod) | `4.6.5` | `4.6.7` |
| [@tanstack/eslint-plugin-query](https://github.com/TanStack/query/tree/HEAD/packages/eslint-plugin-query) | `5.104.1` | `5.104.3` |
| [@tanstack/router-plugin](https://github.com/TanStack/router/tree/HEAD/packages/router-plugin) | `1.168.42` | `1.168.46` |
| [@testing-library/user-event](https://github.com/testing-library/user-event) | `14.6.7` | `14.6.8` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `24.19.1` | `24.19.2` |
| [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8) | `5.0.3` | `5.0.5` |
| [happy-dom](https://github.com/capricorn86/happy-dom) | `20.14.5` | `21.0.1` |
| [oxc-transform-react](https://github.com/oxc-project/oxc/tree/HEAD/napi/transform-react) | `0.153.0` | `0.154.0` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) | `0.72.0` | `0.73.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.87.0` | `1.88.0` |
| [oxlint-tsgolint](https://github.com/oxc-project/tsgolint) | `7.0.2003` | `7.0.2004` |
| [ultracite](https://github.com/haydenbleasel/ultracite) | `7.12.4` | `8.0.1` |
| [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.3.3` | `8.3.4` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `5.0.3` | `5.0.5` |


Updates `@aws-sdk/client-acm` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-acm/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-acm)

Updates `@aws-sdk/client-bedrock` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-bedrock)

Updates `@aws-sdk/client-bedrock-agent` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-agent/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-bedrock-agent)

Updates `@aws-sdk/client-bedrock-agent-runtime` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-agent-runtime/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-bedrock-agent-runtime)

Updates `@aws-sdk/client-bedrock-runtime` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-runtime/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-bedrock-runtime)

Updates `@aws-sdk/client-ec2` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ec2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-ec2)

Updates `@aws-sdk/client-ecr` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-ecr)

Updates `@aws-sdk/client-ecs` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecs/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-ecs)

Updates `@aws-sdk/client-eks` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-eks/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-eks)

Updates `@aws-sdk/client-elastic-load-balancing-v2` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-elastic-load-balancing-v2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-elastic-load-balancing-v2)

Updates `@aws-sdk/client-iam` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-iam/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-iam)

Updates `@aws-sdk/client-rds` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-rds/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-rds)

Updates `@aws-sdk/client-s3` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-s3)

Updates `@aws-sdk/client-sts` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/clients/client-sts)

Updates `@aws-sdk/lib-storage` from 3.1147.0 to 3.1152.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1152.0/lib/lib-storage)

Updates `@base-ui/react` from 1.8.0 to 1.9.0
- [Release notes](https://github.com/mui/base-ui/releases)
- [Changelog](https://github.com/mui/base-ui/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mui/base-ui/commits/v1.9.0/packages/react)

Updates `@tanstack/react-query` from 5.104.1 to 5.104.3
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/react-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/react-query@5.104.3/packages/react-query)

Updates `@tanstack/react-router` from 1.170.41 to 1.170.44
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.44/packages/react-router)

Updates `lucide-react` from 1.52.0 to 1.57.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.57.0/packages/lucide-react)

Updates `react` from 19.3.0 to 19.3.1
- [Release notes](https://github.com/react/react/releases)
- [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/react/react/commits/v19.3.1/packages/react)

Updates `react-dom` from 19.3.0 to 19.3.1
- [Release notes](https://github.com/react/react/releases)
- [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/react/react/commits/v19.3.1/packages/react-dom)

Updates `react-hook-form` from 7.89.0 to 7.90.0
- [Release notes](https://github.com/react-hook-form/react-hook-form/releases)
- [Changelog](https://github.com/react-hook-form/react-hook-form/blob/master/CHANGELOG.md)
- [Commits](https://github.com/react-hook-form/react-hook-form/compare/v7.89.0...v7.90.0)

Updates `zod` from 4.6.5 to 4.6.7
- [Release notes](https://github.com/colinhacks/zod/releases)
- [Commits](https://github.com/colinhacks/zod/compare/v4.6.5...v4.6.7)

Updates `@tanstack/eslint-plugin-query` from 5.104.1 to 5.104.3
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/eslint-plugin-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/eslint-plugin-query@5.104.3/packages/eslint-plugin-query)

Updates `@tanstack/router-plugin` from 1.168.42 to 1.168.46
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/router-plugin/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/router-plugin@1.168.46/packages/router-plugin)

Updates `@testing-library/user-event` from 14.6.7 to 14.6.8
- [Release notes](https://github.com/testing-library/user-event/releases)
- [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/user-event/compare/v14.6.7...v14.6.8)

Updates `@types/node` from 24.19.1 to 24.19.2
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@vitest/coverage-v8` from 5.0.3 to 5.0.5
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v5.0.5/packages/coverage-v8)

Updates `happy-dom` from 20.14.5 to 21.0.1
- [Release notes](https://github.com/capricorn86/happy-dom/releases)
- [Commits](https://github.com/capricorn86/happy-dom/compare/v20.14.5...v21.0.1)

Updates `oxc-transform-react` from 0.153.0 to 0.154.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/napi/transform-react/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/crates_v0.154.0/napi/transform-react)

Updates `oxfmt` from 0.72.0 to 0.73.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.73.0/npm/oxfmt)

Updates `oxlint` from 1.87.0 to 1.88.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.88.0/npm/oxlint)

Updates `oxlint-tsgolint` from 7.0.2003 to 7.0.2004
- [Release notes](https://github.com/oxc-project/tsgolint/releases)
- [Commits](https://github.com/oxc-project/tsgolint/compare/v7.0.2003...v7.0.2004)

Updates `ultracite` from 7.12.4 to 8.0.1
- [Release notes](https://github.com/haydenbleasel/ultracite/releases)
- [Commits](https://github.com/haydenbleasel/ultracite/compare/ultracite@7.12.4...ultracite@8.0.1)

Updates `vite` from 8.3.3 to 8.3.4
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.3.4/packages/vite)

Updates `vitest` from 5.0.3 to 5.0.5
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v5.0.5/packages/vitest)

---
updated-dependencies:
- dependency-name: "@aws-sdk/client-acm"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-agent"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-agent-runtime"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-runtime"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ec2"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ecr"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ecs"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-eks"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-elastic-load-balancing-v2"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-iam"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-rds"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-s3"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-sts"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/lib-storage"
  dependency-version: 3.1152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@base-ui/react"
  dependency-version: 1.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/react-query"
  dependency-version: 5.104.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/react-router"
  dependency-version: 1.170.44
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: lucide-react
  dependency-version: 1.57.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: react
  dependency-version: 19.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: react-dom
  dependency-version: 19.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: react-hook-form
  dependency-version: 7.90.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: zod
  dependency-version: 4.6.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/eslint-plugin-query"
  dependency-version: 5.104.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/router-plugin"
  dependency-version: 1.168.46
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@testing-library/user-event"
  dependency-version: 14.6.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 24.19.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@vitest/coverage-v8"
  dependency-version: 5.0.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: happy-dom
  dependency-version: 21.0.1
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-dependencies
- dependency-name: oxc-transform-react
  dependency-version: 0.154.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxfmt
  dependency-version: 0.73.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxlint
  dependency-version: 1.88.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxlint-tsgolint
  dependency-version: 7.0.2004
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: ultracite
  dependency-version: 8.0.1
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-dependencies
- dependency-name: vite
  dependency-version: 8.3.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: vitest
  dependency-version: 5.0.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 09:43:30 +11:00
dependabot[bot] 3da9d9e60b chore: bump pnpm/action-setup (#1215)
Bumps the github-actions group with 1 update: [pnpm/action-setup](https://github.com/pnpm/action-setup).


Updates `pnpm/action-setup` from 6.0.10 to 6.1.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v6.0.10...v6.1.0)

---
updated-dependencies:
- dependency-name: pnpm/action-setup
  dependency-version: 6.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 09:43:04 +11:00
dependabot[bot] 04644bfbc9 chore: bump the go-dependencies group with 13 updates (#1214)
Bumps the go-dependencies group with 13 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.47.1` | `1.47.3` |
| [github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream](https://github.com/aws/aws-sdk-go-v2) | `1.7.20` | `1.7.22` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.33.7` | `1.33.9` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.20.7` | `1.20.9` |
| [github.com/aws/aws-sdk-go-v2/feature/ec2/imds](https://github.com/aws/aws-sdk-go-v2) | `1.20.1` | `1.20.3` |
| [github.com/containerd/containerd/v2](https://github.com/containerd/containerd) | `2.4.1` | `2.4.2` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.3.2` | `5.5.0` |
| [github.com/klauspost/compress](https://github.com/klauspost/compress) | `1.20.1` | `1.20.3` |
| [github.com/klauspost/cpuid/v2](https://github.com/klauspost/cpuid) | `2.4.0` | `2.4.1` |
| [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) | `2.15.0` | `2.15.1` |
| [github.com/oracle/oci-go-sdk/v65](https://github.com/oracle/oci-go-sdk) | `65.126.2` | `65.127.0` |
| [github.com/pterm/pterm](https://github.com/pterm/pterm) | `0.12.83` | `0.12.84` |
| [golang.org/x/tools](https://github.com/golang/tools) | `0.51.0` | `0.52.0` |


Updates `github.com/aws/aws-sdk-go-v2` from 1.47.1 to 1.47.3
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.1...v1.47.3)

Updates `github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream` from 1.7.20 to 1.7.22
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/notifications/v1.7.20...service/notifications/v1.7.22)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.33.7 to 1.33.9
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.7...config/v1.33.9)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.20.7 to 1.20.9
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.7...service/kms/v1.20.9)

Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.20.1 to 1.20.3
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.20.1...v1.20.3)

Updates `github.com/containerd/containerd/v2` from 2.4.1 to 2.4.2
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](https://github.com/containerd/containerd/compare/v2.4.1...v2.4.2)

Updates `github.com/go-chi/chi/v5` from 5.3.2 to 5.5.0
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.3.2...v5.5.0)

Updates `github.com/klauspost/compress` from 1.20.1 to 1.20.3
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.20.1...v1.20.3)

Updates `github.com/klauspost/cpuid/v2` from 2.4.0 to 2.4.1
- [Release notes](https://github.com/klauspost/cpuid/releases)
- [Commits](https://github.com/klauspost/cpuid/compare/v2.4.0...v2.4.1)

Updates `github.com/nats-io/nats-server/v2` from 2.15.0 to 2.15.1
- [Release notes](https://github.com/nats-io/nats-server/releases)
- [Changelog](https://github.com/nats-io/nats-server/blob/main/RELEASES.md)
- [Commits](https://github.com/nats-io/nats-server/compare/v2.15.0...v2.15.1)

Updates `github.com/oracle/oci-go-sdk/v65` from 65.126.2 to 65.127.0
- [Release notes](https://github.com/oracle/oci-go-sdk/releases)
- [Changelog](https://github.com/oracle/oci-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/oracle/oci-go-sdk/compare/v65.126.2...v65.127.0)

Updates `github.com/pterm/pterm` from 0.12.83 to 0.12.84
- [Release notes](https://github.com/pterm/pterm/releases)
- [Commits](https://github.com/pterm/pterm/compare/v0.12.83...v0.12.84)

Updates `golang.org/x/tools` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.51.0...v0.52.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.47.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream
  dependency-version: 1.7.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.33.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.20.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds
  dependency-version: 1.20.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/containerd/containerd/v2
  dependency-version: 2.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.20.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/klauspost/cpuid/v2
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/nats-io/nats-server/v2
  dependency-version: 2.15.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/oracle/oci-go-sdk/v65
  dependency-version: 65.127.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/pterm/pterm
  dependency-version: 0.12.84
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: golang.org/x/tools
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 09:01:24 +11:00
Josh Siviour 71301983bd feat: add read-only EC2 platform inventory for account termination (#1212)
* feat: add read-only EC2 platform inventory for account termination

* fix: drop placement-group member instance IDs from the platform inventory reply
2026-10-16 12:05:32 +10:00
Josh Siviour 359eb7a386 docs: add account lifecycle PR checklist (#1213) 2026-10-16 12:01:14 +10:00
Josh Siviour ef782a2758 test: characterise EC2 platform resource residue and direct-NATS disclosure for account termination (#1211) 2026-10-16 11:25:57 +10:00
Josh Siviour a7a774a7fc docs: post-ADR-0001 source-tree doc fixes and move planning material to Mulga (#1210)
* docs: update source-tree docs and comments after the ADR-0001 package moves

* docs: move package-boundary planning material to Mulga
2026-10-15 16:47:03 +10:00
Josh Siviour bd14e2d67d refactor: move read-only GetResources aggregation to domains/tagging and retire gateway/tagging (#1209)
* test: characterise current tagging read, write and aggregation behaviour

* docs: record tagging capability discovery and narrowed ADR-0001 cut

* test: update ACM tagging characterisation after key preservation

* feat: add read-only TaggedLister capability in domains/tagging

* feat: add AWSGW tag index and ELBv2 TaggedLister adapters

* refactor: move GetResources aggregation to domains/tagging

* docs: close out the ADR-0001 tagging boundary

* docs: repoint tagging migration citations to the rebased commits
2026-10-15 14:58:44 +10:00
Josh Siviour b5879f44ff ADR-0001 package-boundary migration checkpoint (#1190)
* refactor: move telemetry file naming to contract

* docs: record telemetry contract move

* refactor: move aws xml response helpers

* docs: record aws xml helper move

* refactor: move aws tag projections to foundation

* docs: record aws tag projection move

* refactor: move account key to state foundation

* docs: record account key state move

* refactor: move host privilege escalation to runtime

* docs: record host privilege move

* chore: resolve lint findings from package-boundary moves

* refactor: move process supervision helpers to runtime host

* docs: record process supervision move

* refactor: move nbd endpoint helpers to compute runtime

* docs: record nbd endpoint move

* refactor: move image catalogue to operator

* docs: record image catalogue move

* refactor: move resource id generation to aws foundation

* docs: record resource id move

* refactor: move predastore role to runtime roles

* docs: point service manifest at acm domain

* docs: record predastore role move

* refactor: move northstar role to runtime roles

* docs: record northstar role move

* refactor: move awsgw role to runtime roles

* docs: record awsgw role move

* refactor: move error payload helpers to aws errors

* docs: record error payload move

* refactor: move account identifiers to aws identifiers

* docs: record account identifier move

* refactor: move NATS transport to messaging foundation

* docs: record NATS transport move

* refactor: move system instance contract to EC2 domain

* docs: record system instance move

* refactor: move system VPC builder to network domain

* docs: record system VPC move

* refactor: move instance metadata service to EC2 domain

* docs: record guest metadata move and dependency-rule debt

* refactor: move DNS control plane to DNS domain

* docs: record DNS domain move

* refactor: move base zone bootstrap to DNS domain

* docs: record base zone bootstrap move

* refactor: replace SystemAccountID with GlobalAccountID

* docs: record SystemAccountID removal

* refactor: extract CA key-pair loader to foundation pki

* docs: record foundation pki extraction

* refactor: move tenant CA paths to ACM domain

* docs: record tenant CA path move

* refactor: move IPsec certificate handling to network host

* docs: record IPsec certificate move

* refactor: move Ochre serving lifecycle to the Ochre domain

handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.

* docs: record Ochre domain move

* refactor: move Ochre vector store to the Ochre domain

handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.

* docs: record Ochre vector store move

* refactor: move Hugging Face Hub client to the Ochre domain

gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.

* docs: record Hugging Face Hub client move and Ochre gateway debt

* refactor: move EC2 tag mutation helpers to the EC2 domain

utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.

* docs: record EC2 tag mutation move and migrate split finding

* refactor: move network packages under the network domain

Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.

* docs: record network domain move

* refactor: move standalone EC2 resource packages to the EC2 domain

Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.

* docs: record standalone EC2 resource moves

* refactor: move EC2 VPC networking packages to the EC2 domain

Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.

* docs: record EC2 VPC networking moves

* refactor: move EC2 compute and storage packages to the EC2 domain

Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.

* docs: record EC2 compute and storage moves

* refactor: move ECR credential provider to the EKS agent tree

Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.

* docs: record ECR credential provider move

* refactor: move EKS gateway publish helper to the EKS agent tree

Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.

* docs: record EKS gateway publish move

* refactor: move EKS gateway fetch helper to the EKS agent tree

Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.

* docs: record EKS gateway fetch move

* refactor: move EKS webhook cert helper to the EKS agent tree

cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.

* docs: record EKS webhook cert move

* refactor: move EKS konnectivity cert helper to the EKS agent tree

cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.

* docs: record EKS konnectivity cert move

* refactor: move EKS token webhook to the EKS agent tree

cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.

* docs: record EKS token webhook move

* refactor: move ECS agent runtime and credentials packages to the ECS agent tree

* docs: record ECS agent subpackage move

* refactor: move ECS agent implementation to the ECS agent tree

cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.

* docs: record ECS agent move

* refactor: move RDS guest agent to agents/rds/agent

* docs: record RDS guest agent move

* refactor: move installer packages to operator/installer

* docs: record installer move

* refactor: move spx command tree to operator/cli

Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.

* docs: record spx command tree move

* chore: accept known vulnerabilities with an expiry

The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.

* refactor: move EC2 account action adapters to domains/ec2/awsapi

* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi

* refactor: move EC2 key action adapters to domains/ec2/awsapi

* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi

* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi

* docs: record EC2 standalone action adapter moves

* refactor: move EC2 zone action adapters to domains/ec2/awsapi

* refactor: move EC2 vpc action adapters to domains/ec2/awsapi

* refactor: move EC2 eigw action adapters to domains/ec2/awsapi

* refactor: move EC2 eip action adapters to domains/ec2/awsapi

* refactor: move EC2 igw action adapters to domains/ec2/awsapi

* refactor: move EC2 natgw action adapters to domains/ec2/awsapi

* refactor: move EC2 routetable action adapters to domains/ec2/awsapi

* docs: record EC2 VPC networking action adapter moves

* refactor: move EC2 image action adapters to domains/ec2/awsapi

* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi

* refactor: move EC2 volume action adapters to domains/ec2/awsapi

* refactor: move EC2 tags action adapters to domains/ec2/awsapi

* docs: record EC2 storage and tag action adapter moves

* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi

* refactor: move EC2 idempotency adapter to domains/ec2/awsapi

* refactor: move EC2 instance action adapters to domains/ec2/awsapi

* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi

* docs: record EC2 instance action adapter moves

* refactor: move AWS REST-JSON router to ingress/aws/rest

* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit

* docs: record AWS ingress helper moves and tooling conformance

* docs: correct historical lifecycle references

* test: characterize ECR control-plane dispatch before registration seam

* feat: add AWS ingress service registration contract

* docs: record ingress registration phases 0 and 1

* refactor: route registered AWS services through the ingress dispatch registry

The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.

The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.

* docs: record ingress dispatch registry consumption

* refactor: register the ECR control plane from its domain

domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.

Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.

* docs: record ECR control-plane registration

* fix: compose the ECR control plane in the integration harness

The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.

* docs: record ingress pipeline deferral

* refactor: extract AWS error envelope rendering to ingress

ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.

The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.

* docs: record error envelope extraction

* refactor: move AWS service selection into ingress dispatch

Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.

* docs: record ingress service selection move

* test: characterize ACM control-plane dispatch before registration

Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.

* refactor: register ACM control plane from its domain

domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.

* docs: record ACM registration

* docs: record ECS, EKS and RDS source inventory and first-resource ranking

* docs: record EKS control-plane secrets persisted in the cluster record

* chore: drop containerd advisory acceptance now that dev ships v2.4.1

* test: characterize RDS DB subnet group lifecycle before extraction

* docs: record RDS DB subnet group lifecycle contract

* refactor: extract the RDS DB subnet group lifecycle owner

domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.

* docs: record RDS DB subnet group owner extraction

* docs: correct RDS subnet group tagging debt and conformance scope

* test: characterize ECS task definition lifecycle before extraction

* docs: record ECS task definition lifecycle contract

* refactor: extract ECS task definition owner into domains/ecs/taskdefinition

Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.

The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.

List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.

* docs: record ECS task definition owner extraction

* test: characterize EKS access entry lifecycle before extraction

* docs: record EKS access entry lifecycle contract

* refactor: extract EKS access entry owner into domains/eks/access

Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.

Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.

List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.

* docs: record EKS access entry owner extraction

* test: characterize RDS engine catalogue before extraction

* fix: return InvalidParameterValue for a missing RDS master password

ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.

* docs: record RDS engine catalogue boundary and evidence

* refactor: extract RDS engine catalogue into domains/rds/engine

Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.

The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.

The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.

* docs: record RDS engine catalogue owner move

* test: characterize RDS DB parameter group lifecycle before extraction

* docs: record RDS DB parameter group lifecycle contract

* refactor: extract RDS DB parameter group owner

Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.

* docs: record RDS DB parameter group owner extraction

* test: characterize EKS add-on lifecycle and guest wire

Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.

* docs: record EKS add-on lifecycle contract

* fix: require iam:PassRole for EKS add-on service account role

CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.

* docs: record EKS add-on PassRole check

* refactor: extract EKS add-on owner

Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.

* docs: record EKS add-on owner extraction

* refactor: extract EKS add-on guest wire contract

Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.

* docs: record EKS add-on wire contract extraction

* docs: update EKS add-on inventory and remaining edges

* docs: draft EKS add-on lifecycle correction design for review

* chore: run contracts tests in the default test, coverage and race targets

* docs: apply review decisions to EKS add-on lifecycle design

* test: characterize EKS add-on v1 compatibility with added fields

Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.

* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories

* docs: draft EKS add-on v2 guest contract design for review

* docs: qualify historical architecture references

* docs: apply review decisions and ADR-0006 to EKS add-on v2 design

* fix(eks): bind internal report publishing to the caller's cluster

PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.

PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.

* fix(eks): bind webhook token review to the caller's cluster

WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.

WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.

* fix(eks): validate add-on serviceAccountRoleArn syntax

CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.

The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.

* docs: record EKS internal-route and role-ARN security corrections

* docs: add EKS add-on release membership and activation case study

* fix: point UI build, CI and lint rules at moved package paths

* docs: record EKS add-on activation review decisions

* refactor: move quota admission to domains/admission/quota

Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.

* docs: reconcile package-boundary record for ADR-0001 close-out

Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.

* test: move KV migration engine tests to foundation/state/migrate

The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.

* refactor: move config and object-store upgrade workflow to operator/upgrade

Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.

* docs: record migrate move to operator/upgrade

* test: enforce ADR-0001 layering with a blocking ratchet

Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.

Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.

* refactor: drop the legacy handlers_ecr import alias

Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.

* chore: mark spf13/pflag as a direct dependency

operator/cli/wiring.go imports pflag directly. Same version; no upgrade.

* docs: record test-package-check and nilaway as unverified gates

* refactor: move the layering test to module-root architecture/

The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.

* fix: update paths left stale by the package moves

The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.

* refactor: move the default-VPC subject into contracts/ec2/v1

utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.

* docs: qualify the migrate split and record admin and Viperblock findings

The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.

* refactor: move operator-only admin code to operator/admin

Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.

foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.

* docs: record operator/admin split in package-boundary migration

* test: pin viperblockd fence, abandon and dirty-marker wire shapes

Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.

* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts

Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.

Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.

* refactor: split the Viperblock EBS adapter out of services/viperblockd

Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.

The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.

Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.

* refactor: move the viperblockd role to runtime/roles/viperblockd

With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.

* docs: record viperblockd contract, adapter and role moves

* test: characterise vpc.* network-realisation routes

Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.

Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).

* feat: add contracts/network/v1 for vpc.* realisation routes

Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.

* refactor: use contracts/network/v1 for vpc.* producers and consumers

Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.

Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.

No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.

* refactor: replace utils/vpcd_event.go with a typed network projection client

Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.

utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.

* refactor: route the remaining fire-and-forget vpc.* producers through projection

CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.

* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils

The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.

utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.

* docs: record the network contract, projection client and utils removal

* test: characterise instance-state persistence, recovery and generation handling

Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.

* test: freeze direct instance-state consumers

Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.

* docs: record ADR-0007 slice 0 instance-state inventory

Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.

* test: narrow the instance-state ratchet to instance-state access

Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.

* docs: record the narrowed instance-state ratchet commit

* docs: design the instance-state transition record and activation gate

* docs: record the instance-state transition-record design commit

* docs: record slice 1a activation-gate decisions

* feat: add instance transition-record codec with format marker

Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.

Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.

No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.

* docs: record slice 1b step 1 codec and awsgw rollout qualification

Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its
surface, preservation at every object level, the changed-array limitation,
marker handling, the rejection classes and its capable-writers-only scope. C2
supersedes P1's top-level-only preservation for the codec. G19 records that
old read-only gateway consumers remain safe but DescribeInstanceStatus gains
unconfirmed/degraded visibility only after awsgw upgrades. The migration record
gains the commit row and states that no production path uses the codec yet.

* refactor: move the instance transition-record codec to domains/ec2/instance/state

The codec defines how the canonical persisted instance record evolves, with
its assignment and observation sections, which is EC2 instance ownership rather
than node-local VM supervision. It moves unchanged into its own package and
keeps wrapping the legacy vm.InstanceRecord alias and its spec and status
types, which stay in runtime/compute/vm for now.

The domain-to-runtime import is recorded as new dependency-rule debt with its
removal condition and allow-listed against that entry. The instance-state
ratchet now names the new file as an owner file; its allowlist is unchanged.

* fix: refuse array changes that would drop element extensions

A changed array is replaced whole by the writer's value, which silently lost
any member a newer writer had added inside its elements. Encode now inspects
the array as read: if any element carries a member its typed schema does not
know, at any depth, it returns ErrRecordExtensionConflict naming the array and
the first such member. Arrays without such members are still replaced whole,
unchanged arrays are still kept verbatim, and arrays inside unknown members are
never inspected.

* docs: record the codec move and the array extension-conflict rule

Records 8db9771fb and c617914a1. T1 and Section 10 now name
domains/ec2/instance/state as the eventual owner of the persisted instance
record and its evolution, with vm.InstanceRecord as the temporary legacy type
it wraps. C3 replaces the changed-array limitation with the refusal: a changed
array with element extensions fails with ErrRecordExtensionConflict and is never
silently dropped. P2 stays a conservative rule until finalisation.

* fix: refuse removals that would drop record extension fields

A capable writer may no longer remove or replace a known object, map entry or
array when its stored value carries members the writer's schema does not know.
The check that guarded changed arrays now runs wherever the merge drops or
overwrites a stored value instead of merging into it: a known member deleted or
set to null, a map entry removed or stored under a key the typed decode
rewrote, a value of another JSON kind, and a changed array. Encode returns
ErrRecordExtensionConflict naming the changed container and the first unknown
member. Removal stays allowed when the stored value has no extensions, and a
change merged member by member still keeps unknown members.

* docs: record the uniform extension-conflict rule

Adds design clause C8 for 0d7b4266d: a capable writer may not remove or
replace a known object, map entry or array whose stored value carries unknown
structured extension fields; removal without extensions stays allowed. C3 and
C5 point at it. The migration record gains the commit row, updated subsection
wording and a pointer-bump follow-up deferring make docs-portal to the Mulga
gitlink PR so the portal evaluates the proposed Spinifex revision.

* refactor: inject vpcd's IMDS instance-state readers from the instance state boundary

* docs: cite the vpcd reader injection commit in the migration record

* refactor: move vpcd to runtime/roles/vpcd

* docs: cite the vpcd move commit in the migration record

* refactor: move the ECS agent wire vocabulary to contracts/ecs/v1

Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy
shim. Subject strings, JSON fields and types are byte-identical; the old-agent
compatibility test moves unchanged and golden pins cover the remaining
message types. The README route table records that only service-reconcile
has a live producer today; the agent reuses these payload types through the
gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon.

* refactor: move system-image promotion to domains/ec2/image

Move images_promote.go and its tests from admin to domains/ec2/image:
SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage,
GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and
promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI
callers import ec2image. admin now holds only account IDs and key generators.

* docs: record the ECS contract and image-promotion moves and the remaining-roots matrix

* refactor: move the bootstrap account identifiers to foundation/aws/identifiers

DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed,
platform-reserved identity of the first human-facing bootstrap account.
Names and values are unchanged and stay fixed. Callers in daemon, gateway,
awsgw, accountteardown and the operator CLI use awsidentifiers; admin now
holds only the IAM credential generators.

* docs: cite the bootstrap account identifier move in the migration record

* refactor: inject the STS cluster JWKS lookup from handlers/eks

handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and
InvalidIdentityToken mapping; handlers/eks implements it as
OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the
integration harness inject the reader. STS no longer imports any EKS
package, including tests, and the handlers/sts -> handlers/eks layering
exception is removed.

* docs: cite the STS JWKS resolver cut in the migration record

* refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin

GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to
foundation/aws/credentials as a stateless format primitive. handlers/iam
and operator/cli import it directly. The admin legacy root is deleted
and joins the retired roots so it cannot return.

* docs: cite the admin retirement in the migration record

* revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint

The instance transition-record codec, its tests and the slice 1a design
had no production caller. They leave this package-boundary checkpoint
and will be reapplied on a dedicated ADR-0007 branch. Slice 0
characterisation and the vpcd IMDS readers in
domains/ec2/instance/state stay.

* docs: cite the ADR-0007 codec revert in the migration record

* fix: declare the ECS action subjects in service-interfaces.yaml

Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then
failed on the 34 ECS action subjects the daemon registers. Declare them as
ecs.*, the same wildcard convention as rds.*.

* chore: run manifest-check and manifest-lint in preflight

CI runs both, but preflight did not, so manifest drift could pass local
checks and still fail Build and Test.

* docs: record resumed preflight enforcement and dev merge ports

* fix: restore dev changes dropped by earlier merges

Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored.

* docs: record accountteardown as a frozen legacy root pending cleanup capabilities

---------

Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
2026-10-15 14:42:53 +10:00
Julian Sommer 0f025e8332 chore: add lambda to roadmap 2026-10-15 14:10:53 +10:00
Julian Sommer 4c4ca253b4 fix(ci): grant contents: write only to the release jobs that edit the release
The release workflow granted contents: write to every job, including the distro builds and the self-hosted verify job, which only read. Only release, promote, mark-failed and iso create or edit the GitHub release, so they alone get write and the rest default to read.
2026-10-15 13:18:16 +10:00
Julian Sommer 3a3807f08e fix(ci): build releases without the shared Go module cache
setup-go caches by default, and a release or dev prerelease run restores caches that any dev or main run saved. A poisoned module cache from one of those runs would then flow into the shipped binaries, so both release builds now start from a clean cache.
2026-10-15 13:16:59 +10:00
Josh Siviour 26f12e7107 fix: prove volume holder absent by node identity before teardown force-delete (#1208)
Account teardown's force path cleared a volume's attachment and deleted it
without proving the holding instance gone, and the delete guard judged its
instance view complete by counting replies, so one node answering twice
could stand in for the silent node hosting the guest.

The holder must now be terminated or definitively InvalidInstanceID.NotFound
on a describe naming it, confirmed on a listing every configured node
answered by identity. Anything else retains the volume, reports it stuck
naming the volume and instance, and keeps the account TERMINATING.

DeleteVolume's in-use guard takes the configured node set and uses the same
identity-checked listing, so public DeleteVolume also refuses while any
configured node is unanswered. That refusal keeps the listing's retryable
ServiceUnavailable and Retry-After; other guard failures stay InternalError.
2026-10-15 13:13:36 +10:00
Julian Sommer 5865fc418e chore(ci): lock workflow action dependencies with actions.lock (#1207) 2026-10-15 13:55:16 +11:00
Julian Sommer 02c02f6632 chore: pin base images by digest and let dependabot refresh them 2026-10-15 11:42:56 +10:00
Julian Sommer 9075b13fe8 fix(ci): pin go-junit-report through go.mod instead of installing @latest 2026-10-15 11:42:52 +10:00
Julian Sommer 600fb41012 fix(ci): fail UI CI when the committed dist differs from a clean build 2026-10-15 11:21:35 +10:00
Julian Sommer 236dd2e87d fix(ci): build releases and system AMIs against the sibling tags go.mod requires 2026-10-15 11:19:37 +10:00
Julian Sommer 96bcb4746d fix(ci): never run the mkosi image job for fork PRs on self-hosted runners 2026-10-15 11:09:44 +10:00
Julian Sommer 316a1b7c03 refactor(spinifexui): replace clsx and tailwind-merge with cn (#1206) 2026-10-15 11:50:21 +11:00
Julian Sommer 7fdabe1207 refactor(spinifexui): enforce shadcn no-restyle with component variants (#1205) 2026-10-15 11:37:03 +11:00
Julian Sommer 9498481161 refactor(spinifexui): remove dead popup animation classes (#1204) 2026-10-15 11:28:46 +11:00
Josh Siviour f3bebacbe5 fix: fail closed on incomplete instance inventory during account teardown (#1200)
Account teardown listed instances account-wide through a path that returns
whatever nodes answered. A silent or erroring node, a node answering twice,
or a failed instance-bucket query gave a short list with no error, which
teardown counted as drained.

Add DescribeAccountInstancesComplete, which requires a clean reply from
every configured node by node ID plus both instance buckets, and otherwise
returns a retryable ServiceUnavailable. Thread the configured node set into
the teardown engine from the admin API and the CLI. The engine already
aborts on a listing error, so the account stays TERMINATING and storage
teardown does not run.
2026-10-15 10:23:22 +10:00
Julian Sommer 2a19384fe1 feat(spinifexui): enable @shadcn/lint design-system rules (#1203) 2026-10-15 11:17:27 +11:00
Julian Sommer b6e50c488c fix(spinifexui): guard admin routes in beforeLoad so a full page load renders (#1202) 2026-10-15 10:50:58 +11:00
Julian Sommer 5ad2346b3b chore: add contribution license checkbox to PR template 2026-10-15 09:12:25 +10:00
Julian Sommer 4981828497 docs: private-CA trust for containerised tooling pushing to ECR 2026-10-15 09:03:20 +10:00
Julian Sommer 838de354c2 fix: keep the bootstrap 403 visible to heartbeats in rds-agent test
The fake control plane returned the scripted 403 once, then failed every
retry with "returned no config", overwriting the failure the heartbeat
reports. The test only passed if a beat landed within one 1ms retry
interval. A persistent fake error keeps the 403 on every retry.
2026-10-15 09:00:04 +10:00
Julian Sommer 29aef59dae fix: enforce HttpPutResponseHopLimit on the IMDSv2 token response (#1201) 2026-10-15 09:52:10 +11:00
Julian Sommer 6644e6dc9f test: isolate JetStream recovery tests and parallelise slow daemon tests 2026-10-14 14:03:33 +10:00
Julian Sommer a0b4d840e2 test: use 1024-bit keys and parallelise ACM handler tests 2026-10-14 13:54:15 +10:00
Julian Sommer bec3a78167 test: use 1024-bit signing key in OCI api client tests 2026-10-14 13:51:48 +10:00
Julian Sommer 966a28f546 fix: reject key-pair names that traverse out of the caller's keys (#1199) 2026-10-14 14:15:35 +11:00
Julian Sommer c861642d44 fix: require iam:PassRole on the role in AddRoleToInstanceProfile (#1198) 2026-10-14 14:00:29 +11:00
Julian Sommer 72d234e95a fix: bound RunInstances counts and make vCPU quota arithmetic overflow-safe (#1197) 2026-10-14 12:46:44 +11:00
Julian Sommer c09f0b5f11 fix: bound per-tap IMDS server timeouts and connections (#1196)
Set read, write and idle timeouts and a header cap on each per-tap IMDS
http.Server, and cap its listener at 32 connections, so one guest cannot
hold unbounded vpcd descriptors and goroutines with idle keep-alives.
2026-10-14 12:39:34 +11:00
Julian Sommer bc51816ad8 docs: rename site badge and drop Docs index link 2026-10-14 10:53:18 +10:00
Julian Sommer ebc36d6f0c chore: cleanup stale scripts 2026-10-14 10:39:38 +10:00
Ben Duncan 261b695e53 docs: document disabling overlay IPsec for trusted fabrics
The VPC networking guide had no mention of IPsec, so an operator reading it
had no way to discover that the geneve overlay is encrypted by default or
what turning that off is worth. Add a section covering the default-on
posture, the topologies where disabling is and is not appropriate, the
measured cost on the ConnectX-5 reference cluster, and the full procedure.

Correct the performance-tuning guide's claim that a manual
"ovn-nbctl set NB_Global . ipsec=false" is transient because the daemon
reconciles it. That holds for the enable direction only: ReconcileOVNIPSec
returns early when IPsec is disabled, so it stops requesting encryption but
never clears NB_Global.ipsec or other_config:ipsec_encapsulation. Both are
left reading true, which makes the control plane disagree with the config.
Document the manual steps as a required complement to the config change.
2026-10-14 08:09:51 +10:00
Julian Sommer e4d90b30d3 fix(spinifexui): end the session in every tab on sign-out or sign-in 2026-10-14 08:09:51 +10:00
Julian Sommer bf3b830152 fix(spinifexui): refresh the role page after instance profile membership changes
Adding a role to an instance profile, removing it, or deleting the profile
only refreshed the profile list, so the role page kept its cached profiles
for up to five minutes.
2026-10-14 08:09:51 +10:00
Julian Sommer b087d5d469 fix(spinifexui): confirm destructive route, rule, gateway, EIP and version actions
Deleting a route, disassociating a subnet or Elastic IP, revoking a
security-group rule, detaching an internet gateway and deleting a launch
template version each ran on one click. They now open a confirmation
dialog, and a failed launch template version delete is reported instead of
silently leaving the row.
2026-10-14 08:09:51 +10:00
Julian Sommer fd9d69f05b fix(spinifexui): delete the launch-wizard security group when a launch fails
A failed RunInstances or ingress authorize left launch-wizard-N behind, so
the retry, which reuses the same default name, failed with
InvalidGroup.Duplicate. The security group list now also refreshes after a
failed launch.
2026-10-14 08:09:51 +10:00
Julian Sommer 75471404b6 fix(spinifexui): validate the VPC wizard's real subnet plan and report every created subnet
The wizard validated only custom CIDRs, so a default plan that overran a small
VPC, or a custom CIDR overlapping a default, passed the form and left a
half-built VPC. Validation, the mutation, the preview and the CLI panel now
share one resolved CIDR list, and subnet errors are shown.

Subnets are created with allSettled so a failure still reports every subnet
that was made, and the calculator packs private subnets after the public ones
when the VPC midpoint leaves no room.
2026-10-14 08:09:51 +10:00
Julian Sommer 848a662aac fix(spinifexui): accept /0 to /32 for the launch wizard's custom SSH source
The custom CIDR used isValidCidr's VPC defaults of /16 to /28, so a single-IP
source such as 203.0.113.5/32, 10.0.0.0/8 and 0.0.0.0/0 were all rejected.
AWS accepts any prefix for a security-group rule source.

Rebuilds frontend/dist, which also picks up the signature-error logout fix.
2026-10-14 08:09:51 +10:00
Julian Sommer ab45ea84ad fix(spinifexui): stop logging out on signature errors
SignatureDoesNotMatch and IncompleteSignature no longer clear the session:
clock skew and path rewrites also produce them, so the user was logged out
repeatedly over a request problem.
2026-10-14 08:09:51 +10:00
Julian Sommer f50630ed4b fix(spinifexui): stop silent connections blocking the TLS split listener
Accept read the protocol-detection byte inline with a 5s deadline, so each
client that sent nothing stalled every new visitor. Accept now returns the
TLS-wrapped conn immediately and the first byte is sniffed on the first
handshake Read, in the connection's own goroutine.
2026-10-14 08:09:51 +10:00
Julian Sommer abb3da8af2 chore: keep only the newest 5 dev prereleases 2026-10-14 08:09:51 +10:00
Julian Sommer 7e008c4689 fix: read single-host predastore health from /readyz in Service Metrics (#1195) 2026-10-14 08:09:51 +10:00
Julian Sommer d91312d2f9 fix: return NotFound/Malformed for unknown instance IDs on lifecycle verbs (#1194) 2026-10-14 08:09:51 +10:00
Josh Siviour c5d96ee56c fix: preserve ACM private key on certificate tag changes (#1193) 2026-10-14 08:09:51 +10:00
Julian Sommer 788483807f fix: golangci-lint version in CONTRIBUTING.md 2026-10-14 08:09:51 +10:00
Julian Sommer 429c5f13e5 fix: README 2026-10-12 10:31:44 +10:00
Julian Sommer 49d3990252 fix: update README 2026-10-12 10:13:11 +10:00
Julian Sommer dbcda95b3c chore: remove docs dotpoint 2026-10-12 09:24:11 +10:00
Julian Sommer 562e19aeca fix: remove coverage badge 2026-10-12 09:23:11 +10:00
Julian Sommer 19c35326a3 chore: fix lint issues 2026-10-12 08:58:04 +10:00
Julian Sommer 68b1e18c72 chore: bump golangci-lint to v2.14.0 for Go 1.27.2 export data 2026-10-12 08:32:43 +10:00
dependabot[bot] 9f9b7be9f7 chore: bump the go-dependencies group across 1 directory with 12 updates (#1191)
Bumps the go-dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.33.6` | `1.33.7` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.20.6` | `1.20.7` |
| [github.com/oracle/oci-go-sdk/v65](https://github.com/oracle/oci-go-sdk) | `65.126.1` | `65.126.2` |
| [go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp](https://github.com/open-telemetry/opentelemetry-go-contrib) | `0.71.0` | `0.72.0` |
| [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/metric](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/sdk/metric](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/trace](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [golang.org/x/sync](https://github.com/golang/sync) | `0.23.0` | `0.24.0` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.48.0` | `0.49.0` |
| [golang.org/x/tools](https://github.com/golang/tools) | `0.50.0` | `0.51.0` |



Updates `github.com/aws/aws-sdk-go-v2/config` from 1.33.6 to 1.33.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.6...config/v1.33.7)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.20.6 to 1.20.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.6...service/mq/v1.20.7)

Updates `github.com/oracle/oci-go-sdk/v65` from 65.126.1 to 65.126.2
- [Release notes](https://github.com/oracle/oci-go-sdk/releases)
- [Changelog](https://github.com/oracle/oci-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/oracle/oci-go-sdk/compare/v65.126.1...v65.126.2)

Updates `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` from 0.71.0 to 0.72.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go-contrib/compare/zpages/v0.71.0...zpages/v0.72.0)

Updates `go.opentelemetry.io/otel` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/metric` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/sdk` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/sdk/metric` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/trace` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...v1.47.0)

Updates `golang.org/x/sync` from 0.23.0 to 0.24.0
- [Commits](https://github.com/golang/sync/compare/v0.23.0...v0.24.0)

Updates `golang.org/x/sys` from 0.48.0 to 0.49.0
- [Commits](https://github.com/golang/sys/compare/v0.48.0...v0.49.0)

Updates `golang.org/x/tools` from 0.50.0 to 0.51.0
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.33.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.20.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/oracle/oci-go-sdk/v65
  dependency-version: 65.126.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
  dependency-version: 0.72.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/metric
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/sdk/metric
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/trace
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/sys
  dependency-version: 0.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/tools
  dependency-version: 0.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
2026-10-12 09:22:17 +11:00
Julian Sommer 68d3018b3e chore: update golang.org/x/net 2026-10-12 08:15:03 +10:00
Julian Sommer 1a24217b2c chore: rebuild frontend 2026-10-12 08:08:49 +10:00
dependabot[bot] c09f15b50f chore: bump the npm-dependencies group (#1192)
Bumps the npm-dependencies group in /spinifex/services/spinifexui/frontend with 25 updates:

| Package | From | To |
| --- | --- | --- |
| [@aws-sdk/client-acm](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-acm) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-bedrock](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-bedrock-agent](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-agent) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-bedrock-agent-runtime](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-agent-runtime) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-bedrock-runtime](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-bedrock-runtime) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-ec2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ec2) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-ecs](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecs) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-eks](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-eks) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-elastic-load-balancing-v2](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-elastic-load-balancing-v2) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-iam](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-iam) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-rds](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-rds) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/client-sts](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-sts) | `3.1144.0` | `3.1147.0` |
| [@aws-sdk/lib-storage](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/lib/lib-storage) | `3.1144.0` | `3.1147.0` |
| [@tanstack/react-query](https://github.com/TanStack/query/tree/HEAD/packages/react-query) | `5.104.0` | `5.104.1` |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.49.0` | `1.52.0` |
| [@tanstack/eslint-plugin-query](https://github.com/TanStack/query/tree/HEAD/packages/eslint-plugin-query) | `5.104.0` | `5.104.1` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `24.19.0` | `24.19.1` |
| [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/tree/HEAD/packages/plugin-react) | `6.1.1` | `6.1.2` |
| [oxc-transform-react](https://github.com/oxc-project/oxc/tree/HEAD/napi/transform-react) | `0.152.0` | `0.153.0` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) | `0.71.0` | `0.72.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.86.0` | `1.87.0` |
| [ultracite](https://github.com/haydenbleasel/ultracite) | `7.12.2` | `7.12.4` |
| [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.3.1` | `8.3.3` |


Updates `@aws-sdk/client-acm` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-acm/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-acm)

Updates `@aws-sdk/client-bedrock` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-bedrock)

Updates `@aws-sdk/client-bedrock-agent` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-agent/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-bedrock-agent)

Updates `@aws-sdk/client-bedrock-agent-runtime` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-agent-runtime/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-bedrock-agent-runtime)

Updates `@aws-sdk/client-bedrock-runtime` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-bedrock-runtime/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-bedrock-runtime)

Updates `@aws-sdk/client-ec2` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ec2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-ec2)

Updates `@aws-sdk/client-ecr` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-ecr)

Updates `@aws-sdk/client-ecs` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecs/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-ecs)

Updates `@aws-sdk/client-eks` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-eks/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-eks)

Updates `@aws-sdk/client-elastic-load-balancing-v2` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-elastic-load-balancing-v2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-elastic-load-balancing-v2)

Updates `@aws-sdk/client-iam` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-iam/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-iam)

Updates `@aws-sdk/client-rds` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-rds/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-rds)

Updates `@aws-sdk/client-s3` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-s3)

Updates `@aws-sdk/client-sts` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/clients/client-sts)

Updates `@aws-sdk/lib-storage` from 3.1144.0 to 3.1147.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1147.0/lib/lib-storage)

Updates `@tanstack/react-query` from 5.104.0 to 5.104.1
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/react-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/react-query@5.104.1/packages/react-query)

Updates `lucide-react` from 1.49.0 to 1.52.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.52.0/packages/lucide-react)

Updates `@tanstack/eslint-plugin-query` from 5.104.0 to 5.104.1
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/eslint-plugin-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/eslint-plugin-query@5.104.1/packages/eslint-plugin-query)

Updates `@types/node` from 24.19.0 to 24.19.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@vitejs/plugin-react` from 6.1.1 to 6.1.2
- [Release notes](https://github.com/vitejs/vite-plugin-react/releases)
- [Changelog](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite-plugin-react/commits/plugin-react@6.1.2/packages/plugin-react)

Updates `oxc-transform-react` from 0.152.0 to 0.153.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/napi/transform-react/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/crates_v0.153.0/napi/transform-react)

Updates `oxfmt` from 0.71.0 to 0.72.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.72.0/npm/oxfmt)

Updates `oxlint` from 1.86.0 to 1.87.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.87.0/npm/oxlint)

Updates `ultracite` from 7.12.2 to 7.12.4
- [Release notes](https://github.com/haydenbleasel/ultracite/releases)
- [Commits](https://github.com/haydenbleasel/ultracite/compare/ultracite@7.12.2...ultracite@7.12.4)

Updates `vite` from 8.3.1 to 8.3.3
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.3.3/packages/vite)

---
updated-dependencies:
- dependency-name: "@aws-sdk/client-acm"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-agent"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-agent-runtime"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-bedrock-runtime"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ec2"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ecr"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-ecs"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-eks"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-elastic-load-balancing-v2"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-iam"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-rds"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-s3"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/client-sts"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@aws-sdk/lib-storage"
  dependency-version: 3.1147.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/react-query"
  dependency-version: 5.104.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: lucide-react
  dependency-version: 1.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: "@tanstack/eslint-plugin-query"
  dependency-version: 5.104.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@types/node"
  dependency-version: 24.19.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: "@vitejs/plugin-react"
  dependency-version: 6.1.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: oxc-transform-react
  dependency-version: 0.153.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxfmt
  dependency-version: 0.72.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: oxlint
  dependency-version: 1.87.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: ultracite
  dependency-version: 7.12.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: vite
  dependency-version: 8.3.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 09:07:38 +11:00
Julian Sommer de4be25d80 chore: go 1.27.2 2026-10-12 08:06:57 +10:00
6002 changed files with 515611 additions and 354379 deletions
+3
View File
@@ -0,0 +1,3 @@
# The committed bundle carries the AWS SDK's template strings verbatim,
# including trailing spaces that are part of their values.
spinifex/runtime/roles/spinifexui/frontend/dist/** -whitespace
+6 -6
View File
@@ -6,10 +6,10 @@
# Set the matching branch protection rule on `dev` and `main` to
# "Require review from Code Owners" or this file is advisory only.
# Network layer is governed by ADR-0006 invariant tests.
# Network layer is governed by its invariant tests.
# Changes here must respect those clauses
# see spinifex/network/invariants/
# docs/development/proposals/az-local-architecture/0006-spinifex-network-layer-contract.md
/spinifex/network/ @joshsiv-mulga
/spinifex/vpcd/ @joshsiv-mulga
/spinifex/network/invariants/ @joshsiv-mulga
# see spinifex/domains/network/invariants/
# mulga docs/development/proposals/region-local-architecture/0006-spinifex-network-layer-contract.md (proposal, not accepted)
/spinifex/domains/network/ @joshsiv-mulga
/spinifex/runtime/roles/vpcd/ @joshsiv-mulga
/spinifex/domains/network/invariants/ @joshsiv-mulga
+1 -1
View File
@@ -8,7 +8,7 @@ Actions job summary and emits one `::error` annotation per failed test case.
```yaml
- name: Publish test summary
if: always()
uses: ./spinifex/.github/actions/junit-report # or your own checkout path
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
title: "Go E2E results"
+1 -1
View File
@@ -27,7 +27,7 @@
<line x1="510.0" y1="212.0" x2="510.0" y2="220.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
<rect x="28.0" y="224.0" width="964.0" height="94.0" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
<text x="48.0" y="251.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="700" text-anchor="start" letter-spacing="1.5">AWS GATEWAY</text>
<text x="48.0" y="272.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
<text x="48.0" y="272.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">spinifex/runtime/roles/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
<rect x="46.0" y="287.0" width="68.2" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="80.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">TLS 1.3</text>
<rect x="123.2" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

+1 -1
View File
@@ -51,7 +51,7 @@
<!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
<rect x="16" y="154" width="868" height="104" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
<text x="32" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY · port 9999</text>
<text x="876" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a" text-anchor="end">spinifex/services/awsgw/awsgw.go</text>
<text x="876" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a" text-anchor="end">spinifex/runtime/roles/awsgw/awsgw.go</text>
<!-- Two-column feature bullets -->
<g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 12 KiB

+33 -46
View File
@@ -19,8 +19,8 @@
</defs>
<rect width="900" height="480" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · multi-AZ</text>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · PUBLIC SUBNETS · SPREAD ACROSS NODES</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · server_count = 3</text>
<!-- Internet -->
<rect x="350" y="54" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
@@ -41,58 +41,45 @@
<line x1="230" y1="184" x2="230" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<line x1="670" y1="184" x2="670" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
<!-- public_a -->
<g transform="translate(108, 202)">
<rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<rect width="244" height="150" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">10.20.1.0/24</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
<rect x="20" y="70" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="85" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">NAT GW</text>
<rect x="22" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="122" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
<rect x="22" y="70" width="200" height="36" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="122" y="86" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">nginx-alb-1</text>
<text x="122" y="100" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">public IP · Ubuntu · cloud-init</text>
<rect x="22" y="110" width="200" height="36" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="122" y="126" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">nginx-alb-3</text>
<text x="122" y="140" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">public IP · Ubuntu · cloud-init</text>
</g>
<!-- public_b -->
<g transform="translate(548, 202)">
<rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<rect width="244" height="150" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">10.20.2.0/24</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="56" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="71" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
<rect x="22" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="122" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
<rect x="22" y="70" width="200" height="36" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="122" y="86" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">nginx-alb-2</text>
<text x="122" y="100" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">public IP · Ubuntu · cloud-init</text>
</g>
<!-- SNAT arrow from public_a NAT → private row -->
<line x1="230" y1="306" x2="230" y2="340" stroke="#F87171" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.65"/>
<polygon points="226,336 234,336 230,344" fill="#F87171" opacity="0.8"/>
<rect x="256" y="314" width="240" height="20" rx="10" fill="#0a0a0a" stroke="#F87171" stroke-width="0.6" stroke-opacity="0.55"/>
<text x="376" y="328" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">SNAT for private subnets (egress)</text>
<!-- private_a -->
<g transform="translate(108, 350)">
<rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_a</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-a</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_1</text>
<text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
</g>
<!-- private_b -->
<g transform="translate(548, 350)">
<rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_b</text>
<text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-b</text>
<line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_2</text>
<text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
</g>
<!-- spread placement group -->
<rect x="108" y="372" width="684" height="88" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="128" y="394" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">nginx-alb-spread</text>
<text x="772" y="394" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">placement group · strategy spread · one server per node</text>
<line x1="124" y1="402" x2="776" y2="402" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
<rect x="128" y="412" width="204" height="36" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="230" y="428" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">node1</text>
<text x="230" y="442" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">one web server</text>
<rect x="348" y="412" width="204" height="36" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="450" y="428" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">node2</text>
<text x="450" y="442" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">one web server</text>
<rect x="568" y="412" width="204" height="36" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="670" y="428" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">node3</text>
<text x="670" y="442" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">one web server</text>
</svg>

Before

Width:  |  Height:  |  Size: 7.1 KiB

After

Width:  |  Height:  |  Size: 7.5 KiB

@@ -0,0 +1,110 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="580" viewBox="0 0 900 580">
<defs>
<linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#0a0a0a"/>
<stop offset="100%" style="stop-color:#0d1117"/>
</linearGradient>
<linearGradient id="vpc-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#58a6ff14"/>
<stop offset="100%" style="stop-color:#58a6ff05"/>
</linearGradient>
<linearGradient id="node-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#3fb9501f"/>
<stop offset="100%" style="stop-color:#3fb95008"/>
</linearGradient>
<linearGradient id="svc-grad" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" style="stop-color:#a855f71f"/>
<stop offset="100%" style="stop-color:#a855f708"/>
</linearGradient>
<marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0,0 L10,5 L0,10 z" fill="#71717a"/>
</marker>
</defs>
<rect width="900" height="580" fill="url(#bg)" rx="12"/>
<text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ON VULTR · ONE NODE OR THREE</text>
<text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">--nodes 1 or 3 · vx1-g-16c-64g or bare metal</text>
<rect x="30" y="52" width="250" height="56" rx="8" fill="#e4e4e70d" stroke="#e4e4e7" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="155.0" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#e4e4e7" text-anchor="middle" font-weight="700">Your machine</text>
<text x="155.0" y="94" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">deploy-vultr.sh · OpenTofu or Terraform</text>
<rect x="325" y="52" width="250" height="56" rx="8" fill="#3fb9500d" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="450.0" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
<text x="450.0" y="94" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">customers reach your EC2 and load balancers</text>
<rect x="620" y="52" width="250" height="56" rx="8" fill="#58a6ff12" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
<text x="745.0" y="76" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#79c0ff" text-anchor="middle" font-weight="700">Vultr API</text>
<text x="745.0" y="94" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a5d6ff" text-anchor="middle">VPC · servers · reserved IPs</text>
<rect x="30" y="160" width="840" height="400" rx="10" fill="url(#vpc-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.55" stroke-dasharray="6,4"/>
<text x="50" y="184" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#79c0ff" font-weight="700">Vultr VPC</text>
<text x="850" y="184" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#58a6ff" text-anchor="end">private network 10.10.8.0/24</text>
<line x1="155" y1="108" x2="155" y2="158" stroke="#71717a" stroke-width="1.2" opacity="0.8" marker-end="url(#arrow)"/>
<text x="165" y="138" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="start">Terraform builds · SSH installs</text>
<line x1="450" y1="108" x2="450" y2="158" stroke="#71717a" stroke-width="1.2" opacity="0.8" marker-end="url(#arrow)"/>
<text x="460" y="138" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="start">reserved IPs</text>
<line x1="745" y1="158" x2="745" y2="110" stroke="#71717a" stroke-width="1.2" opacity="0.8" marker-end="url(#arrow)"/>
<text x="735" y="138" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" text-anchor="end">Spinifex calls the API</text>
<rect x="60" y="205" width="250" height="118" rx="8" fill="url(#node-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
<text x="78" y="229" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">node1</text>
<text x="292" y="229" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">Ubuntu 24.04 · Spinifex</text>
<line x1="74" y1="237" x2="296" y2="237" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.6"/>
<rect x="76" y="250" width="74" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="113.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">AWS API</text>
<rect x="160" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="190.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">EBS</text>
<rect x="230" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="260.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">S3</text>
<text x="185" y="300" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">VPC networking · your instances</text>
<rect x="325" y="205" width="250" height="118" rx="8" fill="url(#node-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" stroke-dasharray="5,3"/>
<text x="343" y="229" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">node2</text>
<text x="557" y="229" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">with --nodes 3</text>
<line x1="339" y1="237" x2="561" y2="237" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.6"/>
<rect x="341" y="250" width="74" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="378.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">AWS API</text>
<rect x="425" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="455.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">EBS</text>
<rect x="495" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="525.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">S3</text>
<text x="450" y="300" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">VPC networking · your instances</text>
<rect x="590" y="205" width="250" height="118" rx="8" fill="url(#node-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" stroke-dasharray="5,3"/>
<text x="608" y="229" font-family="'SF Mono', ui-monospace, monospace" font-size="12.5" fill="#3fb950" font-weight="700">node3</text>
<text x="822" y="229" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">with --nodes 3</text>
<line x1="604" y1="237" x2="826" y2="237" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.6"/>
<rect x="606" y="250" width="74" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="643.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">AWS API</text>
<rect x="690" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="720.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">EBS</text>
<rect x="760" y="250" width="60" height="24" rx="5" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="790.0" y="266" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle">S3</text>
<text x="715" y="300" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">VPC networking · your instances</text>
<line x1="310" y1="264" x2="325" y2="264" stroke="#58a6ff" stroke-width="1.6" stroke-dasharray="3,3" opacity="0.9"/>
<line x1="575" y1="264" x2="590" y2="264" stroke="#58a6ff" stroke-width="1.6" stroke-dasharray="3,3" opacity="0.9"/>
<text x="450" y="348" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#79c0ff" text-anchor="middle">nodes cluster over the private network: control plane, EBS replicas and S3 shards</text>
<rect x="60" y="370" width="780" height="166" rx="8" fill="url(#svc-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
<text x="80" y="394" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">AWS services across the fleet</text>
<text x="820" y="394" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">spread across the nodes; with three, moved to a survivor if one fails</text>
<line x1="76" y1="402" x2="824" y2="402" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.6"/>
<rect x="76" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="119.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">EC2</text>
<text x="119.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">instances</text>
<rect x="169" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="212.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">ALB / NLB</text>
<text x="212.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">load balancers</text>
<rect x="262" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="305.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">EKS</text>
<text x="305.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">Kubernetes</text>
<rect x="355" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="398.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">ECS</text>
<text x="398.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">containers</text>
<rect x="448" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="491.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">RDS</text>
<text x="491.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">databases</text>
<rect x="541" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="584.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">S3</text>
<text x="584.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">object storage</text>
<rect x="634" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="677.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">EBS</text>
<text x="677.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">volumes</text>
<rect x="727" y="418" width="86" height="52" rx="6" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
<text x="770.0" y="440" font-family="'SF Mono', ui-monospace, monospace" font-size="11.5" fill="#d8b4fe" text-anchor="middle" font-weight="700">IAM</text>
<text x="770.0" y="457" font-family="'Segoe UI', system-ui, sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">access</text>
<text x="450" y="500" font-family="'Segoe UI', system-ui, sans-serif" font-size="10.5" fill="#a1a1aa" text-anchor="middle">Use the AWS CLI, SDKs and Terraform against your own endpoint, unmodified.</text>
<text x="450" y="520" font-family="'Segoe UI', system-ui, sans-serif" font-size="9.5" fill="#71717a" text-anchor="middle">Public addresses are Vultr reserved IPs, attached to the node each resource runs on.</text>
</svg>

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 725 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 743 KiB

+26 -6
View File
@@ -23,11 +23,6 @@ updates:
timezone: Australia/Sydney
cooldown:
default-days: 3
ignore:
- dependency-name: "*"
update-types:
- "version-update:semver-minor"
- "version-update:semver-patch"
groups:
github-actions:
patterns:
@@ -35,7 +30,7 @@ updates:
- package-ecosystem: npm
target-branch: dev
directory: /spinifex/services/spinifexui/frontend
directory: /spinifex/runtime/roles/spinifexui/frontend
schedule:
interval: weekly
day: sunday
@@ -50,3 +45,28 @@ updates:
npm-dependencies:
patterns:
- "*"
# Base images are pinned by digest, so this is what moves them. The golang
# version is deliberate and changes with go.mod; only its digest moves here.
- package-ecosystem: docker
target-branch: dev
directories:
- /build
- /scripts
- /docs/terraform-workbooks/demo-app
schedule:
interval: weekly
day: sunday
timezone: Australia/Sydney
cooldown:
default-days: 3
ignore:
- dependency-name: golang
update-types:
- "version-update:semver-major"
- "version-update:semver-minor"
- "version-update:semver-patch"
groups:
docker-images:
patterns:
- "*"
+56 -1
View File
@@ -165,7 +165,7 @@
"runner": "ci-multi",
"extra": "",
"suite": "instancerecovery",
"timeout_minutes": 105
"timeout_minutes": 125
},
{
"cell": 32,
@@ -190,5 +190,60 @@
"extra": "",
"suite": "kvquorum",
"timeout_minutes": 55
},
{
"cell": 34,
"install": "binary",
"topo": "single",
"net": "dhcp",
"host": "oracle-10",
"arch": "x86_64",
"runner": "ci-single",
"extra": "",
"suite": ""
},
{
"cell": 35,
"install": "binary",
"topo": "single",
"net": "static",
"host": "oracle-10",
"arch": "x86_64",
"runner": "ci-single",
"extra": "reboot-single",
"suite": ""
},
{
"cell": 36,
"install": "binary",
"topo": "single",
"net": "nat",
"host": "oracle-10",
"arch": "x86_64",
"runner": "ci-single",
"extra": "nat-single",
"suite": ""
},
{
"cell": 37,
"install": "binary",
"topo": "real-multi",
"net": "dhcp",
"host": "oracle-10",
"arch": "x86_64",
"runner": "ci-multi",
"extra": "",
"suite": ""
},
{
"cell": 38,
"install": "binary",
"topo": "real-multi",
"net": "nat",
"host": "oracle-10",
"arch": "x86_64",
"runner": "ci-multi",
"extra": "nat-multi",
"suite": ""
}
]
+18
View File
@@ -5,3 +5,21 @@
## Changes
<!-- The notable changes, as bullets. -->
## Account-scoped lifecycle impact
<!-- Complete this section only when the PR changes account-scoped resource lifecycle, account termination, resource cleanup, or a lifecycle boundary. Otherwise write: N/A. -->
- Lifecycle owner and resource family:
- Owner-local ordering, recovery, retained-state, or force impact:
- Cross-domain dependency or ownership impact:
- Evidence added or rechecked:
- ADR-0008 lifecycle register / ADR-0001 migration-record / bead updates:
## Mulga design
<!-- One line, so a reviewer knows whether the design records in the private mulga repository need to change. Write either "Updated: <mulga PR or bead id>" or "Not needed: <reason>". This is a reminder, not a gate. -->
## Community Contribution License
- [ ] All community contributions in this pull request are licensed to Mulga under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). By creating this pull request I represent that I have the right to license these contributions to Mulga under the Apache License 2.0.
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Resolve the tag of each sibling repo (viperblock, predastore, northstar,
# bluebottle) a release build checks out. A release ships what go.mod requires,
# so each sibling is built at the version go.mod names, never a branch tip. A
# pseudo-version has no tag to check out, so it fails the release rather than
# silently building something else.
#
# Takes the path to spinifex's go.mod and writes viperblock=<tag>,
# predastore=<tag>, northstar=<tag> and bluebottle=<tag> to $GITHUB_OUTPUT for
# the checkout steps to consume.
set -euo pipefail
gomod="${1:?usage: $0 <path to spinifex go.mod>}"
: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
for repo in viperblock predastore northstar bluebottle; do
module="github.com/mulgadc/${repo}"
version=$(awk -v m="$module" '$1 == m { print $2; exit }' "$gomod")
if [[ -z "$version" ]]; then
echo "::error::${gomod} does not require ${module}"
exit 1
fi
if [[ ! "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ || "$version" =~ [0-9]{14}-[0-9a-f]{12}$ ]]; then
echo "::error::${module} is required at ${version}, which is not a release tag; tag ${repo} and bump go.mod before releasing"
exit 1
fi
echo "${repo}: building against ${version}"
echo "${repo}=${version}" >>"$GITHUB_OUTPUT"
done
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env python3
"""Warn when a PR body has no filled "Mulga design" line. Reads PR_BODY; never fails the job."""
import os
import re
COMMENT_RE = re.compile(r"<!--.*?-->", re.S)
SECTION_RE = re.compile(r"^#{1,6}\s*Mulga design\s*$", re.I)
LINE_RE = re.compile(r"^\s*(Updated|Not needed)\s*:\s*(.*?)\s*$", re.I)
PLACEHOLDER_RE = re.compile(r"<[^<>]*>")
def filled(body):
inside = False
for line in COMMENT_RE.sub("", body or "").splitlines():
if SECTION_RE.match(line):
inside = True
elif inside and line.lstrip().startswith("#"):
return False
elif inside:
m = LINE_RE.match(line)
if m and PLACEHOLDER_RE.sub("", m.group(2)).strip():
return True
return False
def main():
if filled(os.environ.get("PR_BODY", "")):
print("The Mulga design line is filled in.")
else:
print('::warning title=Mulga design::Add a line under "## Mulga design": "Updated: <mulga PR or bead id>" or "Not needed: <reason>".')
return 0
if __name__ == "__main__":
raise SystemExit(main())
+8 -4
View File
@@ -28,6 +28,9 @@
# that runs identically on either, and because a quota left switched on would
# cap every suite sharing the environment — so it must be proved to restore
# itself wherever it runs, not only on one node.
# route53 belongs to both topologies because every node serves every hosted
# zone from one shared bucket: on one node it proves the zone loads at all, on
# three that each node's sync reaches the same set.
# storagefault runs last in both lists and belongs to both topologies, for
# different reasons. On one node it is RS(1,0), so freezing the only predastore
# is a total outage and the guest-survival assertion is unambiguous. On three it
@@ -37,7 +40,7 @@
# received. It is last because it SIGSTOPs a cluster-wide
# service: anything sharing the environment would see the same outage and report
# a failure that says nothing about itself.
E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota storagefault"
E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota route53 storagefault"
# Suites runnable against a multi-node environment.
#
@@ -57,7 +60,7 @@ E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota
# replicated rather than sitting on one node, so it runs before the pair that
# take a node away — a cluster whose buckets are already wrong should report
# that, not report a recovery failing.
E2E_SUITES_MULTI="multinode lb cert quota kvquorum lbrecovery instancerecovery storagefault"
E2E_SUITES_MULTI="multinode lb cert quota route53 kvquorum lbrecovery instancerecovery storagefault"
# grep -xE alternation form, for narrowing a requested set to the eligible one.
E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
@@ -114,8 +117,9 @@ e2e_suite_timeout() {
# storage case waits for a pause measured at 4m47s and then holds four
# minutes to prove nothing moved. Three tests, each taking a node away and
# bringing it back, and the partition case adds a lease validity, a lease
# TTL and a boot on top of everything the other two wait for.
instancerecovery) echo "90m" ;;
# TTL and a boot on top of everything the other two wait for. The data
# test lets its guest age past the pending watchdog twice, about 20 more.
instancerecovery) echo "110m" ;;
# lbrecovery pays for a load balancer twice: once to stand one up with two
# backends placed around it, and once for the recovery itself — a stale
# owner, a lease, a boot, HAProxy's own start, and its health checks against
@@ -0,0 +1,23 @@
import unittest
from mulga_design_check import filled
class FilledTest(unittest.TestCase):
def test_accepts_either_form(self):
self.assertTrue(filled("## Mulga design\nUpdated: mulga#275"))
self.assertTrue(filled("## Mulga design\n\nnot needed: internal refactor\n## Next"))
def test_rejects_unanswered(self):
self.assertFalse(filled(""))
self.assertFalse(filled("## Mulga design\n<!-- Updated: x -->"))
self.assertFalse(filled("## Mulga design\nUpdated: <mulga PR or bead id>"))
self.assertFalse(filled("## Mulga design\nUpdated:"))
def test_scoped_to_the_section(self):
self.assertFalse(filled("Updated: mulga#1"))
self.assertFalse(filled("## Mulga design\n## Other\nUpdated: mulga#1"))
if __name__ == "__main__":
unittest.main()
+145
View File
@@ -0,0 +1,145 @@
# This file is machine-generated by `gh actions-lock`.
# Do not edit by hand; run `gh actions-lock` to update.
# Docs: https://gh.io/actions-lockfile
version: 'v0.0.3'
workflows:
'.github/workflows/apt-packages.yml':
- 'actions/checkout@v7.0.1'
'.github/workflows/dev-release.yml':
- 'actions/checkout@v7.0.1'
- 'actions/setup-go@v7.0.0'
- 'docker/setup-buildx-action@v4.4.1'
'.github/workflows/e2e-baremetal.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/e2e-cloudvendor-nightly.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/upload-artifact@v7.0.2'
- 'hashicorp/setup-terraform@v4.0.1'
'.github/workflows/e2e-cloudvendor-vultr.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/upload-artifact@v7.0.2'
- 'hashicorp/setup-terraform@v4.0.1'
'.github/workflows/e2e-ddil.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/e2e-nightly.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/download-artifact@v8.0.2'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/e2e-suites.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/setup-go@v7.0.0'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/e2e.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/download-artifact@v8.0.2'
- 'actions/setup-go@v7.0.0'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/gpu-e2e.yml':
- 'actions/checkout@v7.0.1'
- 'actions/setup-go@v7.0.0'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/image-scripts.yml':
- 'actions/checkout@v7.0.1'
- 'actions/setup-go@v7.0.0'
'.github/workflows/mulga-design.yml':
- 'actions/checkout@v7.0.1'
'.github/workflows/perf-3node.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/setup-go@v7.0.0'
- 'actions/upload-artifact@v7.0.2'
'.github/workflows/publish-system-ami.yml':
- 'actions/checkout@v7.0.1'
- 'actions/setup-go@v7.0.0'
'.github/workflows/release.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/download-artifact@v8.0.2'
- 'actions/setup-go@v7.0.0'
- 'actions/upload-artifact@v7.0.2'
- 'docker/setup-buildx-action@v4.4.1'
- 'softprops/action-gh-release@v3.0.3'
'.github/workflows/segscan-oracle.yml':
- 'actions/checkout@v7.0.1'
- 'actions/create-github-app-token@v3.3.0'
- 'actions/setup-go@v7.0.0'
'.github/workflows/spinifex.yml':
- 'actions/cache@v6.1.0'
- 'actions/checkout@v7.0.1'
- 'actions/setup-go@v7.0.0'
- 'golangci/golangci-lint-action@v9.3.0'
'.github/workflows/ui.yml':
- 'actions/checkout@v7.0.1'
- 'actions/setup-node@v7.1.0'
- 'actions/upload-artifact@v7.0.2'
- 'pnpm/action-setup@v6.1.0'
dependencies:
'actions/cache@v6.1.0':
ref: 'v6.1.0'
commit: 'sha1-55cc8345863c7cc4c66a329aec7e433d2d1c52a9'
owner_id: 44036562
repo_id: 215566462
'actions/checkout@v7.0.1':
ref: 'v7.0.1'
commit: 'sha1-3d3c42e5aac5ba805825da76410c181273ba90b1'
owner_id: 44036562
repo_id: 197814629
'actions/create-github-app-token@v3.3.0':
ref: 'v3.3.0'
commit: 'sha1-64ab729a6294000f5aa4a4bc0b73d1f32bc45510'
owner_id: 44036562
repo_id: 642580244
'actions/download-artifact@v8.0.2':
ref: 'v8.0.2'
commit: 'sha1-9000827ccba6bdab643e8b6fd33ac0654aef8333'
owner_id: 44036562
repo_id: 192626254
'actions/setup-go@v7.0.0':
ref: 'v7.0.0'
commit: 'sha1-b7ad1dad31e06c5925ef5d2fc7ad053ef454303e'
owner_id: 44036562
repo_id: 192624594
'actions/setup-node@v7.1.0':
ref: 'v7.1.0'
commit: 'sha1-949feb2413d6458794dcd2491c4babbbce0c15c1'
owner_id: 44036562
repo_id: 189476904
'actions/upload-artifact@v7.0.2':
ref: 'v7.0.2'
commit: 'sha1-cf430e030ddbb5b0abf93d22962f4752f3646cd9'
owner_id: 44036562
repo_id: 192625955
'docker/setup-buildx-action@v4.4.1':
ref: 'v4.4.1'
commit: 'sha1-f87e5991a6d7451dcb8d9637bfbc97413f497069'
owner_id: 5429470
repo_id: 288485773
'golangci/golangci-lint-action@v9.3.0':
ref: 'v9.3.0'
commit: 'sha1-ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a'
owner_id: 35628013
repo_id: 232539879
'hashicorp/setup-terraform@v4.0.1':
ref: 'v4.0.1'
commit: 'sha1-dfe3c3f87815947d99a8997f908cb6525fc44e9e'
owner_id: 761456
repo_id: 258271226
'pnpm/action-setup@v6.1.0':
ref: 'v6.1.0'
commit: 'sha1-ea17c68df8912ef543352723c149a84f56e3d413'
owner_id: 21320719
repo_id: 262289747
'softprops/action-gh-release@v3.0.3':
ref: 'v3.0.3'
commit: 'sha1-efb35369e0ad2afab669f228072c1b0d510eae64'
owner_id: 2242
repo_id: 204253808
+16 -2
View File
@@ -2,7 +2,8 @@ name: Apt Packages
# Path-filtered like image-scripts.yml: this resolves setup.sh's runtime package
# list against each supported suite's real indices, so a renamed, dropped, or
# ambiguously virtual package fails here rather than on a user's machine.
# ambiguously virtual package fails here rather than on a user's machine. The
# dnf job does the same for Oracle Linux, through setup.sh's own repo setup.
on:
push:
branches:
@@ -11,6 +12,7 @@ on:
paths:
- scripts/setup.sh
- scripts/lint-apt-packages.sh
- scripts/lint-dnf-packages.sh
- .github/workflows/apt-packages.yml
pull_request:
branches:
@@ -19,6 +21,7 @@ on:
paths:
- scripts/setup.sh
- scripts/lint-apt-packages.sh
- scripts/lint-dnf-packages.sh
- .github/workflows/apt-packages.yml
workflow_dispatch:
@@ -40,7 +43,18 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
- name: Resolve runtime package list
run: ./scripts/lint-apt-packages.sh ${{ matrix.image }}
dnf_packages:
name: Resolve dnf list (oraclelinux:10)
runs-on: ubuntu-26.04
steps:
- name: Checkout Code
uses: actions/checkout@v7.0.1
- name: Resolve runtime package list
run: ./scripts/lint-dnf-packages.sh oraclelinux:10
+13 -15
View File
@@ -34,7 +34,7 @@ jobs:
# Full history and tags, because the tag name is git describe and a shallow
# clone has no tag to describe against.
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
fetch-depth: 0
@@ -42,40 +42,41 @@ jobs:
# dev, not each repository's default branch, which is what release.yml takes.
# A dev build of spinifex against main of the others is a pairing nobody runs.
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: dev
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: dev
path: viperblock
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: dev
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: dev
path: bluebottle
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v4.4.1
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
# The tag carries the describe string so the commit is recoverable from the
# version alone, and the -dev suffix is what release.yml's own convention
@@ -101,9 +102,9 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
- name: Build microVM artifacts (lb-agent + kernel + initramfs)
- name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
if: steps.version.outputs.exists != 'true'
run: cd spinifex && make build-microvm-image
run: cd spinifex && make build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image
- name: Build amd64 Tarball
if: steps.version.outputs.exists != 'true'
@@ -143,6 +144,7 @@ jobs:
--notes-file "$RUNNER_TEMP/notes.md" \
"dist/spinifex-$TAG-linux-amd64.tar.gz" \
"dist/spinifex-$TAG-linux-amd64.tar.gz.sha256" \
dist/qboot-*.tar.gz \
"scripts/setup.sh"
# The assertion that matters. A prerelease must never become the latest
@@ -167,11 +169,7 @@ jobs:
# Kept rather than unbounded: the Worker reads the first page of releases to
# resolve the channel, so enough prereleases would eventually push the real
# ones off it, and each build carries two assets.
#
# Twenty, not five. A pinned dev tag is how an external evaluator installs a
# specific build, and five is about a week of them -- short enough that the tag
# someone was given stops resolving while they are still using it.
# ones off it. Five leaves a few recent builds to roll back to.
- name: Prune old dev prereleases
if: steps.version.outputs.exists != 'true'
env:
@@ -179,7 +177,7 @@ jobs:
run: |
set -euo pipefail
gh api "repos/$GITHUB_REPOSITORY/releases?per_page=100" \
--jq 'map(select(.prerelease and (.tag_name | endswith("-dev")))) | .[20:] | .[].tag_name' \
--jq 'map(select(.prerelease and (.tag_name | endswith("-dev")))) | .[5:] | .[].tag_name' \
> stale.txt
while read -r tag; do
[ -n "$tag" ] || continue
+8 -7
View File
@@ -77,18 +77,18 @@ jobs:
if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
ref: ${{ inputs.ref || github.ref }}
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
@@ -151,7 +151,8 @@ jobs:
fi
- name: Install go-junit-report on runner
run: go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
run: GOWORK=off go install github.com/jstemmer/go-junit-report/v2
# The eks suite's CreateCluster boots a control-plane VM from the eks-node
# AMI. Built here because libguestfs needs source and network, which the
@@ -221,20 +222,20 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: ${{ env.BM_CELL }}
- name: Analyze e2e failures
if: always()
uses: ./.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "E2E failure analysis (cell ${{ env.BM_CELL }})"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-artifacts-cell-${{ env.BM_CELL }}-${{ github.run_id }}
@@ -76,11 +76,11 @@ jobs:
- name: Put the Terraform state outside the checkout
run: echo "OCI_STATE_ROOT=$RUNNER_WORKSPACE/oci-state" >> "$GITHUB_ENV"
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
@@ -92,14 +92,14 @@ jobs:
# build-install-artifacts.sh lives in mulga. The workbooks and their driver
# are in spinifex, so only the one directory is needed here.
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
path: mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
- uses: hashicorp/setup-terraform@v4
- uses: hashicorp/setup-terraform@v4.0.1
with:
terraform_wrapper: false
@@ -273,7 +273,7 @@ jobs:
# collect step above chose, so terraform.tfstate cannot reach an artifact.
- name: Upload the run
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: e2e-oci-${{ github.run_id }}
path: |
+142
View File
@@ -0,0 +1,142 @@
name: E2E Cloud Vendor Vultr
# On demand only. Deploys a ref to Vultr with deploy-vultr.sh, runs the same
# published Terraform workbooks as the hypervisor nightly and OCI, and destroys
# the deployment, failing if anything is left billing.
on:
workflow_dispatch:
inputs:
nodes:
description: "1 or 3 nodes"
required: false
default: "1"
type: choice
options: ["1", "3"]
plan:
description: "Vultr plan: vx1-g-16c-64g (VM) or vbm-8c-64gb-amd (bare metal)"
required: false
default: vx1-g-16c-64g
region:
description: "Vultr region"
required: false
default: ord
workbooks:
description: "Space-separated workbooks (blank = the driver's full list)"
required: false
source:
description: "tree (build this ref) or release (the published installer)"
required: false
default: tree
type: choice
options: [tree, release]
channel:
description: "With source=release: latest, or dev for the newest prerelease"
required: false
default: latest
keep_on_fail:
description: "Leave a failed deployment up to inspect. It keeps billing."
required: false
type: boolean
default: false
runner:
description: "Runner label (blank = the VULTR_E2E_RUNNER variable, else ubuntu-26.04)"
required: false
permissions:
contents: read
# Deployments are named per run, but the account's reserved-IP quota is shared,
# so a second dispatch queues rather than competing with the first.
concurrency:
group: e2e-vultr
cancel-in-progress: false
jobs:
vultr:
# Hosted by default: the job only drives Terraform and SSH. The API key's
# access control must admit this runner, so a hosted runner needs a key
# whose list holds 0.0.0.0/0; a self-hosted runner can use a fixed address.
runs-on: ${{ inputs.runner || vars.VULTR_E2E_RUNNER || 'ubuntu-26.04' }}
timeout-minutes: 240
env:
VULTR_NAME: e2e-${{ github.run_id }}
VULTR_NODES: ${{ inputs.nodes }}
VULTR_PLAN: ${{ inputs.plan }}
VULTR_REGION: ${{ inputs.region }}
VULTR_SOURCE: ${{ inputs.source }}
VULTR_CHANNEL: ${{ inputs.channel }}
VULTR_ARTIFACT_DIR: ${{ github.workspace }}/e2e-vultr
SPX_REF: ${{ github.ref_name }}
steps:
- uses: actions/checkout@v7.0.1
with:
path: spinifex
fetch-depth: 0
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
# build-install-artifacts.sh lives in mulga; the workbooks and their driver
# are in spinifex, so only the one directory is needed here.
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
path: mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
- uses: hashicorp/setup-terraform@v4.0.1
with:
terraform_wrapper: false
- name: Run Vultr
working-directory: spinifex/scripts/terraform/vultr-spx
env:
VULTR_API_KEY: ${{ secrets.VULTR_API_KEY }}
MULGA_ROOT: ${{ github.workspace }}/mulga
# Through the environment, never interpolated into the script: a
# dispatch input is untrusted text to the shell.
WORKBOOKS_INPUT: ${{ inputs.workbooks }}
VULTR_KEEP_ON_FAIL: ${{ inputs.keep_on_fail && '1' || '0' }}
run: |
set -uo pipefail
if [ -n "$WORKBOOKS_INPUT" ]; then export WORKBOOKS="$WORKBOOKS_INPUT"; fi
./e2e-vultr.sh 2>&1 | tee "$RUNNER_TEMP/e2e-vultr.log"
exit "${PIPESTATUS[0]}"
# always(): a cancelled or timed-out job is the one that would leave
# servers and reserved IPs billing. destroy works from this run's state.
- name: Sweep anything left behind
if: always() && !inputs.keep_on_fail
working-directory: spinifex/scripts/terraform/vultr-spx
env:
VULTR_API_KEY: ${{ secrets.VULTR_API_KEY }}
run: ./e2e-vultr.sh --sweep-only
- name: Publish the verdict
if: always()
run: |
{
echo "### Vultr: ${VULTR_NODES} x ${VULTR_PLAN} in ${VULTR_REGION}, ${VULTR_SOURCE} build of ${SPX_REF}"
echo '```'
cat "$VULTR_ARTIFACT_DIR/verdict.txt" 2>/dev/null || echo "no verdict written"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Logs and the verdict only. The throwaway SSH key and the Terraform state
# under ~/.spinifex never reach an artifact.
- name: Upload the run
if: always()
uses: actions/upload-artifact@v7.0.2
with:
name: e2e-vultr-${{ github.run_id }}
path: |
${{ runner.temp }}/e2e-vultr.log
${{ github.workspace }}/e2e-vultr/*.log
${{ github.workspace }}/e2e-vultr/verdict.txt
${{ github.workspace }}/e2e-vultr/node*/*.log
retention-days: 14
+8 -8
View File
@@ -38,19 +38,19 @@ jobs:
- name: Clean Workspace
run: sudo rm -rf "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
name: Checkout Spinifex
with:
path: spinifex
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
name: Checkout Mulga (tofu-cluster)
with:
repository: mulgadc/mulga
@@ -81,7 +81,7 @@ jobs:
# Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
# the VMs, so a telemetry doc points back at this job's log directly.
- name: Resolve CI job id
uses: ./spinifex/.github/actions/otel-job-id
uses: $/.github/actions/otel-job-id
with:
job-name: DDIL Real Multi-Node
@@ -175,7 +175,7 @@ jobs:
- name: Publish test summary
if: always()
uses: ./spinifex/.github/actions/junit-report
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
title: "DDIL scenario results"
@@ -185,7 +185,7 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./spinifex/.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
cell: ddil
@@ -214,7 +214,7 @@ jobs:
- name: Upload DDIL Results
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: ddil-results-${{ github.run_id }}
path: ${{ runner.temp }}/ddil-results
@@ -234,7 +234,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-ddil-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
+43 -39
View File
@@ -10,7 +10,7 @@ on:
workflow_dispatch:
inputs:
cells:
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 37-38 Oracle Linux on three nodes (multi-node subset with IPsec, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
required: false
permissions:
@@ -33,6 +33,7 @@ concurrency:
env:
GIT_BRANCH: ${{ github.ref_name }}
UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-26.04.qcow2
ORACLE_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-oracle-10.qcow2
SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
@@ -43,7 +44,7 @@ jobs:
cells: ${{ steps.pick.outputs.cells }}
any: ${{ steps.pick.outputs.any }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
sparse-checkout: |
.github/e2e-cells.json
@@ -128,21 +129,21 @@ jobs:
if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
@@ -151,7 +152,7 @@ jobs:
# links each cell's telemetry to the job log that produced it. Runs
# before Bootstrap bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env.
- name: Resolve CI job id
uses: ./spinifex/.github/actions/otel-job-id
uses: $/.github/actions/otel-job-id
with:
job-name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
@@ -195,7 +196,8 @@ jobs:
# back onto the debian gold.
TOFU_VAR_ARGS=""
case "$MATRIX_HOST" in
ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE} -var=host_os=ubuntu" ;;
oracle-*) TOFU_VAR_ARGS="-var=host_image_override=${ORACLE_GOLD_IMAGE} -var=host_os=oracle" ;;
esac
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
@@ -687,7 +689,7 @@ jobs:
'echo "host: $(hostname)"; echo "loadavg: $(cat /proc/loadavg)";
grep -E "^(MemTotal|MemAvailable|SwapFree):" /proc/meminfo;
echo "cpu(steal is field 8): $(grep "^cpu " /proc/stat)";
echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"' 2>&1
echo "running qemu: $(pgrep -c "qemu-system-x86|qemu-kvm" || echo 0)"' 2>&1
} >> "${ARTIFACT_DIR}/node-load.txt"
exit 0
@@ -795,7 +797,8 @@ jobs:
- name: Install go-junit-report
if: always()
run: |
go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
GOWORK=off go -C spinifex install github.com/jstemmer/go-junit-report/v2
echo "$HOME/go/bin" >> "$GITHUB_PATH"
- name: Convert test logs to JUnit XML
@@ -812,20 +815,20 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./spinifex/.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: ${{ matrix.cell }}
- name: Analyze e2e failures
if: always()
uses: ./spinifex/.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "E2E failure analysis (cell ${{ matrix.cell }})"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
@@ -837,7 +840,7 @@ jobs:
# pulling nineteen journals it has no use for.
- name: Upload junit only
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: junit-cell-${{ matrix.cell }}-${{ github.run_id }}
path: ${{ env.ARTIFACT_DIR }}/junit-*.xml
@@ -857,7 +860,8 @@ jobs:
# different image path than apply did.
TOFU_VAR_ARGS=""
case "$MATRIX_HOST" in
ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE} -var=host_os=ubuntu" ;;
oracle-*) TOFU_VAR_ARGS="-var=host_image_override=${ORACLE_GOLD_IMAGE} -var=host_os=oracle" ;;
esac
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
@@ -868,7 +872,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-cell-${{ matrix.cell }}-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
@@ -973,20 +977,20 @@ jobs:
{ sudo mkdir -p "$E2E_ROOT/work" && sudo chown "$(id -u):$(id -g)" "$E2E_ROOT" "$E2E_ROOT/work"; }
echo "TMPDIR=$E2E_ROOT/work" >> "$GITHUB_ENV"
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
ref: ${{ env.GIT_BRANCH }}
path: mulga/spinifex
@@ -998,25 +1002,25 @@ jobs:
run: bash .github/resolve-sibling-refs.sh
working-directory: mulga/spinifex
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: mulga/viperblock
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: mulga/predastore
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: mulga/northstar
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
@@ -1151,7 +1155,7 @@ jobs:
/tmp/smoke-$GITHUB_RUN_ID.log | tail -1)
[ -n "$qemu" ] && cp "$qemu" /tmp/cell21-artifacts-$GITHUB_RUN_ID/qemu.log || true
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-artifacts-cell-21-${{ github.run_id }}
@@ -1220,20 +1224,20 @@ jobs:
# disks, which is what the tmpfs assertion below is about.
echo "WORK_DIR=$E2E_ROOT/work/cell22-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
ref: ${{ env.GIT_BRANCH }}
path: mulga/spinifex
@@ -1245,25 +1249,25 @@ jobs:
run: bash .github/resolve-sibling-refs.sh
working-directory: mulga/spinifex
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: mulga/viperblock
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: mulga/predastore
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: mulga/northstar
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
@@ -1317,7 +1321,7 @@ jobs:
cp "$WORK_DIR"/*-serial.log /tmp/cell22-artifacts-$GITHUB_RUN_ID/ 2>/dev/null ||
echo "no per-node serial logs — the run passed and cleaned up after itself"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-artifacts-cell-22-${{ github.run_id }}
@@ -1369,7 +1373,7 @@ jobs:
echo "PERF_RESULTS_ROOT=$E2E_ROOT/predastore-perf-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
@@ -1384,7 +1388,7 @@ jobs:
working-directory: spinifex
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
@@ -1437,7 +1441,7 @@ jobs:
# Report-only. The baseline comparison and the -10% gate are mulga-wpjsx;
# until the run-to-run spread is known from real nights, a threshold would
# be a guess, and a gate that cries wolf gets muted.
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: predastore-perf-${{ github.run_id }}
@@ -1496,7 +1500,7 @@ jobs:
echo "STRESS_RESULTS_ROOT=$E2E_ROOT/predastore-stress-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
@@ -1508,7 +1512,7 @@ jobs:
working-directory: spinifex
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
@@ -1563,7 +1567,7 @@ jobs:
mkdir -p "$STRESS_RESULTS_ROOT"
make e2e-stress 2>&1 | tee "$STRESS_RESULTS_ROOT/e2e-stress.log"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: predastore-stress-${{ github.run_id }}
@@ -1600,13 +1604,13 @@ jobs:
if: always()
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
sparse-checkout: .github/scripts/e2e-run-summary.py
sparse-checkout-cone-mode: false
- name: Collect junit
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
pattern: junit-cell-*-${{ github.run_id }}
path: junit
+33 -31
View File
@@ -103,25 +103,25 @@ jobs:
| sort -r | xargs -r -I{} sudo umount -l -R {} || true
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
@@ -157,7 +157,7 @@ jobs:
# the env: the suite legs share this env and its baked telemetry.env, so
# the provisioner is the only job the daemon telemetry can belong to.
- name: Resolve CI job id
uses: ./spinifex/.github/actions/otel-job-id
uses: $/.github/actions/otel-job-id
with:
job-name: Single env · provision
@@ -300,7 +300,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-single-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
@@ -323,18 +323,19 @@ jobs:
SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
ARTIFACT_DIR: /tmp/spinifex-e2e-single-${{ github.run_id }}-${{ matrix.suite }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
- name: Install go-junit-report
run: |
go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
GOWORK=off go -C spinifex install github.com/jstemmer/go-junit-report/v2
echo "$HOME/go/bin" >> "$GITHUB_PATH"
- name: Run ${{ matrix.suite }}
@@ -390,7 +391,7 @@ jobs:
- name: Publish test summary
if: always()
uses: ./spinifex/.github/actions/junit-report
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
title: "E2E ${{ matrix.suite }} results"
@@ -400,20 +401,20 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./spinifex/.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: single
- name: Analyze e2e failures
if: always()
uses: ./spinifex/.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "E2E ${{ matrix.suite }} failure analysis"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-single-${{ matrix.suite }}-${{ github.run_id }}
@@ -444,7 +445,7 @@ jobs:
# Reuse the persisted mulga/ tofu-cluster workspace from single_provision.
# Re-checkout only spinifex (for ci-fmt + diag scripts); checking out mulga
# again would wipe the local tofu state and orphan the VM.
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
@@ -467,7 +468,7 @@ jobs:
| tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
true
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-single-diag-${{ github.run_id }}
@@ -523,7 +524,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-single-td-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
@@ -571,25 +572,25 @@ jobs:
| sort -r | xargs -r -I{} sudo umount -l -R {} || true
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
@@ -606,7 +607,7 @@ jobs:
# See the matching note on single_provision.
- name: Resolve CI job id
uses: ./spinifex/.github/actions/otel-job-id
uses: $/.github/actions/otel-job-id
with:
job-name: Multi env · provision
@@ -714,7 +715,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-multi-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
@@ -737,18 +738,19 @@ jobs:
NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
ARTIFACT_DIR: /tmp/spinifex-e2e-multi-${{ github.run_id }}-${{ matrix.suite }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
- name: Install go-junit-report
run: |
go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
GOWORK=off go -C spinifex install github.com/jstemmer/go-junit-report/v2
echo "$HOME/go/bin" >> "$GITHUB_PATH"
- name: Run ${{ matrix.suite }}
@@ -804,7 +806,7 @@ jobs:
- name: Publish test summary
if: always()
uses: ./spinifex/.github/actions/junit-report
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
title: "Multi-Node E2E ${{ matrix.suite }} results"
@@ -814,20 +816,20 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./spinifex/.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: multi
- name: Analyze e2e failures
if: always()
uses: ./spinifex/.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "Multi-Node E2E ${{ matrix.suite }} failure analysis"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-multi-${{ matrix.suite }}-${{ github.run_id }}
@@ -854,7 +856,7 @@ jobs:
SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
@@ -908,7 +910,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-multi-td-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
+41 -25
View File
@@ -54,7 +54,7 @@ jobs:
single_timeout: ${{ steps.budgets.outputs.single_timeout }}
multi_timeout: ${{ steps.budgets.outputs.multi_timeout }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
sparse-checkout: .github/scripts
@@ -142,25 +142,25 @@ jobs:
# stale suite-status.txt / analysis dirs can't leak into this attempt.
rm -rf "${ARTIFACT_DIR}"
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
@@ -178,7 +178,7 @@ jobs:
# Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
# the VM, so a telemetry doc points back at this job's log directly.
- name: Resolve CI job id
uses: ./spinifex/.github/actions/otel-job-id
uses: $/.github/actions/otel-job-id
with:
job-name: Single-Node E2E
@@ -372,7 +372,8 @@ jobs:
- name: Install go-junit-report on runner
run: |
go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
GOWORK=off go -C spinifex install github.com/jstemmer/go-junit-report/v2
echo "$HOME/go/bin" >> "$GITHUB_PATH"
- name: Ship test binaries to VM
@@ -512,7 +513,7 @@ jobs:
- name: Publish test summary
if: always()
uses: ./spinifex/.github/actions/junit-report
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
title: "E2E results"
@@ -522,20 +523,27 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./spinifex/.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: single
- name: Analyze e2e failures
if: always()
uses: ./spinifex/.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "E2E failure analysis"
- uses: actions/upload-artifact@v7
# The WAN capture is most of the artifact and only matters when a run
# failed. Anything short of a clean pass (suite rc, a failed setup or
# collect step, a missing rc) keeps it.
- name: Drop the packet capture from a passing single-node run
if: always() && steps.e2e.outputs.rc == '0' && job.status == 'success'
run: rm -f "${ARTIFACT_DIR}"/net-diag/wan.pcap
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-artifacts-${{ github.run_id }}
@@ -579,7 +587,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-single-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
@@ -642,25 +650,25 @@ jobs:
# stale suite-status.txt / analysis dirs can't leak into this attempt.
rm -rf "${ARTIFACT_DIR}"
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
@@ -677,7 +685,7 @@ jobs:
# See the matching note on e2e_single.
- name: Resolve CI job id
uses: ./spinifex/.github/actions/otel-job-id
uses: $/.github/actions/otel-job-id
with:
job-name: Multi-Node E2E
@@ -758,7 +766,8 @@ jobs:
- name: Install go-junit-report on runner
run: |
go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
GOWORK=off go -C spinifex install github.com/jstemmer/go-junit-report/v2
echo "$HOME/go/bin" >> "$GITHUB_PATH"
- name: Ship test binaries to node1
@@ -965,7 +974,7 @@ jobs:
- name: Publish test summary
if: always()
uses: ./spinifex/.github/actions/junit-report
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
title: "Multi-Node E2E results"
@@ -975,20 +984,27 @@ jobs:
- name: Ship test results to Elasticsearch
if: always()
continue-on-error: true
uses: ./spinifex/.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: multi
- name: Analyze e2e failures
if: always()
uses: ./spinifex/.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "Multi-Node E2E failure analysis"
- uses: actions/upload-artifact@v7
# The WAN captures are 90% of the artifact and only matter when a run
# failed. Anything short of a clean pass (suite rc, a failed setup or
# collect step, a missing rc) keeps them.
- name: Drop the packet captures from a passing multi-node run
if: always() && steps.e2e.outputs.rc == '0' && job.status == 'success'
run: rm -f "${ARTIFACT_DIR}"/net-diag-node*/wan.pcap
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: e2e-multi-artifacts-${{ github.run_id }}
@@ -1035,7 +1051,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-multi-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
@@ -1105,14 +1121,14 @@ jobs:
# the aggregation script below treats an absent suite-status.txt as
# "no failures from that leg".
- name: Download single-node artifacts
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
continue-on-error: true
with:
name: e2e-artifacts-${{ github.run_id }}
path: ${{ github.workspace }}/es-summary-single
- name: Download multi-node artifacts
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
continue-on-error: true
with:
name: e2e-multi-artifacts-${{ github.run_id }}
+8 -7
View File
@@ -95,10 +95,10 @@ jobs:
if: steps.gate.outputs.paused != 'true'
run: sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
if: steps.gate.outputs.paused != 'true'
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
if: steps.gate.outputs.paused != 'true'
with:
go-version-file: go.mod
@@ -187,7 +187,8 @@ jobs:
- name: Install go-junit-report on runner
if: steps.gate.outputs.paused != 'true'
run: |
go install github.com/jstemmer/go-junit-report/v2@latest
# Version from go.mod's tool directive, checked against go.sum.
GOWORK=off go install github.com/jstemmer/go-junit-report/v2
echo "$HOME/go/bin" >> "$GITHUB_PATH"
- name: Ship GPU e2e test binary to wattle
@@ -243,7 +244,7 @@ jobs:
- name: Publish test summary
if: always() && steps.gate.outputs.paused != 'true'
uses: ./.github/actions/junit-report
uses: $/.github/actions/junit-report
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
title: "GPU E2E results"
@@ -251,20 +252,20 @@ jobs:
- name: Ship test results to Elasticsearch
if: always() && steps.gate.outputs.paused != 'true'
continue-on-error: true
uses: ./.github/actions/junit-to-es
uses: $/.github/actions/junit-to-es
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
cell: gpu
- name: Analyze e2e failures
if: always() && steps.gate.outputs.paused != 'true'
uses: ./.github/actions/e2e-analyze
uses: $/.github/actions/e2e-analyze
with:
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
log-dir: ${{ env.ARTIFACT_DIR }}
title: "GPU E2E failure analysis"
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always() && steps.gate.outputs.paused != 'true'
with:
name: gpu-e2e-artifacts-${{ github.run_id }}
+6 -3
View File
@@ -29,13 +29,16 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
- name: Run image script tests + shellcheck
run: make test-images
eks_mkosi_image:
name: Build mkosi EKS image
# A fork PR's code must never run on a persistent self-hosted runner: it
# shares disk, caches and credentials with the release jobs.
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: [self-hosted, ci-single]
concurrency:
group: eks-mkosi-image-${{ github.ref }}
@@ -43,10 +46,10 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: "go.mod"
# Self-hosted: the module + build cache already persists on local
+33
View File
@@ -0,0 +1,33 @@
name: Mulga design
# Reminds the author to say whether the Mulga design records need to change.
# It reads only the pull request body, uses no secrets, and warns without ever
# failing: the check is a prompt, not a gate.
on:
pull_request:
branches:
- dev
types: [opened, edited, reopened]
permissions:
contents: read
jobs:
mulga_design:
name: Mulga design line
if: ${{ !endsWith(github.actor, '[bot]') }}
runs-on: ubuntu-26.04
steps:
- name: Checkout Code
uses: actions/checkout@v7.0.1
with:
sparse-checkout: .github/scripts
- name: Check the Mulga design line
env:
PR_BODY: ${{ github.event.pull_request.body }}
working-directory: .github/scripts
run: |
python3 -m unittest -q test_mulga_design_check
python3 mulga_design_check.py
+7 -7
View File
@@ -80,25 +80,25 @@ jobs:
echo "TMPDIR=$E2E_ROOT/work" >> "$GITHUB_ENV"
echo "PERF_RESULTS_ROOT=$E2E_ROOT/predastore-perf-3node-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
sparse-checkout: scripts/tofu-cluster
path: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
path: spinifex
- uses: actions/setup-go@v7
- uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
@@ -116,7 +116,7 @@ jobs:
run: bash .github/resolve-sibling-refs.sh
working-directory: spinifex
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
id: predastore
with:
repository: mulgadc/predastore
@@ -366,7 +366,7 @@ jobs:
# Report-only, matching cell-23: the baseline comparison and any
# regression gate are future work once run-to-run spread is known from
# real bare-metal runs.
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
name: predastore-perf-3node-${{ github.run_id }}
@@ -407,7 +407,7 @@ jobs:
- name: Upload tofu log
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: tofu-log-perf3node-${{ github.run_id }}
path: ${{ env.TOFU_LOG }}
+18 -30
View File
@@ -17,28 +17,12 @@ name: Publish System AMIs
# spx admin images import --name spinifex-rds-postgres
# spx admin images import --name spinifex-rds-mariadb
#
# Publishing overwrites the mirror artifact, so it is a deliberate action, not
# something every push should trigger: it runs manually, or from release.yml
# once a tag has passed its gates. The `variant` input scopes which images
# build — the Ubuntu GPU legs are 16G and much slower (NVIDIA driver download +
# DKMS build), so publishing only the CPU AMIs stays cheap.
# Publishing overwrites the mirror artifact, so it runs only from release.yml
# once a tag has passed its gates, with siblings at the tags go.mod requires.
# The `variant` input scopes which images build — the Ubuntu GPU legs are 16G
# and much slower (NVIDIA driver download + DKMS build).
on:
workflow_dispatch:
inputs:
variant:
description: "Which images to build"
type: choice
default: all
options:
- all
- cpu
- gpu
- rds
dry_run:
description: "Build + convert only; skip the R2 upload"
type: boolean
default: false
workflow_call:
inputs:
variant:
@@ -94,40 +78,44 @@ jobs:
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
- name: Resolve sibling release tags from go.mod
id: siblings
run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: dev
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: dev
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: dev
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: dev
ref: ${{ steps.siblings.outputs.bluebottle }}
path: bluebottle
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: "spinifex/go.mod"
# Self-hosted runner: the Go module + build cache already persists on
+68 -34
View File
@@ -5,51 +5,60 @@ on:
tags: [ "v*" ]
permissions:
contents: write
contents: read
jobs:
distro-amd64:
runs-on: ubuntu-26.04
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
- name: Resolve sibling release tags from go.mod
id: siblings
run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
path: bluebottle
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v4.4.1
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
- name: Build microVM artifacts (lb-agent + kernel + initramfs)
run: cd spinifex && make build-microvm-image
- name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
run: cd spinifex && make build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image
- name: Build amd64 Tarball
env:
@@ -57,48 +66,57 @@ jobs:
run: cd spinifex && make distro-amd64 VERSION="$VERSION"
- name: Upload amd64 Artifacts
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: distro-amd64
path: |
spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz
spinifex/dist/spinifex-${{ github.ref_name }}-linux-amd64.tar.gz.sha256
spinifex/dist/qboot-*.tar.gz
spinifex/scripts/setup.sh
distro-arm64:
runs-on: ubuntu-26.04-arm
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
- name: Resolve sibling release tags from go.mod
id: siblings
run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
path: bluebottle
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v4.4.1
- name: Build arm64 Tarball
env:
@@ -106,7 +124,7 @@ jobs:
run: cd spinifex && make distro-arm64 VERSION="$VERSION"
- name: Upload arm64 Artifacts
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: distro-arm64
path: |
@@ -116,21 +134,23 @@ jobs:
release:
needs: [ distro-amd64, distro-arm64 ]
runs-on: ubuntu-26.04
permissions:
contents: write
steps:
- name: Download amd64 Artifacts
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
name: distro-amd64
path: artifacts
- name: Download arm64 Artifacts
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
name: distro-arm64
path: artifacts
- name: Create Draft GitHub Release
uses: softprops/action-gh-release@v3
uses: softprops/action-gh-release@v3.0.3
with:
draft: true
files: |
@@ -140,7 +160,7 @@ jobs:
generate_release_notes: true
- name: Upload Tarball for Verify Job
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v7.0.2
with:
name: release-assets
path: artifacts/
@@ -167,14 +187,14 @@ jobs:
| sort -r | xargs -r -I{} sudo umount -l -R {} || true
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
repositories: mulga
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
@@ -183,13 +203,13 @@ jobs:
# Only the shared shell helpers: this job builds nothing from source, but
# it drives the same libvirt provider the e2e cells retry around.
- uses: actions/checkout@v7
- uses: actions/checkout@v7.0.1
with:
sparse-checkout: .github/scripts
path: spinifex
- name: Download Release Assets
uses: actions/download-artifact@v8
uses: actions/download-artifact@v8.0.2
with:
name: release-assets
path: release-assets
@@ -231,6 +251,8 @@ jobs:
promote:
needs: [ verify, baremetal ]
runs-on: ubuntu-26.04
permissions:
contents: write
env:
TAG: ${{ github.ref_name }}
REPO: ${{ github.repository }}
@@ -263,6 +285,8 @@ jobs:
needs: [ verify, baremetal ]
if: failure()
runs-on: ubuntu-26.04
permissions:
contents: write
env:
TAG: ${{ github.ref_name }}
REPO: ${{ github.repository }}
@@ -280,6 +304,8 @@ jobs:
iso:
needs: release
runs-on: [ self-hosted, ci-single ]
permissions:
contents: write
# ISO build (~60-90m) overlaps the parallel baremetal run (~120m); the
# upload gate then waits for baremetal to finish, so the job must outlast it.
timeout-minutes: 180
@@ -294,7 +320,7 @@ jobs:
sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
@@ -302,37 +328,45 @@ jobs:
repositories: mulga
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
ref: ${{ github.ref }}
- name: Resolve sibling release tags from go.mod
id: siblings
run: bash spinifex/.github/resolve-release-sibling-refs.sh spinifex/go.mod
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
path: bluebottle
- name: Checkout Mulga iso-builder
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
@@ -343,7 +377,7 @@ jobs:
run: sudo make -C mulga/scripts/iso-builder clean
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: spinifex/go.mod
cache: false
+9 -9
View File
@@ -3,7 +3,7 @@
#
# tests/integration/segscan_oracle_test.go needs the mulga umbrella repo's
# scripts/segscan checked out alongside spinifex (it is a separate Go module,
# `package main`, and cannot be imported — see spinifex/testutil/segscanoracle).
# `package main`, and cannot be imported — see internal/testkit/segscanoracle).
# spinifex.yml's jobs check out only spinifex/viperblock/predastore and run on
# every PR; adding a second-repo checkout there would make every spinifex PR
# depend on mulga main being healthy and pay that checkout cost on every run,
@@ -25,7 +25,7 @@ jobs:
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
fetch-depth: 2
@@ -38,34 +38,34 @@ jobs:
working-directory: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
path: bluebottle
- uses: actions/create-github-app-token@v3
- uses: actions/create-github-app-token@v3.3.0
id: app-token
with:
client-id: ${{ secrets.APP_ID }}
@@ -73,7 +73,7 @@ jobs:
repositories: mulga
- name: Checkout Mulga (scripts/segscan only)
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/mulga
token: ${{ steps.app-token.outputs.token }}
@@ -81,7 +81,7 @@ jobs:
path: mulga
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
cache-dependency-path: "spinifex/go.sum"
go-version-file: "spinifex/go.mod"
+29 -32
View File
@@ -27,7 +27,7 @@ jobs:
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
fetch-depth: 2
@@ -40,28 +40,28 @@ jobs:
working-directory: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
@@ -71,7 +71,7 @@ jobs:
# the last go.sum change and never refreshed it. Both caches are managed
# below instead.
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: "spinifex/go.mod"
cache: false
@@ -80,7 +80,7 @@ jobs:
# setup-go hashed spinifex/go.sum alone, which left the sibling modules'
# dependencies out of the key and re-downloading on every run.
- name: Cache Go modules
uses: actions/cache@v6
uses: actions/cache@v6.1.0
with:
path: ~/go/pkg/mod
key: gomod-${{ hashFiles('spinifex/go.sum', 'viperblock/go.sum', 'predastore/go.sum', 'northstar/go.sum', 'bluebottle/go.sum') }}
@@ -93,7 +93,7 @@ jobs:
# a PR reads the base branch's entry without spending a minute uploading
# one only that PR would ever read.
- name: Restore Go build cache
uses: actions/cache/restore@v6
uses: actions/cache/restore@v6.1.0
with:
path: ~/.cache/go-build
key: gobuild-${{ github.job }}-${{ github.run_id }}
@@ -119,9 +119,6 @@ jobs:
- name: Build Spinifex
run: make -C spinifex build
- name: Manifest check + drift lint
run: make -C spinifex manifest-check manifest-lint
- name: Run Tests with Coverage
env:
SPINIFEX_REQUIRE_CONFORMANCE_TOOLS: "1"
@@ -150,7 +147,7 @@ jobs:
- name: Save Go build cache
if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
uses: actions/cache/save@v6
uses: actions/cache/save@v6.1.0
with:
path: ~/.cache/go-build
key: gobuild-${{ github.job }}-${{ github.run_id }}
@@ -161,7 +158,7 @@ jobs:
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
@@ -173,28 +170,28 @@ jobs:
working-directory: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
@@ -204,7 +201,7 @@ jobs:
# the last go.sum change and never refreshed it. Both caches are managed
# below instead.
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: "spinifex/go.mod"
cache: false
@@ -213,7 +210,7 @@ jobs:
# setup-go hashed spinifex/go.sum alone, which left the sibling modules'
# dependencies out of the key and re-downloading on every run.
- name: Cache Go modules
uses: actions/cache@v6
uses: actions/cache@v6.1.0
with:
path: ~/go/pkg/mod
key: gomod-${{ hashFiles('spinifex/go.sum', 'viperblock/go.sum', 'predastore/go.sum', 'northstar/go.sum', 'bluebottle/go.sum') }}
@@ -226,7 +223,7 @@ jobs:
# a PR reads the base branch's entry without spending a minute uploading
# one only that PR would ever read.
- name: Restore Go build cache
uses: actions/cache/restore@v6
uses: actions/cache/restore@v6.1.0
with:
path: ~/.cache/go-build
key: gobuild-${{ github.job }}-${{ github.run_id }}
@@ -251,7 +248,7 @@ jobs:
- name: Save Go build cache
if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
uses: actions/cache/save@v6
uses: actions/cache/save@v6.1.0
with:
path: ~/.cache/go-build
key: gobuild-${{ github.job }}-${{ github.run_id }}
@@ -262,7 +259,7 @@ jobs:
steps:
- name: Checkout Spinifex
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
path: spinifex
fetch-depth: 2
@@ -275,28 +272,28 @@ jobs:
working-directory: spinifex
- name: Checkout Viperblock
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/viperblock
ref: ${{ steps.siblings.outputs.viperblock }}
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/predastore
ref: ${{ steps.siblings.outputs.predastore }}
path: predastore
- name: Checkout Northstar
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/northstar
ref: ${{ steps.siblings.outputs.northstar }}
path: northstar
- name: Checkout Bluebottle
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
repository: mulgadc/bluebottle
ref: ${{ steps.siblings.outputs.bluebottle }}
@@ -306,7 +303,7 @@ jobs:
# the last go.sum change and never refreshed it. Both caches are managed
# below instead.
- name: Setup Go
uses: actions/setup-go@v7
uses: actions/setup-go@v7.0.0
with:
go-version-file: "spinifex/go.mod"
cache: false
@@ -315,7 +312,7 @@ jobs:
# setup-go hashed spinifex/go.sum alone, which left the sibling modules'
# dependencies out of the key and re-downloading on every run.
- name: Cache Go modules
uses: actions/cache@v6
uses: actions/cache@v6.1.0
with:
path: ~/go/pkg/mod
key: gomod-${{ hashFiles('spinifex/go.sum', 'viperblock/go.sum', 'predastore/go.sum', 'northstar/go.sum', 'bluebottle/go.sum') }}
@@ -328,7 +325,7 @@ jobs:
# a PR reads the base branch's entry without spending a minute uploading
# one only that PR would ever read.
- name: Restore Go build cache
uses: actions/cache/restore@v6
uses: actions/cache/restore@v6.1.0
with:
path: ~/.cache/go-build
key: gobuild-${{ github.job }}-${{ github.run_id }}
@@ -338,9 +335,9 @@ jobs:
run: go work init ./spinifex ./viperblock ./predastore ./northstar ./bluebottle
- name: Lint
uses: golangci/golangci-lint-action@v9
uses: golangci/golangci-lint-action@v9.3.0
with:
version: v2.13.0
version: v2.14.0
working-directory: spinifex
- name: Govulncheck
@@ -363,7 +360,7 @@ jobs:
- name: Save Go build cache
if: ${{ !cancelled() && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') }}
uses: actions/cache/save@v6
uses: actions/cache/save@v6.1.0
with:
path: ~/.cache/go-build
key: gobuild-${{ github.job }}-${{ github.run_id }}
+113 -12
View File
@@ -6,14 +6,24 @@ on:
- main
- dev
paths:
- spinifex/services/spinifexui/frontend/**
- spinifex/runtime/roles/spinifexui/frontend/**
- spinifex/runtime/roles/spinifexui/*.go
- spinifex/operator/admin/instance_metrics*.go
- spinifex/operator/admin/live_streams*.go
- spinifex/runtime/roles/qmpcollector/**
- contracts/telemetry/**
- .github/workflows/ui.yml
pull_request:
branches:
- main
- dev
paths:
- spinifex/services/spinifexui/frontend/**
- spinifex/runtime/roles/spinifexui/frontend/**
- spinifex/runtime/roles/spinifexui/*.go
- spinifex/operator/admin/instance_metrics*.go
- spinifex/operator/admin/live_streams*.go
- spinifex/runtime/roles/qmpcollector/**
- contracts/telemetry/**
- .github/workflows/ui.yml
workflow_dispatch:
@@ -26,25 +36,25 @@ jobs:
runs-on: ubuntu-26.04
defaults:
run:
working-directory: spinifex/services/spinifexui/frontend
working-directory: spinifex/runtime/roles/spinifexui/frontend
steps:
- name: Checkout Code
uses: actions/checkout@v7
uses: actions/checkout@v7.0.1
with:
sparse-checkout: spinifex/services/spinifexui/frontend
sparse-checkout: spinifex/runtime/roles/spinifexui/frontend
- name: Setup pnpm
uses: pnpm/action-setup@v6
uses: pnpm/action-setup@v6.1.0
with:
package_json_file: spinifex/services/spinifexui/frontend/package.json
package_json_file: spinifex/runtime/roles/spinifexui/frontend/package.json
- name: Setup Node.js
uses: actions/setup-node@v7
uses: actions/setup-node@v7.1.0
with:
cache: "pnpm"
cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml
node-version-file: spinifex/services/spinifexui/frontend/.nvmrc
cache-dependency-path: spinifex/runtime/roles/spinifexui/frontend/pnpm-lock.yaml
node-version-file: spinifex/runtime/roles/spinifexui/frontend/.nvmrc
- name: Install Dependencies
run: pnpm install --frozen-lockfile
@@ -58,5 +68,96 @@ jobs:
- name: Build Frontend
run: pnpm build
- name: Check for generated route tree drift
run: git diff --exit-code -- src/routeTree.gen.ts
# spx embeds the committed dist/, so a bundle that differs from a clean
# build of this source would ship JavaScript CI never built. Porcelain
# rather than git diff, so added and deleted chunks count too.
- name: Check for generated file drift
run: |
drift=$(git status --porcelain -- src/routeTree.gen.ts src/lib/cn-tables.gen.ts dist)
if [ -n "$drift" ]; then
echo "::error::Generated files differ from a clean build. Run pnpm build and commit the result."
echo "$drift"
exit 1
fi
# The pinned image lives in one file that scripts/ui-playwright.sh also reads,
# so make ui-baselines and the job below draw screenshots in the same image.
ui_e2e_image:
name: UI End-to-End Image
runs-on: ubuntu-26.04
outputs:
image: ${{ steps.read.outputs.image }}
steps:
- name: Checkout Code
uses: actions/checkout@v7.0.1
with:
sparse-checkout: spinifex/runtime/roles/spinifexui/frontend/playwright.image
sparse-checkout-cone-mode: false
- name: Read Playwright Image
id: read
run: echo "image=$(tr -d '[:space:]' < spinifex/runtime/roles/spinifexui/frontend/playwright.image)" >> "$GITHUB_OUTPUT"
# The browser suite: the console against a faked API on Vite dev, plus the
# built dist under the production CSP. It runs in Playwright's own image, which
# carries the browsers, so screenshots rasterise as they did for the baselines.
ui_e2e:
name: UI End-to-End
needs: ui_e2e_image
runs-on: ubuntu-26.04
timeout-minutes: 20
container:
image: ${{ needs.ui_e2e_image.outputs.image }}
options: --ipc=host
defaults:
run:
working-directory: spinifex/runtime/roles/spinifexui/frontend
steps:
- name: Checkout Code
uses: actions/checkout@v7.0.1
with:
sparse-checkout: spinifex/runtime/roles/spinifexui/frontend
- name: Setup pnpm
uses: pnpm/action-setup@v6.1.0
with:
package_json_file: spinifex/runtime/roles/spinifexui/frontend/package.json
- name: Setup Node.js
uses: actions/setup-node@v7.1.0
with:
cache: "pnpm"
cache-dependency-path: spinifex/runtime/roles/spinifexui/frontend/pnpm-lock.yaml
node-version-file: spinifex/runtime/roles/spinifexui/frontend/.nvmrc
- name: Install Dependencies
run: pnpm install --frozen-lockfile
# The image's browsers belong to one Playwright release, so a lockfile
# bump without an image bump would otherwise fail on a missing executable.
- name: Check Playwright Matches the Image
env:
IMAGE: ${{ needs.ui_e2e_image.outputs.image }}
run: |
version=$(pnpm exec playwright --version | awk '{print $2}')
case "$IMAGE" in
*":v$version-"*) ;;
*) echo "::error::Lockfile has Playwright $version but the image is $IMAGE. Update playwright.image."; exit 1 ;;
esac
- name: Build Frontend
run: pnpm build
- name: Run End-to-End Suite
run: pnpm e2e
- name: Upload Screenshot Diffs
if: failure()
uses: actions/upload-artifact@v7.0.2
with:
name: ui-e2e-test-results
path: spinifex/runtime/roles/spinifexui/frontend/test-results/
if-no-files-found: ignore
retention-days: 14
+10 -1
View File
@@ -19,7 +19,16 @@ bin/
dist/
build/microvm/vmlinuz
build/microvm/initramfs.cpio.gz
!spinifex/services/spinifexui/frontend/dist/
build/microvm/qboot.rom
build/microvm/qboot.LICENSE
build/microvm/sqs-store/vmlinuz
build/microvm/sqs-store/initramfs.cpio.gz
build/microvm/sqs-store/.stage.*/
build/microvm/sqs-worker/vmlinuz
build/microvm/sqs-worker/initramfs.cpio.gz
build/microvm/sqs-worker/.stage.*/
build/qboot/
!spinifex/runtime/roles/spinifexui/frontend/dist/
tmp/
*.test
*.out
+4 -3
View File
@@ -71,17 +71,18 @@ linters:
- pkg: math/rand$
desc: Use math/rand/v2 instead, see https://go.dev/blog/randv2
# The EC2 control plane reaches block storage through ebsprovider, not
# The control plane reaches block storage through the EBS provider, not
# by importing the engine. viperblock.New starts a chunk uploader, so
# every construction is a writer against a volume viperblockd serves.
# Nothing under handlers/ may import viperblock, in any form, tests
# No domain or legacy handler may import viperblock, in any form, tests
# included. A types-only import counts: one became an engine import.
ec2-control-plane:
files:
- '**/spinifex/domains/**'
- '**/spinifex/handlers/**'
deny:
- pkg: github.com/mulgadc/viperblock
desc: Use the ebsprovider.EBSProvider interface; the control plane must not construct or name viperblock types.
desc: Use the providers/ebs EBSProvider interface; the control plane must not construct or name viperblock types.
embeddedstructfieldcheck:
forbid-mutex: true
+5 -4
View File
@@ -16,6 +16,7 @@ Every AWS call enters through a SigV4-authenticated gateway, is published to NAT
| [`docs/COMMANDS.md`](docs/COMMANDS.md) | The `spx admin` CLI and the systemd units it manages. |
| [`docs/coverage/`](docs/coverage/README.md) | Which AWS API operations each service implements. |
| [`scripts/README.md`](scripts/README.md) | Install, dev-environment, and image-build scripts. |
| [`docs/PACKAGE_BOUNDARY_MIGRATION.md`](docs/PACKAGE_BOUNDARY_MIGRATION.md) | ADR-0001 implementation inventory and incremental source moves. |
User-facing docs are published at [docs.mulgadc.com](https://docs.mulgadc.com).
@@ -53,14 +54,14 @@ AWS_PROFILE=spinifex aws ec2 describe-instance-types
A list of instance types means the cluster is up.
`make preflight` also needs `shellcheck` and `golangci-lint` (CI pins v2.13.0):
`make preflight` also needs `shellcheck` and `golangci-lint`
```bash
sudo apt-get install -y shellcheck
curl -sSfL https://golangci-lint.run/install.sh | sh -s -- -b "$(go env GOPATH)/bin" v2.13.0
curl -sSfL https://golangci-lint.run/install.sh | sh -s -- -b "$(go env GOPATH)/bin" v2.14.0
```
The web console additionally needs Node.js 24 and `pnpm`.
The web console additionally needs `pnpm` and the Node.js version pinned in `spinifex/runtime/roles/spinifexui/frontend/.nvmrc`.
## Development Workflow
@@ -75,7 +76,7 @@ The web console additionally needs Node.js 24 and `pnpm`.
Follow the services with `journalctl -u 'spinifex-*' -f`.
The web console lives in `spinifex/services/spinifexui/frontend/` (`pnpm install`, `pnpm dev`, `pnpm lint`, `pnpm test`). `spx` embeds its checked-in `dist/`, so run `make build-ui` and commit the rebuilt `dist/` with your change.
The web console lives in `spinifex/runtime/roles/spinifexui/frontend/` (`pnpm install`, `pnpm dev`, `pnpm lint`, `pnpm test`). `spx` embeds its checked-in `dist/`, so run `make build-ui` and commit the rebuilt `dist/` with your change.
## Creating a Pull Request
+116 -63
View File
@@ -54,12 +54,25 @@ build: go_build build-installer build-lb-agent generate-aws-model-coverage
# Build spinifex-ui frontend (requires pnpm)
build-ui:
@echo -e "\n....Building spinifex-ui frontend...."
cd spinifex/services/spinifexui/frontend && pnpm build
cd spinifex/runtime/roles/spinifexui/frontend && pnpm build
# Playwright suite for the console, against Vite dev and the built dist under
# the console's CSP. Not part of preflight; needs `pnpm exec playwright install chromium`.
test-ui-e2e: build-ui
cd spinifex/runtime/roles/spinifexui/frontend && pnpm e2e
# The same suite inside the pinned Playwright image CI uses (podman or docker),
# so screenshots compare as they will on the runner. ui-baselines redraws them.
ui-e2e-container:
scripts/ui-playwright.sh compare
ui-baselines:
scripts/ui-playwright.sh baselines
# GO commands
VERSION ?= $(shell git describe --tags --always --dirty --exclude '*-dev')
COMMIT ?= $(shell git rev-parse --short HEAD)
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/spinifex/operator/cli.Version=$(VERSION) -X github.com/mulgadc/spinifex/spinifex/operator/cli.Commit=$(COMMIT)
go_build:
@echo -e "\n....Building $(GO_PROJECT_NAME)"
@@ -140,20 +153,74 @@ $(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
bin/lb-agent:
$(MAKE) build-lb-agent
build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
# The SQS store microVM image: a separate artifact from the LB image above, not
# installed or released by any target yet.
SQS_STORE_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-store
SQS_STORE_MICROVM_ARTIFACTS := $(SQS_STORE_MICROVM_OUT_DIR)/vmlinuz $(SQS_STORE_MICROVM_OUT_DIR)/initramfs.cpio.gz
SQS_STORE_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_STORE_MICROVM_OUT_DIR)/init.sh $(SQS_STORE_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_STORE_MICROVM_OUT_DIR)/,sqs-store-lib.sh sqs-store-prepare sqs-store-run sqs-store-stop) bin/nats-server
install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
$(SQS_STORE_MICROVM_ARTIFACTS) &: $(SQS_STORE_MICROVM_INPUTS)
./scripts/build-sqs-store-microvm-image.sh
# The dedicated SQS store server, built from the go.mod-pinned module so the
# image ships exactly the reviewed version. Static for the musl guest.
bin/nats-server: go.mod go.sum
@mkdir -p bin
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ github.com/nats-io/nats-server/v2
build-sqs-store-microvm-image: $(SQS_STORE_MICROVM_ARTIFACTS) ## Build the SQS store microVM kernel and initramfs
# The shared SQS worker microVM image: the store image's /init and kernel with
# the worker binary instead of nats-server. Not installed or released yet.
SQS_WORKER_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-worker
SQS_WORKER_MICROVM_ARTIFACTS := $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz
SQS_WORKER_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_STORE_MICROVM_OUT_DIR)/init.sh $(SQS_WORKER_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_WORKER_MICROVM_OUT_DIR)/,sqs-worker-lib.sh sqs-worker-prepare sqs-worker-ready sqs-worker-stop) bin/sqs-worker
$(SQS_WORKER_MICROVM_ARTIFACTS) &: $(SQS_WORKER_MICROVM_INPUTS)
SQS_MICROVM_PROFILE=worker ./scripts/build-sqs-store-microvm-image.sh
bin/sqs-worker: go.mod go.sum $(shell find cmd/sqs-worker spinifex/agents/sqs -name '*.go' ! -name '*_test.go')
@mkdir -p bin
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/sqs-worker
build-sqs-worker-microvm-image: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Build the shared SQS worker microVM kernel and initramfs
# qboot is the microvm firmware. The source tarball ships with each release
# because qboot is GPL-2.0.
QBOOT_COMMIT := $(shell sed -n 's/^QBOOT_COMMIT="\(.*\)"$$/\1/p' scripts/build-qboot.sh)
QBOOT_SOURCE := build/qboot/qboot-$(QBOOT_COMMIT).tar.gz
QBOOT_ARTIFACTS := $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE $(QBOOT_SOURCE)
$(QBOOT_ARTIFACTS) &: scripts/build-qboot.sh
./scripts/build-qboot.sh
build-microvm-image: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Build microVM kernel, initramfs and qboot firmware (incremental — skips when up to date)
install-sqs-store-microvm: $(SQS_STORE_MICROVM_ARTIFACTS) ## Install the SQS store microVM image to /usr/share/spinifex/microvm/sqs-store/
sudo install -d /usr/share/spinifex/microvm/sqs-store
sudo rm -f /usr/share/spinifex/microvm/sqs-store/initramfs.cpio.gz
sudo install -m 0644 $(SQS_STORE_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-store/vmlinuz
sudo install -m 0644 $(SQS_STORE_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-store/initramfs.cpio.gz
install-sqs-worker-microvm: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Install the SQS worker microVM image to /usr/share/spinifex/microvm/sqs-worker/
sudo install -d /usr/share/spinifex/microvm/sqs-worker
sudo rm -f /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-worker/vmlinuz
sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
install-microvm: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
sudo install -d /usr/share/spinifex/microvm
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom /usr/share/spinifex/microvm/qboot.rom
sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.LICENSE /usr/share/spinifex/microvm/qboot.LICENSE
# the pre-commit gate: manifest checks, lint, vuln, and the unit and e2e-harness tiers.
# the pre-commit gate: lint, vuln, and the unit and e2e-harness tiers.
# integration and race tests skipped to keep quick, they run in CI.
#
# -j2 overlaps the only two heavy gates, lint and test-cover: alone each leaves
# most of the machine idle, so pairing them cuts a cold run by about a quarter.
# -Otarget keeps each gate's output in one block instead of interleaving them.
# manifest-check manifest-lint test-package-check removed, can add back later
preflight:
@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts
@echo -e "\n ✅ Preflight passed — safe to commit."
@@ -167,7 +234,7 @@ preflight:
# reason: between them they install the software, wire OVN, erase the state and
# remove the install, so a shellcheck warning in one is an outage or a wrong
# deletion, not a style note.
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh scripts/lib/stage-sqs-store-microvm.sh
test-build-scripts:
@echo -e "\n....Running build/scripts/**/*_test.sh...."
@@ -175,13 +242,14 @@ test-build-scripts:
echo "-- $$t"; \
sh "$$t" || exit 1; \
done
@echo -e "\n....Running scripts/*_test.sh...."
@for t in $$(find scripts -maxdepth 1 -name '*_test.sh' | sort); do \
@echo -e "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
@for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
echo "-- $$t"; \
sh "$$t" || exit 1; \
done
@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
@echo " test-build-scripts ok"
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -220,7 +288,7 @@ aws-model-coverage: generate-aws-model-coverage
@go run ./cmd/aws-model-coverage
# Segscan storage oracle: needs the mulga umbrella repo's scripts/segscan
# checked out alongside spinifex (see spinifex/testutil/segscanoracle), which
# checked out alongside spinifex (see internal/testkit/segscanoracle), which
# is not the default local or CI layout, so this is a separate target from
# test-integration rather than folded into it. Skips itself when segscan's
# source isn't found.
@@ -228,27 +296,10 @@ test-segscan-oracle:
@echo -e "\n....Running segscan storage oracle test...."
$(_Q)LOG_IGNORE=1 go test -tags=integration,segscanoracle -timeout 120s ./tests/integration/... -run TestSegscanOracle $(_RACEQ)
# Validate docs/service-interfaces.yaml. Schema check + cross-reference
# of services/suites/fixtures + on-disk path existence.
manifest-check:
@echo -e "\n....Checking service-interfaces.yaml...."
@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
# Drift guards: direct-create fixture lint + NATS subject lint,
# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
# drift beyond the baseline.
manifest-lint:
@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
# Accept current drift into the baseline. Run after an intentional change.
manifest-lint-update:
@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
# Run unit tests
test:
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -timeout 180s ./spinifex/... ./cmd/... ./internal/...
LOG_IGNORE=1 go test -timeout 180s ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/...
# Empty locally, where reusing a cached result between runs is the point.
# CI passes -count=1 so a green run means the suite executed against that commit
@@ -259,24 +310,13 @@ GOTESTFLAGS ?=
COVERPROFILE ?= coverage.out
test-cover:
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 180s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... ./cmd/... ./internal/... $(_COVQ)
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 180s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/... $(_COVQ)
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
# Refresh the README coverage badge. Measured as awesome-go documents
# (-coverpkg=./... over ./...), with cmd/ entrypoints left out of the total.
coverage-badge:
@echo -e "\n....Measuring whole-module coverage for the README badge...."
LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 600s -covermode=atomic -coverpkg=./... -coverprofile=coverage-badge.out ./...
@grep -v '/cmd/' coverage-badge.out > coverage-badge-nocmd.out
@pct=$$(go tool cover -func=coverage-badge-nocmd.out | tail -1 | awk '{print $$NF}' | tr -d '%'); \
color=$$(awk -v p="$$pct" 'BEGIN { print (p >= 80) ? "brightgreen" : (p >= 70) ? "yellow" : "red" }'); \
sed -i -E "s|badge/coverage-[0-9.]+%25-[a-z]+|badge/coverage-$${pct}%25-$${color}|" README.md; \
echo "Coverage $${pct}% ($${color}); README badge updated"
# Run unit tests with race detector
test-race:
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -timeout 300s ./spinifex/... ./cmd/... ./internal/... $(_RACEQ)
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -timeout 300s ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/... $(_RACEQ)
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
@@ -344,14 +384,20 @@ fuzz:
# Microvm artifacts are reinstalled when they already exist on disk — the rule's
# input timestamps drive a rebuild only if anything actually changed. On a fresh
# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
# `make install-microvm` explicitly the first time.
# `make install-microvm` explicitly the first time. They go in before the
# preflight, which checks the microvm firmware.
deploy: build
sudo install -m 755 bin/spx /usr/local/bin/spx
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
$(MAKE) install-microvm; \
else \
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
fi
@if [ "$${SKIP_ASSET_PREFLIGHT:-}" != "1" ]; then \
if ! sudo /usr/local/bin/spx admin preflight; then \
echo ""; \
echo "[deploy] host-asset preflight failed: this node's helper scripts or sudoers"; \
echo " grants don't match the binary just installed. Remediate with:"; \
echo "[deploy] host-asset preflight failed: this node's helper scripts, sudoers"; \
echo " grants, QEMU or microvm firmware don't match the binary just installed. Remediate with:"; \
echo " scripts/update-nodes.sh (from the mulga root)"; \
echo " or:"; \
echo " make reinstall"; \
@@ -361,11 +407,6 @@ deploy: build
else \
echo "[deploy] SKIP_ASSET_PREFLIGHT=1 — skipping host-asset preflight"; \
fi
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
$(MAKE) install-microvm; \
else \
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
fi
sudo systemctl daemon-reload
sudo systemctl restart spinifex.target
@@ -376,23 +417,32 @@ reinstall:
clean:
rm -f ./bin/$(GO_PROJECT_NAME)
rm -rf spinifex/services/spinifexui/frontend/dist
rm -rf spinifex/runtime/roles/spinifexui/frontend/dist
# On dnf hosts the runtime packages and repositories come from setup.sh, so the
# two cannot drift; nbdkit-devel, for building the plugin, is only in CodeReady.
install-system:
@echo -e "\n....Installing system dependencies for $(ARCH)...."
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
ovmf qemu-efi-aarch64 \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch dhcpcd-base
@pm=$$(INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && distro_pkg_manager') || exit 1; \
if [ "$$pm" = dnf ]; then \
INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && setup_sudo && detect_os && install_dnf_deps' && \
sudo dnf config-manager --set-enabled ol10_codeready_builder && \
sudo dnf install -y nbdkit-devel pkgconf-pkg-config make gcc openssh-clients git sudo; \
else \
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
ovmf qemu-efi-aarch64 \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch dhcpcd-base; \
fi
install-go:
@echo -e "\n....Installing Go 1.27.0 for $(ARCH) ($(GO_ARCH))...."
@echo -e "\n....Installing Go 1.27.3 for $(ARCH) ($(GO_ARCH))...."
@if [ ! -d "/usr/local/go" ]; then \
curl -L https://go.dev/dl/go1.27.0.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
curl -L https://go.dev/dl/go1.27.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
else \
echo "Go already installed in /usr/local/go"; \
fi
@@ -449,14 +499,17 @@ distro-amd64:
-f build/Dockerfile.distro \
--output type=local,dest=dist/amd64/ \
../
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ] && [ -f $(MICROVM_OUT_DIR)/qboot.rom ] && [ -f $(QBOOT_SOURCE) ]; then \
echo "[distro-amd64] staging microvm artifacts into tarball"; \
mkdir -p dist/amd64/microvm; \
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE dist/amd64/microvm/; \
cp $(QBOOT_SOURCE) dist/; \
else \
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
fi
./scripts/lib/stage-sqs-store-microvm.sh $(SQS_STORE_MICROVM_OUT_DIR) dist/amd64
./scripts/lib/stage-sqs-store-microvm.sh $(SQS_WORKER_MICROVM_OUT_DIR) dist/amd64 sqs-worker
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
@@ -475,7 +528,7 @@ distro-arm64:
distro-clean:
rm -rf dist/
.PHONY: test-package-check build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover coverage-badge test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
.PHONY: test-package-check build build-ui test-ui-e2e ui-e2e-container ui-baselines build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image install-microvm install-sqs-store-microvm install-sqs-worker-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck nilaway \
+83 -170
View File
@@ -4,225 +4,138 @@
<p align="center">
<a href="https://go.dev"><img src="https://img.shields.io/badge/Go-1.27+-00ADD8?style=flat-square&logo=go&logoColor=white" alt="Go"></a>
<a href="https://pkg.go.dev/github.com/mulgadc/spinifex"><img src="https://img.shields.io/badge/pkg.go.dev-reference-007D9C?style=flat-square&logo=go&logoColor=white" alt="Go Reference"></a>
<a href="https://github.com/mulgadc/spinifex/blob/main/Makefile"><img src="https://img.shields.io/badge/coverage-82.9%25-brightgreen?style=flat-square" alt="Coverage"></a>
<a href="https://pkg.go.dev/github.com/mulgadc/spinifex"><img src="https://img.shields.io/badge/pkg.go.dev-Reference-007D9C?style=flat-square&logo=go&logoColor=white" alt="Go Reference"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-3fb950?style=flat-square" alt="License"></a>
<a href="https://mulgadc.com"><img src="https://img.shields.io/badge/home-mulga-orange?style=flat-square&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9IiNmZmYiIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTE2LjcxOCA4Ljg5MWMtMS4yODUgMS4zNy0zLjE3OCAyLjMxNy00LjY5NSAzLjQ0LS44NTQuNjMtMy4wODkgMi4yNy0xLjYxIDMuMzQ0IDEuNzg4IDEuMjk4IDYuMjQzLjE2NCA3Ljc4NS0xLjI2Ljg0LS43NzUuODE0LTEuODIyLS4zMjgtMi4yNTktMS4yMzUtLjQ3Mi0yLjY2LS4xMTEtMy45MTMuMDg3LS4wNDIuMDA3LS4xMzEuMDU1LS4xMjMtLjAyLjUyMy0uMzM2Ljk5My0uNzUgMS41MDMtMS4xMDIuNDkyLS4zNCAxLjA5Ny0uODI3IDEuNy0uODM4IDEuODcxLS4wMzQgMy43OTkuODkgNC4yODcgMi44MTUuODExIDMuMjAzLTMuMDA2IDUuNzE1LTUuNzg0IDUuOTQybDEuNjE0LS43NDdjLjYwNS0uMzYgMS4yMTctLjczNCAxLjc1Mi0xLjE5Ni4xMzMtLjExNS4yMy0uMjYzLjM0Mi0uMzczLjAyNy0uMDI2LjMwMi0uMTQ0LjE0NC0uMTUtMS40NDEuOTQ1LTMuMTI3IDEuNTcyLTQuODEzIDEuOTMyLS41OC4xMjMtMS4xOTUuMjQyLTEuNzg1LjI1Ny4wMTUuMDguMDg3LjA2NC4xNC4wOC40NzYuMTU0IDEuMDIuMjQ1IDEuNTE2LjMyLjA2OC4wNTItLjAwNi4wNzQtLjA2LjA4MS0uNDQ4LjA1OC0uOTIzLjE0My0xLjM3LjE2Ni0xLjI3LjA2NC0yLjU1LS4wNjgtMy43NzctLjM4bC0uNjktMS45NTdjLS4wNi4wMDYtLjA2My4wNi0uMDc5LjEwMy0uMDYuMTctLjMwNSAxLjQxNy0uMzg3IDEuNDM1LS42Ni0uMi0xLjI1MS0uNjQ3LTEuNzM2LTEuMTMybDEuMTUtMi4wNi0xLjgzNSAxLjAxMWMtLjI1OC0uNTgzLS4zMDUtMS4yNDMtLjIxOS0xLjg3bDIuMzc1LTEuMThjLS43MzktLjA0OS0xLjQ4Ni4wNjgtMi4yMi4xNDUuMTI1LS4zNi4yNzctLjcxOC40NjItMS4wNTEuMDY3LS4xMjIuNDI2LS43MjEuNTItLjcyNC44Ni4xNjQgMS43ODYuMjQxIDIuNjQ4LjA3NGwtMS44Ny0uNzcxLS4wNjktLjA5M2MuMzgyLS4zNi43Ny0uNzE3IDEuMTk5LTEuMDI0LjgwNC4yMzQgMS42My40MjIgMi40NzMuNDMzTDkuMzUgOS4zNDJhNyA3IDAgMCAxIC41NjctLjQwMWMuMTYtLjEwMy42MDktLjQwMi43NzItLjM4LjY1LjIyMSAxLjMyNC4zNzggMi4wMS40MzMtLjMxMy0uMzI1LS45MDktLjU2NC0xLjIxLS44NjgtLjAzMy0uMDM0LS4wNTgtLjAzNi0uMDQzLS4wOTguNzg3LS40NTIgMS42NjItLjkwMSAyLjM0LTEuNTE3LjY5LS42MjkgMS40MjEtMS42NTYuOTI0LTIuNjA1LjU3My4xODIgMS4wNjcuOTA2IDEuMDU0IDEuNTEyLS4wMzQgMS42NzYtMS44MjIgMy4xNy0yLjk0MyA0LjIyMiAxLjczMi0uNzI4IDMuNzE0LTIuMjMgMy43MS00LjMwNS0uMDAzLTEuNjg4LTEuNTQtMi4zNjUtMi45OTMtMi41MThhMy4yIDMuMiAwIDAgMS0uMzg1IDEuMDg3Yy0uNDI4LjcxOS0xLjMwMiAxLjE2OC0xLjc1OCAxLjkxNS0uMzExLjUxLS4zNyAxLjE5NS0xLjAzMSAxLjM5LS4yNy4wOC0uNjE3LjA5My0uODk3LjA5NGwuNjE1LS4zMjVjLjY4OS0uNTA2LjY3Ny0xLjQyIDEuMDk5LTIuMS0uMDUtLjA3LS41NDUtLjItLjY2LS4yMjktLjUxLS4xMjUtMS4zNzUtLjI4NS0xLjg4Ni0uMjUtLjE1Ny4wMS0uODY5LjEyMS0uOTMuMjQyLS4wODguMTc3LjM0MS45My40NjggMS4xMDEuMDI1LjAzNS4wNzcuMDI2LjA4LjAzLjAyNC4wMzEuMDA3LjEwNS0uMDYuMDhhNSA1IDAgMCAxLS40MDQtLjI4M2MtLjUwMy0uMzg4LTEuMDc4LS45NzQtMS40OTktMS40NDctLjA2OC0uMTY2LS4xNi0uMzUuMDAyLS40ODcuODg4LS41NSAxLjc2OC0xLjEyNiAyLjY3LTEuNjUxIDIuNjcxLTEuNTU4IDUuNzExLTMuMzE4IDguMjY3LS40NjUgMi4xMTkgMi4zNjYgMS41MTQgNS4yMTItLjUxMiA3LjM3MnptLTguMjI0LTUuMzhjLjY5OS0uMTIyIDIuMDE4LjU3MiAyLjIzNS0uNDU2LjAyMS0uMS0uMDM2LS4xNTcuMDItLjI1NS4wNDMtLjA3My4yODYtLjI1LjM3LS4zMTguMjctLjIxNy41NzctLjM4Ny44NDItLjYxMi0xLjAyOS4xNjYtMi4wNjUuNzU2LTIuOTY0IDEuMjc3LS4xNDkuMDg2LS4zMS4xNzYtLjQ1MS4yNzMtLjAzNy4wMjUtLjA3NS0uMDA2LS4wNTMuMDltLTQuODMgMTAuODQ0Yy0xLjcxNSAxLjg4MS0xLjMyNSA0LjU3LjQ5NCA2LjIxNyAxLjgxMSAxLjY0MSA0LjY2IDIuMjIzIDcuMDQ4IDIuMTA4IDIuMTUyLS4xMDMgNC4zMzctLjgxMiA2LjQ2LS4wOTEuODI1LjI4IDEuNTQ2LjgwNSAyLjE2IDEuNDExLS4wODMtLjQtLjMyNC0uODE5LS41NTgtMS4xNTktMi4xMy0zLjA5NS02LjI3LTEuOTM1LTkuNDI2LTIuNTUzLTEuNzExLS4zMzUtMy40OTEtMS4xMTgtNC41MzMtMi41NjctLjkwMS0xLjI1My0xLjA0Mi0yLjczLS41OTUtNC4xOTQtLjA5MS0uMDktLjk1Mi43MjEtMS4wNS44MjhtNi4zNjYtOC42NjdjLS4zODIuMzM5LS43ODcuNjYtMS4yMTIuOTQ4bC0xLjIwNy41OWMxLjAyMi4wMzkgMi4wOC0uNTQ2IDIuNDItMS41MzciLz48L3N2Zz4=" alt="mulgadc.com"></a>
<a href="https://docs.mulgadc.com"><img src="https://img.shields.io/badge/docs-spinifex-0969da?style=flat-square&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9IiNmZmYiIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTYuNSAySDIwYTEgMSAwIDAgMSAxIDF2MTQuNWExIDEgMCAwIDEtMSAxSDYuNWExLjUgMS41IDAgMCAwIDAgM0gyMWExIDEgMCAwIDEgMCAySDYuNUEzLjUgMy41IDAgMCAxIDMgMjBWNS41QTMuNSAzLjUgMCAwIDEgNi41IDJ6bTEgNGExIDEgMCAwIDAgMCAyaDlhMSAxIDAgMCAwIDAtMmgtOXptMCA0YTEgMSAwIDAgMCAwIDJoOWExIDEgMCAwIDAgMC0yaC05eiIvPjwvc3ZnPg==" alt="docs.mulgadc.com"></a>
<a href="https://mulgadc.com"><img src="https://img.shields.io/badge/Site-mulga-orange?style=flat-square&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9IiNmZmYiIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTE2LjcxOCA4Ljg5MWMtMS4yODUgMS4zNy0zLjE3OCAyLjMxNy00LjY5NSAzLjQ0LS44NTQuNjMtMy4wODkgMi4yNy0xLjYxIDMuMzQ0IDEuNzg4IDEuMjk4IDYuMjQzLjE2NCA3Ljc4NS0xLjI2Ljg0LS43NzUuODE0LTEuODIyLS4zMjgtMi4yNTktMS4yMzUtLjQ3Mi0yLjY2LS4xMTEtMy45MTMuMDg3LS4wNDIuMDA3LS4xMzEuMDU1LS4xMjMtLjAyLjUyMy0uMzM2Ljk5My0uNzUgMS41MDMtMS4xMDIuNDkyLS4zNCAxLjA5Ny0uODI3IDEuNy0uODM4IDEuODcxLS4wMzQgMy43OTkuODkgNC4yODcgMi44MTUuODExIDMuMjAzLTMuMDA2IDUuNzE1LTUuNzg0IDUuOTQybDEuNjE0LS43NDdjLjYwNS0uMzYgMS4yMTctLjczNCAxLjc1Mi0xLjE5Ni4xMzMtLjExNS4yMy0uMjYzLjM0Mi0uMzczLjAyNy0uMDI2LjMwMi0uMTQ0LjE0NC0uMTUtMS40NDEuOTQ1LTMuMTI3IDEuNTcyLTQuODEzIDEuOTMyLS41OC4xMjMtMS4xOTUuMjQyLTEuNzg1LjI1Ny4wMTUuMDguMDg3LjA2NC4xNC4wOC40NzYuMTU0IDEuMDIuMjQ1IDEuNTE2LjMyLjA2OC4wNTItLjAwNi4wNzQtLjA2LjA4MS0uNDQ4LjA1OC0uOTIzLjE0My0xLjM3LjE2Ni0xLjI3LjA2NC0yLjU1LS4wNjgtMy43NzctLjM4bC0uNjktMS45NTdjLS4wNi4wMDYtLjA2My4wNi0uMDc5LjEwMy0uMDYuMTctLjMwNSAxLjQxNy0uMzg3IDEuNDM1LS42Ni0uMi0xLjI1MS0uNjQ3LTEuNzM2LTEuMTMybDEuMTUtMi4wNi0xLjgzNSAxLjAxMWMtLjI1OC0uNTgzLS4zMDUtMS4yNDMtLjIxOS0xLjg3bDIuMzc1LTEuMThjLS43MzktLjA0OS0xLjQ4Ni4wNjgtMi4yMi4xNDUuMTI1LS4zNi4yNzctLjcxOC40NjItMS4wNTEuMDY3LS4xMjIuNDI2LS43MjEuNTItLjcyNC44Ni4xNjQgMS43ODYuMjQxIDIuNjQ4LjA3NGwtMS44Ny0uNzcxLS4wNjktLjA5M2MuMzgyLS4zNi43Ny0uNzE3IDEuMTk5LTEuMDI0LjgwNC4yMzQgMS42My40MjIgMi40NzMuNDMzTDkuMzUgOS4zNDJhNyA3IDAgMCAxIC41NjctLjQwMWMuMTYtLjEwMy42MDktLjQwMi43NzItLjM4LjY1LjIyMSAxLjMyNC4zNzggMi4wMS40MzMtLjMxMy0uMzI1LS45MDktLjU2NC0xLjIxLS44NjgtLjAzMy0uMDM0LS4wNTgtLjAzNi0uMDQzLS4wOTguNzg3LS40NTIgMS42NjItLjkwMSAyLjM0LTEuNTE3LjY5LS42MjkgMS40MjEtMS42NTYuOTI0LTIuNjA1LjU3My4xODIgMS4wNjcuOTA2IDEuMDU0IDEuNTEyLS4wMzQgMS42NzYtMS44MjIgMy4xNy0yLjk0MyA0LjIyMiAxLjczMi0uNzI4IDMuNzE0LTIuMjMgMy43MS00LjMwNS0uMDAzLTEuNjg4LTEuNTQtMi4zNjUtMi45OTMtMi41MThhMy4yIDMuMiAwIDAgMS0uMzg1IDEuMDg3Yy0uNDI4LjcxOS0xLjMwMiAxLjE2OC0xLjc1OCAxLjkxNS0uMzExLjUxLS4zNyAxLjE5NS0xLjAzMSAxLjM5LS4yNy4wOC0uNjE3LjA5My0uODk3LjA5NGwuNjE1LS4zMjVjLjY4OS0uNTA2LjY3Ny0xLjQyIDEuMDk5LTIuMS0uMDUtLjA3LS41NDUtLjItLjY2LS4yMjktLjUxLS4xMjUtMS4zNzUtLjI4NS0xLjg4Ni0uMjUtLjE1Ny4wMS0uODY5LjEyMS0uOTMuMjQyLS4wODguMTc3LjM0MS45My40NjggMS4xMDEuMDI1LjAzNS4wNzcuMDI2LjA4LjAzLjAyNC4wMzEuMDA3LjEwNS0uMDYuMDhhNSA1IDAgMCAxLS40MDQtLjI4M2MtLjUwMy0uMzg4LTEuMDc4LS45NzQtMS40OTktMS40NDctLjA2OC0uMTY2LS4xNi0uMzUuMDAyLS40ODcuODg4LS41NSAxLjc2OC0xLjEyNiAyLjY3LTEuNjUxIDIuNjcxLTEuNTU4IDUuNzExLTMuMzE4IDguMjY3LS40NjUgMi4xMTkgMi4zNjYgMS41MTQgNS4yMTItLjUxMiA3LjM3MnptLTguMjI0LTUuMzhjLjY5OS0uMTIyIDIuMDE4LjU3MiAyLjIzNS0uNDU2LjAyMS0uMS0uMDM2LS4xNTcuMDItLjI1NS4wNDMtLjA3My4yODYtLjI1LjM3LS4zMTguMjctLjIxNy41NzctLjM4Ny44NDItLjYxMi0xLjAyOS4xNjYtMi4wNjUuNzU2LTIuOTY0IDEuMjc3LS4xNDkuMDg2LS4zMS4xNzYtLjQ1MS4yNzMtLjAzNy4wMjUtLjA3NS0uMDA2LS4wNTMuMDltLTQuODMgMTAuODQ0Yy0xLjcxNSAxLjg4MS0xLjMyNSA0LjU3LjQ5NCA2LjIxNyAxLjgxMSAxLjY0MSA0LjY2IDIuMjIzIDcuMDQ4IDIuMTA4IDIuMTUyLS4xMDMgNC4zMzctLjgxMiA2LjQ2LS4wOTEuODI1LjI4IDEuNTQ2LjgwNSAyLjE2IDEuNDExLS4wODMtLjQtLjMyNC0uODE5LS41NTgtMS4xNTktMi4xMy0zLjA5NS02LjI3LTEuOTM1LTkuNDI2LTIuNTUzLTEuNzExLS4zMzUtMy40OTEtMS4xMTgtNC41MzMtMi41NjctLjkwMS0xLjI1My0xLjA0Mi0yLjczLS41OTUtNC4xOTQtLjA5MS0uMDktLjk1Mi43MjEtMS4wNS44MjhtNi4zNjYtOC42NjdjLS4zODIuMzM5LS43ODcuNjYtMS4yMTIuOTQ4bC0xLjIwNy41OWMxLjAyMi4wMzkgMi4wOC0uNTQ2IDIuNDItMS41MzciLz48L3N2Zz4=" alt="mulgadc.com"></a>
<a href="https://docs.mulgadc.com"><img src="https://img.shields.io/badge/Docs-spinifex-0969da?style=flat-square&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9IiNmZmYiIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTYuNSAySDIwYTEgMSAwIDAgMSAxIDF2MTQuNWExIDEgMCAwIDEtMSAxSDYuNWExLjUgMS41IDAgMCAwIDAgM0gyMWExIDEgMCAwIDEgMCAySDYuNUEzLjUgMy41IDAgMCAxIDMgMjBWNS41QTMuNSAzLjUgMCAwIDEgNi41IDJ6bTEgNGExIDEgMCAwIDAgMCAyaDlhMSAxIDAgMCAwIDAtMmgtOXptMCA0YTEgMSAwIDAgMCAwIDJoOWExIDEgMCAwIDAgMC0yaC05eiIvPjwvc3ZnPg==" alt="docs.mulgadc.com"></a>
</p>
<p align="center">
<a href="#what-is-spinifex">What is Spinifex?</a> ·
<a href="#the-platform">Platform</a> ·
<a href="#three-ways-to-deploy">Deploy</a> ·
<a href="#aws-compatibility">AWS Compatibility</a> ·
<a href="#live-demo">Live Demo</a> ·
<a href="#core-components">Components</a> ·
<a href="#architecture-at-a-glance">Architecture</a> ·
<a href="#self-hosted-installation">Installation</a> ·
<a href="https://docs.mulgadc.com">Docs</a>
<a href="#try-it-live">Live Demo</a> ·
<a href="#services">Services</a> ·
<a href="#deploy-anywhere">Deploy</a> ·
<a href="#install">Install</a> ·
<a href="#architecture">Architecture</a>
</p>
---
# Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your software already uses on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line.
## What is Spinifex?
Spinifex is an open-source infrastructure platform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
Built for teams that need:
- The AWS tooling you already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
- **Your AWS tooling, unchanged.** The AWS CLI, SDKs, and Terraform work as they are.
- **Full control of the stack.** Your hardware, data, and keys, with no external control plane, running through disconnection.
- **Open core under AGPL-3.0.** Auditable, yours to keep, with no lock-in.
You change *where* your software runs, not *what* it is.
## The Platform
## Try It Live
Get a real Spinifex endpoint in under a minute, with no card and no install. The sandbox is yours for 72 hours, with access keys, the web console, and EC2, EBS, S3, VPC, IAM, EKS, and RDS ready to call. Point your existing AWS CLI or Terraform straight at it.
**[Create your sandbox →](https://mulgadc.com/signup)**
## Same Commands, Your Cloud
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-images
aws ec2 run-instances --image-id ami-xxxxxxxx --instance-type t3.micro
aws s3 mb s3://my-bucket
```
For Terraform, start from the [example workbooks](https://docs.mulgadc.com/docs/nginx-webserver).
## Why Not LocalStack or Floci?
[LocalStack](https://github.com/localstack/localstack) and [Floci](https://github.com/floci-io/floci) are emulators for local development and CI testing. An emulator returns a plausible response; Spinifex boots the machine. A call launches a real KVM guest, attaches a replicated Viperblock volume, or writes an erasure-coded object into Predastore, so Spinifex can run your production workloads on hardware you own. Test against an emulator, then run in production on Spinifex.
## Services
The platform serves **405 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
| Service | What it is | Built on | Status |
| --- | --- | --- | --- |
| **EC2** | Compute | QEMU/KVM, with spot instances and GPU passthrough | Available |
| **EBS** | Block storage | [Viperblock](https://github.com/mulgadc/viperblock): replicated, with a WAL on local NVMe | Available |
| **S3** | Object storage | [Predastore](https://github.com/mulgadc/predastore): Reed-Solomon erasure coding | Available |
| **VPC** | Networking | OVN, with security groups | Available |
| **IAM** | Identity & auth | SigV4 across every service | Available |
| **ALB / NLB** | Load balancers | HAProxy (L7), nginx `stream` (L4), with ACM certificates | Available |
| **EKS** | Kubernetes | k3s, with HA control planes and managed node groups | Available |
| **ECR** | Container registry | Predastore, deduplicated per account | Available |
| **ECS** | Container service | Spinifex instances | Available |
| **RDS** | Databases | PostgreSQL and MariaDB, with automated backups | Available |
| **Route 53** | DNS | [Northstar](https://github.com/mulgadc/northstar): authoritative DNS with hosted zones | Available |
| **Bedrock** | AI deployment | vLLM for text generation and Hugging Face TEI for embeddings and reranking | Q4 2026 |
| **Lambda** | Serverless functions | | Q1 2027 |
| **SQS** | Message queues | | Q1 2027 |
Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Deploy Anywhere
<p align="center">
<img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
- **Neocloud:** lift and shift onto partner GPU capacity (H100 / H200 / B200). Change an endpoint, keep the software.
- **On-premise:** a multi-node HA cluster in your own data centre, so the stack, the data, and the jurisdiction stay yours.
- **Edge:** air-gapped sites, vehicles, vessels, factories, and clinics, with compute next to the data.
## Three Ways To Deploy
## Install
The same AWS-compatible platform runs in three shapes. Pick the one that matches your hardware.
Spinifex runs on Ubuntu 26.04 or Debian 13. The full guides are at [docs.mulgadc.com](https://docs.mulgadc.com).
### Neocloud
>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
Lift and shift onto a partner Neocloud without rewriting workloads. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
### On-premise
Host Spinifex on your own hardware in your own data centre. You get a multi-node HA cluster integrated with your storage and networking, so the stack, the data, and the jurisdiction stay yours.
### Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
## AWS Compatibility
Spinifex speaks the AWS API surface. The AWS SDKs, AWS CLI, and Terraform all work unchanged, whether you deploy at the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
| --- | --- | --- |
| **EC2** | Compute | Available |
| **EBS** | Block Storage | Available |
| **S3** | Object Storage | Available |
| **VPC** | Networking | Available |
| **IAM** | Identity & Auth | Available |
| **ALB / NLB** | Load Balancers | Available |
| **EKS** | Kubernetes | Available |
| **ECR** | Container Registry | Available |
| **ECS** | Container Service | Available |
| **RDS** | Databases | Available |
| **Bedrock** | AI Deployment | Q4 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Live Demo
Create a live sandbox in under a minute. The fastest way to experience the stack. Try Spinifex against one of your own workloads on infrastructure we run for you.
* Live in under a minute. No card, no install.
* Yours for 72 hours to test and experiment.
* Real AWS-compatible endpoint, access keys and console.
* EC2, EBS, S3, VPC, IAM, EKS and RDS, ready to call.
* Point your existing AWS CLI or Terraform straight at it.
Sign up at [https://mulgadc.com/signup](https://mulgadc.com/signup)
## Core Components
### Spinifex (Compute and Platform Services – EC2, EKS, ECS, RDS)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing an EC2 compatible API. It manages lifecycle operations like start, stop, and terminate through QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs with the AWS CLI, SDKs, or Terraform, without Kubernetes or a heavyweight orchestrator.
The managed services build on that same instance layer. An EKS node group, an ECS container instance, and an RDS database are Spinifex VMs on your own hardware, each driven through its own AWS API.
- **EC2** – VM lifecycle on bare metal, cloud-init and IMDS metadata, key pairs, tags, spot instances, capacity reservations, and GPU passthrough
- **EKS** – Managed Kubernetes on k3s, with HA control planes, managed node groups, cluster add-ons, an OIDC provider, and IAM access entries
- **ECS** – Task and service scheduling across container instances, with awsvpc ENIs, capacity providers, rolling deployments, and GPU-aware placement
- **ECR** – Private container registry backed by Predastore, with content-addressed blobs deduplicated per account and lifecycle policies
- **RDS** – Managed PostgreSQL and MariaDB, with parameter groups, automated backups, and DNS endpoints served by Northstar
- **ALB / NLB** – Layer 7 load balancing on HAProxy and layer 4 on nginx `stream`, with health checks and ACM-issued listener certificates
- **VPC and IAM** – OVN-backed networking and security groups, with roles, policies, and SigV4 auth applied across every service
### Viperblock (Block Storage – EBS Alternative)
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- High performance WAL logs on local NVMe drives, reducing IO traffic to S3
- In-memory read/write block cache
### Predastore (Object Storage – S3-Compatible)
[Predastore](https://github.com/mulgadc/predastore) is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Predastore chunks data and distributes it across nodes using Reed-Solomon erasure coding, so it survives node loss and suits large-scale or low-bandwidth sites.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
### Northstar (Authoritative DNS)
[Northstar](https://github.com/mulgadc/northstar) is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
## Architecture at a Glance
<p align="center">
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Self-hosted Installation
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for installing Spinifex.
### Bare Metal ISO
The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
**Bare metal (recommended):** download the [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) and follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb).
```bash
curl -fLO https://iso.mulgadc.com/spinifex.iso
```
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware.
### Single Node Install
>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
**Single node:**
```bash
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
### Development Setup
**From source:** see the [Source Install](https://docs.mulgadc.com/docs/install-source) guide.
For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation.
## Spinifex UI
Spinifex ships with a built-in web console, an optional alternative to the AWS CLI, SDKs, and Terraform. If you know the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
## Web Console
<p align="center">
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console dashboard" width="900">
</p>
Each node serves the console on port `3000` over TLS, and it comes up with `spinifex.target`:
Spinifex ships with a built-in web console for your instances, volumes, buckets, networks, and IAM. See [Launching the Web UI](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui).
- **Same API, different surface.** The UI issues the same AWS SigV4 calls the CLI does, so RBAC, audit trails, and IAM policies apply the same way.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed. There is no separate user database.
- **Self-hosted, works offline.** The Spinifex binary embeds the UI and each node serves it directly, with no external CDN or analytics calls.
## Architecture
For the full walkthrough, covering first-time TLS certificate trust, login, and a feature tour, see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
<p align="center">
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
## Development Philosophy
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless, so you scale horizontally by starting more. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
### Built by Engineers, For Engineers
## How We Build Spinifex
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Spinifex is built by infrastructure engineers with deep AWS experience, including former AWS team members, drawing on decades of combined work across AWS, enterprise infrastructure, and edge computing. Engineers architect and implement it, with **Claude Code** accelerating parts of the work.
### AI-Assisted Development
Spinifex is architected and implemented by experienced engineers, with **Claude Code** accelerating certain development tasks. The split keeps the quality and reliability standards that production infrastructure demands:
**How We Use Claude Code:**
- **Code Generation**: Boilerplate AWS API structures and handlers
- **Documentation**: Comprehensive development guides and API documentation
- **Testing**: Test case generation and validation scenarios
- **Refactoring**: Large-scale code restructuring and optimization
**What Remains Human-Driven:**
- **Architecture Decisions**: Core system design and scalability choices
- **Security Implementation**: Authentication, encryption, and threat modeling
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
| Claude Code assists with | Engineers own |
| --- | --- |
| Boilerplate AWS API structures and handlers | Architecture and scalability decisions |
| Development guides and API documentation | Authentication, encryption, and threat modelling |
| Test case generation and validation scenarios | Real-world performance tuning and benchmarking |
| Large-scale restructuring and refactoring | Deployment strategy and production operations |
## Trademarks
@@ -230,4 +143,4 @@ Spinifex is architected and implemented by experienced engineers, with **Claude
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE), with a commercial license available. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE), with a commercial license available.
@@ -0,0 +1,92 @@
package instancestate
// allowance is one recorded production consumer of raw instance state.
type allowance struct {
File, Symbol string
Inv string // inventory item in docs/PACKAGE_BOUNDARY_MIGRATION.md
}
// allowlist mirrors the ADR-0007 slice 0 inventory in
// docs/PACKAGE_BOUNDARY_MIGRATION.md. It only shrinks: delete an entry as soon
// as its use is gone, and never add one to make a new consumer pass.
var allowlist = []allowance{
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "daemon.LocalStatePath", Inv: "INV-01"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "daemon.ReadLocalState", Inv: "INV-01"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "daemon.MarshalLocalState", Inv: "INV-01"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "daemon.WriteLocalStateBytes", Inv: "INV-01"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "JetStreamManager.InitKVBucket", Inv: "INV-02"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "JetStreamManager.InitTerminatedInstanceBucket", Inv: "INV-02"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "JetStreamManager.WriteNodeMarkerBestEffort", Inv: "INV-03"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "JetStreamManager.WriteRunningSet", Inv: "INV-03"},
{File: "spinifex/runtime/daemon/instance_record_repair.go", Symbol: "JetStreamManager.WriteRunningSet", Inv: "INV-03"},
{File: "spinifex/runtime/daemon/daemon.go", Symbol: "JetStreamManager.ListTerminatedInstances", Inv: "INV-04"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.LoadState", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.WriteStoppedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.LoadStoppedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.DeleteStoppedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.UpdateStoppedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.ClaimStoppedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.ListStoppedInstances", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.WriteTerminatedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.UpdateTerminatedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.ListTerminatedInstances", Inv: "INV-05"},
{File: "spinifex/runtime/daemon/vm_adapters.go", Symbol: "JetStreamManager.DeleteTerminatedInstance", Inv: "INV-05"},
{File: "spinifex/runtime/compute/vm/migrate.go", Symbol: "JetStreamManager.WriteStoppedInstance", Inv: "INV-06"},
{File: "spinifex/runtime/compute/vm/migrate.go", Symbol: "JetStreamManager.WriteTerminatedInstance", Inv: "INV-06"},
{File: "spinifex/runtime/compute/vm/shutdown.go", Symbol: "JetStreamManager.WriteTerminatedInstance", Inv: "INV-06"},
{File: "spinifex/runtime/compute/vm/orphan_qemu_reaper.go", Symbol: "JetStreamManager.ListTerminatedInstances", Inv: "INV-06"},
{File: "spinifex/runtime/compute/vm/teardown_reaper.go", Symbol: "JetStreamManager.ListTerminatedInstances", Inv: "INV-06"},
{File: "spinifex/runtime/compute/vm/teardown_reaper.go", Symbol: "JetStreamManager.UpdateTerminatedInstance", Inv: "INV-06"},
{File: "spinifex/runtime/compute/vm/teardown_reaper.go", Symbol: "JetStreamManager.DeleteTerminatedInstance", Inv: "INV-06"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.LoadStoppedInstance", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.UpdateStoppedInstance", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.ListStoppedInstances", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.ListTerminatedInstances", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.WriteStoppedInstance", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.ClaimStoppedInstance", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.WriteTerminatedInstance", Inv: "INV-07"},
{File: "spinifex/domains/ec2/instance/service_impl.go", Symbol: "JetStreamManager.DeleteStoppedInstance", Inv: "INV-07"},
{File: "spinifex/runtime/daemon/daemon_handlers_image.go", Symbol: "JetStreamManager.LoadStoppedInstance", Inv: "INV-08"},
{File: "spinifex/runtime/daemon/daemon_handlers_image.go", Symbol: "JetStreamManager.UpdateStoppedInstance", Inv: "INV-08"},
{File: "spinifex/runtime/daemon/instance_recovery.go", Symbol: "vm.InstanceRecord", Inv: "INV-10"},
{File: "spinifex/runtime/daemon/instance_recovery.go", Symbol: "JetStreamManager.ListInstanceRecords", Inv: "INV-10"},
{File: "spinifex/runtime/daemon/instance_recovery.go", Symbol: "JetStreamManager.ClaimRecoverableInstance", Inv: "INV-10"},
{File: "spinifex/runtime/daemon/instance_recovery.go", Symbol: "JetStreamManager.ReleaseRecoveredInstance", Inv: "INV-10"},
{File: "spinifex/runtime/daemon/instance_recovery.go", Symbol: "JetStreamManager.AbandonRecovery", Inv: "INV-10"},
{File: "spinifex/runtime/daemon/instance_recovery.go", Symbol: "JetStreamManager.LoadInstanceRecord", Inv: "INV-10"},
{File: "spinifex/runtime/daemon/dns_reconcile.go", Symbol: "daemon.InstanceRecordPrefix", Inv: "INV-11"},
{File: "spinifex/runtime/daemon/dns_reconcile.go", Symbol: "daemon.InstanceStateBucket", Inv: "INV-11"},
{File: "spinifex/runtime/daemon/dns_reconcile.go", Symbol: "JetStreamManager.ListInstanceRecords", Inv: "INV-11"},
{File: "spinifex/runtime/daemon/eni_orphan_reaper.go", Symbol: "vm.InstanceRecord", Inv: "INV-12"},
{File: "spinifex/runtime/daemon/eni_orphan_reaper.go", Symbol: "JetStreamManager.ListInstanceRecords", Inv: "INV-12"},
{File: "spinifex/runtime/daemon/eni_orphan_reaper.go", Symbol: "JetStreamManager.ListTerminatedInstanceRecords", Inv: "INV-12"},
{File: "spinifex/domains/ec2/instance/state/imds_readers.go", Symbol: "daemon.LocalState", Inv: "INV-13"},
{File: "spinifex/domains/ec2/instance/state/imds_readers.go", Symbol: "daemon.LocalStatePath", Inv: "INV-13"},
{File: "spinifex/domains/ec2/instance/state/imds_readers.go", Symbol: "daemon.ReadLocalState", Inv: "INV-13"},
{File: "spinifex/domains/ec2/instance/state/imds_readers.go", Symbol: "daemon.InstanceStateBucket", Inv: "INV-13"},
{File: "spinifex/domains/ec2/instance/state/imds_readers.go", Symbol: "JetStreamManager.InitKVBucket", Inv: "INV-13"},
{File: "spinifex/domains/ec2/guestmetadata/instance_lookup.go", Symbol: "vm.InstanceRecord", Inv: "INV-14"},
{File: "spinifex/domains/ec2/guestmetadata/instance_lookup.go", Symbol: "vm.VMFromRecord", Inv: "INV-14"},
{File: "spinifex/domains/ec2/guestmetadata/instance_lookup.go", Symbol: "JetStreamManager.LoadInstanceRecord", Inv: "INV-14"},
{File: "spinifex/domains/network/reconcile/intent.go", Symbol: `literal "spinifex-instance-state"`, Inv: "INV-15"},
{File: "spinifex/domains/network/reconcile/intent.go", Symbol: "vm.InstanceRecord", Inv: "INV-15"},
{File: "spinifex/runtime/roles/awsgw/awsgw.go", Symbol: "daemon.InstanceStateBucket", Inv: "INV-16"},
{File: "spinifex/runtime/roles/awsgw/awsgw.go", Symbol: "daemon.InstanceRecordPrefix", Inv: "INV-16"},
{File: "spinifex/runtime/roles/awsgw/awsgw.go", Symbol: "vm.InstanceRecord", Inv: "INV-16"},
{File: "spinifex/runtime/compute/cache/cache.go", Symbol: "vm.InstanceRecord", Inv: "INV-17"},
{File: "spinifex/runtime/compute/cache/cache.go", Symbol: "vm.VMFromRecord", Inv: "INV-17"},
{File: "spinifex/domains/admission/quota/records.go", Symbol: "vm.InstanceRecord", Inv: "INV-18"},
}
@@ -0,0 +1,200 @@
package instancestate
import (
"fmt"
"go/parser"
"go/token"
"os"
"path/filepath"
"testing"
)
type report struct {
unrecorded []use
stale []allowance
malformed []string
}
// check compares the scanned uses with allow.
func check(s scan, allow []allowance) report {
var rep report
allowed := map[use]bool{}
for _, a := range allow {
u := use{File: a.File, Symbol: a.Symbol}
switch {
case allowed[u]:
rep.malformed = append(rep.malformed, "duplicate entry "+u.String())
case a.Inv == "":
rep.malformed = append(rep.malformed, "entry "+u.String()+" cites no inventory item")
}
allowed[u] = true
}
for _, u := range s.sorted() {
if !allowed[u] {
rep.unrecorded = append(rep.unrecorded, u)
}
}
for _, a := range allow {
if _, ok := s[use{File: a.File, Symbol: a.Symbol}]; !ok {
rep.stale = append(rep.stale, a)
}
}
return rep
}
// TestADR0007_S6_NoNewRawInstanceStateConsumers is the blocking ratchet: no
// production file may start naming the daemon's instance-state bucket, keys,
// local file, raw record or JetStreamManager instance methods.
func TestADR0007_S6_NoNewRawInstanceStateConsumers(t *testing.T) {
root, err := moduleRoot()
if err != nil {
t.Fatal(err)
}
s, files, err := scanModule(root)
if err != nil {
t.Fatal(err)
}
// Guards against a walk that silently finds nothing and passes vacuously.
if files < 500 || len(s) < 20 {
t.Fatalf("scanned %d files and found %d uses under %s; the walk has lost the tree", files, len(s), root)
}
if len(allowlist) == 0 {
t.Fatal("the allowlist is empty, so the stale check would pass vacuously")
}
rep := check(s, allowlist)
t.Run("NoUnrecordedConsumer", func(t *testing.T) {
for _, u := range rep.unrecorded {
t.Errorf("ADR-0007 S6 (consumers receive narrow state capabilities rather than raw stores): %s at %s; "+
"consume an instance-domain capability instead of the daemon's raw instance state", u, s[u])
}
})
t.Run("Allowlist_NoStaleEntries", func(t *testing.T) {
for _, a := range rep.stale {
t.Errorf("allowlist entry %s uses %s (%s) no longer matches; delete it and update the inventory",
a.File, a.Symbol, a.Inv)
}
})
t.Run("Allowlist_WellFormed", func(t *testing.T) {
for _, m := range rep.malformed {
t.Errorf("allowlist_test.go: %s", m)
}
t.Logf("%d allow-listed uses", len(allowlist))
})
t.Run("OwnerFilesExist", func(t *testing.T) {
for _, f := range ownerFiles {
if _, err := os.Stat(filepath.Join(root, f)); err != nil {
t.Errorf("owner file %s is gone; remove it from ownerFiles and re-scan its consumers: %v", f, err)
}
}
})
}
const faultSource = `package %s
import (
"encoding/json"
spxd "github.com/mulgadc/spinifex/spinifex/runtime/daemon"
"github.com/mulgadc/spinifex/spinifex/runtime/compute/vm"
)
type holder struct {
LoadInstanceRecord func() // a field name, not a use
m *spxd.JetStreamManager
}
func (h holder) WriteRunningSet() {} // a declaration, not a use
func use(h holder, raw []byte, jsManager *spxd.JetStreamManager, other interface{ LoadState() }) {
_ = spxd.InstanceStateBucket
_ = spxd.LocalStatePath("")
var r vm.InstanceRecord
_ = json.Unmarshal(raw, &r)
_ = vm.VMFromRecord(&r)
_ = (&vm.VM{}).Record()
_, _ = h.m.LoadInstanceRecord("i-1")
_, _, _ = jsManager.LoadState("node")
other.LoadState()
_ = "spinifex-instance-state"
cluster, _ := spxd.NewJetStreamManager(nil)
_ = cluster.InitClusterStateBucket()
_ = cluster.WriteRunningSet("node", nil)
}
`
func parseFault(t *testing.T, rel, pkg string) scan {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, rel, fmt.Sprintf(faultSource, pkg), parser.SkipObjectResolution)
if err != nil {
t.Fatal(err)
}
out := scan{}
scanFile(fset, rel, f, out)
return out
}
// TestADR0007_S6_RatchetDetectsFaults injects a new consumer, a stale entry
// and a malformed entry, and checks the matcher's declaration exclusions.
func TestADR0007_S6_RatchetDetectsFaults(t *testing.T) {
const rel = "spinifex/domains/widget/w.go"
s := parseFault(t, rel, "widget")
want := []string{
"daemon.InstanceStateBucket", "daemon.LocalStatePath",
"vm.InstanceRecord", "vm.VMFromRecord", "VM.Record()",
"JetStreamManager.LoadInstanceRecord", "JetStreamManager.LoadState",
"JetStreamManager.WriteRunningSet",
`literal "spinifex-instance-state"`,
}
for _, sym := range want {
if _, ok := s[use{File: rel, Symbol: sym}]; !ok {
t.Errorf("injected use of %s was not detected; got %v", sym, s.sorted())
}
}
if len(s) != len(want) {
t.Errorf("want exactly %d uses (the manager type, its constructor, cluster-state methods, field names, method declarations and an unrelated LoadState excluded), got %v",
len(want), s.sorted())
}
stale := allowance{File: "spinifex/domains/gone/g.go", Symbol: "vm.InstanceRecord", Inv: "INV-00"}
recorded := allowance{File: rel, Symbol: "vm.InstanceRecord", Inv: "INV-00"}
rep := check(s, []allowance{recorded, stale, {File: rel, Symbol: "vm.VMFromRecord"}, recorded})
if len(rep.unrecorded) != len(want)-2 {
t.Errorf("want %d unrecorded uses, got %v", len(want)-2, rep.unrecorded)
}
if len(rep.stale) != 1 || rep.stale[0] != stale {
t.Errorf("want stale entry %+v, got %+v", stale, rep.stale)
}
if len(rep.malformed) != 2 {
t.Errorf("want a missing-inventory and a duplicate entry reported, got %v", rep.malformed)
}
}
// Inside the owning packages the symbols are bare identifiers, and the scan
// must see those too.
func TestADR0007_S6_RatchetSeesBareIdentifiersInOwnerPackages(t *testing.T) {
src := map[string]string{
"spinifex/runtime/daemon/new_consumer.go": "package daemon\n\nfunc f() string { return InstanceStateBucket + LocalStatePath(\"\") }\n",
"spinifex/runtime/compute/vm/new_reader.go": "package vm\n\nfunc g(r *InstanceRecord) *VM { return VMFromRecord(r) }\n",
}
want := map[string][]string{
"spinifex/runtime/daemon/new_consumer.go": {"daemon.InstanceStateBucket", "daemon.LocalStatePath"},
"spinifex/runtime/compute/vm/new_reader.go": {"vm.InstanceRecord", "vm.VMFromRecord"},
}
for rel, code := range src {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, rel, code, parser.SkipObjectResolution)
if err != nil {
t.Fatal(err)
}
out := scan{}
scanFile(fset, rel, f, out)
for _, sym := range want[rel] {
if _, ok := out[use{File: rel, Symbol: sym}]; !ok {
t.Errorf("%s: bare use of %s not detected; got %v", rel, sym, out.sorted())
}
}
}
}
+302
View File
@@ -0,0 +1,302 @@
// Package instancestate freezes the production consumers of the daemon's raw
// instance-state persistence, so ADR-0007 S6 and S7 start from a known set
// that can only shrink.
package instancestate
import (
"errors"
"fmt"
"go/ast"
"go/build/constraint"
"go/parser"
"go/token"
"io/fs"
"os"
"path/filepath"
"runtime"
"sort"
"strconv"
"strings"
)
const (
daemonPath = "github.com/mulgadc/spinifex/spinifex/runtime/daemon"
vmPath = "github.com/mulgadc/spinifex/spinifex/runtime/compute/vm"
daemonDir = "spinifex/runtime/daemon"
vmDir = "spinifex/runtime/compute/vm"
)
// daemonSymbols are the daemon's instance-state bucket, key and local-file
// identifiers. The JetStreamManager type and constructor are not listed: they
// also serve cluster state, so only the instance-state methods below count.
var daemonSymbols = set(
"InstanceStateBucket", "InstanceStateBucketVersion",
"TerminatedInstanceBucket", "TerminatedInstanceBucketVersion",
"InstanceRecordPrefix", "InstanceStatePrefix", "NodePresencePrefix",
"StoppedInstancePrefix", "TerminatedInstancePrefix",
"LocalState", "LocalStateSchemaVersion", "LocalStatePath", "ReadLocalState",
"MarshalLocalState", "WriteLocalStateBytes", "LocalStateFileName",
"LocalStateFileMode", "DefaultLocalStateDir",
)
// vmSymbols are the raw persisted record and its conversion.
var vmSymbols = set("InstanceRecord", "InstanceSpec", "InstanceStatus", "VMFromRecord")
// jsmMethods are JetStreamManager's instance-state methods. Matched by name, so
// a call through an interface that mirrors them counts as a consumer too.
var jsmMethods = set(
"WriteInstanceRecord", "LoadInstanceRecord", "UpdateInstanceRecord",
"DeleteInstanceRecord", "ListInstanceRecords",
"WriteTerminatedInstanceRecord", "LoadTerminatedInstanceRecord",
"UpdateTerminatedInstanceRecord", "DeleteTerminatedInstanceRecord",
"ListTerminatedInstanceRecords",
"WriteNodeMarker", "WriteNodeMarkerBestEffort", "WriteRunningSet",
"WriteStoppedInstance", "LoadStoppedInstance", "DeleteStoppedInstance",
"ClaimStoppedInstance", "ClaimRecoverableInstance", "ReleaseRecoveredInstance",
"AbandonRecovery", "UpdateStoppedInstance", "ListStoppedInstances",
"WriteTerminatedInstance", "UpdateTerminatedInstance", "ListTerminatedInstances",
"DeleteTerminatedInstance", "LoadTerminatedInstance",
"InitKVBucket", "InitTerminatedInstanceBucket",
)
// ambiguousJSMMethods share a name with unrelated methods, so they count only
// when called on a field or variable named like a JetStreamManager handle.
var ambiguousJSMMethods = set("LoadState", "DeleteState")
// literals duplicate a bucket or file name instead of naming the owner's symbol.
var literals = set(`"spinifex-instance-state"`, `"spinifex-terminated-instances"`, `"instance-state.json"`)
// ownerFiles define the symbols above. They are the implementation being
// migrated, not consumers of it, and are listed so a deleted one is noticed.
var ownerFiles = []string{
"spinifex/runtime/daemon/instance_membership.go",
"spinifex/runtime/daemon/instance_records.go",
"spinifex/runtime/daemon/instance_records_migrate.go",
"spinifex/runtime/daemon/instance_running_set.go",
"spinifex/runtime/daemon/jetstream.go",
"spinifex/runtime/daemon/local_state.go",
"spinifex/runtime/compute/vm/record.go",
}
func set(names ...string) map[string]bool {
m := make(map[string]bool, len(names))
for _, n := range names {
m[n] = true
}
return m
}
// use is one consumer of one symbol, keyed by file so a new file is a new use.
type use struct {
File, Symbol string
}
func (u use) String() string { return u.File + " uses " + u.Symbol }
// scan maps each use to the first file:line it appears at.
type scan map[use]string
func moduleRoot() (string, error) {
_, file, _, ok := runtime.Caller(0)
if !ok {
return "", errors.New("cannot locate test source")
}
return filepath.Abs(filepath.Join(filepath.Dir(file), "..", ".."))
}
var excludedTopDirs = map[string]bool{"tests": true, "docs": true, "architecture": true}
// scanModule parses every non-test production Go file under root.
func scanModule(root string) (scan, int, error) {
out := scan{}
files := 0
fset := token.NewFileSet()
owners := set(ownerFiles...)
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil {
return err
}
rel = filepath.ToSlash(rel)
if d.IsDir() {
if rel == "." {
return nil
}
name := d.Name()
if strings.HasPrefix(name, ".") || strings.HasPrefix(name, "_") ||
name == "testdata" || name == "vendor" || name == "node_modules" || excludedTopDirs[rel] {
return filepath.SkipDir
}
if _, err := os.Stat(filepath.Join(path, "go.mod")); err == nil {
return filepath.SkipDir
}
return nil
}
if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
return nil
}
src, err := os.ReadFile(path)
if err != nil {
return err
}
f, err := parser.ParseFile(fset, path, src, parser.ParseComments|parser.SkipObjectResolution)
if err != nil {
return err
}
if ignoredByBuildTag(f.Comments, f.Package) {
return nil
}
files++
if owners[rel] {
return nil
}
scanFile(fset, rel, f, out)
return nil
})
return out, files, err
}
// scanFile records every use in f, which lives at module-relative path rel.
func scanFile(fset *token.FileSet, rel string, f *ast.File, out scan) {
dir := filepath.ToSlash(filepath.Dir(rel))
daemonName, vmName := importName(f, daemonPath, "daemon"), importName(f, vmPath, "vm")
inDaemon, inVM := dir == daemonDir, dir == vmDir
seesVM := inDaemon || inVM || daemonName != "" || vmName != ""
record := func(symbol string, pos token.Pos) {
u := use{File: rel, Symbol: symbol}
if _, ok := out[u]; !ok {
out[u] = fmt.Sprintf("%s:%d", rel, fset.Position(pos).Line)
}
}
var visit func(n ast.Node) bool
visit = func(n ast.Node) bool {
switch n := n.(type) {
case *ast.ImportSpec:
return false
case *ast.BasicLit:
if n.Kind == token.STRING && literals[n.Value] {
record("literal "+n.Value, n.Pos())
}
case *ast.CallExpr:
if sel, ok := n.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Record" && len(n.Args) == 0 && seesVM {
record("VM.Record()", sel.Sel.Pos())
}
case *ast.SelectorExpr:
if x, ok := n.X.(*ast.Ident); ok {
switch {
case daemonName != "" && x.Name == daemonName && daemonSymbols[n.Sel.Name]:
record("daemon."+n.Sel.Name, n.Sel.Pos())
case vmName != "" && x.Name == vmName && vmSymbols[n.Sel.Name]:
record("vm."+n.Sel.Name, n.Sel.Pos())
}
}
if jsmMethods[n.Sel.Name] || (ambiguousJSMMethods[n.Sel.Name] && jsmReceivers[receiverName(n.X)]) {
record("JetStreamManager."+n.Sel.Name, n.Sel.Pos())
}
// The selected name is a field or method, never a bare package symbol.
ast.Inspect(n.X, visit)
return false
case *ast.FuncDecl:
// A method declared with one of these names is an implementation,
// not a use; its body is still scanned.
if n.Recv != nil {
ast.Inspect(n.Recv, visit)
}
if n.Body != nil {
ast.Inspect(n.Body, visit)
}
if n.Type != nil {
ast.Inspect(n.Type, visit)
}
return false
case *ast.Field:
// Interface method and struct field names are declarations.
if n.Type != nil {
ast.Inspect(n.Type, visit)
}
if n.Tag != nil {
ast.Inspect(n.Tag, visit)
}
return false
case *ast.Ident:
if inDaemon && daemonSymbols[n.Name] {
record("daemon."+n.Name, n.Pos())
}
if inVM && vmSymbols[n.Name] {
record("vm."+n.Name, n.Pos())
}
}
return true
}
ast.Inspect(f, visit)
}
var jsmReceivers = set("jsManager", "jsm", "js")
// receiverName is the last name in x: "js" for both js and a.js.
func receiverName(x ast.Expr) string {
switch x := x.(type) {
case *ast.Ident:
return x.Name
case *ast.SelectorExpr:
return x.Sel.Name
}
return ""
}
// importName returns the name f refers to path by, or "" if f does not import it.
func importName(f *ast.File, path, def string) string {
for _, imp := range f.Imports {
p, err := strconv.Unquote(imp.Path.Value)
if err != nil || p != path {
continue
}
if imp.Name != nil {
if imp.Name.Name == "_" || imp.Name.Name == "." {
return ""
}
return imp.Name.Name
}
return def
}
return ""
}
// ignoredByBuildTag reports a `//go:build ignore` file, which is a standalone
// program rather than part of its directory's package.
func ignoredByBuildTag(groups []*ast.CommentGroup, pkgPos token.Pos) bool {
for _, cg := range groups {
if cg.Pos() >= pkgPos {
break
}
for _, c := range cg.List {
if !constraint.IsGoBuild(c.Text) {
continue
}
expr, err := constraint.Parse(c.Text)
if err == nil && expr.String() == "ignore" {
return true
}
}
}
return false
}
func (s scan) sorted() []use {
out := make([]use, 0, len(s))
for u := range s {
out = append(out, u)
}
sort.Slice(out, func(i, j int) bool {
if out[i].File != out[j].File {
return out[i].File < out[j].File
}
return out[i].Symbol < out[j].Symbol
})
return out
}
+139
View File
@@ -0,0 +1,139 @@
package layering
// allowance is one recorded forbidden import, allowed until its debt clears.
type allowance struct {
From, To string
Debt string // subject of the migration record's debt entry or close-out row
Reason string
}
// allowlist mirrors the dependency-rule debt and close-out table in
// docs/PACKAGE_BOUNDARY_MIGRATION.md. It only shrinks: add an edge only with
// recorded debt, and delete an entry as soon as its import is gone.
var allowlist = []allowance{
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/eip", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/igw", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/image", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/instance", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/key", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/launchtemplate", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/placementgroup", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/routetable", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/snapshot", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/volume", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "operator/accountlifecycle", To: "domains/ec2/awsapi/vpc", Debt: "accountteardown (dependency)", Reason: "EC2 reapers call the awsapi action adapters; needs domain-owned cleanup capabilities"},
{From: "agents/ecs/agent", To: "domains/ecs", Debt: "agents/ecs/agent", Reason: "assignment/state wire types from the ECS implementation; needs a guest/controller contract"},
{From: "agents/eks/tokenwebhook", To: "domains/eks/cluster", Debt: "agents/eks/tokenwebhook", Reason: "WebhookTokenReviewResult from the EKS implementation; needs an EKS contract or projection"},
{From: "agents/rds/agent", To: "domains/rds/instance", Debt: "agents/rds/agent (dependency, narrowed)", Reason: "command protocol, wire and record types; needs a guest/controller wire contract"},
{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/route53/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/elbv2/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/rds/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/ec2/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/ecs/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/eks/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/iam/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/sts/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "runtime/daemon", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
{From: "domains/acm", To: "domains/iam", Debt: "ACM: the IAM AES-GCM helper import (ACM relocation residual)", Reason: "IAM AES-GCM helpers; needs a shared crypto boundary"},
{From: "domains/admission/quota", To: "domains/ec2/awsapi/eip", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "describe adapter used as a counter; needs a reader capability"},
{From: "domains/admission/quota", To: "domains/ec2/awsapi/instance", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "vcpu.go counts instances through the instance adapter; needs an EC2 reader capability"},
{From: "domains/admission/quota", To: "domains/ec2/awsapi/volume", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "describe adapter used as a counter; needs a reader capability"},
{From: "domains/admission/quota", To: "domains/ec2/awsapi/vpc", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "describe adapter used as a counter; needs a reader capability"},
{From: "domains/admission/quota", To: "domains/ec2/instancetypes", Debt: "domains/admission/quota → domains/ec2/instancetypes", Reason: "DefaultVCPUs sizing; needs a consumer-owned capability or EC2 projection"},
{From: "domains/admission/quota", To: "domains/elbv2/awsapi", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "live ELBv2 counter; needs a reader capability"},
{From: "domains/admission/quota", To: "domains/rds/instance", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "live RDS counter; needs a reader capability"},
{From: "domains/admission/quota", To: "runtime/compute/vm", Debt: "domains/admission/quota (dependency, moved from handlers/quota unchanged)", Reason: "instance-record vCPU sweep; needs a reader capability"},
{From: "domains/dns", To: "domains/network/reconcile", Debt: "domains/dns", Reason: "takes the reconcile lock from network's implementation"},
{From: "domains/dns", To: "runtime/compute/vm", Debt: "domains/dns", Reason: "domain-to-runtime import of VM state"},
{From: "domains/ec2/awsapi/instance", To: "domains/iam", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "IAM edge needs an IAM capability"},
{From: "domains/ec2/awsapi/instance", To: "runtime/compute/cache", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "node liveness and VM state; needs an EC2-owned projection"},
{From: "domains/ec2/awsapi/instance", To: "runtime/compute/vm", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "node liveness and VM state; needs an EC2-owned projection"},
{From: "domains/ec2/eip", To: "domains/network/topology", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ec2/guestmetadata", To: "domains/dns", Debt: "Guest metadata: the cross-domain and runtime imports above", Reason: "guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From: "domains/ec2/guestmetadata", To: "domains/iam", Debt: "Guest metadata: the cross-domain and runtime imports above", Reason: "guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From: "domains/ec2/guestmetadata", To: "domains/sts", Debt: "Guest metadata: the cross-domain and runtime imports above", Reason: "guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From: "domains/ec2/guestmetadata", To: "runtime/compute/vm", Debt: "Guest metadata: the cross-domain and runtime imports above", Reason: "guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From: "domains/ec2/instance", To: "domains/dns", Debt: "domains/ec2/instance", Reason: "cross-domain implementation import of DNS"},
{From: "domains/ec2/instance", To: "domains/network/topology", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ec2/instance", To: "runtime/compute/gpu", Debt: "domains/ec2/instance", Reason: "VM/GPU runtime state read directly; needs a domain-owned projection"},
{From: "domains/ec2/instance", To: "runtime/compute/vm", Debt: "domains/ec2/instance", Reason: "VM/GPU runtime state read directly; needs a domain-owned projection"},
{From: "domains/ec2/instance/state", To: "runtime/daemon", Debt: "domains/ec2/instance/state (dependency, IMDS readers)", Reason: "wraps the daemon's local-file reader and JetStreamManager bucket open unchanged for vpcd's IMDS lookup"},
{From: "domains/ec2/instance/state", To: "runtime/compute/vm", Debt: "domains/ec2/instance/state (dependency)", Reason: "the vpcd IMDS local reader returns vm.VM"},
{From: "domains/ec2/volume", To: "runtime/compute/vm", Debt: "Domain → runtime", Reason: "VM runtime state read directly; needs a domain-owned projection"},
{From: "domains/ec2/vpc", To: "domains/network/external", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ec2/vpc", To: "domains/network/external/dhcp", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ec2/vpc", To: "domains/network/identifiers", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ec2/vpc", To: "domains/network/topology", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ecr/auth", To: "domains/iam", Debt: "Cross-domain", Reason: "IAM implementation import; needs an IAM capability"},
{From: "domains/network/host", To: "runtime/compute/vm", Debt: "domains/network", Reason: "domain-to-runtime import"},
{From: "domains/network/host", To: "runtime/host/command", Debt: "domains/network", Reason: "domain-to-runtime import"},
{From: "domains/network/reconcile", To: "domains/ec2/eip", Debt: "domains/network", Reason: "reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From: "domains/network/reconcile", To: "domains/ec2/igw", Debt: "domains/network", Reason: "reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From: "domains/network/reconcile", To: "domains/ec2/natgw", Debt: "domains/network", Reason: "reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From: "domains/network/reconcile", To: "domains/ec2/routetable", Debt: "domains/network", Reason: "reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From: "domains/network/reconcile", To: "domains/ec2/vpc", Debt: "domains/network", Reason: "reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From: "domains/network/reconcile", To: "runtime/compute/vm", Debt: "domains/network", Reason: "domain-to-runtime import"},
{From: "domains/ochre", To: "domains/ec2/systeminstance", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
{From: "domains/ochre", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "secret encrypt/decrypt helpers; needs a shared crypto boundary"},
{From: "domains/ochre", To: "domains/network/systemvpc", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
{From: "domains/ochre", To: "runtime/compute/gpu", Debt: "Domain → runtime", Reason: "GPU runtime state read directly; needs a domain-owned projection"},
{From: "domains/ochre/vector", To: "domains/iam", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
{From: "domains/rds/awsapi", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "enginecatalog.go calls the instance adapter; needs an EC2 capability"},
{From: "domains/sts/awsapi", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "GetCallerIdentity takes IAMService; needs an IAM capability"},
{From: "domains/ecs", To: "domains/ec2/ebs/metadata", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/ecs", To: "domains/ec2/eip", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/ecs", To: "domains/ec2/instancetypes", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/ecs", To: "domains/elbv2", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/ecs", To: "domains/iam", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/eks/cluster", To: "domains/dns", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "domains/ec2/ebs/metadata", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "domains/ec2/instancetypes", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "domains/ec2/placementgroup", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "domains/ec2/systeminstance", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "domains/network/systemvpc", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "domains/iam", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/eks/cluster", To: "runtime/compute/vm", Debt: "close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)", Reason: "legacy EKS calls another domain's or runtime implementation directly"},
{From: "domains/elbv2", To: "domains/acm", Debt: "domains/elbv2 (cross-domain)", Reason: "ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From: "domains/elbv2", To: "domains/dns", Debt: "domains/elbv2 (cross-domain)", Reason: "ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From: "domains/elbv2", To: "domains/ec2/systeminstance", Debt: "domains/elbv2 (cross-domain)", Reason: "ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From: "domains/elbv2", To: "domains/ec2/vpc", Debt: "domains/elbv2 (cross-domain)", Reason: "ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From: "domains/elbv2", To: "domains/network/topology", Debt: "domains/elbv2 (cross-domain)", Reason: "ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From: "domains/elbv2", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "system-instance role ensurer; needs an IAM capability"},
{From: "domains/elbv2", To: "agents/elbv2", Debt: "domains/elbv2/{agent_types,haproxy,health_checker,nginx}.go import lbagent", Reason: "health-report and cert/PID-path vocabulary; needs a guest/controller contract"},
{From: "domains/rds/instance", To: "domains/dns", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/rds/instance", To: "domains/ec2/instance", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/rds/instance", To: "domains/ec2/instancetypes", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/rds/instance", To: "domains/ec2/systeminstance", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/rds/instance", To: "domains/network/systemvpc", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/rds/instance", To: "domains/iam", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/rds/instance", To: "runtime/compute/vm", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/sts", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "STS reads IAM service, trust-policy and OIDC-provider state; needs an IAM capability"},
}
+163
View File
@@ -0,0 +1,163 @@
// Package layering enforces ADR-0001's package-boundary dependency rules over
// the module's production import graph, ratcheted by a recorded-debt allowlist.
package layering
import (
"errors"
"fmt"
"go/ast"
"go/build/constraint"
"go/parser"
"go/token"
"io/fs"
"os"
"path/filepath"
"runtime"
"sort"
"strconv"
"strings"
)
const modulePath = "github.com/mulgadc/spinifex"
// edge is one package-to-package import, both sides named as in the allowlist:
// application packages relative to spinifex/, module-root ones (cmd/,
// contracts/, internal/) relative to the module root.
type edge struct {
From, To string
}
func (e edge) String() string { return e.From + " -> " + e.To }
// graph is the production import graph restricted to this module.
type graph struct {
// pkgs maps a package name to its module-relative directory.
pkgs map[string]string
// imports records the first file:line at which each edge appears.
imports map[edge]string
}
func newGraph() *graph {
return &graph{pkgs: map[string]string{}, imports: map[edge]string{}}
}
// shortName drops the application-root prefix so names read like the ADR and
// the migration record; classify rejects app-root dirs that would collide.
func shortName(full string) string {
if rest, ok := strings.CutPrefix(full, "spinifex/"); ok {
return rest
}
return full
}
func moduleRoot() (string, error) {
_, file, _, ok := runtime.Caller(0)
if !ok {
return "", errors.New("cannot locate test source")
}
// architecture/layering/graph_test.go -> module root.
return filepath.Abs(filepath.Join(filepath.Dir(file), "..", ".."))
}
// excludedTopDirs hold no production code: tests/ is test-only, docs/ only
// embeds documentation and architecture/ is repository governance tooling.
var excludedTopDirs = map[string]bool{"tests": true, "docs": true, "architecture": true}
// loadGraph parses the imports of every non-test Go file under root, so it
// sees every build-tag variant and needs no Go toolchain or workspace.
func loadGraph(root string) (*graph, error) {
g := newGraph()
fset := token.NewFileSet()
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil {
return err
}
rel = filepath.ToSlash(rel)
if d.IsDir() {
if rel == "." {
return nil
}
name := d.Name()
if strings.HasPrefix(name, ".") || strings.HasPrefix(name, "_") ||
name == "testdata" || name == "vendor" || name == "node_modules" || excludedTopDirs[rel] {
return filepath.SkipDir
}
if _, err := os.Stat(filepath.Join(path, "go.mod")); err == nil {
return filepath.SkipDir
}
return nil
}
if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
return nil
}
f, err := parser.ParseFile(fset, path, nil, parser.ImportsOnly|parser.ParseComments)
if err != nil {
return err
}
if ignoredByBuildTag(f.Comments, f.Package) {
return nil
}
dir := filepath.ToSlash(filepath.Dir(rel))
if dir == "." {
dir = ""
}
from := shortName(dir)
g.pkgs[from] = dir
for _, imp := range f.Imports {
p, err := strconv.Unquote(imp.Path.Value)
if err != nil {
return err
}
target, ok := strings.CutPrefix(p, modulePath+"/")
if !ok {
continue
}
e := edge{From: from, To: shortName(target)}
if _, seen := g.imports[e]; !seen {
pos := fset.Position(imp.Pos())
g.imports[e] = fmt.Sprintf("%s:%d", rel, pos.Line)
}
}
return nil
})
return g, err
}
// ignoredByBuildTag reports a `//go:build ignore` file, which is a standalone
// program (a generator) rather than part of its directory's package.
func ignoredByBuildTag(groups []*ast.CommentGroup, pkgPos token.Pos) bool {
for _, cg := range groups {
if cg.Pos() >= pkgPos {
break
}
for _, c := range cg.List {
if !constraint.IsGoBuild(c.Text) {
continue
}
expr, err := constraint.Parse(c.Text)
if err == nil && expr.String() == "ignore" {
return true
}
}
}
return false
}
// sortedEdges returns the graph's edges in a stable order.
func (g *graph) sortedEdges() []edge {
out := make([]edge, 0, len(g.imports))
for e := range g.imports {
out = append(out, e)
}
sort.Slice(out, func(i, j int) bool {
if out[i].From != out[j].From {
return out[i].From < out[j].From
}
return out[i].To < out[j].To
})
return out
}
+240
View File
@@ -0,0 +1,240 @@
package layering
import (
"fmt"
"sort"
"strings"
"testing"
)
type violation struct {
rule string
edge edge
pos string
}
type report struct {
unmapped []string
violations []violation // every rule hit, allowlisted or not
unrecorded []violation // hits whose edge has no allowlist entry
stale []allowance // entries whose edge no rule rejects any more
malformed []string
}
// check classifies g, applies every rule and compares the result with allow.
func check(g *graph, allow []allowance) report {
var rep report
classes := map[string]class{}
for name, dir := range g.pkgs {
c, ok := classify(dir)
if !ok || c.layer == layerTests {
rep.unmapped = append(rep.unmapped, fmt.Sprintf("%s (directory %s)", name, dir))
continue
}
classes[name] = c
}
allowed := map[edge]bool{}
for _, a := range allow {
e := edge{From: a.From, To: a.To}
switch {
case allowed[e]:
rep.malformed = append(rep.malformed, "duplicate entry "+e.String())
case a.Debt == "" || a.Reason == "":
rep.malformed = append(rep.malformed, "entry "+e.String()+" needs both Debt and Reason")
}
allowed[e] = true
}
hit := map[edge]bool{}
for _, e := range g.sortedEdges() {
from, ok := classes[e.From]
if !ok {
continue
}
to, ok := classify(fullPath(e.To))
if !ok {
rep.unmapped = append(rep.unmapped, fmt.Sprintf("%s (imported by %s at %s)", e.To, e.From, g.imports[e]))
continue
}
for _, r := range rules {
if !r.subject(from, e.From) || !r.forbids(from, to, e.From) {
continue
}
v := violation{rule: r.id, edge: e, pos: g.imports[e]}
rep.violations = append(rep.violations, v)
hit[e] = true
if !allowed[e] {
rep.unrecorded = append(rep.unrecorded, v)
}
}
}
for _, a := range allow {
if !hit[edge{From: a.From, To: a.To}] {
rep.stale = append(rep.stale, a)
}
}
sort.Strings(rep.unmapped)
return rep
}
func ruleByID(id string) rule {
for _, r := range rules {
if r.id == id {
return r
}
}
panic("unknown rule " + id)
}
// TestADR0001_Layering is the blocking ratchet over the real import graph.
func TestADR0001_Layering(t *testing.T) {
root, err := moduleRoot()
if err != nil {
t.Fatal(err)
}
g, err := loadGraph(root)
if err != nil {
t.Fatal(err)
}
// Guards against a walk that silently finds nothing and passes vacuously.
if len(g.pkgs) < 100 || len(g.imports) < 500 {
t.Fatalf("scanned %d packages and %d module imports under %s; the walk has lost the tree and every rule would pass vacuously",
len(g.pkgs), len(g.imports), root)
}
rep := check(g, allowlist)
t.Run("Classification_EveryPackageMapped", func(t *testing.T) {
for _, u := range rep.unmapped {
t.Errorf("ADR-0001 current-to-target map: package %s has no layer; place it under an ADR-0001 home or classify it deliberately in layers_test.go", u)
}
})
for _, r := range rules {
t.Run(r.id, func(t *testing.T) {
subjects := 0
for name, dir := range g.pkgs {
if c, ok := classify(dir); ok && r.subject(c, name) {
subjects++
}
}
if subjects == 0 {
t.Fatalf("no scanned package is governed by %s; the rule would pass vacuously", r.id)
}
var snippets []string
for _, v := range rep.unrecorded {
if v.rule != r.id {
continue
}
t.Errorf("ADR-0001 %s: %s imports %s (%s); %s",
r.clause, v.edge.From, v.edge.To, v.pos, r.hint)
snippets = append(snippets, fmt.Sprintf("{From: %q, To: %q, Debt: \"<migration-record debt subject>\", Reason: \"<why>\"},", v.edge.From, v.edge.To))
}
if len(snippets) > 0 {
t.Logf("fix the import; only debt already recorded in docs/PACKAGE_BOUNDARY_MIGRATION.md may be allow-listed:\n%s",
strings.Join(snippets, "\n"))
}
})
}
t.Run("Allowlist_NoStaleEntries", func(t *testing.T) {
for _, a := range rep.stale {
t.Errorf("allowlist entry %s -> %s (%s) no longer matches a forbidden import; the debt is paid, so delete the entry from allowlist_test.go",
a.From, a.To, a.Debt)
}
})
t.Run("Allowlist_WellFormed", func(t *testing.T) {
for _, m := range rep.malformed {
t.Errorf("allowlist_test.go: %s", m)
}
t.Logf("%d allow-listed edges", len(allowlist))
})
}
// TestADR0001_LayeringCheckerDetectsFaults injects a forbidden edge, a stale
// allowlist entry and an unmapped package into a synthetic graph.
func TestADR0001_LayeringCheckerDetectsFaults(t *testing.T) {
g := newGraph()
for _, dir := range []string{
"spinifex/foundation/widget", "spinifex/domains/alpha/thing",
"spinifex/domains/beta/store", "spinifex/misc",
} {
g.pkgs[shortName(dir)] = dir
}
forbidden := edge{From: "foundation/widget", To: "domains/alpha/thing"}
g.imports[forbidden] = "spinifex/foundation/widget/w.go:7"
allowedEdge := edge{From: "domains/alpha/thing", To: "domains/beta/store"}
g.imports[allowedEdge] = "spinifex/domains/alpha/thing/t.go:9"
staleEntry := allowance{From: "domains/beta/store", To: "domains/alpha/thing", Debt: "synthetic", Reason: "stale"}
rep := check(g, []allowance{
{From: allowedEdge.From, To: allowedEdge.To, Debt: "synthetic", Reason: "recorded"},
staleEntry,
})
if len(rep.unrecorded) != 1 || rep.unrecorded[0].edge != forbidden ||
rep.unrecorded[0].rule != ruleByID("R1_FoundationImportsOnlyFoundation").id {
t.Errorf("want only %s reported under R1, got %+v", forbidden, rep.unrecorded)
}
if len(rep.stale) != 1 || rep.stale[0] != staleEntry {
t.Errorf("want stale entry %+v, got %+v", staleEntry, rep.stale)
}
if len(rep.unmapped) != 1 || !strings.HasPrefix(rep.unmapped[0], "misc ") {
t.Errorf("want unmapped package misc, got %v", rep.unmapped)
}
var allowedHit bool
for _, v := range rep.violations {
allowedHit = allowedHit || v.edge == allowedEdge
}
if !allowedHit {
t.Errorf("allow-listed cross-domain edge %s was not detected by R3, so the allowlist entry would read as stale", allowedEdge)
}
}
// TestADR0001_InterimDaemonDirectionCoversRuntimeDaemon: the daemon left its
// legacy root, so gateway importing runtime/daemon must still be rejected, and
// the former handlers packages are held to it by R5 at their domain homes.
func TestADR0001_InterimDaemonDirectionCoversRuntimeDaemon(t *testing.T) {
daemon, _ := classify("spinifex/runtime/daemon")
checks := []struct{ rule, from string }{
{"Interim_DaemonGatewayHandlersDirection", "spinifex/gateway"},
{"R5_DomainImportsNoRuntimeOrOperator", "spinifex/domains/ecs"},
}
for _, c := range checks {
r := ruleByID(c.rule)
f, _ := classify(c.from)
if !r.subject(f, c.from) || !r.forbids(f, daemon, c.from) {
t.Errorf("%s -> runtime/daemon is not rejected by %s", c.from, c.rule)
}
}
other, _ := classify("spinifex/runtime/host/process")
if gw, _ := classify("spinifex/gateway"); ruleByID("Interim_DaemonGatewayHandlersDirection").forbids(gw, other, "spinifex/gateway") {
t.Error("gateway -> runtime/host/process is rejected, but it is not the daemon")
}
}
// TestADR0002_S7_RequestRecordRejectsOutsiders: the request record is readable
// by the gateway and ingress/aws only.
func TestADR0002_S7_RequestRecordRejectsOutsiders(t *testing.T) {
record, ok := classify("spinifex/ingress/aws/reqctx")
if !ok || !record.requestRecord {
t.Fatalf("ingress/aws/reqctx is not classified as the request record: %+v", record)
}
r := ruleByID("S7_RequestRecordPrivateToIngress")
for name, wantRejected := range map[string]bool{
"domains/ecs/awsapi": true,
"runtime/roles/awsgw": true,
"operator/admin": true,
"gateway": false,
"ingress/aws/dispatch": false,
} {
from, ok := classify(fullPath(name))
if !ok {
t.Fatalf("%s is unclassified", name)
}
if got := r.forbids(from, record, name); got != wantRejected {
t.Errorf("%s -> ingress/aws/reqctx rejected=%v, want %v", name, got, wantRejected)
}
}
}
+159
View File
@@ -0,0 +1,159 @@
package layering
import (
"slices"
"strings"
)
type layer string
const (
layerFoundation layer = "foundation"
layerContracts layer = "contracts"
layerDomain layer = "domains"
layerIngress layer = "ingress"
layerRuntime layer = "runtime"
layerOperator layer = "operator"
layerAgents layer = "agents"
layerProviders layer = "providers"
layerBootstrap layer = "bootstrap"
layerCmd layer = "cmd"
layerInternal layer = "internal"
layerTestkit layer = "testkit"
layerLegacy layer = "legacy"
// layerTests is valid only as an import target; tests/ is never scanned.
layerTests layer = "tests"
)
// domainKind distinguishes the parts of a domain the rules treat differently.
type domainKind int
const (
notDomain domainKind = iota
domainImpl // domains/<d>/... resource implementation
domainAWSAPI // domains/<d>/awsapi/... AWS action adapters
domainProj // domains/<d>/projection/... authorised view
legacyImpl // handlers/<d>/... legacy domain implementation
legacyAdapter // gateway/<d>/... legacy AWS action adapters
)
type class struct {
layer layer
legacy string // legacy root name when layer is layerLegacy
domain string // owning domain for domain-shaped and agent packages
kind domainKind
daemon bool // runtime/daemon, the interim daemon -> gateway -> handlers root
// requestRecord marks ingress/aws/reqctx, the request record ADR-0002:S7 keeps private to ingress.
requestRecord bool
}
func (c class) domainShaped() bool { return c.kind != notDomain }
// isLegacy reports whether c sits under one of the named legacy roots.
func (c class) isLegacy(roots ...string) bool {
if c.layer != layerLegacy {
return false
}
if len(roots) == 0 {
return true
}
return slices.Contains(roots, c.legacy)
}
// legacyRoots are pre-ADR-0001 application roots still standing.
var legacyRoots = map[string]bool{
"gateway": true,
}
// gatewayDomains names the domain each gateway/<service> adapter belongs to;
// Bedrock is the AWS surface of the Ochre domain.
var gatewayDomains = map[string]string{
"bedrock": "ochre", "ecs": "ecs", "eks": "eks", "elbv2": "elbv2", "iam": "iam",
"rds": "rds", "sts": "sts",
}
// classify maps a module-relative package directory to its layer. ok is false
// for a package that has no deliberate classification.
func classify(full string) (class, bool) {
parts := strings.Split(full, "/")
switch parts[0] {
case "cmd":
return class{layer: layerCmd}, len(parts) >= 2
case "contracts":
return class{layer: layerContracts}, len(parts) >= 2
case "internal":
if len(parts) < 2 {
return class{}, false
}
if parts[1] == "testkit" || parts[1] == "tooling" {
return class{layer: layerTestkit}, true
}
return class{layer: layerInternal}, true
case "tests":
return class{layer: layerTests}, true
case "spinifex":
return classifyApp(parts[1:])
}
return class{}, false
}
func classifyApp(app []string) (class, bool) {
if len(app) == 0 {
return class{}, false
}
switch app[0] {
case "foundation":
return class{layer: layerFoundation}, true
case "ingress":
return class{layer: layerIngress, requestRecord: len(app) >= 3 && app[1] == "aws" && app[2] == "reqctx"}, true
case "runtime":
return class{layer: layerRuntime, daemon: len(app) >= 2 && app[1] == "daemon"}, true
case "operator":
return class{layer: layerOperator}, true
case "providers":
return class{layer: layerProviders}, true
case "bootstrap":
return class{layer: layerBootstrap}, true
case "agents":
c := class{layer: layerAgents}
if len(app) >= 2 {
c.domain = app[1]
}
return c, true
case "domains":
if len(app) < 2 {
return class{}, false
}
c := class{layer: layerDomain, domain: app[1], kind: domainImpl}
if len(app) >= 3 {
switch app[2] {
case "awsapi":
c.kind = domainAWSAPI
case "projection":
c.kind = domainProj
}
}
return c, true
}
if !legacyRoots[app[0]] {
return class{}, false
}
c := class{layer: layerLegacy, legacy: app[0]}
switch {
case app[0] == "handlers" && len(app) >= 2:
c.domain, c.kind = app[1], legacyImpl
case app[0] == "gateway" && len(app) >= 2 && gatewayDomains[app[1]] != "":
c.domain, c.kind = gatewayDomains[app[1]], legacyAdapter
}
return c, true
}
// fullPath reverses shortName for classification.
func fullPath(name string) string {
top, _, _ := strings.Cut(name, "/")
switch top {
case "cmd", "contracts", "internal", "tests", "docs":
return name
}
return "spinifex/" + name
}
+63
View File
@@ -0,0 +1,63 @@
package layering
import (
"os"
"path/filepath"
"testing"
)
// frozenLegacyPackages is every production package still under a legacy root.
// It only shrinks: a new entry is a new legacy package, a missing one is a
// completed move whose line must be deleted.
var frozenLegacyPackages = []string{
"gateway",
}
// retiredLegacyRoots were legacy roots whose every package has moved to its
// ADR-0001 target; recreating one would reopen a closed migration row.
var retiredLegacyRoots = []string{"accountteardown", "admin", "daemon", "handlers", "lbagent", "services", "utils", "vpcd"}
// TestADR0001_LegacyRootsFrozen: the current-to-target map gives every legacy
// root a destination, so none may gain a package and no retired root may return.
func TestADR0001_LegacyRootsFrozen(t *testing.T) {
root, err := moduleRoot()
if err != nil {
t.Fatal(err)
}
g, err := loadGraph(root)
if err != nil {
t.Fatal(err)
}
present := map[string]bool{}
for name, dir := range g.pkgs {
if c, ok := classify(dir); ok && c.layer == layerLegacy {
present[name] = true
}
}
if len(present) == 0 {
t.Fatal("found no legacy packages; the freeze would pass vacuously")
}
frozen := map[string]bool{}
for _, p := range frozenLegacyPackages {
frozen[p] = true
if !present[p] {
t.Errorf("legacy package %s no longer exists; delete it from frozenLegacyPackages", p)
}
}
for p := range present {
if !frozen[p] {
t.Errorf("ADR-0001 current-to-target map: %s is a new package under a legacy root; create it at its ADR-0001 target home instead", p)
}
}
for _, r := range retiredLegacyRoots {
entries, err := os.ReadDir(filepath.Join(root, "spinifex", r))
if err != nil && !os.IsNotExist(err) {
t.Fatal(err)
}
if len(entries) > 0 {
t.Errorf("ADR-0001 current-to-target map: spinifex/%s is a retired legacy root; give the code its target home instead", r)
}
}
}
+184
View File
@@ -0,0 +1,184 @@
package layering
import (
"slices"
"strings"
)
// rule is one ADR-0001 dependency rule over a classified import edge.
type rule struct {
id string // subtest name, led by the ADR-0001 clause it encodes
clause string // the ADR text the rule enforces, quoted in failures
hint string // one-line fix direction
// subject reports whether an importer is governed by the rule; the
// non-empty guard requires at least one governed package per rule.
subject func(from class, fromName string) bool
// forbids reports whether the rule rejects the edge.
forbids func(from, to class, fromName string) bool
}
// awsgwRole is the composition root ADR-0002 lets wire domain registrations.
const awsgwRole = "runtime/roles/awsgw"
// qualificationBinaries are the developer/qualification mains ADR-0001 lets
// keep reusable source in internal/testkit or internal/tooling.
var qualificationBinaries = map[string]bool{
"cmd/aws-model-coverage": true, "cmd/dhcptest": true, "cmd/spx-loadgen": true,
}
func isLayer(c class, layers ...layer) bool {
return slices.Contains(layers, c.layer)
}
// importsDomainImplementation is any domain-owned package other than a named
// projection, in its target or legacy home.
func importsDomainImplementation(to class) bool {
return to.domainShaped() && to.kind != domainProj
}
var rules = []rule{
{
id: "R1_FoundationImportsOnlyFoundation",
clause: "rule 1: foundation imports neither domains, ingress, runtime nor operator (enforced conservatively: foundation imports no module package above foundation)",
hint: "move the dependency's needed primitive into foundation or invert it through a parameter",
subject: func(f class, _ string) bool { return f.layer == layerFoundation },
forbids: func(_, t class, _ string) bool { return t.layer != layerFoundation },
},
{
id: "R2_IngressImportsNoImplementation",
clause: "rule 2: ingress/aws imports no domain implementation; domains supply their own action registration through the ingress registration contract",
hint: "take the capability through the registration contract wired by runtime/roles/awsgw",
subject: func(f class, _ string) bool { return f.layer == layerIngress },
forbids: func(_, t class, _ string) bool {
return !isLayer(t, layerFoundation, layerContracts, layerIngress)
},
},
{
id: "R3_DomainImportsNoOtherDomainImplementation",
clause: "rule 3: a domain does not read another domain's private records, migrations or repository; it uses a consumer-owned capability, a named projection, or a versioned contracts/<domain>/v1 contract",
hint: "declare a narrow capability in the consumer, or import the owner's projection or contracts/<domain>/v1",
subject: func(f class, _ string) bool { return f.domainShaped() },
forbids: func(f, t class, _ string) bool {
return f.domainShaped() && importsDomainImplementation(t) && t.domain != f.domain
},
},
{
id: "R4_ProviderImportsNoDomainImplementation",
clause: "rule 4: a provider adapter implements a provider contract and cannot acquire AWS-visible resource authority by reading a domain's private state",
hint: "pass the needed values into the provider contract instead of importing the domain",
subject: func(f class, _ string) bool { return f.layer == layerProviders },
forbids: func(_, t class, _ string) bool { return t.domainShaped() || t.isLegacy() },
},
{
id: "R5_DomainImportsNoRuntimeOrOperator",
clause: "rule 5: operator and runtime are composition consumers; they wire domain modules, so a domain does not import them",
hint: "expose the runtime or operator state through a domain-owned projection or a capability wired at the composition root",
subject: func(f class, _ string) bool { return f.domainShaped() },
forbids: func(f, t class, _ string) bool {
return f.domainShaped() && isLayer(t, layerRuntime, layerOperator)
},
},
{
id: "R5_AWSAPIAdaptersImportedOnlyByOwnerOrComposition",
clause: "rule 5: operator and runtime call public use cases; a domain's awsapi action adapters are owned by that domain and wired only by the composition root",
hint: "call an owner-provided capability instead of the AWS action adapter",
subject: func(_ class, _ string) bool { return true },
forbids: func(f, t class, fromName string) bool {
if t.kind != domainAWSAPI || fromName == awsgwRole {
return false
}
return !f.domainShaped() || f.domain != t.domain
},
},
{
id: "R5_OperatorImportsNoLegacyImplementation",
clause: "rule 5 and the cmd/spinifex map row: operator use cases reach domains through public capabilities and do not read domain-private state",
hint: "reach the domain through its public capability; do not import a legacy implementation root",
subject: func(f class, _ string) bool { return f.layer == layerOperator },
forbids: func(f, t class, _ string) bool { return f.layer == layerOperator && t.isLegacy() },
},
{
id: "R6_ProductionImportsNoTestkit",
clause: "rule 6: test support is under internal/testkit; production packages do not import it",
hint: "move the shared code to its production owner or keep the import in a _test.go file",
subject: func(f class, n string) bool { return f.layer != layerTestkit && !qualificationBinaries[n] },
forbids: func(f, t class, n string) bool {
return f.layer != layerTestkit && !qualificationBinaries[n] && isLayer(t, layerTestkit, layerTests)
},
},
{
id: "Contracts_ImportOnlyFoundationAndContracts",
clause: "decision item 1 and the area table: contracts own cross-process vocabulary, not application interfaces, domain persistence or transport implementation",
hint: "keep wire types self-contained; depend only on foundation or another contract",
subject: func(f class, _ string) bool { return f.layer == layerContracts },
forbids: func(_, t class, _ string) bool { return !isLayer(t, layerFoundation, layerContracts) },
},
{
id: "Domains_ImportNoAgentsIngressOrLegacy",
clause: "area table and domain module shape: a domain owns no other domain's implementation; only its awsapi adapters may use the generic ingress/aws machinery",
hint: "depend on a contract or capability; leave ingress use to the domain's awsapi package",
subject: func(f class, _ string) bool { return f.domainShaped() },
forbids: func(f, t class, _ string) bool {
if !f.domainShaped() {
return false
}
switch {
case t.layer == layerAgents:
return true
case t.layer == layerIngress:
return f.kind != domainAWSAPI && f.kind != legacyAdapter
case t.isLegacy("gateway") && !t.domainShaped():
return true
case t.isLegacy("handlers", "gateway"):
return f.layer == layerDomain
}
return false
},
},
{
id: "Agents_ImportNoDomainOrLegacyImplementation",
clause: "area table: agents/<domain> own in-guest and service-role behaviour, not tenant API authority or another domain's private state",
hint: "move the shared wire vocabulary behind a guest/controller contract under contracts/",
subject: func(f class, _ string) bool { return f.layer == layerAgents },
forbids: func(f, t class, _ string) bool {
return f.layer == layerAgents && (t.domainShaped() || t.isLegacy())
},
},
{
id: "Q22_STSImportsNoEKSImplementation",
clause: "Q-22 prohibits the STS-to-EKS dependency in its present form; cut it through an identity-owned contract or an EKS projection",
hint: "consume an EKS projection or an identity-owned contract instead of the EKS implementation",
subject: func(f class, _ string) bool { return f.domainShaped() && f.domain == "sts" },
forbids: func(f, t class, _ string) bool {
return f.domainShaped() && f.domain == "sts" && t.domain == "eks" &&
(importsDomainImplementation(t) || t.layer == layerAgents)
},
},
{
id: "S7_RequestRecordPrivateToIngress",
clause: "ADR-0002:S7: the request record is private to ingress, so no domain, runtime or operator package reads it; only the gateway and packages under ingress/aws may import ingress/aws/reqctx",
hint: "receive what you need through dispatch.Invocation instead of the request record",
subject: func(_ class, _ string) bool { return true },
forbids: func(_, t class, fromName string) bool {
if !t.requestRecord {
return false
}
return fromName != "gateway" && !strings.HasPrefix(fromName, "ingress/aws/")
},
},
{
id: "Interim_DaemonGatewayHandlersDirection",
clause: "during migration the existing daemon -> gateway -> handlers rule remains the guardrail (Q-20, Q-29)",
hint: "keep the dependency pointing daemon -> gateway -> handlers; invert it through a callback or capability",
subject: func(f class, _ string) bool { return f.isLegacy("handlers", "gateway") },
forbids: func(f, t class, _ string) bool {
switch {
case f.isLegacy("handlers"):
return t.isLegacy("gateway") || t.daemon
case f.isLegacy("gateway"):
return t.daemon
}
return false
},
},
}
+10 -3
View File
@@ -1,6 +1,6 @@
ARG VERSION=dev
FROM golang:1.27-bookworm AS builder
FROM golang:1.27.3-bookworm@sha256:972b4110ba0f47539a975c31e52c64765e1ff19f2d74469ae4a595c6676e80a5 AS builder
ARG VERSION
@@ -35,16 +35,23 @@ COPY bluebottle/ ./bluebottle/
# Build spinifex binary (spx)
RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
cd spinifex && GOFIPS140=v1.0.0 go build -ldflags "-s -w \
-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
-X github.com/mulgadc/spinifex/spinifex/operator/cli.Version=${VERSION} \
-X github.com/mulgadc/spinifex/spinifex/operator/cli.Commit=${COMMIT}" \
-o /out/spx cmd/spinifex/main.go
# Build nbdkit viperblock plugin (.so)
RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
# One tarball ships to every OS. The ceiling is what both binaries need today,
# so a rise is a decision rather than an accident; every supported OS has 2.39+.
RUN spinifex/scripts/check-glibc-ceiling.sh 2.34 /out/spx /out/nbdkit-viperblock-plugin.so
# Copy setup scripts and systemd units
RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
RUN cd spinifex/scripts && cp install-node.sh smoke-test.sh node-reset.sh uninstall-spx.sh /out/
RUN mkdir -p /out/udev && cp spinifex/build/udev/* /out/udev/
RUN mkdir -p /out/lib && cp spinifex/scripts/lib/distro.sh /out/lib/distro.sh
RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex
+96
View File
@@ -0,0 +1,96 @@
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devtmpfs dev /dev
# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
# already enumerated by the time PID 1 runs — no modprobe needed.
# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
#
# modprobe needs a usable modules.dep; when the image is built on a host whose
# depmod can't produce one, busybox modprobe silently no-ops and the module
# never loads — fw_cfg then never appears and init panics. insmod by explicit
# path needs no modules.dep, so fall back to it. Handles .ko and .ko.gz.
load_mod() {
modprobe "$1" 2>/dev/null && return 0
for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
insmod "$ko" 2>/dev/null && return 0
case "$ko" in
*.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
&& insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
esac
done
return 1
}
load_mod qemu_fw_cfg || load_mod fw_cfg_sysfs || true
# virtio_net depends on net_failover, which depends on failover. The load_mod
# insmod fallback loads a single .ko with no dependency resolution, and
# modules.dep is absent when the image is built on a kmod-less host (depmod
# skipped), so modprobe can't resolve the chain either. Load it explicitly
# bottom-up so the NIC comes up regardless of how the image was built —
# without this the agent has no network and the LB hangs in provisioning.
load_mod failover || true
load_mod net_failover || true
load_mod virtio_net || true
NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
i=0
while [ "$i" -lt 25 ]; do
[ -f "$NETCFG" ] && break
sleep 0.04
i=$((i + 1))
done
if [ ! -f "$NETCFG" ]; then
# Dump host/guest fw_cfg state to the serial console before init exits so a
# "netcfg not available" panic is diagnosable from the captured console log
# (module-not-loaded vs qemu-not-exposing-by_name vs wrong machine type).
echo "[init] FATAL: fw_cfg netcfg not available" >&2
echo "[init] dmi: vendor=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null) product=$(cat /sys/class/dmi/id/product_name 2>/dev/null) bios=$(cat /sys/class/dmi/id/bios_version 2>/dev/null)" >&2
echo "[init] qemu_fw_cfg dir present: $([ -d /sys/firmware/qemu_fw_cfg ] && echo yes || echo NO)" >&2
echo "[init] modules: $(lsmod 2>/dev/null | awk 'NR>1{print $1}' | tr '\n' ',')" >&2
echo "[init] by_name listing:" >&2
ls -la /sys/firmware/qemu_fw_cfg/by_name/ 2>&1 | sed 's/^/[init] /' >&2
ls -la /sys/firmware/qemu_fw_cfg/by_name/opt/ 2>&1 | sed 's/^/[init] /' >&2
echo "[init] dmesg fw_cfg:" >&2
dmesg 2>/dev/null | grep -i fw_cfg | sed 's/^/[init] /' >&2
exit 1
fi
. "$NETCFG"
for n in 0 1 2 3 4 5; do
eval mac=\$NIC${n}_MAC
[ -z "$mac" ] && continue
iface=""
for attempt in 1 2 3 4 5; do
iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
[ -n "$iface" ] && break
sleep 0.02
done
if [ -z "$iface" ]; then
echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
continue
fi
eval cidr=\$NIC${n}_CIDR
eval gw=\$NIC${n}_GW
eval default=\$NIC${n}_DEFAULT
eval rdst=\$NIC${n}_ROUTE_DST
eval rvia=\$NIC${n}_ROUTE_VIA
ip link set "$iface" up
[ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
[ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
[ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
done
# The store image takes no lb-agent environment. A CA bundle is optional here:
# nothing in this image dials out yet.
CA_CERT=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw
mkdir -p /etc/ssl/certs
[ -f "$CA_CERT" ] && cp "$CA_CERT" /etc/ssl/certs/ca-certificates.crt
exec /sbin/init
+10
View File
@@ -0,0 +1,10 @@
# The power button loads here, not in /init: busybox init has its SIGUSR2
# (poweroff) handler installed before sysinit runs, so a press can never reach
# the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
::sysinit:/sbin/modprobe tiny_power_button power_signal=12
::sysinit:/usr/local/lib/sqs-store/sqs-store-prepare
::respawn:/usr/local/lib/sqs-store/sqs-store-run
::ctrlaltdel:/sbin/reboot
# One action: the store stops and its disk unmounts first, and the generic
# umount -a -r runs only if the store is gone.
::shutdown:/usr/local/lib/sqs-store/sqs-store-stop
+224
View File
@@ -0,0 +1,224 @@
# sqs-store-lib.sh — shared by the SQS store guest's prepare, run and stop
# steps. Sourced, never executed. Every path and command is overridable so the
# host test can drive the same code against a fake fw_cfg, sysfs and tools.
#
# Guest contract, read from fw_cfg under opt/spinifex/sqs-store/:
# data-serial virtio-blk serial of the already-formatted ext4 data disk
# listen-addr IPv4 address on a configured guest NIC; the only listener
# client-nkey NATS user NKey public key allowed to connect; never a seed
# Anything absent, malformed or ambiguous refuses the store. Nothing formats.
SQS_FWCFG=${SQS_FWCFG:-/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/sqs-store}
SQS_SYSBLOCK=${SQS_SYSBLOCK:-/sys/block}
SQS_DEVDIR=${SQS_DEVDIR:-/dev}
# The host test cannot create block nodes, so it relaxes this to -e.
SQS_NODE_TEST=${SQS_NODE_TEST:--b}
SQS_RUN=${SQS_RUN:-/run/sqs-store}
SQS_DATA=${SQS_DATA:-/data}
SQS_CONSOLE=${SQS_CONSOLE:-/dev/console}
SQS_MOUNTS=${SQS_MOUNTS:-/proc/mounts}
SQS_PORT=${SQS_PORT:-4222}
SQS_STOP_BUDGET_TENTHS=${SQS_STOP_BUDGET_TENTHS:-120}
NATS_BIN=${NATS_BIN:-/usr/local/bin/nats-server}
E2FSCK=${E2FSCK:-e2fsck}
MOUNT=${MOUNT:-mount}
UMOUNT=${UMOUNT:-umount}
IP=${IP:-ip}
sqs_log() { echo "[sqs-store] $*" >> "$SQS_CONSOLE"; }
sqs_fatal() { sqs_log "FATAL: $*"; return 1; }
# sqs_fwcfg <name>: print one fw_cfg value, refusing a missing, empty, oversized
# or multi-token file. The value is not echoed on failure.
sqs_fwcfg() {
f="$SQS_FWCFG/$1/raw"
[ -f "$f" ] || { sqs_fatal "fw_cfg $1 not supplied"; return 1; }
[ "$(wc -c < "$f")" -le 128 ] || { sqs_fatal "fw_cfg $1 is oversized"; return 1; }
v=$(cat "$f")
case "$v" in
"" | *[[:space:]]*) sqs_fatal "fw_cfg $1 must be exactly one value"; return 1 ;;
esac
printf '%s' "$v"
}
# A volume serial is the volume ID with dashes stripped: vol + 17 hex.
sqs_valid_serial() {
[ "${#1}" -eq 20 ] || return 1
case "$1" in vol*) ;; *) return 1 ;; esac
case "${1#vol}" in *[!0-9a-f]*) return 1 ;; esac
}
# A NATS user public NKey: "U" then 55 base32 characters. The checksum is
# left to the pinned server's own config test.
sqs_valid_user_nkey() {
[ "${#1}" -eq 56 ] || return 1
case "$1" in U*) ;; *) return 1 ;; esac
case "$1" in *[!A-Z2-7]*) return 1 ;; esac
}
sqs_valid_ipv4() {
case "$1" in *[!0-9.]* | .* | *. | *..*) return 1 ;; esac
old_ifs=$IFS; IFS=.
# shellcheck disable=SC2086 # split on dots on purpose
set -- $1
IFS=$old_ifs
[ "$#" -eq 4 ] || return 1
for o in "$@"; do
[ "${#o}" -le 3 ] && [ "$o" -le 255 ] || return 1
case "$o" in 0?*) return 1 ;; esac
done
}
# sqs_find_disk <serial>: print the /dev node of the one block device whose
# sysfs serial matches exactly. Zero or several matches refuse.
sqs_find_disk() {
found="" n=0
for b in "$SQS_SYSBLOCK"/*; do
[ -r "$b/serial" ] || continue
[ "$(cat "$b/serial")" = "$1" ] || continue
found=${b##*/}
n=$((n + 1))
done
[ "$n" -eq 1 ] || { sqs_fatal "expected one disk with serial $1, found $n"; return 1; }
test "$SQS_NODE_TEST" "$SQS_DEVDIR/$found" || { sqs_fatal "disk $found has no block device node"; return 1; }
printf '%s' "$SQS_DEVDIR/$found"
}
# sqs_has_addr <ipv4>: the address is assigned to a non-loopback interface.
sqs_has_addr() {
"$IP" -o -4 addr show 2>/dev/null \
| awk -v a="$1" '$2 != "lo" { split($4, p, "/"); if (p[1] == a) f = 1 } END { exit !f }'
}
# sqs_alive <pid>: running or stopped, not exited (a zombie counts as exited).
sqs_alive() { [ -r "/proc/$1/status" ] && ! grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
# sqs_starttime <pid>: kernel start time (stat field 22). It survives exec, and
# pairing it with the pid reduces PID-reuse risk; without a pidfd the check and
# the signal stay separate steps, so it is not an absolute guarantee.
sqs_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20; }
# sqs_same <pid> <starttime>: the recorded process is still alive.
sqs_same() { [ -n "$2" ] && sqs_alive "$1" && [ "$(sqs_starttime "$1")" = "$2" ]; }
# sqs_write_config <addr> <nkey> <file>: the dedicated server's config. Its
# max_payload admits a 1 MiB SQS message with its headers (ADR-0011:S4); the
# 1MB NATS default does not.
sqs_write_config() {
cat > "$3" <<EOF
listen: "$1:$SQS_PORT"
server_name: "sqs-store"
max_payload: 2MiB
jetstream {
store_dir: "$SQS_DATA"
sync_interval: always
}
authorization {
users = [ { nkey: "$2" } ]
}
EOF
}
# sqs_prepare: validate the contract, check and mount the data disk, write the
# server config and mark the store ready. Returns non-zero, unready, on any refusal.
sqs_prepare() {
rm -f "$SQS_RUN/ready"
mkdir -p "$SQS_RUN"
serial=$(sqs_fwcfg data-serial) || return 1
sqs_valid_serial "$serial" || { sqs_fatal "data-serial is not a volume serial"; return 1; }
addr=$(sqs_fwcfg listen-addr) || return 1
sqs_valid_ipv4 "$addr" || { sqs_fatal "listen-addr is not an IPv4 address"; return 1; }
nkey=$(sqs_fwcfg client-nkey) || return 1
sqs_valid_user_nkey "$nkey" || { sqs_fatal "client-nkey is not a user public NKey"; return 1; }
sqs_has_addr "$addr" || { sqs_fatal "listen-addr $addr is not on any guest NIC"; return 1; }
# The server checks the config, including the NKey checksum, before the
# disk is touched.
umask 077
sqs_write_config "$addr" "$nkey" "$SQS_RUN/nats.conf" || return 1
if ! "$NATS_BIN" -t -c "$SQS_RUN/nats.conf" > "$SQS_RUN/config-test.out" 2>&1; then
sqs_fatal "server config test failed: $(sed 's/U[A-Z2-7]\{55\}/<nkey>/g' "$SQS_RUN/config-test.out" | tr '\n' ' ')"
return 1
fi
dev=$(sqs_find_disk "$serial") || return 1
rc=0
"$E2FSCK" -p "$dev" > "$SQS_RUN/fsck.out" 2>&1 || rc=$?
sqs_log "e2fsck -p $dev rc=$rc"
# 0 clean, 1 corrected: proceed. Anything higher, including a disk that is
# not ext4 at all, refuses; the guest never formats.
[ "$rc" -le 1 ] || { sqs_fatal "e2fsck refused $dev: $(tr '\n' ' ' < "$SQS_RUN/fsck.out")"; return 1; }
mkdir -p "$SQS_DATA"
"$MOUNT" -t ext4 -o nodev,nosuid,noexec "$dev" "$SQS_DATA" || { sqs_fatal "mount $dev failed"; return 1; }
: > "$SQS_RUN/ready"
sqs_log "prepared: $dev mounted at $SQS_DATA, config written for $addr:$SQS_PORT; server not started yet"
}
# sqs_run: the respawned supervisor body. Its identity is recorded before the
# stopping check: a stop that misses it set stopping first, so this parks; a
# stop that reads it signals this process whether or not it has exec'd yet.
sqs_run() {
mkdir -p "$SQS_RUN"
echo "$$ $(sqs_starttime $$)" > "$SQS_RUN/pid"
if [ -e "$SQS_RUN/stopping" ]; then
sqs_log "run: shutdown in progress, not starting"
return 2
fi
if [ ! -e "$SQS_RUN/ready" ]; then
sqs_log "run: store not prepared, not starting"
return 2
fi
sqs_log "run: starting pid=$$"
# ADR-0011:S11 bounds the store's Go heap; the guest must be larger.
GOMEMLIMIT=512MiB exec "$NATS_BIN" -c "$SQS_RUN/nats.conf"
}
# sqs_stop: first shutdown action. Block any respawn, SIGTERM the recorded run
# process (a server, or a start not yet exec'd), SIGKILL only after the budget,
# then sync and unmount. Each signal rechecks the identity; a survivor keeps the disk mounted.
sqs_stop() {
mkdir -p "$SQS_RUN"
: > "$SQS_RUN/stopping"
sqs_log "stop: begin"
rec=$(cat "$SQS_RUN/pid" 2>/dev/null || true)
pid=${rec%% *} st=${rec#* }
[ "$st" != "$rec" ] || st=""
if [ -n "$pid" ] && sqs_same "$pid" "$st"; then
kill -TERM "$pid" 2>/dev/null || true
sqs_log "stop: SIGTERM sent to run pid=$pid"
i=0
while sqs_same "$pid" "$st" && [ "$i" -lt "$SQS_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
if sqs_same "$pid" "$st"; then
sqs_log "stop: run pid=$pid still running after $((SQS_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
kill -KILL "$pid" 2>/dev/null || true
j=0
while sqs_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
fi
if sqs_same "$pid" "$st"; then
sqs_log "stop: run pid=$pid STILL ALIVE, leaving $SQS_DATA mounted"
return 1
fi
sqs_log "stop: store exited after $((i / 10)).$((i % 10))s"
else
sqs_log "stop: no running store"
fi
sync
if grep -q " $SQS_DATA " "$SQS_MOUNTS" 2>/dev/null; then
rc=0
"$UMOUNT" "$SQS_DATA" || rc=$?
sqs_log "stop: umount $SQS_DATA rc=$rc"
fi
}
# sqs_shutdown: the only shutdown action. The generic read-only remount runs
# only after the store stopped; a surviving writer keeps every mount as it is.
sqs_shutdown() {
if ! sqs_stop; then
sync
sqs_log "shutdown: store survived, skipping umount -a -r"
return 1
fi
rc=0
"$UMOUNT" -a -r || rc=$?
echo "[init] shutdown: umount -a -r rc=$rc" >> "$SQS_CONSOLE"
}
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
# sysinit: load the disk drivers, then validate the guest contract, check and
# mount the data disk and write the server config. A refusal leaves the store
# unprepared, so the run step never starts it.
. /usr/local/lib/sqs-store/sqs-store-lib.sh
modprobe virtio_blk 2>/dev/null || true
modprobe ext4 2>/dev/null || true
sqs_prepare || sqs_log "store refused; it will not start"
exit 0
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
# Respawned by init. A store that must not run parks here instead of exiting,
# so init does not respawn it in a tight loop.
. /usr/local/lib/sqs-store/sqs-store-lib.sh
sqs_run
exec sleep 2147483647
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
# The only shutdown action: stop the store and unmount its disk, then run the
# generic read-only remount only if the store is gone.
. /usr/local/lib/sqs-store/sqs-store-lib.sh
sqs_shutdown
+8
View File
@@ -0,0 +1,8 @@
# The power button loads here, not in /init: busybox init has its SIGUSR2
# (poweroff) handler installed before sysinit runs, so a press can never reach
# the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
::sysinit:/sbin/modprobe tiny_power_button power_signal=12
::sysinit:/usr/local/lib/sqs-worker/sqs-worker-prepare
::respawn:/usr/local/lib/sqs-worker/sqs-worker-ready
::ctrlaltdel:/sbin/reboot
::shutdown:/usr/local/lib/sqs-worker/sqs-worker-stop
+115
View File
@@ -0,0 +1,115 @@
# sqs-worker-lib.sh — shared by the SQS worker guest's readiness and stop
# steps. Sourced, never executed. Every path and command is overridable so the
# host test can drive the same code against test-owned processes and stub tools.
#
# store-ready exists only while the running watcher's last check passed. A
# check proves the store accepted the worker key, not the store server's
# identity, and a store failure is noticed up to one interval plus call time late.
SQSW_RUN=${SQSW_RUN:-/run/sqs-worker}
SQSW_DATA=${SQSW_DATA:-/data}
SQSW_CONSOLE=${SQSW_CONSOLE:-/dev/console}
SQSW_MOUNTS=${SQSW_MOUNTS:-/proc/mounts}
SQSW_BIN=${SQSW_BIN:-/usr/local/bin/sqs-worker}
SQSW_STOP_BUDGET_TENTHS=${SQSW_STOP_BUDGET_TENTHS:-100}
SQSW_RESPAWN_GAP=${SQSW_RESPAWN_GAP:-5}
SQSW_PARK_INTERVAL=${SQSW_PARK_INTERVAL:-30}
UMOUNT=${UMOUNT:-umount}
sqsw_log() { echo "[sqs-worker] $*" >> "$SQSW_CONSOLE"; }
# sqsw_alive <pid>: running or stopped, not exited (a zombie counts as exited).
sqsw_alive() { [ -r "/proc/$1/status" ] && ! grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
# sqsw_starttime <pid>: kernel start time (stat field 22). It survives exec, and
# pairing it with the pid reduces PID-reuse risk; it is not an absolute guarantee.
sqsw_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20; }
# sqsw_same <pid> <starttime>: the recorded process is still alive.
sqsw_same() { [ -n "$2" ] && sqsw_alive "$1" && [ "$(sqsw_starttime "$1")" = "$2" ]; }
# sqsw_run: the respawned readiness body. A predecessor that was killed or
# crashed could not remove its marker, so every start removes it first. Its
# identity is recorded before the stopping check, as the store's run step does.
sqsw_run() {
mkdir -p "$SQSW_RUN"
echo "$$ $(sqsw_starttime $$)" > "$SQSW_RUN/ready.pid"
if [ -e "$SQSW_RUN/store-ready" ]; then
rm -f "$SQSW_RUN/store-ready"
sqsw_log "ready: removed a store-ready marker left by an earlier watcher"
fi
# Returning here would make init respawn and log at once, forever. Park
# instead, logging once: shutdown stops this recorded process, and a key
# that appears later lets the watcher start.
if [ -e "$SQSW_RUN/stopping" ] || [ ! -s "$SQSW_RUN/public-key" ]; then
if [ -e "$SQSW_RUN/stopping" ]; then
sqsw_log "ready: shutdown in progress, not starting"
else
sqsw_log "ready: no worker key; readiness not started, waiting"
fi
while [ -e "$SQSW_RUN/stopping" ] || [ ! -s "$SQSW_RUN/public-key" ]; do
sleep "$SQSW_PARK_INTERVAL"
done
fi
# Init respawns at once; a watcher that keeps exiting restarts at most
# once per gap instead of in a tight loop.
now=$(date +%s)
last=$(cat "$SQSW_RUN/ready.last" 2>/dev/null || echo 0)
wait=$((last + SQSW_RESPAWN_GAP - now))
if [ "$wait" -gt 0 ] && [ "$wait" -le "$SQSW_RESPAWN_GAP" ]; then sleep "$wait"; fi
date +%s > "$SQSW_RUN/ready.last"
sqsw_log "ready: starting watcher pid=$$"
exec "$SQSW_BIN" ready >> "$SQSW_CONSOLE" 2>&1
}
# sqsw_stop: block any respawn, drop the marker, SIGTERM the recorded watcher,
# SIGKILL only after the budget, and unmount the worker disk only once the
# watcher is confirmed gone. A survivor keeps the disk mounted.
sqsw_stop() {
mkdir -p "$SQSW_RUN"
: > "$SQSW_RUN/stopping"
rm -f "$SQSW_RUN/store-ready"
rec=$(cat "$SQSW_RUN/ready.pid" 2>/dev/null || true)
pid=${rec%% *} st=${rec#* }
[ "$st" != "$rec" ] || st=""
if [ -n "$pid" ] && sqsw_same "$pid" "$st"; then
kill -TERM "$pid" 2>/dev/null || true
sqsw_log "stop: SIGTERM sent to readiness watcher pid=$pid"
i=0
while sqsw_same "$pid" "$st" && [ "$i" -lt "$SQSW_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
if sqsw_same "$pid" "$st"; then
sqsw_log "stop: readiness watcher pid=$pid still running after $((SQSW_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
kill -KILL "$pid" 2>/dev/null || true
j=0
while sqsw_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
fi
if sqsw_same "$pid" "$st"; then
rm -f "$SQSW_RUN/store-ready"
sqsw_log "stop: readiness watcher pid=$pid STILL ALIVE, leaving $SQSW_DATA mounted"
return 1
fi
sqsw_log "stop: readiness watcher pid=$pid exited"
else
sqsw_log "stop: no running readiness watcher"
fi
rm -f "$SQSW_RUN/store-ready" "$SQSW_RUN/ready.pid"
sync
if grep -q " $SQSW_DATA " "$SQSW_MOUNTS" 2>/dev/null; then
rc=0
"$UMOUNT" "$SQSW_DATA" || rc=$?
sqsw_log "stop: umount $SQSW_DATA rc=$rc"
fi
}
# sqsw_shutdown: the only shutdown action. The generic read-only remount runs
# only after the watcher stopped; a survivor keeps every mount as it is.
sqsw_shutdown() {
if ! sqsw_stop; then
sync
sqsw_log "shutdown: readiness watcher survived, skipping umount -a -r"
return 1
fi
rc=0
"$UMOUNT" -a -r || rc=$?
echo "[init] shutdown: umount -a -r rc=$rc" >> "$SQSW_CONSOLE"
}
+8
View File
@@ -0,0 +1,8 @@
#!/bin/sh
# sysinit: load the disk drivers, then select, check and mount the worker disk
# and load or create the worker's store key on it. Any refusal is logged; the
# key is never read or created on a disk that is not proven mounted.
modprobe virtio_blk 2>/dev/null || true
modprobe ext4 2>/dev/null || true
/usr/local/bin/sqs-worker boot >> /dev/console 2>&1 || echo "[sqs-worker] boot refused" >> /dev/console
exit 0
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
# Respawned by init. A watcher that must not run parks here instead of exiting,
# so init does not respawn it in a tight loop.
. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
sqsw_run
exec sleep 2147483647
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
# The only shutdown action: stop the readiness watcher and unmount the worker
# disk, then run the generic read-only remount only if the watcher is gone.
. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
sqsw_shutdown
+5 -4
View File
@@ -1,10 +1,10 @@
# spinifex-unit-version: 3
# spinifex-unit-version: 5
[Unit]
Description=Spinifex Daemon (VM Orchestration)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
Wants=openvswitch-ipsec.service
After=spinifex-nats.service spinifex-viperblock.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service openvswitch-ipsec.service spinifex-hostnet.service
# Wants (not Requires): boot without openvswitch-ipsec, survive OVS restarts via hostnet
Wants=openvswitch-ipsec.service spinifex-hostnet.service
[Service]
EnvironmentFile=-/etc/spinifex/telemetry.env
@@ -14,6 +14,7 @@ Type=simple
Slice=spinifex-system.slice
User=spinifex-daemon
Group=spinifex
ExecStartPre=/usr/local/bin/spx admin hostnet check
ExecStart=/usr/local/bin/spx service spinifex start
Restart=on-failure
RestartSec=5
+25
View File
@@ -0,0 +1,25 @@
# spinifex-unit-version: 2
[Unit]
Description=Spinifex host network wiring (veth pairs, mgmt address, encap route)
# PartOf/WantedBy ovs-vswitchd: re-applied whenever OVS starts; daemon and vpcd only Want it
Requires=ovs-vswitchd.service
After=ovs-vswitchd.service NetworkManager-wait-online.service
PartOf=ovs-vswitchd.service
# Stopping OVS stops ovn-controller through its Requires= chain and starting OVS
# does not bring it back, so the start that re-applies hostnet restarts it too
Wants=ovn-controller.service
Before=spinifex.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/spx admin hostnet apply
# Covers the 120s wait behind setup-ovn.sh plus the 60s wait for links
TimeoutStartSec=240
TimeoutStopSec=10
ProtectSystem=full
ProtectHome=yes
PrivateTmp=yes
[Install]
WantedBy=spinifex.target ovs-vswitchd.service
+4 -3
View File
@@ -1,9 +1,9 @@
# spinifex-unit-version: 1
# spinifex-unit-version: 3
[Unit]
Description=Spinifex VPC Daemon (OVN Networking)
PartOf=spinifex.target
After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service openvswitch-switch.service ovn-controller.service
Wants=spinifex-nats.service ovn-controller.service
After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service ovsdb-server.service ovs-vswitchd.service ovn-controller.service spinifex-hostnet.service
Wants=spinifex-nats.service ovn-controller.service spinifex-hostnet.service
[Service]
EnvironmentFile=-/etc/spinifex/telemetry.env
@@ -11,6 +11,7 @@ Type=simple
Slice=spinifex-system.slice
User=spinifex-vpcd
Group=spinifex
ExecStartPre=/usr/local/bin/spx admin hostnet check
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
ExecStart=/usr/local/bin/spx service vpcd start
Restart=on-failure
+2 -2
View File
@@ -1,9 +1,9 @@
# spinifex-unit-version: 1
# spinifex-unit-version: 4
[Unit]
Description=Spinifex Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer ovsdb-server.service ovs-vswitchd.service ovn-controller.service
[Install]
WantedBy=multi-user.target
+34 -2
View File
@@ -4,10 +4,22 @@ import (
"flag"
"fmt"
"os"
"slices"
"github.com/mulgadc/predastore/s3api"
"github.com/mulgadc/spinifex/internal/awsmodel"
acmawsapi "github.com/mulgadc/spinifex/spinifex/domains/acm/awsapi"
ec2awsapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi"
awsapi "github.com/mulgadc/spinifex/spinifex/domains/ecr/awsapi"
ecsawsapi "github.com/mulgadc/spinifex/spinifex/domains/ecs/awsapi"
eksawsapi "github.com/mulgadc/spinifex/spinifex/domains/eks/awsapi"
elbv2awsapi "github.com/mulgadc/spinifex/spinifex/domains/elbv2/awsapi"
iamawsapi "github.com/mulgadc/spinifex/spinifex/domains/iam/awsapi"
rdsawsapi "github.com/mulgadc/spinifex/spinifex/domains/rds/awsapi"
route53awsapi "github.com/mulgadc/spinifex/spinifex/domains/route53/awsapi"
stsawsapi "github.com/mulgadc/spinifex/spinifex/domains/sts/awsapi"
"github.com/mulgadc/spinifex/spinifex/gateway"
"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
)
@@ -43,10 +55,30 @@ func fail(err error) {
}
func compareAll() ([]awsmodel.OperationCoverage, error) {
dispatch := gateway.AWSOperationInventory()
// Registered services contribute their declared inventory, which needs no
// live dependencies; nothing here serves a request.
inventories := gateway.AWSOperationInventory(map[string]dispatch.Inventory{
awsapi.ServiceName: awsapi.OperationInventory(),
acmawsapi.ServiceName: acmawsapi.OperationInventory(),
route53awsapi.ServiceName: route53awsapi.OperationInventory(),
elbv2awsapi.InventoryName: elbv2awsapi.OperationInventory(),
rdsawsapi.ServiceName: rdsawsapi.OperationInventory(),
ec2awsapi.ServiceName: ec2awsapi.OperationInventory(),
ecsawsapi.ServiceName: ecsawsapi.OperationInventory(),
eksawsapi.ServiceName: eksawsapi.OperationInventory(),
iamawsapi.ServiceName: iamawsapi.OperationInventory(),
stsawsapi.ServiceName: stsawsapi.OperationInventory(),
})
// The loop below walks the embedded models, so a served service with no
// model would otherwise drop out of the report without a word.
for name := range inventories {
if !slices.Contains(awsmodel.Services(), awsmodel.Service(name)) {
return nil, fmt.Errorf("no embedded AWS model for served service %s", name)
}
}
coverages := make([]awsmodel.OperationCoverage, 0, len(awsmodel.Services()))
for _, service := range awsmodel.Services() {
inventory, ok := dispatch[string(service)]
inventory, ok := inventories[string(service)]
var modelInventory awsmodel.DispatchInventory
switch {
case ok:
+1 -1
View File
@@ -23,7 +23,7 @@ import (
"github.com/insomniacslk/dhcp/dhcpv4"
"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
"github.com/mulgadc/spinifex/spinifex/domains/network/external/dhcp"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
)
+5 -165
View File
@@ -1,175 +1,15 @@
// ecr-credential-provider is a kubelet exec credential provider that lets a
// Spinifex EKS worker node pull images from the internal Spinifex ECR registry.
//
// kubelet invokes it per-pull, feeding a CredentialProviderRequest on STDIN and
// reading a CredentialProviderResponse on STDOUT. The provider fetches the
// node IAM-role credentials from IMDS, SigV4-signs GetAuthorizationToken against
// the AWS gateway's ECR endpoint, and returns the minted AWS:<jwt> as Basic auth
// for the requested registry host.
//
// Static node config (gateway URL, CA, region) is read from the cloud-init env
// file /etc/spinifex-eks/agent.env (KEY=value); real env vars override it.
// ecr-credential-provider is the kubelet exec credential provider for Spinifex
// EKS worker nodes; the implementation is in agents/eks/credentialprovider.
package main
import (
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"time"
"github.com/mulgadc/spinifex/internal/ecrauth"
"github.com/mulgadc/spinifex/internal/guestenv"
"github.com/mulgadc/spinifex/internal/imdscreds"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
"github.com/mulgadc/spinifex/spinifex/agents/eks/credentialprovider"
)
const (
credProviderAPIVersion = "credentialprovider.kubelet.k8s.io/v1"
defaultCacheDuration = "10h"
// expiryMargin trims the token's real TTL so kubelet re-invokes the provider
// before the JWT actually expires mid-run.
expiryMargin = 30 * time.Minute
defaultEnvFile = "/etc/spinifex-eks/agent.env"
defaultGatewayCA = "/etc/spinifex-eks/gateway-ca.pem"
defaultIMDSBase = "http://169.254.169.254/latest"
)
// CredentialProviderRequest is the kubelet exec-provider request on STDIN.
type CredentialProviderRequest struct {
APIVersion string `json:"apiVersion"`
Kind string `json:"kind"`
Image string `json:"image"`
}
// AuthConfig is a single registry credential entry.
type AuthConfig struct {
Username string `json:"username"`
Password string `json:"password"`
}
// CredentialProviderResponse is the kubelet exec-provider response on STDOUT.
type CredentialProviderResponse struct {
APIVersion string `json:"apiVersion"`
Kind string `json:"kind"`
CacheKeyType string `json:"cacheKeyType"`
CacheDuration string `json:"cacheDuration"`
Auth map[string]AuthConfig `json:"auth"`
}
// config holds the static node settings the provider needs.
type config struct {
GatewayURL string
GatewayCA string
Region string
IMDSBase string
}
func main() {
cfg := loadConfig(defaultEnvFile)
httpClient, err := ecrauth.GatewayHTTPClient(cfg.GatewayCA)
if err != nil {
slog.Error("ecr-credential-provider: build gateway client", "err", err)
emitEmpty(os.Stdout)
os.Exit(1)
}
if err := run(os.Stdin, os.Stdout, cfg, httpClient); err != nil {
slog.Error("ecr-credential-provider: run failed", "err", err)
// Still emit a valid empty-auth response so kubelet has a parseable reply.
emitEmpty(os.Stdout)
os.Exit(1)
}
}
// run is the testable core: it reads a CredentialProviderRequest from in,
// resolves credentials via IMDS + the gateway, and writes a
// CredentialProviderResponse to out. The httpClient must trust the gateway CA.
func run(in io.Reader, out io.Writer, cfg config, httpClient *http.Client) error {
var req CredentialProviderRequest
if err := json.NewDecoder(in).Decode(&req); err != nil {
return fmt.Errorf("decode request: %w", err)
}
if req.Image == "" {
return fmt.Errorf("request image is empty")
}
imageHost := ecrauth.HostFromImage(req.Image)
creds, err := imdscreds.Fetch(httpClient, cfg.IMDSBase)
if err != nil {
return fmt.Errorf("fetch IMDS credentials: %w", err)
}
tok, err := ecrauth.GetAuthorizationToken(cfg.Region, cfg.GatewayURL, httpClient,
creds.AccessKeyID, creds.SecretAccessKey, creds.SessionToken)
if err != nil {
return fmt.Errorf("get authorization token: %w", err)
}
auth := AuthConfig{Username: tok.Username, Password: tok.Password}
authMap := map[string]AuthConfig{imageHost: auth}
if tok.ProxyHost != "" && tok.ProxyHost != imageHost {
authMap[tok.ProxyHost] = auth
}
resp := CredentialProviderResponse{
APIVersion: credProviderAPIVersion,
Kind: "CredentialProviderResponse",
CacheKeyType: "Registry",
CacheDuration: cacheDurationFrom(tok.ExpiresAt),
Auth: authMap,
}
return json.NewEncoder(out).Encode(&resp)
}
// cacheDurationFrom derives a Go-duration string from the token expiry minus a
// safety margin, falling back to the default when expiry is unknown or too near.
func cacheDurationFrom(expiresAt time.Time) string {
if expiresAt.IsZero() {
return defaultCacheDuration
}
d := time.Until(expiresAt) - expiryMargin
if d <= 0 {
return defaultCacheDuration
}
return d.Round(time.Second).String()
}
// loadConfig reads the cloud-init env file then lets real env vars override. The
// IMDS base is fixed (not in the env file) but overridable for tests.
func loadConfig(envFile string) config {
get := guestenv.Load(envFile).Get
cfg := config{
GatewayURL: get("EKS_GATEWAY_URL"),
GatewayCA: get("EKS_GATEWAY_CA"),
Region: get("EKS_REGION"),
IMDSBase: get("EKS_IMDS_BASE"),
}
if cfg.GatewayCA == "" {
cfg.GatewayCA = defaultGatewayCA
}
if cfg.IMDSBase == "" {
cfg.IMDSBase = defaultIMDSBase
}
return cfg
}
// emitEmpty writes a valid CredentialProviderResponse with no auth entries.
func emitEmpty(out io.Writer) {
resp := CredentialProviderResponse{
APIVersion: credProviderAPIVersion,
Kind: "CredentialProviderResponse",
CacheKeyType: "Registry",
CacheDuration: defaultCacheDuration,
Auth: map[string]AuthConfig{},
}
if err := json.NewEncoder(out).Encode(&resp); err != nil {
slog.Error("ecr-credential-provider: encode empty response", "err", err)
}
os.Exit(credentialprovider.Main(os.Stdin, os.Stdout))
}
-126
View File
@@ -1,126 +0,0 @@
package main
import (
"bytes"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// fakeBackend serves both the IMDS role/credentials endpoints and the gateway
// ECR GetAuthorizationToken on one TLS server, routed by request path.
func fakeBackend(t *testing.T) *httptest.Server {
t.Helper()
mux := http.NewServeMux()
// IMDSv2 token. Real IMDS echoes the requested TTL back as a response
// header; the SDK client parses it from there, not the body.
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPut {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
_, _ = w.Write([]byte("imds-v2-token"))
})
// IMDS role name.
mux.HandleFunc("/latest/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/latest/meta-data/iam/security-credentials/" {
if r.Header.Get("X-Aws-Ec2-Metadata-Token") != "imds-v2-token" {
w.WriteHeader(http.StatusUnauthorized)
return
}
_, _ = w.Write([]byte("node-role"))
return
}
// Role credentials.
_ = json.NewEncoder(w).Encode(map[string]string{
"Code": "Success",
"Type": "AWS-HMAC",
"AccessKeyId": "AKIATEST",
"SecretAccessKey": "secrettest",
"Token": "sessiontoken",
"AccountId": "123456789012",
})
})
// Gateway ECR GetAuthorizationToken (X-Amz-Target dispatch).
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
w.WriteHeader(http.StatusBadRequest)
return
}
authToken := base64.StdEncoding.EncodeToString([]byte("AWS:testjwt"))
w.Header().Set("Content-Type", "application/x-amz-json-1.1")
_ = json.NewEncoder(w).Encode(map[string]any{
"authorizationData": []map[string]any{{
"authorizationToken": authToken,
"proxyEndpoint": "https://123456789012.dkr.ecr.us-east-1.spinifex.internal",
}},
})
})
srv := httptest.NewTLSServer(mux)
t.Cleanup(srv.Close)
return srv
}
func clientTrusting(srv *httptest.Server) *http.Client {
pool := x509.NewCertPool()
pool.AddCert(srv.Certificate())
return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
}
func TestRun_ReturnsBasicAuth(t *testing.T) {
srv := fakeBackend(t)
client := clientTrusting(srv)
cfg := config{
GatewayURL: srv.URL,
Region: "us-east-1",
IMDSBase: srv.URL + "/latest",
}
const image = "123456789012.dkr.ecr.us-east-1.spinifex.internal/spinifex-demo:latest"
reqBody, err := json.Marshal(CredentialProviderRequest{
APIVersion: credProviderAPIVersion,
Kind: "CredentialProviderRequest",
Image: image,
})
if err != nil {
t.Fatalf("marshal request: %v", err)
}
var out bytes.Buffer
if err := run(bytes.NewReader(reqBody), &out, cfg, client); err != nil {
t.Fatalf("run: %v", err)
}
var resp CredentialProviderResponse
if err := json.Unmarshal(out.Bytes(), &resp); err != nil {
t.Fatalf("unmarshal response: %v\n%s", err, out.String())
}
if resp.Kind != "CredentialProviderResponse" {
t.Errorf("Kind = %q, want CredentialProviderResponse", resp.Kind)
}
if resp.CacheKeyType != "Registry" {
t.Errorf("CacheKeyType = %q, want Registry", resp.CacheKeyType)
}
host := "123456789012.dkr.ecr.us-east-1.spinifex.internal"
auth, ok := resp.Auth[host]
if !ok {
t.Fatalf("auth map missing host %q; got %v", host, resp.Auth)
}
if auth.Username != "AWS" {
t.Errorf("username = %q, want AWS", auth.Username)
}
if auth.Password != "testjwt" {
t.Errorf("password = %q, want testjwt", auth.Password)
}
}
-214
View File
@@ -1,214 +0,0 @@
package main
import (
"bufio"
"context"
"fmt"
"log/slog"
"net/http"
"os"
goruntime "runtime"
"strconv"
"strings"
"time"
"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// version is the agent build version, reported in the register message.
// Overridable via -ldflags "-X main.version=...".
var version = "dev"
// Agent wires the ecs-agent's runtime seams: a gateway control-plane client, a
// container runtime, and an ECR resolver. It registers the host as a container
// instance at boot, heartbeats (re-registers) while alive, polls the gateway for
// task assignments, and runs them through containerd, reporting state back over
// the gateway. It never connects to NATS — the bus stays host-internal.
type Agent struct {
cfg config
id identity
cp controlPlane
resolver ctrruntime.Resolver
// rt holds the container runtime, which is resolved by a loop rather than
// once at boot. Empty until the containerd socket answers, and emptied again
// if it stops answering.
rt *runtimeGate
reg *registrar
hb *heartbeater
// netns builds awsvpc task network namespaces (nil-safe; bridge/host skip it).
netns *taskNetns
// cred serves task IAM role credentials at 169.254.170.2 (nil in unit tests).
cred *credEndpoint
// gpu is the local pinning ledger over the host's discovered GPU UUIDs.
// Always non-nil (newAgent seeds an empty one); New() reseeds it from
// nvidia-smi discovery.
gpu *gpuLedger
// dialRuntime connects to the container runtime. nil in unit tests, which
// drive the gate directly.
dialRuntime func() (ctrruntime.Runtime, error)
closers []func() error
}
// newAgent assembles an Agent from already-built seams. Tests use this directly
// with fakes; New builds the production seams and delegates here. rt may be nil
// when containerd is unavailable; the assign path then reports the task STOPPED
// rather than crashing, and the watch loop keeps dialling for it.
func newAgent(cfg config, id identity, cp controlPlane, rt ctrruntime.Runtime, resolver ctrruntime.Resolver) *Agent {
gate := newRuntimeGate(rt)
a := &Agent{
cfg: cfg,
id: id,
cp: cp,
resolver: resolver,
rt: gate,
netns: newTaskNetns(execNetRunner{}),
gpu: newGPULedger(id.Capacity.GPUIDs),
}
// Registration reports the host's capacity only while a runtime is there to
// use it, so an agent that cannot run a container advertises none and takes
// no placement.
a.reg = newRegistrar(cp, id, gate.available)
a.hb = newHeartbeater(cp, id, cfg.Heartbeat, gate.available)
return a
}
// New builds an Agent from config: it resolves the host identity from IMDS,
// builds the gateway control-plane client, the ECR resolver and (best-effort)
// the containerd runtime. A containerd connect failure is logged, not fatal —
// registration and heartbeat still run so the instance is visible while the
// runtime recovers. The ECR gateway client is built lazily on first image pull
// (not here), so a missing or malformed gateway CA does not stop registration.
func New(cfg config) (*Agent, error) {
imdsClient := &http.Client{Timeout: 5 * time.Second}
meta, err := fetchInstanceMetadata(imdsClient, cfg.IMDSBase)
if err != nil {
return nil, fmt.Errorf("instance metadata: %w", err)
}
host, _ := os.Hostname()
id := identity{
AccountID: meta.AccountID,
ClusterName: cfg.ClusterName,
InstanceID: meta.InstanceID,
AZ: meta.AZ,
InstanceType: meta.InstanceType,
Hostname: host,
Capacity: detectCapacity(discoverNvidiaGPUs(execCommandRunner)),
AgentVersion: version,
}
creds := credentials.NewIMDSProvider(imdsClient, cfg.IMDSBase)
resolver := newLazyECRResolver(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA)
// A dial that fails here is not fatal and not final: the node starts the
// container daemon and the agent back to back, so the socket is often not
// accepting yet. Run's watch loop keeps dialling.
rt, perr := ctrruntime.New(cfg.ContainerdSocket)
if perr != nil {
slog.Warn("ecs-agent: containerd unavailable at boot, will keep retrying", "err", perr)
}
cp, err := newGatewayControlPlane(cfg, creds)
if err != nil {
return nil, fmt.Errorf("build gateway control-plane: %w", err)
}
a := newAgent(cfg, id, cp, rt, resolver)
a.cred = newCredEndpoint(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA,
cfg.CredEndpointIP, cfg.CredEndpointPort, execNetRunner{})
a.dialRuntime = func() (ctrruntime.Runtime, error) { return ctrruntime.New(cfg.ContainerdSocket) }
return a, nil
}
// Run registers the instance, starts the heartbeat and assignment-poll loops,
// and blocks until ctx is cancelled, then tears down.
func (a *Agent) Run(ctx context.Context) error {
if err := a.reg.Register(); err != nil {
slog.Error("ecs-agent: initial registration failed", "err", err)
} else {
slog.Info("ecs-agent: registered", "cluster", a.id.ClusterName, "instance", a.id.InstanceID)
}
if a.cred != nil {
if err := a.cred.Start(); err != nil {
slog.Error("ecs-agent: credential endpoint start failed", "err", err)
}
}
go a.hb.Run(ctx)
// Keeps the runtime current for the life of the process: dials while there
// is none, and drops one that stops answering.
if a.dialRuntime != nil {
go a.rt.watch(ctx, a.dialRuntime, a.cfg.RuntimeRetry)
}
// Re-adopt containers still running from before this restart, then poll with
// their tasks pre-seeded so re-delivered assignments are acked, not re-run.
go a.pollAssignments(ctx, a.reconcile(ctx))
<-ctx.Done()
return a.Stop()
}
// Stop closes the runtime. Safe to call once.
func (a *Agent) Stop() error {
var firstErr error
if a.cred != nil {
_ = a.cred.Stop()
}
for _, c := range a.closers {
if err := c(); err != nil && firstErr == nil {
firstErr = err
}
}
a.rt.clear()
return firstErr
}
// detectCapacity reports the host's total CPU units (1 vCPU = 1024), memory,
// and GPU count/UUIDs from the caller's nvidia-smi discovery (nil on a
// non-GPU host).
func detectCapacity(gpuUUIDs []string) bus.InstanceCapacity {
return bus.InstanceCapacity{
CPU: goruntime.NumCPU() * 1024,
MemoryMiB: memTotalMiB(),
GPU: len(gpuUUIDs),
GPUIDs: gpuUUIDs,
}
}
// memTotalMiB reads MemTotal from /proc/meminfo, returning 0 if unavailable.
func memTotalMiB() int {
f, err := os.Open("/proc/meminfo")
if err != nil {
return 0
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := scanner.Text()
if !strings.HasPrefix(line, "MemTotal:") {
continue
}
fields := strings.Fields(line)
if len(fields) < 2 {
return 0
}
kb, err := strconv.Atoi(fields[1])
if err != nil {
return 0
}
return kb / 1024
}
return 0
}
-66
View File
@@ -1,66 +0,0 @@
package main
import (
"context"
"testing"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
)
func TestAgent_RunRegistersThenStopsOnContext(t *testing.T) {
cp := &fakeCP{}
cfg := config{Heartbeat: 5 * time.Millisecond, PollInterval: 5 * time.Millisecond}
puller := &ctrruntime.FakePuller{}
a := newAgent(cfg, testIdentity(), cp, puller, nil)
a.closers = append(a.closers, puller.Close)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
// Give it time to register + re-register (heartbeat) a few times.
time.Sleep(30 * time.Millisecond)
cancel()
select {
case err := <-done:
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
case <-time.After(time.Second):
t.Fatal("Run did not return after cancel")
}
// One boot register + at least one heartbeat re-register.
if cp.registerCount() < 2 {
t.Errorf("registers = %d, want >= 2 (boot + heartbeat)", cp.registerCount())
}
if !puller.Closed {
t.Error("Stop did not close the puller")
}
}
func TestDetectCapacity_Positive(t *testing.T) {
c := detectCapacity(nil)
if c.CPU <= 0 {
t.Errorf("CPU = %d, want > 0", c.CPU)
}
// MemoryMiB may be 0 on platforms without /proc/meminfo; just assert non-negative.
if c.MemoryMiB < 0 {
t.Errorf("MemoryMiB = %d, want >= 0", c.MemoryMiB)
}
if c.GPU != 0 || c.GPUIDs != nil {
t.Errorf("no GPU UUIDs given: want GPU=0/GPUIDs=nil, got %d/%v", c.GPU, c.GPUIDs)
}
}
func TestDetectCapacity_WithGPUUUIDs(t *testing.T) {
c := detectCapacity([]string{"GPU-aaa", "GPU-bbb"})
if c.GPU != 2 {
t.Errorf("GPU = %d, want 2", c.GPU)
}
if len(c.GPUIDs) != 2 {
t.Errorf("GPUIDs = %v, want 2 entries", c.GPUIDs)
}
}
-96
View File
@@ -1,96 +0,0 @@
package main
import (
"strconv"
"time"
"github.com/mulgadc/spinifex/internal/guestenv"
)
const (
defaultEnvFile = "/etc/spinifex-ecs/agent.env"
defaultGatewayCA = "/etc/spinifex-ecs/gateway-ca.pem"
defaultIMDSBase = "http://169.254.169.254/latest"
defaultContainerdSocket = "/run/containerd/containerd.sock"
defaultHeartbeat = 30 * time.Second
defaultPollInterval = 5 * time.Second
)
// config holds the static settings the agent reads at boot. Cluster identity
// (account ID, instance ID) is discovered at runtime from IMDS, not configured;
// only the cluster *name* the instance was launched into is static. AccessKey /
// SecretKey are legacy seeded creds, optional now that the agent SigV4-signs
// gateway calls with instance-role credentials from IMDS; the agent never holds
// a NATS token.
type config struct {
GatewayURL string
GatewayCA string
Region string
ClusterName string
AccessKey string
SecretKey string
IMDSBase string
ContainerdSocket string
Heartbeat time.Duration
PollInterval time.Duration
RuntimeRetry time.Duration
CredEndpointIP string
CredEndpointPort int
}
// loadConfig reads the cloud-init env file then lets real env vars override.
func loadConfig(envFile string) config {
get := guestenv.Load(envFile).Get
cfg := config{
GatewayURL: get("ECS_GATEWAY_URL"),
GatewayCA: get("ECS_GATEWAY_CA"),
Region: get("ECS_REGION"),
ClusterName: get("ECS_CLUSTER"),
AccessKey: get("ECS_ACCESS_KEY"),
SecretKey: get("ECS_SECRET_KEY"),
IMDSBase: get("ECS_IMDS_BASE"),
ContainerdSocket: get("ECS_CONTAINERD_SOCKET"),
}
if cfg.GatewayCA == "" {
cfg.GatewayCA = defaultGatewayCA
}
if cfg.IMDSBase == "" {
cfg.IMDSBase = defaultIMDSBase
}
if cfg.ContainerdSocket == "" {
cfg.ContainerdSocket = defaultContainerdSocket
}
if cfg.ClusterName == "" {
cfg.ClusterName = "default"
}
cfg.Heartbeat = defaultHeartbeat
if v := get("ECS_HEARTBEAT_INTERVAL"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.Heartbeat = d
}
}
cfg.PollInterval = defaultPollInterval
if v := get("ECS_POLL_INTERVAL"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.PollInterval = d
}
}
cfg.RuntimeRetry = defaultRuntimeRetry
if v := get("ECS_RUNTIME_RETRY"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.RuntimeRetry = d
}
}
cfg.CredEndpointIP = get("ECS_CRED_ENDPOINT_IP")
if cfg.CredEndpointIP == "" {
cfg.CredEndpointIP = defaultCredEndpointIP
}
cfg.CredEndpointPort = defaultCredEndpointPort
if v := get("ECS_CRED_ENDPOINT_PORT"); v != "" {
if p, err := strconv.Atoi(v); err == nil && p > 0 {
cfg.CredEndpointPort = p
}
}
return cfg
}
-73
View File
@@ -1,73 +0,0 @@
package main
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestLoadConfig_Defaults(t *testing.T) {
// Ensure env vars don't leak in from the host.
for _, k := range []string{"ECS_GATEWAY_URL", "ECS_GATEWAY_CA", "ECS_REGION",
"ECS_CLUSTER", "ECS_ACCESS_KEY", "ECS_SECRET_KEY", "ECS_IMDS_BASE",
"ECS_CONTAINERD_SOCKET", "ECS_HEARTBEAT_INTERVAL", "ECS_POLL_INTERVAL"} {
t.Setenv(k, "")
}
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
if cfg.GatewayCA != defaultGatewayCA {
t.Errorf("GatewayCA = %q, want default", cfg.GatewayCA)
}
if cfg.IMDSBase != defaultIMDSBase {
t.Errorf("IMDSBase = %q, want default", cfg.IMDSBase)
}
if cfg.ContainerdSocket != defaultContainerdSocket {
t.Errorf("ContainerdSocket = %q, want default", cfg.ContainerdSocket)
}
if cfg.ClusterName != "default" {
t.Errorf("ClusterName = %q, want default", cfg.ClusterName)
}
if cfg.Heartbeat != defaultHeartbeat {
t.Errorf("Heartbeat = %v, want %v", cfg.Heartbeat, defaultHeartbeat)
}
if cfg.PollInterval != defaultPollInterval {
t.Errorf("PollInterval = %v, want %v", cfg.PollInterval, defaultPollInterval)
}
}
func TestLoadConfig_FileThenEnvOverride(t *testing.T) {
dir := t.TempDir()
envFile := filepath.Join(dir, "agent.env")
body := "# comment\nECS_GATEWAY_URL=https://gw.file\nECS_REGION=us-west-2\nECS_CLUSTER=prod\n"
if err := os.WriteFile(envFile, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("ECS_GATEWAY_URL", "https://gw.env")
t.Setenv("ECS_ACCESS_KEY", "AKIAENV")
cfg := loadConfig(envFile)
if cfg.GatewayURL != "https://gw.env" {
t.Errorf("GatewayURL = %q, want env override", cfg.GatewayURL)
}
if cfg.Region != "us-west-2" {
t.Errorf("Region = %q, want from file", cfg.Region)
}
if cfg.ClusterName != "prod" {
t.Errorf("ClusterName = %q, want from file", cfg.ClusterName)
}
if cfg.AccessKey != "AKIAENV" {
t.Errorf("AccessKey = %q, want env", cfg.AccessKey)
}
}
func TestLoadConfig_HeartbeatOverride(t *testing.T) {
t.Setenv("ECS_HEARTBEAT_INTERVAL", "5s")
if cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env")); cfg.Heartbeat != 5*time.Second {
t.Errorf("Heartbeat = %v, want 5s", cfg.Heartbeat)
}
// Garbage value falls back to the default, not zero.
t.Setenv("ECS_HEARTBEAT_INTERVAL", "not-a-duration")
if cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env")); cfg.Heartbeat != defaultHeartbeat {
t.Errorf("Heartbeat = %v, want default on bad value", cfg.Heartbeat)
}
}
-51
View File
@@ -1,51 +0,0 @@
package main
import (
"context"
"log/slog"
"time"
)
// heartbeater keeps the container instance alive by periodically re-registering
// it through the gateway. Real ECS folds liveness into ACS; v1 folds it into an
// idempotent RegisterContainerInstance, which refreshes the instance's LastSeen
// so the scheduler can tell live instances from dead ones (no bus.Heartbeat).
type heartbeater struct {
cp controlPlane
id identity
interval time.Duration
// runtimeReady reports whether a container runtime is resolved, so a beat
// advertises capacity only while there is something to run it with. nil
// means always ready.
runtimeReady func() bool
}
func newHeartbeater(cp controlPlane, id identity, interval time.Duration, runtimeReady func() bool) *heartbeater {
if interval <= 0 {
interval = defaultHeartbeat
}
return &heartbeater{cp: cp, id: id, interval: interval, runtimeReady: runtimeReady}
}
// beat re-registers the instance once, refreshing its LastSeen.
func (h *heartbeater) beat() error {
ready := h.runtimeReady == nil || h.runtimeReady()
return h.cp.Register(advertisedIdentity(h.id, ready))
}
// Run beats every interval until ctx is cancelled. A failed beat is logged and
// retried on the next tick rather than killing the loop.
func (h *heartbeater) Run(ctx context.Context) {
ticker := time.NewTicker(h.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := h.beat(); err != nil {
slog.Warn("ecs-agent: heartbeat (re-register) failed", "err", err)
}
}
}
}
+13 -13
View File
@@ -1,13 +1,6 @@
// ecs-agent runs inside a Spinifex ECS container instance (a guest VM booted from
// the ECS-AMI, with containerd baked in). It registers the host with the ECS
// scheduler through the AWS gateway over TLS+SigV4 (never NATS), heartbeats by
// re-registering, polls the gateway for task assignments, runs them through
// containerd, and reports state back over the gateway.
//
// Static config (gateway URL, CA, region, cluster, seeded IAM creds, containerd
// socket) is read from the cloud-init env file /etc/spinifex-ecs/agent.env
// (KEY=value); real env vars override it. Host identity (account, instance, AZ)
// comes from IMDS at boot.
// Command ecs-agent runs inside a Spinifex ECS container instance; the
// implementation is in agents/ecs/agent. This binary owns process lifecycle:
// signal handling, startup and run failure logging, and the exit code.
package main
import (
@@ -18,12 +11,19 @@ import (
"syscall"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
ecsagent "github.com/mulgadc/spinifex/spinifex/agents/ecs/agent"
)
func main() {
cfg := loadConfig(defaultEnvFile)
// version is the agent build version, reported in the register message.
// Overridable via -ldflags "-X main.version=...".
var version = "dev"
agent, err := New(cfg)
func main() {
cfg := ecsagent.LoadConfig(ecsagent.DefaultEnvFile)
cfg.AgentVersion = version
agent, err := ecsagent.New(cfg)
if err != nil {
slog.Error("ecs-agent: startup failed", "err", err)
os.Exit(1)
-66
View File
@@ -1,66 +0,0 @@
package main
import (
"context"
"fmt"
"io"
"net/http"
"strings"
"github.com/aws/aws-sdk-go-v2/feature/ec2/imds"
"github.com/mulgadc/spinifex/internal/imdscreds"
)
// instanceMetadata is the IMDS-derived identity of the host the agent runs on.
type instanceMetadata struct {
AccountID string
InstanceID string
AZ string
InstanceType string
}
// fetchInstanceMetadata reads the instance-id, AZ and account-id from IMDSv2
// via the SDK IMDS client. base is the IMDS root, e.g.
// "http://169.254.169.254/latest".
func fetchInstanceMetadata(client *http.Client, base string) (instanceMetadata, error) {
imdsClient := imdscreds.NewClient(client, base)
ctx := context.Background()
instanceID, err := getMetadata(ctx, imdsClient, "instance-id")
if err != nil {
return instanceMetadata{}, fmt.Errorf("instance-id: %w", err)
}
az, err := getMetadata(ctx, imdsClient, "placement/availability-zone")
if err != nil {
return instanceMetadata{}, fmt.Errorf("availability-zone: %w", err)
}
doc, err := imdsClient.GetInstanceIdentityDocument(ctx, &imds.GetInstanceIdentityDocumentInput{})
if err != nil {
return instanceMetadata{}, fmt.Errorf("identity document: %w", err)
}
// The instance type comes off the identity document already in hand rather
// than a further meta-data read; the scheduler sizes the host's awsvpc ENI
// capacity from it.
return instanceMetadata{
AccountID: doc.AccountID,
InstanceID: strings.TrimSpace(instanceID),
AZ: strings.TrimSpace(az),
InstanceType: strings.TrimSpace(doc.InstanceType),
}, nil
}
// getMetadata reads a single meta-data path and returns its body as a string.
func getMetadata(ctx context.Context, imdsClient *imds.Client, path string) (string, error) {
out, err := imdsClient.GetMetadata(ctx, &imds.GetMetadataInput{Path: path})
if err != nil {
return "", err
}
defer out.Content.Close()
body, err := io.ReadAll(out.Content)
if err != nil {
return "", err
}
return string(body), nil
}
-82
View File
@@ -1,82 +0,0 @@
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func metadataStub(t *testing.T, enforceV2 bool) *httptest.Server {
t.Helper()
mux := http.NewServeMux()
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPut {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
// Real IMDS echoes the requested TTL back as a response header; the SDK
// client parses it from there, not the body.
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
_, _ = w.Write([]byte("v2-token"))
})
guard := func(w http.ResponseWriter, r *http.Request) bool {
if enforceV2 && r.Header.Get("X-Aws-Ec2-Metadata-Token") != "v2-token" {
w.WriteHeader(http.StatusUnauthorized)
return false
}
return true
}
mux.HandleFunc("/latest/meta-data/instance-id", func(w http.ResponseWriter, r *http.Request) {
if guard(w, r) {
_, _ = w.Write([]byte("i-0abc123"))
}
})
mux.HandleFunc("/latest/meta-data/placement/availability-zone", func(w http.ResponseWriter, r *http.Request) {
if guard(w, r) {
_, _ = w.Write([]byte("us-east-1a"))
}
})
mux.HandleFunc("/latest/dynamic/instance-identity/document", func(w http.ResponseWriter, r *http.Request) {
if guard(w, r) {
_, _ = w.Write([]byte(`{"accountId":"123456789012","region":"us-east-1"}`))
}
})
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv
}
func TestFetchInstanceMetadata_OK(t *testing.T) {
srv := metadataStub(t, true)
got, err := fetchInstanceMetadata(srv.Client(), srv.URL+"/latest")
if err != nil {
t.Fatalf("fetch: %v", err)
}
if got.InstanceID != "i-0abc123" || got.AZ != "us-east-1a" || got.AccountID != "123456789012" {
t.Errorf("metadata mismatch: %+v", got)
}
}
func TestFetchInstanceMetadata_TokenRequired(t *testing.T) {
// Server returns 405 for non-PUT token; a GET-only client would 401 downstream.
srv := metadataStub(t, true)
if _, err := fetchInstanceMetadata(srv.Client(), srv.URL+"/latest"); err != nil {
t.Fatalf("v2 path should succeed: %v", err)
}
}
func TestFetchInstanceMetadata_BadDocument(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600")
_, _ = w.Write([]byte("t"))
})
mux.HandleFunc("/latest/meta-data/instance-id", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("i-1")) })
mux.HandleFunc("/latest/meta-data/placement/availability-zone", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("z")) })
mux.HandleFunc("/latest/dynamic/instance-identity/document", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("not json")) })
srv := httptest.NewServer(mux)
t.Cleanup(srv.Close)
if _, err := fetchInstanceMetadata(srv.Client(), srv.URL+"/latest"); err == nil {
t.Fatal("expected parse error on bad identity document")
}
}
-115
View File
@@ -1,115 +0,0 @@
package main
import (
"context"
"log/slog"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// reconcile re-syncs the gateway with what this instance is actually running
// after an agent restart. It lists the runtime's containers and, for each task
// in this cluster, either re-adopts it (still running: re-register IAM
// credentials, re-attach the exit-wait, refresh RUNNING) or reports it STOPPED
// and clears its leftovers. The returned set seeds pollAssignments so the
// gateway's re-delivered assignments for these tasks are acked but not re-run.
//
// Reporting the dead ones is the load-bearing half. The exit-wait is cancelled
// when the agent shuts down, so a container that dies while the agent is down
// has nobody to report its exit; the gateway holds the task RUNNING forever,
// its service never replaces it, and the target never recovers.
//
// It is best-effort: a nil runner or a List error returns an empty set and the
// agent falls back to its pre-reconcile behaviour (poll + run).
func (a *Agent) reconcile(ctx context.Context) map[string]bool {
adopted := map[string]bool{}
runner := a.rt.get()
if runner == nil {
return adopted
}
containers, err := runner.List(ctx)
if err != nil {
slog.Warn("ecs-agent: reconcile list failed", "err", err)
return adopted
}
// Group this cluster's containers back into their tasks, running or not. The
// Assign is reconstructed from labels — enough to re-register creds and tear
// down netns.
type task struct {
as *bus.Assign
running []ctrruntime.Container
dead []ctrruntime.Container
}
tasks := map[string]*task{}
for _, c := range containers {
if c.Labels[labelClusterName] != a.id.ClusterName {
continue
}
taskID := c.Labels[labelTaskID]
if taskID == "" {
continue
}
t := tasks[taskID]
if t == nil {
t = &task{as: &bus.Assign{
TaskID: taskID,
ClusterName: a.id.ClusterName,
CredID: c.Labels[labelCredID],
TaskRoleARN: c.Labels[labelTaskRoleARN],
ENIMacAddress: c.Labels[labelENIMac],
}}
tasks[taskID] = t
}
if c.Running {
t.running = append(t.running, c)
} else {
t.dead = append(t.dead, c)
}
}
for taskID, t := range tasks {
// A task with nothing running is over, whatever the gateway still believes.
// Mixed running/dead is left to the running path: the exit-wait re-attached
// below reports the individual container when it is reaped.
if len(t.running) == 0 {
a.reportStoppedTask(ctx, t.as, t.dead)
adopted[taskID] = true
continue
}
a.registerTaskCreds(t.as)
statuses := make([]bus.ContainerStatus, 0, len(t.running))
for _, c := range t.running {
name := c.Labels[labelContainerName]
go a.waitContainer(ctx, t.as, name, c.ID)
statuses = append(statuses, bus.ContainerStatus{Name: name, Status: bus.TaskStatusRunning, ContainerID: c.ID})
}
a.reportTaskState(t.as, bus.TaskStatusRunning, "", statuses)
adopted[taskID] = true
slog.Info("ecs-agent: re-adopted running task", "task", taskID, "containers", len(t.running))
}
return adopted
}
// reportStoppedTask clears a task's leftover containers and tells the gateway it
// is STOPPED, so the scheduler stops counting a task this instance is not
// running and its service can place a replacement.
func (a *Agent) reportStoppedTask(ctx context.Context, as *bus.Assign, dead []ctrruntime.Container) {
runner := a.rt.get()
statuses := make([]bus.ContainerStatus, 0, len(dead))
for _, c := range dead {
if runner == nil {
break
}
if err := runner.Remove(ctx, c.ID); err != nil {
slog.Warn("ecs-agent: reconcile remove failed", "task", as.TaskID, "container", c.ID, "err", err)
}
statuses = append(statuses, bus.ContainerStatus{
Name: c.Labels[labelContainerName], Status: bus.TaskStatusStopped, ContainerID: c.ID,
})
}
a.reportTaskState(as, bus.TaskStatusStopped, "Container exited while the agent was not running", statuses)
slog.Info("ecs-agent: reported stopped task", "task", as.TaskID, "containers", len(dead))
}
-250
View File
@@ -1,250 +0,0 @@
package main
import (
"context"
"errors"
"slices"
"testing"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
const testTaskRole = "arn:aws:iam::111122223333:role/task"
// adoptedContainer builds a runtime.Container as the reconciler would discover it,
// stamping only the mulga.ecs.* labels that are non-empty.
func adoptedContainer(taskID, name, cluster, credID, role, mac string, running bool) ctrruntime.Container {
labels := map[string]string{
labelTaskID: taskID,
labelContainerName: name,
labelClusterName: cluster,
}
if credID != "" {
labels[labelCredID] = credID
}
if role != "" {
labels[labelTaskRoleARN] = role
}
if mac != "" {
labels[labelENIMac] = mac
}
return ctrruntime.Container{ID: containerID(taskID, name), Labels: labels, Running: running}
}
func waitedFor(t *testing.T, rt *ctrruntime.FakePuller, id string) {
t.Helper()
deadline := time.After(time.Second)
for !slices.Contains(rt.Waits(), id) {
select {
case <-deadline:
t.Fatalf("exit-wait never re-attached for %s; waited=%v", id, rt.Waits())
case <-time.After(2 * time.Millisecond):
}
}
}
// A running, correctly-labeled container is re-adopted: its credentials are
// re-registered, its RUNNING state refreshed, and its exit-wait re-attached.
func TestReconcile_AdoptsRunningLabeledContainer(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{
WaitErr: errors.New("blocked"),
Containers: []ctrruntime.Container{
adoptedContainer("t-001", "web", "default", "t-001", testTaskRole, "52:54:00:de:ad:01", true),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
adopted := a.reconcile(context.Background())
if !adopted["t-001"] {
t.Fatalf("t-001 not adopted; got %v", adopted)
}
if got := a.cred.roles["t-001"]; got != testTaskRole {
t.Errorf("cred not re-registered: roles[t-001] = %q, want %q", got, testTaskRole)
}
if n := countStatus(cp.taskStates(), "t-001", bus.TaskStatusRunning); n != 1 {
t.Errorf("RUNNING reports for t-001 = %d, want 1", n)
}
waitedFor(t, rt, containerID("t-001", "web"))
}
// A task whose container died while the agent was down must be reported STOPPED,
// not silently skipped. The exit-wait is cancelled on agent shutdown, so nothing
// else ever reports that exit: skipping it leaves the gateway holding the task
// RUNNING forever, its service never places a replacement, and the load
// balancer target stays unhealthy with no path back.
func TestReconcile_ReportsTaskWhoseContainerDied(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{
Containers: []ctrruntime.Container{
adoptedContainer("t-dead", "web", "default", "t-dead", testTaskRole, "52:54:00:de:ad:02", false),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
adopted := a.reconcile(context.Background())
if n := countStatus(cp.taskStates(), "t-dead", bus.TaskStatusStopped); n != 1 {
t.Errorf("STOPPED reports for t-dead = %d, want 1", n)
}
if n := countStatus(cp.taskStates(), "t-dead", bus.TaskStatusRunning); n != 0 {
t.Errorf("t-dead must not be reported RUNNING, got %d such reports", n)
}
// Seeded so a re-delivered assignment for the dead task is acked, not re-run.
if !adopted["t-dead"] {
t.Errorf("t-dead not seeded into the adopted set; got %v", adopted)
}
if want := containerID("t-dead", "web"); !slices.Contains(rt.Removed, want) {
t.Errorf("leftover container %s not removed; removed=%v", want, rt.Removed)
}
}
// A task with one live and one dead container is still running: the live half is
// re-adopted and the re-attached exit-wait reports the dead one when it is
// reaped, so reconcile must not declare the whole task STOPPED.
func TestReconcile_PartiallyDeadTaskStaysRunning(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{
WaitErr: errors.New("blocked"),
Containers: []ctrruntime.Container{
adoptedContainer("t-mixed", "web", "default", "t-mixed", testTaskRole, "52:54:00:de:ad:03", true),
adoptedContainer("t-mixed", "sidecar", "default", "t-mixed", testTaskRole, "52:54:00:de:ad:03", false),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
a.reconcile(context.Background())
if n := countStatus(cp.taskStates(), "t-mixed", bus.TaskStatusRunning); n != 1 {
t.Errorf("RUNNING reports for t-mixed = %d, want 1", n)
}
if n := countStatus(cp.taskStates(), "t-mixed", bus.TaskStatusStopped); n != 0 {
t.Errorf("t-mixed must not be reported STOPPED while a container still runs, got %d", n)
}
}
// pollAssignments seeded with an adopted task acks its re-delivered assignment
// but does not re-run it, while a genuinely new task still dispatches.
func TestPollAssignments_SeededTaskNotRerun(t *testing.T) {
mk := func(id string) bus.Assign {
return bus.Assign{
AccountID: "123456789012", ClusterName: "default", InstanceID: "i-abc123", TaskID: id,
Containers: []bus.AssignContainer{{Name: "web", Image: "registry/web:1", Command: []string{"/bin/true"}}},
}
}
adopted, fresh := mk("t-001"), mk("t-002")
cp := &fakeCP{pollReplies: [][]bus.Assign{
{adopted, fresh},
{adopted, fresh},
nil,
}}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
go a.pollAssignments(ctx, map[string]bool{"t-001": true})
deadline := time.After(time.Second)
for countStatus(cp.taskStates(), "t-002", bus.TaskStatusRunning) < 1 {
select {
case <-deadline:
t.Fatal("fresh task t-002 was never dispatched")
case <-time.After(2 * time.Millisecond):
}
}
time.Sleep(20 * time.Millisecond)
cancel()
if n := countStatus(cp.taskStates(), "t-001", bus.TaskStatusRunning); n != 0 {
t.Errorf("adopted task t-001 was re-run %d times, want 0", n)
}
if n := countStatus(cp.taskStates(), "t-002", bus.TaskStatusRunning); n != 1 {
t.Errorf("fresh task t-002 dispatched %d times, want 1", n)
}
var t1Acked bool
for _, ack := range cp.acks() {
for _, id := range ack {
if id == "t-001" {
t1Acked = true
}
}
}
if !t1Acked {
t.Errorf("adopted task t-001 was never acked; acks=%v", cp.acks())
}
}
// Containers belonging to another cluster are ignored entirely: this agent
// speaks for its own cluster only, and must neither adopt nor report them.
//
// A stopped container of this cluster IS handled — see
// TestReconcile_ReportsTaskWhoseContainerDied. It lands in the adopted set so
// its re-delivered assignment is acked rather than re-run, after reconcile has
// reported the task STOPPED.
func TestReconcile_IgnoresOtherClusterContainers(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{
WaitErr: errors.New("blocked"),
Containers: []ctrruntime.Container{
adoptedContainer("t-other", "web", "other", "", "", "", true),
adoptedContainer("t-other-dead", "web", "other", "", "", "", false),
adoptedContainer("t-ok", "web", "default", "", "", "", true),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, nil)
adopted := a.reconcile(context.Background())
if len(adopted) != 1 || !adopted["t-ok"] {
t.Fatalf("adopted = %v, want only t-ok", adopted)
}
for _, id := range []string{"t-other", "t-other-dead"} {
if n := countStatus(cp.taskStates(), id, bus.TaskStatusStopped); n != 0 {
t.Errorf("reported %d STOPPED states for another cluster's task %s, want 0", n, id)
}
}
}
// A nil runner or a List error degrades to an empty set without panicking.
func TestReconcile_DegradesGracefully(t *testing.T) {
noRunner := newAgent(config{}, testIdentity(), &fakeCP{}, nil, nil)
if got := noRunner.reconcile(context.Background()); len(got) != 0 {
t.Errorf("nil runner: adopted = %v, want empty", got)
}
rt := &ctrruntime.FakePuller{ListErr: errors.New("boom")}
listErr := newAgent(config{}, testIdentity(), &fakeCP{}, rt, nil)
if got := listErr.reconcile(context.Background()); len(got) != 0 {
t.Errorf("list error: adopted = %v, want empty", got)
}
}
// taskLabels emits the optional cred/role/MAC labels only when set.
func TestTaskLabels_OmitsEmptyOptionalLabels(t *testing.T) {
full := taskLabels(&bus.Assign{
TaskID: "t1", ClusterName: "c", CredID: "x", TaskRoleARN: testTaskRole, ENIMacAddress: "mac",
}, "web")
for k, want := range map[string]string{
labelCredID: "x", labelTaskRoleARN: testTaskRole, labelENIMac: "mac",
} {
if full[k] != want {
t.Errorf("full labels[%q] = %q, want %q", k, full[k], want)
}
}
bare := taskLabels(&bus.Assign{TaskID: "t1", ClusterName: "c"}, "web")
if len(bare) != 3 {
t.Errorf("bare labels = %v, want only taskID/containerName/clusterName", bare)
}
for _, k := range []string{labelCredID, labelTaskRoleARN, labelENIMac} {
if _, ok := bare[k]; ok {
t.Errorf("bare labels unexpectedly carry %q", k)
}
}
}
-78
View File
@@ -1,78 +0,0 @@
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ecs"
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// identity is the container instance's stable identity, assembled at boot from
// IMDS (account, instance, AZ) and config (cluster). It is the join key the
// scheduler uses to track this instance.
type identity struct {
AccountID string
ClusterName string
InstanceID string
AZ string
InstanceType string
Hostname string
Capacity bus.InstanceCapacity
AgentVersion string
}
// instanceAttributes builds the built-in container-instance attributes from the
// host's identity. Only attributes whose value was actually discovered are
// reported, so an empty one is absent rather than present and blank.
func instanceAttributes(id identity) []*ecs.Attribute {
var attrs []*ecs.Attribute
add := func(name, value string) {
if value == "" {
return
}
attrs = append(attrs, &ecs.Attribute{Name: aws.String(name), Value: aws.String(value)})
}
add(handlers_ecs.AttrInstanceType, id.InstanceType)
add(handlers_ecs.AttrAvailabilityZone, id.AZ)
return attrs
}
// advertisedIdentity is the identity to register with, given whether a
// container runtime is available. A host with no runtime can run nothing, so it
// advertises no capacity: the scheduler places nothing on it, and the control
// plane reads the zero as an involuntary drain. The identity is otherwise
// unchanged, so the instance stays visible rather than disappearing.
func advertisedIdentity(id identity, runtimeReady bool) identity {
if runtimeReady {
return id
}
id.Capacity = bus.InstanceCapacity{}
return id
}
// registrar registers the host as a container instance through the gateway.
type registrar struct {
cp controlPlane
id identity
// runtimeReady reports whether a container runtime is resolved. nil means
// always ready, which is what a test with no gate wants.
runtimeReady func() bool
}
func newRegistrar(cp controlPlane, id identity, runtimeReady func() bool) *registrar {
return &registrar{cp: cp, id: id, runtimeReady: runtimeReady}
}
// ready reports the runtime state, defaulting to ready when no gate is wired.
func (r *registrar) ready() bool { return r.runtimeReady == nil || r.runtimeReady() }
// Register calls the gateway's RegisterContainerInstance. The scheduler records
// (or refreshes) the container instance on receipt.
func (r *registrar) Register() error {
if err := r.cp.Register(advertisedIdentity(r.id, r.ready())); err != nil {
return fmt.Errorf("register instance %s: %w", r.id.InstanceID, err)
}
return nil
}
-257
View File
@@ -1,257 +0,0 @@
package main
import (
"context"
"errors"
"slices"
"sync"
"testing"
"time"
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// fakeCP is a test double for the gateway control-plane. It records registers,
// task-state reports and GPU reports, and replays a scripted queue of poll
// responses.
type fakeCP struct {
mu sync.Mutex
registers int
lastRegistered identity
states []bus.TaskState
gpuReports []gpuReport
pollAcks [][]string
pollStopAcks [][]string
pollReplies [][]bus.Assign
stopReplies [][]bus.StopDirective
stopPending []bus.StopDirective
pollCalls int
registerErr bool
submitErr bool
}
// gpuReport is one recorded ReportTaskGPU call.
type gpuReport struct {
cluster string
task string
containers []handlers_ecs.ContainerGPUReport
}
var _ controlPlane = (*fakeCP)(nil)
func (f *fakeCP) Register(id identity) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.registerErr {
return errors.New("register boom")
}
f.registers++
f.lastRegistered = id
return nil
}
// registered returns the identity of the most recent Register call.
func (f *fakeCP) registered() identity {
f.mu.Lock()
defer f.mu.Unlock()
return f.lastRegistered
}
func (f *fakeCP) SubmitTaskState(st bus.TaskState) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.submitErr {
return errors.New("submit boom")
}
f.states = append(f.states, st)
return nil
}
func (f *fakeCP) PollAssignments(_, _ string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.pollAcks = append(f.pollAcks, append([]string(nil), ackAssigns...))
f.pollStopAcks = append(f.pollStopAcks, append([]string(nil), ackStops...))
var out []bus.Assign
if f.pollCalls < len(f.pollReplies) {
out = f.pollReplies[f.pollCalls]
}
var stops []bus.StopDirective
if f.pollCalls < len(f.stopReplies) {
stops = f.stopReplies[f.pollCalls]
}
// stopPending is the durable inbox the gateway keeps: an acked directive is
// deleted, and everything still in it is redelivered on every poll.
f.stopPending = slices.DeleteFunc(f.stopPending, func(sd bus.StopDirective) bool {
return slices.Contains(ackStops, sd.TaskID)
})
stops = slices.Concat(stops, f.stopPending)
f.pollCalls++
return out, stops, nil
}
func (f *fakeCP) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
f.mu.Lock()
defer f.mu.Unlock()
f.gpuReports = append(f.gpuReports, gpuReport{cluster: cluster, task: task, containers: containers})
return nil
}
func (f *fakeCP) gpuReportsFor(taskID string) []gpuReport {
f.mu.Lock()
defer f.mu.Unlock()
var out []gpuReport
for _, r := range f.gpuReports {
if r.task == taskID {
out = append(out, r)
}
}
return out
}
func (f *fakeCP) registerCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.registers
}
func (f *fakeCP) taskStates() []bus.TaskState {
f.mu.Lock()
defer f.mu.Unlock()
return append([]bus.TaskState(nil), f.states...)
}
func (f *fakeCP) acks() [][]string {
f.mu.Lock()
defer f.mu.Unlock()
return append([][]string(nil), f.pollAcks...)
}
func (f *fakeCP) stopAcks() [][]string {
f.mu.Lock()
defer f.mu.Unlock()
return append([][]string(nil), f.pollStopAcks...)
}
func testIdentity() identity {
return identity{
AccountID: "123456789012",
ClusterName: "default",
InstanceID: "i-abc123",
AZ: "us-east-1a",
Hostname: "ecs-host",
Capacity: bus.InstanceCapacity{CPU: 2048, MemoryMiB: 4096},
AgentVersion: "test",
}
}
// testIdentityWithGPUs is testIdentity seeded with discovered GPU UUIDs, so
// newAgent's ledger has real devices to pin from.
func testIdentityWithGPUs(uuids ...string) identity {
id := testIdentity()
id.Capacity.GPU = len(uuids)
id.Capacity.GPUIDs = uuids
return id
}
func TestRegistrar_Register(t *testing.T) {
cp := &fakeCP{}
if err := newRegistrar(cp, testIdentity(), nil).Register(); err != nil {
t.Fatalf("Register: %v", err)
}
if cp.registerCount() != 1 {
t.Errorf("registers = %d, want 1", cp.registerCount())
}
}
func TestRegistrar_RegisterError(t *testing.T) {
cp := &fakeCP{registerErr: true}
if err := newRegistrar(cp, testIdentity(), nil).Register(); err == nil {
t.Fatal("expected register error")
}
}
func TestHeartbeater_Beat(t *testing.T) {
cp := &fakeCP{}
if err := newHeartbeater(cp, testIdentity(), time.Second, nil).beat(); err != nil {
t.Fatalf("beat: %v", err)
}
if cp.registerCount() != 1 {
t.Errorf("beat should re-register once, got %d", cp.registerCount())
}
}
func TestHeartbeater_DefaultInterval(t *testing.T) {
h := newHeartbeater(&fakeCP{}, testIdentity(), 0, nil)
if h.interval != defaultHeartbeat {
t.Errorf("interval = %v, want default", h.interval)
}
}
func TestHeartbeater_RunStopsOnContext(t *testing.T) {
cp := &fakeCP{}
h := newHeartbeater(cp, testIdentity(), 5*time.Millisecond, nil)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { h.Run(ctx); close(done) }()
time.Sleep(25 * time.Millisecond)
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("Run did not stop on context cancel")
}
if cp.registerCount() == 0 {
t.Error("expected at least one re-register before cancel")
}
}
// TestRegistrar_NoRuntimeAdvertisesNoCapacity is the placement half of the boot
// bug: an agent with no container runtime kept registering its full CPU and
// memory, so the scheduler placed tasks it could only fail to start. Capacity is
// advertised only while a runtime is there to use it.
func TestRegistrar_NoRuntimeAdvertisesNoCapacity(t *testing.T) {
cp := &fakeCP{}
ready := false
r := newRegistrar(cp, testIdentity(), func() bool { return ready })
if err := r.Register(); err != nil {
t.Fatalf("register: %v", err)
}
if got := cp.registered().Capacity; got.CPU != 0 || got.MemoryMiB != 0 {
t.Fatalf("advertised capacity with no runtime: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
}
ready = true
if err := r.Register(); err != nil {
t.Fatalf("register: %v", err)
}
want := testIdentity().Capacity
if got := cp.registered().Capacity; got.CPU != want.CPU || got.MemoryMiB != want.MemoryMiB {
t.Fatalf("capacity not restored once the runtime resolved: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
}
}
// TestHeartbeater_NoRuntimeAdvertisesNoCapacity is the same rule on the beat,
// which is how a runtime lost after boot reaches the control plane.
func TestHeartbeater_NoRuntimeAdvertisesNoCapacity(t *testing.T) {
cp := &fakeCP{}
h := newHeartbeater(cp, testIdentity(), time.Second, func() bool { return false })
if err := h.beat(); err != nil {
t.Fatalf("beat: %v", err)
}
if got := cp.registered().Capacity; got.CPU != 0 || got.MemoryMiB != 0 {
t.Fatalf("beat advertised capacity with no runtime: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
}
// The instance still identifies itself, so it stays visible as drained rather
// than vanishing from the cluster.
if cp.registered().InstanceID != testIdentity().InstanceID {
t.Error("beat dropped the instance identity")
}
}
+17 -170
View File
@@ -1,196 +1,43 @@
// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
// control-plane resource from the host through the AWS gateway, instead of
// dialing core NATS directly. It is the read-side companion to
// eks-gateway-publish.
//
// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
// as tab-separated lines the on-VM addon-sync shell agent consumes without a
// JSON parser:
//
// <addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
//
// The configuration values are base64-encoded so embedded tabs/newlines cannot
// break the line framing.
// control-plane resource from the host through the AWS gateway, writing it to
// stdout as TSV; the implementation is in agents/eks/gatewayfetch.
//
// Usage:
//
// eks-gateway-fetch -resource addons # GET /clusters/{cluster}/internal-addons/{accountId}
// eks-gateway-fetch -resource recovery -instance-id i-… # GET /clusters/{cluster}/internal-recovery/{accountId}/{instanceId}
//
// The recovery resource returns a single epoch\taction\tsnapshot line the on-VM
// k3s-recovery agent applies before k3s starts (etcd cluster-reset / wipe-rejoin).
//
// Flags default to environment variables seeded by cloud-init:
// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME, EKS_INSTANCE_ID.
package main
import (
"bufio"
"encoding/base64"
"encoding/json"
"context"
"flag"
"fmt"
"io"
"log/slog"
"os"
"time"
"github.com/mulgadc/spinifex/internal/eksgw"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
)
// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
// here to keep this tiny VM binary free of the handler package's dependency
// graph.
type stagedAddon struct {
AddonName string `json:"addonName"`
AddonVersion string `json:"addonVersion"`
ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
ConfigurationValues string `json:"configurationValues"`
}
type internalAddonsResponse struct {
Addons []stagedAddon `json:"addons"`
}
const (
maxAttempts = 30
retryDelay = 5 * time.Second
"github.com/mulgadc/spinifex/spinifex/agents/eks/gatewayfetch"
)
func main() {
var (
gatewayURL string
gatewayCA string
accessKey string
secretKey string
region string
accountID string
clusterName string
resource string
instanceID string
)
flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
flag.StringVar(&instanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons | recovery")
cfg := gatewayfetch.Config{Out: os.Stdout}
flag.StringVar(&cfg.GatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
flag.StringVar(&cfg.GatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
flag.StringVar(&cfg.AccessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
flag.StringVar(&cfg.SecretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
flag.StringVar(&cfg.Region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
flag.StringVar(&cfg.AccountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
flag.StringVar(&cfg.ClusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
flag.StringVar(&cfg.InstanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
flag.StringVar(&cfg.Resource, "resource", "addons", "Resource to fetch: addons | recovery")
flag.Parse()
switch {
case accountID == "":
fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
case clusterName == "":
fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
case resource != "addons" && resource != "recovery":
fatal("--resource must be addons or recovery")
case resource == "recovery" && instanceID == "":
fatal("--instance-id is required for --resource recovery (or set EKS_INSTANCE_ID)")
if err := gatewayfetch.Run(context.Background(), cfg); err != nil {
slog.Error("eks-gateway-fetch: " + err.Error())
os.Exit(1)
}
client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
if err != nil {
fatal(fmt.Sprintf("build gateway client: %v", err))
}
var (
path string
emit func(io.Writer, []byte) error
)
switch resource {
case "recovery":
path = "/clusters/" + clusterName + "/internal-recovery/" + accountID + "/" + instanceID
emit = emitRecoveryTSV
default:
path = "/clusters/" + clusterName + "/internal-addons/" + accountID
emit = emitAddonsTSV
}
var lastErr error
for attempt := 1; attempt <= maxAttempts; attempt++ {
body, err := client.Get(path)
if err != nil {
lastErr = err
slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
if attempt < maxAttempts {
time.Sleep(retryDelay)
}
continue
}
if werr := emit(os.Stdout, body); werr != nil {
fatal(fmt.Sprintf("render response: %v", werr))
}
return
}
fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
}
// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
// every locally-rendered manifest), which is the correct steady state for a
// cluster with no managed add-ons.
func emitAddonsTSV(out io.Writer, body []byte) error {
var resp internalAddonsResponse
if err := json.Unmarshal(body, &resp); err != nil {
return fmt.Errorf("unmarshal internal-addons response: %w", err)
}
w := bufio.NewWriter(out)
for _, a := range resp.Addons {
cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
return err
}
}
return w.Flush()
}
// recoveryDirective mirrors the gateway's RecoveryDirective wire shape; duplicated
// here to keep this tiny VM binary free of the handler package's dependency graph.
type recoveryDirective struct {
Epoch int64 `json:"epoch"`
Action string `json:"action"`
Snapshot string `json:"snapshot"`
SnapshotRequired bool `json:"snapshotRequired"`
}
type internalRecoveryResponse struct {
Directive recoveryDirective `json:"directive"`
}
// emitRecoveryTSV decodes the recovery directive and writes a single
// epoch\taction\tsnapshot\tsnapshotRequired line the on-VM k3s-recovery agent
// consumes without a JSON parser. An unset action defaults to "none" (steady
// state); snapshotRequired is 1 when the snapshot MUST restore (fresh DR seed
// aborts boot on fetch failure) else 0.
func emitRecoveryTSV(out io.Writer, body []byte) error {
var resp internalRecoveryResponse
if err := json.Unmarshal(body, &resp); err != nil {
return fmt.Errorf("unmarshal internal-recovery response: %w", err)
}
d := resp.Directive
if d.Action == "" {
d.Action = "none"
}
required := "0"
if d.SnapshotRequired {
required = "1"
}
if _, err := fmt.Fprintf(out, "%d\t%s\t%s\t%s\n", d.Epoch, d.Action, d.Snapshot, required); err != nil {
return err
}
return nil
}
func fatal(msg string) {
slog.Error("eks-gateway-fetch: " + msg)
os.Exit(1)
}
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"bytes"
"encoding/base64"
"testing"
)
func TestEmitAddonsTSV(t *testing.T) {
body := []byte(`{"addons":[
{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
"serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
"configurationValues":"{\"replicaCount\":2}"}
]}`)
var buf bytes.Buffer
if err := emitAddonsTSV(&buf, body); err != nil {
t.Fatalf("emitAddonsTSV: %v", err)
}
cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
want := "spinifex-noop\t0.1.0\t\t\n" +
"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
if buf.String() != want {
t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
}
}
func TestEmitAddonsTSV_Empty(t *testing.T) {
var buf bytes.Buffer
if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
t.Fatalf("emitAddonsTSV: %v", err)
}
if buf.Len() != 0 {
t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
}
}
func TestEmitAddonsTSV_BadJSON(t *testing.T) {
var buf bytes.Buffer
if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
t.Fatal("expected error on malformed JSON")
}
}
+17 -108
View File
@@ -1,13 +1,6 @@
// eks-gateway-publish runs inside the EKS K3s control-plane VM and relays a
// single publication to the host through the AWS gateway, instead of dialing
// core NATS directly.
//
// It reads a JSON payload on stdin, wraps it as
// {accountId, channel, kind, payload}, SigV4-signs (service "eks") an HTTPS
// POST to {gateway}/clusters/{cluster}/internal-publish, and retries with
// backoff until the gateway returns 2xx or the attempt budget is exhausted —
// so a degraded link surfaces as a non-zero exit rather than a silently
// dropped message (the failure mode of fire-and-forget `nats pub`).
// single JSON publication from stdin to the host through the AWS gateway; the
// implementation is in agents/eks/gatewaypublish.
//
// Usage:
//
@@ -20,115 +13,31 @@
package main
import (
"bytes"
"encoding/json"
"context"
"flag"
"fmt"
"io"
"log/slog"
"os"
"time"
"github.com/mulgadc/spinifex/internal/eksgw"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
)
const (
// Retry budget: the control plane reaches readiness before this runs, so a
// failing POST means a degraded link, not a cold start. Bounded so a stuck
// boot still terminates the OpenRC service.
maxAttempts = 30
retryDelay = 5 * time.Second
"github.com/mulgadc/spinifex/spinifex/agents/eks/gatewaypublish"
)
type publishBody struct {
AccountID string `json:"accountId"`
Channel string `json:"channel"`
Kind string `json:"kind"`
Payload json.RawMessage `json:"payload"`
}
func main() {
var (
gatewayURL string
gatewayCA string
accessKey string
secretKey string
region string
accountID string
clusterName string
channel string
kind string
)
flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
cfg := gatewaypublish.Config{Payload: os.Stdin}
flag.StringVar(&cfg.GatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
flag.StringVar(&cfg.GatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
flag.StringVar(&cfg.AccessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
flag.StringVar(&cfg.SecretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
flag.StringVar(&cfg.Region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
flag.StringVar(&cfg.AccountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
flag.StringVar(&cfg.ClusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
flag.StringVar(&cfg.Channel, "channel", "", "Publish channel: bootstrap|state|addon")
flag.StringVar(&cfg.Kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
flag.Parse()
switch {
case accountID == "":
fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
case clusterName == "":
fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
case channel != "bootstrap" && channel != "state" && channel != "addon":
fatal("--channel must be bootstrap, state or addon")
case channel == "bootstrap" && kind == "":
fatal("--kind is required for the bootstrap channel")
if err := gatewaypublish.Run(context.Background(), cfg); err != nil {
slog.Error("eks-gateway-publish: " + err.Error())
os.Exit(1)
}
payload, err := io.ReadAll(os.Stdin)
if err != nil {
fatal(fmt.Sprintf("read stdin payload: %v", err))
}
if len(bytes.TrimSpace(payload)) == 0 {
fatal("empty stdin payload")
}
if !json.Valid(payload) {
fatal("stdin payload is not valid JSON")
}
body, err := json.Marshal(publishBody{
AccountID: accountID,
Channel: channel,
Kind: kind,
Payload: json.RawMessage(payload),
})
if err != nil {
fatal(fmt.Sprintf("marshal request body: %v", err))
}
client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
if err != nil {
fatal(fmt.Sprintf("build gateway client: %v", err))
}
path := "/clusters/" + clusterName + "/internal-publish"
var lastErr error
for attempt := 1; attempt <= maxAttempts; attempt++ {
if _, err := client.Post(path, body); err != nil {
lastErr = err
slog.Warn("eks-gateway-publish: attempt failed",
"channel", channel, "kind", kind, "attempt", attempt, "err", err)
if attempt < maxAttempts {
time.Sleep(retryDelay)
}
continue
}
slog.Info("eks-gateway-publish: published", "channel", channel, "kind", kind)
return
}
fatal(fmt.Sprintf("publish failed after %d attempts: %v", maxAttempts, lastErr))
}
func fatal(msg string) {
slog.Error("eks-gateway-publish: " + msg)
os.Exit(1)
}
+15 -223
View File
@@ -1,16 +1,6 @@
// Command eks-konnectivity-cert mints, then reuses, an ECDSA-P256 serving
// certificate for the on-node konnectivity-server, signed by the K3s cluster CA.
//
// The konnectivity-agent (DaemonSet on workers) verifies the server it dials
// using the in-pod ServiceAccount CA (/var/run/secrets/.../ca.crt), which in
// K3s is the same server-ca that signs the apiserver. Signing the konnectivity
// serving cert with that CA is therefore what lets the agent validate the
// reverse-tunnel endpoint with no separate CA distribution.
//
// The eks-node AMI carries no openssl, so this is the Go equivalent: load the
// K3s server CA cert+key, issue a leaf (ServerAuth) cert carrying the cluster's
// konnectivity SANs (the NLB private-endpoint IP, public endpoint IP, NLB DNS),
// and cache it under -dir so the OpenRC service is byte-stable across restarts.
// Command eks-konnectivity-cert mints, then reuses, a K3s-CA-signed serving
// certificate for the on-node konnectivity-server; the implementation is in
// agents/eks/konnectivitycert.
//
// Usage:
//
@@ -23,225 +13,27 @@
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"context"
"flag"
"fmt"
"log/slog"
"math/big"
"net"
"os"
"path/filepath"
"strings"
"time"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
"github.com/mulgadc/spinifex/spinifex/agents/eks/konnectivitycert"
)
func main() {
dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
caCert := flag.String("ca-cert", "", "PEM path of the signing (K3s server) CA cert")
caKey := flag.String("ca-key", "", "PEM path of the signing CA private key")
cn := flag.String("cn", "konnectivity-server", "certificate common name")
sansCSV := flag.String("sans", "", "comma-separated SANs (IP or DNS)")
cfg := konnectivitycert.Config{Out: os.Stdout}
flag.StringVar(&cfg.Dir, "dir", "", "directory to cache tls.crt/tls.key in")
flag.StringVar(&cfg.CACert, "ca-cert", "", "PEM path of the signing (K3s server) CA cert")
flag.StringVar(&cfg.CAKey, "ca-key", "", "PEM path of the signing CA private key")
flag.StringVar(&cfg.CN, "cn", "konnectivity-server", "certificate common name")
flag.StringVar(&cfg.SANs, "sans", "", "comma-separated SANs (IP or DNS)")
flag.Parse()
if *dir == "" || *caCert == "" || *caKey == "" || *sansCSV == "" {
fatal("eks-konnectivity-cert: -dir, -ca-cert, -ca-key and -sans are required")
if err := konnectivitycert.Run(context.Background(), cfg); err != nil {
slog.Error("eks-konnectivity-cert: " + err.Error())
os.Exit(1)
}
dns, ips := splitSANs(*sansCSV)
if len(dns) == 0 && len(ips) == 0 {
fatal("eks-konnectivity-cert: -sans lists no usable names")
}
certPath, keyPath, err := ensureCert(*dir, *caCert, *caKey, *cn, dns, ips)
if err != nil {
fatal("eks-konnectivity-cert: " + err.Error())
}
fmt.Printf("%s\t%s\n", certPath, keyPath)
}
// ensureCert returns the cached cert/key paths under dir, minting and persisting
// a fresh CA-signed pair when none is present, readable, and still SAN-current.
func ensureCert(dir, caCertPath, caKeyPath, cn string, dns []string, ips []net.IP) (string, string, error) {
certPath := filepath.Join(dir, "tls.crt")
keyPath := filepath.Join(dir, "tls.key")
if certCoversSANs(certPath, keyPath, dns, ips) {
return certPath, keyPath, nil
}
caCert, caKey, err := loadCA(caCertPath, caKeyPath)
if err != nil {
return "", "", err
}
certPEM, keyPEM, err := issueLeaf(caCert, caKey, cn, dns, ips)
if err != nil {
return "", "", err
}
if err = os.MkdirAll(dir, 0o700); err != nil {
return "", "", fmt.Errorf("mkdir %s: %w", dir, err)
}
if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
return "", "", fmt.Errorf("write %s: %w", certPath, err)
}
if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
return "", "", fmt.Errorf("write %s: %w", keyPath, err)
}
return certPath, keyPath, nil
}
// certCoversSANs reports whether the cached cert exists, parses, has a key, and
// already carries every requested SAN — so a cert minted before the endpoint
// set changed is re-issued rather than served stale.
func certCoversSANs(certPath, keyPath string, dns []string, ips []net.IP) bool {
c, err := os.ReadFile(certPath)
if err != nil {
return false
}
if k, kerr := os.ReadFile(keyPath); kerr != nil || len(k) == 0 {
return false
}
der := decodeFirst(c)
if der == nil {
return false
}
crt, err := x509.ParseCertificate(der)
if err != nil {
return false
}
have := make(map[string]struct{}, len(crt.DNSNames)+len(crt.IPAddresses))
for _, n := range crt.DNSNames {
have[n] = struct{}{}
}
for _, ip := range crt.IPAddresses {
have[ip.String()] = struct{}{}
}
for _, n := range dns {
if _, ok := have[n]; !ok {
return false
}
}
for _, ip := range ips {
if _, ok := have[ip.String()]; !ok {
return false
}
}
return true
}
// loadCA parses the PEM CA cert + EC/PKCS8 private key used to sign the leaf.
func loadCA(certPath, keyPath string) (*x509.Certificate, *ecdsa.PrivateKey, error) {
certPEM, err := os.ReadFile(certPath)
if err != nil {
return nil, nil, fmt.Errorf("read ca cert %s: %w", certPath, err)
}
der := decodeFirst(certPEM)
if der == nil {
return nil, nil, fmt.Errorf("ca cert %s: no PEM block", certPath)
}
caCert, err := x509.ParseCertificate(der)
if err != nil {
return nil, nil, fmt.Errorf("parse ca cert: %w", err)
}
keyPEM, err := os.ReadFile(keyPath)
if err != nil {
return nil, nil, fmt.Errorf("read ca key %s: %w", keyPath, err)
}
keyDER := decodeFirst(keyPEM)
if keyDER == nil {
return nil, nil, fmt.Errorf("ca key %s: no PEM block", keyPath)
}
caKey, err := parseECKey(keyDER)
if err != nil {
return nil, nil, err
}
return caCert, caKey, nil
}
// parseECKey accepts the EC SEC1 or PKCS8 encodings K3s may write for the CA key.
func parseECKey(der []byte) (*ecdsa.PrivateKey, error) {
if k, err := x509.ParseECPrivateKey(der); err == nil {
return k, nil
}
k8, err := x509.ParsePKCS8PrivateKey(der)
if err != nil {
return nil, fmt.Errorf("parse ca key (sec1 and pkcs8 failed): %w", err)
}
ec, ok := k8.(*ecdsa.PrivateKey)
if !ok {
return nil, fmt.Errorf("ca key is %T, want ecdsa", k8)
}
return ec, nil
}
// issueLeaf mints an ECDSA-P256 ServerAuth leaf for cn + SANs, signed by caCert/caKey.
func issueLeaf(caCert *x509.Certificate, caKey *ecdsa.PrivateKey, cn string, dns []string, ips []net.IP) (certPEM, keyPEM []byte, err error) {
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return nil, nil, fmt.Errorf("generate key: %w", err)
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return nil, nil, fmt.Errorf("serial: %w", err)
}
now := time.Now().UTC()
tmpl := x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: cn},
NotBefore: now.Add(-time.Hour),
NotAfter: now.AddDate(10, 0, 0),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
DNSNames: dns,
IPAddresses: ips,
}
der, err := x509.CreateCertificate(rand.Reader, &tmpl, caCert, &priv.PublicKey, caKey)
if err != nil {
return nil, nil, fmt.Errorf("create cert: %w", err)
}
keyDER, err := x509.MarshalECPrivateKey(priv)
if err != nil {
return nil, nil, fmt.Errorf("marshal key: %w", err)
}
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
return certPEM, keyPEM, nil
}
// splitSANs partitions the CSV into DNS names and IPs (an entry that parses as an
// IP goes to IPAddresses, else DNSNames), matching how SAN matching is typed.
func splitSANs(s string) (dns []string, ips []net.IP) {
for p := range strings.SplitSeq(s, ",") {
p = strings.TrimSpace(p)
if p == "" {
continue
}
if ip := net.ParseIP(p); ip != nil {
ips = append(ips, ip)
continue
}
dns = append(dns, p)
}
return dns, ips
}
// decodeFirst returns the DER bytes of the first PEM block, or nil.
func decodeFirst(pemBytes []byte) []byte {
blk, _ := pem.Decode(pemBytes)
if blk == nil {
return nil
}
return blk.Bytes
}
func fatal(msg string) {
slog.Error(msg)
os.Exit(1)
}
-153
View File
@@ -1,153 +0,0 @@
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"math/big"
"net"
"os"
"path/filepath"
"testing"
"time"
)
// writeTestCA mints a self-signed EC CA (cert + SEC1 key PEM) into dir, returning
// the two paths — a stand-in for K3s' /var/lib/rancher/k3s/server/tls/server-ca.*.
func writeTestCA(t *testing.T, dir string) (certPath, keyPath string) {
t.Helper()
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatalf("gen ca key: %v", err)
}
tmpl := x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "k3s-server-ca"},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().AddDate(10, 0, 0),
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature,
BasicConstraintsValid: true,
IsCA: true,
}
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
if err != nil {
t.Fatalf("create ca cert: %v", err)
}
keyDER, err := x509.MarshalECPrivateKey(priv)
if err != nil {
t.Fatalf("marshal ca key: %v", err)
}
certPath = filepath.Join(dir, "server-ca.crt")
keyPath = filepath.Join(dir, "server-ca.key")
if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
t.Fatalf("write ca cert: %v", err)
}
if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}), 0o600); err != nil {
t.Fatalf("write ca key: %v", err)
}
return certPath, keyPath
}
func leafFrom(t *testing.T, certPath string) *x509.Certificate {
t.Helper()
b, err := os.ReadFile(certPath)
if err != nil {
t.Fatalf("read leaf: %v", err)
}
crt, err := x509.ParseCertificate(decodeFirst(b))
if err != nil {
t.Fatalf("parse leaf: %v", err)
}
return crt
}
func TestEnsureCertSignsLeafWithSANsThatChainsToCA(t *testing.T) {
dir := t.TempDir()
caCert, caKey := writeTestCA(t, dir)
dns, ips := splitSANs("10.32.100.4,203.0.113.9,eks-toc.example")
certPath, keyPath, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips)
if err != nil {
t.Fatalf("ensureCert: %v", err)
}
if _, err := os.Stat(keyPath); err != nil {
t.Fatalf("key not written: %v", err)
}
leaf := leafFrom(t, certPath)
if leaf.IsCA {
t.Error("konnectivity serving cert must be a leaf, not a CA")
}
// The agent dials the NLB private-endpoint IP and verifies via the in-pod CA,
// so the IP must be a SAN or the tunnel TLS handshake fails.
if err := leaf.VerifyHostname("10.32.100.4"); err != nil {
t.Errorf("IP SAN 10.32.100.4 missing: %v", err)
}
// The leaf must chain to the K3s server CA — that CA is the in-pod ca.crt the
// agent trusts, so no separate CA distribution is needed.
caPEM, _ := os.ReadFile(caCert)
roots := x509.NewCertPool()
if !roots.AppendCertsFromPEM(caPEM) {
t.Fatal("append CA failed")
}
if _, err := leaf.Verify(x509.VerifyOptions{
Roots: roots,
DNSName: "eks-toc.example",
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
}); err != nil {
t.Errorf("leaf does not chain to the K3s server CA: %v", err)
}
}
func TestEnsureCertReusesThenReissuesOnSANChange(t *testing.T) {
dir := t.TempDir()
caCert, caKey := writeTestCA(t, dir)
dns, ips := splitSANs("10.32.100.4,eks-toc.example")
c1, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips)
if err != nil {
t.Fatalf("first ensureCert: %v", err)
}
first, _ := os.ReadFile(c1)
// Same SANs → cached pair reused byte-for-byte (a churning cert would bounce
// the konnectivity-server on every restart).
if _, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips); err != nil {
t.Fatalf("second ensureCert: %v", err)
}
if second, _ := os.ReadFile(c1); string(first) != string(second) {
t.Error("ensureCert re-minted instead of reusing the SAN-current cached cert")
}
// A new endpoint in the SAN set must force a re-issue.
dns2, ips2 := splitSANs("10.32.100.4,eks-toc.example,10.32.100.99")
if _, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns2, ips2); err != nil {
t.Fatalf("third ensureCert: %v", err)
}
if third, _ := os.ReadFile(c1); string(first) == string(third) {
t.Error("ensureCert served a stale cert that does not cover the new SAN")
}
if err := leafFrom(t, c1).VerifyHostname("10.32.100.99"); err != nil {
t.Errorf("re-issued cert missing new SAN: %v", err)
}
}
func TestSplitSANsPartitionsIPsAndDNS(t *testing.T) {
dns, ips := splitSANs(" 10.0.0.1 , host.example , , 2001:db8::1 ")
if len(dns) != 1 || dns[0] != "host.example" {
t.Errorf("dns = %v, want [host.example]", dns)
}
wantIPs := []net.IP{net.ParseIP("10.0.0.1"), net.ParseIP("2001:db8::1")}
if len(ips) != len(wantIPs) {
t.Fatalf("ips = %v, want %v", ips, wantIPs)
}
for i, ip := range ips {
if !ip.Equal(wantIPs[i]) {
t.Errorf("ips[%d] = %v, want %v", i, ip, wantIPs[i])
}
}
}
-121
View File
@@ -1,121 +0,0 @@
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/base64"
"encoding/pem"
"fmt"
"math/big"
"net"
"os"
"time"
"github.com/mulgadc/bluebottle/pkg/tlsconfig"
)
// serverTLSConfig is the webhook's loopback serving TLS config.
func serverTLSConfig(cert tls.Certificate) *tls.Config {
return &tls.Config{
Certificates: []tls.Certificate{cert},
MinVersion: tls.VersionTLS13,
CurvePreferences: tlsconfig.Curves,
}
}
// ensureServingCert loads the persisted self-signed serving cert/key, or mints
// a fresh ECDSA-P256 pair (SAN 127.0.0.1 / localhost) and persists it. Reusing
// across restarts keeps the CA the apiserver loaded into its webhook kubeconfig
// valid even if the webhook process is restarted. Returns the parsed cert and
// the certificate PEM (for embedding as the kubeconfig CA bundle).
func ensureServingCert(certPath, keyPath string) (tls.Certificate, []byte, error) {
// Fall through to regenerate on a missing/corrupt/incompatible persisted
// pair. The cert PEM is re-read because tls.Certificate keeps only DER.
if cert, err := tls.LoadX509KeyPair(certPath, keyPath); err == nil {
if certPEM, err := os.ReadFile(certPath); err == nil {
return cert, certPEM, nil
}
}
certPEM, keyPEM, err := generateSelfSigned()
if err != nil {
return tls.Certificate{}, nil, err
}
if err := os.WriteFile(certPath, certPEM, 0o644); err != nil {
return tls.Certificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
}
if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
return tls.Certificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
}
cert, err := tls.X509KeyPair(certPEM, keyPEM)
if err != nil {
return tls.Certificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
}
return cert, certPEM, nil
}
func generateSelfSigned() (certPEM, keyPEM []byte, err error) {
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return nil, nil, fmt.Errorf("generate key: %w", err)
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return nil, nil, fmt.Errorf("serial: %w", err)
}
now := time.Now().UTC()
tmpl := x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: "eks-token-webhook"},
NotBefore: now.Add(-time.Hour),
NotAfter: now.AddDate(10, 0, 0),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
IsCA: true,
IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1), net.IPv6loopback},
DNSNames: []string{"localhost"},
}
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
if err != nil {
return nil, nil, fmt.Errorf("create cert: %w", err)
}
keyDER, err := x509.MarshalECPrivateKey(priv)
if err != nil {
return nil, nil, fmt.Errorf("marshal key: %w", err)
}
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
return certPEM, keyPEM, nil
}
// writeAPIServerKubeconfig writes the kubeconfig kube-apiserver loads via
// --authentication-token-webhook-config-file. It points the apiserver at the
// loopback webhook and trusts its self-signed serving cert as the CA bundle.
func writeAPIServerKubeconfig(path, addr string, certPEM []byte) error {
caB64 := base64.StdEncoding.EncodeToString(certPEM)
kubeconfig := fmt.Sprintf(`apiVersion: v1
kind: Config
clusters:
- name: mulga-eks-token-webhook
cluster:
certificate-authority-data: %s
server: https://%s/authenticate
users:
- name: kube-apiserver
contexts:
- name: webhook
context:
cluster: mulga-eks-token-webhook
user: kube-apiserver
current-context: webhook
`, caB64, addr)
if err := os.WriteFile(path, []byte(kubeconfig), 0o600); err != nil {
return fmt.Errorf("write %s: %w", path, err)
}
return nil
}
+25 -179
View File
@@ -1,76 +1,46 @@
// eks-token-webhook is the K8s TokenReview webhook authenticator baked into
// the eks-server AMI. Per kubectl call the kube-apiserver POSTs the bearer
// token (produced by `aws eks get-token`); the webhook relays it through the
// AWS gateway broker (SigV4 HTTPS POST to /clusters/{name}/token-review), which
// host-side runs the STS verify (cross-cluster x-k8s-aws-id pin) + AccessEntry
// KV lookup and returns the resolved identity. The webhook never speaks core
// NATS — same gateway-broker model as ELBv2's lb-agent and eks-gateway-publish.
//
// It binds 127.0.0.1 only; the kube-apiserver is the sole client (loopback).
// On startup it writes the apiserver webhook kubeconfig (with its self-signed
// serving CA) so k3s can be pointed at it via
// --authentication-token-webhook-config-file.
//
// SigV4 creds come from the AWS SDK chain (IMDS instance-role) when static
// EKS_ACCESS_KEY/EKS_SECRET_KEY are absent, matching the sibling helpers.
// Command eks-token-webhook is the K8s TokenReview webhook authenticator baked
// into the eks-server AMI; the implementation is in agents/eks/tokenwebhook.
// It reads its configuration from the first-boot env cloud-init seeds onto the
// VM and binds loopback only.
package main
import (
"encoding/json"
"context"
"errors"
"flag"
"fmt"
"log/slog"
"net/http"
"os"
"time"
"github.com/mulgadc/spinifex/internal/eksgw"
handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
"github.com/mulgadc/spinifex/spinifex/agents/eks/tokenwebhook"
)
// config is the webhook's runtime configuration, sourced from the first-boot
// env (/etc/spinifex-eks/first-boot.env) cloud-init seeds onto the VM.
type config struct {
addr string
gatewayURL string
gatewayCA string
accessKey string
secretKey string
region string
accountID string
clusterName string
certPath string
keyPath string
kubeconfig string
}
func loadConfig() (config, error) {
c := config{
addr: flagAddr,
gatewayURL: os.Getenv("EKS_GATEWAY_URL"),
gatewayCA: os.Getenv("EKS_GATEWAY_CA"),
accessKey: os.Getenv("EKS_ACCESS_KEY"),
secretKey: os.Getenv("EKS_SECRET_KEY"),
region: os.Getenv("EKS_REGION"),
accountID: os.Getenv("EKS_ACCOUNT_ID"),
clusterName: os.Getenv("EKS_CLUSTER_NAME"),
certPath: envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
keyPath: envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
kubeconfig: envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
func loadConfig() (tokenwebhook.Config, error) {
c := tokenwebhook.Config{
Addr: flagAddr,
GatewayURL: os.Getenv("EKS_GATEWAY_URL"),
GatewayCA: os.Getenv("EKS_GATEWAY_CA"),
AccessKey: os.Getenv("EKS_ACCESS_KEY"),
SecretKey: os.Getenv("EKS_SECRET_KEY"),
Region: os.Getenv("EKS_REGION"),
AccountID: os.Getenv("EKS_ACCOUNT_ID"),
ClusterName: os.Getenv("EKS_CLUSTER_NAME"),
CertPath: envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
KeyPath: envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
Kubeconfig: envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
ReadyAddr: os.Getenv("EKS_WEBHOOK_READY_ADDR"),
}
// Static SigV4 creds are optional: when absent, eksgw.New signs with the
// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's
// sibling helpers use. The CP VM launches on an instance profile, so
// buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY by design.
switch {
case c.gatewayURL == "":
case c.GatewayURL == "":
return c, errors.New("EKS_GATEWAY_URL not set")
case c.accountID == "":
case c.AccountID == "":
return c, errors.New("EKS_ACCOUNT_ID not set")
case c.clusterName == "":
case c.ClusterName == "":
return c, errors.New("EKS_CLUSTER_NAME not set")
}
return c, nil
@@ -97,132 +67,8 @@ func main() {
os.Exit(1)
}
if err := run(cfg); err != nil {
if err := tokenwebhook.Run(context.Background(), cfg); err != nil {
slog.Error("eks-token-webhook fatal", "err", err)
os.Exit(1)
}
}
func run(cfg config) error {
// Serving cert + apiserver kubeconfig. Persisted and reused across restarts
// so a webhook crash does not invalidate the CA the apiserver already loaded.
tlsCert, certPEM, err := ensureServingCert(cfg.certPath, cfg.keyPath)
if err != nil {
return fmt.Errorf("serving cert: %w", err)
}
if err := writeAPIServerKubeconfig(cfg.kubeconfig, cfg.addr, certPEM); err != nil {
return fmt.Errorf("write apiserver kubeconfig: %w", err)
}
client, err := eksgw.New(cfg.gatewayURL, cfg.gatewayCA, cfg.accessKey, cfg.secretKey, cfg.region)
if err != nil {
return fmt.Errorf("build gateway client: %w", err)
}
authr := &authenticator{
accountID: cfg.accountID,
clusterName: cfg.clusterName,
review: gatewayReviewer(client, cfg.clusterName, cfg.accountID),
}
mux := http.NewServeMux()
mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
})
mux.HandleFunc("/authenticate", authr.handle)
server := &http.Server{
Addr: cfg.addr,
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
TLSConfig: serverTLSConfig(tlsCert),
}
slog.Info("eks-token-webhook listening", "addr", cfg.addr, "cluster", cfg.clusterName)
if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("serve: %w", err)
}
return nil
}
// tokenReviewRequest is the body POSTed to /clusters/{name}/token-review. The
// webhook holds system SigV4 creds (system account, not the cluster account),
// so accountId names the cluster account explicitly — same as PublishInternal.
type tokenReviewRequest struct {
AccountID string `json:"accountId"`
Token string `json:"token"`
}
// gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
// the gateway and decode the resolved identity. A non-2xx / transport error is
// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
// silent deny, distinguishing "webhook broker down" from "token rejected".
func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (handlers_eks.WebhookTokenReviewResult, error) {
path := "/clusters/" + clusterName + "/token-review"
return func(token string) (handlers_eks.WebhookTokenReviewResult, error) {
body, err := json.Marshal(tokenReviewRequest{AccountID: accountID, Token: token})
if err != nil {
return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("marshal request: %w", err)
}
respBody, err := client.Post(path, body)
if err != nil {
return handlers_eks.WebhookTokenReviewResult{}, err
}
var res handlers_eks.WebhookTokenReviewResult
if err := json.Unmarshal(respBody, &res); err != nil {
return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("decode response: %w", err)
}
return res, nil
}
}
// authenticator resolves a TokenReview by relaying to the gateway broker.
type authenticator struct {
accountID string
clusterName string
review func(token string) (handlers_eks.WebhookTokenReviewResult, error)
}
func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
var review tokenReview
if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
res, err := a.review(review.Spec.Token)
if err != nil {
// Broker fault (gateway down, transport error): a 5xx tells the apiserver
// to retry rather than treating a transient outage as a hard deny.
slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
return
}
status := tokenReviewStatus{Authenticated: res.Authenticated}
if res.Authenticated {
status.User = userInfo{
Username: res.Username,
UID: res.UID,
Groups: res.Groups,
}
// Confirm the token for the audiences the apiserver asked about; an empty
// status.audiences is rejected when --api-audiences is configured.
status.Audiences = review.Spec.Audiences
}
slog.Info("TokenReview decision", "authenticated", status.Authenticated, "username", status.User.Username, "audiences", status.Audiences)
resp := tokenReview{
APIVersion: "authentication.k8s.io/v1",
Kind: "TokenReview",
Status: status,
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(resp); err != nil {
slog.Error("encode TokenReview response", "err", err)
}
}
+2 -117
View File
@@ -1,127 +1,12 @@
package main
import (
"encoding/base64"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// postReview drives authenticator.handle with a TokenReview carrying token and
// audiences, returning the decoded response status and HTTP code.
func postReview(t *testing.T, a *authenticator, token string, audiences []string) (tokenReviewStatus, int) {
t.Helper()
body, err := json.Marshal(tokenReview{
APIVersion: "authentication.k8s.io/v1",
Kind: "TokenReview",
Spec: tokenReviewSpec{Token: token, Audiences: audiences},
})
require.NoError(t, err)
req := httptest.NewRequest(http.MethodPost, "/authenticate", strings.NewReader(string(body)))
w := httptest.NewRecorder()
a.handle(w, req)
if w.Code != http.StatusOK {
return tokenReviewStatus{}, w.Code
}
var resp tokenReview
require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp))
return resp.Status, w.Code
}
func TestHandle_GrantsAndStampsAudiences(t *testing.T) {
a := &authenticator{
accountID: "111122223333",
clusterName: "alpha",
review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
return handlers_eks.WebhookTokenReviewResult{
Authenticated: true,
Username: "arn:aws:iam::111122223333:role/admin",
UID: "AROAEXAMPLE:session",
Groups: []string{"system:masters"},
}, nil
},
}
status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
require.Equal(t, http.StatusOK, code)
require.True(t, status.Authenticated)
assert.Equal(t, "arn:aws:iam::111122223333:role/admin", status.User.Username)
assert.Equal(t, "AROAEXAMPLE:session", status.User.UID)
assert.Equal(t, []string{"system:masters"}, status.User.Groups)
assert.Equal(t, []string{"https://kubernetes.default.svc"}, status.Audiences)
}
func TestHandle_DeniesWithoutStampingAudiences(t *testing.T) {
a := &authenticator{
clusterName: "alpha",
review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
return handlers_eks.WebhookTokenReviewResult{Authenticated: false}, nil
},
}
status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
require.Equal(t, http.StatusOK, code)
assert.False(t, status.Authenticated)
assert.Empty(t, status.User.Username)
assert.Empty(t, status.Audiences, "unauthenticated review must not stamp audiences")
}
// A broker fault is a 5xx (retryable), not a silent deny — a transient gateway
// outage must not look identical to a rejected token.
func TestHandle_BrokerFaultReturns5xx(t *testing.T) {
a := &authenticator{
clusterName: "alpha",
review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
return handlers_eks.WebhookTokenReviewResult{}, errors.New("gateway returned 503")
},
}
_, code := postReview(t, a, "tok", nil)
assert.Equal(t, http.StatusServiceUnavailable, code)
}
func TestEnsureServingCert_PersistsAndReuses(t *testing.T) {
dir := t.TempDir()
certPath := filepath.Join(dir, "wh.crt")
keyPath := filepath.Join(dir, "wh.key")
c1, pem1, err := ensureServingCert(certPath, keyPath)
require.NoError(t, err)
require.NotEmpty(t, pem1)
require.NotNil(t, serverTLSConfig(c1))
// Second call reuses the persisted pair (same cert bytes).
_, pem2, err := ensureServingCert(certPath, keyPath)
require.NoError(t, err)
assert.Equal(t, pem1, pem2)
}
func TestWriteAPIServerKubeconfig(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "wh.kubeconfig")
certPEM := []byte("-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n")
require.NoError(t, writeAPIServerKubeconfig(path, "127.0.0.1:8443", certPEM))
data, err := os.ReadFile(path)
require.NoError(t, err)
body := string(data)
assert.Contains(t, body, "server: https://127.0.0.1:8443/authenticate")
assert.Contains(t, body, base64.StdEncoding.EncodeToString(certPEM))
assert.Contains(t, body, "current-context: webhook")
}
// The CP VM launches on an instance profile, so cloud-init omits the static
// EKS_ACCESS_KEY/EKS_SECRET_KEY and eksgw.New signs with IMDS creds. loadConfig
// must accept absent static keys; gating on them aborts every IMDS-mode boot,
@@ -148,8 +33,8 @@ func TestLoadConfig_AcceptsAbsentStaticCreds(t *testing.T) {
})
cfg, err := loadConfig()
require.NoError(t, err)
assert.Empty(t, cfg.accessKey)
assert.Empty(t, cfg.secretKey)
assert.Empty(t, cfg.AccessKey)
assert.Empty(t, cfg.SecretKey)
// Genuinely-required vars are still enforced.
setEnv(t, map[string]string{
+13 -127
View File
@@ -1,12 +1,6 @@
// Command eks-webhook-cert mints, then reuses, a self-signed ECDSA-P256 serving
// certificate for an in-cluster admission webhook and prints its base64 PEM so
// the addon renderer can embed it as the webhook Secret + each caBundle.
//
// The eks-node AMI carries no openssl, so this is the Go equivalent of the
// upstream Helm chart's install-time genSelfSignedCert: one cert per cluster,
// cached on disk under -dir, stable across addon-sync ticks. The single cert is
// its own CA (IsCA + ServerAuth), so the same PEM serves as both the TLS leaf
// and the caBundle the apiserver trusts.
// Command eks-webhook-cert mints, then reuses, a self-signed serving certificate
// for an in-cluster admission webhook and prints its base64 PEM; the
// implementation is in agents/eks/webhookcert.
//
// Usage:
//
@@ -17,133 +11,25 @@
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/base64"
"encoding/pem"
"context"
"flag"
"fmt"
"log/slog"
"math/big"
"os"
"path/filepath"
"strings"
"time"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
"github.com/mulgadc/spinifex/spinifex/agents/eks/webhookcert"
)
func main() {
dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
cn := flag.String("cn", "", "certificate common name")
dnsCSV := flag.String("dns", "", "comma-separated DNS SANs")
cfg := webhookcert.Config{Out: os.Stdout}
flag.StringVar(&cfg.Dir, "dir", "", "directory to cache tls.crt/tls.key in")
flag.StringVar(&cfg.CN, "cn", "", "certificate common name")
flag.StringVar(&cfg.DNS, "dns", "", "comma-separated DNS SANs")
flag.Parse()
if *dir == "" || *cn == "" || *dnsCSV == "" {
fatal("eks-webhook-cert: -dir, -cn and -dns are required")
if err := webhookcert.Run(context.Background(), cfg); err != nil {
slog.Error("eks-webhook-cert: " + err.Error())
os.Exit(1)
}
dnsNames := splitCSV(*dnsCSV)
if len(dnsNames) == 0 {
fatal("eks-webhook-cert: -dns lists no names")
}
certPEM, keyPEM, err := ensureCert(*dir, *cn, dnsNames)
if err != nil {
fatal("eks-webhook-cert: " + err.Error())
}
b := base64.StdEncoding.EncodeToString
fmt.Printf("%s\t%s\t%s\n", b(certPEM), b(certPEM), b(keyPEM))
}
// ensureCert returns the cached cert/key under dir, minting and persisting a
// fresh self-signed pair (for cn + dnsNames) when none is present or readable.
func ensureCert(dir, cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
certPath := filepath.Join(dir, "tls.crt")
keyPath := filepath.Join(dir, "tls.key")
if c, cerr := os.ReadFile(certPath); cerr == nil {
if k, kerr := os.ReadFile(keyPath); kerr == nil {
if _, perr := x509.ParseCertificate(decodeFirst(c)); perr == nil && len(k) > 0 {
return c, k, nil
}
}
}
certPEM, keyPEM, err = generateSelfSigned(cn, dnsNames)
if err != nil {
return nil, nil, err
}
if err = os.MkdirAll(dir, 0o700); err != nil {
return nil, nil, fmt.Errorf("mkdir %s: %w", dir, err)
}
if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
return nil, nil, fmt.Errorf("write %s: %w", certPath, err)
}
if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
return nil, nil, fmt.Errorf("write %s: %w", keyPath, err)
}
return certPEM, keyPEM, nil
}
// generateSelfSigned mints an ECDSA-P256 self-signed cert valid as both a CA and
// a server cert, with the given common name and DNS SANs.
func generateSelfSigned(cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return nil, nil, fmt.Errorf("generate key: %w", err)
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return nil, nil, fmt.Errorf("serial: %w", err)
}
now := time.Now().UTC()
tmpl := x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: cn},
NotBefore: now.Add(-time.Hour),
NotAfter: now.AddDate(10, 0, 0),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
IsCA: true,
DNSNames: dnsNames,
}
der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
if err != nil {
return nil, nil, fmt.Errorf("create cert: %w", err)
}
keyDER, err := x509.MarshalECPrivateKey(priv)
if err != nil {
return nil, nil, fmt.Errorf("marshal key: %w", err)
}
certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
return certPEM, keyPEM, nil
}
// decodeFirst returns the DER bytes of the first PEM block, or nil.
func decodeFirst(pemBytes []byte) []byte {
blk, _ := pem.Decode(pemBytes)
if blk == nil {
return nil
}
return blk.Bytes
}
func splitCSV(s string) []string {
var out []string
for p := range strings.SplitSeq(s, ",") {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}
func fatal(msg string) {
slog.Error(msg)
os.Exit(1)
}
-81
View File
@@ -1,81 +0,0 @@
package main
import (
"crypto/x509"
"os"
"path/filepath"
"testing"
)
const (
testCN = "aws-load-balancer-webhook-service.kube-system.svc"
testDNS = "aws-load-balancer-webhook-service.kube-system.svc.cluster.local"
)
func parseLeaf(t *testing.T, certPEM []byte) *x509.Certificate {
t.Helper()
leaf, err := x509.ParseCertificate(decodeFirst(certPEM))
if err != nil {
t.Fatalf("parse cert: %v", err)
}
return leaf
}
func TestEnsureCertMintsServingCert(t *testing.T) {
dir := t.TempDir()
certPEM, keyPEM, err := ensureCert(dir, testCN, []string{testCN, testDNS})
if err != nil {
t.Fatalf("ensureCert: %v", err)
}
if len(keyPEM) == 0 {
t.Fatal("empty key PEM")
}
leaf := parseLeaf(t, certPEM)
// Both webhook service DNS names must be present, or the apiserver's TLS
// dial to the webhook fails SAN verification.
for _, want := range []string{testCN, testDNS} {
if err := leaf.VerifyHostname(want); err != nil {
t.Errorf("SAN %q missing: %v", want, err)
}
}
if !leaf.IsCA {
t.Error("cert must be a CA so the same PEM serves as the webhook caBundle")
}
// The single self-signed cert must verify against itself as a root — this is
// exactly what the apiserver does with caBundle == the serving cert.
roots := x509.NewCertPool()
roots.AddCert(leaf)
if _, err := leaf.Verify(x509.VerifyOptions{
Roots: roots,
DNSName: testCN,
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
}); err != nil {
t.Errorf("self-signed cert does not verify as its own CA: %v", err)
}
}
func TestEnsureCertReusesCachedPair(t *testing.T) {
dir := t.TempDir()
cert1, key1, err := ensureCert(dir, testCN, []string{testCN})
if err != nil {
t.Fatalf("first ensureCert: %v", err)
}
cert2, key2, err := ensureCert(dir, testCN, []string{testCN})
if err != nil {
t.Fatalf("second ensureCert: %v", err)
}
if string(cert1) != string(cert2) || string(key1) != string(key2) {
t.Error("ensureCert regenerated instead of reusing the cached pair; a churning cert re-applies the addon every tick")
}
// The private key must not be world-readable on disk.
info, err := os.Stat(filepath.Join(dir, "tls.key"))
if err != nil {
t.Fatalf("stat key: %v", err)
}
if info.Mode().Perm()&0o077 != 0 {
t.Errorf("tls.key mode %o is group/other-accessible", info.Mode().Perm())
}
}
+3 -3
View File
@@ -6,9 +6,9 @@ import (
"strings"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
"github.com/mulgadc/spinifex/cmd/installer/autoinstall"
"github.com/mulgadc/spinifex/cmd/installer/install"
"github.com/mulgadc/spinifex/cmd/installer/ui"
"github.com/mulgadc/spinifex/spinifex/operator/installer/autoinstall"
"github.com/mulgadc/spinifex/spinifex/operator/installer/install"
"github.com/mulgadc/spinifex/spinifex/operator/installer/ui"
)
func main() {
-696
View File
@@ -1,696 +0,0 @@
package ui
import (
"fmt"
"net"
"strconv"
"strings"
"github.com/charmbracelet/bubbles/textinput"
tea "github.com/charmbracelet/bubbletea"
"github.com/charmbracelet/lipgloss"
"github.com/mulgadc/spinifex/cmd/installer/install"
"github.com/mulgadc/spinifex/cmd/installer/netprobe"
)
// foldedNIC marks a role with no interface of its own, which collapses onto
// the plane above it (vpc <- lan <- wan).
const foldedNIC = -1
// roleField identifies one editable element on the per-role screen. Which of
// these are actually shown depends on the plane, the addressing mode, whether
// the NIC is wireless, and whether Advanced is on.
type roleField int
const (
roleFieldNIC roleField = iota
roleFieldMethod
roleFieldIP
roleFieldMask
roleFieldGateway
roleFieldDNS
roleFieldVLAN
roleFieldMTU
roleFieldSSID
roleFieldWiFiPass
)
// roleForm is the editable state for one plane. The three forms are the whole
// of the installer's network configuration.
type roleForm struct {
plane install.Plane
// nic indexes into model.nics, or foldedNIC when the role is folded.
nic int
dhcp bool
focus roleField
ip textinput.Model
mask textinput.Model
gateway textinput.Model
dns textinput.Model
vlan textinput.Model
mtu textinput.Model
ssid textinput.Model
wifiPass textinput.Model
}
func newRoleForm(plane install.Plane, nic int) roleForm {
text := func(placeholder string, limit int) textinput.Model {
in := textinput.New()
in.Prompt = ""
in.Placeholder = placeholder
if limit > 0 {
in.CharLimit = limit
}
return in
}
f := roleForm{
plane: plane,
nic: nic,
dhcp: plane == install.PlaneWAN,
ip: text(defaultIPPlaceholder(plane), 0),
mask: text("255.255.255.0", 0),
gateway: text("192.168.1.1", 0),
dns: text("1.1.1.1, 8.8.8.8", 0),
vlan: text("untagged", 4),
mtu: text("1500", 5),
ssid: text("Network SSID", 64),
wifiPass: text("WiFi password", 128),
}
f.wifiPass.EchoMode = textinput.EchoPassword
return f
}
// defaultIPPlaceholder hints the addressing convention for each plane so an
// operator filling in a rack does not have to remember it.
func defaultIPPlaceholder(plane install.Plane) string {
switch plane {
case install.PlaneLAN:
return "10.0.0.3"
case install.PlaneVPC:
return "10.1.0.3"
default:
return "192.168.1.10"
}
}
func (f *roleForm) folded() bool { return f.nic == foldedNIC }
// canFold reports whether this role may be left unbound. wan always needs an
// uplink, so only lan and vpc can fold.
func (f *roleForm) canFold() bool { return f.plane != install.PlaneWAN }
// visibleFields returns the fields shown for this form, in tab order. Hiding
// rather than disabling keeps the default screen narrow: a DHCP role shows two
// lines, and VLAN/MTU only appear once Advanced is on.
func (f *roleForm) visibleFields(isWiFi, advanced bool) []roleField {
if f.folded() {
return []roleField{roleFieldNIC}
}
fields := []roleField{roleFieldNIC, roleFieldMethod}
if !f.dhcp {
fields = append(fields, roleFieldIP, roleFieldMask)
// Default route and resolvers are properties of the node, not of each
// plane, and wan is the only one that carries either: it holds the
// default route, so it is the only plane that can reach an upstream
// resolver. On DHCP the lease supplies both.
if f.plane == install.PlaneWAN {
fields = append(fields, roleFieldGateway, roleFieldDNS)
}
}
if advanced {
fields = append(fields, roleFieldVLAN, roleFieldMTU)
}
if isWiFi {
fields = append(fields, roleFieldSSID, roleFieldWiFiPass)
}
return fields
}
// input returns the textinput backing a field, or nil for the non-text fields.
func (f *roleForm) input(field roleField) *textinput.Model {
switch field {
case roleFieldIP:
return &f.ip
case roleFieldMask:
return &f.mask
case roleFieldGateway:
return &f.gateway
case roleFieldDNS:
return &f.dns
case roleFieldVLAN:
return &f.vlan
case roleFieldMTU:
return &f.mtu
case roleFieldSSID:
return &f.ssid
case roleFieldWiFiPass:
return &f.wifiPass
default:
return nil
}
}
// blurAll drops focus from every input so only the focused field has a cursor.
func (f *roleForm) blurAll() {
for _, field := range []roleField{
roleFieldIP, roleFieldMask, roleFieldGateway, roleFieldDNS,
roleFieldVLAN, roleFieldMTU, roleFieldSSID, roleFieldWiFiPass,
} {
f.input(field).Blur()
}
}
// focusCurrent gives the cursor to the focused field, if it is a text field.
func (f *roleForm) focusCurrent() {
f.blurAll()
if in := f.input(f.focus); in != nil {
in.Focus()
}
}
// toRole converts the form into the install package's role model.
func (f *roleForm) toRole(nics []netprobe.NIC) install.NetworkRole {
if f.folded() || f.nic >= len(nics) {
return install.NetworkRole{}
}
role := install.NetworkRole{
Interface: nics[f.nic].Name,
DHCPMode: f.dhcp,
VLAN: atoiOr(f.vlan.Value(), 0),
MTU: atoiOr(f.mtu.Value(), 0),
}
if !f.dhcp {
role.Address = strings.TrimSpace(f.ip.Value())
role.Mask = strings.TrimSpace(f.mask.Value())
role.DNS = parseDNS(f.dns.Value())
if f.plane == install.PlaneWAN {
role.Gateway = strings.TrimSpace(f.gateway.Value())
}
}
if nics[f.nic].IsWiFi {
role.WiFiSSID = strings.TrimSpace(f.ssid.Value())
role.WiFiPass = f.wifiPass.Value()
}
return role
}
func atoiOr(s string, fallback int) int {
n, err := strconv.Atoi(strings.TrimSpace(s))
if err != nil {
return fallback
}
return n
}
// ── Roles overview screen ─────────────────────────────────────────────────────
// continueRow is the cursor position below the three role rows.
const continueRow = 3
func (m model) handleRolesKey(key string) (tea.Model, tea.Cmd) {
switch key {
case "up", "k":
if m.roleCursor > 0 {
m.roleCursor--
}
case "down", "j":
if m.roleCursor < continueRow {
m.roleCursor++
}
case "a":
// Advanced reveals VLAN and MTU on every role at once, rather than
// per-role, so the operator flips it on for the whole node.
m.advanced = !m.advanced
m.validationErr = ""
case "enter":
if m.roleCursor == continueRow {
if err := m.buildConfig().Validate(); err != nil {
m.validationErr = capitalise(err.Error())
return m, nil
}
m.validationErr = ""
m.screen = screenIdentity
m.hostnameInput.Focus()
return m, nil
}
m.validationErr = ""
m.screen = screenNetworkRole
m.roles[m.roleCursor].focus = roleFieldNIC
m.roles[m.roleCursor].blurAll()
case "esc":
m.screen = screenDiskConfirm
m.eraseInput.Focus()
}
return m, nil
}
func (m model) viewNetworkRoles(w int) string {
title := styleTitle.Render("Network Planes")
subtitle := styleMuted.Render("A role left unbound folds onto the plane above it.")
adv := styleMuted.Render("Advanced: off")
if m.advanced {
adv = styleSuccess.Render("Advanced: on")
}
header := fmt.Sprintf(" %-*s%-*s", colRole, "ROLE", colIface, "INTERFACE")
if m.advanced {
header += fmt.Sprintf("%-*s%-*s", colVLAN, "VLAN", colMTU, "MTU")
}
header += "ADDRESS"
lines := []string{title, subtitle, "", adv, "", styleMuted.Render(header)}
for i := range m.roles {
row := m.renderRoleRow(i)
if i == m.roleCursor {
lines = append(lines, styleSelected.Render("> "+row))
} else {
lines = append(lines, " "+row)
}
if detail := m.roleHardware(i, bodyWidth(w)); detail != "" {
lines = append(lines, detail)
}
}
lines = append(lines, "")
cont := " Continue"
if m.roleCursor == continueRow {
cont = styleSelected.Render("> Continue")
}
lines = append(lines, cont)
if m.validationErr != "" {
lines = append(lines, "", styleError.Render(m.validationErr))
}
lines = append(lines, styleHelp.Render("↑/↓ move • Enter edit • a advanced • Esc back"))
body := lipgloss.JoinVertical(lipgloss.Left, lines...)
return lipgloss.Place(w, m.height, lipgloss.Center, lipgloss.Center,
styleBox.Width(min(w-4, 78)).Render(body),
)
}
// renderRoleRow renders one line of the overview table. A folded role names
// the plane it collapsed onto rather than showing a blank, so the operator can
// always see where the traffic will actually go.
func (m model) renderRoleRow(i int) string {
f := &m.roles[i]
name := fmt.Sprintf("%-5s", f.plane)
if f.folded() {
_, landed := m.buildConfig().Resolve(f.plane)
return fmt.Sprintf("%-*s%s", colRole, name, styleMuted.Render("folds onto "+string(landed)))
}
iface := styleMuted.Render("—")
if f.nic < len(m.nics) {
iface = nicSummary(m.nics[f.nic])
}
addr := "DHCP"
if !f.dhcp {
addr = formatCIDR(f.ip.Value(), f.mask.Value())
if addr == "" {
addr = styleWarning.Render("not set")
}
}
row := padCell(name, colRole) + padCell(iface, colIface)
if m.advanced {
row += padCell(valueOr(f.vlan.Value(), "—"), colVLAN) + padCell(valueOr(f.mtu.Value(), "—"), colMTU)
}
return row + addr
}
// roleHardware is the vendor/model line beneath a role row, indented under the
// interface column. It is what tells an operator that wan really is the 25gbe
// Mellanox and not the onboard 1gbe.
func (m model) roleHardware(i, width int) string {
f := &m.roles[i]
if f.folded() || f.nic >= len(m.nics) {
return ""
}
indent := 2 + colRole
return strings.Repeat(" ", indent) + styleMuted.Render(nicHardware(m.nics[f.nic], width-indent))
}
// nicSummary renders name, speed and link state, colouring the state so a port
// with no cable stands out rather than having to be read for.
func nicSummary(n netprobe.NIC) string {
state := styleWarning.Render(n.State)
if n.Carrier {
state = styleSuccess.Render(n.State)
}
speed := n.Speed
if speed == "" {
speed = "—"
}
return fmt.Sprintf("%s %s %s", n.Name, speed, state)
}
// nicHardware is the detail line: what the card is, plus the predictable name
// it also answers to, which is the one written on cabling notes. The alt name
// is dropped rather than truncated when the two will not fit together.
func nicHardware(n netprobe.NIC, width int) string {
hw := n.Hardware()
const separator = " · "
if n.AltName != "" && len([]rune(hw))+len(separator)+len([]rune(n.AltName)) <= width {
return hw + separator + n.AltName
}
return truncate(hw, width)
}
// bodyWidth is the text width inside the framed body. Detail lines are trimmed
// to it rather than left to wrap, which would push the table out of alignment.
func bodyWidth(w int) int {
return min(w-4, 78) - 4
}
// padCell pads a cell to width by display width, which the %-*s verb cannot do
// once a cell carries the ANSI escapes of a style.
func padCell(s string, width int) string {
return lipgloss.PlaceHorizontal(width, lipgloss.Left, s)
}
// Column widths for the overview table, shared by the header and the rows.
const (
colRole = 6
colIface = 32
colVLAN = 7
colMTU = 7
)
// focusMark gives the focused text field a visible caret without every field
// rendering its own prompt.
func focusMark(focused bool) string {
if focused {
return styleLabel.Render("> ")
}
return " "
}
// formatCIDR renders an address and mask the way an operator writes them,
// falling back to the raw mask when it is not dotted-decimal.
func formatCIDR(addr, mask string) string {
addr = strings.TrimSpace(addr)
mask = strings.TrimSpace(mask)
if addr == "" {
return ""
}
if mask == "" {
return addr
}
if ip := net.ParseIP(mask); ip != nil && ip.To4() != nil {
if ones, bits := net.IPMask(ip.To4()).Size(); bits != 0 {
return fmt.Sprintf("%s/%d", addr, ones)
}
}
return addr + "/" + strings.TrimPrefix(mask, "/")
}
func valueOr(v, fallback string) string {
if v = strings.TrimSpace(v); v != "" {
return v
}
return fallback
}
// ── Per-role editor screen ────────────────────────────────────────────────────
func (m model) handleRoleKey(key string, msg tea.KeyMsg) (tea.Model, tea.Cmd) {
f := &m.roles[m.roleCursor]
fields := f.visibleFields(m.nicIsWiFi(f.nic), m.advanced)
pos := fieldIndex(fields, f.focus)
switch key {
case "esc":
f.blurAll()
m.screen = screenNetworkRoles
m.validationErr = ""
return m, nil
case "enter":
if err := m.validateRole(m.roleCursor); err != "" {
m.validationErr = err
return m, nil
}
f.blurAll()
m.screen = screenNetworkRoles
m.validationErr = ""
return m, nil
case "tab", "down":
pos = (pos + 1) % len(fields)
f.focus = fields[pos]
f.focusCurrent()
return m, nil
case "shift+tab", "up":
pos = (pos - 1 + len(fields)) % len(fields)
f.focus = fields[pos]
f.focusCurrent()
return m, nil
case "left", "right":
// Only the two selector fields consume arrows; text fields need them
// for cursor movement.
switch f.focus {
case roleFieldNIC:
return m.cycleNIC(m.roleCursor, key == "right"), nil
case roleFieldMethod:
f.dhcp = key == "left"
f.focusCurrent()
return m, nil
}
}
if in := f.input(f.focus); in != nil {
var cmd tea.Cmd
*in, cmd = in.Update(msg)
return m, cmd
}
return m, nil
}
// cycleNIC steps the interface selection, including the fold option for lan
// and vpc. Wrapping keeps a single key usable on a node with one NIC.
func (m model) cycleNIC(idx int, forward bool) model {
f := &m.roles[idx]
low := 0
if f.canFold() {
low = foldedNIC
}
high := len(m.nics) - 1
if forward {
if f.nic >= high {
f.nic = low
} else {
f.nic++
}
return m
}
if f.nic <= low {
f.nic = high
} else {
f.nic--
}
return m
}
func (m model) nicIsWiFi(idx int) bool {
return idx >= 0 && idx < len(m.nics) && m.nics[idx].IsWiFi
}
func (m model) viewNetworkRole(w int) string {
f := &m.roles[m.roleCursor]
title := styleTitle.Render(fmt.Sprintf("Configure %s plane", strings.ToUpper(string(f.plane))))
subtitle := styleMuted.Render(planeDescription(f.plane))
lines := []string{title, subtitle, ""}
for _, field := range f.visibleFields(m.nicIsWiFi(f.nic), m.advanced) {
lines = append(lines, m.renderRoleField(f, field, bodyWidth(w)), "")
}
if m.validationErr != "" {
lines = append(lines, styleError.Render(m.validationErr), "")
}
lines = append(lines, styleHelp.Render("Tab/↑↓ move • ←/→ change • Enter done • Esc cancel"))
body := lipgloss.JoinVertical(lipgloss.Left, lines...)
return lipgloss.Place(w, m.height, lipgloss.Center, lipgloss.Center,
styleBox.Width(min(w-4, 78)).Render(body),
)
}
// planeDescription explains what rides each plane, so the choice is not just a
// three-letter name.
func planeDescription(p install.Plane) string {
switch p {
case install.PlaneWAN:
return "Public ingress and egress, EIPs and NAT (br-wan)."
case install.PlaneLAN:
return "Storage replication, NATS mesh and OVN control (br-lan)."
default:
return "Geneve tunnels carrying EC2 traffic between nodes (br-vpc)."
}
}
// roleLabelWidth is the field-label column on the editor screen, shared so a
// wrapped value can indent to line up underneath it.
const roleLabelWidth = 14
func (m model) renderRoleField(f *roleForm, field roleField, width int) string {
focused := f.focus == field
label := styleLabel.Render(fmt.Sprintf("%-*s", roleLabelWidth, roleFieldLabel(field)))
switch field {
case roleFieldNIC:
return label + m.renderNICChoice(f, focused, width)
case roleFieldMethod:
return label + renderToggle([]string{"DHCP (automatic)", "Static"}, boolToIdx(!f.dhcp), focused)
default:
if in := f.input(field); in != nil {
return label + focusMark(focused) + in.View()
}
return label
}
}
func roleFieldLabel(field roleField) string {
switch field {
case roleFieldNIC:
return "Interface"
case roleFieldMethod:
return "IP method"
case roleFieldIP:
return "IP address"
case roleFieldMask:
return "Subnet mask"
case roleFieldGateway:
return "Gateway"
case roleFieldDNS:
return "DNS"
case roleFieldVLAN:
return "VLAN id"
case roleFieldMTU:
return "MTU"
case roleFieldSSID:
return "WiFi SSID"
default:
return "WiFi password"
}
}
// renderNICChoice shows the selected interface on the field line and its
// hardware identity on a line of its own beneath. Vendor and model are what
// make a port identifiable, and they do not fit alongside the selector.
func (m model) renderNICChoice(f *roleForm, focused bool, width int) string {
var text, detail string
switch {
case f.folded():
_, landed := m.buildConfig().Resolve(f.plane)
text = fmt.Sprintf("(fold onto %s)", landed)
case f.nic < len(m.nics):
text = nicSummary(m.nics[f.nic])
detail = nicHardware(m.nics[f.nic], width-roleLabelWidth-2)
default:
text = "(none detected)"
}
if focused {
text = styleSelected.Render(" ← " + text + " → ")
} else {
text = styleLabel.Render("[ " + text + " ]")
}
if detail == "" {
return text
}
return text + "\n" + strings.Repeat(" ", roleLabelWidth+2) + styleMuted.Render(detail)
}
func renderToggle(options []string, selected int, focused bool) string {
var parts []string
for i, opt := range options {
switch {
case i == selected && focused:
parts = append(parts, styleSelected.Render(" "+opt+" "))
case i == selected:
parts = append(parts, styleLabel.Render("["+opt+"]"))
default:
parts = append(parts, styleMuted.Render(opt))
}
}
return strings.Join(parts, " ")
}
func boolToIdx(b bool) int {
if b {
return 1
}
return 0
}
func fieldIndex(fields []roleField, want roleField) int {
for i, f := range fields {
if f == want {
return i
}
}
return 0
}
// validateRole checks one plane's entries. Cross-role rules (a shared NIC
// without distinct VLAN ids, an unbound wan) belong to install.Config.Validate
// and run when the operator continues.
func (m model) validateRole(idx int) string {
f := &m.roles[idx]
if f.folded() {
return ""
}
if len(m.nics) == 0 {
return "No network interfaces detected"
}
if !f.dhcp {
if net.ParseIP(strings.TrimSpace(f.ip.Value())) == nil {
return "Enter a valid IP address"
}
if !validSubnetMask(f.mask.Value()) {
return "Enter a dotted-decimal subnet mask, e.g. 255.255.255.0 or 255.255.0.0"
}
if f.plane == install.PlaneWAN && net.ParseIP(strings.TrimSpace(f.gateway.Value())) == nil {
return "Enter a valid gateway address"
}
}
if v := strings.TrimSpace(f.vlan.Value()); v != "" {
id, err := strconv.Atoi(v)
if err != nil || id < 1 || id > 4094 {
return "VLAN id must be between 1 and 4094"
}
}
if v := strings.TrimSpace(f.mtu.Value()); v != "" {
mtu, err := strconv.Atoi(v)
if err != nil || mtu < 68 || mtu > 9216 {
return "MTU must be between 68 and 9216"
}
}
if m.nicIsWiFi(f.nic) && strings.TrimSpace(f.ssid.Value()) == "" {
return "Enter the WiFi network SSID"
}
return ""
}
func capitalise(s string) string {
if s == "" {
return s
}
return strings.ToUpper(s[:1]) + s[1:]
}
-357
View File
@@ -1,357 +0,0 @@
package ui
import (
"strings"
"testing"
"github.com/mulgadc/spinifex/cmd/installer/install"
"github.com/mulgadc/spinifex/cmd/installer/netprobe"
)
func nics(n int) []netprobe.NIC {
out := make([]netprobe.NIC, 0, n)
names := []string{"eno1", "ens1f0np0", "ens1f1np1", "ens2f0np0"}
for i := range n {
out = append(out, netprobe.NIC{Name: names[i], Speed: "25Gbps", Vendor: "Mellanox", State: "online"})
}
return out
}
// The pre-fill is the whole of the simple path: an operator with a
// conventional node should be able to press Continue without editing anything.
func TestNewModelPrefillsRolesByNICCount(t *testing.T) {
tests := []struct {
name string
count int
wantLAN, wantVPC int
}{
{"one NIC folds everything onto wan", 1, foldedNIC, foldedNIC},
{"two NICs dedicate the second to lan, vpc folds", 2, 1, foldedNIC},
{"three NICs give each plane its own", 3, 1, 2},
{"four NICs still only bind three", 4, 1, 2},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(tt.count))
if got := m.roles[0].nic; got != 0 {
t.Errorf("wan nic = %d, want 0", got)
}
if got := m.roles[1].nic; got != tt.wantLAN {
t.Errorf("lan nic = %d, want %d", got, tt.wantLAN)
}
if got := m.roles[2].nic; got != tt.wantVPC {
t.Errorf("vpc nic = %d, want %d", got, tt.wantVPC)
}
})
}
}
// Hiding rather than disabling is what keeps the default screen narrow.
func TestVisibleFields(t *testing.T) {
tests := []struct {
name string
mutate func(*roleForm)
isWiFi bool
advanced bool
want []roleField
}{
{
name: "wan on DHCP asks two questions",
mutate: func(f *roleForm) { f.dhcp = true },
want: []roleField{roleFieldNIC, roleFieldMethod},
},
{
name: "wan static adds a gateway",
mutate: func(f *roleForm) { f.dhcp = false },
want: []roleField{roleFieldNIC, roleFieldMethod, roleFieldIP, roleFieldMask, roleFieldGateway, roleFieldDNS},
},
{
name: "advanced reveals VLAN and MTU",
mutate: func(f *roleForm) { f.dhcp = true },
advanced: true,
want: []roleField{roleFieldNIC, roleFieldMethod, roleFieldVLAN, roleFieldMTU},
},
{
name: "wireless adds credentials",
mutate: func(f *roleForm) { f.dhcp = true },
isWiFi: true,
want: []roleField{roleFieldNIC, roleFieldMethod, roleFieldSSID, roleFieldWiFiPass},
},
{
name: "a folded role has nothing to configure",
mutate: func(f *roleForm) { f.nic = foldedNIC },
want: []roleField{roleFieldNIC},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f := newRoleForm(install.PlaneWAN, 0)
tt.mutate(&f)
got := f.visibleFields(tt.isWiFi, tt.advanced)
if len(got) != len(tt.want) {
t.Fatalf("visibleFields() = %v, want %v", got, tt.want)
}
for i := range got {
if got[i] != tt.want[i] {
t.Fatalf("visibleFields() = %v, want %v", got, tt.want)
}
}
})
}
}
// A non-wan role must be able to reach the folded state, and wan must not.
func TestCycleNICFoldReachableOnlyOffWAN(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(2))
// lan: 1 -> 0 -> folded -> wraps to the last NIC.
m.roles[1].nic = 1
for _, want := range []int{0, foldedNIC, 1} {
m = m.cycleNIC(1, false)
if m.roles[1].nic != want {
t.Fatalf("lan nic = %d, want %d", m.roles[1].nic, want)
}
}
// wan cycles only through real interfaces.
m.roles[0].nic = 0
for range 4 {
m = m.cycleNIC(0, false)
if m.roles[0].nic == foldedNIC {
t.Fatal("wan reached the folded state, but it always needs an uplink")
}
}
}
func TestToRoleReadsVLANAndMTU(t *testing.T) {
f := newRoleForm(install.PlaneLAN, 0)
f.dhcp = false
f.ip.SetValue("10.0.0.2")
f.mask.SetValue("255.255.0.0")
f.vlan.SetValue("10")
f.mtu.SetValue("9000")
role := f.toRole(nics(1))
if role.VLAN != 10 || role.MTU != 9000 {
t.Errorf("VLAN/MTU = %d/%d, want 10/9000", role.VLAN, role.MTU)
}
if role.Link() != "eno1.10" {
t.Errorf("Link() = %q, want eno1.10", role.Link())
}
// Only wan carries a default route, even if a gateway was typed earlier.
f.gateway.SetValue("10.0.0.1")
if got := f.toRole(nics(1)).Gateway; got != "" {
t.Errorf("lan Gateway = %q, want empty", got)
}
}
func TestToRoleFoldedIsEmpty(t *testing.T) {
f := newRoleForm(install.PlaneVPC, foldedNIC)
if !f.toRole(nics(2)).Folded() {
t.Error("a folded form must produce a folded role")
}
}
func TestValidateRole(t *testing.T) {
tests := []struct {
name string
plane install.Plane
mutate func(*roleForm)
wantErr string
}{
{"valid static lan", install.PlaneLAN, func(f *roleForm) {
f.dhcp = false
f.ip.SetValue("10.0.0.2")
f.mask.SetValue("255.255.0.0")
}, ""},
{"bad ip", install.PlaneLAN, func(f *roleForm) {
f.dhcp = false
f.ip.SetValue("not-an-ip")
f.mask.SetValue("255.255.0.0")
}, "valid IP"},
{"wan needs a gateway", install.PlaneWAN, func(f *roleForm) {
f.dhcp = false
f.ip.SetValue("216.218.163.99")
f.mask.SetValue("255.255.255.224")
}, "gateway"},
{"VLAN out of range", install.PlaneLAN, func(f *roleForm) {
f.dhcp = true
f.vlan.SetValue("9000")
}, "VLAN id must be"},
{"MTU out of range", install.PlaneLAN, func(f *roleForm) {
f.dhcp = true
f.mtu.SetValue("70000")
}, "MTU must be"},
{"folded roles skip validation", install.PlaneVPC, func(f *roleForm) {
f.nic = foldedNIC
f.ip.SetValue("nonsense")
}, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(3))
m.roles[0] = newRoleForm(tt.plane, 0)
tt.mutate(&m.roles[0])
got := m.validateRole(0)
if tt.wantErr == "" {
if got != "" {
t.Fatalf("validateRole() = %q, want no error", got)
}
return
}
if !strings.Contains(got, tt.wantErr) {
t.Fatalf("validateRole() = %q, want it to mention %q", got, tt.wantErr)
}
})
}
}
// Continue must refuse a layout install.Config.Validate would reject, rather
// than failing later during the install itself.
func TestContinueBlocksOnSharedNICWithoutVLANs(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(2))
m.screen = screenNetworkRoles
m.roleCursor = continueRow
m.roles[1].nic = 0 // lan onto the wan NIC, both untagged
next, _ := m.handleRolesKey("enter")
got := next.(model)
if got.screen != screenNetworkRoles {
t.Error("expected to stay on the roles screen when the layout is invalid")
}
if !strings.Contains(got.validationErr, "both bind eno1") {
t.Errorf("validationErr = %q, want it to name the shared interface", got.validationErr)
}
// Distinct VLAN ids make the same pairing legal.
got.roles[0].vlan.SetValue("30")
got.roles[1].vlan.SetValue("10")
next, _ = got.handleRolesKey("enter")
if next.(model).screen != screenIdentity {
t.Error("distinct VLAN ids should allow two roles to share one NIC")
}
}
func TestAdvancedToggle(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(3))
m.screen = screenNetworkRoles
next, _ := m.handleRolesKey("a")
if !next.(model).advanced {
t.Fatal("'a' should turn Advanced on")
}
next, _ = next.(model).handleRolesKey("a")
if next.(model).advanced {
t.Fatal("'a' should turn Advanced back off")
}
}
// The default route and the resolvers are node-level, and wan is the only plane
// that carries either, so the east-west editors must not ask for them. Their
// bridges are link-local and activate manually — a resolver pinned there is
// unreachable, and a gateway would fight the real default route.
func TestVisibleFieldsOmitsRoutingOffWAN(t *testing.T) {
for _, plane := range []install.Plane{install.PlaneLAN, install.PlaneVPC} {
t.Run(string(plane), func(t *testing.T) {
f := newRoleForm(plane, 1)
f.dhcp = false
got := f.visibleFields(false, false)
want := []roleField{roleFieldNIC, roleFieldMethod, roleFieldIP, roleFieldMask}
if len(got) != len(want) {
t.Fatalf("visibleFields() = %v, want %v", got, want)
}
for i := range got {
if got[i] != want[i] {
t.Fatalf("visibleFields() = %v, want %v", got, want)
}
}
})
}
}
// The mask field must accept only what netgen can consume. It previously
// advertised and accepted a prefix length, which passed the TUI and then failed
// the install itself with "invalid mask: 24".
func TestValidSubnetMaskRejectsPrefixLength(t *testing.T) {
for _, bad := range []string{"24", "/24", "8", "", "255.255.0", "255.0.255.0", "::1"} {
if validSubnetMask(bad) {
t.Errorf("validSubnetMask(%q) = true, want false", bad)
}
}
for _, good := range []string{"255.255.255.0", "255.255.0.0", "255.0.0.0", "255.255.255.252"} {
if !validSubnetMask(good) {
t.Errorf("validSubnetMask(%q) = false, want true", good)
}
}
}
// The hardware detail is the point of the NIC table: an operator has to be able
// to tell which physical port each plane landed on. It gets a line of its own,
// under the interface column, rather than being squeezed into the row.
func TestRoleHardwareRendersOnItsOwnLine(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, []netprobe.NIC{{
Name: "ens1f0np0", Vendor: "Mellanox Technologies", Model: "MT27710",
AltName: "enp132s0f0np0", Speed: "25 Gbps", Carrier: true, State: "online",
}})
view := m.viewNetworkRoles(100)
if !strings.Contains(view, "ens1f0np0 25 Gbps online") {
t.Errorf("expected the identity line in:\n%s", view)
}
detail := m.roleHardware(0, bodyWidth(100))
if !strings.Contains(detail, "Mellanox Technologies MT27710") {
t.Errorf("roleHardware = %q, want the vendor and model", detail)
}
if !strings.Contains(detail, "enp132s0f0np0") {
t.Errorf("roleHardware = %q, want the alternative name", detail)
}
// Every line has to fit the box, or lipgloss wraps it and the columns shear.
for line := range strings.SplitSeq(view, "\n") {
if len([]rune(line)) > 100 {
t.Errorf("line exceeds the terminal width: %q", line)
}
}
}
// A narrow box drops the alternative name before it truncates the model, since
// half a model name identifies nothing.
func TestNICHardwareDropsAltNameBeforeTruncating(t *testing.T) {
n := netprobe.NIC{Vendor: "Broadcom Inc. and subsidiaries", Model: "NetXtreme BCM5720", AltName: "enp1s0f1"}
if got := nicHardware(n, 60); got != "Broadcom Inc. and subsidiaries NetXtreme BCM5720" {
t.Errorf("nicHardware(60) = %q, want the name without the alt name", got)
}
if got := nicHardware(n, 20); got != "Broadcom Inc. and s…" {
t.Errorf("nicHardware(20) = %q, want a marked truncation", got)
}
}
// A role bound to a port with no hardware database entry still gets something
// to identify the card by, never the bare word "unknown".
func TestRoleHardwareFallsBackToDriver(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, []netprobe.NIC{{
Name: "eno1", Driver: "bnxt_en", DeviceID: "14e4:165f", State: "no cable",
}})
if got := m.roleHardware(0, bodyWidth(100)); !strings.Contains(got, "bnxt_en [14e4:165f]") {
t.Errorf("roleHardware = %q, want the driver and device id", got)
}
}
// A folded role has no interface, so it must not render an orphan detail line
// under the plane it collapsed onto.
func TestRoleHardwareEmptyWhenFolded(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda"}}, nics(1))
if got := m.roleHardware(1, bodyWidth(100)); got != "" {
t.Errorf("roleHardware for a folded role = %q, want empty", got)
}
}
-139
View File
@@ -1,139 +0,0 @@
package ui
import (
"slices"
"strings"
"testing"
"github.com/mulgadc/spinifex/cmd/installer/install"
)
// diskModel is a storage screen with n unselected disks and ext4 chosen, which
// is where the wizard starts.
func diskModel(n int) model {
m := model{fsCursor: slices.Index(install.AllFSTypes, install.FSExt4)}
for i := range n {
m.disks = append(m.disks, install.Disk{
Path: string(rune('a'+i)) + "-disk",
Bytes: 200 << 30,
LogicalBlockSize: 512,
PhysicalBlockSize: 512,
})
}
return m
}
func rolePaths(cfg install.DiskConfig) map[install.DiskRole]string {
out := map[install.DiskRole]string{}
for _, rm := range cfg.Roles {
out[rm.Role] = rm.Disk.Path
}
return out
}
// A single-disk machine should need no storage input at all, so the disk that
// newModel preselects has to arrive already holding the os role.
func TestPreselectedDiskIsAValidConfiguration(t *testing.T) {
m := newModel([]install.Disk{{Path: "/dev/sda", Bytes: 200 << 30, LogicalBlockSize: 512}}, nil)
cfg := m.storage()
if err := cfg.Validate(); err != nil {
t.Fatalf("the preselected disk must be installable as-is: %v", err)
}
if cfg.Primary().Path != "/dev/sda" {
t.Errorf("Primary() = %q, want the preselected disk", cfg.Primary().Path)
}
}
// Picking disks in order gives the layout an operator almost always wants, so
// the role key is only needed for the exceptions.
func TestSelectingDisksAssignsRolesInOrder(t *testing.T) {
m := diskModel(3)
for i := range 3 {
m = m.toggleDisk(i)
}
got := rolePaths(m.storage())
for role, want := range map[install.DiskRole]string{
install.RoleOS: "a-disk", install.RoleSpinifex: "b-disk", install.RolePredastore: "c-disk",
} {
if got[role] != want {
t.Errorf("%s = %q, want %q", role, got[role], want)
}
}
if err := m.storage().Validate(); err != nil {
t.Errorf("the default assignment must be valid: %v", err)
}
}
// Deselecting has to drop the role with the disk, or every later role shifts
// onto the wrong drive.
func TestDeselectingADiskReleasesItsRole(t *testing.T) {
m := diskModel(3)
for i := range 3 {
m = m.toggleDisk(i)
}
m = m.toggleDisk(1) // drop the spinifex drive
got := rolePaths(m.storage())
if got[install.RoleOS] != "a-disk" || got[install.RolePredastore] != "c-disk" {
t.Fatalf("remaining roles moved: %v", got)
}
if _, ok := got[install.RoleSpinifex]; ok {
t.Errorf("the spinifex role should be free again: %v", got)
}
}
// The two-drive predastore layout is only reachable by reassigning, so cycling
// swaps rather than refusing a role another disk already holds.
func TestCycleRoleSwapsWithTheDiskThatHoldsIt(t *testing.T) {
m := diskModel(2)
m = m.toggleDisk(0).toggleDisk(1)
// b-disk: spinifex → predastore, which no disk holds, so no swap.
m = m.cycleRole(1)
got := rolePaths(m.storage())
if got[install.RoleOS] != "a-disk" || got[install.RolePredastore] != "b-disk" {
t.Fatalf("roles = %v, want os on a-disk and predastore on b-disk", got)
}
// Cycling past the end wraps onto os, which a-disk holds, so they trade.
m = m.cycleRole(1)
got = rolePaths(m.storage())
if got[install.RoleOS] != "b-disk" || got[install.RolePredastore] != "a-disk" {
t.Fatalf("roles = %v, want the two disks to have swapped", got)
}
if err := m.storage().Validate(); err != nil {
t.Errorf("a swap must always leave a valid assignment: %v", err)
}
}
// ZFS pairs mirrors by selection order and has no roles to assign.
func TestStorageLeavesRolesUnsetOnZFS(t *testing.T) {
m := diskModel(2)
m.fsCursor = slices.Index(install.AllFSTypes, install.FSZFSRAID1)
m = m.toggleDisk(0).toggleDisk(1)
cfg := m.storage()
if len(cfg.Roles) != 0 {
t.Errorf("Roles = %v, want none on a pool", cfg.Roles)
}
if !slices.Equal(cfg.Paths(), []string{"a-disk", "b-disk"}) {
t.Errorf("Paths() = %v, want the selection order preserved", cfg.Paths())
}
}
// A three-drive selection looks like an array and is not one, so the mapping
// and the absence of redundancy both have to be on screen before the erase.
func TestConfirmScreenStatesTheLayoutAndTheRisk(t *testing.T) {
m := diskModel(3)
m.height = 40
for i := range 3 {
m = m.toggleDisk(i)
}
got := m.viewDiskConfirm(100)
for _, want := range []string{"/var/lib/spinifex/predastore", "c-disk", "losing one loses"} {
if !strings.Contains(got, want) {
t.Errorf("confirm screen is missing %q:\n%s", want, got)
}
}
}
+1 -1
View File
@@ -19,7 +19,7 @@ import (
"syscall"
"github.com/mulgadc/bluebottle/pkg/otelsetup"
"github.com/mulgadc/spinifex/spinifex/lbagent"
lbagent "github.com/mulgadc/spinifex/spinifex/agents/elbv2"
_ "github.com/mulgadc/bluebottle/pkg/fipsboot"
)
-424
View File
@@ -1,424 +0,0 @@
package main
import (
"bufio"
"context"
"errors"
"fmt"
"log/slog"
"net/url"
"os"
"strconv"
"strings"
"time"
"github.com/mulgadc/spinifex/internal/gwsign"
"github.com/mulgadc/spinifex/internal/rdsgw"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
"github.com/mulgadc/spinifex/spinifex/otelsetup"
)
// Overridable via -ldflags "-X main.version=...".
var version = "dev"
// The gateway resolves the authoritative DB instance from the request's
// credentials, so DBInstanceIdentifier is an assertion to be checked, not
// trusted.
type identity struct {
DBInstanceIdentifier string
AgentVersion string
EngineVersion string
}
// Every method is a SigV4-signed Query request, so the NATS bus stays
// host-internal.
type controlPlane interface {
Register(ctx context.Context, id identity) (*handlers_rds.RegisterDBInstanceOutput, error)
SubmitState(ctx context.Context, id identity, health handlers_rds.EngineHealth, message string) (*handlers_rds.SubmitDBStateChangeOutput, error)
// Returns Mode=initialize with the master password for as long as a payload
// staged for this VM generation is unacknowledged, and Mode=attach after.
GetBootstrapConfig(ctx context.Context, id identity) (*handlers_rds.GetDBBootstrapConfigOutput, error)
// Reports that PostgreSQL durably applied the staged password, which is what
// lets the control plane destroy it.
AcknowledgeBootstrap(ctx context.Context, id identity, pending *pendingBootstrap) error
PollCommands(ctx context.Context, id identity, replies []handlers_rds.CommandReply, wait time.Duration) ([]handlers_rds.Command, error)
}
// Bounds a register, heartbeat or bootstrap request. The long poll sets its own
// deadline from the wait it asked the gateway to hold.
const callTimeout = 15 * time.Second
// Keeps the poll's deadline past the requested wait, so a gateway answering at
// the end of its window is not cut off by the client.
const pollSlack = 10 * time.Second
type gatewayControlPlane struct {
client *rdsgw.Client
}
var _ controlPlane = (*gatewayControlPlane)(nil)
// Signs with the instance-role credentials the SDK chain resolves from IMDS,
// against the pinned gateway CA.
func newGatewayControlPlane(cfg config) (*gatewayControlPlane, error) {
signer, err := gwsign.NewIMDS(context.Background(), cfg.Region)
if err != nil {
return nil, fmt.Errorf("build IMDS signer: %w", err)
}
// The client timeout is the ceiling for the longest call, the long poll;
// shorter calls carry their own deadline.
client, err := rdsgw.New(cfg.GatewayURL, cfg.GatewayCA, signer, cfg.Region, cfg.PollWait+pollSlack)
if err != nil {
return nil, err
}
return &gatewayControlPlane{client: client}, nil
}
func (g *gatewayControlPlane) Register(ctx context.Context, id identity) (*handlers_rds.RegisterDBInstanceOutput, error) {
params := url.Values{}
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
setIfPresent(params, "AgentVersion", id.AgentVersion)
setIfPresent(params, "EngineVersion", id.EngineVersion)
ctx, cancel := context.WithTimeout(ctx, callTimeout)
defer cancel()
var out handlers_rds.RegisterDBInstanceOutput
if err := g.client.Call(ctx, "RegisterDBInstance", params, &out); err != nil {
return nil, err
}
return &out, nil
}
func (g *gatewayControlPlane) SubmitState(ctx context.Context, id identity, health handlers_rds.EngineHealth, message string) (*handlers_rds.SubmitDBStateChangeOutput, error) {
params := url.Values{"EngineHealth": {string(health)}}
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
setIfPresent(params, "EngineVersion", id.EngineVersion)
setIfPresent(params, "Message", message)
ctx, cancel := context.WithTimeout(ctx, callTimeout)
defer cancel()
var out handlers_rds.SubmitDBStateChangeOutput
if err := g.client.Call(ctx, "SubmitDBStateChange", params, &out); err != nil {
return nil, err
}
return &out, nil
}
func (g *gatewayControlPlane) GetBootstrapConfig(ctx context.Context, id identity) (*handlers_rds.GetDBBootstrapConfigOutput, error) {
params := url.Values{}
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
ctx, cancel := context.WithTimeout(ctx, callTimeout)
defer cancel()
var out handlers_rds.GetDBBootstrapConfigOutput
if err := g.client.Call(ctx, "GetDBBootstrapConfig", params, &out); err != nil {
return nil, err
}
return &out, nil
}
func (g *gatewayControlPlane) AcknowledgeBootstrap(ctx context.Context, id identity, pending *pendingBootstrap) error {
params := url.Values{
"PayloadId": {pending.payloadID},
"VMGeneration": {strconv.FormatInt(pending.vmGeneration, 10)},
}
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
setIfPresent(params, "DataVolumeId", pending.dataVolumeID)
ctx, cancel := context.WithTimeout(ctx, callTimeout)
defer cancel()
var out handlers_rds.AcknowledgeDBBootstrapOutput
if err := g.client.Call(ctx, "AcknowledgeDBBootstrap", params, &out); err != nil {
return err
}
if !out.Acknowledged {
return fmt.Errorf("the control plane did not acknowledge bootstrap payload %s", pending.payloadID)
}
return nil
}
// Declared here so the in-guest binary does not link the server side of the API.
type pollDBCommandsOutput struct {
Commands []handlers_rds.Command `xml:"Commands>member"`
}
func (g *gatewayControlPlane) PollCommands(ctx context.Context, id identity, replies []handlers_rds.CommandReply, wait time.Duration) ([]handlers_rds.Command, error) {
params := url.Values{"WaitTimeSeconds": {strconv.FormatInt(int64(wait.Seconds()), 10)}}
setIfPresent(params, "DBInstanceIdentifier", id.DBInstanceIdentifier)
// Replies ride the poll rather than a separate action, so replying and
// asking for the next command are one round trip.
for i, reply := range replies {
member := fmt.Sprintf("Replies.member.%d.", i+1)
params.Set(member+"CommandId", reply.CommandID)
params.Set(member+"Status", reply.Status)
setIfPresent(params, member+"Message", reply.Message)
}
ctx, cancel := context.WithTimeout(ctx, wait+pollSlack)
defer cancel()
var out pollDBCommandsOutput
if err := g.client.Call(ctx, "PollDBCommands", params, &out); err != nil {
return nil, err
}
return out.Commands, nil
}
// Only sets a parameter that has a value, so the gateway can tell "no
// assertion" from "asserted as blank".
func setIfPresent(params url.Values, key, value string) {
if value != "" {
params.Set(key, value)
}
}
type Agent struct {
cfg config
id identity
cp controlPlane
probe *engineProbe
engine engine
handoffWriter func(string, *handlers_rds.GetDBBootstrapConfigOutput) error
dataMountWaiter func(context.Context, string, string) error
// Set by bootstrap when the fetch replayed a staged payload; nil leaves the
// acknowledgement step a no-op, which is every attach boot.
pending *pendingBootstrap
hb *heartbeater
cmd *commander
guard *paramGuard
}
// Assembles from already-built parts, so tests can pass fakes; New builds the
// production control plane and delegates here.
func newAgent(cfg config, cp controlPlane, probe *engineProbe) (*Agent, error) {
if cfg.MountsFile == "" {
cfg.MountsFile = defaultMountsFile
}
id := identity{
DBInstanceIdentifier: cfg.DBInstanceIdentifier,
AgentVersion: version,
EngineVersion: cfg.EngineVersion,
}
a := &Agent{
cfg: cfg, id: id, cp: cp, probe: probe,
handoffWriter: writeHandoff, dataMountWaiter: waitForDataMount,
}
eng, err := newEngine(cfg, execCommandRunner, execSessionRunner, probe)
if err != nil {
return nil, err
}
a.engine = eng
a.hb = newHeartbeater(cp, probe, a.engine, handlers_rds.HeartbeatInterval)
a.cmd = newCommander(cp, newCommandRegistry(a.engine, newGuestStorage(cfg, execCommandRunner)), cfg.PollWait)
a.guard = newParamGuard(a.engine, probe, cp)
return a, nil
}
// New builds the agent from cfg, failing if no gateway URL is set. It does not
// wait for IMDS: the register loop rides out a datapath still coming up.
func New(cfg config) (*Agent, error) {
if cfg.GatewayURL == "" {
return nil, fmt.Errorf("no gateway URL configured (RDS_GATEWAY_URL)")
}
cp, err := newGatewayControlPlane(cfg)
if err != nil {
return nil, err
}
probe, err := newProbe(cfg, execProbeRunner)
if err != nil {
return nil, err
}
return newAgent(cfg, cp, probe)
}
func (a *Agent) Run(ctx context.Context) error {
// Named before the first dial: when register cannot reach the gateway, the
// address it is dialing is what separates a mgmt NIC that came up without one
// from a control plane that is genuinely down.
slog.Info("rds-agent: starting",
"gatewayURL", a.cfg.GatewayURL, "dbInstanceIdentifier", a.id.DBInstanceIdentifier,
"agentVersion", a.id.AgentVersion)
// Registration comes first: its response is the agent's identity.
if err := a.register(ctx); err != nil {
return err
}
a.hb.id, a.cmd.id, a.guard.id = a.id, a.id, a.id
// Ahead of the handoff, so rds-init never unblocks and no datadir is touched.
// An agent running on the wrong image would initialise one engine over
// another's data, coming up healthy and empty beside data nothing references.
if err := a.refuseEngineMismatch(ctx); err != nil {
return err
}
// Beating before the bootstrap keeps a stuck boot visible as a live VM with
// a down engine rather than as silence.
go a.hb.Run(ctx)
if err := a.bootstrap(ctx); err != nil {
return err
}
// The handoff must land before this wait: rds-datadir needs it to identify
// and authorize the volume. Commands and the parameter guard wait because
// their durable state belongs on that mount, never on the disposable boot disk.
if err := a.dataMountWaiter(ctx, a.cfg.MountsFile, a.cfg.DataMount); err != nil {
// A shutdown cancels this wait; main treats the wrapped context.Canceled
// as a clean stop, exactly as it does for register and bootstrap above.
return fmt.Errorf("wait for data mount %s: %w", a.cfg.DataMount, err)
}
// The receipt it waits on lives on that mount, so this cannot run any earlier.
// A cancelled wait is a clean stop here too.
if err := a.acknowledgeBootstrap(ctx); err != nil {
return err
}
// Directives are only meaningful against a bootstrapped, mounted engine.
go a.cmd.Run(ctx)
// Started after the handoff, so the window it measures covers the engine's
// own start rather than the time rds-init spent waiting for this agent.
go a.guard.Run(ctx)
<-ctx.Done()
return nil
}
// The engine cloud-init says this VM was launched as, against the engine the
// image bakes. An empty expectation is no assertion rather than a disagreement:
// a VM launched before the control plane emitted RDS_ENGINE carries none.
func (a *Agent) refuseEngineMismatch(ctx context.Context) error {
expected := strings.ToLower(strings.TrimSpace(a.cfg.Engine))
if expected == "" || expected == a.cfg.BakedEngine {
return nil
}
message := fmt.Sprintf("this VM was launched as engine %q but its image bakes %q; refusing to bootstrap",
expected, a.cfg.BakedEngine)
slog.ErrorContext(ctx, "rds-agent: "+message)
// Reported before returning, so the control plane learns why the instance
// never came up rather than only that it never did.
if _, err := a.cp.SubmitState(ctx, a.id, handlers_rds.EngineHealthUnhealthy, message); err != nil {
slog.ErrorContext(ctx, "rds-agent: reporting the engine mismatch failed", "err", err)
}
return errors.New(message)
}
const dataMountPollInterval = time.Second
// waitForDataMount blocks the stateful agent loops until the kernel reports the
// configured data mount. Registration, heartbeat, and bootstrap run before it.
func waitForDataMount(ctx context.Context, mountsFile, target string) error {
ticker := time.NewTicker(dataMountPollInterval)
defer ticker.Stop()
for {
mounted, err := mountTableContains(mountsFile, target)
if err != nil {
slog.WarnContext(ctx, "rds-agent: could not read mount table while waiting for data volume",
"mountsFile", mountsFile, "dataMount", target, "err", err)
} else if mounted {
slog.InfoContext(ctx, "rds-agent: data volume mount is ready", "dataMount", target)
return nil
}
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
}
}
}
func mountTableContains(path, target string) (bool, error) {
f, err := os.Open(path)
if err != nil {
return false, err
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) >= 2 && unescapeMountField(fields[1]) == target {
return true, nil
}
}
if err := scanner.Err(); err != nil {
return false, err
}
return false, nil
}
// Linux mount tables encode whitespace and backslashes as octal escapes.
func unescapeMountField(field string) string {
replacer := strings.NewReplacer(`\134`, `\`, `\040`, " ", `\011`, "\t", `\012`, "\n")
return replacer.Replace(field)
}
// Boot-retry bounds. Tight to start, since the usual cause is a control plane
// still creating this instance's record, and capped so an outage does not turn
// a booting fleet into a retry storm. Vars so tests can shorten the wait.
var (
retryMin = 1 * time.Second
retryMax = 30 * time.Second
)
// With the production backoff, attempt five is about 15 seconds into the
// failure and still precedes the first 30-second heartbeat.
const retryErrorAttempt = 5
// Wrapping an error in this stops retryObserved. For the failures the control
// plane decides from its own record, no retry can change the answer, and looping
// on one only keeps the caller blocked.
var errRetryTerminal = errors.New("terminal failure")
// Never gives up on its own: an agent that stopped trying would leave the
// control plane with no signal at all.
func retry(ctx context.Context, what string, fn func(context.Context) error) error {
return retryObserved(ctx, what, fn, nil)
}
// onFailure lets boot-critical callers publish the latest error while the retry
// remains blocked. It must not block because it runs on the retry path itself.
func retryObserved(ctx context.Context, what string, fn func(context.Context) error, onFailure func(error)) error {
delay := retryMin
for attempt := 1; ; attempt++ {
err := fn(ctx)
if err == nil {
return nil
}
if ctx.Err() != nil {
return fmt.Errorf("%s: %w", what, ctx.Err())
}
if onFailure != nil {
onFailure(err)
}
if errors.Is(err, errRetryTerminal) {
slog.ErrorContext(ctx, "rds-agent: "+what+" failed terminally, not retrying", "err", err)
return fmt.Errorf("%s: %w", what, err)
}
if attempt >= retryErrorAttempt {
slog.ErrorContext(ctx, "rds-agent: "+what+" still failing, retrying",
"attempt", attempt, "retry_in_ms", otelsetup.Millis(delay), "err", err)
} else {
slog.WarnContext(ctx, "rds-agent: "+what+" failed, retrying",
"attempt", attempt, "retry_in_ms", otelsetup.Millis(delay), "err", err)
}
select {
case <-ctx.Done():
return fmt.Errorf("%s: %w", what, ctx.Err())
case <-time.After(delay):
}
delay = min(delay*2, retryMax)
}
}
-873
View File
@@ -1,873 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/mulgadc/spinifex/internal/gwsign"
"github.com/mulgadc/spinifex/internal/rdsgw"
"github.com/mulgadc/spinifex/spinifex/awserrors"
gateway_rds "github.com/mulgadc/spinifex/spinifex/gateway/rds"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
"github.com/mulgadc/spinifex/spinifex/utils"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeControlPlane is a scriptable controlPlane. Every call records what the
// agent sent, so a test asserts on the request as well as the reaction.
type fakeControlPlane struct {
mu sync.Mutex
registerOut *handlers_rds.RegisterDBInstanceOutput
registerErrs []error
registerReqs []identity
bootstrapOut *handlers_rds.GetDBBootstrapConfigOutput
bootstrapErrs []error
bootstrapReqs []identity
// Scripted responses ahead of bootstrapOut, so a test can answer the first
// fetch differently from the ones the agent retries with.
bootstrapQueue []*handlers_rds.GetDBBootstrapConfigOutput
ackErrs []error
ackReqs []pendingBootstrap
states []submittedState
pollReplies [][]handlers_rds.CommandReply
pollQueue [][]handlers_rds.Command
pollErr error
polled chan struct{}
}
type submittedState struct {
id identity
health handlers_rds.EngineHealth
message string
}
var _ controlPlane = (*fakeControlPlane)(nil)
func newFakeControlPlane() *fakeControlPlane {
return &fakeControlPlane{
registerOut: &handlers_rds.RegisterDBInstanceOutput{
DBInstanceIdentifier: "db-resolved", HeartbeatIntervalSeconds: 30,
},
polled: make(chan struct{}, 64),
}
}
// nextErr pops the next scripted error, so a test can fail a call n times and
// then let it succeed.
func nextErr(errs *[]error) error {
if len(*errs) == 0 {
return nil
}
err := (*errs)[0]
*errs = (*errs)[1:]
return err
}
func (f *fakeControlPlane) Register(_ context.Context, id identity) (*handlers_rds.RegisterDBInstanceOutput, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.registerReqs = append(f.registerReqs, id)
if err := nextErr(&f.registerErrs); err != nil {
return nil, err
}
return f.registerOut, nil
}
func (f *fakeControlPlane) SubmitState(_ context.Context, id identity, health handlers_rds.EngineHealth, message string) (*handlers_rds.SubmitDBStateChangeOutput, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.states = append(f.states, submittedState{id: id, health: health, message: message})
// Zero means "keep the cadence you have", so a test's fast interval is not
// reset by every beat.
return &handlers_rds.SubmitDBStateChangeOutput{Acknowledged: true}, nil
}
func (f *fakeControlPlane) GetBootstrapConfig(_ context.Context, id identity) (*handlers_rds.GetDBBootstrapConfigOutput, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.bootstrapReqs = append(f.bootstrapReqs, id)
if err := nextErr(&f.bootstrapErrs); err != nil {
return nil, err
}
if len(f.bootstrapQueue) > 0 {
out := f.bootstrapQueue[0]
f.bootstrapQueue = f.bootstrapQueue[1:]
return out, nil
}
return f.bootstrapOut, nil
}
func (f *fakeControlPlane) AcknowledgeBootstrap(_ context.Context, _ identity, pending *pendingBootstrap) error {
f.mu.Lock()
defer f.mu.Unlock()
f.ackReqs = append(f.ackReqs, *pending)
return nextErr(&f.ackErrs)
}
func (f *fakeControlPlane) PollCommands(ctx context.Context, _ identity, replies []handlers_rds.CommandReply, _ time.Duration) ([]handlers_rds.Command, error) {
f.mu.Lock()
f.pollReplies = append(f.pollReplies, slices.Clone(replies))
err := f.pollErr
var out []handlers_rds.Command
if len(f.pollQueue) > 0 {
out, f.pollQueue = f.pollQueue[0], f.pollQueue[1:]
}
f.mu.Unlock()
select {
case f.polled <- struct{}{}:
default:
}
if err != nil {
return nil, err
}
if len(out) > 0 {
return out, nil
}
// An idle window: block until the agent hangs up, as the gateway's long poll
// does, rather than spinning the caller's loop.
<-ctx.Done()
return nil, ctx.Err()
}
func (f *fakeControlPlane) ackRequests() []pendingBootstrap {
f.mu.Lock()
defer f.mu.Unlock()
return slices.Clone(f.ackReqs)
}
func (f *fakeControlPlane) snapshotStates() []submittedState {
f.mu.Lock()
defer f.mu.Unlock()
return append([]submittedState(nil), f.states...)
}
func (f *fakeControlPlane) snapshotReplies() [][]handlers_rds.CommandReply {
f.mu.Lock()
defer f.mu.Unlock()
out := make([][]handlers_rds.CommandReply, len(f.pollReplies))
for i := range f.pollReplies {
out[i] = slices.Clone(f.pollReplies[i])
}
return out
}
// testConfig stamps the engine a real rds-postgres image bakes, so the agent
// resolves the same guest layout it would in a guest, and points its file output
// at a temp dir.
func testConfig(t *testing.T) config {
t.Helper()
cfg := testLoadConfig(t, enginePostgres)
cfg.GatewayURL = "https://gw.test"
cfg.HandoffDir = filepath.Join(t.TempDir(), "spinifex-rds")
cfg.PollWait = time.Second
return cfg
}
// newTestAgent builds the agent over a probe the test drives directly.
func newTestAgent(t *testing.T, cfg config, cp controlPlane, run probeRunner) *Agent {
t.Helper()
a, err := newAgent(cfg, cp, newPostgresProbe(cfg, run))
if err != nil {
t.Fatalf("newAgent: %v", err)
}
return a
}
// staticProbe is a probeRunner that always reports the same exit code.
func staticProbe(code int) probeRunner {
return func(context.Context, string, ...string) (int, string, error) { return code, "", nil }
}
func bootstrapOutput(password *string) *handlers_rds.GetDBBootstrapConfigOutput {
mode := handlers_rds.BootstrapModeAttach
if password != nil {
mode = handlers_rds.BootstrapModeInitialize
}
return &handlers_rds.GetDBBootstrapConfigOutput{
Mode: mode,
DBInstanceIdentifier: "db-resolved",
Engine: "postgres",
MasterUsername: "master",
MasterUserPassword: password,
Port: 5432,
ListenAddress: testListenAddress,
ClientCIDR: testClientCIDR,
DataVolumeID: "vol-data-01",
DataVolumeSerial: "voldata01",
VMGeneration: 1,
PayloadID: testPayloadID,
BootstrapPending: password != nil,
}
}
const testPayloadID = "bp-0123456789abcdef"
// The customer ENI address the engine binds, and the VPC range its client
// authentication admits. writeHandoff refuses a config without either.
const (
testListenAddress = "10.0.1.20"
testClientCIDR = "10.0.0.0/16"
)
// Writes the completion receipt rds-init leaves behind, so the acknowledgement
// step finds what a finished initialization would have produced.
func writeReceipt(t *testing.T, a *Agent, payloadID, dbInstanceIdentifier string) {
t.Helper()
path := a.receiptPath()
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
t.Fatalf("mkdir receipt dir: %v", err)
}
body := fmt.Sprintf("%s=%s\n%s=%s\n", receiptPayload, payloadID, receiptDBID, dbInstanceIdentifier)
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatalf("write receipt: %v", err)
}
}
// runAgent runs the agent until its bootstrap has landed, then cancels. It
// returns once Run has exited so no goroutine outlives the test.
func runAgent(t *testing.T, a *Agent) error {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
waitFor(t, func() bool {
_, err := os.Stat(filepath.Join(a.cfg.HandoffDir, handoffEnvFile))
return err == nil
}, "handoff file to be written")
cancel()
select {
case err := <-done:
// The cancellation above is this helper's own, so Run reporting it is the
// expected clean stop rather than a failure, exactly as main treats it.
if errors.Is(err, context.Canceled) {
return nil
}
return err
case <-time.After(5 * time.Second):
t.Fatal("Run did not exit after cancellation")
return nil
}
}
// waitFor polls cond until it holds or the test times out.
func waitFor(t *testing.T, cond func() bool, what string) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if cond() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("timed out waiting for %s", what)
}
func TestRun_RegistersThenDeliversBootstrap(t *testing.T) {
password := "s3cret"
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(&password)
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(0))
if err := runAgent(t, a); err != nil {
t.Fatalf("Run: %v", err)
}
if len(cp.registerReqs) != 1 {
t.Fatalf("registered %d times, want 1", len(cp.registerReqs))
}
// The agent had no configured identifier, so the gateway's resolution is
// what every later call must carry.
if got := cp.bootstrapReqs[0].DBInstanceIdentifier; got != "db-resolved" {
t.Errorf("bootstrap sent identifier %q, want the registered db-resolved", got)
}
env := readFile(t, filepath.Join(cfg.HandoffDir, handoffEnvFile))
if !strings.Contains(env, "RDS_MODE='initialize'") || !strings.Contains(env, "RDS_MASTER_PASSWORD='s3cret'") {
t.Errorf("handoff env = %q, want initialize mode carrying the password", env)
}
// The probe follows the port the control plane assigned, not the default.
if got := a.probe.port.Load(); got != 5432 {
t.Errorf("probe port = %d, want 5432", got)
}
}
// A reboot re-fetches in attach mode. The agent must write that payload as it
// stands rather than refusing it or inventing a password.
func TestRun_AttachModeNeedsNoPassword(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
env := readFile(t, filepath.Join(cfg.HandoffDir, handoffEnvFile))
if !strings.Contains(env, "RDS_MODE='attach'") {
t.Errorf("handoff env = %q, want attach mode", env)
}
if strings.Contains(env, "RDS_MASTER_PASSWORD") {
t.Errorf("handoff env = %q, want no password key at all in attach mode", env)
}
}
// The record may not exist yet when a freshly launched VM first calls, so both
// boot-critical calls have to ride out failures instead of exiting.
func TestRun_RetriesRegisterAndBootstrap(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cp.registerErrs = []error{errors.New("DBInstanceNotFound")}
cp.bootstrapErrs = []error{errors.New("ServerInternal")}
cfg := testConfig(t)
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
if len(cp.registerReqs) != 2 {
t.Errorf("register attempts = %d, want 2 (one failure, one success)", len(cp.registerReqs))
}
if len(cp.bootstrapReqs) != 2 {
t.Errorf("bootstrap attempts = %d, want 2 (one failure, one success)", len(cp.bootstrapReqs))
}
}
// The fetch mutates nothing, so a handoff that cannot be written is retried
// against the payload already in hand rather than costing a second fetch.
func TestBootstrap_RetriesHandoffWithoutRefetching(t *testing.T) {
cp := newFakeControlPlane()
password := "staged-secret"
cp.bootstrapOut = bootstrapOutput(&password)
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(0))
writes := 0
a.handoffWriter = func(_ string, got *handlers_rds.GetDBBootstrapConfigOutput) error {
writes++
if got.MasterUserPassword == nil || *got.MasterUserPassword != password {
t.Fatalf("handoff attempt %d lost the initialize password", writes)
}
if writes == 1 {
return errors.New("temporary tmpfs write error")
}
return nil
}
if err := a.bootstrap(t.Context()); err != nil {
t.Fatalf("bootstrap: %v", err)
}
if len(cp.bootstrapReqs) != 1 {
t.Errorf("bootstrap fetches = %d, want 1", len(cp.bootstrapReqs))
}
if writes != 2 {
t.Errorf("handoff writes = %d, want 2", writes)
}
if a.pending == nil || a.pending.payloadID != testPayloadID {
t.Errorf("pending bootstrap = %+v, want the staged payload %s", a.pending, testPayloadID)
}
}
// A daemon that predates the encrypted payload answers initialize with nothing
// to initialise with. Writing that handoff would let rds-init run initdb with an
// empty master password, so it is retried until an upgraded node answers.
func TestBootstrap_RetriesInitializeWithNoPassword(t *testing.T) {
cp := newFakeControlPlane()
empty := ""
password := "staged-secret"
cp.bootstrapQueue = []*handlers_rds.GetDBBootstrapConfigOutput{{
Mode: handlers_rds.BootstrapModeInitialize, DBInstanceIdentifier: "db-resolved",
MasterUsername: "master", MasterUserPassword: &empty,
}}
cp.bootstrapOut = bootstrapOutput(&password)
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(0))
var handed []*handlers_rds.GetDBBootstrapConfigOutput
a.handoffWriter = func(_ string, got *handlers_rds.GetDBBootstrapConfigOutput) error {
handed = append(handed, got)
return nil
}
if err := a.bootstrap(t.Context()); err != nil {
t.Fatalf("bootstrap: %v", err)
}
if len(handed) != 1 {
t.Fatalf("handoff writes = %d, want 1 once a usable payload arrived", len(handed))
}
if handed[0].MasterUserPassword == nil || *handed[0].MasterUserPassword != password {
t.Error("the agent wrote a handoff for a fetch that carried no master password")
}
}
func TestAcknowledgeBootstrap_WaitsForTheReceiptThenRetiresIt(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1, dataVolumeID: "vol-data-01"}
// rds-init runs after the agent registers, so the receipt is absent when the
// acknowledgement step first looks and appears some way into the boot.
if err := a.checkBootstrapReceipt(a.pending); err == nil {
t.Fatal("a missing receipt must not read as a completed bootstrap")
}
writeReceipt(t, a, testPayloadID, "db-resolved")
if err := a.acknowledgeBootstrap(t.Context()); err != nil {
t.Fatalf("acknowledgeBootstrap: %v", err)
}
if len(cp.ackRequests()) != 1 {
t.Fatalf("acknowledgements = %d, want 1", len(cp.ackRequests()))
}
if got := cp.ackRequests()[0]; got != *a.pending {
t.Errorf("acknowledged %+v, want %+v", got, *a.pending)
}
if _, err := os.Stat(a.receiptPath()); !os.IsNotExist(err) {
t.Error("the receipt must be removed once the record holds the audit trail")
}
}
// The receipt rides along in every snapshot of the data volume, so a restored
// instance's volume carries the source instance's receipt. Acknowledging on one
// would destroy the ciphertext this instance still needs.
func TestAcknowledgeBootstrap_IgnoresAForeignReceipt(t *testing.T) {
tests := map[string]struct{ payloadID, dbID string }{
"another payload": {payloadID: "bp-somebody-else", dbID: "db-resolved"},
"another instance": {payloadID: testPayloadID, dbID: "db-the-snapshot-source"},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
writeReceipt(t, a, tc.payloadID, tc.dbID)
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
defer cancel()
if err := a.acknowledgeBootstrap(ctx); err == nil {
t.Fatal("acknowledgeBootstrap returned nil, want it still waiting on a real receipt")
}
if n := len(cp.ackRequests()); n != 0 {
t.Errorf("acknowledgements = %d, want 0", n)
}
})
}
}
// An attach boot has nothing staged, so it must not go looking for a receipt the
// datadir will never grow.
func TestAcknowledgeBootstrap_IsANoOpOnAttach(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newTestAgent(t, cfg, cp, staticProbe(0))
if err := a.acknowledgeBootstrap(t.Context()); err != nil {
t.Fatalf("acknowledgeBootstrap: %v", err)
}
if n := len(cp.ackRequests()); n != 0 {
t.Errorf("acknowledgements = %d, want 0 when nothing was staged", n)
}
}
// The image is the authority on which engine the agent runs. A VM launched as
// another engine has to stop ahead of the handoff: rds-init would otherwise
// initialise one engine over the other's datadir and come up healthy and empty
// beside data nothing references.
func TestRun_RefusesAnEngineTheImageDoesNotBake(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
cfg.Engine = "mariadb"
a := newTestAgent(t, cfg, cp, staticProbe(0))
err := a.Run(t.Context())
if err == nil {
t.Fatal("Run returned nil for a VM launched as an engine this image does not bake")
}
if !strings.Contains(err.Error(), "mariadb") {
t.Errorf("error = %v, want it to name the engine the VM was launched as", err)
}
if _, statErr := os.Stat(filepath.Join(cfg.HandoffDir, handoffEnvFile)); !os.IsNotExist(statErr) {
t.Error("the agent wrote a bootstrap handoff for an engine it refused to run")
}
if n := len(cp.bootstrapReqs); n != 0 {
t.Errorf("bootstrap fetches = %d, want none before the refusal", n)
}
// Reported rather than only logged, so the control plane learns why the
// instance never came up rather than only that it never did.
states := cp.snapshotStates()
if len(states) != 1 || states[0].health != handlers_rds.EngineHealthUnhealthy {
t.Fatalf("submitted states = %+v, want a single unhealthy report", states)
}
}
// The same assertion when it agrees is not a gate: the boot goes on exactly as
// it does for a VM launched before the control plane sent one at all.
func TestRun_ProceedsWhenTheDeliveredEngineMatchesTheImage(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
cfg.Engine = enginePostgres
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
}
// A configured identifier is asserted on the wire so a mis-provisioned VM is
// rejected by the gateway instead of adopting whatever it resolves to.
func TestRun_SendsConfiguredIdentifier(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
cfg.DBInstanceIdentifier = "db-configured"
cfg.EngineVersion = "18.1"
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
sent := cp.registerReqs[0]
if sent.DBInstanceIdentifier != "db-configured" || sent.EngineVersion != "18.1" {
t.Errorf("register sent %+v, want the configured identifier and engine version", sent)
}
}
func TestRun_WaitsForDataMountBeforePollingCommands(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(2))
waiting := make(chan struct{})
release := make(chan struct{})
a.dataMountWaiter = func(ctx context.Context, mountsFile, target string) error {
if mountsFile != defaultMountsFile || target != cfg.DataMount {
t.Errorf("mount waiter got %q/%q, want configured defaults", mountsFile, target)
}
close(waiting)
select {
case <-ctx.Done():
return ctx.Err()
case <-release:
return nil
}
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
select {
case <-waiting:
case <-ctx.Done():
t.Fatal("agent did not reach the data-mount wait after writing the handoff")
}
if _, err := os.Stat(filepath.Join(cfg.HandoffDir, handoffEnvFile)); err != nil {
t.Fatalf("handoff was not present before mount wait: %v", err)
}
select {
case <-cp.polled:
t.Fatal("command poll started before the data volume was mounted")
case <-time.After(50 * time.Millisecond):
}
close(release)
select {
case <-cp.polled:
case <-ctx.Done():
t.Fatal("command poll did not start after the data mount became ready")
}
cancel()
if err := <-done; err != nil {
t.Fatalf("Run: %v", err)
}
}
func TestMountTableContains(t *testing.T) {
mounts := filepath.Join(t.TempDir(), "mounts")
require.NoError(t, os.WriteFile(mounts, []byte(
"/dev/vda1 / ext4 rw 0 0\n/dev/vdb /var/lib/postgresql ext4 rw 0 0\n"+
"/dev/vdc /path\\040with\\040spaces xfs rw 0 0\nmalformed\n"), 0o600))
tests := []struct {
target string
want bool
}{
{target: "/var/lib/postgresql", want: true},
{target: "/path with spaces", want: true},
{target: "/missing", want: false},
}
for _, tc := range tests {
t.Run(tc.target, func(t *testing.T) {
got, err := mountTableContains(mounts, tc.target)
require.NoError(t, err)
assert.Equal(t, tc.want, got)
})
}
}
func TestRun_HeartbeatIncludesBootstrapFailure(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapErrs = []error{errors.New("403 AccessDenied: instance profile is not authorised")}
cp.registerOut.HeartbeatIntervalSeconds = 0
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(2))
a.hb.interval = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
waitFor(t, func() bool {
for _, state := range cp.snapshotStates() {
if strings.Contains(state.message, "bootstrap fetch failed") &&
strings.Contains(state.message, "403 AccessDenied") {
return true
}
}
return false
}, "the bootstrap failure to reach a heartbeat")
cancel()
<-done
}
func TestRun_HeartbeatsEngineHealth(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
// Declining to set a cadence leaves the agent's own, which the test shortens
// rather than waiting out the 30s production interval.
cp.registerOut.HeartbeatIntervalSeconds = 0
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.hb.interval = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
waitFor(t, func() bool { return len(cp.snapshotStates()) > 0 }, "a heartbeat")
cancel()
<-done
beat := cp.snapshotStates()[0]
if beat.health != handlers_rds.EngineHealthHealthy {
t.Errorf("heartbeat health = %q, want healthy", beat.health)
}
if beat.id.DBInstanceIdentifier != "db-resolved" {
t.Errorf("heartbeat identifier = %q, want db-resolved", beat.id.DBInstanceIdentifier)
}
}
// The cadence is the control plane's to set: it feeds the staleness window the
// recovery reconciler judges an instance against, so the guest adopts it rather
// than keeping its own.
func TestRegister_AdoptsControlPlaneCadence(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cp.registerOut.HeartbeatIntervalSeconds = 45
cfg := testConfig(t)
a := newTestAgent(t, cfg, cp, staticProbe(0))
if err := runAgent(t, a); err != nil {
t.Fatalf("Run: %v", err)
}
if a.hb.interval != 45*time.Second {
t.Errorf("heartbeat interval = %v, want the 45s the control plane returned", a.hb.interval)
}
}
// The agent being up says nothing about the engine. A down engine has to reach
// the control plane as such, or auto-recovery has nothing to act on.
func TestRun_HeartbeatReportsDownEngine(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cp.registerOut.HeartbeatIntervalSeconds = 0
cfg := testConfig(t)
probe := newPostgresProbe(cfg, staticProbe(2))
// The engine has served before, so a silent one now is a failure rather
// than a boot still in progress.
probe.seenHealthy = true
a, err := newAgent(cfg, cp, probe)
if err != nil {
t.Fatalf("newAgent: %v", err)
}
a.hb.interval = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan error, 1)
go func() { done <- a.Run(ctx) }()
waitFor(t, func() bool { return len(cp.snapshotStates()) > 0 }, "a heartbeat")
cancel()
<-done
beat := cp.snapshotStates()[0]
if beat.health != handlers_rds.EngineHealthUnhealthy {
t.Errorf("heartbeat health = %q, want unhealthy while the agent is up", beat.health)
}
if beat.message == "" {
t.Error("unhealthy heartbeat carried no message explaining the probe result")
}
}
// The mirror the agent decodes poll responses into has to stay aligned with the
// gateway type that produces them; nothing else in the build ties the two.
func TestPollDBCommandsOutput_MatchesGatewayShape(t *testing.T) {
issued := time.Date(2026, 7, 28, 1, 2, 3, 0, time.UTC)
// Rendered exactly as the gateway's typed adapter renders it, so a change to
// either side's tags fails here.
payload := utils.GenerateIAMXMLPayload("PollDBCommands", &gateway_rds.PollDBCommandsOutput{
Commands: []handlers_rds.Command{{
CommandID: "cmd-1",
Type: "reload-parameters",
Parameters: []handlers_rds.Parameter{{Name: "work_mem", Value: "8MB"}},
IssuedAt: &issued,
}},
})
body, err := utils.MarshalToXML(payload)
if err != nil {
t.Fatalf("marshal poll result: %v", err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/xml")
_, _ = w.Write(body)
}))
defer srv.Close()
client, err := rdsgw.New(srv.URL, "", gwsign.NewStatic("AKIATEST", "secret"), "us-east-1", time.Second)
if err != nil {
t.Fatalf("rdsgw.New: %v", err)
}
var got pollDBCommandsOutput
if err := client.Call(context.Background(), "PollDBCommands", nil, &got); err != nil {
t.Fatalf("Call: %v", err)
}
if len(got.Commands) != 1 {
t.Fatalf("decoded %d commands, want 1", len(got.Commands))
}
cmd := got.Commands[0]
if cmd.CommandID != "cmd-1" || cmd.Type != "reload-parameters" {
t.Errorf("command = %+v, want cmd-1/reload-parameters", cmd)
}
if len(cmd.Parameters) != 1 || cmd.Parameters[0].Value != "8MB" {
t.Errorf("Parameters = %+v, want the seeded member", cmd.Parameters)
}
if cmd.IssuedAt == nil || !cmd.IssuedAt.Equal(issued) {
t.Errorf("IssuedAt = %v, want %v", cmd.IssuedAt, issued)
}
}
func readFile(t *testing.T, path string) string {
t.Helper()
body, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
return string(body)
}
// AccessDenied is decided by the control plane's own record, so no retry can
// change it. Looping on one would leave the command poller and the parameter
// guard unstarted for the life of a VM whose engine is serving normally.
func TestAcknowledgeBootstrap_StopsOnATerminalDenial(t *testing.T) {
cp := newFakeControlPlane()
cp.ackErrs = []error{&rdsgw.APIError{
Action: "AcknowledgeDBBootstrap", StatusCode: 403,
Code: awserrors.ErrorAccessDenied,
Message: "bootstrap payload bp-x at generation 1 does not match the current state",
}}
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
writeReceipt(t, a, testPayloadID, "db-resolved")
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
if err := a.acknowledgeBootstrap(ctx); err != nil {
t.Fatalf("acknowledgeBootstrap = %v, want nil so the stateful loops still start", err)
}
if n := len(cp.ackRequests()); n != 1 {
t.Errorf("acknowledgements = %d, want 1 attempt and no retry", n)
}
if a.hb.bootstrapFailure.Load() == nil {
t.Error("a terminal denial must still reach the operator through the heartbeat")
}
if _, err := os.Stat(a.receiptPath()); err != nil {
t.Error("a denied acknowledgement must leave the receipt for a later attempt")
}
}
// rds-init runs after this agent, so the receipt is absent for the whole initdb
// window. Reporting that would put a failure on every healthy create's beat, and
// then into the reason a create that timed out for some other cause records.
func TestAcknowledgeBootstrap_DoesNotReportAnAbsentReceiptAsAFailure(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
defer cancel()
if err := a.acknowledgeBootstrap(ctx); err == nil {
t.Fatal("acknowledgeBootstrap returned nil, want it still waiting for rds-init")
}
if got := a.hb.bootstrapFailure.Load(); got != nil {
t.Errorf("bootstrap failure = %q, want none while the receipt has simply not been written", *got)
}
// A receipt that exists but names another instance is a real mismatch and is
// reported, which is what separates the two cases.
writeReceipt(t, a, testPayloadID, "db-the-snapshot-source")
mismatch, cancelMismatch := context.WithTimeout(t.Context(), 50*time.Millisecond)
defer cancelMismatch()
if err := a.acknowledgeBootstrap(mismatch); err == nil {
t.Fatal("a foreign receipt must not be acknowledged")
}
if a.hb.bootstrapFailure.Load() == nil {
t.Error("a receipt naming another instance must be reported")
}
}
-237
View File
@@ -1,237 +0,0 @@
package main
import (
"context"
"fmt"
"log/slog"
"os"
"path/filepath"
"strconv"
"strings"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// The staged payload this boot has to acknowledge once PostgreSQL has applied
// it. Nil on an attach boot, which makes the acknowledgement step a no-op.
type pendingBootstrap struct {
payloadID string
vmGeneration int64
dataVolumeID string
}
// Fetches the boot material and writes the handoff rds-init is blocked waiting
// for, then points the health probe at the assigned port.
func (a *Agent) bootstrap(ctx context.Context) error {
var cfg *handlers_rds.GetDBBootstrapConfigOutput
if err := retryObserved(ctx, "bootstrap fetch", func(ctx context.Context) error {
fetched, err := a.cp.GetBootstrapConfig(ctx, a.id)
if err != nil {
return err
}
if fetched == nil {
return fmt.Errorf("bootstrap fetch returned no config")
}
// A control plane predating the encrypted payload answers initialize with
// nothing to initialise with. Retrying re-dials the queue group until an
// upgraded node answers, so a mixed-version fleet heals within one boot.
if fetched.Mode == handlers_rds.BootstrapModeInitialize &&
(fetched.MasterUserPassword == nil || *fetched.MasterUserPassword == "") {
return fmt.Errorf("bootstrap fetch returned mode=%s with no master password", fetched.Mode)
}
cfg = fetched
return nil
}, func(err error) {
a.hb.setBootstrapFailure("bootstrap fetch", err)
}); err != nil {
return err
}
a.hb.clearBootstrapFailure()
// The fetch mutates nothing, so a handoff that cannot be written is retried
// against the same staged payload rather than against an attach response.
if err := retryObserved(ctx, "bootstrap handoff", func(context.Context) error {
return a.handoffWriter(a.cfg.HandoffDir, cfg)
}, func(err error) {
a.hb.setBootstrapFailure("bootstrap handoff", err)
}); err != nil {
return err
}
a.hb.clearBootstrapFailure()
if cfg.Port > 0 {
a.probe.setPort(int(cfg.Port))
}
if cfg.BootstrapPending && cfg.PayloadID != "" {
a.pending = &pendingBootstrap{
payloadID: cfg.PayloadID,
vmGeneration: cfg.VMGeneration,
dataVolumeID: cfg.DataVolumeID,
}
}
// Mode is logged but never branched on: rds-init decides whether to initdb
// from the state of the datadir.
slog.Info("rds-agent: bootstrap config delivered",
"mode", cfg.Mode, "port", cfg.Port, "parameters", len(cfg.Parameters),
"bootstrapPending", cfg.BootstrapPending)
return nil
}
// The handoff files rds-init reads. They live on tmpfs, so nothing survives a
// reboot and the next boot re-fetches rather than reusing a stale password or a
// re-minted cert.
const (
handoffEnvFile = "bootstrap.env"
handoffParamsFile = "parameters.conf"
handoffCertFile = "server.crt"
handoffKeyFile = "server.key"
// Root-only: only rds-init reads these, and one holds the master password.
handoffMode = 0o600
handoffDirMode = 0o700
)
// bootstrap.env is written last and renamed into place, so its appearance —
// which rds-init waits on — means the whole handoff is complete.
func writeHandoff(dir string, cfg *handlers_rds.GetDBBootstrapConfigOutput) error {
if cfg.MasterUsername == "" {
return fmt.Errorf("bootstrap config carries no master username")
}
// Half a handoff would leave rds-init deciding what to bind and who to admit,
// and the only answers it could reach on its own are the wildcards this
// contract exists to remove.
if cfg.ListenAddress == "" {
return fmt.Errorf("bootstrap config carries no listen address for the engine to bind")
}
if cfg.ClientCIDR == "" {
return fmt.Errorf("bootstrap config carries no client CIDR to scope client authentication to")
}
if err := os.MkdirAll(dir, handoffDirMode); err != nil {
return fmt.Errorf("create handoff dir %s: %w", dir, err)
}
// MkdirAll leaves an existing directory's mode alone, and this one may have
// been created more permissively earlier in the boot.
if err := os.Chmod(dir, handoffDirMode); err != nil {
return fmt.Errorf("secure handoff dir %s: %w", dir, err)
}
params, err := renderParameters(cfg.Engine, cfg.Parameters)
if err != nil {
return err
}
if err := writeHandoffFile(dir, handoffParamsFile, params); err != nil {
return err
}
// Cert and key are written as a pair; half of one would start the engine
// with TLS configured against a key it does not have.
if cfg.ServingCertificate != "" && cfg.ServingPrivateKey != "" {
if err := writeHandoffFile(dir, handoffCertFile, cfg.ServingCertificate); err != nil {
return err
}
if err := writeHandoffFile(dir, handoffKeyFile, cfg.ServingPrivateKey); err != nil {
return err
}
}
return writeHandoffFile(dir, handoffEnvFile, renderBootstrapEnv(cfg))
}
// Every value is single-quoted because rds-init sources this with `.`: an
// unquoted password holding a space, `$` or `;` would be word-split or executed.
func renderBootstrapEnv(cfg *handlers_rds.GetDBBootstrapConfigOutput) string {
var b strings.Builder
b.WriteString("# Written by rds-agent. Regenerated on every boot; edits are lost.\n")
writeEnvLine(&b, "RDS_MODE", cfg.Mode)
writeEnvLine(&b, "RDS_DB_INSTANCE_IDENTIFIER", cfg.DBInstanceIdentifier)
writeEnvLine(&b, "RDS_MASTER_USERNAME", cfg.MasterUsername)
writeEnvLine(&b, "RDS_LISTEN_ADDRESS", cfg.ListenAddress)
writeEnvLine(&b, "RDS_CLIENT_CIDR", cfg.ClientCIDR)
writeEnvLine(&b, "RDS_DATA_VOLUME_ID", cfg.DataVolumeID)
writeEnvLine(&b, "RDS_DATA_VOLUME_SERIAL", cfg.DataVolumeSerial)
writeEnvLine(&b, "RDS_VM_GENERATION", strconv.FormatInt(cfg.VMGeneration, 10))
writeEnvLine(&b, "RDS_FORMAT_AUTHORIZED", strconv.FormatBool(cfg.FormatAuthorized))
// Neither needs volume access, which is why the assertion is made here: the
// agent runs before rds-datadir mounts the volume, so rds-init is the first
// component that can compare it against the completion receipt on disk.
writeEnvLine(&b, "RDS_BOOTSTRAP_PENDING", boolFlag(cfg.BootstrapPending))
writeEnvLine(&b, "RDS_PAYLOAD_ID", cfg.PayloadID)
// Present only in initialize mode: an attach fetch has no password to write
// and rds-init must not find a stale one.
if cfg.MasterUserPassword != nil {
writeEnvLine(&b, "RDS_MASTER_PASSWORD", *cfg.MasterUserPassword)
}
if cfg.DBName != "" {
writeEnvLine(&b, "RDS_DB_NAME", cfg.DBName)
}
if cfg.Port > 0 {
writeEnvLine(&b, "RDS_PORT", strconv.FormatInt(cfg.Port, 10))
}
return b.String()
}
// rds-init compares against 1 rather than sourcing a Go boolean, so the two
// halves of the guard cannot drift on spelling.
func boolFlag(v bool) string {
if v {
return "1"
}
return "0"
}
func writeEnvLine(b *strings.Builder, key, value string) {
b.WriteString(key)
b.WriteString("=")
b.WriteString(shellQuote(value))
b.WriteString("\n")
}
// Single quotes suppress every shell expansion; an embedded quote is closed,
// escaped and reopened.
func shellQuote(s string) string {
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
}
// postgresql.conf syntax, which MariaDB's option files also parse. Values are
// quoted so a space or unit suffix survives. Names are the engine's startup
// spellings, not the customer's: the wrong one leaves the engine unable to boot.
func renderParameters(engineName string, params []handlers_rds.Parameter) (string, error) {
engine, err := handlers_rds.LookupEngine(engineName)
if err != nil {
return "", fmt.Errorf("render the resolved parameters: %w", err)
}
var b strings.Builder
b.WriteString("# Resolved parameter group, written by rds-agent.\n")
for _, p := range params {
if p.Name == "" {
continue
}
fmt.Fprintf(&b, "%s = '%s'\n", engine.OptionFileName(p.Name), strings.ReplaceAll(p.Value, "'", "''"))
}
return b.String(), nil
}
// Atomic: a temp file in the same directory, renamed over the target. The mode
// is set at creation, so content is never briefly readable at the process umask.
func writeHandoffFile(dir, name, content string) error {
path := filepath.Join(dir, name)
tmp := path + ".new"
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, handoffMode)
if err != nil {
return fmt.Errorf("create %s: %w", tmp, err)
}
// A failed write must not leave the temp file behind holding a password.
if _, err := f.WriteString(content); err != nil {
f.Close()
os.Remove(tmp)
return fmt.Errorf("write %s: %w", tmp, err)
}
if err := f.Close(); err != nil {
os.Remove(tmp)
return fmt.Errorf("close %s: %w", tmp, err)
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("install %s: %w", path, err)
}
return nil
}
-280
View File
@@ -1,280 +0,0 @@
package main
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
func TestWriteHandoff_WritesEveryFileRootOnly(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
password := "s3cret"
err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeInitialize,
Engine: "postgres",
MasterUsername: "master",
ListenAddress: "10.0.1.20",
ClientCIDR: "10.0.0.0/16",
MasterUserPassword: &password,
DBName: "appdb",
Port: 5433,
DataVolumeID: "vol-data-01",
DataVolumeSerial: "voldata01",
VMGeneration: 4,
FormatAuthorized: true,
Parameters: []handlers_rds.Parameter{
{Name: "max_connections", Value: "200"},
{Name: "shared_buffers", Value: "256MB"},
},
ServingCertificate: "CERT",
ServingPrivateKey: "KEY",
})
if err != nil {
t.Fatalf("writeHandoff: %v", err)
}
env := readFile(t, filepath.Join(dir, handoffEnvFile))
for _, want := range []string{
"RDS_MODE='initialize'",
"RDS_MASTER_USERNAME='master'",
"RDS_MASTER_PASSWORD='s3cret'",
"RDS_DB_NAME='appdb'",
"RDS_PORT='5433'",
"RDS_DATA_VOLUME_ID='vol-data-01'",
"RDS_DATA_VOLUME_SERIAL='voldata01'",
"RDS_VM_GENERATION='4'",
"RDS_FORMAT_AUTHORIZED='true'",
} {
if !strings.Contains(env, want) {
t.Errorf("handoff env missing %s:\n%s", want, env)
}
}
params := readFile(t, filepath.Join(dir, handoffParamsFile))
if !strings.Contains(params, "max_connections = '200'") || !strings.Contains(params, "shared_buffers = '256MB'") {
t.Errorf("parameters.conf = %q, want both resolved settings", params)
}
if got := readFile(t, filepath.Join(dir, handoffCertFile)); got != "CERT" {
t.Errorf("server.crt = %q, want CERT", got)
}
if got := readFile(t, filepath.Join(dir, handoffKeyFile)); got != "KEY" {
t.Errorf("server.key = %q, want KEY", got)
}
// The handoff holds a master password and a private key; anything readable
// beyond root is a disclosure to every process in the guest.
if mode := statMode(t, dir); mode != handoffDirMode {
t.Errorf("handoff dir mode = %#o, want %#o", mode, handoffDirMode)
}
for _, name := range []string{handoffEnvFile, handoffParamsFile, handoffCertFile, handoffKeyFile} {
if mode := statMode(t, filepath.Join(dir, name)); mode != handoffMode {
t.Errorf("%s mode = %#o, want %#o", name, mode, handoffMode)
}
}
}
// Attach mode carries no password, and the key must be absent rather than
// present and empty — rds-init reads an empty value as "no password" only by
// accident, and a later boot would inherit it.
func TestWriteHandoff_AttachOmitsPassword(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
MasterUsername: "master",
ListenAddress: "10.0.1.20",
ClientCIDR: "10.0.0.0/16",
Port: 5432,
}); err != nil {
t.Fatalf("writeHandoff: %v", err)
}
env := readFile(t, filepath.Join(dir, handoffEnvFile))
if strings.Contains(env, "RDS_MASTER_PASSWORD") {
t.Errorf("handoff env = %q, want no RDS_MASTER_PASSWORD key", env)
}
for _, want := range []string{
"RDS_DATA_VOLUME_ID=''",
"RDS_DATA_VOLUME_SERIAL=''",
"RDS_VM_GENERATION='0'",
"RDS_FORMAT_AUTHORIZED='false'",
} {
if !strings.Contains(env, want) {
t.Errorf("handoff env missing fail-closed field %s: %q", want, env)
}
}
}
// The env file is sourced by a shell, so a password holding shell metacharacters
// must survive as data rather than being split, expanded or executed.
func TestWriteHandoff_QuotesShellMetacharacters(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
password := `a b$(id);'"\` + "`whoami`"
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeInitialize,
Engine: "postgres",
MasterUsername: "master",
ListenAddress: "10.0.1.20",
ClientCIDR: "10.0.0.0/16",
MasterUserPassword: &password,
Port: 5432,
}); err != nil {
t.Fatalf("writeHandoff: %v", err)
}
env := readFile(t, filepath.Join(dir, handoffEnvFile))
line := findLine(t, env, "RDS_MASTER_PASSWORD=")
if got := shellSource(t, line, "RDS_MASTER_PASSWORD"); got != password {
t.Errorf("sourced password = %q, want %q (line was %s)", got, password, line)
}
}
// A serving cert with no key would have rds-init configure TLS against a key
// that is not there; the pair is delivered together or not at all.
func TestWriteHandoff_SkipsHalfATLSPair(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
MasterUsername: "master",
ListenAddress: "10.0.1.20",
ClientCIDR: "10.0.0.0/16",
Port: 5432,
ServingCertificate: "CERT",
}); err != nil {
t.Fatalf("writeHandoff: %v", err)
}
if _, err := os.Stat(filepath.Join(dir, handoffCertFile)); !os.IsNotExist(err) {
t.Errorf("server.crt exists without a key; stat err = %v", err)
}
}
// A config with no master username cannot bootstrap anything, and writing it
// would only move the failure into rds-init after the boot has committed to it.
func TestWriteHandoff_RejectsConfigWithNoMasterUser(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
Port: 5432,
})
if err == nil {
t.Fatal("writeHandoff accepted a config with no master username, want an error")
}
if _, statErr := os.Stat(filepath.Join(dir, handoffEnvFile)); !os.IsNotExist(statErr) {
t.Errorf("handoff env was written despite the rejection; stat err = %v", statErr)
}
}
// A directory left over from an earlier boot at a looser mode must be tightened,
// not accepted as it is found.
func TestWriteHandoff_TightensExistingDirectory(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatalf("seed dir: %v", err)
}
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
MasterUsername: "master",
ListenAddress: "10.0.1.20",
ClientCIDR: "10.0.0.0/16",
Port: 5432,
}); err != nil {
t.Fatalf("writeHandoff: %v", err)
}
if mode := statMode(t, dir); mode != handoffDirMode {
t.Errorf("handoff dir mode = %#o, want it tightened to %#o", mode, handoffDirMode)
}
}
func TestShellQuote_EscapesEmbeddedQuotes(t *testing.T) {
for _, in := range []string{"", "plain", "it's", `''`, "new\nline", `$HOME`} {
if got := shellSource(t, "V="+shellQuote(in), "V"); got != in {
t.Errorf("shellQuote(%q) sourced back as %q", in, got)
}
}
}
// shellSource evaluates one KEY=value assignment in a real shell and returns
// what the variable ends up holding. The quoting is only correct if a shell
// agrees, so the test asks one instead of re-implementing its rules.
func shellSource(t *testing.T, line, name string) string {
t.Helper()
out, err := exec.Command("/bin/sh", "-c", line+"\nprintf %s \"$"+name+"\"\n").Output()
if err != nil {
t.Fatalf("sourcing %q: %v", line, err)
}
return string(out)
}
func statMode(t *testing.T, path string) os.FileMode {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Mode().Perm()
}
func findLine(t *testing.T, body, prefix string) string {
t.Helper()
for line := range strings.SplitSeq(body, "\n") {
if strings.HasPrefix(line, prefix) {
return line
}
}
t.Fatalf("no line starting %q in:\n%s", prefix, body)
return ""
}
// The include rds-init installs is what the engine starts on, so the spelling
// written here is the one that has to boot.
func TestRenderParameters_UsesTheEngineStartupSpelling(t *testing.T) {
params := []handlers_rds.Parameter{
{Name: "time_zone", Value: "SYSTEM"},
{Name: "max_connections", Value: "85"},
}
rendered, err := renderParameters("mariadb", params)
if err != nil {
t.Fatalf("renderParameters: %v", err)
}
if !strings.Contains(rendered, "default_time_zone = 'SYSTEM'\n") {
t.Errorf("time_zone was not written under its startup spelling:\n%s", rendered)
}
if strings.Contains(rendered, "\ntime_zone = ") {
t.Errorf("time_zone reached the option file, which mariadbd refuses:\n%s", rendered)
}
if !strings.Contains(rendered, "max_connections = '85'\n") {
t.Errorf("an unmapped parameter was not written unchanged:\n%s", rendered)
}
}
func TestRenderParameters_PostgresKeepsEveryName(t *testing.T) {
rendered, err := renderParameters("postgres", []handlers_rds.Parameter{
{Name: "shared_buffers", Value: "262144"},
})
if err != nil {
t.Fatalf("renderParameters: %v", err)
}
if !strings.Contains(rendered, "shared_buffers = '262144'\n") {
t.Errorf("postgres name was rewritten:\n%s", rendered)
}
}
// Failing closed beats writing a file the engine may refuse: the bad include
// would already be on the data volume by the time the server rejected it.
func TestRenderParameters_RefusesAnUnknownEngine(t *testing.T) {
if _, err := renderParameters("", nil); err == nil {
t.Fatal("renderParameters accepted a config naming no engine, want an error")
}
}

Some files were not shown because too many files have changed in this diff Show More