release: update go.mod for v1.13.0

* test: add in-process integration test tier

Roughly half of the behavioural tests under tests/e2e/ assert control-plane
API shapes, authz and error codes that need no live environment, yet they can
only run against a tofu-provisioned environment on a scarce self-hosted
runner. This adds the tier they can move down to.

The real gateway router runs in-process: SetupRoutes() is a pure function of
GatewayConfig, so it serves from an httptest server over embedded NATS
JetStream with real IAM/STS services and genuine SigV4 verification. Only the
daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node
would answer are stubbed per test.

Nothing is provisioned and no fixed ports are bound, so the tier runs
concurrently on a shared build host and is safe as a CI front gate ahead of
the live suites. It is build-tagged and kept out of preflight, which stays
unit-only.

* test(integration): add IAM/STS client accessors and port the zero-stub authz tests

Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a
test can act as a principal other than the seeded root user, and stubs the
node-discovery subject in StartGateway so call paths that trigger discovery
resolve immediately instead of riding out the fan-out timeout.

Ports the four enforcement tests that guard on ec2:DescribeRegions, a
gateway-local action, so they need no daemon-side stub at all.

* test(integration): port remaining IAM/STS authz tests from e2e

Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication,
TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle,
TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and
TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process
integration tier. Each test now self-bootstraps its own users/policies/roles
instead of relying on shared suite-level state, since every integration test
gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs
ec2.IamProfileAssociation.describe because DeleteInstanceProfile always
counts live associations over NATS regardless of which guard actually fires.
TestIAMCleanup is not ported: it only swept a shared long-lived E2E
environment, and every ordering/dependency assertion it made is already
covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and
TestIAMRolesAndProfiles's own delete-conflict guards.

* test(e2e): remove IAM/STS authz tests migrated to the integration tier

All 13 signed-request authz tests plus TestIAMCleanup now have faithful,
mutation-proven equivalents in tests/integration/ — the gateway dispatches
IAM/STS actions directly with no daemon hop, so the in-process tier is a
faithful and far cheaper substitute. The iam/ e2e suite now runs only
TestIAMInstanceProfileAssociation and TestIMDS.

Drop the now fully-dead iam_helpers_test.go alongside the six deleted test
files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no
remaining caller anywhere in the repo once iam_groups_test.go was gone.

* feat(integration): daemon-lite harness for key/tags/routetable/vpc

Adds DaemonLite, an in-process stand-in for a live spinifex daemon that
subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service
implementations directly to their NATS subjects, backed by memory and
embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement,
TestRouteTableValidation, and TestReplaceRouteConvergence from
tests/e2e/single so the integration tier drives genuine daemon-side
business logic instead of StubSubject canned replies.

vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg"
acknowledgements are stubbed with a fixed success reply — the SG record
itself is still written by the real service impl before that event fires,
so the stub only unblocks the synchronous round-trip rather than
replacing tested behavior.

* ci(e2e): gate live E2E jobs behind the in-process integration tier

Runs the in-process integration suite first, on the ci-single runner host
rather than inside a provisioned VM, so a broken control plane is caught
before either live job pays the Spinifex install. Both e2e_single and
e2e_multi now require it, and e2e_status treats a skipped integration job as
a failure so the gating status cannot go green on a gate that never ran.

Drops the shadow-mode suite selector: targeted per-diff suite selection is
not the direction being pursued, and leaving a non-functional shadow job in
the gating workflow only invited it to be mistaken for a live gate.

* test(e2e): remove single/ tests superseded by daemon-lite ports

TestKeyPairs, TestTagManagement, TestRouteTableValidation, and
TestReplaceRouteConvergence now have proven-equivalent ports in
tests/integration/ (mutation testing confirmed parity). Delete the
e2e originals and their runX helpers, along with keypair_test.go's
and tags_test.go's file-local helpers left with no caller.
generateImportPubKey is shared by other single/ tests so it moves to
helpers_test.go instead of being deleted. Slims the manifest-lint
baseline for the removed create-call entries.

* ci: run the lb e2e suite on multinode only

Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the
suite ran twice per e2e run: 354s on single and 654s on multi. Balancing
across instances on one node is not a deployment mode we support, and
ELBv2 proper requires two AZs, so the single-node run was redundant.

Drops ~354s from the single-node job, which is the gate critical path.

* test(integration): port 3 of 7 ECR tests to the in-process tier

Wires the gateway's ECR control plane (repository/policy/lifecycle
metadata over an in-process daemon-lite NATS subscriber, blob and
manifest bytes via a memory object store, JWT issuer/verifier keyed off
the existing IAM master key) into the integration harness, and ports
TestECRControlPlane, TestECRGetLoginPassword, and
TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each
proven equivalent by mutation (evaluatePrincipalPolicy forced to allow,
and the repo-describe path forced to not-found) before its E2E original
was removed.

The other 4 authorization tests (PullOnly, ReadOnly,
DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E
tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass
that AttachRolePolicy/AttachGroupPolicy already have, so attaching
AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails
NoSuchEntity in both tiers alike. Left untouched pending a fix to that
shared IAM code.

* fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy

AttachUserPolicy called getPolicyByARN unconditionally, so attaching an
AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and
AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store
them opaquely; the user path was missed when that guard was introduced.

ListAttachedUserPolicies had the mirror of the same gap: it resolved every
ARN through getPolicyByARN and silently dropped the ones it could not find,
so a managed ARN would disappear from the listing rather than round-trip.

Evaluation already handled these correctly via resolveAttachedPolicy, so
only the write and list paths needed the guard.

* test(integration): port remaining 4 ECR authorization tests

Ports TestECRIAMAuthorization_PullOnly, _ReadOnly,
_DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from
tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs.
Each was proven equivalent by mutation before its E2E original was
removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when
evaluatePrincipalPolicy is forced to always allow, and
DeactivatedKeyReturns401 fails when the access-key-active check in
resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and
a clean run confirmed before committing.

tests/e2e/ecr/iam_authorization_test.go is now fully removed — every
test it held has an in-process equivalent, and its helpers had no other
callers in the package.

* ci: select e2e suites by allow-list instead of deny-list

Both jobs filtered SUITES by exclusion, so any suite named in neither
deny-list ran in both jobs. That is how lb came to run twice per e2e run,
and it makes adding a suite silently fan it out everywhere.

Each job now names the suites it is eligible to run. Selection is still
driven by SUITES, so this is behaviour-preserving for the current default:
single/iam/cert on the single-node job, multinode/lb/cert on the multi-node
job, exactly as before. eks, ecs and storagegrowth are listed as eligible on
the single-node job but are not yet in the default SUITES.

* test(e2e): delete seven tests with no coverage

TestFinalClusterStats and TestInstanceLaunch assert nothing beyond
what other callers of their shared helpers already imply. The
multinode baseline, volume, guest-SSH, and preflight tests duplicate
single-node coverage or exercise per-guest behaviour that isn't
cluster-specific. TestMultinodeInstanceDistribution can't fail: three
independent RunInstances calls with no placement group make a strict
spread assertion flaky and a loose one meaningless.

Preflight checks (/dev/kvm writability, peer SSH reachability) move
into requireMultiNodeFixture so the package fixture singleton still
fails fast with a clear message instead of disappearing along with
the dedicated test. The remaining callers of needInstanceTrio
(CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still
trigger the shared trio launch on their own, so removing the
distribution and volume-lifecycle tests doesn't leave it unbuilt.

Also slims the manifest-lint baseline entry for the deleted
multinode volume test.

* fix(e2e): require full 3-node spread in TestMultinodeSpread

SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2,
and only failed below that. A 33% spread failure (two instances
colocated) passed with a log line. The 243.6s NAT-egress scenario that
follows only proves what it claims to if the private trio actually
landed on three distinct chassis, so that premise needs to be a hard
assertion, not a warning.

* refactor(e2e): drop the unused spread-assertion harness helpers

AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when
the instance-distribution test was removed. Beyond being dead, the assertion
helper documented minNodes=1 as an acceptable logging-only mode inherited from
the bash driver, which is the pattern that let a placement test pass while
every instance landed on one node.

InstanceHostingNode and its ps helpers stay; five multinode tests use them.

* fix(ec2): stop leaked goroutines on prepareEFIVolume error paths

newViperblock starts the chunk uploader and WAL syncer unconditionally,
but prepareEFIVolume only released them on the happy path via Close().
Six error returns between acquisition and Close left both goroutines
running. Add a defer immediately after acquisition, matching the
sibling prepareRootVolume; Close() remains the durability boundary.

* test(predastore): add real predastore fixture for integration tests

Extract predastore_integration_test.go's inline daemon startup into a
reusable testutil/predastore package so tests/integration can point
DaemonLite-style wiring at a real S3-compatible backend instead of an
unreachable host. Kept as its own leaf package rather than folding into
the shared testutil package: it transitively imports predastore's
quicclient, whose package-level connection pool starts a permanent
goroutine at init that would trip goleak checks in any of the ~80
other packages that import testutil for unrelated NATS helpers.

* test(integration): migrate CreateVolume to assert real predastore persistence

Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the
real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore
off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise
viperblock's real S3 backend end to end instead of stopping at input
validation.

* test(integration): add real storage-state oracle for CreateVolume

DescribeVolumes only proves the handler's own read path agrees with its
own write path, both via the same deserialization; it cannot catch a bug
in a field it never re-derives (e.g. a unit-conversion error in the bytes
handed to viperblock, since VolumeMetadata.SizeGiB is copied from the
request directly). vbscan reads a volume's persisted config.json and
chunk objects straight over S3, independent of any spinifex handler code,
so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock
actually wrote rather than only on what the handler claims.

vbscan is a local reimplementation of scripts/vbrefscan's read-only scan
rather than an import: spinifex cannot depend on mulga, the module that
contains it. It only reimplements the config/chunk-listing half of
vbrefscan — the checkpoint-based reference-integrity half needs
viperblock.ParseBlockCheckpointBytes, which does not exist in the
viperblock version spinifex currently depends on (v1.12.1) and is also
moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a
freshly created volume has no checkpoint to scan regardless of version.

* refactor(test): move the predastore daemon fixture out of the coverage scope

test-cover runs go test ./spinifex/..., so anything under spinifex/ is
compiled and measured. The daemon fixture cannot be exercised by unit
tests, so it reported a coverage gap no unit test could close. The e2e
harness already avoids this by living under tests/; the fixture now
does the same. vbscan stays put, being ordinary library code with its
own unit tests.

* test(integration): migrate RunInstances multi-count expansion off live guests

RunInstances with MinCount/MaxCount=2 expands into a single per-node
targeted launch (distributeInstances in gateway/ec2/instance/placement.go)
before the daemon-facing NATS hop, so the real gateway router plus a
stubbed daemon response is sufficient to prove it — no throwaway VM is
needed to exercise the expansion logic itself.

The live single-node E2E test's WaitForInstanceState/TerminateInstances
sequence was teardown hygiene for the real guests it booted, not an
assertion of the expansion behavior, so it is dropped rather than
emulated. The migrated test additionally asserts the per-node launch
request itself carries MinCount=MaxCount=2, proving the count actually
propagates rather than the launch succeeding by coincidence or via the
zero-daemon degenerate path in placement.go.

* docs: drop a doc reference to the deleted live test

* test(integration): migrate launch-template RunInstances expansion off live guests

RunInstances --launch-template resolves ImageId/InstanceType from the
stored template entirely gateway-side (expandLaunchTemplate, before
validation/placement/the daemon-facing NATS hop), so proving it forwards
the resolved fields never needed a real guest boot. Ports the assertion
to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real
LaunchTemplateServiceImpl over embedded JetStream) plus a direct
subscription on the per-node launch subject to capture what the gateway
actually dispatched, rather than only the client-visible response.

The live test keeps its CRUD/versioning lifecycle (create/modify/delete
version, tag-filtered describe) — none of it touches a guest — and drops
only the RunInstances-from-template + wait-for-running + terminate tail,
whose only job was waiting out a real boot to read back fields already
proven resolved.

* test(integration): migrate KV-only account-scoping checks off live tenants

Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account
isolation assertions from tests/e2e/single/account_test.go to the
in-process integration tier. Each of these resources resolves ownership
from a plain account-scoped KV key (utils.AccountKey(accountID, id), or
an account-keyed settings record) with no vm.Manager or real guest
involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl
and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key
services. Two independently minted tenant accounts (via
IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/
AttachUserPolicy, in-process — the same calls runAccountCreate makes
without the CLI/stdout round-trip) replace the live tier's carousel so
isolation is checked between real, separately owned accounts.

Left live: instance scoping (Step2) and volume scoping (Step3) —
checkInstanceOwnership, the actual cross-account gate for stop/start/
reboot/terminate/attach-volume, runs daemon-side only after a real
vm.Manager lookup, so it cannot be exercised without a live guest.
Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot
(viperblock, out of DaemonLite's scope), so it stays live entirely rather
than split off only its two KV-only assertions. Global-resources (Step9)
carries no account branching in the handlers at all, so asserting its
equality across two accounts wouldn't exercise anything a fake node
responder didn't already trivially satisfy — not migrated.

* test(integration): migrate launch-template CRUD lifecycle off live guests

TestLaunchTemplates booted zero guests: its RunInstances coverage had
already moved to TestRunInstances_LaunchTemplateExpansion, leaving only
template CRUD/versioning assertions occupying a full live-cluster slot.
Port the remaining create/describe/version/modify/delete lifecycle,
including the SourceVersion merge and a nested timestamp round-trip, into
TestLaunchTemplates_CRUDLifecycle against the in-process integration
gateway.

* refactor(tests): move spot instance lifecycle coverage to integration tier

RequestSpotInstances is a mock over on-demand RunInstances (no bidding,
interruption, or reclamation), so booting real VMs in the e2e tier proved
nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe,
cancel, close-on-terminate, insufficient-capacity error) to a NATS-level
integration test that exercises the real gateway orchestration and daemon
service impl without a live guest. Real-VM-boot assertions are already
covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests.

* fix(tests): share one embedded NATS server across the integration package

ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide
sync.Once singleton bound to whichever *nats.Conn first initialises it.
Every integration test booted its own embedded NATS server, so the first
RunInstances-calling test captured the singleton, and any later test
inherited a KV handle bound to that server's now-closed connection.

TestMain now boots one embedded NATS+JetStream server for the whole
package; every StartGateway call connects into its own dynamically
provisioned NATS account (accountAuthenticator in shared_nats.go), giving
each test a private JetStream namespace despite the shared server, and
none of the per-test connections are closed until every test has finished
- whichever one the singleton latches onto stays live for the run.

Also wire the suite into the required PR gate: it was previously only
invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration
(front gate)" job, so a regression here never blocked a PR. The whole
package runs in single-digit seconds, so it now runs inline in
`make preflight` and as its own step in spinifex.yml's build_and_test job.

* test(integration): migrate serial-console access checks off live guests

Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings
logic with no guest involved, so it moves to the in-process integration tier
against the real AccountSettingsServiceImpl wired into DaemonLite.

* test(integration): migrate DescribeImages round-trip off live guests

runImage's AMI describe round-trip needs no real cluster: it registers an
AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by
a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an
in-memory object store.

* test(integration): migrate predastore object lifecycle off live guests

runPredastoreObjectLifecycle only needs a real predastore daemon, not a
tofu-provisioned cluster: it now runs against tests/fixtures/predastore's
in-process fixture, covering the same create/put/get/delete/list contract.

* test(integration): migrate ListFoundationModels catalog check off live cluster

ListFoundationModels is pure gateway-side static catalog logic with no
NATS/daemon hop, so it moves to the integration tier via a new BedrockClient
harness accessor; this tier configures no credential resolver, so the
Anthropic entry's absence is asserted explicitly rather than left optional.

* test(integration): cover RunInstances client-token idempotency

Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently.

* test(integration): cover RunInstances launch-template expansion

Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway.

* test(integration): cover RunInstances malformed-input rejection

Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging.

* test(integration): cover RunInstances iam:PassRole enforcement

Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN.

* test(integration): cover RunInstances placement-group strategy routing

Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node.

* test(integration): cover RunInstances capacity distribution across nodes

Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first.

* test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair

Both booted their own instance pair to check SG-to-SG datapath enforcement
and same-default-SG east-west connectivity. Merging them around one shared
client/target pair cuts four boots to two, and converting the inline
harness.Step sequence into named t.Run stages (PortGroupMembership,
AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked,
ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail
attribution instead of one opaque test result.

SameSGComms now runs last and joins the default SG onto each ENI via
ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny
assertion has already recorded its own result, since the default SG's
self-reference rule admits all traffic between co-members and would
otherwise defeat the denied-traffic check.

* test(e2e): merge default-SG reachability, guest DNS, and egress enforcement

DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress
each booted their own instance to check one facet of "a guest is reachable
only when policy says so": ingress deny/authorize, resolver + WAN egress,
and egress ACL enforcement. Merging them around one shared guest cuts
three boots to one, with named t.Run stages (Deny, Authorize, Resolver,
WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution.

The egress revoke/restore round-trip now mutates this scenario's own
dedicated SG instead of the shared default SG the rest of the suite
depends on, isolating its egress-rule mutation from every other test.

* fix(ci): single-quote e2e test_run regex across the ssh boundary

* fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway

* test(e2e): merge single-node guest churn tests into one durability scenario

Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile,
TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance
into TestGuestChurnDurability, a single shared-guest scenario that
re-verifies a data-integrity sentinel sha256 after every perturbation
stage (ENI hotplug, daemon restart, ENI detach, instance-type resize,
stop/start, reboot). This proves the sentinel survives the full churn
sequence rather than just one isolated stop/start.

* fix(vpc): clear subnet drop gate when installing egress reroute

A subnet with an attached IGW but no 0.0.0.0/0 route carries a
priority-1100 drop gate on the VPC router. When a route later lands
(NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but
the drop gate was never removed, so the higher-priority gate wins and
black-holes the subnet's egress in lr_in_policy before it reaches routing
or SNAT. This left NAT-gateway private-subnet egress with 100% packet
loss: the SNAT stage saw zero packets and no conntrack was committed.

Remove the subnet's drop gate for the same prefix once a reroute is
installed. Idempotent and prefix-scoped, so a specific-prefix reroute is
a no-op and leaves the default gate intact.

* test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests

Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline,
TestNATGateway, and TestInstanceEIP into one staged scenario,
TestVPCEgressPaths, around one scenario-owned VPC and its two guests,
cutting five guest boots to three.

The public guest serves PublicSubnetEgress, then the NAT Gateway
bastion hop, then (last, since association rewrites its public IP)
the EIP flip. The private guest serves the NAT Gateway path, launched
directly into the same VPC's already-existing private subnet instead
of a dedicated one. RouteBeforeSubnet keeps its own second VPC and
instance since it exercises the implicit main-route-table code path,
a different one than the explicit route table this scenario's own VPC
uses for its public subnet — collapsing the two would drop coverage of
whichever path got merged away.

The NAT Gateway up/down flip now repeats for an env-tunable round
count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other
merged single-node scenario.

* test: TEMPORARY mutation-check — revert a795c2b2 drop-gate clear

Reverts the RemoveSubnetEgressDrop call in ensureSubnetEgressAtPriority
to confirm TestVPCEgressPaths/Round1/NATGatewayUp actually detects the
regression it guards against. This commit will be reverted immediately
after the live run confirms a failure.

* test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop

Restores the real a795c2b2 fix in ensureSubnetEgressAtPriority (the
first mutation attempt only reverted this call site and was masked by
the independent gate-recompute path in routetable service_impl.go's
CreateRoute, which also clears the drop gate on every 0.0.0.0/0 route
add and predates a795c2b2). Neutralizing RemoveSubnetEgressDrop itself
disables gate-clearing regardless of caller, reproducing the literal
regression. Will be reverted immediately after live verification.

* Revert "test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop"

Mutation-check confirmed: neutralizing RemoveSubnetEgressDrop reproduced
a795c2b2's regression exactly (stale priority-1100 drop gate outranking
the priority-1000 IGW reroute, confirmed via OVN flow dump; PublicSubnetEgress
failed with an SSH handshake timeout to the public instance). Restoring
the real fix; igw.go is now byte-identical to a795c2b2.

* fix(e2e): stop pruning shared multinode SSH key pem mid-run

The multinode package fixture cached Fixture.Artifacts once, computed
from harness.ArtifactDir(t, env) against whichever top-level Test*
happened to win the pkgFixOnce race. That helper both names the
directory after that one test and registers a t.Cleanup that removes
it once that test passes. needKeyPair wrote the shared EC2 key pair's
pem into that same directory, so as soon as the winning test finished,
its own artifact-dir cleanup deleted the pem every later test in the
package still needed to SSH into fresh guests, even though the
underlying EC2 key pair itself was untouched (its DeleteKeyPair
cleanup already correctly drains at process Close() in TestMain).

Fix: give Fixture an ArtifactDir(t) method — the same per-call pattern
already used by the gpu/single/storagegrowth/iam fixtures — so each
diagnostic dump site recomputes its own dir against the CURRENT test.
The shared key pair pem now lives under Fixture.TmpDir, which is
genuinely package-scoped and only removed once in TestMain after
m.Run() completes.

* refactor(e2e): parallelize LB e2e scenarios on multinode clusters

Each of the 8 TestLoadBalancer scenarios already stands up its own
dedicated LB + target group (ModifyListener/ListenerRules mutate their
LB's listener in place, so it can never be shared). The bottleneck was
running them serially: each teardown made createActiveLB back off
waiting for the previous LB's sys.micro slot to be reclaimed before the
next one could stand up.

On multinode clusters (peer != "", i.e. the 3-node cluster this suite
targets) there's spare capacity to stand up all 8 dedicated LBs at once,
so scenarios now run as t.Parallel() subtests instead of serial
create/teardown/backoff. Since each scenario owns an independent LB,
one never reaching active only fails that scenario — it no longer
cascades into five sequential 5-minute timeouts, so the serial fail-fast
gate is skipped in this mode; concurrency itself bounds the worst case
to a single timeout window.

Single-node dev boxes (no peer) keep the original serial path,
including the fail-fast gate, since they lack the spare capacity for
concurrent LB standup. No subtest body, assertion, or call signature
changed — only the dispatch/gating logic in TestLoadBalancer.

* fix(ci): gate e2e jobs on the suites they actually run

* fix(e2e): serialize internet-facing LB subtests among themselves

Live multinode verification showed InternetFacing_ALB and
InternetFacing_NLB running concurrently each clobber the other's
public-LB DNS record mid-registration, so neither ever resolves within
the 90s wait. The old serial suite never had two internet-facing LBs
alive at once, so this never surfaced.

Drop t.Parallel() from the three internet-facing subtests so their
register+resolve windows never overlap each other, while leaving the
internal subtests parallel as before. They may still run alongside the
internal group, which resolves internal (not public) DNS and isn't
affected.

* fix(e2e): give each ALB LB scenario its own security group

Concurrent internal subtests all created ALBs with no explicit
SecurityGroups, so each ALB's ENI fell back to the one shared VPC
default SG. One scenario authorizing/reconfiguring its own listener
racing another mutating the same SG made Internal_ALB_ModifyListener
intermittently fail to serve on the modified port under parallel load.

Each ALB scenario now creates a dedicated SG (createScenarioSG),
authorized only for the ports that scenario's listener(s) use, and
passes it to CreateLoadBalancer. The shared app-instance/probe-client
SG is untouched, since SG ingress is enforced at the destination ENI
regardless of the source ENI's SG. NLBs are unaffected: they already
get an auto-managed per-LB SG when none is supplied.

* test(e2e): annotate intentional create-path calls for manifest-lint

The vpcegress and per-scenario LB security-group tests create raw AWS
resources on purpose — the egress datapath, the route-before-subnet
ordering, and per-scenario SG isolation are the subjects under test, so
they cannot route through a shared Ensure* fixture. Mark each with
// e2e:allow-create and slim the baseline of entries for the removed
single-node network tests.

* test(e2e): restore live spot terminate-close and cancel-running coverage

The integration-tier spot test calls CloseForInstance directly and infers
cancel-keeps-running from the SIR's InstanceId, because DaemonLite has no
vm.Manager to drive the real TerminateInstances -> daemon teardown cleaner
-> RemoveFromSpotRequest -> CloseForInstance chain or to observe a real
VM's running state. Add a thin live e2e test that drives both terminal
transitions through a real backing VM: cancel one spot-backed instance and
observe it stays running, terminate the other and observe its SIR reach
closed through the genuine async delegation.

* fix(network): refresh orphan-EIP prune liveness to spare mid-pass launches

pruneOrphanEIPs listed OVN NAT rows live but matched them against the intent
snapshot captured at the start of the reconcile pass. Because the apply phase
can block for tens of seconds on guest-port datapath waits, a guest launched
during the pass had a live dnat_and_snat row absent from the stale snapshot, so
the prune swept it and blackholed the guest's auto-assigned public IP until the
next pass re-added it — an intermittent multinode SSH-dark flake.

Add an optional FreshIntent re-read callback (wired in vpcd to LoadIntentFromKV)
and union its ports into the live set before pruning, so a mid-pass launch
counts as live. Skip the prune entirely when the re-read fails rather than risk
a false sweep against a snapshot known to be stale. Self-healing and idempotent:
a genuine orphan absent from both snapshot and fresh read is still pruned next
drift. Single-node behaviour is unchanged (callback nil in tests).

.githooks/pre-commit

@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-branch=$(git rev-parse --abbrev-ref HEAD)
-
-# Only enforce preflight on dev and main
-if [ "$branch" != "dev" ] && [ "$branch" != "main" ]; then
-  exit 0
-fi
-
-# Only run preflight if Go files were staged
-if git diff --cached --name-only | grep -q '\.go$'; then
-  echo "Go files staged on $branch — running make preflight..."
-  make preflight
-else
-  echo "No Go files staged, skipping preflight."
-fi

.github/assets/architecture.svg

@@ -1,225 +1,114 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
-  <defs>
-    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#161b22"/>
-      <stop offset="45%" style="stop-color:#1f2028"/>
-      <stop offset="100%" style="stop-color:#161b22"/>
-    </linearGradient>
-    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#3fb95022"/>
-      <stop offset="100%" style="stop-color:#3fb95008"/>
-    </linearGradient>
-    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#DC262633"/>
-      <stop offset="100%" style="stop-color:#7F1D1D14"/>
-    </linearGradient>
-    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#F87171"/>
-      <stop offset="50%" style="stop-color:#DC2626"/>
-      <stop offset="100%" style="stop-color:#7F1D1D"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="620" fill="url(#bg)" rx="12"/>
-
-  <!-- Title bar -->
-  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
-  <text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
-  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
-
-  <!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
-  <rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
-  <text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
-  <text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
-
-  <g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
-    <rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-    <rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-    <rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
-
-    <rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
-
-    <rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
-
-    <rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
-
-    <rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
-  </g>
-
-  <!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
-  <line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
-  <text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
-
-  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
-  <rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
-  <text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
-  <text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
-
-  <g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
-    <rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
-
-    <rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
-
-    <rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
-
-    <rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
-
-    <rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
-
-    <rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
-  </g>
-
-  <!-- ═══ NATS BUS ═══════════════════════════════════════ -->
-  <line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
-  <text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
-  <text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.*   ebs.*   s3.*   vpc.*   iam.*</text>
-  <text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
-
-  <!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
-  <g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
-    <line x1="100" y1="306" x2="100" y2="328"/>
-    <line x1="275" y1="306" x2="275" y2="328"/>
-    <line x1="450" y1="306" x2="450" y2="328"/>
-    <line x1="625" y1="306" x2="625" y2="328"/>
-    <line x1="800" y1="306" x2="800" y2="328"/>
-  </g>
-
-  <!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
-  <!-- EC2 -->
-  <g transform="translate(28, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
-  </g>
-
-  <!-- EBS -->
-  <g transform="translate(204, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
-  </g>
-
-  <!-- S3 -->
-  <g transform="translate(380, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
-  </g>
-
-  <!-- VPC -->
-  <g transform="translate(556, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
-  </g>
-
-  <!-- IAM -->
-  <g transform="translate(732, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
-  </g>
-
-  <!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
-  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
-    <line x1="100" y1="410" x2="100" y2="440"/>
-    <line x1="276" y1="410" x2="276" y2="440"/>
-    <line x1="452" y1="410" x2="452" y2="440"/>
-    <line x1="628" y1="410" x2="628" y2="440"/>
-    <line x1="804" y1="410" x2="804" y2="440"/>
-  </g>
-
-  <!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
-  <rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
-  <text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
-
-  <!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
-  <!-- QEMU/KVM -->
-  <g transform="translate(28, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
-  </g>
-
-  <!-- Viperblock -->
-  <g transform="translate(204, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
-  </g>
-
-  <!-- Predastore -->
-  <g transform="translate(380, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
-  </g>
-
-  <!-- OVN/OVS -->
-  <g transform="translate(556, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
-  </g>
-
-  <!-- IAM store -->
-  <g transform="translate(732, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
-  </g>
-
-  <!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
-  <g transform="translate(16, 558)">
-    <rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
-    <text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      <tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
-      <tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
-    </text>
-    <text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
-      No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
-    </text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="770" viewBox="0 0 1020 770" role="img" aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
+<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
+<rect x="0.0" y="0.0" width="1020.0" height="770.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="28.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ARCHITECTURE</text>
+<text x="992.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">stateless · message-driven · horizontally scalable</text>
+<text x="28.0" y="72.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS SDK → Gateway → NATS → service daemons</text>
+<rect x="28.0" y="92.0" width="964.0" height="92.0" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="119.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">CLIENTS</text>
+<text x="48.0" y="140.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#5B636C" font-weight="400" text-anchor="start">Standard AWS tooling, unmodified. No shims, no custom SDK.</text>
+<rect x="46.0" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="80.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli</text>
+<rect x="123.2" y="153.0" width="81.4" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="163.9" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Terraform</text>
+<rect x="213.6" y="153.0" width="55.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="241.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">boto3</text>
+<rect x="277.6" y="153.0" width="88.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="321.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-sdk-go</text>
+<rect x="374.6" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="408.7" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">kubectl</text>
+<rect x="451.8" y="153.0" width="61.6" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="482.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<rect x="522.4" y="153.0" width="74.8" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="559.8" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">your app</text>
+<line x1="510.0" y1="186.0" x2="510.0" y2="192.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="429.9" y="191.0" width="160.3" height="22.0" rx="11" fill="#1E242B" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.4"/>
+<text x="510.0" y="205.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="400" text-anchor="middle" fill-opacity="0.9">HTTPS · SigV4 · :9999</text>
+<line x1="510.0" y1="212.0" x2="510.0" y2="220.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="224.0" width="964.0" height="94.0" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="48.0" y="251.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="700" text-anchor="start" letter-spacing="1.5">AWS GATEWAY</text>
+<text x="48.0" y="272.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
+<rect x="46.0" y="287.0" width="68.2" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="80.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">TLS 1.3</text>
+<rect x="123.2" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="167.2" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">SigV4 auth</text>
+<rect x="220.2" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="267.5" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">policy eval</text>
+<rect x="323.8" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="371.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">req routing</text>
+<rect x="427.4" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="471.4" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">rate-limit</text>
+<rect x="524.4" y="287.0" width="81.4" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="565.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">audit log</text>
+<line x1="510.0" y1="320.0" x2="510.0" y2="332.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="url(#nats)" fill-opacity="1.0" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55"/>
+<text x="50.0" y="365.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">NATS BUS</text>
+<text x="178.0" y="364.6" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#F5F3EE" font-weight="400" text-anchor="start" fill-opacity="0.92">ec2.*  ebs.*  s3.*  vpc.*  iam.*</text>
+<text x="972.0" y="363.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F5F3EE" font-weight="400" text-anchor="end" fill-opacity="0.9">queue groups · fan-out · request/reply</text>
+<line x1="119.6" y1="388.0" x2="119.6" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="314.8" y1="388.0" x2="314.8" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="510.0" y1="388.0" x2="510.0" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="705.2" y1="388.0" x2="705.2" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="900.4" y1="388.0" x2="900.4" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<rect x="28.0" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="44.0" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ec2.* daemon</text>
+<line x1="44.0" y1="448.0" x2="195.2" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="44.0" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">RunInstances</text>
+<text x="44.0" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">TerminateInstances</text>
+<text x="44.0" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QMP · cloud-init</text>
+<rect x="223.2" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="239.2" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ebs.* daemon</text>
+<line x1="239.2" y1="448.0" x2="390.4" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="239.2" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">CreateVolume</text>
+<text x="239.2" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">AttachVolume</text>
+<text x="239.2" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">snapshots · WAL</text>
+<rect x="418.4" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="434.4" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">s3.* daemon</text>
+<line x1="434.4" y1="448.0" x2="585.6" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="434.4" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">PutObject · GetObject</text>
+<text x="434.4" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Multipart · presigned</text>
+<text x="434.4" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">erasure-coded</text>
+<rect x="613.6" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="629.6" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">vpc.* daemon</text>
+<line x1="629.6" y1="448.0" x2="780.8" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="629.6" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Subnets · SGs · RTs</text>
+<text x="629.6" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">ALB · NAT · IGW</text>
+<text x="629.6" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN · OVS</text>
+<rect x="808.8" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="824.8" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">iam.* daemon</text>
+<line x1="824.8" y1="448.0" x2="976.0" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="824.8" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Users · Roles · Keys</text>
+<text x="824.8" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Policy evaluation</text>
+<text x="824.8" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">audit trail</text>
+<line x1="119.6" y1="516.0" x2="119.6" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="314.8" y1="516.0" x2="314.8" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="510.0" y1="516.0" x2="510.0" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="705.2" y1="516.0" x2="705.2" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="900.4" y1="516.0" x2="900.4" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<rect x="28.0" y="530.0" width="964.0" height="30.0" rx="9" fill="#1E242B" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="548.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle" letter-spacing="1">BACKEND RESOURCES  ·  local hardware  ·  no cloud dependency</text>
+<rect x="28.0" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="119.6" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">QEMU / KVM</text>
+<text x="119.6" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">hypervisor</text>
+<text x="119.6" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">host kernel · virtio</text>
+<rect x="223.2" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="314.8" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Viperblock</text>
+<text x="314.8" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">block storage</text>
+<text x="314.8" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">NVMe WAL · NBD</text>
+<rect x="418.4" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Predastore</text>
+<text x="510.0" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">object storage</text>
+<text x="510.0" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Reed-Solomon</text>
+<rect x="613.6" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="705.2" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">OVN / OVS</text>
+<text x="705.2" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">SDN fabric</text>
+<text x="705.2" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">overlays · flow rules</text>
+<rect x="808.8" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="900.4" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Identity store</text>
+<text x="900.4" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">users · keys · rules</text>
+<text x="900.4" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">local · audit-logged</text>
+<rect x="28.0" y="670.0" width="964.0" height="72.0" rx="11" fill="#181D23" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="697.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#EE4B12" font-weight="700">Queue groups</tspan><tspan fill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspan fill="#EE4B12" font-weight="700">Fan-out</tspan><tspan fill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
+<text x="48.0" y="720.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#5B636C">Daemons are </tspan><tspan fill="#EE4B12" font-weight="700">stateless</tspan><tspan fill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
+</svg>

.github/assets/platform.svg

@@ -1,144 +1,84 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
-  <defs>
-    <linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
-    </linearGradient>
-    <linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
-    </linearGradient>
-    <linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
-      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
-    </linearGradient>
-    <linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
-
-  <!-- Subtle mesh dots background -->
-  <g opacity="0.12" fill="#7F1D1D">
-    <circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
-    <circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
-    <circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
-    <circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
-    <circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
-    <circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
-  </g>
-
-  <!-- Title -->
-  <text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
-  <text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
-
-  <!-- Layer 4: Applications (top) -->
-  <g transform="translate(60, 95)">
-    <rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
-    <text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
-
-    <g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
-      <rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-      <rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-      <rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
-
-      <rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
-
-      <rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
-
-      <rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 188)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 3: Spinifex (highlighted) -->
-  <g transform="translate(60, 215)">
-    <rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
-    <text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
-
-    <g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
-      <!-- EC2 -->
-      <rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
-      <text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
-      <text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
-
-      <!-- EBS -->
-      <rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
-      <text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
-      <text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
-
-      <!-- S3 -->
-      <rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
-      <text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
-      <text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
-
-      <!-- VPC -->
-      <rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
-      <text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
-      <text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
-
-      <!-- IAM -->
-      <rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
-      <text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
-      <text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
-
-      <!-- Control -->
-      <rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
-      <text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
-      <text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 360)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 2: OS -->
-  <g transform="translate(60, 387)">
-    <rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 454)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 1: Hardware -->
-  <g transform="translate(60, 481)">
-    <rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="664" viewBox="0 0 1000 664" role="img" aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
+<rect x="0.0" y="0.0" width="1000.0" height="664.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="204.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.12" stroke-dasharray="2 4"/>
+<rect x="25.0" y="201.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.2" stroke-dasharray="2 4"/>
+<rect x="22.0" y="198.0" width="950.0" height="438.0" rx="16" fill="#181D23" fill-opacity="0.25" stroke="#9AA1A8" stroke-width="1.3" stroke-opacity="0.45" stroke-dasharray="2 4"/>
+<text x="36.0" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Apps</text>
+<rect x="136.0" y="34.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="66.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">Your infrastructure &amp; apps</text>
+<rect x="516.7" y="51.0" width="89.3" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="561.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">AI training</text>
+<rect x="613.0" y="51.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="651.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">Inference</text>
+<rect x="696.7" y="51.0" width="79.0" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="736.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Web &amp; APIs</text>
+<rect x="782.7" y="51.0" width="72.7" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="819.1" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Databases</text>
+<rect x="862.4" y="51.0" width="91.6" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="908.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Batch &amp; data</text>
+<text x="36.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Tooling</text>
+<rect x="136.0" y="102.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="134.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">AWS tooling</text>
+<rect x="600.9" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="633.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">aws-cli</text>
+<rect x="672.0" y="119.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="710.4" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">Terraform</text>
+<rect x="755.7" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="787.8" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">kubectl</text>
+<rect x="826.8" y="119.0" width="66.4" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="860.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">AWS SDKs</text>
+<rect x="900.2" y="119.0" width="53.8" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="927.1" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<text x="484.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">↓  AWS API · SigV4</text>
+<text x="624.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">RESPONSE  ↑</text>
+<text x="136.0" y="231.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">One portable stack.</text>
+<text x="136.0" y="249.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
+<rect x="752.3" y="222.0" width="72.4" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="788.5" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">NEOCLOUD</text>
+<rect x="832.7" y="222.0" width="66.1" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="865.7" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">ON-PREM</text>
+<rect x="906.8" y="222.0" width="47.2" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="930.4" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">EDGE</text>
+<text x="152.0" y="295.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS orchestration</text>
+<text x="956.0" y="293.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.9">Gateway → NATS → daemons</text>
+<rect x="150.0" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EC2</text>
+<text x="164.0" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Compute</text>
+<text x="164.0" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QEMU / KVM</text>
+<rect x="422.7" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EBS</text>
+<text x="436.7" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Block storage</text>
+<text x="436.7" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Viperblock</text>
+<rect x="695.3" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">S3</text>
+<text x="709.3" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Object storage</text>
+<text x="709.3" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Predastore</text>
+<rect x="150.0" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">VPC</text>
+<text x="164.0" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Networking</text>
+<text x="164.0" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN / OVS</text>
+<rect x="422.7" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">IAM</text>
+<text x="436.7" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Identity</text>
+<text x="436.7" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">SigV4 · policies</text>
+<rect x="695.3" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EKS</text>
+<text x="709.3" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Kubernetes</text>
+<text x="709.3" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">K8s-conformant</text>
+<rect x="150.0" y="452.0" width="808.0" height="32.0" rx="8" fill="#14181D" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0" stroke-dasharray="3 3"/>
+<text x="164.0" y="472.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">+ ALB · ECS · ECR · RDS and more of the core AWS services</text>
+<text x="944.0" y="471.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">See all →</text>
+<text x="36.0" y="386.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="start">Spinifex</text>
+<rect x="136.0" y="268.0" width="836.0" height="230.0" rx="12" fill="#EE4B12" fill-opacity="0.07" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="36.0" y="536.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">OS</text>
+<rect x="136.0" y="510.0" width="836.0" height="46.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="537.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">Operating system</text>
+<text x="952.0" y="536.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · Open vSwitch</text>
+<text x="36.0" y="596.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Hardware</text>
+<rect x="136.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">CPU</text>
+<text x="533.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Intel · AMD · ARM</text>
+<rect x="559.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="579.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">GPU</text>
+<text x="956.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">NVIDIA · AMD</text>
+</svg>

.github/resolve-sibling-refs.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Resolve which branch of each sibling repo (viperblock, predastore) the CI
+# workspace should build against. A cross-repo change is developed on a branch
+# of the SAME name in each repo; build against that branch when it exists so the
+# set can be tested together, and fall back to dev otherwise. This mirrors the
+# resolution the e2e path already does in
+# mulga/scripts/tofu-cluster/build-install-artifacts.sh, so unit CI and e2e CI
+# agree on how sibling branches are selected.
+#
+# Reads $BRANCH (the triggering head/ref name) and writes viperblock=<ref> and
+# predastore=<ref> to $GITHUB_OUTPUT for the checkout steps to consume.
+set -euo pipefail
+
+: "${BRANCH:?BRANCH must be set to the triggering branch name}"
+: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
+
+for repo in viperblock predastore; do
+  if git ls-remote --exit-code --heads \
+      "https://github.com/mulgadc/${repo}.git" "$BRANCH" >/dev/null 2>&1; then
+    ref="$BRANCH"
+  else
+    ref="dev"
+  fi
+  echo "${repo}: building against ${ref}"
+  echo "${repo}=${ref}" >>"$GITHUB_OUTPUT"
+done

.github/workflows/e2e-suites.yml

@@ -107,7 +107,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -294,7 +294,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -539,7 +539,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -663,7 +663,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false

.github/workflows/e2e.yml

@@ -37,49 +37,52 @@ env:
   SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
-  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
-  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
-  # can compare predicted vs actual before flipping it on.
-  select_suites:
-    name: Select Suites (shadow)
-    runs-on: ubuntu-26.04
-    timeout-minutes: 5
+  # In-process integration front gate. Provisions nothing -- no tofu, no
+  # docker, no Spinifex daemons, no fixed ports -- so it runs directly on the
+  # runner host rather than inside a provisioned VM, and completes in seconds.
+  #
+  # It runs on ci-single rather than a GitHub-hosted runner so it shares the
+  # toolchain and sibling-repo build of the live suites, and so the run stays
+  # entirely on the LAN that hosts the Elastic sink. Holding a ci-single slot
+  # for seconds is negligible against the 50-minute e2e_single that follows.
+  #
+  # e2e_single and e2e_multi both require this job to succeed, so a red
+  # integration run is caught before either provisions a tofu env or pays the
+  # Spinifex install -- which is the expensive half, not the tofu apply.
+  integration:
+    name: Integration (front gate)
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 10
     steps:
+      # This runner is shared with e2e_single, which leaves root-owned tofu and
+      # mount artifacts behind; a plain checkout would fail on them. Unmount
+      # anything still bound under the workspace before wiping it.
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
       - uses: actions/checkout@v7
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: go.mod
           cache: false
-      - name: Compute targeted suite selection
-        run: |
-          set -euo pipefail
-          BASE="${{ github.event.before }}"
-          HEAD="${{ github.sha }}"
-          # First push to a branch reports an all-zero before-SHA; fall back to
-          # the first parent so the diff is still meaningful.
-          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
-            BASE="${HEAD}~1"
-          fi
-          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
-          go run ./tests/e2e/select-suites/cmd/select-suites \
-            --manifest docs/service-interfaces.yaml \
-            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
-            2>>"$GITHUB_STEP_SUMMARY" \
-            | tee -a "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
+      - name: Run in-process integration tests
+        run: make test-integration
 
   e2e_single:
     name: Single-Node E2E
-    # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-multinode' only). Push triggers and the default SUITES list
-    # both include single/cert/lb so the inputs.suites null-check keeps the
-    # job running on plain PR pushes.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-iam')
+    needs: [integration]
+    # The integration front gate must be green: a red integration run stops
+    # here, before this job provisions a tofu env or pays the Spinifex install.
+    #
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists (single|iam|cert|eks|ecs|storagegrowth). Gating on a suite the
+    # job won't run — lb, which only executes in the multinode job — would
+    # provision a tofu VM and pay the Spinifex install only to run nothing and
+    # tear down. Push triggers keep running via the null-check on inputs.suites.
+    if: needs.integration.result == 'success' && (github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-iam') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-ecs') || contains(github.event.inputs.suites, 'e2e-storagegrowth'))
     runs-on: [self-hosted, ci-single]
     timeout-minutes: 50
     env:
@@ -134,7 +137,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -312,14 +315,20 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list: a suite runs here only if named. Deny-lists
+          # silently fan a newly added suite out to every job, which is how lb ended up
+          # running twice per e2e run.
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE 'single|iam|cert|eks|ecs|storagegrowth' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # The value is expanded into a command that a remote shell re-parses over
+            # ssh, so single-quote it — otherwise regex metacharacters like ( | ) $ are
+            # taken as shell syntax on the VM and bash fails before the test binary runs.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -486,7 +495,14 @@ jobs:
 
   e2e_multi:
     name: Multi-Node E2E
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
+    needs: [integration]
+    # Gated on the integration front gate for the same reason as e2e_single:
+    # the 3-node env is never stood up if the control plane is already broken.
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists (multinode|lb|cert) so lb — which only runs here — still
+    # provisions the 3-node env when requested on its own, and the env stays
+    # unprovisioned when none of them are asked for.
+    if: needs.integration.result == 'success' && (github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-cert'))
     runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
@@ -541,7 +557,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -697,14 +713,17 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks|iam)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list — see the single-node job for why.
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE 'multinode|lb|cert' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # Single-quote for the remote shell — see the single-node job for the
+            # metacharacter-injection reason.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -911,7 +930,7 @@ jobs:
   e2e_status:
     name: Report E2E Status
     if: always()
-    needs: [e2e_single, e2e_multi]
+    needs: [integration, e2e_single, e2e_multi]
     runs-on: ubuntu-26.04
     timeout-minutes: 5
     steps:
@@ -920,18 +939,166 @@ jobs:
           GH_TOKEN: ${{ github.token }}
           REPORT_SHA: ${{ github.sha }}
         run: |
+          # integration must have actually succeeded -- unlike e2e_single/multi,
+          # "skipped" is never an acceptable result for it (it has no `if:` of
+          # its own), so a red or skipped integration run fails the gate here
+          # even though e2e_single/e2e_multi themselves report "skipped" (their
+          # `if:` never let them start).
+          integration_ok=true
+          case "${{ needs.integration.result }}" in success) ;; *) integration_ok=false ;; esac
           single_ok=true; multi_ok=true
           case "${{ needs.e2e_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
           case "${{ needs.e2e_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
-          if $single_ok && $multi_ok; then
+          if $integration_ok && $single_ok && $multi_ok; then
             STATE="success"
             DESC="E2E suites passed"
           else
             STATE="failure"
-            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_multi.result }}"
+            DESC="E2E failed: integration=${{ needs.integration.result }} single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_multi.result }}"
           fi
           gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
             -f state="$STATE" \
             -f context=e2e \
             -f description="$DESC" \
             -f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+
+  # Emits one summary doc per run to the central Elastic sink, keyed by
+  # labels.ci_run_id, so Kibana can list "did run X pass, and where are its
+  # artifacts" without hand-aggregating traces-apm*/logs-apm.error*. This is
+  # a separate self-hosted job (not a step on e2e_status) because e2e_status
+  # runs on GitHub-hosted ubuntu-26.04, which has no route to the private
+  # 192.168.1.13 sink, and moving e2e_status off GH-hosted would risk the PR
+  # status-report check.
+  e2e_es_summary:
+    name: ES CI Run Summary
+    needs: [integration, e2e_single, e2e_multi]
+    if: always()
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 5
+    permissions:
+      actions: read
+    env:
+      # Repo var overrides; defaults to the LAN Elastic sink reachable from
+      # ci-single/ci-multi runners. Anonymous ES requests are superuser on
+      # this host, so no credentials are needed for the POST.
+      MULGA_ES_SINK: ${{ vars.MULGA_ES_SINK || '192.168.1.13:9200' }}
+    steps:
+      # Either leg may not have run (e.g. a workflow_dispatch that only
+      # requested single-node suites), so a missing artifact is tolerated —
+      # the aggregation script below treats an absent suite-status.txt as
+      # "no failures from that leg".
+      - name: Download single-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-single
+
+      - name: Download multi-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-multi-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-multi
+
+      - name: Build and post CI run summary
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          RUN_ID: ${{ github.run_id }}
+          RUN_ATTEMPT: ${{ github.run_attempt }}
+          WORKFLOW: ${{ github.workflow }}
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+          SHA: ${{ github.sha }}
+          SERVER_URL: ${{ github.server_url }}
+          INTEGRATION_RESULT: ${{ needs.integration.result }}
+          SINGLE_RESULT: ${{ needs.e2e_single.result }}
+          MULTI_RESULT: ${{ needs.e2e_multi.result }}
+          SINGLE_DIR: ${{ github.workspace }}/es-summary-single
+          MULTI_DIR: ${{ github.workspace }}/es-summary-multi
+        run: |
+          set -euo pipefail
+
+          SINGLE_STATUS="${SINGLE_DIR}/suite-status.txt"
+          MULTI_STATUS="${MULTI_DIR}/suite-status.txt"
+
+          # Collect "<suite>: FAILED" lines from both legs' suite-status.txt
+          # (either may be absent — a leg that didn't run, or a passing leg
+          # that never wrote the file). Accumulate as newline-separated text
+          # rather than a bash array so this stays correct under `set -u`
+          # even when nothing failed.
+          FAILED_SUITES=""
+          for f in "$SINGLE_STATUS" "$MULTI_STATUS"; do
+            [ -f "$f" ] || continue
+            while IFS= read -r line; do
+              case "$line" in
+                *": FAILED") FAILED_SUITES="${FAILED_SUITES}${line%: FAILED}"$'\n' ;;
+              esac
+            done < "$f"
+          done
+
+          FAILED_JSON=$(printf '%s' "$FAILED_SUITES" | jq -R -s 'split("\n") | map(select(length > 0))')
+          FAILED_COUNT=$(printf '%s' "$FAILED_JSON" | jq 'length')
+
+          # Mirror e2e_status's integration_ok/single_ok/multi_ok logic exactly
+          # so the PR status and this ES doc can never disagree on overall
+          # pass/fail. Like e2e_status, "skipped" is only acceptable for
+          # single/multi (their `if:` may legitimately not select them); a
+          # skipped or failed integration always fails the gate.
+          integration_ok=true
+          case "$INTEGRATION_RESULT" in success) ;; *) integration_ok=false ;; esac
+          single_ok=true
+          case "$SINGLE_RESULT" in success | skipped) ;; *) single_ok=false ;; esac
+          multi_ok=true
+          case "$MULTI_RESULT" in success | skipped) ;; *) multi_ok=false ;; esac
+
+          STATE=success
+          if [ "$integration_ok" != true ] || [ "$single_ok" != true ] || [ "$multi_ok" != true ] || [ "$FAILED_COUNT" -gt 0 ]; then
+            STATE=failure
+          fi
+
+          RUN_STARTED_AT=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}" --jq .run_started_at)
+          START_EPOCH=$(date -u -d "${RUN_STARTED_AT}" +%s)
+          NOW_EPOCH=$(date -u +%s)
+          DURATION=$((NOW_EPOCH - START_EPOCH))
+
+          DOC=$(jq -n \
+            --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            --arg run_id "${RUN_ID}" \
+            --arg workflow "${WORKFLOW}" \
+            --arg branch "${BRANCH}" \
+            --arg sha "${SHA}" \
+            --arg run_attempt "${RUN_ATTEMPT}" \
+            --arg state "${STATE}" \
+            --arg integration_result "${INTEGRATION_RESULT}" \
+            --arg single_result "${SINGLE_RESULT}" \
+            --arg multi_result "${MULTI_RESULT}" \
+            --argjson duration "${DURATION}" \
+            --arg artifact_url "${SERVER_URL}/${REPO}/actions/runs/${RUN_ID}" \
+            --argjson failed_suites "${FAILED_JSON}" \
+            '{
+              "@timestamp": $ts,
+              labels: { ci_run_id: $run_id },
+              ci: { workflow: $workflow, branch: $branch, sha: $sha, run_attempt: $run_attempt },
+              state: $state,
+              integration_result: $integration_result,
+              single_result: $single_result,
+              multi_result: $multi_result,
+              failed_suites: $failed_suites,
+              duration_seconds: $duration,
+              artifact_url: $artifact_url
+            }')
+
+          echo "$DOC" | jq .
+
+          HTTP_CODE=$(curl -sS -o /tmp/es-summary-resp.json -w '%{http_code}' \
+            -X POST "http://${MULGA_ES_SINK}/mulga-ci-runs/_doc" \
+            -H 'Content-Type: application/json' \
+            -d "${DOC}")
+
+          if [ "$HTTP_CODE" != "200" ] && [ "$HTTP_CODE" != "201" ]; then
+            echo "::error::ES POST to ${MULGA_ES_SINK}/mulga-ci-runs failed (HTTP ${HTTP_CODE})"
+            cat /tmp/es-summary-resp.json
+            exit 1
+          fi
+          echo "CI run summary doc posted to ${MULGA_ES_SINK}/mulga-ci-runs (HTTP ${HTTP_CODE})"

.github/workflows/gpu-e2e.yml

@@ -131,7 +131,7 @@ jobs:
       - uses: actions/checkout@v7
         if: steps.gate.outputs.paused != 'true'
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         if: steps.gate.outputs.paused != 'true'
         with:
           go-version-file: go.mod

.github/workflows/publish-eks-ami.yml

@@ -95,7 +95,7 @@ jobs:
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: "spinifex/go.mod"
           # Self-hosted runner: the Go module + build cache already persists on
@@ -105,7 +105,13 @@ jobs:
           cache: false
 
       - name: Setup Go Workspace
-        run: go work init ./spinifex ./viperblock ./predastore
+        # Idempotent: the ci-single runner reuses one workspace across matrix
+        # legs and runs, and checkout never scrubs the root go.work. Remove any
+        # stale file so each leg re-inits cleanly instead of failing on
+        # "go.work already exists".
+        run: |
+          rm -f go.work go.work.sum
+          go work init ./spinifex ./viperblock ./predastore
 
       - name: Install build + upload tooling
         run: |

.github/workflows/release.yml

@@ -32,7 +32,7 @@ jobs:
         uses: docker/setup-buildx-action@v4
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version: "1.26.5"
 
@@ -280,7 +280,7 @@ jobs:
         run: sudo make -C mulga/scripts/iso-builder clean
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false

.github/workflows/spinifex.yml

@@ -32,22 +32,29 @@ jobs:
           path: spinifex
           fetch-depth: 2
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -61,6 +68,15 @@ jobs:
       - name: Manifest check + drift lint
         run: make -C spinifex manifest-check manifest-lint
 
+      # In-process integration tier (gateway.SetupRoutes over embedded NATS
+      # JetStream). Runs here, on every PR, rather than only relying on
+      # e2e.yml's "Integration (front gate)" job: that job is push-triggered
+      # against main/dev on a self-hosted runner, so it never blocks a PR
+      # before merge. This step provisions nothing and completes in seconds,
+      # so it belongs as an ordinary required PR check, not a separate job.
+      - name: Run Integration Tests
+        run: make -C spinifex test-integration
+
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
@@ -80,22 +96,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -116,22 +139,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"

.github/workflows/ui.yml

@@ -40,7 +40,7 @@ jobs:
           package_json_file: spinifex/services/spinifexui/frontend/package.json
 
       - name: Setup Node.js
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v7
         with:
           cache: "pnpm"
           cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml

.golangci.yml

@@ -15,21 +15,24 @@ linters:
     - bodyclose # checks whether HTTP response body is closed successfully
     - canonicalheader # checks whether net/http.Header uses canonical header keys
     - decorder # checks declaration order of types, consts, vars, and funcs
+    - depguard # blocks imports of packages with a preferred replacement
     - durationcheck # checks for two durations multiplied together
     - embeddedstructfieldcheck # checks embedded sync.Mutex/RWMutex in structs
     - errcheck # checks for unchecked errors in type assertions and I/O
     - errchkjson # checks types passed to json encoding functions, flags omittable error checks
     - errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
+    - errorlint # catches ==/!= on errors, type assertions on errors, and %v instead of %w — all break on wrapped errors
     - exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
     - fatcontext # detects nested contexts in loops
-    - forcetypeassert # flags unchecked type assertions that can panic
     - gocheckcompilerdirectives # validates go compiler directive comments (//go:)
+    - godot # checks that declaration comments end in a period
     - goprintffuncname # checks that printf-like functions are named with f at the end
     - gosec # inspects source code for security problems
     - govet # reports suspicious constructs (Printf alignment, shadow, etc.)
     - ineffassign # detects assignments to existing variables that are not used
     - makezero # finds slice declarations with non-zero initial length followed by append
     - mirror # catches wrong mirror patterns in bytes/strings usage
+    - misspell # finds commonly misspelled English words
     - musttag # checks structs passed to marshal/unmarshal have proper tags
     - modernize # modernizes code to use newer Go idioms (go fix analyzers)
     - nakedret # finds naked returns in functions
@@ -43,6 +46,8 @@ linters:
     - recvcheck # checks for receiver type consistency (pointer vs value)
     - sloglint # enforces consistent log/slog usage (key-value pairs, no globals)
     - staticcheck # extensive static analysis checks
+    - testifylint # checks common testify mistakes (assert.Len over assert.Equal on len, etc.)
+    - tparallel # detects inappropriate usage of t.Parallel()
     - unconvert # removes unnecessary type conversions
     - usestdlibvars # detects hardcoded values that should use stdlib constants
     - usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
@@ -53,8 +58,17 @@ linters:
     embeddedstructfieldcheck:
       forbid-mutex: true
 
+    depguard:
+      rules:
+        main:
+          deny:
+            - pkg: log$
+              desc: Use log/slog instead, see https://go.dev/blog/slog
+            - pkg: math/rand$
+              desc: Use math/rand/v2 instead, see https://go.dev/blog/randv2
+
     errcheck:
-      check-type-assertions: true
+      check-type-assertions: true # replaces forcetypeassert
       exclude-functions:
         - (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
 
@@ -111,6 +125,12 @@ linters:
       patterns:
         - ".*"
 
+    testifylint:
+      disable:
+        - encoded-compare # misidentifies non-JSON strings when identifiers contain "JSON"
+        - float-compare # false positives
+        - require-error # assert-on-error is deliberate where a test continues to collect further failures
+
   exclusions:
     presets:
       - std-error-handling

Makefile

@@ -123,7 +123,7 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness test-integration
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -133,6 +133,18 @@ test-harness:
 	@echo -e "\n....Running e2e harness unit tests...."
 	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
 
+# In-process integration tier: the real gateway router against embedded NATS
+# JetStream, with only the daemon-side NATS subjects stubbed. Build-tagged
+# `integration` so it's skipped by the default `go test ./spinifex/...` and by
+# `test-cover`/`test-race`. Nothing provisioned — no tofu, no docker, no
+# Spinifex daemons, so the whole package runs in well under a minute — part
+# of `preflight` (and its own PR-blocking CI step) rather than the
+# self-hosted, push-triggered live e2e tiers, so a regression here is caught
+# before it can be merged, not just after.
+test-integration:
+	@echo -e "\n....Running in-process integration tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
+
 # Validate docs/service-interfaces.yaml. Schema check + cross-reference
 # of services/suites/fixtures + on-disk path existence. Subject content
 # vs source is enforced separately in Bead 5 drift lint.
@@ -297,7 +309,7 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
+.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness test-integration manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
+  <img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
 </p>
 
 <p align="center">
@@ -11,6 +11,8 @@
 <p align="center">
   <a href="#what-is-spinifex">What is Spinifex?</a> ·
   <a href="#the-platform">Platform</a> ·
+  <a href="#three-ways-to-deploy">Deploy</a> ·
+  <a href="#aws-compatibility">AWS services</a> ·
   <a href="#core-components">Components</a> ·
   <a href="#architecture-at-a-glance">Architecture</a> ·
   <a href="#installation">Installation</a> ·
@@ -19,31 +21,90 @@
 
 ---
 
-# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
+# Spinifex: the open, AWS-compatible cloud you run yourself
 
-Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
+Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your
+software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control.
+Move workloads off the hyperscalers, run AI at a fraction of the cost, or take
+the cloud to the edge, without rewriting a line. The exact build you ship to AWS,
+Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
 
 ## What is Spinifex?
 
-Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
+Spinifex is an open-source infrastructure platform that recreates the AWS service
+surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises
+environments. Run cloud-native software without a hyperscaler.
 
-It's designed for developers and operators who need:
+Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering
+underneath: distributed object storage with erasure coding, block storage built
+to survive failure, and compute on bare metal. That depth is the reason your
+software runs unchanged, instead of rewritten.
 
-- Cloud-like infrastructure without the cloud
-- Minimal dependencies, full control
-- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
-- A secure cloud environment you control and own the entire hardware, network and software stack
+It's built for teams that need:
 
-You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
+- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
+- Full control of the stack: your hardware, your network, your data, your keys
+- A cloud that keeps running offline, through disconnection, with no external control plane
+- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
+
+You change *where* your software runs, not *what* it is.
 
 ## The Platform
 
 <p align="center">
-  <img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
+  <img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
 </p>
 
 From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
+## Three ways to deploy
+
+One AWS-compatible surface, three deployment shapes. Pick the one that matches
+your reality; the platform on top is the same.
+
+### Neocloud
+
+Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and
+onto our Neocloud partner ecosystem without rewriting them. GPU capacity
+(H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the
+software.
+
+### On-premise
+
+Bring your own hardware and host Spinifex in your own data centre. A real
+multi-node HA cluster integrated with your storage and networking, with full
+control of stack, data, and jurisdiction. A predictable bill, no egress
+surprises, an auditable open-source core.
+
+### Edge
+
+Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels,
+factories, and clinics. Compute next to the data, running through disconnection,
+on hardware you own. The same AWS APIs your software already uses.
+
+## AWS compatibility
+
+Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform:
+everything you deploy on AWS deploys on Spinifex unchanged. At the edge,
+on-premise, or on a partner Neocloud.
+
+| Service | What it is | Status |
+| --- | --- | --- |
+| **EC2** | Compute | Available |
+| **EBS** | Block storage | Available |
+| **S3** | Object storage | Available |
+| **VPC** | Networking | Available |
+| **IAM** | Identity & auth | Available |
+| **ALB / NLB** | Load balancers | Available |
+| **EKS** | Kubernetes | Available |
+| **ECR** | Container registry | Available |
+| **ECS** | Container service | Available |
+| **RDS** | Databases | Q3 2026 |
+
+Roadmap items (Q3 2026) ship under the same AWS API surface. Code written for AWS
+today keeps working the moment they land.
+[Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
+
 ## Core Components
 
 ### Spinifex (Compute Service – EC2 Alternative)
@@ -75,19 +136,23 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 ## Architecture at a Glance
 
 <p align="center">
-  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
+  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
 </p>
 
-Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
+Every AWS API call is authenticated at the gateway, published to a NATS subject,
+and answered by whichever daemon claims it. Daemons are stateless: scale
+horizontally by starting more. No etcd, no Kubernetes, no external control plane.
+Just systemd units, a NATS cluster, and your hardware. Deep dive:
+**[`docs/DESIGN.md`](docs/DESIGN.md)**.
 
 ## Key Features
 
-AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-
-- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-- Works Offline – No reliance on centralized cloud services or external networks.
-- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
+- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
+- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
+- **Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
+- **Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
+- **Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
+- **Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
 
 ## Installation
 

build/systemd/spinifex-northstar.service

@@ -0,0 +1,54 @@
+[Unit]
+Description=Spinifex Northstar (Authoritative DNS)
+PartOf=spinifex.target
+After=spinifex-nats.service spinifex-predastore.service
+Wants=spinifex-predastore.service
+
+[Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+Type=simple
+Slice=spinifex-system.slice
+User=spinifex-northstar
+Group=spinifex
+ExecStart=/usr/local/bin/spx service northstar start
+Restart=on-failure
+RestartSec=5
+OOMScoreAdjust=-500
+Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/northstar/
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/northstar /etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/northstar /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+# RG-9: exactly CAP_NET_BIND_SERVICE so the unprivileged user binds :53 (no root).
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+CapabilityBoundingSet=CAP_NET_BIND_SERVICE
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
+[Install]
+WantedBy=spinifex.target

build/systemd/spinifex.target

@@ -2,7 +2,7 @@
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

cmd/dhcptest/main.go

@@ -14,7 +14,7 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"log"
+	"log/slog"
 	"net"
 	"os"
 	"os/exec"
@@ -46,18 +46,21 @@ func main() {
 	case *spinifexID != "":
 		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
 		if err != nil {
-			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+			slog.Error("Could not parse generated MAC", "spinifex-id", *spinifexID, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
 	case *macStr != "":
 		hwAddr, err = net.ParseMAC(*macStr)
 		if err != nil {
-			log.Fatalf("parse --mac %q: %v", *macStr, err)
+			slog.Error("Could not parse --mac", "mac", *macStr, "error", err)
+			os.Exit(1)
 		}
 	default:
 		hwAddr, err = ifaceMAC(*iface)
 		if err != nil {
-			log.Fatalf("get MAC for %s: %v", *iface, err)
+			slog.Error("Could not get MAC for interface", "iface", *iface, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("using real interface MAC: %s\n", hwAddr)
 	}
@@ -83,7 +86,7 @@ func main() {
 
 	if *promisc {
 		if err := setPromisc(*iface, true); err != nil {
-			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+			slog.Warn("Failed to set promisc on interface", "iface", *iface, "error", err)
 		} else {
 			fmt.Printf("set %s promisc ON\n\n", *iface)
 		}
@@ -96,12 +99,13 @@ func main() {
 		nclient4.WithRetry(*retries),
 	}
 	if *verbose {
-		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: slogPrintfer{}}))
 	}
 
 	client, err := nclient4.New(*iface, opts...)
 	if err != nil {
-		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+		slog.Error("Could not create nclient4 client", "iface", *iface, "error", err)
+		os.Exit(1)
 	}
 	defer client.Close()
 
@@ -141,6 +145,14 @@ func main() {
 	}
 }
 
+// slogPrintfer adapts nclient4's Printfer to slog. nclient4 only emits through
+// it under --verbose, so the packet summaries log at info level to stay visible.
+type slogPrintfer struct{}
+
+func (slogPrintfer) Printf(format string, v ...any) {
+	slog.Info(fmt.Sprintf(format, v...), "component", "nclient4")
+}
+
 // generateMAC is the same deterministic hash used by vpcd's topology.go.
 // It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
 func generateMAC(resourceID string) string {

cmd/ecs-agent/credendpoint.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"net"
@@ -143,7 +144,7 @@ func (c *credEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	}
 	creds, roleARN, err := c.fetch(r.Context(), credID)
 	if err != nil {
-		if err == errUnknownCredID {
+		if errors.Is(err, errUnknownCredID) {
 			http.Error(w, "not found", http.StatusNotFound)
 			return
 		}

cmd/installer/install/install.go

@@ -539,7 +539,7 @@ func installBootloader(disk string) error {
 	); biosErr != nil {
 		if efiErr != nil {
 			// Both targets failed — the system will not boot.
-			return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
+			return fmt.Errorf("both bootloader targets failed (EFI: %w; BIOS: %w)", efiErr, biosErr)
 		}
 		return biosErr
 	}

cmd/spinifex/cmd/admin.go

@@ -60,13 +60,16 @@ var natsConfTemplate string
 //go:embed templates/predastore-multinode.toml
 var predastoreMultiNodeTemplate string
 
+//go:embed templates/northstar.toml
+var northstarTomlTemplate string
+
 var supportedArchs = map[string]bool{
 	"x86_64":  true,
 	"aarch64": true, // alias for arm64
 	"arm64":   true,
 }
 
-// TODO: Confirm suppported platform types
+// TODO: Confirm suppported platform types.
 var supportedPlatforms = map[string]bool{
 	"Linux/UNIX": true,
 	"Windows":    true,
@@ -334,7 +337,6 @@ func init() {
 	adminInitCmd.Flags().String("external-gateway", "", "WAN gateway IP (auto-detected from default route)")
 	adminInitCmd.Flags().String("gateway-ip", "", "OVN gateway router's external IP for SNAT (default: pool range_start for pool mode, required for nat mode without DHCP)")
 	adminInitCmd.Flags().Int("external-prefix-len", 24, "External pool subnet prefix length (auto-detected)")
-	adminInitCmd.Flags().Bool("no-external", false, "Disable external networking (overlay-only, no internet for VMs)")
 	adminInitCmd.Flags().Bool("gpu-passthrough", false, "Enable VFIO GPU passthrough (sets gpu_passthrough = true in daemon config)")
 	adminInitCmd.Flags().Bool("ipsec", true, "Encrypt intra-AZ Geneve via OVN native IPsec (cluster-wide); disable only for trusted single-rack lab")
 
@@ -386,6 +388,27 @@ func init() {
 	}
 }
 
+// amiVolumeSizeGiB returns the smallest whole GiB that still holds sizeBytes.
+//
+// Rounding up is load-bearing. The image is copied into a root volume of
+// exactly this size, so a volume smaller than the image truncates it and the
+// guest comes up with no root partition — it stalls on the root device until
+// systemd drops it to an emergency shell, and nothing on the way there reports
+// an undersized volume. Flooring (plain integer division) undersizes every
+// image that is not an exact multiple of a GiB, which is why this went unseen
+// while every system image happened to be a round 16 GiB.
+//
+// The downstream guard cannot cover for a wrong answer here: floorVolumeSizeToAMI
+// raises a caller's requested size to this value, so it inherits the mistake
+// rather than catching it.
+func amiVolumeSizeGiB(sizeBytes int64) uint64 {
+	const bytesPerGiB = 1024 * 1024 * 1024
+	if sizeBytes <= 0 {
+		return 0
+	}
+	return utils.SafeInt64ToUint64((sizeBytes + bytesPerGiB - 1) / bytesPerGiB)
+}
+
 func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	var image utils.Images
 
@@ -592,7 +615,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	manifest.AMIMetadata.RootDeviceType = "ebs"
 	manifest.AMIMetadata.Virtualization = "hvm"
 	manifest.AMIMetadata.ImageOwnerAlias = "system"
-	manifest.AMIMetadata.VolumeSizeGiB = utils.SafeInt64ToUint64(imageStat.Size() / 1024 / 1024 / 1024)
+	manifest.AMIMetadata.VolumeSizeGiB = amiVolumeSizeGiB(imageStat.Size())
 	manifest.AMIMetadata.BootMode = image.BootMode
 	manifest.AMIMetadata.Distro = image.Distro
 	manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
@@ -656,6 +679,12 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		os.Exit(1)
 	}
 
+	// Import is an interactive command whose output is a progress bar, so the
+	// volume's routine mount chatter is noise here: a fresh import has no prior
+	// state, making the "no state found" / 404 lines expected rather than
+	// notable. Scoping the logger to this VB (New copies it onto the backend
+	// too) keeps it off the process-wide default. Errors still surface, both
+	// through this logger and as the returned error the caller prints.
 	vbConfig := viperblock.VB{
 		VolumeName: volumeId,
 		VolumeSize: utils.SafeInt64ToUint64(imageStat.Size()),
@@ -668,6 +697,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		VolumeConfig:      manifest,
 		MasterKey:         mkey,
 		EncryptionEnabled: mkey != nil,
+		Logger:            slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError})),
 	}
 
 	// Bake the deployment CA into the image trust store so a stock cloud image
@@ -938,7 +968,7 @@ func runimagesPromoteCmd(cmd *cobra.Command, args []string) {
 	fmt.Printf("✅ Promoted %s to system image (owner: %s).\n", imageID, admin.SystemOwnerAlias)
 }
 
-// List remote images available
+// List remote images available.
 func runimagesListCmd(cmd *cobra.Command, args []string) {
 	//fmt.Println(availableImages)
 
@@ -965,7 +995,7 @@ func runimagesListCmd(cmd *cobra.Command, args []string) {
 	pterm.Println("spx admin images import --name <image-name>")
 }
 
-// TODO: Move all logic to a module, use minimal application logic in viper commands
+// TODO: Move all logic to a module, use minimal application logic in viper commands.
 func runAdminInit(cmd *cobra.Command, args []string) {
 	if os.Getuid() != 0 {
 		fmt.Fprintln(os.Stderr, "⚠️  Warning: 'spx admin init' is not running as root.")
@@ -1018,7 +1048,6 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	externalGateway, _ := cmd.Flags().GetString("external-gateway")
 	externalPrefixLen, _ := cmd.Flags().GetInt("external-prefix-len")
 	gatewayIP, _ := cmd.Flags().GetString("gateway-ip")
-	noExternal, _ := cmd.Flags().GetBool("no-external")
 	gpuPassthrough, _ := cmd.Flags().GetBool("gpu-passthrough")
 	ipsecEnabled, _ := cmd.Flags().GetBool("ipsec")
 
@@ -1051,59 +1080,56 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	// Auto-detect network topology
 	var poolStart, poolEnd string
 	var detectedNet *admin.DetectedNetwork
-	if !noExternal {
-		detected, err := admin.DetectNetwork()
-		if err != nil {
-			fmt.Fprintf(os.Stderr, "⚠️  Network auto-detection failed: %v\n", err)
-			fmt.Fprintf(os.Stderr, "   Use --no-external to skip, or specify flags manually.\n")
+	detected, err := admin.DetectNetwork()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "⚠️  Network auto-detection failed: %v\n", err)
+		fmt.Fprintf(os.Stderr, "   Use --external-mode=nat for outbound-only VMs on a non-bridgeable uplink, or specify --external-* flags manually.\n")
+	} else {
+		detectedNet = detected
+
+		// Print detected topology
+		fmt.Println("\n🔍 Detected network topology:")
+		fmt.Printf("  %-14s %-18s %-20s %-16s %s\n", "Interface", "IP", "Subnet", "Gateway", "Role")
+		for _, iface := range detected.Interfaces {
+			gw := "—"
+			if iface.Gateway != "" {
+				gw = iface.Gateway
+			}
+			fmt.Printf("  %-14s %-18s %-20s %-16s %s\n", iface.Name, iface.IP, iface.Subnet, gw, strings.ToUpper(iface.Role))
+		}
+		if detected.LANCount == 0 {
+			fmt.Println("\n  Mode: single-NIC (veth-bridged external)")
 		} else {
-			detectedNet = detected
+			fmt.Printf("\n  Mode: %d LAN + 1 WAN (veth-bridged external)\n", detected.LANCount)
+		}
 
-			// Print detected topology
-			fmt.Println("\n🔍 Detected network topology:")
-			fmt.Printf("  %-14s %-18s %-20s %-16s %s\n", "Interface", "IP", "Subnet", "Gateway", "Role")
-			for _, iface := range detected.Interfaces {
-				gw := "—"
-				if iface.Gateway != "" {
-					gw = iface.Gateway
-				}
-				fmt.Printf("  %-14s %-18s %-20s %-16s %s\n", iface.Name, iface.IP, iface.Subnet, gw, strings.ToUpper(iface.Role))
+		// Apply auto-detected values when flags not explicitly set
+		if detected.WAN != nil {
+			if externalIface == "" {
+				externalIface = detected.WAN.Name
 			}
-			if detected.LANCount == 0 {
-				fmt.Println("\n  Mode: single-NIC (veth-bridged external)")
-			} else {
-				fmt.Printf("\n  Mode: %d LAN + 1 WAN (veth-bridged external)\n", detected.LANCount)
+			if externalGateway == "" {
+				externalGateway = detected.WAN.Gateway
+			}
+			if !cmd.Flags().Changed("external-prefix-len") {
+				externalPrefixLen = detected.WAN.PrefixLen
 			}
 
-			// Apply auto-detected values when flags not explicitly set
-			if detected.WAN != nil {
-				if externalIface == "" {
-					externalIface = detected.WAN.Name
-				}
-				if externalGateway == "" {
-					externalGateway = detected.WAN.Gateway
-				}
-				if !cmd.Flags().Changed("external-prefix-len") {
-					externalPrefixLen = detected.WAN.PrefixLen
-				}
-
-				// Default mode: always "pool". Source defaults to "static"; if
-				// --external-pool is omitted the validator below will error with
-				// a SuggestPoolRange hint.
-				if externalMode == "" && !cmd.Flags().Changed("external-mode") {
-					if isNonBridgeableUplink(detected.WAN.Name) {
-						fmt.Fprintf(os.Stderr, "\n❌ Detected WAN interface %s cannot be bridged (WiFi/cellular/PPP).\n", detected.WAN.Name)
-						fmt.Fprintf(os.Stderr, "   Use routed NAT mode instead (outbound-only VM networking):\n")
-						fmt.Fprintf(os.Stderr, "     ./scripts/setup-ovn.sh --management --nat-uplink\n")
-						fmt.Fprintf(os.Stderr, "     spx admin init --external-mode=nat\n")
-						os.Exit(1)
-					}
-					externalMode = "pool"
+			// Default mode: always "pool". Source defaults to "static"; if
+			// --external-pool is omitted the validator below will error with
+			// a SuggestPoolRange hint.
+			if externalMode == "" && !cmd.Flags().Changed("external-mode") {
+				if isNonBridgeableUplink(detected.WAN.Name) {
+					fmt.Fprintf(os.Stderr, "\n❌ Detected WAN interface %s cannot be bridged (WiFi/cellular/PPP).\n", detected.WAN.Name)
+					fmt.Fprintf(os.Stderr, "   Use routed NAT mode instead (outbound-only VM networking):\n")
+					fmt.Fprintf(os.Stderr, "     ./scripts/setup-ovn.sh --management --nat-uplink\n")
+					fmt.Fprintf(os.Stderr, "     spx admin init --external-mode=nat\n")
+					os.Exit(1)
 				}
+				externalMode = "pool"
 			}
 		}
 	}
-
 	// Validate external networking flags
 	if externalMode != "" && externalMode != "pool" && externalMode != "nat" {
 		fmt.Fprintf(os.Stderr, "❌ Error: --external-mode must be 'pool', 'nat', or empty, got: %s\n", externalMode)
@@ -1209,8 +1235,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		os.Exit(1)
 	}
 
-	// Resolve the off-host advertise IP. DetectNetwork may have been skipped
-	// when --no-external is set; detect lazily if we need the WAN IP.
+	// Resolve the off-host advertise IP. DetectNetwork may have failed earlier;
+	// detect lazily if we still need the WAN IP.
 	if advertiseFlag == "" && (bindIP == "0.0.0.0" || bindIP == "127.0.0.1") && detectedNet == nil {
 		if d, derr := admin.DetectNetwork(); derr == nil {
 			detectedNet = d
@@ -1384,6 +1410,29 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	}
 	spxRoot = filepath.Clean(spxRoot)
 
+	// Generate dedicated, bucket-scoped credentials for the northstar DNS
+	// service. Rendered into predastore.toml ([[auth]]) and northstar.toml so
+	// the resolver reads zone files read-only from its own S3 bucket.
+	//
+	// Generated above the multi-node dispatch because the pair is cluster-wide:
+	// a node's predastore only honours the keys rendered into its own config, so
+	// every node must present this same pair to read the distributed zone bucket.
+	northstarAccessKey, err := admin.GenerateAWSAccessKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating northstar access key: %v\n", err)
+		os.Exit(1)
+	}
+	northstarSecretKey, err := admin.GenerateAWSSecretKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating northstar secret key: %v\n", err)
+		os.Exit(1)
+	}
+	northstarCreds := admin.NorthstarCredentials{
+		AccessKey: northstarAccessKey,
+		SecretKey: northstarSecretKey,
+		Bucket:    admin.NorthstarBucketName,
+	}
+
 	// Determine if this is a multi-node formation. Operator intent comes from
 	// --nodes, not from whether --bind was left at the 0.0.0.0 default.
 	isMultiNode := nodes >= 2
@@ -1420,7 +1469,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		runAdminInitMultiNode(cmd, accessKey, secretKey, accountID, adminAccessKey, adminSecretKey,
 			masterKey, viperblockKey, natsToken, clusterName,
 			configDir, spxRoot, certPath, region, az, node, bindIP, advertiseIP, clusterBind, email,
-			port, nodes, formationTimeoutStr, tokenTTLStr, services, networkConfig)
+			port, nodes, formationTimeoutStr, tokenTTLStr, services, networkConfig, northstarCreds)
 		return
 	}
 
@@ -1437,6 +1486,10 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 	portStr := strconv.Itoa(port)
 
+	// The keys are cluster-wide and generated above the dispatch; the config path
+	// is node-local, so it is derived here from this node's own config dirs.
+	northstarConfigPath := filepath.Join(dirs.Northstar, "northstar.toml")
+
 	// Parse multi-node predastore configuration (legacy flag-based approach for single-node)
 	var predastoreNodeID int
 	if predastoreNodesStr != "" {
@@ -1466,7 +1519,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		}
 
 		// Generate multi-node predastore.toml
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval, northstarCreds)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1531,6 +1584,14 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		IPSecEnabled:   ipsecEnabled,
 
 		EncryptionKeyFile: viperblockKeyPath,
+
+		NorthstarAccessKey:      northstarCreds.AccessKey,
+		NorthstarSecretKey:      northstarCreds.SecretKey,
+		NorthstarBucket:         northstarCreds.Bucket,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+		NorthstarConfigPath:     northstarConfigPath,
+		PoolDNSServers:          dnsServers,
 	}
 
 	// Print external networking summary
@@ -1601,10 +1662,15 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 // runAdminInitMultiNode handles the multi-node formation path for admin init.
 // It starts a formation server, registers this node, waits for all nodes to join,
 // then generates configs with complete cluster topology.
+//
+// The northstar credentials are generated by the caller and provision this
+// node's predastore with the zone bucket. The same pair is distributed to
+// joiners, since each node's predastore only honours the keys in its own config.
 func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID, adminAccessKey, adminSecretKey string,
 	masterKey, viperblockKey []byte, natsToken, clusterName,
 	configDir, spxRoot, certPath, region, az, node, bindIP, advertiseIP, clusterBind, email string,
-	port, expectedNodes int, formationTimeoutStr, tokenTTLStr string, services []string, networkConfig *formation.NetworkConfig) {
+	port, expectedNodes int, formationTimeoutStr, tokenTTLStr string, services []string, networkConfig *formation.NetworkConfig,
+	northstarCreds admin.NorthstarCredentials) {
 	formationTimeout, err := time.ParseDuration(formationTimeoutStr)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "❌ Error: Invalid --formation-timeout: %v\n", err)
@@ -1656,6 +1722,12 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		Region:         region,
 		AdminAccessKey: adminAccessKey,
 		AdminSecretKey: adminSecretKey,
+
+		// Joiners provision their own predastore with this pair, so every
+		// node's resolver can read the distributed zone bucket via its local
+		// endpoint. The bucket is a constant, so it is derived node-side.
+		NorthstarAccessKey: northstarCreds.AccessKey,
+		NorthstarSecretKey: northstarCreds.SecretKey,
 	}
 
 	fs := formation.NewFormationServer(expectedNodes, creds, string(caCertData), string(caKeyData), networkConfig, joinToken, tokenTTL)
@@ -1727,11 +1799,15 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 	portStr := strconv.Itoa(port)
 
+	// The keys are cluster-wide and generated above the dispatch; the config path
+	// is node-local, so it is derived here from this node's own config dirs.
+	northstarConfigPath := filepath.Join(dirs.Northstar, "northstar.toml")
+
 	// Generate multi-node predastore config
 	var predastoreNodeID int
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval, northstarCreds)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1771,12 +1847,19 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		PredastoreNodeID:          predastoreNodeID,
 		CompactionIntervalSeconds: compactionInterval,
 		Services:                  services,
-		RemoteNodes:               buildRemoteNodes(allNodes, node),
+		RemoteNodes:               buildRemoteNodes(allNodes, node, northstarConfigPath),
 
 		OperatorEmail: email,
 
 		EncryptionKeyFile: viperblockKeyPath,
 
+		NorthstarAccessKey:      northstarCreds.AccessKey,
+		NorthstarSecretKey:      northstarCreds.SecretKey,
+		NorthstarBucket:         northstarCreds.Bucket,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+		NorthstarConfigPath:     northstarConfigPath,
+
 		// Multi-endpoint OVN NB/SB list across the RAFT quorum; the init node's
 		// own address leads, the rest provide failover.
 		OVNNBAddr: ovnNBAddr,
@@ -2179,12 +2262,15 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 	portStr := strconv.Itoa(port)
 
+	northstarCreds, northstarConfigPath := northstarFromFormation(creds, dirs)
+
 	// Generate multi-node predastore config
 	var predastoreNodeID int
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP, compactionInterval)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP, compactionInterval,
+			northstarCreds)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -2228,12 +2314,19 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		PredastoreNodeID:          predastoreNodeID,
 		CompactionIntervalSeconds: compactionInterval,
 		Services:                  services,
-		RemoteNodes:               buildRemoteNodes(statusResp.Nodes, node),
+		RemoteNodes:               buildRemoteNodes(statusResp.Nodes, node, northstarConfigPath),
 
 		OperatorEmail: email,
 
 		EncryptionKeyFile: viperblockKeyPath,
 
+		NorthstarAccessKey:      northstarCreds.AccessKey,
+		NorthstarSecretKey:      northstarCreds.SecretKey,
+		NorthstarBucket:         northstarCreds.Bucket,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+		NorthstarConfigPath:     northstarConfigPath,
+
 		// Multi-endpoint OVN NB/SB list across the RAFT quorum so the client
 		// fails over instead of pinning to a single init node.
 		OVNNBAddr: ovnNBAddr,
@@ -2290,7 +2383,16 @@ func resolveAdvertiseIP(bindIP, advertiseFlag string, detected *admin.DetectedNe
 // buildRemoteNodes converts formation NodeInfo into RemoteNode entries,
 // excluding the local node. This puts all cluster members into spinifex.toml
 // so config is the source of truth for expected cluster membership.
-func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode string) []admin.RemoteNode {
+//
+// northstarConfigPath is the local node's own path, republished for every peer:
+// every node in a formed cluster runs northstar, and the seed set must be
+// identical on all of them or the base zone's NS records get pinned to whichever
+// node wins the create-if-absent race. --config-dir is per-node and does not
+// cross the wire, so the value is accurate whenever nodes share a config dir and
+// inert when they do not — only its emptiness is ever read back. Passing it
+// empty (no credentials distributed) leaves peers with no stanza, so no node is
+// advertised as a resolver it cannot be.
+func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode, northstarConfigPath string) []admin.RemoteNode {
 	var remote []admin.RemoteNode
 	for name, n := range allNodes {
 		if name == localNode {
@@ -2303,11 +2405,12 @@ func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode string)
 			host = n.BindIP
 		}
 		remote = append(remote, admin.RemoteNode{
-			Name:     name,
-			Host:     host,
-			Region:   n.Region,
-			AZ:       n.AZ,
-			Services: n.Services,
+			Name:                name,
+			Host:                host,
+			Region:              n.Region,
+			AZ:                  n.AZ,
+			Services:            n.Services,
+			NorthstarConfigPath: northstarConfigPath,
 		})
 	}
 	sort.Slice(remote, func(i, j int) bool {
@@ -2523,6 +2626,7 @@ type configDirs struct {
 	Viperblock string
 	NATS       string
 	Spinifex   string
+	Northstar  string
 }
 
 // createConfigSubdirs creates the standard config subdirectories under configDir.
@@ -2533,8 +2637,9 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
+		Northstar:  filepath.Join(configDir, "northstar"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex, dirs.Northstar} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2542,14 +2647,55 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	return dirs, nil
 }
 
+// northstarFromFormation derives a joining node's northstar credentials and
+// node-local config path from the cluster-wide pair distributed at formation.
+//
+// The keys cross the wire because a node's predastore only honours the keys in
+// its own config, so every node must present the same pair to read the
+// distributed zone bucket. The bucket name and config path are node-local
+// constants, so they are derived here rather than carried.
+//
+// A leader that predates credential distribution sends no keys. That yields a
+// zero pair and an empty path, so the node renders no northstar config at all
+// rather than a resolver holding a key its own predastore would reject.
+func northstarFromFormation(creds *formation.SharedCredentials, dirs configDirs) (admin.NorthstarCredentials, string) {
+	if creds.NorthstarAccessKey == "" || creds.NorthstarSecretKey == "" {
+		return admin.NorthstarCredentials{}, ""
+	}
+	return admin.NorthstarCredentials{
+		AccessKey: creds.NorthstarAccessKey,
+		SecretKey: creds.NorthstarSecretKey,
+		Bucket:    admin.NorthstarBucketName,
+	}, filepath.Join(dirs.Northstar, "northstar.toml")
+}
+
 // generateAndWriteConfigs renders the standard config files (spinifex.toml,
 // awsgw.toml, nats.conf, and optionally predastore.toml) from templates.
 func generateAndWriteConfigs(dirs configDirs, spinifexTomlPath string, settings admin.ConfigSettings, skipPredastore bool) error {
+	// A one-sided pair must disable Northstar wholesale. Rendering only the
+	// public stanza or only the secret file advertises a resolver that cannot run.
+	northstarEnabled := settings.NorthstarAccessKey != "" && settings.NorthstarSecretKey != ""
+	if !northstarEnabled {
+		settings.NorthstarAccessKey = ""
+		settings.NorthstarSecretKey = ""
+		settings.NorthstarBucket = ""
+		settings.NorthstarConfigPath = ""
+	}
+
 	configs := []admin.ConfigFile{
 		{Name: "spinifex.toml", Path: spinifexTomlPath, Template: spinifexTomlTemplate},
 		{Name: filepath.Join(dirs.AWSGW, "awsgw.toml"), Path: filepath.Join(dirs.AWSGW, "awsgw.toml"), Template: awsgwTomlTemplate},
 		{Name: filepath.Join(dirs.NATS, "nats.conf"), Path: filepath.Join(dirs.NATS, "nats.conf"), Template: natsConfTemplate},
 	}
+	// northstar.toml is only rendered when scoped DNS credentials were
+	// provisioned. A cluster formed by a leader that predates their distribution
+	// leaves the keys empty, yielding no northstar config rather than a partial
+	// one pointing at a bucket the local predastore does not serve.
+	if northstarEnabled {
+		configs = append(configs, admin.ConfigFile{
+			Name: filepath.Join(dirs.Northstar, "northstar.toml"), Path: filepath.Join(dirs.Northstar, "northstar.toml"), Template: northstarTomlTemplate,
+		})
+	}
 	if !skipPredastore {
 		configs = append(configs, admin.ConfigFile{
 			Name: filepath.Join(dirs.Predastore, "predastore.toml"), Path: filepath.Join(dirs.Predastore, "predastore.toml"), Template: predastoreTomlTemplate,
@@ -2580,6 +2726,7 @@ func finalizeNodeSetup(dataDir, certPath, adminAccessKey, adminSecretKey, region
 func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkConfig) {
 	settings.IPSecEnabled = nc.IPSecEnabled
 	settings.ExternalMode = nc.ExternalMode
+	settings.PoolDNSServers = nc.PoolDNSServers
 	if nc.ExternalMode != "" {
 		settings.Pools = []admin.PoolData{{
 			Name:       nc.PoolName,
@@ -2913,7 +3060,7 @@ func detectDNSServers(iface string) []string {
 }
 
 // parseDNSFromResolvectl extracts IP addresses from resolvectl dns output.
-// Format: "Link 2 (enp0s3): 192.168.1.1 8.8.8.8 1.1.1.1"
+// Format: "Link 2 (enp0s3): 192.168.1.1 8.8.8.8 1.1.1.1".
 func parseDNSFromResolvectl(output string) []string {
 	var servers []string
 	for line := range strings.SplitSeq(output, "\n") {

cmd/spinifex/cmd/admin_nat_test.go

@@ -138,8 +138,8 @@ func TestIsNonBridgeableUplink(t *testing.T) {
 
 func TestBridgeModeFor(t *testing.T) {
 	assert.Equal(t, "nat", bridgeModeFor("nat"))
-	assert.Equal(t, "", bridgeModeFor("pool"))
-	assert.Equal(t, "", bridgeModeFor(""))
+	assert.Empty(t, bridgeModeFor("pool"))
+	assert.Empty(t, bridgeModeFor(""))
 }
 
 func TestSpinifexTomlTemplate_NATModeWithPublicPool(t *testing.T) {

cmd/spinifex/cmd/admin_test.go

@@ -10,7 +10,9 @@ import (
 	"testing"
 
 	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/formation"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	toml "github.com/pelletier/go-toml/v2"
@@ -51,7 +53,7 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 		"node2": {Name: "node2", BindIP: "10.0.0.2"}, // legacy joiner
 		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "203.0.113.3"},
 	}
-	got := buildRemoteNodes(nodes, "node3")
+	got := buildRemoteNodes(nodes, "node3", "")
 	if assert.Len(t, got, 2) {
 		assert.Equal(t, "node1", got[0].Name)
 		assert.Equal(t, "203.0.113.1", got[0].Host, "advertise wins when set")
@@ -60,6 +62,29 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 	}
 }
 
+// Every peer must carry the northstar stanza, since a node that describes only
+// its own northstar computes a single-entry seed set and pins the base zone's NS
+// records to whichever node won the create-if-absent race.
+func TestBuildRemoteNodes_NorthstarConfigPath(t *testing.T) {
+	nodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2"},
+	}
+
+	got := buildRemoteNodes(nodes, "node2", "/etc/spinifex/northstar/northstar.toml")
+	if assert.Len(t, got, 1) {
+		assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", got[0].NorthstarConfigPath)
+	}
+
+	// No distributed credentials means no node runs northstar. Publishing peers
+	// as resolvers anyway would point guests at 169.254.169.253 for a service
+	// that never starts.
+	got = buildRemoteNodes(nodes, "node2", "")
+	if assert.Len(t, got, 1) {
+		assert.Empty(t, got[0].NorthstarConfigPath)
+	}
+}
+
 func TestResolveAdvertiseIP(t *testing.T) {
 	wan := &admin.DetectedNetwork{WAN: &admin.DetectedInterface{IP: "192.168.1.21"}}
 	noWAN := &admin.DetectedNetwork{}
@@ -146,6 +171,44 @@ func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
 	assert.NotContains(t, string(data), "advertise =")
 }
 
+// The northstar stanza carries the non-secret domains so confined services
+// (vpcd) resolve DNS names without reading the 0600 northstar.toml.
+func TestSpinifexTomlTemplate_NorthstarDomains(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:                    "node1",
+		Az:                      "ap-southeast-2a",
+		Port:                    "4432",
+		Region:                  "ap-southeast-2",
+		BindIP:                  "10.11.12.1",
+		AccessKey:               "AKIATEST",
+		SecretKey:               "SECRET",
+		AccountID:               "123456789012",
+		NatsToken:               "token",
+		ConfigDir:               dir,
+		OVNNBAddr:               "tcp:127.0.0.1:6641",
+		OVNSBAddr:               "tcp:127.0.0.1:6642",
+		NorthstarConfigPath:     "/etc/spinifex/northstar/northstar.toml",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `config_path = "/etc/spinifex/northstar/northstar.toml"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+
+	// The stanza must round-trip into NorthstarConfig so the resolvers read it.
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	assert.Equal(t, "spx3.net", cfg.Nodes["node1"].Northstar.DefaultDomain)
+	assert.Equal(t, "compute.internal", cfg.Nodes["node1"].Northstar.InternalDomain)
+}
+
 // Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
 // alias (per mulga-998 D3). Prevents the footgun where operators inherited the
 // old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
@@ -210,7 +273,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 		{ID: 3, Host: "10.0.0.3"},
 	}
 	content, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
@@ -218,7 +281,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 
 	// Specific bind IP → stays as-is.
 	content2, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
@@ -524,15 +587,15 @@ const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_f
 func TestAdminInitCmd_ExternalDHCPFlagSchema(t *testing.T) {
 	sourceFlag := adminInitCmd.Flags().Lookup("external-source")
 	require.NotNil(t, sourceFlag, "--external-source must be defined")
-	assert.Equal(t, "", sourceFlag.DefValue)
+	assert.Empty(t, sourceFlag.DefValue)
 
 	bindFlag := adminInitCmd.Flags().Lookup("external-bind-bridge")
 	require.NotNil(t, bindFlag, "--external-bind-bridge must be defined")
-	assert.Equal(t, "", bindFlag.DefValue)
+	assert.Empty(t, bindFlag.DefValue)
 
 	poolFlag := adminInitCmd.Flags().Lookup("external-pool")
 	require.NotNil(t, poolFlag)
-	assert.Equal(t, "", poolFlag.DefValue)
+	assert.Empty(t, poolFlag.DefValue)
 }
 
 // DHCP-sourced pool must emit source + bind_bridge into spinifex.toml so the
@@ -631,6 +694,297 @@ func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
 	}
 }
 
+// predastoreBucketAuth parses the buckets and auth entries a northstar-enabled
+// multi-node predastore must render.
+type predastoreBucketAuth struct {
+	Buckets []struct {
+		Name string `toml:"name"`
+		Type string `toml:"type"`
+	} `toml:"buckets"`
+	Auth []struct {
+		AccessKeyID     string `toml:"access_key_id"`
+		SecretAccessKey string `toml:"secret_access_key"`
+		Policy          []struct {
+			Bucket  string   `toml:"bucket"`
+			Actions []string `toml:"actions"`
+		} `toml:"policy"`
+	} `toml:"auth"`
+}
+
+// policyFor returns the actions the given key is granted on the given bucket,
+// and whether the key holds any policy for it at all.
+func (c predastoreBucketAuth) policyFor(accessKey, bucket string) ([]string, bool) {
+	for _, auth := range c.Auth {
+		if auth.AccessKeyID != accessKey {
+			continue
+		}
+		for _, p := range auth.Policy {
+			if p.Bucket == bucket {
+				return p.Actions, true
+			}
+		}
+	}
+	return nil, false
+}
+
+func (c predastoreBucketAuth) hasBucket(name string) bool {
+	for _, b := range c.Buckets {
+		if b.Name == name {
+			return true
+		}
+	}
+	return false
+}
+
+// Northstar on a multi-node cluster needs three things from predastore, all
+// gated on the same credential: the zone bucket, write access for the system
+// key (which is what BootstrapBaseZone seeds zones with), and the resolver's
+// own read-only key. The template rendered all three correctly, but the caller
+// never populated the credential, so multi-node clusters silently got none of
+// them.
+func TestPredastoreMultinodeTemplate_NorthstarProvisioned(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{AccessKey: "NSAK", SecretKey: "NSSK", Bucket: admin.NorthstarBucketName},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.True(t, cfg.hasBucket(admin.NorthstarBucketName), "northstar zone bucket not rendered")
+
+	// The zone bucket must be readable from every node's local predastore, so
+	// a resolver on any node can serve it.
+	for _, b := range cfg.Buckets {
+		if b.Name == admin.NorthstarBucketName {
+			assert.Equal(t, "distributed", b.Type)
+		}
+	}
+
+	// BootstrapBaseZone writes zone files with the system key, not the
+	// resolver's. Without this grant, seeding fails on every node.
+	systemActions, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	require.True(t, ok, "system key has no policy for the northstar bucket")
+	assert.Contains(t, systemActions, "s3:GetObject", "bootstrap must check whether the zone already exists")
+	assert.Contains(t, systemActions, "s3:PutObject", "bootstrap must create a missing zone")
+
+	// The resolver's key stays read-only and scoped to the zone bucket alone.
+	nsActions, ok := cfg.policyFor("NSAK", admin.NorthstarBucketName)
+	require.True(t, ok, "northstar key has no policy for the northstar bucket")
+	assert.ElementsMatch(t, []string{"s3:ListBucket", "s3:GetObject"}, nsActions)
+	for _, auth := range cfg.Auth {
+		if auth.AccessKeyID == "NSAK" {
+			assert.Equal(t, "NSSK", auth.SecretAccessKey)
+		}
+	}
+
+	_, ok = cfg.policyFor("NSAK", "predastore")
+	assert.False(t, ok, "northstar key must not reach the predastore bucket")
+}
+
+// Without a credential the northstar stanzas must be omitted wholesale rather
+// than half-rendered: an empty-keyed [[auth]] entry would let anyone read the
+// zone bucket.
+func TestPredastoreMultinodeTemplate_NorthstarOmittedWithoutCredentials(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.False(t, cfg.hasBucket(admin.NorthstarBucketName))
+	_, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	assert.False(t, ok, "system key granted northstar access without a northstar credential")
+	assert.Len(t, cfg.Auth, 1, "only the system key should be rendered")
+}
+
+// A joiner must provision its predastore with the leader's pair verbatim, since
+// each node's predastore only honours the keys in its own config. Per-node
+// generation would have node A's predastore reject node B's resolver.
+func TestNorthstarFromFormation_UsesDistributedPair(t *testing.T) {
+	dirs := configDirs{Northstar: "/etc/spinifex/northstar"}
+	creds := &formation.SharedCredentials{NorthstarAccessKey: "NSAK", NorthstarSecretKey: "NSSK"}
+
+	got, path := northstarFromFormation(creds, dirs)
+
+	assert.Equal(t, "NSAK", got.AccessKey)
+	assert.Equal(t, "NSSK", got.SecretKey)
+	assert.Equal(t, admin.NorthstarBucketName, got.Bucket, "bucket is a constant, derived node-side")
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", path)
+}
+
+// A leader predating credential distribution, or returning only half a pair,
+// must yield no Northstar config. Advertising a resolver without usable keys
+// would send guests to a service that cannot read its zones.
+func TestNorthstarFromFormation_IncompletePairYieldsNoConfig(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "legacy leader"},
+		{name: "access key only", access: "NSAK"},
+		{name: "secret key only", secret: "NSSK"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, path := northstarFromFormation(&formation.SharedCredentials{
+				NorthstarAccessKey: tt.access,
+				NorthstarSecretKey: tt.secret,
+			}, configDirs{Northstar: "/etc/spinifex/northstar"})
+
+			assert.Equal(t, admin.NorthstarCredentials{}, got)
+			assert.Empty(t, path, "config path without credentials advertises a resolver that cannot read its zones")
+		})
+	}
+}
+
+// multinodeNorthstarSettings mirrors what both formation paths hand
+// generateAndWriteConfigs once the distributed pair is wired through.
+func multinodeNorthstarSettings(configPath string) admin.ConfigSettings {
+	s := northstarSettings()
+	s.Node = "node1"
+	s.Az = "ap-southeast-2a"
+	s.Port = "4432"
+	s.AccountID = "123456789012"
+	s.OVNNBAddr = "tcp:127.0.0.1:6641"
+	s.OVNSBAddr = "tcp:127.0.0.1:6642"
+	s.NorthstarConfigPath = configPath
+	return s
+}
+
+// The multi-node formation paths must produce both halves of a working
+// resolver: the northstar.toml the service reads, and the spinifex.toml stanza
+// that makes configuredNorthstarNodeNames count this node. Rendering was
+// previously reachable only from the single-node path.
+func TestGenerateAndWriteConfigs_MultinodeRendersNorthstar(t *testing.T) {
+	dirs, err := createConfigSubdirs(t.TempDir())
+	require.NoError(t, err)
+	nsPath := filepath.Join(dirs.Northstar, "northstar.toml")
+	spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+	require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, multinodeNorthstarSettings(nsPath), true))
+
+	nsContent, err := os.ReadFile(nsPath)
+	require.NoError(t, err)
+	assert.Contains(t, string(nsContent), `access_key = "AKIANORTHSTAR"`)
+
+	// config_path is the sole predicate for "this node runs northstar", so its
+	// presence is what puts the node in the nameserver set and the resolver list.
+	cfg, err := config.LoadConfig(spinifexTomlPath)
+	require.NoError(t, err)
+	assert.Equal(t, nsPath, cfg.Nodes["node1"].Northstar.ConfigPath)
+}
+
+// Incomplete credentials must yield neither file nor stanza. A stanza without
+// a usable northstar.toml would point guests at a resolver that never starts.
+func TestGenerateAndWriteConfigs_NoNorthstarWithoutCompleteCredentials(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "no credentials"},
+		{name: "access key only", access: "AKIANORTHSTAR"},
+		{name: "secret key only", secret: "northstar-secret"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			dirs, err := createConfigSubdirs(t.TempDir())
+			require.NoError(t, err)
+			spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+			settings := multinodeNorthstarSettings(filepath.Join(dirs.Northstar, "northstar.toml"))
+			settings.NorthstarAccessKey = tt.access
+			settings.NorthstarSecretKey = tt.secret
+			require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, settings, false))
+
+			_, err = os.Stat(filepath.Join(dirs.Northstar, "northstar.toml"))
+			assert.True(t, os.IsNotExist(err), "northstar.toml rendered without complete credentials")
+
+			cfg, err := config.LoadConfig(spinifexTomlPath)
+			require.NoError(t, err)
+			assert.Empty(t, cfg.Nodes["node1"].Northstar.ConfigPath)
+
+			predastoreContent, err := os.ReadFile(filepath.Join(dirs.Predastore, "predastore.toml"))
+			require.NoError(t, err)
+			var predastoreCfg predastoreBucketAuth
+			require.NoError(t, toml.Unmarshal(predastoreContent, &predastoreCfg))
+			assert.False(t, predastoreCfg.hasBucket(admin.NorthstarBucketName))
+			assert.Len(t, predastoreCfg.Auth, 1, "incomplete pair rendered a Northstar auth entry")
+		})
+	}
+}
+
+// renderClusterNode renders one node's spinifex.toml as its own formation path
+// would: its local northstar stanza plus a stanza for every peer.
+func renderClusterNode(t *testing.T, node string, allNodes map[string]formation.NodeInfo) *config.ClusterConfig {
+	t.Helper()
+	const nsPath = "/etc/spinifex/northstar/northstar.toml"
+
+	settings := multinodeNorthstarSettings(nsPath)
+	settings.Node = node
+	settings.BindIP = allNodes[node].BindIP
+	settings.AdvertiseIP = allNodes[node].AdvertiseIP
+	settings.RemoteNodes = buildRemoteNodes(allNodes, node, nsPath)
+
+	path := filepath.Join(t.TempDir(), "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	return cfg
+}
+
+// Every node must derive the same nameserver set from its own rendered config.
+// EnsureBaseZone is create-if-absent and never overwrites, so if each node saw
+// only itself the zone's NS records would be pinned to whichever node won the
+// startup race — permanently, for the life of the zone object. Identical sets
+// make that race benign: whoever writes, writes the same bytes.
+func TestSpinifexTomlTemplate_PeerNorthstarSeedsAreUniform(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "192.168.1.33"},
+	}
+
+	wantIPs := []string{"192.168.1.31", "192.168.1.32", "192.168.1.33"}
+	wantHosts := []string{"ns1", "ns2", "ns3"}
+	for _, node := range []string{"node1", "node2", "node3"} {
+		cfg := renderClusterNode(t, node, allNodes)
+		seeds := handlers_dns.NameserverSeeds(cfg)
+		require.Len(t, seeds, 3)
+		for i, seed := range seeds {
+			assert.Equal(t, wantHosts[i], seed.Host)
+			assert.Equal(t, wantIPs[i], seed.IP)
+		}
+		assert.Equal(t, wantIPs, handlers_dns.ResolverNameserverIPs(cfg),
+			"%s derived a different nameserver set from its own config", node)
+	}
+}
+
+// A peer's stanza carries config_path alone: it is the only field ever read back
+// from a peer, and the domains would be fiction — the local node cannot know
+// what a peer configured.
+func TestSpinifexTomlTemplate_PeerNorthstarStanzaIsPathOnly(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+	}
+
+	cfg := renderClusterNode(t, "node1", allNodes)
+	peer := cfg.Nodes["node2"].Northstar
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", peer.ConfigPath)
+	assert.Empty(t, peer.DefaultDomain, "peer domains are unknowable from here")
+	assert.Empty(t, peer.InternalDomain)
+}
+
 // Unset knob must leave production config byte-identical: no [compaction] block
 // and the original trailing bytes unchanged (single-node has no trailing
 // newline, multinode keeps one).
@@ -644,7 +998,7 @@ func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
 		"unset single-node tail changed: %q", single[len(single)-40:])
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.NotContains(t, multi, "[compaction]")
@@ -665,7 +1019,7 @@ func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
 	assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	var multiCfg compactionConfig
@@ -702,3 +1056,40 @@ func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
 	})
 	assert.Contains(t, out, "interval_seconds = 45")
 }
+
+// The image is written into a root volume of exactly the size this returns, so
+// anything less than the image size truncates the image and the guest never
+// finds its root. The old flooring conversion undersized every non-round image.
+func TestAMIVolumeSizeGiB(t *testing.T) {
+	const giB int64 = 1024 * 1024 * 1024
+
+	tests := []struct {
+		name  string
+		bytes int64
+		want  uint64
+	}{
+		// The regression: a real 4.94 GiB mkosi image floored to 4 GiB and the
+		// guest hung with no root partition.
+		{name: "non-round image rounds up", bytes: 5306470400, want: 5},
+		// Exact multiples must not gain a spare GiB — the round sizes are the
+		// common case and were the only ones the old code got right.
+		{name: "exact multiple is unchanged", bytes: 16 * giB, want: 16},
+		{name: "one byte over a multiple rounds up", bytes: 16*giB + 1, want: 17},
+		{name: "one byte under a multiple rounds up", bytes: 16*giB - 1, want: 16},
+		// Sub-GiB images still need a whole GiB to live in.
+		{name: "sub-GiB image gets a full GiB", bytes: 1, want: 1},
+		{name: "empty image", bytes: 0, want: 0},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := amiVolumeSizeGiB(tt.bytes)
+			assert.Equal(t, tt.want, got)
+			// The invariant the guest depends on, stated directly.
+			if tt.bytes > 0 {
+				assert.GreaterOrEqual(t, int64(got)*giB, tt.bytes,
+					"volume must be large enough to hold the image")
+			}
+		})
+	}
+}

cmd/spinifex/cmd/northstar_template_test.go

@@ -0,0 +1,89 @@
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	toml "github.com/pelletier/go-toml/v2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func renderTemplate(t *testing.T, tmpl string, settings admin.ConfigSettings) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "out.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, tmpl, settings))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	var parsed map[string]any
+	require.NoError(t, toml.Unmarshal(data, &parsed), "rendered template must be valid TOML")
+	return string(data)
+}
+
+func northstarSettings() admin.ConfigSettings {
+	return admin.ConfigSettings{
+		Region:                  "ap-southeast-2",
+		BindIP:                  "0.0.0.0",
+		AdvertiseIP:             "192.168.1.31",
+		AccessKey:               "AKIASYSTEM",
+		SecretKey:               "SYSTEMSECRET",
+		NatsToken:               "NATSTOKEN",
+		PoolDNSServers:          []string{"1.1.1.1", "8.8.8.8"},
+		NorthstarAccessKey:      "AKIANORTHSTAR",
+		NorthstarSecretKey:      "NORTHSTARSECRET",
+		NorthstarBucket:         admin.NorthstarBucketName,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+	}
+}
+
+func TestNorthstarTomlTemplate(t *testing.T) {
+	content := renderTemplate(t, northstarTomlTemplate, northstarSettings())
+
+	// Northstar binds :5300 on the wildcard and the privileged :53 on the node
+	// WAN IP (avoids the systemd-resolved stub on 127.0.0.53:53).
+	assert.Contains(t, content, `listen     = "0.0.0.0:5300,192.168.1.31:53"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `bucket     = "northstar"`)
+	assert.Contains(t, content, `access_key = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `secret_key = "NORTHSTARSECRET"`)
+	// 0.0.0.0 bind must resolve to a loopback S3 endpoint, not a dial to 0.0.0.0.
+	assert.Contains(t, content, `endpoint   = "https://127.0.0.1:8443"`)
+	assert.Contains(t, content, `nameservers = ["1.1.1.1:53", "8.8.8.8:53"]`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+	var parsed struct {
+		Quotas struct {
+			Enabled              bool `toml:"enabled"`
+			RecordsPerHostedZone int  `toml:"records_per_hosted_zone"`
+		} `toml:"quotas"`
+	}
+	require.NoError(t, toml.Unmarshal([]byte(content), &parsed))
+	assert.False(t, parsed.Quotas.Enabled)
+	assert.Equal(t, 10_000, parsed.Quotas.RecordsPerHostedZone)
+	assert.NotContains(t, content, "nats_url")
+	assert.NotContains(t, content, "NATSTOKEN")
+}
+
+// When provisioned, predastore.toml gains the northstar bucket plus a scoped
+// read-only [[auth]] entry — the isolation guarantee.
+func TestPredastoreTemplateWithNorthstar(t *testing.T) {
+	content := renderTemplate(t, predastoreTomlTemplate, northstarSettings())
+
+	assert.Contains(t, content, `name = "northstar"`)
+	assert.Contains(t, content, `access_key_id = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `{ bucket = "northstar", actions = ["s3:ListBucket", "s3:GetObject"] }`)
+}
+
+// Without northstar creds (e.g. cluster-join paths), predastore.toml renders
+// exactly as before — no empty bucket/auth stanzas.
+func TestPredastoreTemplateWithoutNorthstar(t *testing.T) {
+	s := northstarSettings()
+	s.NorthstarAccessKey = ""
+	s.NorthstarSecretKey = ""
+	content := renderTemplate(t, predastoreTomlTemplate, s)
+
+	assert.NotContains(t, content, `name = "northstar"`)
+	assert.NotContains(t, content, "AKIANORTHSTAR")
+}

cmd/spinifex/cmd/paths.go

@@ -8,7 +8,7 @@ import (
 
 // DefaultConfigDir returns the default configuration directory.
 // Production: /etc/spinifex (when /etc/spinifex exists)
-// Development: ~/spinifex/config
+// Development: ~/spinifex/config.
 func DefaultConfigDir() string {
 	if isProductionLayout() {
 		return "/etc/spinifex"
@@ -18,7 +18,7 @@ func DefaultConfigDir() string {
 
 // DefaultDataDir returns the default data directory.
 // Production: /var/lib/spinifex
-// Development: ~/spinifex
+// Development: ~/spinifex.
 func DefaultDataDir() string {
 	if isProductionLayout() {
 		return "/var/lib/spinifex"
@@ -28,7 +28,7 @@ func DefaultDataDir() string {
 
 // LogDirFor returns the log directory for a given data directory.
 // Production: /var/log/spinifex (matches systemd ReadWritePaths)
-// Development: <dataDir>/logs (supports custom per-node data dirs)
+// Development: <dataDir>/logs (supports custom per-node data dirs).
 func LogDirFor(dataDir string) string {
 	if isProductionLayout() {
 		return "/var/log/spinifex"

cmd/spinifex/cmd/root.go

@@ -15,7 +15,7 @@ var (
 	appConfig *config.ClusterConfig
 )
 
-// rootCmd represents the base command when called without any subcommands
+// rootCmd represents the base command when called without any subcommands.
 var rootCmd = &cobra.Command{
 	Use:   "spx",
 	Short: "Spinifex - Open source AWS-compatible platform for secure edge deployments",

cmd/spinifex/cmd/service.go

@@ -8,11 +8,13 @@ import (
 	"strings"
 
 	"github.com/mulgadc/spinifex/spinifex/config"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	"github.com/mulgadc/spinifex/spinifex/network/external"
 	"github.com/mulgadc/spinifex/spinifex/otelsetup"
 	"github.com/mulgadc/spinifex/spinifex/service"
 	"github.com/mulgadc/spinifex/spinifex/services/awsgw"
 	"github.com/mulgadc/spinifex/spinifex/services/nats"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
 	"github.com/mulgadc/spinifex/spinifex/services/predastore"
 	"github.com/mulgadc/spinifex/spinifex/services/qmpcollector"
 	"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
@@ -201,7 +203,7 @@ var predastoreStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for viperblock
+// Repeat for viperblock.
 var viperblockStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the viperblock service",
@@ -295,6 +297,14 @@ var viperblockStartCmd = &cobra.Command{
 			shardWAL = *nodeConfig.Viperblock.ShardWAL
 		}
 
+		// Resolve chunk GC setting: default false unless explicitly set to true.
+		// GC is not a per-volume toggle — flipping it on applies to every VB this
+		// node constructs (nbdkit plugin + volume service).
+		gcEnabled := false
+		if nodeConfig.Viperblock.GCEnabled != nil {
+			gcEnabled = *nodeConfig.Viperblock.GCEnabled
+		}
+
 		encryptionKeyFile := nodeConfig.Viperblock.EncryptionKeyFile
 		if envKey := viper.GetString("viperblock-encryption-key-file"); envKey != "" {
 			encryptionKeyFile = envKey
@@ -315,6 +325,7 @@ var viperblockStartCmd = &cobra.Command{
 			BaseDir:           nodeConfig.Predastore.BaseDir,
 			NodeName:          clusterConfig.Node,
 			ShardWAL:          shardWAL,
+			GCEnabled:         gcEnabled,
 			EncryptionKeyFile: encryptionKeyFile,
 		})
 
@@ -365,7 +376,7 @@ var viperblockStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for nats
+// Repeat for nats.
 var natsStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the nats service",
@@ -434,7 +445,7 @@ var natsStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for spinifex
+// Repeat for spinifex.
 var spinifexStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the spinifex service",
@@ -793,20 +804,23 @@ var vpcdStartCmd = &cobra.Command{
 		defer initTelemetry("vpcd", false)()
 
 		svc, err := service.New("vpcd", &vpcd.Config{
-			NatsHost:          nodeConfig.NATS.Host,
-			NatsToken:         nodeConfig.NATS.ACL.Token,
-			NatsCACert:        nodeConfig.NATS.CACert,
-			OVNNBAddr:         nodeConfig.VPCD.OVNNBAddr,
-			OVNSBAddr:         nodeConfig.VPCD.OVNSBAddr,
-			BaseDir:           nodeConfig.BaseDir,
-			Debug:             false,
-			ExternalMode:      clusterConfig.Network.ExternalMode,
-			ExternalPools:     extPools,
-			Bootstrap:         bootstrap,
-			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			BridgeMode:        nodeConfig.VPCD.BridgeMode,
-			AZ:                nodeConfig.AZ,
-			NATExemptCIDRs:    clusterConfig.Network.NATExemptCIDRs,
+			NatsHost:                nodeConfig.NATS.Host,
+			NatsToken:               nodeConfig.NATS.ACL.Token,
+			NatsCACert:              nodeConfig.NATS.CACert,
+			OVNNBAddr:               nodeConfig.VPCD.OVNNBAddr,
+			OVNSBAddr:               nodeConfig.VPCD.OVNSBAddr,
+			BaseDir:                 nodeConfig.BaseDir,
+			Debug:                   false,
+			ExternalMode:            clusterConfig.Network.ExternalMode,
+			ExternalPools:           extPools,
+			Bootstrap:               bootstrap,
+			ExternalInterface:       nodeConfig.VPCD.ExternalInterface,
+			BridgeMode:              nodeConfig.VPCD.BridgeMode,
+			AZ:                      nodeConfig.AZ,
+			NorthstarBaseDomain:     handlers_dns.ResolveBaseDomain(&nodeConfig),
+			NorthstarInternalDomain: handlers_dns.ResolveInternalDomain(&nodeConfig),
+			ResolverNameservers:     handlers_dns.ResolverNameserverIPs(clusterConfig),
+			NATExemptCIDRs:          clusterConfig.Network.NATExemptCIDRs,
 		})
 		if err != nil {
 			fmt.Println("Error starting vpcd service:", err)
@@ -849,6 +863,157 @@ var vpcdStatusCmd = &cobra.Command{
 	},
 }
 
+var northstarCmd = &cobra.Command{
+	Use:   "northstar",
+	Short: "Manage the northstar (DNS) service",
+}
+
+// northstarStartOptions contains the command inputs that select Northstar configuration.
+type northstarStartOptions struct {
+	configFile      string
+	configOverride  string
+	baseDirOverride string
+}
+
+// northstarStarter is the service operation required by Northstar activation.
+type northstarStarter interface {
+	Start() (int, error)
+}
+
+// northstarStartDependencies isolates startup side effects for behavioral tests.
+type northstarStartDependencies struct {
+	loadConfig        func(string) (*config.ClusterConfig, error)
+	bootstrapBaseZone func(string, *config.ClusterConfig) error
+	newService        func(*northstar.Config) (northstarStarter, error)
+}
+
+var northstarStartCmd = &cobra.Command{
+	Use:           "start",
+	Short:         "Start the northstar service",
+	SilenceErrors: true,
+	SilenceUsage:  true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		fmt.Println("Starting northstar service...")
+
+		defer initTelemetry("northstar", false)()
+
+		return runNorthstarStart(northstarStartOptions{
+			configFile:      viper.GetString("config"),
+			configOverride:  viper.GetString("northstar-config"),
+			baseDirOverride: viper.GetString("base-dir"),
+		}, northstarStartDependencies{
+			loadConfig:        loadRequiredClusterConfig,
+			bootstrapBaseZone: northstar.BootstrapBaseZone,
+			newService: func(cfg *northstar.Config) (northstarStarter, error) {
+				return service.New("northstar", cfg)
+			},
+		})
+	},
+}
+
+// loadRequiredClusterConfig loads the service's explicit cluster configuration.
+func loadRequiredClusterConfig(path string) (*config.ClusterConfig, error) {
+	if path == "" {
+		return nil, fmt.Errorf("config file is not set")
+	}
+	if _, err := os.Stat(path); err != nil {
+		return nil, fmt.Errorf("stat cluster config %s: %w", path, err)
+	}
+	clusterConfig, err := config.LoadConfig(path)
+	if err != nil {
+		return nil, fmt.Errorf("load cluster config %s: %w", path, err)
+	}
+	return clusterConfig, nil
+}
+
+// runNorthstarStart starts configured Northstar nodes and cleanly skips nodes where it is optional.
+func runNorthstarStart(options northstarStartOptions, deps northstarStartDependencies) error {
+	clusterConfig, err := deps.loadConfig(options.configFile)
+	if err != nil {
+		return err
+	}
+
+	nodeConfig, ok := clusterConfig.Nodes[clusterConfig.Node]
+	if !ok {
+		return fmt.Errorf("node %q not found in cluster config", clusterConfig.Node)
+	}
+
+	configPath := resolveNorthstarConfigPath(nodeConfig.Northstar.ConfigPath, options.configOverride)
+	if options.configOverride != "" {
+		fmt.Println("Overwriting northstar config path to:", options.configOverride)
+	}
+	if configPath == "" {
+		// An absent path disables this optional service and must not trigger systemd restarts.
+		slog.Info("northstar is not configured on this node; skipping service start")
+		return nil
+	}
+
+	baseDir := nodeConfig.BaseDir
+	if options.baseDirOverride != "" {
+		baseDir = options.baseDirOverride
+	}
+
+	// Seed the default_domain base zone before the read-only daemon starts.
+	// The S3 polling path remains the backstop when this best-effort seed fails.
+	if err := deps.bootstrapBaseZone(configPath, clusterConfig); err != nil {
+		slog.Warn("northstar base zone bootstrap failed (continuing)", "error", err)
+	}
+
+	svc, err := deps.newService(&northstar.Config{
+		ConfigPath: configPath,
+		BasePath:   baseDir,
+		NodeID:     nodeConfig.Predastore.NodeID,
+		NatsHost:   nodeConfig.NATS.Host,
+		NatsToken:  nodeConfig.NATS.ACL.Token,
+		NatsCACert: nodeConfig.NATS.CACert,
+	})
+	if err != nil {
+		return fmt.Errorf("create northstar service: %w", err)
+	}
+
+	if _, err := svc.Start(); err != nil {
+		return fmt.Errorf("start northstar service: %w", err)
+	}
+	fmt.Println("northstar service started")
+	return nil
+}
+
+// resolveNorthstarConfigPath applies the explicit override before the node config.
+func resolveNorthstarConfigPath(nodePath, override string) string {
+	if override != "" {
+		return override
+	}
+	return nodePath
+}
+
+var northstarStopCmd = &cobra.Command{
+	Use:   "stop",
+	Short: "Stop the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("Stopping northstar service...")
+
+		svc, err := service.New("northstar", &northstar.Config{})
+		if err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			return
+		}
+
+		if err = svc.Stop(); err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			os.Exit(1)
+		}
+		fmt.Println("northstar service stopped")
+	},
+}
+
+var northstarStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Get status of the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("northstar service status: ...")
+	},
+}
+
 var qmpCollectorCmd = &cobra.Command{
 	Use:   "qmp-collector",
 	Short: "Manage the qmp-collector (guest metrics) service",
@@ -1115,6 +1280,17 @@ func init() {
 	vpcdCmd.AddCommand(vpcdStopCmd)
 	vpcdCmd.AddCommand(vpcdStatusCmd)
 
+	// northstar
+	serviceCmd.AddCommand(northstarCmd)
+
+	northstarCmd.PersistentFlags().String("northstar-config", "", "Path to northstar.toml (overrides config file)")
+	viper.BindEnv("northstar-config", "SPINIFEX_NORTHSTAR_CONFIG")
+	viper.BindPFlag("northstar-config", northstarCmd.PersistentFlags().Lookup("northstar-config"))
+
+	northstarCmd.AddCommand(northstarStartCmd)
+	northstarCmd.AddCommand(northstarStopCmd)
+	northstarCmd.AddCommand(northstarStatusCmd)
+
 	// qmp-collector
 	serviceCmd.AddCommand(qmpCollectorCmd)
 

cmd/spinifex/cmd/service_test.go

@@ -0,0 +1,207 @@
+package cmd
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type stubNorthstarService struct {
+	starts   int
+	startErr error
+}
+
+func (s *stubNorthstarService) Start() (int, error) {
+	s.starts++
+	return 0, s.startErr
+}
+
+func TestRunNorthstarStartActivation(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name           string
+		nodePath       string
+		override       string
+		wantPath       string
+		wantConfigured bool
+	}{
+		{
+			name: "unconfigured node is skipped",
+		},
+		{
+			name:           "node config starts service",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			wantPath:       "/etc/spinifex/northstar/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override starts service",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override takes precedence",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			clusterConfig := &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {
+						BaseDir:   "/var/lib/spinifex/northstar",
+						Northstar: config.NorthstarConfig{ConfigPath: tt.nodePath},
+					},
+				},
+			}
+			stub := &stubNorthstarService{}
+			bootstrapCalls := 0
+			factoryCalls := 0
+			var startedConfig *northstar.Config
+			deps := northstarStartDependencies{
+				loadConfig: func(string) (*config.ClusterConfig, error) {
+					return clusterConfig, nil
+				},
+				bootstrapBaseZone: func(path string, cfg *config.ClusterConfig) error {
+					bootstrapCalls++
+					assert.Equal(t, tt.wantPath, path)
+					assert.Same(t, clusterConfig, cfg)
+					return nil
+				},
+				newService: func(cfg *northstar.Config) (northstarStarter, error) {
+					factoryCalls++
+					startedConfig = cfg
+					return stub, nil
+				},
+			}
+
+			err := runNorthstarStart(northstarStartOptions{
+				configFile:     "/etc/spinifex/spinifex.toml",
+				configOverride: tt.override,
+			}, deps)
+			require.NoError(t, err)
+
+			if !tt.wantConfigured {
+				assert.Zero(t, bootstrapCalls)
+				assert.Zero(t, factoryCalls)
+				assert.Zero(t, stub.starts)
+				assert.Nil(t, startedConfig)
+				return
+			}
+			assert.Equal(t, 1, bootstrapCalls)
+			assert.Equal(t, 1, factoryCalls)
+			assert.Equal(t, 1, stub.starts)
+			require.NotNil(t, startedConfig)
+			assert.Equal(t, tt.wantPath, startedConfig.ConfigPath)
+		})
+	}
+}
+
+func TestLoadRequiredClusterConfigRejectsMissingFile(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		path string
+	}{
+		{name: "empty path"},
+		{name: "missing path", path: t.TempDir() + "/missing.toml"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			_, err := loadRequiredClusterConfig(tt.path)
+			require.Error(t, err)
+		})
+	}
+}
+
+func TestRunNorthstarStartReturnsLoadFailure(t *testing.T) {
+	t.Parallel()
+
+	loadErr := errors.New("cluster config unavailable")
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return nil, loadErr
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, loadErr)
+}
+
+func TestRunNorthstarStartContinuesAfterBootstrapFailure(t *testing.T) {
+	t.Parallel()
+
+	bootstrapErr := errors.New("predastore unavailable")
+	stub := &stubNorthstarService{}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return bootstrapErr },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.NoError(t, err)
+	assert.Equal(t, 1, stub.starts)
+}
+
+func TestRunNorthstarStartRejectsMissingNode(t *testing.T) {
+	t.Parallel()
+
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{Node: "missing", Nodes: map[string]config.Config{}}, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.EqualError(t, err, `node "missing" not found in cluster config`)
+}
+
+func TestRunNorthstarStartReturnsConfiguredFailure(t *testing.T) {
+	t.Parallel()
+
+	startErr := errors.New("configured startup failed")
+	stub := &stubNorthstarService{startErr: startErr}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return nil },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, startErr)
+	assert.Equal(t, 1, stub.starts)
+}

cmd/spinifex/cmd/templates/northstar.toml

@@ -0,0 +1,40 @@
+# Northstar DNS service configuration (TOML).
+# Written by `spx admin init`; read by the spinifex northstar service.
+# Contains bucket-scoped S3 credentials — keep mode 0600.
+
+# UDP+TCP bind addresses (comma separated). :5300 binds the wildcard for local
+# tooling; :53 binds the node WAN IP specifically (via CAP_NET_BIND_SERVICE) so
+# instances resolve against the node directly. Binding the WAN IP rather than
+# 0.0.0.0 avoids the systemd-resolved stub on 127.0.0.53:53. A future per-VPC
+# resolver VIP (OVN) will front :53.
+listen     = "0.0.0.0:5300,{{if .AdvertiseIP}}{{.AdvertiseIP}}{{else}}{{.BindIP}}{{end}}:53"
+dot_listen = ""
+doh_listen = ""
+tls_cert   = ""
+tls_key    = ""
+
+# Authoritative base domain for internal resource names.
+default_domain = "{{.NorthstarDefaultDomain}}"
+
+# AWS-parity private DNS zone for instance ip-<addr> records (IMDS hostname).
+internal_domain = "{{.NorthstarInternalDomain}}"
+
+# Seconds between S3 zone re-scans (anti-entropy; live changes arrive via NATS).
+sync_interval = 30
+
+# DNS (Route 53) service quotas (default off, mirrors [quota] in awsgw.toml).
+# Applied at daemon start; restart to change.
+[quotas]
+enabled = false
+records_per_hosted_zone = 10000   # AWS default; enforced only when enabled
+
+[s3]
+endpoint   = "https://{{if eq .BindIP "0.0.0.0"}}127.0.0.1{{else}}{{.BindIP}}{{end}}:8443"
+bucket     = "{{.NorthstarBucket}}"
+region     = "{{.Region}}"
+access_key = "{{.NorthstarAccessKey}}"
+secret_key = "{{.NorthstarSecretKey}}"
+
+# Forwarders for non-authoritative (recursive) queries. Empty => REFUSED.
+[upstream]
+nameservers = [{{range $i, $s := .PoolDNSServers}}{{if $i}}, {{end}}"{{$s}}:53"{{end}}]

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -46,13 +46,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/predastore.toml

@@ -75,13 +75,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/spinifex.toml

@@ -129,6 +129,14 @@ secretkey = "{{.SecretKey}}"
 {{- if gt .PredastoreNodeID 0}}
 node_id = {{.PredastoreNodeID}}
 {{- end}}
+
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Node}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+default_domain = "{{.NorthstarDefaultDomain}}"
+internal_domain = "{{.NorthstarInternalDomain}}"
+{{- end}}
 {{- range .RemoteNodes}}
 
 [nodes.{{.Name}}]
@@ -139,4 +147,9 @@ az = "{{.AZ}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Name}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+{{- end}}
 {{- end}}

cmd/spinifex/cmd/version.go

@@ -8,7 +8,7 @@ import (
 )
 
 // Version variables set via ldflags at build time.
-// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0"
+// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0".
 var (
 	Version = "dev"
 	Commit  = "unknown"

docs/COMMANDS.md

@@ -31,6 +31,9 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp
 | `spx service vpcd start` | — | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) |
 | `spx service vpcd stop` | — | Stops the vpcd service |
 | `spx service vpcd status` | — | Reports vpcd service status |
+| `spx service northstar start` | `--northstar-config` (overrides `nodes.<node>.northstar.config_path`) | Loads cluster config → reads `northstar.toml` → starts the northstar DNS server (authoritative for internal `*.spx3.net`, recursive via upstream forwarders), syncing zones from its S3 bucket. Guests resolve via `169.254.169.253`, served by vpcd's per-tap DNS shim which forwards to northstar |
+| `spx service northstar stop` | — | Stops the northstar service |
+| `spx service northstar status` | — | Reports northstar service status |
 | `spx service qmp-collector start` | — | Starts the guest-metrics collector (polls per-VM telemetry QMP sockets + tap counters, publishes CloudWatch-shaped series to NATS `metrics.ec2.*`) |
 | `spx service qmp-collector stop` | — | Stops the qmp-collector service |
 | `spx service qmp-collector status` | — | Reports qmp-collector service status |
@@ -54,7 +57,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--no-external` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/`. `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/`. `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
 | `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |
 
 ### Version

docs/reference-architectures/RTX-Pro-6000-EKS/README.md

@@ -0,0 +1,453 @@
+---
+title: "EKS AI Platform on Bare Metal: llama.cpp + YOLO on Spinifex"
+description: "Deploy a GPU-accelerated AI inference platform — an OpenAI-compatible LLM API and a real-time CV stream — in Kubernetes on bare-metal hardware, managed entirely with standard AWS tooling."
+category: "Reference Architectures"
+tags:
+  - nvidia
+  - rtx-pro-6000
+  - eks
+  - kubernetes
+  - llama.cpp
+  - yolo
+  - ecr
+  - bare-metal
+  - terraform
+resources:
+  - title: "eks-ai-platform (Terraform workbook, Dockerfiles, scripts)"
+    url: "https://github.com/mulgadc/eks-ai-platform"
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching EKS Clusters"
+    url: "/docs/eks"
+  - title: "llama.cpp"
+    url: "https://github.com/ggml-org/llama.cpp"
+  - title: "Ultralytics YOLO"
+    url: "https://docs.ultralytics.com"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services such as EC2, S3, EBS and EKS, to bare-metal, edge, and on-prem deployments. It exposes a fully AWS-compatible API, so any tooling that works against AWS — the `aws` CLI, OpenTofu, `kubectl` — works against a Spinifex node unchanged, with a single profile swap.
+
+This guide walks through the use of Spinifex to deploy a self-contained AI inference platform on Supermicro's bare metal X14 platform using only standard AWS tooling. We use Terraform to create resources (wrapped in simple `make` commands) in the exact same way you would create AWS resources.
+
+
+Specifically, we create an EKS cluster with two worker nodes, each consisting of a g7e.2xlarge EC2 instance with an attached GPU via VFIO passthrough, an ALB to route traffic to each node, ECR for storing and managing our workload images, and all of the associated security and certificate management requirements (IAM, ACM) you would expect from real AWS.
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Chassis** | Supermicro X14 2U CloudDC with Intel Xeon 6730P |
+| **GPUs** | 2× NVIDIA RTX Pro 6000 Blackwell Server Edition (96 GiB GDDR7 each, 192 GiB total) |
+| **Storage** | 4× NVMe SSD: 2× 1.5 TB, 2× 880 GB — one 1.5 TB drive carries the OS; the remaining three back Predastore |
+| **Spinifex instance family** | `g7e` — one RTX Pro 6000 per instance via VFIO PCIe passthrough |
+| **Kubernetes** | k3s, managed via Spinifex EKS API |
+| **API endpoint** | `https://<host>:9999` (AWS-compatible) |
+
+### Workloads
+Three workloads run in the `inference` namespace of an EKS cluster backed by two GPU worker nodes:
+
+| Pod | Role | GPU | Model | Image source |
+|---|---|---|---|---|
+| `llm-server` | OpenAI-compatible chat API (llama.cpp) | 1× RTX Pro 6000 | Llama 3.2 3B Instruct Q4_K_M GGUF | ECR `llm-server:latest` |
+| `yolo-stream` | MJPEG object-detection stream (CUDA) | 1× RTX Pro 6000 | YOLO11x | ECR `yolo-stream:latest` |
+| `ai-dashboard` | Web UI: chat + live detection feed | CPU only | — | ECR `ai-dashboard:latest` |
+
+Both GPU workloads bake their model weights into the Docker image at build time — `llm-server`'s GGUF weights (~2 GiB) and `yolo-stream`'s YOLO11x checkpoint (~109 MB) are present in the image when it starts.
+
+`yolo-stream` renders a 1280×720 sample video through YOLO11x once on startup (~25 s), caches the annotated frames in memory, then serves `/stream` from that cache — smooth MJPEG playback decoupled from per-frame inference cost after the initial warm-up.
+
+The two GPU nodes are split one-per-workload. With one RTX Pro 6000 per node and both pods requesting `nvidia.com/gpu: "1"`, the scheduler cannot fit both on the same node, so the split happens naturally without explicit affinity rules.
+
+
+
+## Architecture
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/ai-platform-request-flow-2.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
+
+### AWS services exercised
+
+| Service | Role |
+|---|---|
+| **IAM** | Cluster role, node role (with ECR, LBC, EBS-CSI permissions inline), viewer access entry |
+| **ECR** | Private registry for all three images; source of truth for builds, though nodes receive images via sideload rather than live OCI pull for this demo |
+| **EC2** | 2× GPU microVM (`g7e.2xlarge`), one RTX Pro 6000 each via VFIO PCIe passthrough |
+| **EKS** | Cluster, GPU nodegroup (`desired_size = 2`), LBC and EBS-CSI managed addons, access entries |
+| **ELBv2** | ALB provisioned by the LBC addon; single shared IngressGroup across all three services |
+| **ACM** | Self-signed cert (`ai-platform.spinifex.local`) imported and attached to the ALB HTTPS listener |
+| **EBS (Viperblock)** | 200 GB root volume per GPU worker node, provisioned by the nodegroup (`disk_size = 200`)|
+
+All permissions for the LBC and EBS-CSI addons are attached directly to the node role. Both addons support IRSA, but fall back to the node's instance profile when no `service_account_role_arn` is supplied — sufficient for a single-cluster deployment.
+
+## Prerequisites
+
+### On the Spinifex host
+
+**1. Configure host-local VPC networking**
+
+Spinifex uses bridged networking via OVN. The X14 exposes a single public IP on one physical NIC — before installing Spinifex, ensure `br-wan` exists with the physical NIC enslaved to it, and attach a host-local address range for EC2 instance communication:
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24             # VM pool gateway — host-local
+      - <existing-wan-ip>/<prefix>  # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: <upstream-gateway>
+```
+
+Apply with `sudo netplan apply`. For outbound connectivity from instances through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+
+The full process is described in the [VPC Networking](/docs/vpc-networking#host-local-subnet-no-upstream-router) guide.
+
+**2. Install Spinifex**
+
+Follow the [Single Node Install](/docs/install) guide. This installs Spinifex and starts all services, however `spinifex.toml` needs to be edited to wire up the bridge created in the previous step, as described in the following section.
+
+**3. Configure spinifex.toml and restart services**
+
+Edit `/etc/spinifex/spinifex.toml` to point the external pool at the bridge address range created in step 1:
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+Then restart all services:
+
+```bash
+sudo systemctl restart spinifex.target
+sudo systemctl status spinifex.target
+```
+
+**4. Bind the GPUs to VFIO**
+
+```bash
+sudo spx admin gpu setup
+# Reboot, then:
+sudo spx admin gpu enable
+```
+
+Confirm both GPUs are bound:
+
+```bash
+lspci -d 10de: -nn
+# Expect: NVIDIA Corporation Device [10de:2bb5] appearing twice
+```
+
+**5. Attach Predastore storage**
+
+The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. Predastore is distributed across the three non-OS drives — one storage node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.
+
+Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.
+
+```bash
+lsblk   # identify the OS drive and the three data drives
+
+# Stop services so Predastore isn't writing while we relocate its data directories
+sudo systemctl stop spinifex.target
+
+# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+
+# Drive 1: non-OS 1.5 TB (e.g. /dev/nvme1n1)
+sudo mkdir -p /mnt/nvme-1
+sudo mount /dev/nvme1n1 /mnt/nvme-1
+sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+
+sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
+sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+
+echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# Drive 2: first ~880 GB drive (e.g. /dev/nvme2n1)
+sudo mkdir -p /mnt/nvme-2
+sudo mount /dev/nvme2n1 /mnt/nvme-2
+sudo mkdir -p /mnt/nvme-2/nodes /mnt/nvme-2/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-2 /mnt/nvme-2/nodes/node-2
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-2    /mnt/nvme-2/db/node-2
+
+sudo ln -s /mnt/nvme-2/nodes/node-2 /var/lib/spinifex/predastore/distributed/nodes/node-2
+sudo ln -s /mnt/nvme-2/db/node-2    /var/lib/spinifex/predastore/distributed/db/node-2
+
+echo "/dev/nvme2n1  /mnt/nvme-2  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# Drive 3: second ~880 GB drive (e.g. /dev/nvme3n1)
+sudo mkdir -p /mnt/nvme-3
+sudo mount /dev/nvme3n1 /mnt/nvme-3
+sudo mkdir -p /mnt/nvme-3/nodes /mnt/nvme-3/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-3 /mnt/nvme-3/nodes/node-3
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-3    /mnt/nvme-3/db/node-3
+
+sudo ln -s /mnt/nvme-3/nodes/node-3 /var/lib/spinifex/predastore/distributed/nodes/node-3
+sudo ln -s /mnt/nvme-3/db/node-3    /var/lib/spinifex/predastore/distributed/db/node-3
+
+echo "/dev/nvme3n1  /mnt/nvme-3  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# --- End of per-drive block ---
+
+sudo systemctl start spinifex.target
+```
+
+Verify the nodes are healthy before proceeding:
+
+```bash
+export AWS_PROFILE=spinifex
+aws s3 ls
+# Should return without error (empty bucket list is fine)
+```
+
+**6. Verify the GPU instance type**
+
+```bash
+sudo spx admin gpu status
+```
+
+This will print confirmation that GPU passthrough has been configured correctly along with the available GPU instance types. The RTX Pro 6000 Blackwell Server Edition (PCI device `10de:2bb5`) maps to the `g7e` family. The workbook defaults to `g7e.2xlarge` (one GPU per node); override with `GPU_TYPE=g7e.4xlarge` (or the size your host reports) if needed. Do not use `g7e.12xlarge` — that is the 2× GPU size.
+
+### Local tooling
+
+- [OpenTofu](https://opentofu.org/) >= 1.6
+- [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- [Docker](https://docs.docker.com/get-docker/)
+
+
+### Clone the workbook
+
+```bash
+git clone https://github.com/mulgadc/eks-ai-platform
+cd eks-ai-platform
+```
+
+## Instructions
+
+### 1. Import the EKS GPU node AMI
+
+GPU worker nodes require the `ecr-credential-provider` binary so kubelet can call `GetAuthorizationToken` against ECR. This binary is included in the dedicated EKS GPU node AMI in the Spinifex image catalogue. List available images and import it:
+
+```bash
+spx admin images list
+# Look for the EKS GPU node image
+
+spx admin images import --name spinifex-eks-node-gpu
+```
+
+Confirm the AMI is registered:
+
+```bash
+aws ec2 describe-images --query 'Images[*].[Name,ImageId]' --output table
+```
+
+### 2. Provision the cluster
+
+```bash
+make infra ENDPOINT=https://<host>:9999
+```
+
+This provisions the VPC (`10.33.0.0/16`, two public and two private subnets with a NAT gateway), IAM roles, ECR repositories, EKS cluster, GPU nodegroup (2× `g7e.2xlarge`, 200 GB disk each), LBC and EBS-CSI managed addons, a self-signed ACM certificate, and the NodePort security group rules the ALB needs to reach the worker nodes.
+
+Update your kubeconfig once the cluster reports `ACTIVE`:
+
+```bash
+$(tofu -chdir=workbook output -raw update_kubeconfig)
+kubectl get nodes
+# Expect: 2 Ready nodes in the gpu-workers nodegroup
+```
+
+The Makefile wraps `tofu -chdir=workbook apply -var spinifex_endpoint=... -var gpu_instance_type=...` — running Tofu directly is equivalent and lets you pass any additional variables. The workbook provisions `aws_vpc`, `aws_subnet` (two public, two private), `aws_eks_cluster`, `aws_eks_node_group` (two `g7e.2xlarge` nodes, each with a 200 GB Viperblock root volume via `disk_size = 200`), `aws_eks_addon` for LBC and EBS-CSI, three `aws_ecr_repository` resources, three IAM roles, and a self-signed `aws_acm_certificate` — all via Spinifex's AWS-compatible endpoint at `:9999`.
+
+### 3. Build and push container images
+
+```bash
+make images
+```
+
+This authenticates to ECR, then builds and pushes all three images:
+
+- **`llm-server`** — based on `ghcr.io/ggml-org/llama.cpp:server-cuda`; downloads Llama 3.2 3B Instruct Q4_K_M GGUF (~2 GiB) from Hugging Face at build time and bakes it into the image. Exposes an OpenAI-compatible `/v1/chat/completions` API.
+- **`yolo-stream`** — based on `pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime`; installs Ultralytics and downloads YOLO11x weights at build time. PyTorch 2.7.1+cu128 is required: the RTX Pro 6000 Blackwell is compute capability sm_120, and earlier PyTorch releases ship no sm_120 kernels.
+- **`ai-dashboard`** — lightweight Flask proxy (`FROM python:3.11-slim`) that aggregates the LLM API and YOLO stream into a single page.
+
+The ECR registry URI always includes `:9999` — for example, `<account>.dkr.ecr.ap-southeast-2.<suffix>:9999`. Use the `ecr_registry` Tofu output directly in `docker login` and image references; do not construct the hostname manually.
+
+Image URIs come from `tofu -chdir=workbook output -raw ecr_registry`. ECR authentication uses the same API as AWS: `aws ecr get-login-password` calls `GetAuthorizationToken` against the Spinifex ECR endpoint and returns a short-lived JWT that Docker accepts as a registry password. The `make images` target is equivalent to running those `docker build` and `docker push` commands directly against `$REGISTRY` from that Tofu output.
+
+### 4. Sideload images onto GPU worker nodes
+
+```bash
+make sideload
+```
+
+ECR is the source of truth for all three images — authentication, push, and registry management all work identically to AWS. In a standard EKS deployment, nodes would pull images directly from ECR at scheduling time. In this demo, live pulls of these image sizes (~2 GiB for `llm-server`, ~4.5 GiB for `yolo-stream`) through the Spinifex ECR gateway proved unreliable — large transfers stalled or failed mid-stream, a combination of network conditions on this single-host setup and a rough edge in early Spinifex ECR support. This step works around that by staging the images directly into each node's containerd store before the pods are scheduled.
+
+It exports each image from the local Docker daemon, serves the tarballs over HTTP from the Spinifex host, and imports them directly into each worker node's containerd store via short-lived privileged pods. All three Deployments use `imagePullPolicy: IfNotPresent` and depend on the images already being present in containerd.
+
+The script also labels the two GPU nodes deterministically (`workload=llm-server` / `workload=yolo-stream`, sorted by node name), and the Deployments use matching `nodeSelector` values so each pod lands on the node that already has its image. `ai-dashboard` has no GPU requirement and is imported on both nodes since it can schedule onto either.
+
+Expect several minutes for `yolo-stream`'s image (CUDA + PyTorch + YOLO11x weights, ~4.5 GiB).
+
+This step has no Tofu equivalent — it operates directly on the running cluster via `kubectl`. The ECR registry URI is read from the Tofu state; node labelling uses `kubectl label` and image import uses short-lived privileged pods that run `ctr images import` into each node's containerd store.
+
+### 5. Deploy workloads
+
+```bash
+make workloads ENDPOINT=https://<host>:9999
+```
+
+This deploys the NVIDIA GPU Operator via Helm, then all three application Deployments, ClusterIP and NodePort services, and the shared ALB Ingresses.
+
+The GPU Operator requires two adjustments for k3s:
+
+- `driver.enabled=false` — the NVIDIA driver is pre-built into the GPU AMI at image creation time; the Operator installs only the toolkit and device plugin.
+- `CONTAINERD_SOCKET=/run/k3s/containerd/containerd.sock` — k3s bundles its own containerd at a different socket and config path than the standalone containerd default. Without this override the toolkit DaemonSet crash-loops with `no such file or directory`.
+
+Watch the GPU Operator complete, then the inference pods come up:
+
+```bash
+kubectl -n gpu-operator get pods -w
+kubectl -n inference get pods -w
+```
+
+Confirm each GPU node reports `nvidia.com/gpu: 1` in allocatables:
+
+```bash
+$(tofu -chdir=workbook output -raw gpu_verify_hint)
+```
+
+All three routes share a single ALB via `alb.ingress.kubernetes.io/group.name`. Explicit `group.order` values (`/v1` = 10, `/stream` = 20, `/` = 100) ensure the dashboard's catch-all path evaluates last — without them, the LBC sorts Ingress resources alphabetically, which places the catch-all first and swallows the other routes.
+
+Retrieve the ALB IP (the DNS name `*.elb.spinifex.local` is a label, not a resolvable entry):
+
+```bash
+$(tofu -chdir=workbook output -raw alb_ip_hint)
+ALB_IP=<output>
+```
+
+Validate the LLM endpoint:
+
+```bash
+curl -sk https://$ALB_IP/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model":"llama-3.2-3b-instruct","messages":[{"role":"user","content":"What is Spinifex?"}]}' \
+  | python3 -m json.tool
+```
+
+Open `https://$ALB_IP/` to reach the dashboard. If accessing from a remote machine:
+
+```bash
+ssh -L 8443:$ALB_IP:443 <spinifex-host>
+# Open https://ai-platform.spinifex.local:8443/
+# Add 127.0.0.1 ai-platform.spinifex.local to /etc/hosts if the browser requires hostname match
+```
+
+The workloads module reads cluster coordinates, ECR image URIs, NodePort values, and the ACM cert ARN from the parent module's Tofu state, then creates the NVIDIA GPU Operator `helm_release`, three `kubernetes_deployment_v1` resources, six `kubernetes_service_v1` resources (ClusterIP + NodePort per workload), and three `kubernetes_ingress_v1` resources — all through the Terraform Kubernetes and Helm providers, which authenticate to the cluster via `aws eks get-token`.
+
+### Dashboard
+
+After the successful completion of the `make workloads` step, the dashboard should be available and displaying the outputs of the two worker nodes; YOLO computer vision in the left pane, and a LLM chat in the right pane, as shown in the images below.
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/yolo.png" alt="YOLO real-time object detection stream"></p>
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/llm-chat.png" alt="LLM chat interface"></p>
+
+
+
+### 6. Teardown
+
+```bash
+make destroy ENDPOINT=https://<host>:9999
+```
+
+Workloads are destroyed before infra. Both GPU worker instances terminate, immediately returning their RTX Pro 6000s to the Spinifex pool.
+
+The Makefile runs the two-module destroy sequence — `tofu -chdir=workbook/workloads destroy` first, then `tofu -chdir=workbook destroy` — because the parent module's security group rules are referenced by the ALB created in the workloads layer. Running Tofu directly in that order is equivalent.
+
+## Troubleshooting
+
+### `llm-server` or `yolo-stream` pod stuck in `Pending`
+
+The GPU Operator DaemonSet must complete before `nvidia.com/gpu` appears in node allocatables:
+
+```bash
+kubectl -n gpu-operator get daemonset -w
+kubectl get node -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
+```
+
+If a pod shows `Insufficient nvidia.com/gpu` on a node that looks healthy, the `workload=` labels from `make sideload` may be stale — for example, after a nodegroup recreation that assigned new node names. Re-run `make sideload` to relabel the nodes and re-import the images.
+
+### GPU worker instances fail to launch (`bind ... to vfio-pci: invalid argument`)
+
+On hosts where the GPU's IOMMU group contains an upstream PCIe root-port bridge (no ACS isolation), Spinifex attempts to bind the bridge to `vfio-pci`. Bridges are non-endpoint devices that `vfio-pci` refuses to bind, causing the instance to crash immediately after launch:
+
+```
+GPU claim failed... bind IOMMU group member 0000:14:02.0: bind 0000:14:02.0 to vfio-pci: invalid argument
+```
+
+Check IOMMU group membership with `lspci -nnk` and `/sys/kernel/iommu_groups/*/devices/`. This is fixed in Spinifex by excluding bridge-class PCI devices from the bind lifecycle — ensure your Spinifex build includes that fix. If the failed bind left a bridge without a driver, restore it:
+
+```bash
+echo | sudo tee /sys/bus/pci/devices/<bridge-addr>/driver_override
+echo <bridge-addr> | sudo tee /sys/bus/pci/drivers/pcieport/bind
+```
+
+If the nodegroup got wedged in `CREATING` after hitting this, delete it and let Terraform recreate it:
+
+```bash
+aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
+```
+
+### Nodegroup stuck `CREATING` with healthy nodes, Tofu times out
+
+`tofu apply` hangs for 20 minutes and fails with `workers did not become Ready: timed out`, even though `kubectl get nodes` shows both workers `Ready`. The cause is a missing `eks.amazonaws.com/nodegroup` label on the node — without it, the control plane never tallies the nodegroup as satisfied. Confirm with:
+
+```bash
+kubectl get node <worker> -o jsonpath='{.metadata.labels}'
+```
+
+This is fixed upstream in Spinifex. If stuck, delete both the nodegroup and cluster and let Terraform recreate them cleanly:
+
+```bash
+aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
+aws eks delete-cluster --name ai-platform
+```
+
+### ALB returns 502 immediately after rollout
+
+Both `llm-server` and `yolo-stream` require a few seconds after container start before their readiness probes pass — CUDA and GPU Operator initialisation contribute to the delay. The ALB marks targets unhealthy during this window. Watch the pods become ready:
+
+```bash
+kubectl -n inference get pods -w
+kubectl -n inference logs -f deploy/llm-server
+```
+
+## Conclusion
+
+This guide demonstrates how Spinifex turns a single bare-metal chassis into a production-shaped AI serving platform managed entirely with standard AWS tooling. IAM roles, ECR repositories, an EKS cluster, GPU worker nodes, addons, and an ALB are all provisioned with the same Terraform resources and AWS CLI commands that work on real AWS — with a single `AWS_PROFILE` swap.
+
+The RTX Pro 6000 Blackwell Server Edition's 96 GiB GDDR7 fits substantial GPU workloads in a single PCIe slot, and Spinifex's `g7e` instance family exposes each GPU as a standard EC2 instance. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node and retain the full EKS workflow — from `aws ecr get-login-password` to `kubectl get ingress` — on hardware they own. ECR acts as the canonical registry throughout: image build, push, and authentication are identical to AWS, with direct kubelet pulls being the intended path as Spinifex's ECR gateway matures.

go.mod

@@ -5,8 +5,9 @@ go 1.26.5
 require (
 	github.com/aws/aws-sdk-go v1.55.8
 	github.com/aws/aws-sdk-go-v2 v1.42.1
-	github.com/aws/aws-sdk-go-v2/config v1.32.29
-	github.com/aws/aws-sdk-go-v2/credentials v1.19.28
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
+	github.com/aws/aws-sdk-go-v2/config v1.32.30
+	github.com/aws/aws-sdk-go-v2/credentials v1.19.29
 	github.com/cenkalti/backoff/v4 v4.3.0
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
@@ -18,8 +19,10 @@ require (
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/klauspost/cpuid/v2 v2.4.0
-	github.com/mulgadc/predastore v1.12.0
-	github.com/mulgadc/viperblock v1.12.0
+	github.com/miekg/dns v1.1.72
+	github.com/mulgadc/northstar v1.13.0
+	github.com/mulgadc/predastore v1.13.0
+	github.com/mulgadc/viperblock v1.13.0
 	github.com/nats-io/nats-server/v2 v2.14.3
 	github.com/nats-io/nats.go v1.52.0
 	github.com/opencontainers/runtime-spec v1.3.0
@@ -64,12 +67,15 @@ require (
 	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 // indirect
-	github.com/aws/aws-sdk-go-v2/service/signin v1.4.0 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.32.0 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.44.0 // indirect
-	github.com/aws/smithy-go v1.27.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 // indirect
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.105.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/signin v1.4.1 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.32.1 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.1 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.44.1 // indirect
+	github.com/aws/smithy-go v1.27.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -104,7 +110,7 @@ require (
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fatih/color v1.19.0 // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
-	github.com/fsnotify/fsnotify v1.9.0 // indirect
+	github.com/fsnotify/fsnotify v1.10.1 // indirect
 	github.com/gabriel-vasile/mimetype v1.4.13 // indirect
 	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
@@ -189,7 +195,7 @@ require (
 	golang.org/x/text v0.40.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
 	golang.org/x/tools v0.47.0 // indirect
-	golang.org/x/vuln v1.1.4 // indirect
+	golang.org/x/vuln v1.4.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260618152121-87f3d3e198d3 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3 // indirect
 	google.golang.org/grpc v1.81.1 // indirect

go.sum

@@ -531,13 +531,13 @@ github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDk
 github.com/aws/aws-sdk-go-v2/config v1.18.14/go.mod h1:0pI6JQBHKwd0JnwAZS3VCapLKMO++UL2BOkWwyyzTnA=
 github.com/aws/aws-sdk-go-v2/config v1.27.4/go.mod h1:zq2FFXK3A416kiukwpsd+rD4ny6JC7QSkp4QdN1Mp2g=
 github.com/aws/aws-sdk-go-v2/config v1.29.12/go.mod h1:xse1YTjmORlb/6fhkWi8qJh3cvZi4JoVNhc+NbJt4kI=
-github.com/aws/aws-sdk-go-v2/config v1.32.29 h1:BcMHHnpiWKogf+gGfpj3K1w+Sktz29XDo/cPSAPO3FU=
-github.com/aws/aws-sdk-go-v2/config v1.32.29/go.mod h1:+Kbhn8Es4kPUph3F/0W7avykytc+Jh2Ld9/msv9ljV4=
+github.com/aws/aws-sdk-go-v2/config v1.32.30 h1:XwsEzpTJfQYJbFicz/QMLwAZdyeNVVoOEkbF7R3gPJk=
+github.com/aws/aws-sdk-go-v2/config v1.32.30/go.mod h1:Ud32SuMc+/9BGxfpSVld7HrE2o05JwKmXY4M3jOQNZU=
 github.com/aws/aws-sdk-go-v2/credentials v1.13.14/go.mod h1:85ckagDuzdIOnZRwws1eLKnymJs3ZM1QwVC1XcuNGOY=
 github.com/aws/aws-sdk-go-v2/credentials v1.17.4/go.mod h1:+30tpwrkOgvkJL1rUZuRLoxcJwtI/OkeBLYnHxJtVe0=
 github.com/aws/aws-sdk-go-v2/credentials v1.17.65/go.mod h1:4zyjAuGOdikpNYiSGpsGz8hLGmUzlY8pc8r9QQ/RXYQ=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.28 h1:zTXJSsNcoO91/mTXsZoYf0AK8dvNPiA58/VtyGXR+wM=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.28/go.mod h1:Kd9E0JzDBW/q1xbsHFrev/GnbAf5J0Ng8xoyc7HZ91Q=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.29 h1:WHZGssHH887cO0ox07SIQZsFx3MKD4ps6w0xUEmnKYQ=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.29/go.mod h1:Mhl0xR6zjguiuj00XRx2wMx22sAltk7oya39sT7fdg8=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.23/go.mod h1:mOtmAg65GT1HIL/HT/PynwPbS+UG0BgCZ6vhkPqnxWo=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.2/go.mod h1:iRlGzMix0SExQEviAyptRWRGdYNo3+ufW/lCzvKVTUc=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30/go.mod h1:Jpne2tDnYiFascUEs2AWHJL9Yp7A5ZVy3TNyxaAjD6M=
@@ -571,31 +571,31 @@ github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 h1:/Z5jmNrK
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30/go.mod h1:lEzEZnOosE7zi8Z6royW1cFJTD9fpab4Ul1SBrllewk=
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 h1:uao4A3QZ5UmB326V6KF+qRpv9Tjz7IlnlnTbbANntlU=
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31/go.mod h1:I/1+z0VwL1GhQyLgkoHDlygpUZ+iTAwOQ/NsftiUL2I=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 h1:XptwLL+UHXgafYMIHTy59IRovLbhz3znkxY2uS/pbXU=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0/go.mod h1:zdmCoFO/dSI7GlrwsPqFJI+WlFnSU4Tc8TJnlXrM1Do=
-github.com/aws/aws-sdk-go-v2/service/signin v1.4.0 h1:sLzmJGCMv+C8KqiJgEqDLB6vxaJGmobRh4rr//ZpA3w=
-github.com/aws/aws-sdk-go-v2/service/signin v1.4.0/go.mod h1:mxC0nT/C8wMMS97DemZPzvUZxvIt+2Iq+eS3JdFZGgg=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.105.2 h1:5C00eQYpTrgQXnp6V3P6P7zPElna3AXvlukbANE6nJI=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.105.2/go.mod h1:zdmCoFO/dSI7GlrwsPqFJI+WlFnSU4Tc8TJnlXrM1Do=
+github.com/aws/aws-sdk-go-v2/service/signin v1.4.1 h1:V7ZZ300WPXGjvkyore5DGe0ljVPOxCXie/thWdtSBXE=
+github.com/aws/aws-sdk-go-v2/service/signin v1.4.1/go.mod h1:mxC0nT/C8wMMS97DemZPzvUZxvIt+2Iq+eS3JdFZGgg=
 github.com/aws/aws-sdk-go-v2/service/sso v1.12.3/go.mod h1:jtLIhd+V+lft6ktxpItycqHqiVXrPIRjWIsFIlzMriw=
 github.com/aws/aws-sdk-go-v2/service/sso v1.20.1/go.mod h1:RsYqzYr2F2oPDdpy+PdhephuZxTfjHQe7SOBcZGoAU8=
 github.com/aws/aws-sdk-go-v2/service/sso v1.25.2/go.mod h1:qs4a9T5EMLl/Cajiw2TcbNt2UNo/Hqlyp+GiuG4CFDI=
-github.com/aws/aws-sdk-go-v2/service/sso v1.32.0 h1:qjMmry/cBDee1E/2gyvel0uRYCi3mwRZ2hf6N+GAodo=
-github.com/aws/aws-sdk-go-v2/service/sso v1.32.0/go.mod h1:u8af9Nqkmqnr96f7v9nHqzZT9XBwbXEkTiqT4ROuJSE=
+github.com/aws/aws-sdk-go-v2/service/sso v1.32.1 h1:gYFYh4iLLcAOJRLNPY2aD2g9DIhKn4eof8UkIrr1rTk=
+github.com/aws/aws-sdk-go-v2/service/sso v1.32.1/go.mod h1:u8af9Nqkmqnr96f7v9nHqzZT9XBwbXEkTiqT4ROuJSE=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.3/go.mod h1:zVwRrfdSmbRZWkUkWjOItY7SOalnFnq/Yg2LVPqDjwc=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.1/go.mod h1:YjAPFn4kGFqKC54VsHs5fn5B6d+PCY2tziEa3U/GB5Y=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.0/go.mod h1:MlYRNmYu/fGPoxBQVvBYr9nyr948aY/WLUvwBMBJubs=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0 h1:fpOlDPI55HdszaxapEGk6HsGosOUaM2YPWJpjMgp8UI=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0/go.mod h1:DMPWJBjYs6+3+f/qhBFEFPPlQ6NlhWjai3dJNvipJ84=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.1 h1:arjT9Cm3/WYbGmD5TUZHk4UQn4Lle1fUNZs5FC6CtF0=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.1/go.mod h1:DMPWJBjYs6+3+f/qhBFEFPPlQ6NlhWjai3dJNvipJ84=
 github.com/aws/aws-sdk-go-v2/service/sts v1.18.4/go.mod h1:1mKZHLLpDMHTNSYPJ7qrcnCQdHCWsNQaT0xRvq2u80s=
 github.com/aws/aws-sdk-go-v2/service/sts v1.28.1/go.mod h1:uQ7YYKZt3adCRrdCBREm1CD3efFLOUNH77MrUCvx5oA=
 github.com/aws/aws-sdk-go-v2/service/sts v1.33.17/go.mod h1:cQnB8CUnxbMU82JvlqjKR2HBOm3fe9pWorWBza6MBJ4=
-github.com/aws/aws-sdk-go-v2/service/sts v1.44.0 h1:bLZ0PolJ8J+HkJHztcXORUpHXBye2U8298lCEMi6ZCU=
-github.com/aws/aws-sdk-go-v2/service/sts v1.44.0/go.mod h1:9gdl4RrflIdpDb2TlXshWgR1F9TeCkvqDx77Vpr4Z/Q=
+github.com/aws/aws-sdk-go-v2/service/sts v1.44.1 h1:RvfHDg+xvAeZ+5741vUEjpOVtYSIm93W2zhx10Xtydw=
+github.com/aws/aws-sdk-go-v2/service/sts v1.44.1/go.mod h1:9gdl4RrflIdpDb2TlXshWgR1F9TeCkvqDx77Vpr4Z/Q=
 github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
 github.com/aws/smithy-go v1.20.1/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
 github.com/aws/smithy-go v1.22.2/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg=
 github.com/aws/smithy-go v1.22.3/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI=
-github.com/aws/smithy-go v1.27.3 h1:F3Zb497UhhskkfpJmfkXswyo+t0sh9OTBnIHjogWbVY=
-github.com/aws/smithy-go v1.27.3/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/smithy-go v1.27.4 h1:JQcphmBN4f0q/sPqXqROIItRNV/hy10cgu7CsFy616M=
+github.com/aws/smithy-go v1.27.4/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/bazelbuild/rules_go v0.49.0/go.mod h1:Dhcz716Kqg1RHNWos+N6MlXNkjNP2EwZQ0LukRKJfMs=
@@ -814,8 +814,8 @@ github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8/go.mod h1:ZhphrRTfi2
 github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
 github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
-github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
+github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@@ -1276,6 +1276,8 @@ github.com/mdlayher/packet v1.1.2/go.mod h1:GEu1+n9sG5VtiRE4SydOmX5GTwyyYlteZiFU
 github.com/mdlayher/socket v0.5.1 h1:VZaqt6RkGkt2OE9l3GcC6nZkqD3xKeQLyfleW/uBcos=
 github.com/mdlayher/socket v0.5.1/go.mod h1:TjPLHI1UgwEv5J1B5q0zTZq12A/6H7nKmtTanQE37IQ=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
+github.com/miekg/dns v1.1.72 h1:vhmr+TF2A3tuoGNkLDFK9zi36F2LS+hKTRW0Uf8kbzI=
+github.com/miekg/dns v1.1.72/go.mod h1:+EuEPhdHOsfk6Wk5TT2CzssZdqkmFhf8r+aVyDEToIs=
 github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY=
 github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3/go.mod h1:RagcQ7I8IeTMnF8JTXieKnO4Z6JCsikNEzj0DwauVzE=
 github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
@@ -1315,10 +1317,12 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.12.0 h1:JmoF//cIr3DgJhGypNVVLtaBZGCVUDzOo6z63407x54=
-github.com/mulgadc/predastore v1.12.0/go.mod h1:kgTncuCaB11e3eCQeZF0ydEFG0biuOlrZ4DlHNzwvZY=
-github.com/mulgadc/viperblock v1.12.0 h1:1PYG7GQ1cBbHbLI7QB7IzVu6EeKhrtcAJ+VAvNczjs8=
-github.com/mulgadc/viperblock v1.12.0/go.mod h1:5MGciKEvqpwLv2pdjk9j7PR8NHtXPs4pJzJi4UJojeU=
+github.com/mulgadc/northstar v1.13.0 h1:L9PyYZ9Klg7dv2kE5+DHOcgPHelaZRDIvnwBsbHsdGI=
+github.com/mulgadc/northstar v1.13.0/go.mod h1:erQE/PzK9UMtgH8Y+uI5RdiboO/nqe1CUJbIpkae/FM=
+github.com/mulgadc/predastore v1.13.0 h1:jDPpLJP/GA4WK2t+9lksTSFDuQiV8vgDl6ISnPJRUuo=
+github.com/mulgadc/predastore v1.13.0/go.mod h1:38YSuk9zvYmbmgyM13WTkIQPJtvo2KmGJBYJul6NeAE=
+github.com/mulgadc/viperblock v1.13.0 h1:NuqmPRm26/j58GKbzcyshm0uZwdobzdjKK1qEsb2pCY=
+github.com/mulgadc/viperblock v1.13.0/go.mod h1:sW0hxZjN7WDQQZ39L5mE58fVLbSv3mBP3/ifFPdwJtU=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -2570,8 +2574,8 @@ golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnps
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
-golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
-golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
+golang.org/x/vuln v1.4.0 h1:FpmTZiV4PyqY3lFfuCkz1JftEXb/+8M2NEkjJM5TF4g=
+golang.org/x/vuln v1.4.0/go.mod h1:FJ7XyKs83nAdxQ7PMsia2PoynwZMJ/QajXVMBBIgFe8=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

scripts/images/eks-node-gpu/k3s-agent.service

@@ -20,8 +20,9 @@ TasksMax=infinity
 LimitNOFILE=1048576
 LimitNPROC=infinity
 LimitCORE=infinity
-# Cloud-init drops K3S_URL/K3S_TOKEN/K3S_NODE_NAME/K3S_NODE_LABEL here at
-# RunInstances time per the Nodegroup bootstrap flow.
+# Cloud-init drops K3S_URL/K3S_TOKEN/K3S_NODE_NAME (plus SPINIFEX_NODEGROUP and,
+# for GPU workers, SPINIFEX_GPU_NODE) here at RunInstances time per the Nodegroup
+# bootstrap flow; mulga-eks-provider-id reads the SPINIFEX_* vars for node-labels.
 EnvironmentFile=-/etc/spinifex-eks/agent.env
 # Tee k3s (and its embedded kubelet's klog) stderr to the serial console as well
 # as journald: Ubuntu logs to journald only, which is off-box-invisible on a

scripts/images/eks-node-gpu/mulga-eks-provider-id.service

@@ -1,6 +1,13 @@
 [Unit]
 Description=Set K3s provider-id + topology labels from IMDS for EBS CSI
-After=network-online.target
+# After cloud-config.service so cloud-init has written /etc/spinifex-eks/agent.env
+# (write_files completes by the config stage): this script folds the nodegroup +
+# GPU node-labels (read from agent.env) into the same node-label list, so agent.env
+# must exist before it runs. NOT cloud-final.service: that unit is ordered after
+# multi-user.target and k3s-agent.service (ordered After this unit) is
+# WantedBy=multi-user.target, so chaining onto cloud-final forms an ordering cycle
+# that makes systemd skip both this unit and k3s-agent, so the worker never joins.
+After=network-online.target cloud-config.service
 Wants=network-online.target
 Before=k3s-agent.service
 

scripts/images/eks-node/addons/nvidia-device-plugin/0.17.4/10-daemonset.yaml

@@ -1,6 +1,7 @@
 # nvidia-device-plugin 0.17.4 — CDI-mode DaemonSet exposing nvidia.com/gpu
 # allocatable on GPU-tainted nodes. Gated to GPU nodes via nodeSelector on the
-# K3S_NODE_LABEL nvidia.com/gpu.present=true (nodegroup_userdata.go); inert
+# node-label nvidia.com/gpu.present=true (written by mulga-eks-provider-id from
+# the SPINIFEX_GPU_NODE agent.env signal); inert
 # elsewhere. --device-list-strategy=cdi-cri makes the plugin generate a CDI spec
 # for GPU workloads (containerd 2.x resolves the CRI-passed CDI devices; no
 # enable_cdi drop-in needed on k3s v1.32.5). The plugin's OWN container needs

scripts/images/eks-node/mulga-eks-provider-id.initd

@@ -2,14 +2,16 @@
 # mulga-eks-provider-id — boot oneshot that writes a K3s drop-in from IMDS
 # before K3s starts: the kubelet provider-id plus region/zone topology labels,
 # the node metadata the aws-ebs-csi-driver needs for NodeGetInfo + provisioning.
-# Ordered after local (which installs the IMDS on-link route) and before both
-# K3s roles. Role-agnostic: enabled on every node (server + agent).
+# Also folds the worker's nodegroup + GPU node-labels (from agent.env) into the
+# same node-label list, so it must run after cloud-init has written agent.env
+# (cloud-config/cloud-final stages, same as eks-node-role) and before the role
+# selector starts K3s. Role-agnostic: enabled on every node (server + agent).
 
 description="Set K3s provider-id + topology labels from IMDS for EBS CSI"
 
 depend() {
-    after net local
-    before k3s k3s-agent
+    after net local cloud-config cloud-final
+    before k3s k3s-agent eks-node-role
 }
 
 start() {

scripts/images/eks-node/mulga-eks-provider-id.sh

@@ -17,13 +17,22 @@ set -eu
 # family"). This drop-in supplies all four; region is derived from the AZ
 # (trailing zone letters stripped).
 #
-# K3s merges /etc/rancher/k3s/config.yaml.d/*.yaml over config.yaml; list keys
-# such as kubelet-arg are concatenated, so this adds the flag without touching
-# the cloud-init-written main config. Role-agnostic: runs on server + agent.
+# This drop-in is also the SINGLE writer of the node-label list. K3s replaces
+# (not merges) node-label across config.yaml.d files, so the last-sorted file
+# wins outright and any other node-label drop-in would silently wipe the rest.
+# The worker's nodegroup label (and, on GPU workers, the nvidia.com/gpu.present
+# label + NoSchedule taint) therefore ride here too, read from agent.env, so
+# they share the one list with the topology labels. On the K3s server there is
+# no agent.env, so only the topology labels are written — behavior unchanged.
+#
+# kubelet-arg lists ARE concatenated by k3s across drop-ins, so the provider-id
+# flag coexists with other files' kubelet-arg without touching the main config.
+# Role-agnostic: runs on server + agent.
 
 IMDS="http://169.254.169.254/latest"
 DROPIN_DIR=/etc/rancher/k3s/config.yaml.d
 DROPIN="${DROPIN_DIR}/10-provider-id.yaml"
+AGENT_ENV=/etc/spinifex-eks/agent.env
 LOGTAG="mulga-eks-provider-id"
 
 log() { echo "[${LOGTAG}] $*"; }
@@ -79,13 +88,32 @@ case "${itype}" in
     *) itype="m5.large" ;;
 esac
 
+# Fold the worker's nodegroup + GPU node-labels into the same list. agent.env is
+# written by cloud-init and absent on the server; source it only if present.
+nodegroup=""
+gpu_node=""
+if [ -f "${AGENT_ENV}" ]; then
+    # shellcheck disable=SC1090
+    . "${AGENT_ENV}"
+    nodegroup="${SPINIFEX_NODEGROUP:-}"
+    gpu_node="${SPINIFEX_GPU_NODE:-}"
+fi
+
 mkdir -p "${DROPIN_DIR}"
-cat > "${DROPIN}" <<EOF
-kubelet-arg:
-  - "provider-id=aws:///${az}/${iid}"
-node-label:
-  - "topology.kubernetes.io/region=${region}"
-  - "topology.kubernetes.io/zone=${az}"
-  - "node.kubernetes.io/instance-type=${itype}"
-EOF
-log "wrote provider-id aws:///${az}/${iid}, region=${region} zone=${az} type=${itype}"
+{
+    echo "kubelet-arg:"
+    echo "  - \"provider-id=aws:///${az}/${iid}\""
+    echo "node-label:"
+    echo "  - \"topology.kubernetes.io/region=${region}\""
+    echo "  - \"topology.kubernetes.io/zone=${az}\""
+    echo "  - \"node.kubernetes.io/instance-type=${itype}\""
+    if [ -n "${nodegroup}" ]; then
+        echo "  - \"eks.amazonaws.com/nodegroup=${nodegroup}\""
+    fi
+    if [ "${gpu_node}" = "true" ]; then
+        echo "  - \"nvidia.com/gpu.present=true\""
+        echo "node-taint:"
+        echo "  - \"nvidia.com/gpu=present:NoSchedule\""
+    fi
+} > "${DROPIN}"
+log "wrote provider-id aws:///${az}/${iid}, region=${region} zone=${az} type=${itype} nodegroup=${nodegroup:-<none>} gpu=${gpu_node:-false}"

scripts/publish-system-image.sh

@@ -67,7 +67,10 @@ if [[ -z "${IMAGE_NAME:-}" ]]; then
 fi
 
 # Mirror build-system-image.sh's derived paths so we reuse its cached raw.
-BUILD_DIR="/tmp/${IMAGE_NAME}-image-build"
+# Honor SYSTEM_IMAGE_BUILD_DIR exactly as the build script does — the workflow
+# points it off tmpfs so the 16G Ubuntu GPU raw fits, and the publish step must
+# look where the build actually wrote, not a hardcoded /tmp.
+BUILD_DIR="${SYSTEM_IMAGE_BUILD_DIR:-/tmp}/${IMAGE_NAME}-image-build"
 RAW_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-${DISTRO}.raw"
 
 OBJECT_BASE="${AMI_NAME:-$IMAGE_NAME}-${ARCH}"

scripts/reset-dev-env.sh

@@ -275,5 +275,41 @@ sleep 5
 echo "==> Building and installing microVM artifacts"
 cd "$PROJECT_ROOT" && make build-lb-agent install-microvm
 
+# --- Authorize SSH ingress on the default security group ---
+# Default SG denies all inbound, so smoke-test.sh's SSH probe times out.
+# Open port 22 on the default VPC's default SG before the smoke test runs.
+aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
+
+echo "==> Waiting for EC2 daemon"
+DAEMON_TIMEOUT=60
+DAEMON_ELAPSED=0
+while [ $DAEMON_ELAPSED -lt $DAEMON_TIMEOUT ]; do
+    if aws_as_user ec2 describe-security-groups --output text >/dev/null 2>&1; then
+        break
+    fi
+    sleep 2
+    DAEMON_ELAPSED=$((DAEMON_ELAPSED + 2))
+done
+if [ $DAEMON_ELAPSED -ge $DAEMON_TIMEOUT ]; then
+    echo "❌ EC2 daemon not ready after ${DAEMON_TIMEOUT}s"
+    exit 1
+fi
+
+echo "==> Authorizing SSH (port 22) on default security group"
+DEFAULT_VPC_ID=$(aws_as_user ec2 describe-vpcs \
+    --filters "Name=is-default,Values=true" \
+    --query 'Vpcs[0].VpcId' --output text)
+DEFAULT_SG_ID=$(aws_as_user ec2 describe-security-groups \
+    --filters "Name=vpc-id,Values=$DEFAULT_VPC_ID" "Name=group-name,Values=default" \
+    --query 'SecurityGroups[0].GroupId' --output text)
+if [ -z "$DEFAULT_SG_ID" ] || [ "$DEFAULT_SG_ID" = "None" ]; then
+    echo "❌ Could not locate default security group in VPC $DEFAULT_VPC_ID"
+    exit 1
+fi
+aws_as_user ec2 authorize-security-group-ingress \
+    --group-id "$DEFAULT_SG_ID" --protocol tcp --port 22 --cidr 0.0.0.0/0 \
+    >/dev/null 2>&1 || true
+echo "  Authorized SSH on $DEFAULT_SG_ID ($DEFAULT_VPC_ID)"
+
 # --- Smoke test ---
 sudo -u "$INVOKING_USER" "$SCRIPT_DIR/smoke-test.sh"

scripts/setup.sh

@@ -15,7 +15,8 @@
 #   VERBOSE                    Set to 1 to echo "[setup] <stage>" before each top-level step.
 #   SETUP_STAGES               Comma-separated subset of stages to run:
 #                                deps, aws, users, sudoers, files, directories,
-#                                env, systemd, logrotate, udev, fixown, migrations
+#                                env, systemd, logrotate, udev, fixown, resolved,
+#                                migrations
 #                              Unset = run every stage appropriate for the current mode.
 
 set -e
@@ -139,11 +140,12 @@ create_service_users() {
         [gw]="/var/lib/spinifex/awsgw"
         [daemon]="/var/lib/spinifex/spinifex"
         [storage]="/var/lib/spinifex/predastore"
+        [northstar]="/var/lib/spinifex/northstar"
         [viperblock]="/var/lib/spinifex/viperblock"
         [vpcd]="/var/lib/spinifex"
         [ui]="/var/lib/spinifex"
     )
-    for svc in nats gw daemon storage viperblock vpcd ui; do
+    for svc in nats gw daemon storage northstar viperblock vpcd ui; do
         local user="spinifex-${svc}"
         if ! id "$user" > /dev/null 2>&1; then
             $SUDO useradd --system --no-create-home \
@@ -175,7 +177,7 @@ create_service_users() {
     # Daemon consumes viperblock's NBD socket — join the producer-typed group.
     $SUDO usermod -aG spinifex-viperblock spinifex-daemon
 
-    info "Service users created (spinifex-{nats,gw,daemon,storage,viperblock,vpcd,ui})"
+    info "Service users created (spinifex-{nats,gw,daemon,storage,northstar,viperblock,vpcd,ui})"
 }
 
 # --- Install scoped sudoers rules ---
@@ -421,6 +423,12 @@ TMPEOF
     $SUDO chown "spinifex-storage:$SPINIFEX_GROUP" /etc/spinifex/predastore
     $SUDO chmod 0750 /etc/spinifex/predastore
 
+    # Northstar holds northstar.toml (bucket-scoped S3 creds, written 0600 by
+    # `spx admin init`); fix_file_ownership reassigns it to spinifex-northstar.
+    $SUDO mkdir -p /etc/spinifex/northstar
+    $SUDO chown "spinifex-northstar:$SPINIFEX_GROUP" /etc/spinifex/northstar
+    $SUDO chmod 0750 /etc/spinifex/northstar
+
     $SUDO mkdir -p /etc/spinifex/awsgw
     $SUDO chown "spinifex-gw:$SPINIFEX_GROUP" /etc/spinifex/awsgw
     $SUDO chmod 0750 /etc/spinifex/awsgw
@@ -446,6 +454,10 @@ TMPEOF
     $SUDO chown "spinifex-storage:$SPINIFEX_GROUP" /var/lib/spinifex/predastore
     $SUDO chmod 0700 /var/lib/spinifex/predastore
 
+    $SUDO mkdir -p /var/lib/spinifex/northstar
+    $SUDO chown "spinifex-northstar:$SPINIFEX_GROUP" /var/lib/spinifex/northstar
+    $SUDO chmod 0700 /var/lib/spinifex/northstar
+
     $SUDO mkdir -p /var/lib/spinifex/viperblock
     $SUDO chown "spinifex-viperblock:$SPINIFEX_GROUP" /var/lib/spinifex/viperblock
     $SUDO chmod 0700 /var/lib/spinifex/viperblock
@@ -517,6 +529,7 @@ fix_file_ownership() {
     for entry in \
         nats:spinifex-nats \
         predastore:spinifex-storage \
+        northstar:spinifex-northstar \
         spinifex:spinifex-daemon \
         viperblock:spinifex-viperblock \
         vpcd:spinifex-vpcd \
@@ -537,6 +550,10 @@ fix_file_ownership() {
         $SUDO chown -R "spinifex-storage:$SPINIFEX_GROUP" /etc/spinifex/predastore \
             || fatal "Failed to set ownership on /etc/spinifex/predastore"
     fi
+    if [ -d /etc/spinifex/northstar ]; then
+        $SUDO chown -R "spinifex-northstar:$SPINIFEX_GROUP" /etc/spinifex/northstar \
+            || fatal "Failed to set ownership on /etc/spinifex/northstar"
+    fi
     if [ -d /etc/spinifex/awsgw ]; then
         $SUDO chown -R "spinifex-gw:$SPINIFEX_GROUP" /etc/spinifex/awsgw \
             || fatal "Failed to set ownership on /etc/spinifex/awsgw"
@@ -777,6 +794,79 @@ EOF
     info "Swap enabled: $swapfile (${size_mb}MiB), vm.swappiness=10"
 }
 
+# --- Configure host split DNS for Spinifex zones (systemd-resolved) ---
+# When systemd-resolved manages the host resolver, route the Spinifex
+# authoritative zones (default_domain + compute.internal) to the local northstar
+# resolver on the node WAN IP, while every other name keeps using the existing
+# link/DHCP DNS. The "~" routing-domain prefix restricts the DNS= server to
+# exactly those zones (resolved.conf(5)). northstar binds <wan>:53 (not
+# 127.0.0.1), so DNS= must be the WAN IP. Idempotent; no-op without resolved.
+setup_resolved_dns() {
+    stage "configuring systemd-resolved split DNS for Spinifex zones"
+
+    # ISO chroot has no live resolver to reconfigure; firstboot handles it.
+    if [ "${ISO_BUILD:-0}" = "1" ]; then
+        info "Resolved split DNS skipped (ISO_BUILD=1; firstboot configures it)"
+        return
+    fi
+
+    # Only act when systemd-resolved is the active resolver; otherwise the
+    # operator must point their resolver at the node :53 manually.
+    if ! $SUDO systemctl is-active --quiet systemd-resolved 2>/dev/null; then
+        info "systemd-resolved not active — skipping host split DNS"
+        info "  To resolve the Spinifex zones, forward them to the node WAN IP on :53"
+        return
+    fi
+
+    # northstar binds :53 on the node WAN IP. Detect the default-route source.
+    local wan_ip
+    wan_ip=$(ip -4 route get 1.1.1.1 2>/dev/null | grep -oE 'src [0-9.]+' | awk '{print $2}')
+    [ -z "$wan_ip" ] && wan_ip=$(ip -4 route show default 2>/dev/null | grep -oE 'src [0-9.]+' | awk '{print $2}')
+    if [ -z "$wan_ip" ]; then
+        warn "Could not detect a WAN IP — skipping resolved split DNS"
+        warn "  Re-run once the WAN is up: sudo SETUP_STAGES=resolved $0"
+        return
+    fi
+
+    # Default zones match `spx admin init` defaults; honour a custom
+    # default_domain if northstar.toml already exists (post-init re-run).
+    local base_domain="spx3.net" internal_domain="compute.internal"
+    local ns_toml=/etc/spinifex/northstar/northstar.toml
+    if $SUDO test -f "$ns_toml"; then
+        local cfg_domain
+        cfg_domain=$($SUDO grep -E '^[[:space:]]*default_domain[[:space:]]*=' "$ns_toml" 2>/dev/null \
+            | head -1 | sed -E 's/.*=[[:space:]]*"?([^"#]*[^"# ])"?.*/\1/')
+        [ -n "$cfg_domain" ] && base_domain="$cfg_domain"
+    fi
+
+    local dropin_dir=/etc/systemd/resolved.conf.d
+    local dropin="$dropin_dir/spinifex-dns.conf"
+    local tmp
+    tmp=$(mktemp)
+    cat > "$tmp" << EOF
+# Generated by setup.sh — route the Spinifex authoritative zones to the local
+# northstar resolver on the node WAN IP. The "~" routing-domain prefix restricts
+# this DNS= server to exactly these zones; all other names use the link DNS.
+# northstar binds <wan>:53 (not 127.0.0.1), so DNS= is the WAN IP.
+[Resolve]
+DNS=${wan_ip}
+Domains=~${base_domain} ~${internal_domain}
+EOF
+
+    # Skip the resolved restart when nothing changed (idempotent re-runs).
+    if $SUDO test -f "$dropin" && $SUDO cmp -s "$tmp" "$dropin"; then
+        rm -f "$tmp"
+        info "Host split DNS already current: ${base_domain} + ${internal_domain} -> ${wan_ip}:53"
+        return
+    fi
+
+    $SUDO mkdir -p "$dropin_dir"
+    $SUDO install -m 0644 "$tmp" "$dropin"
+    rm -f "$tmp"
+    $SUDO systemctl restart systemd-resolved 2>/dev/null || true
+    info "Host split DNS: ${base_domain} + ${internal_domain} -> ${wan_ip}:53 (northstar)"
+}
+
 # --- Main ---
 main() {
     info "Spinifex installer"
@@ -813,6 +903,7 @@ main() {
     stage_enabled logrotate  && install_logrotate
     stage_enabled udev       && install_udev
     stage_enabled swap       && setup_swap
+    stage_enabled resolved   && setup_resolved_dns
 
     # Migrations are only safe on a live system (need a running NATS and a
     # persisted config file). Skip under ISO_BUILD and under any explicit

spinifex/admin/admin.go

@@ -21,6 +21,7 @@ import (
 	"text/template"
 	"time"
 
+	"github.com/mulgadc/spinifex/spinifex/config"
 	toml "github.com/pelletier/go-toml/v2"
 	"gopkg.in/ini.v1"
 )
@@ -32,6 +33,12 @@ type RemoteNode struct {
 	Region   string
 	AZ       string
 	Services []string
+
+	// NorthstarConfigPath marks the peer as running northstar. Its emptiness is
+	// the only thing read from a peer's stanza — no code opens a peer's path —
+	// and it must be rendered for every northstar peer so each node derives the
+	// same nameserver set for the base zone seed.
+	NorthstarConfigPath string
 }
 
 type ConfigSettings struct {
@@ -107,6 +114,20 @@ type ConfigSettings struct {
 	// Empty means no key was provisioned and volumes are written cleartext
 	// (legacy mode); the template omits the field entirely in that case.
 	EncryptionKeyFile string
+
+	// Northstar (DNS) settings. The northstar service reads zones from a
+	// dedicated, read-only S3 bucket using bucket-scoped credentials rendered
+	// into predastore.toml ([[auth]]) and northstar.toml ([s3]).
+	NorthstarAccessKey      string
+	NorthstarSecretKey      string
+	NorthstarBucket         string // S3 bucket holding zone files (default "northstar")
+	NorthstarDefaultDomain  string // authoritative base domain (default "spx3.net")
+	NorthstarInternalDomain string // AWS-parity private zone (default "compute.internal")
+	NorthstarConfigPath     string // path to northstar.toml, rendered into spinifex.toml
+
+	// PoolDNSServers are the host-detected upstream DNS servers northstar forwards
+	// recursive queries to, rendered into northstar.toml's [recursion] nameservers.
+	PoolDNSServers []string
 }
 
 // PoolData is one [[network.external_pools]] block rendered into spinifex.toml.
@@ -142,7 +163,7 @@ type ConfigFile struct {
 func GenerateConfigFiles(configs []ConfigFile, configSettings ConfigSettings) error {
 	for _, cfg := range configs {
 		if err := GenerateConfigFile(cfg.Path, cfg.Template, configSettings); err != nil {
-			return fmt.Errorf("error creating %s: %v", cfg.Name, err)
+			return fmt.Errorf("error creating %s: %w", cfg.Name, err)
 		}
 		fmt.Printf("✅ Created: %s\n", cfg.Name)
 	}
@@ -219,7 +240,10 @@ func GenerateCertificatesIfNeeded(configDir string, force bool, bindIP string, a
 
 	if force || !FileExists(serverCertPath) || !FileExists(serverKeyPath) {
 		extraDNS := AWSGWServiceDNSNames(awsRegion, internalSuffix)
-		if err := GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, []string{bindIP}, extraDNS); err != nil {
+		// Always pin the canonical mgmt-bridge IP; the control plane publishes to
+		// it regardless of whether br-mgmt is up when this cert is minted.
+		extraIPs := []string{bindIP, config.DefaultMgmtBridgeIP}
+		if err := GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, extraIPs, extraDNS); err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
 			os.Exit(1)
 		}
@@ -247,7 +271,9 @@ func GenerateServerCertOnly(configDir string, bindIP, awsRegion, internalSuffix
 	}
 
 	extraDNS := AWSGWServiceDNSNames(awsRegion, internalSuffix)
-	return GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, []string{bindIP}, extraDNS)
+	// Always pin the canonical mgmt-bridge IP (see GenerateCertificatesIfNeeded).
+	extraIPs := []string{bindIP, config.DefaultMgmtBridgeIP}
+	return GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, extraIPs, extraDNS)
 }
 
 func CreateServiceDirectories(spxRoot string) {
@@ -350,6 +376,8 @@ func SetServiceOwnership() {
 		"/var/lib/spinifex/nats":       "spinifex-nats",
 		"/etc/spinifex/predastore":     "spinifex-storage",
 		"/var/lib/spinifex/predastore": "spinifex-storage",
+		"/etc/spinifex/northstar":      "spinifex-northstar",
+		"/var/lib/spinifex/northstar":  "spinifex-northstar",
 		"/etc/spinifex/viperblock":     "spinifex-viperblock",
 		"/var/lib/spinifex/spinifex":   "spinifex-daemon",
 		"/var/lib/spinifex/viperblock": "spinifex-viperblock",
@@ -481,6 +509,30 @@ func GenerateAWSSecretKey() (string, error) {
 	return base64.StdEncoding.EncodeToString(bytes), nil
 }
 
+// Northstar DNS defaults baked into config files at init time.
+const (
+	// NorthstarBucketName is the S3 bucket holding DNS zone files.
+	NorthstarBucketName = "northstar"
+	// NorthstarDefaultDomain is the authoritative base domain for internal names.
+	NorthstarDefaultDomain = "spx3.net"
+	// NorthstarInternalDomain is the AWS-parity private zone for ip-<addr> names.
+	NorthstarInternalDomain = "compute.internal"
+)
+
+// NorthstarCredentials is the bucket-scoped credential pair the northstar DNS
+// service reads zone files with, together with the bucket that scopes it.
+//
+// Every node in a cluster must be provisioned with the same pair: the zone
+// bucket is distributed, and each node's predastore only honours the keys
+// rendered into its own config, so a per-node pair would have node A's
+// predastore reject node B's resolver. A zero value provisions no northstar
+// credentials at all, which renders a config without any northstar stanza.
+type NorthstarCredentials struct {
+	AccessKey string
+	SecretKey string
+	Bucket    string
+}
+
 // SystemAccountID returns the system/root account ID (000000000000).
 // Used for service-to-service auth credentials baked into config files.
 func SystemAccountID() string {
@@ -498,7 +550,7 @@ func DefaultAccountName() string {
 	return "spinifex"
 }
 
-// GenerateNATSToken generates a secure random token for NATS
+// GenerateNATSToken generates a secure random token for NATS.
 func GenerateNATSToken() (string, error) {
 	bytes := make([]byte, 32)
 	if _, err := rand.Read(bytes); err != nil {
@@ -836,7 +888,12 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP string)
 // GenerateMultiNodePredastoreConfig produces a complete predastore.toml for a
 // multi-node Predastore cluster. Each node gets its own DB entry (port 6660)
 // and shard entry (port 9991) on a distinct IP. Node ID 1 is the bootstrap leader.
-func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNodeConfig, accessKey, secretKey, region, natsToken, configDir, bindIP string, compactionIntervalSeconds int) (string, error) {
+//
+// A populated northstar credential provisions the zone bucket, grants the
+// system key write access to it, and adds the read-only entry the resolver
+// authenticates with. A zero value omits all three, yielding a config no
+// northstar service can use.
+func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNodeConfig, accessKey, secretKey, region, natsToken, configDir, bindIP string, compactionIntervalSeconds int, northstar NorthstarCredentials) (string, error) {
 	if len(nodes) < 2 {
 		return "", fmt.Errorf("multi-node predastore requires at least 2 nodes, got %d", len(nodes))
 	}
@@ -850,7 +907,17 @@ func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNod
 		ConfigDir                 string
 		BindIP                    string
 		CompactionIntervalSeconds int
-	}{nodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionIntervalSeconds}
+		NorthstarAccessKey        string
+		NorthstarSecretKey        string
+		NorthstarBucket           string
+	}{
+		Nodes: nodes, AccessKey: accessKey, SecretKey: secretKey, Region: region,
+		NatsToken: natsToken, ConfigDir: configDir, BindIP: bindIP,
+		CompactionIntervalSeconds: compactionIntervalSeconds,
+		NorthstarAccessKey:        northstar.AccessKey,
+		NorthstarSecretKey:        northstar.SecretKey,
+		NorthstarBucket:           northstar.Bucket,
+	}
 
 	tmpl, err := template.New("predastore-multinode").Parse(templateStr)
 	if err != nil {

spinifex/admin/admin_test.go

@@ -11,10 +11,28 @@ import (
 	"strings"
 	"testing"
 
+	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
+// certHasIP reports whether a PEM-encoded cert carries ip as an IP SAN.
+func certHasIP(t *testing.T, certPath, ip string) bool {
+	t.Helper()
+	certPEM, err := os.ReadFile(certPath)
+	require.NoError(t, err)
+	block, _ := pem.Decode(certPEM)
+	require.NotNil(t, block)
+	cert, err := x509.ParseCertificate(block.Bytes)
+	require.NoError(t, err)
+	for _, got := range cert.IPAddresses {
+		if got.Equal(net.ParseIP(ip)) {
+			return true
+		}
+	}
+	return false
+}
+
 // --- Key / Token generation ---
 
 func TestGenerateAWSAccessKey_Format(t *testing.T) {
@@ -432,7 +450,7 @@ func TestGenerateSignedCert(t *testing.T) {
 		baseBlock, _ := pem.Decode(basePEM)
 		baseParsed, _ := x509.ParseCertificate(baseBlock.Bytes)
 
-		assert.Equal(t, len(baseParsed.IPAddresses), len(cert.IPAddresses),
+		assert.Len(t, cert.IPAddresses, len(baseParsed.IPAddresses),
 			"passing duplicate/special IPs should not add extra entries")
 	})
 
@@ -622,6 +640,8 @@ func TestGenerateSignedCert_IncludesHostname(t *testing.T) {
 // --- Certificate orchestrator ---
 
 // TestGenerateCertificatesIfNeeded uses subtests to share the initial generation.
+//
+//nolint:tparallel // subtests mutate the shared dir in order: SkipsWhenAllExist pins the CA modtime that ForcePreservesCARegeneratesServerCert then regenerates against
 func TestGenerateCertificatesIfNeeded(t *testing.T) {
 	t.Parallel()
 	dir := t.TempDir()
@@ -672,6 +692,17 @@ func TestGenerateCertificatesIfNeeded(t *testing.T) {
 		_, err = srvCert.Verify(x509.VerifyOptions{Roots: pool, KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}})
 		assert.NoError(t, err, "re-signed server cert must verify against preserved CA")
 	})
+
+	// The mgmt-bridge IP must be a SAN even though br-mgmt is not a live
+	// interface here (interface enumeration cannot discover it), mirroring a
+	// host where br-mgmt is down when the cert is minted. Without the explicit
+	// pin the control-plane publish to https://<mgmt-ip> would fail cert verify.
+	t.Run("MgmtBridgeIPAlwaysInSAN", func(t *testing.T) {
+		certDir := t.TempDir()
+		GenerateCertificatesIfNeeded(certDir, false, "10.0.0.5", "us-east-1", "spinifex.internal")
+		assert.True(t, certHasIP(t, filepath.Join(certDir, "server.pem"), config.DefaultMgmtBridgeIP),
+			"server cert must carry the canonical mgmt-bridge IP SAN regardless of br-mgmt state")
+	})
 }
 
 func TestGenerateServerCertOnly(t *testing.T) {
@@ -681,6 +712,7 @@ func TestGenerateServerCertOnly(t *testing.T) {
 	require.NoError(t, GenerateCACert(filepath.Join(caDir, "ca.pem"), filepath.Join(caDir, "ca.key")))
 
 	t.Run("Success", func(t *testing.T) {
+		t.Parallel()
 		dir := t.TempDir()
 		// Copy CA files into test dir
 		caCert, _ := os.ReadFile(filepath.Join(caDir, "ca.pem"))
@@ -708,9 +740,14 @@ func TestGenerateServerCertOnly(t *testing.T) {
 		// AWS-parity ECR SANs must be present.
 		assert.Contains(t, cert.DNSNames, "ecr.us-east-1.spinifex.internal")
 		assert.Contains(t, cert.DNSNames, "*.dkr.ecr.us-east-1.spinifex.internal")
+
+		// Joining nodes must also pin the mgmt-bridge IP regardless of br-mgmt state.
+		assert.True(t, certHasIP(t, filepath.Join(dir, "server.pem"), config.DefaultMgmtBridgeIP),
+			"server cert must carry the canonical mgmt-bridge IP SAN")
 	})
 
 	t.Run("MissingCA", func(t *testing.T) {
+		t.Parallel()
 		dir := t.TempDir()
 		err := GenerateServerCertOnly(dir, "10.0.0.5", "us-east-1", "spinifex.internal")
 		assert.Error(t, err)
@@ -773,18 +810,49 @@ host = "{{.Host}}"
 		{ID: 3, Host: "10.0.0.3"},
 	}
 
-	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0)
+	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0, NorthstarCredentials{})
 	require.NoError(t, err)
 	assert.Contains(t, result, `host = "10.0.0.1"`)
 	assert.Contains(t, result, `host = "10.0.0.3"`)
 }
 
+// The northstar template fields were declared but never assigned, so every
+// multi-node predastore config silently rendered the empty-key path: no zone
+// bucket and no credential the resolver could authenticate with.
+func TestGenerateMultiNodePredastoreConfig_NorthstarCredentialsReachTemplate(t *testing.T) {
+	tmpl := `access = "{{.NorthstarAccessKey}}" secret = "{{.NorthstarSecretKey}}" bucket = "{{.NorthstarBucket}}"`
+	nodes := []PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"},
+		{ID: 2, Host: "10.0.0.2"},
+	}
+
+	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0,
+		NorthstarCredentials{AccessKey: "NSAK", SecretKey: "NSSK", Bucket: "northstar"})
+	require.NoError(t, err)
+	assert.Equal(t, `access = "NSAK" secret = "NSSK" bucket = "northstar"`, result)
+}
+
+// A zero credential must leave every northstar field empty, which is what the
+// production template's guards key off to omit the stanzas entirely.
+func TestGenerateMultiNodePredastoreConfig_NoNorthstarCredentials(t *testing.T) {
+	tmpl := `access = "{{.NorthstarAccessKey}}" secret = "{{.NorthstarSecretKey}}" bucket = "{{.NorthstarBucket}}"`
+	nodes := []PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"},
+		{ID: 2, Host: "10.0.0.2"},
+	}
+
+	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0,
+		NorthstarCredentials{})
+	require.NoError(t, err)
+	assert.Equal(t, `access = "" secret = "" bucket = ""`, result)
+}
+
 func TestGenerateMultiNodePredastoreConfig_MinimumNodes(t *testing.T) {
 	tmpl := "{{range .Nodes}}{{.ID}}{{end}}"
 
 	_, err := GenerateMultiNodePredastoreConfig(tmpl, []PredastoreNodeConfig{
 		{ID: 1, Host: "10.0.0.1"},
-	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0)
+	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0, NorthstarCredentials{})
 	assert.Error(t, err)
 	assert.Contains(t, err.Error(), "at least 2 nodes")
 }
@@ -792,7 +860,7 @@ func TestGenerateMultiNodePredastoreConfig_MinimumNodes(t *testing.T) {
 func TestGenerateMultiNodePredastoreConfig_InvalidTemplate(t *testing.T) {
 	_, err := GenerateMultiNodePredastoreConfig("{{.Unclosed", []PredastoreNodeConfig{
 		{ID: 1, Host: "a"}, {ID: 2, Host: "b"}, {ID: 3, Host: "c"},
-	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0)
+	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0, NorthstarCredentials{})
 	assert.Error(t, err)
 	assert.Contains(t, err.Error(), "failed to parse")
 }

spinifex/admin/images_remove.go

@@ -320,7 +320,7 @@ func readAMIConfig(store objectstore.ObjectStore, bucket, imageID string) (viper
 
 	var state viperblock.VBState
 	if err := json.Unmarshal(viperblock.StateBody(body), &state); err != nil {
-		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %v", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
+		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %w", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
 	}
 	return state.VolumeConfig.AMIMetadata, nil
 }
@@ -352,7 +352,7 @@ func readVolumeConfig(store objectstore.ObjectStore, bucket, volumeID string) (*
 
 	var w volumeConfigWrapper
 	if err := json.Unmarshal(viperblock.StateBody(body), &w); err != nil {
-		return nil, fmt.Errorf("%w: %s: %v", errCorruptVolumeConfig, key, err)
+		return nil, fmt.Errorf("%w: %s: %w", errCorruptVolumeConfig, key, err)
 	}
 	return &w.VolumeConfig, nil
 }

spinifex/admin/images_remove_test.go

@@ -3,7 +3,6 @@ package admin
 import (
 	"bytes"
 	"encoding/json"
-	"errors"
 	"testing"
 
 	"github.com/aws/aws-sdk-go/aws"
@@ -157,7 +156,7 @@ func TestRemoveSystemImage_AccountOwned_ForceBypasses(t *testing.T) {
 
 	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
 	require.NoError(t, err)
-	assert.Greater(t, res.ObjectsDeleted, 0)
+	assert.Positive(t, res.ObjectsDeleted)
 	assert.Equal(t, 0, store.Count())
 }
 
@@ -196,7 +195,7 @@ func TestRemoveSystemImage_DependentVolume_Direct_Refused(t *testing.T) {
 	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
 	require.Error(t, err)
 	var depErr *DependentError
-	require.True(t, errors.As(err, &depErr))
+	require.ErrorAs(t, err, &depErr)
 	assert.Contains(t, depErr.Dependents.Volumes, "vol-aaa")
 }
 
@@ -214,7 +213,7 @@ func TestRemoveSystemImage_DependentVolume_Transitive_Refused(t *testing.T) {
 	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
 	require.Error(t, err)
 	var depErr *DependentError
-	require.True(t, errors.As(err, &depErr))
+	require.ErrorAs(t, err, &depErr)
 	assert.Contains(t, depErr.Dependents.Volumes, "vol-bbb")
 	assert.Contains(t, depErr.Dependents.Snapshots, derivedSnap)
 }
@@ -232,7 +231,7 @@ func TestRemoveSystemImage_DependentAMI_Refused(t *testing.T) {
 	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
 	require.Error(t, err)
 	var depErr *DependentError
-	require.True(t, errors.As(err, &depErr))
+	require.ErrorAs(t, err, &depErr)
 	assert.Contains(t, depErr.Dependents.AMIs, "ami-acct-002")
 }
 
@@ -245,7 +244,7 @@ func TestRemoveSystemImage_Force_OverridesDependents(t *testing.T) {
 
 	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
 	require.NoError(t, err)
-	assert.Greater(t, res.ObjectsDeleted, 0)
+	assert.Positive(t, res.ObjectsDeleted)
 	// vol-orphan/config.json remains; the AMI is gone.
 	_, err = store.GetObject(t.Context(), &awss3.GetObjectInput{
 		Bucket: aws.String(testRemoveBucket),

spinifex/admin/ipsec_test.go

@@ -14,6 +14,8 @@ import (
 
 // TestGenerateIPSecPeerCert shares a CA across subtests so the 4096-bit
 // CA-key generation runs once (~0.7s).
+//
+//nolint:tparallel // the parent cannot be parallel: it overrides the charonCATrustDir/charonRereadCAs package vars for the duration of the test
 func TestGenerateIPSecPeerCert(t *testing.T) {
 	// Redirect the charon trust-store symlink at a temp dir so the test
 	// doesn't try to write to /etc/ipsec.d/cacerts on the host. Restored

spinifex/admin/telemetry_test.go

@@ -10,7 +10,6 @@ import (
 	"time"
 
 	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
 )
 
 func TestReadMachineID_ReturnsNonEmpty(t *testing.T) {
@@ -29,8 +28,14 @@ func TestSendTelemetry_PostsCorrectPayload(t *testing.T) {
 		assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
 
 		body, err := io.ReadAll(r.Body)
-		require.NoError(t, err)
-		require.NoError(t, json.Unmarshal(body, &received))
+		if !assert.NoError(t, err) {
+			http.Error(w, "read request body", http.StatusBadRequest)
+			return
+		}
+		if !assert.NoError(t, json.Unmarshal(body, &received)) {
+			http.Error(w, "decode request body", http.StatusBadRequest)
+			return
+		}
 
 		w.WriteHeader(http.StatusNoContent)
 	}))

spinifex/awsec2query/parser_test.go

@@ -110,7 +110,7 @@ func TestQueryParamsToStruct_StopInstances(t *testing.T) {
 	assert.Len(t, input.InstanceIds, 2)
 	assert.Equal(t, "i-1234567890abcdef0", aws.StringValue(input.InstanceIds[0]))
 	assert.Equal(t, "i-0987654321fedcba0", aws.StringValue(input.InstanceIds[1]))
-	assert.Equal(t, true, aws.BoolValue(input.Force))
+	assert.True(t, aws.BoolValue(input.Force))
 }
 
 func TestQueryParamsToStruct_StartInstances(t *testing.T) {
@@ -249,7 +249,7 @@ func TestQueryParamsToStruct_BlockDeviceMappingsWithEBS(t *testing.T) {
 	assert.NotNil(t, bdm.Ebs)
 	assert.Equal(t, int64(100), aws.Int64Value(bdm.Ebs.VolumeSize))
 	assert.Equal(t, "gp3", aws.StringValue(bdm.Ebs.VolumeType))
-	assert.Equal(t, true, aws.BoolValue(bdm.Ebs.DeleteOnTermination))
+	assert.True(t, aws.BoolValue(bdm.Ebs.DeleteOnTermination))
 }
 
 func TestQueryParamsToStruct_BlockDeviceMappingsWithEphemeral(t *testing.T) {
@@ -312,8 +312,8 @@ func TestQueryParamsToStruct_MultipleBlockDeviceMappings(t *testing.T) {
 	assert.Equal(t, "snap-1234567890abcdef0", aws.StringValue(bdm1.Ebs.SnapshotId))
 	assert.Equal(t, int64(30), aws.Int64Value(bdm1.Ebs.VolumeSize))
 	assert.Equal(t, "gp3", aws.StringValue(bdm1.Ebs.VolumeType))
-	assert.Equal(t, true, aws.BoolValue(bdm1.Ebs.Encrypted))
-	assert.Equal(t, true, aws.BoolValue(bdm1.Ebs.DeleteOnTermination))
+	assert.True(t, aws.BoolValue(bdm1.Ebs.Encrypted))
+	assert.True(t, aws.BoolValue(bdm1.Ebs.DeleteOnTermination))
 
 	// Second mapping - Empty EBS
 	bdm2 := input.BlockDeviceMappings[1]
@@ -472,7 +472,7 @@ func TestQueryParamsToStruct_LocationNameDryRun(t *testing.T) {
 	err := QueryParamsToStruct(args, input)
 
 	assert.NoError(t, err)
-	assert.Equal(t, true, aws.BoolValue(input.DryRun))
+	assert.True(t, aws.BoolValue(input.DryRun))
 }
 
 func TestQueryParamsToStruct_NonStructPointer(t *testing.T) {

spinifex/awserrors/awserrors.go

@@ -449,7 +449,7 @@ var (
 	ErrorVpnGatewayLimitExceeded                        = "VpnGatewayLimitExceeded"
 	ErrorZonesMismatched                                = "ZonesMismatched"
 
-	// IAM-specific error codes
+	// IAM-specific error codes.
 	ErrorIAMNoSuchEntity            = "NoSuchEntity"
 	ErrorIAMEntityAlreadyExists     = "EntityAlreadyExists"
 	ErrorIAMDeleteConflict          = "DeleteConflict"
@@ -458,7 +458,7 @@ var (
 	ErrorIAMMalformedPolicyDocument = "MalformedPolicyDocument"
 	ErrorAccessDenied               = "AccessDenied"
 
-	// ECR-specific error codes
+	// ECR-specific error codes.
 	ErrorRepositoryNotFound       = "RepositoryNotFoundException"
 	ErrorRepositoryPolicyNotFound = "RepositoryPolicyNotFoundException"
 	ErrorLifecyclePolicyNotFound  = "LifecyclePolicyNotFoundException"
@@ -473,7 +473,7 @@ var (
 	ErrorTooManyTags              = "TooManyTagsException"
 	ErrorOperationNotSupported    = "OperationNotSupportedException"
 
-	// ELBv2-specific error codes
+	// ELBv2-specific error codes.
 	ErrorELBv2LoadBalancerNotFound         = "LoadBalancerNotFound"
 	ErrorELBv2TargetGroupNotFound          = "TargetGroupNotFound"
 	ErrorELBv2ListenerNotFound             = "ListenerNotFound"
@@ -500,7 +500,7 @@ var (
 	ErrorELBv2CertificateNotFound          = "CertificateNotFound"
 	ErrorELBv2SSLPolicyNotFound            = "SSLPolicyNotFound"
 
-	// Bedrock/bedrock-runtime error codes
+	// Bedrock/bedrock-runtime error codes.
 	ErrorValidationException = "ValidationException"
 	// ErrorResourceNotFoundException aliases ErrorEKSResourceNotFound: both
 	// services use the same wire code, so it is not a second ErrorLookup entry.

spinifex/config/config.go

@@ -27,6 +27,13 @@ const (
 	DefaultAWSInternalSuffix = "spinifex.internal"
 )
 
+// DefaultMgmtBridgeIP is the canonical br-mgmt host address the control plane
+// advertises: the EKS gateway URL, predastore endpoint, and node-group userdata
+// all target it. Kept in sync with setup-ovn.sh MGMT_CIDR. Server certs must
+// always carry this as a SAN so publish succeeds even when br-mgmt happens to be
+// down at cert-generation time (interface enumeration would otherwise miss it).
+const DefaultMgmtBridgeIP = "10.15.8.1"
+
 // AWSConfig holds cluster-wide AWS-parity settings shared across services.
 // Region scopes the default AWS region; InternalSuffix is the internal DNS
 // suffix used to build service endpoints (e.g. ecr.{region}.{suffix}).
@@ -77,7 +84,7 @@ type BootstrapConfig struct {
 	SubnetCidr string `mapstructure:"subnet_cidr"`
 }
 
-// Config holds all configuration for the application
+// Config holds all configuration for the application.
 type Config struct {
 	// Node config
 	Node string `json:"Node" mapstructure:"node"`
@@ -95,6 +102,7 @@ type Config struct {
 	Viperblock ViperblockConfig `json:"Viperblock" mapstructure:"viperblock"`
 	AWSGW      AWSGWConfig      `json:"AWSGW" mapstructure:"awsgw"`
 	VPCD       VPCDConfig       `json:"VPCD" mapstructure:"vpcd"`
+	Northstar  NorthstarConfig  `json:"Northstar" mapstructure:"northstar"`
 
 	BaseDir string `json:"BaseDir" mapstructure:"base_dir"`
 	WalDir  string `json:"WalDir" mapstructure:"wal_dir"`
@@ -116,6 +124,13 @@ type ViperblockConfig struct {
 	// EncryptionKeyFile is the path to the shared 32-byte AES-256 master key for viperblock at-rest encryption.
 	// Empty means cleartext. When set, all VB instances must load it via masterkey.LoadShared.
 	EncryptionKeyFile string `json:"EncryptionKeyFile" mapstructure:"encryption_key_file"`
+
+	// GCEnabled turns on viperblock chunk garbage collection (mark-and-sweep,
+	// snapshot-ancestry gated) on every VB this node constructs: the nbdkit
+	// plugin backing a mounted volume, and the short-lived VBs used by the
+	// volume service. Default false when nil so existing deployments keep
+	// today's behavior until explicitly opted in.
+	GCEnabled *bool `json:"GCEnabled" mapstructure:"gc_enabled"`
 }
 
 // VPCDConfig holds the VPC daemon (vpcd) configuration.
@@ -126,6 +141,17 @@ type VPCDConfig struct {
 	BridgeMode        string `json:"BridgeMode" mapstructure:"bridge_mode"`               // "direct" or "veth" (auto-detected if empty)
 }
 
+// NorthstarConfig holds the per-node northstar DNS service configuration.
+type NorthstarConfig struct {
+	// ConfigPath is the path to northstar.toml written by `spx admin init`.
+	ConfigPath string `json:"ConfigPath" mapstructure:"config_path"`
+	// DefaultDomain and InternalDomain mirror the northstar zone domains as
+	// non-secret values so producers (daemon, vpcd) can resolve DNS names
+	// without reading the credential-bearing northstar.toml.
+	DefaultDomain  string `json:"DefaultDomain" mapstructure:"default_domain"`
+	InternalDomain string `json:"InternalDomain" mapstructure:"internal_domain"`
+}
+
 // ParseEndpoints splits a comma-separated OVSDB endpoint list (NB/SB RAFT
 // cluster) into individual endpoints, trimming whitespace and dropping empties.
 // A single endpoint yields a one-element slice; empty input yields nil. Both the
@@ -167,7 +193,7 @@ type GPUModelOverride struct {
 	MIGProfile string `json:"MIGProfile" mapstructure:"mig_profile"`
 }
 
-// DaemonConfig holds the daemon configuration
+// DaemonConfig holds the daemon configuration.
 type DaemonConfig struct {
 	Host              string             `json:"Host" mapstructure:"host"`
 	TLSKey            string             `json:"TLSKey" mapstructure:"tlskey"`
@@ -180,7 +206,7 @@ type DaemonConfig struct {
 	MIGProfile string `json:"MIGProfile" mapstructure:"mig_profile"`
 }
 
-// NATSConfig holds the NATS configuration
+// NATSConfig holds the NATS configuration.
 type NATSConfig struct {
 	Host   string  `json:"Host" mapstructure:"host"`
 	CACert string  `json:"CACert" mapstructure:"cacert"`
@@ -188,12 +214,12 @@ type NATSConfig struct {
 	Sub    NATSSub `json:"Sub" mapstructure:"sub"`
 }
 
-// NATSACL holds the NATS ACL configuration
+// NATSACL holds the NATS ACL configuration.
 type NATSACL struct {
 	Token string `json:"Token" mapstructure:"token"`
 }
 
-// NATSSub holds the NATS subscription configuration
+// NATSSub holds the NATS subscription configuration.
 type NATSSub struct {
 	Subject string `json:"Subject" mapstructure:"subject"`
 }
@@ -235,7 +261,7 @@ func (c Config) GetServices() []string {
 	return c.Services
 }
 
-// LoadConfig loads the configuration from file and environment variables
+// LoadConfig loads the configuration from file and environment variables.
 func LoadConfig(configPath string) (*ClusterConfig, error) {
 	// Set environment variable prefix
 	viper.SetEnvPrefix("SPINIFEX")

spinifex/config/config_test.go

@@ -313,7 +313,7 @@ func TestNodeBaseDir_HappyPath(t *testing.T) {
 
 func TestNodeBaseDir_NilConfig(t *testing.T) {
 	var cc *ClusterConfig
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 func TestNodeBaseDir_EmptyNode(t *testing.T) {
@@ -323,7 +323,7 @@ func TestNodeBaseDir_EmptyNode(t *testing.T) {
 			"node1": {BaseDir: "/data/node1"},
 		},
 	}
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 func TestNodeBaseDir_NodeNotInMap(t *testing.T) {
@@ -333,7 +333,7 @@ func TestNodeBaseDir_NodeNotInMap(t *testing.T) {
 			"node1": {BaseDir: "/data/node1"},
 		},
 	}
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 func TestNodeBaseDir_EmptyBaseDir(t *testing.T) {
@@ -343,7 +343,7 @@ func TestNodeBaseDir_EmptyBaseDir(t *testing.T) {
 			"node1": {BaseDir: ""},
 		},
 	}
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 // Tests for NetworkConfig (external pools)
@@ -785,6 +785,50 @@ shardwal = true
 	assert.True(t, *n.Viperblock.ShardWAL)
 }
 
+func TestLoadConfig_ViperblockGCEnabled_DefaultNil(t *testing.T) {
+	resetViper(t)
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+
+	toml := `
+node = "n1"
+
+[nodes.n1]
+region = "us-east-1"
+`
+	require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
+
+	cfg, err := LoadConfig(path)
+	require.NoError(t, err)
+
+	n := cfg.Nodes["n1"]
+	assert.Nil(t, n.Viperblock.GCEnabled, "GCEnabled should be nil when not configured (defaults to false in service)")
+}
+
+func TestLoadConfig_ViperblockGCEnabled_True(t *testing.T) {
+	resetViper(t)
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+
+	toml := `
+node = "n1"
+
+[nodes.n1]
+region = "us-east-1"
+
+[nodes.n1.viperblock]
+gc_enabled = true
+`
+	require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
+
+	cfg, err := LoadConfig(path)
+	require.NoError(t, err)
+
+	n := cfg.Nodes["n1"]
+	require.NotNil(t, n.Viperblock.GCEnabled, "GCEnabled should be set when explicitly configured")
+	assert.True(t, *n.Viperblock.GCEnabled)
+}
+
 func TestParseEndpoints(t *testing.T) {
 	cases := []struct {
 		name string

spinifex/daemon/daemon.go

@@ -34,6 +34,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_acm "github.com/mulgadc/spinifex/spinifex/handlers/acm"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	handlers_ec2_account "github.com/mulgadc/spinifex/spinifex/handlers/ec2/account"
 	handlers_ec2_eigw "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eigw"
 	handlers_ec2_eip "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eip"
@@ -116,7 +117,7 @@ type ResourceManager struct {
 // handler's GatePublisher hook — the two services are wired together below.
 var _ handlers_ec2_igw.GatePublisher = (*handlers_ec2_routetable.RouteTableServiceImpl)(nil)
 
-// Daemon represents the main daemon service
+// Daemon represents the main daemon service.
 type Daemon struct {
 	node                  string
 	clusterConfig         *config.ClusterConfig
@@ -124,6 +125,10 @@ type Daemon struct {
 	natsConn              *nats.Conn
 	resourceMgr           *ResourceManager
 	instanceService       *handlers_ec2_instance.InstanceServiceImpl
+	dnsWriter             *handlers_dns.Writer
+	dnsReconciler         *handlers_dns.Reconciler
+	dnsBaseDomain         string
+	dnsInternalDomain     string
 	keyService            *handlers_ec2_key.KeyServiceImpl
 	imageService          *handlers_ec2_image.ImageServiceImpl
 	volumeService         *handlers_ec2_volume.VolumeServiceImpl
@@ -343,7 +348,7 @@ func (d *Daemon) onNATSReconnect(_ *nats.Conn) {
 // execCommand wraps exec.Command so tests can substitute a fake implementation.
 var execCommand = exec.Command
 
-// getSystemMemory returns the total system memory in GB
+// getSystemMemory returns the total system memory in GB.
 func getSystemMemory() (float64, error) {
 	switch runtime.GOOS {
 	case "darwin":
@@ -523,7 +528,7 @@ func (rm *ResourceManager) instanceMemChargeMiB(it *ec2.InstanceTypeInfo) int64
 		nbdkitChargeMiB(defaultMainVolumes, defaultAuxVolumes, rm.nbdkitMainMiB, rm.nbdkitAuxMiB)
 }
 
-// GetInstanceTypeInfos returns all instance types as ec2.InstanceTypeInfo for AWS API compatibility
+// GetInstanceTypeInfos returns all instance types as ec2.InstanceTypeInfo for AWS API compatibility.
 func (rm *ResourceManager) GetInstanceTypeInfos() []*ec2.InstanceTypeInfo {
 	rm.mu.RLock()
 	defer rm.mu.RUnlock()
@@ -666,12 +671,12 @@ func (rm *ResourceManager) GetResourceStats() (totalVCPU int, totalMemGB float64
 	return totalVCPU, totalMemGB, reservedVCPU, reservedMemGB, allocVCPU, allocMemGB, caps
 }
 
-// SetConfigPath sets the configuration file path for cluster management
+// SetConfigPath sets the configuration file path for cluster management.
 func (d *Daemon) SetConfigPath(path string) {
 	d.configPath = path
 }
 
-// NewDaemon creates a new daemon instance
+// NewDaemon creates a new daemon instance.
 func NewDaemon(cfg *config.ClusterConfig) (*Daemon, error) {
 	ctx, cancel := context.WithCancel(context.Background())
 
@@ -1317,6 +1322,10 @@ func (d *Daemon) startCluster() error {
 	// Create services before loading/launching instances, since LaunchInstance depends on them
 	store := objectstore.NewS3ObjectStoreFromConfig(admin.DialTarget(d.config.Predastore.Host), d.config.Predastore.Region, d.config.Predastore.AccessKey, d.config.Predastore.SecretKey)
 	d.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(d.config, d.resourceMgr.instanceTypes, d.natsConn, store, d.vmMgr, d.resourceMgr, d.jsManager)
+	d.dnsWriter = handlers_dns.NewWriter(d.config, d.clusterConfig, d.natsConn)
+	d.dnsReconciler = handlers_dns.NewReconciler(d.config, d.natsConn, d.dnsDesiredSet)
+	d.dnsBaseDomain = handlers_dns.ResolveBaseDomain(d.config)
+	d.dnsInternalDomain = handlers_dns.ResolveInternalDomain(d.config)
 	d.keyService = handlers_ec2_key.NewKeyServiceImpl(d.config)
 	d.imageService = handlers_ec2_image.NewImageServiceImpl(d.config)
 
@@ -1614,6 +1623,24 @@ func (d *Daemon) startCluster() error {
 		return fmt.Errorf("failed to subscribe to NATS topics: %w", err)
 	}
 
+	// DNS record writer: the single queue-group consumer of
+	// dns.recordset.change. No-op when northstar S3 is not configured.
+	if sub, err := d.dnsWriter.Subscribe(d.natsConn); err != nil {
+		return fmt.Errorf("failed to subscribe DNS record writer: %w", err)
+	} else if sub != nil {
+		d.natsSubscriptions[handlers_dns.SubjectRecordsetChange] = sub
+		slog.Info("Subscribed DNS record writer", "subject", handlers_dns.SubjectRecordsetChange, "queue", handlers_dns.QueueGroup)
+	}
+
+	// DNS drift backstop: periodically rebuild managed records
+	// from the live cross-tenant inventory and converge the zone. It publishes
+	// through the same queue-group writer, so every node running it serialises on
+	// one writer and never races the zone. No-op when northstar is not configured.
+	if d.dnsReconciler.Enabled() {
+		go d.dnsReconciler.Run(d.ctx)
+		slog.Info("Started DNS reconcile backstop", "interval", handlers_dns.DefaultReconcileInterval)
+	}
+
 	// Initialize per-instance-type NATS subscriptions for capacity-aware routing.
 	d.resourceMgr.initSubscriptions(d.natsConn, d.handleEC2RunInstances, d.handleSystemLaunchInstance, d.node)
 
@@ -2340,7 +2367,7 @@ func (rm *ResourceManager) allocate(instanceType *ec2.InstanceTypeInfo) error {
 	return nil
 }
 
-// deallocate releases resources for an instance and updates NATS subscriptions
+// deallocate releases resources for an instance and updates NATS subscriptions.
 func (rm *ResourceManager) deallocate(instanceType *ec2.InstanceTypeInfo) {
 	rm.mu.Lock()
 	vCPUs := instanceTypeVCPUs(instanceType)

spinifex/daemon/daemon_handlers.go

@@ -122,7 +122,7 @@ func handleNATSRequestWithPrincipal[I any, O any](msg *nats.Msg, serviceFn func(
 	respondWithJSON(msg, output)
 }
 
-// handleEC2Events processes incoming EC2 instance events (start, stop, terminate, attach-volume)
+// handleEC2Events processes incoming EC2 instance events (start, stop, terminate, attach-volume).
 func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 	ctx, span := utils.StartConsumerSpan(msg)
 	defer span.End()
@@ -213,7 +213,7 @@ func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 
 // --- Admin / node management handlers ---
 
-// handleHealthCheck processes NATS health check requests
+// handleHealthCheck processes NATS health check requests.
 func (d *Daemon) handleHealthCheck(msg *nats.Msg) {
 	configHash, err := d.computeConfigHash()
 	if err != nil {
@@ -239,7 +239,7 @@ func (d *Daemon) handleHealthCheck(msg *nats.Msg) {
 }
 
 // handleNodeDiscover responds to node discovery requests with this node's ID
-// Used by the gateway to dynamically discover active spinifex nodes in the cluster
+// Used by the gateway to dynamically discover active spinifex nodes in the cluster.
 func (d *Daemon) handleNodeDiscover(msg *nats.Msg) {
 	response := types.NodeDiscoverResponse{
 		Node: d.node,

spinifex/daemon/daemon_handlers_eks_test.go

@@ -105,7 +105,7 @@ func TestDaemonHandleEKS_AllHandlersDispatchToService(t *testing.T) {
 		{"eks.UntagResource", d.handleEKSUntagResource, notImpl},
 		{"eks.ListTagsForResource", d.handleEKSListTagsForResource, notImpl},
 	}
-	require.Equal(t, 30, len(cases), "expected exactly one handler per non-lifecycle AWS EKS action")
+	require.Len(t, cases, 30, "expected exactly one handler per non-lifecycle AWS EKS action")
 
 	for _, c := range cases {
 		t.Run(c.subject, func(t *testing.T) {

spinifex/daemon/daemon_handlers_instance.go

@@ -13,6 +13,7 @@ import (
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	"github.com/mulgadc/spinifex/spinifex/filterutil"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	handlers_ec2_instance "github.com/mulgadc/spinifex/spinifex/handlers/ec2/instance"
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/utils"
@@ -381,6 +382,20 @@ func (d *Daemon) handleEC2DescribeInstances(msg *nats.Msg) {
 					instanceCopy.PublicIpAddress = aws.String(instance.PublicIP)
 				}
 
+				// Project the AWS-shaped DNS names from the current IPs, the same
+				// way PublicIpAddress/State are derived here. Mirrors the records
+				// the control-plane writer publishes to northstar.
+				publicDNS, privateDNS := handlers_dns.EC2DNSNames(
+					d.config.Region, d.dnsBaseDomain, d.dnsInternalDomain,
+					aws.StringValue(instanceCopy.PublicIpAddress), aws.StringValue(instanceCopy.PrivateIpAddress),
+				)
+				if publicDNS != "" {
+					instanceCopy.PublicDnsName = aws.String(publicDNS)
+				}
+				if privateDNS != "" {
+					instanceCopy.PrivateDnsName = aws.String(privateDNS)
+				}
+
 				// Map internal status to AWS state, projecting Spinifex-only states
 				// (e.g. error -> stopped) so SDK/UI clients see a valid label.
 				if info, ok := vm.EC2APIState(instance.Status); ok {

spinifex/daemon/daemon_handlers_test.go

@@ -938,7 +938,7 @@ func TestRespondWithVolumeAttachment(t *testing.T) {
 	assert.Equal(t, "/dev/sdf", *attachment.Device)
 	assert.Equal(t, "attached", *attachment.State)
 	assert.NotNil(t, attachment.AttachTime)
-	assert.Equal(t, false, *attachment.DeleteOnTermination)
+	assert.False(t, *attachment.DeleteOnTermination)
 }
 
 // --- handleEC2ModifyVolume tests ---
@@ -1712,7 +1712,7 @@ func TestHandleEC2DescribeInstanceAttribute_DefaultAttribute_DisableApiTerminati
 	err = json.Unmarshal(reply.Data, &output)
 	require.NoError(t, err)
 	require.NotNil(t, output.DisableApiTermination)
-	assert.Equal(t, true, *output.DisableApiTermination.Value)
+	assert.True(t, *output.DisableApiTermination.Value)
 }
 
 func TestHandleEC2DescribeInstanceAttribute_DefaultAttribute_ShutdownBehavior(t *testing.T) {
@@ -2119,11 +2119,11 @@ func TestHandleNodeStatus(t *testing.T) {
 	assert.Equal(t, "us-west-2a", resp.AZ)
 	assert.GreaterOrEqual(t, resp.Uptime, int64(0))
 	assert.Equal(t, 2, resp.VMCount)
-	assert.Greater(t, resp.TotalVCPU, 0)
+	assert.Positive(t, resp.TotalVCPU)
 	assert.Greater(t, resp.TotalMemGB, 0.0)
 	assert.Equal(t, daemon.resourceMgr.reservedVCPU, resp.ReservedVCPU, "ReservedVCPU must be populated on wire")
 	assert.InDelta(t, daemon.resourceMgr.reservedMem, resp.ReservedMemGB, 0.001, "ReservedMemGB must be populated on wire")
-	assert.Greater(t, resp.ReservedVCPU, 0, "default reserve is non-zero")
+	assert.Positive(t, resp.ReservedVCPU, "default reserve is non-zero")
 	assert.Greater(t, resp.ReservedMemGB, 0.0, "default reserve is non-zero")
 }
 
@@ -2144,7 +2144,7 @@ func TestHandleNodeStatus_NoVMs(t *testing.T) {
 
 	assert.Equal(t, 0, resp.VMCount)
 	assert.Equal(t, "Ready", resp.Status)
-	assert.Greater(t, resp.ReservedVCPU, 0, "default reserve is exposed even when no VMs")
+	assert.Positive(t, resp.ReservedVCPU, "default reserve is exposed even when no VMs")
 	assert.Greater(t, resp.ReservedMemGB, 0.0, "default reserve is exposed even when no VMs")
 }
 
@@ -2195,7 +2195,7 @@ func TestHandleNodeVMs(t *testing.T) {
 	vm1 := vmsByID["i-vm-1"]
 	assert.Equal(t, "running", vm1.Status)
 	assert.Equal(t, instanceType, vm1.InstanceType)
-	assert.Greater(t, vm1.VCPU, 0)
+	assert.Positive(t, vm1.VCPU)
 	assert.Greater(t, vm1.MemoryGB, 0.0)
 	assert.Equal(t, launchTime.Unix(), vm1.LaunchTime)
 
@@ -2528,7 +2528,7 @@ func TestHandleEC2CreateInternetGateway_SuccessPath(t *testing.T) {
 	require.NoError(t, err)
 	require.NotNil(t, output.InternetGateway)
 	assert.NotEmpty(t, *output.InternetGateway.InternetGatewayId)
-	assert.True(t, len(*output.InternetGateway.InternetGatewayId) > 4)
+	assert.Greater(t, len(*output.InternetGateway.InternetGatewayId), 4)
 }
 
 func TestHandleEC2CreateSubnet_SuccessPath(t *testing.T) {
@@ -2744,7 +2744,7 @@ func TestHandleEC2DescribeInstanceTypes_NoFilter(t *testing.T) {
 	err = json.Unmarshal(reply.Data, &output)
 	require.NoError(t, err)
 	assert.NotNil(t, output.InstanceTypes)
-	assert.Greater(t, len(output.InstanceTypes), 0)
+	assert.NotEmpty(t, output.InstanceTypes)
 }
 
 // --- handleEC2StartStoppedInstance: instance type not available ---

spinifex/daemon/daemon_handlers_volume.go

@@ -176,7 +176,7 @@ func (d *Daemon) handleEC2DescribeVolumesModifications(msg *nats.Msg) {
 	handleNATSRequest(msg, d.volumeService.DescribeVolumesModifications)
 }
 
-// handleEC2ModifyVolume processes incoming EC2 ModifyVolume requests
+// handleEC2ModifyVolume processes incoming EC2 ModifyVolume requests.
 func (d *Daemon) handleEC2ModifyVolume(msg *nats.Msg) {
 	ctx, span := utils.StartConsumerSpan(msg)
 	defer span.End()

spinifex/daemon/daemon_handlers_vpc.go

@@ -189,7 +189,7 @@ func (d *Daemon) ensureDefaultVPCInfrastructureFor(ctx context.Context, accountI
 	}
 }
 
-// ensureDefaultIGWRoute adds 0.0.0.0/0 → igw-xxx to the main route table if not present
+// ensureDefaultIGWRoute adds 0.0.0.0/0 → igw-xxx to the main route table if not present.
 func (d *Daemon) ensureDefaultIGWRoute(ctx context.Context, accountID, vpcID string) {
 	// Find the main route table for this VPC
 	vpcFilter := "vpc-id"

spinifex/daemon/daemon_test.go

@@ -78,7 +78,7 @@ func drainAndClose(t *testing.T, nc *nats.Conn) {
 	}
 }
 
-// createTestDaemon creates a test daemon instance with minimal configuration
+// createTestDaemon creates a test daemon instance with minimal configuration.
 func createTestDaemon(t *testing.T, natsURL string) *Daemon {
 	// Create a temporary directory for test data
 	tmpDir := t.TempDir()
@@ -150,7 +150,7 @@ func createTestDaemon(t *testing.T, natsURL string) *Daemon {
 	return daemon
 }
 
-// getTestInstanceType returns a valid instance type for testing based on the system's CPU
+// getTestInstanceType returns a valid instance type for testing based on the system's CPU.
 func getTestInstanceType(t *testing.T) string {
 	t.Helper()
 	rm, err := NewResourceManager(nil, nil, nil)
@@ -182,13 +182,13 @@ func seedTestAMI(t *testing.T, store *objectstore.MemoryObjectStore, bucket, ima
 	require.NoError(t, err)
 }
 
-// TestResourceManager tests resource manager functionality
+// TestResourceManager tests resource manager functionality.
 func TestResourceManager(t *testing.T) {
 	rm, err := NewResourceManager(nil, nil, nil)
 	require.NoError(t, err)
 
 	require.NotNil(t, rm)
-	assert.Greater(t, rm.hostVCPU, 0)
+	assert.Positive(t, rm.hostVCPU)
 	assert.Greater(t, rm.hostMemGB, float64(0))
 
 	// Test allocation using the first available instance type (dynamic based on CPU)
@@ -247,7 +247,7 @@ func TestResourceManager(t *testing.T) {
 
 		// Test requesting more than available
 		maxPossible := rm.canAllocate(microType, 1000)
-		assert.Greater(t, maxPossible, 0, "Should be able to allocate at least 1")
+		assert.Positive(t, maxPossible, "Should be able to allocate at least 1")
 		assert.LessOrEqual(t, maxPossible, 1000, "Should not exceed requested count")
 
 		// Test requesting exactly 1
@@ -267,7 +267,7 @@ func TestResourceManager(t *testing.T) {
 	})
 }
 
-// TestGetInstanceTypeInfos tests the GetInstanceTypeInfos method
+// TestGetInstanceTypeInfos tests the GetInstanceTypeInfos method.
 func TestGetInstanceTypeInfos(t *testing.T) {
 	rm, err := NewResourceManager(nil, nil, nil)
 	require.NoError(t, err)
@@ -276,7 +276,7 @@ func TestGetInstanceTypeInfos(t *testing.T) {
 
 	require.NotEmpty(t, infos, "Should return at least one instance type")
 	// With generation-specific families, minimum is 7 (unknown/burstable-only) up to 31 (current-gen)
-	assert.True(t, len(infos) >= 7,
+	assert.GreaterOrEqual(t, len(infos), 7,
 		"Should have at least 7 instance types, got %d", len(infos))
 
 	// Verify structure of returned instance type info
@@ -295,7 +295,7 @@ func TestGetInstanceTypeInfos(t *testing.T) {
 	}
 }
 
-// TestGetAvailableInstanceTypeInfos_ResourceFiltering tests that instance types are filtered by available resources
+// TestGetAvailableInstanceTypeInfos_ResourceFiltering tests that instance types are filtered by available resources.
 func TestGetAvailableInstanceTypeInfos_ResourceFiltering(t *testing.T) {
 	rm, err := NewResourceManager(nil, nil, nil)
 	require.NoError(t, err)
@@ -313,7 +313,7 @@ func TestGetAvailableInstanceTypeInfos_ResourceFiltering(t *testing.T) {
 	// be filtered out.
 	assert.LessOrEqual(t, len(initialAvailable), len(allTypes),
 		"Available types should be <= total types")
-	assert.Greater(t, len(initialAvailable), 0, "Should have at least one available type")
+	assert.NotEmpty(t, initialAvailable, "Should have at least one available type")
 
 	// Verify all initially available types fit within schedulable resources.
 	schedulableVCPU := rm.hostVCPU - rm.reservedVCPU
@@ -373,11 +373,11 @@ func TestGetAvailableInstanceTypeInfos_ResourceFiltering(t *testing.T) {
 	// Deallocate and verify we get the same available types as before
 	rm.deallocate(nanoType)
 	afterDeallocation := rm.GetAvailableInstanceTypeInfos(false)
-	assert.Equal(t, len(initialAvailable), len(afterDeallocation),
+	assert.Len(t, afterDeallocation, len(initialAvailable),
 		"Should have same available types after deallocation")
 }
 
-// TestHandleEC2DescribeInstanceTypes tests the DescribeInstanceTypes handler
+// TestHandleEC2DescribeInstanceTypes tests the DescribeInstanceTypes handler.
 func TestHandleEC2DescribeInstanceTypes(t *testing.T) {
 	natsURL := sharedNATSURL
 
@@ -403,7 +403,7 @@ func TestHandleEC2DescribeInstanceTypes(t *testing.T) {
 		require.NoError(t, err, "Should unmarshal response")
 
 		require.NotNil(t, output.InstanceTypes, "InstanceTypes should not be nil")
-		assert.Greater(t, len(output.InstanceTypes), 0, "Should return at least one instance type")
+		assert.NotEmpty(t, output.InstanceTypes, "Should return at least one instance type")
 
 		// Verify CPU architecture is correct
 		expectedArch := "x86_64"
@@ -461,7 +461,7 @@ func TestHandleEC2DescribeInstanceTypes(t *testing.T) {
 		}
 
 		// Verify counts match (all available types should be returned)
-		assert.Equal(t, len(expectedTypes), len(output.InstanceTypes),
+		assert.Len(t, output.InstanceTypes, len(expectedTypes),
 			"Returned instance types count should match available types count")
 
 		t.Logf("Verified %d instance types match expected list", len(output.InstanceTypes))
@@ -527,7 +527,7 @@ func TestHandleEC2DescribeInstanceTypes(t *testing.T) {
 		err = json.Unmarshal(reply.Data, &afterAllocationOutput)
 		require.NoError(t, err)
 
-		assert.Equal(t, initialCount, len(afterAllocationOutput.InstanceTypes),
+		assert.Len(t, afterAllocationOutput.InstanceTypes, initialCount,
 			"no-filter response must list the same supported types before and after allocation")
 
 		afterTypes := make(map[string]bool)
@@ -660,7 +660,7 @@ func TestHandleEC2DescribeInstanceTypes(t *testing.T) {
 				expectedSlots++
 			}
 		}
-		assert.Equal(t, expectedSlots, len(output.InstanceTypes),
+		assert.Len(t, output.InstanceTypes, expectedSlots,
 			"Should have %d slots for types fitting 2 vCPU / 16GB", expectedSlots)
 
 		// Now increase capacity to test duplicate slots
@@ -694,7 +694,7 @@ func TestHandleEC2DescribeInstanceTypes(t *testing.T) {
 	})
 }
 
-// TestDaemon_BootAllocation verifies that resources are correctly reconstructed on startup
+// TestDaemon_BootAllocation verifies that resources are correctly reconstructed on startup.
 func TestDaemon_BootAllocation(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -774,7 +774,7 @@ func TestDaemon_BootAllocation(t *testing.T) {
 	assert.Equal(t, expectedMem, daemon.resourceMgr.allocatedMem)
 }
 
-// TestCanAllocate_CountEdgeCases tests edge cases for canAllocate with count parameter
+// TestCanAllocate_CountEdgeCases tests edge cases for canAllocate with count parameter.
 func TestCanAllocate_CountEdgeCases(t *testing.T) {
 	t.Run("MinCount_equals_MaxCount", func(t *testing.T) {
 		rm, err := NewResourceManager(nil, nil, nil)
@@ -961,7 +961,7 @@ func TestAllocate_NoOvercommitUnderContention(t *testing.T) {
 	// Size the pool on the full per-instance charge (guest -m + nbdkit, RG-6),
 	// not bare guest -m, so capacityFor slots still fit exactly after RG-6.
 	memGB := float64(rm.instanceMemChargeMiB(microType)) / 1024.0
-	require.Greater(t, vCPUs, 0, "micro type must report vCPU count")
+	require.Positive(t, vCPUs, "micro type must report vCPU count")
 	require.Greater(t, memGB, float64(0), "micro type must report memory")
 
 	const capacityFor = 8 // slots
@@ -1018,7 +1018,7 @@ func TestAllocate_NoOvercommitUnderContention(t *testing.T) {
 	assert.InDelta(t, memGB*float64(capacityFor), finalMem, 0.001, "allocated memory must not exceed schedulable pool")
 }
 
-// TestDescribeInstances_ReservationGrouping tests that instances are grouped by reservation ID
+// TestDescribeInstances_ReservationGrouping tests that instances are grouped by reservation ID.
 func TestDescribeInstances_ReservationGrouping(t *testing.T) {
 	natsURL := sharedNATSURL
 
@@ -1209,7 +1209,7 @@ func TestDescribeInstances_ReservationGrouping(t *testing.T) {
 	})
 }
 
-// TestRunInstances_CountValidation tests MinCount/MaxCount validation scenarios
+// TestRunInstances_CountValidation tests MinCount/MaxCount validation scenarios.
 func TestRunInstances_CountValidation(t *testing.T) {
 	natsURL := sharedNATSURL
 	instanceType := getTestInstanceType(t)
@@ -1330,9 +1330,9 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		}
 
 		// Each fittable type gets 2 subscriptions: queue group + node-specific
-		assert.Equal(t, fittableTypes*2, len(rm.instanceSubs),
+		assert.Len(t, rm.instanceSubs, fittableTypes*2,
 			"should subscribe to all instance types that fit (queue + node-specific)")
-		assert.Greater(t, len(rm.instanceSubs), 0,
+		assert.NotEmpty(t, rm.instanceSubs,
 			"should subscribe to at least some instance types")
 
 		// Verify topics follow the expected pattern
@@ -1353,7 +1353,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		rm.initSubscriptions(nc, handler, nil, "test-node")
 
 		initialCount := len(rm.instanceSubs)
-		require.Greater(t, initialCount, 0)
+		require.Positive(t, initialCount)
 
 		// Allocate all resources so nothing fits
 		rm.mu.Lock()
@@ -1370,7 +1370,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		queueSubs, nodeSubs := countSubsBySuffix(rm.instanceSubs, ".test-node")
 		assert.Equal(t, 0, queueSubs,
 			"queue subscriptions should drop when the node is full")
-		assert.Greater(t, nodeSubs, 0,
+		assert.Positive(t, nodeSubs,
 			"node-targeted subscriptions persist regardless of capacity")
 	})
 
@@ -1394,7 +1394,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		rm.updateInstanceSubscriptions()
 		queueSubs, nodeSubs := countSubsBySuffix(rm.instanceSubs, ".test-node")
 		assert.Equal(t, 0, queueSubs, "queue subscriptions drop when the node is full")
-		assert.Greater(t, nodeSubs, 0, "node-targeted subscriptions persist when full")
+		assert.Positive(t, nodeSubs, "node-targeted subscriptions persist when full")
 
 		// Free all resources
 		rm.mu.Lock()
@@ -1403,7 +1403,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		rm.mu.Unlock()
 		rm.updateInstanceSubscriptions()
 
-		assert.Equal(t, expectedCount, len(rm.instanceSubs),
+		assert.Len(t, rm.instanceSubs, expectedCount,
 			"should resubscribe to all types when resources are freed")
 	})
 
@@ -1426,7 +1426,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		rm.updateInstanceSubscriptions()
 
 		// Count subscribed types — should be less than total but more than zero
-		assert.Greater(t, len(rm.instanceSubs), 0,
+		assert.NotEmpty(t, rm.instanceSubs,
 			"should still be subscribed to small instance types")
 		assert.Less(t, len(rm.instanceSubs), len(rm.instanceTypes)*2,
 			"should not be subscribed to large instance types")
@@ -1448,7 +1448,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		rm.initSubscriptions(nc, handler, nil, "test-node")
 
 		initialCount := len(rm.instanceSubs)
-		require.Greater(t, initialCount, 0)
+		require.Positive(t, initialCount)
 
 		// Find a .micro type that fits (2 vCPU, 1 GB — always fits)
 		var microType *ec2.InstanceTypeInfo
@@ -1467,7 +1467,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 			require.NoError(t, err)
 			allocated++
 		}
-		require.Greater(t, allocated, 0)
+		require.Positive(t, allocated)
 
 		// Should have fewer subscriptions now (or zero)
 		assert.Less(t, len(rm.instanceSubs), initialCount,
@@ -1477,7 +1477,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 		for range allocated {
 			rm.deallocate(microType)
 		}
-		assert.Equal(t, initialCount, len(rm.instanceSubs),
+		assert.Len(t, rm.instanceSubs, initialCount,
 			"deallocating should restore all subscriptions")
 	})
 
@@ -1557,7 +1557,7 @@ func TestInstanceTypeSubscriptions(t *testing.T) {
 	})
 }
 
-// TestResourceManager_ConcurrentAccess tests thread safety of resource manager
+// TestResourceManager_ConcurrentAccess tests thread safety of resource manager.
 func TestResourceManager_ConcurrentAccess(t *testing.T) {
 	rm, err := NewResourceManager(nil, nil, nil)
 	require.NoError(t, err)
@@ -1738,7 +1738,7 @@ func TestInstanceCleanerAdapter_DeleteVolumes_DeleteOnTermination_False(t *testi
 	assert.False(t, ebsDeletedVolumes["vol-keep"], "Root volume with DeleteOnTermination=false should NOT be deleted")
 }
 
-// TestHandleEC2Events_AttachVolume tests the attach-volume handler in handleEC2Events
+// TestHandleEC2Events_AttachVolume tests the attach-volume handler in handleEC2Events.
 func TestHandleEC2Events_AttachVolume(t *testing.T) {
 	natsURL := sharedNATSURL
 
@@ -1840,7 +1840,7 @@ func TestHandleEC2Events_AttachVolume(t *testing.T) {
 	})
 }
 
-// TestHandleEC2Events_DetachVolume tests the detach-volume handler in handleEC2Events
+// TestHandleEC2Events_DetachVolume tests the detach-volume handler in handleEC2Events.
 func TestHandleEC2Events_DetachVolume(t *testing.T) {
 	natsURL := sharedNATSURL
 
@@ -2267,7 +2267,7 @@ func TestDetachVolume_SuccessPath(t *testing.T) {
 	assert.Equal(t, "vol-root", *instance.Instance.BlockDeviceMappings[0].Ebs.VolumeId)
 }
 
-// TestDetachVolume_ForceFlag tests that force=true continues past device_del failure
+// TestDetachVolume_ForceFlag tests that force=true continues past device_del failure.
 func TestDetachVolume_ForceFlag(t *testing.T) {
 	natsURL := sharedNATSURL
 
@@ -2493,7 +2493,7 @@ func TestDetachVolume_BlockdevDelFailure(t *testing.T) {
 	assert.True(t, bdmFound, "Volume must remain in BlockDeviceMappings when blockdev-del fails")
 }
 
-// TestDetachVolume_SuccessWithDeviceMatch tests detach with correct --device cross-check
+// TestDetachVolume_SuccessWithDeviceMatch tests detach with correct --device cross-check.
 func TestDetachVolume_SuccessWithDeviceMatch(t *testing.T) {
 	natsURL := sharedNATSURL
 
@@ -4304,12 +4304,12 @@ func TestReloadGPUTypes_CallsUpdateInstanceSubscriptions(t *testing.T) {
 
 	rm.reloadGPUTypes([]instancetypes.GPUModel{instancetypes.NVIDIAt4}, nil, gpuMgr)
 	subsAfter := len(rm.instanceSubs)
-	require.Greater(t, subsAfter, 0, "subscriptions added after reload — updateInstanceSubscriptions ran")
+	require.Positive(t, subsAfter, "subscriptions added after reload — updateInstanceSubscriptions ran")
 
 	// Each g4dn size produces both queue and node-specific topics; record the
 	// count so we can prove a second call does not double-subscribe.
 	rm.reloadGPUTypes([]instancetypes.GPUModel{instancetypes.NVIDIAt4}, nil, gpuMgr)
-	assert.Equal(t, subsAfter, len(rm.instanceSubs),
+	assert.Len(t, rm.instanceSubs, subsAfter,
 		"second reload with identical models must not double-subscribe — proves idempotent invocation")
 }
 

spinifex/daemon/dns_reconcile.go

@@ -0,0 +1,59 @@
+package daemon
+
+import (
+	"github.com/aws/aws-sdk-go/aws"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
+	"github.com/mulgadc/spinifex/spinifex/vm"
+)
+
+// dnsDesiredSet builds the full desired managed-record set for the reconcile
+// backstop. It spans all tenants: every running instance on this
+// node plus every active load balancer and EKS cluster across all accounts.
+// Prune authority is granted per record class only when that class enumerated
+// completely, so a transient store error can never delete another tenant's live
+// records — the reconcile only ever repairs, never over-prunes, on a partial view.
+func (d *Daemon) dnsDesiredSet() handlers_dns.DesiredSet {
+	ds := handlers_dns.DesiredSet{}
+	ds.Changes = append(ds.Changes, d.desiredEC2DNSChanges()...)
+
+	if d.elbv2Service != nil {
+		if ch, ok := d.elbv2Service.DesiredDNSChanges(); ok {
+			ds.Changes = append(ds.Changes, ch...)
+			ds.Prunable.ELB = true
+		}
+	}
+	if d.eksService != nil {
+		if ch, ok := d.eksService.DesiredDNSChanges(); ok {
+			ds.Changes = append(ds.Changes, ch...)
+			ds.Prunable.EKS = true
+		}
+	}
+	return ds
+}
+
+// desiredEC2DNSChanges returns UPSERTs for this node's running instances. EC2
+// records are node-local — vmMgr holds only this node's VMs — so they are never
+// pruned by the reconcile; the terminate hook owns EC2 record removal. The
+// domains mirror the lifecycle publish so re-asserting is a no-op when in sync.
+func (d *Daemon) desiredEC2DNSChanges() []handlers_dns.Change {
+	var changes []handlers_dns.Change
+	d.vmMgr.View(func(vms map[string]*vm.VM) {
+		for _, v := range vms {
+			if v == nil || v.Status != vm.StateRunning {
+				continue
+			}
+			privateIP := ""
+			if v.Instance != nil {
+				privateIP = aws.StringValue(v.Instance.PrivateIpAddress)
+			}
+			if v.PublicIP == "" && privateIP == "" {
+				continue
+			}
+			changes = append(changes, handlers_dns.EC2Changes(
+				handlers_dns.ActionUpsert, d.config.Region,
+				d.dnsBaseDomain, d.dnsInternalDomain, v.PublicIP, privateIP,
+			)...)
+		}
+	})
+	return changes
+}

spinifex/daemon/heartbeat_test.go

@@ -36,11 +36,11 @@ func TestBuildHeartbeat(t *testing.T) {
 	assert.Equal(t, []string{"daemon", "nats", "viperblock"}, h.Services)
 	assert.Equal(t, 0, h.VMCount)
 	assert.Equal(t, 0, h.AllocatedVCPU)
-	assert.Greater(t, h.AvailableVCPU, 0)
+	assert.Positive(t, h.AvailableVCPU)
 	assert.Greater(t, h.AvailableMem, 0.0)
 	assert.Equal(t, rm.reservedVCPU, h.ReservedVCPU, "ReservedVCPU must be populated from ResourceManager")
 	assert.InDelta(t, rm.reservedMem, h.ReservedMem, 0.001, "ReservedMem must be populated from ResourceManager")
-	assert.Greater(t, h.ReservedVCPU, 0, "default reserve is non-zero")
+	assert.Positive(t, h.ReservedVCPU, "default reserve is non-zero")
 	assert.Greater(t, h.ReservedMem, 0.0, "default reserve is non-zero")
 }
 

spinifex/daemon/jetstream.go

@@ -16,20 +16,20 @@ import (
 )
 
 const (
-	// InstanceStateBucket is the name of the KV bucket for storing instance state
+	// InstanceStateBucket is the name of the KV bucket for storing instance state.
 	InstanceStateBucket = "spinifex-instance-state"
-	// ClusterStateBucket is the name of the KV bucket for cluster state (heartbeats, shutdown markers, service maps)
+	// ClusterStateBucket is the name of the KV bucket for cluster state (heartbeats, shutdown markers, service maps).
 	ClusterStateBucket = "spinifex-cluster-state"
-	// InstanceStatePrefix is the key prefix for per-node instance state entries
+	// InstanceStatePrefix is the key prefix for per-node instance state entries.
 	InstanceStatePrefix = "node."
-	// StoppedInstancePrefix is the key prefix for stopped instances in shared KV
+	// StoppedInstancePrefix is the key prefix for stopped instances in shared KV.
 	StoppedInstancePrefix = "instance."
-	// TerminatedInstanceBucket is the name of the KV bucket for terminated instances (auto-expiry via TTL)
+	// TerminatedInstanceBucket is the name of the KV bucket for terminated instances (auto-expiry via TTL).
 	TerminatedInstanceBucket = "spinifex-terminated-instances"
-	// TerminatedInstancePrefix is the key prefix for terminated instances
+	// TerminatedInstancePrefix is the key prefix for terminated instances.
 	TerminatedInstancePrefix = "terminated."
 
-	// Schema versions for daemon KV buckets
+	// Schema versions for daemon KV buckets.
 	InstanceStateBucketVersion      = 1
 	ClusterStateBucketVersion       = 1
 	TerminatedInstanceBucketVersion = 1
@@ -44,7 +44,7 @@ type KVSyncObserver interface {
 	RecordKVSyncFailure(bucket string, err error)
 }
 
-// JetStreamManager manages JetStream KV store operations for instance state
+// JetStreamManager manages JetStream KV store operations for instance state.
 type JetStreamManager struct {
 	js           nats.JetStreamContext
 	kv           nats.KeyValue // spinifex-instance-state
@@ -86,7 +86,7 @@ func NewJetStreamManager(nc *nats.Conn, replicas int) (*JetStreamManager, error)
 	}, nil
 }
 
-// InitKVBucket initializes the KV bucket, creating it if it doesn't exist
+// InitKVBucket initializes the KV bucket, creating it if it doesn't exist.
 func (m *JetStreamManager) InitKVBucket() error {
 	// Try to get the existing bucket first
 	kv, err := m.js.KeyValue(InstanceStateBucket)
@@ -705,7 +705,7 @@ func (m *JetStreamManager) LoadState(nodeID string) (map[string]*vm.VM, error) {
 	return state.VMS, nil
 }
 
-// DeleteState removes the instance state from the KV store for the given node
+// DeleteState removes the instance state from the KV store for the given node.
 func (m *JetStreamManager) DeleteState(nodeID string) error {
 	if m.kv == nil {
 		return errors.New("KV bucket not initialized")

spinifex/daemon/jetstream_test.go

@@ -14,7 +14,7 @@ import (
 	"github.com/stretchr/testify/require"
 )
 
-// TestJetStreamManager_WriteAndLoadState tests round-trip write and load of instance state
+// TestJetStreamManager_WriteAndLoadState tests round-trip write and load of instance state.
 func TestJetStreamManager_WriteAndLoadState(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -65,7 +65,7 @@ func TestJetStreamManager_WriteAndLoadState(t *testing.T) {
 	assert.Equal(t, "t3.small", loadedInstances["i-test-002"].InstanceType)
 }
 
-// TestJetStreamManager_LoadState_KeyNotFound tests that LoadState returns empty state when key doesn't exist
+// TestJetStreamManager_LoadState_KeyNotFound tests that LoadState returns empty state when key doesn't exist.
 func TestJetStreamManager_LoadState_KeyNotFound(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -86,7 +86,7 @@ func TestJetStreamManager_LoadState_KeyNotFound(t *testing.T) {
 	assert.Empty(t, instances, "Should return empty VMS map")
 }
 
-// TestJetStreamManager_BucketCreation tests that InitKVBucket creates the bucket when it doesn't exist
+// TestJetStreamManager_BucketCreation tests that InitKVBucket creates the bucket when it doesn't exist.
 func TestJetStreamManager_BucketCreation(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -105,7 +105,7 @@ func TestJetStreamManager_BucketCreation(t *testing.T) {
 	assert.NotNil(t, jsm.kv, "KV bucket should be initialized")
 }
 
-// TestJetStreamManager_BucketReconnection tests that InitKVBucket connects to existing bucket
+// TestJetStreamManager_BucketReconnection tests that InitKVBucket connects to existing bucket.
 func TestJetStreamManager_BucketReconnection(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -150,7 +150,7 @@ func TestJetStreamManager_BucketReconnection(t *testing.T) {
 	assert.Equal(t, vm.StateRunning, loadedInstances["i-persist"].Status)
 }
 
-// TestJetStreamManager_DeleteState tests deleting state from the KV store
+// TestJetStreamManager_DeleteState tests deleting state from the KV store.
 func TestJetStreamManager_DeleteState(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -190,7 +190,7 @@ func TestJetStreamManager_DeleteState(t *testing.T) {
 	assert.Empty(t, loadedInstances, "Should return empty state after deletion")
 }
 
-// TestJetStreamManager_DeleteState_NonExistent tests deleting state that doesn't exist
+// TestJetStreamManager_DeleteState_NonExistent tests deleting state that doesn't exist.
 func TestJetStreamManager_DeleteState_NonExistent(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -209,7 +209,7 @@ func TestJetStreamManager_DeleteState_NonExistent(t *testing.T) {
 	require.NoError(t, err, "Deleting non-existent state should not error")
 }
 
-// TestJetStreamManager_WriteState_UpdateExisting tests that writing state updates existing entry
+// TestJetStreamManager_WriteState_UpdateExisting tests that writing state updates existing entry.
 func TestJetStreamManager_WriteState_UpdateExisting(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -257,7 +257,7 @@ func TestJetStreamManager_WriteState_UpdateExisting(t *testing.T) {
 	assert.NotNil(t, loadedInstances["i-new"], "Should have new instance")
 }
 
-// TestJetStreamManager_MultipleNodes tests storing state for multiple nodes
+// TestJetStreamManager_MultipleNodes tests storing state for multiple nodes.
 func TestJetStreamManager_MultipleNodes(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -304,7 +304,7 @@ func TestJetStreamManager_MultipleNodes(t *testing.T) {
 	assert.False(t, exists, "Node-1 should not have node-2's instances")
 }
 
-// TestJetStreamManager_KVNotInitialized tests error handling when KV is not initialized
+// TestJetStreamManager_KVNotInitialized tests error handling when KV is not initialized.
 func TestJetStreamManager_KVNotInitialized(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -327,7 +327,7 @@ func TestJetStreamManager_KVNotInitialized(t *testing.T) {
 	assert.Error(t, err, "DeleteState should error when KV not initialized")
 }
 
-// TestJetStreamManager_UpdateReplicas tests updating replica count for the KV bucket
+// TestJetStreamManager_UpdateReplicas tests updating replica count for the KV bucket.
 func TestJetStreamManager_UpdateReplicas(t *testing.T) {
 	natsURL := sharedJSNATSURL
 
@@ -358,7 +358,7 @@ func TestJetStreamManager_UpdateReplicas(t *testing.T) {
 	// This test verifies the basic functionality works.
 }
 
-// TestJetStreamManager_UpdateReplicas_NoInit tests UpdateReplicas when JS not initialized
+// TestJetStreamManager_UpdateReplicas_NoInit tests UpdateReplicas when JS not initialized.
 func TestJetStreamManager_UpdateReplicas_NoInit(t *testing.T) {
 	// Test with nil JetStream context
 	jsm := &JetStreamManager{
@@ -372,7 +372,7 @@ func TestJetStreamManager_UpdateReplicas_NoInit(t *testing.T) {
 
 // --- Stopped instance KV tests ---
 
-// TestJetStreamManager_WriteAndLoadStoppedInstance tests round-trip write and load of a stopped instance
+// TestJetStreamManager_WriteAndLoadStoppedInstance tests round-trip write and load of a stopped instance.
 func TestJetStreamManager_WriteAndLoadStoppedInstance(t *testing.T) {
 	nc, err := nats.Connect(sharedJSNATSURL)
 	require.NoError(t, err)
@@ -407,7 +407,7 @@ func TestJetStreamManager_WriteAndLoadStoppedInstance(t *testing.T) {
 	_ = jsm.DeleteStoppedInstance(testVM.ID)
 }
 
-// TestJetStreamManager_LoadStoppedInstance_NotFound tests that LoadStoppedInstance returns nil for missing key
+// TestJetStreamManager_LoadStoppedInstance_NotFound tests that LoadStoppedInstance returns nil for missing key.
 func TestJetStreamManager_LoadStoppedInstance_NotFound(t *testing.T) {
 	nc, err := nats.Connect(sharedJSNATSURL)
 	require.NoError(t, err)
@@ -423,7 +423,7 @@ func TestJetStreamManager_LoadStoppedInstance_NotFound(t *testing.T) {
 	assert.Nil(t, loaded)
 }
 
-// TestJetStreamManager_DeleteStoppedInstance tests deleting a stopped instance (including non-existent)
+// TestJetStreamManager_DeleteStoppedInstance tests deleting a stopped instance (including non-existent).
 func TestJetStreamManager_DeleteStoppedInstance(t *testing.T) {
 	nc, err := nats.Connect(sharedJSNATSURL)
 	require.NoError(t, err)
@@ -653,7 +653,7 @@ func TestJetStreamManager_UpdateStoppedInstance_NoResurrectAfterClaim(t *testing
 	assert.Nil(t, stillGone, "the stopped record must stay deleted after the losing update")
 }
 
-// TestJetStreamManager_ListStoppedInstances tests listing multiple stopped instances
+// TestJetStreamManager_ListStoppedInstances tests listing multiple stopped instances.
 func TestJetStreamManager_ListStoppedInstances(t *testing.T) {
 	nc, err := nats.Connect(sharedJSNATSURL)
 	require.NoError(t, err)
@@ -695,7 +695,7 @@ func TestJetStreamManager_ListStoppedInstances(t *testing.T) {
 	}
 }
 
-// TestJetStreamManager_StoppedInstances_NoInterference tests that stopped instances don't interfere with per-node state
+// TestJetStreamManager_StoppedInstances_NoInterference tests that stopped instances don't interfere with per-node state.
 func TestJetStreamManager_StoppedInstances_NoInterference(t *testing.T) {
 	nc, err := nats.Connect(sharedJSNATSURL)
 	require.NoError(t, err)
@@ -735,7 +735,7 @@ func TestJetStreamManager_StoppedInstances_NoInterference(t *testing.T) {
 	_ = jsm.DeleteState("interference-test-node")
 }
 
-// TestJetStreamManager_StoppedInstance_KVNotInitialized tests error handling when KV is not initialized
+// TestJetStreamManager_StoppedInstance_KVNotInitialized tests error handling when KV is not initialized.
 func TestJetStreamManager_StoppedInstance_KVNotInitialized(t *testing.T) {
 	nc, err := nats.Connect(sharedJSNATSURL)
 	require.NoError(t, err)

spinifex/daemon/peer_health_test.go

@@ -182,7 +182,7 @@ func TestPeerDaemonPort_FallbackToSelf(t *testing.T) {
 
 	// Unparseable peer Daemon.Host + no self config — empty.
 	d2 := &Daemon{}
-	assert.Equal(t, "", d2.peerDaemonPort(config.Config{Daemon: config.DaemonConfig{Host: "garbage"}}))
+	assert.Empty(t, d2.peerDaemonPort(config.Config{Daemon: config.DaemonConfig{Host: "garbage"}}))
 }
 
 // TestMonitorPeerReachability_EndToEnd runs the real goroutine against the

spinifex/daemon/roles_test.go

@@ -64,7 +64,7 @@ func TestFetchNATSRole_ClusterFollower(t *testing.T) {
 
 func TestFetchNATSRole_Unreachable(t *testing.T) {
 	role := fetchNATSRole("http://127.0.0.1:1/varz", &http.Client{})
-	assert.Equal(t, "", role)
+	assert.Empty(t, role)
 }
 
 func TestFetchNATSRole_BadJSON(t *testing.T) {
@@ -74,7 +74,7 @@ func TestFetchNATSRole_BadJSON(t *testing.T) {
 	defer srv.Close()
 
 	role := fetchNATSRole(srv.URL, srv.Client())
-	assert.Equal(t, "", role)
+	assert.Empty(t, role)
 }
 
 func TestFetchPredastoreRole_Leader(t *testing.T) {
@@ -99,7 +99,7 @@ func TestFetchPredastoreRole_Follower(t *testing.T) {
 
 func TestFetchPredastoreRole_Unreachable(t *testing.T) {
 	role := fetchPredastoreRole("https://127.0.0.1:1/status", &http.Client{})
-	assert.Equal(t, "", role)
+	assert.Empty(t, role)
 }
 
 func TestFetchPredastoreRole_BadJSON(t *testing.T) {
@@ -109,5 +109,5 @@ func TestFetchPredastoreRole_BadJSON(t *testing.T) {
 	defer srv.Close()
 
 	role := fetchPredastoreRole(srv.URL, srv.Client())
-	assert.Equal(t, "", role)
+	assert.Empty(t, role)
 }

spinifex/daemon/state_test.go

@@ -742,7 +742,7 @@ func TestRestoreInstances_RunningResetsLaunchTime(t *testing.T) {
 	// LaunchTime should be reset to approximately now, not the stale 30min-ago time
 	assert.True(t, instance.Instance.LaunchTime.After(before) || instance.Instance.LaunchTime.Equal(before),
 		"LaunchTime should be reset to now, got %v (stale was %v)", *instance.Instance.LaunchTime, staleTime)
-	assert.True(t, time.Since(*instance.Instance.LaunchTime) < vm.PendingWatchdogTimeout,
+	assert.Less(t, time.Since(*instance.Instance.LaunchTime), vm.PendingWatchdogTimeout,
 		"LaunchTime should be within watchdog timeout window")
 }
 

spinifex/formation/formation.go

@@ -99,6 +99,17 @@ type SharedCredentials struct {
 	// Admin credentials (generated by init node, shared to join nodes)
 	AdminAccessKey string `json:"admin_access_key,omitempty"`
 	AdminSecretKey string `json:"admin_secret_key,omitempty"`
+
+	// Northstar credentials are the bucket-scoped, read-only pair the DNS
+	// resolver reads zone files with. Every node needs the identical pair:
+	// the zone bucket is distributed, but each node's predastore only honours
+	// the keys rendered into its own config.
+	//
+	// Both are omitempty so a cluster formed by an init node predating
+	// northstar distribution yields no northstar config on joiners at all,
+	// rather than a config holding a key no predastore recognises.
+	NorthstarAccessKey string `json:"northstar_access_key,omitempty"`
+	NorthstarSecretKey string `json:"northstar_secret_key,omitempty"`
 }
 
 // FormationServer is a lightweight HTTPS server that coordinates cluster formation.

spinifex/formation/formation_test.go

@@ -71,7 +71,7 @@ func TestGenerateJoinToken(t *testing.T) {
 	token1, err := GenerateJoinToken()
 	require.NoError(t, err)
 
-	assert.True(t, len(token1) > len("spx_join_"))
+	assert.Greater(t, len(token1), len("spx_join_"))
 	assert.Equal(t, "spx_join_", token1[:9])
 	assert.Len(t, token1[9:], 16)
 
@@ -371,6 +371,53 @@ func TestStatusEndpointViperblockKey(t *testing.T) {
 	assert.Equal(t, "dmlwZXJibG9jay1zaGFyZWQta2V5LWJhc2U2NA==", sr.ViperblockKey)
 }
 
+// The northstar resolver on every node reads the distributed zone bucket with
+// one shared, bucket-scoped pair, so joiners must receive the init node's keys
+// verbatim over the formation channel.
+func TestStatusEndpointNorthstarCredentials(t *testing.T) {
+	t.Parallel()
+	creds := testCreds()
+	creds.NorthstarAccessKey = "AKIANORTHSTAR1234567"
+	creds.NorthstarSecretKey = "northstarSecretKey1234567890"
+	fs := NewFormationServer(2, creds, "ca-cert", "ca-key", nil, testToken, testTokenTTL)
+	ts := testServer(t, fs)
+	defer ts.Close()
+
+	// Before completion the keys ride on Credentials, which is withheld entirely
+	resp := authGet(t, ts.URL+"/formation/status", testToken)
+	var sr StatusResponse
+	require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
+	resp.Body.Close()
+	assert.Nil(t, sr.Credentials)
+
+	require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
+	require.NoError(t, fs.RegisterNode(testNode("node2", "10.0.0.2")))
+
+	resp = authGet(t, ts.URL+"/formation/status", testToken)
+	defer resp.Body.Close()
+	require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
+	require.True(t, sr.Complete)
+	require.NotNil(t, sr.Credentials)
+	assert.Equal(t, "AKIANORTHSTAR1234567", sr.Credentials.NorthstarAccessKey)
+	assert.Equal(t, "northstarSecretKey1234567890", sr.Credentials.NorthstarSecretKey)
+}
+
+// An init node predating northstar distribution sends no northstar keys. The
+// joiner must then see a clean zero value and render no northstar config, not
+// a half-populated one holding a key its predastore never received.
+func TestSharedCredentialsNorthstarOmittedWhenUnset(t *testing.T) {
+	t.Parallel()
+	encoded, err := json.Marshal(testCreds())
+	require.NoError(t, err)
+	assert.NotContains(t, string(encoded), "northstar_access_key")
+	assert.NotContains(t, string(encoded), "northstar_secret_key")
+
+	var decoded SharedCredentials
+	require.NoError(t, json.Unmarshal(encoded, &decoded))
+	assert.Empty(t, decoded.NorthstarAccessKey)
+	assert.Empty(t, decoded.NorthstarSecretKey)
+}
+
 func TestHealthEndpoint_NoAuth(t *testing.T) {
 	t.Parallel()
 	fs := NewFormationServer(1, testCreds(), "", "", nil, testToken, testTokenTTL)

spinifex/gateway/acm/certificates.go

@@ -10,7 +10,7 @@ import (
 	"github.com/nats-io/nats.go"
 )
 
-// ImportCertificate — CertificateManager.ImportCertificate
+// ImportCertificate — CertificateManager.ImportCertificate.
 func ImportCertificate(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.ImportCertificateOutput, error) {
 	input := new(acm.ImportCertificateInput)
 	if err := unmarshalIfBody(body, input); err != nil {
@@ -19,7 +19,7 @@ func ImportCertificate(ctx context.Context, natsConn *nats.Conn, accountID strin
 	return handlers_acm.NewNATSACMService(natsConn).ImportCertificate(ctx, input, accountID)
 }
 
-// DescribeCertificate — CertificateManager.DescribeCertificate
+// DescribeCertificate — CertificateManager.DescribeCertificate.
 func DescribeCertificate(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.DescribeCertificateOutput, error) {
 	input := new(acm.DescribeCertificateInput)
 	if err := unmarshalIfBody(body, input); err != nil {
@@ -28,7 +28,7 @@ func DescribeCertificate(ctx context.Context, natsConn *nats.Conn, accountID str
 	return handlers_acm.NewNATSACMService(natsConn).DescribeCertificate(ctx, input, accountID)
 }
 
-// ListCertificates — CertificateManager.ListCertificates
+// ListCertificates — CertificateManager.ListCertificates.
 func ListCertificates(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.ListCertificatesOutput, error) {
 	input := new(acm.ListCertificatesInput)
 	if err := unmarshalIfBody(body, input); err != nil {
@@ -37,7 +37,7 @@ func ListCertificates(ctx context.Context, natsConn *nats.Conn, accountID string
 	return handlers_acm.NewNATSACMService(natsConn).ListCertificates(ctx, input, accountID)
 }
 
-// DeleteCertificate — CertificateManager.DeleteCertificate
+// DeleteCertificate — CertificateManager.DeleteCertificate.
 func DeleteCertificate(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.DeleteCertificateOutput, error) {
 	input := new(acm.DeleteCertificateInput)
 	if err := unmarshalIfBody(body, input); err != nil {
@@ -46,7 +46,7 @@ func DeleteCertificate(ctx context.Context, natsConn *nats.Conn, accountID strin
 	return handlers_acm.NewNATSACMService(natsConn).DeleteCertificate(ctx, input, accountID)
 }
 
-// ListTagsForCertificate — CertificateManager.ListTagsForCertificate
+// ListTagsForCertificate — CertificateManager.ListTagsForCertificate.
 func ListTagsForCertificate(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.ListTagsForCertificateOutput, error) {
 	input := new(acm.ListTagsForCertificateInput)
 	if err := unmarshalIfBody(body, input); err != nil {
@@ -55,7 +55,7 @@ func ListTagsForCertificate(ctx context.Context, natsConn *nats.Conn, accountID
 	return handlers_acm.NewNATSACMService(natsConn).ListTagsForCertificate(ctx, input, accountID)
 }
 
-// AddTagsToCertificate — CertificateManager.AddTagsToCertificate
+// AddTagsToCertificate — CertificateManager.AddTagsToCertificate.
 func AddTagsToCertificate(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.AddTagsToCertificateOutput, error) {
 	input := new(acm.AddTagsToCertificateInput)
 	if err := unmarshalIfBody(body, input); err != nil {
@@ -64,7 +64,7 @@ func AddTagsToCertificate(ctx context.Context, natsConn *nats.Conn, accountID st
 	return handlers_acm.NewNATSACMService(natsConn).AddTagsToCertificate(ctx, input, accountID)
 }
 
-// RemoveTagsFromCertificate — CertificateManager.RemoveTagsFromCertificate
+// RemoveTagsFromCertificate — CertificateManager.RemoveTagsFromCertificate.
 func RemoveTagsFromCertificate(ctx context.Context, natsConn *nats.Conn, accountID string, body []byte) (*acm.RemoveTagsFromCertificateOutput, error) {
 	input := new(acm.RemoveTagsFromCertificateInput)
 	if err := unmarshalIfBody(body, input); err != nil {

spinifex/gateway/acm_test.go

@@ -25,7 +25,7 @@ func setupACMRequest(target, body string) *http.Request {
 func TestACMActionFromTarget(t *testing.T) {
 	assert.Equal(t, "ImportCertificate", acmActionFromTarget("CertificateManager.ImportCertificate"))
 	assert.Equal(t, "ListCertificates", acmActionFromTarget("ListCertificates"))
-	assert.Equal(t, "", acmActionFromTarget(""))
+	assert.Empty(t, acmActionFromTarget(""))
 }
 
 func TestACMActionsMap_AllActionsRegistered(t *testing.T) {

spinifex/gateway/bedrock/anthropic.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"encoding/json"
 	"errors"
+	"fmt"
 	"io"
 	"log/slog"
 	"net/http"
@@ -58,6 +59,7 @@ type anthropicRequest struct {
 	Temperature   *float64           `json:"temperature,omitempty"`
 	TopP          *float64           `json:"top_p,omitempty"`
 	StopSequences []string           `json:"stop_sequences,omitempty"`
+	Stream        bool               `json:"stream,omitempty"`
 }
 
 type anthropicUsage struct {
@@ -115,6 +117,12 @@ func (b *anthropicBoundProvider) Converse(ctx context.Context, modelID string, i
 	return b.inner.Converse(ctx, modelID, input, b.apiKey)
 }
 
+var _ ConverseStreamProvider = (*anthropicBoundProvider)(nil)
+
+func (b *anthropicBoundProvider) ConverseStream(ctx context.Context, modelID string, input *bedrockruntime.ConverseStreamInput) (converseStreamSource, error) {
+	return b.inner.ConverseStream(ctx, modelID, input, b.apiKey)
+}
+
 // newAnthropicProvider returns a Provider that calls the Anthropic Messages
 // API with apiKey.
 func newAnthropicProvider(apiKey string) Provider {
@@ -249,3 +257,227 @@ func anthropicToConverseOutput(ar anthropicResponse, latency time.Duration) *bed
 		},
 	}
 }
+
+// ConverseStream translates input to an Anthropic Messages request with
+// stream:true, opens the SSE connection, and returns a converseStreamSource
+// that reframes it into the normalized Bedrock ConverseStream taxonomy.
+// Anthropic's own event taxonomy already matches Bedrock's nearly 1:1.
+func (p *anthropicProvider) ConverseStream(ctx context.Context, modelID string, input *bedrockruntime.ConverseStreamInput, apiKey string) (converseStreamSource, error) {
+	req := buildAnthropicRequest(modelID, converseStreamToConverseInput(input))
+	req.Stream = true
+
+	reqBody, err := json.Marshal(req)
+	if err != nil {
+		slog.Error("anthropic stream: failed to marshal request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+
+	httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, p.baseURL+anthropicMessagesPath, bytes.NewReader(reqBody)) //nolint:gosec // G704: p.baseURL is the hardcoded Anthropic API endpoint (or an httptest stub in tests), never user input
+	if err != nil {
+		slog.Error("anthropic stream: failed to build request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+	httpReq.Header.Set("X-Api-Key", apiKey)
+	httpReq.Header.Set("Anthropic-Version", anthropicAPIVersion)
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.Header.Set("Accept", "text/event-stream")
+
+	resp, err := p.httpClient.Do(httpReq) //nolint:gosec // G704: httpReq targets p.baseURL, not user input
+	if err != nil {
+		slog.Error("anthropic stream: request failed", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorServiceUnavailableException)
+	}
+
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		respBody, _ := io.ReadAll(resp.Body)
+		_ = resp.Body.Close()
+		slog.Error("anthropic stream: upstream error", "model", modelID, "status", resp.StatusCode, "body", string(respBody))
+		return nil, errors.New(mapUpstreamStatus(resp.StatusCode))
+	}
+
+	return &anthropicConverseStreamSource{
+		resp:    resp,
+		scanner: newSSEScanner(resp.Body),
+		start:   time.Now(),
+	}, nil
+}
+
+// anthropicConverseStreamSource reframes an Anthropic Messages SSE stream
+// into the normalized Bedrock ConverseStream taxonomy. Anthropic emits one
+// SSE event per normalized Bedrock event, except message_delta (captures
+// stop_reason/output tokens, emitted as messageStop) and message_stop
+// (triggers the trailing metadata event, carrying accumulated usage).
+type anthropicConverseStreamSource struct {
+	resp    *http.Response
+	scanner *sseScanner
+	start   time.Time
+
+	inputTokens  int64
+	outputTokens int64
+}
+
+var _ converseStreamSource = (*anthropicConverseStreamSource)(nil)
+
+func (s *anthropicConverseStreamSource) Close() error {
+	return s.resp.Body.Close()
+}
+
+func (s *anthropicConverseStreamSource) Next(_ context.Context) (ConverseStreamEvent, bool, error) {
+	for {
+		ev, ok, err := s.scanner.Next()
+		if err != nil {
+			if errors.Is(err, io.EOF) {
+				return ConverseStreamEvent{}, false, nil
+			}
+			return ConverseStreamEvent{}, false, newStreamFault(err)
+		}
+		if !ok {
+			continue
+		}
+
+		switch ev.Event {
+		case "", "ping":
+			continue
+		case "message_start":
+			return s.handleMessageStart(ev.Data)
+		case "content_block_start":
+			return s.handleContentBlockStart(ev.Data)
+		case "content_block_delta":
+			return s.handleContentBlockDelta(ev.Data)
+		case "content_block_stop":
+			return s.handleContentBlockStop(ev.Data)
+		case "message_delta":
+			return s.handleMessageDelta(ev.Data)
+		case "message_stop":
+			return s.handleMessageStop()
+		case "error":
+			return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("anthropic stream: upstream error event: %s", ev.Data))
+		default:
+			continue // forward-compat: skip unrecognised event types
+		}
+	}
+}
+
+func (s *anthropicConverseStreamSource) handleMessageStart(data string) (ConverseStreamEvent, bool, error) {
+	var payload struct {
+		Message struct {
+			Role  string `json:"role"`
+			Usage struct {
+				InputTokens int64 `json:"input_tokens"`
+			} `json:"usage"`
+		} `json:"message"`
+	}
+	if err := json.Unmarshal([]byte(data), &payload); err != nil {
+		return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("anthropic stream: decode message_start: %w", err))
+	}
+	s.inputTokens = payload.Message.Usage.InputTokens
+	role := payload.Message.Role
+	if role == "" {
+		role = bedrockruntime.ConversationRoleAssistant
+	}
+	return ConverseStreamEvent{
+		Kind:         converseStreamEventMessageStart,
+		MessageStart: &bedrockruntime.MessageStartEvent{Role: aws.String(role)},
+	}, true, nil
+}
+
+func (s *anthropicConverseStreamSource) handleContentBlockStart(data string) (ConverseStreamEvent, bool, error) {
+	var payload struct {
+		Index        int64 `json:"index"`
+		ContentBlock struct {
+			Type string `json:"type"`
+			ID   string `json:"id"`
+			Name string `json:"name"`
+		} `json:"content_block"`
+	}
+	if err := json.Unmarshal([]byte(data), &payload); err != nil {
+		return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("anthropic stream: decode content_block_start: %w", err))
+	}
+	start := &bedrockruntime.ContentBlockStart{}
+	if payload.ContentBlock.Type == "tool_use" {
+		start.ToolUse = &bedrockruntime.ToolUseBlockStart{
+			ToolUseId: aws.String(payload.ContentBlock.ID),
+			Name:      aws.String(payload.ContentBlock.Name),
+		}
+	}
+	return ConverseStreamEvent{
+		Kind: converseStreamEventContentBlockStart,
+		ContentBlockStart: &bedrockruntime.ContentBlockStartEvent{
+			ContentBlockIndex: aws.Int64(payload.Index),
+			Start:             start,
+		},
+	}, true, nil
+}
+
+func (s *anthropicConverseStreamSource) handleContentBlockDelta(data string) (ConverseStreamEvent, bool, error) {
+	var payload struct {
+		Index int64 `json:"index"`
+		Delta struct {
+			Type        string `json:"type"`
+			Text        string `json:"text"`
+			PartialJSON string `json:"partial_json"`
+		} `json:"delta"`
+	}
+	if err := json.Unmarshal([]byte(data), &payload); err != nil {
+		return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("anthropic stream: decode content_block_delta: %w", err))
+	}
+	delta := &bedrockruntime.ContentBlockDelta{}
+	if payload.Delta.Type == "input_json_delta" {
+		delta.ToolUse = &bedrockruntime.ToolUseBlockDelta{Input: aws.String(payload.Delta.PartialJSON)}
+	} else {
+		delta.Text = aws.String(payload.Delta.Text)
+	}
+	return ConverseStreamEvent{
+		Kind: converseStreamEventContentBlockDelta,
+		ContentBlockDelta: &bedrockruntime.ContentBlockDeltaEvent{
+			ContentBlockIndex: aws.Int64(payload.Index),
+			Delta:             delta,
+		},
+	}, true, nil
+}
+
+func (s *anthropicConverseStreamSource) handleContentBlockStop(data string) (ConverseStreamEvent, bool, error) {
+	var payload struct {
+		Index int64 `json:"index"`
+	}
+	if err := json.Unmarshal([]byte(data), &payload); err != nil {
+		return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("anthropic stream: decode content_block_stop: %w", err))
+	}
+	return ConverseStreamEvent{
+		Kind:             converseStreamEventContentBlockStop,
+		ContentBlockStop: &bedrockruntime.ContentBlockStopEvent{ContentBlockIndex: aws.Int64(payload.Index)},
+	}, true, nil
+}
+
+func (s *anthropicConverseStreamSource) handleMessageDelta(data string) (ConverseStreamEvent, bool, error) {
+	var payload struct {
+		Delta struct {
+			StopReason string `json:"stop_reason"`
+		} `json:"delta"`
+		Usage struct {
+			OutputTokens int64 `json:"output_tokens"`
+		} `json:"usage"`
+	}
+	if err := json.Unmarshal([]byte(data), &payload); err != nil {
+		return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("anthropic stream: decode message_delta: %w", err))
+	}
+	s.outputTokens = payload.Usage.OutputTokens
+	return ConverseStreamEvent{
+		Kind:        converseStreamEventMessageStop,
+		MessageStop: &bedrockruntime.MessageStopEvent{StopReason: aws.String(mapAnthropicStopReason(payload.Delta.StopReason))},
+	}, true, nil
+}
+
+func (s *anthropicConverseStreamSource) handleMessageStop() (ConverseStreamEvent, bool, error) {
+	return ConverseStreamEvent{
+		Kind: converseStreamEventMetadata,
+		Metadata: &bedrockruntime.ConverseStreamMetadataEvent{
+			Usage: &bedrockruntime.TokenUsage{
+				InputTokens:  aws.Int64(s.inputTokens),
+				OutputTokens: aws.Int64(s.outputTokens),
+				TotalTokens:  aws.Int64(s.inputTokens + s.outputTokens),
+			},
+			Metrics: &bedrockruntime.ConverseStreamMetrics{LatencyMs: aws.Int64(time.Since(s.start).Milliseconds())},
+		},
+	}, true, nil
+}

spinifex/gateway/bedrock/anthropic_test.go

@@ -20,7 +20,10 @@ func TestAnthropicProvider_Converse_MapsResponse(t *testing.T) {
 
 	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		capturedAPIKey = r.Header.Get("X-Api-Key")
-		require.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody))
+		if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody)) {
+			http.Error(w, "decode request body", http.StatusBadRequest)
+			return
+		}
 		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte(`{
@@ -126,6 +129,214 @@ func TestAnthropicBoundProvider_Converse(t *testing.T) {
 	assert.Equal(t, "bound response", *out.Output.Message.Content[0].Text)
 }
 
+// anthropicStreamFixture is a canned Anthropic Messages streaming SSE body
+// covering the full taxonomy, including a keepalive ping and a
+// tool_use content block to exercise structural passthrough.
+const anthropicStreamFixture = `event: message_start
+data: {"type":"message_start","message":{"id":"msg_1","role":"assistant","usage":{"input_tokens":12}}}
+
+event: content_block_start
+data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
+
+event: ping
+data: {"type":"ping"}
+
+event: content_block_delta
+data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
+
+event: content_block_delta
+data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" there"}}
+
+event: content_block_stop
+data: {"type":"content_block_stop","index":0}
+
+event: content_block_start
+data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_1","name":"get_weather"}}
+
+event: content_block_delta
+data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"city\":"}}
+
+event: content_block_stop
+data: {"type":"content_block_stop","index":1}
+
+event: message_delta
+data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":5}}
+
+event: message_stop
+data: {"type":"message_stop"}
+
+`
+
+func TestAnthropicProvider_ConverseStream_MapsSSEToTaxonomy(t *testing.T) {
+	var capturedBody anthropicRequest
+	var capturedAPIKey string
+
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		capturedAPIKey = r.Header.Get("X-Api-Key")
+		assert.Equal(t, "text/event-stream", r.Header.Get("Accept"))
+		assert.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody))
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(anthropicStreamFixture))
+	}))
+	defer ts.Close()
+
+	p := &anthropicProvider{httpClient: ts.Client(), baseURL: ts.URL}
+
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hi")}}},
+		},
+	}
+
+	src, err := p.ConverseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", input, "sk-test-key")
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	assert.Equal(t, "sk-test-key", capturedAPIKey)
+	assert.True(t, capturedBody.Stream)
+
+	events := drainConverseStream(t, src)
+	require.Len(t, events, 10)
+
+	kinds := make([]converseStreamEventKind, len(events))
+	for i, ev := range events {
+		kinds[i] = ev.Kind
+	}
+	assert.Equal(t, []converseStreamEventKind{
+		converseStreamEventMessageStart,
+		converseStreamEventContentBlockStart,
+		converseStreamEventContentBlockDelta,
+		converseStreamEventContentBlockDelta,
+		converseStreamEventContentBlockStop,
+		converseStreamEventContentBlockStart,
+		converseStreamEventContentBlockDelta,
+		converseStreamEventContentBlockStop,
+		converseStreamEventMessageStop,
+		converseStreamEventMetadata,
+	}, kinds)
+
+	assert.Equal(t, bedrockruntime.ConversationRoleAssistant, *events[0].MessageStart.Role)
+	assert.Equal(t, "Hello", *events[2].ContentBlockDelta.Delta.Text)
+	assert.Equal(t, " there", *events[3].ContentBlockDelta.Delta.Text)
+
+	toolStart := events[5].ContentBlockStart
+	require.NotNil(t, toolStart.Start.ToolUse)
+	assert.Equal(t, "toolu_1", *toolStart.Start.ToolUse.ToolUseId)
+	assert.Equal(t, "get_weather", *toolStart.Start.ToolUse.Name)
+
+	toolDelta := events[6].ContentBlockDelta
+	require.NotNil(t, toolDelta.Delta.ToolUse)
+	assert.Equal(t, `{"city":`, *toolDelta.Delta.ToolUse.Input)
+
+	assert.Equal(t, bedrockruntime.StopReasonToolUse, *events[8].MessageStop.StopReason)
+	assert.Equal(t, int64(12), *events[9].Metadata.Usage.InputTokens)
+	assert.Equal(t, int64(5), *events[9].Metadata.Usage.OutputTokens)
+	assert.Equal(t, int64(17), *events[9].Metadata.Usage.TotalTokens)
+}
+
+func TestAnthropicConverseStreamSource_MessageStopEmitsMetadata(t *testing.T) {
+	raw := `event: message_start
+data: {"type":"message_start","message":{"role":"assistant","usage":{"input_tokens":4}}}
+
+event: message_delta
+data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":2}}
+
+event: message_stop
+data: {"type":"message_stop"}
+
+`
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(raw))
+	}))
+	defer ts.Close()
+
+	p := &anthropicProvider{httpClient: ts.Client(), baseURL: ts.URL}
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hi")}}}},
+	}
+	src, err := p.ConverseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", input, "k")
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	events := drainConverseStream(t, src)
+	require.Len(t, events, 3)
+	assert.Equal(t, converseStreamEventMetadata, events[2].Kind)
+	assert.Equal(t, int64(4), *events[2].Metadata.Usage.InputTokens)
+	assert.Equal(t, int64(2), *events[2].Metadata.Usage.OutputTokens)
+	assert.Equal(t, int64(6), *events[2].Metadata.Usage.TotalTokens)
+}
+
+func TestAnthropicConverseStreamSource_ErrorEventSurfacesAsStreamFault(t *testing.T) {
+	raw := "event: error\ndata: {\"error\":{\"message\":\"overloaded\"}}\n\n"
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(raw))
+	}))
+	defer ts.Close()
+
+	p := &anthropicProvider{httpClient: ts.Client(), baseURL: ts.URL}
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hi")}}}},
+	}
+	src, err := p.ConverseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", input, "k")
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	_, ok, err := src.Next(context.Background())
+	assert.False(t, ok)
+	require.Error(t, err)
+	var fault *streamFaultError
+	assert.ErrorAs(t, err, &fault)
+	assert.Contains(t, err.Error(), "overloaded")
+}
+
+func TestAnthropicProvider_ConverseStream_401ReturnsAccessDenied(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusUnauthorized)
+		_, _ = w.Write([]byte(`{"error": {"message": "invalid x-api-key"}}`))
+	}))
+	defer ts.Close()
+
+	p := &anthropicProvider{httpClient: ts.Client(), baseURL: ts.URL}
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hi")}}}},
+	}
+	_, err := p.ConverseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", input, "bad-key")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorAccessDeniedException, err.Error())
+}
+
+func TestAnthropicBoundProvider_ConverseStream(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		assert.Equal(t, "sk-bound-test", r.Header.Get("X-Api-Key"))
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(anthropicStreamFixture))
+	}))
+	defer ts.Close()
+
+	p := newAnthropicProvider("sk-bound-test")
+	bound, ok := p.(*anthropicBoundProvider)
+	require.True(t, ok)
+	bound.inner.httpClient = ts.Client()
+	bound.inner.baseURL = ts.URL
+
+	sp, ok := p.(ConverseStreamProvider)
+	require.True(t, ok)
+
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hi")}}}},
+	}
+	src, err := sp.ConverseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", input)
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	events := drainConverseStream(t, src)
+	require.NotEmpty(t, events)
+	assert.Equal(t, converseStreamEventMessageStart, events[0].Kind)
+}
+
 func TestAnthropicModelID(t *testing.T) {
 	cases := []struct {
 		name, in, want string

spinifex/gateway/bedrock/contract_test.go

@@ -0,0 +1,240 @@
+package gateway_bedrock
+
+import (
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"regexp"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/awstesting/unit"
+	"github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamtest"
+	"github.com/aws/aws-sdk-go/service/bedrockruntime"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// This file is the contract test the plan's acceptance criterion calls for:
+// feed our production reframers' output through the *real* aws-sdk-go v1
+// bedrockruntime client (not just decode the raw bytes ourselves), proving a
+// genuine SDK consumer can read what we frame — byte framing, headers, and
+// JSON payload shape all have to be right for this to work at all.
+
+// contractSession stands up an httptest server (TLS+h2, matching a real
+// event-stream transport) running handler, and returns a session pointed at
+// it plus a bedrockruntime client built on that session.
+func contractSession(t *testing.T, handler http.HandlerFunc) *bedrockruntime.BedrockRuntime {
+	t.Helper()
+	sess, cleanup, err := eventstreamtest.SetupEventStreamSession(t, handler, true)
+	require.NoError(t, err)
+	t.Cleanup(cleanup)
+	return bedrockruntime.New(sess, &aws.Config{Credentials: unit.Session.Config.Credentials})
+}
+
+var converseStreamPathPattern = regexp.MustCompile(`^/model/([^/]+)/converse-stream$`)
+var invokeStreamPathPattern = regexp.MustCompile(`^/model/([^/]+)/invoke-with-response-stream$`)
+
+func TestContract_ConverseStream_SelfHost_RealSDKConsumerDecodesFrames(t *testing.T) {
+	vllmStub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(vllmStreamFixture))
+	}))
+	defer vllmStub.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	endpoints := NewStaticEndpointResolver(map[string]string{modelID: vllmStub.URL})
+
+	svc := contractSession(t, func(w http.ResponseWriter, r *http.Request) {
+		m := converseStreamPathPattern.FindStringSubmatch(r.URL.Path)
+		if !assert.NotNil(t, m, "unexpected path %s", r.URL.Path) {
+			return
+		}
+		body, err := readAll(r)
+		if !assert.NoError(t, err) {
+			return
+		}
+		err = ConverseStream(r.Context(), w, "000000000001", m[1], body, nil, endpoints)
+		assert.NoError(t, err)
+	})
+
+	resp, err := svc.ConverseStream(&bedrockruntime.ConverseStreamInput{
+		ModelId: aws.String(modelID),
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	})
+	require.NoError(t, err)
+	defer func() { _ = resp.GetStream().Close() }()
+
+	var kinds []string
+	var deltas []string
+	for event := range resp.GetStream().Events() {
+		switch e := event.(type) {
+		case *bedrockruntime.MessageStartEvent:
+			kinds = append(kinds, "messageStart")
+			assert.Equal(t, bedrockruntime.ConversationRoleAssistant, *e.Role)
+		case *bedrockruntime.ContentBlockStartEvent:
+			kinds = append(kinds, "contentBlockStart")
+		case *bedrockruntime.ContentBlockDeltaEvent:
+			kinds = append(kinds, "contentBlockDelta")
+			deltas = append(deltas, *e.Delta.Text)
+		case *bedrockruntime.ContentBlockStopEvent:
+			kinds = append(kinds, "contentBlockStop")
+		case *bedrockruntime.MessageStopEvent:
+			kinds = append(kinds, "messageStop")
+			assert.Equal(t, bedrockruntime.StopReasonEndTurn, *e.StopReason)
+		case *bedrockruntime.ConverseStreamMetadataEvent:
+			kinds = append(kinds, "metadata")
+			assert.Equal(t, int64(8), *e.Usage.InputTokens)
+			assert.Equal(t, int64(3), *e.Usage.OutputTokens)
+		default:
+			t.Fatalf("unexpected event type %T", event)
+		}
+	}
+	require.NoError(t, resp.GetStream().Err())
+
+	assert.Equal(t, []string{
+		"messageStart", "contentBlockStart", "contentBlockDelta", "contentBlockDelta",
+		"contentBlockStop", "messageStop", "metadata",
+	}, kinds)
+	assert.Equal(t, []string{"Hello", " world"}, deltas)
+}
+
+func TestContract_ConverseStream_Anthropic_RealSDKConsumerDecodesFrames(t *testing.T) {
+	anthropicStub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(anthropicStreamFixture))
+	}))
+	defer anthropicStub.Close()
+
+	modelID := "anthropic.claude-3-5-sonnet-20240620-v1:0"
+
+	// The top-level ConverseStream entrypoint routes provider-direct traffic
+	// through Router, which hardcodes the real Anthropic base URL — there is
+	// no injection seam for it (by design: it's a live vendor endpoint, not
+	// a pinned self-host one). So this contract test drives the same
+	// production pieces (anthropicProvider.ConverseStream + pumpConverseStream
+	// + frameWriter) directly against the stub, rather than through
+	// ConverseStream(ctx, w, ...): the framing/encoding under test is
+	// identical either way, since both operations share pumpConverseStream.
+	svc := contractSession(t, func(w http.ResponseWriter, r *http.Request) {
+		p := &anthropicProvider{httpClient: anthropicStub.Client(), baseURL: anthropicStub.URL}
+		src, err := p.ConverseStream(r.Context(), modelID, &bedrockruntime.ConverseStreamInput{
+			Messages: []*bedrockruntime.Message{
+				{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hi")}}},
+			},
+		}, "sk-test")
+		if !assert.NoError(t, err) {
+			return
+		}
+		defer func() { _ = src.Close() }()
+
+		fw, err := newFrameWriter(w)
+		if !assert.NoError(t, err) {
+			return
+		}
+		pumpConverseStream(r.Context(), fw, src, modelID)
+	})
+
+	resp, err := svc.ConverseStream(&bedrockruntime.ConverseStreamInput{ModelId: aws.String(modelID)})
+	require.NoError(t, err)
+	defer func() { _ = resp.GetStream().Close() }()
+
+	var kinds []string
+	var sawToolUse bool
+	for event := range resp.GetStream().Events() {
+		switch e := event.(type) {
+		case *bedrockruntime.MessageStartEvent:
+			kinds = append(kinds, "messageStart")
+		case *bedrockruntime.ContentBlockStartEvent:
+			kinds = append(kinds, "contentBlockStart")
+			if e.Start != nil && e.Start.ToolUse != nil {
+				sawToolUse = true
+				assert.Equal(t, "get_weather", *e.Start.ToolUse.Name)
+			}
+		case *bedrockruntime.ContentBlockDeltaEvent:
+			kinds = append(kinds, "contentBlockDelta")
+		case *bedrockruntime.ContentBlockStopEvent:
+			kinds = append(kinds, "contentBlockStop")
+		case *bedrockruntime.MessageStopEvent:
+			kinds = append(kinds, "messageStop")
+			assert.Equal(t, bedrockruntime.StopReasonToolUse, *e.StopReason)
+		case *bedrockruntime.ConverseStreamMetadataEvent:
+			kinds = append(kinds, "metadata")
+			assert.Equal(t, int64(12), *e.Usage.InputTokens)
+			assert.Equal(t, int64(5), *e.Usage.OutputTokens)
+		default:
+			t.Fatalf("unexpected event type %T", event)
+		}
+	}
+	require.NoError(t, resp.GetStream().Err())
+
+	assert.Equal(t, []string{
+		"messageStart", "contentBlockStart", "contentBlockDelta", "contentBlockDelta", "contentBlockStop",
+		"contentBlockStart", "contentBlockDelta", "contentBlockStop", "messageStop", "metadata",
+	}, kinds)
+	assert.True(t, sawToolUse)
+}
+
+func TestContract_InvokeModelWithResponseStream_SelfHost_RealSDKConsumerDecodesFrames(t *testing.T) {
+	llamaStub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(llamaCompletionsStreamFixture))
+	}))
+	defer llamaStub.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	endpoints := NewStaticEndpointResolver(map[string]string{modelID: llamaStub.URL})
+
+	svc := contractSession(t, func(w http.ResponseWriter, r *http.Request) {
+		m := invokeStreamPathPattern.FindStringSubmatch(r.URL.Path)
+		if !assert.NotNil(t, m, "unexpected path %s", r.URL.Path) {
+			return
+		}
+		body, err := readAll(r)
+		if !assert.NoError(t, err) {
+			return
+		}
+		err = InvokeModelWithResponseStream(r.Context(), w, "000000000001", m[1], body, nil, endpoints, r.Header.Get("Content-Type"))
+		assert.NoError(t, err)
+	})
+
+	resp, err := svc.InvokeModelWithResponseStream(&bedrockruntime.InvokeModelWithResponseStreamInput{
+		ModelId: aws.String(modelID),
+		Body:    []byte(`{"prompt":"hello","max_gen_len":128}`),
+	})
+	require.NoError(t, err)
+	defer func() { _ = resp.GetStream().Close() }()
+
+	var generations []string
+	var sawFinal bool
+	for event := range resp.GetStream().Events() {
+		part, ok := event.(*bedrockruntime.PayloadPart)
+		require.True(t, ok, "unexpected event type %T", event)
+
+		var probe struct {
+			Generation string `json:"generation"`
+			StopReason string `json:"stop_reason"`
+		}
+		require.NoError(t, json.Unmarshal(part.Bytes, &probe))
+		generations = append(generations, probe.Generation)
+		if probe.StopReason != "" {
+			sawFinal = true
+			assert.Equal(t, "stop", probe.StopReason)
+		}
+	}
+	require.NoError(t, resp.GetStream().Err())
+
+	assert.Equal(t, []string{"Hello", " world", ""}, generations)
+	assert.True(t, sawFinal)
+}
+
+func readAll(r *http.Request) ([]byte, error) {
+	defer func() { _ = r.Body.Close() }()
+	return io.ReadAll(r.Body)
+}

spinifex/gateway/bedrock/converse_stream.go

@@ -0,0 +1,165 @@
+package gateway_bedrock
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log/slog"
+	"net/http"
+
+	"github.com/aws/aws-sdk-go/private/protocol/json/jsonutil"
+	"github.com/aws/aws-sdk-go/service/bedrockruntime"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+)
+
+// converseStreamEventKind tags which aws-sdk-go v1 event struct a
+// ConverseStreamEvent carries, driving both the frame's ":event-type" header
+// and which field payload() marshals.
+type converseStreamEventKind int
+
+const (
+	converseStreamEventMessageStart converseStreamEventKind = iota
+	converseStreamEventContentBlockStart
+	converseStreamEventContentBlockDelta
+	converseStreamEventContentBlockStop
+	converseStreamEventMessageStop
+	converseStreamEventMetadata
+)
+
+// eventType is the Bedrock wire ":event-type" for kind.
+func (k converseStreamEventKind) eventType() string {
+	switch k {
+	case converseStreamEventMessageStart:
+		return "messageStart"
+	case converseStreamEventContentBlockStart:
+		return "contentBlockStart"
+	case converseStreamEventContentBlockDelta:
+		return "contentBlockDelta"
+	case converseStreamEventContentBlockStop:
+		return "contentBlockStop"
+	case converseStreamEventMessageStop:
+		return "messageStop"
+	case converseStreamEventMetadata:
+		return "metadata"
+	default:
+		return "unknown"
+	}
+}
+
+// ConverseStreamEvent is one normalized Bedrock ConverseStream event. Exactly
+// one payload field is set, matching Kind; reframers (vLLM, Anthropic) build
+// these from their own SSE so the writer/pump stays backend-agnostic.
+type ConverseStreamEvent struct {
+	Kind              converseStreamEventKind
+	MessageStart      *bedrockruntime.MessageStartEvent
+	ContentBlockStart *bedrockruntime.ContentBlockStartEvent
+	ContentBlockDelta *bedrockruntime.ContentBlockDeltaEvent
+	ContentBlockStop  *bedrockruntime.ContentBlockStopEvent
+	MessageStop       *bedrockruntime.MessageStopEvent
+	Metadata          *bedrockruntime.ConverseStreamMetadataEvent
+}
+
+// payload JSON-marshals the event struct matching Kind via jsonutil.BuildJSON
+// (the SDK's own restjson marshaler), honouring the locationName tags a real
+// SDK consumer's UnmarshalEvent expects — mirroring WriteJSONResponse.
+func (e ConverseStreamEvent) payload() ([]byte, error) {
+	switch e.Kind {
+	case converseStreamEventMessageStart:
+		return jsonutil.BuildJSON(e.MessageStart)
+	case converseStreamEventContentBlockStart:
+		return jsonutil.BuildJSON(e.ContentBlockStart)
+	case converseStreamEventContentBlockDelta:
+		return jsonutil.BuildJSON(e.ContentBlockDelta)
+	case converseStreamEventContentBlockStop:
+		return jsonutil.BuildJSON(e.ContentBlockStop)
+	case converseStreamEventMessageStop:
+		return jsonutil.BuildJSON(e.MessageStop)
+	case converseStreamEventMetadata:
+		return jsonutil.BuildJSON(e.Metadata)
+	default:
+		return nil, fmt.Errorf("converse stream: unknown event kind %d", e.Kind)
+	}
+}
+
+// converseStreamSource is the backend-agnostic reframer contract: Next
+// returns the next normalized event, or (zero,false,nil) at clean EOF.
+type converseStreamSource interface {
+	Next(ctx context.Context) (ConverseStreamEvent, bool, error)
+	Close() error
+}
+
+// ConverseStream is the bedrock-runtime ConverseStream entry point used by
+// the gateway route table. Unlike the JSON-dispatch handlers it owns w
+// directly: a pre-stream failure (unknown model, unresolved credential,
+// upstream connect error) returns an awserrors code for the normal
+// ErrorHandler envelope. Once the first frame is written it always returns
+// nil — any later failure is an in-band exception event, since the HTTP
+// status can no longer change.
+func ConverseStream(ctx context.Context, w http.ResponseWriter, accountID, modelID string, body []byte, resolver CredentialResolver, endpointResolver EndpointResolver) error {
+	input := new(bedrockruntime.ConverseStreamInput)
+	if len(body) > 0 {
+		if err := json.Unmarshal(body, input); err != nil {
+			return errors.New(awserrors.ErrorValidationException)
+		}
+	}
+
+	src, err := NewRouter(resolver, endpointResolver).ConverseStream(ctx, accountID, modelID, input)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		if cerr := src.Close(); cerr != nil {
+			slog.Error("converse-stream: failed to close upstream source", "model", modelID, "err", cerr)
+		}
+	}()
+
+	fw, err := newFrameWriter(w)
+	if err != nil {
+		return err
+	}
+
+	pumpConverseStream(ctx, fw, src, modelID)
+	return nil
+}
+
+// pumpConverseStream drains src and writes each normalized event as a frame,
+// in AWS order (messageStart -> per block contentBlockStart/Delta*/Stop ->
+// messageStop -> metadata). A mid-stream Next error surfaces as an in-band
+// exception event and stops the pump; a write/flush error also stops the
+// pump silently, since the client connection is already broken and the
+// HTTP status can no longer change either way.
+func pumpConverseStream(ctx context.Context, fw *frameWriter, src converseStreamSource, modelID string) {
+	for {
+		event, ok, err := src.Next(ctx)
+		if err != nil {
+			excType := excInternalServerException
+			var fault *streamFaultError
+			if errors.As(err, &fault) {
+				excType = excModelStreamErrorException
+			}
+			slog.Error("converse-stream: upstream fault", "model", modelID, "err", err)
+			if werr := fw.writeException(excType, exceptionPayload(err)); werr != nil {
+				slog.Error("converse-stream: failed to write exception frame", "model", modelID, "err", werr)
+			}
+			return
+		}
+		if !ok {
+			return
+		}
+
+		payload, err := event.payload()
+		if err != nil {
+			slog.Error("converse-stream: failed to marshal event", "model", modelID, "kind", event.Kind, "err", err)
+			if werr := fw.writeException(excInternalServerException, exceptionPayload(err)); werr != nil {
+				slog.Error("converse-stream: failed to write exception frame", "model", modelID, "err", werr)
+			}
+			return
+		}
+
+		if werr := fw.writeEvent(event.Kind.eventType(), payload); werr != nil {
+			slog.Error("converse-stream: failed to write frame, aborting", "model", modelID, "err", werr)
+			return
+		}
+	}
+}

spinifex/gateway/bedrock/converse_stream_test.go

@@ -0,0 +1,204 @@
+package gateway_bedrock
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	esv2 "github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream"
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/bedrockruntime"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeConverseStreamSource drives pumpConverseStream directly with a scripted
+// event/error sequence, independent of any real backend.
+type fakeConverseStreamSource struct {
+	events []ConverseStreamEvent
+	err    error
+	closed bool
+}
+
+func (f *fakeConverseStreamSource) Next(_ context.Context) (ConverseStreamEvent, bool, error) {
+	if len(f.events) > 0 {
+		ev := f.events[0]
+		f.events = f.events[1:]
+		return ev, true, nil
+	}
+	if f.err != nil {
+		return ConverseStreamEvent{}, false, f.err
+	}
+	return ConverseStreamEvent{}, false, nil
+}
+
+func (f *fakeConverseStreamSource) Close() error {
+	f.closed = true
+	return nil
+}
+
+// decodeAllFrames decodes every event-stream message in raw via the v2
+// package Decoder, returning each frame's ":message-type"/":event-type" (or
+// ":exception-type") header and payload.
+type decodedFrame struct {
+	MessageType string
+	Type        string
+	Payload     []byte
+}
+
+func decodeAllFrames(t *testing.T, raw []byte) []decodedFrame {
+	t.Helper()
+	dec := esv2.NewDecoder()
+	r := bytes.NewReader(raw)
+	var out []decodedFrame
+	for r.Len() > 0 {
+		msg, err := dec.Decode(r, nil)
+		require.NoError(t, err)
+		mt := headerString(t, msg, ":message-type")
+		typeHeader := ":event-type"
+		if mt == "exception" {
+			typeHeader = ":exception-type"
+		}
+		out = append(out, decodedFrame{MessageType: mt, Type: headerString(t, msg, typeHeader), Payload: msg.Payload})
+	}
+	return out
+}
+
+func TestPumpConverseStream_MidStreamFaultWritesExceptionFrame(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	src := &fakeConverseStreamSource{
+		events: []ConverseStreamEvent{
+			{Kind: converseStreamEventMessageStart, MessageStart: &bedrockruntime.MessageStartEvent{Role: aws.String("assistant")}},
+		},
+		err: newStreamFault(errors.New("upstream connection reset")),
+	}
+
+	pumpConverseStream(context.Background(), fw, src, "test-model")
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 2)
+	assert.Equal(t, "event", frames[0].MessageType)
+	assert.Equal(t, "messageStart", frames[0].Type)
+	assert.Equal(t, "exception", frames[1].MessageType)
+	assert.Equal(t, excModelStreamErrorException, frames[1].Type)
+	assert.JSONEq(t, `{"message":"upstream connection reset"}`, string(frames[1].Payload))
+
+	// The status must still be 200: post-header, a fault cannot change it.
+	assert.Equal(t, http.StatusOK, rec.Code)
+}
+
+func TestPumpConverseStream_NonFaultErrorUsesInternalServerException(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	src := &fakeConverseStreamSource{err: errors.New("plain internal error")}
+	pumpConverseStream(context.Background(), fw, src, "test-model")
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 1)
+	assert.Equal(t, excInternalServerException, frames[0].Type)
+}
+
+func TestPumpConverseStream_CleanEOFWritesNoExceptionFrame(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	src := &fakeConverseStreamSource{
+		events: []ConverseStreamEvent{
+			{Kind: converseStreamEventMessageStart, MessageStart: &bedrockruntime.MessageStartEvent{Role: aws.String("assistant")}},
+			{Kind: converseStreamEventMessageStop, MessageStop: &bedrockruntime.MessageStopEvent{StopReason: aws.String(bedrockruntime.StopReasonEndTurn)}},
+		},
+	}
+	pumpConverseStream(context.Background(), fw, src, "test-model")
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 2)
+	assert.Equal(t, "messageStart", frames[0].Type)
+	assert.Equal(t, "messageStop", frames[1].Type)
+}
+
+func TestConverseStream_UnknownModelReturnsResourceNotFoundPreHeader(t *testing.T) {
+	rec := httptest.NewRecorder()
+	body := []byte(`{"messages":[{"role":"user","content":[{"text":"hi"}]}]}`)
+
+	err := ConverseStream(context.Background(), rec, "000000000001", "does.not-exist-v1:0", body, nil, nil)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
+	// A pre-stream failure must not have written anything: the gateway's
+	// ErrorHandler owns turning this into the JSON error envelope.
+	assert.Equal(t, 0, rec.Body.Len())
+}
+
+func TestConverseStream_MalformedBodyReturnsValidationException(t *testing.T) {
+	rec := httptest.NewRecorder()
+	err := ConverseStream(context.Background(), rec, "000000000001", "meta.llama3-70b-instruct-v1:0", []byte("{not-json"), nil, nil)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorValidationException, err.Error())
+}
+
+func TestConverseStream_SelfHostHappyPath_WritesFramedTaxonomy(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(vllmStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	rec := httptest.NewRecorder()
+	body, err := json.Marshal(&bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	})
+	require.NoError(t, err)
+
+	err = ConverseStream(context.Background(), rec, "000000000001", modelID, body, nil, NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	require.NoError(t, err)
+
+	assert.Equal(t, http.StatusOK, rec.Code)
+	assert.Equal(t, eventStreamContentType, rec.Header().Get("Content-Type"))
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 7)
+	assert.Equal(t, "messageStart", frames[0].Type)
+	assert.Equal(t, "contentBlockStart", frames[1].Type)
+	assert.Equal(t, "contentBlockDelta", frames[2].Type)
+	assert.Equal(t, "contentBlockDelta", frames[3].Type)
+	assert.Equal(t, "contentBlockStop", frames[4].Type)
+	assert.Equal(t, "messageStop", frames[5].Type)
+	assert.Equal(t, "metadata", frames[6].Type)
+}
+
+func TestConverseStream_NonFlusherWriter_ReturnsPreHeaderError(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(vllmStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	w := &nonFlusherWriter{}
+	body, err := json.Marshal(&bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	})
+	require.NoError(t, err)
+
+	err = ConverseStream(context.Background(), w, "000000000001", modelID, body, nil, NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInternalError, err.Error())
+	assert.False(t, w.wroteHeader)
+}

spinifex/gateway/bedrock/credentials.go

@@ -2,6 +2,7 @@ package gateway_bedrock
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"sync"
 
@@ -88,7 +89,7 @@ func (s *CredentialStore) Resolve(_ context.Context, accountID, vendor string) (
 				return "", false, fmt.Errorf("decrypt credential for %s/%s: %w", accountID, vendor, err)
 			}
 			return plaintext, true, nil
-		case err != nats.ErrKeyNotFound:
+		case !errors.Is(err, nats.ErrKeyNotFound):
 			return "", false, fmt.Errorf("kv get credential for %s/%s: %w", accountID, vendor, err)
 		}
 	}

spinifex/gateway/bedrock/eventstream.go

@@ -0,0 +1,146 @@
+package gateway_bedrock
+
+import (
+	"encoding/json"
+	"errors"
+	"net/http"
+
+	esv2 "github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream"
+	"github.com/aws/aws-sdk-go/private/protocol/json/jsonutil"
+	"github.com/aws/aws-sdk-go/service/bedrockruntime"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+)
+
+// eventStreamContentType is the wire content type for both ConverseStream and
+// InvokeModelWithResponseStream responses.
+const eventStreamContentType = "application/vnd.amazon.eventstream"
+
+// Bedrock in-band exception types, used as the frame ":exception-type"
+// header once WriteHeader(200) has already been sent and the HTTP status can
+// no longer change. Every pre-header failure instead returns a normal
+// awserrors code for the ordinary JSON error envelope.
+const (
+	excInternalServerException   = "internalServerException"
+	excModelStreamErrorException = "modelStreamErrorException"
+)
+
+// streamFaultError marks a reframer error as originating from the upstream
+// connection (a broken/malformed provider stream) rather than our own
+// internal encoding, so the pump surfaces it as modelStreamErrorException
+// instead of the default internalServerException.
+type streamFaultError struct{ err error }
+
+// newStreamFault wraps err as a streamFaultError, or returns nil for a nil err.
+func newStreamFault(err error) error {
+	if err == nil {
+		return nil
+	}
+	return &streamFaultError{err: err}
+}
+
+func (f *streamFaultError) Error() string { return f.err.Error() }
+func (f *streamFaultError) Unwrap() error { return f.err }
+
+// exceptionMessage is the payload shape for an in-band exception frame,
+// matching the "message" field every generated Bedrock exception struct
+// (InternalServerException, ModelStreamErrorException, ...) carries.
+type exceptionMessage struct {
+	Message string `json:"message"`
+}
+
+// exceptionPayload marshals err as an exception frame payload. Marshaling a
+// single-string struct cannot fail; the fallback exists only to satisfy the
+// type checker.
+func exceptionPayload(err error) []byte {
+	body, marshalErr := json.Marshal(exceptionMessage{Message: err.Error()})
+	if marshalErr != nil {
+		return []byte(`{"message":"internal error"}`)
+	}
+	return body
+}
+
+// flushSupported reports whether w (or anything it wraps via Unwrap, per the
+// otelsetup statusRecorder fix) implements http.Flusher, without invoking
+// Flush — a pure capability probe, so a non-Flusher writer can be rejected
+// pre-header via the normal awserrors path rather than after WriteHeader(200)
+// has already committed the response to 200.
+func flushSupported(w http.ResponseWriter) bool {
+	for {
+		if _, ok := w.(http.Flusher); ok {
+			return true
+		}
+		u, ok := w.(interface{ Unwrap() http.ResponseWriter })
+		if !ok {
+			return false
+		}
+		w = u.Unwrap()
+	}
+}
+
+// frameWriter writes AWS event-stream frames to an HTTP response, flushing
+// after each one so a streaming SDK consumer sees bytes incrementally rather
+// than buffered until the handler returns. One frameWriter serves both
+// ConverseStream and InvokeModelWithResponseStream.
+type frameWriter struct {
+	w    http.ResponseWriter
+	enc  *esv2.Encoder
+	ctrl *http.ResponseController
+}
+
+// newFrameWriter probes w for flush support (pre-header safe: flushSupported
+// has no side effects on failure), then sets the event-stream content type
+// and writes the 200 header. Once this returns successfully, no further
+// failure may change the HTTP status — only in-band exception frames.
+func newFrameWriter(w http.ResponseWriter) (*frameWriter, error) {
+	if !flushSupported(w) {
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+	w.Header().Set("Content-Type", eventStreamContentType)
+	w.WriteHeader(http.StatusOK)
+	return &frameWriter{
+		w:    w,
+		enc:  esv2.NewEncoder(),
+		ctrl: http.NewResponseController(w),
+	}, nil
+}
+
+// writeMessage encodes headers+payload as one event-stream message and
+// flushes it to the client.
+func (fw *frameWriter) writeMessage(headers esv2.Headers, payload []byte) error {
+	headers.Set(":content-type", esv2.StringValue("application/json"))
+	if err := fw.enc.Encode(fw.w, esv2.Message{Headers: headers, Payload: payload}); err != nil {
+		return err
+	}
+	return fw.ctrl.Flush()
+}
+
+// writeEvent frames payload as a normal ":message-type: event" frame of
+// eventType (e.g. "messageStart", "chunk").
+func (fw *frameWriter) writeEvent(eventType string, payload []byte) error {
+	var headers esv2.Headers
+	headers.Set(":message-type", esv2.StringValue("event"))
+	headers.Set(":event-type", esv2.StringValue(eventType))
+	return fw.writeMessage(headers, payload)
+}
+
+// writeException frames payload as an in-band ":message-type: exception"
+// frame of excType. Used only after the 200 header has already been sent —
+// the post-header error contract forbids changing the HTTP status.
+func (fw *frameWriter) writeException(excType string, payload []byte) error {
+	var headers esv2.Headers
+	headers.Set(":message-type", esv2.StringValue("exception"))
+	headers.Set(":exception-type", esv2.StringValue(excType))
+	return fw.writeMessage(headers, payload)
+}
+
+// writeChunk frames a provider-native chunk as the
+// InvokeModelWithResponseStream "chunk" event, base64-encoding nativeChunk
+// into Bedrock's PayloadPart{Bytes} shape via the SDK's own restjson
+// marshaler (jsonutil.BuildJSON), matching WriteJSONResponse's convention.
+func (fw *frameWriter) writeChunk(nativeChunk []byte) error {
+	payload, err := jsonutil.BuildJSON(&bedrockruntime.PayloadPart{Bytes: nativeChunk})
+	if err != nil {
+		return err
+	}
+	return fw.writeEvent("chunk", payload)
+}

spinifex/gateway/bedrock/eventstream_test.go

@@ -0,0 +1,146 @@
+package gateway_bedrock
+
+import (
+	"bytes"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	esv2 "github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// decodeFrame decodes one event-stream message off raw using the
+// aws-sdk-go-v2 eventstream package's own Decoder, proving our encoded bytes
+// round-trip through prelude/headers/CRC framing exactly as a real consumer
+// would decode them.
+func decodeFrame(t *testing.T, raw []byte) esv2.Message {
+	t.Helper()
+	dec := esv2.NewDecoder()
+	msg, err := dec.Decode(bytes.NewReader(raw), nil)
+	require.NoError(t, err)
+	return msg
+}
+
+func headerString(t *testing.T, msg esv2.Message, name string) string {
+	t.Helper()
+	v := msg.Headers.Get(name)
+	require.NotNilf(t, v, "header %q not present", name)
+	return v.String()
+}
+
+func TestFrameWriter_WriteEvent_RoundTripsThroughDecoder(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	require.NoError(t, fw.writeEvent("messageStart", []byte(`{"role":"assistant"}`)))
+
+	assert.Equal(t, http.StatusOK, rec.Code)
+	assert.Equal(t, eventStreamContentType, rec.Header().Get("Content-Type"))
+
+	msg := decodeFrame(t, rec.Body.Bytes())
+	assert.Equal(t, "event", headerString(t, msg, ":message-type"))
+	assert.Equal(t, "messageStart", headerString(t, msg, ":event-type"))
+	assert.Equal(t, "application/json", headerString(t, msg, ":content-type"))
+	assert.JSONEq(t, `{"role":"assistant"}`, string(msg.Payload))
+}
+
+func TestFrameWriter_WriteException_RoundTripsThroughDecoder(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	require.NoError(t, fw.writeException(excModelStreamErrorException, exceptionPayload(errors.New("upstream broke"))))
+
+	msg := decodeFrame(t, rec.Body.Bytes())
+	assert.Equal(t, "exception", headerString(t, msg, ":message-type"))
+	assert.Equal(t, excModelStreamErrorException, headerString(t, msg, ":exception-type"))
+	assert.JSONEq(t, `{"message":"upstream broke"}`, string(msg.Payload))
+}
+
+func TestFrameWriter_WriteChunk_Base64EncodesPayload(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	require.NoError(t, fw.writeChunk([]byte(`{"type":"content_block_delta"}`)))
+
+	msg := decodeFrame(t, rec.Body.Bytes())
+	assert.Equal(t, "chunk", headerString(t, msg, ":event-type"))
+	assert.JSONEq(t, `{"bytes":"eyJ0eXBlIjoiY29udGVudF9ibG9ja19kZWx0YSJ9"}`, string(msg.Payload))
+}
+
+func TestFrameWriter_MultipleFrames_EachDecodesIndependently(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	require.NoError(t, fw.writeEvent("messageStart", []byte(`{"role":"assistant"}`)))
+	require.NoError(t, fw.writeEvent("contentBlockDelta", []byte(`{"contentBlockIndex":0}`)))
+
+	body := rec.Body.Bytes()
+	dec := esv2.NewDecoder()
+	r := bytes.NewReader(body)
+
+	msg1, err := dec.Decode(r, nil)
+	require.NoError(t, err)
+	assert.Equal(t, "messageStart", headerString(t, msg1, ":event-type"))
+
+	msg2, err := dec.Decode(r, nil)
+	require.NoError(t, err)
+	assert.Equal(t, "contentBlockDelta", headerString(t, msg2, ":event-type"))
+
+	assert.Equal(t, 0, r.Len(), "decoder should have consumed the entire body across both frames")
+}
+
+// nonFlusherWriter implements only the bare http.ResponseWriter methods: no
+// Flush, no Unwrap. It stands in for a hand-rolled ResponseWriter that
+// cannot support streaming, proving newFrameWriter rejects it pre-header
+// (Header/WriteHeader/Write are never called) rather than writing a 200 and
+// then failing.
+type nonFlusherWriter struct {
+	header      http.Header
+	wroteHeader bool
+}
+
+func (w *nonFlusherWriter) Header() http.Header {
+	if w.header == nil {
+		w.header = http.Header{}
+	}
+	return w.header
+}
+
+func (w *nonFlusherWriter) Write(b []byte) (int, error) { return len(b), nil }
+
+func (w *nonFlusherWriter) WriteHeader(int) { w.wroteHeader = true }
+
+func TestNewFrameWriter_NonFlusherWriter_CleanPreHeaderError(t *testing.T) {
+	w := &nonFlusherWriter{}
+	fw, err := newFrameWriter(w)
+	require.Error(t, err)
+	assert.Nil(t, fw)
+	assert.Equal(t, awserrors.ErrorInternalError, err.Error())
+	assert.False(t, w.wroteHeader, "newFrameWriter must not write a header when the writer can't support streaming")
+}
+
+func TestFlushSupported_WalksUnwrapChain(t *testing.T) {
+	rec := httptest.NewRecorder()
+	wrapped := &unwrappingWriter{inner: rec}
+	assert.True(t, flushSupported(wrapped))
+}
+
+// unwrappingWriter has no Flush of its own but exposes Unwrap, mirroring
+// otelsetup's fixed statusRecorder: flushSupported must walk through it to
+// find rec's Flush.
+type unwrappingWriter struct {
+	inner http.ResponseWriter
+}
+
+func (w *unwrappingWriter) Header() http.Header         { return w.inner.Header() }
+func (w *unwrappingWriter) Write(b []byte) (int, error) { return w.inner.Write(b) }
+func (w *unwrappingWriter) WriteHeader(code int)        { w.inner.WriteHeader(code) }
+func (w *unwrappingWriter) Unwrap() http.ResponseWriter { return w.inner }

spinifex/gateway/bedrock/invoke.go

@@ -78,3 +78,71 @@ func (rt *InvokeRouter) InvokeModel(ctx context.Context, accountID, modelID stri
 func InvokeModel(ctx context.Context, accountID, modelID string, body []byte, resolver CredentialResolver, endpointResolver EndpointResolver) ([]byte, string, error) {
 	return NewInvokeRouter(resolver, endpointResolver).InvokeModel(ctx, accountID, modelID, body)
 }
+
+// InvokeStreamAdapter is the optional streaming capability an InvokeAdapter
+// may implement. Both shipped families (Llama/vLLM, Anthropic) do;
+// InvokeStreamRouter type-asserts the resolved InvokeAdapter to it rather
+// than widening InvokeAdapter itself, mirroring ConverseStreamProvider.
+type InvokeStreamAdapter interface {
+	InvokeModelWithResponseStream(ctx context.Context, modelID string, body []byte) (invokeStreamSource, error)
+}
+
+// InvokeStreamRouter resolves a modelId to its catalog entry and dispatches
+// to the matching InvokeStreamAdapter, resolving self-host endpoints and
+// provider credentials as needed.
+type InvokeStreamRouter struct {
+	resolver         CredentialResolver
+	endpointResolver EndpointResolver
+}
+
+// NewInvokeStreamRouter constructs an InvokeStreamRouter. A nil resolver or
+// endpointResolver falls back to a resolver/resolver that finds nothing, so
+// an InvokeStreamRouter is always safe to use even before the real stores
+// are wired in.
+func NewInvokeStreamRouter(resolver CredentialResolver, endpointResolver EndpointResolver) *InvokeStreamRouter {
+	if resolver == nil {
+		resolver = NoopCredentialResolver
+	}
+	if endpointResolver == nil {
+		endpointResolver = NewStaticEndpointResolver(nil)
+	}
+	return &InvokeStreamRouter{resolver: resolver, endpointResolver: endpointResolver}
+}
+
+// InvokeModelWithResponseStream routes modelID to its family adapter via the
+// catalog, exactly like InvokeRouter.InvokeModel, then requires the resolved
+// adapter to also implement InvokeStreamAdapter.
+func (rt *InvokeStreamRouter) InvokeModelWithResponseStream(ctx context.Context, accountID, modelID string, body []byte) (invokeStreamSource, error) {
+	entry, ok := lookupCatalogEntry(modelID)
+	if !ok {
+		return nil, errors.New(awserrors.ErrorResourceNotFoundException)
+	}
+
+	var a InvokeAdapter
+	switch {
+	case entry.Provider == tierSelfHost:
+		a = newLlamaInvokeAdapter(rt.endpointResolver)
+	case strings.HasPrefix(entry.Provider, providerPrefix):
+		switch strings.TrimPrefix(entry.Provider, providerPrefix) {
+		case vendorAnthropic:
+			key, ok, err := rt.resolver.Resolve(ctx, accountID, vendorAnthropic)
+			if err != nil {
+				return nil, err
+			}
+			if !ok {
+				return nil, errors.New(awserrors.ErrorAccessDeniedException)
+			}
+			a = newAnthropicInvokeAdapter(key)
+		default:
+			return nil, errors.New(awserrors.ErrorResourceNotFoundException)
+		}
+	default:
+		return nil, errors.New(awserrors.ErrorResourceNotFoundException)
+	}
+
+	sa, ok := a.(InvokeStreamAdapter)
+	if !ok {
+		return nil, errors.New(awserrors.ErrorValidationException)
+	}
+	return sa.InvokeModelWithResponseStream(ctx, modelID, body)
+}

spinifex/gateway/bedrock/invoke_anthropic.go

@@ -40,6 +40,12 @@ func (b *boundAnthropicInvokeAdapter) InvokeModel(ctx context.Context, modelID s
 	return b.inner.InvokeModel(ctx, modelID, body, b.apiKey)
 }
 
+var _ InvokeStreamAdapter = (*boundAnthropicInvokeAdapter)(nil)
+
+func (b *boundAnthropicInvokeAdapter) InvokeModelWithResponseStream(ctx context.Context, modelID string, body []byte) (invokeStreamSource, error) {
+	return b.inner.InvokeModelWithResponseStream(ctx, modelID, body, b.apiKey)
+}
+
 // newAnthropicInvokeAdapter returns an InvokeAdapter that forwards to the
 // Anthropic Messages API with apiKey.
 func newAnthropicInvokeAdapter(apiKey string) InvokeAdapter {
@@ -100,3 +106,88 @@ func (a *anthropicInvokeAdapter) InvokeModel(ctx context.Context, modelID string
 
 	return respBody, "application/json", nil
 }
+
+// InvokeModelWithResponseStream rewrites the request body exactly like
+// InvokeModel (drops anthropic_version, injects model), sets stream:true,
+// and returns a source that forwards each Anthropic SSE "data:" line
+// verbatim as the chunk payload — Bedrock's Claude invoke-stream bytes *are*
+// the Anthropic event JSON, so no shape translation is needed.
+func (a *anthropicInvokeAdapter) InvokeModelWithResponseStream(ctx context.Context, modelID string, body []byte, apiKey string) (invokeStreamSource, error) {
+	var fields map[string]json.RawMessage
+	if err := json.Unmarshal(body, &fields); err != nil {
+		slog.Error("anthropic invoke-stream: failed to parse request body", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorValidationException)
+	}
+	delete(fields, "anthropic_version")
+
+	modelJSON, err := json.Marshal(anthropicModelID(modelID))
+	if err != nil {
+		slog.Error("anthropic invoke-stream: failed to marshal model id", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+	fields["model"] = modelJSON
+	fields["stream"] = json.RawMessage("true")
+
+	reqBody, err := json.Marshal(fields)
+	if err != nil {
+		slog.Error("anthropic invoke-stream: failed to marshal request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+
+	httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, a.baseURL+anthropicMessagesPath, bytes.NewReader(reqBody)) //nolint:gosec // G704: a.baseURL is the hardcoded Anthropic API endpoint (or an httptest stub in tests), never user input
+	if err != nil {
+		slog.Error("anthropic invoke-stream: failed to build request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+	httpReq.Header.Set("X-Api-Key", apiKey)
+	httpReq.Header.Set("Anthropic-Version", anthropicAPIVersion)
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.Header.Set("Accept", "text/event-stream")
+
+	resp, err := a.httpClient.Do(httpReq) //nolint:gosec // G704: httpReq targets a.baseURL, not user input
+	if err != nil {
+		slog.Error("anthropic invoke-stream: request failed", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorServiceUnavailableException)
+	}
+
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		respBody, _ := io.ReadAll(resp.Body)
+		_ = resp.Body.Close()
+		slog.Error("anthropic invoke-stream: upstream error", "model", modelID, "status", resp.StatusCode, "body", string(respBody))
+		return nil, errors.New(mapUpstreamStatus(resp.StatusCode))
+	}
+
+	return &anthropicInvokeStreamSource{resp: resp, scanner: newSSEScanner(resp.Body)}, nil
+}
+
+// anthropicInvokeStreamSource forwards each Anthropic SSE "data:" line
+// verbatim as the chunk payload, skipping keepalive "ping" events.
+type anthropicInvokeStreamSource struct {
+	resp    *http.Response
+	scanner *sseScanner
+}
+
+var _ invokeStreamSource = (*anthropicInvokeStreamSource)(nil)
+
+func (s *anthropicInvokeStreamSource) Close() error {
+	return s.resp.Body.Close()
+}
+
+func (s *anthropicInvokeStreamSource) Next(_ context.Context) ([]byte, bool, error) {
+	for {
+		ev, ok, err := s.scanner.Next()
+		if err != nil {
+			if errors.Is(err, io.EOF) {
+				return nil, false, nil
+			}
+			return nil, false, newStreamFault(err)
+		}
+		if !ok {
+			continue
+		}
+		if ev.Event == "ping" || ev.Data == "" {
+			continue
+		}
+		return []byte(ev.Data), true, nil
+	}
+}

spinifex/gateway/bedrock/invoke_anthropic_test.go

@@ -22,7 +22,10 @@ func TestAnthropicInvokeAdapter_InvokeModel_RewritesRequestAndReturnsBodyVerbati
 		capturedAPIKey = r.Header.Get("X-Api-Key")
 		capturedAnthropicVersion = r.Header.Get("Anthropic-Version")
 		capturedContentType = r.Header.Get("Content-Type")
-		require.NoError(t, json.NewDecoder(r.Body).Decode(&captured))
+		if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&captured)) {
+			http.Error(w, "decode request body", http.StatusBadRequest)
+			return
+		}
 		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte(canned))
@@ -74,6 +77,90 @@ func TestAnthropicInvokeAdapter_InvokeModel_401ReturnsAccessDenied(t *testing.T)
 // hardcodes the real Anthropic base URL, so — same package as production code
 // — the test reaches into the unexported inner client to redirect it at an
 // httptest stub instead of the network.
+func TestAnthropicInvokeAdapter_InvokeModelWithResponseStream_ForwardsSSEVerbatim(t *testing.T) {
+	const raw = `event: message_start
+data: {"type":"message_start","message":{"id":"msg_1"}}
+
+event: ping
+data: {"type":"ping"}
+
+event: content_block_delta
+data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"hi"}}
+
+event: message_stop
+data: {"type":"message_stop"}
+
+`
+	var captured map[string]json.RawMessage
+
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		assert.Equal(t, "text/event-stream", r.Header.Get("Accept"))
+		assert.NoError(t, json.NewDecoder(r.Body).Decode(&captured))
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(raw))
+	}))
+	defer ts.Close()
+
+	a := &anthropicInvokeAdapter{httpClient: ts.Client(), baseURL: ts.URL}
+	reqBody := []byte(`{"anthropic_version":"bedrock-2023-05-31","max_tokens":256,"messages":[{"role":"user","content":[{"type":"text","text":"hi"}]}]}`)
+
+	src, err := a.InvokeModelWithResponseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", reqBody, "sk-test-key")
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	var streamField bool
+	require.NoError(t, json.Unmarshal(captured["stream"], &streamField))
+	assert.True(t, streamField)
+	_, hasVersion := captured["anthropic_version"]
+	assert.False(t, hasVersion)
+
+	chunks := drainInvokeStream(t, src)
+	require.Len(t, chunks, 3, "ping is skipped; message_start, content_block_delta, message_stop forward verbatim")
+	assert.JSONEq(t, `{"type":"message_start","message":{"id":"msg_1"}}`, string(chunks[0]))
+	assert.JSONEq(t, `{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"hi"}}`, string(chunks[1]))
+	assert.JSONEq(t, `{"type":"message_stop"}`, string(chunks[2]))
+}
+
+func TestAnthropicInvokeAdapter_InvokeModelWithResponseStream_401ReturnsAccessDenied(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusUnauthorized)
+		_, _ = w.Write([]byte(`{"error": {"message": "invalid x-api-key"}}`))
+	}))
+	defer ts.Close()
+
+	a := &anthropicInvokeAdapter{httpClient: ts.Client(), baseURL: ts.URL}
+	_, err := a.InvokeModelWithResponseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", []byte(`{"anthropic_version":"bedrock-2023-05-31","messages":[]}`), "bad-key")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorAccessDeniedException, err.Error())
+}
+
+func TestBoundAnthropicInvokeAdapter_InvokeModelWithResponseStream(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		assert.Equal(t, "sk-bound-test", r.Header.Get("X-Api-Key"))
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("data: {\"type\":\"message_stop\"}\n\n"))
+	}))
+	defer ts.Close()
+
+	adapter := newAnthropicInvokeAdapter("sk-bound-test")
+	bound, ok := adapter.(*boundAnthropicInvokeAdapter)
+	require.True(t, ok)
+	bound.inner.httpClient = ts.Client()
+	bound.inner.baseURL = ts.URL
+
+	sa, ok := adapter.(InvokeStreamAdapter)
+	require.True(t, ok)
+
+	src, err := sa.InvokeModelWithResponseStream(context.Background(), "anthropic.claude-3-5-sonnet-20240620-v1:0", []byte(`{"anthropic_version":"bedrock-2023-05-31","messages":[]}`))
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	chunks := drainInvokeStream(t, src)
+	require.Len(t, chunks, 1)
+	assert.JSONEq(t, `{"type":"message_stop"}`, string(chunks[0]))
+}
+
 func TestBoundAnthropicInvokeAdapter_InvokeModel(t *testing.T) {
 	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		assert.Equal(t, "sk-bound-test", r.Header.Get("X-Api-Key"))

spinifex/gateway/bedrock/invoke_llama.go

@@ -5,9 +5,12 @@ import (
 	"context"
 	"encoding/json"
 	"errors"
+	"fmt"
 	"io"
 	"log/slog"
 	"net/http"
+	"strings"
+	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 )
@@ -32,11 +35,50 @@ type llamaInvokeResponse struct {
 
 // llamaCompletionsRequest is the OpenAI /v1/completions request vLLM serves.
 type llamaCompletionsRequest struct {
-	Model       string   `json:"model"`
-	Prompt      string   `json:"prompt"`
-	MaxTokens   int      `json:"max_tokens,omitempty"`
-	Temperature *float64 `json:"temperature,omitempty"`
-	TopP        *float64 `json:"top_p,omitempty"`
+	Model         string             `json:"model"`
+	Prompt        string             `json:"prompt"`
+	MaxTokens     int                `json:"max_tokens,omitempty"`
+	Temperature   *float64           `json:"temperature,omitempty"`
+	TopP          *float64           `json:"top_p,omitempty"`
+	Stream        bool               `json:"stream,omitempty"`
+	StreamOptions *vllmStreamOptions `json:"stream_options,omitempty"`
+}
+
+// llamaCompletionsStreamChunk is one OpenAI /v1/completions streaming SSE
+// "data:" chunk.
+type llamaCompletionsStreamChunk struct {
+	Choices []llamaCompletionsStreamChoice `json:"choices"`
+	Usage   *llamaCompletionsUsage         `json:"usage"`
+}
+
+type llamaCompletionsStreamChoice struct {
+	Text         string  `json:"text"`
+	FinishReason *string `json:"finish_reason"`
+}
+
+// llamaInvokeStreamChunk is a non-final Bedrock-native Llama
+// invoke-with-response-stream chunk.
+type llamaInvokeStreamChunk struct {
+	Generation string `json:"generation"`
+}
+
+// llamaInvocationMetrics is Bedrock's per-invocation latency/token summary,
+// carried on the final Llama invoke-with-response-stream chunk.
+type llamaInvocationMetrics struct {
+	InputTokenCount   int   `json:"inputTokenCount"`
+	OutputTokenCount  int   `json:"outputTokenCount"`
+	InvocationLatency int64 `json:"invocationLatency"`
+	FirstByteLatency  int64 `json:"firstByteLatency"`
+}
+
+// llamaInvokeStreamFinalChunk is the final Bedrock-native Llama
+// invoke-with-response-stream chunk, carrying stop/usage metadata.
+type llamaInvokeStreamFinalChunk struct {
+	Generation           string                 `json:"generation"`
+	PromptTokenCount     int                    `json:"prompt_token_count"`
+	GenerationTokenCount int                    `json:"generation_token_count"`
+	StopReason           string                 `json:"stop_reason"`
+	InvocationMetrics    llamaInvocationMetrics `json:"amazon-bedrock-invocationMetrics"`
 }
 
 type llamaCompletionsChoice struct {
@@ -164,3 +206,172 @@ func (a *llamaInvokeAdapter) InvokeModel(ctx context.Context, modelID string, bo
 
 	return out, "application/json", nil
 }
+
+var _ InvokeStreamAdapter = (*llamaInvokeAdapter)(nil)
+
+// InvokeModelWithResponseStream resolves modelID's endpoint, opens a
+// streaming OpenAI /v1/completions request (stream:true,
+// stream_options.include_usage for the trailing usage chunk), and returns a
+// source that translates each chunk to the Bedrock-native Llama streaming
+// shape.
+func (a *llamaInvokeAdapter) InvokeModelWithResponseStream(ctx context.Context, modelID string, body []byte) (invokeStreamSource, error) {
+	baseURL, ok, err := a.endpointResolver.Endpoint(ctx, modelID)
+	if err != nil {
+		slog.Error("llama invoke-stream: endpoint resolution failed", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorServiceUnavailableException)
+	}
+	if !ok {
+		return nil, errors.New(awserrors.ErrorModelNotReadyException)
+	}
+
+	var lr llamaInvokeRequest
+	if err := json.Unmarshal(body, &lr); err != nil {
+		slog.Error("llama invoke-stream: failed to parse request body", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorValidationException)
+	}
+
+	reqBody, err := json.Marshal(llamaCompletionsRequest{
+		Model:         modelID,
+		Prompt:        lr.Prompt,
+		MaxTokens:     lr.MaxGenLen,
+		Temperature:   lr.Temperature,
+		TopP:          lr.TopP,
+		Stream:        true,
+		StreamOptions: &vllmStreamOptions{IncludeUsage: true},
+	})
+	if err != nil {
+		slog.Error("llama invoke-stream: failed to marshal request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+
+	httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+llamaCompletionsPath, bytes.NewReader(reqBody)) //nolint:gosec // G704: baseURL is a resolved pinned self-host endpoint, not user input
+	if err != nil {
+		slog.Error("llama invoke-stream: failed to build request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.Header.Set("Accept", "text/event-stream")
+
+	resp, err := a.httpClient.Do(httpReq) //nolint:gosec // G704: httpReq targets the resolved pinned self-host endpoint, not user input
+	if err != nil {
+		slog.Error("llama invoke-stream: request failed", "model", modelID, "endpoint", baseURL, "err", err)
+		return nil, errors.New(awserrors.ErrorServiceUnavailableException)
+	}
+
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		respBody, _ := io.ReadAll(resp.Body)
+		_ = resp.Body.Close()
+		slog.Error("llama invoke-stream: upstream error", "model", modelID, "status", resp.StatusCode, "body", string(respBody))
+		return nil, errors.New(mapUpstreamStatus(resp.StatusCode))
+	}
+
+	return &llamaInvokeStreamSource{
+		resp:    resp,
+		scanner: newSSEScanner(resp.Body),
+		start:   time.Now(),
+	}, nil
+}
+
+// llamaInvokeStreamSource translates an OpenAI /v1/completions SSE stream
+// into the Bedrock-native Llama invoke-with-response-stream chunk shape: a
+// {"generation": "..."} chunk per text delta, and a final chunk carrying
+// stop_reason, token counts, and amazon-bedrock-invocationMetrics. The final
+// chunk's token counts arrive on vLLM's trailing usage-only chunk (via
+// stream_options.include_usage), which follows the finish_reason chunk, so
+// emitting the final chunk is deferred until that usage chunk (or EOF) is
+// seen — otherwise the token counts would always be zero.
+type llamaInvokeStreamSource struct {
+	resp    *http.Response
+	scanner *sseScanner
+	start   time.Time
+
+	gotFirstByte                   bool
+	firstByte                      time.Time
+	promptTokens, completionTokens int
+	pendingFinal                   *llamaInvokeStreamFinalChunk
+}
+
+var _ invokeStreamSource = (*llamaInvokeStreamSource)(nil)
+
+func (s *llamaInvokeStreamSource) Close() error {
+	return s.resp.Body.Close()
+}
+
+func (s *llamaInvokeStreamSource) Next(_ context.Context) ([]byte, bool, error) {
+	for {
+		ev, ok, err := s.scanner.Next()
+		if err != nil {
+			if errors.Is(err, io.EOF) {
+				if s.pendingFinal != nil {
+					return s.emitFinal()
+				}
+				return nil, false, nil
+			}
+			return nil, false, newStreamFault(err)
+		}
+		if !ok {
+			continue
+		}
+		data := strings.TrimSpace(ev.Data)
+		if data == "" || data == "[DONE]" {
+			continue
+		}
+
+		var chunk llamaCompletionsStreamChunk
+		if err := json.Unmarshal([]byte(ev.Data), &chunk); err != nil {
+			return nil, false, newStreamFault(fmt.Errorf("llama invoke-stream: decode chunk: %w", err))
+		}
+
+		if !s.gotFirstByte {
+			s.gotFirstByte = true
+			s.firstByte = time.Now()
+		}
+		if chunk.Usage != nil {
+			s.promptTokens = chunk.Usage.PromptTokens
+			s.completionTokens = chunk.Usage.CompletionTokens
+		}
+		if len(chunk.Choices) == 0 {
+			// The trailing usage-only chunk: emit the deferred final chunk
+			// now that token counts are known.
+			if s.pendingFinal != nil {
+				return s.emitFinal()
+			}
+			continue
+		}
+
+		choice := chunk.Choices[0]
+		if choice.FinishReason != nil {
+			s.pendingFinal = &llamaInvokeStreamFinalChunk{
+				Generation: choice.Text,
+				StopReason: mapLlamaStopReason(*choice.FinishReason),
+			}
+			continue
+		}
+
+		out, err := json.Marshal(llamaInvokeStreamChunk{Generation: choice.Text})
+		if err != nil {
+			return nil, false, newStreamFault(fmt.Errorf("llama invoke-stream: marshal chunk: %w", err))
+		}
+		return out, true, nil
+	}
+}
+
+// emitFinal fills in token counts/metrics on the deferred final chunk and
+// marshals it.
+func (s *llamaInvokeStreamSource) emitFinal() ([]byte, bool, error) {
+	final := *s.pendingFinal
+	s.pendingFinal = nil
+	final.PromptTokenCount = s.promptTokens
+	final.GenerationTokenCount = s.completionTokens
+	final.InvocationMetrics = llamaInvocationMetrics{
+		InputTokenCount:   s.promptTokens,
+		OutputTokenCount:  s.completionTokens,
+		InvocationLatency: time.Since(s.start).Milliseconds(),
+		FirstByteLatency:  s.firstByte.Sub(s.start).Milliseconds(),
+	}
+	out, err := json.Marshal(final)
+	if err != nil {
+		return nil, false, newStreamFault(fmt.Errorf("llama invoke-stream: marshal final chunk: %w", err))
+	}
+	return out, true, nil
+}

spinifex/gateway/bedrock/invoke_llama_test.go

@@ -16,7 +16,10 @@ func TestLlamaInvokeAdapter_InvokeModel_MapsRequestAndResponse(t *testing.T) {
 	var captured llamaCompletionsRequest
 
 	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		require.NoError(t, json.NewDecoder(r.Body).Decode(&captured))
+		if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&captured)) {
+			http.Error(w, "decode request body", http.StatusBadRequest)
+			return
+		}
 		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte(`{
@@ -99,3 +102,106 @@ func TestLlamaInvokeAdapter_InvokeModel_UnresolvedEndpointReturnsModelNotReady(t
 	require.Error(t, err)
 	assert.Equal(t, awserrors.ErrorModelNotReadyException, err.Error())
 }
+
+// llamaCompletionsStreamFixture is a canned OpenAI /v1/completions streaming
+// SSE body: two text deltas, a finish_reason chunk, a trailing usage-only
+// chunk, then [DONE].
+const llamaCompletionsStreamFixture = `data: {"choices":[{"text":"Hello","finish_reason":null}]}
+
+data: {"choices":[{"text":" world","finish_reason":null}]}
+
+data: {"choices":[{"text":"","finish_reason":"stop"}]}
+
+data: {"choices":[],"usage":{"prompt_tokens":6,"completion_tokens":2}}
+
+data: [DONE]
+
+`
+
+func drainInvokeStream(t *testing.T, src invokeStreamSource) [][]byte {
+	t.Helper()
+	var chunks [][]byte
+	for {
+		chunk, ok, err := src.Next(context.Background())
+		require.NoError(t, err)
+		if !ok {
+			return chunks
+		}
+		chunks = append(chunks, chunk)
+	}
+}
+
+func TestLlamaInvokeAdapter_InvokeModelWithResponseStream_TranslatesChunks(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		var req llamaCompletionsRequest
+		assert.NoError(t, json.NewDecoder(r.Body).Decode(&req))
+		assert.True(t, req.Stream)
+		if !assert.NotNil(t, req.StreamOptions) {
+			return
+		}
+		assert.True(t, req.StreamOptions.IncludeUsage)
+
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(llamaCompletionsStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	a := newLlamaInvokeAdapter(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	a.httpClient = ts.Client()
+
+	src, err := a.InvokeModelWithResponseStream(context.Background(), modelID, []byte(`{"prompt":"hello","max_gen_len":128}`))
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	chunks := drainInvokeStream(t, src)
+	require.Len(t, chunks, 3)
+
+	var c1, c2 llamaInvokeStreamChunk
+	require.NoError(t, json.Unmarshal(chunks[0], &c1))
+	assert.Equal(t, "Hello", c1.Generation)
+	require.NoError(t, json.Unmarshal(chunks[1], &c2))
+	assert.Equal(t, " world", c2.Generation)
+
+	var final llamaInvokeStreamFinalChunk
+	require.NoError(t, json.Unmarshal(chunks[2], &final))
+	assert.Empty(t, final.Generation)
+	assert.Equal(t, "stop", final.StopReason)
+	assert.Equal(t, 6, final.PromptTokenCount)
+	assert.Equal(t, 2, final.GenerationTokenCount)
+	assert.Equal(t, 6, final.InvocationMetrics.InputTokenCount)
+	assert.Equal(t, 2, final.InvocationMetrics.OutputTokenCount)
+	assert.GreaterOrEqual(t, final.InvocationMetrics.InvocationLatency, int64(0))
+}
+
+func TestLlamaInvokeAdapter_InvokeModelWithResponseStream_UnresolvedEndpointReturnsModelNotReady(t *testing.T) {
+	a := newLlamaInvokeAdapter(NewStaticEndpointResolver(nil))
+
+	_, err := a.InvokeModelWithResponseStream(context.Background(), "meta.llama3-70b-instruct-v1:0", []byte(`{"prompt":"hello"}`))
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorModelNotReadyException, err.Error())
+}
+
+func TestLlamaInvokeStreamSource_MidStreamDecodeErrorSurfacesAsStreamFault(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("data: {not-json\n\n"))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	a := newLlamaInvokeAdapter(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	a.httpClient = ts.Client()
+
+	src, err := a.InvokeModelWithResponseStream(context.Background(), modelID, []byte(`{"prompt":"hello"}`))
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	_, ok, err := src.Next(context.Background())
+	assert.False(t, ok)
+	require.Error(t, err)
+	var fault *streamFaultError
+	assert.ErrorAs(t, err, &fault)
+}

spinifex/gateway/bedrock/invoke_stream.go

@@ -0,0 +1,77 @@
+package gateway_bedrock
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"net/http"
+)
+
+// invokeStreamSource is the backend-agnostic InvokeModelWithResponseStream
+// reframer contract: Next returns the next provider-native chunk (verbatim
+// for Anthropic passthrough, translated to the Bedrock-native shape for
+// Llama), or (nil,false,nil) at clean EOF.
+type invokeStreamSource interface {
+	Next(ctx context.Context) (chunk []byte, ok bool, err error)
+	Close() error
+}
+
+// InvokeModelWithResponseStream is the bedrock-runtime
+// InvokeModelWithResponseStream entry point used by the gateway route table.
+// Unlike the JSON-dispatch handlers it owns w directly: a pre-stream failure
+// (unknown model, unresolved credential, upstream connect error) returns an
+// awserrors code for the normal ErrorHandler envelope. Once the first frame
+// is written it always returns nil — any later failure is an in-band
+// exception event, since the HTTP status can no longer change.
+// requestContentType is the client's declared Content-Type, logged only.
+func InvokeModelWithResponseStream(ctx context.Context, w http.ResponseWriter, accountID, modelID string, body []byte, resolver CredentialResolver, endpointResolver EndpointResolver, requestContentType string) error {
+	src, err := NewInvokeStreamRouter(resolver, endpointResolver).InvokeModelWithResponseStream(ctx, accountID, modelID, body)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		if cerr := src.Close(); cerr != nil {
+			slog.Error("invoke-with-response-stream: failed to close upstream source", "model", modelID, "err", cerr)
+		}
+	}()
+
+	fw, err := newFrameWriter(w)
+	if err != nil {
+		return err
+	}
+
+	slog.Debug("invoke-with-response-stream: streaming", "model", modelID, "request_content_type", requestContentType)
+	pumpInvokeStream(ctx, fw, src, modelID)
+	return nil
+}
+
+// pumpInvokeStream drains src and writes each provider-native chunk as a
+// "chunk" frame. A mid-stream Next error surfaces as an in-band exception
+// event and stops the pump; a write/flush error also stops the pump
+// silently, since the client connection is already broken and the HTTP
+// status can no longer change either way.
+func pumpInvokeStream(ctx context.Context, fw *frameWriter, src invokeStreamSource, modelID string) {
+	for {
+		chunk, ok, err := src.Next(ctx)
+		if err != nil {
+			excType := excInternalServerException
+			var fault *streamFaultError
+			if errors.As(err, &fault) {
+				excType = excModelStreamErrorException
+			}
+			slog.Error("invoke-with-response-stream: upstream fault", "model", modelID, "err", err)
+			if werr := fw.writeException(excType, exceptionPayload(err)); werr != nil {
+				slog.Error("invoke-with-response-stream: failed to write exception frame", "model", modelID, "err", werr)
+			}
+			return
+		}
+		if !ok {
+			return
+		}
+
+		if werr := fw.writeChunk(chunk); werr != nil {
+			slog.Error("invoke-with-response-stream: failed to write frame, aborting", "model", modelID, "err", werr)
+			return
+		}
+	}
+}

spinifex/gateway/bedrock/invoke_stream_test.go

@@ -0,0 +1,120 @@
+package gateway_bedrock
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeInvokeStreamSource drives pumpInvokeStream directly with a scripted
+// chunk/error sequence, independent of any real backend.
+type fakeInvokeStreamSource struct {
+	chunks [][]byte
+	err    error
+	closed bool
+}
+
+func (f *fakeInvokeStreamSource) Next(_ context.Context) ([]byte, bool, error) {
+	if len(f.chunks) > 0 {
+		c := f.chunks[0]
+		f.chunks = f.chunks[1:]
+		return c, true, nil
+	}
+	if f.err != nil {
+		return nil, false, f.err
+	}
+	return nil, false, nil
+}
+
+func (f *fakeInvokeStreamSource) Close() error {
+	f.closed = true
+	return nil
+}
+
+func TestPumpInvokeStream_MidStreamFaultWritesExceptionFrame(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	src := &fakeInvokeStreamSource{
+		chunks: [][]byte{[]byte(`{"generation":"hi"}`)},
+		err:    newStreamFault(errors.New("upstream connection reset")),
+	}
+	pumpInvokeStream(context.Background(), fw, src, "test-model")
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 2)
+	assert.Equal(t, "chunk", frames[0].Type)
+	assert.Equal(t, "exception", frames[1].MessageType)
+	assert.Equal(t, excModelStreamErrorException, frames[1].Type)
+}
+
+func TestPumpInvokeStream_NonFaultErrorUsesInternalServerException(t *testing.T) {
+	rec := httptest.NewRecorder()
+	fw, err := newFrameWriter(rec)
+	require.NoError(t, err)
+
+	src := &fakeInvokeStreamSource{err: errors.New("plain internal error")}
+	pumpInvokeStream(context.Background(), fw, src, "test-model")
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 1)
+	assert.Equal(t, excInternalServerException, frames[0].Type)
+}
+
+func TestInvokeModelWithResponseStream_UnknownModelReturnsResourceNotFoundPreHeader(t *testing.T) {
+	rec := httptest.NewRecorder()
+	err := InvokeModelWithResponseStream(context.Background(), rec, "000000000001", "does.not-exist-v1:0", []byte(`{}`), nil, nil, "application/json")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
+	assert.Equal(t, 0, rec.Body.Len())
+}
+
+func TestInvokeModelWithResponseStream_SelfHostHappyPath_WritesChunkFrames(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(llamaCompletionsStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	rec := httptest.NewRecorder()
+	body := []byte(`{"prompt":"hello","max_gen_len":128}`)
+
+	err := InvokeModelWithResponseStream(context.Background(), rec, "000000000001", modelID, body, nil, NewStaticEndpointResolver(map[string]string{modelID: ts.URL}), "application/json")
+	require.NoError(t, err)
+
+	assert.Equal(t, http.StatusOK, rec.Code)
+	assert.Equal(t, eventStreamContentType, rec.Header().Get("Content-Type"))
+
+	frames := decodeAllFrames(t, rec.Body.Bytes())
+	require.Len(t, frames, 3)
+	for _, f := range frames {
+		assert.Equal(t, "chunk", f.Type)
+	}
+}
+
+func TestInvokeModelWithResponseStream_NonFlusherWriter_ReturnsPreHeaderError(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(llamaCompletionsStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	w := &nonFlusherWriter{}
+	body := []byte(`{"prompt":"hello"}`)
+
+	err := InvokeModelWithResponseStream(context.Background(), w, "000000000001", modelID, body, nil, NewStaticEndpointResolver(map[string]string{modelID: ts.URL}), "application/json")
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInternalError, err.Error())
+	assert.False(t, w.wroteHeader)
+}

spinifex/gateway/bedrock/invoke_test.go

@@ -82,3 +82,43 @@ func TestInvokeModel_PackageEntryPoint_UnknownModel(t *testing.T) {
 	require.Error(t, err)
 	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
 }
+
+func TestInvokeStreamRouter_SelfHostSuccess(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(llamaCompletionsStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	rt := NewInvokeStreamRouter(nil, NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+
+	src, err := rt.InvokeModelWithResponseStream(context.Background(), "000000000001", modelID, []byte(`{"prompt":"hello"}`))
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	chunks := drainInvokeStream(t, src)
+	assert.Len(t, chunks, 3)
+}
+
+func TestInvokeStreamRouter_UnknownModelReturnsResourceNotFound(t *testing.T) {
+	rt := NewInvokeStreamRouter(nil, nil)
+	_, err := rt.InvokeModelWithResponseStream(context.Background(), "000000000001", "does.not-exist-v1:0", []byte(`{}`))
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
+}
+
+func TestInvokeStreamRouter_AnthropicNoCredentialReturnsAccessDenied(t *testing.T) {
+	rt := NewInvokeStreamRouter(stubCredentialResolver{ok: false}, nil)
+	_, err := rt.InvokeModelWithResponseStream(context.Background(), "000000000001", "anthropic.claude-3-5-sonnet-20240620-v1:0", []byte(`{}`))
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorAccessDeniedException, err.Error())
+}
+
+func TestInvokeStreamRouter_SelfHostNoEndpointReturnsModelNotReady(t *testing.T) {
+	rt := NewInvokeStreamRouter(nil, nil)
+	_, err := rt.InvokeModelWithResponseStream(context.Background(), "000000000001", "meta.llama3-70b-instruct-v1:0", []byte(`{"prompt":"hello"}`))
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorModelNotReadyException, err.Error())
+}

spinifex/gateway/bedrock/provider.go

@@ -84,3 +84,62 @@ func (rt *Router) Converse(ctx context.Context, accountID, modelID string, input
 func Converse(ctx context.Context, accountID, modelID string, input *bedrockruntime.ConverseInput, resolver CredentialResolver, endpointResolver EndpointResolver) (*bedrockruntime.ConverseOutput, error) {
 	return NewRouter(resolver, endpointResolver).Converse(ctx, accountID, modelID, input)
 }
+
+// ConverseStreamProvider is the optional streaming capability a Provider may
+// implement. Both shipped backends (vLLM, Anthropic) do; Router.ConverseStream
+// type-asserts the resolved Provider to it rather than widening Provider
+// itself, so a future non-streaming-capable backend still compiles.
+type ConverseStreamProvider interface {
+	ConverseStream(ctx context.Context, modelID string, input *bedrockruntime.ConverseStreamInput) (converseStreamSource, error)
+}
+
+// converseStreamToConverseInput adapts a ConverseStreamInput to a
+// ConverseInput carrying only the fields buildVLLMRequest/buildAnthropicRequest
+// read (Messages, System, InferenceConfig). The two generated types are
+// otherwise structurally identical field-for-field; this lets the streaming
+// path reuse the exact same request builders as Converse.
+func converseStreamToConverseInput(input *bedrockruntime.ConverseStreamInput) *bedrockruntime.ConverseInput {
+	return &bedrockruntime.ConverseInput{
+		Messages:        input.Messages,
+		System:          input.System,
+		InferenceConfig: input.InferenceConfig,
+	}
+}
+
+// ConverseStream routes modelID to its provider via the catalog, exactly like
+// Converse, then requires the resolved provider to also implement
+// ConverseStreamProvider.
+func (rt *Router) ConverseStream(ctx context.Context, accountID, modelID string, input *bedrockruntime.ConverseStreamInput) (converseStreamSource, error) {
+	entry, ok := lookupCatalogEntry(modelID)
+	if !ok {
+		return nil, errors.New(awserrors.ErrorResourceNotFoundException)
+	}
+
+	var p Provider
+	switch {
+	case entry.Provider == tierSelfHost:
+		p = newVLLMProvider(rt.endpointResolver)
+	case strings.HasPrefix(entry.Provider, providerPrefix):
+		switch strings.TrimPrefix(entry.Provider, providerPrefix) {
+		case vendorAnthropic:
+			key, ok, err := rt.resolver.Resolve(ctx, accountID, vendorAnthropic)
+			if err != nil {
+				return nil, err
+			}
+			if !ok {
+				return nil, errors.New(awserrors.ErrorAccessDeniedException)
+			}
+			p = newAnthropicProvider(key)
+		default:
+			return nil, errors.New(awserrors.ErrorResourceNotFoundException)
+		}
+	default:
+		return nil, errors.New(awserrors.ErrorResourceNotFoundException)
+	}
+
+	sp, ok := p.(ConverseStreamProvider)
+	if !ok {
+		return nil, errors.New(awserrors.ErrorValidationException)
+	}
+	return sp.ConverseStream(ctx, modelID, input)
+}

spinifex/gateway/bedrock/router_test.go

@@ -92,6 +92,54 @@ func TestConverse_PackageEntryPoint_UnknownModel(t *testing.T) {
 	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
 }
 
+func converseStreamInput() *bedrockruntime.ConverseStreamInput {
+	return &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	}
+}
+
+func TestRouter_ConverseStream_SelfHostSuccess(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(vllmStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	rt := NewRouter(nil, NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+
+	src, err := rt.ConverseStream(context.Background(), "000000000001", modelID, converseStreamInput())
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	events := drainConverseStream(t, src)
+	assert.Equal(t, converseStreamEventMessageStart, events[0].Kind)
+}
+
+func TestRouter_ConverseStream_UnknownModelReturnsResourceNotFound(t *testing.T) {
+	rt := NewRouter(nil, nil)
+	_, err := rt.ConverseStream(context.Background(), "000000000001", "does.not-exist-v1:0", converseStreamInput())
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
+}
+
+func TestRouter_ConverseStream_AnthropicNoCredentialReturnsAccessDenied(t *testing.T) {
+	rt := NewRouter(stubCredentialResolver{ok: false}, nil)
+	_, err := rt.ConverseStream(context.Background(), "000000000001", "anthropic.claude-3-5-sonnet-20240620-v1:0", converseStreamInput())
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorAccessDeniedException, err.Error())
+}
+
+func TestRouter_ConverseStream_SelfHostNoEndpointReturnsModelNotReady(t *testing.T) {
+	rt := NewRouter(nil, nil)
+	_, err := rt.ConverseStream(context.Background(), "000000000001", "meta.llama3-70b-instruct-v1:0", converseStreamInput())
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorModelNotReadyException, err.Error())
+}
+
 func TestNewRouter_NilArgumentsFallBackToNoops(t *testing.T) {
 	rt := NewRouter(nil, nil)
 	require.NotNil(t, rt)

spinifex/gateway/bedrock/sse.go

@@ -0,0 +1,71 @@
+package gateway_bedrock
+
+import (
+	"bufio"
+	"io"
+	"strings"
+)
+
+// sseMaxLineBuf is the maximum single-line size the scanner accepts. Data
+// lines carry one JSON chunk each; 1MiB comfortably covers any single
+// streamed delta.
+const sseMaxLineBuf = 1024 * 1024
+
+// sseEvent is one parsed text/event-stream record: the optional "event:"
+// type (empty when absent, per the SSE spec's implicit "message" default)
+// and its "data:" lines joined with "\n".
+type sseEvent struct {
+	Event string
+	Data  string
+}
+
+// sseScanner reads text/event-stream framing shared by vLLM/OpenAI (bare
+// "data:" lines) and Anthropic ("event:"+"data:" pairs): blank-line
+// terminated records, skipping comment lines (leading ':') and blank
+// keepalive lines between records.
+type sseScanner struct {
+	sc *bufio.Scanner
+}
+
+// newSSEScanner wraps r for SSE line scanning.
+func newSSEScanner(r io.Reader) *sseScanner {
+	sc := bufio.NewScanner(r)
+	sc.Buffer(make([]byte, 0, 64*1024), sseMaxLineBuf)
+	return &sseScanner{sc: sc}
+}
+
+// Next returns the next SSE record, or (zero, false, err) at EOF/read error.
+// err is io.EOF at a clean stream end.
+func (s *sseScanner) Next() (sseEvent, bool, error) {
+	var event string
+	var dataLines []string
+	sawAny := false
+
+	for s.sc.Scan() {
+		line := s.sc.Text()
+		if line == "" {
+			if sawAny {
+				return sseEvent{Event: event, Data: strings.Join(dataLines, "\n")}, true, nil
+			}
+			continue // keepalive blank line between records
+		}
+		switch {
+		case strings.HasPrefix(line, ":"):
+			// comment/keepalive, ignore — does not start a record
+		case strings.HasPrefix(line, "event:"):
+			sawAny = true
+			event = strings.TrimSpace(strings.TrimPrefix(line, "event:"))
+		case strings.HasPrefix(line, "data:"):
+			sawAny = true
+			dataLines = append(dataLines, strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " "))
+		}
+	}
+
+	if err := s.sc.Err(); err != nil {
+		return sseEvent{}, false, err
+	}
+	if sawAny {
+		return sseEvent{Event: event, Data: strings.Join(dataLines, "\n")}, true, nil
+	}
+	return sseEvent{}, false, io.EOF
+}

spinifex/gateway/bedrock/sse_test.go

@@ -0,0 +1,104 @@
+package gateway_bedrock
+
+import (
+	"io"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestSSEScanner_VLLMStyleBareDataLines(t *testing.T) {
+	raw := "data: {\"a\":1}\n\ndata: {\"a\":2}\n\ndata: [DONE]\n\n"
+	sc := newSSEScanner(strings.NewReader(raw))
+
+	ev1, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Empty(t, ev1.Event)
+	assert.Equal(t, `{"a":1}`, ev1.Data)
+
+	ev2, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, `{"a":2}`, ev2.Data)
+
+	ev3, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, "[DONE]", ev3.Data)
+
+	_, ok, err = sc.Next()
+	assert.False(t, ok)
+	assert.ErrorIs(t, err, io.EOF)
+}
+
+func TestSSEScanner_AnthropicStyleEventAndData(t *testing.T) {
+	raw := "event: message_start\ndata: {\"type\":\"message_start\"}\n\nevent: ping\ndata: {}\n\n"
+	sc := newSSEScanner(strings.NewReader(raw))
+
+	ev1, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, "message_start", ev1.Event)
+	assert.Equal(t, `{"type":"message_start"}`, ev1.Data)
+
+	ev2, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, "ping", ev2.Event)
+}
+
+func TestSSEScanner_MultiLineData(t *testing.T) {
+	raw := "data: line one\ndata: line two\n\n"
+	sc := newSSEScanner(strings.NewReader(raw))
+
+	ev, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, "line one\nline two", ev.Data)
+}
+
+func TestSSEScanner_SkipsCommentsAndKeepaliveBlankLines(t *testing.T) {
+	raw := ":keepalive\n\n:another comment\ndata: {\"a\":1}\n\n"
+	sc := newSSEScanner(strings.NewReader(raw))
+
+	ev, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, `{"a":1}`, ev.Data)
+}
+
+func TestSSEScanner_TrailingRecordWithoutFinalBlankLine(t *testing.T) {
+	raw := "data: {\"a\":1}\n" // no trailing blank line before EOF
+	sc := newSSEScanner(strings.NewReader(raw))
+
+	ev, ok, err := sc.Next()
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, `{"a":1}`, ev.Data)
+
+	_, ok, err = sc.Next()
+	assert.False(t, ok)
+	assert.ErrorIs(t, err, io.EOF)
+}
+
+func TestSSEScanner_EmptyStreamIsCleanEOF(t *testing.T) {
+	sc := newSSEScanner(strings.NewReader(""))
+	_, ok, err := sc.Next()
+	assert.False(t, ok)
+	assert.ErrorIs(t, err, io.EOF)
+}
+
+type errReader struct{ err error }
+
+func (r errReader) Read(_ []byte) (int, error) { return 0, r.err }
+
+func TestSSEScanner_ReadErrorPropagates(t *testing.T) {
+	boom := assert.AnError
+	sc := newSSEScanner(errReader{err: boom})
+	_, ok, err := sc.Next()
+	assert.False(t, ok)
+	assert.ErrorIs(t, err, boom)
+}

spinifex/gateway/bedrock/vllm.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"encoding/json"
 	"errors"
+	"fmt"
 	"io"
 	"log/slog"
 	"net/http"
@@ -47,12 +48,31 @@ type vllmMessage struct {
 }
 
 type vllmChatRequest struct {
-	Model       string        `json:"model"`
-	Messages    []vllmMessage `json:"messages"`
-	MaxTokens   *int64        `json:"max_tokens,omitempty"`
-	Temperature *float64      `json:"temperature,omitempty"`
-	TopP        *float64      `json:"top_p,omitempty"`
-	Stop        []string      `json:"stop,omitempty"`
+	Model         string             `json:"model"`
+	Messages      []vllmMessage      `json:"messages"`
+	MaxTokens     *int64             `json:"max_tokens,omitempty"`
+	Temperature   *float64           `json:"temperature,omitempty"`
+	TopP          *float64           `json:"top_p,omitempty"`
+	Stop          []string           `json:"stop,omitempty"`
+	Stream        bool               `json:"stream,omitempty"`
+	StreamOptions *vllmStreamOptions `json:"stream_options,omitempty"`
+}
+
+// vllmStreamOptions requests the trailing usage-only chunk vLLM/OpenAI
+// otherwise omits from a streaming response.
+type vllmStreamOptions struct {
+	IncludeUsage bool `json:"include_usage"`
+}
+
+// vllmStreamChunk is one OpenAI chat-completions streaming SSE "data:" chunk.
+type vllmStreamChunk struct {
+	Choices []vllmStreamChoice `json:"choices"`
+	Usage   *vllmUsage         `json:"usage"`
+}
+
+type vllmStreamChoice struct {
+	Delta        vllmMessage `json:"delta"`
+	FinishReason *string     `json:"finish_reason"`
 }
 
 type vllmChoice struct {
@@ -222,3 +242,212 @@ func vllmToConverseOutput(cr vllmChatResponse, latency time.Duration) (*bedrockr
 		Metrics: &bedrockruntime.ConverseMetrics{LatencyMs: aws.Int64(latency.Milliseconds())},
 	}, nil
 }
+
+var _ ConverseStreamProvider = (*vllmProvider)(nil)
+
+// ConverseStream resolves modelID's endpoint, opens a streaming OpenAI
+// chat-completions request (stream:true, stream_options.include_usage for
+// the trailing usage chunk), and returns a converseStreamSource that
+// reframes the SSE into the normalized Bedrock ConverseStream taxonomy.
+func (p *vllmProvider) ConverseStream(ctx context.Context, modelID string, input *bedrockruntime.ConverseStreamInput) (converseStreamSource, error) {
+	baseURL, ok, err := p.endpointResolver.Endpoint(ctx, modelID)
+	if err != nil {
+		slog.Error("vllm stream: endpoint resolution failed", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorServiceUnavailableException)
+	}
+	if !ok {
+		return nil, errors.New(awserrors.ErrorModelNotReadyException)
+	}
+
+	req := buildVLLMRequest(modelID, converseStreamToConverseInput(input))
+	req.Stream = true
+	req.StreamOptions = &vllmStreamOptions{IncludeUsage: true}
+
+	reqBody, err := json.Marshal(req)
+	if err != nil {
+		slog.Error("vllm stream: failed to marshal request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+
+	httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+vllmChatCompletionsPath, bytes.NewReader(reqBody)) //nolint:gosec // G704: baseURL is a resolved pinned self-host endpoint, not user input
+	if err != nil {
+		slog.Error("vllm stream: failed to build request", "model", modelID, "err", err)
+		return nil, errors.New(awserrors.ErrorInternalError)
+	}
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.Header.Set("Accept", "text/event-stream")
+
+	resp, err := p.httpClient.Do(httpReq) //nolint:gosec // G704: httpReq targets the resolved pinned self-host endpoint, not user input
+	if err != nil {
+		slog.Error("vllm stream: request failed", "model", modelID, "endpoint", baseURL, "err", err)
+		return nil, errors.New(awserrors.ErrorServiceUnavailableException)
+	}
+
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		respBody, _ := io.ReadAll(resp.Body)
+		_ = resp.Body.Close()
+		slog.Error("vllm stream: upstream error", "model", modelID, "status", resp.StatusCode, "body", string(respBody))
+		return nil, errors.New(mapUpstreamStatus(resp.StatusCode))
+	}
+
+	return &vllmConverseStreamSource{
+		resp:    resp,
+		scanner: newSSEScanner(resp.Body),
+		start:   time.Now(),
+	}, nil
+}
+
+// vllmConverseStreamSource reframes a vLLM (OpenAI chat-completions) SSE
+// stream into the normalized Bedrock ConverseStream event taxonomy:
+// messageStart+contentBlockStart on the first chunk, contentBlockDelta per
+// content delta, contentBlockStop+messageStop on finish_reason, metadata on
+// the trailing usage-only chunk (or defensively at EOF if the upstream never
+// sends one). Events are queued because a single upstream chunk can fan out
+// to more than one normalized event.
+type vllmConverseStreamSource struct {
+	resp    *http.Response
+	scanner *sseScanner
+	start   time.Time
+
+	queue           []ConverseStreamEvent
+	started         bool
+	stopped         bool
+	metadataEmitted bool
+	inputTokens     int64
+	outputTokens    int64
+}
+
+var _ converseStreamSource = (*vllmConverseStreamSource)(nil)
+
+func (s *vllmConverseStreamSource) Close() error {
+	return s.resp.Body.Close()
+}
+
+func (s *vllmConverseStreamSource) Next(_ context.Context) (ConverseStreamEvent, bool, error) {
+	for len(s.queue) == 0 {
+		ev, ok, err := s.scanner.Next()
+		if err != nil {
+			if errors.Is(err, io.EOF) {
+				s.closeOut(bedrockruntime.StopReasonEndTurn)
+				if !s.metadataEmitted {
+					s.emitMetadata()
+				}
+				break
+			}
+			return ConverseStreamEvent{}, false, newStreamFault(err)
+		}
+		if !ok {
+			continue
+		}
+		data := strings.TrimSpace(ev.Data)
+		if data == "" || data == "[DONE]" {
+			continue
+		}
+
+		var chunk vllmStreamChunk
+		if err := json.Unmarshal([]byte(ev.Data), &chunk); err != nil {
+			return ConverseStreamEvent{}, false, newStreamFault(fmt.Errorf("vllm stream: decode chunk: %w", err))
+		}
+		s.consume(chunk)
+	}
+
+	if len(s.queue) == 0 {
+		return ConverseStreamEvent{}, false, nil
+	}
+	next := s.queue[0]
+	s.queue = s.queue[1:]
+	return next, true, nil
+}
+
+// consume folds one decoded chat-completion chunk into the queued normalized
+// events and running usage totals.
+func (s *vllmConverseStreamSource) consume(chunk vllmStreamChunk) {
+	if chunk.Usage != nil {
+		s.inputTokens = chunk.Usage.PromptTokens
+		s.outputTokens = chunk.Usage.CompletionTokens
+	}
+	if len(chunk.Choices) == 0 {
+		// The trailing usage-only chunk from stream_options.include_usage.
+		// Close the block/message before metadata so the taxonomy stays ordered
+		// even if the upstream sends usage without a prior finish_reason chunk.
+		if chunk.Usage != nil {
+			s.closeOut(bedrockruntime.StopReasonEndTurn)
+			s.emitMetadata()
+		}
+		return
+	}
+
+	choice := chunk.Choices[0]
+	s.ensureStarted()
+
+	if choice.Delta.Content != "" {
+		s.queue = append(s.queue, ConverseStreamEvent{
+			Kind: converseStreamEventContentBlockDelta,
+			ContentBlockDelta: &bedrockruntime.ContentBlockDeltaEvent{
+				ContentBlockIndex: aws.Int64(0),
+				Delta:             &bedrockruntime.ContentBlockDelta{Text: aws.String(choice.Delta.Content)},
+			},
+		})
+	}
+
+	if choice.FinishReason != nil {
+		s.closeOut(mapVLLMFinishReason(*choice.FinishReason))
+	}
+}
+
+// ensureStarted queues messageStart+contentBlockStart exactly once, so the stop
+// events can never appear without a preceding start on a zero-content stream.
+func (s *vllmConverseStreamSource) ensureStarted() {
+	if s.started {
+		return
+	}
+	s.started = true
+	s.queue = append(s.queue,
+		ConverseStreamEvent{
+			Kind:         converseStreamEventMessageStart,
+			MessageStart: &bedrockruntime.MessageStartEvent{Role: aws.String(bedrockruntime.ConversationRoleAssistant)},
+		},
+		ConverseStreamEvent{
+			Kind:              converseStreamEventContentBlockStart,
+			ContentBlockStart: &bedrockruntime.ContentBlockStartEvent{ContentBlockIndex: aws.Int64(0), Start: &bedrockruntime.ContentBlockStart{}},
+		},
+	)
+}
+
+// closeOut queues contentBlockStop+messageStop exactly once.
+func (s *vllmConverseStreamSource) closeOut(stopReason string) {
+	if s.stopped {
+		return
+	}
+	s.ensureStarted()
+	s.stopped = true
+	s.queue = append(s.queue,
+		ConverseStreamEvent{
+			Kind:             converseStreamEventContentBlockStop,
+			ContentBlockStop: &bedrockruntime.ContentBlockStopEvent{ContentBlockIndex: aws.Int64(0)},
+		},
+		ConverseStreamEvent{
+			Kind:        converseStreamEventMessageStop,
+			MessageStop: &bedrockruntime.MessageStopEvent{StopReason: aws.String(stopReason)},
+		},
+	)
+}
+
+// emitMetadata queues the trailing metadata event exactly once.
+func (s *vllmConverseStreamSource) emitMetadata() {
+	if s.metadataEmitted {
+		return
+	}
+	s.metadataEmitted = true
+	s.queue = append(s.queue, ConverseStreamEvent{
+		Kind: converseStreamEventMetadata,
+		Metadata: &bedrockruntime.ConverseStreamMetadataEvent{
+			Usage: &bedrockruntime.TokenUsage{
+				InputTokens:  aws.Int64(s.inputTokens),
+				OutputTokens: aws.Int64(s.outputTokens),
+				TotalTokens:  aws.Int64(s.inputTokens + s.outputTokens),
+			},
+			Metrics: &bedrockruntime.ConverseStreamMetrics{LatencyMs: aws.Int64(time.Since(s.start).Milliseconds())},
+		},
+	})
+}

spinifex/gateway/bedrock/vllm_test.go

@@ -18,7 +18,10 @@ func TestVLLMProvider_Converse_MapsResponse(t *testing.T) {
 	var captured vllmChatRequest
 
 	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		require.NoError(t, json.NewDecoder(r.Body).Decode(&captured))
+		if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&captured)) {
+			http.Error(w, "decode request body", http.StatusBadRequest)
+			return
+		}
 		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(http.StatusOK)
 		_, _ = w.Write([]byte(`{
@@ -71,3 +74,228 @@ func TestVLLMProvider_Converse_UnresolvedEndpointReturnsModelNotReady(t *testing
 	require.Error(t, err)
 	assert.Equal(t, awserrors.ErrorModelNotReadyException, err.Error())
 }
+
+// vllmStreamFixture is a canned OpenAI chat-completions streaming SSE body:
+// a role-only first chunk, two content deltas, a finish_reason chunk, a
+// trailing usage-only chunk (stream_options.include_usage), then [DONE].
+const vllmStreamFixture = `data: {"choices":[{"delta":{"role":"assistant"},"finish_reason":null}]}
+
+data: {"choices":[{"delta":{"content":"Hello"},"finish_reason":null}]}
+
+data: {"choices":[{"delta":{"content":" world"},"finish_reason":null}]}
+
+data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
+
+data: {"choices":[],"usage":{"prompt_tokens":8,"completion_tokens":3}}
+
+data: [DONE]
+
+`
+
+// drainConverseStream reads src to clean EOF, returning the ordered events.
+func drainConverseStream(t *testing.T, src converseStreamSource) []ConverseStreamEvent {
+	t.Helper()
+	var events []ConverseStreamEvent
+	for {
+		ev, ok, err := src.Next(context.Background())
+		require.NoError(t, err)
+		if !ok {
+			return events
+		}
+		events = append(events, ev)
+	}
+}
+
+func TestVLLMProvider_ConverseStream_MapsSSEToTaxonomy(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		assert.Equal(t, "text/event-stream", r.Header.Get("Accept"))
+		var req vllmChatRequest
+		assert.NoError(t, json.NewDecoder(r.Body).Decode(&req))
+		assert.True(t, req.Stream)
+		if !assert.NotNil(t, req.StreamOptions) {
+			return
+		}
+		assert.True(t, req.StreamOptions.IncludeUsage)
+
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(vllmStreamFixture))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	p := newVLLMProvider(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	p.httpClient = ts.Client()
+
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	}
+
+	src, err := p.ConverseStream(context.Background(), modelID, input)
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	events := drainConverseStream(t, src)
+	require.Len(t, events, 7)
+
+	kinds := make([]converseStreamEventKind, len(events))
+	for i, ev := range events {
+		kinds[i] = ev.Kind
+	}
+	assert.Equal(t, []converseStreamEventKind{
+		converseStreamEventMessageStart,
+		converseStreamEventContentBlockStart,
+		converseStreamEventContentBlockDelta,
+		converseStreamEventContentBlockDelta,
+		converseStreamEventContentBlockStop,
+		converseStreamEventMessageStop,
+		converseStreamEventMetadata,
+	}, kinds)
+
+	assert.Equal(t, bedrockruntime.ConversationRoleAssistant, *events[0].MessageStart.Role)
+	assert.Equal(t, int64(0), *events[1].ContentBlockStart.ContentBlockIndex)
+	assert.Equal(t, "Hello", *events[2].ContentBlockDelta.Delta.Text)
+	assert.Equal(t, " world", *events[3].ContentBlockDelta.Delta.Text)
+	assert.Equal(t, int64(0), *events[4].ContentBlockStop.ContentBlockIndex)
+	assert.Equal(t, bedrockruntime.StopReasonEndTurn, *events[5].MessageStop.StopReason)
+	assert.Equal(t, int64(8), *events[6].Metadata.Usage.InputTokens)
+	assert.Equal(t, int64(3), *events[6].Metadata.Usage.OutputTokens)
+	assert.Equal(t, int64(11), *events[6].Metadata.Usage.TotalTokens)
+	assert.GreaterOrEqual(t, *events[6].Metadata.Metrics.LatencyMs, int64(0))
+}
+
+func TestVLLMProvider_ConverseStream_UnresolvedEndpointReturnsModelNotReady(t *testing.T) {
+	p := newVLLMProvider(NewStaticEndpointResolver(nil))
+
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	}
+
+	_, err := p.ConverseStream(context.Background(), "meta.llama3-70b-instruct-v1:0", input)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorModelNotReadyException, err.Error())
+}
+
+func TestVLLMProvider_ConverseStream_UpstreamNon2xxMapsStatus(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusTooManyRequests)
+		_, _ = w.Write([]byte(`{"error":"rate limited"}`))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	p := newVLLMProvider(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	p.httpClient = ts.Client()
+
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	}
+
+	_, err := p.ConverseStream(context.Background(), modelID, input)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorThrottlingException, err.Error())
+}
+
+// A usage-only chunk that arrives without a preceding finish_reason chunk must
+// still yield contentBlockStop+messageStop before metadata, not metadata first.
+func TestVLLMProvider_ConverseStream_UsageWithoutFinishReasonKeepsOrder(t *testing.T) {
+	sse := "data: {\"choices\":[{\"delta\":{\"role\":\"assistant\",\"content\":\"Hi\"}}]}\n\n" +
+		"data: {\"choices\":[],\"usage\":{\"prompt_tokens\":8,\"completion_tokens\":3}}\n\n" +
+		"data: [DONE]\n\n"
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(sse))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	p := newVLLMProvider(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	p.httpClient = ts.Client()
+
+	src, err := p.ConverseStream(context.Background(), modelID, &bedrockruntime.ConverseStreamInput{})
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	events := drainConverseStream(t, src)
+	kinds := make([]converseStreamEventKind, len(events))
+	for i, ev := range events {
+		kinds[i] = ev.Kind
+	}
+	assert.Equal(t, []converseStreamEventKind{
+		converseStreamEventMessageStart,
+		converseStreamEventContentBlockStart,
+		converseStreamEventContentBlockDelta,
+		converseStreamEventContentBlockStop,
+		converseStreamEventMessageStop,
+		converseStreamEventMetadata,
+	}, kinds)
+}
+
+// A zero-content stream (upstream closes before any choice chunk) must still be
+// well-formed: messageStart before the synthesized stop events, never orphaned.
+func TestVLLMProvider_ConverseStream_EmptyStreamIsWellFormed(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	p := newVLLMProvider(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	p.httpClient = ts.Client()
+
+	src, err := p.ConverseStream(context.Background(), modelID, &bedrockruntime.ConverseStreamInput{})
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	events := drainConverseStream(t, src)
+	require.NotEmpty(t, events)
+	assert.Equal(t, converseStreamEventMessageStart, events[0].Kind)
+	kinds := make([]converseStreamEventKind, len(events))
+	for i, ev := range events {
+		kinds[i] = ev.Kind
+	}
+	assert.Equal(t, []converseStreamEventKind{
+		converseStreamEventMessageStart,
+		converseStreamEventContentBlockStart,
+		converseStreamEventContentBlockStop,
+		converseStreamEventMessageStop,
+		converseStreamEventMetadata,
+	}, kinds)
+}
+
+func TestVLLMConverseStreamSource_MidStreamDecodeErrorSurfacesAsStreamFault(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("data: {not-json\n\n"))
+	}))
+	defer ts.Close()
+
+	modelID := "meta.llama3-70b-instruct-v1:0"
+	p := newVLLMProvider(NewStaticEndpointResolver(map[string]string{modelID: ts.URL}))
+	p.httpClient = ts.Client()
+
+	input := &bedrockruntime.ConverseStreamInput{
+		Messages: []*bedrockruntime.Message{
+			{Role: aws.String(bedrockruntime.ConversationRoleUser), Content: []*bedrockruntime.ContentBlock{{Text: aws.String("hello")}}},
+		},
+	}
+
+	src, err := p.ConverseStream(context.Background(), modelID, input)
+	require.NoError(t, err)
+	defer func() { _ = src.Close() }()
+
+	_, ok, err := src.Next(context.Background())
+	assert.False(t, ok)
+	require.Error(t, err)
+	var fault *streamFaultError
+	assert.ErrorAs(t, err, &fault)
+}

spinifex/gateway/bedrock_request_test.go

@@ -200,3 +200,102 @@ func TestBedrockRuntimeRequest_MissingAccountIDReturnsServerInternal(t *testing.
 	require.Error(t, err)
 	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
 }
+
+// newVLLMStreamStub stands up an httptest server answering the OpenAI
+// chat-completions streaming (SSE) wire, for the ConverseStream self-host path.
+func newVLLMStreamStub(t *testing.T) *httptest.Server {
+	t.Helper()
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(`data: {"choices":[{"delta":{"role":"assistant"},"finish_reason":null}]}
+
+data: {"choices":[{"delta":{"content":"hi from vllm stream"},"finish_reason":null}]}
+
+data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
+
+data: {"choices":[],"usage":{"prompt_tokens":4,"completion_tokens":2}}
+
+data: [DONE]
+
+`))
+	}))
+	t.Cleanup(ts.Close)
+	return ts
+}
+
+// newLlamaCompletionsStreamStub stands up an httptest server answering the
+// OpenAI completions streaming (SSE) wire, for the
+// InvokeModelWithResponseStream self-host path.
+func newLlamaCompletionsStreamStub(t *testing.T) *httptest.Server {
+	t.Helper()
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/event-stream")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(`data: {"choices":[{"text":"hi from llama stream","finish_reason":null}]}
+
+data: {"choices":[{"text":"","finish_reason":"stop"}]}
+
+data: {"choices":[],"usage":{"prompt_tokens":5,"completion_tokens":3}}
+
+data: [DONE]
+
+`))
+	}))
+	t.Cleanup(ts.Close)
+	return ts
+}
+
+func TestBedrockRuntimeRequest_ConverseStream(t *testing.T) {
+	ts := newVLLMStreamStub(t)
+	gw := newBedrockRequestGateway(t, ts.URL)
+
+	body := `{"messages":[{"role":"user","content":[{"text":"hello"}]}]}`
+	req := bedrockRequestWithAccount(http.MethodPost, "/model/"+bedrockTestLlamaModelID+"/converse-stream", body)
+	w := httptest.NewRecorder()
+	require.NoError(t, gw.BedrockRuntime_Request(w, req))
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "application/vnd.amazon.eventstream", w.Header().Get("Content-Type"))
+	assert.NotZero(t, w.Body.Len())
+}
+
+func TestBedrockRuntimeRequest_ConverseStream_UnknownModelReturnsResourceNotFound(t *testing.T) {
+	ts := newVLLMStreamStub(t)
+	gw := newBedrockRequestGateway(t, ts.URL)
+
+	body := `{"messages":[{"role":"user","content":[{"text":"hello"}]}]}`
+	req := bedrockRequestWithAccount(http.MethodPost, "/model/does.not-exist-v1:0/converse-stream", body)
+	w := httptest.NewRecorder()
+	err := gw.BedrockRuntime_Request(w, req)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
+	// Pre-stream failure: BedrockRuntime_Request must not have written
+	// anything, leaving ErrorHandler free to write the JSON envelope.
+	assert.Zero(t, w.Body.Len())
+}
+
+func TestBedrockRuntimeRequest_InvokeModelWithResponseStream(t *testing.T) {
+	ts := newLlamaCompletionsStreamStub(t)
+	gw := newBedrockRequestGateway(t, ts.URL)
+
+	body := `{"prompt":"hello","max_gen_len":128}`
+	req := bedrockRequestWithAccount(http.MethodPost, "/model/"+bedrockTestLlamaModelID+"/invoke-with-response-stream", body)
+	w := httptest.NewRecorder()
+	require.NoError(t, gw.BedrockRuntime_Request(w, req))
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "application/vnd.amazon.eventstream", w.Header().Get("Content-Type"))
+	assert.NotZero(t, w.Body.Len())
+}
+
+func TestBedrockRuntimeRequest_InvokeModelWithResponseStream_UnknownModelReturnsResourceNotFound(t *testing.T) {
+	ts := newLlamaCompletionsStreamStub(t)
+	gw := newBedrockRequestGateway(t, ts.URL)
+
+	body := `{"prompt":"hello"}`
+	req := bedrockRequestWithAccount(http.MethodPost, "/model/does.not-exist-v1:0/invoke-with-response-stream", body)
+	w := httptest.NewRecorder()
+	err := gw.BedrockRuntime_Request(w, req)
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorResourceNotFoundException, err.Error())
+	assert.Zero(t, w.Body.Len())
+}