Compare commits

..

36 Commits

Author SHA1 Message Date
Josh Siviour 0fc93e975a refactor: serve EC2 actions from domains/ec2/awsapi (#1354)
* test: characterise EC2 through the signed gateway pipeline

* refactor: serve EC2 actions from domains/ec2/awsapi over composed capabilities

The EC2 action table, idempotency and validation move out of the gateway behind a Deps struct. Quota is a consumer-side interface and the instance adapters take a narrow InstanceProfiles interface instead of the whole IAM service. The gateway only composes Deps and calls Dispatch. The 18 gateway to EC2 awsapi allowlist rows are deleted.

* test: drive every EC2 action through Dispatch in its own package

Covers the request errors, the gateway-local actions without NATS and each routed action reaching its handler against a stub daemon.

* docs: record the EC2 capability move in the package boundary migration

* docs: record the EC2 capability move PR number
2026-10-31 00:30:51 +10:00
Josh Siviour d066786faa feat: operator-launched SQS worker and store system VMs (#1353)
* feat: persist the SQS worker's store key on its own disk

The shared SQS worker generates its store-client NKey inside the guest and
keeps it in an owner-only directory on its own persistent disk, so no platform
component creates or unseals the seed. The key is synced and linked into place
without replacing anything; a missing, mismatched, unreadable or loosely
permissioned key fails closed instead of being regenerated.

* fix: make the SQS worker key durably named before returning it

Every successful call now syncs the key directory and its entry in the parent
before handing out a key, so a retry after a failed directory sync cannot
return a key whose name is not yet durable. Only the key directory is created,
inside a parent that must exist, so an unmounted worker disk fails closed. The
first-start decision is explicit: an existing store's authorized key is
required and never defaults to generating a new one.

* docs: state that workerkey cannot detect an unmounted worker disk

An existing parent directory does not prove the worker disk is mounted, since
the mountpoint can exist on the initramfs after a failed mount. The error now
says only that the parent is missing, and the package documents that its
caller must verify the expected data volume is mounted first.

* feat: add the SQS worker guest bootstrap

The shared SQS worker microVM boots with sqs-worker boot, which selects
exactly the disk whose serial fw_cfg names, checks and mounts it at /data,
and proves from the mount table that this disk is what is mounted before
loading or creating its store key there. A missing, empty or unknown store
status, a contradictory or invalid store key, a missing or ambiguous disk, a
failed check or mount, an empty mountpoint or a foreign mount refuses without
creating a key. Only the public key is logged and published.

The image is built by the store image script under SQS_MICROVM_PROFILE=worker,
with the worker binary, no nats-server and an empty /data mountpoint; the
store image's contents are unchanged.

* fix: refuse replayable new-store status at SQS worker boot

A static fw_cfg store-status of "new" survives a guest reset, so after the
store exists it could still reach key creation and replace a lost worker key.
Boot now accepts only "existing" with the store's authorized public key and
refuses "new" before touching any disk, until a current authoritative
first-store decision exists. workerkey.ForNewStore remains, uncalled.

* feat: add SQS first-store claim and worker key acquisition

The SQS domain records one first-store decision per Region in its own KV
bucket, moving claimed -> key-recorded -> launch-intent under CAS and bound
to the worker's instance ID, claim ID and generation. Key creation is
authorized only while no public key is recorded; afterwards only the
recorded key is returned, and a launch intent requires it. Older
generations, same-generation rivals and unreadable records authorize
nothing, and records hold no private key material.

workerkey.Acquire syncs a new key to disk and returns it only after its
public key is recorded, reusing the on-disk key after a crash before
recording and failing closed on a missing or different key afterwards.
No transport carries the decision yet.

* fix: read SQS first-store decisions from the stream leader

KV.Get uses a direct get, which a lagging replica can answer, so a stale
claimed read could authorize key creation after the leader had recorded
the worker's public key, or make a superseded generation look current.
Reads now use the SQS store's leader-answered stream message get, and
every authorizing answer is also confirmed by a compare-and-swap at the
revision read, so a stale or unavailable read authorizes nothing.

* feat(sqs): add regional worker identity record and instance index

The SQS domain keeps one worker identity record per Region with a monotonic
generation, claim ID, pinned node and encrypted data volume, moving from
launching to bound or failed under compare-and-swap. A launch intent takes
the next generation before any instance exists, so the previous worker stops
being current; binding writes the instance index entry first.

Authorize accepts an instance only while the Region's leader-read record is
bound to exactly that instance, generation and claim, confirmed by a
compare-and-swap at the read revision, so an old, orphaned, rewritten or
unbound index entry, or a lagging replica, authorizes nothing.

Both this bucket and the first-store bucket now open through the kvutil
chokepoint and refuse a stored configuration that could expire, cap or evict
records or holds fewer replicas than the cluster needs. Nothing calls the
store yet.

* fix(sqs): make worker launch exclusive and harden durable buckets

BeginLaunch now refuses a new claim while the previous attempt is launching
or bound, including one whose outcome is unknown, so only one caller ever
holds an intent to start a VM on the pinned volume. A replacement takes the
next generation only after FailLaunch or the new RecordStopped confirms that
nothing holds the volume, and each request names the generation it
supersedes, so a replayed request cannot take a generation it never saw.

The durable bucket opener also refuses non-limits retention, per-message
TTLs, asynchronous flush, a message-size cap, any discard policy other than
discard-new or a per-subject discard, unacknowledged writes, a sealed
stream, and a mirror or sources.

* feat: add internal SQS worker first-store gateway actions

Serve ClaimFirstStore and RecordWorkerPublicKey under the internal
spinifex-sqs signing service, apart from the public AmazonSQS inventory.
Only an sqsWorkerRole session in the system account passes the class
gate, then the role policy; the worker identity is resolved from the
current workerident record by the STS-set session name, never from the
request, and any extra parameter is refused. Seeds and other non-public
keys are refused without being recorded, logged or echoed.

The gateway binds the worker-identity and first-store buckets per
request and never creates them, so it provisions nothing and answers
ServiceUnavailable until a worker launch has created them.

* fix(sqs): fence replaced worker generations out of first-store

A request that passed worker authorization before a confirmed stop and
replacement could reach an unclaimed first-store record ahead of the
successor and create or record the old generation's key. The launch path
now raises a monotonic first-store fence past the replaced generation
before BeginLaunch, which refuses a replacement until a leader read shows
it; firststore refuses every call from a generation below the fence.

The internal adapter no longer logs caller-chosen parameter names.

* feat(sqs): launch the shared SQS worker on its pinned node

Add the SQS worker system-instance kind and a domain launcher that
realizes the Region's worker from its identity record without ever
starting a second VM:

- domains/sqs/workerlaunch returns a bound worker as is, prepares the
  data volume on first launch only, refuses a volume whose own record
  is not encrypted, marks the intent dispatched before launching and
  never resends a dispatched launch: an ambiguous outcome stays
  launching until reconciled. A replacement starts only after a
  confirmed stop or failure and the first-store fence, reusing the
  pinned volume.
- workerident gains MarkDispatched and Current; firststore gains a
  leader-read RecordedKey.
- The daemon launches the worker through the shared direct-boot path
  with its encrypted volume attached at boot and kept on terminate, a
  required instance profile and a mandatory mgmt NIC. Its firmware
  config carries only public values: the data serial, the store
  decision, the gateway URL on this node's bridge and the daemon's own
  cluster CA. A relaunch is refused on any other node.
- sqsWorkerRole is ensured in the system account granting exactly the
  two internal actions; an existing role whose trust admits more than
  EC2 is refused. The role contract lives in domains/sqs/workerrole so
  the composition root does not import the gateway adapter.

The worker network allocator and any caller remain unwired, and the
guest still refuses to create a key for a new store.

* fix: verify SQS worker role trust before granting and report unclaimed volumes

Check that an existing or racing-created sqsWorkerRole trusts only EC2
before writing its policy, and accept the instance profile only when a
re-read shows it carries exactly that role. Refuse the first launch
before allocating when the cluster storage key is not configured, and
return the ID of any volume created but not confirmed in a launch
intent instead of dropping it.

* feat: add private SQS system VPC and worker ENI allocator

Add systemvpc.EnsurePrivate, which builds one VPC and one private subnet
from the VPC and subnet surface only, so it cannot create an internet
gateway, route table, NAT gateway or EIP. The daemon's SQS worker network
adapter uses it with a [sqs] supernet defaulting to 10.240.0.0/14 and
finds or creates one ENI per launch claim in the VPC default security
group, refusing a duplicated claim or an address outside the subnet.

Reword UnclaimedVolumeError: a lost BeginLaunch reply may still have
recorded the volume.

* feat: worker takes its first-store decision from the gateway at boot

The SQS worker guest no longer trusts a static fw_cfg store status. After
selecting, checking and verifying the mount of its own data disk, it asks
the internal spinifex-sqs gateway service for its current first-store
decision, signed with instance-metadata credentials only, over TLS 1.3
trusting only the cluster CA. It then creates and records or loads the
key on that disk; a key lost after recording is never replaced.

- internal/sqsworkergw: SigV4 client for ClaimFirstStore and
  RecordWorkerPublicKey; refuses non-public keys, non-decisions and
  redirects; errors carry only the parsed code
- gwsign.NewInstanceRole: IMDS-only credentials, no env/profile fallback
- workerboot: retries only lost replies and ServiceUnavailable/Throttling
- launch contract carries the Region instead of a store status or key

* fix: worker retries a denial while its launch is still binding it

The worker VM can reach the gateway before its launch binds the instance
identity, and that early call is legitimately AccessDenied. The guest
treated the denial as final and stayed unprepared even after binding.
AccessDenied is now a pending authorization the guest asks about again
within its existing bound; a denied claim creates nothing, and a worker
that is never bound, or was replaced, gives up at the bound.

* feat(sqs): add operator EnsureSQSWorker action and node launch handler

Add the super-admin EnsureSQSWorker(NodeId, InstanceType) action on the
internal spinifex operator service. It validates the node against the
cluster and as a single subject token, then requests
sqs.worker.Ensure.v1.<node> with no queue group, so only that node answers
and the worker stays pinned.

The node handler re-checks the named node and instance type and runs one
composed workerlaunch.Launcher (real volume preparer, role, private network
and direct-boot launch), one launch at a time, in the daemon's own context
with a 30-minute bound. A caller timeout does not cancel the launch; a
retry reports the bound worker, or IncorrectState for a dispatched launch
whose outcome is unknown, which is never resent. The reply names the bound
VM and does not claim the worker or SQS is ready. Nothing starts a worker
at daemon startup.

* feat: ship the SQS worker image and check it before allocating

Build the SQS worker microVM image in both release workflows, stage it into
the tarball as microvm-sqs-worker, and install it with setup.sh under the
store image's rules: a release without it removes an earlier one and an
incomplete one is refused. Add make install-sqs-worker-microvm.

The worker launcher now asks a Host capability whether this node can boot a
worker before it prepares a volume, records or resumes an intent, fences a
replaced generation, allocates the network or dispatches. The daemon checks
for a non-empty kernel and initramfs, its mgmt bridge and the cluster CA, and
the EnsureSQSWorker handler reports a refusal as ServiceUnavailable, so a
call after the image is installed launches instead of leaving an ambiguous
dispatched intent.

Cover missing and incomplete images through the admin action and handler,
a stopped worker left untouched, and an ENI create whose reply is lost.

* fix: never repeat an uncertain SQS worker ENI create

The launcher now looks for the claim's ENI without creating one, marks the
launch intent NetworkAttempted, and only then sends the single create. Once
marked, a missing ENI fails closed with IncorrectState instead of a second
create that could give the claim two interfaces; a visible ENI is reused.

Adds a test that buildSQSWorkerLauncher wires the daemon's real host check.

* feat(sqs): operator-triggered SQS store launch on the worker's node

Add the super-admin EnsureSQSStore action and its node handler. The
store is launched only on the node the Region's worker is bound to,
with the public key that worker recorded in the first-store decision;
the daemon never receives the worker's seed.

A per-Region store launch record marks each step before its uncertain
create (volume, ENI in the worker's private subnet, VM launch), so a
step whose result is not visible fails closed and is never resent.
Volumes are refused unless their own record is encrypted, and their
IDs are kept. The store guest runs under GOMEMLIMIT=512MiB and
instance types at or below that are refused; this is a floor, not a
validated size.

* fix: retry uncreated SQS store volumes and fix the store instance type

A volume preparation that failed before sending any create (a missing
tool, scratch or format failure, or no cluster storage key) now carries
storevolume.ErrNotCreated. The store launcher withdraws its volume mark
and returns ServiceUnavailable, so a retry may try again once the host
is fixed; any create that was sent stays one-shot.

The store launch record now keeps the operator's instance type. A
request for a different type is refused as IncorrectState before
dispatch and after binding, and the reply names the launched type,
which the admin action checks.

* feat: SQS worker store readiness and store size check

The worker's one-shot ready step mounts and verifies its disk, asks the
internal spinifex-sqs DescribeStore action for the bound store, loads its
key only if it matches the recorded public key, and checks the store's
JetStream with it. It retries within a bound while the store is pending or
not answering, and stops on a mismatched or refused key. The NKey
authenticates the worker only; the store server is not authenticated.

EnsureSQSStore now also refuses a different volume size once the store is
bound, instead of returning the existing store.

* fix(sqs): keep worker store readiness current until shutdown

The readiness step now runs as a guest-local watch loop with no overall
deadline: each pass is bounded, a pending or unreachable store is retried
every few seconds, and a ready store is rechecked periodically. The
store-ready marker is removed on any failed pass, cleared at start and
removed on exit; shutdown stops the watcher by its recorded PID. Terminal
refusals such as a rejected or mismatched key back off for minutes rather
than spinning. The NKey still authenticates only the worker to the store.

* feat(sqs): supervise the worker readiness watcher and add spx admin sqs

Run the worker guest's readiness watcher under init respawn, rate-limited,
and remove any stale store-ready marker before each start, so a killed or
crashed watcher cannot leave a false readiness marker. Shutdown blocks
respawn, removes the marker, stops the recorded watcher by PID and start
time with SIGKILL escalation, and unmounts /data only once it has exited;
a surviving watcher leaves every mount in place.

Add spx admin sqs ensure-worker and ensure-store, which call the existing
EnsureSQSWorker and EnsureSQSStore actions through the signed spinifex
gateway path with the standard AWS credential chain. Both refuse missing
inputs before sending anything and report only the bound VM, never
readiness.

* fix: park SQS worker readiness and fix the worker instance type

A readiness start that finds shutdown in progress or no worker key now
logs once and waits in place instead of exiting, so init's respawn does
not restart and log it in a loop; shutdown stops it like a watcher.

The worker identity record keeps each generation's instance type. A
request for another type is refused as IncorrectState before dispatch
and after binding, and only a replacement may change it. The ensure
reply, admin action and spx admin sqs ensure-worker report and check
the recorded type.

* fix: record the SQS worker volume intent before creating it

The first worker launch prepared its data volume before any record
existed, so a create that applied but returned no ID left nothing for the
next call to find, and it could prepare a second volume.

The launch intent is now recorded first, and its one volume create is
marked before it is sent. A Prepare proven to have sent nothing withdraws
the mark and is retryable; any other outcome is never resent and returns
IncorrectState, naming the volume when its ID is known. An intent that
has sent no create may still be re-typed.

* docs: document spx admin sqs ensure-worker and ensure-store

Record the flags, the caller and owning accounts, the bound-versus-ready
distinction and the current retry and recovery limits.

* docs: distinguish SQS ensure timeouts from refusals that created nothing

A gateway timeout means only that no reply arrived; the launch may still be
running. The worker type locks once any step has been attempted.

* docs: state what each SQS ensure ServiceUnavailable establishes

No refusal proves earlier attempts created nothing; a retry checks the
durable attempt and continues only when that is safe.
2026-10-31 00:17:25 +10:00
Josh Siviour 55054ad0ac fix: interim guard against opening a volume behind a live owner holding unsealed writes (#1352)
* test: relay a capacity refusal from an owner holding unsealed writes

* fix: refuse a start elsewhere when the owner holds unsealed writes

* fix: bound the unsealed-writes check well under the start forward timeout

* test: refuse a volume takeover while the marker owner is live

* fix: refuse a volume takeover while the previous owner is live

* fix: retry EKS control-plane starts refused for capacity or a held volume

* docs: describe the interim unsealed-writes takeover guards

* test: give the viperblockd conformance harness a cluster-state bucket

The takeover guard reads node liveness from the cluster-state bucket the
daemon creates. The harness had none, so the real reader errored and the
guard refused every takeover. Create the bucket the way the daemon does so
the real reader runs and finds no heartbeat.

* fix: leave no dirty marker behind when opening a volume that does not exist

An open takes the lease and writes the dirty marker before the volume's
state is read. When the volume turned out to be absent the marker stayed,
so a later open on another node met the takeover guard and was refused
as held elsewhere instead of answering not found. Clear the marker this
open wrote when the state load reports the volume missing.
2026-10-30 23:53:41 +10:00
Josh Siviour 27304bbfd6 refactor: serve the four Bedrock services from one Ochre registration (#1351)
* test: characterise the four Bedrock services through the SigV4 pipeline

* refactor: serve the four Bedrock services from one ochre registration

The Bedrock adapters move from gateway to domains/ochre/awsapi behind a
single dispatch registration. The sub-service scopes are aliases, a
sub-service scope selects its dispatcher outright and only the shared
bedrock scope is split by path. The thirteen Bedrock fields leave
GatewayConfig for awsapi.Deps, and the bodyscope allowlist row for
domains/ochre is deleted.

* docs: record the Bedrock registration in the package boundary migration

* docs: record the Bedrock registration PR number
2026-10-30 23:06:24 +10:00
Josh Siviour a52b078678 refactor: move the ECR /v2 bridge out of the gateway (#1350)
* test: pin the ECR /v2 pipeline through the gateway handler

* feat: rehydrate a principal from an access key id in credverify

Adds credential.Rehydrator and CallerARN on Principal, implemented beside Authenticate and Resolve on the same key, session and user lookups, with a parity table against the gateway's ECR rehydrate.

* refactor: share the credverify key and session lookups with Rehydrate

Authenticate, Resolve and Rehydrate now call one lookup per credential kind, so the checks cannot drift. The parity table also asserts the gateway and credverify caller ARN builders agree.

* refactor: move registry host matching to ingress/http

The matcher takes its service labels from the caller and exposes the parsed account and region through accessors, so ingress names no service.

* refactor: extract the identity-policy evaluator into policyauth

The SigV4 gateway path and the ECR registry authorize through one evaluator and one set of request condition keys, over the credential principal.

* refactor: extract account admission into accountadmission

The SigV4 gateway path asks one Admitter for account status, with the ACTIVE-only cache moved with it, so the ECR registry can use the same capability.

* refactor: move the ECR /v2 bridge out of the gateway

The bridge, token exchange and per-operation authorization now live in
domains/ecr/registry behind registry-owned Principal, Rehydrator and
Authorizer types, so the package imports no ingress code. The awsgw role
adapts credverify, policyauth and host matching to them, and the gateway
keeps only a RegistryHandler mounted at /v2. The bridge refuses to
compose without a verifier, issuer, rehydrator, authorizer or admission,
so the open-registry mode for a nil verifier is gone.

* test: carry the retired ECR principal parity rows into credverify

Global root, a verifier with no IAM, a record with no principal type and
the gone, replaced and legacy session verdicts for user and role sessions
were only exercised by the gateway tests that went with the old path.

* docs: record the ECR /v2 bridge move in the package boundary migration

* docs: record the ECR /v2 bridge PR number
2026-10-30 15:57:37 +10:00
Josh Siviour 26d53602a3 fix: accept AWS-format IRSA trust-policy condition keys (#1349)
* test: cover AWS-form IRSA condition keys and StringLike on web identity

* fix: match AWS-form IRSA condition keys and allow StringLike on web identity

* fix: keep the provider ARN host-path unnormalised for web identity

* feat: log the reason a web identity trust policy denies

* fix: satisfy errname, modernize and gofmt on the denial error

* test: cover the stock AWS IRSA trust policy in the EKS e2e

* test: make the IRSA denial prove the policy was evaluated
2026-10-30 14:34:08 +10:00
Josh Siviour 060204514e refactor: register ECS and EKS through the ingress seam with shared PassRole (#1348)
* test: pin ECS and EKS dispatch through the SigV4 pipeline

* refactor: register ECS and EKS on the ingress seam

EKS resolves the cluster and nodegroup role through passrole.Check with the
IAM role resolver, in the order it had. Responses and check order are unchanged.

* fix: match AWS role handling for ECS and EKS PassRole cases

ECS refuses a foreign task role before PassRole, accepts a role that is not an ARN or does not exist after PassRole, and reports a RunTask override role it cannot assume. EKS CreateCluster and CreateNodegroup refuse a foreign or non-role ARN with AccessDeniedException before PassRole.

* test: cover ECS and EKS awsapi registration dispatch

* fix: report EKS parameter faults as InvalidParameterException

* docs: record ECS and EKS gateway migration and EKS divergences
2026-10-30 14:18:53 +10:00
Josh Siviour 12cea4f6f7 feat: add IAM role resolution and the shared iam:PassRole flow (#1347)
* test: pin the current role-resolution gaps in EC2, ECS and EKS

* feat: add the shared iam:PassRole flow with classified role resolution

* feat: resolve same-account roles for iam:PassRole in domains/iam
2026-10-30 12:54:44 +10:00
Josh Siviour 099a69b23b refactor: register IAM and STS through the ingress seam (#1346)
* docs: point the /admin migration row at spinifex#1345

* test: characterise IAM and STS dispatch through the SigV4 pipeline

* refactor: register IAM through the ingress seam

* refactor: register STS through the ingress seam

* docs: point the IAM and STS migration rows at spinifex#1346
2026-10-30 12:08:15 +10:00
Josh Siviour 6462ccc41f refactor: serve the /admin operator API from the spinifex registration (#1345)
* refactor: move the /admin handlers and wire types into operator/admin

* refactor: serve /admin through the spinifex registration

* docs: record the /admin move into the spinifex registration
2026-10-30 10:50:47 +10:00
Josh Siviour ae8ee1e5fe refactor: register RDS through the ingress seam (#1344)
* refactor(gateway): register RDS through the ingress seam

RDS dispatch moves from gateway/rds.go to domains/rds/awsapi.NewRegistration, composed by awsgw and the integration harness.
The caller comes from dispatch.Actor and the principal-class gate still runs before the policy check.

* docs: point the RDS migration row at spinifex#1344

* chore: group the imports merged in the actor projection test
2026-10-30 09:59:15 +10:00
Josh Siviour 488f1c7bf9 feat: SQS store guest on a prepared cold-attached EBS volume (not wired) (#1343)
* feat: build a separate SQS store microVM image

Package the go.mod-pinned nats-server, ext4 support and a clean ACPI
power-down into a microVM kernel and initramfs of their own. The build
checks the packed archive's contents and refuses a nats-server whose
version differs from the pin. The image does not start the store and
nothing installs or releases it; the LB image is unchanged.

* fix: publish the SQS store image only after its gates pass

Stage the kernel and initramfs and move them into place only once the
content, size and guest-RAM gates pass, so a failed build cannot leave a
pair make takes as up to date. The old initramfs is removed first and the
new one lands last, so an interrupted publish never leaves a mixed pair.
Check the module list without piping into grep -q, which under pipefail
reported a present module as missing.

* feat: start the SQS store guest on its checked data disk

The SQS store image now prepares, runs and stops a dedicated JetStream
server. It reads a narrow fw_cfg contract (data disk serial, listen
address, one user NKey public key), refuses anything absent, malformed
or ambiguous, has the pinned server test the config, then checks and
mounts exactly one matching ext4 disk. It never formats and has no
ephemeral fallback. The server listens only on the supplied guest NIC
address, admits only that NKey and syncs every write.

Shutdown stops the recorded run process first, whether it is the server
or a start not yet exec'd, and runs the generic read-only remount only
once it is gone. A store that survives SIGKILL leaves every mount alone.

* fix: report the SQS store as prepared, not listening, before it starts

* feat: launch the SQS store guest on its own versioned subject

An explicit SQS store kind carries an existing system-account data volume
and the one NATS user public key the store accepts. It is served only on
system.LaunchSQSStore.v1, so a daemon without this code has no responder
and the launch fails instead of booting a load balancer without its disk;
the legacy launch path refuses the kind on both send and receive.

Every refusal happens before capacity, an ENI or a VM record exists. The
volume goes through the normal EBS mount and terminate detach and is
never deleted on termination. The store boots its own image with CPU
mitigations on, gets only its network, disk serial, listen address and
key, and cannot be relaunched after a host restart yet.

* feat: ship the SQS store microVM image in releases and installs

Release builds now build the SQS store image and stage it beside the load
balancer's as microvm-sqs-store/, so an older installer that copies
microvm/* without -r still works. setup.sh installs it to
/usr/share/spinifex/microvm/sqs-store/, where the daemon looks for it.

A release without a complete image never leaves an older one usable:
staging drops a previously staged image, and the installer removes an
earlier installed pair before warning (missing) or failing (incomplete).
Only the two image files are removed. The initramfs is installed last, so
an interrupted install leaves no bootable mismatched pair.

* feat: restart the SQS store guest on its launching node

Persist the non-secret guest config of an SQS store instance (launching
node, data volume, listen address, client public NKey and rendered
network config) in the instance record, and rebuild its fw_cfg files
from it on relaunch. The relaunch refuses on any other node, or when the
record is missing or malformed, before writing any file; the data volume
is remounted through the normal EBS path and kept on terminate.

Cross-node recovery now never selects or claims an SQS store record, so
no other node spends retry budget on it or relaunches it. Same-node
restore is unchanged, as are load balancer restarts. Older daemons know
neither the exclusion nor the record field, so this does not prevent a
cross-node relaunch by a mixed-version cluster.

* feat: hold prepared system volumes in creating until released

Add an internal CreateVolumeForPreparation path that writes a new
volume's first record as creating, and CompleteVolumePreparation to
release an unattached creating volume in the caller's account to
available. Ordinary CreateVolume is unchanged.

Attach and delete already refuse a creating volume. ModifyVolume,
ForceDetachVolume and CreateSnapshot now refuse it with IncorrectState
so nothing can expand, release or capture a volume mid-preparation.

* feat: prepare fresh SQS store volumes with an ext4 filesystem

The SQS store guest never formats a disk, so a new volume gets its
filesystem on the host. The preparer checks for mkfs.ext4 and qemu-img,
builds an ext4 image in private scratch, creates the volume in
"creating", publishes it on this node, copies the image with the NBD
import helper, unpublishes, and only then releases it as available.

It never writes a volume it did not just create. A volume is deleted
only when no export can still be live: after a confirmed unpublish, or
when the export was not NBD. Any publish error, a failed or ambiguous
unpublish, or a failed release retains the volume and reports its ID.
A publish refused as in-use or already-exists sends no unpublish, since
another export may exist. Cleanup ignores caller cancellation but is
bounded at 30s.

The raw-image copy moves to runtime/compute/nbd unchanged, and the
domain receives it as an injected writer. The volume service gains a
discard path for an unattached volume still in "creating".

* fix: report unknown state and match qemu-img lookup in volume prep

A failed release may still have made the volume available, so the
retained-volume error now says its state is unknown rather than
claiming it is still creating.

The copy runs qemu-img by name through PATH, so the pre-allocation
check now uses a PATH lookup for it too. mkfs.ext4 keeps the sbin
fallback because it runs from the resolved path.

* docs: describe the unwired SQS store system instance

Records what the guest image, versioned launch, same-node restart and
fresh-volume preparation provide, that nothing calls them, and what is
untested or missing before production use.
2026-10-30 08:16:43 +10:00
Ben Duncan 3617e4a045 fix: log an expected outcome at its own level, not ERROR (#1342)
A dispatch failure was logged at ERROR whatever the caller was about to
be given, so DryRunOperation, which AWS documents as the success answer
to a dry run, read as a fault of ours along with every NotFound and
malformed parameter. The level now comes from the status the caller
receives, resolved the way ErrorHandler resolves it.

QEMU announces its own exit on stderr, so a guest stopping on the
SIGTERM the daemon sent it was logged at ERROR too.
2026-10-29 14:38:41 -05:00
Josh Siviour 24281cb860 fix: count volume lease validity from the write, not its reply (#1340)
* fix: count volume lease validity from the write, not its reply

The server starts a lease entry's TTL when it applies the write, but the
holder started its validity window when the acknowledgement arrived, so a
slow reply stretched validity past the server's expiry and a peer could
acquire the volume while the holder still considered it valid.

Validity now counts from when each create, generation publish and renewal
was sent. An acquisition acknowledged after validity lapsed is refused as
retryable, and a lost or lapsed lease still awaiting its fence is no
longer shared with another open on the same node. A renewal that lands
after the lease was given up, or after its validity elapsed, cannot revive
it: the lapse check and the update run under the lease lock, and fencing
runs once. The renew loop also fences at the validity deadline itself
rather than at the next check, which could consume the whole margin.

This bounds local validity only; it is not backend-enforced fencing, and a
paused host can still write after its validity ends.

* docs: describe volume lease validity, fencing and their limits
2026-10-29 21:38:22 +10:00
Josh Siviour e58838bc67 refactor: verify SigV4 credentials through an ingress capability (#1339)
* refactor(gateway): verify SigV4 credentials behind a CredentialVerifier

The authenticator no longer imports the IAM and STS domains. It hands the
signature check to a verifier as a callback, so the secret never leaves the
adapter, and runs the account-active check between Authenticate and Resolve.
A nil verifier fails every request closed.

The adapter lives in runtime/roles/awsgw/credverify. ResolveCallerUser and the
principal-type constants move from domains/sts/awsapi to domains/sts so the
adapter can use them without importing an awsapi package.

* fix(gateway): guard a nil credential verifier and tighten its tests

A typed-nil credverify.Verifier now fails closed instead of panicking.
The session verdict test goes through the real verifier into the lockout boundary.
Test setups build the verifier from the same mock as the gateway services.

* test(gateway): commit the verifier test helper
2026-10-29 21:30:06 +10:00
Josh Siviour f0e4ffe70c refactor: give seam services the verified caller's identity facts (#1338)
* test: pin the caller facts the gateway gives a registered service per principal kind

* refactor(gateway): seal the verified caller into one request-context value

* feat(dispatch): give registered services the caller's principal ARN, user ID, user name and session name

Actor gains PrincipalARN, UserID, UserName and SessionName, filled from the
sealed principal, and RoleName(), which keeps the account-checked role ARN
rule in one place. No consumer reads them yet.

* refactor(gateway): read the caller from dispatch.Actor in the RDS, EKS, STS, IAM and admin handlers

callerActor builds the Actor the seam hands services, and the seven readers that
took fields off the sealed principal now take them from it. RDS and EKS take the
role name from Actor.RoleName(). No behaviour change.
2026-10-29 16:13:41 +10:00
Josh Siviour 18adea9486 refactor: register the spinifex operator service through the ingress seam (#1337)
* test: characterize the spinifex Query and /admin surfaces through the full gateway pipeline

* refactor: register the spinifex Query service from operator/admin

* docs: point the spinifex registration migration row at the PR
2026-10-29 14:55:50 +10:00
Julian Sommer f06def6894 chore: update operation count 2026-10-29 13:50:59 +10:00
Julian Sommer ef7d30578f chore(awsmodel): sync AWS models to api-models-aws e10d6a1c (2026-10-28)
Seven models changed and add 26 unimplemented operations. The request sweep gains 33 requests with no new findings.
2026-10-29 13:35:39 +10:00
Julian Sommer b76679e571 fix(integration): restore ELBv2 to the request conformance sweep
The ELBv2 move to the ingress seam added its inventory to the coverage tool but not to TestRequestConformance, so 36 operations and 45 findings dropped out while the suite passed. A served service with no implemented operations now fails the sweep.
2026-10-29 13:34:12 +10:00
Josh Siviour d7d922e963 refactor: register Tagging and ELBv2 through the ingress seam (#1336)
* test: characterize Tagging dispatch, authorization and GetResources responses through the gateway pipeline

* refactor: register Resource Groups Tagging through the ingress seam

* test: characterize ELBv2 through the full gateway pipeline

* refactor: register ELBv2 through the ingress seam

* test: cover the ELBv2 registration, query parser and load balancer limit

* test: characterize RDS through the full gateway pipeline

* docs: point the Tagging and ELBv2 migration rows at the PR

* test: register Tagging and ELBv2 on the integration harness gateway
2026-10-29 13:27:43 +10:00
Julian Sommer 64e3711893 fix(awsmodel): accept InvalidGatewayID.NotFound in the EC2 error catalog
AWS's EC2 error reference documents it, and DeleteEgressOnlyInternetGateway and ReplaceRoute answer it, so the integration suite reported two false violations.
2026-10-29 13:27:09 +10:00
Josh Siviour ad9568a6c6 refactor: move gateway/spx, admin wire types and EKS OIDC discovery out of gateway (#1334)
* refactor: move gateway/spx to operator/admin

* test: pin the admin wire types and method list in golden JSON

* refactor: move the admin wire types and method list to contracts/operator/v1

* refactor: serve EKS OIDC discovery and JWKS from the EKS domain

* refactor: assert the EKS OIDC publisher satisfies the gateway interface

* docs: cite the pull request in the gateway root-move rows
2026-10-29 12:22:16 +10:00
Josh Siviour dedb3d8d75 ci: keep the E2E WAN packet captures only for runs that did not pass (#1335)
The captures are about 90% of each E2E artifact and nothing reads them unless
a run failed, so a clean single-node or multi-node pass drops them before the
artifact upload.
2026-10-29 12:21:30 +10:00
Julian Sommer cc1186fa48 fix: grant the build user passwordless sudo for the appliance build when it lacks it, and remove the grant at seal 2026-10-29 11:10:51 +10:00
Julian Sommer dbdb827a69 fix: install tar on a minimal Oracle Linux build system before copying the appliance payload 2026-10-29 11:05:35 +10:00
Josh Siviour c94b0f22f6 feat: accept EnaSupport on RegisterImage, report it on instances, and name refused parameters (#1332)
* feat: accept EnaSupport on RegisterImage and name the refused parameter in its errors

EnaSupport true and false are stored on the image record, returned by DescribeImages when set and carried through CopyImage. Every validation error in ValidateRegisterImageInput keeps its code and now says which parameter and value it refused.

* feat: set EnaSupport on images created from an instance

AWS takes an image's EnaSupport from the instance it was made from, and every instance here reports it, so the source AMI's value is not inherited.

* docs: record RegisterImage EnaSupport support and the refused parameters

* fix: report EnaSupport on DescribeInstances and RunInstances
2026-10-29 10:53:30 +10:00
Julian Sommer 71567d0998 fix: show the appliance password rules before the first-login change 2026-10-29 10:51:06 +10:00
Josh Siviour 11e8b68ddc refactor: remove the unused host SQS message store service (#1333)
* refactor: remove the host SQS store systemd unit

The SQS message store will run inside system-owned instances rather than
as a host daemon, so spinifex.target no longer starts it. Bump the target
to version 4 and drop the unit from the embedded set and its invariants.

* refactor: remove host SQS store config provisioning

The SQS store will run in system-owned instances, so admin init and join
no longer generate or distribute a host store token, routes or config,
and the formation join payload drops the store token field.

* refactor: remove host SQS store service command and role

The SQS store will run in system-owned instances, so drop the unused
spx service sqs-store command, its service registration and the host
sqsstore role. The domains/sqs/message JetStream store is unaffected.

* refactor: remove host SQS store provisioning from install scripts

The SQS store will run in system-owned instances, so setup no longer
creates the spinifex-sqsstore user, its directories or the 4223/4249
peer firewall ports, and install, reset and uninstall stop handling the
retired unit, user and data directory.

* docs: drop the host SQS store from firewall and connection tables

Spinifex no longer ships a host SQS store service, so remove its ports
from the host firewall and network connection references and say in
DESIGN.md that no host service, unit, user or firewall rule exists for it.
2026-10-29 10:48:05 +10:00
Julian Sommer 8f4754da13 fix: install the appliance with the setup.sh from the release tarball, since install.mulgadc.com serves the stable one 2026-10-29 10:30:48 +10:00
Julian Sommer 8b965d24ce fix: refuse Route 53 requests that break the model's bounds with InvalidInput (#1329) 2026-10-29 11:29:45 +11:00
Julian Sommer ac2620df58 fix: report an SSH failure in build-oracle-appliance.sh as one, not as missing sudo 2026-10-29 10:26:09 +10:00
Julian Sommer 2feb264863 feat: Oracle Linux multi-node clusters with OVN IPsec (#1327) 2026-10-29 11:17:22 +11:00
Josh Siviour ddadfa2812 chore: remove unused e2e suite-selection manifest and drift lints (#1331)
The targeted suite selector (select-suites) has not run in CI since its shadow-mode step was removed in #556.
The service-interfaces manifest, manifest-check and manifest-lint existed only to feed it, so they go too.
Also drops the now-meaningless e2e:allow-create markers, keeping their rationale as plain comments.
2026-10-29 10:04:31 +10:00
Josh Siviour 5923143062 feat: set X-Forwarded-For, -Proto and -Port on ALB requests (#1330)
* feat: render X-Forwarded-For, -Proto and -Port headers in ALB HAProxy frontends

Append joins every client X-Forwarded-For line and adds the client address as AWS does, preserve and remove follow the load balancer attribute, and Proto and Port are always overwritten.

* feat: validate the ALB forwarding attributes and re-render config when they change

ModifyLoadBalancerAttributes rejects bad xff_header_processing.mode and xff_client_port.enabled values with ValidationError before storing anything, and regenerates the stored config when either key changes.

* test: assert the ALB X-Forwarded headers and the remove mode in the lb e2e

The app instances echo the request headers on /__headers. The internal ALB is driven from the probe client and the HTTPS ALB from the runner, covering the caller address, a client value, spoofed Proto and Port, mode remove and a rejected mode.

* test: let the lb probe client trigger accept a null header list

* docs: document the ALB forwarding headers and their attributes
2026-10-29 09:33:39 +10:00
Julian Sommer 46ca3dfba1 chore: comment 2026-10-29 09:13:32 +10:00
513 changed files with 39087 additions and 12025 deletions
+2 -2
View File
@@ -102,9 +102,9 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
- name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot)
- name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
if: steps.version.outputs.exists != 'true'
run: cd spinifex && make build-microvm-image
run: cd spinifex && make build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image
- name: Build amd64 Tarball
if: steps.version.outputs.exists != 'true'
+14
View File
@@ -536,6 +536,13 @@ jobs:
log-dir: ${{ env.ARTIFACT_DIR }}
title: "E2E failure analysis"
# The WAN capture is most of the artifact and only matters when a run
# failed. Anything short of a clean pass (suite rc, a failed setup or
# collect step, a missing rc) keeps it.
- name: Drop the packet capture from a passing single-node run
if: always() && steps.e2e.outputs.rc == '0' && job.status == 'success'
run: rm -f "${ARTIFACT_DIR}"/net-diag/wan.pcap
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
@@ -990,6 +997,13 @@ jobs:
log-dir: ${{ env.ARTIFACT_DIR }}
title: "Multi-Node E2E failure analysis"
# The WAN captures are 90% of the artifact and only matter when a run
# failed. Anything short of a clean pass (suite rc, a failed setup or
# collect step, a missing rc) keeps them.
- name: Drop the packet captures from a passing multi-node run
if: always() && steps.e2e.outputs.rc == '0' && job.status == 'success'
run: rm -f "${ARTIFACT_DIR}"/net-diag-node*/wan.pcap
- uses: actions/upload-artifact@v7.0.2
if: always()
with:
+2 -2
View File
@@ -57,8 +57,8 @@ jobs:
go-version-file: spinifex/go.mod
cache: false
- name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot)
run: cd spinifex && make build-microvm-image
- name: Build microVM artifacts (lb-agent + kernel + initramfs + qboot + SQS store and worker images)
run: cd spinifex && make build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image
- name: Build amd64 Tarball
env:
-3
View File
@@ -119,9 +119,6 @@ jobs:
- name: Build Spinifex
run: make -C spinifex build
- name: Manifest check + drift lint
run: make -C spinifex manifest-check manifest-lint
- name: Run Tests with Coverage
env:
SPINIFEX_REQUIRE_CONFORMANCE_TOOLS: "1"
+6
View File
@@ -21,6 +21,12 @@ build/microvm/vmlinuz
build/microvm/initramfs.cpio.gz
build/microvm/qboot.rom
build/microvm/qboot.LICENSE
build/microvm/sqs-store/vmlinuz
build/microvm/sqs-store/initramfs.cpio.gz
build/microvm/sqs-store/.stage.*/
build/microvm/sqs-worker/vmlinuz
build/microvm/sqs-worker/initramfs.cpio.gz
build/microvm/sqs-worker/.stage.*/
build/qboot/
!spinifex/runtime/roles/spinifexui/frontend/dist/
tmp/
+50 -22
View File
@@ -140,6 +140,38 @@ $(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
bin/lb-agent:
$(MAKE) build-lb-agent
# The SQS store microVM image: a separate artifact from the LB image above, not
# installed or released by any target yet.
SQS_STORE_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-store
SQS_STORE_MICROVM_ARTIFACTS := $(SQS_STORE_MICROVM_OUT_DIR)/vmlinuz $(SQS_STORE_MICROVM_OUT_DIR)/initramfs.cpio.gz
SQS_STORE_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_STORE_MICROVM_OUT_DIR)/init.sh $(SQS_STORE_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_STORE_MICROVM_OUT_DIR)/,sqs-store-lib.sh sqs-store-prepare sqs-store-run sqs-store-stop) bin/nats-server
$(SQS_STORE_MICROVM_ARTIFACTS) &: $(SQS_STORE_MICROVM_INPUTS)
./scripts/build-sqs-store-microvm-image.sh
# The dedicated SQS store server, built from the go.mod-pinned module so the
# image ships exactly the reviewed version. Static for the musl guest.
bin/nats-server: go.mod go.sum
@mkdir -p bin
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ github.com/nats-io/nats-server/v2
build-sqs-store-microvm-image: $(SQS_STORE_MICROVM_ARTIFACTS) ## Build the SQS store microVM kernel and initramfs
# The shared SQS worker microVM image: the store image's /init and kernel with
# the worker binary instead of nats-server. Not installed or released yet.
SQS_WORKER_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-worker
SQS_WORKER_MICROVM_ARTIFACTS := $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz
SQS_WORKER_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_STORE_MICROVM_OUT_DIR)/init.sh $(SQS_WORKER_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_WORKER_MICROVM_OUT_DIR)/,sqs-worker-lib.sh sqs-worker-prepare sqs-worker-ready sqs-worker-stop) bin/sqs-worker
$(SQS_WORKER_MICROVM_ARTIFACTS) &: $(SQS_WORKER_MICROVM_INPUTS)
SQS_MICROVM_PROFILE=worker ./scripts/build-sqs-store-microvm-image.sh
bin/sqs-worker: go.mod go.sum $(shell find cmd/sqs-worker spinifex/agents/sqs -name '*.go' ! -name '*_test.go')
@mkdir -p bin
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/sqs-worker
build-sqs-worker-microvm-image: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Build the shared SQS worker microVM kernel and initramfs
# qboot is the microvm firmware. The source tarball ships with each release
# because qboot is GPL-2.0.
QBOOT_COMMIT := $(shell sed -n 's/^QBOOT_COMMIT="\(.*\)"$$/\1/p' scripts/build-qboot.sh)
@@ -151,6 +183,18 @@ $(QBOOT_ARTIFACTS) &: scripts/build-qboot.sh
build-microvm-image: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Build microVM kernel, initramfs and qboot firmware (incremental — skips when up to date)
install-sqs-store-microvm: $(SQS_STORE_MICROVM_ARTIFACTS) ## Install the SQS store microVM image to /usr/share/spinifex/microvm/sqs-store/
sudo install -d /usr/share/spinifex/microvm/sqs-store
sudo rm -f /usr/share/spinifex/microvm/sqs-store/initramfs.cpio.gz
sudo install -m 0644 $(SQS_STORE_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-store/vmlinuz
sudo install -m 0644 $(SQS_STORE_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-store/initramfs.cpio.gz
install-sqs-worker-microvm: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Install the SQS worker microVM image to /usr/share/spinifex/microvm/sqs-worker/
sudo install -d /usr/share/spinifex/microvm/sqs-worker
sudo rm -f /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-worker/vmlinuz
sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
install-microvm: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
sudo install -d /usr/share/spinifex/microvm
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
@@ -158,15 +202,14 @@ install-microvm: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Install microVM arti
sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom /usr/share/spinifex/microvm/qboot.rom
sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.LICENSE /usr/share/spinifex/microvm/qboot.LICENSE
# the pre-commit gate: manifest checks, lint, vuln, and the unit and e2e-harness tiers.
# the pre-commit gate: lint, vuln, and the unit and e2e-harness tiers.
# integration and race tests skipped to keep quick, they run in CI.
#
# -j2 overlaps the only two heavy gates, lint and test-cover: alone each leaves
# most of the machine idle, so pairing them cuts a cold run by about a quarter.
# -Otarget keeps each gate's output in one block instead of interleaving them.
# manifest-check manifest-lint test-package-check removed, can add back later
preflight:
@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts manifest-check manifest-lint
@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts
@echo -e "\n ✅ Preflight passed — safe to commit."
# Shell suites + shellcheck for build/scripts/, the systemd-unit helpers that
@@ -178,7 +221,7 @@ preflight:
# reason: between them they install the software, wire OVN, erase the state and
# remove the install, so a shellcheck warning in one is an outage or a wrong
# deletion, not a style note.
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh scripts/lib/stage-sqs-store-microvm.sh
test-build-scripts:
@echo -e "\n....Running build/scripts/**/*_test.sh...."
@@ -240,23 +283,6 @@ test-segscan-oracle:
@echo -e "\n....Running segscan storage oracle test...."
$(_Q)LOG_IGNORE=1 go test -tags=integration,segscanoracle -timeout 120s ./tests/integration/... -run TestSegscanOracle $(_RACEQ)
# Validate docs/service-interfaces.yaml. Schema check + cross-reference
# of services/suites/fixtures + on-disk path existence.
manifest-check:
@echo -e "\n....Checking service-interfaces.yaml...."
@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
# Drift guards: direct-create fixture lint + NATS subject lint,
# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
# drift beyond the baseline.
manifest-lint:
@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
# Accept current drift into the baseline. Run after an intentional change.
manifest-lint-update:
@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
# Run unit tests
test:
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -469,6 +495,8 @@ distro-amd64:
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
fi
./scripts/lib/stage-sqs-store-microvm.sh $(SQS_STORE_MICROVM_OUT_DIR) dist/amd64
./scripts/lib/stage-sqs-store-microvm.sh $(SQS_WORKER_MICROVM_OUT_DIR) dist/amd64 sqs-worker
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
@@ -487,7 +515,7 @@ distro-arm64:
distro-clean:
rm -rf dist/
.PHONY: test-package-check build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
.PHONY: test-package-check build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image build-sqs-store-microvm-image build-sqs-worker-microvm-image install-microvm install-sqs-store-microvm install-sqs-worker-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck nilaway \
+3 -1
View File
@@ -54,7 +54,7 @@ For Terraform, start from the [example workbooks](https://docs.mulgadc.com/docs/
## Services
The platform serves **392 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
The platform serves **405 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
| Service | What it is | Built on | Status |
| --- | --- | --- | --- |
@@ -68,8 +68,10 @@ The platform serves **392 AWS API operations** across the services below. See ex
| **ECR** | Container registry | Predastore, deduplicated per account | Available |
| **ECS** | Container service | Spinifex instances | Available |
| **RDS** | Databases | PostgreSQL and MariaDB, with automated backups | Available |
| **Route 53** | DNS | [Northstar](https://github.com/mulgadc/northstar): authoritative DNS with hosted zones | Available |
| **Bedrock** | AI deployment | vLLM for text generation and Hugging Face TEI for embeddings and reranking | Q4 2026 |
| **Lambda** | Serverless functions | | Q1 2027 |
| **SQS** | Message queues | | Q1 2027 |
Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
+9 -32
View File
@@ -29,9 +29,15 @@ var allowlist = []allowance{
{From: "agents/rds/agent", To: "domains/rds/instance", Debt: "agents/rds/agent (dependency, narrowed)", Reason: "command protocol, wire and record types; needs a guest/controller wire contract"},
{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/route53/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/acm/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/ecr/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/route53/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/elbv2/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/rds/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/ecs/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/eks/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/iam/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "cmd/aws-model-coverage", To: "domains/sts/awsapi", Debt: "cmd/aws-model-coverage", Reason: "reads declared service inventories; named exception, must not construct a registration"},
{From: "runtime/daemon", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
@@ -53,9 +59,6 @@ var allowlist = []allowance{
{From: "domains/ec2/awsapi/instance", To: "runtime/compute/cache", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "node liveness and VM state; needs an EC2-owned projection"},
{From: "domains/ec2/awsapi/instance", To: "runtime/compute/vm", Debt: "domains/ec2/awsapi/instance (dependency)", Reason: "node liveness and VM state; needs an EC2-owned projection"},
{From: "domains/ec2/awsapi/spotinstance", To: "domains/admission/quota", Debt: "domains/ec2/awsapi/spotinstance (dependency)", Reason: "RequestSpotInstances.go uses IAMService and quota.Service"},
{From: "domains/ec2/awsapi/spotinstance", To: "domains/iam", Debt: "domains/ec2/awsapi/spotinstance (dependency)", Reason: "RequestSpotInstances.go uses IAMService and quota.Service"},
{From: "domains/ec2/eip", To: "domains/network/topology", Debt: "EC2 → network implementation", Reason: "EC2 reads network implementation; clears with the EC2-owned projection"},
{From: "domains/ec2/guestmetadata", To: "domains/dns", Debt: "Guest metadata: the cross-domain and runtime imports above", Reason: "guest-metadata relocation residual; needs capabilities, projections or contracts"},
@@ -92,42 +95,18 @@ var allowlist = []allowance{
{From: "domains/ochre", To: "domains/ec2/systeminstance", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
{From: "domains/ochre", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "secret encrypt/decrypt helpers; needs a shared crypto boundary"},
{From: "domains/ochre", To: "domains/network/systemvpc", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
{From: "domains/ochre", To: "ingress/aws/bodyscope", Debt: "domains/ochre (action adapters)", Reason: "Bedrock action adapters still live in the Ochre package; they move to domains/ochre/awsapi"},
{From: "domains/ochre", To: "runtime/compute/gpu", Debt: "Domain → runtime", Reason: "GPU runtime state read directly; needs a domain-owned projection"},
{From: "domains/ochre/vector", To: "domains/iam", Debt: "Cross-domain", Reason: "consumes another domain's implementation rather than a capability"},
{From: "gateway", To: "domains/ec2/awsapi", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/account", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/capacityreservation", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/eigw", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/eip", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/idem", Debt: "gateway/ec2.go (registration; ec2_idempotency.go)", Reason: "dispatch-time idempotency wrapper; moves with the EC2 dispatch map"},
{From: "gateway", To: "domains/ec2/awsapi/igw", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/image", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/instance", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/key", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/launchtemplate", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/natgw", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/placementgroup", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/routetable", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/snapshot", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/spotinstance", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/tags", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/volume", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/vpc", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ec2/awsapi/zone", Debt: "gateway/ec2.go (registration)", Reason: "EC2 dispatch map stays in gateway until the ingress registration seam"},
{From: "domains/rds/awsapi", To: "domains/ec2/awsapi/instance", Debt: "Instance adapter consumers outside EC2", Reason: "enginecatalog.go calls the instance adapter; needs an EC2 capability"},
{From: "domains/sts/awsapi", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "GetCallerIdentity takes IAMService; needs an IAM capability"},
{From: "gateway", To: "domains/eks/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/iam/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/rds/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/sts/awsapi", Debt: "gateway (service registration)", Reason: "IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/ecs/awsapi", Debt: "gateway (service registration)", Reason: "ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
{From: "gateway", To: "domains/elbv2/awsapi", Debt: "gateway (service registration)", Reason: "ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
{From: "domains/ecs", To: "domains/ec2/ebs/metadata", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/ecs", To: "domains/ec2/eip", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
{From: "domains/ecs", To: "domains/ec2/instancetypes", Debt: "close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)", Reason: "legacy ECS calls another domain's implementation as an internal API"},
@@ -160,6 +139,4 @@ var allowlist = []allowance{
{From: "domains/rds/instance", To: "runtime/compute/vm", Debt: "close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)", Reason: "legacy RDS calls another domain's or runtime implementation directly"},
{From: "domains/sts", To: "domains/iam", Debt: "IAM implementation consumers in STS, Ochre and ELBv2", Reason: "STS reads IAM service, trust-policy and OIDC-provider state; needs an IAM capability"},
{From: "operator/cli", To: "gateway", Debt: "operator/cli (dependency)", Reason: "command tree imports implementations; needs public capabilities"},
}
-1
View File
@@ -11,7 +11,6 @@ import (
// completed move whose line must be deleted.
var frozenLegacyPackages = []string{
"gateway",
"gateway/spx",
}
// retiredLegacyRoots were legacy roots whose every package has moved to its
+96
View File
@@ -0,0 +1,96 @@
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devtmpfs dev /dev
# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
# already enumerated by the time PID 1 runs — no modprobe needed.
# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
#
# modprobe needs a usable modules.dep; when the image is built on a host whose
# depmod can't produce one, busybox modprobe silently no-ops and the module
# never loads — fw_cfg then never appears and init panics. insmod by explicit
# path needs no modules.dep, so fall back to it. Handles .ko and .ko.gz.
load_mod() {
modprobe "$1" 2>/dev/null && return 0
for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
insmod "$ko" 2>/dev/null && return 0
case "$ko" in
*.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
&& insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
esac
done
return 1
}
load_mod qemu_fw_cfg || load_mod fw_cfg_sysfs || true
# virtio_net depends on net_failover, which depends on failover. The load_mod
# insmod fallback loads a single .ko with no dependency resolution, and
# modules.dep is absent when the image is built on a kmod-less host (depmod
# skipped), so modprobe can't resolve the chain either. Load it explicitly
# bottom-up so the NIC comes up regardless of how the image was built —
# without this the agent has no network and the LB hangs in provisioning.
load_mod failover || true
load_mod net_failover || true
load_mod virtio_net || true
NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
i=0
while [ "$i" -lt 25 ]; do
[ -f "$NETCFG" ] && break
sleep 0.04
i=$((i + 1))
done
if [ ! -f "$NETCFG" ]; then
# Dump host/guest fw_cfg state to the serial console before init exits so a
# "netcfg not available" panic is diagnosable from the captured console log
# (module-not-loaded vs qemu-not-exposing-by_name vs wrong machine type).
echo "[init] FATAL: fw_cfg netcfg not available" >&2
echo "[init] dmi: vendor=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null) product=$(cat /sys/class/dmi/id/product_name 2>/dev/null) bios=$(cat /sys/class/dmi/id/bios_version 2>/dev/null)" >&2
echo "[init] qemu_fw_cfg dir present: $([ -d /sys/firmware/qemu_fw_cfg ] && echo yes || echo NO)" >&2
echo "[init] modules: $(lsmod 2>/dev/null | awk 'NR>1{print $1}' | tr '\n' ',')" >&2
echo "[init] by_name listing:" >&2
ls -la /sys/firmware/qemu_fw_cfg/by_name/ 2>&1 | sed 's/^/[init] /' >&2
ls -la /sys/firmware/qemu_fw_cfg/by_name/opt/ 2>&1 | sed 's/^/[init] /' >&2
echo "[init] dmesg fw_cfg:" >&2
dmesg 2>/dev/null | grep -i fw_cfg | sed 's/^/[init] /' >&2
exit 1
fi
. "$NETCFG"
for n in 0 1 2 3 4 5; do
eval mac=\$NIC${n}_MAC
[ -z "$mac" ] && continue
iface=""
for attempt in 1 2 3 4 5; do
iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
[ -n "$iface" ] && break
sleep 0.02
done
if [ -z "$iface" ]; then
echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
continue
fi
eval cidr=\$NIC${n}_CIDR
eval gw=\$NIC${n}_GW
eval default=\$NIC${n}_DEFAULT
eval rdst=\$NIC${n}_ROUTE_DST
eval rvia=\$NIC${n}_ROUTE_VIA
ip link set "$iface" up
[ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
[ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
[ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
done
# The store image takes no lb-agent environment. A CA bundle is optional here:
# nothing in this image dials out yet.
CA_CERT=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw
mkdir -p /etc/ssl/certs
[ -f "$CA_CERT" ] && cp "$CA_CERT" /etc/ssl/certs/ca-certificates.crt
exec /sbin/init
+10
View File
@@ -0,0 +1,10 @@
# The power button loads here, not in /init: busybox init has its SIGUSR2
# (poweroff) handler installed before sysinit runs, so a press can never reach
# the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
::sysinit:/sbin/modprobe tiny_power_button power_signal=12
::sysinit:/usr/local/lib/sqs-store/sqs-store-prepare
::respawn:/usr/local/lib/sqs-store/sqs-store-run
::ctrlaltdel:/sbin/reboot
# One action: the store stops and its disk unmounts first, and the generic
# umount -a -r runs only if the store is gone.
::shutdown:/usr/local/lib/sqs-store/sqs-store-stop
+216
View File
@@ -0,0 +1,216 @@
# sqs-store-lib.sh — shared by the SQS store guest's prepare, run and stop
# steps. Sourced, never executed. Every path and command is overridable so the
# host test can drive the same code against a fake fw_cfg, sysfs and tools.
#
# Guest contract, read from fw_cfg under opt/spinifex/sqs-store/:
# data-serial virtio-blk serial of the already-formatted ext4 data disk
# listen-addr IPv4 address on a configured guest NIC; the only listener
# client-nkey NATS user NKey public key allowed to connect; never a seed
# Anything absent, malformed or ambiguous refuses the store. Nothing formats.
SQS_FWCFG=${SQS_FWCFG:-/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/sqs-store}
SQS_SYSBLOCK=${SQS_SYSBLOCK:-/sys/block}
SQS_DEVDIR=${SQS_DEVDIR:-/dev}
# The host test cannot create block nodes, so it relaxes this to -e.
SQS_NODE_TEST=${SQS_NODE_TEST:--b}
SQS_RUN=${SQS_RUN:-/run/sqs-store}
SQS_DATA=${SQS_DATA:-/data}
SQS_CONSOLE=${SQS_CONSOLE:-/dev/console}
SQS_MOUNTS=${SQS_MOUNTS:-/proc/mounts}
SQS_PORT=${SQS_PORT:-4222}
SQS_STOP_BUDGET_TENTHS=${SQS_STOP_BUDGET_TENTHS:-120}
NATS_BIN=${NATS_BIN:-/usr/local/bin/nats-server}
E2FSCK=${E2FSCK:-e2fsck}
MOUNT=${MOUNT:-mount}
UMOUNT=${UMOUNT:-umount}
IP=${IP:-ip}
sqs_log() { echo "[sqs-store] $*" >> "$SQS_CONSOLE"; }
sqs_fatal() { sqs_log "FATAL: $*"; return 1; }
# sqs_fwcfg <name>: print one fw_cfg value, refusing a missing, empty, oversized
# or multi-token file. The value is not echoed on failure.
sqs_fwcfg() {
f="$SQS_FWCFG/$1/raw"
[ -f "$f" ] || { sqs_fatal "fw_cfg $1 not supplied"; return 1; }
[ "$(wc -c < "$f")" -le 128 ] || { sqs_fatal "fw_cfg $1 is oversized"; return 1; }
v=$(cat "$f")
case "$v" in
"" | *[[:space:]]*) sqs_fatal "fw_cfg $1 must be exactly one value"; return 1 ;;
esac
printf '%s' "$v"
}
# A volume serial is the volume ID with dashes stripped: vol + 17 hex.
sqs_valid_serial() {
[ "${#1}" -eq 20 ] || return 1
case "$1" in vol*) ;; *) return 1 ;; esac
case "${1#vol}" in *[!0-9a-f]*) return 1 ;; esac
}
# A NATS user public NKey: "U" then 55 base32 characters. The checksum is
# left to the pinned server's own config test.
sqs_valid_user_nkey() {
[ "${#1}" -eq 56 ] || return 1
case "$1" in U*) ;; *) return 1 ;; esac
case "$1" in *[!A-Z2-7]*) return 1 ;; esac
}
sqs_valid_ipv4() {
case "$1" in *[!0-9.]* | .* | *. | *..*) return 1 ;; esac
old_ifs=$IFS; IFS=.
# shellcheck disable=SC2086 # split on dots on purpose
set -- $1
IFS=$old_ifs
[ "$#" -eq 4 ] || return 1
for o in "$@"; do
[ "${#o}" -le 3 ] && [ "$o" -le 255 ] || return 1
case "$o" in 0?*) return 1 ;; esac
done
}
# sqs_find_disk <serial>: print the /dev node of the one block device whose
# sysfs serial matches exactly. Zero or several matches refuse.
sqs_find_disk() {
found="" n=0
for b in "$SQS_SYSBLOCK"/*; do
[ -r "$b/serial" ] || continue
[ "$(cat "$b/serial")" = "$1" ] || continue
found=${b##*/}
n=$((n + 1))
done
[ "$n" -eq 1 ] || { sqs_fatal "expected one disk with serial $1, found $n"; return 1; }
test "$SQS_NODE_TEST" "$SQS_DEVDIR/$found" || { sqs_fatal "disk $found has no block device node"; return 1; }
printf '%s' "$SQS_DEVDIR/$found"
}
# sqs_has_addr <ipv4>: the address is assigned to a non-loopback interface.
sqs_has_addr() {
"$IP" -o -4 addr show 2>/dev/null \
| awk -v a="$1" '$2 != "lo" { split($4, p, "/"); if (p[1] == a) f = 1 } END { exit !f }'
}
# sqs_alive <pid>: running or stopped, not exited (a zombie counts as exited).
sqs_alive() { [ -r "/proc/$1/status" ] && ! grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
# sqs_starttime <pid>: kernel start time (stat field 22). It survives exec, and
# pairing it with the pid reduces PID-reuse risk; without a pidfd the check and
# the signal stay separate steps, so it is not an absolute guarantee.
sqs_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20; }
# sqs_same <pid> <starttime>: the recorded process is still alive.
sqs_same() { [ -n "$2" ] && sqs_alive "$1" && [ "$(sqs_starttime "$1")" = "$2" ]; }
# sqs_prepare: validate the contract, check and mount the data disk, write the
# server config and mark the store ready. Returns non-zero, unready, on any refusal.
sqs_prepare() {
rm -f "$SQS_RUN/ready"
mkdir -p "$SQS_RUN"
serial=$(sqs_fwcfg data-serial) || return 1
sqs_valid_serial "$serial" || { sqs_fatal "data-serial is not a volume serial"; return 1; }
addr=$(sqs_fwcfg listen-addr) || return 1
sqs_valid_ipv4 "$addr" || { sqs_fatal "listen-addr is not an IPv4 address"; return 1; }
nkey=$(sqs_fwcfg client-nkey) || return 1
sqs_valid_user_nkey "$nkey" || { sqs_fatal "client-nkey is not a user public NKey"; return 1; }
sqs_has_addr "$addr" || { sqs_fatal "listen-addr $addr is not on any guest NIC"; return 1; }
# The server checks the config, including the NKey checksum, before the
# disk is touched.
umask 077
cat > "$SQS_RUN/nats.conf" <<EOF
listen: "$addr:$SQS_PORT"
server_name: "sqs-store"
jetstream {
store_dir: "$SQS_DATA"
sync_interval: always
}
authorization {
users = [ { nkey: "$nkey" } ]
}
EOF
if ! "$NATS_BIN" -t -c "$SQS_RUN/nats.conf" > "$SQS_RUN/config-test.out" 2>&1; then
sqs_fatal "server config test failed: $(sed 's/U[A-Z2-7]\{55\}/<nkey>/g' "$SQS_RUN/config-test.out" | tr '\n' ' ')"
return 1
fi
dev=$(sqs_find_disk "$serial") || return 1
rc=0
"$E2FSCK" -p "$dev" > "$SQS_RUN/fsck.out" 2>&1 || rc=$?
sqs_log "e2fsck -p $dev rc=$rc"
# 0 clean, 1 corrected: proceed. Anything higher, including a disk that is
# not ext4 at all, refuses; the guest never formats.
[ "$rc" -le 1 ] || { sqs_fatal "e2fsck refused $dev: $(tr '\n' ' ' < "$SQS_RUN/fsck.out")"; return 1; }
mkdir -p "$SQS_DATA"
"$MOUNT" -t ext4 -o nodev,nosuid,noexec "$dev" "$SQS_DATA" || { sqs_fatal "mount $dev failed"; return 1; }
: > "$SQS_RUN/ready"
sqs_log "prepared: $dev mounted at $SQS_DATA, config written for $addr:$SQS_PORT; server not started yet"
}
# sqs_run: the respawned supervisor body. Its identity is recorded before the
# stopping check: a stop that misses it set stopping first, so this parks; a
# stop that reads it signals this process whether or not it has exec'd yet.
sqs_run() {
mkdir -p "$SQS_RUN"
echo "$$ $(sqs_starttime $$)" > "$SQS_RUN/pid"
if [ -e "$SQS_RUN/stopping" ]; then
sqs_log "run: shutdown in progress, not starting"
return 2
fi
if [ ! -e "$SQS_RUN/ready" ]; then
sqs_log "run: store not prepared, not starting"
return 2
fi
sqs_log "run: starting pid=$$"
# ADR-0011:S11 bounds the store's Go heap; the guest must be larger.
GOMEMLIMIT=512MiB exec "$NATS_BIN" -c "$SQS_RUN/nats.conf"
}
# sqs_stop: first shutdown action. Block any respawn, SIGTERM the recorded run
# process (a server, or a start not yet exec'd), SIGKILL only after the budget,
# then sync and unmount. Each signal rechecks the identity; a survivor keeps the disk mounted.
sqs_stop() {
mkdir -p "$SQS_RUN"
: > "$SQS_RUN/stopping"
sqs_log "stop: begin"
rec=$(cat "$SQS_RUN/pid" 2>/dev/null || true)
pid=${rec%% *} st=${rec#* }
[ "$st" != "$rec" ] || st=""
if [ -n "$pid" ] && sqs_same "$pid" "$st"; then
kill -TERM "$pid" 2>/dev/null || true
sqs_log "stop: SIGTERM sent to run pid=$pid"
i=0
while sqs_same "$pid" "$st" && [ "$i" -lt "$SQS_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
if sqs_same "$pid" "$st"; then
sqs_log "stop: run pid=$pid still running after $((SQS_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
kill -KILL "$pid" 2>/dev/null || true
j=0
while sqs_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
fi
if sqs_same "$pid" "$st"; then
sqs_log "stop: run pid=$pid STILL ALIVE, leaving $SQS_DATA mounted"
return 1
fi
sqs_log "stop: store exited after $((i / 10)).$((i % 10))s"
else
sqs_log "stop: no running store"
fi
sync
if grep -q " $SQS_DATA " "$SQS_MOUNTS" 2>/dev/null; then
rc=0
"$UMOUNT" "$SQS_DATA" || rc=$?
sqs_log "stop: umount $SQS_DATA rc=$rc"
fi
}
# sqs_shutdown: the only shutdown action. The generic read-only remount runs
# only after the store stopped; a surviving writer keeps every mount as it is.
sqs_shutdown() {
if ! sqs_stop; then
sync
sqs_log "shutdown: store survived, skipping umount -a -r"
return 1
fi
rc=0
"$UMOUNT" -a -r || rc=$?
echo "[init] shutdown: umount -a -r rc=$rc" >> "$SQS_CONSOLE"
}
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
# sysinit: load the disk drivers, then validate the guest contract, check and
# mount the data disk and write the server config. A refusal leaves the store
# unprepared, so the run step never starts it.
. /usr/local/lib/sqs-store/sqs-store-lib.sh
modprobe virtio_blk 2>/dev/null || true
modprobe ext4 2>/dev/null || true
sqs_prepare || sqs_log "store refused; it will not start"
exit 0
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
# Respawned by init. A store that must not run parks here instead of exiting,
# so init does not respawn it in a tight loop.
. /usr/local/lib/sqs-store/sqs-store-lib.sh
sqs_run
exec sleep 2147483647
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
# The only shutdown action: stop the store and unmount its disk, then run the
# generic read-only remount only if the store is gone.
. /usr/local/lib/sqs-store/sqs-store-lib.sh
sqs_shutdown
+8
View File
@@ -0,0 +1,8 @@
# The power button loads here, not in /init: busybox init has its SIGUSR2
# (poweroff) handler installed before sysinit runs, so a press can never reach
# the pre-exec shell PID 1. A press before this line is lost; the host re-sends.
::sysinit:/sbin/modprobe tiny_power_button power_signal=12
::sysinit:/usr/local/lib/sqs-worker/sqs-worker-prepare
::respawn:/usr/local/lib/sqs-worker/sqs-worker-ready
::ctrlaltdel:/sbin/reboot
::shutdown:/usr/local/lib/sqs-worker/sqs-worker-stop
+115
View File
@@ -0,0 +1,115 @@
# sqs-worker-lib.sh — shared by the SQS worker guest's readiness and stop
# steps. Sourced, never executed. Every path and command is overridable so the
# host test can drive the same code against test-owned processes and stub tools.
#
# store-ready exists only while the running watcher's last check passed. A
# check proves the store accepted the worker key, not the store server's
# identity, and a store failure is noticed up to one interval plus call time late.
SQSW_RUN=${SQSW_RUN:-/run/sqs-worker}
SQSW_DATA=${SQSW_DATA:-/data}
SQSW_CONSOLE=${SQSW_CONSOLE:-/dev/console}
SQSW_MOUNTS=${SQSW_MOUNTS:-/proc/mounts}
SQSW_BIN=${SQSW_BIN:-/usr/local/bin/sqs-worker}
SQSW_STOP_BUDGET_TENTHS=${SQSW_STOP_BUDGET_TENTHS:-100}
SQSW_RESPAWN_GAP=${SQSW_RESPAWN_GAP:-5}
SQSW_PARK_INTERVAL=${SQSW_PARK_INTERVAL:-30}
UMOUNT=${UMOUNT:-umount}
sqsw_log() { echo "[sqs-worker] $*" >> "$SQSW_CONSOLE"; }
# sqsw_alive <pid>: running or stopped, not exited (a zombie counts as exited).
sqsw_alive() { [ -r "/proc/$1/status" ] && ! grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
# sqsw_starttime <pid>: kernel start time (stat field 22). It survives exec, and
# pairing it with the pid reduces PID-reuse risk; it is not an absolute guarantee.
sqsw_starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | cut -d' ' -f20; }
# sqsw_same <pid> <starttime>: the recorded process is still alive.
sqsw_same() { [ -n "$2" ] && sqsw_alive "$1" && [ "$(sqsw_starttime "$1")" = "$2" ]; }
# sqsw_run: the respawned readiness body. A predecessor that was killed or
# crashed could not remove its marker, so every start removes it first. Its
# identity is recorded before the stopping check, as the store's run step does.
sqsw_run() {
mkdir -p "$SQSW_RUN"
echo "$$ $(sqsw_starttime $$)" > "$SQSW_RUN/ready.pid"
if [ -e "$SQSW_RUN/store-ready" ]; then
rm -f "$SQSW_RUN/store-ready"
sqsw_log "ready: removed a store-ready marker left by an earlier watcher"
fi
# Returning here would make init respawn and log at once, forever. Park
# instead, logging once: shutdown stops this recorded process, and a key
# that appears later lets the watcher start.
if [ -e "$SQSW_RUN/stopping" ] || [ ! -s "$SQSW_RUN/public-key" ]; then
if [ -e "$SQSW_RUN/stopping" ]; then
sqsw_log "ready: shutdown in progress, not starting"
else
sqsw_log "ready: no worker key; readiness not started, waiting"
fi
while [ -e "$SQSW_RUN/stopping" ] || [ ! -s "$SQSW_RUN/public-key" ]; do
sleep "$SQSW_PARK_INTERVAL"
done
fi
# Init respawns at once; a watcher that keeps exiting restarts at most
# once per gap instead of in a tight loop.
now=$(date +%s)
last=$(cat "$SQSW_RUN/ready.last" 2>/dev/null || echo 0)
wait=$((last + SQSW_RESPAWN_GAP - now))
if [ "$wait" -gt 0 ] && [ "$wait" -le "$SQSW_RESPAWN_GAP" ]; then sleep "$wait"; fi
date +%s > "$SQSW_RUN/ready.last"
sqsw_log "ready: starting watcher pid=$$"
exec "$SQSW_BIN" ready >> "$SQSW_CONSOLE" 2>&1
}
# sqsw_stop: block any respawn, drop the marker, SIGTERM the recorded watcher,
# SIGKILL only after the budget, and unmount the worker disk only once the
# watcher is confirmed gone. A survivor keeps the disk mounted.
sqsw_stop() {
mkdir -p "$SQSW_RUN"
: > "$SQSW_RUN/stopping"
rm -f "$SQSW_RUN/store-ready"
rec=$(cat "$SQSW_RUN/ready.pid" 2>/dev/null || true)
pid=${rec%% *} st=${rec#* }
[ "$st" != "$rec" ] || st=""
if [ -n "$pid" ] && sqsw_same "$pid" "$st"; then
kill -TERM "$pid" 2>/dev/null || true
sqsw_log "stop: SIGTERM sent to readiness watcher pid=$pid"
i=0
while sqsw_same "$pid" "$st" && [ "$i" -lt "$SQSW_STOP_BUDGET_TENTHS" ]; do sleep 0.1; i=$((i + 1)); done
if sqsw_same "$pid" "$st"; then
sqsw_log "stop: readiness watcher pid=$pid still running after $((SQSW_STOP_BUDGET_TENTHS / 10))s, SIGKILL"
kill -KILL "$pid" 2>/dev/null || true
j=0
while sqsw_same "$pid" "$st" && [ "$j" -lt 30 ]; do sleep 0.1; j=$((j + 1)); done
fi
if sqsw_same "$pid" "$st"; then
rm -f "$SQSW_RUN/store-ready"
sqsw_log "stop: readiness watcher pid=$pid STILL ALIVE, leaving $SQSW_DATA mounted"
return 1
fi
sqsw_log "stop: readiness watcher pid=$pid exited"
else
sqsw_log "stop: no running readiness watcher"
fi
rm -f "$SQSW_RUN/store-ready" "$SQSW_RUN/ready.pid"
sync
if grep -q " $SQSW_DATA " "$SQSW_MOUNTS" 2>/dev/null; then
rc=0
"$UMOUNT" "$SQSW_DATA" || rc=$?
sqsw_log "stop: umount $SQSW_DATA rc=$rc"
fi
}
# sqsw_shutdown: the only shutdown action. The generic read-only remount runs
# only after the watcher stopped; a survivor keeps every mount as it is.
sqsw_shutdown() {
if ! sqsw_stop; then
sync
sqsw_log "shutdown: readiness watcher survived, skipping umount -a -r"
return 1
fi
rc=0
"$UMOUNT" -a -r || rc=$?
echo "[init] shutdown: umount -a -r rc=$rc" >> "$SQSW_CONSOLE"
}
+8
View File
@@ -0,0 +1,8 @@
#!/bin/sh
# sysinit: load the disk drivers, then select, check and mount the worker disk
# and load or create the worker's store key on it. Any refusal is logged; the
# key is never read or created on a disk that is not proven mounted.
modprobe virtio_blk 2>/dev/null || true
modprobe ext4 2>/dev/null || true
/usr/local/bin/sqs-worker boot >> /dev/console 2>&1 || echo "[sqs-worker] boot refused" >> /dev/console
exit 0
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
# Respawned by init. A watcher that must not run parks here instead of exiting,
# so init does not respawn it in a tight loop.
. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
sqsw_run
exec sleep 2147483647
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
# The only shutdown action: stop the readiness watcher and unmount the worker
# disk, then run the generic read-only remount only if the watcher is gone.
. /usr/local/lib/sqs-worker/sqs-worker-lib.sh
sqsw_shutdown
-57
View File
@@ -1,57 +0,0 @@
# spinifex-unit-version: 1
[Unit]
Description=Spinifex SQS Message Store (dedicated JetStream)
PartOf=spinifex.target
ConditionPathExists=/etc/spinifex/sqs-store/sqs-store.conf
[Service]
EnvironmentFile=-/etc/spinifex/telemetry.env
Type=simple
Slice=spinifex-system.slice
User=spinifex-sqsstore
Group=spinifex
ExecStart=/usr/local/bin/spx service sqs-store start
Restart=on-failure
RestartSec=5
# JetStream store shutdown and raft stepdown have no internal deadline.
TimeoutStopSec=20
OOMScoreAdjust=-500
# Soft Go heap limit for the store; MemoryHigh/Max bound a runaway on top.
Environment=GOMEMLIMIT=512MiB
MemoryHigh=10%
MemoryMax=20%
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/sqs-store/sqs-store.conf
# Filesystem access
NoNewPrivileges=yes
ProtectSystem=strict
PrivateDevices=yes
ReadOnlyPaths=/etc/spinifex/sqs-store /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
ReadWritePaths=/var/lib/spinifex/sqs-store /var/log/spinifex
# Hardening
ProtectHome=yes
PrivateTmp=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictRealtime=yes
RestrictNamespaces=yes
ProtectClock=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectProc=invisible
SystemCallArchitectures=native
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
CapabilityBoundingSet=
ProtectHostname=yes
KeyringMode=private
RemoveIPC=yes
LimitCORE=0
LimitNOFILE=65536
LimitNPROC=4096
[Install]
WantedBy=spinifex.target
+2 -2
View File
@@ -1,9 +1,9 @@
# spinifex-unit-version: 3
# spinifex-unit-version: 4
[Unit]
Description=Spinifex Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=spinifex-nats.service spinifex-sqs-store.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer ovsdb-server.service ovs-vswitchd.service ovn-controller.service
Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer ovsdb-server.service ovs-vswitchd.service ovn-controller.service
[Install]
WantedBy=multi-user.target
+12
View File
@@ -10,7 +10,13 @@ import (
"github.com/mulgadc/spinifex/internal/awsmodel"
acmawsapi "github.com/mulgadc/spinifex/spinifex/domains/acm/awsapi"
awsapi "github.com/mulgadc/spinifex/spinifex/domains/ecr/awsapi"
ecsawsapi "github.com/mulgadc/spinifex/spinifex/domains/ecs/awsapi"
eksawsapi "github.com/mulgadc/spinifex/spinifex/domains/eks/awsapi"
elbv2awsapi "github.com/mulgadc/spinifex/spinifex/domains/elbv2/awsapi"
iamawsapi "github.com/mulgadc/spinifex/spinifex/domains/iam/awsapi"
rdsawsapi "github.com/mulgadc/spinifex/spinifex/domains/rds/awsapi"
route53awsapi "github.com/mulgadc/spinifex/spinifex/domains/route53/awsapi"
stsawsapi "github.com/mulgadc/spinifex/spinifex/domains/sts/awsapi"
"github.com/mulgadc/spinifex/spinifex/gateway"
"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
@@ -54,6 +60,12 @@ func compareAll() ([]awsmodel.OperationCoverage, error) {
awsapi.ServiceName: awsapi.OperationInventory(),
acmawsapi.ServiceName: acmawsapi.OperationInventory(),
route53awsapi.ServiceName: route53awsapi.OperationInventory(),
elbv2awsapi.InventoryName: elbv2awsapi.OperationInventory(),
rdsawsapi.ServiceName: rdsawsapi.OperationInventory(),
ecsawsapi.ServiceName: ecsawsapi.OperationInventory(),
eksawsapi.ServiceName: eksawsapi.OperationInventory(),
iamawsapi.ServiceName: iamawsapi.OperationInventory(),
stsawsapi.ServiceName: stsawsapi.OperationInventory(),
})
// The loop below walks the embedded models, so a served service with no
// model would otherwise drop out of the report without a word.
+120
View File
@@ -0,0 +1,120 @@
// sqs-worker runs inside the shared SQS worker microVM.
//
// Usage:
//
// sqs-worker boot
// sqs-worker ready
//
// boot selects and mounts the worker's data disk named by fw_cfg, asks the
// gateway for its current first-store decision with its instance-role
// credentials, then loads or creates the store key on that disk and
// publishes only the public key. ready then keeps checking the bound store's
// JetStream with that key until it is stopped, keeping a readiness marker only
// while the check passes; it proves the store accepted the worker key, not
// the store server's identity.
package main
import (
"context"
"errors"
"fmt"
"os"
"os/signal"
"syscall"
"time"
"github.com/mulgadc/spinifex/internal/gwsign"
"github.com/mulgadc/spinifex/internal/sqsworkergw"
"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerboot"
"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
)
// The cluster CA the daemon supplies; the only root the client trusts.
const caPath = "/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw"
// newClaimer signs with the IMDS instance-role credentials only; there is
// no static-key fallback.
func newClaimer() workerboot.NewClaimer {
return func(region, gatewayURL string) (workerkey.Claimer, error) {
return newClient(region, gatewayURL)
}
}
func newClient(region, gatewayURL string) (*sqsworkergw.Client, error) {
ca, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("read cluster CA: %w", err)
}
return sqsworkergw.New(gatewayURL, region, ca, gwsign.NewInstanceRole())
}
type describer struct{ c *sqsworkergw.Client }
func (d describer) DescribeStore(ctx context.Context) (workerboot.Store, error) {
st, err := d.c.DescribeStore(ctx)
return workerboot.Store(st), err
}
func main() {
if len(os.Args) != 2 || (os.Args[1] != "boot" && os.Args[1] != "ready") {
fmt.Fprintln(os.Stderr, "usage: sqs-worker boot|ready")
os.Exit(2)
}
logf := func(msg string) { fmt.Fprintf(os.Stderr, "[sqs-worker] %s\n", msg) }
run := boot
if os.Args[1] == "ready" {
run = ready
}
if err := run(logf); err != nil {
logf("FATAL: " + err.Error())
os.Exit(1)
}
}
func config(logf func(string)) workerboot.Config {
return workerboot.Config{
FwCfgDir: "/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/sqs-worker",
SysBlock: "/sys/block",
DevDir: "/dev",
MountInfo: "/proc/self/mountinfo",
DataDir: "/data",
RunDir: "/run/sqs-worker",
Disk: workerboot.SystemDisk{E2fsck: "/sbin/e2fsck"},
IsBlockDevice: workerboot.IsBlockDevice,
NewClaimer: newClaimer(),
Attempts: 60,
Wait: 5 * time.Second,
Log: logf,
}
}
func boot(logf func(string)) error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
_, err := workerboot.Boot(ctx, config(logf))
return err
}
// ready runs until init stops it; the marker is removed on the way out.
func ready(logf func(string)) error {
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer cancel()
err := workerboot.Watch(ctx, workerboot.ReadyConfig{
Config: config(logf),
NewDescriber: func(region, gatewayURL string) (workerboot.StoreDescriber, error) {
c, err := newClient(region, gatewayURL)
if err != nil {
return nil, err
}
return describer{c}, nil
},
Checker: workerboot.NATSChecker{},
CallTimeout: 30 * time.Second,
Refresh: 30 * time.Second,
Refused: 5 * time.Minute,
})
if errors.Is(err, context.Canceled) {
return nil
}
return err
}
+29
View File
@@ -0,0 +1,29 @@
package clusterv1
import "time"
// ClusterStateBucket is the KV bucket holding each daemon's heartbeat alongside
// other cluster-wide state.
const ClusterStateBucket = "spinifex-cluster-state"
// HeartbeatKey is the ClusterStateBucket key a node writes its heartbeat under.
func HeartbeatKey(node string) string {
return "heartbeat." + node
}
// HeartbeatTimestamp is the read side of a heartbeat for node liveness. It
// decodes only the timestamp, so a reader does not depend on the capacity
// figures the daemon writes beside it.
type HeartbeatTimestamp struct {
Timestamp string `json:"timestamp"`
}
// Time returns when the heartbeat was written. A timestamp that is missing or
// unparseable vouches for nothing, so ok is false.
func (h HeartbeatTimestamp) Time() (written time.Time, ok bool) {
parsed, err := time.Parse(time.RFC3339, h.Timestamp)
if err != nil {
return time.Time{}, false
}
return parsed.UTC(), true
}
+35
View File
@@ -0,0 +1,35 @@
package clusterv1_test
import (
"encoding/json"
"testing"
"time"
clusterv1 "github.com/mulgadc/spinifex/contracts/cluster/v1"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The bucket and key are the deployed layout: a rolling upgrade has old daemons
// writing and new readers reading, so these strings cannot change.
func TestHeartbeatKey_IsTheDeployedLayout(t *testing.T) {
assert.Equal(t, "spinifex-cluster-state", clusterv1.ClusterStateBucket)
assert.Equal(t, "heartbeat.node-a", clusterv1.HeartbeatKey("node-a"))
}
func TestHeartbeatTimestamp_ReadsAFullDaemonHeartbeat(t *testing.T) {
full := `{"node":"node-a","epoch":3,"timestamp":"2026-10-30T01:02:03Z","vm_count":4,"allocated_vcpu":8}`
var h clusterv1.HeartbeatTimestamp
require.NoError(t, json.Unmarshal([]byte(full), &h))
written, ok := h.Time()
require.True(t, ok)
assert.True(t, written.Equal(time.Date(2026, 10, 30, 1, 2, 3, 0, time.UTC)))
}
func TestHeartbeatTimestamp_UndatedBeatVouchesForNothing(t *testing.T) {
for _, ts := range []string{"", "yesterday"} {
_, ok := clusterv1.HeartbeatTimestamp{Timestamp: ts}.Time()
assert.False(t, ok, "timestamp %q", ts)
}
}
+7
View File
@@ -0,0 +1,7 @@
package ec2v1
// VolumeHeldElsewhereText is the text of a start refused because another node
// holds one of the instance's volumes. It crosses NATS as message text, so
// readers match on it; the provider that refuses and every reader share this
// string so they cannot drift.
const VolumeHeldElsewhereText = "volume is leased by another owner"
+5
View File
@@ -8,3 +8,8 @@ administration. It is separate from AWS tenant-facing service contracts.
roles and Reed-Solomon configuration. It is an operational status contract,
not a generic provider abstraction; do not change its subject or JSON shape
incompatibly. Add a new version when that is required.
`admin.go` holds the `/admin/<Method>` request and response bodies the private admin surface shares with the CLI.
`AdminMethodNames` is the list of callable methods, and the gateway router builds its accept set from it.
`AccountQuotaRequest`, `AccountQuotaResponse` and `DeleteAccountResponse` stay in the gateway until the `/admin` handlers move, because they embed types the contracts layer may not import.
Do not change a field name, JSON tag or omitempty incompatibly; add a new version when that is required.
+76
View File
@@ -0,0 +1,76 @@
package operatorv1
import "slices"
// adminMethodNames is the callable /admin/<Method> list. The gateway router
// builds its accept set from it, so a name cannot be served without being listed.
var adminMethodNames = []string{
"CreateAccount",
"DeleteAccount",
"DescribeAccountDeletion",
"GetAccountQuota",
"ListAccounts",
"PutAccountQuota",
}
// AdminMethodNames returns the callable /admin/<Method> names in sorted order.
// Tooling that mints a credential grants these by name, so the list has to be
// the one the router serves rather than a copy that can drift out of step.
func AdminMethodNames() []string {
return slices.Sorted(slices.Values(adminMethodNames))
}
// Deletion job states.
const (
DeletionStateRunning = "RUNNING"
DeletionStateCompleted = "COMPLETED"
DeletionStateFailed = "FAILED"
)
// CreateAccountRequest is the /admin/CreateAccount request body.
type CreateAccountRequest struct {
Name string `json:"name"`
ClientToken string `json:"clientToken"`
Source string `json:"source,omitempty"`
}
// CreateAccountResponse is the /admin/CreateAccount success body. It is stored
// verbatim (with the secret encrypted) so a replay returns an identical answer.
type CreateAccountResponse struct {
AccountID string `json:"accountId"`
AccountName string `json:"accountName"`
AdminUser string `json:"adminUser"`
AccessKeyID string `json:"accessKeyId"`
SecretAccessKey string `json:"secretAccessKey"`
DefaultVpcID string `json:"defaultVpcId"`
}
// DeleteAccountRequest is the /admin/DeleteAccount request body.
type DeleteAccountRequest struct {
AccountID string `json:"accountId"`
AccountName string `json:"accountName"`
ClientToken string `json:"clientToken"`
Force bool `json:"force,omitempty"`
DryRun bool `json:"dryRun,omitempty"`
}
// DescribeAccountDeletionRequest is the /admin/DescribeAccountDeletion body.
type DescribeAccountDeletionRequest struct {
AccountID string `json:"accountId"`
}
// AccountSummary is one row of /admin/ListAccounts. It carries no credential
// material: the caller can already create and delete accounts, but that is no
// reason for a listing to hand out keys.
type AccountSummary struct {
AccountID string `json:"accountId"`
AccountName string `json:"accountName"`
Status string `json:"status"`
CreatedAt string `json:"createdAt,omitempty"`
}
// ListAccountsResponse is the /admin/ListAccounts success body.
type ListAccountsResponse struct {
Accounts []AccountSummary `json:"accounts"`
Count int `json:"count"`
}
+65
View File
@@ -0,0 +1,65 @@
package operatorv1
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/require"
)
func goldenJSON(t *testing.T, v any, want string) {
t.Helper()
data, err := json.Marshal(v)
require.NoError(t, err)
require.JSONEq(t, want, string(data))
}
func TestAdminWire_CreateAccountRequest(t *testing.T) {
goldenJSON(t, CreateAccountRequest{Name: "acme", ClientToken: "tok-1", Source: "cli"},
`{"name":"acme","clientToken":"tok-1","source":"cli"}`)
goldenJSON(t, CreateAccountRequest{Name: "acme", ClientToken: "tok-1"},
`{"name":"acme","clientToken":"tok-1"}`)
}
func TestAdminWire_CreateAccountResponse(t *testing.T) {
goldenJSON(t, CreateAccountResponse{
AccountID: "000000000042", AccountName: "acme", AdminUser: "admin",
AccessKeyID: "AKIAEXAMPLE", SecretAccessKey: "secret", DefaultVpcID: "vpc-1",
}, `{"accountId":"000000000042","accountName":"acme","adminUser":"admin","accessKeyId":"AKIAEXAMPLE","secretAccessKey":"secret","defaultVpcId":"vpc-1"}`)
}
func TestAdminWire_DeleteAccountRequest(t *testing.T) {
goldenJSON(t, DeleteAccountRequest{AccountID: "000000000042", AccountName: "acme", ClientToken: "tok-2", Force: true, DryRun: true},
`{"accountId":"000000000042","accountName":"acme","clientToken":"tok-2","force":true,"dryRun":true}`)
goldenJSON(t, DeleteAccountRequest{AccountID: "000000000042", AccountName: "acme", ClientToken: "tok-2"},
`{"accountId":"000000000042","accountName":"acme","clientToken":"tok-2"}`)
}
func TestAdminWire_DescribeAccountDeletionRequest(t *testing.T) {
goldenJSON(t, DescribeAccountDeletionRequest{AccountID: "000000000042"}, `{"accountId":"000000000042"}`)
}
func TestAdminWire_DeletionStates(t *testing.T) {
require.Equal(t, "RUNNING", DeletionStateRunning)
require.Equal(t, "COMPLETED", DeletionStateCompleted)
require.Equal(t, "FAILED", DeletionStateFailed)
}
func TestAdminWire_ListAccountsResponse(t *testing.T) {
goldenJSON(t, ListAccountsResponse{
Accounts: []AccountSummary{
{AccountID: "000000000042", AccountName: "acme", Status: "ACTIVE", CreatedAt: "2026-01-02T03:04:05Z"},
{AccountID: "000000000043", AccountName: "bolt", Status: "TERMINATING"},
},
Count: 2,
}, `{"accounts":[
{"accountId":"000000000042","accountName":"acme","status":"ACTIVE","createdAt":"2026-01-02T03:04:05Z"},
{"accountId":"000000000043","accountName":"bolt","status":"TERMINATING"}],"count":2}`)
}
func TestAdminWire_AdminMethodNames(t *testing.T) {
require.Equal(t, []string{
"CreateAccount", "DeleteAccount", "DescribeAccountDeletion",
"GetAccountQuota", "ListAccounts", "PutAccountQuota",
}, AdminMethodNames())
}
+16 -1
View File
@@ -136,7 +136,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. |
| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is allowed, and is preferred over leaving the instance unable to run, but it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is refused while that node is alive (its daemon heartbeat is under 60 seconds old, or cannot be read) and is otherwise allowed, in which case it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
| `spx admin storage status` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Sends the signed `GetStorageStatus` action to the AWS gateway with credentials from the standard AWS chain (e.g. `AWS_PROFILE=spinifex`) and prints what the console's Service Metrics page shows: predastore encoding, meta and blob node counts, cluster health, a meta node table (health, raft state, leader, term, commit and applied index) and the blob nodes. Cluster health is `Degraded` when any meta node is unreachable or the nodes reporting raft state do not agree on one leader. A meta node read only through its host's `/readyz` shows healthy with blank raft fields. |
### Upgrade Management
@@ -192,6 +192,21 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
|---------|-------|-------------|
| `spx admin eks restore-snapshot` | `--cluster` (required), `--snapshot` (optional, defaults to the latest snapshot in predastore), `--account` (optional, defaults to the bootstrap account) | Single-CP total-loss DR path (fail-safe): validates the snapshot exists in predastore BEFORE any mutation (a typo'd/missing key hard-fails, never resets into an empty datastore) → launches a fresh control-plane VM as a cluster-init seed (replaying the persisted create-time launch template) → sets a required-snapshot `RecoveryDirective` (`cluster-reset`) so the boot-time recovery agent aborts rather than resets-into-empty if it cannot fetch the snapshot → persists the replacement in cluster meta BEFORE re-pointing the NLB (so an NLB failure is convergeable by the reconciler, returned as a provisional status, not a hard error) → re-points the cluster NLB's apiserver and konnectivity target groups from the old CP's ENI to the new one → fences the old CP with retries, failing loudly if it cannot be confirmed terminated (split-brain guard). Any failure before the meta commit unwinds the fresh CP (terminate + clear directive) so a re-run does not stack a second resetting control plane. The returned status is provisional — success means the sequence completed, not that etcd is restored and serving; verify cluster health. HA clusters (a spread with a potentially surviving quorum) are rejected — recover those via quorum reformation instead. |
### SQS System Instances
These commands start the first SQS system instances on one node. They do not serve the SQS API, and a bound VM is not a ready service.
Both send a signed `spinifex` admin action (`EnsureSQSWorker`, `EnsureSQSStore`) to the AWS gateway using the standard AWS credential chain, for example `AWS_PROFILE=spinifex`. The caller must be in the super-admin account (`000000000001`) and hold `spinifex:<Action>`. The VMs, volumes, private VPC and ENIs belong to the system account (`000000000000`). `--endpoint`, `--region` and `--ca-bundle` default to this node's gateway, region and `config/ca.pem`; off a node, `--endpoint` and `--region` are required. Each call waits up to 11 minutes.
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin sqs ensure-worker` | `--node` (required), `--instance-type` (required), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's one SQS worker VM on `--node`, or reports the VM already bound there: node, region, instance, instance type and generation. The worker is pinned to its first node; another node is refused with `InvalidParameterValue`, and there is no takeover. It gets a private system VPC with one /24 subnet and no internet gateway, NAT or Elastic IP, and an encrypted 1 GiB system-account data volume on which the guest generates its own store key. The platform records only the public key. A different `--instance-type` is accepted only while no volume, network interface or launch step has been attempted; once any has, it is `IncorrectState`, whether or not the step completed. |
| `spx admin sqs ensure-store` | `--node` (required, the worker's node), `--instance-type` (required, more than 512 MiB memory), `--volume-size-gib` (required, 1–16384), `--endpoint`, `--region`, `--ca-bundle` | Launches the Region's SQS store VM on the worker's node with an encrypted system-account data volume, or reports the VM already bound there: node, region, instance, instance type and address. Needs the worker bound with its public key recorded, else `IncorrectState`. The store accepts only that key. A different size, type or key than the first call is `IncorrectState`. |
`ServiceUnavailable` never proves that no SQS resources exist, because an earlier attempt may already have created some. Its message says what it does establish. When no node serves the subject, this request was not delivered. When a volume create was not sent, that particular create did not happen. When the node cannot boot the guest yet or its launcher lacks a capability, this request stopped before allocating or recording anything. When the gateway times out waiting for the node's reply, nothing is known: the launch may still be running, so never assume it created nothing. In every case, call the command again with the same arguments. A retry first checks the durable record of the attempt and continues only when that is safe; it can proceed to the first launch after a create that was proven unsent, and otherwise reports the recorded outcome as bound or `IncorrectState`, or returns `ServiceUnavailable` again. `IncorrectState` from a volume, network interface or launch step means that step was recorded as sent but its outcome was never observed. It is never resent, the message names the volume when its ID is known, and no tool reconciles it yet: recovery is manual.
`bound` says only that the VM was launched for the current record. The worker decides store readiness itself, by checking that the store is bound to its own public key and that a NATS connection with that key succeeds; nothing reports that readiness to the operator. The worker-to-store connection does not authenticate the store server. Real VPC routing, IMDS and gateway TLS from the guest, and worker-to-store traffic on a real interface have not been tested.
## Private Admin API
Not an AWS API: a Spinifex-internal surface on the AWS gateway, JSON in and JSON out, one method per path segment at `POST /admin/<Method>`. It exists so a self-service signup form, a provisioning system or a load-test harness can create and remove tenant accounts without an operator shell on a node.
+153 -1
View File
@@ -243,6 +243,39 @@ Hot-attaching a volume to a running direct-boot microVM is not supported, becaus
This covers only requests that are already mounted when the instance starts; nothing yet supplies data volumes to a system instance, and the path is verified with fake-QEMU unit tests, not with a real EBS volume in a guest.
#### Volume leases
Before opening an engine on a volume, the Viperblock worker takes a lease: one entry per volume in the `VIPERBLOCK_VOLUME_LEASES` JetStream KV bucket, which expires 45 seconds after its last write (`providers/ebs/viperblock/volume_lease.go`).
A renewal becomes eligible once the last confirmed write is 10 seconds old, and the holder checks for that periodically, so the rewrite lands some time after that point.
Each rewrite is conditioned on the revision the holder last wrote, and a refused write that finds the entry gone or naming another owner or generation fences the export as a takeover.
It may keep writing for 40 seconds after the last write the store acknowledged, measured from when that write was sent, because the server starts the entry's expiry when it applies the write and never earlier.
A slow acknowledgement therefore shortens the remaining validity rather than extending it, and an acquisition acknowledged after its validity has already run out is refused as retryable.
For a holder that cannot renew, a timer is armed for the validity deadline, leaving a nominal 5 seconds before the entry can expire; when it fires, the export's nbdkit is fenced with SIGKILL (`volume_fence.go`).
The fence runs on its own goroutine, so Go scheduling, that goroutine and process termination all delay the kill, and enough delay can use up the whole margin; a kill held in uninterruptible sleep, which the fence waits up to 30 seconds for, is the extreme case.
Once a lease is lost or has lapsed it stays lost: a renewal that succeeds afterwards is ignored, and another open on the same node cannot share it while the fence runs.
These rules bound only the holder's own clock.
The storage backend does not reject a stale writer, so a paused or partitioned host that resumes after its validity can still write; generation enforcement at the write boundary is required by Q-24 in the umbrella repository's `docs/specs/viperblock/rfc3-1_overview.md` and is not implemented.
An export that survives a daemon restart writes without a renewed lease until recovery adopts it, and a survivor that recovery cannot adopt while a guest still uses it is left running without a lease or a fence.
#### Unsealed writes and start elsewhere (interim)
A volume is unsealed while its node holds writes the backend has not confirmed; the node records this in a dirty marker naming itself as owner (`providers/ebs/viperblock/volume_dirty.go`).
Opening such a volume on another node loses those writes, so two guards stop it while the previous owner may still hold them.
This is an interim guard until the instance lifecycle work defines ownership, and it is not a fence.
When `ec2.start` is forwarded to the node that last ran a stopped instance and that node refuses for capacity, the forwarding node normally starts the instance locally.
If the refusing node still holds unsealed writes for any of the instance's volumes, it sets the `X-Unsealed-Writes` NATS reply header and the forwarder relays `InsufficientInstanceCapacity` instead of starting elsewhere.
The check asks the owner's viperblockd over the per-node `unsealed` verb with a 5 second bound, and a check that fails or times out counts as holding, so the refusal is kept.
Independently, a node that finds another node's dirty marker on a volume refuses to open it when that node's daemon heartbeat (`heartbeat.<node>` in the cluster-state bucket) is under 60 seconds old.
It also refuses when the heartbeat cannot be read or no heartbeat source is wired, and the refusal maps to the held-elsewhere error that recovery retries.
A previous owner that has stopped heartbeating is opened over with a warning, as before.
The heartbeat age is compared across node clocks, so the 60 second window is also the clock-skew allowance.
EKS control-plane recovery treats both refusals, capacity and held-elsewhere, as retryable: it keeps retrying the stopped member and does not count them against its restart attempts.
### Predastore (S3)
Object storage used for:
@@ -274,7 +307,7 @@ storage-provider contract.
`domains/sqs/message` defines the store-neutral interface the SQS domain reaches its message store through (ADR-0011:S6), with a JetStream adapter in `domains/sqs/message/jetstream` and a shared conformance suite in `domains/sqs/message/conformance`.
It is an internal primitive: nothing outside the SQS domain uses it, no daemon or service is wired to it, and it does not itself deliver an SQS API.
The dedicated JetStream server ADR-0011 describes is not deployed by this code.
The dedicated JetStream server ADR-0011 describes is not deployed by this code, and Spinifex ships no host service, unit, user or firewall rule for it; the guest that could run it is described under *SQS store system instance (not wired)*.
The interface covers durable and idempotent append, body and metadata reads, conditional state writes and deletes, per-namespace counts, namespace enumeration, an ordered change feed, a bounded paginated scan, namespace deletion and a sweep of tombstoned bodies.
A namespace carries an immutable message-shard and state-shard allocation chosen when the queue is created, so a later change in shard counts never moves an existing queue.
@@ -293,6 +326,125 @@ If that consumer delivers nothing for about two seconds while the stream holds a
Coverage is unit and conformance tests on in-process R1 and R3 servers; there is no end-to-end or multi-host validation of this code, and earlier multi-host evidence predates the final feed.
### SQS store system instance (not wired)
Spinifex can build, install, launch, restart and give storage to a dedicated SQS store microVM, but nothing calls any of it.
No SQS API, domain service, admin command or daemon start-up path creates a store, so none of this is in production use.
The guest image is built by `make build-sqs-store-microvm-image`, separate from the load balancer image, and bundles the `nats-server` version pinned in `go.mod` plus the ext4 tools.
Release tarballs carry it as `microvm-sqs-store/`, and `setup.sh` installs it to `/usr/share/spinifex/microvm/sqs-store/`.
A release without a complete image removes any previously installed one, so a launch is then refused rather than booting a stale image.
At boot the guest reads three firmware config values: the data disk's serial, an IPv4 address on one of its interfaces, and one NATS user NKey public key.
It mounts exactly one matching disk at `/data` after `e2fsck -p`, refuses a missing, duplicate, unformatted or unrepairable disk, and never formats.
JetStream keeps its data on `/data` with `sync_interval: always` (ADR-0011:S3), and the server listens only on that address and accepts only that key.
On shutdown the store is stopped before `/data` is unmounted; a store that survives SIGKILL leaves every mount in place.
A launch uses the versioned subject `system.LaunchSQSStore.v1`, which a daemon without this code does not serve, so it fails with no responder instead of booting another guest.
The legacy `system.LaunchInstance` path refuses the store kind on both send and receive.
Every input check runs before capacity is reserved, and the data volume must be an available system-account volume.
It is mounted through the normal EBS path and detached, never deleted, on terminate, which is the protection standard EBS gives today and nothing stronger.
CPU mitigations stay enabled for the store, unlike the load balancer.
A successful launch means the VM is running, not that NATS is ready.
The instance record keeps the store's non-secret guest settings so a restart on the node that launched it rewrites the firmware config files.
A relaunch on any other node is refused, and cross-node recovery never claims a store.
Both checks exist only in daemons with this code: an older daemon can still claim and relaunch a store elsewhere, and an explicit start served by another node claims the instance before its relaunch is refused.
`domains/sqs/storevolume` prepares a fresh data volume, because the guest never formats.
It checks for `mkfs.ext4` and `qemu-img` before allocating, creates the volume in `creating`, writes an empty ext4 image to it over NBD on the local node, unpublishes it, and only then makes it `available`.
While a volume is `creating`, attach, delete, modify, force-detach and snapshot are refused.
A volume is deleted only after a confirmed unpublish; any publish error, an unconfirmed unpublish or a failed release keeps it and reports its ID, with its state reported as unknown after a failed release.
Evidence is local only: unit tests, real QEMU boots against disposable scratch disks, and one run of the preparer against the `qemunbdd` provider whose volume kept acknowledged writes across a reboot.
Viperblock, volume encryption, the EC2 volume record path, multiple nodes and host reachability of the guest address are not tested.
Credential issuance and rotation, store placement and NATS readiness checks do not exist, and there is no failover: a lost node leaves its store unavailable.
### SQS worker guest (operator-launched)
The shared SQS worker guest holds the SQS store's client credential, so that no platform component does (ADR-0011:S2).
Only an operator activates it, with the super-admin `EnsureSQSWorker` action described below; nothing starts it at daemon startup, and no worker service runs after boot.
`make build-sqs-worker-microvm-image` builds it with the store image's script and network `/init` under `SQS_MICROVM_PROFILE=worker`; it carries the static `sqs-worker` binary and the ext4 tools, no `nats-server`, and an empty `/data` mountpoint.
Releases build it beside the store image and ship it as `microvm-sqs-worker`, which `setup.sh` installs to `/usr/share/spinifex/microvm/sqs-worker` with the store image's rules: a release without it removes an earlier one, and an incomplete one is refused.
At boot `sqs-worker boot` reads three firmware config values under `opt/spinifex/sqs-worker/`: the worker disk's serial, its Region and the gateway URL; none is a first-store decision, because a static value survives a guest reset.
A missing, empty or malformed value refuses before any disk is touched.
It selects exactly one disk with that serial, runs `e2fsck -p`, mounts it at `/data` and then confirms from `/proc/self/mountinfo` that this disk, as a whole ext4 filesystem, is what is mounted there; an empty mountpoint, a foreign mount or a failed check or mount refuses, and the guest never formats.
Only then does it ask the gateway for the current decision with `ClaimFirstStore`, signed by `internal/sqsworkergw` with credentials from the instance metadata service only (`gwsign.NewInstanceRole`, no environment, profile or static-key fallback), over TLS 1.3 that trusts only the cluster CA from `opt/spinifex/ca-cert` and never follows a redirect.
Under that decision `workerkey.Acquire` creates the key on `/data/sqs-worker-key`, owner-only and synced, records only its public key and then uses it, or loads the recorded key; a missing or different key once one is recorded refuses and is never replaced.
The VM can start before its launch binds the instance, so its first calls are legitimately `AccessDenied`; that denial creates nothing and is asked again, as are a lost reply and a `ServiceUnavailable` or `Throttling` refusal, all within one bound (60 attempts 5 seconds apart), after which a worker that was never bound, or was replaced, refuses the boot.
A signature or other refusal, or an answer that is neither "create" nor one valid public key, refuses the boot at once.
`domains/sqs/firststore` is the SQS domain's first-store decision: one record per Region in the `spinifex-sqs-first-store` KV bucket moves from `claimed` to `key-recorded` to `launch-intent` under compare-and-swap, bound to the worker's instance ID, claim ID and generation.
A claim authorizes key creation only while no public key is recorded; once one is, every claim returns only that key, and a store launch intent needs it recorded first.
An older generation, a different identity at the same generation or an unreadable record authorizes nothing, and the record holds no private key material.
Reads are stream message gets answered by the stream leader, never direct gets a lagging replica can answer, and every authorizing answer is also confirmed by a compare-and-swap at the revision it read, so a stale or unavailable read authorizes nothing.
The authenticated caller that will carry this decision must derive the worker's instance ID and generation from authoritative instance state; a generation supplied in a request cannot itself authorize a takeover.
`domains/sqs/workerident` is that authoritative instance state: one record per Region in the `spinifex-sqs-worker-identity` KV bucket holds a monotonic generation, claim ID, pinned node and encrypted data volume, and moves from `launching` to `bound` to `stopped`, or from `launching` to `failed`, under compare-and-swap.
A launch intent is refused while the previous attempt is `launching` or `bound`, including one whose outcome is unknown, so only one caller ever holds an intent to start a VM on the pinned volume.
A replacement takes the next generation only after `FailLaunch` or `RecordStopped` records that nothing holds the volume, and each request names the generation it supersedes, so a replayed request cannot take a generation it never observed; there is no automatic takeover.
The launch path then raises the first-store fence to the successor's generation with `firststore.Fence` before `BeginLaunch`, which refuses a replacement until a leader read shows that fence; the fence only rises, and `firststore` refuses every call from a generation below it.
Only the current claim ID is recorded, so claim IDs are minted fresh per attempt and a VM launch must be idempotent on generation and claim together.
Binding writes the instance's index entry first and then binds the record to that instance.
The index is only a lookup: `Authorize` accepts an instance only while the Region's record is `bound` to exactly that instance, generation and claim, confirmed by a compare-and-swap at the leader-read revision, so an old, orphaned, rewritten or unbound entry authorizes nothing.
A same-instance reboot keeps its instance ID and identity; a replacement takes a new generation, and the first cut refuses another node or data volume and an unencrypted volume, although the encryption flag is the caller's claim and the launch wiring must verify it from the volume's metadata.
Both buckets are opened through `OpenDurableKV`, which refuses an existing bucket whose effective stream configuration could expire, evict, lose, lag or refuse records: non-limits retention, an age, per-message or delete-marker TTL, memory storage, asynchronous flush, a size, count or message-size cap, a discard policy other than discard-new, unacknowledged writes, a sealed stream, a mirror or sources, or fewer replicas than the cluster needs.
`workerkey.Acquire` drives that decision: it syncs a new key to disk, records its public key and only then returns it, so a crash before recording reuses the key already on disk.
`domains/sqs/awsapi/worker` carries the decision to the worker as `ClaimFirstStore` and `RecordWorkerPublicKey`, served under the internal `spinifex-sqs` signing service, which no AWS SDK signs for; the public AmazonSQS action inventory is untouched.
The gateway admits only an assumed-role session of `sqsWorkerRole` in the system account, then the role's policy, and only then resolves the session name, the instance ID STS set, through `workerident.Authorize`; the request carries no identity, and any parameter beyond the action's own is refused.
The first-store identity is the current record's instance, claim and generation, and a worker bound to another Region is refused.
A request that passed `Authorize` just before a confirmed stop and replacement still carries the old generation; the fence was raised before the successor began, so `firststore` refuses it whether or not the successor has claimed yet.
The adapter never logs a caller-chosen parameter name, because one could carry a secret.
`RecordWorkerPublicKey` accepts only a valid public user key, so a seed is refused without being recorded, logged or echoed.
The gateway binds both buckets per request and never creates them, so it provisions nothing and answers `ServiceUnavailable` until a worker launch has created them; the worker launch below ensures the role.
The private key stays on the worker disk; the guest logs and publishes only the public key to `/run/sqs-worker/public-key`.
`domains/sqs/workerlaunch` realizes the worker on the node it runs on, its only node: a bound record is returned as is; otherwise it takes the intent, then on the first launch marks the intent's one volume create, prepares the volume and records its ID, reads the volume's own record and refuses it unless encrypted, ensures the role, finds or creates the network for the claim and reads the recorded store key, and marks the intent dispatched before launching.
A dispatched launch is never sent again: a lost reply or error leaves the intent `launching` and dispatched, and every later call returns `ErrOutcomeUnknown` until the outcome is reconciled, since the launch path is not idempotent.
A failure before dispatch leaves an undispatched intent that the next call resumes with the same claim; a confirmed stop or failure is replaced only after `firststore.Fence` excludes the old generation, reusing the pinned volume.
The launch passes no store decision or key: the guest asks the gateway at boot.
The intent exists before any volume, so concurrent first launches allocate nothing for the losers. A Prepare proven to have sent no create withdraws the mark and returns `ServiceUnavailable`, or `UnsupportedOperation` when the cluster storage key is not configured, and a retry may prepare the volume; a create that may have applied is never sent again, and every later call returns `IncorrectState` until it is reconciled. A volume the call knows of but cannot use is reported with its ID in a `VolumeError` and left in place, never deleted. An intent that has sent no create may be re-typed; one that has keeps its type until replaced. The worker role is created with EC2-only trust, and an existing role is checked for that trust before any policy is granted; the instance profile must carry exactly the worker role, and one holding any other role is refused unchanged.
The daemon's `LaunchSQSWorkerInstance` boots the `sqs-worker` image through the same direct-boot path as the store, with the data volume attached at boot and kept on terminate, a required instance profile, and a required mgmt NIC; it re-reads the volume's own record and refuses one that is not available and encrypted, before anything is allocated.
Its firmware config adds the gateway URL on this node's mgmt bridge and the cluster CA from the daemon's own trust store; the caller cannot supply either, and no value written is secret.
A relaunch rebuilds that config from the VM record on the node that launched it and is refused anywhere else, and membership takeover already skips SQS-managed instances.
`sqsWorkerRole` is ensured in the system account granting exactly `spinifex-sqs:ClaimFirstStore` and `spinifex-sqs:RecordWorkerPublicKey`, from the `domains/sqs/workerrole` contract the gateway gate also reads; an existing role whose trust admits more than the EC2 service is refused.
Evidence is local only: unit tests with a fake firmware config, sysfs and mount table, tests running the boot through the real client against the real `spinifex-sqs` registration and records, and QEMU boots of the built image against disposable scratch disks and a host-side test gateway and metadata service, showing the first boot creates the key, reboots reuse it, a key lost after recording is never replaced, and the seed reaches no log or request.
The worker's network is a private SQS system VPC per Region, built by `systemvpc.EnsurePrivate` from the `[sqs]` supernet (default `10.240.0.0/14`): one VPC and one /24 subnet, with no internet gateway, NAT gateway, EIP, route table or daemon-to-store node port. Each launch claim gets one ENI in that subnet, in the VPC default security group, found again by its claim on a retry; two ENIs for one claim are refused. The launch intent is marked before the one create is sent, so a create whose ENI is not yet visible is never sent again: the launch fails closed with `IncorrectState` until the ENI appears or the outcome is reconciled. Worker-to-store TCP and the guest's route to the metadata service are not yet proven by any traffic test.
The super-admin `EnsureSQSWorker` action on the internal `spinifex` operator service takes a `NodeId`, which must be a node of this cluster and a single subject token, and an `InstanceType`, and sends a request to `sqs.worker.Ensure.v1.<node>` with no queue group, so only that node answers and the worker stays pinned.
That node's daemon re-checks the node the request names and the instance type, and refuses as `ServiceUnavailable` before allocating a volume or network or recording an intent unless the worker image's kernel and initramfs are non-empty files, its mgmt bridge exists and the cluster CA is readable; a call after the image is installed then launches.
It then runs its one composed launcher for its own Region, one launch at a time, in the daemon's own context with a 30-minute bound, so a caller's timeout does not cancel it.
A reply lost to that timeout is reported as `ServiceUnavailable`; calling again reports the bound worker, or `IncorrectState` for a dispatched launch whose outcome is unknown, which is never resent; a Region pinned to another node is refused as `InvalidParameterValue`.
The reply names the bound VM's node, Region, instance ID and generation; bound is not ready, and it says nothing about the guest's boot, its key or the store.
Stop/detach evidence for `RecordStopped` is not wired, the guest's metadata route and gateway TLS on a real ENI and worker-to-store TCP are unproven, and Viperblock's at-rest key is held by the platform.
The worker identity record holds the instance type of its generation; a request for another type is refused as `IncorrectState`, before dispatch and after binding, the reply names the recorded type, and only a replacement after a confirmed stop may change it.
Operators call both actions with `spx admin sqs ensure-worker --node --instance-type` and `spx admin sqs ensure-store --node --instance-type --volume-size-gib`, signed for the `spinifex` service with the standard AWS credential chain like the other `spx admin` gateway commands.
Each command refuses missing inputs before sending anything and reports only the bound VM; a bound VM is not a ready worker or store.
The super-admin `EnsureSQSStore` action takes a `NodeId`, an `InstanceType` and a `VolumeSizeGiB` and sends a request to `sqs.store.Ensure.v1.<node>` with no queue group; only the node the Region's worker is bound to can launch the store.
That node refuses an instance type whose memory does not exceed 512 MiB, the store's `GOMEMLIMIT` under ADR-0011:S11; this is a floor, not a validated size, and no type is defaulted.
It refuses before recording or allocating anything unless the store image and the cluster storage key are present, the worker is bound on this node, and that worker's identity has recorded its public key in the first-store decision, which it then advances to a launch intent.
The store's own launch record in `spinifex-sqs-store-launch` holds only that public key, the claim, the node and the operator's instance type and size, and marks each step before its uncertain create: the volume, the ENI in the worker's private subnet, and the launch.
A later request for a different instance type or size is refused as `IncorrectState`, before dispatch and after binding, and the reply names the type the store was launched with.
A volume preparation that failed before sending any create, such as a missing `mkfs.ext4` or a scratch or format failure, withdraws its mark and returns `ServiceUnavailable`, so a retry may try again once the host is fixed.
Any other marked step whose result is not visible is reported as `IncorrectState` and never resent; a volume that is not encrypted by its own record is refused as `UnsupportedOperation` with its ID kept, and the daemon re-checks the volume's encryption and the guest's size at launch.
The daemon never receives the worker's seed; the store's NATS server accepts only that public key, and the reply's bound VM and address are not readiness, which only the worker can check.
After boot, the worker's long-running `sqs-worker ready` step mounts and verifies its disk again and calls the internal `spinifex-sqs` `DescribeStore` action, which admits only the current worker and returns the bound store's address and recorded public key, or pending.
It loads its key only if it matches that public key, then connects to the store's NATS on port 4222 with it and asks JetStream for account info, each pass bounded to 30 seconds.
The step runs until shutdown with no overall deadline, so a store the operator launches at any time is picked up: it asks again every 5 seconds while the store is pending, not answering or the reply was lost, and re-checks a ready store every 30 seconds.
A pass writes only the address and public key to `/run/sqs-worker/store-ready`; any failed pass removes that file, and so does the step's own exit.
A start that finds shutdown in progress or no worker key logs once and waits in place rather than exiting, so init does not respawn it in a loop; shutdown stops it like a running watcher.
Init respawns the step, at most once every 5 seconds, and every start removes a marker left by a watcher that was killed or crashed before the new watcher runs.
A ready store that fails is noticed up to one re-check interval plus one bounded call later, so the marker can be up to about a minute stale.
Shutdown blocks further respawns, removes the marker, and stops the watcher by its recorded PID and start time, escalating to SIGKILL after a budget; it unmounts the worker disk only once the watcher is confirmed gone, and a survivor leaves every mount in place.
A mismatched key, a store that refuses the key, a key missing from the disk or an unclassified gateway refusal is asked again only every 5 minutes, so it never spins and still recovers if the operator fixes it; a missing key is never replaced.
The NKey authenticates the worker to the store, but nothing authenticates the store to the worker: there is no TLS, so a pass means only that something at that address accepted the key.
This is store readiness for the worker alone; there is no SQS API, and no production sizing or support is claimed.
## Configuration
Cluster configuration (`spinifex/bootstrap/config/config.go`):
+27 -14
View File
@@ -351,8 +351,22 @@ while the branch is in flight; they do not describe the umbrella repository's
| `spinifex#1269` | `spinifex/handlers/elbv2` | `spinifex/domains/elbv2` (package `elbv2`) | A mechanical whole-package relocation under the 2026-10-26 ADR-0001 amendment: the ELBv2 control plane keeps its files, tests and current dependencies, and importers alias it `elbv2domain`. No identifier, API, NATS subject, KV bucket, metric, error string or behaviour changes. Its allowlist entries are now sourced from `domains/elbv2`. Q-30's data-plane boundary stays deferred, recorded as `agents/elbv2` (Q-30 data-plane boundary) debt. |
| `spinifex#1271` | `spinifex/handlers/ecs` | `spinifex/domains/ecs` (package `ecs`) | A mechanical whole-package relocation: ADR-0001's `handlers/ecs` row targets `domains/ecs`, the files are already organised by resource, and resource packages are introduced only where callers need a compiler boundary. The ECS control plane keeps its files, tests and current dependencies alongside the already-extracted `domains/ecs/taskdefinition`, and importers alias it `ecsdomain`. No identifier, API, NATS subject, KV bucket, metric, error string or behaviour changes. Its allowlist entries are now sourced from `domains/ecs`. `handlers` has no package left and is a retired legacy root; R5 holds `domains/ecs` to the interim rule's ban on importing the daemon. |
| `spinifex#1273` | `spinifex/gateway/{ecs,elbv2}` | `spinifex/domains/{ecs,elbv2}/awsapi` (`awsapi`, imported as `ecsawsapi`, `elbv2awsapi`) | A mechanical relocation of the ECS and ELBv2 AWS action adapters to their ADR-0001 domain homes, as `spinifex#1265` did for IAM, STS, EKS and RDS; APIs, NATS subjects, KV buckets and behaviour are unchanged. The root `gateway` dispatch still imports each package, recorded as `gateway` (service registration) debt, and `domains/admission/quota` keeps its live ELBv2 counter import, now of `domains/elbv2/awsapi`. |
| `spinifex#1336` | `spinifex/gateway/tagging.go` (`Tagging_Request`, `taggingActions`, `taggingActionFromTarget`) and the `GatewayConfig.TaggingResources` field | `spinifex/domains/tagging/awsapi` (`NewRegistration`, `Deps`, `Actions`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | Resource Groups Tagging registers through the ADR-0002 seam like ACM and Route 53. The check order is kept: action, scope, authorization, account, body, aggregator, so a denied caller still gets `AccessDenied` before any body error and an unwired aggregator answers `ServerInternal`. The `checkPolicyResources` stand-in of `*` for every action is unchanged. The legacy switch case, the `supportedServices` entry and the `jsonErrorService` case are removed, and the registration declares the JSON 1.1 envelope. No allowlist row existed for this path. The nil-NATS check moves to the gateway-wide gate, which answers `ServiceUnavailableException` before dispatch as it did for every service. `cmd/aws-model-coverage` output is byte-identical because there is no embedded Tagging model. The characterization tests that pin the pipeline behaviour were committed first and pass unchanged against the registration; only the `withTagging` test helper changed. |
| `spinifex#1336` | `spinifex/gateway/elbv2.go` (`ELBv2_Request`, `elbv2Actions`, `elbv2Handler`, `accountLoadBalancerLimit`) | `spinifex/domains/elbv2/awsapi` (`NewRegistration`, `Deps`, `Quota`, `ActionNames`, `OperationInventory`), `spinifex/domains/admission/quota` (`Service.LoadBalancerLimit`), `spinifex/ingress/aws/query` (`ParseArgs`), composed by `spinifex/runtime/roles/awsgw` | ELBv2 registers through the ADR-0002 seam as the first AWS Query service. The check order is kept: query parse, action, account, input parse, scope, authorization, NATS, dispatch, so a denied caller still gets `AccessDenied` before the handler runs. The gateway members ELBv2 read are now supplied as `Deps`: `NATSConn` as `Deps.NATS`, `Quota` as the `Deps.Quota` interface declared in `awsapi` because `domains/admission/quota` imports `awsapi`, `Region` as `Invocation.Region`, and the policy gate as `Invocation.Authorize`. The `DescribeAccountLimits` cap resolution moves to `Service.LoadBalancerLimit` unchanged, and a nil `Deps.Quota` enforces nothing and reports the AWS default. AWSGW now wires the registry after it builds the quota service, because the registration captures it. The registration declares the XML envelope and the inventory name `elasticloadbalancingv2`, so `cmd/aws-model-coverage` keys it explicitly and its output is byte-identical. `gateway.ParseAWSQueryArgs` delegates to `query.ParseArgs`. The legacy switch case, the `supportedServices` entry, the inventory entry and the `gateway -> domains/elbv2/awsapi` allowlist row are removed. `cmd/aws-model-coverage` imports `domains/elbv2/awsapi` for the inventory, under a fourth named-exception allowlist row whose reason is now generic for all four. Finding, not fixed here: the XML for `DescribeAccountLimits` and `DescribeLoadBalancers` lists member fields in a different order from run to run because the marshal is map-ordered, so the characterization tests pin decoded content rather than bytes. The characterization tests that pin the pipeline behaviour were committed first and pass unchanged against the registration; only the `withELBv2` and `serveELBv2` test helpers and the setup of two older gateway tests changed. |
| `spinifex#1337` | `spinifex/gateway/spinifex.go` (`Spinifex_Request`, `spinifexAdminActions`) | `spinifex/operator/admin` (`NewRegistration`, `Deps`, `ModelAccess`, `ActionNames`), composed by `spinifex/runtime/roles/awsgw` | The `spinifex` Query service registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, authorization with the wildcard resource, account, the super-admin account gate, then the action, so a denied caller learns nothing about which actions exist. The gateway members the actions read are now `Deps`: `Version` and `Commit` strings, `NATSConn` as `Deps.NATS`, `RootCAs`, `DiscoverActiveNodes` as `Deps.ActiveNodes`, `BedrockAccessAdmin` as the `ModelAccess` interface, and the credential and access resolvers as `Deps.Credentials` and `Deps.Access`. The query actions need only the caller account, so the actor is not widened. The cluster-unavailable answer stays in the shared request path ahead of authorization. `/admin/{method}` moved onto the same registration in `mulga-3nbla.7`, recorded below. |
| `spinifex#1344` | `spinifex/gateway/rds.go` (`RDS_Request`, `rdsCaller`) | `spinifex/domains/rds/awsapi` (`NewRegistration`, `Deps`, `Quota`, `CallerFromActor`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | RDS registers through the ADR-0002 seam with the XML envelope it had. The check order is kept: query parse, action, account, the principal-class gate for the internal agent actions, scope, authorization, NATS, quota and dispatch, so a denied caller still gets `AccessDenied` before the handler runs and no customer grant is evaluated against an agent action. The gateway members RDS read are now supplied as `Deps`: `NATSConn` as `Deps.NATS`, `ExpectedNodes` as `Deps.ExpectedNodes`, `Quota.EnforceRDSInstances` as the `Deps.Quota` interface, `Region` as `Invocation.Region` and the policy gate as `Invocation.Authorize`. A nil `Deps.Quota` skips the DB-instance cap. The caller comes from `dispatch.Actor`: the session name for an assumed-role session and the user name otherwise, and the role name from `Actor.RoleName()`, which parses the underlying role ARN and never the session name. The account comes from `Invocation.AccountID`, so a caller with no composable ARN still reaches the policy check and is refused there as before. The legacy switch case, the `supportedServices` entry, the inventory entry and the `gateway -> domains/rds/awsapi` allowlist row are removed, and `cmd/aws-model-coverage` reads the inventory under a fifth named-exception allowlist row. The characterization tests that pin the pipeline behaviour were committed first with spinifex#1336 and pass unchanged against the registration; only the `withRDS` and `serveRDS` test helpers and the setup of the direct-call RDS tests changed. |
| `#1277` | `spinifex/accountteardown` | `spinifex/operator/accountlifecycle` (package `accountlifecycle`, imported as `accountteardown`) | A mechanical whole-package relocation under the 2026-10-27 ADR-0001 amendment: the account deletion coordinator keeps its APIs, NATS subjects, KV access and behaviour; importers keep the `accountteardown` name as an alias, so only their import paths change. Replacing its direct domain and adapter cleanup calls with owner capabilities stays deferred debt. |
| `#1285` | `spinifex/gateway/requestaudit.go` and `slogRequestLogger` from `spinifex/gateway/gateway.go` | `spinifex/ingress/aws/audit` (package `audit`) | The first generic-ingress extraction from the root gateway: the request-audit middleware and access-log middleware move unchanged in behaviour. The gateway's authentication flow and dispatch still populate the record, now through the exported `audit.From` and `Record` setters at the same call sites; the package imports only `foundation/telemetry` and `ingress/http`. Its unit tests moved with it; the tests that run it behind SigV4 stay in `gateway`. |
| `spinifex#1334` | `spinifex/gateway/spx` (`spx.go`, `storage.go`, `promote_image.go` and their tests) | `spinifex/operator/admin` (package `admin`) | The version, node, VM and storage-status gather helpers and the image-promote NATS request move unchanged with no shim; the `spinifex` service dispatch that calls them stays in `gateway/spinifex.go` until the `spinifex` registration slice. `operator/admin` already imports `domains/ec2/image` and `contracts/operator/v1`, so the move adds no edge and the layering ratchet needed no allowlist row. The `gateway/spx` frozen-legacy entry is removed. |
| `spinifex#1334` | `spinifex/gateway` (`CreateAccountRequest`/`Response`, `DeleteAccountRequest`, `DescribeAccountDeletionRequest`, `DeletionState*`, `AccountSummary`, `ListAccountsResponse`, `AdminMethodNames`) | `contracts/operator/v1` (`admin.go`) | The `/admin` bodies the CLI shares with the gateway move unchanged in field names, JSON tags and omitempty, pinned by golden JSON written against the gateway types first and moved with them. `AdminMethodNames` is now owned by the contract and the gateway builds its `adminMethods` accept set from it, with a test that the router accepts exactly that list. `AccountQuotaRequest`/`AccountQuotaResponse` (embed `quota.Overrides` from `domains/admission/quota`) and `DeleteAccountResponse` (embeds `accountlifecycle.Result`) stay in `gateway`: `contracts` may import only `foundation` and `contracts`, so they are settled when the `/admin` handlers move to `operator/admin`. `accountlifecycle.Stage` serialises as an integer, not its name; the golden test pins that. |
| `spinifex#1345` | `spinifex/gateway/admin*.go` (the `/admin/{method}` handlers, gates and body cap) and the `AccountQuotaRequest`/`AccountQuotaResponse`/`DeleteAccountResponse` wire types | `spinifex/operator/admin` (`rest.go`, `account_*.go`), served by the same `spinifex` registration | `/admin/{method}` stays a chi route in `gateway`, ahead of the service selector, but `Admin_Request` now only mints the request ID and header and hands the request to the registered entry through `dispatchRegistered`, the path every Query request takes. `dispatch.Invocation` gains `Service`, the credential-scope service, so the scope gate is the first identity gate in the dispatcher and a key signed for another service is denied exactly as before. The order is unchanged: verb and method, then the five fail-closed gates (scope, super-admin account, IAM user, identity, `spinifex:<Method>` policy), then the cluster check, then the 8 KiB body cap. The dispatcher returns errors and the route renders them as the JSON envelope with `requestId`, then logs once from the real result, so a denial or a 5xx is never logged as a completed request; the registration's `ResolveAction` labels the method, which removes the `auth.go` special case. The caller is read from `dispatch.Actor` (`UserName` for the gate and `updatedBy`). `BucketStore` and `SignupMaxAccounts` leave `GatewayConfig` for `operator/admin.Deps`, composed in `awsgw` beside `IAM`, `Quota`, `ExpectedNodes` and `NodeIDs`. The three wire types that could not enter `contracts` move to `operator/admin` with their handlers, `operator/cli` imports them from there, and the last `operator/cli` → `gateway` allowlist entry is deleted. |
| `spinifex#1346` | `spinifex/gateway/iam.go` (`IAM_Request`, the IAM action table, `getUserForCaller`, the PassRole check) | `spinifex/domains/iam/awsapi` (`NewRegistration`, `Deps`, `ActionNames`), composed by `spinifex/runtime/roles/awsgw` | IAM registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, service and account checks, then one parse of the input, the `GetUser` default for a missing `UserName`, scope resolution, authorization, the `iam:PassRole` check for `AddRoleToInstanceProfile`, then the handler. The caller is read from `dispatch.Actor` (`PrincipalType`, `UserName`), with no ARN parsing. The dependencies are `Deps.IAM` and `Deps.CountInstanceProfileAssociations`, composed in `awsgw`, and the gateway keeps no IAM fallback. The `cmd/aws-model-coverage` named exception gains an `iam` row. |
| `spinifex#1346` | `spinifex/gateway/sts.go` (`STS_Request`, the STS action table, the anonymous STS dispatch) | `spinifex/domains/sts/awsapi` (`NewRegistration`, `Deps`, `ActionNames`, `NewAnonymousAssumeRoleWithWebIdentity`), composed by `spinifex/runtime/roles/awsgw` | STS registers through the ADR-0002 seam with the XML error envelope it had. The check order is kept: query parse, action, service check, caller, the `AssumeRole` identity gate scoped to the stored role ARN, then the handler. The caller is read from `dispatch.Actor` (`CallerARN`, `UserName`, `SessionName`, `UserID`, `AccessKeyID`), with no ARN parsing. The gateway's authorization failure wraps a generic `dispatch.PolicyDenialError`, and STS matches it to answer with the caller, action and resource, so the gateway holds no STS-specific denial code. An unsigned `AssumeRoleWithWebIdentity` stays ahead of SigV4 in the gateway and reaches the handler through `GatewayConfig.AnonymousSTS`, a func composed in `awsgw`, never the STS service. The `cmd/aws-model-coverage` named exception gains an `sts` row. |
| `spinifex#1348` | `spinifex/gateway/ecs.go` (`ECS_Request`) and `spinifex/gateway/eks.go` (`EKS_Request`, the EKS route table) | `spinifex/domains/ecs/awsapi` and `spinifex/domains/eks/awsapi` (`NewRegistration`, `Deps`, `OperationInventory`), composed by `spinifex/runtime/roles/awsgw` | ECS (AWS JSON 1.1) and EKS (REST JSON) register through the ADR-0002 seam with the JSON error envelopes they had, both declared as AWS JSON 1.1 (see the envelope divergence below). The check order and every response are kept, pinned by the pipeline characterisation tests committed first. EKS and ECS resolve roles through `foundation/aws/passrole` `Check` with `domains/iam` `RoleResolver`: a malformed or foreign role is refused first, a missing role is authorized as written before it is reported, and a resolver outage is returned unchanged. Role handling now follows the AWS capture of 2026-10-30. ECS RegisterTaskDefinition refuses a foreign role with `ClientException` "Role is not valid" before PassRole and accepts a nonexistent, invented-path or non-ARN role from a caller who can pass roles. ECS RunTask refuses a foreign `taskRoleArn` or `executionRoleArn` override the same way and answers a nonexistent or invented-path one with "ECS was unable to assume the role". EKS CreateCluster and CreateNodegroup answer a non-ARN or foreign role with `AccessDeniedException` "Cross-account pass role is not allowed.". EKS public routes now report every parameter fault as `InvalidParameterException`, the code the EKS model declares, while the internal control-plane routes keep `InvalidParameterValue` because the guest agents match on it. Two deliberate EKS divergences remain. First, CreateCluster still refuses a nonexistent or invented-path role with `InvalidParameterException` "role ... does not exist", where AWS accepts it and the cluster later ends FAILED. Our EKS only stores, validates and echoes the cluster role and never assumes it, so accepting would leave an ACTIVE cluster bound to a role that does not exist. Second, CreateNodegroup keeps its existing refusal of a nonexistent role because AWS was not captured for it. A third divergence is the error envelope. EKS is restJson1 but registers `ErrorEnvelopeJSON`, so errors carry `application/x-amz-json-1.1` and a `__type` body where AWS sends `application/json` and a `message` body. The `X-Amzn-Errortype` header is correct and SDKs read the code from it, so impact is low. Switching to `ErrorEnvelopeRESTJSON` also changes the throttle code and the default internal-error code (`ServiceException` where the EKS model declares `ServerException`), so it is left for its own change. Verified live on env14: the ECS and EKS role cases above, and a valid CreateCluster reaching ACTIVE with the control-plane agent publishing through the gateway. The EKS route table and its tests move with the registration, and the gateway keeps no ECS or EKS fallback. The `cmd/aws-model-coverage` named exception gains an `ecs` and an `eks` row, and the two `gateway` allowlist rows are removed. |
| `spinifex#1350` | `spinifex/gateway` (`ecr.go`, `ecrauth_middleware.go`, the `/v2/token` exchange, `ecr_principal.go` with `ResolveECRPrincipal`, the per-operation `/v2` authorization, the registry host matcher, `ctxECRPrincipal`) | `spinifex/domains/ecr/registry` (`Bridge`, `NewBridge`, `BridgeConfig`, and its own `Principal`, `Rehydrator`, `Authorizer`, `Admission` types with `ErrInvalidCredential` and `ErrUnavailable`), `spinifex/ingress/http` (`HostMatcher`, `TargetAccount`, `RequestClientIP`; names no service), `spinifex/ingress/aws/credential` (`Rehydrator`, `Principal.CallerARN`), and under `spinifex/runtime/roles/awsgw`: `credverify` (`Rehydrate`, sharing the SigV4 key and session lookups), `policyauth`, `accountadmission`, `registryadapt` | ADR-0001 amendment 2026-10-29 and ADR-0005:S2: the OCI `/v2` bridge and its token auth move out of the gateway, and ECR resolves its principal through the credential capability instead of reading `IAMService` and `STSService` directly. ADR-0002:S5 keeps `/v2` outside the seam, so the gateway keeps only a `RegistryHandler` mount that answers 501 when unset. `domains/ecr/registry` imports no ingress package (`TestADR0001_Layering` unchanged): the principal, rehydrator and authorizer are registry-local types and `registryadapt` maps `credential.Failure` codes onto `ErrInvalidCredential` (401) and `ErrUnavailable` (503). No widening of `dispatch.Actor` or `dispatch.Invocation`. The SigV4 path and the bridge share one policy evaluator (`policyauth`) and one account-admission check. Deliberate change: the open-registry mode for a nil token verifier is deleted and `NewBridge` refuses a nil Verifier, Issuer, Rehydrator, Authorizer or Admission, so tests that want an unauthenticated registry mount `registry.Handler` directly. The pre-move pipeline characterisation (status codes, error bodies, the `WWW-Authenticate` challenge, referrers, token exchange, revoked and deleted principals) passes unchanged apart from setup, and the retired `ResolveECRPrincipal` parity rows live on as `credverify` `Rehydrate` rows and `Bridge` claim-check rows. Live on env14 (DHCP external IP): crane, skopeo and docker push and pull (`tests/e2e/ecr`), a cosign sign and verify in the legacy tag mode and in OCI 1.1 referrers mode with the signature returned by `GET /v2/{repo}/referrers/{digest}`, `/v2/token` refresh from fresh logins, and a deactivated access key answering 401 on `/v2/token`, manifest GET, crane pull and crane push, then 200 again once reactivated. The ECR signing-key deadlock bead (`mulga-6wqce`) is untouched. |
| `spinifex#1351` | `spinifex/gateway/{bedrock,bedrockruntime,bedrockagent,bedrockagentruntime}.go` (`Bedrock_Request`, `BedrockRuntime_Request`, `BedrockAgent_Request`, `BedrockAgentRuntime_Request`, the four route tables), `spinifex/domains/ochre/authz.go`, and the 13 `Bedrock*` fields of `GatewayConfig` | `spinifex/domains/ochre/awsapi` (`NewRegistration`, `Deps`, `Quota`, `OperationInventory`, `ServiceName`), composed by `spinifex/runtime/roles/awsgw` | The four Bedrock services register through the ADR-0002 seam as one registration, per ADR-0002:S1 (one per signing service) and the ADR-0001 amendment of 2026-10-06: the service is `bedrock` and `bedrock-runtime`, `bedrock-agent` and `bedrock-agent-runtime` are aliases. A request signed with a sub-service scope is dispatched to that sub-service whatever its path, so a path owned by another sub-service answers `InvalidActionException` as before; only the shared `bedrock` scope is split by path. The pipeline characterisation tests committed first (status, error type and envelope per service, metering order, nil-capability answers) pass unchanged apart from setup. The 13 gateway fields become `awsapi.Deps`: `Credentials`, `Endpoints`, `Access`, `Recorder`, `Logging`, `Provisioned`, `Guardrails`, `Embedder`, `KB`, `DataSources`, `Vector` and `Quota`, with the documented nil fallbacks (no grants denies every model, a nil store errors rather than panics); `BedrockAccessAdmin` had no reader outside one test and is gone, and `Deps.NATS` is dropped because the nil-NATS answer is given by the gateway-wide gate before dispatch. `Logging`, `Provisioned`, `Guardrails`, `KB` and `DataSources` are the concrete exported store types, a temporary edge until Ochre exposes capability interfaces for them. The wire-typed action functions stay in the `domains/ochre` root and the SDK-typed adapters move; `authz.go` moves with them, which removes the `domains/ochre` to `ingress/aws/bodyscope` allowlist row, the only allowlist change. The legacy switch cases, the `supportedServices` and `jsonErrorService` entries and the path-based sub-service rewrite in `GetService` are removed, so `ec2` is the only service left on the legacy table; the rewrite's rows are pinned in `awsapi` as `resolveSubservice` and dispatch tests. The registration declares the AWS JSON 1.1 envelope it had and `cmd/aws-model-coverage` is not wired, so no coverage allowlist row is added. Recorded divergences, not changed by this move: Bedrock is restJson1 in AWS but answers with the JSON 1.1 envelope (`application/x-amz-json-1.1`, `__type` body), and CreateGuardrail answers 200 where AWS answers 202; both are tracked by `mulga-xug8b`. No change to `dispatch.Actor` or `dispatch.Invocation`. |
| `spinifex#1354` | `spinifex/gateway/ec2.go` (the EC2 action table, `ec2Handler`, `ec2Actions`, the dispatch pipeline), `ec2_idempotency.go`, `ec2_validation.go`, and the EC2 token-bucket fields of `GatewayConfig` | `spinifex/domains/ec2/awsapi` (`Handler`, `NewHandler`, `Deps`, `Quota`, `Dispatch`, `ActionNames`, `InputHasField`), composed by `spinifex/gateway/ec2.go` | Slice 9a of the EC2 move: the capabilities first, with no behaviour change. The EC2 action table, idempotency and validation move to `domains/ec2/awsapi` behind `Deps{NATS, IAM, Quota, ActiveNodes, ExpectedNodes, NodeIDs, AZ, Endpoint, InstanceStatus}`; the gateway keeps the composition and `EC2_Request`, still on the legacy switch until slice 9b registers EC2 through the ADR-0002 seam. `Quota` is declared in `awsapi` because `domains/admission/quota` imports the EC2 action packages and cannot be imported back, and the gateway wraps `quota.Service` to supply the NATS instance-type resolver its retype gate takes. The instance and spot-instance adapters take the two-method `InstanceProfiles` interface (`ResolveInstanceProfile`, `CanonicalResourceARN`) instead of `iamdomain.IAMService`, which pays off the `spotinstance` to `domains/admission/quota` and `domains/iam` allowlist rows. Allowlist changes are 20 deletions and no additions: the 18 `gateway` to `domains/ec2/awsapi/*` rows and those 2. The pipeline characterisation tests committed first (error envelopes, check order, gateway-local actions, idempotent replay, PassRole, ImportKeyPair padding, launch-template expansion before authorization, RunInstances instance-profile refusal order) pass unchanged. Package-local tests drive every routed action through `Dispatch` against a stub daemon. No change to `dispatch.Actor` or `dispatch.Invocation`. Live on env14: DescribeRegions, DescribeAvailabilityZones and DescribeAccountAttributes, an idempotent CreateVolume replay and its parameter mismatch, the DryRun and MaxResults errors, a no-policy user refused with UnauthorizedOperation (including on the gateway-local DescribeRegions), an assumed-role session allowed and refused per its policy, and one RunInstances to running then terminated. |
| `spinifex#1334` | `spinifex/gateway/oidc_discovery.go` (`eksAccountKV`, the discovery-document and JWKS read and construction) | `spinifex/domains/eks/cluster` (`OIDCPublisher`, `OIDCDiscoveryDocument`), composed by `spinifex/runtime/roles/awsgw` | The EKS domain now reads its own account bucket and builds the per-cluster discovery document and JWKS responses (ADR-0005:S4), and the gateway only maps the unauthenticated `/oidc/eks/...` routes onto an `OIDCPublisher` interface it is handed. Responses are byte-identical, pinned by `oidc_discovery_current_behaviour_test.go`, whose assertions are unchanged. The gateway no longer imports `domains/eks/cluster` in production code. `STS` keeps its own `ClusterJWKSResolver` (`OIDCJWKSReader`), untouched. Recorded debt, not repaired here: a stored JWKS is served verbatim with 200 even when it is not valid JSON. |
The EC2 contract row is intentionally different from the directory moves: it creates
a compatibility boundary. `ec2.cmd.*` retains its deployed one-token NATS
subscription and permission shape, while callers use
@@ -387,9 +401,9 @@ the canonical IAM/STS principal before dispatch, because ECR must not depend on
gateway identity context or ARN construction.
All implemented ECR control-plane actions now dispatch through explicit ECR
capabilities. The remaining ECR-named gateway code is generic `/v2/*` route
assembly, SigV4/registry authentication and policy enforcement; it remains in
gateway rather than inverting those dependencies into the ECR domain.
capabilities. The OCI `/v2` bridge, its token exchange and its principal
resolution now live in `domains/ecr/registry`; the gateway keeps only the
`RegistryHandler` mount, and `awsgw` composes the bridge.
### ACM relocation residuals
@@ -438,7 +452,7 @@ These residuals were recorded by individual slices and must be resolved once the
- `runtime/roles/vpcd` (placement): the role still mixes process/role supervision with network-controller responsibilities. `lease_reaper.go` (orphaned external-pool DHCP lease sweep), `ovn_controller_watchdog.go` (OVN controller liveness and restart), `host_eip_loop.go` (host-EIP reconcile loop) and `lease_datapath_reconcile.go` (lease datapath reconcile) are periodic network-control loops, not role wiring; `vpcd.go` itself also carries bridge-mode detection and verification, chassis-list parsing, external-CIDR resolution, DHCP manager startup and lease/IP-change handling, and gateway/host-EIP allocator and neighbour flush/prime construction alongside `Service.Start`. The future placement question is whether these belong in `domains/network`. Removal condition: an accepted decision on network-controller placement, and extraction of those loops and helpers behind the role.
- ECR control-plane registration: `awsapi.Deps.NATS` is a raw `*nats.Conn`, not a narrow capability. Every NATS-backed `Handler` passes it to `domains/ecr.NewNATSMetaStore`, which calls `natsmsg.NATSRequest`, so narrowing it means changing the exported `Handler` type, every handler and the meta store; it must not be broadened further meanwhile.
- ACM control-plane registration: `domains/acm/awsapi.Deps.NATS` (`registration.go`) is likewise a raw `*nats.Conn` carried by every `Handler`, for the same reason; it must not be broadened further meanwhile.
- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `domains/ecr/awsapi`, `domains/acm/awsapi`, `domains/route53/awsapi` and `gateway` to read ECR's, ACM's and Route 53's declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
- `cmd/aws-model-coverage` imports `ingress/aws/dispatch`, `gateway` and the `awsapi` package of each registered service (`domains/ecr/awsapi`, `domains/acm/awsapi`, `domains/route53/awsapi`, `domains/elbv2/awsapi`, `domains/rds/awsapi`, `domains/ecs/awsapi`, `domains/eks/awsapi`) to read their declared inventories. It is a non-serving tool and the only named exception to the ADR-0002 composition invariant; it must not construct a registration.
- Ingress pipeline (ADR-0002 phase 4): the request-audit record, the trace action enricher, the throttle key functions and the gateway request-context keys stay in `gateway` until the auth/IAM and request-context boundary is designed. SigV4 authentication writes the audit record and context keys from four places, so moving them now would export exactly the setters ADR-0002 says must not be exported solely to make a directory move possible.
- NAT projection delivery is inconsistent and is preserved as found: EIP `AssociateAddress` sends `vpc.add-nat` unacknowledged (`AnnounceNATAdd`), skipping the 45s OVN barrier every other add path waits on, and `DeleteNetworkInterface` sends `vpc.delete-nat` unacknowledged (`AnnounceNATDelete`), skipping the ack and the ErrNoResponders retry. The reconciler's orphan sweep is the only backstop for both. Clears when those call sites use `AddNAT`/`AddNATBestEffort` and `RemoveNAT`, a reliability change of its own.
- `vpc.add-system-egress` has a `vpcd` subscriber and a contract entry but no producer (its former `domains/ec2/eip` publisher is gone); `domains/eks/cluster` still publishes the delete. Either a producer is missing or the add route is dead; it needs an owner decision.
@@ -446,7 +460,7 @@ These residuals were recorded by individual slices and must be resolved once the
- `foundation/messaging/nats/nats_trace.go` still names its OpenTelemetry tracer `github.com/mulgadc/spinifex/spinifex/utils`. It is an emitted instrumentation-scope name that dashboards may filter on, so renaming it is a telemetry change, not a path fix.
- `runtime/compute/cache.New`: two construction-time panics carry a lint exemption pending a constructor that returns an error.
- `domains/ec2/systeminstance`: managed-service parent metadata is only the `ManagedBy` tag value.
- `domains/ochre` (action adapters): since `spinifex#1266` the Bedrock action layer (handlers, action authorization, event-stream framing and the CRUD, catalogue and Converse/Invoke entry points) shares package `ochre` with the domain core, stores and inference engine, and imports `ingress/aws/bodyscope`. Moving it to `domains/ochre/awsapi` is not mechanical: the adapters reach package-private store internals (the guardrail and provisioned-throughput store handles and get/put/update methods), and adapter tests override the package-private model catalogue. It needs core capabilities first, then the move; the top-level `gateway/bedrock*.go` HTTP handlers additionally wait for the ingress registration seam.
- `domains/ochre` (action adapters): cleared by the Bedrock registration in `spinifex#1351`. The SDK-typed adapters and `authz.go` moved to `domains/ochre/awsapi`, the `ingress/aws/bodyscope` allowlist row is deleted, and the adapters take the concrete exported Ochre stores through `awsapi.Deps` as a temporary edge until Ochre exposes capability interfaces for them.
- `domains/network`: `reconcile/intent.go` builds desired network state by reading EC2's KV buckets and decoding EC2's private VPC, subnet, ENI, route-table, gateway, EIP, NAT and security-group records, and `reconcile/drift.go` watches those buckets. ADR-0001 names this pair explicitly: EC2 and network communicate through a capability/projection or versioned contract, never by one being the other's KV reader. Step 7 fix: an EC2-owned `domains/ec2/projection` exposing network intent with list and watch (it may stay KV-backed, so no data moves); a narrow intent interface declared by `reconcile`; wiring at the composition root; and the duplicated `PurposeEIP`/`PurposeIGWLRP` constants and mirrored `SGRule`/`SGEvent` folded into the projection or `contracts/ec2/v1`. The drift-watch coverage test and reconcile scenario tests are the evidence the projection must keep. `domains/network` also imports `runtime/compute/vm` and `runtime/host/command` (domain-to-runtime).
- `domains/ec2/instance`: imports `runtime/compute/vm` and `runtime/compute/gpu` (domain-to-runtime) and `domains/dns` (cross-domain implementation import).
- `domains/ec2/instance/state` (dependency): the vpcd IMDS local reader imports `runtime/compute/vm` to return `vm.VM`. Removal condition: that type moves into a domain-owned package that `runtime/compute/vm` consumes (runtime to domain is allowed).
@@ -456,14 +470,13 @@ These residuals were recorded by individual slices and must be resolved once the
- `operator/installer/firstboot` (security): `Config.InstallCallback` is rendered into `spinifex-firstboot.sh`, which is world-readable (0755). The field is currently unused (`fireInstallCallback` reads `SPINIFEX_INSTALL_CALLBACK` from the environment instead), but any future wiring must not place secrets or tokenised callback URLs in the generated script; sensitive callback credentials should reach first boot through a protected channel such as a 0600 file or a credential store.
- `operator/cli` (known security risk): `spx admin join` contacts the formation server with `InsecureSkipVerify`, sends the join token as a bearer credential and receives the cluster CA certificate and key, master key, Viperblock key and shared credentials, so an on-path attacker during formation can capture the token and the cluster root secrets or substitute a CA. This predates the move; lint surfaced it when the tree left `cmd/`. It is accepted for now and carries a line-level gosec annotation stating it is not a false positive. Removal condition: authenticate formation TLS with a pinned key or certificate delivered with the join material, or an out-of-band CA, with tests proving a mismatched server certificate fails before credentials are sent.
- `domains/eks/cluster` (potential security risk): `ClusterMeta.ControlPlaneTemplate` (`domains/eks/cluster/service_impl.go`) clears the template's credentials but keeps `OIDCPrivateKeyPEM` and the k3s `JoinToken`, so both are persisted in the `eks-account-{acct}` cluster record, which the gateway also reads. Whether that record is encrypted at rest is unconfirmed; if it is not, a reader of the bucket can forge service-account tokens for the cluster's OIDC issuer and join nodes to it. This predates the refactor and is not fixed in it. Removal condition: the key and token are held by a dedicated secret store (a future KMS or Secrets Manager capability is the likely owner) and the cluster record keeps only a reference, with a test proving the persisted record contains neither secret.
- `operator/cli` (dependency): the command tree still imports domain and service implementations directly (`handlers/{iam,eks}`, `domains/admission/quota`, `gateway`, `domains/ochre`, `vpcd`, the Viperblock adapter and runtime roles). The ADR's `cmd/spinifex` current-to-target map row requires command packages to reach domains through public capabilities; splitting supported use cases into `operator/*` packages is later work.
- `operator/cli` (dependency): the command tree still imports domain and service implementations directly (`handlers/{iam,eks}`, `domains/admission/quota`, `domains/ochre`, `vpcd`, the Viperblock adapter and runtime roles). The ADR's `cmd/spinifex` current-to-target map row requires command packages to reach domains through public capabilities; splitting supported use cases into `operator/*` packages is later work.
- `providers/ebs/viperblock` → `runtime/compute/nbd` and `runtime/host/process` (dependency, moved unchanged from `services/viperblockd`): the adapter formats NBD endpoints and manages nbdkit processes through runtime packages. ADR-0001 does not list providers → runtime as forbidden and the layering test does not enforce it (same category as `providers/ebs/conformance` → `runtime/compute/nbd`), but a provider should not depend on node runtime. Clears when NBD endpoint naming has a provider-facing home (a candidate for `contracts/ebs/v1`) and process supervision is injected by the role. The adapter also imports `providers/objectstore`, a provider-to-provider edge that is allowed.
- `gateway/ec2.go` (registration): the EC2 dispatch map stays a `GatewayConfig` method in `gateway` and imports every `domains/ec2/awsapi/*` package; `gateway/ec2_idempotency.go` also imports `domains/ec2/awsapi/idem` (`KVBucket`, `TTL`, `TokenAndParams`) for the dispatch-time idempotency wrapper. ADR-0001 makes the domain own its action registration; moving the map waits on the generic ingress registration seam and must not be done as a package move.
- `gateway` (service registration): the root `gateway` dispatch for IAM, STS, EKS, RDS, ECS and ELBv2 imports `domains/{iam,sts,eks,rds,ecs,elbv2}/awsapi`, which only AWSGW may wire. It clears when each service registers through the ADR-0002 ingress registration seam; as with EC2, moving the dispatch must not be done as a package move.
- `gateway/ec2.go` (registration): the action table, idempotency and validation moved to `domains/ec2/awsapi` behind `Handler` and `Deps` (slice 9a, `spinifex#1354`), but the gateway still composes the handler and serves it from the legacy switch, so `gateway` keeps importing `domains/ec2/awsapi` and `domains/ec2/awsapi/instance`. Registering EC2 through the ADR-0002 seam and emptying the legacy switch is slice 9b.
- `gateway` (service registration): the root `gateway` dispatch for IAM, STS, EKS, RDS and ECS imports `domains/{iam,sts,eks,rds,ecs}/awsapi`, which only AWSGW may wire. It clears when each service registers through the ADR-0002 ingress registration seam; as with EC2, moving the dispatch must not be done as a package move.
- `operator/accountlifecycle` (dependency): its EC2 reapers call `domains/ec2/awsapi/*` action adapters directly, as they did under `gateway/ec2`. The amended ADR-0001 `accountteardown` row keeps replacing these with domain-owned cleanup capabilities as deferred debt; it was not a relocation prerequisite. It also holds a read-only consumer of the EC2 platform-inventory contract (`contracts/ec2/v1`, `ec2.v1.PlatformInventory`), served by `domains/ec2/accountinventory`; only the explicit dry run (`Engine.DryRun`) calls it, to log a shadow comparison, and the key-pair, placement-group and launch-template reapers still use the `awsapi` adapters. A domain-owned cleanup capability for the proven-safe subset of those three kinds (key pairs and empty placement groups no instance, spot request or live launch-template version names, and launch-template residue; never live launch templates) exists as `domains/ec2/accountcleanup` (a Go interface with injected owners and reference readers, no NATS subject); nothing calls it yet. It also holds the StageAccount settings-seal reaper, a consumer of `contracts/ec2/v1`'s `ec2.v1.SealAccountSettings` and per-node `ec2.v1.AccountSettingsSealCapability.<node>` subjects, served by `domains/ec2/account` through the daemon; the reaper reads no KV and imports no domain package, reaching both subjects only through NATS and the configured node set `NewClusterEngine` already receives.
- `domains/admission/quota` (dependency, moved from `handlers/quota` unchanged): `live.go` imports `domains/elbv2/awsapi` and `domains/rds/instance` for its live ELBv2 and RDS counters; `records.go` imports `runtime/compute/vm` for the instance-record vCPU sweep; `live.go`, `vcpu.go` and `storage.go` call `domains/ec2/awsapi/{vpc,eip,volume}` describe adapters to count VPCs, subnets, addresses and volumes, as they did under `gateway/ec2`. The ADR's `handlers/quota` current-to-target map row replaces these with reader capabilities once their final locations exist.
- `domains/ec2/awsapi/instance` (dependency): imports `domains/iam` (`IAMService` in `DescribeInstances.go`, `RunInstances.go` and `IamInstanceProfileAssociation.go`; `InstanceProfile` in the latter two), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge needs an IAM capability; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
- `domains/ec2/awsapi/spotinstance` (dependency): imports `domains/iam.IAMService` and `domains/admission/quota.Service` in `RequestSpotInstances.go`.
- `domains/ec2/awsapi/instance` (dependency): imports `domains/iam` for the `InstanceProfile` type that `InstanceProfiles.ResolveInstanceProfile` returns (`RunInstances.go`, `IamInstanceProfileAssociation.go`), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge is now the two-method `InstanceProfiles` interface plus that type; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
- Instance adapter consumers outside EC2: `domains/rds/awsapi/enginecatalog.go`, `runtime/daemon/eks_cp_control.go`, `runtime/daemon/rds_deps.go`, `domains/admission/quota`'s `vcpu.go` and `gateway/iam.go` (`CountInstanceProfileAssociations`) call `domains/ec2/awsapi/instance` directly; each needs an EC2 capability rather than the AWS action adapter.
- `runtime/daemon` (Q-28 ownership and deployment): the daemon moved whole, keeping its current ownership, deployment and `daemon → gateway → handlers` dependencies. Q-28's write-side consolidation and any daemon decomposition or split remain deferred by ADR-0001 and are not paid by the move.
@@ -673,11 +686,11 @@ series above; "needs decision" is open.
| `handlers/elbv2` | — | Done. The implementation moved to `domains/elbv2` (package `elbv2`) with no API, NATS subject, storage or behaviour change. Q-30's data-plane boundary remains deferred debt; the amended ADR-0001 row makes it not a relocation prerequisite. Its cross-domain, IAM and `agents/elbv2` imports remain dependency-rule debt, with their allowlist entries now sourced from `domains/elbv2`. |
| `handlers/iam/mock` | — | Done. Moved with the implementation to `domains/iam/mock`. |
| `gateway` (root) | Generic HTTP route assembly, SigV4/registry authentication, the policy gate, request-audit middleware, request-context keys, and the legacy per-service dispatch switch for every service not yet registered. | Deferred: the ADR-0002 phases above (auth/IAM and request-context boundary design) gate moving the audit, context-key and legacy-dispatch mechanics. |
| `gateway/bedrock` | — | Done. The whole package moved into `domains/ochre` in `spinifex#1266` with its package-private core and tests intact and no behaviour change. Separating the action adapters into `domains/ochre/awsapi` remains deferred, recorded as `domains/ochre` (action adapters) debt above. |
| `gateway/bedrock` | — | Done. The whole package moved into `domains/ochre` in `spinifex#1266` with its package-private core and tests intact and no behaviour change. Separating the action adapters into `domains/ochre/awsapi` remains deferred, recorded as `domains/ochre` (action adapters) debt above. The action adapters then moved to `domains/ochre/awsapi` behind one ADR-0002 registration in `spinifex#1351`. |
| `gateway/{iam,sts,eks,rds}` | — | Done. The action adapters moved to `domains/{iam,sts,eks,rds}/awsapi` in `spinifex#1265` with no API, NATS subject, storage or behaviour change. ADR-0001 does not make ADR-0002 registration a prerequisite for relocating a domain's adapters, as the EC2 adapter moves already showed; the root `gateway` dispatch that still imports them is recorded `gateway` (service registration) debt. |
| `gateway/{ecs,elbv2}` | — | Done. The action adapters moved to `domains/{ecs,elbv2}/awsapi` in `spinifex#1273` with no API, NATS subject, storage or behaviour change; the root `gateway` dispatch that still imports them is recorded `gateway` (service registration) debt. |
| `gateway/spx` | `promote_image.go` and `spx.go` dispatch (`storage.go` already consumes `contracts/operator/v1`). | Needs decision: which package owns operator-plane action dispatch registration — an `ingress/aws`-style seam for the operator plane, or `gateway` stays its home. |
| `gateway/tagging` | — | Done. `domains/tagging` owns the read-only cross-resource `GetResources` aggregation and its API semantics (`0d830e901`, `aedd0b2c4`); AWSGW composes the owner listers, the EC2 tag index and ELBv2 (`f3545e3ea`); the root `gateway` keeps only the legacy dispatch in `gateway/tagging.go`. The package is removed, and its two allowlist rows and frozen-legacy entry are gone. Owner tag mutation capabilities remain future direction, so the RDS and ECS tag-writing edges above stay open. |
| `gateway/spx` | — | Done. The gather helpers, storage-status config and image-promote request moved to `operator/admin` with no behaviour change and the package is removed. The `spinifex` Query dispatch and the `/admin/{method}` handlers that call them now live in `operator/admin` as one seam registration. |
| `gateway/tagging` | — | Done. `domains/tagging` owns the read-only cross-resource `GetResources` aggregation and its API semantics (`0d830e901`, `aedd0b2c4`); AWSGW composes the owner listers, the EC2 tag index and ELBv2 (`f3545e3ea`); the dispatch moved to `domains/tagging/awsapi` and the root `gateway` no longer serves Tagging. The package is removed, and its two allowlist rows and frozen-legacy entry are gone. Owner tag mutation capabilities remain future direction, so the RDS and ECS tag-writing edges above stay open. |
| `services/viperblockd` | — | Done. The adapter is `providers/ebs/viperblock` (with the read-only `dirty` reader), the shared vocabulary is `contracts/viperblockd/legacy/v1`, and the role is `runtime/roles/viperblockd`. `services/` is empty. |
| `utils` | — | Done and deleted. `ec2_subject.go` went to `contracts/ec2/v1`, the vpcd protocol to `contracts/network/v1` with the typed client in `domains/network/projection`, and the master-key loader to `domains/ec2/ebs/encryption`. |
| `vpcd` | — | Done. The IMDS local-file reader and record-space opener moved to `domains/ec2/instance/state` and are injected through `guestmetadata.LocalStateReader`/`RecordLoader` (recorded above), so `vpcd` no longer imports `daemon` directly; its inter-process boundary now exists (`contracts/network/v1`, with `domains/network/subscribers` as the consumer and `domains/network/projection` as the producer client). The package moved unchanged to `runtime/roles/vpcd` (recorded below), and the legacy directory is removed. |
@@ -696,7 +709,7 @@ ADR-0001 completion is structural, so none of these is cleared by remediating se
| Class | Paths | Status |
|---|---|---|
| No missing boundary (candidate moves) | `handlers/ecs/bus` → `contracts/ecs/v1` (`a130f6b8e`); `admin/images_promote.go` → `domains/ec2/image` (`263c58167`); `admin` bootstrap account identifiers → `foundation/aws/identifiers` (`3fdaa54d9`); `admin` credential-format generators → `foundation/aws/credentials` `0368e861a`, retiring `admin` | Done. |
| One narrow missing boundary | Done: `handlers/sts` EKS JWKS lookup → `handlers/eks` `ClusterJWKSResolver` (`254d898a3`); `gateway/tagging` → `domains/tagging` with AWSGW-composed listers (`0d830e901`, `f3545e3ea`, `aedd0b2c4`). Open: `gateway/spx` (owner of operator-plane dispatch registration) | Each needs one decision or one capability, then a mechanical move. |
| One narrow missing boundary | Done: `handlers/sts` EKS JWKS lookup → `handlers/eks` `ClusterJWKSResolver` (`254d898a3`); `gateway/tagging` → `domains/tagging` with AWSGW-composed listers (`0d830e901`, `f3545e3ea`, `aedd0b2c4`); `gateway/spx` → `operator/admin`. Done: the `spinifex` service and `/admin` dispatch registration, owned by `operator/admin` | Each needs one decision or one capability, then a mechanical move. |
| Broader boundary or follows another package | `gateway` root (ADR-0002 auth and request-context phase) | Recorded debt with removal conditions above. |
Per-consumer IAM capabilities are the widest unblocker: they gate STS's IAM reads, Ochre's IAM reads, the ELBv2 role path and the ECS, EKS and RDS ensurers.
-1
View File
@@ -73,7 +73,6 @@ The cluster plane is peer-scoped. On a single-NIC node the planes collapse onto
| Ports | Serving |
|---|---|
| TCP 4222, 4248 | NATS client and cluster routes |
| TCP 4223, 4249 | SQS message store client and cluster routes |
| TCP 4432 | Formation, then the daemon cluster manager |
| TCP and UDP 5300 | Internal DNS |
| TCP 6641, 6642 | OVN northbound and southbound |
+54 -2
View File
@@ -2,8 +2,8 @@
"schema_version": 1,
"model_source": {
"repository": "github.com/aws/api-models-aws",
"commit": "9b1a4929274443d8671ea8acfda7afbe856e29e9",
"commit_date": "2026-09-25"
"commit": "e10d6a1c5ac36af7c0c61d1f67b3c3ebbe41de49",
"commit_date": "2026-10-28"
},
"services": [
{
@@ -300,6 +300,7 @@
"DeleteCapacityManagerDataExport",
"DeleteCarrierGateway",
"DeleteClientVpnEndpoint",
"DeleteClientVpnEndpointAuthorizationPolicy",
"DeleteClientVpnRoute",
"DeleteCoipCidr",
"DeleteCoipPool",
@@ -681,7 +682,9 @@
"GetCapacityManagerMetricData",
"GetCapacityManagerMetricDimensions",
"GetCapacityManagerMonitoredTagKeys",
"GetCapacityManagerSupportabilityScores",
"GetCapacityReservationUsage",
"GetClientVpnEndpointAuthorizationPolicy",
"GetCoipPoolUsage",
"GetConsoleOutput",
"GetConsoleScreenshot",
@@ -695,6 +698,7 @@
"GetHostReservationPurchasePreview",
"GetImageAncestry",
"GetImageBlockPublicAccessState",
"GetImageFindings",
"GetInstanceMetadataDefaults",
"GetInstanceTpmEkPub",
"GetInstanceTypesFromInstanceRequirements",
@@ -734,6 +738,7 @@
"GetSnapshotBlockPublicAccessState",
"GetSpotPlacementScores",
"GetSubnetCidrReservations",
"GetTransitGatewayAttachmentNatConfiguration",
"GetTransitGatewayAttachmentPropagations",
"GetTransitGatewayMeteringPolicyEntries",
"GetTransitGatewayMulticastDomainAssociations",
@@ -766,6 +771,7 @@
"ModifyCapacityReservation",
"ModifyCapacityReservationFleet",
"ModifyClientVpnEndpoint",
"ModifyClientVpnEndpointAuthorizationPolicy",
"ModifyDefaultCreditSpecification",
"ModifyEbsDefaultKmsKeyId",
"ModifyFleet",
@@ -814,6 +820,7 @@
"ModifyTrafficMirrorFilterRule",
"ModifyTrafficMirrorSession",
"ModifyTransitGateway",
"ModifyTransitGatewayAttachmentNatConfiguration",
"ModifyTransitGatewayMeteringPolicy",
"ModifyTransitGatewayPolicyTableEntry",
"ModifyTransitGatewayPrefixListReference",
@@ -1335,6 +1342,7 @@
"DeleteCapacityManagerDataExport",
"DeleteCarrierGateway",
"DeleteClientVpnEndpoint",
"DeleteClientVpnEndpointAuthorizationPolicy",
"DeleteClientVpnRoute",
"DeleteCoipCidr",
"DeleteCoipPool",
@@ -1653,7 +1661,9 @@
"GetCapacityManagerMetricData",
"GetCapacityManagerMetricDimensions",
"GetCapacityManagerMonitoredTagKeys",
"GetCapacityManagerSupportabilityScores",
"GetCapacityReservationUsage",
"GetClientVpnEndpointAuthorizationPolicy",
"GetCoipPoolUsage",
"GetConsoleScreenshot",
"GetDeclarativePoliciesReportSummary",
@@ -1665,6 +1675,7 @@
"GetHostReservationPurchasePreview",
"GetImageAncestry",
"GetImageBlockPublicAccessState",
"GetImageFindings",
"GetInstanceMetadataDefaults",
"GetInstanceTpmEkPub",
"GetInstanceTypesFromInstanceRequirements",
@@ -1701,6 +1712,7 @@
"GetSnapshotBlockPublicAccessState",
"GetSpotPlacementScores",
"GetSubnetCidrReservations",
"GetTransitGatewayAttachmentNatConfiguration",
"GetTransitGatewayAttachmentPropagations",
"GetTransitGatewayMeteringPolicyEntries",
"GetTransitGatewayMulticastDomainAssociations",
@@ -1732,6 +1744,7 @@
"ModifyCapacityReservation",
"ModifyCapacityReservationFleet",
"ModifyClientVpnEndpoint",
"ModifyClientVpnEndpointAuthorizationPolicy",
"ModifyDefaultCreditSpecification",
"ModifyEbsDefaultKmsKeyId",
"ModifyFleet",
@@ -1773,6 +1786,7 @@
"ModifyTrafficMirrorFilterRule",
"ModifyTrafficMirrorSession",
"ModifyTransitGateway",
"ModifyTransitGatewayAttachmentNatConfiguration",
"ModifyTransitGatewayMeteringPolicy",
"ModifyTransitGatewayPolicyTableEntry",
"ModifyTransitGatewayPrefixListReference",
@@ -1897,6 +1911,7 @@
"DescribeImages",
"DescribePullThroughCacheRules",
"DescribeRegistry",
"DescribeReplicationBackfillResults",
"DescribeRepositories",
"DescribeRepositoryCreationTemplates",
"GetAccountSetting",
@@ -1912,6 +1927,7 @@
"ListImageReferrers",
"ListImages",
"ListPullTimeUpdateExclusions",
"ListReplicationBackfills",
"ListTagsForResource",
"PutAccountSetting",
"PutImage",
@@ -1926,6 +1942,8 @@
"SetRepositoryPolicy",
"StartImageScan",
"StartLifecyclePolicyPreview",
"StartReplicationBackfill",
"StopReplicationBackfill",
"TagResource",
"UntagResource",
"UpdateImageStorageClass",
@@ -2031,14 +2049,18 @@
"DescribeImageReplicationStatus",
"DescribeImageSigningStatus",
"DescribePullThroughCacheRules",
"DescribeReplicationBackfillResults",
"DescribeRepositoryCreationTemplates",
"GetAccountSetting",
"GetSigningConfiguration",
"ListImageReferrers",
"ListPullTimeUpdateExclusions",
"ListReplicationBackfills",
"PutAccountSetting",
"PutSigningConfiguration",
"RegisterPullTimeUpdateExclusion",
"StartReplicationBackfill",
"StopReplicationBackfill",
"UpdateImageStorageClass",
"UpdatePullThroughCacheRule",
"UpdateRepositoryCreationTemplate",
@@ -2058,11 +2080,13 @@
"note": "",
"operations": {
"modelled": [
"AssociateGatewayWithCluster",
"ContinueServiceDeployment",
"CreateCapacityProvider",
"CreateCluster",
"CreateDaemon",
"CreateExpressGatewayService",
"CreateGateway",
"CreateService",
"CreateTaskSet",
"DeleteAccountSetting",
@@ -2072,6 +2096,7 @@
"DeleteDaemon",
"DeleteDaemonTaskDefinition",
"DeleteExpressGatewayService",
"DeleteGateway",
"DeleteService",
"DeleteTaskDefinitions",
"DeleteTaskSet",
@@ -2085,12 +2110,16 @@
"DescribeDaemonRevisions",
"DescribeDaemonTaskDefinition",
"DescribeExpressGatewayService",
"DescribeGateway",
"DescribeGatewayResources",
"DescribeGatewayRoute",
"DescribeServiceDeployments",
"DescribeServiceRevisions",
"DescribeServices",
"DescribeTaskDefinition",
"DescribeTaskSets",
"DescribeTasks",
"DisassociateGatewayFromCluster",
"DiscoverPollEndpoint",
"ExecuteCommand",
"GetTaskProtection",
@@ -2101,6 +2130,9 @@
"ListDaemonDeployments",
"ListDaemonTaskDefinitions",
"ListDaemons",
"ListGatewayClusterAssociations",
"ListGatewayRoutes",
"ListGateways",
"ListServiceDeployments",
"ListServices",
"ListServicesByNamespace",
@@ -2131,6 +2163,7 @@
"UpdateContainerInstancesState",
"UpdateDaemon",
"UpdateExpressGatewayService",
"UpdateGateway",
"UpdateService",
"UpdateServicePrimaryTaskSet",
"UpdateTaskProtection",
@@ -2219,15 +2252,18 @@
],
"unsupported": [],
"missing": [
"AssociateGatewayWithCluster",
"ContinueServiceDeployment",
"CreateDaemon",
"CreateExpressGatewayService",
"CreateGateway",
"CreateTaskSet",
"DeleteAccountSetting",
"DeleteAttributes",
"DeleteDaemon",
"DeleteDaemonTaskDefinition",
"DeleteExpressGatewayService",
"DeleteGateway",
"DeleteTaskDefinitions",
"DeleteTaskSet",
"DescribeDaemon",
@@ -2235,9 +2271,13 @@
"DescribeDaemonRevisions",
"DescribeDaemonTaskDefinition",
"DescribeExpressGatewayService",
"DescribeGateway",
"DescribeGatewayResources",
"DescribeGatewayRoute",
"DescribeServiceDeployments",
"DescribeServiceRevisions",
"DescribeTaskSets",
"DisassociateGatewayFromCluster",
"DiscoverPollEndpoint",
"ExecuteCommand",
"GetTaskProtection",
@@ -2245,6 +2285,9 @@
"ListDaemonDeployments",
"ListDaemonTaskDefinitions",
"ListDaemons",
"ListGatewayClusterAssociations",
"ListGatewayRoutes",
"ListGateways",
"ListServiceDeployments",
"PutAccountSettingDefault",
"PutAttributes",
@@ -2257,6 +2300,7 @@
"UpdateContainerAgent",
"UpdateDaemon",
"UpdateExpressGatewayService",
"UpdateGateway",
"UpdateServicePrimaryTaskSet",
"UpdateTaskProtection",
"UpdateTaskSet"
@@ -3099,6 +3143,7 @@
"ApplyPendingMaintenanceAction",
"AuthorizeDBSecurityGroupIngress",
"BacktrackDBCluster",
"CancelDBCapacityReservation",
"CancelExportTask",
"CopyDBClusterParameterGroup",
"CopyDBClusterSnapshot",
@@ -3107,6 +3152,7 @@
"CopyOptionGroup",
"CreateBlueGreenDeployment",
"CreateCustomDBEngineVersion",
"CreateDBCapacityReservation",
"CreateDBCluster",
"CreateDBClusterEndpoint",
"CreateDBClusterParameterGroup",
@@ -3150,6 +3196,7 @@
"DescribeAccountAttributes",
"DescribeBlueGreenDeployments",
"DescribeCertificates",
"DescribeDBCapacityReservations",
"DescribeDBClusterAutomatedBackups",
"DescribeDBClusterBacktracks",
"DescribeDBClusterEndpoints",
@@ -3204,6 +3251,7 @@
"ModifyCertificates",
"ModifyCurrentDBClusterCapacity",
"ModifyCustomDBEngineVersion",
"ModifyDBCapacityReservation",
"ModifyDBCluster",
"ModifyDBClusterEndpoint",
"ModifyDBClusterParameterGroup",
@@ -3354,6 +3402,7 @@
"ApplyPendingMaintenanceAction",
"AuthorizeDBSecurityGroupIngress",
"BacktrackDBCluster",
"CancelDBCapacityReservation",
"CancelExportTask",
"CopyDBClusterParameterGroup",
"CopyDBClusterSnapshot",
@@ -3362,6 +3411,7 @@
"CopyOptionGroup",
"CreateBlueGreenDeployment",
"CreateCustomDBEngineVersion",
"CreateDBCapacityReservation",
"CreateDBClusterEndpoint",
"CreateDBClusterParameterGroup",
"CreateDBClusterSnapshot",
@@ -3392,6 +3442,7 @@
"DescribeAccountAttributes",
"DescribeBlueGreenDeployments",
"DescribeCertificates",
"DescribeDBCapacityReservations",
"DescribeDBClusterAutomatedBackups",
"DescribeDBClusterBacktracks",
"DescribeDBClusterEndpoints",
@@ -3433,6 +3484,7 @@
"ModifyCertificates",
"ModifyCurrentDBClusterCapacity",
"ModifyCustomDBEngineVersion",
"ModifyDBCapacityReservation",
"ModifyDBClusterEndpoint",
"ModifyDBClusterParameterGroup",
"ModifyDBClusterSnapshotAttribute",
+15
View File
@@ -28,6 +28,21 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
An AMI cannot be shared with another account or made public. `ModifyImageAttribute` refuses `LaunchPermission`, `UserIds`, `UserGroups`, `OrganizationArns`, `OrganizationalUnitArns` and `OperationType`, and `ResetImageAttribute` resets only `description`, not `launchPermission`. Each refusal is `InvalidParameterValue`. Terraform's `aws_ami_launch_permission` therefore cannot be created. `description` is the only image attribute that can be modified or reset.
### RegisterImage Registers From a Snapshot
`RegisterImage` takes a root EBS snapshot in `BlockDeviceMappings` and writes an image that points at it.
`EnaSupport` is supported, with `true` and `false` both accepted and stored on the image.
`DescribeImages` returns `EnaSupport` when the image has one and leaves the member out when it was never set, as AWS does.
`CopyImage` carries the value to the copy, and `CreateImage` sets it to `true` because every instance reports ENA support.
`RunInstances` does not read the flag, so an image registered with `EnaSupport=false` is not held back from instance types that need ENA on AWS, since guests use virtio networking either way.
`RegisterImage` refuses `ImageLocation`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport` and `SriovNetSupport`, and any `VirtualizationType` other than `hvm`.
Each refusal is `InvalidParameterValue`, and the message names the parameter and the value, for example `Value (aki-123) for parameter KernelId is invalid. KernelId is not supported.`
The same form is used for a bad `Architecture`, `BootMode` or `Name` length.
A missing `Name`, `BlockDeviceMappings` or root snapshot answers `MissingParameter` and names what is missing.
Registering with `--ena-support`, `--no-ena-support` and neither, copying the image, and three refusals were run on a three-node cluster with the AWS CLI.
Launching an instance from an `EnaSupport` image was not run, so the launch path is unchanged by construction and not by test.
### Operations
| Operation |
+15
View File
@@ -5,3 +5,18 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
### AMI Launch Permissions Are Refused
An AMI cannot be shared with another account or made public. `ModifyImageAttribute` refuses `LaunchPermission`, `UserIds`, `UserGroups`, `OrganizationArns`, `OrganizationalUnitArns` and `OperationType`, and `ResetImageAttribute` resets only `description`, not `launchPermission`. Each refusal is `InvalidParameterValue`. Terraform's `aws_ami_launch_permission` therefore cannot be created. `description` is the only image attribute that can be modified or reset.
### RegisterImage Registers From a Snapshot
`RegisterImage` takes a root EBS snapshot in `BlockDeviceMappings` and writes an image that points at it.
`EnaSupport` is supported, with `true` and `false` both accepted and stored on the image.
`DescribeImages` returns `EnaSupport` when the image has one and leaves the member out when it was never set, as AWS does.
`CopyImage` carries the value to the copy, and `CreateImage` sets it to `true` because every instance reports ENA support.
`RunInstances` does not read the flag, so an image registered with `EnaSupport=false` is not held back from instance types that need ENA on AWS, since guests use virtio networking either way.
`RegisterImage` refuses `ImageLocation`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport` and `SriovNetSupport`, and any `VirtualizationType` other than `hvm`.
Each refusal is `InvalidParameterValue`, and the message names the parameter and the value, for example `Value (aki-123) for parameter KernelId is invalid. KernelId is not supported.`
The same form is used for a bad `Architecture`, `BootMode` or `Name` length.
A missing `Name`, `BlockDeviceMappings` or root snapshot answers `MissingParameter` and names what is missing.
Registering with `--ena-support`, `--no-ena-support` and neither, copying the image, and three refusals were run on a three-node cluster with the AWS CLI.
Launching an instance from an `EnaSupport` image was not run, so the launch path is unchanged by construction and not by test.
+40
View File
@@ -52,6 +52,46 @@ Measured on a three-node cluster with a 5 second interval and thresholds of 2:
A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
### Forwarding headers
An Application Load Balancer sets three headers on every HTTP and HTTPS request it forwards.
`X-Forwarded-For` carries the client address, `X-Forwarded-Proto` is `http` or `https` for the listener that accepted the request, and `X-Forwarded-Port` is that listener's port.
`X-Forwarded-Proto` and `X-Forwarded-Port` are always overwritten, so a value sent by the client never reaches the target.
Network Load Balancers do not touch the stream.
`routing.http.xff_header_processing.mode` decides what happens to `X-Forwarded-For`:
| Mode | Behaviour |
| --- | --- |
| `append` (default) | Keeps what the client sent and adds the client address last, as one comma-separated line such as `203.0.113.9, 192.0.2.10`. |
| `preserve` | Forwards the client's value unchanged and adds nothing. The header is absent when the client sent none. |
| `remove` | Drops the header, including any client value. |
`routing.http.xff_client_port.enabled` (default `false`) appends the client's source port in `append` mode, as `192.0.2.10:54321`.
An IPv6 client is written `[2001:db8::1]:54321`.
The setting has no effect in `preserve` and `remove` modes.
`ModifyLoadBalancerAttributes` accepts only `append`, `preserve` or `remove` for the mode and `true` or `false` for the client port, and rejects anything else with `ValidationError`.
A change re-renders the load balancer configuration, and a call that leaves both values as they were does not.
In a live test each change reached the target within one 3 second poll.
A client that sends `X-Forwarded-For` more than once has its lines joined into the single comma-separated value.
At most eight client lines are joined, and any beyond the eighth are dropped, while the client address is still added last.
Every other `routing.http.*` attribute is stored and returned by `DescribeLoadBalancerAttributes` but has no effect on the data plane.
Measured on a three-node cluster, with the caller on the first node:
| Listener | `X-Forwarded-For` the target saw |
| --- | --- |
| Internet-facing HTTP | The node's real address, `192.168.0.120` |
| Internet-facing HTTPS | The same address, with `https` and port 443 |
| Internal HTTP | The calling instance's private address |
| Internet-facing HTTP with a client value of `203.0.113.9` | `203.0.113.9, 192.168.0.120` on one line |
A packet capture on the load balancer's interface showed HAProxy receiving the real client address, so nothing translates it first.
Behaviour against AWS was not compared, and the cases above were not run with an IPv6 client.
### Operations
| Operation |
+40
View File
@@ -29,3 +29,43 @@ Measured on a three-node cluster with a 5 second interval and thresholds of 2:
A TCP connection held through a Network Load Balancer survived four configuration reloads while the other backend changed state, because the agent applies nginx with a graceful reload and HAProxy with a graceful restart.
The HAProxy configuration with the backup servers was accepted by a live Application Load Balancer, but its rendered text was not dumped and checked by hand.
### Forwarding headers
An Application Load Balancer sets three headers on every HTTP and HTTPS request it forwards.
`X-Forwarded-For` carries the client address, `X-Forwarded-Proto` is `http` or `https` for the listener that accepted the request, and `X-Forwarded-Port` is that listener's port.
`X-Forwarded-Proto` and `X-Forwarded-Port` are always overwritten, so a value sent by the client never reaches the target.
Network Load Balancers do not touch the stream.
`routing.http.xff_header_processing.mode` decides what happens to `X-Forwarded-For`:
| Mode | Behaviour |
| --- | --- |
| `append` (default) | Keeps what the client sent and adds the client address last, as one comma-separated line such as `203.0.113.9, 192.0.2.10`. |
| `preserve` | Forwards the client's value unchanged and adds nothing. The header is absent when the client sent none. |
| `remove` | Drops the header, including any client value. |
`routing.http.xff_client_port.enabled` (default `false`) appends the client's source port in `append` mode, as `192.0.2.10:54321`.
An IPv6 client is written `[2001:db8::1]:54321`.
The setting has no effect in `preserve` and `remove` modes.
`ModifyLoadBalancerAttributes` accepts only `append`, `preserve` or `remove` for the mode and `true` or `false` for the client port, and rejects anything else with `ValidationError`.
A change re-renders the load balancer configuration, and a call that leaves both values as they were does not.
In a live test each change reached the target within one 3 second poll.
A client that sends `X-Forwarded-For` more than once has its lines joined into the single comma-separated value.
At most eight client lines are joined, and any beyond the eighth are dropped, while the client address is still added last.
Every other `routing.http.*` attribute is stored and returned by `DescribeLoadBalancerAttributes` but has no effect on the data plane.
Measured on a three-node cluster, with the caller on the first node:
| Listener | `X-Forwarded-For` the target saw |
| --- | --- |
| Internet-facing HTTP | The node's real address, `192.168.0.120` |
| Internet-facing HTTPS | The same address, with `https` and port 443 |
| Internal HTTP | The calling instance's private address |
| Internet-facing HTTP with a client value of `203.0.113.9` | `203.0.113.9, 192.168.0.120` on one line |
A packet capture on the load balancer's interface showed HAProxy receiving the real client address, so nothing translates it first.
Behaviour against AWS was not compared, and the cases above were not run with an IPv6 client.
+31 -5
View File
@@ -4,7 +4,7 @@ Generated by `make aws-model-coverage`. This file is not published: it carries t
**Not implemented** is the candidate work. **Not applicable** is declared in `pages.json` and is not work: those operations describe something this platform will not have.
Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
Pinned model source: `github.com/aws/api-models-aws@e10d6a1c5ac3, 2026-10-28`.
## acm
@@ -46,7 +46,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
## ec2
`2016-11-15` — 127 of 804 modelled operations implemented (15.8%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
`2016-11-15` — 127 of 811 modelled operations implemented (15.7%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
### Not implemented
@@ -190,6 +190,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
- `DeleteApplicationStatusCheck`
- `DeleteCapacityManagerDataExport`
- `DeleteClientVpnEndpoint`
- `DeleteClientVpnEndpointAuthorizationPolicy`
- `DeleteClientVpnRoute`
- `DeleteCoipCidr`
- `DeleteCoipPool`
@@ -484,7 +485,9 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
- `GetCapacityManagerMetricData`
- `GetCapacityManagerMetricDimensions`
- `GetCapacityManagerMonitoredTagKeys`
- `GetCapacityManagerSupportabilityScores`
- `GetCapacityReservationUsage`
- `GetClientVpnEndpointAuthorizationPolicy`
- `GetCoipPoolUsage`
- `GetConsoleScreenshot`
- `GetDeclarativePoliciesReportSummary`
@@ -496,6 +499,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
- `GetHostReservationPurchasePreview`
- `GetImageAncestry`
- `GetImageBlockPublicAccessState`
- `GetImageFindings`
- `GetInstanceMetadataDefaults`
- `GetInstanceTpmEkPub`
- `GetInstanceTypesFromInstanceRequirements`
@@ -532,6 +536,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
- `GetSnapshotBlockPublicAccessState`
- `GetSpotPlacementScores`
- `GetSubnetCidrReservations`
- `GetTransitGatewayAttachmentNatConfiguration`
- `GetTransitGatewayAttachmentPropagations`
- `GetTransitGatewayMeteringPolicyEntries`
- `GetTransitGatewayMulticastDomainAssociations`
@@ -563,6 +568,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
- `ModifyCapacityReservation`
- `ModifyCapacityReservationFleet`
- `ModifyClientVpnEndpoint`
- `ModifyClientVpnEndpointAuthorizationPolicy`
- `ModifyDefaultCreditSpecification`
- `ModifyEbsDefaultKmsKeyId`
- `ModifyFleet`
@@ -601,6 +607,7 @@ Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
- `ModifyTrafficMirrorFilterRule`
- `ModifyTrafficMirrorSession`
- `ModifyTransitGateway`
- `ModifyTransitGatewayAttachmentNatConfiguration`
- `ModifyTransitGatewayMeteringPolicy`
- `ModifyTransitGatewayPolicyTableEntry`
- `ModifyTransitGatewayPrefixListReference`
@@ -762,7 +769,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
## ecr
`2015-09-21` — 23 of 58 modelled operations implemented (39.7%), 10 stubbed, 0 not applicable, 3 registered outside the pinned model.
`2015-09-21` — 23 of 62 modelled operations implemented (37.1%), 10 stubbed, 0 not applicable, 3 registered outside the pinned model.
### Not implemented
@@ -776,14 +783,18 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `DescribeImageReplicationStatus`
- `DescribeImageSigningStatus`
- `DescribePullThroughCacheRules`
- `DescribeReplicationBackfillResults`
- `DescribeRepositoryCreationTemplates`
- `GetAccountSetting`
- `GetSigningConfiguration`
- `ListImageReferrers`
- `ListPullTimeUpdateExclusions`
- `ListReplicationBackfills`
- `PutAccountSetting`
- `PutSigningConfiguration`
- `RegisterPullTimeUpdateExclusion`
- `StartReplicationBackfill`
- `StopReplicationBackfill`
- `UpdateImageStorageClass`
- `UpdatePullThroughCacheRule`
- `UpdateRepositoryCreationTemplate`
@@ -819,19 +830,22 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
## ecs
`2014-11-13` — 31 of 77 modelled operations implemented (40.3%), 5 stubbed, 0 not applicable, 3 registered outside the pinned model.
`2014-11-13` — 31 of 88 modelled operations implemented (35.2%), 5 stubbed, 0 not applicable, 3 registered outside the pinned model.
### Not implemented
- `AssociateGatewayWithCluster`
- `ContinueServiceDeployment`
- `CreateDaemon`
- `CreateExpressGatewayService`
- `CreateGateway`
- `CreateTaskSet`
- `DeleteAccountSetting`
- `DeleteAttributes`
- `DeleteDaemon`
- `DeleteDaemonTaskDefinition`
- `DeleteExpressGatewayService`
- `DeleteGateway`
- `DeleteTaskDefinitions`
- `DeleteTaskSet`
- `DescribeDaemon`
@@ -839,9 +853,13 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `DescribeDaemonRevisions`
- `DescribeDaemonTaskDefinition`
- `DescribeExpressGatewayService`
- `DescribeGateway`
- `DescribeGatewayResources`
- `DescribeGatewayRoute`
- `DescribeServiceDeployments`
- `DescribeServiceRevisions`
- `DescribeTaskSets`
- `DisassociateGatewayFromCluster`
- `DiscoverPollEndpoint`
- `ExecuteCommand`
- `GetTaskProtection`
@@ -849,6 +867,9 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `ListDaemonDeployments`
- `ListDaemonTaskDefinitions`
- `ListDaemons`
- `ListGatewayClusterAssociations`
- `ListGatewayRoutes`
- `ListGateways`
- `ListServiceDeployments`
- `PutAccountSettingDefault`
- `PutAttributes`
@@ -861,6 +882,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `UpdateContainerAgent`
- `UpdateDaemon`
- `UpdateExpressGatewayService`
- `UpdateGateway`
- `UpdateServicePrimaryTaskSet`
- `UpdateTaskProtection`
- `UpdateTaskSet`
@@ -1085,7 +1107,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
## rds
`2014-10-31` — 27 of 164 modelled operations implemented (16.5%), 0 stubbed, 4 not applicable, 5 registered outside the pinned model.
`2014-10-31` — 27 of 168 modelled operations implemented (16.1%), 0 stubbed, 4 not applicable, 5 registered outside the pinned model.
### Not implemented
@@ -1095,6 +1117,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
- `ApplyPendingMaintenanceAction`
- `AuthorizeDBSecurityGroupIngress`
- `BacktrackDBCluster`
- `CancelDBCapacityReservation`
- `CancelExportTask`
- `CopyDBClusterParameterGroup`
- `CopyDBClusterSnapshot`
@@ -1103,6 +1126,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
- `CopyOptionGroup`
- `CreateBlueGreenDeployment`
- `CreateCustomDBEngineVersion`
- `CreateDBCapacityReservation`
- `CreateDBClusterEndpoint`
- `CreateDBClusterParameterGroup`
- `CreateDBClusterSnapshot`
@@ -1133,6 +1157,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
- `DescribeAccountAttributes`
- `DescribeBlueGreenDeployments`
- `DescribeCertificates`
- `DescribeDBCapacityReservations`
- `DescribeDBClusterAutomatedBackups`
- `DescribeDBClusterBacktracks`
- `DescribeDBClusterEndpoints`
@@ -1174,6 +1199,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
- `ModifyCertificates`
- `ModifyCurrentDBClusterCapacity`
- `ModifyCustomDBEngineVersion`
- `ModifyDBCapacityReservation`
- `ModifyDBClusterEndpoint`
- `ModifyDBClusterParameterGroup`
- `ModifyDBClusterSnapshotAttribute`
+1 -4
View File
@@ -99,9 +99,6 @@ behavior in other words.
| 4432 | Formation server | HTTPS | Cluster (bootstrap only) | Cluster join coordination; active only while a join token is valid. Binds the node's `--bind` (lan) address. See *Formation port lifecycle* below. | Short-lived bearer token + TLS¹ |
| 4222 | spinifex-nats (client) | NATS + TLS | Cluster | Internal service bus for EC2/EBS/VPC/S3 handlers | Token + mutual TLS (cluster CA) |
| 4248 | spinifex-nats (cluster) | NATS + TLS | Cluster | Inter-node NATS federation | Token + mutual TLS (cluster CA) |
| 4223 | spinifex-sqs-store (client) | NATS + TLS | Cluster | SQS message data, separate from the control-plane bus | Store-only token + TLS |
| 4249 | spinifex-sqs-store (cluster) | NATS + TLS | Cluster | Inter-node SQS store replication | Store-only token + mutual TLS (cluster CA) |
| 8223 | spinifex-sqs-store (monitor) | HTTP | Loopback | varz and jsz for the store | Loopback only |
| 5300 | northstar | DNS (UDP + TCP) | Cluster | Forward target for every node's per-instance DNS shim, dialled cross-node. Binds the wildcard by design. | None — restrict by firewall |
| 6660 | predastore (blob node) | QUIC / UDP | Cluster | Erasure-coded object shard transport between hosts. Multi-node clusters only — see *Predastore ports* below. | TLS 1.3, server certificate verified against the cluster CA |
| 7660 | predastore (meta node) | QUIC / UDP | Cluster | Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only. | TLS 1.3, server certificate verified against the cluster CA |
@@ -224,7 +221,7 @@ table inet spinifex_filter {
# Cluster, from peer nodes only. Replace with your nodes' lan-plane
# addresses — a CIDR does not generalise to nodes on different subnets.
ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4223, 4248, 4249, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } udp dport { 5300, 6660, 7660 } accept
# Encap, from peer chassis only. Replace with your nodes' vpc-plane
-521
View File
@@ -1,521 +0,0 @@
# Service interface manifest.
#
# Source of truth for "which services produce/consume which NATS subjects"
# and "which e2e suites cover which services". Consumed by
# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
# a PR diff.
#
# See docs/development/improvements/e2e-targeted-suite-selection.md.
#
# Editing rules
# -------------
# - subscribes / publishes: subjects this service receives / sends. Both
# lists use the literal subject string, or a "<prefix>.*" wildcard for
# pattern subscriptions (e.g. "ebs.snapshot.*").
# - depends_on: subjects this service requires from peers. Used by the
# selector to compute "if X changes, which suites might break".
# - Subjects must match what is actually referenced in source. The lint
# in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
# and fails if any subject is absent from this manifest.
version: 1
services:
awsgw:
path: spinifex/runtime/roles/awsgw
# TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
# No subscribes; only publishes (via NATSRequest).
subscribes: []
publishes:
- "ec2.*"
- "elbv2.*"
- "acm.*"
- "route53.*"
- "iam.*"
- "rds.*"
- "ecs.*"
- "ochre.vector.*"
depends_on:
- "ec2.*"
- "elbv2.*"
- "acm.*"
- "route53.*"
- "iam.*"
- "rds.*"
- "ecs.*"
- "ochre.vector.*"
spinifex_daemon:
path: spinifex/runtime/roles/spinifex
# Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
# Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
# forwards to vpcd / viperblockd / predastore.
#
# Source layout: spinifex/runtime/roles/spinifex (entry), spinifex/runtime/daemon
# (subscribe registration), spinifex/domains/<domain> and the legacy
# spinifex/handlers/{ecs,eks,elbv2,iam,rds} packages (per-subject handler
# bodies). Selector treats these paths as belonging to this service via
# the `additional_paths` field.
additional_paths:
- spinifex/runtime/daemon
- spinifex/domains/ec2/account
- spinifex/domains/ec2/eigw
- spinifex/domains/ec2/eip
- spinifex/domains/ec2/igw
- spinifex/domains/ec2/image
- spinifex/domains/ec2/instance
- spinifex/domains/ec2/key
- spinifex/domains/ec2/launchtemplate
- spinifex/domains/ec2/natgw
- spinifex/domains/ec2/placementgroup
- spinifex/domains/ec2/routetable
- spinifex/domains/ec2/snapshot
- spinifex/domains/ec2/spotinstance
- spinifex/domains/ec2/tags
- spinifex/domains/ec2/volume
- spinifex/domains/ec2/volumestate
- spinifex/domains/ec2/vpc
- spinifex/domains/elbv2
- spinifex/domains/acm
- spinifex/domains/route53
- spinifex/domains/iam
- spinifex/domains/ecs
- spinifex/domains/rds/instance
- spinifex/domains/rds/subnetgroup
- spinifex/domains/rds/parametergroup
- spinifex/domains/rds/engine
- spinifex/domains/ecs/taskdefinition
- spinifex/domains/eks/cluster
- spinifex/domains/eks/access
- spinifex/domains/eks/addon
# Guest add-on wire contract (subject, report, staged-manifest body).
- contracts/eks/v1
# ECS agent/scheduler Layer-2 bus wire contract (subjects, message types).
- contracts/ecs/v1
- spinifex/domains/ochre/vector
subscribes:
# EC2 instance lifecycle.
- "ec2.RunInstances"
- "ec2.DescribeInstances"
- "ec2.DescribeStoppedInstances"
- "ec2.DescribeTerminatedInstances"
- "ec2.DescribeInstanceAttribute"
- "ec2.DescribeInstanceStatus"
- "ec2.DescribeInstanceTypes"
- "ec2.ModifyInstanceAttribute"
- "ec2.ModifyVolume"
- "ec2.start"
- "ec2.terminate"
- "ec2.launch"
# contracts/ec2/v1 owns the command payload and this subject vocabulary.
# The wildcard is intentionally one final instance-ID token, not `>`.
- "ec2.cmd.*"
# EC2 VPC / network ops handled by daemon before delegation to vpcd.
- "ec2.CreateVpc"
- "ec2.DescribeVpcs"
- "ec2.CreateSubnet"
- "ec2.CreateInternetGateway"
- "ec2.CreateKeyPair"
- "ec2.CreateImage"
- "ec2.DescribePlacementGroups"
# Cluster coordination subjects.
- "ec2.ReserveClusterNode"
- "ec2.FinalizeClusterInstances"
# An EIP association reaches whichever node holds the instance, which is
# rarely the node that served the request (fan-out, no queue group).
- "ec2.eni.public-ip-changed"
# contracts/cluster/v1 owns the node status/capacity payload and subject.
- "spinifex.node.status"
# contracts/cluster/v1 owns the VM inventory payload and subject.
- "spinifex.node.vms"
- "spinifex.nodes.discover"
- "spinifex.cluster.shutdown.progress"
# Coordinated cluster-shutdown phases (fan-out, no queue group).
- "spinifex.cluster.shutdown.gate"
- "spinifex.cluster.shutdown.drain"
- "spinifex.cluster.shutdown.storage"
- "spinifex.cluster.shutdown.persist"
- "spinifex.cluster.shutdown.infra"
# Operator-plane Predastore topology contract (contracts/operator/v1).
- "spinifex.storage.config"
# ELBv2 control plane (domains/elbv2/service_nats.go topic map).
- "elbv2.CreateLoadBalancer"
- "elbv2.DeleteLoadBalancer"
- "elbv2.DescribeLoadBalancers"
- "elbv2.CreateTargetGroup"
- "elbv2.DeleteTargetGroup"
- "elbv2.DescribeTargetGroups"
- "elbv2.RegisterTargets"
- "elbv2.DeregisterTargets"
- "elbv2.DescribeTargetHealth"
- "elbv2.CreateListener"
- "elbv2.DeleteListener"
- "elbv2.DescribeListeners"
# ACM-alike certificate store (domains/acm; subscribed in daemon.go).
- "acm.ImportCertificate"
- "acm.DescribeCertificate"
- "acm.ListCertificates"
- "acm.DeleteCertificate"
# Route 53 hosted zones (domains/route53; subscribed in daemon.go).
- "route53.*"
# Operator-only zone activation and delegation, sent by spx admin dns.
- "route53.admin.*"
- "elbv2.AddListenerCertificates"
- "elbv2.RemoveListenerCertificates"
- "elbv2.DescribeListenerCertificates"
- "elbv2.DescribeSSLPolicies"
# RDS control plane (domains/rds/instance/subjects.go). One wildcard rather than
# 30 literals: every action subject is registered from a constant, so a
# per-subject list would be a hand-mirror of that file with nothing
# checking it.
- "rds.*"
# ECS control plane (domains/ecs/service_nats.go, registered in
# daemon.go). One wildcard, as for RDS: every action subject is a literal
# in those two files, so a per-subject list would only mirror them.
- "ecs.*"
# Layer-2 agent bus: register and health are wildcard subscriptions over
# {account}.{db-instance}, so the subject is built at runtime and only the
# prefix is declarable.
- "rds.bus.*"
# ECS Layer-2 scheduler/agent bus (contracts/ecs/v1): account, cluster and
# instance/task IDs are runtime-built, so only the prefix is declarable.
# Only the leader holds these subscriptions, so there is no queue group.
- "ecs.bus.*"
# Ochre vector store tenant surface (domains/ochre/vector/api.go),
# gated behind config.OchreVectorConfig.Enabled (default off).
- "ochre.vector.createIndex"
- "ochre.vector.deleteIndex"
- "ochre.vector.listIndexes"
- "ochre.vector.ingest"
- "ochre.vector.describeJob"
- "ochre.vector.query"
- "ochre.vector.listJobs"
- "ochre.vector.stopJob"
# Fence route owned by contracts/viperblockd/legacy/v1: viperblockd
# announces a torn-down export so this node stops the guest that was
# using it. "ebs.fenced" is the single-node form; a named node is
# addressed as "ebs.<node>.fenced".
- "ebs.fenced"
publishes:
# → vpcd
- "vpc.create"
- "vpc.delete"
- "vpc.create-subnet"
- "vpc.delete-subnet"
- "vpc.create-sg"
- "vpc.delete-sg"
- "vpc.update-sg"
- "vpc.create-port"
- "vpc.delete-port"
- "vpc.update-port-sgs"
# EC2-to-network projection owned by contracts/ec2/v1. The network
# subscriber realizes it in OVN without reading EC2 private state.
- "vpc.igw-attach"
- "vpc.igw-detach"
- "vpc.add-nat-gateway"
- "vpc.delete-nat-gateway"
- "vpc.add-nat"
- "vpc.delete-nat"
- "vpc.dhcp.acquire"
- "vpc.dhcp.release"
# Per-subnet route-table projections owned by domains/ec2/routetable:
# IGW default-route reachability and the egress gate/ungate decision it
# implies, published fire-and-forget on every route-table mutation.
- "vpc.add-igw-route"
- "vpc.delete-igw-route"
- "vpc.gate-subnet-egress"
- "vpc.ungate-subnet-egress"
# Egress-only reroute removal for a single system instance
# (domains/eks/cluster). Fire-and-forget. vpc.add-system-egress has no
# current producer.
- "vpc.delete-system-egress"
# → viperblockd
# The migrated legacy ebs.config and ebs.sync payloads and routes are
# owned by contracts/viperblockd/legacy/v1. They are transitional and
# distinct from the generic providers/ebs protocol; other legacy ebs.*
# routes remain to be migrated separately.
- "ebs.mount"
- "ebs.unmount"
- "ebs.delete"
- "ebs.sync"
- "ebs.snapshot.*"
- "ebs.config"
- "ebs.config.*"
# Abandon route owned by contracts/viperblockd/legacy/v1: the daemon
# asks viperblockd to tear an export down without sealing. "ebs.abandon"
# is the single-node form; a named node addresses "ebs.<node>.abandon".
- "ebs.abandon"
# → predastore (S3 / AMI / keypair store)
- "s3.*"
# → every daemon, so the one holding the instance reports the new address
- "ec2.eni.public-ip-changed"
# IAM lifecycle event
- "iam.account.created"
# → the in-guest rds-agent: set-password and the like, published on the
# per-instance command subject and answered on its reply subject.
- "rds.bus.*"
# ECS service-reconcile self-wake (contracts/ecs/v1): the worker that
# recorded a task-state change is rarely the leader, so this fans the
# wake out and the leader's own "ecs.bus.*" subscription picks it up.
- "ecs.bus.*"
depends_on:
- "vpc.*"
- "ebs.*"
- "s3.*"
vpcd:
path: spinifex/runtime/roles/vpcd
# OVN-backed VPC / subnet / SG / NAT / IGW data plane.
subscribes:
- "vpc.create"
- "vpc.delete"
- "vpc.create-subnet"
- "vpc.delete-subnet"
- "vpc.create-sg"
- "vpc.delete-sg"
- "vpc.update-sg"
- "vpc.create-port"
- "vpc.delete-port"
- "vpc.update-port-sgs"
- "vpc.igw-attach"
- "vpc.igw-detach"
- "vpc.add-nat-gateway"
- "vpc.delete-nat-gateway"
- "vpc.add-nat"
- "vpc.delete-nat"
- "vpc.dhcp.acquire"
- "vpc.dhcp.release"
- "vpc.add-igw-route"
- "vpc.delete-igw-route"
- "vpc.gate-subnet-egress"
- "vpc.ungate-subnet-egress"
- "vpc.add-system-egress"
- "vpc.delete-system-egress"
publishes:
- "vpc.port-status"
- "vpc.dhcp.lease-changed"
- "vpc.dhcp.owner-check"
depends_on: []
viperblockd:
path: spinifex/runtime/roles/viperblockd
# EBS data plane. Backed by the viperblock storage engine in the
# mulga/viperblock submodule. Submodule pointer bump → selector forces
# "run all" (contract changes can be invisible at the spinifex source
# level).
# The migrated legacy ebs.config and ebs.sync routes are versioned by
# contracts/viperblockd/legacy/v1, not by providers/ebs. Other legacy
# ebs.* routes remain to be migrated separately.
# The dirty-marker JetStream KV bucket VIPERBLOCK_VOLUME_DIRTY is owned
# by the same contract; not a NATS subject, so not listed below.
#
# Source layout: spinifex/runtime/roles/viperblockd (role: process lifecycle,
# NATS subscription wiring) and spinifex/providers/ebs/viperblock (the
# Viperblock EBS provider adapter the role wires up). Selector treats the
# adapter path as belonging to this service via `additional_paths`.
additional_paths:
- spinifex/providers/ebs/viperblock
subscribes:
- "ebs.mount"
- "ebs.unmount"
- "ebs.delete"
- "ebs.sync"
- "ebs.snapshot.*"
- "ebs.config"
- "ebs.config.*"
# Abandon route (contracts/viperblockd/legacy/v1): request/reply from
# the daemon, answered by the Viperblock adapter on the named node.
- "ebs.abandon"
publishes:
- "ebs.mount.response"
# Fence route (contracts/viperblockd/legacy/v1): announced after the
# Viperblock adapter tears an export down because its lease moved.
- "ebs.fenced"
- "ebs.unmount.response"
depends_on: []
predastore:
path: spinifex/runtime/roles/predastore
# S3 / AMI / keypair storage. External submodule under mulga/predastore;
# this dir contains only the spinifex-side adapter. Submodule pointer
# bump → selector forces "run all".
subscribes:
- "s3.*"
publishes: []
depends_on: []
nats:
path: spinifex/runtime/roles/nats
# NATS broker bootstrap / cluster config. Not a NATS pub/sub
# participant. Any change here is infra-path; selector forces
# "run all".
subscribes: []
publishes: []
depends_on: []
spinifexui:
path: spinifex/runtime/roles/spinifexui
# Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
# dir do not need any e2e suite to run.
subscribes: []
publishes: []
depends_on: []
covers_no_suite: true
# Fixture → service mapping. Each Ensure* helper in
# tests/e2e/harness/fixtures.go exercises this set of services; the
# Bead 4 selector AST-scans test files for harness.Ensure* call sites
# and resolves coverage via this block.
fixtures:
EnsureKeyPair:
services: [spinifex_daemon, predastore]
EnsureAMI:
services: [spinifex_daemon, predastore, viperblockd]
EnsureDefaultVPC:
services: [vpcd, spinifex_daemon]
EnsureSubnet:
services: [vpcd, spinifex_daemon]
EnsureSG:
services: [vpcd, spinifex_daemon]
EnsureInstance:
services: [spinifex_daemon, viperblockd, vpcd]
EnsureVolume:
services: [spinifex_daemon, viperblockd]
EnsureSnapshot:
services: [spinifex_daemon, viperblockd, predastore]
EnsureNATGateway:
services: [vpcd, spinifex_daemon]
EnsureLoadBalancer:
services: [spinifex_daemon, awsgw]
suites:
e2e-cert:
path: tests/e2e/cert
covers: [awsgw, spinifex_daemon, vpcd, predastore]
e2e-lb:
path: tests/e2e/lb
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# Exercises full ELBv2 control plane (domains/elbv2) + data plane
# (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
# this manifest; revisit if it splits into a standalone service.
e2e-single:
path: tests/e2e/single
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
# snapshot, IAM. Currently runs all phases sequentially under one
# `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
# into per-test fixture-driven coverage so the selector can pick
# subtests, not just whole suites.
e2e-iam:
path: tests/e2e/iam
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# "Everything IAM": the IAM/STS control plane plus the instance-identity
# tests (instance-profile association + IMDS), which boot guest VMs — OVN
# via vpcd, AMI via predastore, root volume via viperblockd, lifecycle via
# spinifex_daemon. Single-mode, VM-booting leg.
e2e-ecr:
path: tests/e2e/ecr
covers: [awsgw, spinifex_daemon, predastore]
# ECR control plane (domains/ecr, gateway/ecr*) + OCI data plane: blobs and
# manifests in predastore, control + auth bridge in awsgw, KV via
# spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
# client binary or registry-host resolution is absent.
e2e-bedrock:
path: tests/e2e/bedrock
covers: [awsgw]
# Ochre (AWS-Bedrock) control plane: static catalog list/get served
# straight from awsgw (gateway/bedrock.go, domains/ochre/) — no NATS
# hop to spinifex_daemon. Converse (Anthropic-direct / self-host vLLM)
# subtests are gated on backend availability and skip by default.
e2e-ecs:
path: tests/e2e/ecs
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# ECS data plane end-to-end: cluster + a real container instance launched
# from the spinifex-ecs-node AMI (boots, registers over the gateway, runs
# tasks through containerd), standalone tasks in awsvpc/bridge/host modes,
# and an awsvpc service fronted by an ELBv2 target group. Boots a guest VM,
# so vpcd (OVN + awsvpc ENI), viperblockd (root volume), predastore (AMI)
# are all in the path.
e2e-rds:
path: tests/e2e/rds
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# RDS create/describe/connect: CreateDBInstance boots a dual-NIC DB VM from
# the engine AMI (predastore) with a Viperblock data volume (viperblockd) and
# a customer ENI (vpcd), and the reconciler flips it to available on the
# in-guest agent's first healthy heartbeat over awsgw. Every test deletes the
# instances it creates, so the suite needs no gate beyond SPINIFEX_E2E.
e2e-ddil:
path: tests/e2e/ddil
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# DDIL = degraded / disconnected / intermittent / low-bandwidth.
# Exercises the full daemon stack under network failure injection;
# any service touching the wire path is in scope.
e2e-route53:
path: tests/e2e/route53
covers: [awsgw, spinifex_daemon, predastore]
# Hosted zones end to end: delegation check, publish to the northstar zone
# store in predastore, and every node answering before a change is INSYNC.
# Suites below are bash-only today. Entries activate as the Go ports
# land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
#
# e2e-multinode:
# path: spinifex/tests/e2e/multinode
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# e2e-reboot:
# path: spinifex/tests/e2e/reboot
# covers: [spinifex_daemon, viperblockd]
# e2e-tofu:
# path: spinifex/tests/e2e/tofu
# covers: [awsgw, spinifex_daemon, vpcd]
# e2e-bm:
# path: spinifex/tests/e2e/baremetal
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
#
# e2e-natuplink activates once CI can provision an external_mode=nat
# node (setup-ovn.sh --nat-uplink + spx admin init --external-mode=nat).
# e2e-natuplink:
# path: spinifex/tests/e2e/natuplink
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# Special-case rules consumed by the selector (Bead 4). Hard-coded in
# the tool, listed here for visibility.
#
# infra_paths_force_all:
# - spinifex/runtime/roles/nats/**
# - spinifex/tests/e2e/harness/**
# - spinifex/tests/e2e/lib/**
# - spinifex/.github/workflows/e2e-go.yml
# - spinifex/.github/scripts/**
# - spinifex/Makefile
# - mulga/scripts/tofu-cluster/**
#
# submodule_force_all:
# - mulga/predastore (submodule pointer)
# - mulga/viperblock (submodule pointer)
#
# many_services_force_all_threshold: 3 # ≥ N services touched → run all
+1 -2
View File
@@ -120,7 +120,7 @@ An ALB whose targets all fail their health check keeps serving from them instead
### `eks`, `ecs`, `ecr` — container services
`eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.
`eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, an IRSA web-identity exchange under the stock AWS trust policy (`StringEquals` on the issuer host-path `:sub`/`:aud`, a mismatched `:sub` denied with `AccessDenied`, a `StringLike` `:sub` granted), and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.
### `rds` — managed databases
@@ -143,7 +143,6 @@ The create/describe/connect path against a live cluster: a DB VM that actually b
- **`ddil`** — Denied, Disrupted, Intermittent, Limited network scenarios.
- **`teardown`** — sweeps leftovers.
- **`harness`** — shared primitives, not a suite. `harness/vpc_egress.go` carries the VPC/subnet/IGW/NAT-gateway fixtures the egress scenarios build on.
- **`manifest-check`, `manifest-lint`** — manifest validation, not cluster behaviour.
## Keeping this honest
+1 -1
View File
@@ -33,6 +33,7 @@ require (
github.com/mulgadc/viperblock v1.23.0
github.com/nats-io/nats-server/v2 v2.15.1
github.com/nats-io/nats.go v1.54.0
github.com/nats-io/nkeys v0.4.16
github.com/opencontainers/runtime-spec v1.3.0
github.com/oracle/oci-go-sdk/v65 v65.128.0
github.com/ovn-kubernetes/libovsdb v0.9.0
@@ -161,7 +162,6 @@ require (
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
+2 -2
View File
@@ -77,8 +77,8 @@ func TestEC2ErrorCatalogMetadataAndClasses(t *testing.T) {
if catalog.metadata.VerifiedOn == "" {
t.Fatal("verifiedOn is empty")
}
if len(catalog.codes) != 114 {
t.Fatalf("catalog code count = %d, want 114", len(catalog.codes))
if len(catalog.codes) != 115 {
t.Fatalf("catalog code count = %d, want 115", len(catalog.codes))
}
if catalog.codes["InvalidParameterValue"] != ec2ErrorClassClient {
t.Fatal("InvalidParameterValue is not classified as client")
+2 -2
View File
@@ -5,6 +5,6 @@ package awsmodel
// ModelCommit is the github.com/aws/api-models-aws commit the embedded
// models were taken from, and ModelCommitDate the day it was committed.
const (
ModelCommit = "9b1a4929274443d8671ea8acfda7afbe856e29e9"
ModelCommitDate = "2026-09-25"
ModelCommit = "e10d6a1c5ac36af7c0c61d1f67b3c3ebbe41de49"
ModelCommitDate = "2026-10-28"
)
+5 -5
View File
@@ -21,13 +21,13 @@ func TestLoadAllModels(t *testing.T) {
shapes int
}{
ACM: {"2015-12-08", 40, 239},
EC2: {"2016-11-15", 804, 4245},
ECR: {"2015-09-21", 58, 404},
ECS: {"2014-11-13", 77, 643},
EKS: {"2017-11-01", 70, 450},
EC2: {"2016-11-15", 811, 4297},
ECR: {"2015-09-21", 62, 426},
ECS: {"2014-11-13", 88, 734},
EKS: {"2017-11-01", 70, 458},
ElasticLoadBalancingV2: {"2015-12-01", 51, 418},
IAM: {"2010-05-08", 180, 624},
RDS: {"2014-10-31", 164, 791},
RDS: {"2014-10-31", 168, 816},
Route53: {"2013-04-01", 71, 409},
S3: {"2006-03-01", 112, 710},
STS: {"2011-06-15", 11, 101},
Binary file not shown.
Binary file not shown.
@@ -50,6 +50,7 @@
"InvalidCapacityReservationId.Malformed",
"InvalidCapacityReservationId.NotFound",
"InvalidEgressOnlyInternetGatewayId.NotFound",
"InvalidGatewayID.NotFound",
"InvalidGroup.Duplicate",
"InvalidGroup.NotFound",
"InvalidGroup.Reserved",
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+3 -2
View File
@@ -111,7 +111,7 @@ func GenerateRequests(service Service, operationName string, options RequestOpti
}
for _, member := range append([]string{""}, optional...) {
if unguessable[member] {
if unguessable[member] && options.Fixtures[operationName+"."+member] == "" && options.Fixtures[member] == "" {
continue
}
value, _, err := generate()
@@ -157,7 +157,8 @@ func GenerateRequests(service Service, operationName string, options RequestOpti
// unguessable members take values the model cannot describe: a pagination
// token the service issued, or a filter name only its documentation lists.
// They get rejection requests but no acceptance request.
// They get rejection requests, and an acceptance request only when a fixture
// supplies a value.
var unguessable = map[string]bool{
"Filter": true,
"Filters": true,
+14
View File
@@ -337,6 +337,20 @@ func TestGenerateRequestsSkipsAcceptanceForUnguessableMembers(t *testing.T) {
require.NotEmpty(t, markerRejections, "a pagination token still gets rejection requests")
}
func TestGenerateRequestsAcceptsAFixtureForAnUnguessableMember(t *testing.T) {
options := testRequestOptions
options.Fixtures = map[string]string{"ListRoles.Marker": "fixture-marker"}
plan, err := GenerateRequests(IAM, "ListRoles", options)
require.NoError(t, err)
var markers []any
for _, request := range plan.Cases {
if request.Acceptance() && request.Member == "Marker" {
markers = append(markers, request.Input["Marker"])
}
}
require.Equal(t, []any{"fixture-marker"}, markers)
}
func TestGenerateRequestsUsesResourceHints(t *testing.T) {
tests := []struct {
service Service
+12
View File
@@ -15,6 +15,8 @@ import (
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds"
"github.com/aws/aws-sdk-go-v2/feature/ec2/imds"
)
// Signer holds a credential provider and signs requests in place.
@@ -42,6 +44,16 @@ func NewIMDS(ctx context.Context, region string) (*Signer, error) {
return &Signer{provider: cfg.Credentials}, nil
}
// NewInstanceRole returns a Signer whose credentials come only from the
// instance metadata service: no environment, profile or static-key fallback.
func NewInstanceRole() *Signer {
return newInstanceRole(imds.New(imds.Options{}))
}
func newInstanceRole(client *imds.Client) *Signer {
return &Signer{provider: aws.NewCredentialsCache(ec2rolecreds.New(func(o *ec2rolecreds.Options) { o.Client = client }))}
}
// EnsureCredentials retrieves once from the provider, surfacing the error so a
// caller can warm the IMDS instance-role datapath before the first Sign. The
// SDK's IMDS client fast-fails each probe (250ms dial), so a cold per-tap
+39
View File
@@ -12,6 +12,7 @@ import (
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/feature/ec2/imds"
"github.com/mulgadc/bluebottle/pkg/sigv4"
)
@@ -123,3 +124,41 @@ func TestNewIMDS_BuildsProvider(t *testing.T) {
t.Fatal("expected non-nil credentials provider")
}
}
// The instance-role signer ignores environment keys and signs only with the
// credentials the metadata service issues.
func TestNewInstanceRole_UsesOnlyIMDS(t *testing.T) {
t.Setenv("AWS_ACCESS_KEY_ID", testAccessKey)
t.Setenv("AWS_SECRET_ACCESS_KEY", testSecretKey)
const imdsKey = "ASIAIMDSONLYEXAMPLE1"
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/latest/api/token":
_, _ = w.Write([]byte("token"))
case "/latest/meta-data/iam/security-credentials/":
_, _ = w.Write([]byte("sqsWorkerRole"))
case "/latest/meta-data/iam/security-credentials/sqsWorkerRole":
_, _ = w.Write([]byte(`{"Code":"Success","AccessKeyId":"` + imdsKey + `","SecretAccessKey":"s","Token":"t","Expiration":"2099-01-01T00:00:00Z"}`))
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
s := newInstanceRole(imds.New(imds.Options{Endpoint: srv.URL}))
req := httptest.NewRequest(http.MethodPost, "https://gw.example/", nil)
if err := s.Sign(req, emptySHA, testService, testRegion); err != nil {
t.Fatal(err)
}
if auth := req.Header.Get("Authorization"); !strings.Contains(auth, "Credential="+imdsKey+"/") {
t.Fatalf("Authorization = %q, want the IMDS credential", auth)
}
srv.Close()
cold := newInstanceRole(imds.New(imds.Options{Endpoint: srv.URL}))
if err := cold.EnsureCredentials(context.Background()); err == nil {
t.Fatal("environment keys were used when IMDS was unreachable")
}
}
const emptySHA = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+242
View File
@@ -0,0 +1,242 @@
// Package sqsworkergw is the shared SQS worker guest's SigV4 client for the
// gateway's internal spinifex-sqs service. It sends only public values: the
// worker's identity is the signing session, and the store seed never leaves
// the guest.
package sqsworkergw
import (
"bytes"
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/xml"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/mulgadc/bluebottle/pkg/tlsconfig"
"github.com/nats-io/nkeys"
)
const (
// The SigV4 credential scope the gateway routes the internal surface on.
signingService = "spinifex-sqs"
apiVersion = "2026-10-30"
callTimeout = 30 * time.Second
maxResponse = 64 << 10
actionClaim = "ClaimFirstStore"
actionDescribe = "DescribeStore"
actionRecord = "RecordWorkerPublicKey"
)
// Signer signs a request in place for service and region.
type Signer interface {
Sign(r *http.Request, payloadHash, service, region string) error
}
// Client calls the internal first-store actions for this worker.
type Client struct {
baseURL string
region string
signer Signer
httpClient *http.Client
}
// New builds a TLS 1.3 client that trusts only caPEM, the cluster CA. An
// https URL, a Region and a CA are all required: there is no fallback.
func New(baseURL, region string, caPEM []byte, signer Signer) (*Client, error) {
u, err := url.Parse(baseURL)
if err != nil || u.Scheme != "https" || u.Host == "" {
return nil, errors.New("sqsworkergw: gateway URL must be an https URL")
}
if region == "" {
return nil, errors.New("sqsworkergw: Region is required")
}
if signer == nil {
return nil, errors.New("sqsworkergw: signer is required")
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(caPEM) {
return nil, errors.New("sqsworkergw: gateway CA has no usable certificates")
}
tlsCfg := &tls.Config{MinVersion: tls.VersionTLS13, CurvePreferences: tlsconfig.Curves, RootCAs: pool}
return &Client{
baseURL: strings.TrimRight(baseURL, "/"),
region: region,
signer: signer,
httpClient: &http.Client{
Timeout: callTimeout,
Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 1, IdleConnTimeout: 30 * time.Second},
// A redirect would resend a signed request somewhere else.
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
},
}, nil
}
// APIError is a gateway refusal; Code is the AWS error code.
type APIError struct {
Action string
StatusCode int
Code string
}
func (e *APIError) Error() string {
return fmt.Sprintf("spinifex-sqs %s: %s (HTTP %d)", e.Action, e.Code, e.StatusCode)
}
// Retryable reports a refusal that a later identical call may pass:
// contention or a store that is not yet available. Every other refusal,
// including AccessDenied, is final for this boot.
func (e *APIError) Retryable() bool {
return e.Code == "ServiceUnavailable" || e.Code == "Throttling" || e.StatusCode == http.StatusServiceUnavailable
}
// Pending reports an AccessDenied, which a worker that has started but is
// not yet bound to its identity legitimately receives. It authorizes
// nothing; the caller may ask again within its own bound.
func (e *APIError) Pending() bool { return e.Code == "AccessDenied" }
// TransientError is a call that may not have reached the gateway, or whose
// reply was lost: signing, sending or reading failed. Both actions are safe
// to repeat, so it is retryable.
type TransientError struct{ Err error }
func (e *TransientError) Error() string { return e.Err.Error() }
func (e *TransientError) Unwrap() error { return e.Err }
func (e *TransientError) Retryable() bool { return true }
type claimResult struct {
CreateKey bool `xml:"CreateKey"`
PublicKey string `xml:"PublicKey"`
}
// Claim asks for this worker's current first-store decision. A decision
// that is neither "create" nor one valid public key is refused here.
func (c *Client) Claim(ctx context.Context) (bool, string, error) {
var res claimResult
if err := c.call(ctx, actionClaim, nil, &res); err != nil {
return false, "", err
}
switch {
case res.CreateKey && res.PublicKey == "":
return true, "", nil
case !res.CreateKey && nkeys.IsValidPublicUserKey(res.PublicKey):
return false, res.PublicKey, nil
}
return false, "", errors.New("spinifex-sqs ClaimFirstStore: response is not a valid decision")
}
// Store is the Region's store launch as the gateway reports it. Bound means
// a VM was launched for it, not that it is ready.
type Store struct {
Bound bool
Address string
PublicKey string
}
type describeResult struct {
State string `xml:"State"`
Address string `xml:"Address"`
PublicKey string `xml:"PublicKey"`
}
// DescribeStore reports this worker's store. A bound reply must carry an
// IPv4 address and a valid public key; a pending one must carry neither.
func (c *Client) DescribeStore(ctx context.Context) (Store, error) {
var res describeResult
if err := c.call(ctx, actionDescribe, nil, &res); err != nil {
return Store{}, err
}
switch {
case res.State == "pending" && res.Address == "" && res.PublicKey == "":
return Store{}, nil
case res.State == "bound" && isIPv4(res.Address) && nkeys.IsValidPublicUserKey(res.PublicKey):
return Store{Bound: true, Address: res.Address, PublicKey: res.PublicKey}, nil
}
return Store{}, errors.New("spinifex-sqs DescribeStore: response is not a valid store state")
}
func isIPv4(s string) bool {
ip := net.ParseIP(s)
return ip != nil && ip.To4() != nil && !strings.Contains(s, ":")
}
// RecordPublicKey records the worker's public key. Anything other than a
// user public key is refused before it is sent.
func (c *Client) RecordPublicKey(ctx context.Context, publicKey string) error {
if !nkeys.IsValidPublicUserKey(publicKey) {
return errors.New("spinifex-sqs RecordWorkerPublicKey: refusing to send a value that is not a user public key")
}
return c.call(ctx, actionRecord, url.Values{"PublicKey": {publicKey}}, nil)
}
func (c *Client) call(ctx context.Context, action string, params url.Values, out any) error {
form := url.Values{"Action": {action}, "Version": {apiVersion}}
maps.Copy(form, params)
body := []byte(form.Encode())
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/", bytes.NewReader(body))
if err != nil {
return fmt.Errorf("spinifex-sqs %s: build request: %w", action, err)
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded; charset=utf-8")
sum := sha256.Sum256(body)
if err := c.signer.Sign(req, hex.EncodeToString(sum[:]), signingService, c.region); err != nil {
return &TransientError{Err: fmt.Errorf("spinifex-sqs %s: sign: %w", action, err)}
}
resp, err := c.httpClient.Do(req)
if err != nil {
return &TransientError{Err: fmt.Errorf("spinifex-sqs %s: send: %w", action, err)}
}
defer resp.Body.Close()
respBody, err := io.ReadAll(io.LimitReader(resp.Body, maxResponse))
if err != nil {
return &TransientError{Err: fmt.Errorf("spinifex-sqs %s: read response: %w", action, err)}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return parseError(action, resp.StatusCode, respBody)
}
if out == nil {
return nil
}
return decodeResult(action, respBody, out)
}
func decodeResult(action string, body []byte, out any) error {
want := action + "Result"
dec := xml.NewDecoder(bytes.NewReader(body))
for {
tok, err := dec.Token()
if err != nil {
return fmt.Errorf("spinifex-sqs %s: response carries no <%s>", action, want)
}
if start, ok := tok.(xml.StartElement); ok && start.Name.Local == want {
if err := dec.DecodeElement(out, &start); err != nil {
return fmt.Errorf("spinifex-sqs %s: decode response: %w", action, err)
}
return nil
}
}
}
// The error body is not echoed: only the parsed code is kept.
func parseError(action string, status int, body []byte) error {
var env struct {
Error struct {
Code string `xml:"Code"`
} `xml:"Error"`
}
code := fmt.Sprintf("HTTP%d", status)
if xml.Unmarshal(body, &env) == nil && env.Error.Code != "" {
code = env.Error.Code
}
return &APIError{Action: action, StatusCode: status, Code: code}
}
+235
View File
@@ -0,0 +1,235 @@
package sqsworkergw
import (
"context"
"encoding/pem"
"errors"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"github.com/nats-io/nkeys"
)
type nopSigner struct{ err error }
func (s nopSigner) Sign(r *http.Request, _, service, region string) error {
if s.err != nil {
return s.err
}
r.Header.Set("Authorization", "test "+service+"/"+region)
return nil
}
func userKey(t *testing.T) (pub, seed string) {
t.Helper()
kp, err := nkeys.CreateUser()
if err != nil {
t.Fatal(err)
}
p, _ := kp.PublicKey()
s, _ := kp.Seed()
return p, string(s)
}
// server answers with body and status, counting and keeping requests.
func server(t *testing.T, status int, body string) (*Client, *atomic.Int32, *[]string) {
t.Helper()
var n atomic.Int32
var seen []string
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
n.Add(1)
b, _ := io.ReadAll(r.Body)
seen = append(seen, r.Header.Get("Authorization")+"|"+string(b))
w.WriteHeader(status)
_, _ = io.WriteString(w, body)
}))
t.Cleanup(srv.Close)
ca := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw})
c, err := New(srv.URL, "ap-southeast-2", ca, nopSigner{})
if err != nil {
t.Fatal(err)
}
return c, &n, &seen
}
func TestNew_RefusesWithoutHTTPSRegionCAOrSigner(t *testing.T) {
ca := []byte("not a certificate")
cases := map[string]func() error{
"http": func() error { _, err := New("http://10.0.0.1:9999", "ap-southeast-2", ca, nopSigner{}); return err },
"no host": func() error { _, err := New("https://", "ap-southeast-2", ca, nopSigner{}); return err },
"no region": func() error { _, err := New("https://10.0.0.1:9999", "", ca, nopSigner{}); return err },
"no signer": func() error { _, err := New("https://10.0.0.1:9999", "ap-southeast-2", ca, nil); return err },
"no CA": func() error { _, err := New("https://10.0.0.1:9999", "ap-southeast-2", ca, nopSigner{}); return err },
}
for name, fn := range cases {
if fn() == nil {
t.Errorf("%s: client built", name)
}
}
}
func TestClaim_SignsForTheInternalServiceAndRegion(t *testing.T) {
c, _, seen := server(t, 200, `<ClaimFirstStoreResponse><ClaimFirstStoreResult><CreateKey>true</CreateKey></ClaimFirstStoreResult></ClaimFirstStoreResponse>`)
create, pub, err := c.Claim(context.Background())
if err != nil || !create || pub != "" {
t.Fatalf("claim = %v %q %v", create, pub, err)
}
if got := (*seen)[0]; !strings.HasPrefix(got, "test spinifex-sqs/ap-southeast-2|") || !strings.Contains(got, "Action=ClaimFirstStore") {
t.Fatalf("request = %q", got)
}
}
// Only "create" or exactly one valid public key is a decision.
func TestClaim_RefusesAnythingButOneDecision(t *testing.T) {
pub, seed := userKey(t)
cases := map[string]string{
"create with a key": `<CreateKey>true</CreateKey><PublicKey>` + pub + `</PublicKey>`,
"neither": `<CreateKey>false</CreateKey>`,
"seed as key": `<CreateKey>false</CreateKey><PublicKey>` + seed + `</PublicKey>`,
"no result": ``,
}
for name, inner := range cases {
t.Run(name, func(t *testing.T) {
body := `<ClaimFirstStoreResponse><ClaimFirstStoreResult>` + inner + `</ClaimFirstStoreResult></ClaimFirstStoreResponse>`
if inner == "" {
body = `<Other/>`
}
c, _, _ := server(t, 200, body)
_, _, err := c.Claim(context.Background())
if err == nil {
t.Fatal("accepted a non-decision")
}
if strings.Contains(err.Error(), seed) {
t.Fatal("the error echoes a seed")
}
})
}
t.Run("existing", func(t *testing.T) {
c, _, _ := server(t, 200, `<ClaimFirstStoreResponse><ClaimFirstStoreResult><CreateKey>false</CreateKey><PublicKey>`+pub+`</PublicKey></ClaimFirstStoreResult></ClaimFirstStoreResponse>`)
create, got, err := c.Claim(context.Background())
if err != nil || create || got != pub {
t.Fatalf("claim = %v %q %v", create, got, err)
}
})
}
func TestRecordPublicKey_SendsOnlyAPublicKey(t *testing.T) {
pub, seed := userKey(t)
c, n, seen := server(t, 200, `<RecordWorkerPublicKeyResponse/>`)
if err := c.RecordPublicKey(context.Background(), seed); err == nil || strings.Contains(err.Error(), seed) {
t.Fatalf("seed err = %v", err)
}
if n.Load() != 0 {
t.Fatal("a seed was sent")
}
if err := c.RecordPublicKey(context.Background(), pub); err != nil {
t.Fatal(err)
}
if !strings.Contains((*seen)[0], "PublicKey="+pub) {
t.Fatalf("request = %q", (*seen)[0])
}
}
// Refusals carry only the parsed code; ServiceUnavailable is the one
// retryable refusal, and transport failures are retryable too.
func TestErrors_ClassifiedWithoutEchoingTheBody(t *testing.T) {
cases := []struct {
status int
code string
retryable bool
}{
{403, "AccessDenied", false},
{400, "InvalidParameterValue", false},
{400, "InvalidState", false},
{403, "SignatureDoesNotMatch", false},
{503, "ServiceUnavailable", true},
{500, "InternalFailure", false},
}
for _, tc := range cases {
c, _, _ := server(t, tc.status, `<ErrorResponse><Error><Code>`+tc.code+`</Code><Message>secret-ish detail</Message></Error></ErrorResponse>`)
_, _, err := c.Claim(context.Background())
api, ok := errors.AsType[*APIError](err)
if !ok || api.Code != tc.code || api.Retryable() != tc.retryable || api.Pending() != (tc.code == "AccessDenied") {
t.Fatalf("%s: err = %v", tc.code, err)
}
if strings.Contains(err.Error(), "secret-ish") {
t.Fatal("the error echoes the response body")
}
}
c, _, _ := server(t, 200, "")
c.baseURL = "https://127.0.0.1:1"
_, _, err := c.Claim(context.Background())
tr, ok := errors.AsType[*TransientError](err)
if !ok || !tr.Retryable() {
t.Fatalf("unreachable err = %v, want TransientError", err)
}
}
// A redirect would resend a signed request elsewhere; it is not followed.
func TestCall_DoesNotFollowRedirects(t *testing.T) {
var followed atomic.Bool
target := httptest.NewTLSServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { followed.Store(true) }))
t.Cleanup(target.Close)
c, _, _ := server(t, 0, "")
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, target.URL, http.StatusTemporaryRedirect)
}))
t.Cleanup(srv.Close)
c.baseURL = srv.URL
if _, _, err := c.Claim(context.Background()); err == nil {
t.Fatal("a redirect was accepted as a decision")
}
if followed.Load() {
t.Fatal("the signed request followed a redirect")
}
}
// Only "pending" with nothing else, or "bound" with an IPv4 address and one
// valid public key, is a store state.
func TestDescribeStore_AcceptsOnlyAValidState(t *testing.T) {
pub, seed := userKey(t)
wrap := func(inner string) string {
return `<DescribeStoreResponse><DescribeStoreResult>` + inner + `</DescribeStoreResult></DescribeStoreResponse>`
}
t.Run("pending", func(t *testing.T) {
c, _, seen := server(t, 200, wrap(`<State>pending</State>`))
st, err := c.DescribeStore(context.Background())
if err != nil || st.Bound {
t.Fatalf("store = %+v %v", st, err)
}
if !strings.Contains((*seen)[0], "Action=DescribeStore") {
t.Fatalf("request = %q", (*seen)[0])
}
})
t.Run("bound", func(t *testing.T) {
c, _, _ := server(t, 200, wrap(`<State>bound</State><Address>10.240.0.9</Address><PublicKey>`+pub+`</PublicKey>`))
st, err := c.DescribeStore(context.Background())
if err != nil || !st.Bound || st.Address != "10.240.0.9" || st.PublicKey != pub {
t.Fatalf("store = %+v %v", st, err)
}
})
for name, inner := range map[string]string{
"pending with address": `<State>pending</State><Address>10.240.0.9</Address>`,
"bound without key": `<State>bound</State><Address>10.240.0.9</Address>`,
"bound with seed": `<State>bound</State><Address>10.240.0.9</Address><PublicKey>` + seed + `</PublicKey>`,
"bound with hostname": `<State>bound</State><Address>store.example</Address><PublicKey>` + pub + `</PublicKey>`,
"bound with IPv6": `<State>bound</State><Address>::ffff:10.240.0.9</Address><PublicKey>` + pub + `</PublicKey>`,
"unknown state": `<State>ready</State>`,
} {
t.Run(name, func(t *testing.T) {
c, _, _ := server(t, 200, wrap(inner))
_, err := c.DescribeStore(context.Background())
if err == nil {
t.Fatal("accepted an invalid store state")
}
if strings.Contains(err.Error(), seed) {
t.Fatal("the error echoes a seed")
}
})
}
}
+1 -1
View File
@@ -10,7 +10,7 @@ Node installation, dev-environment lifecycle, guest image builds, and verificati
| `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after `OVN_RAFT_REJOIN_WAIT` seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks `openvswitch-ipsec.service` and `ipsec.service` for the daemon to unmask when IPsec is on, unless `openvswitch-ipsec.service` is already running; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
| `install-node.sh` | Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs `spx admin init` on the first host and `spx admin join` on the rest, then verifies. Before touching any host it refuses hosts of different OS families (`distro_os_family`) or `spx` versions, and `--ipsec on` when any host uses libreswan. The scripted form of [install-multi-node](../docs/install/install-multi-node/README.md). Destroys the joining nodes' logical network state and crypto — see its `--help`. |
| `dev-install.sh` | Full local dev setup through the production installer: builds from source, packs a tarball, runs `setup.sh`, initialises the cluster, and starts services under systemd. |
| `build-oracle-appliance.sh` | Turns an Oracle Linux 10.2 system reachable over SSH with passwordless sudo into a sealed gold image, then powers it off. It updates OL and reboots into the newest UEK R8, sets SELinux permissive, installs Spinifex without initialising it, and leaves `spinifex.target` and the NATS watchdog disabled. It creates the `spinifex` login with a default password that must be changed at first login, installs `appliance/firstboot.sh` as `spinifex-appliance-firstboot.service`, and wipes the system's identity (machine-id, SSH host keys, OVS chassis, NetworkManager profiles, logs). On each unit's first boot that service puts the first physical NIC with carrier on a DHCP `br-wan`, names the host `spinifex-<mac>`, runs `setup-ovn.sh --management` and `spx admin init --nodes 1`, and shows the IP and login on the console and in `/etc/motd`. `--channel`, `--version` and `--tarball` pick what to install. |
| `build-oracle-appliance.sh` | Turns an Oracle Linux 10.2 system reachable over SSH with sudo into a sealed gold image, then powers it off. It updates OL and reboots into the newest UEK R8, sets SELinux permissive, installs Spinifex without initialising it, and leaves `spinifex.target` and the NATS watchdog disabled. It creates the `spinifex` login with a default password that must be changed at first login, installs `appliance/firstboot.sh` as `spinifex-appliance-firstboot.service`, and wipes the system's identity (machine-id, SSH host keys, OVS chassis, NetworkManager profiles, logs). On each unit's first boot that service puts the first physical NIC with carrier on a DHCP `br-wan`, names the host `spinifex-<mac>`, runs `setup-ovn.sh --management` and `spx admin init --nodes 1`, and shows the IP and login on the console and in `/etc/motd`. `--channel`, `--version` and `--tarball` pick what to install. |
| `clone-deps.sh` | Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
### `lib/distro.sh`: host-OS capability probes
+25 -12
View File
@@ -8,7 +8,7 @@
# prepare.sh appliance spinifex login, first-boot service, console placeholder
# prepare.sh seal wipe this system's identity, then power off
# INSTALL_SPINIFEX_CHANNEL, INSTALL_SPINIFEX_VERSION and INSTALL_SPINIFEX_TARBALL
# pass through to the installer. BUILD_USER is the SSH user to lock at seal.
# pick the release tarball to install. BUILD_USER is the SSH user to lock at seal.
set -euo pipefail
# Shared by every unit built from this image; first login forces a change.
@@ -64,19 +64,24 @@ stage_update() {
stage_install() {
[ ! -e /etc/spinifex/spinifex.toml ] || die "/etc/spinifex/spinifex.toml exists; this system has been initialised, start from a clean OL install"
local args=()
[ -n "${INSTALL_SPINIFEX_CHANNEL:-}" ] && args+=(--channel "$INSTALL_SPINIFEX_CHANNEL")
[ -n "${INSTALL_SPINIFEX_VERSION:-}" ] && args+=(--version "$INSTALL_SPINIFEX_VERSION")
# install.mulgadc.com serves the latest stable setup.sh, which may predate
# Oracle Linux support, so run the setup.sh inside the release tarball.
local dir tarball
dir=$(mktemp -d)
if [ -n "${INSTALL_SPINIFEX_TARBALL:-}" ]; then
# A local build: the payload carries the setup.sh of the checkout the
# tarball was built from, and setup.sh reads the tarball from the env.
[ -f "$HERE/setup.sh" ] || die "no setup.sh in the payload for INSTALL_SPINIFEX_TARBALL"
INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash "$HERE/setup.sh" "${args[@]}"
tarball=$INSTALL_SPINIFEX_TARBALL
else
curl -fsSL -o /tmp/spinifex-setup.sh https://install.mulgadc.com
INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash /tmp/spinifex-setup.sh "${args[@]}"
rm -f /tmp/spinifex-setup.sh
local url="https://install.mulgadc.com/download/${INSTALL_SPINIFEX_VERSION:-${INSTALL_SPINIFEX_CHANNEL:-latest}}/amd64"
tarball=$dir/spinifex.tar.gz
log "downloading $url"
curl -fsSL -o "$tarball" "$url"
curl -fsSL -o "$dir/checksum.sha256" "$url.sha256"
[ "$(awk '{print $1}' "$dir/checksum.sha256")" = "$(sha256sum "$tarball" | awk '{print $1}')" ] \
|| die "checksum mismatch for $url"
fi
tar -xzf "$tarball" -C "$dir" ./setup.sh
INSTALL_SPINIFEX_TARBALL=$tarball INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash "$dir/setup.sh"
rm -rf "$dir"
# The installer enables these on a live host. The ISO image ships them off
# and first boot enables them after init; a clone that started them first
# would run every service with no config.
@@ -96,6 +101,14 @@ stage_appliance() {
chmod 0440 /etc/sudoers.d/spinifex-admin
visudo -cf /etc/sudoers.d/spinifex-admin >/dev/null
# OL's pwquality gives one try per login, then ends the session, so say the
# rules up front. spx admin banner keeps /etc/motd content above its own
# section, and agetty shows /etc/issue.d beside the banner in /etc/issue.
local hint="First login: the new password needs 8+ characters and must not contain 'spinifex'."
printf '%s\n' "$hint" > /etc/motd
install -d /etc/issue.d
printf '%s\n\n' "$hint" > /etc/issue.d/spinifex-password.issue
# sshd takes the first value it reads, so this sorts ahead of the
# 50-cloud-init.conf that templates ship with passwords off.
printf 'PasswordAuthentication yes\n' > /etc/ssh/sshd_config.d/10-spinifex-appliance.conf
@@ -154,7 +167,7 @@ stage_seal() {
home=$(getent passwd "$BUILD_USER" | cut -d: -f6)
rm -f "$home/.ssh/authorized_keys" "$home/.bash_history"
passwd -l "$BUILD_USER" >/dev/null
rm -f "/etc/sudoers.d/90-cloud-init-users"
rm -f /etc/sudoers.d/90-cloud-init-users /etc/sudoers.d/90-spinifex-build
fi
rm -f /root/.ssh/authorized_keys /root/.bash_history /home/spinifex/.bash_history
+14 -5
View File
@@ -9,8 +9,8 @@
# --version <tag> install a specific release
# --tarball <path> install a local release tarball instead
#
# Ports, keys and jump hosts come from ~/.ssh/config. <user> needs
# passwordless sudo.
# Ports, keys and jump hosts come from ~/.ssh/config. <user> needs sudo;
# without passwordless sudo, sudo asks for its password once.
set -euo pipefail
HERE=$(cd "$(dirname "$0")" && pwd)
@@ -59,8 +59,7 @@ push_payload() {
rsh "rm -rf $REMOTE_DIR && mkdir -p $REMOTE_DIR"
tar -C "$PAYLOAD_SRC" -cf - . | rsh "tar -C $REMOTE_DIR -xf -"
if [ -n "$TARBALL" ]; then
# setup.sh from this checkout installs the tarball built from it.
scp "${SSH_OPTS[@]}" -q "$HERE/setup.sh" "$TARBALL" "$TARGET:$REMOTE_DIR/"
scp "${SSH_OPTS[@]}" -q "$TARBALL" "$TARGET:$REMOTE_DIR/"
fi
}
@@ -75,7 +74,17 @@ wait_for_ssh() {
die "$TARGET did not come back over SSH after the reboot"
}
rsh "sudo -n true" || die "$BUILD_USER needs passwordless sudo on the build system"
rsh true || die "cannot SSH to $TARGET; set its key in ~/.ssh/config (IdentityFile) or ssh-add it"
# The stages run sudo with no terminal, so grant passwordless sudo for the
# build when the user lacks it. Seal removes the grant.
if ! rsh "sudo -n true" 2>/dev/null; then
log "Granting $BUILD_USER passwordless sudo for the build; sudo asks for its password once"
ssh "${SSH_OPTS[@]}" -t "$TARGET" "echo '$BUILD_USER ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-spinifex-build >/dev/null && sudo chmod 0440 /etc/sudoers.d/90-spinifex-build" \
|| die "could not grant $BUILD_USER passwordless sudo on the build system"
rsh "sudo -n true" || die "$BUILD_USER still has no passwordless sudo on the build system"
fi
# A minimal OL install has no tar, which the payload copy and the installer need.
rsh "command -v tar >/dev/null || sudo dnf install -y -q tar" || die "cannot install tar on the build system"
log "Copying the payload to $TARGET"
push_payload
+518
View File
@@ -0,0 +1,518 @@
#!/bin/bash
# build-sqs-store-microvm-image.sh — build the SQS store microVM kernel +
# initramfs: the dedicated nats-server, ext4 support, the store's guest steps
# and a clean power-down. The guest never formats a disk; build/microvm/
# sqs-store/sqs-store-lib.sh states its fw_cfg contract. The LB image
# (build-microvm-image.sh) is separate and unchanged.
#
# SQS_MICROVM_PROFILE=worker builds the shared SQS worker guest instead: the
# same network /init and kernel, build/microvm/sqs-worker's steps and
# $REPO_ROOT/bin/sqs-worker, with no nats-server.
#
# Inputs:
# $REPO_ROOT/build/microvm/sqs-store/init.sh, inittab
# $REPO_ROOT/build/microvm/sqs-store/sqs-store-{lib.sh,prepare,run,stop}
# $REPO_ROOT/bin/nats-server (make bin/nats-server, the go.mod-pinned version)
#
# Outputs:
# $REPO_ROOT/build/microvm/sqs-store/vmlinuz
# $REPO_ROOT/build/microvm/sqs-store/initramfs.cpio.gz
#
# Rootfs strategy (first match wins):
# 1. apk available on host → install directly into a temp chroot (Alpine CI/CD)
# 2. docker available → run alpine container, export filesystem
# 3. podman available → same via podman
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
BUILD_DIR="$REPO_ROOT/build/microvm/sqs-store"
INIT_SH="$BUILD_DIR/init.sh"
INITTAB="$BUILD_DIR/inittab"
NATS_SERVER_BIN="$REPO_ROOT/bin/nats-server"
PROFILE="${SQS_MICROVM_PROFILE:-store}"
case "$PROFILE" in
store) ;;
worker)
BUILD_DIR="$REPO_ROOT/build/microvm/sqs-worker"
INITTAB="$BUILD_DIR/inittab"
SQS_WORKER_BIN="$REPO_ROOT/bin/sqs-worker"
;;
*) echo "ERROR: SQS_MICROVM_PROFILE must be store or worker, not $PROFILE" >&2; exit 1 ;;
esac
TAG="[build-sqs-$PROFILE-microvm-image]"
ALPINE_VERSION="${ALPINE_VERSION:-3.24}"
ALPINE_MIRROR="${ALPINE_MIRROR:-https://dl-cdn.alpinelinux.org/alpine}"
# e2fsprogs: mkfs.ext4/e2fsck for a guest that mounts an ext4 data volume.
# kmod: the image's own depmod/modprobe rebuild module metadata below.
ALPINE_PACKAGES="busybox linux-virt iproute2 ca-certificates e2fsprogs kmod"
# Modules kept by name on top of the virtio/fw_cfg/failover globs, each with its
# full depends= and softdep closure. tiny_power_button turns the ACPI power
# button (QMP system_powerdown) into a signal busybox init handles as poweroff.
MICROVM_EXTRA_MODULES="${MICROVM_EXTRA_MODULES:-ext4 tiny_power_button}"
echo "$TAG repo root: $REPO_ROOT"
echo "$TAG build dir: $BUILD_DIR"
mkdir -p "$BUILD_DIR"
# --- Verify non-negotiable tools ---
# fakeroot is mandatory: device nodes are written into the cpio archive inside a
# faked-root environment so no real device node is ever created on the host
# filesystem (which previously required sudo mknod and risked clobbering host
# /dev/null on cleanup).
for tool in cpio gzip find fakeroot; do
if ! command -v "$tool" >/dev/null 2>&1; then
echo "ERROR: required tool not found: $tool" >&2
exit 1
fi
done
if [ "$PROFILE" = worker ]; then
if [ ! -x "$SQS_WORKER_BIN" ]; then
echo "ERROR: $SQS_WORKER_BIN not found — run 'make bin/sqs-worker'" >&2
exit 1
fi
if ! file -b "$SQS_WORKER_BIN" | grep -q "statically linked"; then
echo "ERROR: $SQS_WORKER_BIN is not statically linked — it cannot run on the musl guest" >&2
exit 1
fi
else
# --- Verify the dedicated nats-server is the go.mod-pinned build ---
# A stale or host-installed binary would ship an unreviewed server version.
NATS_SERVER_PIN=$(awk '$1 == "github.com/nats-io/nats-server/v2" { print $2; exit }' "$REPO_ROOT/go.mod")
if [ -z "$NATS_SERVER_PIN" ]; then
echo "ERROR: no github.com/nats-io/nats-server/v2 requirement in $REPO_ROOT/go.mod" >&2
exit 1
fi
if [ ! -x "$NATS_SERVER_BIN" ]; then
echo "ERROR: $NATS_SERVER_BIN not found — run 'make bin/nats-server'" >&2
exit 1
fi
if ! file -b "$NATS_SERVER_BIN" | grep -q "statically linked"; then
echo "ERROR: $NATS_SERVER_BIN is not statically linked — it cannot run on the musl guest" >&2
exit 1
fi
nats_server_version=$("$NATS_SERVER_BIN" --version 2>&1 | awk '{ print $NF; exit }')
if [ "$nats_server_version" != "$NATS_SERVER_PIN" ]; then
echo "ERROR: $NATS_SERVER_BIN reports $nats_server_version, go.mod pins $NATS_SERVER_PIN" >&2
exit 1
fi
echo "$TAG nats-server: $nats_server_version (matches go.mod)"
fi
# --- Create chroot ---
CHROOT_DIR=$(mktemp -d)
# Guard: every destructive op below targets "$CHROOT_DIR/...". An empty or
# unexpected value would aim rm -rf at the host (e.g. /dev). Refuse anything that
# is not a freshly-created temp dir.
case "${CHROOT_DIR:?mktemp -d returned empty}" in
/tmp/*|/var/tmp/*|"${TMPDIR:-/nonexistent}"/*) ;;
*) echo "ERROR: refusing to use unexpected CHROOT_DIR: $CHROOT_DIR" >&2; exit 1 ;;
esac
# Outputs are written here first, beside $BUILD_DIR so publishing is a rename.
STAGE_DIR=$(mktemp -d "$BUILD_DIR/.stage.XXXXXX")
CONTAINER_TOOL=""
CONTAINER_CID=""
cleanup() {
echo "$TAG cleaning up chroot: $CHROOT_DIR"
rm -rf "$CHROOT_DIR"
rm -rf "${STAGE_DIR:?}"
if [ -n "$CONTAINER_CID" ] && [ -n "$CONTAINER_TOOL" ]; then
echo "$TAG cleaning up container: $CONTAINER_CID"
$CONTAINER_TOOL rm -f "$CONTAINER_CID" >/dev/null 2>&1 || true
fi
}
trap cleanup EXIT
# --- Populate rootfs ---
# Strategy 1: native apk (Alpine hosts, CI)
if command -v apk >/dev/null 2>&1; then
echo "$TAG rootfs: native apk"
mkdir -p \
"$CHROOT_DIR/etc/apk" \
"$CHROOT_DIR/lib/apk/db" \
"$CHROOT_DIR/var/cache/apk"
cat > "$CHROOT_DIR/etc/apk/repositories" <<EOF
${ALPINE_MIRROR}/v${ALPINE_VERSION}/main
${ALPINE_MIRROR}/v${ALPINE_VERSION}/community
EOF
apk add \
--root "$CHROOT_DIR" \
--arch x86_64 \
--no-cache \
--allow-untrusted \
--initdb \
$ALPINE_PACKAGES
# Strategy 2: docker -> BuildKit local export.
# Do NOT use `docker export` here. On Docker's overlay2 driver it leaks the
# container's overlay mount (the mount survives `docker rm`), orphaning layers
# under /var/lib/docker that only a daemon restart reclaims; on a busy CI host
# this grows without bound. `buildx --output type=local` writes the image
# filesystem straight to a directory and releases every mount, so nothing leaks.
elif command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
echo "$TAG rootfs: docker buildx --output type=local (alpine:${ALPINE_VERSION})"
BUILD_CTX=$(mktemp -d)
cat > "$BUILD_CTX/Dockerfile" <<DOCKERFILE
FROM alpine:${ALPINE_VERSION}
RUN apk add --no-cache ${ALPINE_PACKAGES}
DOCKERFILE
docker buildx build \
--platform linux/amd64 \
--output "type=local,dest=${CHROOT_DIR}" \
"$BUILD_CTX"
rm -rf "$BUILD_CTX"
# Strategy 3: podman -> run + export. Podman's store does not exhibit the docker
# overlay2 export-mount leak, and podman has no buildx.
elif command -v podman >/dev/null 2>&1; then
echo "$TAG rootfs: podman export (alpine:${ALPINE_VERSION})"
CONTAINER_TOOL=podman
CONTAINER_CID=$(podman run -d \
"alpine:${ALPINE_VERSION}" \
sh -c "apk add --no-cache ${ALPINE_PACKAGES}")
exit_code=$(podman wait "$CONTAINER_CID")
if [ "$exit_code" != "0" ]; then
echo "ERROR: package installation failed in container (exit $exit_code)" >&2
exit 1
fi
podman export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
else
echo "ERROR: no rootfs build tool available." >&2
echo " Install one of: apk (Alpine), docker, podman" >&2
exit 1
fi
# --- /dev: empty mountpoint only ---
# tar extraction without root silently creates 0-byte regular files for device
# nodes. Drop them and leave an empty /dev mountpoint. The two static nodes the
# kernel needs (/dev/console to wire init's stdio, /dev/null) are written into
# the cpio archive under fakeroot at pack time — NOT created on the host. init
# mounts devtmpfs over /dev as its first action for everything else at runtime.
rm -rf "${CHROOT_DIR:?}/dev"
mkdir "$CHROOT_DIR/dev"
# --- Place init script ---
echo "$TAG installing init.sh as /init..."
if [ ! -f "$INIT_SH" ]; then
echo "ERROR: $INIT_SH not found — cannot build initramfs" >&2
exit 1
fi
install -m 0755 "$INIT_SH" "$CHROOT_DIR/init"
# --- Place inittab ---
# Required: without it busybox init runs its default table, not the power-down.
if [ ! -f "$INITTAB" ]; then
echo "ERROR: $INITTAB not found — cannot build initramfs" >&2
exit 1
fi
echo "$TAG installing inittab..."
install -m 0644 "$INITTAB" "$CHROOT_DIR/etc/inittab"
if [ "$PROFILE" = worker ]; then
echo "$TAG installing worker guest steps..."
for step in sqs-worker-lib.sh sqs-worker-prepare sqs-worker-ready sqs-worker-stop; do
if [ ! -f "$BUILD_DIR/$step" ]; then
echo "ERROR: $BUILD_DIR/$step not found — cannot build initramfs" >&2
exit 1
fi
done
mkdir -p "$CHROOT_DIR/usr/local/lib/sqs-worker" "$CHROOT_DIR/usr/local/bin"
install -m 0755 "$BUILD_DIR/sqs-worker-lib.sh" "$BUILD_DIR/sqs-worker-prepare" "$BUILD_DIR/sqs-worker-ready" "$BUILD_DIR/sqs-worker-stop" \
"$CHROOT_DIR/usr/local/lib/sqs-worker/"
install -m 0755 "$SQS_WORKER_BIN" "$CHROOT_DIR/usr/local/bin/sqs-worker"
# The mountpoint ships empty; the worker proves its disk is mounted here.
mkdir -p "$CHROOT_DIR/data"
else
# --- Place the store's guest steps ---
echo "$TAG installing store guest steps..."
mkdir -p "$CHROOT_DIR/usr/local/lib/sqs-store"
for step in sqs-store-lib.sh sqs-store-prepare sqs-store-run sqs-store-stop; do
if [ ! -f "$BUILD_DIR/$step" ]; then
echo "ERROR: $BUILD_DIR/$step not found — cannot build initramfs" >&2
exit 1
fi
done
install -m 0644 "$BUILD_DIR/sqs-store-lib.sh" "$CHROOT_DIR/usr/local/lib/sqs-store/"
install -m 0755 "$BUILD_DIR/sqs-store-prepare" "$BUILD_DIR/sqs-store-run" "$BUILD_DIR/sqs-store-stop" \
"$CHROOT_DIR/usr/local/lib/sqs-store/"
# --- Copy nats-server binary ---
echo "$TAG copying nats-server binary..."
mkdir -p "$CHROOT_DIR/usr/local/bin"
install -m 0755 "$NATS_SERVER_BIN" "$CHROOT_DIR/usr/local/bin/nats-server"
fi
# --- Create required directories for init ---
mkdir -p \
"$CHROOT_DIR/proc" \
"$CHROOT_DIR/sys" \
"$CHROOT_DIR/dev" \
"$CHROOT_DIR/etc/ssl/certs"
# --- Locate kernel ---
KERNEL_IMG=$(find "$CHROOT_DIR/boot" -name "vmlinuz*" | sort | tail -1)
if [ -z "$KERNEL_IMG" ]; then
echo "ERROR: no vmlinuz found in $CHROOT_DIR/boot" >&2
exit 1
fi
echo "$TAG kernel image: $KERNEL_IMG"
KERNEL_VER=$(basename "$KERNEL_IMG" | sed 's/vmlinuz-//')
KERNEL_CONFIG="$CHROOT_DIR/boot/config-${KERNEL_VER}"
# --- Assert fw_cfg module ---
if ! find "$CHROOT_DIR/lib/modules" \
-name "qemu_fw_cfg.ko*" \
-o -name "fw_cfg_sysfs.ko*" \
| grep -q .; then
if ! grep -q "CONFIG_FW_CFG_SYSFS=y" "$KERNEL_CONFIG" 2>/dev/null; then
echo "ERROR: qemu_fw_cfg module missing from initramfs — fw_cfg reads will fail at boot" >&2
exit 1
fi
echo "$TAG fw_cfg: built-in (CONFIG_FW_CFG_SYSFS=y)"
else
echo "$TAG fw_cfg: module found in lib/modules"
fi
# --- Assert ext4 userspace tools (data volume) ---
for fs_tool in mkfs.ext4 e2fsck; do
if [ ! -x "$CHROOT_DIR/sbin/$fs_tool" ] && [ ! -x "$CHROOT_DIR/usr/sbin/$fs_tool" ]; then
echo "ERROR: $fs_tool missing from image — guest cannot check or create an ext4 data volume" >&2
exit 1
fi
done
echo "$TAG e2fsprogs: mkfs.ext4 + e2fsck present"
# --- Strip kernel modules (Phase 2 keep-list approach) ---
# Only modules needed by a virtio-mmio microvm guest are kept:
# virtio*.ko* — virtio bus, net, blk, rng, console, and helpers
# *fw_cfg*.ko* — QEMU fw_cfg sysfs driver (delivers boot blobs)
# *failover*.ko* — net_failover + failover (virtio_net deps for live migration)
# $MICROVM_EXTRA_MODULES — ext4 by default, plus dependency closure
# All other loadable modules are removed. Drivers compiled into the kernel
# (CONFIG_VIRTIO_MMIO=y etc.) are unaffected — they live in vmlinuz, not here.
# module_path prints a module's kernel/ path from the stock modules.dep, by
# name or modules.alias; it fails when the module is not loadable.
module_path() {
local kdir="$1" name="${2//-/_}" path alias_target
path=$(awk -F: -v n="$name" '{ f = $1; sub(/.*\//, "", f); sub(/\.ko(\.gz|\.xz|\.zst)?$/, "", f);
gsub(/-/, "_", f); if (f == n) { print $1; exit } }' "$kdir/modules.dep")
if [ -z "$path" ] && [ -f "$kdir/modules.alias" ]; then
alias_target=$(awk -v n="$2" '$1 == "alias" && $2 == n { print $3; exit }' "$kdir/modules.alias")
[ -n "$alias_target" ] && [ "$alias_target" != "$2" ] && path=$(module_path "$kdir" "$alias_target")
fi
[ -n "$path" ] && printf '%s\n' "$path"
}
# module_builtin reports whether a module name is compiled into the kernel.
module_builtin() {
local kdir="$1" name="${2//-/_}"
[ -f "$kdir/modules.builtin" ] || return 1
awk -v n="$name" '{ f = $0; sub(/.*\//, "", f); sub(/\.ko$/, "", f); gsub(/-/, "_", f);
if (f == n) { found = 1; exit } } END { exit !found }' "$kdir/modules.builtin"
}
# image_kmod runs the image's own kmod (depmod, modprobe) through its musl
# loader, so module metadata matches the guest's modprobe without a host kmod.
image_kmod() {
"$CHROOT_DIR/lib/ld-musl-x86_64.so.1" --library-path "$CHROOT_DIR/lib:$CHROOT_DIR/usr/lib" \
--argv0 "$1" "$CHROOT_DIR/bin/kmod" "${@:2}"
}
echo "$TAG stripping kernel modules to virtio+fw_cfg+${MICROVM_EXTRA_MODULES} only..."
for kver_dir in "$CHROOT_DIR/lib/modules"/*/; do
kver="$(basename "$kver_dir")"
kernel_dir="$kver_dir/kernel"
[ -d "$kernel_dir" ] || continue
if [ ! -s "$kver_dir/modules.dep" ]; then
echo "ERROR: stock ${kver_dir}modules.dep missing — cannot resolve module dependencies" >&2
exit 1
fi
# Extra modules start the closure; the glob families are kept as found.
keep_list=$(mktemp)
: > "$keep_list"
for name in $MICROVM_EXTRA_MODULES; do
if ! path=$(module_path "$kver_dir" "$name"); then
if module_builtin "$kver_dir" "$name"; then continue; fi
echo "ERROR: module $name is neither loadable nor built into $kver" >&2
exit 1
fi
echo "$path" >> "$keep_list"
done
# Close over modules.dep hard dependencies and modules.softdep pre:/post:
# names until the set stops growing. A softdep must be loadable or built in.
while :; do
before=$(sort -u "$keep_list" | wc -l)
for path in $(sort -u "$keep_list"); do
awk -F': ' -v p="$path" '$1 == p { n = split($2, d, " "); for (i = 1; i <= n; i++) print d[i] }' \
"$kver_dir/modules.dep" >> "$keep_list"
mod=$(basename "$path"); mod="${mod%%.ko*}"; mod="${mod//-/_}"
[ -f "$kver_dir/modules.softdep" ] || continue
for soft in $(awk -v n="$mod" '{ m = $2; gsub(/-/, "_", m) }
$1 == "softdep" && m == n { for (i = 3; i <= NF; i++) if ($i != "pre:" && $i != "post:") print $i }' \
"$kver_dir/modules.softdep"); do
if soft_path=$(module_path "$kver_dir" "$soft"); then
echo "$soft_path" >> "$keep_list"
elif ! module_builtin "$kver_dir" "$soft"; then
echo "ERROR: softdep $soft of $mod is neither loadable nor built into $kver" >&2
exit 1
fi
done
done
after=$(sort -u "$keep_list" | wc -l)
[ "$before" -eq "$after" ] && break
done
extra_names=$(sort -u "$keep_list" | while read -r p; do m=$(basename "$p"); m="${m%%.ko*}"; echo "${m//-/_}"; done)
(cd "$kver_dir" && find kernel -name "virtio*.ko*" -o -name "*fw_cfg*.ko*" \
-o -name "*failover*.ko*") >> "$keep_list"
# Copy the keeper set out, preserving each kernel/ relative path.
tmp_keep=$(mktemp -d)
while read -r rel; do
rel="${rel#kernel/}"
mkdir -p "$tmp_keep/$(dirname "$rel")"
cp "$kernel_dir/$rel" "$tmp_keep/$rel"
done < <(sort -u "$keep_list")
rm -f "$keep_list"
# Replace kernel module tree with the keeper set.
rm -rf "$kernel_dir"
mkdir -p "$kernel_dir"
# Restore each kept file into the same relative path.
(cd "$tmp_keep" && find . -name "*.ko*" | while read -r f; do
dest_dir="$kernel_dir/$(dirname "${f#./}")"
mkdir -p "$dest_dir"
cp "$f" "$dest_dir/"
done)
rm -rf "$tmp_keep"
# Decompress kept modules to plain .ko. Alpine ships .ko.gz, but a host
# kmod that lacks gzip support (e.g. Debian trixie's kmod 34) silently
# produces an EMPTY modules.dep from compressed modules — the guest's
# modprobe then no-ops and qemu_fw_cfg never loads. Plain .ko sidesteps the
# host-kmod dependency entirely and the guest needs no decompressor either.
find "$kernel_dir" -name "*.ko.gz" -exec gunzip -f {} +
# Independent closure gate: each extra-closure module's own .modinfo depends=
# must name a kept module or a built-in, so a resolver bug fails here.
kept_names=$(find "$kernel_dir" -name "*.ko" -exec basename {} .ko \; | tr '-' '_' | sort -u)
while read -r ko; do
ko_name=$(basename "$ko" .ko); ko_name="${ko_name//-/_}"
grep -qx "$ko_name" <<< "$extra_names" || continue
deps=$(tr '\0' '\n' < "$ko" | grep -a -m1 '^depends=' | cut -d= -f2 | tr ',' ' ' || true)
for dep in $deps; do
dep="${dep//-/_}"
if ! grep -qx "$dep" <<< "$kept_names" && ! module_builtin "$kver_dir" "$dep"; then
echo "ERROR: $(basename "$ko") depends on $dep, which is neither kept nor built in" >&2
exit 1
fi
done
done < <(find "$kernel_dir" -name "*.ko")
for name in $MICROVM_EXTRA_MODULES; do
if ! grep -qx "${name//-/_}" <<< "$kept_names" && ! module_builtin "$kver_dir" "$name"; then
echo "ERROR: required module $name missing from the stripped module tree" >&2
exit 1
fi
done
echo "$TAG extra module closure: $(echo "$extra_names" | tr '\n' ' ')"
echo "$TAG modules kept: $(echo "$kept_names" | tr '\n' ' ')"
# Rebuild module metadata with the image's own kmod, the binary the guest's
# modprobe runs, so no host depmod is needed. Stock indexes still name .ko.gz
# files, so they are removed first and a failed rebuild cannot ship them.
for f in dep dep.bin alias alias.bin softdep weakdep symbols symbols.bin \
builtin.bin builtin.alias.bin devname; do
rm -f "$kver_dir/modules.$f"
done
if ! image_kmod depmod -b "$CHROOT_DIR" "$kver"; then
echo "ERROR: image depmod failed for $kver" >&2
exit 1
fi
if [ ! -s "$kver_dir/modules.dep" ] || [ ! -s "$kver_dir/modules.dep.bin" ]; then
echo "ERROR: ${kver_dir}modules.dep or modules.dep.bin missing or empty after depmod" >&2
exit 1
fi
# Resolution gate: the image's modprobe must resolve every required module
# to plain .ko files present in the stripped tree.
for name in $MICROVM_EXTRA_MODULES qemu_fw_cfg virtio_net; do
module_builtin "$kver_dir" "$name" && continue
if ! plan=$(image_kmod modprobe -d "$CHROOT_DIR" -S "$kver" --show-depends "$name"); then
echo "ERROR: image modprobe cannot resolve $name for $kver" >&2
exit 1
fi
while read -r verb path _; do
[ "$verb" = "insmod" ] || continue
case "$path" in
"$CHROOT_DIR/lib/modules/$kver/kernel/"*.ko) [ -f "$path" ] && continue ;;
esac
echo "ERROR: modprobe plan for $name names $path, not a kept plain .ko" >&2
exit 1
done <<< "$plan"
echo "$TAG modprobe $name: $(awk '{ n = split($2, a, "/"); printf "%s ", a[n] }' <<< "$plan")"
done
done
# --- Strip package-manager artifacts (not needed at runtime) ---
echo "$TAG stripping package manager artifacts..."
rm -rf \
"$CHROOT_DIR/lib/apk" \
"$CHROOT_DIR/var/cache/apk" \
"$CHROOT_DIR/etc/apk" \
"$CHROOT_DIR/usr/share/man" \
"$CHROOT_DIR/usr/share/doc" \
"$CHROOT_DIR/usr/share/apk" \
"$CHROOT_DIR/usr/include" \
"$CHROOT_DIR/usr/lib/pkgconfig"
# --- Strip debug symbols from binaries ---
# nats-server in /usr/local/bin is left as built (-s -w), so the shipped bytes
# are the ones whose version was checked above.
echo "$TAG stripping debug symbols from binaries..."
find "$CHROOT_DIR/usr/sbin" "$CHROOT_DIR/usr/bin" \
"$CHROOT_DIR/sbin" "$CHROOT_DIR/bin" \
-type f 2>/dev/null | while read -r bin; do
strip --strip-all "$bin" 2>/dev/null || true
done
# --- Copy vmlinuz out before stripping /boot from chroot ---
VMLINUZ_OUT="$STAGE_DIR/vmlinuz"
echo "$TAG copying kernel: $VMLINUZ_OUT"
cp "$KERNEL_IMG" "$VMLINUZ_OUT"
# Drop /boot from the chroot — vmlinuz, Alpine's stock initramfs-virt, System.map
# and kernel config all live here (~25 MiB). None are needed at runtime: the
# guest already has the kernel loaded by QEMU's -kernel, and Alpine's initramfs
# is unused because we replace it with our own /init. Leaving /boot in the cpio
# nearly doubles uncompressed initramfs size and forces a 512 MiB guest just to
# survive decompression.
rm -rf "${CHROOT_DIR:?}/boot"
# --- Build initramfs ---
INITRAMFS_OUT="$STAGE_DIR/initramfs.cpio.gz"
echo "$TAG building initramfs: $INITRAMFS_OUT"
# Create the static device nodes and pack the archive inside a single fakeroot
# session: mknod/cpio see faked device entries and write real char-device
# records into the cpio, while the host filesystem never gets an actual node.
fakeroot sh -c '
cd "$1" || exit 1
mknod -m 600 dev/console c 5 1
mknod -m 666 dev/null c 1 3
find . | cpio --quiet -o -H newc | gzip -9 > "$2"
' _ "$CHROOT_DIR" "$INITRAMFS_OUT"
# --- Gate the staged pair, then publish it ---
# Nothing reaches $BUILD_DIR until every gate passes, so a failed build cannot
# leave a pair the grouped make target would take as up to date.
"$REPO_ROOT/scripts/lib/sqs-store-microvm-publish.sh" "$STAGE_DIR" "$BUILD_DIR" "$PROFILE"
echo ""
echo "$TAG done."
+1 -1
View File
@@ -447,7 +447,7 @@ done
# Matched on the service argv so the ssh shell running this cannot match itself.
for host in "${HOSTS[@]}"; do
$DRY_RUN && { echo " [dry-run] $host: wait for every 'spx service' process to exit"; continue; }
on "$host" "for u in spinifex-predastore spinifex-viperblock spinifex-sqs-store spinifex-nats; do
on "$host" "for u in spinifex-predastore spinifex-viperblock spinifex-nats; do
systemctl is-active --quiet \$u && sudo systemctl stop \$u
done
for _ in \$(seq 1 45); do
+291
View File
@@ -0,0 +1,291 @@
#!/bin/sh
# sqs-store-guest_test.sh — the SQS store guest steps start the store only on
# a valid contract and one checked, mounted disk, never format, and stop the
# store before unmounting it.
#
# Drives build/microvm/sqs-store/sqs-store-lib.sh against a fake fw_cfg, sysfs
# and stub tools. The stop cases signal only test-owned children.
# The case settings below are read by the sourced library, not by this file.
# shellcheck disable=SC2034
set -eu
ROOT=$(CDPATH='' cd -- "$(dirname -- "$0")/../.." && pwd)
LIB=${SQS_GUEST_LIB:-"$ROOT/build/microvm/sqs-store/sqs-store-lib.sh"}
TMP=$(mktemp -d)
CHILDREN=""
cleanup() {
for p in $CHILDREN; do kill -KILL "$p" 2>/dev/null || true; done
rm -rf "$TMP"
}
trap cleanup EXIT
fails=0
pass() { echo " ok - $1"; }
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
check() { if eval "$2"; then pass "$1"; else fail "$1"; fi; }
echo "-- build/microvm/sqs-store/sqs-store-lib.sh"
SERIAL=vol0123456789abcdef0
NKEY=UAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB
ADDR=10.11.12.13
# Stub tools record each call to calls.log; their exit codes come from files.
mkdir -p "$TMP/bin"
cat > "$TMP/bin/e2fsck" <<'EOF'
#!/bin/sh
echo "e2fsck $*" >> "$T/calls.log"
exit "$(cat "$T/fsck.rc")"
EOF
cat > "$TMP/bin/mount" <<'EOF'
#!/bin/sh
echo "mount $*" >> "$T/calls.log"
exit "$(cat "$T/mount.rc")"
EOF
cat > "$TMP/bin/umount" <<'EOF'
#!/bin/sh
echo "umount $*" >> "$T/calls.log"
EOF
for f in mkfs mkfs.ext4 mke2fs; do
printf '#!/bin/sh\necho "mkfs %s $*" >> "$T/calls.log"\n' "$f" > "$TMP/bin/$f"
done
cat > "$TMP/bin/ip" <<'EOF'
#!/bin/sh
cat "$T/ip.out"
EOF
cat > "$TMP/bin/nats-server" <<'EOF'
#!/bin/sh
echo "nats-server $*" >> "$T/calls.log"
[ "$1" = -t ] || echo "GOMEMLIMIT=${GOMEMLIMIT:-unset}" >> "$T/env.log"
[ "$1" = -t ] && exit "$(cat "$T/configtest.rc")"
exit 0
EOF
chmod +x "$TMP/bin"/*
# fresh <name>: a case directory with a valid contract and one matching disk
# among others. Callers then break one thing.
fresh() {
T="$TMP/$1"
export T
mkdir -p "$T/fwcfg" "$T/sys" "$T/dev" "$T/run" "$T/data"
for k in data-serial listen-addr client-nkey; do mkdir -p "$T/fwcfg/$k"; done
printf '%s\n' "$SERIAL" > "$T/fwcfg/data-serial/raw"
printf '%s\n' "$ADDR" > "$T/fwcfg/listen-addr/raw"
printf '%s\n' "$NKEY" > "$T/fwcfg/client-nkey/raw"
disk vda vol00000000000000000
disk vdb "$SERIAL"
echo 0 > "$T/fsck.rc"
echo 0 > "$T/mount.rc"
echo 0 > "$T/configtest.rc"
: > "$T/calls.log"
: > "$T/console"
printf '1: lo inet 127.0.0.1/8 scope host lo\n2: eth0 inet %s/24 brd 10.11.12.255 scope global eth0\n' "$ADDR" > "$T/ip.out"
}
disk() { mkdir -p "$T/sys/$1"; printf '%s\n' "$2" > "$T/sys/$1/serial"; : > "$T/dev/$1"; }
# prepare: run sqs_prepare in a subshell with every path pointed at the case.
prepare() {
(
SQS_FWCFG="$T/fwcfg" SQS_SYSBLOCK="$T/sys" SQS_DEVDIR="$T/dev" SQS_NODE_TEST=-e
SQS_RUN="$T/run" SQS_DATA="$T/data" SQS_CONSOLE="$T/console"
E2FSCK="$TMP/bin/e2fsck" MOUNT="$TMP/bin/mount" IP="$TMP/bin/ip" NATS_BIN="$TMP/bin/nats-server"
PATH="$TMP/bin:$PATH"
export PATH
# shellcheck source=../../build/microvm/sqs-store/sqs-store-lib.sh
. "$LIB"
sqs_prepare
) > /dev/null 2>&1
}
refused() { [ ! -e "$T/run/ready" ] && grep -q FATAL "$T/console"; }
never_mounted() { ! grep -q '^mount ' "$T/calls.log"; }
never_formatted() { ! grep -q '^mkfs ' "$T/calls.log"; }
never_checked() { ! grep -q '^e2fsck ' "$T/calls.log"; }
# --- A valid contract prepares the store ---
fresh valid
check "valid contract is prepared" 'prepare && [ -e "$T/run/ready" ]'
check "the matching disk is checked, then mounted at the data dir" \
'grep -qx "e2fsck -p $T/dev/vdb" "$T/calls.log" && grep -qx "mount -t ext4 -o nodev,nosuid,noexec $T/dev/vdb $T/data" "$T/calls.log"'
check "config listens only on the supplied address" \
'grep -c "listen" "$T/run/nats.conf" | grep -qx 1 && grep -qx "listen: \"$ADDR:4222\"" "$T/run/nats.conf"'
check "config admits only the supplied user NKey" 'grep -q "users = \[ { nkey: \"$NKEY\" } \]" "$T/run/nats.conf"'
check "config syncs JetStream on the data disk" \
'grep -q "store_dir: \"$T/data\"" "$T/run/nats.conf" && grep -q "sync_interval: always" "$T/run/nats.conf"'
check "config is tested by the server before ready" 'grep -qx "nats-server -t -c $T/run/nats.conf" "$T/calls.log"'
check "no auth material on the console" '! grep -q "$NKEY" "$T/console"'
check "prepare reports the disk and config, not a listener" 'grep -q "prepared: $T/dev/vdb mounted" "$T/console" && ! grep -qi "listening" "$T/console"'
fresh fsck-corrected; echo 1 > "$T/fsck.rc"
check "e2fsck exit 1 (corrected) proceeds" 'prepare && [ -e "$T/run/ready" ]'
# --- Disk refusals: nothing mounted, nothing formatted ---
fresh no-disk; rm -rf "$T/sys/vdb"
check "missing disk is refused before any check or mount" '! prepare && refused && never_checked && never_mounted'
fresh wrong-disk; printf 'vol0fffffffffffffff0\n' > "$T/fwcfg/data-serial/raw"
check "disk with another serial is refused" '! prepare && refused && never_checked && never_mounted'
fresh two-disks; disk vdc "$SERIAL"
check "two disks with the serial are refused" '! prepare && refused && never_checked && never_mounted'
fresh bad-serial; printf 'vol-0123456789abcdef\n' > "$T/fwcfg/data-serial/raw"
check "malformed serial is refused" '! prepare && refused && never_mounted'
fresh no-node; rm -f "$T/dev/vdb"
check "disk without a device node is refused" '! prepare && refused && never_mounted'
for rc in 2 4 8; do
fresh "fsck-$rc"; echo "$rc" > "$T/fsck.rc"
check "e2fsck exit $rc is refused without mount or format" '! prepare && refused && never_mounted && never_formatted'
done
fresh mount-fail; echo 32 > "$T/mount.rc"
check "failed mount is refused" '! prepare && refused'
# --- Auth and listener refusals happen before the disk is touched ---
fresh no-nkey; rm -rf "$T/fwcfg/client-nkey"
check "missing NKey is refused before the disk" '! prepare && refused && never_checked && never_mounted'
for bad in seed lower short two empty; do
fresh "nkey-$bad"
case $bad in
seed) v=SUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ;;
lower) v=uaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab ;;
short) v=UAAAA ;;
two) v="$NKEY
$NKEY" ;;
empty) v="" ;;
esac
printf '%s\n' "$v" > "$T/fwcfg/client-nkey/raw"
check "NKey ($bad) is refused before the disk, without echoing it" \
'! prepare && refused && never_checked && { [ -z "$v" ] || ! grep -qF "$v" "$T/console"; }'
done
fresh nkey-checksum; echo 1 > "$T/configtest.rc"
check "NKey the server rejects is refused before the disk" '! prepare && refused && never_checked && never_mounted'
fresh no-addr; rm -rf "$T/fwcfg/listen-addr"
check "missing listen address is refused" '! prepare && refused && never_mounted'
fresh addr-absent; printf '1: lo inet 127.0.0.1/8 scope host lo\n2: eth0 inet 10.11.12.99/24 scope global eth0\n' > "$T/ip.out"
check "listen address not on a guest NIC is refused" '! prepare && refused && never_mounted'
fresh addr-loopback; printf '127.0.0.1\n' > "$T/fwcfg/listen-addr/raw"
check "loopback listen address is refused" '! prepare && refused && never_mounted'
for bad in 0.0.0.0x 1.2.3 1.2.3.256 01.2.3.4; do
fresh "addr-$bad"; printf '%s\n' "$bad" > "$T/fwcfg/listen-addr/raw"
check "malformed listen address $bad is refused" '! prepare && refused && never_mounted'
done
# --- Run: the store starts only when prepared and not stopping ---
run_step() {
(
SQS_RUN="$T/run" SQS_CONSOLE="$T/console" NATS_BIN="$TMP/bin/nats-server"
# shellcheck source=../../build/microvm/sqs-store/sqs-store-lib.sh
. "$LIB"
sqs_run
) > /dev/null 2>&1
}
fresh run-ready; prepare; : > "$T/calls.log"
check "a prepared store starts the server" 'run_step && grep -qx "nats-server -c $T/run/nats.conf" "$T/calls.log"'
fresh run-memlimit; prepare; : > "$T/env.log"
check "the server runs under the ADR-0011:S11 heap limit" 'run_step && grep -qx "GOMEMLIMIT=512MiB" "$T/env.log"'
fresh run-unready
check "an unprepared store never starts the server" '! run_step && ! grep -q "^nats-server" "$T/calls.log"'
fresh run-stopping; prepare; : > "$T/calls.log"; : > "$T/run/stopping"
check "a stopping store never starts the server" '! run_step && ! grep -q "^nats-server" "$T/calls.log"'
# --- Stop: SIGTERM the verified store, then unmount ---
STORE="$TMP/bin/fake-store"
cp "$(command -v sleep)" "$STORE"
stop_step() {
(
SQS_RUN="$T/run" SQS_DATA="$T/data" SQS_CONSOLE="$T/console" SQS_MOUNTS="$T/mounts"
UMOUNT="$TMP/bin/umount" NATS_BIN="$STORE" SQS_STOP_BUDGET_TENTHS=10
# shellcheck source=../../build/microvm/sqs-store/sqs-store-lib.sh
. "$LIB"
sqs_stop
) > /dev/null 2>&1
}
gone() { [ ! -r "/proc/$1/status" ] || grep -q '^State:[[:space:]]*Z' "/proc/$1/status"; }
# record <pid>: write the identity sqs_run would record for that process.
# shellcheck source=../../build/microvm/sqs-store/sqs-store-lib.sh
record() { echo "$1 $( . "$LIB"; sqs_starttime "$1" )" > "$T/run/pid"; }
fresh stop-orderly; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
"$STORE" 60 & pid=$!; CHILDREN="$CHILDREN $pid"; record "$pid"
stop_step
check "stop sends SIGTERM and the store exits" 'gone "$pid" && grep -q "SIGTERM sent" "$T/console" && ! grep -q SIGKILL "$T/console"'
check "stop blocks respawn and unmounts after the store exits" \
'[ -e "$T/run/stopping" ] && grep -qx "umount $T/data" "$T/calls.log"'
fresh stop-hung; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
(trap '' TERM; exec "$STORE" 60) & pid=$!; CHILDREN="$CHILDREN $pid"; record "$pid"
stop_step
check "store ignoring SIGTERM is killed after the budget, then unmounted" \
'gone "$pid" && grep -q SIGKILL "$T/console" && grep -qx "umount $T/data" "$T/calls.log"'
fresh stop-reused; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
"$STORE" 60 & pid=$!; CHILDREN="$CHILDREN $pid"; echo "$pid 1" > "$T/run/pid"
stop_step
check "a recorded PID with another start time is never signalled" '! gone "$pid" && grep -q "no running store" "$T/console"'
kill -KILL "$pid"
# A start held after its stopping check and before its exec: the run step
# blocks writing "run: starting" to a FIFO console until the test reads it.
fresh stop-pending-start; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
: > "$T/run/ready"
mkfifo "$T/run-console"
printf '#!/bin/sh\n: > "%s/started"\nexec "%s" 60\n' "$T" "$STORE" > "$TMP/bin/pending-nats"
chmod +x "$TMP/bin/pending-nats"
SQS_RUN="$T/run" SQS_CONSOLE="$T/run-console" NATS_BIN="$TMP/bin/pending-nats" \
sh -c '. "$1"; sqs_run' sh "$LIB" & pid=$!; CHILDREN="$CHILDREN $pid"
i=0
while [ "$i" -lt 100 ] && ! grep -q "^$pid " "$T/run/pid" 2>/dev/null; do i=$((i + 1)); sleep 0.05; done
check "pending start is held before its exec" 'grep -q "^$pid " "$T/run/pid" && ! gone "$pid" && [ ! -e "$T/started" ]'
stop_step
# Release the FIFO: a start that survived the stop would exec the server now.
cat "$T/run-console" > /dev/null 2>&1 &
reader=$!; CHILDREN="$CHILDREN $reader"
i=0
while [ "$i" -lt 20 ] && ! gone "$pid"; do i=$((i + 1)); sleep 0.05; done
check "stop ends a pending start before it can exec the server" 'gone "$pid" && [ ! -e "$T/started" ]'
check "stop unmounts only after the pending start is gone" \
'grep -q "store exited" "$T/console" && grep -qx "umount $T/data" "$T/calls.log"'
kill -KILL "$reader" 2>/dev/null || true
# --- Shutdown: the generic remount runs only once the store is gone ---
# shutdown_step [survivor-pid]: with a pid, that process is reported alive
# whatever happens to it, as a writer stuck past SIGKILL would be.
shutdown_step() {
(
SQS_RUN="$T/run" SQS_DATA="$T/data" SQS_CONSOLE="$T/console" SQS_MOUNTS="$T/mounts"
UMOUNT="$TMP/bin/umount" SQS_STOP_BUDGET_TENTHS=5
# shellcheck source=../../build/microvm/sqs-store/sqs-store-lib.sh
. "$LIB"
if [ -n "${1:-}" ]; then sqs_same() { [ "$1" = "$survivor" ]; }; survivor=$1; fi
sqs_shutdown
) > /dev/null 2>&1
}
fresh shutdown-orderly; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
"$STORE" 60 & pid=$!; CHILDREN="$CHILDREN $pid"; record "$pid"
check "orderly shutdown succeeds" 'shutdown_step'
check "orderly shutdown unmounts the data disk, then runs umount -a -r" \
'[ "$(grep "^umount" "$T/calls.log" | tr "\n" "|")" = "umount $T/data|umount -a -r|" ]'
fresh shutdown-survivor; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
"$STORE" 60 & pid=$!; CHILDREN="$CHILDREN $pid"; record "$pid"
check "shutdown with a surviving store reports failure" '! shutdown_step "$pid"'
check "a surviving store leaves the data disk and every other mount alone" \
'! grep -q "^umount" "$T/calls.log" && grep -q "STILL ALIVE" "$T/console" && grep -q "skipping umount -a -r" "$T/console"'
fresh stop-unmounted; : > "$T/mounts"
stop_step
check "stop with no data mount does not unmount" '! grep -q "^umount" "$T/calls.log"'
if [ "$fails" -ne 0 ]; then
echo " $fails failure(s)"
exit 1
fi
+110
View File
@@ -0,0 +1,110 @@
#!/bin/bash
# sqs-store-microvm-publish.sh STAGE_DIR OUT_DIR [PROFILE] — gate a staged SQS
# store (default) or worker microVM kernel + initramfs and move them into
# OUT_DIR only if every gate passes.
#
# A failed gate leaves OUT_DIR as it was: a previous valid pair stays, and an
# older pair stays older than the build inputs, so make rebuilds next time.
set -euo pipefail
STAGE_DIR="${1:?usage: sqs-store-microvm-publish.sh STAGE_DIR OUT_DIR}"
OUT_DIR="${2:?usage: sqs-store-microvm-publish.sh STAGE_DIR OUT_DIR}"
VMLINUZ="$STAGE_DIR/vmlinuz"
INITRAMFS="$STAGE_DIR/initramfs.cpio.gz"
PROFILE="${3:-store}"
case "$PROFILE" in
store)
payload="usr/local/bin/nats-server usr/local/lib/sqs-store/sqs-store-lib.sh usr/local/lib/sqs-store/sqs-store-prepare usr/local/lib/sqs-store/sqs-store-run usr/local/lib/sqs-store/sqs-store-stop" ;;
worker)
payload="usr/local/bin/sqs-worker usr/local/lib/sqs-worker/sqs-worker-lib.sh usr/local/lib/sqs-worker/sqs-worker-prepare usr/local/lib/sqs-worker/sqs-worker-ready usr/local/lib/sqs-worker/sqs-worker-stop data" ;;
*) echo "ERROR: profile must be store or worker, not $PROFILE" >&2; exit 1 ;;
esac
tag="[build-sqs-$PROFILE-microvm-image]"
for f in "$VMLINUZ" "$INITRAMFS"; do
if [ ! -s "$f" ]; then
echo "ERROR: staged artifact $f missing or empty" >&2
exit 1
fi
done
# --- Content gate: read the packed archive, not the chroot ---
if ! archive_list=$(zcat "$INITRAMFS" | cpio --quiet -t); then
echo "ERROR: $INITRAMFS is not a readable gzip cpio archive" >&2
exit 1
fi
# shellcheck disable=SC2086 # payload is a word list on purpose
for want in init etc/inittab dev/console dev/null $payload; do
if ! grep -qx "$want" <<< "$archive_list"; then
echo "ERROR: $want missing from $INITRAMFS" >&2
exit 1
fi
done
for fs_tool in mkfs.ext4 e2fsck; do
if ! grep -qxE "(usr/)?sbin/$fs_tool" <<< "$archive_list"; then
echo "ERROR: $fs_tool missing from $INITRAMFS" >&2
exit 1
fi
done
# Module files may spell a name with dashes (tiny-power-button.ko). Normalise
# once: piping into grep -q under pipefail fails when grep exits on a match.
module_list=${archive_list//-/_}
for want_ko in ext4 tiny_power_button; do
if ! grep -q "/$want_ko\.ko\$" <<< "$module_list"; then
echo "ERROR: $want_ko missing from $INITRAMFS" >&2
exit 1
fi
done
if grep -qE '(^|/)(lb-agent|haproxy|nginx)$' <<< "$archive_list"; then
echo "ERROR: LB data-plane binaries found in the SQS store image" >&2
exit 1
fi
# The worker holds the store client key; it must never carry the store server.
if [ "$PROFILE" = worker ] && grep -qE '(^|/)nats-server$' <<< "$archive_list"; then
echo "ERROR: nats-server found in the SQS worker image" >&2
exit 1
fi
echo "$tag content gate: PASS"
echo ""
echo "$tag staged artifacts:"
ls -lh "$VMLINUZ" "$INITRAMFS"
echo ""
# --- Artifact size gate (50 MiB total) ---
ARTIFACT_MAX_MiB="${MICROVM_ARTIFACT_MAX_MiB:-50}"
total_bytes=$(( $(stat -c %s "$VMLINUZ") + $(stat -c %s "$INITRAMFS") ))
max_bytes=$(( ARTIFACT_MAX_MiB * 1024 * 1024 ))
total_mib=$(( (total_bytes + 1048575) / 1048576 ))
if [ "$total_bytes" -gt "$max_bytes" ]; then
echo "ERROR: artifact size ${total_mib} MiB exceeds ${ARTIFACT_MAX_MiB} MiB gate" >&2
echo " Set MICROVM_ARTIFACT_MAX_MiB=N to override during development." >&2
exit 1
fi
echo "$tag artifact size gate: PASS (${total_mib} MiB / ${ARTIFACT_MAX_MiB} MiB)"
# --- Guest-RAM initramfs gate (uncompressed) ---
# The kernel unpacks the whole initramfs into RAM before init, so the
# UNCOMPRESSED size — not the gzip artifact — sets the guest memory floor.
UNCOMPRESSED_MAX_MiB="${MICROVM_INITRAMFS_UNCOMPRESSED_MAX_MiB:-96}"
uncompressed_bytes=$(zcat "$INITRAMFS" | wc -c)
uncompressed_mib=$(( (uncompressed_bytes + 1048575) / 1048576 ))
umax_bytes=$(( UNCOMPRESSED_MAX_MiB * 1024 * 1024 ))
if [ "$uncompressed_bytes" -gt "$umax_bytes" ]; then
echo "ERROR: uncompressed initramfs ${uncompressed_mib} MiB exceeds ${UNCOMPRESSED_MAX_MiB} MiB guest-RAM gate" >&2
echo " The kernel unpacks this into guest RAM before init; a larger" >&2
echo " footprint panics before userspace. Slim the initramfs, or raise the" >&2
echo " instance memory AND MICROVM_INITRAMFS_UNCOMPRESSED_MAX_MiB together." >&2
exit 1
fi
echo "$tag initramfs guest-RAM gate: PASS (${uncompressed_mib} MiB / ${UNCOMPRESSED_MAX_MiB} MiB uncompressed)"
# --- Publish ---
# Two renames are not atomic as a pair, so the old initramfs goes first and the
# new one lands last. Stopped in between, the initramfs is absent: no mixed pair
# exists and the grouped make target is missing, so make rebuilds.
mkdir -p "$OUT_DIR"
rm -f "$OUT_DIR/initramfs.cpio.gz"
mv -f "$VMLINUZ" "$OUT_DIR/vmlinuz"
mv -f "$INITRAMFS" "$OUT_DIR/initramfs.cpio.gz"
echo "$tag published: $OUT_DIR/vmlinuz $OUT_DIR/initramfs.cpio.gz"
@@ -0,0 +1,196 @@
#!/bin/sh
# sqs-store-microvm-publish_test.sh — a staged SQS store microVM pair reaches
# the output directory only when every gate passes, and a failed build cannot
# leave a pair that make then takes as up to date.
#
# Artifacts are small fake archives listing the gated paths; no image is built.
set -eu
LIB_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
PUBLISH="$LIB_DIR/sqs-store-microvm-publish.sh"
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
fails=0
pass() { echo " ok - $1"; }
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
echo "-- lib/sqs-store-microvm-publish.sh"
for tool in cpio gzip make; do
if ! command -v "$tool" >/dev/null 2>&1; then
echo " FAIL - required tool not found: $tool"
exit 1
fi
done
# stage <dir> [omit-path] — a fake staged pair; omit-path is left out of the archive.
stage() {
dir=$1 omit=${2:-}
root="$TMP/root.$$.$(date +%s%N)"
for p in init etc/inittab usr/local/bin/nats-server sbin/mkfs.ext4 sbin/e2fsck \
usr/local/lib/sqs-store/sqs-store-lib.sh usr/local/lib/sqs-store/sqs-store-prepare \
usr/local/lib/sqs-store/sqs-store-run usr/local/lib/sqs-store/sqs-store-stop \
dev/console dev/null lib/modules/6.12-virt/kernel/fs/ext4/ext4.ko \
lib/modules/6.12-virt/kernel/drivers/acpi/tiny-power-button.ko; do
[ "$p" = "$omit" ] && continue
mkdir -p "$root/$(dirname "$p")"
echo "$p" > "$root/$p"
done
# pad=N appends N files after the gated ones, as a real image's listing has.
if [ "${pad:-0}" -gt 0 ]; then
mkdir -p "$root/zz"
(cd "$root/zz" && seq "$pad" | xargs touch)
fi
mkdir -p "$dir"
(cd "$root" && find . -mindepth 1 | sed 's|^\./||' | sort | cpio --quiet -o -H newc) | gzip > "$dir/initramfs.cpio.gz"
echo kernel > "$dir/vmlinuz"
rm -rf "$root"
}
# fingerprint <dir> — content and mtime of both outputs, or "absent".
fingerprint() {
for f in "$1/vmlinuz" "$1/initramfs.cpio.gz"; do
if [ -e "$f" ]; then
printf '%s %s ' "$(cksum < "$f")" "$(stat -c %Y "$f")"
else
printf 'absent '
fi
done
}
# --- A valid pair is published and leaves the stage ---
stage "$TMP/ok/stage"
if bash "$PUBLISH" "$TMP/ok/stage" "$TMP/ok/out" >/dev/null 2>&1 \
&& [ -s "$TMP/ok/out/vmlinuz" ] && [ -s "$TMP/ok/out/initramfs.cpio.gz" ] \
&& [ ! -e "$TMP/ok/stage/vmlinuz" ] && [ ! -e "$TMP/ok/stage/initramfs.cpio.gz" ]; then
pass "valid pair is published"
else
fail "valid pair is published"
fi
# --- A large listing with the modules early still passes the module gate ---
pad=20000 stage "$TMP/large/stage"
if bash "$PUBLISH" "$TMP/large/stage" "$TMP/large/out" >/dev/null 2>&1; then
pass "large archive with early modules is published"
else
fail "large archive with early modules is published"
fi
# --- Each failing gate keeps the previous valid pair exactly ---
for case in nats-server store-stop ext4 size; do
d="$TMP/keep-$case"
stage "$d/prev"
bash "$PUBLISH" "$d/prev" "$d/out" >/dev/null 2>&1
before=$(fingerprint "$d/out")
max=50
case $case in
nats-server) stage "$d/stage" usr/local/bin/nats-server ;;
store-stop) stage "$d/stage" usr/local/lib/sqs-store/sqs-store-stop ;;
ext4) stage "$d/stage" lib/modules/6.12-virt/kernel/fs/ext4/ext4.ko ;;
size) stage "$d/stage"; max=0 ;;
esac
if MICROVM_ARTIFACT_MAX_MiB=$max bash "$PUBLISH" "$d/stage" "$d/out" >/dev/null 2>&1; then
fail "$case gate failure is refused"
elif [ "$(fingerprint "$d/out")" = "$before" ]; then
pass "$case gate failure keeps the previous pair"
else
fail "$case gate failure keeps the previous pair"
fi
done
# --- With no previous pair, a failed gate publishes nothing ---
stage "$TMP/none/stage" usr/local/bin/nats-server
if bash "$PUBLISH" "$TMP/none/stage" "$TMP/none/out" >/dev/null 2>&1; then
fail "failed gate with no previous pair is refused"
elif [ "$(fingerprint "$TMP/none/out")" = "absent absent " ]; then
pass "failed gate with no previous pair publishes nothing"
else
fail "failed gate with no previous pair publishes nothing"
fi
# --- make cannot take a failed rebuild's pair as up to date ---
# The same grouped-target shape as the Makefile; the builder stages a pair
# missing nats-server once BAD exists, the way a failed real build would.
m="$TMP/make"
mkdir -p "$m/out"
cat > "$m/build.sh" <<EOF
#!/bin/sh
set -eu
rm -rf "$m/stage"
. "$m/build.sh.lib"
if [ -e "$m/BAD" ]; then
stage "$m/stage" usr/local/bin/nats-server
else
stage "$m/stage"
fi
bash "$PUBLISH" "$m/stage" "$m/out"
EOF
{
echo "TMP=\"$TMP\""
sed -n '/^stage() {/,/^}/p' "$0"
} > "$m/build.sh.lib"
printf 'out/vmlinuz out/initramfs.cpio.gz &: input\n\tsh build.sh\n' > "$m/Makefile"
echo v1 > "$m/input"
if (cd "$m" && make -s out/vmlinuz >/dev/null 2>&1) && [ -s "$m/out/vmlinuz" ]; then
pass "make builds the first valid pair"
else
fail "make builds the first valid pair"
echo " $fails failure(s)"; exit 1
fi
good=$(fingerprint "$m/out")
touch "$m/BAD"
# Age the published pair so the input is strictly newer, without a future mtime.
past="@$(( $(date +%s) - 10 ))"
touch -d "$past" "$m/out/vmlinuz" "$m/out/initramfs.cpio.gz"
touch "$m/input"
good=$(fingerprint "$m/out")
if (cd "$m" && make -s out/vmlinuz >/dev/null 2>&1); then
fail "make reports the failed rebuild"
else
pass "make reports the failed rebuild"
fi
if (cd "$m" && make -q out/vmlinuz out/initramfs.cpio.gz); then
fail "after a failed rebuild make still sees the pair as stale"
else
pass "after a failed rebuild make still sees the pair as stale"
fi
if [ "$(fingerprint "$m/out")" = "$good" ]; then
pass "failed rebuild leaves the previous valid pair"
else
fail "failed rebuild leaves the previous valid pair"
fi
# --- A publish stopped after the kernel lands leaves no mixed pair ---
# A stub mv lets the first rename through and fails the second: the point at
# which an interrupt would leave a new kernel beside an old initramfs.
i="$TMP/interrupt"
mkdir -p "$i/bin"
real_mv=$(command -v mv)
{
echo '#!/bin/sh'
echo "if [ -e '$i/moved' ]; then exit 1; fi"
echo ": > '$i/moved'"
echo "exec '$real_mv' \"\$@\""
} > "$i/bin/mv"
chmod +x "$i/bin/mv"
stage "$i/prev"
bash "$PUBLISH" "$i/prev" "$i/out" >/dev/null 2>&1
stage "$i/stage"
echo new-kernel > "$i/stage/vmlinuz"
if PATH="$i/bin:$PATH" bash "$PUBLISH" "$i/stage" "$i/out" >/dev/null 2>&1; then
fail "interrupted publish is reported"
elif [ "$(cat "$i/out/vmlinuz")" = new-kernel ] && [ ! -e "$i/out/initramfs.cpio.gz" ]; then
pass "interrupted publish leaves no initramfs beside the new kernel"
else
fail "interrupted publish leaves no initramfs beside the new kernel"
fi
printf 'out/vmlinuz out/initramfs.cpio.gz &: input\n\ttrue\n' > "$i/Makefile"
touch -d "@$(( $(date +%s) - 10 ))" "$i/input"
if (cd "$i" && make -q out/vmlinuz out/initramfs.cpio.gz); then
fail "make does not accept the interrupted publish"
else
pass "make does not accept the interrupted publish"
fi
[ "$fails" -eq 0 ] || { echo " $fails failure(s)"; exit 1; }
+163
View File
@@ -0,0 +1,163 @@
#!/bin/sh
# sqs-worker-guest_test.sh — the SQS worker guest's readiness watcher is
# supervised by init, never leaves a store-ready marker behind a dead watcher,
# and shutdown unmounts the worker disk only once the watcher is gone.
#
# Drives build/microvm/sqs-worker/sqs-worker-lib.sh with test-owned processes
# and a stub umount. Every signal goes only to this test's own children.
# The case settings below are read by the sourced library, not by this file.
# shellcheck disable=SC2034
set -eu
ROOT=$(CDPATH='' cd -- "$(dirname -- "$0")/../.." && pwd)
LIB=${SQSW_GUEST_LIB:-"$ROOT/build/microvm/sqs-worker/sqs-worker-lib.sh"}
INITTAB="$ROOT/build/microvm/sqs-worker/inittab"
TMP=$(mktemp -d)
CHILDREN=""
cleanup() {
for p in $CHILDREN; do kill -KILL "$p" 2>/dev/null || true; done
rm -rf "$TMP"
}
trap cleanup EXIT
fails=0
pass() { echo " ok - $1"; }
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
check() { if eval "$2"; then pass "$1"; else fail "$1"; fi; }
echo "-- build/microvm/sqs-worker/sqs-worker-lib.sh"
mkdir -p "$TMP/bin"
cat > "$TMP/bin/umount" <<'EOF'
#!/bin/sh
echo "umount $*" >> "$T/calls.log"
EOF
# The fake watcher notes whether a marker was already present when it
# started, publishes readiness as the real one does, then stays running.
cat > "$TMP/bin/watcher" <<'EOF'
#!/bin/sh
[ ! -e "$T/run/store-ready" ] || echo stale >> "$T/saw-stale"
echo "$$" >> "$T/watchers"
echo "10.0.0.5 UPUB" > "$T/run/store-ready"
exec sleep 60
EOF
chmod +x "$TMP/bin"/*
fresh() {
T="$TMP/$1"
export T
mkdir -p "$T/run" "$T/data"
echo UPUBLICKEY > "$T/run/public-key"
for f in calls.log console mounts; do : > "$T/$f"; done
}
gone() { [ ! -r "/proc/$1/status" ] || grep -q '^State:[[:space:]]*Z' "/proc/$1/status" 2>/dev/null; }
waitfor() { i=0; while [ "$i" -lt 100 ] && ! eval "$1"; do i=$((i + 1)); sleep 0.05; done; eval "$1"; }
# start_run: one init respawn of the readiness step, in the background.
start_run() {
SQSW_RUN="$T/run" SQSW_CONSOLE="$T/console" SQSW_BIN="$TMP/bin/watcher" SQSW_RESPAWN_GAP="${GAP:-0}" SQSW_PARK_INTERVAL="${PARK:-30}" \
sh -c '. "$1"; sqsw_run' sh "$LIB" & pid=$!
CHILDREN="$CHILDREN $pid"
}
# record <pid>: write the identity sqs_run would record for that process.
# shellcheck source=../../build/microvm/sqs-worker/sqs-worker-lib.sh
record() { echo "$1 $( . "$LIB"; sqsw_starttime "$1" )" > "$T/run/ready.pid"; }
# shutdown_step [survivor-pid]: with a pid, that process is reported alive
# whatever happens to it, as a watcher stuck past SIGKILL would be.
shutdown_step() {
(
SQSW_RUN="$T/run" SQSW_DATA="$T/data" SQSW_CONSOLE="$T/console" SQSW_MOUNTS="$T/mounts"
UMOUNT="$TMP/bin/umount" SQSW_STOP_BUDGET_TENTHS=5
# shellcheck source=../../build/microvm/sqs-worker/sqs-worker-lib.sh
. "$LIB"
if [ -n "${1:-}" ]; then sqsw_same() { [ "$1" = "$survivor" ]; }; survivor=$1; fi
sqsw_shutdown
) > /dev/null 2>&1
}
check "init respawns the readiness step rather than running it once" \
'grep -qx "::respawn:/usr/local/lib/sqs-worker/sqs-worker-ready" "$INITTAB" && ! grep -q "::once:" "$INITTAB"'
# --- A watcher killed after readiness: its marker must not outlive it ---
fresh killed-after-ready
start_run; first=$pid
check "the watcher publishes store-ready" 'waitfor "[ -s \"\$T/run/store-ready\" ]"'
kill -KILL "$first"
check "SIGKILL leaves the marker behind, as a crash would" 'waitfor "gone $first" && [ -e "$T/run/store-ready" ]'
start_run; second=$pid
check "the respawned step starts a new watcher" 'waitfor "[ \"\$(wc -l < \"\$T/watchers\")\" -eq 2 ]"'
check "the respawn removed the dead watcher's marker before the new watcher ran" \
'[ ! -e "$T/saw-stale" ] && grep -q "removed a store-ready marker left by an earlier watcher" "$T/console"'
check "the respawn records the new watcher's identity" 'grep -q "^$second " "$T/run/ready.pid"'
kill -KILL "$second" 2>/dev/null || true
# --- A watcher that keeps exiting is respawned at most once per gap ---
fresh respawn-gap
date +%s > "$T/run/ready.last"
GAP=2 start_run; t0=$(date +%s)
waitfor "[ -s \"\$T/watchers\" ]" || true
t1=$(date +%s)
check "a start right after the previous one waits for the gap" '[ $((t1 - t0)) -ge 1 ]'
kill -KILL "$pid" 2>/dev/null || true
# --- Runs that must not start ---
# A run that cannot start parks rather than exiting, so init does not respawn
# it in a loop; a second is many park intervals, and it must log only once.
fresh run-stopping; : > "$T/run/stopping"; echo stale > "$T/run/store-ready"
PARK=0.05 start_run; waitfor "grep -q 'shutdown in progress' \"\$T/console\"" || true
sleep 1
check "a start during shutdown parks instead of exiting and respawning" '! gone "$pid"'
check "a parked start during shutdown starts no watcher and removes the marker" \
'[ ! -e "$T/watchers" ] && [ ! -e "$T/run/store-ready" ]'
check "a parked start during shutdown logs once" '[ "$(grep -c "shutdown in progress" "$T/console")" -eq 1 ]'
shutdown_step
check "shutdown stops the parked start" 'waitfor "gone $pid"'
fresh run-no-key; rm -f "$T/run/public-key"
PARK=0.05 start_run; waitfor "grep -q 'no worker key' \"\$T/console\"" || true
sleep 1
check "a guest without a worker key parks instead of exiting and respawning" '! gone "$pid" && [ ! -e "$T/watchers" ]'
check "a parked guest without a worker key logs once" '[ "$(grep -c "no worker key" "$T/console")" -eq 1 ]'
echo UPUBLICKEY > "$T/run/public-key"
check "a key that appears later starts the watcher" 'waitfor "[ -s \"\$T/watchers\" ]"'
kill -KILL "$pid" 2>/dev/null || true
fresh run-no-key-stop; rm -f "$T/run/public-key"
PARK=0.05 start_run; waitfor "grep -q 'no worker key' \"\$T/console\"" || true
shutdown_step
check "shutdown stops a start parked for a missing key" 'waitfor "gone $pid" && [ ! -e "$T/watchers" ]'
# --- Shutdown: unmount only once the watcher is confirmed gone ---
fresh shutdown-orderly; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
start_run; waitfor "[ -s \"\$T/run/store-ready\" ]"
check "orderly shutdown succeeds" 'shutdown_step'
check "orderly shutdown confirms the watcher exited, removes the marker, then unmounts" \
'gone "$pid" && grep -q "readiness watcher pid=$pid exited" "$T/console" && [ ! -e "$T/run/store-ready" ] &&
[ "$(grep "^umount" "$T/calls.log" | tr "\n" "|")" = "umount $T/data|umount -a -r|" ]'
check "shutdown blocks any later respawn" '[ -e "$T/run/stopping" ]'
fresh shutdown-hung; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
(trap '' TERM; exec sleep 60) & pid=$!; CHILDREN="$CHILDREN $pid"; record "$pid"
echo "10.0.0.5 UPUB" > "$T/run/store-ready"
shutdown_step
check "a watcher ignoring SIGTERM is killed after the budget, then the disk is unmounted" \
'gone "$pid" && grep -q SIGKILL "$T/console" && grep -qx "umount $T/data" "$T/calls.log"'
fresh shutdown-survivor; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
(trap '' TERM; exec sleep 60) & pid=$!; CHILDREN="$CHILDREN $pid"; record "$pid"
echo "10.0.0.5 UPUB" > "$T/run/store-ready"
check "shutdown with a surviving watcher reports failure" '! shutdown_step "$pid"'
check "a surviving watcher leaves every mount alone and is never reported stopped" \
'! grep -q "^umount" "$T/calls.log" && grep -q "STILL ALIVE" "$T/console" && grep -q "skipping umount -a -r" "$T/console" && ! grep -q "exited" "$T/console"'
check "a surviving watcher still leaves no readiness marker" '[ ! -e "$T/run/store-ready" ]'
kill -KILL "$pid" 2>/dev/null || true
fresh shutdown-reused; echo "dev $T/data ext4 rw 0 0" > "$T/mounts"
(exec sleep 60) & pid=$!; CHILDREN="$CHILDREN $pid"; echo "$pid 1" > "$T/run/ready.pid"
shutdown_step
check "a recorded PID with another start time is never signalled" '! gone "$pid" && grep -q "no running readiness watcher" "$T/console"'
kill -KILL "$pid" 2>/dev/null || true
if [ "$fails" -ne 0 ]; then
echo " $fails failure(s)"
exit 1
fi
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
# stage-sqs-store-microvm.sh <image-dir> <dist-root> [sqs-store|sqs-worker]:
# copy a built SQS microVM image into a release tree as <dist-root>/microvm-<name>.
# Without a complete image it stages nothing and says so, as the LB image does.
set -eu
src=$1 name=${3:-sqs-store}
case "$name" in
sqs-store) label="SQS store" ;;
sqs-worker) label="SQS worker" ;;
*) echo "[distro] unknown SQS microVM image '$name'" >&2; exit 2 ;;
esac
dest=$2/microvm-$name
# An image staged by an earlier build must not ship with this one. Only the
# two image files are removed; anything else in the directory is left alone.
rm -f "$dest/vmlinuz" "$dest/initramfs.cpio.gz"
if [ -s "$src/vmlinuz" ] && [ -s "$src/initramfs.cpio.gz" ]; then
echo "[distro] staging the $label microVM image into the tarball"
mkdir -p "$dest"
cp "$src/vmlinuz" "$src/initramfs.cpio.gz" "$dest/"
else
[ ! -d "$dest" ] || rmdir "$dest" 2> /dev/null || true
echo "[distro] WARNING: $label microVM image missing; the tarball will not include it"
echo "[distro] run 'make build-$name-microvm-image' first"
fi
+1 -5
View File
@@ -190,7 +190,7 @@ run sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
# the wipe and before the verification, failing the reset on a file the script
# had already deleted. update-nodes.sh carries the same wait.
if ! $DRY_RUN; then
for unit in spinifex-predastore spinifex-viperblock spinifex-sqs-store spinifex-nats; do
for unit in spinifex-predastore spinifex-viperblock spinifex-nats; do
if systemctl is-active --quiet "$unit"; then
sudo systemctl stop "$unit" || true
fi
@@ -428,10 +428,6 @@ if ! $KEEP_DATA && [ -d "$DATA_DIR/nats/jetstream" ]; then
log "removing the JetStream store"
run sudo rm -rf "$DATA_DIR/nats/jetstream"
fi
if ! $KEEP_DATA && [ -d "$DATA_DIR/sqs-store/jetstream" ]; then
log "removing the SQS store JetStream data"
run sudo rm -rf "$DATA_DIR/sqs-store/jetstream"
fi
# Directories may survive — a mountpoint cannot be removed, and neither can the
# path leading down to one. Files may not: a surviving file is state that would
+44 -20
View File
@@ -283,7 +283,6 @@ create_service_users() {
# Create per-service users with correct home directories
declare -A SERVICE_HOMES=(
[nats]="/var/lib/spinifex/nats"
[sqsstore]="/var/lib/spinifex/sqs-store"
[gw]="/var/lib/spinifex/awsgw"
[daemon]="/var/lib/spinifex/spinifex"
[storage]="/var/lib/spinifex/predastore"
@@ -292,7 +291,7 @@ create_service_users() {
[vpcd]="/var/lib/spinifex"
[ui]="/var/lib/spinifex"
)
for svc in nats sqsstore gw daemon storage northstar viperblock vpcd ui; do
for svc in nats gw daemon storage northstar viperblock vpcd ui; do
local user="spinifex-${svc}"
if ! id "$user" > /dev/null 2>&1; then
$SUDO useradd --system --no-create-home \
@@ -324,7 +323,7 @@ create_service_users() {
# Daemon consumes viperblock's NBD socket — join the producer-typed group.
$SUDO usermod -aG spinifex-viperblock spinifex-daemon
info "Service users created (spinifex-{nats,sqsstore,gw,daemon,storage,northstar,viperblock,vpcd,ui})"
info "Service users created (spinifex-{nats,gw,daemon,storage,northstar,viperblock,vpcd,ui})"
}
# --- Install scoped sudoers rules ---
@@ -701,7 +700,7 @@ table inet spinifex_filter {
# inventory and running instances to anyone who asks. 8660 is
# predastore's unauthenticated /healthz and /readyz, which name the
# peers a gate cannot reach, so it belongs here for the same reason.
ip saddr $spinifex_peers tcp dport { 4222, 4223, 4248, 4249, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
ip saddr $spinifex_peers tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
ip saddr $spinifex_peers udp dport { 5300, 6660, 7660 } accept
# Encap plane, peer-scoped: Geneve, IKE, NAT-T and ESP. Geneve is a
@@ -1543,6 +1542,47 @@ install_files() {
$SUDO cp "$EXTRACT_DIR/microvm/"* /usr/share/spinifex/microvm/
info " /usr/share/spinifex/microvm/*"
fi
install_sqs_store_image
install_sqs_worker_image
}
# The SQS images ship beside the load balancer's, not inside it: an older
# installer copies microvm/* without -r and would stop at a subdirectory.
install_sqs_store_image() { install_sqs_microvm_image sqs-store "SQS store"; }
install_sqs_worker_image() { install_sqs_microvm_image sqs-worker "SQS worker"; }
# install_sqs_microvm_image <name> <label>
install_sqs_microvm_image() {
local name=$1 label=$2 f
local src="$EXTRACT_DIR/microvm-$name"
local dest="${SPINIFEX_MICROVM_DIR:-/usr/share/spinifex/microvm}/$name"
# A previously installed image belongs to an older release, and this
# release's daemon must not boot it. Only the two image files are removed.
if [ ! -d "$src" ]; then
remove_sqs_microvm_image "$dest" "$label"
warn "This release has no $label microVM image; $label launches will be refused"
return 0
fi
for f in vmlinuz initramfs.cpio.gz; do
if [ ! -s "$src/$f" ]; then
remove_sqs_microvm_image "$dest" "$label"
fatal "The $label microVM image in this release is incomplete: $f is missing"
fi
done
# The initramfs goes last, so an interrupted install leaves no pair the
# daemon would boot: it refuses a launch while either file is missing.
$SUDO install -d "$dest"
$SUDO rm -f "$dest/initramfs.cpio.gz"
$SUDO install -m 0644 "$src/vmlinuz" "$dest/vmlinuz"
$SUDO install -m 0644 "$src/initramfs.cpio.gz" "$dest/initramfs.cpio.gz"
info " $dest/vmlinuz, $dest/initramfs.cpio.gz"
}
# Running VMs keep their already-loaded kernel; the files are read at boot.
remove_sqs_microvm_image() {
[ -e "$1/initramfs.cpio.gz" ] || [ -e "$1/vmlinuz" ] || return 0
$SUDO rm -f "$1/initramfs.cpio.gz" "$1/vmlinuz"
warn "Removed the $2 microVM image left by an earlier install from $1"
}
# --- Create directories ---
@@ -1598,12 +1638,6 @@ TMPEOF
$SUDO chown "spinifex-nats:$SPINIFEX_GROUP" /etc/spinifex/nats
$SUDO chmod 0750 /etc/spinifex/nats
# The SQS store holds its own token: 0700 keeps every other spinifex-*
# service in the shared group out.
$SUDO mkdir -p /etc/spinifex/sqs-store
$SUDO chown "spinifex-sqsstore:$SPINIFEX_GROUP" /etc/spinifex/sqs-store
$SUDO chmod 0700 /etc/spinifex/sqs-store
$SUDO mkdir -p /etc/spinifex/predastore
$SUDO chown "spinifex-storage:$SPINIFEX_GROUP" /etc/spinifex/predastore
$SUDO chmod 0750 /etc/spinifex/predastore
@@ -1631,10 +1665,6 @@ TMPEOF
$SUDO chown "spinifex-nats:$SPINIFEX_GROUP" /var/lib/spinifex/nats
$SUDO chmod 0700 /var/lib/spinifex/nats
$SUDO mkdir -p /var/lib/spinifex/sqs-store
$SUDO chown "spinifex-sqsstore:$SPINIFEX_GROUP" /var/lib/spinifex/sqs-store
$SUDO chmod 0700 /var/lib/spinifex/sqs-store
$SUDO mkdir -p /var/lib/spinifex/spinifex
$SUDO chown "spinifex-daemon:$SPINIFEX_GROUP" /var/lib/spinifex/spinifex
$SUDO chmod 0700 /var/lib/spinifex/spinifex
@@ -1716,7 +1746,6 @@ fix_file_ownership() {
# Per-service data dirs — recursive chown so existing files are accessible
for entry in \
nats:spinifex-nats \
sqs-store:spinifex-sqsstore \
predastore:spinifex-storage \
northstar:spinifex-northstar \
spinifex:spinifex-daemon \
@@ -1735,11 +1764,6 @@ fix_file_ownership() {
$SUDO chown -R "spinifex-nats:$SPINIFEX_GROUP" /etc/spinifex/nats \
|| fatal "Failed to set ownership on /etc/spinifex/nats"
fi
if [ -d /etc/spinifex/sqs-store ]; then
$SUDO chown -R "spinifex-sqsstore:$SPINIFEX_GROUP" /etc/spinifex/sqs-store \
|| fatal "Failed to set ownership on /etc/spinifex/sqs-store"
$SUDO chmod 0700 /etc/spinifex/sqs-store
fi
if [ -d /etc/spinifex/predastore ]; then
$SUDO chown -R "spinifex-storage:$SPINIFEX_GROUP" /etc/spinifex/predastore \
|| fatal "Failed to set ownership on /etc/spinifex/predastore"
+244
View File
@@ -0,0 +1,244 @@
#!/bin/sh
# setup_sqs_store_image_test.sh — tests for shipping the SQS store and worker
# microVM images: staging them into a release tree and installing with setup.sh.
#
# setup.sh is sourced with INSTALL_SPINIFEX_LIB_ONLY=1 so main never runs, and
# installs into a temporary SPINIFEX_MICROVM_DIR with SUDO cleared.
set -eu
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
SETUP="$SCRIPT_DIR/setup.sh"
STAGE="$SCRIPT_DIR/lib/stage-sqs-store-microvm.sh"
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
fails=0
pass() { echo " ok - $1"; }
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
echo "-- SQS store and worker microVM image release and install"
# image <dir> <tag>: a fake built image whose bytes name its tag.
image() {
mkdir -p "$1"
printf 'kernel-%s\n' "$2" > "$1/vmlinuz"
printf 'initramfs-%s\n' "$2" > "$1/initramfs.cpio.gz"
}
# release <dir>: a release tree with the load balancer image, as distro-amd64 builds it.
release() {
rm -rf "$1"
mkdir -p "$1/microvm"
printf 'lb-kernel\n' > "$1/microvm/vmlinuz"
printf 'lb-initramfs\n' > "$1/microvm/initramfs.cpio.gz"
printf 'qboot\n' > "$1/microvm/qboot.rom"
}
# install_image <extract-dir> <microvm-dir> [path-prefix]: runs
# install_sqs_store_image and sets RC and OUT.
install_image() {
RC=0
OUT=$(PATH="${3:-}$PATH" EXTRACT_DIR="$1" SPINIFEX_MICROVM_DIR="$2" INSTALL_SPINIFEX_LIB_ONLY=1 \
bash -c '. "$1"; SUDO=""; install_sqs_store_image' _ "$SETUP" 2>&1) || RC=$?
}
same() { cmp -s "$1" "$2"; }
# --- staging ---
image "$TMP/built" v1
release "$TMP/rel"
sh "$STAGE" "$TMP/built" "$TMP/rel" > "$TMP/stage.out"
if same "$TMP/built/vmlinuz" "$TMP/rel/microvm-sqs-store/vmlinuz" \
&& same "$TMP/built/initramfs.cpio.gz" "$TMP/rel/microvm-sqs-store/initramfs.cpio.gz"; then
pass "stages the built image beside the load balancer's"
else
fail "staged image differs from the build"
fi
if [ ! -e "$TMP/rel/microvm/sqs-store" ] && [ "$(ls "$TMP/rel/microvm" | wc -l)" -eq 3 ]; then
pass "leaves the load balancer's microvm directory as it was"
else
fail "load balancer directory changed: $(ls "$TMP/rel/microvm")"
fi
release "$TMP/rel-none"
mkdir -p "$TMP/half"; printf 'k\n' > "$TMP/half/vmlinuz"
OUT=$(sh "$STAGE" "$TMP/half" "$TMP/rel-none")
if [ ! -e "$TMP/rel-none/microvm-sqs-store" ] && echo "$OUT" | grep -q 'WARNING: SQS store microVM image missing'; then
pass "stages nothing from an incomplete build, and says so"
else
fail "incomplete build staged or unannounced: $OUT"
fi
# --- an older installer meets a release that carries the store image ---
# It copies microvm/* without -r; the sibling directory must not reach it.
mkdir -p "$TMP/old-dest"
if cp "$TMP/rel/microvm/"* "$TMP/old-dest/" 2> "$TMP/old.err" && [ ! -e "$TMP/old-dest/sqs-store" ]; then
pass "an older installer's copy still succeeds and ignores the store image"
else
fail "older installer copy failed: $(cat "$TMP/old.err")"
fi
mkdir -p "$TMP/nested/microvm/sqs-store" "$TMP/nested-dest"
printf 'x\n' > "$TMP/nested/microvm/vmlinuz"
if cp "$TMP/nested/microvm/"* "$TMP/nested-dest/" 2> /dev/null; then
fail "control: a nested directory did not break the older copy, so the test proves nothing"
else
pass "control: a nested directory would have broken the older copy"
fi
# --- install ---
install_image "$TMP/rel" "$TMP/mv"
if [ "$RC" = 0 ] && same "$TMP/built/vmlinuz" "$TMP/mv/sqs-store/vmlinuz" \
&& same "$TMP/built/initramfs.cpio.gz" "$TMP/mv/sqs-store/initramfs.cpio.gz"; then
pass "installs the staged image where the daemon looks"
else
fail "install rc=$RC: $OUT"
fi
mode=$(stat -c '%a' "$TMP/mv/sqs-store/initramfs.cpio.gz" 2> /dev/null || echo none)
if [ "$mode" = 644 ]; then pass "installs the image 0644"; else fail "initramfs mode $mode"; fi
image "$TMP/built2" v2
release "$TMP/rel2"
sh "$STAGE" "$TMP/built2" "$TMP/rel2" > /dev/null
install_image "$TMP/rel2" "$TMP/mv"
if [ "$RC" = 0 ] && same "$TMP/built2/vmlinuz" "$TMP/mv/sqs-store/vmlinuz" \
&& same "$TMP/built2/initramfs.cpio.gz" "$TMP/mv/sqs-store/initramfs.cpio.gz"; then
pass "an upgrade replaces both files"
else
fail "upgrade rc=$RC: $OUT"
fi
# A release without the store image: nothing installed, and a clear warning.
install_image "$TMP/rel-none" "$TMP/mv-none"
if [ "$RC" = 0 ] && [ ! -e "$TMP/mv-none/sqs-store" ] && echo "$OUT" | grep -q 'no SQS store microVM image; SQS store launches will be refused'; then
pass "a release without the image installs nothing and warns"
else
fail "release without image: rc=$RC exists=$([ -e "$TMP/mv-none/sqs-store" ] && echo yes || echo no) $OUT"
fi
# installed <dir>: a store image from an earlier install, plus an unrelated file.
installed() {
image "$1/sqs-store" old
printf 'keep\n' > "$1/sqs-store/operator-note"
}
gone() { [ ! -e "$1/sqs-store/vmlinuz" ] && [ ! -e "$1/sqs-store/initramfs.cpio.gz" ]; }
kept() { [ "$(cat "$1/sqs-store/operator-note" 2> /dev/null)" = keep ]; }
# Upgrading to a release without the image must not leave the old one usable.
installed "$TMP/mv-up"
install_image "$TMP/rel-none" "$TMP/mv-up"
if [ "$RC" = 0 ] && gone "$TMP/mv-up" && kept "$TMP/mv-up" \
&& echo "$OUT" | grep -q 'Removed the SQS store microVM image left by an earlier install' \
&& echo "$OUT" | grep -q 'SQS store launches will be refused'; then
pass "a release without the image removes the earlier one, keeps other files, and warns"
else
fail "release without image over an installed one: rc=$RC $OUT"
fi
# An incomplete store image refuses and removes the earlier pair, because the
# binaries this release installed before it would otherwise boot the old one.
installed "$TMP/mv-bad"
cp -r "$TMP/rel2" "$TMP/rel-bad"
rm "$TMP/rel-bad/microvm-sqs-store/initramfs.cpio.gz"
install_image "$TMP/rel-bad" "$TMP/mv-bad"
if [ "$RC" != 0 ] && echo "$OUT" | grep -q 'incomplete: initramfs.cpio.gz is missing' \
&& gone "$TMP/mv-bad" && kept "$TMP/mv-bad"; then
pass "an incomplete image is refused and the earlier pair is removed"
else
fail "incomplete image: rc=$RC $OUT"
fi
# Restaging from a missing or incomplete build drops an image staged earlier.
for case in missing incomplete; do
image "$TMP/built-$case" new
release "$TMP/rel-$case"
sh "$STAGE" "$TMP/built" "$TMP/rel-$case" > /dev/null
if [ "$case" = missing ]; then rm -r "$TMP/built-$case"; else rm "$TMP/built-$case/initramfs.cpio.gz"; fi
OUT=$(sh "$STAGE" "$TMP/built-$case" "$TMP/rel-$case")
if [ ! -e "$TMP/rel-$case/microvm-sqs-store" ] && echo "$OUT" | grep -q 'tarball will not include it'; then
pass "restaging from a $case build ships no earlier image"
else
fail "restaging from a $case build left: $(ls -R "$TMP/rel-$case/microvm-sqs-store" 2> /dev/null)"
fi
done
mkdir -p "$TMP/rel-other/microvm-sqs-store"
image "$TMP/rel-other/microvm-sqs-store" old
printf 'keep\n' > "$TMP/rel-other/microvm-sqs-store/note"
sh "$STAGE" "$TMP/no-such-build" "$TMP/rel-other" > /dev/null
if [ ! -e "$TMP/rel-other/microvm-sqs-store/vmlinuz" ] && [ "$(cat "$TMP/rel-other/microvm-sqs-store/note")" = keep ]; then
pass "restaging removes only the image files"
else
fail "restaging touched more than the image files"
fi
# An install that stops at the initramfs leaves no pair the daemon would boot.
mkdir -p "$TMP/bin"
cat > "$TMP/bin/install" << 'EOF'
#!/bin/sh
for a in "$@"; do last=$a; done
case "$last" in */initramfs.cpio.gz) echo "install: stopped" >&2; exit 1 ;; esac
exec /usr/bin/install "$@"
EOF
chmod +x "$TMP/bin/install"
install_image "$TMP/rel" "$TMP/mv" "$TMP/bin:"
if [ "$RC" != 0 ] && same "$TMP/built/vmlinuz" "$TMP/mv/sqs-store/vmlinuz" && [ ! -e "$TMP/mv/sqs-store/initramfs.cpio.gz" ]; then
pass "an interrupted install leaves the initramfs missing, not mismatched"
else
fail "interrupted install: rc=$RC initramfs=$([ -e "$TMP/mv/sqs-store/initramfs.cpio.gz" ] && echo present || echo missing)"
fi
# --- SQS worker image: same staging and install path, its own directory ---
install_worker() {
RC=0
OUT=$(EXTRACT_DIR="$1" SPINIFEX_MICROVM_DIR="$2" INSTALL_SPINIFEX_LIB_ONLY=1 \
bash -c '. "$1"; SUDO=""; install_sqs_worker_image' _ "$SETUP" 2>&1) || RC=$?
}
image "$TMP/wbuilt" w1
release "$TMP/wrel"
sh "$STAGE" "$TMP/built" "$TMP/wrel" > /dev/null
sh "$STAGE" "$TMP/wbuilt" "$TMP/wrel" sqs-worker > /dev/null
if same "$TMP/wbuilt/vmlinuz" "$TMP/wrel/microvm-sqs-worker/vmlinuz" \
&& same "$TMP/wbuilt/initramfs.cpio.gz" "$TMP/wrel/microvm-sqs-worker/initramfs.cpio.gz" \
&& same "$TMP/built/vmlinuz" "$TMP/wrel/microvm-sqs-store/vmlinuz"; then
pass "the worker image stages beside the store image"
else
fail "worker staging: $(ls -R "$TMP/wrel")"
fi
install_worker "$TMP/wrel" "$TMP/wmv"
if [ "$RC" = 0 ] && same "$TMP/wbuilt/initramfs.cpio.gz" "$TMP/wmv/sqs-worker/initramfs.cpio.gz" \
&& [ ! -e "$TMP/wmv/sqs-store" ]; then
pass "the worker image installs into its own directory only"
else
fail "worker install: rc=$RC $OUT"
fi
release "$TMP/wrel-none"
OUT=$(sh "$STAGE" "$TMP/no-such-build" "$TMP/wrel-none" sqs-worker)
if [ ! -e "$TMP/wrel-none/microvm-sqs-worker" ] && echo "$OUT" | grep -q 'SQS worker microVM image missing'; then
pass "a missing worker build stages nothing and says so"
else
fail "missing worker build: $OUT"
fi
install_worker "$TMP/wrel-none" "$TMP/wmv"
if [ "$RC" = 0 ] && [ ! -e "$TMP/wmv/sqs-worker/vmlinuz" ] && [ ! -e "$TMP/wmv/sqs-worker/initramfs.cpio.gz" ] \
&& echo "$OUT" | grep -q 'no SQS worker microVM image; SQS worker launches will be refused'; then
pass "a release without the worker image removes the earlier one and warns"
else
fail "release without worker image: rc=$RC $OUT"
fi
cp -r "$TMP/wrel" "$TMP/wrel-bad"
rm "$TMP/wrel-bad/microvm-sqs-worker/vmlinuz"
install_worker "$TMP/wrel" "$TMP/wmv"
install_worker "$TMP/wrel-bad" "$TMP/wmv"
if [ "$RC" != 0 ] && echo "$OUT" | grep -q 'SQS worker microVM image in this release is incomplete: vmlinuz is missing' \
&& [ ! -e "$TMP/wmv/sqs-worker/initramfs.cpio.gz" ]; then
pass "an incomplete worker image is refused and the earlier pair is removed"
else
fail "incomplete worker image: rc=$RC $OUT"
fi
if OUT=$(sh "$STAGE" "$TMP/wbuilt" "$TMP/wrel" lb 2>&1); then
fail "an unknown image name was staged"
else
pass "an unknown image name is refused"
fi
[ "$fails" = 0 ] || { echo "FAIL: $fails"; exit 1; }
echo " setup_sqs_store_image_test ok"
+3 -3
View File
@@ -237,7 +237,7 @@ fi
log "stopping services"
run sudo systemctl stop spinifex.target 2>/dev/null || true
if ! $DRY_RUN; then
for unit in spinifex-predastore spinifex-viperblock spinifex-sqs-store spinifex-nats; do
for unit in spinifex-predastore spinifex-viperblock spinifex-nats; do
if systemctl is-active --quiet "$unit"; then
sudo systemctl stop "$unit" || true
fi
@@ -499,7 +499,7 @@ fi
# userdel warns about a missing home. Order here is cosmetic, not correctness.
# ---------------------------------------------------------------------------
log "removing service users and groups"
for svc in nats sqsstore gw daemon storage northstar viperblock vpcd ui; do
for svc in nats gw daemon storage northstar viperblock vpcd ui; do
user="spinifex-${svc}"
if id -u "$user" >/dev/null 2>&1; then
run sudo userdel "$user" 2>/dev/null || log " NOTE: could not remove user $user"
@@ -635,7 +635,7 @@ if ! $DRY_RUN; then
for u in "${RESTART_UNITS[@]}"; do
systemctl is-failed --quiet "$u" 2>/dev/null && LEFT+=("${u%.service}.service is failed")
done
for svc in nats sqsstore gw daemon storage northstar viperblock vpcd ui; do
for svc in nats gw daemon storage northstar viperblock vpcd ui; do
id -u "spinifex-${svc}" >/dev/null 2>&1 && LEFT+=("user spinifex-${svc}")
done
if command -v nft >/dev/null 2>&1 && sudo nft list table inet spinifex_filter >/dev/null 2>&1; then
@@ -0,0 +1,676 @@
package workerboot
import (
"bytes"
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/pem"
"errors"
"fmt"
"io"
"maps"
"math/big"
"net/http"
"net/http/httptest"
"os"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
"github.com/mulgadc/bluebottle/pkg/iampolicy"
"github.com/mulgadc/spinifex/internal/gwsign"
"github.com/mulgadc/spinifex/internal/sqsworkergw"
testutil "github.com/mulgadc/spinifex/internal/testkit"
"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
sqsworkerapi "github.com/mulgadc/spinifex/spinifex/domains/sqs/awsapi/worker"
"github.com/mulgadc/spinifex/spinifex/domains/sqs/firststore"
"github.com/mulgadc/spinifex/spinifex/domains/sqs/storelaunch"
"github.com/mulgadc/spinifex/spinifex/domains/sqs/workerident"
"github.com/mulgadc/spinifex/spinifex/domains/sqs/workerrole"
awsidentifiers "github.com/mulgadc/spinifex/spinifex/foundation/aws/identifiers"
"github.com/mulgadc/spinifex/spinifex/foundation/state/clustersize"
"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
"github.com/nats-io/nkeys"
)
const (
gwInstance = "i-0abc12345def67890"
gwSuccessor = "i-0fedcba9876543210"
gwVolume = "vol-0a1b2c3d4e5f60718"
)
// session is one instance-role credential: its secret and the instance ID
// STS names the session after.
type session struct{ secret, instance string }
// localGateway stands for the gateway's ingress in front of the real
// spinifex-sqs registration: it verifies each SigV4 signature against the
// issued credential, names the caller from that session alone, and grants
// only the worker role's actions. Every request body is kept for inspection.
type localGateway struct {
t *testing.T
srv *httptest.Server
reg dispatch.Registration
mu sync.Mutex
sessions map[string]session
bodies []string
// dropReply, while set, applies a request and then drops its reply.
dropReply func(action string) bool
// replied, when set, receives each reply's error code, "" on success.
replied chan string
}
func newLocalGateway(t *testing.T, deps sqsworkerapi.Deps) *localGateway {
t.Helper()
g := &localGateway{t: t, reg: sqsworkerapi.NewRegistration(deps), sessions: map[string]session{}}
g.srv = httptest.NewTLSServer(http.HandlerFunc(g.serve))
t.Cleanup(g.srv.Close)
return g
}
func (g *localGateway) issue(accessKey, secret, instance string) *gwsign.Signer {
g.mu.Lock()
defer g.mu.Unlock()
g.sessions[accessKey] = session{secret: secret, instance: instance}
return gwsign.NewStatic(accessKey, secret)
}
func (g *localGateway) caPEM() []byte {
return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: g.srv.Certificate().Raw})
}
func (g *localGateway) serve(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
g.fail(w, http.StatusBadRequest, "MalformedRequest")
return
}
g.mu.Lock()
g.bodies = append(g.bodies, string(body))
g.mu.Unlock()
sess, service, region, ok := g.verify(r, body)
if !ok {
g.fail(w, http.StatusForbidden, "SignatureDoesNotMatch")
return
}
if service != sqsworkerapi.ServiceName {
g.fail(w, http.StatusBadRequest, "InvalidAction")
return
}
r.Body = io.NopCloser(bytes.NewReader(body))
action := ""
if q, err := parseForm(body); err == nil {
action = q
}
rec := httptest.NewRecorder()
err = g.reg.Dispatch(rec, dispatch.Invocation{
Request: r,
AccountID: awsidentifiers.GlobalAccountID,
Service: service,
Region: region,
Authorize: func(svc, act string, _ []string, _ iampolicy.ConditionKeys) error {
if svc != sqsworkerapi.ServiceName || !slices.Contains(workerrole.Actions(), act) {
return &dispatch.PolicyDenialError{}
}
return nil
},
Actor: func() (dispatch.Actor, error) {
return dispatch.Actor{
AccountID: awsidentifiers.GlobalAccountID,
PrincipalType: "assumed-role",
PrincipalARN: "arn:aws:iam::" + awsidentifiers.GlobalAccountID + ":role/" + workerrole.RoleName,
SessionName: sess.instance,
}, nil
},
})
if err != nil {
code := err.Error()
status := http.StatusBadRequest
switch code {
case "AccessDenied":
status = http.StatusForbidden
case "ServiceUnavailable":
status = http.StatusServiceUnavailable
}
g.fail(w, status, code)
return
}
if g.dropReply != nil && g.dropReply(action) {
hj, ok := w.(http.Hijacker)
if !ok {
g.t.Error("cannot drop a reply")
return
}
conn, _, err := hj.Hijack()
if err == nil {
_ = conn.Close()
}
return
}
g.reply("")
maps.Copy(w.Header(), rec.Header())
w.WriteHeader(rec.Code)
_, _ = w.Write(rec.Body.Bytes())
}
func parseForm(body []byte) (string, error) {
r, err := http.NewRequest(http.MethodPost, "/", bytes.NewReader(body))
if err != nil {
return "", err
}
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
if err := r.ParseForm(); err != nil {
return "", err
}
return r.PostForm.Get("Action"), nil
}
// verify recomputes the signature with the issued secret. It returns the
// session, and the service and Region the credential scope names.
func (g *localGateway) verify(r *http.Request, body []byte) (session, string, string, bool) {
authz := r.Header.Get("Authorization")
const prefix = "AWS4-HMAC-SHA256 Credential="
if !strings.HasPrefix(authz, prefix) {
return session{}, "", "", false
}
scope, _, _ := strings.Cut(strings.TrimPrefix(authz, prefix), ",")
parts := strings.Split(scope, "/")
if len(parts) != 5 || parts[4] != "aws4_request" {
return session{}, "", "", false
}
g.mu.Lock()
sess, ok := g.sessions[parts[0]]
g.mu.Unlock()
if !ok {
return session{}, "", "", false
}
sum := sha256.Sum256(body)
hash := hex.EncodeToString(sum[:])
if r.Header.Get("X-Amz-Content-Sha256") != hash {
return session{}, "", "", false
}
at, err := time.Parse("20060102T150405Z", r.Header.Get("X-Amz-Date"))
if err != nil {
return session{}, "", "", false
}
check, err := http.NewRequest(r.Method, "https://"+r.Host+r.URL.RequestURI(), bytes.NewReader(body))
if err != nil {
return session{}, "", "", false
}
check.Header.Set("Content-Type", r.Header.Get("Content-Type"))
check.Header.Set("X-Amz-Content-Sha256", hash)
creds := aws.Credentials{AccessKeyID: parts[0], SecretAccessKey: sess.secret}
if err := v4.NewSigner().SignHTTP(context.Background(), creds, check, hash, parts[3], parts[2], at); err != nil {
return session{}, "", "", false
}
return sess, parts[3], parts[2], check.Header.Get("Authorization") == authz
}
func (g *localGateway) fail(w http.ResponseWriter, status int, code string) {
g.reply(code)
w.Header().Set("Content-Type", "text/xml")
w.WriteHeader(status)
_, _ = fmt.Fprintf(w, "<ErrorResponse><Error><Code>%s</Code></Error></ErrorResponse>", code)
}
func (g *localGateway) reply(code string) {
if g.replied != nil {
g.replied <- code
}
}
func (g *localGateway) sentBodies() []string {
g.mu.Lock()
defer g.mu.Unlock()
return slices.Clone(g.bodies)
}
// gwCluster is the domain's real worker-identity and first-store records.
type gwCluster struct {
ids workerident.Store
fence firststore.Store
stores storelaunch.Records
gw *localGateway
}
func newGWCluster(t *testing.T) gwCluster {
t.Helper()
clustersize.DeclareForTest(t, 1)
_, _, js := testutil.StartTestJetStream(t)
ctx := context.Background()
ids, err := workerident.Open(ctx, js)
if err != nil {
t.Fatal(err)
}
fence, err := firststore.Open(ctx, js)
if err != nil {
t.Fatal(err)
}
stores, err := storelaunch.Open(ctx, js)
if err != nil {
t.Fatal(err)
}
return gwCluster{ids: ids, fence: fence, stores: stores, gw: newLocalGateway(t, sqsworkerapi.JetStreamDeps(js))}
}
func (c gwCluster) launch(t *testing.T, claim string, supersedes uint64, inst string) workerident.Intent {
t.Helper()
ctx := context.Background()
in, err := c.ids.BeginLaunch(ctx, testRegion, workerident.LaunchRequest{InstanceType: "t3.small",
ClaimID: claim, Supersedes: supersedes, NodeID: "node-a", DataVolumeID: gwVolume, DataVolumeEncrypted: true,
})
if err != nil {
t.Fatal(err)
}
if err := c.ids.BindInstance(ctx, in, inst); err != nil {
t.Fatal(err)
}
return in
}
// guestFor points the fake guest's boot at the local gateway, signing with
// the instance's own credential through the real client.
func (c gwCluster) guestFor(t *testing.T, g *guest, accessKey, inst string) {
t.Helper()
signer := c.gw.issue(accessKey, "secret-"+accessKey, inst)
g.fw("data-serial", testSerial)
g.fw("region", testRegion)
g.fw("gateway-url", c.gw.srv.URL)
g.cfg.Wait = 10 * time.Millisecond
g.cfg.NewClaimer = func(region, url string) (workerkey.Claimer, error) {
return sqsworkergw.New(url, region, c.gw.caPEM(), signer)
}
}
func (c gwCluster) assertNoSeedSent(t *testing.T, seed string) {
t.Helper()
seed = strings.TrimSpace(seed)
for _, b := range c.gw.sentBodies() {
if strings.Contains(b, seed) {
t.Fatal("a request carried seed material")
}
}
}
// A bound worker boots: the gateway authorizes creation, the guest creates
// its key on the mounted disk and records only the public key; reboots get
// that same key back from the domain.
func TestGateway_FirstBootCreatesKeyAndRebootsReuseIt(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
pub, err := g.boot()
if err != nil {
t.Fatal(err)
}
seed := g.seed()
for boot := 2; boot <= 3; boot++ {
g.resetMounts()
got, err := g.boot()
if err != nil || got != pub || g.seed() != seed {
t.Fatalf("reboot %d = %q, %v", boot, got, err)
}
}
records := 0
for _, b := range c.gw.sentBodies() {
if strings.Contains(b, "Action=RecordWorkerPublicKey") {
records++
if !strings.Contains(b, "PublicKey="+pub) {
t.Fatal("a record carried a different key")
}
}
}
if records != 1 {
t.Fatalf("records = %d, want 1", records)
}
c.assertNoSeedSent(t, seed)
g.assertNoSeedLeak(nil, seed)
}
// The record's reply is lost after it landed: the boot retries, the domain
// answers with the recorded key, and the same key is used.
func TestGateway_LostRecordReplyKeepsTheKey(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
var dropped bool
c.gw.dropReply = func(action string) bool {
if action == "RecordWorkerPublicKey" && !dropped {
dropped = true
return true
}
return false
}
pub, err := g.boot()
if err != nil || !dropped {
t.Fatalf("boot = %q, %v (dropped=%v)", pub, err, dropped)
}
g.resetMounts()
if again, err := g.boot(); err != nil || again != pub {
t.Fatalf("reboot = %q, %v", again, err)
}
c.assertNoSeedSent(t, g.seed())
}
// After the key is recorded, a lost guest copy fails closed: the domain
// answers with the recorded key and no replacement is created or sent.
func TestGateway_KeyLostAfterRecordingFailsClosed(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
if _, err := g.boot(); err != nil {
t.Fatal(err)
}
if err := os.Remove(g.keyFile()); err != nil {
t.Fatal(err)
}
before := len(c.gw.sentBodies())
g.resetMounts()
if _, err := g.boot(); !errors.Is(err, workerkey.ErrKeyMissing) {
t.Fatalf("err = %v, want ErrKeyMissing", err)
}
if _, err := os.Lstat(g.keyFile()); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a replacement key was created")
}
for _, b := range c.gw.sentBodies()[before:] {
if strings.Contains(b, "RecordWorkerPublicKey") {
t.Fatal("a replacement key was offered")
}
}
}
// An instance the domain never bound, or one already replaced, is refused
// at the first call and creates no key; AccessDenied is not retried.
func TestGateway_UnboundOrReplacedInstanceIsRefused(t *testing.T) {
t.Run("unbound", func(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAOTHER", gwSuccessor)
_, err := g.boot()
api, ok := errors.AsType[*sqsworkergw.APIError](err)
if !ok || api.Code != "AccessDenied" {
t.Fatalf("err = %v, want AccessDenied", err)
}
if n := len(c.gw.sentBodies()); n != g.cfg.Attempts {
t.Fatalf("calls = %d, want the %d attempts and no more", n, g.cfg.Attempts)
}
g.assertNoKey()
})
t.Run("replaced", func(t *testing.T) {
c := newGWCluster(t)
first := c.launch(t, "claim-0001", 0, gwInstance)
ctx := context.Background()
if err := c.ids.RecordStopped(ctx, first, gwInstance); err != nil {
t.Fatal(err)
}
if err := c.fence.Fence(ctx, testRegion, first.Generation+1); err != nil {
t.Fatal(err)
}
c.launch(t, "claim-0002", first.Generation, gwSuccessor)
g := newGuest(t)
c.guestFor(t, g, "ASIAOLD", gwInstance)
if _, err := g.boot(); err == nil {
t.Fatal("the replaced instance booted")
}
g.assertNoKey()
})
}
// A bad signature is refused before any decision, and creates no key.
func TestGateway_BadSignatureCreatesNoKey(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
forged := gwsign.NewStatic("ASIAWORKER1", "not-the-secret")
g.cfg.NewClaimer = func(region, url string) (workerkey.Claimer, error) {
return sqsworkergw.New(url, region, c.gw.caPEM(), forged)
}
if _, err := g.boot(); err == nil {
t.Fatal("a forged signature booted")
}
g.assertNoKey()
}
// The client trusts only the cluster CA it was given: a gateway whose
// certificate another CA signed is never sent a request.
func TestGateway_UntrustedServerIsNotContacted(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
g.cfg.Attempts = 1
signer := c.gw.issue("ASIAWORKER1", "secret-ASIAWORKER1", gwInstance)
g.cfg.NewClaimer = func(region, url string) (workerkey.Claimer, error) {
return sqsworkergw.New(url, region, otherCA(t), signer)
}
if _, err := g.boot(); err == nil || !strings.Contains(err.Error(), "certificate") {
t.Fatalf("err = %v, want a TLS refusal", err)
}
if len(c.gw.sentBodies()) != 0 {
t.Fatal("a request reached a server outside the trusted CA")
}
g.assertNoKey()
}
// otherCA is a fresh self-signed CA unrelated to the test server's.
func otherCA(t *testing.T) []byte {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatal(err)
}
tmpl := &x509.Certificate{
SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "other"},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour),
IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign,
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
}
// The worker VM starts before its launch binds the instance, so its first
// calls are denied. It creates nothing while denied, and boots once bound.
func TestGateway_BootBeforeBindSucceedsAfterBinding(t *testing.T) {
c := newGWCluster(t)
ctx := context.Background()
intent, err := c.ids.BeginLaunch(ctx, testRegion, workerident.LaunchRequest{InstanceType: "t3.small",
ClaimID: "claim-0001", NodeID: "node-a", DataVolumeID: gwVolume, DataVolumeEncrypted: true,
})
if err != nil {
t.Fatal(err)
}
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
g.cfg.Attempts = 1000
c.gw.replied = make(chan string, 2000)
type result struct {
pub string
err error
}
done := make(chan result, 1)
go func() {
pub, err := g.boot()
done <- result{pub, err}
}()
if code := <-c.gw.replied; code != "AccessDenied" {
t.Fatalf("first reply = %q, want AccessDenied before binding", code)
}
g.assertNoKey()
if err := c.ids.BindInstance(ctx, intent, gwInstance); err != nil {
t.Fatal(err)
}
r := <-done
if r.err != nil || r.pub == "" {
t.Fatalf("boot after binding = %q, %v", r.pub, r.err)
}
if g.seed() == "" {
t.Fatal("booted without a key on the worker disk")
}
}
// A replaced worker is never bound again: it keeps being denied, creates
// nothing and gives up within its attempts.
func TestGateway_StaleWorkerGivesUpWithinTheBound(t *testing.T) {
c := newGWCluster(t)
ctx := context.Background()
first := c.launch(t, "claim-0001", 0, gwInstance)
if err := c.ids.RecordStopped(ctx, first, gwInstance); err != nil {
t.Fatal(err)
}
if err := c.fence.Fence(ctx, testRegion, first.Generation+1); err != nil {
t.Fatal(err)
}
c.launch(t, "claim-0002", first.Generation, gwSuccessor)
g := newGuest(t)
c.guestFor(t, g, "ASIAOLD", gwInstance)
g.cfg.Attempts = 5
_, err := g.boot()
if api, ok := errors.AsType[*sqsworkergw.APIError](err); !ok || !api.Pending() {
t.Fatalf("err = %v, want the denial", err)
}
if n := len(c.gw.sentBodies()); n != 5 {
t.Fatalf("calls = %d, want exactly the 5 attempts", n)
}
g.assertNoKey()
}
// bindStore runs the store launch record through to bound for pub at address,
// as the operator's EnsureSQSStore would.
func (c gwCluster) bindStore(t *testing.T, pub, address string) {
t.Helper()
ctx := context.Background()
const claim = "store-claim-0001"
steps := []func() error{
func() error { return c.stores.Begin(ctx, testRegion, claim, "node-a", pub, "t3.medium", 8) },
func() error { return c.stores.MarkVolumeAttempted(ctx, testRegion, claim) },
func() error { return c.stores.RecordVolume(ctx, testRegion, claim, "vol-0b1c2d3e4f5a60718") },
func() error { return c.stores.MarkNetworkAttempted(ctx, testRegion, claim) },
func() error { return c.stores.MarkDispatched(ctx, testRegion, claim) },
func() error { return c.stores.Bind(ctx, testRegion, claim, "i-0123456789abcdef0", address) },
}
for i, step := range steps {
if err := step(); err != nil {
t.Fatalf("store launch step %d: %v", i, err)
}
}
}
// readyFor points the guest's readiness at the local gateway and a store on port.
func (c gwCluster) readyFor(g *guest, accessKey, inst string, port int) ReadyConfig {
signer := c.gw.issue(accessKey, "secret-"+accessKey, inst)
return ReadyConfig{
Config: g.cfg,
NewDescriber: func(region, url string) (StoreDescriber, error) {
cl, err := sqsworkergw.New(url, region, c.gw.caPEM(), signer)
if err != nil {
return nil, err
}
return gwDescriber{cl}, nil
},
Checker: NATSChecker{Port: port, Dial: time.Second},
}
}
type gwDescriber struct{ c *sqsworkergw.Client }
func (d gwDescriber) DescribeStore(ctx context.Context) (Store, error) {
st, err := d.c.DescribeStore(ctx)
return Store(st), err
}
// The whole path with real components: the worker boots and records its key,
// waits while the operator has not launched the store, then reaches the real
// store's JetStream with its own key once the store is bound; after a reboot
// it is ready again with the same key. No request carries the seed.
func TestGateway_ReadyAfterADelayedStoreLaunch(t *testing.T) {
c := newGWCluster(t)
c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
pub, err := g.boot()
if err != nil {
t.Fatal(err)
}
port := storeServer(t, pub)
err = g.watch(c.readyFor(g, "ASIAWORKER1", gwInstance, port), func(call int, ready bool) bool {
if ready {
if call < 4 {
t.Fatalf("ready at call %d, before the store launch", call)
}
return true
}
if call == 3 {
c.bindStore(t, pub, "127.0.0.1")
}
return false
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("watch = %v", err)
}
for boot := 1; boot <= 2; boot++ {
g.resetMounts()
if again, err := g.boot(); err != nil || again != pub {
t.Fatalf("reboot %d = %q %v", boot, again, err)
}
if err := g.watch(c.readyFor(g, "ASIAWORKER1", gwInstance, port), func(_ int, ready bool) bool { return ready }); !errors.Is(err, context.Canceled) {
t.Fatalf("ready after reboot %d: %v", boot, err)
}
}
c.assertNoSeedSent(t, g.seed())
g.assertNoSeedLeak(nil, g.seed())
}
// A store bound to a key other than this worker's is never checked, and a
// replaced worker cannot read the store at all; neither is ever ready.
func TestGateway_ReadyRefusesWrongKeyAndReplacedWorker(t *testing.T) {
c := newGWCluster(t)
first := c.launch(t, "claim-0001", 0, gwInstance)
g := newGuest(t)
c.guestFor(t, g, "ASIAWORKER1", gwInstance)
if _, err := g.boot(); err != nil {
t.Fatal(err)
}
other, _ := nkeys.CreateUser()
otherPub, _ := other.PublicKey()
c.bindStore(t, otherPub, "127.0.0.1")
port := storeServer(t, otherPub)
never := func(call int, ready bool) bool {
if ready {
t.Fatal("ready against a store for another key")
}
return call == 3
}
_ = g.watch(c.readyFor(g, "ASIAWORKER1", gwInstance, port), never)
if !slices.ContainsFunc(g.logs, func(l string) bool { return strings.Contains(l, ErrStoreKeyMismatch.Error()) }) {
t.Fatalf("no mismatch reported: %v", g.logs)
}
if err := c.ids.RecordStopped(context.Background(), first, gwInstance); err != nil {
t.Fatal(err)
}
_ = g.watch(c.readyFor(g, "ASIAWORKER1", gwInstance, port), never)
if !strings.Contains(g.logs[len(g.logs)-1], "AccessDenied") {
t.Fatalf("stopped worker: %v", g.logs[len(g.logs)-1])
}
}
@@ -0,0 +1,81 @@
package workerboot
import (
"context"
"errors"
"fmt"
"net"
"strconv"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/nats-io/nkeys"
)
// StorePort is the port the store guest's NATS listens on.
const StorePort = 4222
// NATSChecker connects with the worker key and asks JetStream for its account
// info. It authenticates the client only: there is no TLS, so a pass says
// something answering at the address accepted the key.
type NATSChecker struct {
// Port overrides StorePort, for tests against a local server.
Port int
// Dial bounds each connection attempt; zero is five seconds.
Dial time.Duration
}
var _ Checker = NATSChecker{}
// StoreUnreachableError is a store that did not answer yet; it is retried.
type StoreUnreachableError struct{ Err error }
func (e *StoreUnreachableError) Error() string { return "store not answering: " + e.Err.Error() }
func (e *StoreUnreachableError) Unwrap() error { return e.Err }
func (e *StoreUnreachableError) Retryable() bool { return true }
// ErrStoreRefusedKey is a store that answered but refused the worker key.
var ErrStoreRefusedKey = errors.New("the store refused the worker key")
// Check passes once JetStream answers for the worker key.
func (c NATSChecker) Check(ctx context.Context, address string, kp nkeys.KeyPair) error {
if ip := net.ParseIP(address); ip == nil || ip.To4() == nil {
return errors.New("store address is not an IPv4 address")
}
pub, err := kp.PublicKey()
if err != nil {
return fmt.Errorf("derive worker public key: %w", err)
}
port := c.Port
if port == 0 {
port = StorePort
}
dial := c.Dial
if dial == 0 {
dial = 5 * time.Second
}
nc, err := nats.Connect("nats://"+net.JoinHostPort(address, strconv.Itoa(port)),
nats.Nkey(pub, kp.Sign),
nats.Timeout(dial),
nats.NoReconnect(),
nats.Name("spinifex-sqs-worker-readiness"),
)
if err != nil {
if errors.Is(err, nats.ErrAuthorization) || errors.Is(err, nats.ErrAuthExpired) {
return fmt.Errorf("%w: %w", ErrStoreRefusedKey, err)
}
return &StoreUnreachableError{Err: err}
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
return &StoreUnreachableError{Err: err}
}
callCtx, cancel := context.WithTimeout(ctx, dial)
defer cancel()
if _, err := js.AccountInfo(callCtx); err != nil {
return &StoreUnreachableError{Err: fmt.Errorf("jetstream account info: %w", err)}
}
return nil
}
+163
View File
@@ -0,0 +1,163 @@
package workerboot
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"time"
"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
"github.com/nats-io/nkeys"
)
// Store is the Region's store launch as the gateway reports it. Bound means a
// VM was launched for the store, not that it serves.
type Store struct {
Bound bool
Address string
PublicKey string
}
// StoreDescriber asks the gateway for this worker's store.
type StoreDescriber interface {
DescribeStore(ctx context.Context) (Store, error)
}
// NewDescriber builds the gateway client the store's address comes from.
type NewDescriber func(region, gatewayURL string) (StoreDescriber, error)
// Checker proves the store's JetStream answers this worker's key at address.
// Only an error with Retryable() true is asked again.
type Checker interface {
Check(ctx context.Context, address string, kp nkeys.KeyPair) error
}
// ReadyConfig is Config plus the readiness capabilities and cadence.
type ReadyConfig struct {
Config
NewDescriber NewDescriber
Checker Checker
// CallTimeout bounds one describe-and-check pass; zero is 30 seconds.
CallTimeout time.Duration
// Refresh is the wait after a pass before checking again; zero is 30s.
Refresh time.Duration
// Refused is the wait after a failure that cannot clear by itself, such
// as a refused or mismatched key; zero is five minutes.
Refused time.Duration
}
// ErrStoreKeyMismatch is a bound store launched for a key other than the one
// on this worker's disk; it is asked again only at the Refused cadence.
var ErrStoreKeyMismatch = errors.New("the store was launched for a different worker key")
const readyFile = "store-ready"
// Watch keeps the readiness marker true until ctx ends: it re-proves the disk,
// then checks the store with the worker's existing key every pass, writing the
// marker on a pass and removing it on any failure and on return. It never
// creates a key. NKey authenticates this client only, not the store server.
func Watch(ctx context.Context, cfg ReadyConfig) error {
marker := filepath.Join(cfg.RunDir, readyFile)
defer func() { _ = os.Remove(marker) }()
if err := os.Remove(marker); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("clear stale store readiness: %w", err)
}
c, err := readContract(cfg.FwCfgDir)
if err != nil {
return err
}
if err := mountWorkerDisk(cfg.Config, c.serial); err != nil {
return err
}
if cfg.NewDescriber == nil || cfg.Checker == nil {
return errors.New("no gateway client or store checker is configured")
}
d, err := cfg.NewDescriber(c.region, c.gatewayURL)
if err != nil {
return fmt.Errorf("build gateway client: %w", err)
}
keys := workerkey.Store{Dir: filepath.Join(cfg.DataDir, keyDirName)}
callTimeout := orDefault(cfg.CallTimeout, 30*time.Second)
refresh := orDefault(cfg.Refresh, 30*time.Second)
refused := orDefault(cfg.Refused, 5*time.Minute)
last := ""
for {
callCtx, cancel := context.WithTimeout(ctx, callTimeout)
st, err := readyOnce(callCtx, cfg, d, keys)
cancel()
delay, state := refresh, ""
if err == nil {
err = writeReady(cfg.RunDir, st)
}
switch {
case ctx.Err() != nil:
return ctx.Err()
case err == nil:
state = fmt.Sprintf("store at %s answered the worker key (client-authenticated only; the store server is not authenticated)", st.Address)
default:
_ = os.Remove(marker)
delay = cfg.Wait
if !pending(err) && !retryable(err) && !errors.Is(err, errStorePending) {
delay = refused
}
state = fmt.Sprintf("store not ready: %v", err)
}
// Only a change is logged, so a long wait does not flood the console.
if state != last {
cfg.Log(state)
last = state
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(max(delay, time.Millisecond)):
}
}
}
func orDefault(d, def time.Duration) time.Duration {
if d <= 0 {
return def
}
return d
}
var errStorePending = errors.New("the store is not yet bound")
func readyOnce(ctx context.Context, cfg ReadyConfig, d StoreDescriber, keys workerkey.Store) (Store, error) {
st, err := d.DescribeStore(ctx)
if err != nil {
return Store{}, err
}
if !st.Bound {
return Store{}, errStorePending
}
kp, err := keys.ForExistingStore(st.PublicKey)
if err != nil {
if errors.Is(err, workerkey.ErrKeyMismatch) {
return Store{}, fmt.Errorf("%w: %w", ErrStoreKeyMismatch, err)
}
return Store{}, err
}
defer kp.Wipe()
if err := cfg.Checker.Check(ctx, st.Address, kp); err != nil {
return Store{}, err
}
return st, nil
}
// writeReady publishes only the address and public key, never the seed.
func writeReady(runDir string, st Store) error {
if err := os.MkdirAll(runDir, 0o750); err != nil {
return fmt.Errorf("create %s: %w", runDir, err)
}
body := "address=" + st.Address + "\npublic=" + st.PublicKey + "\n"
if err := os.WriteFile(filepath.Join(runDir, readyFile), []byte(body), 0o600); err != nil {
return fmt.Errorf("publish store readiness: %w", err)
}
return nil
}
@@ -0,0 +1,438 @@
package workerboot
import (
"context"
"errors"
"net"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nkeys"
)
// describer is the gateway's DescribeStore, answering from a script.
type describer struct {
answers []describeAnswer
calls int
}
type describeAnswer struct {
st Store
err error
}
func (d *describer) DescribeStore(context.Context) (Store, error) {
d.calls++
a := d.answers[min(d.calls, len(d.answers))-1]
return a.st, a.err
}
type checker struct {
errs []error
calls int
seen []string
}
func (c *checker) Check(_ context.Context, address string, kp nkeys.KeyPair) error {
c.calls++
pub, _ := kp.PublicKey()
c.seen = append(c.seen, address+"|"+pub)
if len(c.errs) > 0 {
err := c.errs[0]
c.errs = c.errs[1:]
return err
}
return nil
}
// bootedGuest has run the boot step, so its disk holds a recorded key.
func bootedGuest(t *testing.T) (*guest, string) {
t.Helper()
g := newGuest(t)
g.contract()
pub, err := g.boot()
if err != nil {
t.Fatal(err)
}
return g, pub
}
func (g *guest) readyCfg(d StoreDescriber, c Checker) ReadyConfig {
return ReadyConfig{
Config: g.cfg,
NewDescriber: func(string, string) (StoreDescriber, error) { return d, nil },
Checker: c,
}
}
// hooked asks stop before each describe, after the previous pass has written
// or removed the marker, and ends the watch when stop says so.
type hooked struct {
d StoreDescriber
marker string
stop func(call int, ready bool) bool
cancel context.CancelFunc
calls int
}
func (h *hooked) DescribeStore(ctx context.Context) (Store, error) {
h.calls++
_, err := os.Stat(h.marker)
if h.stop(h.calls, err == nil) {
h.cancel()
return Store{}, context.Canceled
}
return h.d.DescribeStore(ctx)
}
// watch runs Watch with millisecond cadences until stop ends it, then checks
// the marker was removed on the way out.
func (g *guest) watch(cfg ReadyConfig, stop func(call int, ready bool) bool) error {
g.t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
inner := cfg.NewDescriber
cfg.NewDescriber = func(region, url string) (StoreDescriber, error) {
d, err := inner(region, url)
if err != nil {
return nil, err
}
return &hooked{d: d, marker: filepath.Join(cfg.RunDir, readyFile), stop: stop, cancel: cancel}, nil
}
for _, p := range []*time.Duration{&cfg.Wait, &cfg.Refresh, &cfg.Refused} {
if *p == 0 {
*p = time.Millisecond
}
}
err := Watch(ctx, cfg)
if errors.Is(ctx.Err(), context.DeadlineExceeded) {
g.t.Fatal("the watch never reached its stop condition")
}
g.assertNotReady()
return err
}
func bound(pub string) describeAnswer {
return describeAnswer{st: Store{Bound: true, Address: "10.240.0.9", PublicKey: pub}}
}
func (g *guest) marker() string {
g.t.Helper()
b, _ := os.ReadFile(filepath.Join(g.cfg.RunDir, readyFile))
return string(b)
}
// A store launched long after the worker booted, far past the old 15-minute
// bound of 180 attempts, still becomes ready; nothing is published before.
func TestWatch_StoreLaunchedLateBecomesReady(t *testing.T) {
g, pub := bootedGuest(t)
answers := make([]describeAnswer, 201)
answers[200] = bound(pub)
d := &describer{answers: answers}
c := &checker{errs: []error{&StoreUnreachableError{Err: errors.New("connection refused")}}}
published := ""
err := g.watch(g.readyCfg(d, c), func(call int, ready bool) bool {
if ready {
published = g.marker()
return true
}
if call > 202 {
t.Fatalf("not ready by call %d", call)
}
return false
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("watch = %v", err)
}
if d.calls != 202 || c.calls != 2 || c.seen[0] != "10.240.0.9|"+pub {
t.Fatalf("describe %d, check %d, seen %v", d.calls, c.calls, c.seen)
}
if published != "address=10.240.0.9\npublic="+pub+"\n" {
t.Fatalf("published %q", published)
}
notReady := 0
for _, l := range g.logs {
if strings.HasPrefix(l, "store not ready") {
notReady++
}
}
if notReady > 2 {
t.Fatalf("%d not-ready lines for two distinct states", notReady)
}
}
// After a pass, the marker goes as soon as a later check fails, and returns
// once the store answers again.
func TestWatch_LossAfterReadyRemovesTheMarker(t *testing.T) {
cases := map[string]func(pub string) (*describer, *checker){
"store stops answering": func(pub string) (*describer, *checker) {
return &describer{answers: []describeAnswer{bound(pub)}},
&checker{errs: []error{nil, &StoreUnreachableError{Err: errors.New("connection refused")}, nil}}
},
"gateway reply lost": func(pub string) (*describer, *checker) {
return &describer{answers: []describeAnswer{bound(pub), {err: transient{}}, bound(pub)}}, &checker{}
},
"store no longer bound": func(pub string) (*describer, *checker) {
return &describer{answers: []describeAnswer{bound(pub), {}, bound(pub)}}, &checker{}
},
"store now refuses the key": func(pub string) (*describer, *checker) {
return &describer{answers: []describeAnswer{bound(pub)}}, &checker{errs: []error{nil, ErrStoreRefusedKey, nil}}
},
}
for name, mk := range cases {
t.Run(name, func(t *testing.T) {
g, pub := bootedGuest(t)
d, c := mk(pub)
var seen []bool
if err := g.watch(g.readyCfg(d, c), func(call int, ready bool) bool {
seen = append(seen, ready)
return call == 4
}); !errors.Is(err, context.Canceled) {
t.Fatalf("watch = %v", err)
}
if want := []bool{false, true, false, true}; !slices.Equal(seen, want) {
t.Fatalf("marker before each pass = %v, want %v", seen, want)
}
})
}
}
// A marker left by an earlier run is cleared before the first pass.
func TestWatch_ClearsAStaleMarker(t *testing.T) {
g, pub := bootedGuest(t)
mustWrite(t, filepath.Join(g.cfg.RunDir, readyFile), "address=10.0.0.1\n")
d := &describer{answers: []describeAnswer{{}}}
_ = g.watch(g.readyCfg(d, &checker{}), func(call int, ready bool) bool {
if call == 1 && ready {
t.Fatal("a stale marker survived into the first pass")
}
return call == 2
})
_ = pub
}
// Failures that cannot clear by themselves wait the Refused cadence: in the
// window, each is tried once and never published.
func TestWatch_FinalFailuresDoNotSpin(t *testing.T) {
other, _ := nkeys.CreateUser()
otherPub, _ := other.PublicKey()
cases := map[string]struct {
d func(pub string) *describer
c *checker
checks int
}{
"store refuses the key": {func(pub string) *describer { return &describer{answers: []describeAnswer{bound(pub)}} },
&checker{errs: []error{ErrStoreRefusedKey}}, 1},
"store launched for another key": {func(string) *describer { return &describer{answers: []describeAnswer{bound(otherPub)}} },
&checker{}, 0},
"unclassified gateway refusal": {func(string) *describer {
return &describer{answers: []describeAnswer{{err: errors.New("InvalidParameterValue")}}}
}, &checker{}, 0},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
g, pub := bootedGuest(t)
g.cfg.Wait = time.Millisecond
d := tc.d(pub)
cfg := g.readyCfg(d, tc.c)
cfg.Refresh, cfg.Refused = time.Millisecond, time.Hour
ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
defer cancel()
if err := Watch(ctx, cfg); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("watch = %v", err)
}
if d.calls != 1 || tc.c.calls != tc.checks {
t.Fatalf("describe %d, check %d in the window", d.calls, tc.c.calls)
}
g.assertNotReady()
})
}
}
// A mismatched store is asked again at the Refused cadence, so a store the
// operator relaunches for this worker's key is picked up.
func TestWatch_RecoversWhenTheStoreIsRelaunchedForThisKey(t *testing.T) {
g, pub := bootedGuest(t)
other, _ := nkeys.CreateUser()
otherPub, _ := other.PublicKey()
d := &describer{answers: []describeAnswer{bound(otherPub), bound(pub)}}
c := &checker{}
if err := g.watch(g.readyCfg(d, c), func(_ int, ready bool) bool { return ready }); !errors.Is(err, context.Canceled) {
t.Fatalf("watch = %v", err)
}
if d.calls != 2 || c.calls != 1 {
t.Fatalf("describe %d, check %d", d.calls, c.calls)
}
}
// A key missing from the disk is never replaced, however often it is asked.
func TestWatch_LostKeyIsNeverReplaced(t *testing.T) {
g, pub := bootedGuest(t)
dir := filepath.Join(g.cfg.DataDir, keyDirName)
entries, _ := os.ReadDir(dir)
for _, e := range entries {
if err := os.Remove(filepath.Join(dir, e.Name())); err != nil {
t.Fatal(err)
}
}
d := &describer{answers: []describeAnswer{bound(pub)}}
c := &checker{}
_ = g.watch(g.readyCfg(d, c), func(call int, ready bool) bool {
if ready {
t.Fatal("ready without a key")
}
return call == 5
})
if c.calls != 0 {
t.Fatalf("checked %d times without a key", c.calls)
}
if after, _ := os.ReadDir(dir); len(after) != 0 {
t.Fatal("a replacement key was written")
}
}
// Readiness never runs on a disk that is not proven mounted.
func TestWatch_RefusesWithoutTheMountedWorkerDisk(t *testing.T) {
g, pub := bootedGuest(t)
g.resetMounts()
g.disk.noopMount = true
d := &describer{answers: []describeAnswer{bound(pub)}}
if err := Watch(context.Background(), g.readyCfg(d, &checker{})); err == nil || d.calls != 0 {
t.Fatalf("err = %v, describe %d", err, d.calls)
}
}
// After a restart the same key reaches readiness again with no new key.
func TestWatch_SurvivesARestart(t *testing.T) {
g, pub := bootedGuest(t)
for range 2 {
d := &describer{answers: []describeAnswer{bound(pub)}}
if err := g.watch(g.readyCfg(d, &checker{}), func(_ int, ready bool) bool { return ready }); !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
g.resetMounts()
again, err := g.boot()
if err != nil || again != pub {
t.Fatalf("reboot = %q %v", again, err)
}
}
}
func (g *guest) assertNotReady() {
g.t.Helper()
if _, err := os.Stat(filepath.Join(g.cfg.RunDir, readyFile)); !os.IsNotExist(err) {
g.t.Fatalf("readiness was published: %v", err)
}
}
func startStore(t *testing.T, pub string, port int) *server.Server {
t.Helper()
opts := &server.Options{Host: "127.0.0.1", Port: port, JetStream: true, StoreDir: t.TempDir(), NoLog: true,
NoSigs: true, Nkeys: []*server.NkeyUser{{Nkey: pub}}}
ns, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go ns.Start()
if !ns.ReadyForConnections(10 * time.Second) {
t.Fatal("store server did not start")
}
t.Cleanup(ns.Shutdown)
return ns
}
// End to end on the guest with a real store: not ready before it serves,
// ready once it does, not ready again once it stops, and the marker never
// holds the seed.
func TestWatch_RealStoreStartsLateThenStops(t *testing.T) {
g, pub := bootedGuest(t)
seedFile, _ := os.ReadDir(filepath.Join(g.cfg.DataDir, keyDirName))
key, _ := os.ReadFile(filepath.Join(g.cfg.DataDir, keyDirName, seedFile[0].Name()))
l, _ := net.Listen("tcp", "127.0.0.1:0")
port := l.Addr().(*net.TCPAddr).Port
_ = l.Close()
d := &describer{answers: []describeAnswer{{st: Store{Bound: true, Address: "127.0.0.1", PublicKey: pub}}}}
cfg := g.readyCfg(d, NATSChecker{Port: port, Dial: time.Second})
var ns *server.Server
phase := 0
err := g.watch(cfg, func(call int, ready bool) bool {
switch {
case phase == 0 && call == 3:
if ready {
t.Fatal("ready before the store served")
}
ns, phase = startStore(t, pub, port), 1
case phase == 1 && ready:
if strings.Contains(g.marker(), strings.TrimSpace(string(key))) {
t.Fatal("the seed was published")
}
ns.Shutdown()
ns.WaitForShutdown()
phase = 2
case phase == 2:
if ready {
t.Fatal("still ready after the store stopped")
}
return true
}
return false
})
if !errors.Is(err, context.Canceled) || phase != 2 {
t.Fatalf("watch = %v, phase %d", err, phase)
}
}
// storeServer is a real NATS server with JetStream that admits only pub.
func storeServer(t *testing.T, pub string) int {
t.Helper()
opts := &server.Options{
Host: "127.0.0.1", Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true,
Nkeys: []*server.NkeyUser{{Nkey: pub}},
}
ns, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go ns.Start()
if !ns.ReadyForConnections(10 * time.Second) {
t.Fatal("store server did not start")
}
t.Cleanup(ns.Shutdown)
return ns.Addr().(*net.TCPAddr).Port
}
// The real checker passes only with the admitted key; a wrong key is refused
// as final, and nothing listening is retryable.
func TestNATSChecker_AgainstARealStore(t *testing.T) {
kp, _ := nkeys.CreateUser()
pub, _ := kp.PublicKey()
port := storeServer(t, pub)
c := NATSChecker{Port: port, Dial: 2 * time.Second}
if err := c.Check(context.Background(), "127.0.0.1", kp); err != nil {
t.Fatalf("admitted key: %v", err)
}
other, _ := nkeys.CreateUser()
err := c.Check(context.Background(), "127.0.0.1", other)
if !errors.Is(err, ErrStoreRefusedKey) || retryable(err) {
t.Fatalf("wrong key: %v", err)
}
seed, _ := kp.Seed()
if strings.Contains(err.Error(), string(seed)) {
t.Fatal("the error echoes a seed")
}
l, _ := net.Listen("tcp", "127.0.0.1:0")
closed := l.Addr().(*net.TCPAddr).Port
_ = l.Close()
err = NATSChecker{Port: closed, Dial: time.Second}.Check(context.Background(), "127.0.0.1", kp)
if !retryable(err) {
t.Fatalf("nothing listening: %v is not retryable", err)
}
if err := c.Check(context.Background(), "store.example", kp); err == nil || retryable(err) {
t.Fatalf("hostname accepted: %v", err)
}
}
@@ -0,0 +1,45 @@
package workerboot
import (
"errors"
"fmt"
"os"
"os/exec"
"strings"
"syscall"
)
// SystemDisk checks with e2fsck and mounts with the mount syscall.
type SystemDisk struct {
E2fsck string
}
var _ Disk = SystemDisk{}
// Check runs a preen check: 0 is clean and 1 corrected; anything higher,
// including a disk that is not ext4, refuses. Nothing formats.
func (d SystemDisk) Check(dev string) error {
out, err := exec.Command(d.E2fsck, "-p", dev).CombinedOutput()
if err == nil {
return nil
}
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok && exitErr.ExitCode() == 1 {
return nil
}
return fmt.Errorf("e2fsck refused: %w: %s", err, strings.Join(strings.Fields(string(out)), " "))
}
// Mount mounts the disk ext4 with no device nodes, setuid or execution.
func (SystemDisk) Mount(dev, dir string) error {
return syscall.Mount(dev, dir, "ext4", syscall.MS_NODEV|syscall.MS_NOSUID|syscall.MS_NOEXEC, "")
}
// IsBlockDevice reports whether path is a block (not character) device node.
func IsBlockDevice(path string) bool {
fi, err := os.Stat(path)
if err != nil {
return false
}
m := fi.Mode()
return m&os.ModeDevice != 0 && m&os.ModeCharDevice == 0
}
@@ -0,0 +1,361 @@
// Package workerboot prepares the shared SQS worker guest: it selects exactly
// the expected data disk by serial, checks and mounts it, proves that disk is
// what is mounted, and only then loads or creates the worker's store key on it
// under the current first-store decision the gateway returns.
package workerboot
import (
"bufio"
"context"
"errors"
"fmt"
"io/fs"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
"github.com/nats-io/nkeys"
)
// Guest contract, read from fw_cfg under opt/spinifex/sqs-worker/. None of it
// is a decision: whether a store key may be created is asked of the gateway.
//
// data-serial virtio-blk serial of the worker's own ext4 data disk
// region the Region the worker signs its gateway calls for
// gateway-url https URL of the gateway on this node's mgmt bridge
const (
keyDirName = "sqs-worker-key"
maxFwCfg = 128
)
var regionPattern = regexp.MustCompile(`^[a-z]{2}(-[a-z]+)+-[0-9]+$`)
// NewClaimer builds the gateway client the first-store decision comes from.
type NewClaimer func(region, gatewayURL string) (workerkey.Claimer, error)
// Disk checks and mounts the worker data disk. The guest never formats.
type Disk interface {
Check(dev string) error
Mount(dev, dir string) error
}
// Config holds every path the boot step reads or writes, so tests can run it
// against a fake fw_cfg, sysfs and mount table.
type Config struct {
FwCfgDir string
SysBlock string
DevDir string
MountInfo string
DataDir string
RunDir string
Disk Disk
// IsBlockDevice reports whether a /dev node is a block device.
IsBlockDevice func(path string) bool
NewClaimer NewClaimer
// Attempts bounds the decision calls, Wait apart. Only a lost reply, a
// retryable refusal or a pending authorization is asked again.
Attempts int
Wait time.Duration
Log func(msg string)
}
type contract struct {
serial, region, gatewayURL string
}
// Boot runs the boot step and returns the worker's public key. The key is
// loaded or created only under the gateway's current decision; any missing,
// malformed or ambiguous input or answer refuses.
func Boot(ctx context.Context, cfg Config) (string, error) {
c, err := readContract(cfg.FwCfgDir)
if err != nil {
return "", err
}
if err := mountWorkerDisk(cfg, c.serial); err != nil {
return "", err
}
if cfg.NewClaimer == nil {
return "", errors.New("no gateway client is configured")
}
claimer, err := cfg.NewClaimer(c.region, c.gatewayURL)
if err != nil {
return "", fmt.Errorf("build gateway client: %w", err)
}
kp, err := acquire(ctx, cfg, workerkey.Store{Dir: filepath.Join(cfg.DataDir, keyDirName)}, claimer)
if err != nil {
return "", err
}
defer kp.Wipe()
pub, err := kp.PublicKey()
if err != nil {
return "", fmt.Errorf("derive worker public key: %w", err)
}
if err := writePublicKey(cfg.RunDir, pub); err != nil {
return "", err
}
cfg.Log("store key ready: public=" + pub)
return pub, nil
}
// mountWorkerDisk selects the disk with serial, checks and mounts it at
// DataDir unless it is already there, and proves from the mount table that
// exactly that disk is what is mounted.
func mountWorkerDisk(cfg Config, serial string) error {
name, err := findDisk(cfg, serial)
if err != nil {
return err
}
dev := filepath.Join(cfg.DevDir, name)
majMin, err := readTrimmed(filepath.Join(cfg.SysBlock, name, "dev"))
if err != nil {
return fmt.Errorf("read device number of %s: %w", name, err)
}
mounted, err := mountedDevice(cfg.MountInfo, cfg.DataDir)
if err != nil {
return err
}
switch mounted {
case majMin:
cfg.Log(fmt.Sprintf("%s already mounted at %s", dev, cfg.DataDir))
case "":
if err := cfg.Disk.Check(dev); err != nil {
return fmt.Errorf("check %s: %w", dev, err)
}
if err := os.MkdirAll(cfg.DataDir, 0o750); err != nil {
return fmt.Errorf("create mountpoint %s: %w", cfg.DataDir, err)
}
if err := cfg.Disk.Mount(dev, cfg.DataDir); err != nil {
return fmt.Errorf("mount %s at %s: %w", dev, cfg.DataDir, err)
}
default:
return fmt.Errorf("%s holds device %s, not the worker disk %s", cfg.DataDir, mounted, majMin)
}
// Proven from the mount table, not inferred from the mount call: an empty
// mountpoint directory must never receive a key.
mounted, err = mountedDevice(cfg.MountInfo, cfg.DataDir)
if err != nil {
return err
}
if mounted != majMin {
return fmt.Errorf("worker disk %s (%s) is not mounted at %s", dev, majMin, cfg.DataDir)
}
return nil
}
func readContract(dir string) (contract, error) {
var c contract
var err error
if c.serial, err = readFwCfg(dir, "data-serial"); err != nil {
return c, err
}
if !validSerial(c.serial) {
return c, errors.New("data-serial is not a volume serial")
}
if c.region, err = readFwCfg(dir, "region"); err != nil {
return c, err
}
if !regionPattern.MatchString(c.region) {
return c, errors.New("region is not a Region name")
}
if c.gatewayURL, err = readFwCfg(dir, "gateway-url"); err != nil {
return c, err
}
if u, err := url.Parse(c.gatewayURL); err != nil || u.Scheme != "https" || u.Host == "" {
return c, errors.New("gateway-url is not an https URL")
}
return c, nil
}
// acquire runs the decision and key step, repeating it only on an error that
// says it is safe to repeat, including a denial a not-yet-bound worker gets.
// A denial creates nothing, and each repeat re-asks the gateway.
func acquire(ctx context.Context, cfg Config, store workerkey.Store, c workerkey.Claimer) (kp nkeys.KeyPair, err error) {
attempts := max(cfg.Attempts, 1)
for i := range attempts {
kp, err = store.Acquire(ctx, c)
if err == nil || i == attempts-1 {
return kp, err
}
switch {
case pending(err):
cfg.Log(fmt.Sprintf("worker not yet authorized, asking again: %v", err))
case retryable(err):
cfg.Log(fmt.Sprintf("first-store decision not settled, retrying: %v", err))
default:
return kp, err
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(cfg.Wait):
}
}
return nil, err
}
func pending(err error) bool {
p, ok := errors.AsType[interface {
error
Pending() bool
}](err)
return ok && p.Pending()
}
func retryable(err error) bool {
r, ok := errors.AsType[interface {
error
Retryable() bool
}](err)
return ok && r.Retryable()
}
// readFwCfg returns one fw_cfg value: a single token with at most one trailing
// newline. The value is never echoed in an error.
func readFwCfg(dir, name string) (string, error) {
path := filepath.Join(dir, name, "raw")
fi, err := os.Stat(path)
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return "", fmt.Errorf("fw_cfg %s not supplied: %w", name, fs.ErrNotExist)
}
return "", fmt.Errorf("fw_cfg %s unreadable", name)
}
if fi.Size() > maxFwCfg {
return "", fmt.Errorf("fw_cfg %s is oversized", name)
}
b, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("fw_cfg %s unreadable", name)
}
v := strings.TrimSuffix(string(b), "\n")
if v == "" || strings.ContainsAny(v, " \t\r\n\v\f") {
return "", fmt.Errorf("fw_cfg %s must be exactly one value", name)
}
return v, nil
}
// A volume serial is the volume ID with its dash removed: vol + 17 hex.
func validSerial(s string) bool {
if len(s) != 20 || !strings.HasPrefix(s, "vol") {
return false
}
for _, r := range s[3:] {
if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
return false
}
}
return true
}
// findDisk returns the block device name whose serial matches exactly once.
func findDisk(cfg Config, serial string) (string, error) {
entries, err := os.ReadDir(cfg.SysBlock)
if err != nil {
return "", fmt.Errorf("list block devices: %w", err)
}
var found []string
for _, e := range entries {
s, err := readTrimmed(filepath.Join(cfg.SysBlock, e.Name(), "serial"))
if err == nil && s == serial {
found = append(found, e.Name())
}
}
if len(found) != 1 {
return "", fmt.Errorf("expected one disk with serial %s, found %d", serial, len(found))
}
if !cfg.IsBlockDevice(filepath.Join(cfg.DevDir, found[0])) {
return "", fmt.Errorf("disk %s has no block device node", found[0])
}
return found[0], nil
}
// mountedDevice returns the major:minor of the topmost ext4 mount of a whole
// filesystem at dir, or "" when nothing is mounted there.
func mountedDevice(mountInfo, dir string) (string, error) {
f, err := os.Open(mountInfo)
if err != nil {
return "", fmt.Errorf("read mount table: %w", err)
}
defer f.Close()
dir = filepath.Clean(dir)
var majMin string
found := false
sc := bufio.NewScanner(f)
for sc.Scan() {
fields := strings.Fields(sc.Text())
sep := -1
for i, fld := range fields {
if fld == "-" {
sep = i
break
}
}
if sep < 6 || sep+1 >= len(fields) {
continue
}
if unescapeMountPath(fields[4]) != dir {
continue
}
found = true
majMin = ""
if fields[3] == "/" && fields[sep+1] == "ext4" {
majMin = fields[2]
}
}
if err := sc.Err(); err != nil {
return "", fmt.Errorf("read mount table: %w", err)
}
if found && majMin == "" {
return "", fmt.Errorf("%s is mounted, but not as a whole ext4 filesystem", dir)
}
return majMin, nil
}
// unescapeMountPath decodes the octal escapes mountinfo uses for space, tab,
// newline and backslash.
func unescapeMountPath(s string) string {
r := strings.NewReplacer(`\040`, " ", `\011`, "\t", `\012`, "\n", `\134`, `\`)
return r.Replace(s)
}
func readTrimmed(path string) (string, error) {
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
return strings.TrimSpace(string(b)), nil
}
// writePublicKey publishes the public key for later reporting; it is not secret.
func writePublicKey(runDir, pub string) error {
if err := os.MkdirAll(runDir, 0o750); err != nil {
return fmt.Errorf("create %s: %w", runDir, err)
}
tmp, err := os.CreateTemp(runDir, ".public-key-*")
if err != nil {
return fmt.Errorf("write public key: %w", err)
}
defer os.Remove(tmp.Name())
if _, err := tmp.WriteString(pub + "\n"); err != nil {
tmp.Close()
return fmt.Errorf("write public key: %w", err)
}
if err := tmp.Chmod(0o644); err != nil {
tmp.Close()
return fmt.Errorf("write public key: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("write public key: %w", err)
}
if err := os.Rename(tmp.Name(), filepath.Join(runDir, "public-key")); err != nil {
return fmt.Errorf("write public key: %w", err)
}
return nil
}
@@ -0,0 +1,583 @@
package workerboot
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mulgadc/spinifex/spinifex/agents/sqs/workerkey"
"github.com/nats-io/nkeys"
)
const (
testSerial = "vol0a1b2c3d4e5f60718"
testMajMin = "253:16"
testRegion = "ap-southeast-2"
testGatewayURL = "https://10.15.8.1:9999"
)
// guest is a fake guest: fw_cfg, sysfs, /dev and a mount table the fake disk
// edits the way a real mount would.
type guest struct {
t *testing.T
root string
cfg Config
disk *fakeDisk
auth *authority
logs []string
}
// authority stands for the domain's first-store record behind the gateway:
// it authorizes creation until a key is recorded, then only that key.
type authority struct {
recorded string
claimErrs []error
// recordErrs fail RecordPublicKey; landed ones record before failing,
// as a reply lost after the write.
recordErrs []recordErr
claims int
sent []string
builds int
buildErr error
}
type recordErr struct {
err error
landed bool
}
// transient is a refusal the boot may repeat, as a lost reply.
type transient struct{}
func (transient) Error() string { return "connection reset" }
func (transient) Retryable() bool { return true }
// notBoundError is the denial a started but not yet bound worker receives.
type notBoundError struct{}
func (notBoundError) Error() string { return "AccessDenied" }
func (notBoundError) Pending() bool { return true }
func (a *authority) Claim(context.Context) (bool, string, error) {
a.claims++
if len(a.claimErrs) > 0 {
err := a.claimErrs[0]
a.claimErrs = a.claimErrs[1:]
return false, "", err
}
return a.recorded == "", a.recorded, nil
}
func (a *authority) RecordPublicKey(_ context.Context, pub string) error {
a.sent = append(a.sent, pub)
if a.recorded != "" && a.recorded != pub {
return errors.New("InvalidParameterValue")
}
if len(a.recordErrs) > 0 {
e := a.recordErrs[0]
a.recordErrs = a.recordErrs[1:]
if e.landed {
a.recorded = pub
}
return e.err
}
a.recorded = pub
return nil
}
type fakeDisk struct {
g *guest
checkErr error
mountErr error
noopMount bool
checks int
mounts int
}
func (d *fakeDisk) Check(string) error {
d.checks++
return d.checkErr
}
func (d *fakeDisk) Mount(dev, dir string) error {
d.mounts++
if d.mountErr != nil {
return d.mountErr
}
if !d.noopMount {
d.g.appendMount(testMajMin, "/", dir, "ext4", dev)
}
return nil
}
func newGuest(t *testing.T) *guest {
t.Helper()
g := &guest{t: t, root: t.TempDir(), auth: &authority{}}
g.disk = &fakeDisk{g: g}
g.cfg = Config{
FwCfgDir: filepath.Join(g.root, "fwcfg"),
SysBlock: filepath.Join(g.root, "sys", "block"),
DevDir: filepath.Join(g.root, "dev"),
MountInfo: filepath.Join(g.root, "mountinfo"),
DataDir: filepath.Join(g.root, "data"),
RunDir: filepath.Join(g.root, "run"),
Disk: g.disk,
IsBlockDevice: func(p string) bool { _, err := os.Stat(p); return err == nil },
NewClaimer: func(region, url string) (workerkey.Claimer, error) {
g.auth.builds++
if region != testRegion || url != testGatewayURL {
t.Errorf("claimer built for %q %q", region, url)
}
if g.auth.buildErr != nil {
return nil, g.auth.buildErr
}
return g.auth, nil
},
Attempts: 3,
Log: func(msg string) { g.logs = append(g.logs, msg) },
}
for _, d := range []string{g.cfg.SysBlock, g.cfg.DevDir} {
mustMkdir(t, d)
}
g.resetMounts()
// An empty mountpoint, as the initramfs ships one.
mustMkdir(t, g.cfg.DataDir)
g.addDisk("vda", testSerial, testMajMin)
return g
}
func mustMkdir(t *testing.T, d string) {
t.Helper()
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
func mustWrite(t *testing.T, path, content string) {
t.Helper()
mustMkdir(t, filepath.Dir(path))
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
func (g *guest) addDisk(name, serial, majMin string) {
mustWrite(g.t, filepath.Join(g.cfg.SysBlock, name, "serial"), serial+"\n")
mustWrite(g.t, filepath.Join(g.cfg.SysBlock, name, "dev"), majMin+"\n")
mustWrite(g.t, filepath.Join(g.cfg.DevDir, name), "")
}
func (g *guest) fw(name, value string) {
mustWrite(g.t, filepath.Join(g.cfg.FwCfgDir, name, "raw"), value)
}
// contract supplies the whole fw_cfg contract.
func (g *guest) contract() {
g.fw("data-serial", testSerial+"\n")
g.fw("region", testRegion+"\n")
g.fw("gateway-url", testGatewayURL+"\n")
}
func (g *guest) boot() (string, error) {
g.t.Helper()
return Boot(context.Background(), g.cfg)
}
// assertSentOnlyPublicKeys checks nothing but valid public keys left the guest.
func (g *guest) assertSentOnlyPublicKeys(seed string) {
g.t.Helper()
for _, v := range g.auth.sent {
if !nkeys.IsValidPublicUserKey(v) || (seed != "" && strings.Contains(v, seed)) {
g.t.Fatal("the guest sent something other than a public key")
}
}
}
// seedKey stands for a worker key created when the store was first made and
// recorded by the domain; it returns that key's public half.
func (g *guest) seedKey() string {
g.t.Helper()
kp, err := workerkey.Store{Dir: g.keyDir()}.ForNewStore()
if err != nil {
g.t.Fatalf("seed worker key: %v", err)
}
defer kp.Wipe()
pub, err := kp.PublicKey()
if err != nil {
g.t.Fatal(err)
}
g.auth.recorded = pub
return pub
}
func (g *guest) keyFile() string { return filepath.Join(g.keyDir(), "store-client.nk") }
// resetMounts is a reboot: only the initramfs root is mounted.
func (g *guest) resetMounts() {
mustWrite(g.t, g.cfg.MountInfo, "1 0 0:2 / / rw - rootfs rootfs rw\n")
}
func (g *guest) appendMount(majMin, root, dir, fstype, source string) {
f, err := os.OpenFile(g.cfg.MountInfo, os.O_APPEND|os.O_WRONLY, 0)
if err != nil {
g.t.Fatal(err)
}
defer f.Close()
if _, err := fmt.Fprintf(f, "30 1 %s %s %s rw,nodev - %s %s rw\n", majMin, root, strings.ReplaceAll(dir, " ", `\040`), fstype, source); err != nil {
g.t.Fatal(err)
}
}
func (g *guest) keyDir() string { return filepath.Join(g.cfg.DataDir, keyDirName) }
func (g *guest) seed() string {
b, err := os.ReadFile(filepath.Join(g.keyDir(), "store-client.nk"))
if err != nil {
g.t.Fatalf("read key: %v", err)
}
return string(b)
}
func (g *guest) assertNoKey() {
g.t.Helper()
if _, err := os.Lstat(g.keyDir()); !errors.Is(err, os.ErrNotExist) {
g.t.Fatalf("a key directory exists although boot refused: %v", err)
}
if _, err := os.Lstat(filepath.Join(g.cfg.RunDir, "public-key")); !errors.Is(err, os.ErrNotExist) {
g.t.Fatal("a public key was published although boot refused")
}
}
func (g *guest) assertNoSeedLeak(err error, seed string) {
g.t.Helper()
if seed == "" {
return
}
if err != nil && strings.Contains(err.Error(), seed) {
g.t.Fatal("an error exposes the seed")
}
for _, l := range g.logs {
if strings.Contains(l, seed) {
g.t.Fatal("a log line exposes the seed")
}
}
}
// otherPub is a valid store key the worker does not hold.
func otherPub(t *testing.T) string {
t.Helper()
pub, _ := userKey(t)
return pub
}
func userKey(t *testing.T) (pub, seed string) {
t.Helper()
kp, err := nkeys.CreateUser()
if err != nil {
t.Fatal(err)
}
p, err := kp.PublicKey()
if err != nil {
t.Fatal(err)
}
s, err := kp.Seed()
if err != nil {
t.Fatal(err)
}
return p, string(s)
}
func TestBoot_FirstStoreCreatesKeyThenReusesItAcrossReboots(t *testing.T) {
g := newGuest(t)
g.contract()
pub, err := g.boot()
if err != nil {
t.Fatal(err)
}
seed := g.seed()
if g.auth.recorded != pub || len(g.auth.sent) != 1 {
t.Fatalf("recorded %q after %d sends, want %s once", g.auth.recorded, len(g.auth.sent), pub)
}
for boot := 2; boot <= 3; boot++ {
g.resetMounts()
got, err := g.boot()
if err != nil || got != pub || g.seed() != seed {
t.Fatalf("boot %d = %q, %v; the worker key must not change", boot, got, err)
}
}
if len(g.auth.sent) != 1 {
t.Fatalf("a recorded key was sent again: %d sends", len(g.auth.sent))
}
if g.disk.mounts != 3 {
t.Fatalf("mounts = %d, want one per boot", g.disk.mounts)
}
published, err := os.ReadFile(filepath.Join(g.cfg.RunDir, "public-key"))
if err != nil || strings.TrimSpace(string(published)) != pub {
t.Fatalf("published key = %q, %v", published, err)
}
g.assertSentOnlyPublicKeys(seed)
g.assertNoSeedLeak(nil, seed)
}
// Once the domain holds a key, losing the guest's copy never yields a new
// one, whatever a replayed first-start fw_cfg says.
func TestBoot_KeyLostAfterRecordingIsNeverReplaced(t *testing.T) {
g := newGuest(t)
g.contract()
g.seedKey()
if err := os.Remove(g.keyFile()); err != nil {
t.Fatal(err)
}
g.fw("store-status", "new\n")
g.resetMounts()
_, err := g.boot()
if !errors.Is(err, workerkey.ErrKeyMissing) {
t.Fatalf("err = %v, want ErrKeyMissing", err)
}
if _, err := os.Lstat(g.keyFile()); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a replacement key was created for an existing store")
}
if len(g.auth.sent) != 0 || g.auth.claims != 1 {
t.Fatalf("sends=%d claims=%d; a final refusal must not be retried or answered", len(g.auth.sent), g.auth.claims)
}
if _, err := os.Lstat(filepath.Join(g.cfg.RunDir, "public-key")); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a public key was published")
}
}
func TestBoot_RecordedKeyMismatchRefuses(t *testing.T) {
g := newGuest(t)
g.contract()
g.seedKey()
seed := g.seed()
g.auth.recorded = otherPub(t)
_, err := g.boot()
if !errors.Is(err, workerkey.ErrKeyMismatch) {
t.Fatalf("err = %v, want ErrKeyMismatch", err)
}
if g.seed() != seed || len(g.auth.sent) != 0 {
t.Fatal("a mismatch changed or sent the worker key")
}
g.assertNoSeedLeak(err, seed)
}
// A reply lost after the key was recorded is retried, and the retry is
// answered with that same key: one key, recorded once.
func TestBoot_LostRecordReplyKeepsTheSameKey(t *testing.T) {
for _, landed := range []bool{true, false} {
t.Run(fmt.Sprintf("landed=%v", landed), func(t *testing.T) {
g := newGuest(t)
g.contract()
g.auth.recordErrs = []recordErr{{err: transient{}, landed: landed}}
pub, err := g.boot()
if err != nil {
t.Fatal(err)
}
if g.auth.recorded != pub || g.auth.claims != 2 {
t.Fatalf("recorded=%q claims=%d", g.auth.recorded, g.auth.claims)
}
for _, sent := range g.auth.sent {
if sent != pub {
t.Fatal("a second key was sent")
}
}
g.assertSentOnlyPublicKeys(g.seed())
})
}
}
func TestBoot_RetriesOnlyRetryableRefusals(t *testing.T) {
t.Run("transient then decided", func(t *testing.T) {
g := newGuest(t)
g.contract()
g.auth.claimErrs = []error{transient{}, transient{}}
if _, err := g.boot(); err != nil {
t.Fatal(err)
}
if g.auth.claims != 3 {
t.Fatalf("claims = %d, want 3", g.auth.claims)
}
})
t.Run("exhausted", func(t *testing.T) {
g := newGuest(t)
g.contract()
g.auth.claimErrs = []error{transient{}, transient{}, transient{}, transient{}}
if _, err := g.boot(); err == nil {
t.Fatal("boot passed without a decision")
}
if g.auth.claims != 3 {
t.Fatalf("claims = %d, want the 3 attempts", g.auth.claims)
}
g.assertNoKey()
})
t.Run("final", func(t *testing.T) {
g := newGuest(t)
g.contract()
denied := errors.New("AccessDenied")
g.auth.claimErrs = []error{denied}
if _, err := g.boot(); !errors.Is(err, denied) {
t.Fatalf("err = %v, want the denial", err)
}
if g.auth.claims != 1 {
t.Fatalf("claims = %d; a denial must not be retried", g.auth.claims)
}
g.assertNoKey()
})
t.Run("pending then authorized", func(t *testing.T) {
g := newGuest(t)
g.contract()
g.auth.claimErrs = []error{notBoundError{}, notBoundError{}}
if _, err := g.boot(); err != nil {
t.Fatal(err)
}
if g.auth.claims != 3 {
t.Fatalf("claims = %d, want 3", g.auth.claims)
}
})
t.Run("pending never authorized", func(t *testing.T) {
g := newGuest(t)
g.contract()
g.auth.claimErrs = []error{notBoundError{}, notBoundError{}, notBoundError{}, notBoundError{}}
if _, err := g.boot(); !errors.Is(err, notBoundError{}) {
t.Fatalf("err = %v, want the denial", err)
}
if g.auth.claims != 3 {
t.Fatalf("claims = %d, want the 3 attempts", g.auth.claims)
}
g.assertNoKey()
})
t.Run("cancelled while waiting", func(t *testing.T) {
g := newGuest(t)
g.contract()
g.cfg.Wait = time.Hour
g.auth.claimErrs = []error{transient{}}
ctx, cancel := context.WithCancel(context.Background())
g.cfg.Log = func(string) { cancel() }
if _, err := Boot(ctx, g.cfg); !errors.Is(err, context.Canceled) {
t.Fatalf("err = %v, want context.Canceled", err)
}
g.assertNoKey()
})
}
// The gateway is asked only once the worker disk is proven mounted, so a
// disk refusal never reaches it and never creates a key.
func TestBoot_DiskRefusalsNeverAskTheGateway(t *testing.T) {
cases := []struct {
name string
setup func(g *guest)
want string
}{
{"wrong serial", func(g *guest) { g.fw("data-serial", "vol0ffffffffffffffff") }, "found 0"},
{"no disk", func(g *guest) {
if err := os.RemoveAll(filepath.Join(g.cfg.SysBlock, "vda")); err != nil {
t.Fatal(err)
}
}, "found 0"},
{"ambiguous", func(g *guest) { g.addDisk("vdb", testSerial, "253:32") }, "found 2"},
{"no device node", func(g *guest) {
if err := os.Remove(filepath.Join(g.cfg.DevDir, "vda")); err != nil {
t.Fatal(err)
}
}, "no block device node"},
{"check fails", func(g *guest) { g.disk.checkErr = errors.New("e2fsck refused") }, "e2fsck refused"},
{"mount fails", func(g *guest) { g.disk.mountErr = errors.New("invalid argument") }, "invalid argument"},
{"empty mountpoint", func(g *guest) { g.disk.noopMount = true }, "is not mounted at"},
{"foreign device", func(g *guest) { g.appendMount("253:48", "/", g.cfg.DataDir, "ext4", "x") }, "not the worker disk"},
{"bind of a subdirectory", func(g *guest) { g.appendMount(testMajMin, "/sub", g.cfg.DataDir, "ext4", "x") }, "not as a whole ext4 filesystem"},
{"tmpfs", func(g *guest) { g.appendMount("0:40", "/", g.cfg.DataDir, "tmpfs", "x") }, "not as a whole ext4 filesystem"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
g := newGuest(t)
g.contract()
tc.setup(g)
_, err := g.boot()
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("err = %v, want %q", err, tc.want)
}
if g.auth.builds != 0 || g.auth.claims != 0 {
t.Fatal("the gateway was asked before the disk was proven")
}
g.assertNoKey()
})
}
}
func TestBoot_AlreadyMountedWorkerDiskIsReused(t *testing.T) {
g := newGuest(t)
g.contract()
g.seedKey()
g.appendMount(testMajMin, "/", g.cfg.DataDir, "ext4", "/dev/vda")
if _, err := g.boot(); err != nil {
t.Fatal(err)
}
if g.disk.checks != 0 || g.disk.mounts != 0 {
t.Fatal("a mounted worker disk was checked or mounted again")
}
}
func TestBoot_ContractRefusalsTouchNothing(t *testing.T) {
cases := []struct {
name string
setup func(g *guest)
want string
}{
{"no serial", func(g *guest) { mustRemove(t, filepath.Join(g.cfg.FwCfgDir, "data-serial")) }, "data-serial not supplied"},
{"bad serial", func(g *guest) { g.fw("data-serial", "vol-0a1b2c3d4e5f6071") }, "not a volume serial"},
{"no region", func(g *guest) { mustRemove(t, filepath.Join(g.cfg.FwCfgDir, "region")) }, "region not supplied"},
{"empty region", func(g *guest) { g.fw("region", "") }, "exactly one value"},
{"bad region", func(g *guest) { g.fw("region", "Sydney") }, "not a Region name"},
{"no gateway", func(g *guest) { mustRemove(t, filepath.Join(g.cfg.FwCfgDir, "gateway-url")) }, "gateway-url not supplied"},
{"plain http gateway", func(g *guest) { g.fw("gateway-url", "http://10.15.8.1:9999") }, "not an https URL"},
{"no claimer", func(g *guest) { g.cfg.NewClaimer = nil }, "no gateway client"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
g := newGuest(t)
g.contract()
tc.setup(g)
_, err := g.boot()
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("err = %v, want %q", err, tc.want)
}
if g.auth.claims != 0 {
t.Fatal("a refused contract asked the gateway")
}
g.assertNoKey()
})
}
}
func TestBoot_ClaimerBuildFailureCreatesNoKey(t *testing.T) {
g := newGuest(t)
g.contract()
g.auth.buildErr = errors.New("no IMDS credentials")
if _, err := g.boot(); err == nil || !strings.Contains(err.Error(), "no IMDS credentials") {
t.Fatalf("err = %v", err)
}
g.assertNoKey()
}
func mustRemove(t *testing.T, path string) {
t.Helper()
if err := os.RemoveAll(path); err != nil {
t.Fatal(err)
}
}
func TestMountedDevice_UsesTopmostMountAndUnescapes(t *testing.T) {
g := newGuest(t)
dir := filepath.Join(g.root, "data dir")
g.appendMount("253:48", "/", dir, "ext4", "/dev/vdb")
g.appendMount(testMajMin, "/", dir, "ext4", "/dev/vda")
got, err := mountedDevice(g.cfg.MountInfo, dir)
if err != nil || got != testMajMin {
t.Fatalf("mountedDevice = %q, %v; want %s", got, err, testMajMin)
}
}
+45
View File
@@ -0,0 +1,45 @@
package workerkey
import (
"context"
"errors"
"github.com/nats-io/nkeys"
)
// Claimer is the SQS domain's first-store decision for this worker identity.
type Claimer interface {
// Claim returns createKey, or the one public key the store authorizes.
Claim(ctx context.Context) (createKey bool, publicKey string, err error)
RecordPublicKey(ctx context.Context, publicKey string) error
}
// Acquire returns the worker's store key under a current domain decision. A
// new key is usable only once its public key is recorded; a recorded key is
// never replaced, so a missing or different key on disk fails closed.
func (s Store) Acquire(ctx context.Context, c Claimer) (nkeys.KeyPair, error) {
create, authorized, err := c.Claim(ctx)
if err != nil {
return nil, err
}
if !create {
return s.ForExistingStore(authorized)
}
if authorized != "" {
return nil, errors.New("first-store decision is contradictory")
}
// After a crash before recording, this loads the key already on disk.
kp, err := s.ForNewStore()
if err != nil {
return nil, err
}
pub, err := kp.PublicKey()
if err == nil {
err = c.RecordPublicKey(ctx, pub)
}
if err != nil {
kp.Wipe()
return nil, err
}
return kp, nil
}
@@ -0,0 +1,266 @@
package workerkey
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
testutil "github.com/mulgadc/spinifex/internal/testkit"
"github.com/mulgadc/spinifex/spinifex/domains/sqs/firststore"
"github.com/nats-io/nats.go/jetstream"
"github.com/nats-io/nkeys"
)
const region = "ap-southeast-2"
// domainClaimer drives the real first-store state machine for one worker
// identity. A failing record either drops the write or applies it and loses
// the reply.
type domainClaimer struct {
s firststore.Store
id firststore.Identity
failRecord bool
applied bool
}
func (c *domainClaimer) Claim(ctx context.Context) (bool, string, error) {
d, err := c.s.Claim(ctx, region, c.id)
return d.CreateKey, d.PublicKey, err
}
func (c *domainClaimer) RecordPublicKey(ctx context.Context, pub string) error {
if c.failRecord {
c.failRecord = false
if c.applied {
c.s.RecordPublicKey(ctx, region, c.id, pub)
}
return errors.New("nats: timeout")
}
return c.s.RecordPublicKey(ctx, region, c.id, pub)
}
func domain(t *testing.T) (firststore.Store, jetstream.KeyValue) {
t.Helper()
_, _, js := testutil.StartTestJetStream(t)
kv, err := js.CreateKeyValue(context.Background(), firststore.BucketConfig(1))
if err != nil {
t.Fatal(err)
}
return firststore.Store{JS: js, KV: kv}, kv
}
func worker(gen uint64, n byte) firststore.Identity {
return firststore.Identity{InstanceID: "i-0000000000000000" + string('0'+n), ClaimID: "claim-" + string('a'+n) + "-0001", Generation: gen}
}
func workerDisk(t *testing.T) Store {
return Store{Dir: filepath.Join(t.TempDir(), "sqs-worker-key")}
}
func keyPath(s Store) string { return filepath.Join(s.Dir, keyFile) }
func pubOf(t *testing.T, kp nkeys.KeyPair) string {
t.Helper()
p, err := kp.PublicKey()
if err != nil {
t.Fatal(err)
}
return p
}
func recorded(t *testing.T, s firststore.Store, id firststore.Identity) string {
t.Helper()
d, err := s.Claim(context.Background(), region, id)
if err != nil || d.CreateKey {
t.Fatalf("domain decision = %+v, %v; want a recorded key", d, err)
}
return d.PublicKey
}
func TestAcquire_CrashBeforeRecordingReusesTheKeyOnDisk(t *testing.T) {
ctx := context.Background()
fs, _ := domain(t)
disk := workerDisk(t)
c := &domainClaimer{s: fs, id: worker(1, 1), failRecord: true}
if _, err := disk.Acquire(ctx, c); err == nil {
t.Fatal("a key was returned before its public key was recorded")
}
before, err := os.ReadFile(keyPath(disk))
if err != nil {
t.Fatalf("key not synced to disk before recording: %v", err)
}
kp, err := disk.Acquire(ctx, c)
if err != nil {
t.Fatal(err)
}
defer kp.Wipe()
after, _ := os.ReadFile(keyPath(disk))
if string(before) != string(after) {
t.Fatal("the retry replaced the key")
}
if recorded(t, fs, c.id) != pubOf(t, kp) {
t.Fatal("the recorded key is not the key on disk")
}
}
func TestAcquire_LostRecordReplyIsRecoveredOnReboot(t *testing.T) {
ctx := context.Background()
fs, _ := domain(t)
disk := workerDisk(t)
c := &domainClaimer{s: fs, id: worker(1, 1), failRecord: true, applied: true}
if _, err := disk.Acquire(ctx, c); err == nil {
t.Fatal("expected the lost reply to surface")
}
kp, err := disk.Acquire(ctx, c)
if err != nil {
t.Fatal(err)
}
defer kp.Wipe()
if recorded(t, fs, c.id) != pubOf(t, kp) {
t.Fatal("reboot did not use the recorded key")
}
}
func TestAcquire_AfterRecordingKeyLossOrMismatchFailsClosed(t *testing.T) {
ctx := context.Background()
setup := func(t *testing.T, launch bool) (firststore.Store, Store, *domainClaimer, string) {
fs, _ := domain(t)
disk := workerDisk(t)
c := &domainClaimer{s: fs, id: worker(1, 1)}
kp, err := disk.Acquire(ctx, c)
if err != nil {
t.Fatal(err)
}
pub := pubOf(t, kp)
kp.Wipe()
if launch {
if err := fs.RecordLaunchIntent(ctx, region, c.id, pub); err != nil {
t.Fatal(err)
}
}
return fs, disk, c, pub
}
for _, launch := range []bool{false, true} {
name := map[bool]string{false: "key recorded", true: "launch intent"}[launch]
t.Run(name+"/key lost then reboot", func(t *testing.T) {
fs, disk, c, pub := setup(t, launch)
os.Remove(keyPath(disk))
// A relaunched worker generation gets the same answer.
for _, id := range []firststore.Identity{c.id, worker(2, 2)} {
c.id = id
if _, err := disk.Acquire(ctx, c); !errors.Is(err, ErrKeyMissing) {
t.Fatalf("acquire = %v, want ErrKeyMissing", err)
}
if _, err := os.Lstat(keyPath(disk)); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a replacement key was created")
}
}
if recorded(t, fs, c.id) != pub {
t.Fatal("the recorded key changed")
}
})
t.Run(name+"/different key on disk", func(t *testing.T) {
_, _, c, _ := setup(t, launch)
other := workerDisk(t)
kp, err := other.ForNewStore()
if err != nil {
t.Fatal(err)
}
kp.Wipe()
before, _ := os.ReadFile(keyPath(other))
if _, err := other.Acquire(ctx, c); !errors.Is(err, ErrKeyMismatch) {
t.Fatalf("acquire = %v, want ErrKeyMismatch", err)
}
after, _ := os.ReadFile(keyPath(other))
if string(before) != string(after) {
t.Fatal("the mismatched key was replaced")
}
})
}
}
func TestAcquire_RefusedClaimCreatesNoKey(t *testing.T) {
ctx := context.Background()
cases := map[string]func(t *testing.T, fs firststore.Store, kv jetstream.KeyValue) firststore.Identity{
"rival holds the claim": func(t *testing.T, fs firststore.Store, _ jetstream.KeyValue) firststore.Identity {
fs.Claim(ctx, region, worker(1, 1))
return worker(1, 2)
},
"stale replayed generation": func(t *testing.T, fs firststore.Store, _ jetstream.KeyValue) firststore.Identity {
fs.Claim(ctx, region, worker(1, 1))
fs.Claim(ctx, region, worker(2, 2))
return worker(1, 1)
},
"unreadable record": func(t *testing.T, _ firststore.Store, kv jetstream.KeyValue) firststore.Identity {
kv.Put(ctx, "region."+region, []byte("{"))
return worker(1, 1)
},
}
for name, setup := range cases {
t.Run(name, func(t *testing.T) {
fs, kv := domain(t)
id := setup(t, fs, kv)
disk := workerDisk(t)
if _, err := disk.Acquire(ctx, &domainClaimer{s: fs, id: id}); err == nil {
t.Fatal("a refused claim returned a key")
}
if _, err := os.Lstat(keyPath(disk)); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a refused claim created a key")
}
})
}
}
type contradictoryClaimer struct{ pub string }
func (c contradictoryClaimer) Claim(context.Context) (bool, string, error) { return true, c.pub, nil }
func (contradictoryClaimer) RecordPublicKey(context.Context, string) error {
return errors.New("unexpected")
}
func TestAcquire_ContradictoryDecisionCreatesNoKey(t *testing.T) {
kp, _ := nkeys.CreateUser()
disk := workerDisk(t)
if _, err := disk.Acquire(context.Background(), contradictoryClaimer{pub: pubOf(t, kp)}); err == nil {
t.Fatal("a contradictory decision returned a key")
}
if _, err := os.Lstat(keyPath(disk)); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a contradictory decision created a key")
}
}
// A replica that has applied the claim but not the key record must not lead a
// worker whose key is lost to create a replacement.
func TestAcquire_StaleClaimedReadAfterRecordingCreatesNoKey(t *testing.T) {
ctx := context.Background()
fs, kv := domain(t)
disk := workerDisk(t)
id := worker(1, 1)
fs.Claim(ctx, region, id)
claimed, err := kv.Get(ctx, "region."+region)
if err != nil {
t.Fatal(err)
}
kp, err := disk.Acquire(ctx, &domainClaimer{s: fs, id: id})
if err != nil {
t.Fatal(err)
}
kp.Wipe()
os.Remove(keyPath(disk))
_, _, replicaJS := testutil.StartTestJetStream(t)
replica, err := replicaJS.CreateKeyValue(ctx, firststore.BucketConfig(1))
if err != nil {
t.Fatal(err)
}
replica.Put(ctx, "region."+region, claimed.Value())
stale := firststore.Store{JS: replicaJS, KV: fs.KV}
if _, err := disk.Acquire(ctx, &domainClaimer{s: stale, id: id}); err == nil {
t.Fatal("a stale claimed read returned a key")
}
if _, err := os.Lstat(keyPath(disk)); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a stale claimed read created a replacement key")
}
}
+243
View File
@@ -0,0 +1,243 @@
// Package workerkey keeps the shared SQS worker's store-client NKey on the
// worker's own persistent disk. The seed is generated and held only inside the
// worker guest; the platform learns the public key alone.
package workerkey
import (
"bytes"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"syscall"
"github.com/nats-io/nkeys"
)
const (
keyFile = "store-client.nk"
tempPrefix = ".store-client-"
tempSuffix = ".tmp"
// A user seed is 58 bytes; anything much larger is not one.
maxKeyFileSize = 256
)
// Each refusal is fail-closed: the worker never replaces a key it cannot use.
var (
ErrKeyMissing = errors.New("worker store key is missing but the store already exists")
ErrKeyMismatch = errors.New("worker store key does not match the store's authorized key")
ErrUnsafe = errors.New("worker store key path is not owner-only")
ErrCorrupt = errors.New("worker store key file does not hold a user NKey seed")
)
// Store is the key directory on the worker disk. Workerkey cannot tell a
// mounted disk from an empty mountpoint, so the caller must first verify the
// expected data volume is mounted there.
type Store struct {
Dir string
}
// ForNewStore returns the worker's store-client key for a store that the
// caller has positively established does not exist yet, generating it on first
// use. An unreadable or absent store record is not that proof.
func (s Store) ForNewStore() (nkeys.KeyPair, error) {
return s.ensure("")
}
// ForExistingStore returns the worker's key only if it is already on disk and
// is the one the store authorizes; it never generates a key.
func (s Store) ForExistingStore(authorized string) (nkeys.KeyPair, error) {
if !nkeys.IsValidPublicUserKey(authorized) {
return nil, errors.New("authorized store key is not a user public NKey")
}
return s.ensure(authorized)
}
func (s Store) ensure(expected string) (nkeys.KeyPair, error) {
unlock, err := s.prepareDir()
if err != nil {
return nil, err
}
defer unlock()
kp, err := s.load()
if errors.Is(err, fs.ErrNotExist) {
if expected != "" {
return nil, ErrKeyMissing
}
kp, err = s.create()
}
if err != nil {
return nil, err
}
if expected != "" {
pub, err := kp.PublicKey()
if err != nil {
kp.Wipe()
return nil, ErrCorrupt
}
if pub != expected {
kp.Wipe()
return nil, fmt.Errorf("%w: have %s, store authorizes %s", ErrKeyMismatch, pub, expected)
}
}
// Re-synced on every return, so a key whose earlier sync failed is never
// handed out before its name and directory are durable.
if err := s.syncNames(); err != nil {
kp.Wipe()
return nil, err
}
return kp, nil
}
func (s Store) path() string { return filepath.Join(s.Dir, keyFile) }
// prepareDir creates only the key directory, inside a parent that must already
// exist, refuses one that is not owner-only, locks it and removes temporary
// files a crash left behind.
func (s Store) prepareDir() (unlock func(), err error) {
if s.Dir == "" {
return nil, errors.New("worker store key directory is not set")
}
parent, err := os.Stat(filepath.Dir(s.Dir))
if err != nil || !parent.IsDir() {
return nil, errors.New("worker key directory parent is missing")
}
if err := os.Mkdir(s.Dir, 0o700); err != nil && !errors.Is(err, fs.ErrExist) {
return nil, fmt.Errorf("create worker key directory: %w", err)
}
fi, err := os.Lstat(s.Dir)
if err != nil {
return nil, fmt.Errorf("stat worker key directory: %w", err)
}
if !fi.IsDir() || fi.Mode().Perm()&0o077 != 0 {
return nil, ErrUnsafe
}
d, err := os.Open(s.Dir)
if err != nil {
return nil, fmt.Errorf("open worker key directory: %w", err)
}
if err := syscall.Flock(int(d.Fd()), syscall.LOCK_EX); err != nil {
d.Close()
return nil, fmt.Errorf("lock worker key directory: %w", err)
}
unlock = func() { d.Close() }
stale, err := filepath.Glob(filepath.Join(s.Dir, tempPrefix+"*"+tempSuffix))
if err != nil {
unlock()
return nil, fmt.Errorf("list stale worker key files: %w", err)
}
for _, f := range stale {
if err := os.Remove(f); err != nil && !errors.Is(err, fs.ErrNotExist) {
unlock()
return nil, fmt.Errorf("remove stale worker key file: %w", err)
}
}
return unlock, nil
}
// load reads an existing key. Any file that is present but unusable refuses
// rather than being replaced.
func (s Store) load() (nkeys.KeyPair, error) {
fi, err := os.Lstat(s.path())
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() || fi.Mode().Perm()&0o077 != 0 {
return nil, ErrUnsafe
}
if fi.Size() > maxKeyFileSize {
return nil, ErrCorrupt
}
seed, err := os.ReadFile(s.path())
if err != nil {
return nil, fmt.Errorf("read worker store key: %w", err)
}
kp, err := nkeys.FromSeed(bytes.TrimSpace(seed))
if err != nil {
return nil, ErrCorrupt
}
pub, err := kp.PublicKey()
if err != nil || !nkeys.IsValidPublicUserKey(pub) {
kp.Wipe()
return nil, ErrCorrupt
}
return kp, nil
}
// create writes a new seed to a synced temporary file and links it into place
// without replacing anything; the caller syncs the directory names.
func (s Store) create() (nkeys.KeyPair, error) {
kp, err := nkeys.CreateUser()
if err != nil {
return nil, fmt.Errorf("generate worker store key: %w", err)
}
seed, err := kp.Seed()
if err != nil {
kp.Wipe()
return nil, fmt.Errorf("encode worker store key: %w", err)
}
tmp, err := os.CreateTemp(s.Dir, tempPrefix+"*"+tempSuffix)
if err != nil {
kp.Wipe()
return nil, fmt.Errorf("create worker store key file: %w", err)
}
tmpName := tmp.Name()
defer os.Remove(tmpName)
if err := writeSynced(tmp, seed); err != nil {
kp.Wipe()
return nil, err
}
if err := os.Link(tmpName, s.path()); err != nil {
kp.Wipe()
if errors.Is(err, fs.ErrExist) {
return s.load()
}
return nil, fmt.Errorf("install worker store key file: %w", err)
}
return kp, nil
}
// syncNames makes the key's directory entry and the key directory's own entry
// in its parent durable.
func (s Store) syncNames() error {
if err := syncDirFn(s.Dir); err != nil {
return err
}
return syncDirFn(filepath.Dir(s.Dir))
}
func writeSynced(f *os.File, data []byte) error {
if err := f.Chmod(0o600); err != nil {
f.Close()
return fmt.Errorf("restrict worker store key file: %w", err)
}
if _, err := f.Write(data); err != nil {
f.Close()
return fmt.Errorf("write worker store key file: %w", err)
}
if err := f.Sync(); err != nil {
f.Close()
return fmt.Errorf("sync worker store key file: %w", err)
}
if err := f.Close(); err != nil {
return fmt.Errorf("close worker store key file: %w", err)
}
return nil
}
// syncDirFn is replaced in tests to fail a directory sync deterministically.
var syncDirFn = syncDir
func syncDir(dir string) error {
d, err := os.Open(dir)
if err != nil {
return fmt.Errorf("open worker key directory: %w", err)
}
defer d.Close()
if err := d.Sync(); err != nil {
return fmt.Errorf("sync worker key directory: %w", err)
}
return nil
}
@@ -0,0 +1,423 @@
package workerkey
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"syscall"
"testing"
"time"
"github.com/nats-io/nkeys"
)
func newStore(t *testing.T) Store {
t.Helper()
return Store{Dir: filepath.Join(t.TempDir(), "keys")}
}
func publicKey(t *testing.T, kp nkeys.KeyPair) string {
t.Helper()
pub, err := kp.PublicKey()
if err != nil {
t.Fatalf("public key: %v", err)
}
return pub
}
func otherUserKey(t *testing.T) string {
t.Helper()
kp, err := nkeys.CreateUser()
if err != nil {
t.Fatalf("create user: %v", err)
}
return publicKey(t, kp)
}
func readFile(t *testing.T, path string) []byte {
t.Helper()
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
return b
}
func assertNoSeed(t *testing.T, err error, seed []byte) {
t.Helper()
if err != nil && strings.Contains(err.Error(), string(seed)) {
t.Fatalf("error exposes the seed: %v", err)
}
}
func TestEnsure_FirstBootPersistsAndRestartReusesKey(t *testing.T) {
s := newStore(t)
kp, err := s.ForNewStore()
if err != nil {
t.Fatalf("first boot: %v", err)
}
pub := publicKey(t, kp)
dirInfo, err := os.Stat(s.Dir)
if err != nil {
t.Fatal(err)
}
if got := dirInfo.Mode().Perm(); got != 0o700 {
t.Fatalf("directory mode = %o, want 700", got)
}
fileInfo, err := os.Stat(s.path())
if err != nil {
t.Fatal(err)
}
if got := fileInfo.Mode().Perm(); got != 0o600 {
t.Fatalf("key file mode = %o, want 600", got)
}
onDisk, err := nkeys.FromSeed(readFile(t, s.path()))
if err != nil {
t.Fatalf("persisted file is not a seed: %v", err)
}
if publicKey(t, onDisk) != pub {
t.Fatal("persisted seed does not match the returned key")
}
// A restart is a fresh Store over the same disk; neither path may
// generate a replacement.
again, err := Store{Dir: s.Dir}.ForExistingStore(pub)
if err != nil {
t.Fatalf("restart for the existing store: %v", err)
}
if publicKey(t, again) != pub {
t.Fatal("restart for the existing store changed the key")
}
again, err = Store{Dir: s.Dir}.ForNewStore()
if err != nil {
t.Fatalf("repeated first start: %v", err)
}
if publicKey(t, again) != pub {
t.Fatal("repeated first start replaced the key")
}
entries, err := os.ReadDir(s.Dir)
if err != nil {
t.Fatal(err)
}
if len(entries) != 1 {
t.Fatalf("key directory holds %d entries, want only the key", len(entries))
}
}
func TestEnsure_ExistingStoreWithMissingKeyFailsClosed(t *testing.T) {
s := newStore(t)
_, err := s.ForExistingStore(otherUserKey(t))
if !errors.Is(err, ErrKeyMissing) {
t.Fatalf("err = %v, want ErrKeyMissing", err)
}
if _, err := os.Lstat(s.path()); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("a key was generated for an existing store: %v", err)
}
}
func TestEnsure_MismatchedKeyFailsClosedAndKeepsFile(t *testing.T) {
s := newStore(t)
if _, err := s.ForNewStore(); err != nil {
t.Fatal(err)
}
before := readFile(t, s.path())
_, err := Store{Dir: s.Dir}.ForExistingStore(otherUserKey(t))
if !errors.Is(err, ErrKeyMismatch) {
t.Fatalf("err = %v, want ErrKeyMismatch", err)
}
assertNoSeed(t, err, before)
if string(readFile(t, s.path())) != string(before) {
t.Fatal("mismatch replaced the key file")
}
}
func TestEnsure_UnusableKeyFileIsNeverReplaced(t *testing.T) {
seedKP, err := nkeys.CreateUser()
if err != nil {
t.Fatal(err)
}
userSeed, err := seedKP.Seed()
if err != nil {
t.Fatal(err)
}
accountKP, err := nkeys.CreateAccount()
if err != nil {
t.Fatal(err)
}
accountSeed, err := accountKP.Seed()
if err != nil {
t.Fatal(err)
}
cases := []struct {
name string
content []byte
mode os.FileMode
want error
}{
{"empty", nil, 0o600, ErrCorrupt},
{"garbage", []byte("not a seed"), 0o600, ErrCorrupt},
{"account seed", accountSeed, 0o600, ErrCorrupt},
{"oversized", make([]byte, maxKeyFileSize+1), 0o600, ErrCorrupt},
{"group readable", userSeed, 0o640, ErrUnsafe},
{"world readable", userSeed, 0o604, ErrUnsafe},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s := newStore(t)
if err := os.MkdirAll(s.Dir, 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(s.path(), tc.content, 0o600); err != nil {
t.Fatal(err)
}
if err := os.Chmod(s.path(), tc.mode); err != nil {
t.Fatal(err)
}
_, err := s.ForNewStore()
if !errors.Is(err, tc.want) {
t.Fatalf("err = %v, want %v", err, tc.want)
}
assertNoSeed(t, err, userSeed)
assertNoSeed(t, err, accountSeed)
if string(readFile(t, s.path())) != string(tc.content) {
t.Fatal("an unusable key file was replaced")
}
})
}
}
func TestEnsure_RefusesUnsafeDirectoryAndSymlink(t *testing.T) {
t.Run("open directory", func(t *testing.T) {
s := newStore(t)
if err := os.MkdirAll(s.Dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.Chmod(s.Dir, 0o755); err != nil {
t.Fatal(err)
}
if _, err := s.ForNewStore(); !errors.Is(err, ErrUnsafe) {
t.Fatalf("err = %v, want ErrUnsafe", err)
}
if _, err := os.Lstat(s.path()); !errors.Is(err, os.ErrNotExist) {
t.Fatal("a key was written into an open directory")
}
})
t.Run("symlinked key", func(t *testing.T) {
s := newStore(t)
target := newStore(t)
if _, err := target.ForNewStore(); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(s.Dir, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target.path(), s.path()); err != nil {
t.Fatal(err)
}
if _, err := s.ForNewStore(); !errors.Is(err, ErrUnsafe) {
t.Fatalf("err = %v, want ErrUnsafe", err)
}
})
}
func TestEnsure_CrashLeftoverTempIsNotAKey(t *testing.T) {
s := newStore(t)
if err := os.MkdirAll(s.Dir, 0o700); err != nil {
t.Fatal(err)
}
stale := filepath.Join(s.Dir, tempPrefix+"123"+tempSuffix)
if err := os.WriteFile(stale, []byte("partial"), 0o600); err != nil {
t.Fatal(err)
}
// A crash before the link means no store was authorized for it.
if _, err := s.ForExistingStore(otherUserKey(t)); !errors.Is(err, ErrKeyMissing) {
t.Fatalf("err = %v, want ErrKeyMissing", err)
}
if _, err := os.Lstat(stale); !errors.Is(err, os.ErrNotExist) {
t.Fatal("stale temporary key file was kept")
}
}
func TestEnsure_ConcurrentFirstBootAgreesOnOneKey(t *testing.T) {
const rounds, n = 20, 16
for range rounds {
s := newStore(t)
pubs := make([]string, n)
errs := make([]error, n)
start := make(chan struct{})
var wg sync.WaitGroup
for i := range n {
wg.Go(func() {
<-start
kp, err := Store{Dir: s.Dir}.ForNewStore()
if err != nil {
errs[i] = err
return
}
pubs[i], errs[i] = kp.PublicKey()
})
}
close(start)
wg.Wait()
onDisk, err := nkeys.FromSeed(readFile(t, s.path()))
if err != nil {
t.Fatal(err)
}
want := publicKey(t, onDisk)
for i := range n {
if errs[i] != nil {
t.Fatalf("call %d: %v", i, errs[i])
}
if pubs[i] != want {
t.Fatalf("call %d returned a key that is not the persisted one", i)
}
}
}
}
func TestForExistingStore_RejectsInvalidAuthorizedKey(t *testing.T) {
kp, err := nkeys.CreateUser()
if err != nil {
t.Fatal(err)
}
seed, err := kp.Seed()
if err != nil {
t.Fatal(err)
}
// An empty key is what an unreadable store record would yield; it must not
// fall through to first-start generation.
for _, authorized := range []string{"", string(seed)} {
s := newStore(t)
_, err := s.ForExistingStore(authorized)
if err == nil {
t.Fatalf("authorized key %q was accepted", authorized[:min(len(authorized), 2)])
}
assertNoSeed(t, err, seed)
if _, err := os.Lstat(s.Dir); !errors.Is(err, os.ErrNotExist) {
t.Fatal("an invalid authorized key still touched the disk")
}
}
}
func TestForNewStore_RefusesMissingParent(t *testing.T) {
s := Store{Dir: filepath.Join(t.TempDir(), "missing-parent", "keys")}
if _, err := s.ForNewStore(); err == nil {
t.Fatal("a key directory was created without its parent")
}
if _, err := os.Lstat(filepath.Dir(s.Dir)); !errors.Is(err, os.ErrNotExist) {
t.Fatal("the missing parent was created")
}
}
// hookSyncs records every directory sync and fails call number failAt
// (counted from one); zero fails none.
func hookSyncs(t *testing.T, failAt int) *[]string {
t.Helper()
var synced []string
orig := syncDirFn
t.Cleanup(func() { syncDirFn = orig })
syncDirFn = func(dir string) error {
synced = append(synced, dir)
if len(synced) == failAt {
return errors.New("injected directory sync failure")
}
return orig(dir)
}
return &synced
}
func TestEnsure_RetryAfterFailedDirectorySyncKeepsKeyAndSyncs(t *testing.T) {
retries := map[string]func(s Store, pub string) (nkeys.KeyPair, error){
"first start": func(s Store, _ string) (nkeys.KeyPair, error) { return s.ForNewStore() },
"existing store": func(s Store, pub string) (nkeys.KeyPair, error) {
return s.ForExistingStore(pub)
},
}
for name, retry := range retries {
for failAt := 1; failAt <= 2; failAt++ {
t.Run(fmt.Sprintf("%s after sync %d failed", name, failAt), func(t *testing.T) {
s := newStore(t)
hookSyncs(t, failAt)
if _, err := s.ForNewStore(); err == nil {
t.Fatal("a key was returned although a directory sync failed")
}
first := readFile(t, s.path())
onDisk, err := nkeys.FromSeed(first)
if err != nil {
t.Fatal(err)
}
pub := publicKey(t, onDisk)
synced := hookSyncs(t, 0)
kp, err := retry(Store{Dir: s.Dir}, pub)
if err != nil {
t.Fatalf("retry: %v", err)
}
if string(readFile(t, s.path())) != string(first) {
t.Fatal("retry replaced the key")
}
if publicKey(t, kp) != pub {
t.Fatal("retry returned a key other than the persisted one")
}
want := []string{s.Dir, filepath.Dir(s.Dir)}
if len(*synced) != 2 || (*synced)[0] != want[0] || (*synced)[1] != want[1] {
t.Fatalf("retry synced %v, want %v", *synced, want)
}
})
}
}
}
// Another holder of the directory lock, such as a second worker process on
// the same disk, must finish before this one reads, cleans or creates a key.
func TestEnsure_WaitsForDirectoryLock(t *testing.T) {
s := newStore(t)
if err := os.MkdirAll(s.Dir, 0o700); err != nil {
t.Fatal(err)
}
holder, err := os.Open(s.Dir)
if err != nil {
t.Fatal(err)
}
defer holder.Close()
if err := syscall.Flock(int(holder.Fd()), syscall.LOCK_EX); err != nil {
t.Fatal(err)
}
done := make(chan error, 1)
go func() {
_, err := Store{Dir: s.Dir}.ForNewStore()
done <- err
}()
deadline := time.Now().Add(300 * time.Millisecond)
for time.Now().Before(deadline) {
select {
case err := <-done:
t.Fatalf("Ensure returned while another holder had the lock: %v", err)
default:
}
if _, err := os.Lstat(s.path()); err == nil {
t.Fatal("a key was created while another holder had the lock")
}
time.Sleep(10 * time.Millisecond)
}
if err := syscall.Flock(int(holder.Fd()), syscall.LOCK_UN); err != nil {
t.Fatal(err)
}
select {
case err := <-done:
if err != nil {
t.Fatalf("Ensure after unlock: %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Ensure did not proceed after the lock was released")
}
}
+14
View File
@@ -247,6 +247,7 @@ type Config struct {
ACM ACMConfig `json:"ACM" mapstructure:"acm"`
Bedrock BedrockConfig `json:"Bedrock" mapstructure:"bedrock"`
OchreVector OchreVectorConfig `json:"OchreVector" mapstructure:"ochre_vector"`
SQS SQSConfig `json:"SQS" mapstructure:"sqs"`
BaseDir string `json:"BaseDir" mapstructure:"base_dir"`
WalDir string `json:"WalDir" mapstructure:"wal_dir"`
@@ -447,6 +448,19 @@ type BedrockConfig struct {
// (10.248.0.0/14) and EKS control-plane (10.252.0.0/14) supernets.
const BedrockDefaultSystemVPCSupernet = "10.244.0.0/14"
// SQSConfig is a node's [sqs] section. The SQS worker and store VMs share one
// private system VPC per region, with no internet or NAT gateway.
type SQSConfig struct {
// The IPv4 /14 the system VPC's /22 is carved from. It must not overlap the
// RDS, Bedrock or EKS control-plane supernets or any customer VPC CIDR.
SystemVPCSupernet string `json:"SystemVPCSupernet" mapstructure:"system_vpc_supernet"`
}
// SQSDefaultSystemVPCSupernet anchors the SQS system VPC address space at
// 10.240.0.0/14, immediately below and disjoint from the Bedrock, RDS and EKS
// control-plane supernets.
const SQSDefaultSystemVPCSupernet = "10.240.0.0/14"
// OchreVectorConfig gates the Ochre vector store's platform Postgres
// appliance and VectorService NATS surface. Disabled by default: an unset
// section constructs nothing and registers no ochre.vector.* subjects, so an
+21
View File
@@ -3,6 +3,7 @@ package quota
import (
"context"
"errors"
"log/slog"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/elbv2"
@@ -113,3 +114,23 @@ func (s *Service) EnforceRDSInstances(ctx context.Context, natsConn *nats.Conn,
}
return exceeds(len(out.DBInstances), want, limits.RDSInstances)
}
// LoadBalancerLimit is the load balancer cap DescribeAccountLimits reports.
func (s *Service) LoadBalancerLimit(ctx context.Context, accountID string) (int, error) {
// An exempt account has no cap enforced against it, so the AWS default is a
// truer answer than a configured limit nothing consults.
if s.Exempt(accountID) {
return elbv2awsapi.DefaultLoadBalancerLimit, nil
}
_, limits, err := s.GetAccountQuota(ctx, accountID)
if err != nil {
if awserrors.IsErrorCode(err, awserrors.ErrorBlocked) {
return 0, err
}
slog.ErrorContext(ctx, "DescribeAccountLimits: failed to read account quota", "accountID", accountID, "err", err)
return 0, errors.New(awserrors.ErrorInternalError)
}
return limits.LoadBalancers, nil
}
var _ elbv2awsapi.Quota = (*Service)(nil)
@@ -6,6 +6,7 @@ import (
"math"
"testing"
elbv2awsapi "github.com/mulgadc/spinifex/spinifex/domains/elbv2/awsapi"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
)
@@ -82,3 +83,25 @@ func TestEnforceLiveExemptShortCircuits(t *testing.T) {
})
}
}
func TestLoadBalancerLimit_ExemptAccountReportsAWSDefault(t *testing.T) {
var disabled *Service
got, err := disabled.LoadBalancerLimit(t.Context(), "123456789012")
if err != nil {
t.Fatalf("LoadBalancerLimit: %v", err)
}
if got != elbv2awsapi.DefaultLoadBalancerLimit {
t.Errorf("limit = %d, want the AWS default %d", got, elbv2awsapi.DefaultLoadBalancerLimit)
}
}
func TestLoadBalancerLimit_EnabledAccountReportsConfiguredLimit(t *testing.T) {
svc := New(Limits{Enabled: true, LoadBalancers: 7}, nil)
got, err := svc.LoadBalancerLimit(t.Context(), "123456789012")
if err != nil {
t.Fatalf("LoadBalancerLimit: %v", err)
}
if got != 7 {
t.Errorf("limit = %d, want 7", got)
}
}
@@ -5,6 +5,7 @@ import (
"testing"
"github.com/mulgadc/spinifex/internal/testkit"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
"github.com/nats-io/nats.go/jetstream"
"github.com/stretchr/testify/require"
)
@@ -134,3 +135,36 @@ func TestPutAccountQuotaWithoutBucket(t *testing.T) {
svc := New(baseLimits(), nil)
require.Error(t, svc.PutAccountQuota(t.Context(), "000000000002", Overrides{VCPUs: ptr(32)}, "operator"))
}
// The cap DescribeAccountLimits reports follows the account's stored override
// and falls back to the configured baseline for an account without one.
func TestLoadBalancerLimit_FollowsPerAccountOverride(t *testing.T) {
svc, _ := newOverrideService(t)
const upgraded, plain = "000000000002", "000000000003"
require.NoError(t, svc.PutAccountQuota(t.Context(), upgraded, Overrides{LoadBalancers: ptr(31)}, "operator"))
got, err := svc.LoadBalancerLimit(t.Context(), upgraded)
require.NoError(t, err)
require.Equal(t, 31, got)
got, err = svc.LoadBalancerLimit(t.Context(), plain)
require.NoError(t, err)
require.Equal(t, baseLimits().LoadBalancers, got)
}
// A sealed account is refused with Blocked, and an unreadable record is an
// InternalError, never a made-up number.
func TestLoadBalancerLimit_RefusesSealedAndUnreadableRecords(t *testing.T) {
svc, kv := newOverrideService(t)
const sealed, corrupt = "000000000004", "000000000005"
_, err := kv.Put(t.Context(), sealed, []byte(`{"state":"sealed"}`))
require.NoError(t, err)
_, err = kv.Put(t.Context(), corrupt, []byte(`not json`))
require.NoError(t, err)
_, err = svc.LoadBalancerLimit(t.Context(), sealed)
require.EqualError(t, err, awserrors.ErrorBlocked)
_, err = svc.LoadBalancerLimit(t.Context(), corrupt)
require.EqualError(t, err, awserrors.ErrorInternalError)
}
+631
View File
@@ -0,0 +1,631 @@
// Package awsapi serves the EC2 query-protocol actions over composed capabilities.
package awsapi
import (
"context"
"errors"
"log/slog"
"net/http"
"reflect"
"strings"
"uuid"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
ec2accountapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/account"
ec2capacityreservationapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/capacityreservation"
ec2eigwapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/eigw"
ec2eipapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/eip"
ec2igwapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/igw"
ec2imageapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/image"
ec2instanceapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/instance"
ec2keyapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/key"
ec2launchtemplateapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/launchtemplate"
ec2natgwapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/natgw"
ec2placementgroupapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/placementgroup"
ec2routetableapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/routetable"
ec2snapshotapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/snapshot"
ec2spotinstanceapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/spotinstance"
ec2tagsapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/tags"
ec2volumeapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/volume"
ec2vpcapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/vpc"
ec2zoneapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/zone"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
awsxml "github.com/mulgadc/spinifex/spinifex/foundation/aws/xml"
"github.com/mulgadc/spinifex/spinifex/foundation/lifecycle/idempotency"
"github.com/mulgadc/spinifex/spinifex/ingress/aws/query"
"github.com/nats-io/nats.go"
)
// ec2Action parses once so authorization and dispatch share one typed input.
type ec2Action struct {
parse func(map[string]string) (any, error)
dispatch func(ctx context.Context, action string, parsed any, c *call, accountID string) ([]byte, error)
// idempotent records that this action honours ClientToken, so the set of
// actions that do is assertable rather than folklore.
idempotent bool
}
func (h ec2Action) withQueryPreprocessor(preprocess func(map[string]string)) ec2Action {
parse := h.parse
h.parse = func(q map[string]string) (any, error) {
preprocess(q)
return parse(q)
}
return h
}
// requestContext returns r's context, or Background for a nil request
// (some callers and tests dispatch without an *http.Request). The SDK's
// invocation id rides along: it is repeated across retries of one call, so it
// is what lets a mutation recognise its own resend instead of doing the work
// twice.
func requestContext(r *http.Request) context.Context {
if r == nil {
return context.Background()
}
return idempotency.WithKey(r.Context(), r.Header.Get(idempotency.SDKInvocationIDHeader))
}
// parseEC2Input parses query args into an action's input type. Shared by every
// ec2Action constructor so they all consume the same parse.
func parseEC2Input[In any](q map[string]string) (any, error) {
input := new(In)
if err := query.QueryParamsToStruct(q, input); err != nil {
return nil, err
}
return input, nil
}
// ec2Handler creates a type-safe EC2Handler. Parsing is separate from dispatch
// so authorization and execution consume the exact same input.
func ec2Handler[In any](handler func(ctx context.Context, input *In, c *call, accountID string) (any, error)) ec2Action {
return ec2Action{
parse: parseEC2Input[In],
dispatch: func(ctx context.Context, action string, parsed any, c *call, accountID string) ([]byte, error) {
input, ok := parsed.(*In)
if !ok {
return nil, errors.New(awserrors.ErrorInternalError)
}
output, err := handler(ctx, input, c, accountID)
if err != nil {
return nil, err
}
xmlOutput, err := marshalEC2Response(action, output)
if err != nil {
return nil, err
}
return xmlOutput, nil
},
}
}
// ec2ListsAsSet names the list fields AWS omits when empty instead of
// rendering them. Each is rendered as its handler set it, so nil omits it.
var ec2ListsAsSet = map[reflect.Type][]string{
reflect.TypeFor[ec2.CreateKeyPairOutput](): {"Tags"},
reflect.TypeFor[ec2.Image](): {"Tags", "ProductCodes"},
reflect.TypeFor[ec2.CreateSecurityGroupOutput](): {"Tags"},
reflect.TypeFor[ec2.SecurityGroup](): {"Tags"},
reflect.TypeFor[ec2.SecurityGroupRule](): {"Tags"},
reflect.TypeFor[ec2.RevokeSecurityGroupIngressOutput](): {"UnknownIpPermissions"},
reflect.TypeFor[ec2.RevokeSecurityGroupEgressOutput](): {"UnknownIpPermissions"},
reflect.TypeFor[ec2.NetworkInterface](): {"Ipv4Prefixes", "Ipv6Prefixes"},
reflect.TypeFor[ec2.Vpc](): {"Ipv6CidrBlockAssociationSet"},
reflect.TypeFor[ec2.Subnet](): {"Tags"},
}
// marshalEC2Response renders an EC2 handler's output into the action's XML
// envelope. Every EC2 action funnels through here (via ec2Handler and
// ec2HandlerWithReq), so wire-format fixes belong here, not in each handler.
func marshalEC2Response(action string, output any) ([]byte, error) {
// BuildXML omits a nil slice's container element entirely but renders an
// empty one for a non-nil empty slice; AWS mostly renders the latter.
normalized := awsxml.NormalizeOutput(output, ec2ListsAsSet)
// The SDK's generated output structs never carry a RequestId field.
withRequestID := awsxml.WithRequestID(normalized, uuid.NewV4().String())
payload := awsxml.ResponsePayload(action+"Response", withRequestID)
xmlOutput, err := awsxml.Marshal(payload)
if err != nil {
return nil, errors.New("failed to marshal response to XML")
}
return xmlOutput, nil
}
var ec2Actions = map[string]ec2Action{
"DescribeInstances": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstancesInput, c *call, accountID string) (any, error) {
out, err := ec2instanceapi.DescribeInstancesChecked(ctx, input, c.NATS, 0, c.NodeIDs, accountID)
if err != nil {
return out, err
}
ec2instanceapi.EnrichInstanceProfileIDs(out, c.IAM, accountID)
return out, nil
}),
"RunInstances": ec2Handler(func(ctx context.Context, input *ec2.RunInstancesInput, c *call, accountID string) (any, error) {
passRoleCheck := func(roleARN string) error {
return c.passRole(roleARN)
}
launchQuotaCheck := func() error {
return c.Quota.EnforceLaunch(ctx, accountID, aws.StringValue(input.InstanceType), int(aws.Int64Value(input.MaxCount)))
}
reservation, err := ec2instanceapi.RunInstances(ctx, input, c.NATS, c.IAM, accountID, passRoleCheck, launchQuotaCheck, c.ExpectedNodes)
if err != nil {
return nil, err
}
// Charge the actual launched vCPUs; a counter write failure is drift for
// reconcile to correct, so it must not fail the already-successful launch.
if err := c.Quota.ChargeLaunch(ctx, accountID, &reservation); err != nil {
slog.WarnContext(ctx, "RunInstances: vcpu quota charge failed, reconcile will correct", "account", accountID, "err", err)
}
return reservation, nil
}),
"AssociateIamInstanceProfile": ec2Handler(func(ctx context.Context, input *ec2.AssociateIamInstanceProfileInput, c *call, accountID string) (any, error) {
passRoleCheck := func(roleARN string) error {
return c.passRole(roleARN)
}
return ec2instanceapi.AssociateIamInstanceProfile(ctx, input, c.NATS, c.IAM, accountID, passRoleCheck)
}),
"DisassociateIamInstanceProfile": ec2Handler(func(ctx context.Context, input *ec2.DisassociateIamInstanceProfileInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DisassociateIamInstanceProfile(ctx, input, c.NATS, c.ActiveNodes(ctx), accountID)
}),
"ReplaceIamInstanceProfileAssociation": ec2Handler(func(ctx context.Context, input *ec2.ReplaceIamInstanceProfileAssociationInput, c *call, accountID string) (any, error) {
passRoleCheck := func(roleARN string) error {
return c.passRole(roleARN)
}
return ec2instanceapi.ReplaceIamInstanceProfileAssociation(ctx, input, c.NATS, c.IAM, c.ActiveNodes(ctx), accountID, passRoleCheck)
}),
"DescribeIamInstanceProfileAssociations": ec2Handler(func(ctx context.Context, input *ec2.DescribeIamInstanceProfileAssociationsInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DescribeIamInstanceProfileAssociations(ctx, input, c.NATS, c.ActiveNodes(ctx), accountID)
}),
"StartInstances": ec2Handler(func(ctx context.Context, input *ec2.StartInstancesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.StartInstances(ctx, input, c.NATS, accountID)
}),
"StopInstances": ec2Handler(func(ctx context.Context, input *ec2.StopInstancesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.StopInstances(ctx, input, c.NATS, accountID)
}),
"RebootInstances": ec2Handler(func(ctx context.Context, input *ec2.RebootInstancesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.RebootInstances(ctx, input, c.NATS, accountID)
}),
"MonitorInstances": ec2Handler(func(ctx context.Context, input *ec2.MonitorInstancesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.MonitorInstances(ctx, input, c.NATS, accountID)
}),
"UnmonitorInstances": ec2Handler(func(ctx context.Context, input *ec2.UnmonitorInstancesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.UnmonitorInstances(ctx, input, c.NATS, accountID)
}),
"TerminateInstances": ec2Handler(func(ctx context.Context, input *ec2.TerminateInstancesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.TerminateInstances(ctx, input, c.NATS, accountID)
}),
"DescribeInstanceTypes": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstanceTypesInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DescribeInstanceTypes(ctx, input, c.NATS, c.ExpectedNodes, c.NodeIDs, accountID)
}),
"DescribeInstanceTypeOfferings": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstanceTypeOfferingsInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DescribeInstanceTypeOfferings(ctx, input, c.NATS, c.ExpectedNodes, accountID, c.Region, c.AZ)
}),
"DescribeInstanceStatus": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstanceStatusInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DescribeInstanceStatus(ctx, input, c.NATS, c.ActiveNodes(ctx), accountID, c.AZ, c.InstanceStatus)
}),
"GetConsoleOutput": ec2Handler(func(ctx context.Context, input *ec2.GetConsoleOutputInput, c *call, accountID string) (any, error) {
return ec2instanceapi.GetConsoleOutput(ctx, input, c.NATS, accountID)
}),
"GetPasswordData": ec2Handler(func(ctx context.Context, input *ec2.GetPasswordDataInput, c *call, accountID string) (any, error) {
return ec2instanceapi.GetPasswordData(ctx, input, c.NATS, accountID)
}),
"ModifyInstanceAttribute": ec2Handler(func(ctx context.Context, input *ec2.ModifyInstanceAttributeInput, c *call, accountID string) (any, error) {
var delta int
if input.InstanceType != nil {
d, err := c.Quota.EnforceRetype(ctx, c.NATS, func() []string { return c.NodeIDs }, accountID, aws.StringValue(input.InstanceId), aws.StringValue(input.InstanceType.Value))
if err != nil {
return nil, err
}
delta = d
}
out, err := ec2instanceapi.ModifyInstanceAttribute(ctx, input, c.NATS, accountID)
if err != nil {
return nil, err
}
// Charge the retype's vCPU growth; a counter write failure is drift for
// reconcile to correct, so it must not fail the applied retype.
if err := c.Quota.AddVCPU(ctx, accountID, delta); err != nil {
slog.WarnContext(ctx, "ModifyInstanceAttribute: vcpu quota charge failed, reconcile will correct", "account", accountID, "err", err)
}
return out, nil
}),
"DescribeInstanceAttribute": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstanceAttributeInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DescribeInstanceAttribute(ctx, input, c.NATS, c.ActiveNodes(ctx), accountID)
}),
"ModifyInstanceMetadataOptions": ec2Handler(func(ctx context.Context, input *ec2.ModifyInstanceMetadataOptionsInput, c *call, accountID string) (any, error) {
return ec2instanceapi.ModifyInstanceMetadataOptions(ctx, input, c.NATS, accountID)
}),
"DescribeInstanceCreditSpecifications": ec2Handler(func(ctx context.Context, input *ec2.DescribeInstanceCreditSpecificationsInput, c *call, accountID string) (any, error) {
return ec2instanceapi.DescribeInstanceCreditSpecifications(input)
}),
"CreateKeyPair": ec2Handler(func(ctx context.Context, input *ec2.CreateKeyPairInput, c *call, accountID string) (any, error) {
return ec2keyapi.CreateKeyPair(ctx, input, c.NATS, accountID)
}),
"DeleteKeyPair": ec2Handler(func(ctx context.Context, input *ec2.DeleteKeyPairInput, c *call, accountID string) (any, error) {
return ec2keyapi.DeleteKeyPair(ctx, input, c.NATS, accountID)
}),
"DescribeKeyPairs": ec2Handler(func(ctx context.Context, input *ec2.DescribeKeyPairsInput, c *call, accountID string) (any, error) {
return ec2keyapi.DescribeKeyPairs(ctx, input, c.NATS, accountID)
}),
"ImportKeyPair": ec2Handler(func(ctx context.Context, input *ec2.ImportKeyPairInput, c *call, accountID string) (any, error) {
return ec2keyapi.ImportKeyPair(ctx, input, c.NATS, accountID)
}).withQueryPreprocessor(func(q map[string]string) {
// Parser leaves Base64 padding URL-encoded; decode it before dispatch.
if strings.HasSuffix(q["PublicKeyMaterial"], "%3D%3D") {
q["PublicKeyMaterial"] = strings.Replace(q["PublicKeyMaterial"], "%3D%3D", "==", 1)
}
}),
"DescribeImages": ec2Handler(func(ctx context.Context, input *ec2.DescribeImagesInput, c *call, accountID string) (any, error) {
return ec2imageapi.DescribeImages(ctx, input, c.NATS, accountID)
}),
"CreateImage": ec2Handler(func(ctx context.Context, input *ec2.CreateImageInput, c *call, accountID string) (any, error) {
return ec2imageapi.CreateImage(ctx, input, c.NATS, c.ActiveNodes(ctx), accountID)
}),
"DeregisterImage": ec2Handler(func(ctx context.Context, input *ec2.DeregisterImageInput, c *call, accountID string) (any, error) {
return ec2imageapi.DeregisterImage(ctx, input, c.NATS, accountID)
}),
"RegisterImage": ec2Handler(func(ctx context.Context, input *ec2.RegisterImageInput, c *call, accountID string) (any, error) {
return ec2imageapi.RegisterImage(ctx, input, c.NATS, accountID)
}),
"CopyImage": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CopyImageInput, c *call, accountID string) (ec2.CopyImageOutput, error) {
return ec2imageapi.CopyImage(ctx, input, c.NATS, c.Region, accountID)
}),
"DescribeImageAttribute": ec2Handler(func(ctx context.Context, input *ec2.DescribeImageAttributeInput, c *call, accountID string) (any, error) {
return ec2imageapi.DescribeImageAttribute(ctx, input, c.NATS, accountID)
}),
"ModifyImageAttribute": ec2Handler(func(ctx context.Context, input *ec2.ModifyImageAttributeInput, c *call, accountID string) (any, error) {
return ec2imageapi.ModifyImageAttribute(ctx, input, c.NATS, accountID)
}),
"ResetImageAttribute": ec2Handler(func(ctx context.Context, input *ec2.ResetImageAttributeInput, c *call, accountID string) (any, error) {
return ec2imageapi.ResetImageAttribute(ctx, input, c.NATS, accountID)
}),
"DescribeRegions": ec2Handler(func(ctx context.Context, input *ec2.DescribeRegionsInput, c *call, accountID string) (any, error) {
return ec2zoneapi.DescribeRegions(input, c.Region, c.Endpoint)
}),
"DescribeAvailabilityZones": ec2Handler(func(ctx context.Context, input *ec2.DescribeAvailabilityZonesInput, c *call, accountID string) (any, error) {
return ec2zoneapi.DescribeAvailabilityZones(input, c.Region, c.AZ)
}),
"DescribeVolumes": ec2Handler(func(ctx context.Context, input *ec2.DescribeVolumesInput, c *call, accountID string) (any, error) {
return ec2volumeapi.DescribeVolumes(ctx, input, c.NATS, accountID)
}),
"ModifyVolume": ec2Handler(func(ctx context.Context, input *ec2.ModifyVolumeInput, c *call, accountID string) (any, error) {
if err := c.Quota.EnforceVolumeModify(ctx, c.NATS, accountID, aws.StringValue(input.VolumeId), int(aws.Int64Value(input.Size))); err != nil {
return nil, err
}
return ec2volumeapi.ModifyVolume(ctx, input, c.NATS, accountID)
}),
"CreateVolume": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateVolumeInput, c *call, accountID string) (ec2.Volume, error) {
if err := c.Quota.EnforceVolumeCreate(ctx, c.NATS, accountID, int(aws.Int64Value(input.Size))); err != nil {
return ec2.Volume{}, err
}
return ec2volumeapi.CreateVolume(ctx, input, c.NATS, accountID)
}),
"DeleteVolume": ec2Handler(func(ctx context.Context, input *ec2.DeleteVolumeInput, c *call, accountID string) (any, error) {
return ec2volumeapi.DeleteVolume(ctx, input, c.NATS, c.NodeIDs, accountID)
}),
"AttachVolume": ec2Handler(func(ctx context.Context, input *ec2.AttachVolumeInput, c *call, accountID string) (any, error) {
return ec2volumeapi.AttachVolume(ctx, input, c.NATS, accountID)
}),
"DescribeVolumeStatus": ec2Handler(func(ctx context.Context, input *ec2.DescribeVolumeStatusInput, c *call, accountID string) (any, error) {
return ec2volumeapi.DescribeVolumeStatus(ctx, input, c.NATS, accountID)
}),
"DescribeVolumesModifications": ec2Handler(func(ctx context.Context, input *ec2.DescribeVolumesModificationsInput, c *call, accountID string) (any, error) {
return ec2volumeapi.DescribeVolumesModifications(ctx, input, c.NATS, accountID)
}),
"DetachVolume": ec2Handler(func(ctx context.Context, input *ec2.DetachVolumeInput, c *call, accountID string) (any, error) {
return ec2volumeapi.DetachVolume(ctx, input, c.NATS, accountID)
}),
"DescribeAccountAttributes": ec2Handler(func(ctx context.Context, input *ec2.DescribeAccountAttributesInput, c *call, accountID string) (any, error) {
return ec2accountapi.DescribeAccountAttributes(input, func() (string, error) {
return defaultVPCID(ctx, c.NATS, accountID)
})
}),
"EnableEbsEncryptionByDefault": ec2Handler(func(ctx context.Context, input *ec2.EnableEbsEncryptionByDefaultInput, c *call, accountID string) (any, error) {
return ec2accountapi.EnableEbsEncryptionByDefault(ctx, input, c.NATS, accountID)
}),
"DisableEbsEncryptionByDefault": ec2Handler(func(ctx context.Context, input *ec2.DisableEbsEncryptionByDefaultInput, c *call, accountID string) (any, error) {
return ec2accountapi.DisableEbsEncryptionByDefault(ctx, input, c.NATS, accountID)
}),
"GetEbsEncryptionByDefault": ec2Handler(func(ctx context.Context, input *ec2.GetEbsEncryptionByDefaultInput, c *call, accountID string) (any, error) {
return ec2accountapi.GetEbsEncryptionByDefault(ctx, input, c.NATS, accountID)
}),
"GetSerialConsoleAccessStatus": ec2Handler(func(ctx context.Context, input *ec2.GetSerialConsoleAccessStatusInput, c *call, accountID string) (any, error) {
return ec2accountapi.GetSerialConsoleAccessStatus(ctx, input, c.NATS, accountID)
}),
"EnableSerialConsoleAccess": ec2Handler(func(ctx context.Context, input *ec2.EnableSerialConsoleAccessInput, c *call, accountID string) (any, error) {
return ec2accountapi.EnableSerialConsoleAccess(ctx, input, c.NATS, accountID)
}),
"DisableSerialConsoleAccess": ec2Handler(func(ctx context.Context, input *ec2.DisableSerialConsoleAccessInput, c *call, accountID string) (any, error) {
return ec2accountapi.DisableSerialConsoleAccess(ctx, input, c.NATS, accountID)
}),
"CreateTags": ec2Handler(func(ctx context.Context, input *ec2.CreateTagsInput, c *call, accountID string) (any, error) {
return ec2tagsapi.CreateTags(ctx, input, c.NATS, accountID)
}),
"DeleteTags": ec2Handler(func(ctx context.Context, input *ec2.DeleteTagsInput, c *call, accountID string) (any, error) {
return ec2tagsapi.DeleteTags(ctx, input, c.NATS, accountID)
}),
"DescribeTags": ec2Handler(func(ctx context.Context, input *ec2.DescribeTagsInput, c *call, accountID string) (any, error) {
return ec2tagsapi.DescribeTags(ctx, input, c.NATS, accountID)
}),
"CreateSnapshot": ec2Handler(func(ctx context.Context, input *ec2.CreateSnapshotInput, c *call, accountID string) (any, error) {
return ec2snapshotapi.CreateSnapshot(ctx, input, c.NATS, accountID)
}),
"DeleteSnapshot": ec2Handler(func(ctx context.Context, input *ec2.DeleteSnapshotInput, c *call, accountID string) (any, error) {
return ec2snapshotapi.DeleteSnapshot(ctx, input, c.NATS, accountID)
}),
"DescribeSnapshots": ec2Handler(func(ctx context.Context, input *ec2.DescribeSnapshotsInput, c *call, accountID string) (any, error) {
return ec2snapshotapi.DescribeSnapshots(ctx, input, c.NATS, accountID)
}),
"CopySnapshot": ec2Handler(func(ctx context.Context, input *ec2.CopySnapshotInput, c *call, accountID string) (any, error) {
return ec2snapshotapi.CopySnapshot(ctx, input, c.NATS, accountID)
}),
"CreateInternetGateway": ec2Handler(func(ctx context.Context, input *ec2.CreateInternetGatewayInput, c *call, accountID string) (any, error) {
return ec2igwapi.CreateInternetGateway(ctx, input, c.NATS, accountID)
}),
"DeleteInternetGateway": ec2Handler(func(ctx context.Context, input *ec2.DeleteInternetGatewayInput, c *call, accountID string) (any, error) {
return ec2igwapi.DeleteInternetGateway(ctx, input, c.NATS, accountID)
}),
"DescribeInternetGateways": ec2Handler(func(ctx context.Context, input *ec2.DescribeInternetGatewaysInput, c *call, accountID string) (any, error) {
return ec2igwapi.DescribeInternetGateways(ctx, input, c.NATS, accountID)
}),
"AttachInternetGateway": ec2Handler(func(ctx context.Context, input *ec2.AttachInternetGatewayInput, c *call, accountID string) (any, error) {
return ec2igwapi.AttachInternetGateway(ctx, input, c.NATS, accountID)
}),
"DetachInternetGateway": ec2Handler(func(ctx context.Context, input *ec2.DetachInternetGatewayInput, c *call, accountID string) (any, error) {
return ec2igwapi.DetachInternetGateway(ctx, input, c.NATS, accountID)
}),
"CreateEgressOnlyInternetGateway": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateEgressOnlyInternetGatewayInput, c *call, accountID string) (ec2.CreateEgressOnlyInternetGatewayOutput, error) {
return ec2eigwapi.CreateEgressOnlyInternetGateway(ctx, input, c.NATS, accountID)
}),
"DeleteEgressOnlyInternetGateway": ec2Handler(func(ctx context.Context, input *ec2.DeleteEgressOnlyInternetGatewayInput, c *call, accountID string) (any, error) {
return ec2eigwapi.DeleteEgressOnlyInternetGateway(ctx, input, c.NATS, accountID)
}),
"DescribeEgressOnlyInternetGateways": ec2Handler(func(ctx context.Context, input *ec2.DescribeEgressOnlyInternetGatewaysInput, c *call, accountID string) (any, error) {
return ec2eigwapi.DescribeEgressOnlyInternetGateways(ctx, input, c.NATS, accountID)
}),
"CreatePlacementGroup": ec2Handler(func(ctx context.Context, input *ec2.CreatePlacementGroupInput, c *call, accountID string) (any, error) {
return ec2placementgroupapi.CreatePlacementGroup(ctx, input, c.NATS, accountID)
}),
"DeletePlacementGroup": ec2Handler(func(ctx context.Context, input *ec2.DeletePlacementGroupInput, c *call, accountID string) (any, error) {
return ec2placementgroupapi.DeletePlacementGroup(ctx, input, c.NATS, accountID)
}),
"DescribePlacementGroups": ec2Handler(func(ctx context.Context, input *ec2.DescribePlacementGroupsInput, c *call, accountID string) (any, error) {
return ec2placementgroupapi.DescribePlacementGroups(ctx, input, c.NATS, accountID)
}),
"CreateCapacityReservation": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateCapacityReservationInput, c *call, accountID string) (ec2.CreateCapacityReservationOutput, error) {
return ec2capacityreservationapi.CreateCapacityReservation(ctx, input, c.NATS, c.ExpectedNodes, accountID)
}),
"DescribeCapacityReservations": ec2Handler(func(ctx context.Context, input *ec2.DescribeCapacityReservationsInput, c *call, accountID string) (any, error) {
return ec2capacityreservationapi.DescribeCapacityReservations(ctx, input, c.NATS, c.ExpectedNodes, accountID)
}),
"CancelCapacityReservation": ec2Handler(func(ctx context.Context, input *ec2.CancelCapacityReservationInput, c *call, accountID string) (any, error) {
return ec2capacityreservationapi.CancelCapacityReservation(ctx, input, c.NATS, c.ExpectedNodes, accountID)
}),
"CreateLaunchTemplate": ec2Handler(func(ctx context.Context, input *ec2.CreateLaunchTemplateInput, c *call, accountID string) (any, error) {
return ec2launchtemplateapi.CreateLaunchTemplate(ctx, input, c.NATS, accountID)
}),
"CreateLaunchTemplateVersion": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateLaunchTemplateVersionInput, c *call, accountID string) (ec2.CreateLaunchTemplateVersionOutput, error) {
return ec2launchtemplateapi.CreateLaunchTemplateVersion(ctx, input, c.NATS, accountID)
}),
"DeleteLaunchTemplate": ec2Handler(func(ctx context.Context, input *ec2.DeleteLaunchTemplateInput, c *call, accountID string) (any, error) {
return ec2launchtemplateapi.DeleteLaunchTemplate(ctx, input, c.NATS, accountID)
}),
"DeleteLaunchTemplateVersions": ec2Handler(func(ctx context.Context, input *ec2.DeleteLaunchTemplateVersionsInput, c *call, accountID string) (any, error) {
return ec2launchtemplateapi.DeleteLaunchTemplateVersions(ctx, input, c.NATS, accountID)
}),
"ModifyLaunchTemplate": ec2Handler(func(ctx context.Context, input *ec2.ModifyLaunchTemplateInput, c *call, accountID string) (any, error) {
return ec2launchtemplateapi.ModifyLaunchTemplate(ctx, input, c.NATS, accountID)
}),
"DescribeLaunchTemplates": ec2Handler(func(ctx context.Context, input *ec2.DescribeLaunchTemplatesInput, c *call, accountID string) (any, error) {
return ec2launchtemplateapi.DescribeLaunchTemplates(ctx, input, c.NATS, accountID)
}),
"DescribeLaunchTemplateVersions": ec2Handler(func(ctx context.Context, input *ec2.DescribeLaunchTemplateVersionsInput, c *call, accountID string) (any, error) {
return ec2launchtemplateapi.DescribeLaunchTemplateVersions(ctx, input, c.NATS, accountID)
}),
"RequestSpotInstances": ec2Handler(func(ctx context.Context, input *ec2.RequestSpotInstancesInput, c *call, accountID string) (any, error) {
passRoleCheck := func(roleARN string) error {
return c.passRole(roleARN)
}
return ec2spotinstanceapi.RequestSpotInstances(ctx, input, c.NATS, c.IAM, accountID, c.AZ, passRoleCheck, c.Quota, c.ExpectedNodes)
}),
"DescribeSpotInstanceRequests": ec2Handler(func(ctx context.Context, input *ec2.DescribeSpotInstanceRequestsInput, c *call, accountID string) (any, error) {
return ec2spotinstanceapi.DescribeSpotInstanceRequests(ctx, input, c.NATS, accountID)
}),
"CancelSpotInstanceRequests": ec2Handler(func(ctx context.Context, input *ec2.CancelSpotInstanceRequestsInput, c *call, accountID string) (any, error) {
return ec2spotinstanceapi.CancelSpotInstanceRequests(ctx, input, c.NATS, accountID)
}),
"CreateVpc": ec2Handler(func(ctx context.Context, input *ec2.CreateVpcInput, c *call, accountID string) (any, error) {
if err := c.Quota.EnforceVPCs(ctx, c.NATS, accountID, 1); err != nil {
return nil, err
}
return ec2vpcapi.CreateVpc(ctx, input, c.NATS, accountID)
}),
"DeleteVpc": ec2Handler(func(ctx context.Context, input *ec2.DeleteVpcInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DeleteVpc(ctx, input, c.NATS, accountID)
}),
"DescribeVpcs": ec2Handler(func(ctx context.Context, input *ec2.DescribeVpcsInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DescribeVpcs(ctx, input, c.NATS, accountID)
}),
"ModifyVpcAttribute": ec2Handler(func(ctx context.Context, input *ec2.ModifyVpcAttributeInput, c *call, accountID string) (any, error) {
return ec2vpcapi.ModifyVpcAttribute(ctx, input, c.NATS, accountID)
}),
"DescribeVpcAttribute": ec2Handler(func(ctx context.Context, input *ec2.DescribeVpcAttributeInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DescribeVpcAttribute(ctx, input, c.NATS, accountID)
}),
// No VPC endpoint can exist here, because CreateVpcEndpoint is not served, so
// the empty list is the true answer rather than a stand-in for one. It has to
// be answered all the same: the Terraform AWS provider calls this on every VPC
// delete to sweep GuardDuty-managed endpoints, and an InvalidAction there fails
// the destroy and strands the VPC.
"DescribeVpcEndpoints": ec2Handler(func(_ context.Context, _ *ec2.DescribeVpcEndpointsInput, _ *call, _ string) (any, error) {
return ec2.DescribeVpcEndpointsOutput{VpcEndpoints: []*ec2.VpcEndpoint{}}, nil
}),
"CreateSubnet": ec2Handler(func(ctx context.Context, input *ec2.CreateSubnetInput, c *call, accountID string) (any, error) {
if err := c.Quota.EnforceSubnets(ctx, c.NATS, accountID, 1); err != nil {
return nil, err
}
return ec2vpcapi.CreateSubnet(ctx, input, c.NATS, accountID)
}),
"DeleteSubnet": ec2Handler(func(ctx context.Context, input *ec2.DeleteSubnetInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DeleteSubnet(ctx, input, c.NATS, accountID)
}),
"DescribeSubnets": ec2Handler(func(ctx context.Context, input *ec2.DescribeSubnetsInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DescribeSubnets(ctx, input, c.NATS, accountID)
}),
"ModifySubnetAttribute": ec2Handler(func(ctx context.Context, input *ec2.ModifySubnetAttributeInput, c *call, accountID string) (any, error) {
return ec2vpcapi.ModifySubnetAttribute(ctx, input, c.NATS, accountID)
}),
"CreateRouteTable": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateRouteTableInput, c *call, accountID string) (ec2.CreateRouteTableOutput, error) {
return ec2routetableapi.CreateRouteTable(ctx, input, c.NATS, accountID)
}),
"DeleteRouteTable": ec2Handler(func(ctx context.Context, input *ec2.DeleteRouteTableInput, c *call, accountID string) (any, error) {
return ec2routetableapi.DeleteRouteTable(ctx, input, c.NATS, accountID)
}),
"DescribeRouteTables": ec2Handler(func(ctx context.Context, input *ec2.DescribeRouteTablesInput, c *call, accountID string) (any, error) {
return ec2routetableapi.DescribeRouteTables(ctx, input, c.NATS, accountID)
}),
"CreateRoute": ec2Handler(func(ctx context.Context, input *ec2.CreateRouteInput, c *call, accountID string) (any, error) {
return ec2routetableapi.CreateRoute(ctx, input, c.NATS, accountID)
}),
"DeleteRoute": ec2Handler(func(ctx context.Context, input *ec2.DeleteRouteInput, c *call, accountID string) (any, error) {
return ec2routetableapi.DeleteRoute(ctx, input, c.NATS, accountID)
}),
"ReplaceRoute": ec2Handler(func(ctx context.Context, input *ec2.ReplaceRouteInput, c *call, accountID string) (any, error) {
return ec2routetableapi.ReplaceRoute(ctx, input, c.NATS, accountID)
}),
"AssociateRouteTable": ec2Handler(func(ctx context.Context, input *ec2.AssociateRouteTableInput, c *call, accountID string) (any, error) {
return ec2routetableapi.AssociateRouteTable(ctx, input, c.NATS, accountID)
}),
"DisassociateRouteTable": ec2Handler(func(ctx context.Context, input *ec2.DisassociateRouteTableInput, c *call, accountID string) (any, error) {
return ec2routetableapi.DisassociateRouteTable(ctx, input, c.NATS, accountID)
}),
"ReplaceRouteTableAssociation": ec2Handler(func(ctx context.Context, input *ec2.ReplaceRouteTableAssociationInput, c *call, accountID string) (any, error) {
return ec2routetableapi.ReplaceRouteTableAssociation(ctx, input, c.NATS, accountID)
}),
"CreateNetworkInterface": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateNetworkInterfaceInput, c *call, accountID string) (ec2.CreateNetworkInterfaceOutput, error) {
return ec2vpcapi.CreateNetworkInterface(ctx, input, c.NATS, accountID)
}),
"DeleteNetworkInterface": ec2Handler(func(ctx context.Context, input *ec2.DeleteNetworkInterfaceInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DeleteNetworkInterface(ctx, input, c.NATS, accountID)
}),
"DescribeNetworkInterfaces": ec2Handler(func(ctx context.Context, input *ec2.DescribeNetworkInterfacesInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DescribeNetworkInterfaces(ctx, input, c.NATS, accountID)
}),
"ModifyNetworkInterfaceAttribute": ec2Handler(func(ctx context.Context, input *ec2.ModifyNetworkInterfaceAttributeInput, c *call, accountID string) (any, error) {
return ec2vpcapi.ModifyNetworkInterfaceAttribute(ctx, input, c.NATS, accountID)
}),
"AttachNetworkInterface": ec2Handler(func(ctx context.Context, input *ec2.AttachNetworkInterfaceInput, c *call, accountID string) (any, error) {
return ec2vpcapi.AttachNetworkInterface(ctx, input, c.NATS, accountID)
}),
"DetachNetworkInterface": ec2Handler(func(ctx context.Context, input *ec2.DetachNetworkInterfaceInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DetachNetworkInterface(ctx, input, c.NATS, accountID)
}),
"CreateSecurityGroup": ec2Handler(func(ctx context.Context, input *ec2.CreateSecurityGroupInput, c *call, accountID string) (any, error) {
return ec2vpcapi.CreateSecurityGroup(ctx, input, c.NATS, accountID)
}),
"DeleteSecurityGroup": ec2Handler(func(ctx context.Context, input *ec2.DeleteSecurityGroupInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DeleteSecurityGroup(ctx, input, c.NATS, accountID)
}),
"DescribeSecurityGroups": ec2Handler(func(ctx context.Context, input *ec2.DescribeSecurityGroupsInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DescribeSecurityGroups(ctx, input, c.NATS, accountID)
}),
"DescribeSecurityGroupRules": ec2Handler(func(ctx context.Context, input *ec2.DescribeSecurityGroupRulesInput, c *call, accountID string) (any, error) {
return ec2vpcapi.DescribeSecurityGroupRules(ctx, input, c.NATS, accountID)
}),
"GetSecurityGroupsForVpc": ec2Handler(func(ctx context.Context, input *ec2.GetSecurityGroupsForVpcInput, c *call, accountID string) (any, error) {
return ec2vpcapi.GetSecurityGroupsForVpc(ctx, input, c.NATS, accountID)
}),
"AuthorizeSecurityGroupIngress": ec2Handler(func(ctx context.Context, input *ec2.AuthorizeSecurityGroupIngressInput, c *call, accountID string) (any, error) {
return ec2vpcapi.AuthorizeSecurityGroupIngress(ctx, input, c.NATS, accountID)
}),
"AuthorizeSecurityGroupEgress": ec2Handler(func(ctx context.Context, input *ec2.AuthorizeSecurityGroupEgressInput, c *call, accountID string) (any, error) {
return ec2vpcapi.AuthorizeSecurityGroupEgress(ctx, input, c.NATS, accountID)
}),
"RevokeSecurityGroupIngress": ec2Handler(func(ctx context.Context, input *ec2.RevokeSecurityGroupIngressInput, c *call, accountID string) (any, error) {
return ec2vpcapi.RevokeSecurityGroupIngress(ctx, input, c.NATS, accountID)
}),
"RevokeSecurityGroupEgress": ec2Handler(func(ctx context.Context, input *ec2.RevokeSecurityGroupEgressInput, c *call, accountID string) (any, error) {
return ec2vpcapi.RevokeSecurityGroupEgress(ctx, input, c.NATS, accountID)
}),
"UpdateSecurityGroupRuleDescriptionsIngress": ec2Handler(func(ctx context.Context, input *ec2.UpdateSecurityGroupRuleDescriptionsIngressInput, c *call, accountID string) (any, error) {
return ec2vpcapi.UpdateSecurityGroupRuleDescriptionsIngress(ctx, input, c.NATS, accountID)
}),
"UpdateSecurityGroupRuleDescriptionsEgress": ec2Handler(func(ctx context.Context, input *ec2.UpdateSecurityGroupRuleDescriptionsEgressInput, c *call, accountID string) (any, error) {
return ec2vpcapi.UpdateSecurityGroupRuleDescriptionsEgress(ctx, input, c.NATS, accountID)
}),
"ModifySecurityGroupRules": ec2Handler(func(ctx context.Context, input *ec2.ModifySecurityGroupRulesInput, c *call, accountID string) (any, error) {
return ec2vpcapi.ModifySecurityGroupRules(ctx, input, c.NATS, accountID)
}),
"AllocateAddress": ec2Handler(func(ctx context.Context, input *ec2.AllocateAddressInput, c *call, accountID string) (any, error) {
if err := c.Quota.EnforceEIPs(ctx, c.NATS, accountID, 1); err != nil {
return nil, err
}
return ec2eipapi.AllocateAddress(ctx, input, c.NATS, accountID)
}),
"ReleaseAddress": ec2Handler(func(ctx context.Context, input *ec2.ReleaseAddressInput, c *call, accountID string) (any, error) {
return ec2eipapi.ReleaseAddress(ctx, input, c.NATS, accountID)
}),
"AssociateAddress": ec2Handler(func(ctx context.Context, input *ec2.AssociateAddressInput, c *call, accountID string) (any, error) {
return ec2eipapi.AssociateAddress(ctx, input, c.NATS, accountID)
}),
"DisassociateAddress": ec2Handler(func(ctx context.Context, input *ec2.DisassociateAddressInput, c *call, accountID string) (any, error) {
return ec2eipapi.DisassociateAddress(ctx, input, c.NATS, accountID)
}),
"DescribeAddresses": ec2Handler(func(ctx context.Context, input *ec2.DescribeAddressesInput, c *call, accountID string) (any, error) {
return ec2eipapi.DescribeAddresses(ctx, input, c.NATS, accountID)
}),
"DescribeAddressesAttribute": ec2Handler(func(ctx context.Context, input *ec2.DescribeAddressesAttributeInput, c *call, accountID string) (any, error) {
return ec2eipapi.DescribeAddressesAttribute(ctx, input, c.NATS, accountID)
}),
"CreateNatGateway": ec2IdempotentHandler(func(ctx context.Context, input *ec2.CreateNatGatewayInput, c *call, accountID string) (ec2.CreateNatGatewayOutput, error) {
return ec2natgwapi.CreateNatGateway(ctx, input, c.NATS, accountID)
}),
"DeleteNatGateway": ec2Handler(func(ctx context.Context, input *ec2.DeleteNatGatewayInput, c *call, accountID string) (any, error) {
return ec2natgwapi.DeleteNatGateway(ctx, input, c.NATS, accountID)
}),
"DescribeNatGateways": ec2Handler(func(ctx context.Context, input *ec2.DescribeNatGatewaysInput, c *call, accountID string) (any, error) {
return ec2natgwapi.DescribeNatGateways(ctx, input, c.NATS, accountID)
}),
}
// defaultVPCID returns the ID of accountID's default VPC, or "" when it has
// none, as the account's own DescribeVpcs sees it.
func defaultVPCID(ctx context.Context, natsConn *nats.Conn, accountID string) (string, error) {
if natsConn == nil {
return "", errors.New(awserrors.ErrorServerInternal)
}
out, err := ec2vpcapi.DescribeVpcs(ctx, &ec2.DescribeVpcsInput{
Filters: []*ec2.Filter{{Name: aws.String("is-default"), Values: aws.StringSlice([]string{"true"})}},
}, natsConn, accountID)
if err != nil {
return "", err
}
if len(out.Vpcs) == 0 {
return "", nil
}
return aws.StringValue(out.Vpcs[0].VpcId), nil
}
// ec2LocalActions are actions that don't require NATS. DescribeAccountAttributes
// needs it only for default-vpc.
var ec2LocalActions = map[string]bool{
"DescribeRegions": true,
"DescribeAvailabilityZones": true,
"DescribeAccountAttributes": true,
}
// dryRunRequested reports whether a parsed EC2 input carries DryRun=true. Only
// actions whose SDK input models the parameter can answer DryRunOperation.
func dryRunRequested(input any) bool {
v := reflect.ValueOf(input)
if v.Kind() != reflect.Pointer || v.IsNil() || v.Elem().Kind() != reflect.Struct {
return false
}
field := v.Elem().FieldByName("DryRun")
if !field.IsValid() {
return false
}
dryRun, ok := reflect.TypeAssert[*bool](field)
return ok && aws.BoolValue(dryRun)
}
@@ -0,0 +1,32 @@
//test:in-package — reads the unexported ec2Actions route table.
package awsapi
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestImportKeyPair_Base64PaddingWorkaround(t *testing.T) {
// The ImportKeyPair handler decodes URL-encoded Base64 padding (%3D%3D → ==)
// before passing to the generic handler.
handler := ec2Actions["ImportKeyPair"]
require.NotNil(t, handler)
q := map[string]string{
"Action": "ImportKeyPair",
"KeyName": "test-key",
"PublicKeyMaterial": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCZ1FD%3D%3D",
}
_, err := handler.parse(q)
require.NoError(t, err)
assert.True(t, strings.HasSuffix(q["PublicKeyMaterial"], "=="),
"Expected PublicKeyMaterial to end with == but got: %s", q["PublicKeyMaterial"])
assert.NotContains(t, q["PublicKeyMaterial"], "%3D",
"Expected no URL-encoded padding remaining")
}
@@ -1,12 +1,11 @@
//test:in-package — ec2Actions is unexported here, and the completeness test
// exists to compare it against the scope table.
//test:in-package — ec2Actions is unexported, and the completeness test exists
// to compare it against the scope table.
package gateway
package awsapi
import (
"testing"
ec2awsapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi"
"github.com/stretchr/testify/assert"
)
@@ -15,11 +14,11 @@ import (
// behind by a deleted or renamed action fails too.
func TestEC2ScopeTableIsExhaustive(t *testing.T) {
for action := range ec2Actions {
assert.True(t, ec2awsapi.HasScope(action),
assert.True(t, HasScope(action),
"ec2 action %q has no resource scope entry: add one to ec2Scopes in domains/ec2/awsapi/authz.go", action)
}
for _, action := range ec2awsapi.ScopedActions() {
for _, action := range ScopedActions() {
_, ok := ec2Actions[action]
assert.True(t, ok,
"ec2Scopes has an entry for %q, which the dispatch table does not serve: remove it from domains/ec2/awsapi/authz.go", action)
+65
View File
@@ -0,0 +1,65 @@
package awsapi
import (
"context"
"sync"
"github.com/aws/aws-sdk-go/service/ec2"
ec2instanceapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/instance"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// Quota is the account quota the EC2 actions consult and charge. It is declared
// here because domains/admission/quota imports the EC2 action packages to count
// live resources, and its retype gate takes a resolver type this package must
// not name.
type Quota interface {
EnforceLaunch(ctx context.Context, accountID, instanceType string, maxCount int) error
ChargeLaunch(ctx context.Context, accountID string, reservation *ec2.Reservation) error
// EnforceRetype returns the vCPU delta AddVCPU charges once the retype applies.
EnforceRetype(ctx context.Context, nc *nats.Conn, nodeIDs func() []string, accountID, instanceID, newType string) (int, error)
AddVCPU(ctx context.Context, accountID string, delta int) error
EnforceVolumeCreate(ctx context.Context, nc *nats.Conn, accountID string, requestedGiB int) error
EnforceVolumeModify(ctx context.Context, nc *nats.Conn, accountID, volumeID string, newGiB int) error
EnforceEIPs(ctx context.Context, nc *nats.Conn, accountID string, want int) error
EnforceSubnets(ctx context.Context, nc *nats.Conn, accountID string, want int) error
EnforceVPCs(ctx context.Context, nc *nats.Conn, accountID string, want int) error
}
// Deps are the composed capabilities the EC2 control plane dispatches to.
type Deps struct {
NATS *nats.Conn
IAM ec2instanceapi.InstanceProfiles
Quota Quota
// ActiveNodes is the number of daemons currently answering.
ActiveNodes func(ctx context.Context) int
ExpectedNodes int
NodeIDs []string
AZ string
// Endpoint is the gateway's own dialable base URL DescribeRegions advertises.
Endpoint string
InstanceStatus ec2instanceapi.StatusSynthesis
}
// Handler serves EC2 requests over its Deps. It binds the client-token store on
// first use and is safe for concurrent requests.
type Handler struct {
Deps
tokenOnce sync.Once
tokenKV jetstream.KeyValue
tokenErr error
}
// NewHandler returns the handler over deps.
func NewHandler(deps Deps) *Handler { return &Handler{Deps: deps} }
// call is one request's view of the handler: the handler's capabilities plus the
// region and the PassRole gate of the authenticated caller.
type call struct {
*Handler
Region string
passRole func(roleARN string) error
}
@@ -0,0 +1,25 @@
//test:in-package — drives the unexported action table without a request.
package awsapi
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// DescribeRegions must advertise the endpoint the handler was composed with, so
// a caller on a concrete-host deployment is never told to dial its own loopback.
func TestDescribeRegions_UsesAdvertisedEndpoint(t *testing.T) {
c := &call{Handler: NewHandler(Deps{Endpoint: "https://10.0.0.5:9999"}), Region: "ap-southeast-2"}
h := ec2Actions["DescribeRegions"]
input, err := h.parse(map[string]string{})
require.NoError(t, err)
xmlOutput, err := h.dispatch(t.Context(), "DescribeRegions", input, c, "acct-123")
require.NoError(t, err)
body := string(xmlOutput)
assert.Contains(t, body, "<regionEndpoint>https://10.0.0.5:9999</regionEndpoint>")
assert.NotContains(t, body, "localhost")
}
@@ -0,0 +1,172 @@
//test:in-package — drives the unexported action table through Dispatch.
package awsapi
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/bluebottle/pkg/iampolicy"
"github.com/mulgadc/spinifex/internal/testkit"
iamdomain "github.com/mulgadc/spinifex/spinifex/domains/iam"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/arn"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
"github.com/mulgadc/spinifex/spinifex/ingress/aws/dispatch"
"github.com/nats-io/nats.go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type allowQuota struct{}
var _ Quota = allowQuota{}
func (allowQuota) EnforceLaunch(context.Context, string, string, int) error { return nil }
func (allowQuota) ChargeLaunch(context.Context, string, *ec2.Reservation) error { return nil }
func (allowQuota) EnforceRetype(context.Context, *nats.Conn, func() []string, string, string, string) (int, error) {
return 0, nil
}
func (allowQuota) AddVCPU(context.Context, string, int) error { return nil }
func (allowQuota) EnforceVolumeCreate(context.Context, *nats.Conn, string, int) error { return nil }
func (allowQuota) EnforceVolumeModify(context.Context, *nats.Conn, string, string, int) error {
return nil
}
func (allowQuota) EnforceEIPs(context.Context, *nats.Conn, string, int) error { return nil }
func (allowQuota) EnforceSubnets(context.Context, *nats.Conn, string, int) error { return nil }
func (allowQuota) EnforceVPCs(context.Context, *nats.Conn, string, int) error { return nil }
type noProfiles struct{}
func (noProfiles) ResolveInstanceProfile(string, string) (*iamdomain.InstanceProfile, error) {
return nil, errors.New(awserrors.ErrorIAMNoSuchEntity)
}
func (noProfiles) CanonicalResourceARN(string, arn.IAMResourceType, string) (string, error) {
return "", errors.New(awserrors.ErrorIAMNoSuchEntity)
}
func allowAll(string, string, []string, iampolicy.ConditionKeys) error { return nil }
func formRequest(body string) *http.Request {
r := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(body))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
return r
}
func newTestHandler(t *testing.T) *Handler {
t.Helper()
_, nc, _ := testutil.StartTestJetStream(t)
// A daemon that answers every EC2 subject with an empty result, so each
// action reaches its domain call and returns instead of waiting.
sub, err := nc.Subscribe("ec2.>", func(m *nats.Msg) { _ = m.Respond([]byte("{}")) })
require.NoError(t, err)
t.Cleanup(func() { _ = sub.Unsubscribe() })
return NewHandler(Deps{
NATS: nc,
IAM: noProfiles{},
Quota: allowQuota{},
ActiveNodes: func(context.Context) int { return 1 },
ExpectedNodes: 1,
NodeIDs: []string{"node-1"},
AZ: "ap-southeast-2a",
Endpoint: "https://localhost:9999",
})
}
// Every routed action must reach its handler through Dispatch. Inputs are empty,
// so the domain call may reject them, but the wrapper must run and Dispatch must
// answer with a result or an error rather than panic.
func TestDispatch_EveryActionReachesItsHandler(t *testing.T) {
h := newTestHandler(t)
for _, action := range ActionNames() {
t.Run(action, func(t *testing.T) {
w := httptest.NewRecorder()
inv := dispatch.Invocation{
Request: formRequest("Action=" + action),
AccountID: testAccountID,
Service: ServiceName,
Region: testRegion,
Authorize: allowAll,
}
assert.NotPanics(t, func() { _ = h.Dispatch(w, inv) })
})
}
}
func TestDispatch_RequestErrors(t *testing.T) {
h := newTestHandler(t)
deny := func(string, string, []string, iampolicy.ConditionKeys) error {
return errors.New(awserrors.ErrorAccessDenied)
}
cases := []struct {
name string
body string
accountID string
authorize dispatch.AuthorizeFunc
want string
}{
{"missing action", "Foo=bar", testAccountID, allowAll, awserrors.ErrorMissingAction},
{"unknown action", "Action=NoSuchAction", testAccountID, allowAll, awserrors.ErrorInvalidAction},
{"no account", "Action=DescribeVpcs", "", allowAll, awserrors.ErrorInternalError},
{"policy denial", "Action=DescribeVpcs", testAccountID, deny, awserrors.ErrorUnauthorizedOperation},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := h.Dispatch(httptest.NewRecorder(), dispatch.Invocation{
Request: formRequest(tc.body), AccountID: tc.accountID, Service: ServiceName,
Region: testRegion, Authorize: tc.authorize,
})
require.Error(t, err)
assert.Equal(t, tc.want, err.Error())
})
}
}
func TestDispatch_SuccessWritesXML(t *testing.T) {
h := newTestHandler(t)
w := httptest.NewRecorder()
err := h.Dispatch(w, dispatch.Invocation{
Request: formRequest("Action=DescribeRegions"), AccountID: testAccountID,
Service: ServiceName, Region: testRegion, Authorize: allowAll,
})
require.NoError(t, err)
assert.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, "text/xml", w.Header().Get("Content-Type"))
assert.Contains(t, w.Body.String(), "ap-southeast-2")
}
// Without a NATS connection the gateway-local actions still answer and every
// other action reports ServerInternal.
func TestDispatch_NoNATS(t *testing.T) {
h := NewHandler(Deps{AZ: "ap-southeast-2a", Endpoint: "https://localhost:9999"})
inv := func(body string) dispatch.Invocation {
return dispatch.Invocation{
Request: formRequest(body), AccountID: testAccountID, Service: ServiceName,
Region: testRegion, Authorize: allowAll,
}
}
require.NoError(t, h.Dispatch(httptest.NewRecorder(), inv("Action=DescribeRegions")))
err := h.Dispatch(httptest.NewRecorder(), inv("Action=DescribeVpcs"))
require.Error(t, err)
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
}
func TestInputHasField(t *testing.T) {
assert.True(t, InputHasField("CreateVpc", "DryRun"))
assert.False(t, InputHasField("CreateVpc", "NoSuchField"))
assert.False(t, InputHasField("NoSuchAction", "DryRun"))
}
@@ -1,4 +1,4 @@
package gateway
package awsapi
import (
"context"
@@ -13,17 +13,17 @@ import (
)
// render drives a single EC2 action through ec2Handler exactly as the real
// dispatch table does (see ec2Actions in ec2.go), with a stub inner handler
// dispatch table does (see ec2Actions in actions.go), with a stub inner handler
// standing in for the NATS round trip. This exercises the real marshal path
// (currently marshalEC2Response) without needing a live NATS connection.
func render[In any](t *testing.T, action string, out any) string {
t.Helper()
h := ec2Handler(func(_ context.Context, _ *In, _ *GatewayConfig, _ string) (any, error) {
h := ec2Handler(func(_ context.Context, _ *In, _ *call, _ string) (any, error) {
return out, nil
})
input, err := h.parse(map[string]string{})
require.NoError(t, err)
xmlOutput, err := h.dispatch(action, input, &GatewayConfig{}, "acct-123", nil)
xmlOutput, err := h.dispatch(t.Context(), action, input, &call{Handler: NewHandler(Deps{})}, "acct-123")
require.NoError(t, err)
return string(xmlOutput)
}
@@ -1,11 +1,10 @@
package gateway
package awsapi
import (
"context"
"errors"
"fmt"
"log/slog"
"net/http"
ec2idemapi "github.com/mulgadc/spinifex/spinifex/domains/ec2/awsapi/idem"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
@@ -13,40 +12,39 @@ import (
"github.com/nats-io/nats.go/jetstream"
)
// ec2ClientTokenBucket lazily binds the EC2 token bucket, once per gateway
// clientTokenBucket lazily binds the EC2 token bucket, once per handler
// rather than once per request.
func (gw *GatewayConfig) ec2ClientTokenBucket(ctx context.Context) (jetstream.KeyValue, error) {
gw.ec2TokenOnce.Do(func() {
js, err := jetstream.New(gw.NATSConn)
func (h *Handler) clientTokenBucket(ctx context.Context) (jetstream.KeyValue, error) {
h.tokenOnce.Do(func() {
js, err := jetstream.New(h.NATS)
if err != nil {
gw.ec2TokenErr = fmt.Errorf("clienttoken jetstream: %w", err)
h.tokenErr = fmt.Errorf("clienttoken jetstream: %w", err)
return
}
// The bind happens once, so it must not inherit the first caller's
// cancellation: a client that disconnects mid-open would poison the
// bucket for every later request. Deadline-free, so the open falls back
// to the JetStream API's own timeout.
gw.ec2TokenKV, gw.ec2TokenErr = idempotency.OpenBucket(
h.tokenKV, h.tokenErr = idempotency.OpenBucket(
context.WithoutCancel(ctx), js, ec2idemapi.KVBucket, ec2idemapi.TTL)
})
return gw.ec2TokenKV, gw.ec2TokenErr
return h.tokenKV, h.tokenErr
}
// ec2IdempotentHandler is ec2Handler for creates that honour ClientToken: a
// repeat of the same token returns the first call's result instead of creating
// a second resource. Out is pinned so the result can be stored and replayed.
func ec2IdempotentHandler[In, Out any](handler func(ctx context.Context, input *In, gw *GatewayConfig, accountID string) (Out, error)) ec2Action {
func ec2IdempotentHandler[In, Out any](handler func(ctx context.Context, input *In, c *call, accountID string) (Out, error)) ec2Action {
return ec2Action{
idempotent: true,
parse: parseEC2Input[In],
dispatch: func(action string, parsed any, gw *GatewayConfig, accountID string, r *http.Request) ([]byte, error) {
dispatch: func(ctx context.Context, action string, parsed any, c *call, accountID string) ([]byte, error) {
input, ok := parsed.(*In)
if !ok {
return nil, errors.New(awserrors.ErrorInternalError)
}
ctx := requestContext(r)
output, err := runEC2Idempotent(ctx, action, gw, accountID, input, func() (Out, error) {
return handler(ctx, input, gw, accountID)
output, err := runEC2Idempotent(ctx, action, c, accountID, input, func() (Out, error) {
return handler(ctx, input, c, accountID)
})
if err != nil {
return nil, err
@@ -61,7 +59,7 @@ func ec2IdempotentHandler[In, Out any](handler func(ctx context.Context, input *
func runEC2Idempotent[In, Out any](
ctx context.Context,
action string,
gw *GatewayConfig,
c *call,
accountID string,
input *In,
work func() (Out, error),
@@ -74,7 +72,7 @@ func runEC2Idempotent[In, Out any](
return work()
}
kv, kerr := gw.ec2ClientTokenBucket(ctx)
kv, kerr := c.clientTokenBucket(ctx)
if kerr != nil {
// Running anyway would create the duplicate the token was sent to
// prevent, so this fails rather than degrading to no idempotency.
@@ -1,8 +1,7 @@
//test:in-package — the decorator, the action table and the gateway's token
//bucket field are all unexported, and the point of these tests is that the
//table's entries are wired to the decorator.
//test:in-package — the decorator and the action table are unexported, and the
// point of these tests is that the table entries are wired to the decorator.
package gateway
package awsapi
import (
"bytes"
@@ -60,32 +59,32 @@ func responseFields(t *testing.T, doc []byte) []string {
return fields
}
// newTokenGateway returns a gateway whose client-token bucket is already bound
// newTokenCall returns a gateway whose client-token bucket is already bound
// to a test JetStream, so dispatch never reaches a real NATS connection.
func newTokenGateway(t *testing.T) *GatewayConfig {
func newTokenCall(t *testing.T) *call {
t.Helper()
_, nc, _ := testutil.StartTestJetStream(t)
kv, err := idempotency.OpenBucket(t.Context(), testutil.NewJetStream(t, nc), "gw-ec2-tokens", time.Minute)
require.NoError(t, err)
gw := &GatewayConfig{}
// Burns the Once as it binds, so ec2ClientTokenBucket returns this bucket
h := NewHandler(Deps{})
// Burns the Once as it binds, so clientTokenBucket returns this bucket
// rather than dialing.
gw.ec2TokenOnce.Do(func() { gw.ec2TokenKV = kv })
return gw
h.tokenOnce.Do(func() { h.tokenKV = kv })
return &call{Handler: h}
}
// dispatchTwice runs one action twice through the decorator with the same input
// and reports how many times the create actually ran.
func dispatchTwice[In, Out any](t *testing.T, gw *GatewayConfig, action string, input *In, result Out) (calls int, first, second []byte) {
func dispatchTwice[In, Out any](t *testing.T, c *call, action string, input *In, result Out) (calls int, first, second []byte) {
t.Helper()
handler := ec2IdempotentHandler(func(ctx context.Context, in *In, gw *GatewayConfig, accountID string) (Out, error) {
handler := ec2IdempotentHandler(func(ctx context.Context, in *In, c *call, accountID string) (Out, error) {
calls++
return result, nil
})
first, err := handler.dispatch(action, input, gw, idemTestAccount, nil)
first, err := handler.dispatch(t.Context(), action, input, c, idemTestAccount)
require.NoError(t, err)
second, err = handler.dispatch(action, input, gw, idemTestAccount, nil)
second, err = handler.dispatch(t.Context(), action, input, c, idemTestAccount)
require.NoError(t, err)
return calls, first, second
}
@@ -98,8 +97,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
t.Run("CreateVolume", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateVolume",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateVolume",
&ec2.CreateVolumeInput{Size: aws.Int64(8), ClientToken: tok},
ec2.Volume{VolumeId: aws.String("vol-1")})
assert.Equal(t, 1, calls)
@@ -109,8 +108,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
t.Run("CreateNetworkInterface", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateNetworkInterface",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateNetworkInterface",
&ec2.CreateNetworkInterfaceInput{ClientToken: tok},
ec2.CreateNetworkInterfaceOutput{
NetworkInterface: &ec2.NetworkInterface{NetworkInterfaceId: aws.String("eni-1")},
@@ -123,8 +122,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
// Both IDs have to survive the replay: re-creating either one leaks it.
t.Run("CreateNatGateway", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateNatGateway",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateNatGateway",
&ec2.CreateNatGatewayInput{ClientToken: tok},
ec2.CreateNatGatewayOutput{
NatGateway: &ec2.NatGateway{
@@ -142,8 +141,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
t.Run("CreateRouteTable", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateRouteTable",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateRouteTable",
&ec2.CreateRouteTableInput{ClientToken: tok},
ec2.CreateRouteTableOutput{
RouteTable: &ec2.RouteTable{
@@ -161,8 +160,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
t.Run("CreateEgressOnlyInternetGateway", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateEgressOnlyInternetGateway",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateEgressOnlyInternetGateway",
&ec2.CreateEgressOnlyInternetGatewayInput{ClientToken: tok},
ec2.CreateEgressOnlyInternetGatewayOutput{
EgressOnlyInternetGateway: &ec2.EgressOnlyInternetGateway{
@@ -176,8 +175,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
t.Run("CopyImage", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CopyImage",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CopyImage",
&ec2.CopyImageInput{Name: aws.String("copy"), ClientToken: tok},
ec2.CopyImageOutput{ImageId: aws.String("ami-1")})
assert.Equal(t, 1, calls)
@@ -189,8 +188,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
// client that thought it made one call.
t.Run("CreateLaunchTemplateVersion", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateLaunchTemplateVersion",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateLaunchTemplateVersion",
&ec2.CreateLaunchTemplateVersionInput{LaunchTemplateId: aws.String("lt-1"), ClientToken: tok},
ec2.CreateLaunchTemplateVersionOutput{
LaunchTemplateVersion: &ec2.LaunchTemplateVersion{
@@ -205,8 +204,8 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
t.Run("CreateCapacityReservation", func(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, first, second := dispatchTwice(t, gw, "CreateCapacityReservation",
c := newTokenCall(t)
calls, first, second := dispatchTwice(t, c, "CreateCapacityReservation",
&ec2.CreateCapacityReservationInput{ClientToken: tok},
ec2.CreateCapacityReservationOutput{
CapacityReservation: &ec2.CapacityReservation{
@@ -223,14 +222,14 @@ func TestIdempotentHandler_SameTokenCreatesOneResource(t *testing.T) {
// separate pieces of work rather than one replaying the other's response.
func TestIdempotentHandler_TokensAreScopedPerAction(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
c := newTokenCall(t)
tok := aws.String("shared-token")
volumeCalls, _, _ := dispatchTwice(t, gw, "CreateVolume",
volumeCalls, _, _ := dispatchTwice(t, c, "CreateVolume",
&ec2.CreateVolumeInput{ClientToken: tok}, ec2.Volume{VolumeId: aws.String("vol-1")})
require.Equal(t, 1, volumeCalls)
eigwCalls, _, second := dispatchTwice(t, gw, "CreateEgressOnlyInternetGateway",
eigwCalls, _, second := dispatchTwice(t, c, "CreateEgressOnlyInternetGateway",
&ec2.CreateEgressOnlyInternetGatewayInput{ClientToken: tok},
ec2.CreateEgressOnlyInternetGatewayOutput{
EgressOnlyInternetGateway: &ec2.EgressOnlyInternetGateway{
@@ -246,19 +245,19 @@ func TestIdempotentHandler_TokensAreScopedPerAction(t *testing.T) {
// must not run.
func TestIdempotentHandler_ParamMismatchIsRejected(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
c := newTokenCall(t)
calls := 0
handler := ec2IdempotentHandler(func(ctx context.Context, in *ec2.CreateVolumeInput, gw *GatewayConfig, accountID string) (ec2.Volume, error) {
handler := ec2IdempotentHandler(func(ctx context.Context, in *ec2.CreateVolumeInput, c *call, accountID string) (ec2.Volume, error) {
calls++
return ec2.Volume{VolumeId: aws.String("vol-1")}, nil
})
_, err := handler.dispatch("CreateVolume",
&ec2.CreateVolumeInput{Size: aws.Int64(8), ClientToken: aws.String("tok-mm")}, gw, idemTestAccount, nil)
_, err := handler.dispatch(t.Context(), "CreateVolume",
&ec2.CreateVolumeInput{Size: aws.Int64(8), ClientToken: aws.String("tok-mm")}, c, idemTestAccount)
require.NoError(t, err)
_, err = handler.dispatch("CreateVolume",
&ec2.CreateVolumeInput{Size: aws.Int64(16), ClientToken: aws.String("tok-mm")}, gw, idemTestAccount, nil)
_, err = handler.dispatch(t.Context(), "CreateVolume",
&ec2.CreateVolumeInput{Size: aws.Int64(16), ClientToken: aws.String("tok-mm")}, c, idemTestAccount)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorIdempotentParameterMismatch, err.Error())
assert.Equal(t, 1, calls, "a mismatched token must not create anything")
@@ -267,8 +266,8 @@ func TestIdempotentHandler_ParamMismatchIsRejected(t *testing.T) {
// Without a token the client did not ask for idempotency, so each call creates.
func TestIdempotentHandler_NoTokenCreatesEachTime(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
calls, _, _ := dispatchTwice(t, gw, "CreateVolume",
c := newTokenCall(t)
calls, _, _ := dispatchTwice(t, c, "CreateVolume",
&ec2.CreateVolumeInput{Size: aws.Int64(8)}, ec2.Volume{VolumeId: aws.String("vol-1")})
assert.Equal(t, 2, calls)
}
@@ -277,10 +276,10 @@ func TestIdempotentHandler_NoTokenCreatesEachTime(t *testing.T) {
// again rather than replay the failure.
func TestIdempotentHandler_FailedCreateIsRetryable(t *testing.T) {
t.Parallel()
gw := newTokenGateway(t)
c := newTokenCall(t)
boom := errors.New("create failed")
calls := 0
handler := ec2IdempotentHandler(func(ctx context.Context, in *ec2.CreateVolumeInput, gw *GatewayConfig, accountID string) (ec2.Volume, error) {
handler := ec2IdempotentHandler(func(ctx context.Context, in *ec2.CreateVolumeInput, c *call, accountID string) (ec2.Volume, error) {
calls++
if calls == 1 {
return ec2.Volume{}, boom
@@ -289,10 +288,10 @@ func TestIdempotentHandler_FailedCreateIsRetryable(t *testing.T) {
})
input := &ec2.CreateVolumeInput{Size: aws.Int64(8), ClientToken: aws.String("tok-retry")}
_, err := handler.dispatch("CreateVolume", input, gw, idemTestAccount, nil)
_, err := handler.dispatch(t.Context(), "CreateVolume", input, c, idemTestAccount)
require.ErrorIs(t, err, boom, "the handler's own error reaches the caller unchanged")
out, err := handler.dispatch("CreateVolume", input, gw, idemTestAccount, nil)
out, err := handler.dispatch(t.Context(), "CreateVolume", input, c, idemTestAccount)
require.NoError(t, err)
assert.Equal(t, 2, calls)
assert.Contains(t, string(out), "vol-1")
@@ -2,9 +2,9 @@ package image
import (
"context"
"errors"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
ec2image "github.com/mulgadc/spinifex/spinifex/domains/ec2/image"
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
@@ -27,74 +27,86 @@ var validRegisterImageBootModes = map[string]bool{
}
// ValidateRegisterImageInput rejects inputs that ask for behaviour spinifex
// doesn't have (PV, S3 bundles, kernel/ramdisk, IMDS/TPM/ENA hints) rather
// than silently discarding them. Registration is pointer-only: caller supplies
// a snapshot and we write a config.json that references it.
// doesn't have (PV, S3 bundles, kernel/ramdisk, IMDS/TPM hints) rather than
// silently discarding them. Registration is pointer-only: caller supplies a
// snapshot and we write a config.json that references it.
func ValidateRegisterImageInput(input *ec2.RegisterImageInput) error {
if input == nil {
return errors.New(awserrors.ErrorMissingParameter)
return awserrors.Errorf(awserrors.ErrorMissingParameter, "The request must contain the parameter Name")
}
if input.Name == nil || *input.Name == "" {
return errors.New(awserrors.ErrorMissingParameter)
return missingParam("Name")
}
if n := len(*input.Name); n < 3 || n > 128 {
return errors.New(awserrors.ErrorInvalidAMINameMalformed)
return invalidParam(awserrors.ErrorInvalidAMINameMalformed, "Name", *input.Name, "AMI names must be between 3 and 128 characters long.")
}
if input.ImageLocation != nil && *input.ImageLocation != "" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if aws.StringValue(input.ImageLocation) != "" {
return unsupportedParam("ImageLocation", *input.ImageLocation)
}
if input.Architecture != nil && *input.Architecture != "" {
if !validRegisterImageArchitectures[*input.Architecture] {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
if arch := aws.StringValue(input.Architecture); arch != "" && !validRegisterImageArchitectures[arch] {
return invalidParam(awserrors.ErrorInvalidParameterValue, "Architecture", arch, "Supported values are x86_64, arm64 and i386.")
}
if input.VirtualizationType != nil && *input.VirtualizationType != "" && *input.VirtualizationType != "hvm" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if virt := aws.StringValue(input.VirtualizationType); virt != "" && virt != "hvm" {
return invalidParam(awserrors.ErrorInvalidParameterValue, "VirtualizationType", virt, "Only hvm is supported.")
}
if input.BootMode != nil && *input.BootMode != "" {
if !validRegisterImageBootModes[*input.BootMode] {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
if mode := aws.StringValue(input.BootMode); mode != "" && !validRegisterImageBootModes[mode] {
return invalidParam(awserrors.ErrorInvalidParameterValue, "BootMode", mode, "Supported values are bios, uefi and uefi-preferred.")
}
if input.KernelId != nil && *input.KernelId != "" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if aws.StringValue(input.KernelId) != "" {
return unsupportedParam("KernelId", *input.KernelId)
}
if input.RamdiskId != nil && *input.RamdiskId != "" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if aws.StringValue(input.RamdiskId) != "" {
return unsupportedParam("RamdiskId", *input.RamdiskId)
}
if input.TpmSupport != nil && *input.TpmSupport != "" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if aws.StringValue(input.TpmSupport) != "" {
return unsupportedParam("TpmSupport", *input.TpmSupport)
}
if input.ImdsSupport != nil && *input.ImdsSupport != "" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if aws.StringValue(input.ImdsSupport) != "" {
return unsupportedParam("ImdsSupport", *input.ImdsSupport)
}
if input.EnaSupport != nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
if input.SriovNetSupport != nil && *input.SriovNetSupport != "" {
return errors.New(awserrors.ErrorInvalidParameterValue)
if aws.StringValue(input.SriovNetSupport) != "" {
return unsupportedParam("SriovNetSupport", *input.SriovNetSupport)
}
if len(input.BlockDeviceMappings) == 0 {
return errors.New(awserrors.ErrorMissingParameter)
return missingParam("BlockDeviceMapping")
}
root := selectRootBlockDeviceMapping(input.BlockDeviceMappings, input.RootDeviceName)
if root == nil || root.Ebs == nil || root.Ebs.SnapshotId == nil || *root.Ebs.SnapshotId == "" {
return errors.New(awserrors.ErrorMissingParameter)
if name := aws.StringValue(input.RootDeviceName); name != "" && root == nil {
return awserrors.Errorf(awserrors.ErrorMissingParameter,
"Value (%s) for parameter RootDeviceName is invalid. No block device mapping has that DeviceName.", name)
}
return missingParam("SnapshotId in the root block device mapping")
}
if !strings.HasPrefix(*root.Ebs.SnapshotId, "snap-") {
return errors.New(awserrors.ErrorInvalidSnapshotIDMalformed)
return invalidParam(awserrors.ErrorInvalidSnapshotIDMalformed, "SnapshotId", *root.Ebs.SnapshotId, "Expected: 'snap-...'.")
}
return nil
}
// missingParam is EC2's wording for an absent required parameter.
func missingParam(name string) error {
return awserrors.Errorf(awserrors.ErrorMissingParameter, "The request must contain the parameter %s", name)
}
// invalidParam is EC2's wording for a parameter value it refuses.
func invalidParam(code, name, value, reason string) error {
return awserrors.Errorf(code, "Value (%s) for parameter %s is invalid. %s", value, name, reason)
}
// unsupportedParam refuses a parameter spinifex has no behaviour for.
func unsupportedParam(name, value string) error {
return invalidParam(awserrors.ErrorInvalidParameterValue, name, value, name+" is not supported.")
}
// selectRootBlockDeviceMapping returns the BDM matching RootDeviceName if set,
// else the first BDM carrying an EBS snapshot reference.
func selectRootBlockDeviceMapping(mappings []*ec2.BlockDeviceMapping, rootDeviceName *string) *ec2.BlockDeviceMapping {

Some files were not shown because too many files have changed in this diff Show More